ruvector/examples/data/framework/docs/PHYSICS_CLIENTS.md
rUv 38d93a6e8d feat: Add comprehensive dataset discovery framework for RuVector (#104)
* feat: Add comprehensive dataset discovery framework for RuVector

This commit introduces a powerful dataset discovery framework with
integrations for three high-impact public data sources:

## Core Framework (examples/data/framework/)
- DataIngester: Streaming ingestion with batching and deduplication
- CoherenceEngine: Min-cut based coherence signal computation
- DiscoveryEngine: Pattern detection for emerging structures

## OpenAlex Integration (examples/data/openalex/)
- Research frontier radar: Detect emerging fields via boundary motion
- Cross-domain bridge detection: Find connector subgraphs
- Topic graph construction from citation networks
- Full API client with cursor-based pagination

## Climate Integration (examples/data/climate/)
- NOAA GHCN and NASA Earthdata clients
- Sensor network graph construction
- Regime shift detection using min-cut coherence breaks
- Time series vectorization for similarity search
- Seasonal decomposition analysis

## SEC EDGAR Integration (examples/data/edgar/)
- XBRL financial statement parsing
- Peer network construction
- Coherence watch: Detect fundamental vs narrative divergence
- Filing analysis with sentiment and risk extraction
- Cross-company contagion detection

Each integration leverages RuVector's unique capabilities:
- Vector memory for semantic similarity
- Graph structures for relationship modeling
- Dynamic min-cut for coherence signal computation
- Time series embeddings for pattern matching

Discovery thesis: Detect emerging patterns before they have names,
find non-obvious cross-domain bridges, and map causality chains.

* feat: Add working discovery examples for climate and financial data

- Fix borrow checker issues in coherence analysis modules
- Create standalone workspace for data examples
- Add regime_detector.rs for climate network coherence analysis
- Add coherence_watch.rs for SEC EDGAR narrative-fundamental divergence
- Add frontier_radar.rs template for OpenAlex research discovery
- Update Cargo.toml dependencies for example executability
- Add rand dev-dependency for demo data generation

Examples successfully detect:
- Climate regime shifts via min-cut coherence analysis
- Cross-regional teleconnection patterns
- Fundamental vs narrative divergence in SEC filings
- Sector fragmentation signals in financial data

* feat: Add working discovery examples for climate and financial data

- Add RuVector-native discovery engine with Stoer-Wagner min-cut
- Implement cross-domain pattern detection (climate ↔ finance)
- Add cosine similarity for vector-based semantic matching
- Create cross_domain_discovery example demonstrating:
  - 42% cross-domain edge connectivity
  - Bridge formation detection with 0.73-0.76 confidence
  - Climate and finance correlation hypothesis generation

* perf: Add optimized discovery engine with SIMD and parallel processing

Performance improvements:
- 8.84x speedup for vector insertion via parallel batching
- 2.91x SIMD speedup for cosine similarity (chunked + AVX2)
- Incremental graph updates with adjacency caching
- Early termination in Stoer-Wagner min-cut

Statistical analysis features:
- P-value computation for pattern significance
- Effect size (Cohen's d) calculation
- 95% confidence intervals
- Granger-style temporal causality detection

Benchmark results (248 vectors, 3 domains):
- Cross-domain edges: 34.9% of total graph
- Domain coherence: Climate 0.74, Finance 0.94, Research 0.97
- Detected climate-finance temporal correlations

* feat: Add discovery hunter and comprehensive README tutorial

New features:
- Discovery hunter example with multi-phase pattern detection
- Climate extremes, financial stress, and research data generation
- Cross-domain hypothesis generation
- Anomaly injection testing

Documentation:
- Detailed README with step-by-step tutorial
- API reference for OptimizedConfig and patterns
- Performance benchmarks and best practices
- Troubleshooting guide

* feat: Complete discovery framework with all features

HNSW Indexing (754 lines):
- O(log n) approximate nearest neighbor search
- Configurable M, ef_construction parameters
- Cosine, Euclidean, Manhattan distance metrics
- Batch insertion support

API Clients (888 lines):
- OpenAlex: academic works, authors, topics
- NOAA: climate observations
- SEC EDGAR: company filings
- Rate limiting and retry logic

Persistence (638 lines):
- Save/load engine state and patterns
- Gzip compression (3-10x size reduction)
- Incremental pattern appending

CLI Tool (1,109 lines):
- discover, benchmark, analyze, export commands
- Colored terminal output
- JSON and human-readable formats

Streaming (570 lines):
- Async stream processing
- Sliding and tumbling windows
- Real-time pattern detection
- Backpressure handling

Tests (30 unit tests):
- Stoer-Wagner min-cut verification
- SIMD cosine similarity accuracy
- Statistical significance
- Granger causality
- Cross-domain patterns

Benchmarks:
- CLI: 176 vectors/sec @ 2000 vectors
- SIMD: 6.82M ops/sec (2.06x speedup)
- Vector insertion: 1.61x speedup
- Total: 44.74ms for 248 vectors

* feat: Add visualization, export, forecasting, and real data discovery

Visualization (555 lines):
- ASCII graph rendering with box-drawing characters
- Domain-based ANSI coloring (Climate=blue, Finance=green, Research=yellow)
- Coherence timeline sparklines
- Pattern summary dashboard
- Domain connectivity matrix

Export (650 lines):
- GraphML export for Gephi/Cytoscape
- DOT export for Graphviz
- CSV export for patterns and coherence history
- Filtered export by domain, weight, time range
- Batch export with README generation

Forecasting (525 lines):
- Holt's double exponential smoothing for trend
- CUSUM-based regime change detection (70.67% accuracy)
- Cross-domain correlation forecasting (r=1.000)
- Prediction intervals (95% CI)
- Anomaly probability scoring

Real Data Discovery:
- Fetched 80 actual papers from OpenAlex API
- Topics: climate risk, stranded assets, carbon pricing, physical risk, transition risk
- Built coherence graph: 592 nodes, 1049 edges
- Average min-cut: 185.76 (well-connected research cluster)

* feat: Add medical, real-time, and knowledge graph data sources

New API Clients:
- PubMed E-utilities for medical literature search (NCBI)
- ClinicalTrials.gov v2 API for clinical study data
- FDA OpenFDA for drug adverse events and recalls
- Wikipedia article search and extraction
- Wikidata SPARQL queries for structured knowledge

Real-time Features:
- RSS/Atom feed parsing with deduplication
- News aggregator with multiple source support
- WebSocket and REST polling infrastructure
- Event streaming with configurable windows

Examples:
- medical_discovery: PubMed + ClinicalTrials + FDA integration
- multi_domain_discovery: Climate-health-finance triangulation
- wiki_discovery: Wikipedia/Wikidata knowledge graph
- realtime_feeds: News feed aggregation demo

Tested across 70+ unit tests with all domains integrated.

* feat: Add economic, patent, and ArXiv data source clients

New API Clients:
- FredClient: Federal Reserve economic indicators (GDP, CPI, unemployment)
- WorldBankClient: Global development indicators and climate data
- AlphaVantageClient: Stock market daily prices
- ArxivClient: Scientific preprint search with category and date filters
- UsptoPatentClient: USPTO patent search by keyword, assignee, CPC class
- EpoClient: Placeholder for European patent search

New Domain:
- Domain::Economic for economic/financial indicator data

Updated Exports:
- Domain colors and shapes for Economic in visualization and export

Examples:
- economic_discovery: FRED + World Bank integration demo
- arxiv_discovery: AI/ML/Climate paper search demo
- patent_discovery: Climate tech and AI patent search demo

All 85 tests passing. APIs tested with live endpoints.

* feat: Add Semantic Scholar, bioRxiv/medRxiv, and CrossRef research clients

New Research API Clients:
- SemanticScholarClient: Citation graph analysis, paper search, author lookup
  - Methods: search_papers, get_citations, get_references, search_by_field
  - Builds citation networks for graph analysis

- BiorxivClient: Life sciences preprints
  - Methods: search_recent, search_by_category (neuroscience, genomics, etc.)
  - Automatic conversion to Domain::Research

- MedrxivClient: Medical preprints
  - Methods: search_covid, search_clinical, search_by_date_range
  - Automatic conversion to Domain::Medical

- CrossRefClient: DOI metadata and scholarly communication
  - Methods: search_works, get_work, search_by_funder, get_citations
  - Polite pool support for better rate limits

All clients include:
- Rate limiting respecting API guidelines
- Retry logic with exponential backoff
- SemanticVector conversion with rich metadata
- Comprehensive unit tests

Examples:
- biorxiv_discovery: Fetch neuroscience and clinical research
- crossref_demo: Search publications, funders, datasets

Total: 104 tests passing, ~2,500 new lines of code

* feat: Add MCP server with STDIO/SSE transport and optimized discovery

MCP Server Implementation (mcp_server.rs):
- JSON-RPC 2.0 protocol with MCP 2024-11-05 compliance
- Dual transport: STDIO for CLI, SSE for HTTP streaming
- 22 discovery tools exposing all data sources:
  - Research: OpenAlex, ArXiv, Semantic Scholar, CrossRef, bioRxiv, medRxiv
  - Medical: PubMed, ClinicalTrials.gov, FDA
  - Economic: FRED, World Bank
  - Climate: NOAA
  - Knowledge: Wikipedia, Wikidata SPARQL
  - Discovery: Multi-source, coherence analysis, pattern detection
- Resources: discovery://patterns, discovery://graph, discovery://history
- Pre-built prompts: cross_domain_discovery, citation_analysis, trend_detection

Binary Entry Point (bin/mcp_discovery.rs):
- CLI arguments with clap
- Configurable discovery parameters
- STDIO/SSE mode selection

Optimized Discovery Runner:
- Parallel data fetching with tokio::join!
- SIMD-accelerated vector operations (1.1M comparisons/sec)
- 6-phase discovery pipeline with benchmarking
- Statistical significance testing (p-values)
- Cross-domain correlation analysis
- CSV export and hypothesis report generation

Performance Results:
- 180 vectors from 3 sources in 7.5s
- 686 edges computed in 8ms
- SIMD throughput: 1,122,216 comparisons/sec

All 106 tests passing.

* feat: Add space, genomics, and physics data source clients

Add exotic data source integrations:
- Space clients: NASA (APOD, NEO, Mars, DONKI), Exoplanet Archive, SpaceX API, TNS Astronomy
- Genomics clients: NCBI (genes, proteins, SNPs), UniProt, Ensembl, GWAS Catalog
- Physics clients: USGS Earthquakes, CERN Open Data, Argo Ocean, Materials Project

New domains: Space, Genomics, Physics, Seismic, Ocean

All 106 tests passing, SIMD benchmark: 208k comparisons/sec

* chore: Update export/visualization and output files

* docs: Add API client inventory and reference documentation

* fix: Update API clients for 2025 endpoint changes

- ArXiv: Switch from HTTP to HTTPS (export.arxiv.org)
- USPTO: Migrate to PatentSearch API v2 (search.patentsview.org)
  - Legacy API (api.patentsview.org) discontinued May 2025
  - Updated query format from POST to GET
  - Note: May require API authentication
- FRED: Require API key (mandatory as of 2025)
  - Added error handling for missing API key
  - Added response error field parsing

All tests passing, ArXiv discovery confirmed working

* feat: Implement comprehensive 2025 API client library (11,810 lines)

Add 7 new API client modules implementing 35+ data sources:

Academic APIs (1,328 lines):
- OpenAlexClient, CoreClient, EricClient, UnpaywallClient

Finance APIs (1,517 lines):
- FinnhubClient, TwelveDataClient, CoinGeckoClient, EcbClient, BlsClient

Geospatial APIs (1,250 lines):
- NominatimClient, OverpassClient, GeonamesClient, OpenElevationClient

News & Social APIs (1,606 lines):
- HackerNewsClient, GuardianClient, NewsDataClient, RedditClient

Government APIs (2,354 lines):
- CensusClient, DataGovClient, EuOpenDataClient, UkGovClient
- WorldBankGovClient, UNDataClient

AI/ML APIs (2,035 lines):
- HuggingFaceClient, OllamaClient, ReplicateClient
- TogetherAiClient, PapersWithCodeClient

Transportation APIs (1,720 lines):
- GtfsClient, MobilityDatabaseClient
- OpenRouteServiceClient, OpenChargeMapClient

All clients include:
- Async/await with tokio and reqwest
- Mock data fallback for testing without API keys
- Rate limiting with configurable delays
- SemanticVector conversion for RuVector integration
- Comprehensive unit tests (252 total tests passing)
- Full error handling with FrameworkError

* docs: Add API client documentation for new implementations

Add documentation for:
- Geospatial clients (Nominatim, Overpass, Geonames, OpenElevation)
- ML clients (HuggingFace, Ollama, Replicate, Together, PapersWithCode)
- News clients (HackerNews, Guardian, NewsData, Reddit)
- Finance clients implementation notes

* feat: Implement dynamic min-cut tracking system (SODA 2026)

Based on El-Hayek, Henzinger, Li (SODA 2026) subpolynomial dynamic min-cut algorithm.

Core Components (2,626 lines):
- dynamic_mincut.rs (1,579 lines): EulerTourTree, DynamicCutWatcher, LocalMinCutProcedure
- cut_aware_hnsw.rs (1,047 lines): CutAwareHNSW, CoherenceZones, CutGatedSearch

Key Features:
- O(log n) connectivity queries via Euler-tour trees
- n^{o(1)} update time when λ ≤ 2^{(log n)^{3/4}} (vs O(n³) Stoer-Wagner)
- Cut-gated HNSW search that respects coherence boundaries
- Real-time cut monitoring with threshold-based deep evaluation
- Thread-safe structures with Arc<RwLock>

Performance (benchmarked):
- 75x speedup over periodic recomputation
- O(1) min-cut queries vs O(n³) recompute
- ~25µs per edge update

Tests & Benchmarks:
- 36+ unit tests across both modules
- 5 benchmark suites comparing periodic vs dynamic
- Integration with existing OptimizedDiscoveryEngine

This enables real-time coherence tracking in RuVector, transforming
min-cut from an expensive periodic computation to a maintained invariant.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 14:36:41 -05:00

11 KiB

Physics, Seismic, and Ocean Data Clients

Overview

This module provides async API clients for physics, seismic, and ocean data sources, enabling cross-disciplinary discoveries through RuVector's semantic vector search and graph coherence analysis.

New Domains

Three new domains have been added to Domain enum in ruvector_native.rs:

  • Domain::Physics - Particle physics, materials science
  • Domain::Seismic - Earthquake data, seismic activity
  • Domain::Ocean - Ocean temperature, salinity, depth profiles

Clients

1. UsgsEarthquakeClient

USGS Earthquake Hazards Program - Real-time and historical earthquake data worldwide.

Features

  • No API key required (public data)
  • Global earthquake coverage
  • Magnitude, location, depth, tsunami warnings
  • ~5 requests/second rate limit

Methods

use ruvector_data_framework::UsgsEarthquakeClient;

let client = UsgsEarthquakeClient::new()?;

// Get recent earthquakes above minimum magnitude
let recent = client.get_recent(4.5, 7).await?; // Mag 4.5+, last 7 days

// Search by geographic region
let la_quakes = client.search_by_region(
    34.05,    // latitude
    -118.25,  // longitude
    200.0,    // radius in km
    30        // days back
).await?;

// Get significant earthquakes only
let significant = client.get_significant(30).await?;

// Filter by magnitude range
let moderate = client.get_by_magnitude_range(4.0, 6.0, 7).await?;

SemanticVector Metadata

Each earthquake is converted to a SemanticVector with:

metadata: {
    "magnitude": "5.4",
    "place": "Southern California",
    "latitude": "34.05",
    "longitude": "-118.25",
    "depth_km": "10.5",
    "tsunami": "0",
    "significance": "450",
    "status": "reviewed",
    "alert": "green",
    "source": "usgs"
}

2. CernOpenDataClient

CERN Open Data Portal - LHC experiment data, particle physics datasets.

Features

  • No API key required
  • CMS, ATLAS, LHCb, ALICE experiments
  • Collision events, particle physics data
  • Educational and research datasets

Methods

use ruvector_data_framework::CernOpenDataClient;

let client = CernOpenDataClient::new()?;

// Search datasets by keywords
let higgs = client.search_datasets("Higgs").await?;
let top_quark = client.search_datasets("top quark").await?;

// Get specific dataset by record ID
let dataset = client.get_dataset(5500).await?;

// Search by experiment
let cms_data = client.search_by_experiment("CMS").await?;
let atlas_data = client.search_by_experiment("ATLAS").await?;

Available Experiments

  • "CMS" - Compact Muon Solenoid
  • "ATLAS" - A Toroidal LHC ApparatuS
  • "LHCb" - Large Hadron Collider beauty
  • "ALICE" - A Large Ion Collider Experiment

SemanticVector Metadata

metadata: {
    "recid": "12345",
    "title": "CMS 2011 Higgs to two photons dataset",
    "experiment": "CMS",
    "collision_energy": "7TeV",
    "collision_type": "pp",
    "data_type": "Dataset",
    "source": "cern"
}

3. ArgoClient

Argo Float Ocean Data - Global ocean temperature, salinity, pressure profiles.

Features

  • Global ocean coverage (4000+ floats)
  • Temperature and salinity profiles
  • Depth measurements (0-2000m typical)
  • Free public data

Methods

use ruvector_data_framework::ArgoClient;

let client = ArgoClient::new()?;

// Get recent profiles (placeholder - requires Argo GDAC integration)
let recent = client.get_recent_profiles(30).await?;

// Search by region
let atlantic = client.search_by_region(
    0.0,     // latitude
    -30.0,   // longitude
    500.0    // radius km
).await?;

// Temperature-focused profiles
let temp_data = client.get_temperature_profiles().await?;

// Create sample data for testing
let samples = client.create_sample_profiles(50)?;

Note on Implementation

The current Argo client includes a create_sample_profiles() method for demonstration. For production use, integrate with:

SemanticVector Metadata

metadata: {
    "platform_number": "1900001",
    "latitude": "35.5",
    "longitude": "-45.2",
    "temperature": "18.3",
    "salinity": "35.1",
    "depth_m": "500.0",
    "source": "argo"
}

4. MaterialsProjectClient

Materials Project - Computational materials science database (150,000+ materials).

Features

  • Crystal structures and properties
  • Band gaps, formation energies
  • Electronic and mechanical properties
  • Requires free API key from https://materialsproject.org

Methods

use ruvector_data_framework::MaterialsProjectClient;

// API key required
let api_key = std::env::var("MATERIALS_PROJECT_API_KEY")?;
let client = MaterialsProjectClient::new(api_key)?;

// Search by chemical formula
let silicon = client.search_materials("Si").await?;
let iron_oxide = client.search_materials("Fe2O3").await?;
let battery = client.search_materials("LiFePO4").await?;

// Get specific material by ID
let mp_149 = client.get_material("mp-149").await?; // Silicon

// Search by property range
let semiconductors = client.search_by_property(
    "band_gap",
    1.0,  // min eV
    3.0   // max eV
).await?;

let stable = client.search_by_property(
    "formation_energy_per_atom",
    -2.0,  // min eV/atom
    0.0    // max eV/atom
).await?;

Common Properties

  • "band_gap" - Electronic band gap (eV)
  • "formation_energy_per_atom" - Formation energy (eV/atom)
  • "energy_per_atom" - Total energy per atom
  • "density" - Density (g/cm³)
  • "volume" - Volume per atom

SemanticVector Metadata

metadata: {
    "material_id": "mp-149",
    "formula": "Si",
    "band_gap": "1.14",
    "density": "2.33",
    "formation_energy": "0.0",
    "crystal_system": "cubic",
    "elements": "Si",
    "source": "materials_project"
}

Geographic Utilities

The GeoUtils helper provides geographic calculations:

use ruvector_data_framework::GeoUtils;

// Calculate distance between two points (Haversine formula)
let distance_km = GeoUtils::distance_km(
    40.7128, -74.0060,  // NYC
    34.0522, -118.2437  // LA
);
// Returns: ~3936 km

// Check if point is within radius
let within = GeoUtils::within_radius(
    34.05, -118.25,     // Center (LA)
    32.72, -117.16,     // Point (San Diego)
    200.0               // Radius in km
);
// Returns: true

Rate Limiting

All clients implement automatic rate limiting and retry logic:

Client Rate Limit Max Retries Retry Delay
USGS 200ms (~5 req/s) 3 1s exponential
CERN 500ms (~2 req/s) 3 1s exponential
Argo 300ms (~3 req/s) 3 1s exponential
Materials Project 1000ms (1 req/s) 3 1s exponential

Cross-Domain Discovery Examples

1. Earthquake-Climate Correlations

use ruvector_data_framework::{
    UsgsEarthquakeClient, NoaaClient,
    NativeDiscoveryEngine, NativeEngineConfig
};

let mut engine = NativeDiscoveryEngine::new(NativeEngineConfig::default());

// Add earthquake data
let usgs = UsgsEarthquakeClient::new()?;
let earthquakes = usgs.get_recent(5.0, 30).await?;
for eq in earthquakes {
    engine.add_vector(eq);
}

// Add climate data
let noaa = NoaaClient::new(None)?;
let climate = noaa.get_climate_data("GHCND:USW00023174", 30).await?;
for record in climate {
    engine.add_vector(record);
}

// Discover patterns
let patterns = engine.detect_patterns();
for pattern in patterns {
    if !pattern.cross_domain_links.is_empty() {
        println!("Found cross-domain pattern: {}", pattern.description);
    }
}

2. Materials for Particle Detectors

use ruvector_data_framework::{
    CernOpenDataClient, MaterialsProjectClient
};

let cern = CernOpenDataClient::new()?;
let materials = MaterialsProjectClient::new(api_key)?;

// Get particle physics requirements
let detector_data = cern.search_datasets("detector").await?;

// Find materials with suitable properties
let semiconductors = materials.search_by_property("band_gap", 1.0, 3.0).await?;

// Add to discovery engine to find correlations
let mut engine = NativeDiscoveryEngine::new(config);
for data in detector_data {
    engine.add_vector(data);
}
for material in semiconductors {
    engine.add_vector(material);
}

let patterns = engine.detect_patterns();

3. Ocean Temperature & Seismic Activity

use ruvector_data_framework::{
    ArgoClient, UsgsEarthquakeClient
};

let argo = ArgoClient::new()?;
let usgs = UsgsEarthquakeClient::new()?;

// Get ocean data for a region
let ocean = argo.search_by_region(0.0, -30.0, 1000.0).await?;

// Get earthquakes in same region
let quakes = usgs.search_by_region(0.0, -30.0, 1000.0, 90).await?;

// Discover correlations
let mut engine = NativeDiscoveryEngine::new(config);
for profile in ocean {
    engine.add_vector(profile);
}
for eq in quakes {
    engine.add_vector(eq);
}

// Look for cross-domain patterns
let patterns = engine.detect_patterns();
for pattern in patterns.iter().filter(|p| {
    p.cross_domain_links.iter().any(|l|
        (l.source_domain == Domain::Ocean && l.target_domain == Domain::Seismic) ||
        (l.source_domain == Domain::Seismic && l.target_domain == Domain::Ocean)
    )
}) {
    println!("Ocean-Seismic correlation: {}", pattern.description);
}

Running the Example

# Basic example (no API keys required)
cargo run --example physics_discovery

# With Materials Project API key
export MATERIALS_PROJECT_API_KEY="your_key_here"
cargo run --example physics_discovery

Integration with RuVector

All clients convert data to SemanticVector format, enabling:

  1. Vector Similarity Search - Find similar earthquakes, materials, experiments
  2. Graph Coherence Analysis - Detect network fragmentation/consolidation
  3. Cross-Domain Pattern Discovery - Bridge physics, seismic, ocean domains
  4. Temporal Analysis - Track changes over time
  5. Spatial Analysis - Geographic clustering and correlation

Testing

# Run all physics client tests
cargo test physics_clients

# Run specific client tests
cargo test usgs_client
cargo test cern_client
cargo test argo_client
cargo test materials_project_client

# Run geographic utilities tests
cargo test geo_utils

API Documentation

USGS Earthquake API

CERN Open Data Portal

Argo Data

Materials Project

Future Enhancements

  1. Full Argo GDAC Integration - Parse netCDF files directly
  2. CERN Data Caching - Local cache for large datasets
  3. USGS Historical Data - Access to complete historical catalog
  4. Materials Project Batch Queries - Optimize multi-material searches
  5. Real-time Earthquake Streaming - WebSocket for live data
  6. Ocean Current Prediction - ML models for temperature forecasting

License

Part of RuVector Data Discovery Framework. See main LICENSE file.