ruvector/examples/data/framework/docs/VISUALIZATION.md
rUv 38d93a6e8d feat: Add comprehensive dataset discovery framework for RuVector (#104)
* feat: Add comprehensive dataset discovery framework for RuVector

This commit introduces a powerful dataset discovery framework with
integrations for three high-impact public data sources:

## Core Framework (examples/data/framework/)
- DataIngester: Streaming ingestion with batching and deduplication
- CoherenceEngine: Min-cut based coherence signal computation
- DiscoveryEngine: Pattern detection for emerging structures

## OpenAlex Integration (examples/data/openalex/)
- Research frontier radar: Detect emerging fields via boundary motion
- Cross-domain bridge detection: Find connector subgraphs
- Topic graph construction from citation networks
- Full API client with cursor-based pagination

## Climate Integration (examples/data/climate/)
- NOAA GHCN and NASA Earthdata clients
- Sensor network graph construction
- Regime shift detection using min-cut coherence breaks
- Time series vectorization for similarity search
- Seasonal decomposition analysis

## SEC EDGAR Integration (examples/data/edgar/)
- XBRL financial statement parsing
- Peer network construction
- Coherence watch: Detect fundamental vs narrative divergence
- Filing analysis with sentiment and risk extraction
- Cross-company contagion detection

Each integration leverages RuVector's unique capabilities:
- Vector memory for semantic similarity
- Graph structures for relationship modeling
- Dynamic min-cut for coherence signal computation
- Time series embeddings for pattern matching

Discovery thesis: Detect emerging patterns before they have names,
find non-obvious cross-domain bridges, and map causality chains.

* feat: Add working discovery examples for climate and financial data

- Fix borrow checker issues in coherence analysis modules
- Create standalone workspace for data examples
- Add regime_detector.rs for climate network coherence analysis
- Add coherence_watch.rs for SEC EDGAR narrative-fundamental divergence
- Add frontier_radar.rs template for OpenAlex research discovery
- Update Cargo.toml dependencies for example executability
- Add rand dev-dependency for demo data generation

Examples successfully detect:
- Climate regime shifts via min-cut coherence analysis
- Cross-regional teleconnection patterns
- Fundamental vs narrative divergence in SEC filings
- Sector fragmentation signals in financial data

* feat: Add working discovery examples for climate and financial data

- Add RuVector-native discovery engine with Stoer-Wagner min-cut
- Implement cross-domain pattern detection (climate ↔ finance)
- Add cosine similarity for vector-based semantic matching
- Create cross_domain_discovery example demonstrating:
  - 42% cross-domain edge connectivity
  - Bridge formation detection with 0.73-0.76 confidence
  - Climate and finance correlation hypothesis generation

* perf: Add optimized discovery engine with SIMD and parallel processing

Performance improvements:
- 8.84x speedup for vector insertion via parallel batching
- 2.91x SIMD speedup for cosine similarity (chunked + AVX2)
- Incremental graph updates with adjacency caching
- Early termination in Stoer-Wagner min-cut

Statistical analysis features:
- P-value computation for pattern significance
- Effect size (Cohen's d) calculation
- 95% confidence intervals
- Granger-style temporal causality detection

Benchmark results (248 vectors, 3 domains):
- Cross-domain edges: 34.9% of total graph
- Domain coherence: Climate 0.74, Finance 0.94, Research 0.97
- Detected climate-finance temporal correlations

* feat: Add discovery hunter and comprehensive README tutorial

New features:
- Discovery hunter example with multi-phase pattern detection
- Climate extremes, financial stress, and research data generation
- Cross-domain hypothesis generation
- Anomaly injection testing

Documentation:
- Detailed README with step-by-step tutorial
- API reference for OptimizedConfig and patterns
- Performance benchmarks and best practices
- Troubleshooting guide

* feat: Complete discovery framework with all features

HNSW Indexing (754 lines):
- O(log n) approximate nearest neighbor search
- Configurable M, ef_construction parameters
- Cosine, Euclidean, Manhattan distance metrics
- Batch insertion support

API Clients (888 lines):
- OpenAlex: academic works, authors, topics
- NOAA: climate observations
- SEC EDGAR: company filings
- Rate limiting and retry logic

Persistence (638 lines):
- Save/load engine state and patterns
- Gzip compression (3-10x size reduction)
- Incremental pattern appending

CLI Tool (1,109 lines):
- discover, benchmark, analyze, export commands
- Colored terminal output
- JSON and human-readable formats

Streaming (570 lines):
- Async stream processing
- Sliding and tumbling windows
- Real-time pattern detection
- Backpressure handling

Tests (30 unit tests):
- Stoer-Wagner min-cut verification
- SIMD cosine similarity accuracy
- Statistical significance
- Granger causality
- Cross-domain patterns

Benchmarks:
- CLI: 176 vectors/sec @ 2000 vectors
- SIMD: 6.82M ops/sec (2.06x speedup)
- Vector insertion: 1.61x speedup
- Total: 44.74ms for 248 vectors

* feat: Add visualization, export, forecasting, and real data discovery

Visualization (555 lines):
- ASCII graph rendering with box-drawing characters
- Domain-based ANSI coloring (Climate=blue, Finance=green, Research=yellow)
- Coherence timeline sparklines
- Pattern summary dashboard
- Domain connectivity matrix

Export (650 lines):
- GraphML export for Gephi/Cytoscape
- DOT export for Graphviz
- CSV export for patterns and coherence history
- Filtered export by domain, weight, time range
- Batch export with README generation

Forecasting (525 lines):
- Holt's double exponential smoothing for trend
- CUSUM-based regime change detection (70.67% accuracy)
- Cross-domain correlation forecasting (r=1.000)
- Prediction intervals (95% CI)
- Anomaly probability scoring

Real Data Discovery:
- Fetched 80 actual papers from OpenAlex API
- Topics: climate risk, stranded assets, carbon pricing, physical risk, transition risk
- Built coherence graph: 592 nodes, 1049 edges
- Average min-cut: 185.76 (well-connected research cluster)

* feat: Add medical, real-time, and knowledge graph data sources

New API Clients:
- PubMed E-utilities for medical literature search (NCBI)
- ClinicalTrials.gov v2 API for clinical study data
- FDA OpenFDA for drug adverse events and recalls
- Wikipedia article search and extraction
- Wikidata SPARQL queries for structured knowledge

Real-time Features:
- RSS/Atom feed parsing with deduplication
- News aggregator with multiple source support
- WebSocket and REST polling infrastructure
- Event streaming with configurable windows

Examples:
- medical_discovery: PubMed + ClinicalTrials + FDA integration
- multi_domain_discovery: Climate-health-finance triangulation
- wiki_discovery: Wikipedia/Wikidata knowledge graph
- realtime_feeds: News feed aggregation demo

Tested across 70+ unit tests with all domains integrated.

* feat: Add economic, patent, and ArXiv data source clients

New API Clients:
- FredClient: Federal Reserve economic indicators (GDP, CPI, unemployment)
- WorldBankClient: Global development indicators and climate data
- AlphaVantageClient: Stock market daily prices
- ArxivClient: Scientific preprint search with category and date filters
- UsptoPatentClient: USPTO patent search by keyword, assignee, CPC class
- EpoClient: Placeholder for European patent search

New Domain:
- Domain::Economic for economic/financial indicator data

Updated Exports:
- Domain colors and shapes for Economic in visualization and export

Examples:
- economic_discovery: FRED + World Bank integration demo
- arxiv_discovery: AI/ML/Climate paper search demo
- patent_discovery: Climate tech and AI patent search demo

All 85 tests passing. APIs tested with live endpoints.

* feat: Add Semantic Scholar, bioRxiv/medRxiv, and CrossRef research clients

New Research API Clients:
- SemanticScholarClient: Citation graph analysis, paper search, author lookup
  - Methods: search_papers, get_citations, get_references, search_by_field
  - Builds citation networks for graph analysis

- BiorxivClient: Life sciences preprints
  - Methods: search_recent, search_by_category (neuroscience, genomics, etc.)
  - Automatic conversion to Domain::Research

- MedrxivClient: Medical preprints
  - Methods: search_covid, search_clinical, search_by_date_range
  - Automatic conversion to Domain::Medical

- CrossRefClient: DOI metadata and scholarly communication
  - Methods: search_works, get_work, search_by_funder, get_citations
  - Polite pool support for better rate limits

All clients include:
- Rate limiting respecting API guidelines
- Retry logic with exponential backoff
- SemanticVector conversion with rich metadata
- Comprehensive unit tests

Examples:
- biorxiv_discovery: Fetch neuroscience and clinical research
- crossref_demo: Search publications, funders, datasets

Total: 104 tests passing, ~2,500 new lines of code

* feat: Add MCP server with STDIO/SSE transport and optimized discovery

MCP Server Implementation (mcp_server.rs):
- JSON-RPC 2.0 protocol with MCP 2024-11-05 compliance
- Dual transport: STDIO for CLI, SSE for HTTP streaming
- 22 discovery tools exposing all data sources:
  - Research: OpenAlex, ArXiv, Semantic Scholar, CrossRef, bioRxiv, medRxiv
  - Medical: PubMed, ClinicalTrials.gov, FDA
  - Economic: FRED, World Bank
  - Climate: NOAA
  - Knowledge: Wikipedia, Wikidata SPARQL
  - Discovery: Multi-source, coherence analysis, pattern detection
- Resources: discovery://patterns, discovery://graph, discovery://history
- Pre-built prompts: cross_domain_discovery, citation_analysis, trend_detection

Binary Entry Point (bin/mcp_discovery.rs):
- CLI arguments with clap
- Configurable discovery parameters
- STDIO/SSE mode selection

Optimized Discovery Runner:
- Parallel data fetching with tokio::join!
- SIMD-accelerated vector operations (1.1M comparisons/sec)
- 6-phase discovery pipeline with benchmarking
- Statistical significance testing (p-values)
- Cross-domain correlation analysis
- CSV export and hypothesis report generation

Performance Results:
- 180 vectors from 3 sources in 7.5s
- 686 edges computed in 8ms
- SIMD throughput: 1,122,216 comparisons/sec

All 106 tests passing.

* feat: Add space, genomics, and physics data source clients

Add exotic data source integrations:
- Space clients: NASA (APOD, NEO, Mars, DONKI), Exoplanet Archive, SpaceX API, TNS Astronomy
- Genomics clients: NCBI (genes, proteins, SNPs), UniProt, Ensembl, GWAS Catalog
- Physics clients: USGS Earthquakes, CERN Open Data, Argo Ocean, Materials Project

New domains: Space, Genomics, Physics, Seismic, Ocean

All 106 tests passing, SIMD benchmark: 208k comparisons/sec

* chore: Update export/visualization and output files

* docs: Add API client inventory and reference documentation

* fix: Update API clients for 2025 endpoint changes

- ArXiv: Switch from HTTP to HTTPS (export.arxiv.org)
- USPTO: Migrate to PatentSearch API v2 (search.patentsview.org)
  - Legacy API (api.patentsview.org) discontinued May 2025
  - Updated query format from POST to GET
  - Note: May require API authentication
- FRED: Require API key (mandatory as of 2025)
  - Added error handling for missing API key
  - Added response error field parsing

All tests passing, ArXiv discovery confirmed working

* feat: Implement comprehensive 2025 API client library (11,810 lines)

Add 7 new API client modules implementing 35+ data sources:

Academic APIs (1,328 lines):
- OpenAlexClient, CoreClient, EricClient, UnpaywallClient

Finance APIs (1,517 lines):
- FinnhubClient, TwelveDataClient, CoinGeckoClient, EcbClient, BlsClient

Geospatial APIs (1,250 lines):
- NominatimClient, OverpassClient, GeonamesClient, OpenElevationClient

News & Social APIs (1,606 lines):
- HackerNewsClient, GuardianClient, NewsDataClient, RedditClient

Government APIs (2,354 lines):
- CensusClient, DataGovClient, EuOpenDataClient, UkGovClient
- WorldBankGovClient, UNDataClient

AI/ML APIs (2,035 lines):
- HuggingFaceClient, OllamaClient, ReplicateClient
- TogetherAiClient, PapersWithCodeClient

Transportation APIs (1,720 lines):
- GtfsClient, MobilityDatabaseClient
- OpenRouteServiceClient, OpenChargeMapClient

All clients include:
- Async/await with tokio and reqwest
- Mock data fallback for testing without API keys
- Rate limiting with configurable delays
- SemanticVector conversion for RuVector integration
- Comprehensive unit tests (252 total tests passing)
- Full error handling with FrameworkError

* docs: Add API client documentation for new implementations

Add documentation for:
- Geospatial clients (Nominatim, Overpass, Geonames, OpenElevation)
- ML clients (HuggingFace, Ollama, Replicate, Together, PapersWithCode)
- News clients (HackerNews, Guardian, NewsData, Reddit)
- Finance clients implementation notes

* feat: Implement dynamic min-cut tracking system (SODA 2026)

Based on El-Hayek, Henzinger, Li (SODA 2026) subpolynomial dynamic min-cut algorithm.

Core Components (2,626 lines):
- dynamic_mincut.rs (1,579 lines): EulerTourTree, DynamicCutWatcher, LocalMinCutProcedure
- cut_aware_hnsw.rs (1,047 lines): CutAwareHNSW, CoherenceZones, CutGatedSearch

Key Features:
- O(log n) connectivity queries via Euler-tour trees
- n^{o(1)} update time when λ ≤ 2^{(log n)^{3/4}} (vs O(n³) Stoer-Wagner)
- Cut-gated HNSW search that respects coherence boundaries
- Real-time cut monitoring with threshold-based deep evaluation
- Thread-safe structures with Arc<RwLock>

Performance (benchmarked):
- 75x speedup over periodic recomputation
- O(1) min-cut queries vs O(n³) recompute
- ~25µs per edge update

Tests & Benchmarks:
- 36+ unit tests across both modules
- 5 benchmark suites comparing periodic vs dynamic
- Integration with existing OptimizedDiscoveryEngine

This enables real-time coherence tracking in RuVector, transforming
min-cut from an expensive periodic computation to a maintained invariant.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-04 14:36:41 -05:00

397 lines
9.7 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# ASCII Graph Visualization Guide
Terminal-based graph visualization for the RuVector Discovery Framework with ANSI colors, domain clustering, coherence heatmaps, and pattern timeline displays.
## Features
### 🎨 Graph Visualization
- **ASCII art rendering** with box-drawing characters
- **Domain-based coloring** using ANSI escape codes
- 🔵 Climate (Blue)
- 🟢 Finance (Green)
- 🟡 Research (Yellow)
- 🟣 Cross-domain (Magenta)
- **Cluster structure** showing node groupings by domain
- **Cross-domain bridges** displayed as connecting lines
### 📊 Domain Matrix
- Shows connectivity strength between domains
- Diagonal elements show node count per domain
- Off-diagonal elements show cross-domain edge counts
- Color-coded by domain
### 📈 Coherence Timeline
- **ASCII sparkline** chart for temporal coherence values
- **Adaptive scaling** based on value range
- Duration display (days/hours/minutes)
- Time range labels
### 🔍 Pattern Summary
- Pattern count by type with visual bars
- Statistical significance indicators
- Top patterns ranked by confidence
- P-values and effect sizes
### 🖥️ Complete Dashboard
Combines all visualizations into a single comprehensive view.
## API Reference
### Core Functions
#### `render_graph_ascii`
```rust
pub fn render_graph_ascii(
engine: &OptimizedDiscoveryEngine,
width: usize,
height: usize
) -> String
```
Renders the graph as ASCII art with colored domain nodes.
**Parameters:**
- `engine` - The discovery engine containing the graph
- `width` - Canvas width in characters (recommended: 80)
- `height` - Canvas height in characters (recommended: 20)
**Returns:** String containing the ASCII art representation
**Example:**
```rust
use ruvector_data_framework::visualization::render_graph_ascii;
let graph = render_graph_ascii(&engine, 80, 20);
println!("{}", graph);
```
---
#### `render_domain_matrix`
```rust
pub fn render_domain_matrix(
engine: &OptimizedDiscoveryEngine
) -> String
```
Renders a domain connectivity matrix showing connections between domains.
**Returns:** Formatted matrix string with domain statistics
**Example:**
```rust
let matrix = render_domain_matrix(&engine);
println!("{}", matrix);
```
---
#### `render_coherence_timeline`
```rust
pub fn render_coherence_timeline(
history: &[(DateTime<Utc>, f64)]
) -> String
```
Renders coherence timeline as ASCII sparkline/chart.
**Parameters:**
- `history` - Time series of (timestamp, coherence_value) pairs
**Returns:** ASCII chart with sparkline visualization
**Example:**
```rust
let timeline = render_coherence_timeline(&coherence_history);
println!("{}", timeline);
```
---
#### `render_pattern_summary`
```rust
pub fn render_pattern_summary(
patterns: &[SignificantPattern]
) -> String
```
Renders a summary of discovered patterns with statistics.
**Parameters:**
- `patterns` - List of significant patterns to summarize
**Returns:** Formatted summary with pattern breakdown
**Example:**
```rust
let summary = render_pattern_summary(&patterns);
println!("{}", summary);
```
---
#### `render_dashboard`
```rust
pub fn render_dashboard(
engine: &OptimizedDiscoveryEngine,
patterns: &[SignificantPattern],
coherence_history: &[(DateTime<Utc>, f64)]
) -> String
```
Renders a complete dashboard combining all visualizations.
**Parameters:**
- `engine` - Discovery engine with graph data
- `patterns` - Discovered patterns
- `coherence_history` - Time series of coherence values
**Returns:** Complete dashboard string
**Example:**
```rust
let dashboard = render_dashboard(&engine, &patterns, &coherence_history);
println!("{}", dashboard);
```
## Box-Drawing Characters
The module uses Unicode box-drawing characters for structure:
| Character | Unicode | Usage |
|-----------|---------|-------|
| `─` | U+2500 | Horizontal line |
| `│` | U+2502 | Vertical line |
| `┌` | U+250C | Top-left corner |
| `┐` | U+2510 | Top-right corner |
| `└` | U+2514 | Bottom-left corner |
| `┘` | U+2518 | Bottom-right corner |
| `┼` | U+253C | Cross |
| `┬` | U+252C | T-down |
| `┴` | U+2534 | T-up |
| `├` | U+251C | T-right |
| `┤` | U+2524 | T-left |
## ANSI Color Codes
Domain colors are implemented using ANSI escape sequences:
| Domain | Color | Code |
|--------|-------|------|
| Climate | Blue | `\x1b[34m` |
| Finance | Green | `\x1b[32m` |
| Research | Yellow | `\x1b[33m` |
| Cross-domain | Magenta | `\x1b[35m` |
| Reset | Default | `\x1b[0m` |
| Bright | Bold | `\x1b[1m` |
| Dim | Dimmed | `\x1b[2m` |
## Complete Example
```rust
use chrono::{Duration, Utc};
use ruvector_data_framework::optimized::{OptimizedConfig, OptimizedDiscoveryEngine};
use ruvector_data_framework::ruvector_native::{Domain, SemanticVector};
use ruvector_data_framework::visualization::render_dashboard;
use std::collections::HashMap;
fn main() {
// Create engine
let config = OptimizedConfig::default();
let mut engine = OptimizedDiscoveryEngine::new(config);
// Add vectors
let now = Utc::now();
for i in 0..10 {
let vector = SemanticVector {
id: format!("climate_{}", i),
embedding: vec![0.5 + i as f32 * 0.05; 128],
domain: Domain::Climate,
timestamp: now,
metadata: HashMap::new(),
};
engine.add_vector(vector);
}
// Compute coherence over time
let mut coherence_history = Vec::new();
let mut all_patterns = Vec::new();
for step in 0..5 {
let timestamp = now + Duration::hours(step);
let coherence = engine.compute_coherence();
coherence_history.push((timestamp, coherence.mincut_value));
let patterns = engine.detect_patterns_with_significance();
all_patterns.extend(patterns);
}
// Display dashboard
let dashboard = render_dashboard(&engine, &all_patterns, &coherence_history);
println!("{}", dashboard);
}
```
## Terminal Compatibility
The visualization module uses ANSI escape codes and Unicode box-drawing characters. For best results:
### ✅ Recommended Terminals
- **Linux**: GNOME Terminal, Konsole, Alacritty, Kitty
- **macOS**: Terminal.app, iTerm2
- **Windows**: Windows Terminal, ConEmu
- **Cross-platform**: Alacritty, Kitty
### ⚠️ Limited Support
- **Windows CMD**: No ANSI color support (use Windows Terminal instead)
- **Old terminals**: May not support Unicode box-drawing
### 🔧 Environment Variables
```bash
# Ensure Unicode support
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
# Force color output
export FORCE_COLOR=1
```
## Performance Considerations
### Memory
- Graph rendering: O(width × height) for canvas
- Timeline rendering: O(history length)
- Pattern summary: O(pattern count)
### Time Complexity
- Graph layout: O(nodes + edges)
- Timeline chart: O(history samples)
- Pattern summary: O(patterns × log(patterns)) for sorting
### Optimization Tips
1. **Limit canvas size** - Use 80×20 for standard terminals
2. **Sample large datasets** - Timeline auto-samples if > 60 points
3. **Filter patterns** - Only display top N patterns for large lists
## Testing
Run the visualization tests:
```bash
# Run all visualization tests
cargo test --lib visualization
# Run specific test
cargo test --lib test_render_graph_ascii
# Run visualization demo
cargo run --example visualization_demo
```
## Integration with Discovery Pipeline
```rust
use ruvector_data_framework::{DiscoveryPipeline, PipelineConfig};
use ruvector_data_framework::visualization::render_dashboard;
// Create pipeline
let config = PipelineConfig::default();
let mut pipeline = DiscoveryPipeline::new(config);
// Run discovery
let patterns = pipeline.run(data_source).await?;
// Build coherence history from engine
let coherence_history = pipeline.coherence.signals()
.iter()
.map(|s| (s.window.start, s.min_cut_value))
.collect();
// Visualize results
let dashboard = render_dashboard(
&pipeline.discovery_engine,
&patterns,
&coherence_history
);
println!("{}", dashboard);
```
## Customization
### Custom Color Schemes
Modify the color constants in `visualization.rs`:
```rust
const COLOR_CLIMATE: &str = "\x1b[34m"; // Change to your preference
const COLOR_FINANCE: &str = "\x1b[32m";
const COLOR_RESEARCH: &str = "\x1b[33m";
```
### Custom Characters
Replace box-drawing characters:
```rust
const BOX_H: char = '-'; // Use ASCII alternative
const BOX_V: char = '|';
const BOX_TL: char = '+';
```
### Layout Customization
Modify domain positions in `render_graph_ascii`:
```rust
let domain_regions = [
(Domain::Climate, 10, 2), // Top-left
(Domain::Finance, mid_x + 10, 2), // Top-right
(Domain::Research, 10, mid_y + 2), // Bottom-left
];
```
## Troubleshooting
### Colors not displaying
```bash
# Check terminal color support
echo -e "\x1b[34mBlue\x1b[0m"
# Enable color in cargo output
cargo run --color=always
```
### Box characters appear as question marks
```bash
# Verify UTF-8 encoding
locale # Should show UTF-8
# Set UTF-8 locale
export LANG=en_US.UTF-8
```
### Layout issues
- Ensure terminal width ≥ 80 characters
- Use monospace font (recommended: Cascadia Code, Fira Code)
- Adjust canvas size parameters
## Future Enhancements
Planned features for future versions:
- [ ] Interactive terminal UI with cursive/tui-rs
- [ ] Real-time streaming updates
- [ ] Export to SVG/PNG
- [ ] 3D graph visualization (ASCII isometric)
- [ ] Animated transitions between states
- [ ] Custom color themes
- [ ] Responsive layout for different terminal sizes
- [ ] Mouse interaction support
## See Also
- [Optimized Discovery Engine](../src/optimized.rs)
- [Pattern Detection](../src/discovery.rs)
- [Coherence Computation](../src/coherence.rs)
- [Cross-Domain Discovery Example](../examples/cross_domain_discovery.rs)
## License
Part of the RuVector Discovery Framework. See main repository for license information.