mirror of
https://github.com/ruvnet/RuVector.git
synced 2026-05-26 16:04:02 +00:00
25176801e9
Comprehensive benchmark suite for evaluating RuvLTRA models on Claude Code-specific tasks (not HumanEval/MBPP generic coding). Routing Benchmark (96 test cases): - 13 agent types: coder, researcher, reviewer, tester, architect, security-architect, debugger, documenter, refactorer, optimizer, devops, api-docs, planner - Categories: implementation, research, review, testing, architecture, security, debugging, documentation, refactoring, performance, devops, api-documentation, planning, ambiguous - Difficulty levels: easy, medium, hard - Metrics: accuracy by category/difficulty, latency percentiles Embedding Benchmark: - Similarity detection: 36 pairs (high/medium/low/none similarity) - Semantic search: 5 queries with relevance-graded documents - Clustering: 5 task clusters (auth, testing, database, frontend, devops) - Metrics: MRR, NDCG, cluster purity, silhouette score CLI commands: - `ruvllm benchmark routing` - Test agent routing accuracy - `ruvllm benchmark embedding` - Test embedding quality - `ruvllm benchmark full` - Complete evaluation suite Baseline results (keyword router): - Routing: 66.7% accuracy (needs native model for improvement) - Establishes comparison point for model evaluation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| agentic-integration | ||
| agentic-synth | ||
| agentic-synth-examples | ||
| burst-scaling | ||
| cli | ||
| cloud-run | ||
| core | ||
| graph-data-generator | ||
| graph-node | ||
| graph-wasm | ||
| node | ||
| postgres-cli | ||
| router | ||
| router-darwin-arm64 | ||
| router-darwin-x64 | ||
| router-linux-arm64-gnu | ||
| router-linux-x64-gnu | ||
| router-win32-x64-msvc | ||
| rudag | ||
| ruvector | ||
| ruvector-extensions | ||
| ruvllm | ||
| ruvllm-darwin-arm64 | ||
| ruvllm-darwin-x64 | ||
| ruvllm-linux-arm64-gnu | ||
| ruvllm-linux-x64-gnu | ||
| ruvllm-win32-x64-msvc | ||
| rvlite | ||
| sona | ||
| spiking-neural | ||
| tiny-dancer | ||
| tiny-dancer-darwin-arm64 | ||
| tiny-dancer-darwin-x64 | ||
| tiny-dancer-linux-arm64-gnu | ||
| tiny-dancer-linux-x64-gnu | ||
| tiny-dancer-win32-x64-msvc | ||
| wasm | ||