ruvector/docs/metaharness-implementation-plan.md

MetaHarness Integration for RuVector: Comprehensive Benchmark Suite Implementation Plan

Author: Claude Code MetaHarness Architect
Date: 2026-06-21
Phase: Phase 1 MVP (2026-06-21 to 2026-08-30)
Status: In Development

---

Executive Summary

This document outlines the 5-phase implementation plan to integrate MetaHarness with RuVector's benchmark suite, enabling autonomous parameter optimization via Darwin Mode evolution against public leaderboard scores (ANN-Benchmarks, BEIR, VectorDBBench, MTEB).

Key outcomes:

• Phase 1: ANN-Benchmarks compatibility layer + single-dataset harness (4 weeks)
• Phase 2: Parameter sweep framework (3 weeks)
• Phase 3: BEIR + VectorDBBench integration (4 weeks)
• Phase 4: Darwin Mode evolution loop (3 weeks)
• Phase 5: MTEB embedding quality validation (2 weeks)

Total: 16 weeks, 8 concurrent agents, ~12K LOC across TypeScript + Rust.

---

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│  Public Leaderboards (ANN-Benchmarks, BEIR, MTEB)       │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│  MetaHarness Darwin Mode Integration Layer              │
│  (scorePolicy.ts, mutationSurfaces.ts, configSchema.ts) │
└────┬──────────┬──────────────┬────────────┬─────────────┘
     │          │              │            │
┌────▼──┐  ┌────▼──────┐  ┌───▼──────┐  ┌──▼───────┐
│Phase 1│  │ Phase 2    │  │ Phase 3  │  │ Phase 4  │
│ HDF5  │  │ Parameter  │  │ BEIR +   │  │ Darwin   │
│Loader │  │ Sweep      │  │ VDBBench │  │ Mode     │
│SIFT/  │  │ (Grid+     │  │          │  │Evolution │
│GIST   │  │ Random)    │  │          │  │Loop      │
└────┬──┘  └────┬───────┘  └───┬──────┘  └──┬───────┘
     │          │              │            │
     └──────────┴──────────────┴────────────┘
            │
     ┌──────▼──────────────┐
     │  RuVector Core      │
     │  ─────────────────  │
     │  HNSW, RaBitQ,      │
     │  Matryoshka, PQ,    │
     │  Hybrid, LSM-ANN,   │
     │  ColBERT, DiskANN   │
     └─────────────────────┘

---

Phase 1: ANN-Benchmarks Compatibility Layer (4 weeks)

Goal: Load SIFT1M, GIST1M, GloVe datasets; measure recall@10, QPS; single dataset benchmark harness.

Deliverables

File	Lines	Purpose
scripts/benchmark/ann-datasets.ts	400	HDF5 loader, dataset registry
scripts/benchmark/single-dataset-harness.ts	600	SIFT/GIST test runner, metric aggregation
scripts/benchmark/baseline-configs.json	200	RuVector module defaults (HNSW M=12, efConstruction=200, etc.)
scripts/benchmark/result-formatter.ts	300	CSV + JSON output, comparison tables
.github/workflows/benchmark-smoke.yml	100	Daily CI job (SIFT1M subset, 3 configs)
crates/ruvector-bench/src/hdf5_loader.rs	350	Rust HDF5 bindings (via hdf5 crate)
docs/validation/smoke-baseline-2026-06.json	150	Golden baseline for regression detection

Key APIs:

// ann-datasets.ts
interface Dataset {
  name: string;           // "sift1m", "gist1m", "glove-angular"
  dimension: number;      // 128, 960, 100
  train_size: number;     // 100k-1M
  test_size: number;      // 10k
  hdf5_url: string;       // download URL
  download_cache_dir: string;
}

async function loadDataset(ds: Dataset): Promise<{
  train: Float32Array[];
  test: Float32Array[];
  groundtruth: number[][];  // [test_size][100] nearest neighbor IDs
}>;

// single-dataset-harness.ts
interface BenchmarkConfig {
  module: string;          // "hnsw", "rabitq", "matryoshka"
  params: Record<string, any>;
  dataset: Dataset;
}

async function runBenchmark(config: BenchmarkConfig): Promise<{
  recall_at_k: number[];   // [1, 10, 100]
  qps: number;
  latency_p50_ms: number;
  latency_p99_ms: number;
  memory_mb: number;
  build_time_sec: number;
}>;

CI Gate (.github/workflows/benchmark-smoke.yml):

- name: Smoke Benchmark
  run: |
    npm run benchmark:sift1m:smoke
    # Pass if recall@10 >= baseline * 0.98 (allow 2% regression)
    node scripts/benchmark/check-regression.js \
      --baseline docs/validation/smoke-baseline-2026-06.json \
      --tolerance 0.02

Success Criteria:

• Load SIFT1M in <30s
• Run 3 configs in <5min per config
• CSV output matches manual Python benchmark ±1%
• 0 regression on main branch

---

Phase 2: Parameter Sweep Framework (3 weeks)

Goal: Grid + random search over index config space; identify Pareto frontier (recall vs QPS vs memory).

Deliverables

File	Lines	Purpose
scripts/benchmark/sweep-config.json	150	Grid definition (HNSW M∈[4,8,12,16,20,32], efConstruction∈[50,100,200,400])
scripts/benchmark/sweep-harness.ts	800	Grid/random exploration, Pareto ranking
scripts/benchmark/pareto-visualizer.ts	400	2D plots (recall vs QPS, memory vs latency)
crates/ruvector-bench/src/grid_search.rs	500	Parallel config evaluation (rayon)
docs/benchmark-results/phase2-pareto-frontier.json	300	Pareto archive per module

Sweep Grid:

{
  "sweep_spaces": {
    "hnsw": {
      "M": [4, 8, 12, 16, 20, 32],
      "efConstruction": [50, 100, 200, 400],
      "efSearch": [50, 100, 200]
    },
    "rabitq": {
      "bits": [1],
      "rotation": [true],
      "normalize": [true, false]
    },
    "matryoshka": {
      "full_dim": [768],
      "search_dims": [[64], [128, 256], [128, 256, 512]]
    },
    "pq": {
      "M": [8, 16, 32],
      "nbits": [4, 8]
    },
    "hybrid": {
      "sparse_weight": [0.2, 0.5, 0.8],
      "fusion_strategy": ["rrf", "linear", "dbsf"]
    }
  },
  "dataset": "sift1m",
  "sample_strategy": "grid",  // "grid" | "random" | "latin_hypercube"
  "sample_count": 50
}

Key API:

// sweep-harness.ts
interface ParetoPoint {
  config: BenchmarkConfig;
  recall_at_10: number;
  qps: number;
  memory_mb: number;
  p99_ms: number;
  timestamp: string;
}

async function sweepConfigs(
  space: SweepSpace,
  dataset: Dataset,
  maxParallel?: number
): Promise<ParetoPoint[]>;

function rankPareto(points: ParetoPoint[]): {
  dominating: ParetoPoint[];      // non-dominated set
  dominated: ParetoPoint[];
  hypervolume: number;             // Pareto hypervolume
};

Pareto Visualization:

<!-- pareto-frontier.html -->
<svg width="800" height="600">
  <!-- Scatter: X=recall@10, Y=QPS, bubble-size=memory -->
  <!-- Pareto frontier: red line connecting dominating points -->
  <!-- Hover: show config JSON -->
</svg>

Success Criteria:

• Identify 10-15 non-dominated configs per module
• Sweep completes in <2 hours (8 cores)
• Pareto frontier visually separates memory-optimized vs latency-optimized

---

Phase 3: BEIR + VectorDBBench Integration (4 weeks)

Goal: Add retrieval benchmarks (11 BEIR datasets, VectorDBBench workloads); measure NDCG, MRR, MAP.

Deliverables

File	Lines	Purpose
scripts/benchmark/beir-loader.ts	500	BEIR dataset fetcher + corpus indexing
scripts/benchmark/retrieval-harness.ts	700	NDCG@10, MRR, MAP computation
scripts/benchmark/vdb-bench-workloads.ts	400	Insert rate, query latency, memory under workload
crates/ruvector-bench/src/retrieval.rs	600	Batch retrieval, recall@k histogram
docs/benchmark-results/beir-baseline.json	250	BEIR baselines (DPR, GTR, E5)

BEIR Datasets:

{
  "beir_datasets": [
    "trec-covid",      // 169K docs, 50 queries
    "nfcorpus",        // 323K docs, 323 queries
    "nq",              // 3.2M docs, 3.45K queries
    "scifact",         // 5.2K docs, 300 queries
    "trec-news",       // 595K docs, 60 queries
    "dbpedia",         // 4.6M docs, 400 queries
    "trec-web",        // 3.1M docs, 50 queries
    "fever",           // 5.4M docs, 6.8K queries
    "climate-fever",   // 5.4M docs, 1535 queries
    "arguana",         // 8.8K docs, 1406 queries
    "webis-touche2020" // 382K docs, 49 queries
  ],
  "metrics": ["ndcg@10", "mrr", "map", "recall@100"]
}

Key API:

// beir-loader.ts
interface BEIRDataset {
  name: string;
  corpus: Document[];        // {id, text, metadata}
  queries: Query[];           // {id, text}
  qrels: Map<string, Map<string, number>>;  // {query_id -> {doc_id -> relevance}}
}

async function loadBEIRDataset(name: string): Promise<BEIRDataset>;

// retrieval-harness.ts
interface RetrievalMetrics {
  ndcg_at_k: number[];       // [10, 100, 1000]
  mrr: number;
  map: number;
  recall_at_k: number[];
  query_time_ms: number;
}

async function evaluateRetrieval(
  index: VectorIndex,
  dataset: BEIRDataset,
  k: number = 100
): Promise<RetrievalMetrics>;

VectorDBBench Workloads:

{
  "workloads": [
    {
      "name": "insert-heavy",
      "insert_rate": 10000,      // docs/sec
      "query_rate": 1000,
      "duration_sec": 60,
      "k": 10
    },
    {
      "name": "query-heavy",
      "insert_rate": 100,
      "query_rate": 5000,
      "duration_sec": 60,
      "k": 100
    }
  ]
}

Success Criteria:

• BEIR indexing: 5M docs in <5 min
• NDCG@10 ≥ 0.45 on nq dataset (vs DPR baseline 0.49)
• VectorDBBench: sustain 5K QPS for 60 sec without OOM

---

Phase 4: Darwin Mode Evolution Loop (3 weeks)

Goal: MetaHarness Darwin Mode autonomously evolves index configs to maximize composite score.

Deliverables

File	Lines	Purpose
scripts/benchmark/darwin-score-policy.ts	300	Score function composition
scripts/benchmark/mutation-surfaces.ts	400	Mutation definitions for all modules
scripts/benchmark/darwin-harness.ts	600	Main evolution loop, checkpoint strategy
.github/workflows/darwin-evolution.yml	120	Weekly evolution run
docs/darwin/evolution-runs/	per-run	Archive of all runs + winning configs

Score Function (darwin-score-policy.ts):

interface ScoringPolicy {
  baseline: {
    recall_at_10: number;    // 0.85
    qps: number;             // 50000
    memory_mb: number;        // 256
    latency_p99_ms: number;  // 5.0
  };
  weights: {
    recall: 0.4;
    qps: 0.3;
    memory: 0.2;
    latency: 0.1;
  };
}

function computeScore(metrics: BenchmarkMetrics, policy: ScoringPolicy): number {
  const recall_norm = metrics.recall_at_10 / policy.baseline.recall_at_10;
  const qps_norm = Math.log(metrics.qps / policy.baseline.qps);
  const mem_norm = 1 - (metrics.memory_mb / policy.baseline.memory_mb);
  const lat_norm = 1 - (metrics.latency_p99_ms / policy.baseline.latency_p99_ms);
  
  return (
    policy.weights.recall * recall_norm +
    policy.weights.qps * Math.max(0, qps_norm) +  // penalize slowdown
    policy.weights.memory * Math.max(0, mem_norm) +
    policy.weights.latency * Math.max(0, lat_norm)
  );
}

Mutation Surfaces (mutation-surfaces.ts):

type MutationSurface = {
  module: string;
  param: string;
  type: "int" | "float" | "enum" | "boolean";
  range?: [number, number];
  options?: string[];
  mutation_ops: {
    add?: (v: any) => any;
    multiply?: (v: any) => any;
    swap?: (options: string[]) => string;
  };
};

const MUTATION_SURFACES: MutationSurface[] = [
  {
    module: "hnsw",
    param: "M",
    type: "int",
    range: [4, 32],
    mutation_ops: {
      add: (v) => Math.min(v + 2, 32),
      multiply: (v) => Math.max(Math.floor(v * 0.8), 4)
    }
  },
  {
    module: "hnsw",
    param: "efConstruction",
    type: "int",
    range: [50, 400],
    mutation_ops: {
      add: (v) => Math.min(v + 50, 400),
      multiply: (v) => Math.max(Math.floor(v * 1.2), 50)
    }
  },
  {
    module: "rabitq",
    param: "normalize",
    type: "boolean"
  },
  {
    module: "matryoshka",
    param: "search_dims",
    type: "enum",
    options: ["[64]", "[128]", "[256]", "[64,128]", "[128,256]", "[256,512]"]
  },
  // ... 15+ more surfaces across all modules
];

Darwin Loop (darwin-harness.ts):

async function runDarwinEvolution(options: {
  dataset: Dataset;
  max_generations: number;
  population_size: number;
  mutation_rate: number;
  elite_fraction: number;
}): Promise<{
  generation: number;
  best_config: BenchmarkConfig;
  best_score: number;
  population: Array<{config, score}>;
  checkpoint: string;
}[]> {
  // 1. Initialize: Pareto frontier from Phase 2 + random mutations
  let population = [...phasePareto, ...randomMutations(options.population_size)];
  
  // 2. For each generation:
  for (let g = 0; g < options.max_generations; g++) {
    // a. Evaluate all configs
    const evaluated = await Promise.all(
      population.map(cfg => benchmarkAndScore(cfg))
    );
    
    // b. Rank by score, keep elite
    const sorted = evaluated.sort((a, b) => b.score - a.score);
    const elite = sorted.slice(0, Math.ceil(options.elite_fraction * population.size));
    
    // c. Mutate elite to create next generation
    const mutated = elite.flatMap(e =>
      Array(options.population_size / elite.length).fill(null).map(() =>
        mutateConfig(e.config, MUTATION_SURFACES)
      )
    );
    
    population = [...elite.map(e => e.config), ...mutated];
    
    // d. Checkpoint best config
    const best = sorted[0];
    console.log(`[G${g}] best_score=${best.score.toFixed(3)}, best_config=${JSON.stringify(best.config)}`);
    
    yield {
      generation: g,
      best_config: best.config,
      best_score: best.score,
      population: sorted.slice(0, 10),
      checkpoint: `generation-${g}.json`
    };
  }
}

ADR-150 Compliance (graceful degradation):

// darwin-harness.ts
async function initDarwinMode(): Promise<void> {
  try {
    const Darwin = await import("@metaharness/darwin");
    log.info("MetaHarness Darwin Mode loaded");
    return Darwin;
  } catch (e) {
    if (e.code === "MODULE_NOT_FOUND") {
      log.warn("@metaharness/darwin not installed; skipping evolution");
      log.warn("Install via: npm install --optional @metaharness/darwin");
      return null;
    }
    throw e;
  }
}

async function runBenchmark(...) {
  const darwin = await initDarwinMode();
  if (!darwin) {
    // Fallback: run phase 2 grid search instead
    return sweepConfigs(...);
  }
  // Run Darwin evolution
  return runDarwinEvolution(...);
}

Success Criteria:

• Evolve to a config that beats baseline on 3 of 4 metrics
• Checkpoint every generation (JSON archive)
• Zero crashes on missing MetaHarness (graceful degradation)

---

Phase 5: MTEB Embedding Quality Validation (2 weeks)

Goal: Validate embedding quality on MTEB benchmark (170K sentences, 15 retrieval tasks).

Deliverables

File	Lines	Purpose
scripts/benchmark/mteb-loader.ts	300	MTEB dataset fetcher
scripts/benchmark/mteb-harness.ts	400	STS evaluation, clustering scoring
scripts/benchmark/embedding-quality.ts	350	Vector similarity analysis
docs/benchmark-results/mteb-baseline.json	150	Baseline scores

MTEB Datasets:

• Retrieval (15 datasets): trec-covid, scifact, nfcorpus, nq, ...
• STS (semantic textual similarity): 8 datasets
• Clustering: 11 datasets
• Reranking: 4 datasets

Success Criteria:

• All-MiniLM-L6-v2 on nq: NDCG@10 ≥ 0.45
• E5-large-v2 on nq: NDCG@10 ≥ 0.50
• Complete in <10 hours

---

File Structure & Paths

ruvector/
├── scripts/benchmark/
│   ├── ann-datasets.ts                    (Phase 1, 400 lines)
│   ├── single-dataset-harness.ts          (Phase 1, 600 lines)
│   ├── baseline-configs.json              (Phase 1, 200 lines)
│   ├── result-formatter.ts                (Phase 1, 300 lines)
│   ├── check-regression.js                (Phase 1, 150 lines)
│   │
│   ├── sweep-config.json                  (Phase 2, 150 lines)
│   ├── sweep-harness.ts                   (Phase 2, 800 lines)
│   ├── pareto-visualizer.ts               (Phase 2, 400 lines)
│   │
│   ├── beir-loader.ts                     (Phase 3, 500 lines)
│   ├── retrieval-harness.ts               (Phase 3, 700 lines)
│   ├── vdb-bench-workloads.ts             (Phase 3, 400 lines)
│   │
│   ├── darwin-score-policy.ts             (Phase 4, 300 lines)
│   ├── mutation-surfaces.ts               (Phase 4, 400 lines)
│   ├── darwin-harness.ts                  (Phase 4, 600 lines)
│   │
│   ├── mteb-loader.ts                     (Phase 5, 300 lines)
│   ├── mteb-harness.ts                    (Phase 5, 400 lines)
│   ├── embedding-quality.ts               (Phase 5, 350 lines)
│   │
│   └── index.ts                           (master export, 50 lines)
│
├── crates/ruvector-bench/
│   ├── Cargo.toml
│   └── src/
│       ├── hdf5_loader.rs                 (Phase 1, 350 lines)
│       ├── grid_search.rs                 (Phase 2, 500 lines)
│       ├── retrieval.rs                   (Phase 3, 600 lines)
│       └── lib.rs
│
├── .github/workflows/
│   ├── benchmark-smoke.yml                (Phase 1, 100 lines)
│   ├── benchmark-sweep.yml                (Phase 2, 120 lines)
│   ├── benchmark-beir.yml                 (Phase 3, 140 lines)
│   └── darwin-evolution.yml               (Phase 4, 120 lines)
│
├── docs/validation/
│   ├── smoke-baseline-2026-06.json
│   └── manifests/
│       ├── 2026-06-21-sift1m.json
│       ├── 2026-06-21-beir-baseline.json
│       └── ...
│
├── docs/darwin/
│   ├── evolution-runs/
│   │   ├── 2026-07-10-run-1.json
│   │   ├── 2026-07-17-run-2.json
│   │   └── ...
│   └── best-configs-archive.json
│
└── docs/benchmark-results/
    ├── phase2-pareto-frontier.json
    ├── beir-baseline.json
    ├── mteb-baseline.json
    └── leaderboard-summary.html

---

CI/CD Integration

Daily Smoke Test

File: .github/workflows/benchmark-smoke.yml

name: Benchmark Smoke Test
on:
  schedule:
    - cron: "0 6 * * *"  # 6 AM UTC daily
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest-16core
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"
      
      - name: Install dependencies
        run: npm install
      
      - name: Download SIFT1M subset (100K)
        run: |
          curl -L https://ann-benchmarks.com/sift1m.hdf5 | head -c 100MB > sift1m-subset.hdf5
      
      - name: Run smoke benchmark (HNSW only)
        run: |
          npx ts-node scripts/benchmark/single-dataset-harness.ts \
            --dataset sift1m-subset \
            --modules hnsw,rabitq \
            --config baseline-configs.json \
            --output smoke-results.json
        timeout-minutes: 10
      
      - name: Check regression
        run: |
          node scripts/benchmark/check-regression.js \
            --baseline docs/validation/smoke-baseline-2026-06.json \
            --current smoke-results.json \
            --tolerance 0.02
      
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: smoke-results-${{ github.run_id }}
          path: smoke-results.json
      
      - name: Comment on PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const results = require('./smoke-results.json');
            const comment = `## Benchmark Smoke Test
            
            **SIFT1M (100K subset)**
            - HNSW: recall@10=${results.hnsw.recall_at_10.toFixed(3)}, QPS=${results.hnsw.qps.toFixed(0)}
            - RaBitQ: recall@10=${results.rabitq.recall_at_10.toFixed(3)}, QPS=${results.rabitq.qps.toFixed(0)}
            
            [Full results](https://github.com/ruvnet/ruvector/actions/runs/${{ github.run_id }})`;
            
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comment
            });

Weekly Parameter Sweep

File: .github/workflows/benchmark-sweep.yml (runs Phase 2)

name: Weekly Parameter Sweep
on:
  schedule:
    - cron: "0 20 * * 0"  # Sunday 8 PM UTC
  workflow_dispatch:

jobs:
  sweep:
    runs-on: ubuntu-latest-32core
    timeout-minutes: 240  # 4 hours
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Rust
        uses: dtolnay/rust-toolchain@stable
      
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"
      
      - name: Download full datasets
        run: |
          # Download to local cache, skip if cached
          npm run benchmark:download-datasets
      
      - name: Run sweep
        run: |
          npx ts-node scripts/benchmark/sweep-harness.ts \
            --config sweep-config.json \
            --parallel 8 \
            --output pareto-frontier-${{ github.run_id }}.json
      
      - name: Generate Pareto visualizations
        run: |
          npx ts-node scripts/benchmark/pareto-visualizer.ts \
            --input pareto-frontier-${{ github.run_id }}.json \
            --output pareto-frontier-${{ github.run_id }}.html
      
      - name: Commit results
        run: |
          git config user.email "bench@ruvector.local"
          git config user.name "Benchmark Bot"
          mv pareto-frontier-${{ github.run_id }}.json docs/benchmark-results/
          mv pareto-frontier-${{ github.run_id }}.html docs/benchmark-results/
          git add docs/benchmark-results/
          git commit -m "chore(bench): weekly parameter sweep $(date -u +%Y-%m-%d)"
          git push origin main
        if: always()

BEIR & VectorDBBench (Phase 3)

File: .github/workflows/benchmark-beir.yml

name: BEIR & VectorDBBench Benchmark
on:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * 1"  # Monday midnight UTC

jobs:
  beir:
    runs-on: ubuntu-latest-32core
    timeout-minutes: 480  # 8 hours
    steps:
      - uses: actions/checkout@v4
      
      - name: Download BEIR datasets
        run: npm run benchmark:download-beir
        timeout-minutes: 60
      
      - name: Run retrieval benchmark
        run: |
          npx ts-node scripts/benchmark/retrieval-harness.ts \
            --datasets nq,trec-covid,scifact \
            --modules hnsw,matryoshka,hybrid \
            --output beir-results-${{ github.run_id }}.json
      
      - name: Run VectorDBBench workloads
        run: |
          npx ts-node scripts/benchmark/vdb-bench-workloads.ts \
            --dataset nq \
            --config [insert-heavy,query-heavy] \
            --output vdb-results-${{ github.run_id }}.json
      
      - name: Store results
        run: |
          mkdir -p docs/validation/manifests
          mv beir-results-${{ github.run_id }}.json \
             docs/validation/manifests/beir-$(date -u +%Y-%m-%d).json
          mv vdb-results-${{ github.run_id }}.json \
             docs/validation/manifests/vdb-$(date -u +%Y-%m-%d).json
      
      - name: Create witness signature
        run: |
          npx ts-node scripts/benchmark/witness-signer.ts \
            --manifest docs/validation/manifests/beir-$(date -u +%Y-%m-%d).json \
            --sign-with /home/ruvultra/.ssh/id_ed25519
      
      - name: Commit & push
        run: |
          git config user.email "bench@ruvector.local"
          git config user.name "Benchmark Bot"
          git add docs/validation/manifests/
          git commit -m "chore(validation): beir+vdb benchmark $(date -u +%Y-%m-%d)"
          git push origin main

Darwin Evolution (Phase 4)

File: .github/workflows/darwin-evolution.yml

name: Darwin Mode Evolution
on:
  workflow_dispatch:
  schedule:
    - cron: "0 12 * * 3"  # Wednesday noon UTC (weekly)

jobs:
  darwin:
    runs-on: ubuntu-latest-32core
    timeout-minutes: 360  # 6 hours
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: "20"
      
      - name: Install MetaHarness Darwin
        run: |
          npm install --optional @metaharness/darwin
        continue-on-error: true  # OK if missing (ADR-150)
      
      - name: Run Darwin evolution
        run: |
          npx ts-node scripts/benchmark/darwin-harness.ts \
            --dataset sift1m \
            --generations 10 \
            --population-size 20 \
            --output darwin-run-${{ github.run_id }}.json
      
      - name: Extract best config
        run: |
          node -e "
            const run = require('./darwin-run-${{ github.run_id }}.json');
            const best = run.reduce((a,b) => a.best_score > b.best_score ? a : b);
            console.log('Best config (generation', best.generation + ')');
            console.log(JSON.stringify(best.best_config, null, 2));
            console.log('Score:', best.best_score.toFixed(4));
          "
      
      - name: Commit evolution history
        run: |
          mkdir -p docs/darwin/evolution-runs
          mv darwin-run-${{ github.run_id }}.json \
             docs/darwin/evolution-runs/$(date -u +%Y-%m-%d)-run-${{ github.run_number }}.json
          git add docs/darwin/evolution-runs/
          git commit -m "chore(darwin): evolution run $(date -u +%Y-%m-%d)"
          git push origin main
        if: success()

---

Metrics & Success Gates

Phase 1 Gate

• SIFT1M loads in <30s
• Single benchmark run takes <5 min per config
• CSV output within ±1% of manual Python baseline
• Smoke test passes daily with <2% regression tolerance

Phase 2 Gate

• Grid sweep completes in <2 hours (8 cores)
• Identify 10-15 non-dominated Pareto configs
• Pareto frontier is visually correct (no crossing)
• Top 3 configs beat baseline on at least 2 metrics

Phase 3 Gate

• BEIR indexing: 5M docs in <5 min per dataset
• NDCG@10 on NQ ≥ 0.45 (DPR baseline is 0.49)
• VectorDBBench: sustain 5K QPS for 60 sec without OOM
• All 11 BEIR datasets complete without timeout

Phase 4 Gate

• Darwin evolution produces a config beating baseline on 3+ metrics
• Graceful degradation: if @metaharness/darwin missing, falls back to Phase 2
• 100% of evolution runs checkpointed to JSON
• Zero crashes on platform (macOS, Linux, Windows)

Phase 5 Gate

• MTEB evaluation completes in <10 hours
• All-MiniLM-L6-v2 achieves ≥0.45 NDCG@10 on NQ
• E5-large-v2 achieves ≥0.50 NDCG@10 on NQ

---

Effort Estimate

Phase	Team	Weeks	Key Files	Risks
1	2 engineers	4	7 TypeScript, 1 Rust	HDF5 library compatibility
2	1 engineer	3	3 TypeScript, 1 Rust	Grid explosion (need pruning)
3	2 engineers	4	5 TypeScript, 1 Rust	BEIR dataset size (26M docs total)
4	1 engineer	3	3 TypeScript	@metaharness/darwin API stability
5	1 engineer	2	3 TypeScript	MTEB evaluation infrastructure
Total	8	16	21 TypeScript, 3 Rust	Dependency on MetaHarness

---

Dependencies & Risks

External Dependencies

• hdf5 crate (Rust) — used for Phase 1 ANN-Benchmarks loading
• @metaharness/darwin (npm) — optional, Phase 4 only (ADR-150 compliance)
• BEIR corpus — 26M docs, ~200GB compressed (Phase 3)
• MTEB datasets — 170K sentences (Phase 5)

Risks & Mitigations

Risk	Likelihood	Impact	Mitigation
HDF5 library not available on CI	Medium	High	Ship pre-built binaries, fallback to Python subprocess
BEIR dataset download timeout	Medium	High	Cache in GCS, use CDN mirror
MetaHarness Darwin unstable	Low	High	Vendorize snapshot, version-pin with fallback
Parameter sweep explodes (>1000 configs)	Medium	Medium	Implement early pruning, random sampling instead of grid
CI job timeout on large runs	Medium	Medium	Increase timeout, split into multiple jobs

---

Rollout Timeline

2026-06-21 — Phase 1 kickoff (ANN-Benchmarks loader + smoke test)
2026-07-19 — Phase 1 complete, Phase 2 starts (grid sweep)
2026-08-09 — Phase 2 complete, Phase 3 starts (BEIR integration)
2026-09-06 — Phase 3 complete, Phase 4 starts (Darwin evolution)
2026-09-27 — Phase 4 complete, Phase 5 starts (MTEB validation)
2026-10-11 — Phase 5 complete, MVP launch

---

Success Metrics (Post-MVP)

1. Reproducibility: All benchmark runs generate signed witness manifests (ADR-267)
2. Autonomy: Darwin Mode evolves at least 1 config/week that beats baseline
3. Publication: Submit SOTA results to ANN-Benchmarks, VectorDBBench leaderboards
4. Adoption: RuVector users run benchmarks via npm run benchmark:all
5. SOTA Claims: Claim SOTA in 3+ categories (recall@10, memory efficiency, latency)

---

Appendix: ADR-150 Compliance Checklist

• All @metaharness/* packages in optionalDependencies only
• Darwin Mode imports wrapped in try-catch MODULE_NOT_FOUND
• Fallback to Phase 2 grid search if Darwin unavailable
• README includes installation: npm install --optional @metaharness/darwin
• CI smoke test runs without MetaHarness installed
• No hard dependency on @metaharness/* in main code paths