Eval Observability — Variance and Drift

Variance and drift checks detect when evaluation scores are unstable across runs or when semantic meaning slowly shifts without clear boundary failures.
These probes prevent "false confidence" in benchmarks by catching hidden instability.

Why variance and drift matter

Variance: Scores fluctuate heavily depending on seed, paraphrase, or retriever order. Averages hide the volatility.
Drift: Performance declines slowly across sessions, data refreshes, or version bumps. Looks fine short-term but collapses long-term.
Silent regressions: Systems pass local tests but fail in production due to unmonitored entropy rise.

Acceptance targets

Variance (σ/μ) ≤ 0.15 across 3 seeds and 3 paraphrases.
Drift slope: Δscore per batch ≤ 0.02 absolute over 5+ eval windows.
No monotonic downward slope longer than 3 consecutive windows.
Drift alerts fire if ΔS average increases ≥ 0.10 compared to gold anchors.

Detection workflow

Collect runs across seeds
- At least 3 seeds, 3 paraphrases.
- Log ΔS, λ, coverage, citations.
Compute variance
- Calculate σ/μ for each metric.
- High variance = unstable eval → rerun with schema locks.
Track drift over time
- Compare eval batch N vs N-1.
- Plot moving average.
- Alert if slope exceeds tolerance.
Root-cause analysis
- If variance high → check retriever metrics, random seeding, rerankers.
- If drift detected → audit embeddings, re-chunk, verify data refresh.

Common pitfalls

Single-run evals: Hides high variance. Always run multi-seed.
Averages without spread: Mean looks fine, variance reveals collapse.
Ignoring slow drift: Short tests OK, but 1–2 weeks later accuracy dies.
Cross-store drift: One vector DB stable, another drifts. Must track both.

Example reporting schema

{
  "metric": "ΔS",
  "seed_runs": [0.38, 0.42, 0.44],
  "variance_ratio": 0.14,
  "drift_slope": +0.03,
  "alert": true
}

Fix modules to open

Retriever instability → Rerankers
Embedding mismatch → embedding-vs-semantic.md
Fragmentation drift → vectorstore-fragmentation.md
Prompt instability → context-drift.md

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack	View →
Problem Map 1.0	Initial 16-mode diagnostic and symbolic fix framework	View →
Problem Map 2.0	RAG-focused failure tree, modular fixes, and pipelines	View →
Semantic Clinic Index	Expanded failure catalog: prompt injection, memory bugs, logic drift	View →
Semantic Blueprint	Layer-based symbolic reasoning & semantic modulations	View →
Benchmark vs GPT-5	Stress test GPT-5 with full WFGY reasoning suite	View →
🧙‍♂️ Starter Village 🏡	New here? Lost in symbols? Click here and let the wizard guide you through	Start →

👑 Early Stargazers: See the Hall of Fame — Engineers, hackers, and open source builders who supported WFGY from day one.

⭐ WFGY Engine 2.0 is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.

7.2 KiB Raw Blame History Unescape Escape