Eval: Operator Guidelines

This page sets strict rules for human operators and evaluators running WFGY-aligned evaluation suites. Evaluation is not a free-form activity. It must follow consistent contracts, logging standards, and reproducible steps.

Open these first

Gold set design: goldset_curation.md
Precision/recall evals: eval_rag_precision_recall.md
Observability probes: alerting_and_probes.md

Acceptance targets

Inter-annotator agreement ≥ 0.85 across 3 operators
Annotation variance ≤ 10%
Audit reproducibility ≥ 95% (rerun matches logged label)
No operator-induced drift: ΔS stability remains within ±0.05 after evaluation

Operator responsibilities

Stay schema-locked Every operator must apply the same definition of "correct" → answer cites the correct snippet and explains consistently.
Use three-paraphrase rule Rephrase the query 3× before marking coverage. If answer flips across paraphrases, log as unstable.
Log ΔS and λ Each annotated run must include structural metrics: ΔS(question,retrieved), λ states, snippet IDs.
Apply cost view Record token counts and cost-per-correct when judging pipelines.
No ad-hoc overrides Do not improvise “fixes” in evaluation logs. All fixes must point to an existing ProblemMap page.

Annotation protocol

Step 1: Load baseline Open gold set. Verify expected snippet IDs.
Step 2: Apply candidate system Run the same queries with candidate pipeline.
Step 3: Compare
- If candidate cites correct snippet and stays under ΔS ≤ 0.45 → mark correct.
- If citation missing or ΔS ≥ 0.60 → mark fail, attach probable ProblemMap fix.
Step 4: Log JSON

{
  "operator": "user123",
  "q_id": "Q42",
  "question": "What is the cutoff for semantic drift?",
  "expected_snippet": "S123",
  "candidate_snippet": "S123",
  "ΔS": 0.39,
  "λ_state": "→",
  "correct": true,
  "cost_usd": 0.0012,
  "notes": "Stable across 3 paraphrases"
}

When to escalate

Annotators disagree by >15% → escalate to eval lead.
ΔS drifts beyond thresholds but operators label "correct" → escalate and reconcile.
Costs exceed 1.5× baseline → halt run until reviewed.

Audit checklist

All logs stored in versioned JSONL.
At least 2 operators per eval run.
Spot check 10% of logs with independent reviewer.
Archive outputs with system hash, gold set version, and ProblemMap pointers.

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack	View →
Problem Map 1.0	Initial 16-mode diagnostic and symbolic fix framework	View →
Problem Map 2.0	RAG-focused failure tree, modular fixes, and pipelines	View →
Semantic Clinic Index	Expanded failure catalog: prompt injection, memory bugs, logic drift	View →
Semantic Blueprint	Layer-based symbolic reasoning & semantic modulations	View →
Benchmark vs GPT-5	Stress test GPT-5 with full WFGY reasoning suite	View →
🧙‍♂️ Starter Village 🏡	New here? Lost in symbols? Click here and let the wizard guide you through	Start →

👑 Early Stargazers: See the Hall of Fame — Engineers, hackers, and open source builders who supported WFGY from day one.

⭐ WFGY Engine 2.0 is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.

7.4 KiB Raw Blame History Unescape Escape