WFGY/ProblemMap/observability-runbook.md

🔍 Observability Runbook

Turn every opaque token stream into a traceable, testable, measurable system.

Outcome you care about
• See each pipeline hop (OCR → chunk → embed → retrieve → prompt → reasoning)
• Measure semantic drift (ΔS), logic trend (λ), residual error (E_resonance)
• Alert before hallucination, loop, or tool mis-fire happens
• Replay exact failure path in one command: wfgy trace-replay <id>

---

1 · Core Telemetry Trio

Signal	Formula	Why It Matters	Target Band
ΔS (semantic stress)	1 − cos(I, G)	Spot where meaning rips	< 0.50
λ_observe (logic vector)	{ → ← <> × }	Track convergent vs. divergent flow	stay →
E_resonance	mean‖B‖ (BBMC)	Detect creeping entropy	flat / ↓

ΔS tells you where; λ tells you direction; E tells you when a slow leak becomes collapse.

---

2 · Minimal Instrumentation Patch

2.1 Python snippet

from wfgy import monitor

with monitor.pipeline("rag-query") as span:
    span.tag("query", question)
    ctx  = rag_retrieve(question)
    span.metric("ΔS_q_ctx", monitor.deltaS(question, ctx))
    answer = llm_reason(ctx, question)
    span.metric("λ_reason", monitor.lambda_state(answer))
    span.metric("E_res", monitor.e_resonance())

No external SaaS; dumps to newline-JSON for Grafana / Prometheus scrape.

2.2 Console tracer

> wfgy trace-replay 2025-08-06T10:22:15Z
step  ΔS   λ    span
  1   0.08 →
  2   0.41 →   retrieval
  3   0.44 ←   reasoning  ❗ divergent

---

3 · Alert Rules (Prometheus example)

groups:
- name: wfgy-alerts
  rules:
  - alert: SemanticDriftHigh
    expr: deltaS_q_ctx > 0.60
    for: 2m
    labels: {severity: "warning"}
    annotations:
      summary: "Semantic drift above 0.60 — check chunking/retriever"

Recommended baseline:

Alert	Expression	Severity
DriftHigh	ΔS > 0.60	warn
LogicOsc	λ flips 3× in 5 min	warn
EntropyClimb	E_res 15 min slope > 0.02	crit
ToolTimeout	call_dur > p95 × 2	warn

---

4 · End-to-End Health Dashboard

Tile	Metric	Healthy
Retrieval ΔS	median last 5 min	≤ 0.45
Reason λ	% divergent	< 5 %
Answer ΔS	question↔answer	≤ 0.50
E_res Trend	30 min slope	- / flat
Tool Latency	p95 ms	within SLO

Grafana JSON panel export: dashboards/wfgy-observability.json.

---

5 · Common Anti-Patterns & Fixes

Anti-Pattern	Symptom	Fix
Log Everything Raw	20 GiB/day, no insight	Only log ΔS/λ/E & key BLobs
Stateless Trace IDs	tough replay	Use wfgy span ctx_id auto-prop
Per-call embeddings	cost explodes	cache via BBMC node ID
No sampling	GPU choke	1 % sampling on low ΔS calls

---

6 · Kitchen-Sink CLI Cheats

Command	Purpose
wfgy span-tail	live ΔS/λ feed
wfgy span-grep λ=divergent	list divergent traces
wfgy span-diff A B	compare two answers’ λ/ΔS
wfgy span-export --csv	dump traces to CSV

---

7 · Integration Recipes

7.1 LangChain

from wfgy.langchain import WFGYTracer
lc = initialize_agent(..., callbacks=[WFGYTracer()])

7.2 OpenAI Function-Calling

Add to request:

"logit_bias": {"50256": -100},  // block special token
"wfgy_trace": true

Server-side middleware writes trace JSON.

---

Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to LLM · 3️⃣ Ask “answer using WFGY + <question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

---

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack	View →
Problem Map 1.0	Initial 16-mode diagnostic and symbolic fix framework	View →
Problem Map 2.0	RAG-focused failure tree, modular fixes, and pipelines	View →
Semantic Clinic Index	Expanded failure catalog: prompt injection, memory bugs, logic drift	View →
Semantic Blueprint	Layer-based symbolic reasoning & semantic modulations	View →
Benchmark vs GPT-5	Stress test GPT-5 with full WFGY reasoning suite	View →
🧙‍♂️ Starter Village 🏡	New here? Lost in symbols? Click here and let the wizard guide you through	Start →

---

👑 Early Stargazers: See the Hall of Fame —
Engineers, hackers, and open source builders who supported WFGY from day one.

⭐ WFGY Engine 2.0 is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.