WFGY/ProblemMap/ops/failover_and_recovery.md

Failover & recovery — deterministic recovery steps

Purpose: deterministic operator steps to failover or recover critical components (vectorstore, retriever, generator, indexer, controller). Aim to reduce data loss and return to safe state quickly.

---

Basic principles

1. Fail fast to a safe mode — prefer read-only answers or cached responses over uncontrolled writes or risky LLM calls.
2. Preserve evidence — do not truncate logs or delete index segments until investigation complete.
3. Prefer scoped recovery — restart single pod/shard before cluster-wide actions.

---

Scenario A: Vectorstore shard down / index corrupt

Symptoms

• Retriever returns empty sets or inconsistent scores for golden queries.
• Vectorstore pod logs show IO / index errors.

Steps

1. Mark the shard unhealthy in the service registry (so retriever avoids it).

2. If replica exists, route traffic to other replica.

3. Attempt graceful re-open:

   kubectl -n $NS exec deploy/vectorstore -- /bin/sh -c "ctl index reopen shard-5"
   
   
   

4. If reopen fails, restore from latest snapshot (S3) to a new shard:

   • Create new PV and restore snapshot.
   • Start fresh pod pointed to restored PV.

5. Re-run small validation suite (10–50 golden qids) before reintroducing shard.

Post recovery

• Re-index missing docs if necessary; track reindex job progress.
• Add a postmortem entry and schedule a permanent fix.

---

Scenario B: Generator (LLM) provider outage

Symptoms

• LLM errors (5xx), rate-limit responses, or auth failures.

Steps

1. Switch to backup LLM provider (if configured) via config flag:

   # toggle provider in config map or feature flag
   kubectl -n $NS set env deploy/rag-api PROVIDER=backup-provider
   

2. If no backup, enable local fallback:

   • Return cached answers for known qids.
   • Return safe refusal for unknown qids.

3. Throttle traffic and backlog long-running requests to a worker queue.

4. Once provider restored, slowly ramp traffic and compare CHR/precision to baseline.

---

Scenario C: Bootstrap deadlock at startup

Symptoms

• Pods stuck in CrashLoopBackOff or Ready never true; logs show circular dependency or missing migration.

Steps

1. Inspect init containers & migration jobs:

   kubectl -n $NS get jobs
   kubectl -n $NS logs job/migrations
   

2. Run migrations manually in controlled pod:

   kubectl -n $NS run --rm -it migration-runner --image=myimage -- bash -c "python migrate.py"
   

3. Ensure controller component (if any) is up before starting retriever/generator. Use Helm hooks or manual kubectl apply ordering.

4. If necessary, scale down and start components one-by-one.

---

Safety nets & best practices

• Keep automated snapshots of vectorstore daily; keep 7–14 days retention.
• Maintain a tested restore playbook and a “mini-cluster” restore test monthly.
• Automate warm-failover for LLMs: pre-warm API tokens for backup providers.

---

Post-incident

• Triage root cause, assign fixes.
• Add automated test that would have caught this.
• Update runbooks and notify stakeholders.

---

Links

• Deployment checklist → deployment_checklist.md
• Debug playbook → debug_playbook.md
• Live monitoring → live_monitoring_rag.md

---

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

---

Explore More

Module	Description	Link
WFGY Core	Canonical framework entry point	View
Problem Map	Diagnostic map and navigation hub	View
Tension Universe Experiments	MVP experiment field	View
Recognition	Where WFGY is referenced or adopted	View
AI Guide	Anti-hallucination reading protocol for tools	View

If this repository helps, starring it improves discovery for other builders.