7 KiB
Failover & recovery — deterministic recovery steps
Purpose: deterministic operator steps to failover or recover critical components (vectorstore, retriever, generator, indexer, controller). Aim to reduce data loss and return to safe state quickly.
Basic principles
- Fail fast to a safe mode — prefer read-only answers or cached responses over uncontrolled writes or risky LLM calls.
- Preserve evidence — do not truncate logs or delete index segments until investigation complete.
- Prefer scoped recovery — restart single pod/shard before cluster-wide actions.
Scenario A: Vectorstore shard down / index corrupt
Symptoms
- Retriever returns empty sets or inconsistent scores for golden queries.
- Vectorstore pod logs show IO / index errors.
Steps
-
Mark the shard unhealthy in the service registry (so retriever avoids it).
-
If replica exists, route traffic to other replica.
-
Attempt graceful re-open:
kubectl -n $NS exec deploy/vectorstore -- /bin/sh -c "ctl index reopen shard-5" -
If reopen fails, restore from latest snapshot (S3) to a new shard:
- Create new PV and restore snapshot.
- Start fresh pod pointed to restored PV.
-
Re-run small validation suite (10–50 golden qids) before reintroducing shard.
Post recovery
- Re-index missing docs if necessary; track reindex job progress.
- Add a postmortem entry and schedule a permanent fix.
Scenario B: Generator (LLM) provider outage
Symptoms
- LLM errors (5xx), rate-limit responses, or auth failures.
Steps
-
Switch to backup LLM provider (if configured) via config flag:
# toggle provider in config map or feature flag kubectl -n $NS set env deploy/rag-api PROVIDER=backup-provider -
If no backup, enable local fallback:
- Return cached answers for known qids.
- Return safe refusal for unknown qids.
-
Throttle traffic and backlog long-running requests to a worker queue.
-
Once provider restored, slowly ramp traffic and compare CHR/precision to baseline.
Scenario C: Bootstrap deadlock at startup
Symptoms
- Pods stuck in CrashLoopBackOff or
Readynever true; logs show circular dependency or missing migration.
Steps
-
Inspect init containers & migration jobs:
kubectl -n $NS get jobs kubectl -n $NS logs job/migrations -
Run migrations manually in controlled pod:
kubectl -n $NS run --rm -it migration-runner --image=myimage -- bash -c "python migrate.py" -
Ensure controller component (if any) is up before starting retriever/generator. Use Helm hooks or manual
kubectl applyordering. -
If necessary, scale down and start components one-by-one.
Safety nets & best practices
- Keep automated snapshots of vectorstore daily; keep 7–14 days retention.
- Maintain a tested restore playbook and a “mini-cluster” restore test monthly.
- Automate warm-failover for LLMs: pre-warm API tokens for backup providers.
Post-incident
- Triage root cause, assign fixes.
- Add automated test that would have caught this.
- Update runbooks and notify stakeholders.
Links
- Deployment checklist → deployment_checklist.md
- Debug playbook → debug_playbook.md
- Live monitoring → live_monitoring_rag.md
🔗 Quick-Start Downloads (60 sec)
| Tool | Link | 3-Step Setup |
|---|---|---|
| WFGY 1.0 PDF | Engine Paper | 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>” |
| TXT OS (plain-text OS) | TXTOS.txt | 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly |
🧭 Explore More
| Module | Description | Link |
|---|---|---|
| WFGY Core | WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack | View → |
| Problem Map 1.0 | Initial 16-mode diagnostic and symbolic fix framework | View → |
| Problem Map 2.0 | RAG-focused failure tree, modular fixes, and pipelines | View → |
| Semantic Clinic Index | Expanded failure catalog: prompt injection, memory bugs, logic drift | View → |
| Semantic Blueprint | Layer-based symbolic reasoning & semantic modulations | View → |
| Benchmark vs GPT-5 | Stress test GPT-5 with full WFGY reasoning suite | View → |
| 🧙♂️ Starter Village 🏡 | New here? Lost in symbols? Click here and let the wizard guide you through | Start → |
👑 Early Stargazers: See the Hall of Fame —
Engineers, hackers, and open source builders who supported WFGY from day one.
⭐ WFGY Engine 2.0 is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.
