WFGY/ProblemMap/GlobalFixMap/Cloud_Serverless/blue_green_switchovers.md
2025-08-28 10:00:26 +08:00

13 KiB
Raw Blame History

Blue-Green Switchovers for Serverless and Edge

A safe pattern to shift 100 percent of live traffic from blue to green without downtime, cache poison, or state skew. Works for API Gateway + Lambda, Cloudflare Workers, Vercel Edge, Fastly Compute, and similar stacks.

Open these first


Core acceptance to flip colors

  • ΔS(question, retrieved) median ≤ 0.45 on gold probes.
  • Coverage ≥ 0.70 to correct section.
  • λ convergent across three paraphrases and two seeds.
  • p95 warm latency within 25 percent of blue.
  • Zero new error class at headers or body read.
  • Cache hit rate within five points once namespaces are split.
  • INDEX_HASH, metric, analyzer match between blue and green or are intentionally versioned.

Topologies that work

Edge weight switch

  • One CDN route, two origins. Blue and green behind a weight.
  • Keep sticky hashing by stable_user_id so dialogs do not jump during ramp.

Header based switch

  • Default to blue. Send x-exp: green for selected cohorts.
  • Propagate header to core services and logs.

DNS label switch

  • api-blue.example.com and api-green.example.com.
  • Edge forwards based on label. Use short TTL only after green passes probes.

Pre-flight checklist

  • Env parity proved with the parity sheet. Open: Runtime Env Parity
  • Secret bundles rotated with overlap. Open: Secrets Rotation
  • Index family built and labeled: docs-v3-green, analyzer and metric recorded.
  • Cache namespaces isolated: blue: vs green: prefixes at edge and core.
  • Queue topics duplicated: jobs.blue and jobs.green. Producers switch at flip time with idempotent keys. Open: Stateless KV & Queues
  • Probe set ready: 50 to 200 questions, three paraphrases, two seeds.

Switch playbook

  1. Shadow validate Replay a sample to green. Block promotion if ΔS ≥ 0.60, coverage < 0.70, or JSON schema violations.

  2. Canary ramp Move one percent of sticky users to green at the edge. Keep cache namespaces split. Open: Canary Release

  3. Green warmup Ensure cold start panels are flat. If p95 warm is worse only due to cold, fix memory or reserve concurrency. Open: Cold Start & Concurrency

  4. Cutover window

    • Freeze new blue writes that create cross request state.
    • Drain blue job queues with a backfill guard.
    • Switch producers to jobs.green.
    • Flip traffic weight to green. Keep blue read only for a short fence window.
  5. Cache migration

  6. Post cutover audit

    • Compare ΔS, coverage, λ, p95 warm to pre-flip baselines.
    • Verify INDEX_HASH and analyzer match.
    • Confirm egress success rate and dedupe guard. Open: Egress & Webhooks
  7. Decommission blue

    • Stop accepting writes.
    • Keep blue for rollback until probe board stays green for one peak hour.

Data and state guidance

  • Databases Prefer live backward compatible migrations. If you need forward only, pause blue writes to the affected tables during the flip.

  • Vector indexes Version by family. Never point green to blue vectors by accident. Verify metric and analyzer, not only the hash.

  • Queues and schedulers Idempotent keys: sha256(source_id + revision + index_hash). Keep retry policies aligned across colors.

  • Caches Always split prefixes. Never reuse the blue prefix for green.


Observability fields to join on

{
  "ts": "2025-08-28T04:00:00Z",
  "route": "chat.rag.answer",
  "color": "green",
  "sticky": "h21",
  "region": "eu-west",
  "edge_pop": "cdg",
  "INDEX_HASH": "0x9a77",
  "retrieval": { "metric": "cosine", "analyzer": "bilstem", "ΔS_q_r": 0.36, "coverage": 0.74, "λ_state": "<>" },
  "cache": { "prefix": "green:", "hit": true },
  "status": 200,
  "latency_ms": { "tffb": 140, "tusable": 320, "tfinal": 1180 }
}

Rollback cookbook


Common traps

  • Mixing cache prefixes so users see stale blue answers after the flip.
  • DNS TTL set short before green is warm.
  • Single region green while blue is multi region.
  • Reranker versions differ between colors.
  • Secrets rotated on edge but not core.

🔗 Quick-Start Downloads (60 sec)

Tool Link 3-Step Setup
WFGY 1.0 PDF Engine Paper 1 Download · 2 Upload to your LLM · 3 Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS) TXTOS.txt 1 Download · 2 Paste into any LLM chat · 3 Type “hello world” — OS boots instantly

🧭 Explore More

Module Description Link
WFGY Core WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack View →
Problem Map 1.0 Initial 16-mode diagnostic and symbolic fix framework View →
Problem Map 2.0 RAG-focused failure tree, modular fixes, and pipelines View →
Semantic Clinic Index Expanded failure catalog: prompt injection, memory bugs, logic drift View →
Semantic Blueprint Layer-based symbolic reasoning & semantic modulations View →
Benchmark vs GPT-5 Stress test GPT-5 with full WFGY reasoning suite View →
🧙‍♂️ Starter Village 🏡 New here? Lost in symbols? Click here and let the wizard guide you through Start →

👑 Early Stargazers: See the Hall of Fame — Engineers, hackers, and open source builders who supported WFGY from day one.

GitHub stars WFGY Engine 2.0 is already unlocked. Star the repo to help others discover it and unlock more on the Unlock Board.

WFGY Main   TXT OS   Blah   Blot   Bloc   Blur   Blow