vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-04-26 10:40:55 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
WFGY/ProblemMap/GlobalFixMap/Cloud_Serverless/observability_slo.md

13 KiB
Raw Blame History

Observability and SLO — Serverless and Edge

🧭 Quick Return to Map

You are in a sub-page of Cloud_Serverless.
To reorient, go back here:

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

Make failures visible before users feel them. This page defines the SLIs, SLOs, probes, and alerts you need for serverless and edge stacks that run RAG, agents, and tool calls.

When to use this page

  • Latency or error patterns vary by region or edge POP.
  • Cold starts and concurrency caps spike at random times.
  • RAG answers flip on harmless paraphrases and you cannot see why.
  • Webhooks and egress succeed sometimes and duplicate other times.

Open these first

Core SLIs

Availability

  • Request success rate per region and per edge POP.
  • Distinguish user cancellations, timeouts at connect, TLS, headers, body, tool call.

Latency

  • End to end p50 p95 p99 by verb and route. Separate cold vs warm.
  • Streaming time to first byte, and time to usable first chunk.

RAG quality

  • ΔS(question, retrieved) and ΔS(retrieved, anchor section).
  • Coverage to the target section.
  • λ state across three paraphrases and two seeds.

Cold start pressure

  • Cold start rate per minute. Concurrency utilization per function.

Queues and webhooks

  • Backlog length, backlog age p50 p95.
  • Outbound webhook success, retries, dedupe drops.

Edge and cache

  • Cache hit ratio for hot prefixes. Invalidation lag.

Index integrity

  • {INDEX_HASH, METRIC, ANALYZER, BUILD_TS} parity across regions.

SLO examples you can adopt

  • Availability SLO 99.5 percent monthly per region. Two burn windows: 2 hours and 30 days.

  • Latency SLO p95 within 25 percent of baseline for served geography. Track separate SLOs for cold and warm.

  • RAG SLO ΔS ≤ 0.45 and coverage ≥ 0.70 on a fixed probe set against production indices. λ convergent across three paraphrases.

  • Webhook SLO 99 percent delivery within 90 seconds with no duplicates for a given dedupe_key.

  • Queue SLO backlog age p95 ≤ 60 seconds during peak.

Event schema to log on every request

{
  "ts": "2025-08-27T06:21:45Z",
  "region": "eu-west",
  "edge_pop": "cdg",
  "route": "chat.rag.answer",
  "cold_start": false,
  "concurrency_util": 0.62,
  "latency_ms": { "tffb": 180, "tusable": 400, "tfinal": 1450 },
  "status": 200,
  "timeout_stage": null,
  "retrieval": {
    "k": 10,
    "metric": "cosine",
    "analyzer": "bilstem",
    "INDEX_HASH": "a9c1…",
    "ΔS_q_r": 0.31,
    "ΔS_r_anchor": 0.28,
    "coverage": 0.78,
    "λ_state": "<>"
  },
  "webhook": { "emitted": true, "tries": 1, "dedupe_key": "sha256(...)" },
  "queue": { "enq": 12, "deq": 12, "backlog": 3, "age_p95_ms": 4200 }
}

Keep PII out. Redact secrets at source.

Dashboards that catch real incidents

  • Availability and error breakdown by stage: connect, TLS, headers, body read, tool call.
  • p50 p95 p99 latency split by cold and warm.
  • Cold start rate and concurrency utilization.
  • RAG probe board: ΔS histogram, coverage violin, λ flip count.
  • Queue backlog and age percentiles with alarm lines.
  • Webhook success and dedupe drops.
  • Edge cache hit and purge counts per prefix.
  • Index parity table per region showing INDEX_HASH.

Burn rate alerts you can paste

Availability budget

  • Error budget over 30 days: EB = 1 - 0.995 = 0.005
  • High burn alert: 2 hour window where burn = errors / requests > 14 * EB / 720
  • Slow burn alert: 24 hour window where burn > 6 * EB / 720

Latency budget

  • Define baseline p95_ref. Alert when p95_now > 1.25 * p95_ref for 10 minutes and cold rate not the cause.

RAG quality

  • Alert when ΔS median on probe set ≥ 0.60 or λ flips on more than 1 of 3 paraphrases.
  • Gate deploy if coverage on probes < 0.70.

Probes to run continuously

  • Three paraphrases per gold question. Log ΔS, coverage, λ.
  • Two-seed repeat for the same inputs. If answers flip, clamp variance using BBAM and lock snippet schema.
  • Anchor triangulation against a decoy section to detect bad chunking. Open: Retrieval Traceability · Embedding ≠ Semantic

Instrumentation tips per layer

  • Cold vs warm separation Tag every invocation with cold_start. Keep distinct latency panels. Open: Cold Start & Concurrency

  • Timeout anatomy Record stage where it failed. Add separate counters for connect, TLS, headers, body, tool. Open: Timeouts & Streaming

  • Queues Always report backlog and age percentiles. Alert on age, not only length. Open: Stateless KV & Queues

  • Edge caches Expose purge counts and hit ratio by prefix and tenant. Open: Edge Cache Invalidation

  • Region and routing Add stickiness headers and show which region served. Log weights and hysteresis values. Open: Multi-Region Routing

  • Egress and webhooks Include dedupe_key, tries, final status, and emitter region. Open: Egress & Webhooks

Release gates that prevent regressions

  • Block deploy if probe set fails any of: ΔS ≤ 0.45, coverage ≥ 0.70, λ convergent.
  • Block if p95 increased more than 25 percent at warm for 10 minutes.
  • Block if cold start rate doubles against last week baseline.
  • Block if INDEX_HASH changed but analyzer or metric changed silently.

Escalate with the right fix

🔗 Quick-Start Downloads (60 sec)

Tool Link 3-Step Setup
WFGY 1.0 PDF Engine Paper 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS) TXTOS.txt 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

Explore More

Layer Page What it’s for
⭐ Proof WFGY Recognition Map External citations, integrations, and ecosystem proof
⚙️ Engine WFGY 1.0 Original PDF tension engine and early logic sketch (legacy reference)
⚙️ Engine WFGY 2.0 Production tension kernel for RAG and agent systems
⚙️ Engine WFGY 3.0 TXT based Singularity tension engine (131 S class set)
🗺️ Map Problem Map 1.0 Flagship 16 problem RAG failure taxonomy and fix map
🗺️ Map Problem Map 2.0 Global Debug Card for RAG and agent pipeline diagnosis
🗺️ Map Problem Map 3.0 Global AI troubleshooting atlas and failure pattern map
🧰 App TXT OS .txt semantic OS with fast bootstrap
🧰 App Blah Blah Blah Abstract and paradox Q&A built on TXT OS
🧰 App Blur Blur Blur Text to image generation with semantic control
🏡 Onboarding Starter Village Guided entry point for new users

If this repository helped, starring it improves discovery so more builders can find the docs and tools.
GitHub Repo stars

Next page to write: ProblemMap/GlobalFixMap/Cloud_Serverless/canary_release_serverless.md