vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-04-28 11:40:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
WFGY/ProblemMap/GlobalFixMap/Cloud_Serverless/observability_slo.md

13 KiB
Raw Blame History

Observability and SLO — Serverless and Edge

🧭 Quick Return to Map

You are in a sub-page of Cloud_Serverless.
To reorient, go back here:

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

Make failures visible before users feel them. This page defines the SLIs, SLOs, probes, and alerts you need for serverless and edge stacks that run RAG, agents, and tool calls.

When to use this page

  • Latency or error patterns vary by region or edge POP.
  • Cold starts and concurrency caps spike at random times.
  • RAG answers flip on harmless paraphrases and you cannot see why.
  • Webhooks and egress succeed sometimes and duplicate other times.

Open these first

Core SLIs

Availability

  • Request success rate per region and per edge POP.
  • Distinguish user cancellations, timeouts at connect, TLS, headers, body, tool call.

Latency

  • End to end p50 p95 p99 by verb and route. Separate cold vs warm.
  • Streaming time to first byte, and time to usable first chunk.

RAG quality

  • ΔS(question, retrieved) and ΔS(retrieved, anchor section).
  • Coverage to the target section.
  • λ state across three paraphrases and two seeds.

Cold start pressure

  • Cold start rate per minute. Concurrency utilization per function.

Queues and webhooks

  • Backlog length, backlog age p50 p95.
  • Outbound webhook success, retries, dedupe drops.

Edge and cache

  • Cache hit ratio for hot prefixes. Invalidation lag.

Index integrity

  • {INDEX_HASH, METRIC, ANALYZER, BUILD_TS} parity across regions.

SLO examples you can adopt

  • Availability SLO 99.5 percent monthly per region. Two burn windows: 2 hours and 30 days.

  • Latency SLO p95 within 25 percent of baseline for served geography. Track separate SLOs for cold and warm.

  • RAG SLO ΔS ≤ 0.45 and coverage ≥ 0.70 on a fixed probe set against production indices. λ convergent across three paraphrases.

  • Webhook SLO 99 percent delivery within 90 seconds with no duplicates for a given dedupe_key.

  • Queue SLO backlog age p95 ≤ 60 seconds during peak.

Event schema to log on every request

{
  "ts": "2025-08-27T06:21:45Z",
  "region": "eu-west",
  "edge_pop": "cdg",
  "route": "chat.rag.answer",
  "cold_start": false,
  "concurrency_util": 0.62,
  "latency_ms": { "tffb": 180, "tusable": 400, "tfinal": 1450 },
  "status": 200,
  "timeout_stage": null,
  "retrieval": {
    "k": 10,
    "metric": "cosine",
    "analyzer": "bilstem",
    "INDEX_HASH": "a9c1…",
    "ΔS_q_r": 0.31,
    "ΔS_r_anchor": 0.28,
    "coverage": 0.78,
    "λ_state": "<>"
  },
  "webhook": { "emitted": true, "tries": 1, "dedupe_key": "sha256(...)" },
  "queue": { "enq": 12, "deq": 12, "backlog": 3, "age_p95_ms": 4200 }
}

Keep PII out. Redact secrets at source.

Dashboards that catch real incidents

  • Availability and error breakdown by stage: connect, TLS, headers, body read, tool call.
  • p50 p95 p99 latency split by cold and warm.
  • Cold start rate and concurrency utilization.
  • RAG probe board: ΔS histogram, coverage violin, λ flip count.
  • Queue backlog and age percentiles with alarm lines.
  • Webhook success and dedupe drops.
  • Edge cache hit and purge counts per prefix.
  • Index parity table per region showing INDEX_HASH.

Burn rate alerts you can paste

Availability budget

  • Error budget over 30 days: EB = 1 - 0.995 = 0.005
  • High burn alert: 2 hour window where burn = errors / requests > 14 * EB / 720
  • Slow burn alert: 24 hour window where burn > 6 * EB / 720

Latency budget

  • Define baseline p95_ref. Alert when p95_now > 1.25 * p95_ref for 10 minutes and cold rate not the cause.

RAG quality

  • Alert when ΔS median on probe set ≥ 0.60 or λ flips on more than 1 of 3 paraphrases.
  • Gate deploy if coverage on probes < 0.70.

Probes to run continuously

  • Three paraphrases per gold question. Log ΔS, coverage, λ.
  • Two-seed repeat for the same inputs. If answers flip, clamp variance using BBAM and lock snippet schema.
  • Anchor triangulation against a decoy section to detect bad chunking. Open: Retrieval Traceability · Embedding ≠ Semantic

Instrumentation tips per layer

  • Cold vs warm separation Tag every invocation with cold_start. Keep distinct latency panels. Open: Cold Start & Concurrency

  • Timeout anatomy Record stage where it failed. Add separate counters for connect, TLS, headers, body, tool. Open: Timeouts & Streaming

  • Queues Always report backlog and age percentiles. Alert on age, not only length. Open: Stateless KV & Queues

  • Edge caches Expose purge counts and hit ratio by prefix and tenant. Open: Edge Cache Invalidation

  • Region and routing Add stickiness headers and show which region served. Log weights and hysteresis values. Open: Multi-Region Routing

  • Egress and webhooks Include dedupe_key, tries, final status, and emitter region. Open: Egress & Webhooks

Release gates that prevent regressions

  • Block deploy if probe set fails any of: ΔS ≤ 0.45, coverage ≥ 0.70, λ convergent.
  • Block if p95 increased more than 25 percent at warm for 10 minutes.
  • Block if cold start rate doubles against last week baseline.
  • Block if INDEX_HASH changed but analyzer or metric changed silently.

Escalate with the right fix

🔗 Quick-Start Downloads (60 sec)

Tool Link 3-Step Setup
WFGY 1.0 PDF Engine Paper 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS) TXTOS.txt 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

Explore More

Layer Page What it’s for
Proof WFGY Recognition Map External citations, integrations, and ecosystem proof
Engine WFGY 1.0 Original PDF based tension engine
Engine WFGY 2.0 Production tension kernel and math engine for RAG and agents
Engine WFGY 3.0 TXT based Singularity tension engine, 131 S class set
Map Problem Map 1.0 Flagship 16 problem RAG failure checklist and fix map
Map Problem Map 2.0 RAG focused recovery pipeline
Map Problem Map 3.0 Global Debug Card, image as a debug protocol layer
Map Semantic Clinic Symptom to family to exact fix
Map Grandma’s Clinic Plain language stories mapped to Problem Map 1.0
Onboarding Starter Village Guided tour for newcomers
App TXT OS TXT semantic OS, fast boot
App Blah Blah Blah Abstract and paradox Q and A built on TXT OS
App Blur Blur Blur Text to image with semantic control
App Blow Blow Blow Reasoning game engine and memory demo

If this repository helped, starring it improves discovery so more builders can find the docs and tools. GitHub Repo stars

Next page to write: ProblemMap/GlobalFixMap/Cloud_Serverless/canary_release_serverless.md