mirror of
https://github.com/onestardao/WFGY.git
synced 2026-05-02 13:31:07 +00:00
164 lines
4.9 KiB
Markdown
164 lines
4.9 KiB
Markdown
# Live monitoring & alerting — RAG services
|
||
|
||
**Goal:** list of recommended metrics, alert rules and dashboard panels to keep RAG pipelines observable and actionable.
|
||
|
||
---
|
||
|
||
## Core metrics to collect (recommended names)
|
||
|
||
**Service-level**
|
||
- `rag_e2e_latency_seconds` (histogram) — E2E latency (request in → answer out)
|
||
- `rag_error_count_total` — errors per endpoint
|
||
- `rag_request_count_total` — total requests
|
||
|
||
**Retrieval-level**
|
||
- `retriever_qps_total`
|
||
- `retriever_retrieved_docs_count` (per request)
|
||
- `retriever_empty_result_count_total` — unexpected empty sets
|
||
|
||
**Vectorstore**
|
||
- `vectorstore_index_load_time_seconds`
|
||
- `vectorstore_memory_bytes`
|
||
- `vectorstore_indexed_docs_total`
|
||
|
||
**Accuracy/provenance**
|
||
- `rag_citation_hit_rate` (CHR gauge over sliding window)
|
||
- `rag_precision_shipped` (periodic batch scorer push)
|
||
- `rag_under_refusal_count_total`
|
||
|
||
**Infrastructure**
|
||
- `llm_api_rate_limited_total`
|
||
- `llm_api_error_total`
|
||
- `queue_backlog_count` (if using background queues)
|
||
|
||
---
|
||
|
||
## Suggested PromQL alerts (examples)
|
||
|
||
> Tune thresholds to your workload.
|
||
|
||
**A) Latency breach (interactive)**
|
||
```yaml
|
||
alert: RAGHighP95Latency
|
||
expr: histogram_quantile(0.95, sum(rate(rag_e2e_latency_seconds_bucket[5m])) by (le,instance)) > 2
|
||
for: 5m
|
||
labels:
|
||
severity: page
|
||
annotations:
|
||
summary: "RAG p95 > 2s ({{ $labels.instance }})"
|
||
````
|
||
|
||
**B) Error spike**
|
||
|
||
```yaml
|
||
alert: RAGErrorSpike
|
||
expr: increase(rag_error_count_total[5m]) > 50
|
||
for: 2m
|
||
labels: { severity: page }
|
||
```
|
||
|
||
**C) Retriever empty results**
|
||
|
||
```yaml
|
||
alert: RetrieverEmptyResults
|
||
expr: increase(retriever_empty_result_count_total[5m]) > 1
|
||
for: 5m
|
||
labels: { severity: ticket }
|
||
```
|
||
|
||
**D) CHR drop**
|
||
|
||
```yaml
|
||
alert: CHRDrop
|
||
expr: rag_citation_hit_rate < 0.6
|
||
for: 10m
|
||
labels: { severity: ticket }
|
||
```
|
||
|
||
**E) LLM auth failure**
|
||
|
||
```yaml
|
||
alert: LLMAuthFail
|
||
expr: increase(llm_api_error_total{code="401"}[5m]) > 0
|
||
for: 1m
|
||
```
|
||
|
||
---
|
||
|
||
## Dashboard panels (recommended)
|
||
|
||
1. E2E latency (p50/p95/p99) trend.
|
||
2. Requests per second and error rate.
|
||
3. Retriever QPS, avg retrieved docs, empty results.
|
||
4. CHR & Precision (batch scorer push).
|
||
5. Vectorstore memory & disk IO.
|
||
6. LLM provider error & rate-limit metrics.
|
||
|
||
---
|
||
|
||
## Incident play (fast actions)
|
||
|
||
1. If CHR drop → run **diagnostic retrieval** for 10 golden queries (retrieved ids + cosine scores).
|
||
2. If retriever empty → check vectorstore health and index partitions. Restart index shard if needed.
|
||
3. If E2E latency spike with LLM errors → throttle traffic, put a hard rate limit and rollback deploy if needed.
|
||
4. If LLM auth failure → rotate key & redeploy secrets.
|
||
|
||
---
|
||
|
||
## How to integrate scoring metrics
|
||
|
||
* Periodic scorer job should push `rag_citation_hit_rate` and `rag_precision_shipped` as a short-timeseries gauge (per 5–15m window).
|
||
* Use batching: run `score_eval.py` (see `ProblemMap/eval/README.md`) nightly and push summary metrics via a small exporter.
|
||
|
||
---
|
||
|
||
## Troubleshooting queries (prometheus examples)
|
||
|
||
* Check p95 per instance:
|
||
|
||
```promql
|
||
histogram_quantile(0.95, sum(rate(rag_e2e_latency_seconds_bucket[5m])) by (le,instance))
|
||
```
|
||
* CHR trend:
|
||
|
||
```promql
|
||
avg_over_time(rag_citation_hit_rate[30m])
|
||
```
|
||
|
||
---
|
||
|
||
### Links
|
||
|
||
* Deployment checklist → [deployment\_checklist.md](./deployment_checklist.md)
|
||
* Debug playbook → [debug\_playbook.md](./debug_playbook.md)
|
||
* Eval & scoring → [../eval/README.md](../eval/README.md)
|
||
|
||
---
|
||
|
||
|
||
### 🔗 Quick-Start Downloads (60 sec)
|
||
|
||
| Tool | Link | 3-Step Setup |
|
||
|------|------|--------------|
|
||
| **WFGY 1.0 PDF** | [Engine Paper](https://github.com/onestardao/WFGY/blob/main/I_am_not_lizardman/WFGY_All_Principles_Return_to_One_v1.0_PSBigBig_Public.pdf) | 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + \<your question>” |
|
||
| **TXT OS (plain-text OS)** | [TXTOS.txt](https://github.com/onestardao/WFGY/blob/main/OS/TXTOS.txt) | 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly |
|
||
|
||
---
|
||
|
||
<!-- WFGY_FOOTER_START -->
|
||
|
||
### Explore More
|
||
|
||
| Module | Description | Link |
|
||
| --- | --- | --- |
|
||
| WFGY Core | Canonical framework entry point | [View](https://github.com/onestardao/WFGY/tree/main/core/README.md) |
|
||
| Problem Map | Diagnostic map and navigation hub | [View](https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md) |
|
||
| Tension Universe Experiments | MVP experiment field | [View](https://github.com/onestardao/WFGY/tree/main/TensionUniverse/Experiments) |
|
||
| Recognition | Where WFGY is referenced or adopted | [View](https://github.com/onestardao/WFGY/blob/main/recognition/README.md) |
|
||
| AI Guide | Anti-hallucination reading protocol for tools | [View](https://github.com/onestardao/WFGY/blob/main/AI_GUIDE.md) |
|
||
|
||
> If this repository helps, starring it improves discovery for other builders.
|
||
> [](https://github.com/onestardao/WFGY)
|
||
|
||
<!-- WFGY_FOOTER_END -->
|
||
|