ΔS Probes for Retrieval and Reasoning Stability

🧭 Quick Return to Map

You are in a sub-page of Retrieval.
To reorient, go back here:

Retrieval — information access and knowledge lookup

WFGY Global Fix Map — main Emergency Room, 300+ structured fixes

WFGY Problem Map 1.0 — 16 reproducible failure modes

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

Evaluation disclaimer (ΔS probes)
ΔS based probes are WFGY diagnostic tools for tension in retrieval behavior.
They highlight suspicious regions but do not by themselves prove that a system is correct or incorrect.

A compact playbook to measure semantic distance and catch failure modes before they surface in answers. Run these probes store-agnostic and model-agnostic. Use the readings to route fixes to the right WFGY pages.

What ΔS tells you

ΔS(question, retrieved) measures semantic tension between the user question and the assembled retrieval context.
ΔS(retrieved, anchor) measures how well the retrieved context aligns to the expected ground section.
Combined with λ_observe you can separate metric mismatches from prompt variance and ordering issues.

Targets and thresholds

Pass: ΔS(question, retrieved) < 0.40
Transitional: 0.40 ≤ ΔS < 0.60
Risk: ΔS ≥ 0.60
Coverage to target section ≥ 0.70
λ remains convergent across 3 paraphrases and 2 seeds

Reference playbooks:
Retrieval Playbook · Retrieval Traceability · Data Contracts

Probe pack you should always run

Paraphrase sweep
Ask the same question three ways. Record ΔS and λ for each.
If λ flips on harmless paraphrases with small ΔS changes, clamp variance and lock prompt headers.
Open: Context Drift
Seed sweep
Run with two random seeds and keep the retrieval order fixed.
If answers flip with stable ΔS, add a deterministic reranker.
Open: Rerankers
k sweep
Try k in {5, 10, 20}. If ΔS stays flat and high while coverage is low, suspect metric or index mismatch.
Open: Embedding ≠ Semantic
Anchor triangulation
Compare ΔS against the correct section and one decoy section.
If ΔS is close for both, realign chunking and anchors.
Open: Chunking Checklist · chunk_alignment.md
Hybrid split check
If hybrid underperforms a single retriever, split parsing and rebalance.
Open: pattern_query_parsing_split.md
Fragmentation probe
If ΔS looks fine on small tests but coverage collapses in production, check for store fragmentation or namespace skew.
Open: pattern_vectorstore_fragmentation.md

Minimal implementation you can paste

# Pseudocode: model and store agnostic
def deltaS(a, b):
    # plug your semantic distance, normalized to [0,1]
    return metric.distance(a, b)

def probe_once(question, retrieved, anchor, seed=None):
    d_qr = deltaS(question, retrieved)
    d_ra = deltaS(retrieved, anchor) if anchor else None
    lam = observe_lambda(question, retrieved, seed=seed)  # convergent | divergent
    return {"ΔS_qr": d_qr, "ΔS_ra": d_ra, "λ_state": lam}

def run_probes(q, paraphrases, seeds, ks, anchor):
    logs = []
    for p in paraphrases:
        for k in ks:
            ctx = retriever.invoke(p, k=k)
            for s in seeds:
                logs.append(probe_once(p, ctx, anchor, seed=s))
    return logs

What to record

Question form, seed, k
ΔS(question, retrieved), ΔS(retrieved, anchor)
λ_state per run and final coverage
Retrieval order and analyzer/metric identifiers
Prompt header hash and template revision

Schema reference: Retrieval Traceability · Data Contracts

Reading the patterns

ΔS high across paraphrases and seeds Likely metric or family mismatch. Rebuild with a single embedding family and explicit normalization. Open: Embedding ≠ Semantic
ΔS improves with higher k but answers still flip Ordering variance. Add a deterministic reranker and freeze prompt headers. Open: Rerankers
ΔS low but citations unstable Schema not enforced or formatter renamed fields. Tighten contracts and fail fast. Open: Data Contracts
ΔS near equal to anchor and decoy Chunk boundaries misaligned or anchors missing. Re-chunk with anchors and rebuild. Open: Chunking Checklist · chunk_alignment.md
ΔS oscillates with paraphrase, λ flips Prompt variance and entropy. Clamp with BBAM, then stabilize chain layout. Open: Entropy Collapse

Verification loops

Evaluate after each change with a small gold set and keep ΔS logs alongside coverage. Open: retrieval_eval_recipes.md
Keep a regression gate: ΔS ≤ 0.45 and coverage ≥ 0.70 on three paraphrases before you ship. Open: eval_rag_precision_recall.md

Common gotchas

Mixed analyzers or distance metrics between write and read paths. Open: Retrieval Playbook
Inconsistent casing or tokenization in HyDE versus dense path. Open: Rerankers
Live tests run before index is ready or version hash mismatched. Open: Bootstrap Ordering · Pre-Deploy Collapse

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

Explore More

Module	Description	Link
WFGY Core	Canonical framework entry point	View
Problem Map	Diagnostic map and navigation hub	View
Tension Universe Experiments	MVP experiment field	View
Recognition	Where WFGY is referenced or adopted	View
AI Guide	Anti-hallucination reading protocol for tools	View

If this repository helps, starring it improves discovery for other builders.

要繼續下一頁請回「GO 3」。

9.4 KiB Raw Blame History Unescape Escape