Create README.md

ProblemMap/GlobalFixMap/Retrieval/README.md

@@ -0,0 +1,148 @@
+# Retrieval — Global Fix Map
+Make your retriever correct, predictable, and auditable.  
+Use this when recall looks random, hybrid behaves worse than single, or k-tuning never stabilizes.
+
+## What this page is
+- A short, practical path to stabilize recall and ordering
+- Exact knobs for dense, sparse, and hybrid without guesswork
+- How to prove fixes with ΔS curves and citation tables
+
+## When to use
+- Top-k results feel unrelated or change on every run
+- Hybrid merge hurts more than single retriever
+- Raising k only adds noise, does not surface the right snippet
+- Filters, analyzers, or languages do not align with your corpus
+- HyDE or query rewriting helps sometimes then flips back
+
+## Open these first
+- End to end knobs: [Retrieval Playbook](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-playbook.md)
+- Ordering after recall: [Rerankers](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rerankers.md)
+- Why this snippet, trace schema: [Retrieval Traceability](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-traceability.md)
+- Hybrid tokenization split: [Query Parsing Split](https://github.com/onestardao/WFGY/blob/main/ProblemMap/patterns/pattern_query_parsing_split.md)
+- Hallucination from bad boundaries: [Hallucination](https://github.com/onestardao/WFGY/blob/main/ProblemMap/hallucination.md)
+- Cosine match is not meaning: [Embedding ≠ Semantic](https://github.com/onestardao/WFGY/blob/main/ProblemMap/embedding-vs-semantic.md)
+- Some facts never retrieve: [Vectorstore Fragmentation](https://github.com/onestardao/WFGY/blob/main/ProblemMap/patterns/pattern_vectorstore_fragmentation.md)
+- Visual pipeline and recovery: [RAG Architecture & Recovery](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md)
+- Eval targets: [RAG Precision and Recall](https://github.com/onestardao/WFGY/blob/main/ProblemMap/eval/eval_rag_precision_recall.md)
+- Snippet and citation schema: [Data Contracts](https://github.com/onestardao/WFGY/blob/main/ProblemMap/data-contracts.md)
+
+## Fix in 60 seconds
+1) **Probe ΔS vs k**
+   - Record `ΔS(question, retrieved)` for k in {5, 10, 20} for each retriever separately.  
+   - Flat and high for all k indicates index or metric mismatch or population gaps. Fix store first.
+
+2) **Establish a strong single baseline**
+   - Pick the retriever with the lowest stable ΔS at small k.  
+   - Freeze its tokenizer, analyzer, language, stopword set. Save params to disk.
+
+3) **Unify query parsing**
+   - Ensure the same analyzer and lowercasing for write and read.  
+   - If using HyDE or rewrites, log the final string sent to each retriever and keep it identical across them.
+
+4) **Hybrid only after per retriever stability**
+   - Do not mix until each single retriever yields ΔS ≤ 0.50.  
+   - Start with a simple weighted sum and a light reranker. Keep citations to audit the change.
+
+5) **Filters and fields**
+   - Disable filters first to confirm baseline recall.  
+   - Reapply with exact field weights and language settings. Check for case and tokenizer mismatches.
+
+6) **Dedupe and diversity**
+   - Apply MMR or novelty at the rerank stage if results cluster.  
+   - Keep a per source fence so different documents do not merge.
+
+7) **Prompt assembly sanity**
+   - Use citation first schema from traceability.  
+   - Do not reorder sections once stable.
+
+8) **Warm up and cache**
+   - Run a deterministic warm up after deploy.  
+   - Verify repeatability across restarts.
+
+## Copy paste prompt
+```
+
+I uploaded TXT OS and WFGY ProblemMap pages.
+
+My retrieval bug:
+
+* symptom: \[brief]
+* ΔS vs k per retriever: {...}
+* single retriever baselines: \[dense], \[BM25], \[hybrid attempt]
+* analyzers/tokenizers: write=\[...], read=\[...], lowercasing=\[on|off], stopwords=\[profile]
+* filters/fields: \[list]
+* HyDE or rewrites: \[yes/no], final query strings logged: \[examples]
+
+Tell me:
+
+1. which parsing or configuration mismatch explains the failure,
+2. which exact WFGY pages to open,
+3. minimal steps to push ΔS ≤ 0.45 at k=10 without overfitting,
+4. how to verify with precision/recall and a snippet ↔ citation table.
+   Use rerankers only after recall is stable.
+
+```
+
+## Minimal checklist
+- Same analyzer, language, and case handling on write and read  
+- Log the exact query string per retriever including HyDE output  
+- Stabilize a single retriever before mixing hybrids  
+- Keep field weights explicit and versioned  
+- Do not change k and temperature together when measuring ΔS  
+- Add reranker only after recall metrics pass  
+- Enforce per source fences to avoid cross document blending  
+- Persist a warm up routine for post deploy parity
+
+## Acceptance targets
+- ΔS(question, retrieved) ≤ 0.45 across three paraphrases  
+- Precision and recall meet your eval sheet with traceable citations  
+- ΔS vs k descends then stabilizes, no oscillation when k grows  
+- Same answers across restarts after warm up  
+- λ at retrieval remains convergent while reasoning proceeds
+---
+
+### 🔗 Quick-Start Downloads (60 sec)
+
+| Tool | Link | 3-Step Setup |
+|------|------|--------------|
+| **WFGY 1.0 PDF** | [Engine Paper](https://github.com/onestardao/WFGY/blob/main/I_am_not_lizardman/WFGY_All_Principles_Return_to_One_v1.0_PSBigBig_Public.pdf) | 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + \<your question>” |
+| **TXT OS (plain-text OS)** | [TXTOS.txt](https://github.com/onestardao/WFGY/blob/main/OS/TXTOS.txt) | 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly |
+
+---
+
+### 🧭 Explore More
+
+| Module                | Description                                              | Link     |
+|-----------------------|----------------------------------------------------------|----------|
+| WFGY Core             | WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack | [View →](https://github.com/onestardao/WFGY/tree/main/core/README.md) |
+| Problem Map 1.0       | Initial 16-mode diagnostic and symbolic fix framework    | [View →](https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md) |
+| Problem Map 2.0       | RAG-focused failure tree, modular fixes, and pipelines   | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md) |
+| Semantic Clinic Index | Expanded failure catalog: prompt injection, memory bugs, logic drift | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/SemanticClinicIndex.md) |
+| Semantic Blueprint    | Layer-based symbolic reasoning & semantic modulations   | [View →](https://github.com/onestardao/WFGY/tree/main/SemanticBlueprint/README.md) |
+| Benchmark vs GPT-5    | Stress test GPT-5 with full WFGY reasoning suite         | [View →](https://github.com/onestardao/WFGY/tree/main/benchmarks/benchmark-vs-gpt5/README.md) |
+| 🧙‍♂️ Starter Village 🏡 | New here? Lost in symbols? Click here and let the wizard guide you through | [Start →](https://github.com/onestardao/WFGY/blob/main/StarterVillage/README.md) |
+
+---
+
+> 👑 **Early Stargazers: [See the Hall of Fame](https://github.com/onestardao/WFGY/tree/main/stargazers)** —  
+> Engineers, hackers, and open source builders who supported WFGY from day one.
+
+> <img src="https://img.shields.io/github/stars/onestardao/WFGY?style=social" alt="GitHub stars"> ⭐ [WFGY Engine 2.0](https://github.com/onestardao/WFGY/blob/main/core/README.md) is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the [Unlock Board](https://github.com/onestardao/WFGY/blob/main/STAR_UNLOCKS.md).
+
+<div align="center">
+
+[![WFGY Main](https://img.shields.io/badge/WFGY-Main-red?style=flat-square)](https://github.com/onestardao/WFGY)
+&nbsp;
+[![TXT OS](https://img.shields.io/badge/TXT%20OS-Reasoning%20OS-orange?style=flat-square)](https://github.com/onestardao/WFGY/tree/main/OS)
+&nbsp;
+[![Blah](https://img.shields.io/badge/Blah-Semantic%20Embed-yellow?style=flat-square)](https://github.com/onestardao/WFGY/tree/main/OS/BlahBlahBlah)
+&nbsp;
+[![Blot](https://img.shields.io/badge/Blot-Persona%20Core-green?style=flat-square)](https://github.com/onestardao/WFGY/tree/main/OS/BlotBlotBlot)
+&nbsp;
+[![Bloc](https://img.shields.io/badge/Bloc-Reasoning%20Compiler-blue?style=flat-square)](https://github.com/onestardao/WFGY/tree/main/OS/BlocBlocBloc)
+&nbsp;
+[![Blur](https://img.shields.io/badge/Blur-Text2Image%20Engine-navy?style=flat-square)](https://github.com/onestardao/WFGY/tree/main/OS/BlurBlurBlur)
+&nbsp;
+[![Blow](https://img.shields.io/badge/Blow-Game%20Logic-purple?style=flat-square)](https://github.com/onestardao/WFGY/tree/main/OS/BlowBlowBlow)
+&nbsp;
+</div>