WFGY/ProblemMap/GlobalFixMap/Chunking/README.md

Chunking — Global Fix Map

Find and fix the silent breaks at document boundaries.
You use this when sections look fine but retrieved snippets cut mid idea, tables split, or anchors vanish.

What this page is

• A quick route to correct chunk sizes, fences, and anchors.
• Structural rules to stop boundary drift without touching models.
• Checks you can measure and repeat.

When to use

• Top-k looks plausible but citations point to the wrong half of a section.
• Tables or code blocks are split across chunks.
• Headers disappear or merge with the next paragraph.
• Long answers smear topics across two sources.
• Recall is high but precision is noisy around joins.

Open these first

• Boundary checklist: Chunking Checklist
• Hallucination at boundaries: Hallucination
• Snippet schema and trace: Data Contracts · Retrieval Traceability
• Drift in long windows: Context Drift · Entropy Collapse

Fix in 60 seconds

1. Probe ΔS at joins
   • Compute ΔS across adjacent chunks A↔B for the same section title.
   • Trigger: ΔS ≥ 0.50 at the join or ΔS spikes when you remove the header.
2. Add λ_observe markers
   • Ask cite-then-explain. If cite fails and explain passes, boundary drift is the cause.
3. Patch the structure
   • Keep headers and section ids inside each chunk.
   • Do not split code fences, tables, math.
   • Cap plain-text chunks by semantic units, not hard token length.
   • Add section_id, page_no, char_span to every snippet.
   • For very long sections store sub-section anchors.

Copy-paste prompt

I uploaded TXT OS and the WFGY ProblemMap files.

My chunking bug:

* symptom: \[brief]
* traces: \[ΔS at several joins], \[examples of split tables or code], \[λ states]

Tell me:

1. which boundary rule is violated and why,
2. which fix pages to open in this repo,
3. minimal steps to push ΔS(join) ≤ 0.45 and keep λ convergent,
4. how to verify with a snippet ↔ citation table.
   Use BBMC for anchor alignment. If logic still flips, apply BBCR bridge.

Minimal checklist

• One header per chunk. Keep the header text in the chunk body.
• Never cut a table, code block, or list mid item.
• Prefer sentence or paragraph aware splitters.
• Store stable ids: source_id, section_id, sub_id if any.
• Keep cross-refs like figure or table captions with the referenced block.
• Add back-pressure: if a chunk would cut a fence, expand to include the full unit.

Acceptance targets

• ΔS(question, retrieved) ≤ 0.45 on three paraphrases.
• ΔS at adjacent joins ≤ 0.50.
• Cite-then-answer passes consistently.
• λ stays convergent when you reorder non-semantic headers.
• Retrieval coverage to the target section ≥ 0.70.

---

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

---

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack	View →
Problem Map 1.0	Initial 16-mode diagnostic and symbolic fix framework	View →
Problem Map 2.0	RAG-focused failure tree, modular fixes, and pipelines	View →
Semantic Clinic Index	Expanded failure catalog: prompt injection, memory bugs, logic drift	View →
Semantic Blueprint	Layer-based symbolic reasoning & semantic modulations	View →
Benchmark vs GPT-5	Stress test GPT-5 with full WFGY reasoning suite	View →
🧙‍♂️ Starter Village 🏡	New here? Lost in symbols? Click here and let the wizard guide you through	Start →

---

👑 Early Stargazers: See the Hall of Fame —
Engineers, hackers, and open source builders who supported WFGY from day one.

⭐ WFGY Engine 2.0 is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.