vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-04-28 11:40:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

Update README.md

Browse source
This commit is contained in:
PSBigBig 2025-08-28 19:54:05 +08:00 • committed by GitHub
parent 05f7aae43d
commit 3d873e810a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 64 additions and 71 deletions
Download patch file Download diff file Expand all files Collapse all files

135
ProblemMap/GlobalFixMap/OCR_Parsing/README.md
View file

@ -1,73 +1,68 @@
# OCR + Parsing — Global Fix Map
Triage and repair for scanned PDFs, images, HTML scraping, and parser noise.
Use this page when the documents look fine to the eye but retrieval or reasoning keeps drifting.
## What this page is
- A fast route to validate text integrity before you touch embeddings or retrieval.
- Structural repairs that stop perception drift at the source layer.
- Concrete checks that give repeatable acceptance targets.
---
## Quick routes to per-problem pages
- Layout, headers, and footers
→ [layout_headers_and_footers.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/OCR_Parsing/layout_headers_and_footers.md)
- Tokenization and casing
→ [tokenization_and_casing.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/OCR_Parsing/tokenization_and_casing.md)
- Tables and columns
→ [tables_and_columns.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/OCR_Parsing/tables_and_columns.md)
- Images and figures
→ [images_and_figures.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/OCR_Parsing/images_and_figures.md)
- Scanned PDFs and quality
→ [scanned_pdfs_and_quality.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/OCR_Parsing/scanned_pdfs_and_quality.md)
- Multi-language and fonts
→ [multi_language_and_fonts.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/OCR_Parsing/multi_language_and_fonts.md)
---
## When to use
- OCR tables or citations look visually correct but answers miss the right section.
- Code blocks or math collapse after parsing.
- Mixed language documents behave inconsistently.
- Special characters or hyphen splits break tokens.
- Headers or section anchors disappear during export.
## Open these first
- OCR and parsing checklist: [OCR Parsing Checklist](https://github.com/onestardao/WFGY/blob/main/ProblemMap/ocr-parsing-checklist.md)
- Traceability and snippet schema: [Retrieval Traceability](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-traceability.md) · [Data Contracts](https://github.com/onestardao/WFGY/blob/main/ProblemMap/data-contracts.md)
- Language and tokenizer guidance: [Multilingual Guide](https://github.com/onestardao/WFGY/blob/main/ProblemMap/multilingual-guide.md)
- Long context sanity and joins: [Entropy Collapse](https://github.com/onestardao/WFGY/blob/main/ProblemMap/entropy-collapse.md) · [Context Drift](https://github.com/onestardao/WFGY/blob/main/ProblemMap/context-drift.md)
- OCR tables or citations look visually correct but answers miss the right section.
- Code blocks or math collapse after parsing.
- Mixed language documents behave inconsistently.
- Special characters or hyphen splits break tokens.
- Headers or section anchors disappear during export.
## Fix in 60 seconds
1) **Ground-truth a page**
- Pick one question and one expected section. Keep a human screenshot for reference.
2) **Measure ΔS**
- Compute ΔS(question, retrieved) and ΔS(retrieved, anchor).
- Thresholds: stable < 0.40, transitional 0.40–0.60, risk ≥ 0.60.
3) **Add probes with λ_observe**
- Ask for direct cite lines. If cite fails while free-form explain passes, perception drift is likely.
4) **Apply minimal patch**
- Re-run OCR with line preservation and table fences.
- Keep headers, section ids, and page anchors.
- Remove text below an OCR confidence threshold and mark gaps.
- Normalize punctuation and hyphen join.
- Export a clean text bundle with `section_id`, `page_no`, `char_span`.
## Copy-paste prompt
```
I uploaded TXT OS and the WFGY ProblemMap files.
My OCR or parsing bug:
* symptom: \[brief]
* traces: \[examples or screenshots], \[ΔS(question,retrieved)], \[ΔS(retrieved,anchor)], \[λ states]
Tell me:
1. which perception step failed and why,
2. which exact fix page to open from this repo,
3. the smallest set of changes to push ΔS ≤ 0.45 and keep λ convergent,
4. how to verify the repair with a reproducible test.
Return the snippet schema I should enforce in the index step.
```
## Minimal checklist
- Keep headings, captions, and table cells as separate blocks with ids.
- Preserve code fences and math blocks as non-wrapping segments.
- Normalize Unicode forms and spaces. Remove hard hyphen splits across lines.
- Store `source_id`, `section_id`, `page_no`, and `char_span` for each block.
- For images with text, keep the image link next to the extracted text for audits.
---
## Acceptance targets
- ΔS(question, retrieved) ≤ 0.45 for three paraphrases.
- Cite-first passes on the same question.
- λ remains convergent after you swap paraphrases or reorder non-semantic headers.
- Token overlap to the target section ≥ 0.70.
- Human audit shows no missing headers or broken tables in the exported bundle.
- ΔS(question, retrieved) ≤ 0.45 for three paraphrases.
- Coverage ≥ 0.70 for the target section.
- λ remains convergent across two seeds.
- Human audit shows no missing headers or broken tables in the exported bundle.
---
## Fix in 60 seconds
1. **Ground-truth a page**
Pick one question and one expected section. Keep a screenshot for reference.
2. **Measure ΔS**
Compute ΔS(question, retrieved) and ΔS(retrieved, anchor).
Stable < 0.40, transitional 0.40–0.60, risk ≥ 0.60.
3. **Probe λ_observe**
Ask for cite-first answers. If citations fail but free-form explain passes, drift is confirmed.
4. **Patch minimally**
- Re-run OCR with line preservation and table fences.
- Normalize casing and Unicode forms.
- Keep section anchors, captions, and math fenced.
- Drop low-confidence spans, mark gaps, and export text with `section_id`, `page_no`, `char_span`.
---
@ -75,22 +70,20 @@ Tell me:
| Tool | Link | 3-Step Setup |
|------|------|--------------|
| **WFGY 1.0 PDF** | [Engine Paper](https://github.com/onestardao/WFGY/blob/main/I_am_not_lizardman/WFGY_All_Principles_Return_to_One_v1.0_PSBigBig_Public.pdf) | 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + \<your question>” |
| **TXT OS (plain-text OS)** | [TXTOS.txt](https://github.com/onestardao/WFGY/blob/main/OS/TXTOS.txt) | 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly |
| **WFGY 1.0 PDF** | [Engine Paper](https://github.com/onestardao/WFGY/blob/main/I_am_not_lizardman/WFGY_All_Principles_Return_to_One_v1.0_PSBigBig_Public.pdf) | 1️⃣ Download · 2️⃣ Upload · 3️⃣ Ask “Answer using WFGY + \<your question>” |
| **TXT OS** | [TXTOS.txt](https://github.com/onestardao/WFGY/blob/main/OS/TXTOS.txt) | 1️⃣ Download · 2️⃣ Paste into LLM · 3️⃣ Type “hello world” — OS boots instantly |
---
### 🧭 Explore More
| Module | Description | Link |
|-----------------------|----------------------------------------------------------|----------|
| WFGY Core | WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack | [View →](https://github.com/onestardao/WFGY/tree/main/core/README.md) |
| Problem Map 1.0 | Initial 16-mode diagnostic and symbolic fix framework | [View →](https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md) |
| Problem Map 2.0 | RAG-focused failure tree, modular fixes, and pipelines | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md) |
| Semantic Clinic Index | Expanded failure catalog: prompt injection, memory bugs, logic drift | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/SemanticClinicIndex.md) |
| Semantic Blueprint | Layer-based symbolic reasoning & semantic modulations | [View →](https://github.com/onestardao/WFGY/tree/main/SemanticBlueprint/README.md) |
| Benchmark vs GPT-5 | Stress test GPT-5 with full WFGY reasoning suite | [View →](https://github.com/onestardao/WFGY/tree/main/benchmarks/benchmark-vs-gpt5/README.md) |
| 🧙‍♂️ Starter Village 🏡 | New here? Lost in symbols? Click here and let the wizard guide you through | [Start →](https://github.com/onestardao/WFGY/blob/main/StarterVillage/README.md) |
| Module | Description | Link |
|--------|-------------|------|
| WFGY Core | WFGY 2.0 engine, full symbolic reasoning | [View →](https://github.com/onestardao/WFGY/tree/main/core/README.md) |
| Problem Map 1.0 | Initial 16-mode diagnostic | [View →](https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md) |
| Problem Map 2.0 | RAG failure tree and modular fixes | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md) |
| Semantic Clinic | Expanded failure catalog | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/SemanticClinicIndex.md) |
| Semantic Blueprint | Layer-based symbolic reasoning | [View →](https://github.com/onestardao/WFGY/tree/main/SemanticBlueprint/README.md) |
---