WFGY/ProblemMap/GlobalFixMap/Embeddings/duplication_and_near_duplicate_collapse.md

Duplication and Near Duplicate Collapse — Guardrails and Fix Patterns

Use this page when repeated or slightly varied snippets crowd the index, citations look repetitive, or the right section is hidden behind many near copies. The goal is to detect exact and near duplicates before or during indexing, collapse them to a canonical record, and verify with ΔS, coverage, and λ.

Open these first

• Visual map and recovery: rag-architecture-and-recovery.md
• End to end retrieval knobs: retrieval-playbook.md
• Traceability and cite first: retrieval-traceability.md
• Snippet schema and audits: data-contracts.md
• Chunking checklist: chunking-checklist.md
• Vector metric pitfalls: vectorstore-metrics-and-faiss-pitfalls.md
• Rerank safety net: rerankers.md

When to use this page

• Many hits look the same or cite the same wording with different URLs
• Boilerplate or footer text dominates top k
• Small edits produce new vectors that push anchors down
• OCR produces minor variants that inflate recall but hurt precision
• Crawls from mirrors or CDNs duplicate the same document

Acceptance targets

• ΔS(question, retrieved) ≤ 0.45
• Coverage of the target section ≥ 0.70
• λ remains convergent across three paraphrases and two seeds
• Duplicate inflation reduced by at least 90 percent on the gold set

---

Failure signatures → likely cause

• Ten snippets from similar URLs push the correct anchor to rank 8 or worse Likely cause. No canonicalization or collapse of mirrors and tracking params.

• Boilerplate repeats on every page and often ranks ahead of substance Likely cause. No boilerplate mask at chunk time and no duplicate filter.

• OCR corpus looks noisy with hyphenation and line wrap variants Likely cause. OCR cleanup not applied before hashing or embedding.

• Same paragraph exists in multiple language folders with small changes Likely cause. No language aware duplicate detection and no cluster level tie break.

---

Fix in 60 seconds

1. Canonicalize first Normalize URL and source keys. Remove tracking parameters. Pick a canonical per mirror set.

2. Fingerprint text before embedding Store text_sha256 on normalized text. Add a near duplicate signature such as MinHash or SimHash. Refuse ingest if text_sha256 already exists for the same (source_id, section_id, rev).

3. Collapse near duplicates Cluster by MinHash or SimHash then by cosine on vectors inside each cluster. Keep one canonical. Mark others as duplicate_of.

4. Prefer a deterministic tie break Choose the latest rev, highest ocr_conf, canonical domain, and longest snippet that still respects the section boundary.

5. Verify Three paraphrases and two seeds. Require coverage ≥ 0.70 and ΔS ≤ 0.45. Count unique sources in top k.

---

Canonicalization rules

• Normalize URL path. Lowercase host. Strip UTM and session params. Map known mirrors to a single source_id.
• Normalize text. Unicode NFC. Collapse whitespace. Fix OCR soft hyphens. Preserve code blocks and citations that users will query.
• Keep section anchors stable, for example parent_id plus section_id.

---

Fingerprints to store

• Exact text_sha256 on the final normalized text.

• Near duplicate text

  • MinHash over 5 or 7 word shingles with LSH buckets.
  • SimHash 64 bit over character 5 grams.

• Near duplicate vector Cosine threshold inside a small candidate set, for example neighbors at k equals 50 per section.

---

Collapse policy

• Form clusters from near duplicate candidates.
• Select a canonical item per cluster using stable priority: canonical_domain then latest_rev then ocr_conf then longer_snippet_with_same_anchor.
• Write only the canonical vector to the retriever collection.
• Keep duplicate_of pointers and duplicate_cluster_id for audits.
• Surface non canonical variants only in UI expansion, never in the top k pool.

---

Contract fields to add

{
  "canonical_domain": "docs.example.com",
  "canonical_url": "https://docs.example.com/guide#a1",
  "duplicate_cluster_id": "dup:9e44c...",
  "duplicate_of": null,
  "text_sha256": "sha256:...",
  "minhash_sig": ["...","...","..."],
  "simhash64": "0x8f32c1aa44d0beef",
  "ocr_conf": 0.97,
  "boilerplate_mask": "footer,nav,ads",
  "collapse_policy": "canonical_domain>latest_rev>ocr_conf>longer_snippet"
}

---

Probes you can paste into a notebook

Probe A — duplicate rate
Sample 10k snippets. Group by text_sha256. Report pct with count>1. Target < 2 percent after collapse.

Probe B — near duplicate clustering
Run MinHash LSH → sample 100 clusters → within each cluster compute median pairwise cosine. If median > 0.90, collapse is safe.

Probe C — anchor displacement
Before and after collapse run 50 gold queries. Record anchor rank and ΔS. Expect anchor rank to improve or hold and ΔS to drop.

Probe D — boilerplate dominance
Measure fraction of top-20 tokens coming from masked regions. If > 0.20, improve boilerplate mask and re-chunk.

---

Common edge cases and fixes

• Code examples duplicated across pages Preserve code fences. Add a feature flag to down weight boilerplate code unless the query is code scoped.

• Press releases syndicated across domains Use canonical_domain mapping and collapse clusters across domains.

• Multilingual near matches Do not collapse across languages by default. Only collapse if anchors and citations are identical and the user domain is monolingual.

• Versioned docs Keep the latest rev as canonical unless the query explicitly requests an older version.

---

Verification checklist

• Duplicate inflation drops by at least 90 percent on the gold set
• Coverage ≥ 0.70 and ΔS ≤ 0.45 across three paraphrases and two seeds
• λ convergent and top k unique source count increases
• No anchor regression after collapse on the held out suite

---

Copy paste prompt for the LLM step

TXT OS and the WFGY Problem Map are loaded.

My issue: duplicates and near duplicates crowd top-k and hide the correct anchor.
Traces:
- duplicate_rate=...
- cluster_examples=[...]
- ΔS(question,retrieved)=..., coverage=..., λ across 3 paraphrases

Tell me:
1) the failing layer and why,
2) the exact WFGY page to open next,
3) a minimal collapse policy and tie break that I should implement,
4) a verification plan to reach coverage ≥ 0.70 and ΔS ≤ 0.45.
Use BBMC, BBCR, BBPF, BBAM when relevant.

---

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

---

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live. full symbolic reasoning and math stack	View →
Problem Map 1.0	Initial 16 mode diagnostic and symbolic fixes	View →
Problem Map 2.0	RAG focused failure tree and pipelines	View →
Semantic Clinic Index	Prompt injection. memory bugs. logic drift	View →
Semantic Blueprint	Layer based symbolic reasoning. semantic modulations	View →
Benchmark vs GPT-5	Stress test with full WFGY suite	View →
Starter Village	A guided first run	Start →

---

👑 Early Stargazers. See the Hall of Fame Engineers, hackers, and open source builders who supported WFGY from day one.

⭐ WFGY Engine 2.0 is live. ⭐ Star the repo to help others discover it and unlock more on the Unlock Board.