WFGY/ProblemMap/GlobalFixMap/Language/tokenizer_mismatch.md

Tokenizer Mismatch — Guardrails and Fix Patterns

🧭 Quick Return to Map

You are in a sub-page of Language.
To reorient, go back here:

• Language — multilingual processing and semantic alignment
• WFGY Global Fix Map — main Emergency Room, 300+ structured fixes
• WFGY Problem Map 1.0 — 16 reproducible failure modes

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

Stabilize multilingual retrieval when the query and the index segment text differently. Typical pain shows up in Chinese, Japanese, Thai, Khmer, and any mixed-script corpus where whitespace is unreliable.

---

Open these first

• Visual map and recovery: rag-architecture-and-recovery.md
• Chunking checklist: chunking-checklist.md
• Retrieval traceability: retrieval-traceability.md
• Schema fence for snippets: data-contracts.md
• Embedding vs meaning: embedding-vs-semantic.md

Related multilingual pages in this folder:

• Guide overview: multilingual_guide.md
• Script direction and mixing: script_mixing.md
• Locale and analyzer drift: locale_drift.md
• HyDE behavior across languages: hyde_multilingual.md

---

Core acceptance targets

• ΔS(question, retrieved) ≤ 0.45 for both English and target language
• Coverage of target section ≥ 0.70 after repair
• λ remains convergent across three paraphrases in the target language
• E_resonance flat across 50+ queries that mix scripts and numbers

---

What this failure looks like

Symptom	Likely cause	Where to fix
Chinese or Japanese queries return no hits, while English paraphrase works	Query tokenizer uses whitespace, index uses character n-gram or vice versa	Switch to language-aware analyzers; unify query and index pipelines
BM25 recall ok but citations land in the wrong sub-section	Token boundary misalignment between chunker and retriever	Rechunk with stable boundaries and same segmentation as the store
Hybrid retrieval underperforms a single retriever	Mixed analyzers per stage, reranker sees inconsistent text	Normalize text pre-rerank and re-embed with the same tokenizer
High similarity yet wrong meaning	Embedding model trained with different normalization or casing rules	Re-embed with consistent normalization; see Embedding ≠ Semantic

---

Fix in 60 seconds

1. Measure ΔS Run the same question in English and the target language. If ΔS differs by more than 0.15, suspect tokenizer mismatch.

2. Probe λ_observe Paraphrase the non-English query three ways. If λ flips between convergent and divergent when you reorder headers, lock the prompt schema and proceed to analyzer unification.

3. Apply the smallest structural change

• If your store is lexical: set the same analyzer for both index and query.
• If your store is vector-only: normalize and segment before embedding, then re-embed a small gold set and verify.

4. Verify Coverage ≥ 0.70 and ΔS ≤ 0.45 on three paraphrases. Log the analyzer and normalization used.

---

Minimal repair recipes by stack

Elasticsearch / OpenSearch

• Pick one analyzer for CJK fields and use it for both indexing and queries. Options that work in practice:

  • Japanese: kuromoji
  • Korean: nori
  • Chinese: smartcn or ICU + bigram filter

• Add a keyword subfield for exact filters and rerank features.

• Normalize to NFC and convert full-width to half-width for digits and ASCII.

• Rebuild only the affected indices, then re-run the gold set. See: retrieval-playbook.md

BM25 in code or light stores (Chroma, sqlite FTS, etc.)

• For CJK and Thai, use character bigrams or trigrams on both index and query.
• Remove language-specific stopwords when language is unknown.
• Keep punctuation normalization consistent across stages. See: pattern_query_parsing_split.md

Vector stores (FAISS, Milvus, pgvector, Weaviate, Qdrant)

• Do not trust the model to “fix” segmentation. Pre-normalize text:

  • Unicode NFC
  • Lowercase where appropriate
  • Full-width to half-width for numbers and ASCII
  • Optional: insert spaces between CJK and ASCII tokens for stable sentencepiece

• Re-embed both corpus and queries with the exact same normalization script.

• If recall is still low, add a lexical recall stage, then rerank. See: vectorstore-fragmentation.md

---

Diagnostic checklist

• Query and index use the same tokenizer or analyzer.
• Chunker segmentation matches the retriever segmentation.
• Unicode form is consistent; half-width and diacritics normalized.
• Mixed scripts do not flip direction or join tokens incorrectly.
• Rerank stage sees normalized text, not raw captures.

---

Copy-paste tests

Three-language ΔS probe

Question: "<your question>"
Languages: English, Chinese, Japanese

For each language:
1. Retrieve top-k with your current settings.
2. Compute ΔS(question, retrieved). Record λ_state.
3. If ΔS differs by > 0.15 across languages, suspect tokenizer mismatch.
Return a table: language, ΔS, λ_state, analyzer/normalizer used.

Schema lock for reruns

Header order: citations → facts → synthesis → caveats.
If λ flips when you change header order, lock this schema and fix analyzers first.

---

When to escalate

• ΔS remains ≥ 0.60 after analyzer unification. Re-chunk with stable boundaries and re-embed a gold slice. Open: chunking-checklist.md

• Citations still jump across sections after repair. Enforce snippet schema and forbid cross-section reuse. Open: data-contracts.md, retrieval-traceability.md

• Hybrid beats single only sometimes. Align analyzers in both stages and rerank deterministically. Open: rerankers.md

---

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

---

Explore More

Layer	Page	What it’s for
⭐ Proof	WFGY Recognition Map	External citations, integrations, and ecosystem proof
⚙️ Engine	WFGY 1.0	Original PDF tension engine and early logic sketch (legacy reference)
⚙️ Engine	WFGY 2.0	Production tension kernel for RAG and agent systems
⚙️ Engine	WFGY 3.0	TXT based Singularity tension engine (131 S class set)
🗺️ Map	Problem Map 1.0	Flagship 16 problem RAG failure taxonomy and fix map
🗺️ Map	Problem Map 2.0	Global Debug Card for RAG and agent pipeline diagnosis
🗺️ Map	Problem Map 3.0	Global AI troubleshooting atlas and failure pattern map
🧰 App	TXT OS	.txt semantic OS with fast bootstrap
🧰 App	Blah Blah Blah	Abstract and paradox Q&A built on TXT OS
🧰 App	Blur Blur Blur	Text to image generation with semantic control
🏡 Onboarding	Starter Village	Guided entry point for new users

If this repository helped, starring it improves discovery so more builders can find the docs and tools.