vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-04-28 11:40:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
WFGY/ProblemMap/GlobalFixMap/Language/script_mixing.md

10 KiB
Raw Blame History

Script Mixing — Guardrails and Fix Patterns

🧭 Quick Return to Map

You are in a sub-page of Language.
To reorient, go back here:

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

Keep retrieval stable when a single query or snippet mixes scripts and directions. Common cases: CJK + Latin acronyms, Arabic or Hebrew with numbers and English terms, Devanagari with Latin product names, and datasets where full-width digits appear beside half-width ASCII.

Open these first

Related in this folder:

Core acceptance targets

  • ΔS(question, retrieved) ≤ 0.45 for mixed-script queries
  • Coverage of the target section ≥ 0.70 after repair
  • λ remains convergent across three paraphrases that include different script orderings
  • E_resonance flat on long windows with numerals, punctuation, and brand names mixed in

What this failure looks like

Symptom Likely cause Where to fix
Arabic or Hebrew queries return partial hits or broken citations Bidirectional marks and numerals flip visual order; analyzer not bidi-aware Normalize directionality and digits before indexing and querying
CJK text with Latin acronyms splits unpredictably Mixed width digits, zero-width chars, or inconsistent spacing rules Pre-normalize width, strip zero-width, add script-boundary spacing for embedding
English brand + Thai sentence retrieves far sections Different analyzers per stage cause token joins and drops Unify analyzer and pre-segment at script transitions
High similarity but wrong meaning on acronyms Casing and width normalization inconsistent between corpus and query Apply the same ASCII, width, and case rules in both pipelines

Fix in 60 seconds

  1. Measure ΔS Run the original mixed-script query and a variant where scripts are separated by spaces. If ΔS improves by ≥ 0.10, you have a script-mixing normalization gap.

  2. Probe λ_observe Swap the order of scripts in the query, keep semantics identical. If λ flips or citations jump, lock prompt headers and fix normalization and analyzer alignment first.

  3. Apply the smallest structural change

  • Normalize Unicode to NFC, convert full-width to half-width for digits and ASCII.
  • Remove zero-width characters, directional isolates from raw text.
  • Ensure the same analyzer is used for both index and query, or pre-segment before embedding.
  1. Verify Coverage ≥ 0.70 and ΔS ≤ 0.45 on three paraphrases with different script orders.

Minimal repair recipes by stack

Elasticsearch / OpenSearch

  • Use ICU chain for mixed scripts. Typical pipeline: icu_normalizer (NFC) → icu_transform (full-width to half-width) → icu_folding → optional CJK bigram filter.
  • For Arabic or Persian add arabic_normalization or persian_normalization.
  • Strip bidi control chars in a char filter.
  • Set the same analyzer for index and search_analyzer on the field.
  • Create a keyword subfield for exact acronyms and model names. Reference: retrieval-playbook.md

BM25 in code or light stores

  • Preprocess text with a normalization step that performs: Unicode NFC, width fold for digits and ASCII, lowercasing where safe, removal of zero-width and bidi marks.
  • For CJK, insert temporary spaces at script boundaries or use character bigrams for both index and query.
  • Keep identical punctuation rules across stages. Open: pattern_query_parsing_split.md

Vector stores (FAISS, Milvus, Qdrant, Weaviate, pgvector)

  • Normalize text before embedding with the same script rules for corpus and queries.
  • Add lightweight lexical recall (BM25) to catch brand names and numerals, then rerank deterministically.
  • Re-embed only a gold slice to validate, then batch the full rebuild. Open: vectorstore-fragmentation.md

Diagnostic checklist

  • The same normalization code runs for ingest and query.
  • Width folding, casing, digit policy are identical across stages.
  • Bidi control marks removed or isolated consistently.
  • Chunk boundaries do not split inside script transitions that carry meaning.
  • Rerank stage views the normalized text, not raw captures.

Copy-paste tests

Script order probe

Q0: original mixed-script query
Q1: same words, scripts reordered
Q2: same words, add spaces at script boundaries

Return a table with ΔS per query, λ_state, and whether citations stayed in the same section.

Bidi and width sanity

Given a sentence with Arabic text, ASCII digits, and an English acronym:
1) Remove bidi marks and normalize widths.
2) Show the token sequence used by the retriever.
3) Verify that numbers appear in logical order and acronyms stay intact.

When to escalate

  • ΔS remains ≥ 0.60 after normalization and analyzer unification. Re-chunk with stable boundaries and re-embed a gold slice. Open: chunking-checklist.md

  • Citations still jump between sections on mixed-script inputs. Enforce snippet schema and forbid cross-section reuse. Open: data-contracts.md, retrieval-traceability.md

  • Hybrid retrieval underperforms a single retriever. Align normalization rules before rerank, and make rerank deterministic. Open: rerankers.md

🔗 Quick-Start Downloads (60 sec)

Tool Link 3-Step Setup
WFGY 1.0 PDF Engine Paper 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS) TXTOS.txt 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

Explore More

Module Description Link
WFGY Core Canonical framework entry point View
Problem Map Diagnostic map and navigation hub View
Tension Universe Experiments MVP experiment field View
Recognition Where WFGY is referenced or adopted View
AI Guide Anti-hallucination reading protocol for tools View

If this repository helps, starring it improves discovery for other builders.
GitHub Repo stars