mirror of
https://github.com/onestardao/WFGY.git
synced 2026-04-28 11:40:07 +00:00
Create input_language_switching.md
This commit is contained in:
PSBigBig
GitHub
parent
137f21066d
commit
657157fc5e
1 changed files with 192 additions and 0 deletions
|
|
@ -0,0 +1,192 @@
|
|||
# Input Language Switching — Guardrails and Fix Pattern
|
||||
|
||||
Use this page when the **user’s keyboard or input language flips mid-thread** and retrieval or reasoning starts drifting. Classic signs include mixed scripts in one query, wrong tokenizer chosen by the stack, or partial IME composition text being sent to the server.
|
||||
|
||||
## When to use
|
||||
|
||||
* Users alternate between EN ↔ CJK ↔ RTL in the same session.
|
||||
* Queries look semantically fine but **retriever recall collapses after a language flip**.
|
||||
* You see **half words** or odd segments caused by IME composition text being submitted.
|
||||
* Autocorrect or predictive text silently changes script or diacritics.
|
||||
|
||||
## Acceptance targets
|
||||
|
||||
* ΔS(question, retrieved) ≤ 0.45 across **3 paraphrases** in the active language.
|
||||
* Coverage of target section ≥ 0.70 after the switch.
|
||||
* λ remains **convergent** when toggling `lang=xx-BCP47` between the last two user turns.
|
||||
* E\_resonance stays flat over 20-40 turns.
|
||||
|
||||
---
|
||||
|
||||
## Open these first
|
||||
|
||||
* Language tokenization mismatches
|
||||
→ [tokenizer\_mismatch.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/tokenizer_mismatch.md)
|
||||
* Script mixing in a single query
|
||||
→ [script\_mixing.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/script_mixing.md)
|
||||
* Diacritics and normalization policy
|
||||
→ [diacritics\_and\_folding.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/diacritics_and_folding.md)
|
||||
* Locale drift across turns
|
||||
→ [locale\_drift.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/locale_drift.md)
|
||||
* CJK segmentation specifics
|
||||
→ [cjk\_segmentation\_wordbreak.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/cjk_segmentation_wordbreak.md)
|
||||
* Contract your snippet schema
|
||||
→ [data-contracts.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/data-contracts.md)
|
||||
* End to end retrieval knobs
|
||||
→ [retrieval-playbook.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-playbook.md)
|
||||
|
||||
---
|
||||
|
||||
## Diagnose quickly
|
||||
|
||||
1. **Spot the flip**
|
||||
Log BCP-47 tags per turn. If `lang_prev != lang_now`, record a switch event with `t_switch`.
|
||||
|
||||
2. **IME composition fence**
|
||||
Check inputs for composition markers or incomplete tokens. If composition not complete, **do not send** to retrieval or indexing.
|
||||
|
||||
3. **Tokenizer audit**
|
||||
Confirm the retriever tokenizer matches `lang_now`. English-only analyzers on CJK or RTL usually cause flat high ΔS.
|
||||
|
||||
4. **Punctuation and width**
|
||||
Look for full-width digits or punctuation introduced by the keyboard. If present, apply the width normalizer from the punctuation page.
|
||||
Open: [digits\_width\_punctuation.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/digits_width_punctuation.md)
|
||||
|
||||
5. **Direction and collation**
|
||||
If RTL or locale sorting changed, set `dir` and collation explicitly in the request.
|
||||
Open: existing RTL guide and collation page:
|
||||
[locale\_collation\_and\_sorting.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/locale_collation_and_sorting.md)
|
||||
|
||||
---
|
||||
|
||||
## Fix pattern
|
||||
|
||||
**A. Add an Input Contract to every user turn**
|
||||
|
||||
Required fields you attach to the message payload before retrieval:
|
||||
|
||||
```json
|
||||
{
|
||||
"raw_text": "...",
|
||||
"normalized_text": "...",
|
||||
"lang_bcp47": "xx-YY",
|
||||
"script": "Latn|Cyrl|Arab|Hans|Hant|Kana|... ",
|
||||
"dir": "ltr|rtl",
|
||||
"ime_composition_complete": true,
|
||||
"keyboard_hint": "gboard|ios|system|unknown"
|
||||
}
|
||||
```
|
||||
|
||||
* `normalized_text` must follow your **LanguageLocale** normalization policy
|
||||
See: [diacritics\_and\_folding.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/diacritics_and_folding.md)
|
||||
|
||||
**B. Route to the correct analyzer**
|
||||
|
||||
* Select **tokenizer/analyzer** based on `lang_bcp47` and `script`.
|
||||
* For CJK, switch to CJK-aware analyzer and re-ranker; for mixed text, run a **language split** then merge with a deterministic re-rank.
|
||||
See: [cjk\_segmentation\_wordbreak.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/cjk_segmentation_wordbreak.md) and [rerankers.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rerankers.md)
|
||||
|
||||
**C. Guard IME composition**
|
||||
|
||||
* Add a client or gateway **composition fence**. If `ime_composition_complete=false`, buffer and wait.
|
||||
* Only proceed to retrieval once confirmed complete.
|
||||
|
||||
**D. Stabilize semantics after the flip**
|
||||
|
||||
* Run a **three-paraphrase probe** in the active language. If ΔS stays flat and high, rebuild or switch index metric.
|
||||
Open: [embedding-vs-semantic.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/embedding-vs-semantic.md)
|
||||
Then follow: [retrieval-playbook.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-playbook.md)
|
||||
|
||||
---
|
||||
|
||||
## Minimal recipe you can copy
|
||||
|
||||
1. Detect `lang_bcp47` and `script` on the **client**.
|
||||
2. Normalize text using your LanguageLocale policy.
|
||||
3. If IME not complete, **do nothing**. Wait until completion.
|
||||
4. Choose the analyzer and reranker given `lang_bcp47`.
|
||||
5. Retrieve with schema-locked snippets and store: `lang_bcp47`, `script`, `dir`.
|
||||
6. Probe ΔS on 3 paraphrases in the active language. Clamp λ if needed.
|
||||
7. Enforce cite-then-explain with **snippet and citation schema**.
|
||||
Open: [retrieval-traceability.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-traceability.md) and [data-contracts.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/data-contracts.md)
|
||||
|
||||
---
|
||||
|
||||
## Copy-paste prompt for the LLM step
|
||||
|
||||
```
|
||||
You have TXTOS and WFGY Problem Map loaded.
|
||||
|
||||
Input meta:
|
||||
- lang_bcp47 = {xx-YY}
|
||||
- script = {Latn|Hans|Hant|Arab|...}
|
||||
- dir = {ltr|rtl}
|
||||
- ime_composition_complete = {true|false}
|
||||
|
||||
Tasks:
|
||||
1) If ime_composition_complete=false, return: {"action":"wait_for_ime"} only.
|
||||
2) Use the active language for paraphrases and answers. No silent translation.
|
||||
3) Validate cite-then-explain. If citations missing, return the fix tip and stop.
|
||||
4) If ΔS(question,retrieved) ≥ 0.60, suggest the smallest structural fix referencing:
|
||||
retrieval-playbook, tokenizer_mismatch, script_mixing, diacritics_and_folding.
|
||||
Return JSON:
|
||||
{ "answer":"...", "citations":[...], "ΔS":0.xx, "λ_state":"...", "lang_used":"xx-YY" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Observability
|
||||
|
||||
* Log the **distribution of `lang_bcp47`** across the session. A spike at switch events plus ΔS spikes indicates analyzer mismatch.
|
||||
* Track **ΔS vs k** right after a switch. Flat-high curves suggest metric or index misalignment.
|
||||
* Record `ime_wait_count`. A high count with good outcomes is normal and preferable to composition leakage.
|
||||
|
||||
---
|
||||
|
||||
## Common gotchas
|
||||
|
||||
* Mobile keyboards emit **full-width digits** or punctuation after language switch. Normalize width before retrieval.
|
||||
Open: [digits\_width\_punctuation.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/digits_width_punctuation.md)
|
||||
* Predictive text inserts **diacritics** that your index fold does not expect. Align fold policy.
|
||||
Open: [diacritics\_and\_folding.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/diacritics_and_folding.md)
|
||||
* Mixed language metadata causes wrong **reroute** to an English index. Validate `lang_bcp47` at the gateway.
|
||||
Open: [mixed\_locale\_metadata.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/GlobalFixMap/LanguageLocale/mixed_locale_metadata.md)
|
||||
|
||||
---
|
||||
|
||||
## Escalate
|
||||
|
||||
* If ΔS remains ≥ 0.60 after analyzer and normalization fixes, **split the index by language** or switch to a **multilingual embedding** and rebuild with correct metric.
|
||||
Open: [embedding-vs-semantic.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/embedding-vs-semantic.md) and [retrieval-playbook.md](https://github.com/onestardao/WFGY/blob/main/ProblemMap/retrieval-playbook.md)
|
||||
|
||||
---
|
||||
|
||||
**Next suggested page to generate**
|
||||
`ProblemMap/GlobalFixMap/LanguageLocale/nonstandard_whitespace.md`
|
||||
Focus on NBSP, NNBSP, thin-space, zero-width joiner/non-joiner, and their impact on retrieval.
|
||||
|
||||
---
|
||||
|
||||
> 👑 **Early Stargazers: [See the Hall of Fame](https://github.com/onestardao/WFGY/tree/main/stargazers)** —
|
||||
> Engineers, hackers, and open source builders who supported WFGY from day one.
|
||||
|
||||
> <img src="https://img.shields.io/github/stars/onestardao/WFGY?style=social" alt="GitHub stars"> ⭐ [WFGY Engine 2.0](https://github.com/onestardao/WFGY/blob/main/core/README.md) is already unlocked. ⭐ Star the repo to help others discover it and unlock more on the [Unlock Board](https://github.com/onestardao/WFGY/blob/main/STAR_UNLOCKS.md).
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://github.com/onestardao/WFGY)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlahBlahBlah)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlotBlotBlot)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlocBlocBloc)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlurBlurBlur)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlowBlowBlow)
|
||||
|
||||
|
||||
</div>
|
||||
Loading…
Add table
Add a link
Reference in a new issue