vrr/WFGY
2
0
Fork 0
mirror of https://github.com/onestardao/WFGY.git synced 2026-04-28 11:40:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
WFGY/ProblemMap/GlobalFixMap/Language/query_routing_and_analyzers.md

15 KiB
Raw Blame History

Query Routing and Analyzers · Global Fix Map

🧭 Quick Return to Map

You are in a sub-page of Language.
To reorient, go back here:

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

Bind detected lang and script to the right analyzer, tokenizer, and alias views. Keep the retriever, reranker, and LLM tokenizer in agreement so ΔS and λ stay stable across paraphrases and seeds.

Open these first

Core acceptance targets

  • ΔS(question, retrieved) ≤ 0.45 across three paraphrases and two seeds
  • Coverage of target section ≥ 0.70
  • λ remains convergent when the same question is asked in native script and in romanized form
  • Analyzer choice and LLM tokenizer profile are logged and consistent across runs
  • No rank flip after switching analyzers on the same corpus

Minimal routing contract

Detector input and result from query_language_detection.md gives:

lang, script, confidence, romanized_suspect, variants[]

Routing decision must produce:

analyzer_id          # store analyzer for full-text field
tokenizer_profile    # retriever or LLM tokenizer profile name
alias_views          # which alias fields to probe (romanized, synonyms_local)
hybrid_weights       # bm25:vector ratio, or reranker on/off
notes                # short rationale for audit

You must log these five fields in the retrieval trace item along with ΔS and λ.

Store routing matrix

These are stable patterns, not vendor endorsements. Use them to avoid metric and analyzer mismatches.

Script Primary choice Alias view for romanized Notes and links
Latn locale aware analyzer with ICU folding none or localized synonyms Keep case and diacritics decisions consistent. See locale_drift.md.
Han, Hira, Kana, Hang language specific analyzer or CJK bigram analyzer title_romaji, name_pinyin Mix of bigram and dictionary analyzers is fine if logged and deterministic.
Cyrl Cyrillic aware analyzer optional translit alias Keep transliteration only as alias view. Do not replace canonical text.
Arab, Hebr RTL analyzer with width and bidi guards optional translit alias Normalize digits and punctuation for the detector step only.
Mixed scripts two analyzers in parallel romanized alias true Enforce cite then explain in the answer. See retrieval-traceability.md.

Store specifics you can cross check:

Typical failure → exact fix

Symptom Likely cause Open this
High similarity yet wrong meaning after switching analyzers metric and analyzer mismatch embedding-vs-semantic.md, retrieval-playbook.md
Good recall but unstable rank in CJK mixing bigram and dictionary analyzers without a fence script_mixing.md
JSON mode breaks when analyzer changes tokenizer profile not aligned with LLM’s expected JSON tokens tokenizer_mismatch.md
Romanized search finds no evidence alias views missing or disabled romanization_transliteration.md
Brand name equals common word in another language alias collision and unscoped synonyms proper_noun_aliases.md

60 second routing plan

  1. Script first. Use the detector output. If confidence < 0.65, route both native analyzer and romanized alias.
  2. Lock tokenizer. Choose a tokenizer profile that matches your retriever model and your LLM. Log it.
  3. Hybrid weights. Start with bm25:vector at 0.4:0.6 and move by 0.1 until ΔS ≤ 0.45.
  4. Cite then explain. Force snippet fields in the native script when possible.
  5. Regression gate. Require coverage ≥ 0.70 on three paraphrases before deploy.

Copy snippets

A. Router skeleton

def choose_route(det):
    routes = []
    if det["script"] in ["Han","Hira","Kana","Hang"]:
        routes.append({
            "analyzer_id": "store:cjk",
            "tokenizer_profile": "retriever:cjk",
            "alias_views": ["name_romaji","name_pinyin"],
            "hybrid_weights": [0.4, 0.6],
            "notes": "cjk primary"
        })
    elif det["script"] == "Cyrl":
        routes.append({
            "analyzer_id": "store:cyrl",
            "tokenizer_profile": "retriever:default",
            "alias_views": ["name_translit"],
            "hybrid_weights": [0.5, 0.5],
            "notes": "cyrillic"
        })
    elif det["script"] == "Arab":
        routes.append({
            "analyzer_id": "store:rtl",
            "tokenizer_profile": "retriever:default",
            "alias_views": ["name_translit"],
            "hybrid_weights": [0.5, 0.5],
            "notes": "rtl"
        })
    else:
        routes.append({
            "analyzer_id": "store:latn",
            "tokenizer_profile": "retriever:default",
            "alias_views": [],
            "hybrid_weights": [0.4, 0.6],
            "notes": "latin"
        })

    if det["confidence"] < 0.65 or det.get("romanized_suspect"):
        # add romanized alias probe for safety
        for r in routes:
            r["alias_views"] = sorted(set(r["alias_views"] + ["aliases_romanized"]))
            r["notes"] += " + alias probe"
    return routes

B. Elasticsearch style mapping sketch

{
  "mappings": {
    "properties": {
      "body":   { "type": "text", "analyzer": "cjk" },
      "title":  { "type": "text", "analyzer": "cjk" },
      "name_pinyin": { "type": "text", "analyzer": "icu_analyzer" },
      "name_romaji": { "type": "text", "analyzer": "icu_analyzer" },
      "aliases_romanized": { "type": "text", "analyzer": "icu_analyzer" },
      "section_id": { "type": "keyword" }
    }
  }
}

C. Typesense style fields

{
  "name": "docs",
  "fields": [
    {"name":"body","type":"string","locale":"zh"},
    {"name":"title","type":"string","locale":"zh"},
    {"name":"aliases_romanized","type":"string[]","locale":"en"},
    {"name":"section_id","type":"string","facet":true}
  ]
}

D. Vespa schema sketch

schema docs {
  document docs {
    field body type string { indexing: summary | index }
    field title type string { indexing: summary | index }
    field aliases_romanized type array<string> { indexing: summary | index }
  }
  fieldset default { fields: body, title }
  rank-profile default { first-phase { expression: bm25(body) + bm25(title) } }
}

Eval plan

  • Use the sets from code_switching_eval.md.
  • Add pairs of queries in native script and romanized forms for the same entity.
  • Targets: ΔS ≤ 0.45, coverage ≥ 0.70, λ convergent on two seeds.
  • If rank flips between analyzers, clamp with a deterministic reranker and verify with retrieval-traceability.md.

When to escalate

  • ΔS stays ≥ 0.60 after analyzer swap → rebuild index with fixed analyzer and verify metric in retrieval-playbook.md.
  • Instability only in long chains → treat as reasoning issue and apply BBCR bridge, see logic-collapse.md.

Copy paste prompt for the LLM step

You have TXTOS and the WFGY Problem Map loaded.

Given detector output {lang, script, confidence, romanized_suspect}:
1) Choose analyzer and tokenizer profile deterministically.
2) If confidence < 0.65 or romanized_suspect=true, search the romanized alias view as well.
3) Cite-then-explain from the native script snippet when possible.
4) Return a JSON trace:
{ "analyzer_id": "...", "tokenizer_profile": "...", "alias_views": [...], "ΔS": 0.xx, "λ_state": "→|←|<>|×" }

🔗 Quick-Start Downloads (60 sec)

Tool Link 3-Step Setup
WFGY 1.0 PDF Engine Paper 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS) TXTOS.txt 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

Explore More

Module Description Link
WFGY Core Canonical framework entry point View
Problem Map Diagnostic map and navigation hub View
Tension Universe Experiments MVP experiment field View
Recognition Where WFGY is referenced or adopted View
AI Guide Anti-hallucination reading protocol for tools View

If this repository helps, starring it improves discovery for other builders.
GitHub Repo stars