Code-Switching Eval — Gold Set and Probes

A compact harness to verify retrieval and reasoning under code-switching and multilingual inputs. Use this to measure ΔS and λ across three variants of each question: source language, target language, and mixed script or code-switched.

Open these first

Visual map and recovery → rag-architecture-and-recovery.md
End to end retrieval knobs → retrieval-playbook.md
Why this snippet, how to cite → retrieval-traceability.md
Snippet schema fence → data-contracts.md
Embedding vs meaning → embedding-vs-semantic.md
Chunk boundary sanity → chunking-checklist.md
Tokenizer issues → tokenizer_mismatch.md
Mixed scripts inside one query → script_mixing.md
Locale normalization and variants → locale_drift.md
End to end recipes → multilingual_guide.md

Acceptance targets

ΔS(question, retrieved) ≤ 0.45 for each of the three variants
Coverage of the target section ≥ 0.70
λ remains convergent across three paraphrases and two seeds
E_resonance stays flat on long windows that mix scripts

Gold set format

Create a small file you can run in any stack. CSV or JSONL both fine. Recommended CSV schema:

id,question_src,question_l2,question_mix,anchor_doc_id,anchor_offsets,answer_short,notes
ex001,"what is the dosage of abc?","藥品abc的劑量是多少？","abc dosage要怎麼算？",doc_042,"[120, 180]","10 mg BID","CJK mix"

Rules

question_src is your primary language.
question_l2 is the same meaning in the second language.
question_mix is a natural code-switch that users would actually type.
anchor_doc_id and anchor_offsets point to the authoritative span.
Keep 20 to 50 rows for a first pass.

60-second runner outline

You can paste this into any quick script. Replace the retrieval call with your own client.

# minimal pseudo-runner for ΔS and λ probes
from collections import defaultdict

K_LIST = [5, 10, 20]
VARIANTS = ["question_src", "question_l2", "question_mix"]

def retrieve(q, k):
    # call your store, return list of {doc_id, offsets, text}
    return call_retriever(query=q, top_k=k)

def delta_s(a, b):
    # use your ΔS implementation. lower is better.
    return compute_delta_s(a, b)

def lambda_state(history):
    # classify →, ←, <>, × based on stability of attention or order
    return classify_lambda(history)

def run_row(row):
    out = []
    for v in VARIANTS:
        q = row[v]
        lambda_hist = []
        best = None
        for k in K_LIST:
            hits = retrieve(q, k)
            if not hits:
                lambda_hist.append("×")
                continue
            # pick your best hit by rerank or score
            best = hits[0]
            s_qr = delta_s(q, best["text"])
            s_ra = delta_s(best["text"], anchor_text(row["anchor_doc_id"], row["anchor_offsets"]))
            lambda_hist.append((k, s_qr, s_ra))
        out.append({
            "variant": v,
            "lambda": lambda_state(lambda_hist),
            "s_qr@5": lambda_hist[0][1] if lambda_hist and isinstance(lambda_hist[0], tuple) else 1.0,
            "s_qr@10": lambda_hist[1][1] if len(lambda_hist) > 1 and isinstance(lambda_hist[1], tuple) else 1.0,
            "s_qr@20": lambda_hist[2][1] if len(lambda_hist) > 2 and isinstance(lambda_hist[2], tuple) else 1.0,
        })
    return out

Output columns you want in a flat table

id, variant, λ_state, ΔS@5, ΔS@10, ΔS@20, pass
Mark pass when ΔS ≤ 0.45 and λ is convergent.

Probe plan

Lock analyzers Use the same normalization at index and query. If you must change anything, run the whole gold set again. Only one variable at a time.
Triplet run For each row run question_src, question_l2, question_mix at k ∈ {5, 10, 20}. Log ΔS and λ per variant.
Three-paraphrase run For the worst failing variant, add three small paraphrases in the same language. If λ flips while citations stay valid, apply schema locks and BBAM.
Anchor triangulation Compare ΔS to the anchor span and to a decoy in the same document. If both similar, re-chunk and re-embed.

Decision rules

Only the mixed variant fails Suspect script mixing or analyzer split. Open script_mixing.md. Also verify you did not change casing or width in only one path.
L2 fails, source and mix pass Likely tokenizer mismatch or missing segmenter. Open tokenizer_mismatch.md.
zh-Hans vs zh-Hant do not co-retrieve Add variant mapping and normalize digits and punctuation. Open locale_drift.md.
All three variants fail with high similarity Embedding is not multilingual or metric is wrong. Open embedding-vs-semantic.md and the Retrieval Playbook to rebuild.
Hybrid worse than single Fix query parsing split and reranker weights. Open patterns/pattern_query_parsing_split.md and rerankers.md.

Copy-paste prompt for the LLM judge step

You have TXTOS and the WFGY Problem Map loaded.

Context:
- gold id: {id}
- variant: {variant}
- retrieved snippet: "{snippet}"
- expected anchor: "{anchor_excerpt}"
- ΔS(question, retrieved) = {s_qr}
- ΔS(retrieved, anchor) = {s_ra}

Tasks:
1) Decide pass or fail for retrieval alignment. If fail, name the failing layer.
2) Return the exact WFGY page to fix it.
3) Give a minimal repair plan with 3 steps.
4) Output JSON only:
   {"pass": true|false, "fail_layer": "...", "open": [".../page1.md",".../page2.md"], "plan": ["step1","step2","step3"]}

Minimal dataset checklist

At least one trilingual example with names that have native and romanized forms
At least one code-switch that uses Latin terms inside CJK or Indic sentences
At least one RTL or diacritic heavy example
Keep a small “sanity” row that always passes after a clean rebuild

What good looks like

Pass rate ≥ 90 percent on question_src and question_l2
Mixed variant pass rate ≥ 80 percent after analyzer unification
No λ flip in the three-paraphrase probe for the same variant
ΔS median ≤ 0.40 on all variants after fixes

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

🧭 Explore More

Module	Description	Link
WFGY Core	WFGY 2.0 engine is live: full symbolic reasoning architecture and math stack	View →
Problem Map 1.0	Initial 16-mode diagnostic and symbolic fix framework	View →
Problem Map 2.0	RAG-focused failure tree, modular fixes, and pipelines	View →
Semantic Clinic Index	Expanded failure catalog: prompt injection, memory bugs, logic drift	View →
Semantic Blueprint	Layer-based symbolic reasoning & semantic modulations	View →
Benchmark vs GPT-5	Stress test GPT-5 with full WFGY reasoning suite	View →
Starter Village	New here? Lost in symbols? Click here and let the wizard guide you through	Start →

👑 Early Stargazers: See the Hall of Fame — Engineers, hackers, and open source builders who supported WFGY from day one. Hall of Fame • WFGY Engine 2.0 • Unlock Board

12 KiB Raw Blame History Unescape Escape