mirror of
https://github.com/onestardao/WFGY.git
synced 2026-04-28 11:40:07 +00:00
Create pattern_query_parsing_split.md
This commit is contained in:
PSBigBig
GitHub
parent
67c5e15ea3
commit
168623ec03
1 changed files with 314 additions and 0 deletions
314
ProblemMap/patterns/pattern_query_parsing_split.md
Normal file
314
ProblemMap/patterns/pattern_query_parsing_split.md
Normal file
|
|
@ -0,0 +1,314 @@
|
|||
# Pattern — Query Parsing Split (Multi-Intent / Wrong Sub-Intent First)
|
||||
|
||||
**Scope**
|
||||
A single user query actually contains **multiple intents** (lookup + policy + transformation + generation), but the pipeline treats it as **one** retrieval/generation ask. The system answers the **easiest/earliest** sub-intent and ignores the rest, or mixes intents, causing off-topic retrieval and wrong acceptance decisions.
|
||||
|
||||
**Why it matters**
|
||||
Multi-intent queries are common (“compare A vs B and give a summary with citations”). If you don’t split, retrieval pools and prompts blur constraints, you get **false grounding**, and audit trails become meaningless (“which intent did this citation serve?”).
|
||||
|
||||
---
|
||||
|
||||
## 1) Signals & Fast Triage
|
||||
|
||||
**Likely symptoms**
|
||||
- The answer handles **only part** of the question (e.g., explains A but not B or the comparison).
|
||||
- Retrieved chunks mix unrelated facets (policy + tutorial + changelog) → noisy context, low CHR.
|
||||
- Auditor (Example 04) flips `VALID` ↔ `NOT_IN_CONTEXT` depending on which fragment the model latched onto.
|
||||
- Example 02 labels skew to `query_parse_error`.
|
||||
|
||||
**Deterministic checks (no LLM)**
|
||||
- **Separator heuristics**: query contains `and`, `vs`, `;`, `,`, numbered lists `1) 2)`, or colon-scoped asks (“X: do Y, then Z”).
|
||||
- **Verb phase count**: ≥2 finite verbs across different objects (`compare`, `explain`, `implement`, `deploy`).
|
||||
- **Constraint tokens**: presence of at least one **data** intent (`find`, `lookup`, `cite`) and one **action** intent (`summarize`, `generate`, `rewrite`).
|
||||
If ≥2 signals → treat as **multi-intent** and split.
|
||||
|
||||
---
|
||||
|
||||
## 2) Minimal Reproducible Case
|
||||
|
||||
`data/chunks.json`:
|
||||
|
||||
```json
|
||||
[
|
||||
{"id":"pA#1","text":"Policy A: Only domain example.com is allowed."},
|
||||
{"id":"pB#1","text":"Policy B: Allow *.company.com and partner domains."},
|
||||
{"id":"pC#1","text":"How to edit email settings in the dashboard."}
|
||||
]
|
||||
````
|
||||
|
||||
User query:
|
||||
**“Compare Policy A vs B with citations, then draft an email asking IT to switch our domain.”**
|
||||
|
||||
Naive pipelines either:
|
||||
|
||||
1. Summarize A **or** B only, **or**
|
||||
2. Draft the email **without** grounded comparison.
|
||||
|
||||
---
|
||||
|
||||
## 3) Root Causes
|
||||
|
||||
* **Single-turn monolith**: retrieval runs once on the whole sentence; constraints collide.
|
||||
* **No intent schema**: pipeline can’t represent “first compare (grounded), then draft (un-grounded).”
|
||||
* **Prompt overloading**: one template tries to do comparison + generation + policy proof.
|
||||
* **Acceptance gate blind**: Auditor validates a claim that mixes two intents.
|
||||
|
||||
---
|
||||
|
||||
## 4) Standard Fix (Ordered, Minimal, Measurable)
|
||||
|
||||
**Step 1 — Detect & Split**
|
||||
|
||||
* Run deterministic heuristics (Section 1) to produce **sub-intents** with **roles**: `COMPARE`, `LOOKUP`, `DRAFT`, `REWRITE`, etc.
|
||||
* Each sub-intent gets its **own** retrieval pool and acceptance rule.
|
||||
|
||||
**Step 2 — Bind Contracts per Sub-Intent**
|
||||
|
||||
* Evidence-only template for **grounded** intents (`COMPARE`, `LOOKUP`) → requires `citations: [id,...]`.
|
||||
* Free-form template for **creative** intents (`DRAFT`) → must **echo** the grounded summary id (handoff contract) but does **not** add new citations.
|
||||
|
||||
**Step 3 — Sequence with Handoffs**
|
||||
|
||||
* Output of grounded step → `summary.claim`, `citations`.
|
||||
* Draft step **may** rephrase but cannot introduce new factual claims; it references the **handoff id**.
|
||||
|
||||
**Step 4 — Accept or Refuse**
|
||||
|
||||
* Accept only if **grounded** step is `VALID` (Example 04) **and** draft step references the correct handoff id.
|
||||
* If grounded step is `NOT_IN_CONTEXT`, overall request returns refusal with explanation.
|
||||
|
||||
**Step 5 — Evaluate**
|
||||
|
||||
* Example 08: score **per-intent** precision and CHR; drafts are graded on **schema compliance**, not truth.
|
||||
|
||||
---
|
||||
|
||||
## 5) Reference Implementation — Python (stdlib only)
|
||||
|
||||
Create `tools/intent_split.py`.
|
||||
|
||||
```python
|
||||
# tools/intent_split.py -- rule-based multi-intent splitter + per-intent contracts
|
||||
import re, json, os, time, urllib.request, uuid
|
||||
|
||||
GROUND_REFUSAL = "not in context"
|
||||
|
||||
def split_intents(q: str):
|
||||
text = q.strip()
|
||||
# crude separators
|
||||
parts = re.split(r"\bthen\b|;| and then | && | -> ", text, flags=re.IGNORECASE)
|
||||
intents = []
|
||||
for p in parts:
|
||||
role = "LOOKUP"
|
||||
if re.search(r"\bcompare|vs\b", p, re.IGNORECASE): role = "COMPARE"
|
||||
if re.search(r"\bdraft|email|write|generate|compose\b", p, re.IGNORECASE): role = "DRAFT"
|
||||
intents.append({"id": str(uuid.uuid4())[:8], "role": role, "text": p.strip()})
|
||||
# if single fragment but has both compare + draft keywords, split into two logical intents
|
||||
if len(intents)==1 and re.search(r"\bcompare|vs\b", text, re.IGNORECASE) and re.search(r"\bdraft|email|write|generate\b", text, re.IGNORECASE):
|
||||
intents = [
|
||||
{"id": str(uuid.uuid4())[:8], "role":"COMPARE", "text": text},
|
||||
{"id": str(uuid.uuid4())[:8], "role":"DRAFT", "text": text}
|
||||
]
|
||||
return intents
|
||||
|
||||
def retrieve(chunks, q, k=6):
|
||||
qs = set(w for w in re.split(r"\W+", q.lower()) if len(w)>=3)
|
||||
scored = []
|
||||
for c in chunks:
|
||||
toks = re.split(r"\W+", c["text"].lower())
|
||||
overlap = sum(1 for t in toks if t in qs)
|
||||
scored.append((overlap, c))
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
return [c for s,c in scored[:k]]
|
||||
|
||||
def build_compare_prompt(q, ctx, allowed):
|
||||
ctxs = "\n\n".join(f"[{c['id']}] {c['text']}" for c in ctx)
|
||||
return (
|
||||
"Task: Compare the two policies strictly from EVIDENCE.\n"
|
||||
"Output JSON ONLY: { claim: string, citations: [id,...] }\n"
|
||||
f"If not provable, reply exactly '{GROUND_REFUSAL}'.\n\n"
|
||||
f"Question: {q}\nEVIDENCE:\n{ctxs}\n"
|
||||
)
|
||||
|
||||
def build_draft_prompt(summary_json):
|
||||
return (
|
||||
"Task: Draft a short email referencing the grounded comparison.\n"
|
||||
"You MUST echo {handoff_id} exactly and MUST NOT add new policy facts.\n"
|
||||
"Output JSON ONLY: { email: string, handoff_id: string }\n\n"
|
||||
f"Grounded summary:\n{json.dumps(summary_json)}\n"
|
||||
)
|
||||
|
||||
def call_openai(prompt, model=os.getenv("OPENAI_MODEL","gpt-4o-mini")):
|
||||
key=os.getenv("OPENAI_API_KEY"); assert key, "OPENAI_API_KEY"
|
||||
body = json.dumps({"model":model,"messages":[{"role":"user","content":prompt}],"temperature":0}).encode()
|
||||
req = urllib.request.Request("https://api.openai.com/v1/chat/completions", data=body, headers={"Content-Type":"application/json","Authorization":f"Bearer {key}"})
|
||||
with urllib.request.urlopen(req) as r:
|
||||
j=json.loads(r.read().decode()); return j["choices"][0]["message"]["content"].strip()
|
||||
|
||||
def parse_json(text):
|
||||
s=text.find("{"); e=text.rfind("}")
|
||||
if s<0 or e<=s: return None
|
||||
try: return json.loads(text[s:e+1])
|
||||
except: return None
|
||||
|
||||
def run(q, chunks):
|
||||
turns=[]
|
||||
intents = split_intents(q)
|
||||
handoff=None
|
||||
for it in intents:
|
||||
if it["role"] in ("LOOKUP","COMPARE"):
|
||||
ctx = retrieve(chunks, it["text"], k=6)
|
||||
allowed = [c["id"] for c in ctx]
|
||||
out = parse_json(call_openai(build_compare_prompt(it["text"], ctx, allowed)))
|
||||
if not out or (isinstance(out, dict) and out.get("claim","").strip().lower()==GROUND_REFUSAL):
|
||||
return {"status":"REFUSAL", "reason":"grounding_failed"}
|
||||
# schema & scope checks
|
||||
if not set(out.get("citations",[])).issubset(set(allowed)):
|
||||
return {"status":"REJECT", "reason":"citation_out_of_scope"}
|
||||
handoff = {"handoff_id": str(uuid.uuid4())[:8], "summary": out}
|
||||
turns.append({"intent": it, "ctx_ids": allowed, "out": out, "handoff_id": handoff["handoff_id"]})
|
||||
elif it["role"]=="DRAFT":
|
||||
if not handoff: return {"status":"REJECT", "reason":"draft_without_grounding"}
|
||||
draft = parse_json(call_openai(build_draft_prompt({"handoff_id": handoff["handoff_id"], **handoff["summary"]})))
|
||||
if not draft or draft.get("handoff_id") != handoff["handoff_id"]:
|
||||
return {"status":"REJECT", "reason":"handoff_mismatch"}
|
||||
turns.append({"intent": it, "out": draft, "handoff_id": handoff["handoff_id"]})
|
||||
return {"status":"OK", "turns": turns}
|
||||
|
||||
if __name__=="__main__":
|
||||
chunks = json.load(open("data/chunks.json",encoding="utf8"))
|
||||
print(json.dumps(run("Compare Policy A vs B with citations, then draft an email asking IT to switch our domain.", chunks), indent=2))
|
||||
```
|
||||
|
||||
**Pass criteria**
|
||||
|
||||
* For the sample query, the first turn is a **grounded** `COMPARE` with citations to `pA#1`/`pB#1`.
|
||||
* The `DRAFT` turn echoes the **handoff\_id** and contains no new policy facts.
|
||||
* If comparison is `not in context`, overall **REFUSAL** (no email is drafted).
|
||||
|
||||
---
|
||||
|
||||
## 6) Node Quick Variant (split only, no LLM call)
|
||||
|
||||
Create `tools/intent_split.mjs`.
|
||||
|
||||
```js
|
||||
// tools/intent_split.mjs -- detect multi-intent; emit a small plan
|
||||
export function splitIntents(q){
|
||||
const text = q.trim();
|
||||
const cuts = text.split(/(?:\bthen\b|;| and then | && | -> )/i).map(s=>s.trim()).filter(Boolean);
|
||||
const parts = cuts.length ? cuts : [text];
|
||||
return parts.map(p=>{
|
||||
let role = "LOOKUP";
|
||||
if (/\bcompare|vs\b/i.test(p)) role = "COMPARE";
|
||||
if (/\bdraft|email|write|generate|compose\b/i.test(p)) role = "DRAFT";
|
||||
return { role, text: p };
|
||||
});
|
||||
}
|
||||
|
||||
// CLI
|
||||
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||
const q = process.argv.slice(2).join(" ");
|
||||
console.log(JSON.stringify(splitIntents(q), null, 2));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7) Acceptance Criteria (ship/no-ship)
|
||||
|
||||
A multi-intent response **may ship** only if:
|
||||
|
||||
1. Each **grounded** sub-intent has `citations ⊆ retrieved_ids` and passes Auditor `VALID`.
|
||||
2. **Creative** sub-intents (draft/rewrite) echo a valid `handoff_id` from a `VALID` grounded step.
|
||||
3. If any grounded sub-intent returns `not in context`, the overall request refuses (no partial answers).
|
||||
4. Example 08 per-intent gates pass (CHR for grounded, compliance for drafts).
|
||||
|
||||
---
|
||||
|
||||
## 8) Prevention (contracts & defaults)
|
||||
|
||||
* **Query schema**: `role: {COMPARE|LOOKUP|DRAFT|REWRITE}`, `text`, optional `constraints`.
|
||||
* **Router default**: split when ≥2 deterministic signals fire; otherwise single-intent.
|
||||
* **Template isolation**: distinct prompts per role; never mix compare + draft in the same prompt.
|
||||
* **UI hinting**: suggest quick toggles (“Compare” / “Draft”) for power users; cut ambiguity at the source.
|
||||
|
||||
---
|
||||
|
||||
## 9) Debug Workflow (10 minutes)
|
||||
|
||||
1. Run the splitter; print the plan.
|
||||
2. Execute grounded step(s) first and log citations.
|
||||
3. Ensure draft step references a real `handoff_id`.
|
||||
4. If grounded fails → return refusal; do **not** proceed.
|
||||
5. Re-score with Example 08; CHR should improve while over-refusal stays controlled.
|
||||
|
||||
---
|
||||
|
||||
## 10) Common Traps & Fixes
|
||||
|
||||
* **Draft first** temptation → ungrounded emails. Always ground **before** drafting.
|
||||
* **One big retrieval** for all roles → tail noise. Retrieve **per role**.
|
||||
* **Auditor on drafts** → meaningless. Audit only **grounded** claims; drafts check **schema** and **handoff**.
|
||||
* **Partial shipping** (“we answered the easy half”) → inconsistent UX. Refuse on missing grounded parts.
|
||||
|
||||
---
|
||||
|
||||
## 11) Minimal Checklist (copy into PR)
|
||||
|
||||
* [ ] Split multi-intent queries deterministically into roles.
|
||||
* [ ] Grounded steps use evidence-only template + citations.
|
||||
* [ ] Drafts echo `handoff_id`; no new facts.
|
||||
* [ ] Acceptance gate enforces per-role rules; no partial ship.
|
||||
* [ ] Example 08 gates pass per intent.
|
||||
|
||||
---
|
||||
|
||||
## References to hands-on examples
|
||||
|
||||
* **Example 01** — Guarded baseline (evidence-only + refusal)
|
||||
* **Example 02** — Reflection triage (`query_parse_error`)
|
||||
* **Example 03** — Retrieval stabilization for each sub-intent
|
||||
* **Example 04** — Acceptance gate (Scholar/Auditor + handoff)
|
||||
* **Example 08** — Eval per intent (CHR for grounded, compliance for drafts)
|
||||
|
||||
---
|
||||
|
||||
### 🧭 Explore More
|
||||
|
||||
| Module | Description | Link |
|
||||
|-----------------------|----------------------------------------------------------|----------|
|
||||
| WFGY Core | Standalone semantic reasoning engine for any LLM | [View →](https://github.com/onestardao/WFGY/tree/main/core/README.md) |
|
||||
| Problem Map 1.0 | Initial 16-mode diagnostic and symbolic fix framework | [View →](https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md) |
|
||||
| Problem Map 2.0 | RAG-focused failure tree, modular fixes, and pipelines | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/rag-architecture-and-recovery.md) |
|
||||
| Semantic Clinic Index | Expanded failure catalog: prompt injection, memory bugs, logic drift | [View →](https://github.com/onestardao/WFGY/blob/main/ProblemMap/SemanticClinicIndex.md) |
|
||||
| Semantic Blueprint | Layer-based symbolic reasoning & semantic modulations | [View →](https://github.com/onestardao/WFGY/tree/main/SemanticBlueprint/README.md) |
|
||||
| Benchmark vs GPT-5 | Stress test GPT-5 with full WFGY reasoning suite | [View →](https://github.com/onestardao/WFGY/tree/main/benchmarks/benchmark-vs-gpt5/README.md) |
|
||||
|
||||
|
||||
---
|
||||
|
||||
> 👑 **Early Stargazers: [See the Hall of Fame](https://github.com/onestardao/WFGY/tree/main/stargazers)** —
|
||||
> Engineers, hackers, and open source builders who supported WFGY from day one.
|
||||
|
||||
> <img src="https://img.shields.io/github/stars/onestardao/WFGY?style=social" alt="GitHub stars"> ⭐ Help reach 10,000 stars by 2025-09-01 to unlock Engine 2.0 for everyone ⭐ **[Star WFGY on GitHub](https://github.com/onestardao/WFGY)**
|
||||
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://github.com/onestardao/WFGY)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlahBlahBlah)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlotBlotBlot)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlocBlocBloc)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlurBlurBlur)
|
||||
|
||||
[](https://github.com/onestardao/WFGY/tree/main/OS/BlowBlowBlow)
|
||||
|
||||
</div>
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue