From 01ef7d7dc42e776a7c473a890924e62ac9f4d088 Mon Sep 17 00:00:00 2001 From: yiliang114 <1204183885@qq.com> Date: Tue, 18 Aug 2026 12:28:45 +0900 Subject: [PATCH] fix(memory): end the initial recall wait on the fast result, widen tokenization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 100 ms initial budget was a fixed cost, and the evidence for it measured the wrong thing. Deterministic *scoring* is microseconds, but the fast result is only published once recall has enumerated, read, and parsed the memory tree — and this branch removed the 200-document cap for recall, so that scan grows with the tree. recall-scan-latency.test.ts adds that measurement against a real temporary tree: ~29 ms at 200 topics, ~70 ms at 500, ~130 ms at 1000. So for any tree small enough to scan in time — the ordinary case — the fast result was in hand tens of milliseconds before the budget expired, and the rest of the budget was spent waiting on a model selector this design already assumes will miss it. The wait now ends on whichever comes first: recall settling, the fast result being published, cancellation, or the ceiling. The preference order is unchanged, because the code after the wait still prefers a settled recall. Past roughly a thousand topics the scan alone exceeds the ceiling and the turn pays the full budget for nothing; that is recorded as a known limitation rather than fixed, since the fix is a persistent catalog. Tokenization kept only [a-z0-9]{3,} runs, so Cyrillic, Greek, Arabic, and accented Latin produced no tokens at all and the deterministic path was unconditionally silent for them. Keep whole runs of non-CJK letters, marks, and digits instead. CJK is excluded per character rather than by alternation order: \p{L} also matches Han, so a Latin-initial run would otherwise swallow the CJK after it and turn abc漢字 into one token. Scripts without word separators outside the CJK set still collapse to one run, which is recorded rather than claimed as segmentation. Two smaller follow-ups. The active-tool alias set is now derived once per recall instead of once per scanned document, which mattered little under the old 200-document cap and more without it. And the eval prints the Recall@5 a query-blind random scorer would score on this corpus (20%), with a test holding that floor at or below 25%, because a small corpus flatters every design and the headline was unreadable without it. --- ...-08-08-native-memory-recall-reliability.md | 78 +++++++- docs/design/auto-memory/memory-system.md | 28 ++- packages/core/src/core/client.test.ts | 81 ++++++-- packages/core/src/core/client.ts | 43 ++++- .../__fixtures__/auto-memory-recall-eval.json | 42 ++++ packages/core/src/memory/recall-eval.test.ts | 54 +++++- .../src/memory/recall-scan-latency.test.ts | 182 ++++++++++++++++++ packages/core/src/memory/recall.test.ts | 51 +++++ packages/core/src/memory/recall.ts | 106 ++++++---- 9 files changed, 603 insertions(+), 62 deletions(-) create mode 100644 packages/core/src/memory/recall-scan-latency.test.ts diff --git a/docs/design/2026-08-08-native-memory-recall-reliability.md b/docs/design/2026-08-08-native-memory-recall-reliability.md index 7693f8c3b5..1616fcb93d 100644 --- a/docs/design/2026-08-08-native-memory-recall-reliability.md +++ b/docs/design/2026-08-08-native-memory-recall-reliability.md @@ -27,7 +27,9 @@ Keep a single recall lifecycle and model-primary selection. Add one deterministic delivery stage in front of it — not the two-stage shared-scan Fast/Refined architecture originally proposed in RFC #7040. -- Give user-query recall a fixed 100 ms initial wait budget. +- Give user-query recall a 100 ms initial wait **ceiling**, not a fixed cost. + The wait ends on whichever comes first: recall settling, the deterministic + result being published, cancellation, or the ceiling. - Deliver a result that settles inside the budget in the initial prompt. - If the budget expires and the deterministic candidate pass found relevant documents, deliver that bounded result instead of nothing. @@ -52,6 +54,37 @@ The 100 ms budget stays internal, per RFC #7040's direction of a small fixed internal budget determined by benchmark rather than exposed as public configuration; telemetry can show whether a later change is justified. +### The budget is a ceiling because the scan, not the selector, decides + +The fast result is published once recall has enumerated, read, and parsed the +memory tree — and this design removed the 200-document cap for recall, so that +scan grows with the tree. `recall-scan-latency.test.ts` measures the wall-clock +time from the recall call to that publication against a real temporary tree: + +| topics | median | share of the 100 ms budget | +| ------ | ------- | -------------------------- | +| 200 | ~29 ms | ~29% | +| 500 | ~70 ms | ~70% | +| 1000 | ~130 ms | ~130% | + +Two conclusions follow, and neither is visible in the deterministic _scoring_ +cost, which is microseconds. + +First, for any tree small enough to scan in time — which is the ordinary case, +where a user holds tens of topics rather than hundreds — the fast result is in +hand long before the ceiling. Spending the remainder waits for a model selector +that this design already assumes will miss the budget, so it is close to pure +added latency on every user turn. The wait therefore ends on the fast result. +The preference order is unchanged: whatever ends the wait, a settled recall is +still delivered in preference to the fast result. + +Second, past roughly a thousand topics the scan alone exceeds the ceiling. Such +a turn spends the whole budget and still delivers nothing, which is worse than +the zero-wait behaviour this design replaced. Ending the wait early does not fix +that case; it bounds it and removes the cost everywhere else. A persistent +catalog is the actual fix and remains out of scope, per +`2026-08-09-bounded-memory-recall-candidates.md`. + ### `MAX_RELEVANT_DOCS` is per delivery, not per turn `MAX_RELEVANT_DOCS = 5` bounds one prompt. It does not bound a turn. A turn @@ -98,7 +131,12 @@ Improve the deterministic scorer, which now serves both the fast path and the selector-failure fallback: - normalize query and document text with Unicode NFKC; -- keep ASCII alphanumeric tokens of at least three characters; +- keep runs of at least three non-CJK letters, marks, and digits as whole + tokens. `\p{L}`-based rather than `[a-z0-9]`, so Cyrillic, Greek, Arabic, + and accented Latin produce tokens instead of none. CJK is excluded per + character rather than by alternation order, because `\p{L}` also matches + Han and a Latin-initial run would otherwise swallow the CJK after it and + turn `abc漢字` into a single token; - generate Unicode code-point bigrams for Han, Hiragana, Katakana, and Hangul runs; - ignore isolated CJK characters; @@ -128,10 +166,17 @@ selector-failure fallback: ## Verification Recall quality is measured in `packages/core/src/memory/recall-eval.test.ts` -against a 48-case labeled corpus, scored both by the shipped scorer and by a -frozen copy of the pre-change one so "no regression" is reproducible rather -than asserted. Delivery is measured separately in `recall-delivery-eval.test.ts`, -because a correct selection that never reaches the model is worth nothing. +against a 51-case, 25-document labeled corpus, scored both by the shipped +scorer and by a frozen copy of the pre-change one so "no regression" is +reproducible rather than asserted. Delivery is measured separately in +`recall-delivery-eval.test.ts`, because a correct selection that never reaches +the model is worth nothing, and scan latency in `recall-scan-latency.test.ts`, +because a correct selection that is not ready in time reaches nothing either. + +The eval prints the corpus size and the Recall@5 a query-blind random scorer +would achieve on it (5 of 25 documents, so 20%), and a test keeps that floor +at or below 25% with the measured result well clear of it. A small corpus +flatters every design; the floor is what makes the headline readable. - Recall settling inside the budget is delivered initially. - A budget expiry with deterministic candidates delivers that bounded result @@ -141,8 +186,17 @@ because a correct selection that never reaches the model is worth nothing. - A fast result never crosses a query boundary. - No-result queries stay silent under both designs. - A labeled set covers Chinese, English, Japanese, Korean, mixed text, - NFKC normalization, body-only matches, no-result queries, and answerable - queries that share no token with their document. + NFKC normalization, body-only matches, no-result queries, answerable + queries that share no token with their document, and alphabetic scripts + outside ASCII and CJK (Cyrillic, Greek, accented Latin). +- The fast result is published inside the initial ceiling for tree sizes a + user can plausibly reach, measured against a real temporary memory tree + rather than modelled. +- The initial wait ends as soon as the deterministic result is published and + does not run to the ceiling; a wait with nothing to deliver still runs to + the ceiling and then proceeds without memory. +- The active-tool alias set is derived once per recall rather than once per + scanned document. - Score ties are broken by recency rather than by document type, so a user-typed document is not pushed out of the two-document fast result by a tied feedback, project, or reference document. @@ -180,6 +234,14 @@ because a correct selection that never reaches the model is worth nothing. it, so requiring a lexical match did not create the gap, but it does keep the fast path silent there. This is why the headline tool-free delivery figure is 92.3% and not 100%: the residual 7.7% is exactly that slice. +- Past roughly a thousand topics in one scope, the memory-tree scan alone + exceeds the initial ceiling, so the turn spends the whole budget and still + delivers nothing. Ending the wait on the fast result bounds this rather than + removing it; the real fix is a persistent catalog, which is out of scope. +- Scripts written without word separators and outside the CJK set — Thai, + Khmer, Lao — now produce a token where they previously produced none, but + the token is the whole run. That is not segmentation, and such a query will + usually still match nothing. - Recall can see older documents outside the shared 200-document scanner cap, but non-recall callers, including Forget, keep the existing capped scanner. A broader manageability pass is separate from this recall-only change. diff --git a/docs/design/auto-memory/memory-system.md b/docs/design/auto-memory/memory-system.md index 002def9d60..47226d84a1 100644 --- a/docs/design/auto-memory/memory-system.md +++ b/docs/design/auto-memory/memory-system.md @@ -361,7 +361,7 @@ flowchart TD I -- 有文档 --> J[strategy: model] I -- 无文档 --> K[strategy: none\n仍然返回空] H -- "失败/异常" --> L[复用已计算的启发式排序] - F -- 否 --> M[tokenize query\nNFKC + ASCII token + CJK bigram\n最多 64 个 token] + F -- 否 --> M[tokenize query\nNFKC + 非 CJK 字母整串 + CJK bigram\n最多 64 个 token] M --> N[scoreDocument 打分\ntitle +4 / description +3 / body +1\n词法命中后类型加成最多 +2] N --> O[过滤 score=0 的文档\n按分数降序、mtime 降序、输入顺序排列\n取 Top 5] L --> O @@ -391,6 +391,16 @@ flowchart TD | query token 出现在 body 前 1200 字符 | +1（每个 token） | | 至少一次词法命中后，token 是类型特征关键词 | +1，整篇文档最多 +2 | +> **Tokenize 规则**：NFKC 归一化并转小写后，Han/Hiragana/Katakana/Hangul 连续片段 +> 按 code point bigram 切分（单字不产生 token）；其余至少 3 个字母、组合符或数字的 +> 连续片段整串保留。后者基于 `\p{L}` 而非 `[a-z0-9]`，因此西里尔、希腊、阿拉伯和 +> 带重音拉丁文都能产生 token。CJK 是**逐字符**排除的，不能只依赖正则分支顺序—— +> `\p{L}` 也匹配 Han，否则 `abc漢字` 会被并成一个 token。Thai/Khmer/Lao 这类 +> 无分词符又不在 CJK 集合内的文字，会整段变成一个 token：比之前完全没有 token 强， +> 但不是分词。 +> +> **同分排序**：按 mtime 降序，再按输入顺序（稳定排序），**不按 type**。 + **每种类型的特征关键词**： - `user`：user, preference, background, role, terse @@ -418,11 +428,12 @@ flowchart TD ```mermaid flowchart TD - A[UserQuery 到达\n启动 Recall Prefetch] --> B{100 ms 内\nRecall 是否完成?\nINITIAL_MEMORY_RECALL_WAIT_MS} - B -- 是 --> C{选中结果非空?} + A[UserQuery 到达\n启动 Recall Prefetch] --> B{等待结束\n以先到者为准:\nRecall 完成 / Fast 就绪 /\n取消 / 100 ms 上限} + B --> B1{Recall 是否完成?} + B1 -- 是 --> C{选中结果非空?} C -- 是 --> C1[注入首轮 Prompt\nphase: refined] C -- 否 --> C0[丢弃\nno_relevant_results] - B -- 否 --> D{是否有确定性\nFast 结果?} + B1 -- 否 --> D{是否有确定性\nFast 结果?} D -- 是 --> E[注入首轮 Prompt\nphase: fast\n最多 2 篇 MAX_FAST_RECALL_DOCS] D -- 否 --> F[首轮不注入] E --> G[Recall 继续运行] @@ -447,6 +458,15 @@ flowchart TD 用户级 Memory 最重要的场景。Fast 结果复用 `selectModelCandidateDocuments` 为 Model Manifest 已经算好的候选，不产生额外扫描或 I/O。 +**100 ms 是上限而不是固定开销**：Fast 结果在 Recall 扫完 Memory 树之后才发布， +所以真正决定它能否赶上的是**扫描耗时**，不是打分耗时（后者是微秒级）。 +`recall-scan-latency.test.ts` 在真实临时 Memory 树上实测：200 篇约 29 ms、 +500 篇约 70 ms、1000 篇约 130 ms。对能在预算内扫完的树（普通用户的常见情况）， +Fast 就绪后继续等待只是在等一个本设计已经假定赶不上的 Model Selector， +因此等待会在 Fast 就绪时立即结束。超过约 1000 篇时扫描本身就超预算， +该轮会付满 100 ms 且什么都投不到——提前结束等待只能把这种情况**限制住**， +消除不了它。 + **Fast 阶段的边界**：Fast 结果就是确定性结果，因此它只能解决**时机**问题， 解决不了**匹配**问题。与文档没有任何词面重叠的 Query 产生不了 Fast 结果， 这类 Query 在无工具回合仍然拿不到 Memory——只有 Model Selector 能覆盖它们， diff --git a/packages/core/src/core/client.test.ts b/packages/core/src/core/client.test.ts index 0b9a56784f..34b082b3f8 100644 --- a/packages/core/src/core/client.test.ts +++ b/packages/core/src/core/client.test.ts @@ -5156,10 +5156,9 @@ hello ), ); - // Held for the budget, then the fast result goes out instead of nothing. - await vi.advanceTimersByTimeAsync(99); - expect(mockTurnRunFn).not.toHaveBeenCalled(); - await vi.advanceTimersByTimeAsync(1); + // The deterministic result was already published, so the budget has + // nothing left to wait for and the request goes out without spending it. + await vi.advanceTimersByTimeAsync(0); await done; expect(mockTurnRunFn).toHaveBeenCalledWith( @@ -5171,6 +5170,58 @@ hello ); }); + it('ends the initial wait as soon as the deterministic result arrives', async () => { + vi.useFakeTimers(); + // Stands in for the memory-tree scan: the fast result is not ready when + // the wait begins, but lands well before the budget expires. + const SCAN_MS = 30; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, SCAN_MS); + // Selector never settles — stands in for a slow round trip. + return new Promise(() => {}); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-early-return', + ), + ); + + await vi.advanceTimersByTimeAsync(SCAN_MS - 1); + expect(mockTurnRunFn).not.toHaveBeenCalled(); + // The remaining ~70 ms of budget is never spent. + await vi.advanceTimersByTimeAsync(1); + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + expect.any(AbortSignal), + ); + + await vi.advanceTimersByTimeAsync(100); + await done; + }); + it('still delivers the model-selected result at ToolResult after a fast initial delivery', async () => { vi.useFakeTimers(); let settleRecall: @@ -5507,12 +5558,18 @@ hello it('delivers no fast result when the turn is cancelled inside the initial window', async () => { vi.useFakeTimers(); const controller = new AbortController(); + // The fast result must still be in flight when the abort lands, + // otherwise the wait would already have ended on its arrival and there + // would be no window left to cancel inside. mockMemoryManager.recall.mockImplementation((_root, _query, options) => { - options.onFastResult?.({ - prompt: '## Relevant memory\n\nFast deterministic result.', - selectedDocs: [fastDoc('/m/fast.md', '- terse')], - strategy: 'heuristic', - }); + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, 80); return new Promise(() => {}); }); @@ -5737,9 +5794,11 @@ hello }); it('should hold the main request for exactly the initial recall budget when recall never settles', async () => { - // Recall never settles. Fake timers pin the budget contract: the + // Recall never settles and never publishes a deterministic result, so + // nothing can end the wait early. Fake timers pin the ceiling: the // request must still be blocked 1 ms inside the budget and proceed, - // without memory, the moment the budget expires. + // without memory, the moment the budget expires. This is also the shape + // of a memory tree whose scan is slower than the budget. vi.useFakeTimers(); mockMemoryManager.recall.mockReturnValue(new Promise(() => {})); diff --git a/packages/core/src/core/client.ts b/packages/core/src/core/client.ts index 3d6e5c966d..d39052fb64 100644 --- a/packages/core/src/core/client.ts +++ b/packages/core/src/core/client.ts @@ -304,6 +304,15 @@ function sameActiveGoalProjection( * 3. Aborted-and-discarded by every cleanup path (resetChat, * MaxSessionTurns, etc.) or replaced when a new UserQuery arrives. */ +/** + * Publication slot for recall's deterministic result, plus a one-shot + * listener for its arrival. + */ +type MemoryFastResultBox = { + current: RelevantAutoMemoryPromptResult | null; + onArrive?: () => void; +}; + type MemoryPrefetchHandle = { promise: Promise; /** Set by promise.finally(). null until the promise settles. */ @@ -320,8 +329,11 @@ type MemoryPrefetchHandle = { * Deterministic result published by recall before it blocks on the model * selector. A box rather than a plain field because recall can invoke the * callback before this handle object exists. + * + * `onArrive` lets the bounded initial wait stop as soon as there is + * something to deliver, instead of always spending the whole budget. */ - fastResultRef: { current: RelevantAutoMemoryPromptResult | null }; + fastResultRef: MemoryFastResultBox; /** True after the fast result was injected — prevents double-inject and double-log. */ fastDelivered: boolean; /** Paths injected by the fast phase, excluded from the later refined delivery. */ @@ -911,11 +923,32 @@ export class GeminiClient { return null; } - if (handle.settledAt === null && waitMs > 0) { + // `waitMs` is a ceiling, not a fixed cost. The wait ends on whichever + // comes first: recall settling, the deterministic result being published, + // cancellation, or the budget expiring. + // + // Ending on the fast result matters more than it looks. That result is + // published once recall has scanned the memory tree, which is milliseconds + // for an ordinary tree — while the model selector is a network round trip + // that this design already assumes will miss the budget. Spending the rest + // of the budget after the fast result is in hand therefore buys an + // outcome that almost never arrives, and charges every user turn for it. + // See `recall-scan-latency.test.ts` for the scan measurements. + // + // The preference order is unchanged: whatever ends the wait, the code + // below still prefers a settled recall over the fast result. + if ( + handle.settledAt === null && + handle.fastResultRef.current === null && + waitMs > 0 + ) { await new Promise((resolve) => { const finish = () => { clearTimeout(timer); handle.controller.signal.removeEventListener('abort', finish); + if (handle.fastResultRef.onArrive === finish) { + handle.fastResultRef.onArrive = undefined; + } resolve(); }; @@ -926,6 +959,7 @@ export class GeminiClient { handle.controller.signal.addEventListener('abort', finish, { once: true, }); + handle.fastResultRef.onArrive = finish; void handle.promise.then(finish, finish); } }); @@ -2862,9 +2896,7 @@ export class GeminiClient { } else { signal.addEventListener('abort', onParentAbort, { once: true }); } - const fastResultRef: { - current: RelevantAutoMemoryPromptResult | null; - } = { current: null }; + const fastResultRef: MemoryFastResultBox = { current: null }; const promise = this.config .getMemoryManager() .recall( @@ -2877,6 +2909,7 @@ export class GeminiClient { abortSignal: controller.signal, onFastResult: (result) => { fastResultRef.current = result; + fastResultRef.onArrive?.(); }, }, ) diff --git a/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json b/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json index 635e07fd4e..8adf013b54 100644 --- a/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json +++ b/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json @@ -153,6 +153,27 @@ "title": "Deployment 发布说明", "description": "Release 流程", "body": "Mixed-language deployment notes." + }, + { + "id": "ru-deploy", + "type": "project", + "title": "Процесс развёртывания", + "description": "Контрольный список релиза", + "body": "Перед выкатом проверить мониторинг и переключатели отката." + }, + { + "id": "el-auth", + "type": "reference", + "title": "Ρύθμιση ταυτοποίησης", + "description": "Επίλυση προβλημάτων σύνδεσης", + "body": "Έλεγχος λήξης συνεδρίας και δικαιωμάτων." + }, + { + "id": "fr-perf", + "type": "project", + "title": "Réduction du démarrage à froid", + "description": "Objectifs de performance trimestriels", + "body": "Mesurer la latence côté serveur avant toute optimisation." } ], "cases": [ @@ -479,6 +500,27 @@ "category": "semantic-no-lexical", "query": "首屏加载太久了", "relevantIds": ["zh-perf"] + }, + { + "id": "other-script-ru-deploy", + "category": "other-script", + "query": "процесс развёртывания", + "relevantIds": ["ru-deploy"], + "expectedTopId": "ru-deploy" + }, + { + "id": "other-script-el-auth", + "category": "other-script", + "query": "ρύθμιση ταυτοποίησης", + "relevantIds": ["el-auth"], + "expectedTopId": "el-auth" + }, + { + "id": "other-script-fr-perf", + "category": "other-script", + "query": "démarrage à froid", + "relevantIds": ["fr-perf"], + "expectedTopId": "fr-perf" } ] } diff --git a/packages/core/src/memory/recall-eval.test.ts b/packages/core/src/memory/recall-eval.test.ts index 5ee294aa36..d342613c2f 100644 --- a/packages/core/src/memory/recall-eval.test.ts +++ b/packages/core/src/memory/recall-eval.test.ts @@ -43,6 +43,7 @@ const categories = new Set([ 'nfkc', 'body-only', 'semantic-no-lexical', + 'other-script', 'no-result', ] as const); @@ -322,6 +323,15 @@ const isEnglish = (testCase: EvalCase) => testCase.category === 'english'; */ const isSemanticNoLexical = (testCase: EvalCase) => testCase.category === 'semantic-no-lexical'; +/** + * Alphabetic scripts outside ASCII and CJK — Cyrillic, Greek, and accented + * Latin here. The pre-change tokenizer kept only `[a-z0-9]{3,}` runs, so + * these queries produced no tokens at all and the deterministic path was + * unconditionally silent; the shipped tokenizer keeps whole non-CJK letter + * runs instead. + */ +const isOtherScript = (testCase: EvalCase) => + testCase.category === 'other-script'; const isCjk = (testCase: EvalCase) => testCase.category === 'chinese' || testCase.category === 'japanese' || @@ -334,11 +344,26 @@ function formatPercent(value: number | null): string { return value === null ? 'n/a' : `${(value * 100).toFixed(1)}%`; } +/** + * Expected Recall@5 of a scorer that ignores the query and returns five + * documents drawn uniformly at random from the corpus. Each labeled document + * has a `RECALL_AT / corpusSize` chance of being among them, so the expected + * per-case recall — and therefore the mean over any slice — is that ratio. + * + * Printed beside the measured columns because a small corpus flatters the + * headline: on a pool this size "100% Recall@5" is a much weaker statement + * than it reads, and a reader comparing designs needs the floor to calibrate + * against. It is exact rather than sampled, so the table stays deterministic. + */ +function randomBaselineRecallAt5(corpusSize: number): number { + return corpusSize === 0 ? 0 : Math.min(1, RECALL_AT / corpusSize); +} + describe('auto-memory recall evaluation', () => { it('loads a labeled corpus covering every required category', () => { const fixture = loadFixture(); expect(fixture.cases.length).toBeGreaterThanOrEqual(30); - expect(fixture.cases.length).toBeLessThanOrEqual(50); + expect(fixture.cases.length).toBeLessThanOrEqual(60); expect(new Set(fixture.cases.map((testCase) => testCase.category))).toEqual( categories, ); @@ -434,6 +459,7 @@ describe('auto-memory recall evaluation', () => { ['english', isEnglish], ['cjk', isCjk], ['mixed', isMixed], + ['other-script', isOtherScript], ['semantic-no-lexical', isSemanticNoLexical], ] as const; @@ -445,6 +471,14 @@ describe('auto-memory recall evaluation', () => { '| --- | --- | --- | --- |', ]; + const randomFloor = randomBaselineRecallAt5(fixture.docs.length); + lines.splice( + 2, + 0, + `corpus: ${fixture.docs.length} documents, ${fixture.cases.length} cases — a query-blind random scorer returning ${RECALL_AT} documents scores ${formatPercent(randomFloor)} Recall@5 on this pool`, + '', + ); + for (const [label, filter] of rows) { const before = evaluate( fixture, @@ -477,6 +511,24 @@ describe('auto-memory recall evaluation', () => { expect(lines.length).toBeGreaterThan(5); }); + /** + * Guards the headline against a corpus so small that Recall@5 is nearly + * free. This is a property of the fixture, not of the scorer: shrink the + * corpus far enough and every metric approaches 100% for any design. + */ + it('keeps the corpus large enough for Recall@5 to discriminate', () => { + const fixture = loadFixture(); + const randomFloor = randomBaselineRecallAt5(fixture.docs.length); + + expect(randomFloor).toBeLessThanOrEqual(0.25); + const after = evaluate( + fixture, + selectRelevantAutoMemoryDocuments, + (testCase) => !isSemanticNoLexical(testCase), + ); + expect(after.recallAt5!).toBeGreaterThan(randomFloor * 3); + }); + it('produces deterministic summaries', () => { const fixture = loadFixture(); expect(evaluate(fixture, selectRelevantAutoMemoryDocuments)).toEqual( diff --git a/packages/core/src/memory/recall-scan-latency.test.ts b/packages/core/src/memory/recall-scan-latency.test.ts new file mode 100644 index 0000000000..7f3672f53f --- /dev/null +++ b/packages/core/src/memory/recall-scan-latency.test.ts @@ -0,0 +1,182 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import * as fs from 'node:fs/promises'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import type { Config } from '../config/config.js'; +import { getAutoMemoryFilePath } from './paths.js'; +import { resolveRelevantAutoMemoryPromptForQuery } from './recall.js'; +import { selectRelevantAutoMemoryDocumentsByModel } from './relevanceSelector.js'; +import { ensureAutoMemoryScaffold } from './store.js'; + +/** + * Measures the part of the initial-turn budget nothing else measures. + * + * `recall-delivery-eval.test.ts` times the deterministic *scoring*, which is + * microseconds. That is not what decides whether the fast path delivers. The + * fast result is published from `onFastResult`, which fires only after recall + * has enumerated, read, and parsed every topic file — and this branch removed + * the 200-document cap for recall, so that scan grows with the memory tree. + * If the scan alone exceeds `INITIAL_MEMORY_RECALL_WAIT_MS`, the turn pays the + * full budget *and* delivers nothing, which is strictly worse than before. + * + * So this file measures wall-clock time from the recall call to the fast + * callback, against a real temporary memory tree, with the model selector + * mocked to hang the way a network round trip does. + * + * Timings are machine-dependent and CI is shared, so the assertions are + * deliberately loose; the printed table is the artifact worth reading. What + * is asserted is the structural claim: the fast result lands well inside the + * budget at memory-tree sizes users can plausibly reach. + */ + +vi.mock('./relevanceSelector.js', () => ({ + selectRelevantAutoMemoryDocumentsByModel: vi.fn(), +})); + +/** Mirrors INITIAL_MEMORY_RECALL_WAIT_MS in client.ts. */ +const INITIAL_BUDGET_MS = 100; +const TOPIC_COUNTS = [200, 500, 1000] as const; +const REPEATS = 5; + +let tempDir: string; +const projectRootByCount = new Map(); + +async function buildMemoryTree(topicCount: number): Promise { + const projectRoot = path.join(tempDir, `project-${topicCount}`); + await fs.mkdir(projectRoot, { recursive: true }); + await ensureAutoMemoryScaffold( + projectRoot, + new Date('2026-04-01T00:00:00.000Z'), + ); + + const referenceDir = path.dirname( + getAutoMemoryFilePath(projectRoot, 'reference/topic-0000.md'), + ); + await fs.mkdir(referenceDir, { recursive: true }); + + // Bodies are sized like real notes rather than one-liners: the scan reads + // and parses whole files, so a corpus of stubs would understate the cost. + const filler = 'Historical note about an unrelated subsystem. '.repeat(20); + await Promise.all( + Array.from({ length: topicCount }, (_, index) => + fs.writeFile( + path.join(referenceDir, `topic-${String(index).padStart(4, '0')}.md`), + [ + '---', + 'type: reference', + `name: Topic ${index}`, + `description: Reference note number ${index} about deployment history`, + '---', + '', + filler, + index === topicCount - 1 ? 'The saved codeword is SCANBENCH.' : '', + '', + ].join('\n'), + 'utf-8', + ), + ), + ); + + return projectRoot; +} + +/** Wall-clock ms from the recall call until the fast result is published. */ +async function measureTimeToFastResultMs(projectRoot: string): Promise { + let elapsed = Number.NaN; + const startedAt = performance.now(); + const recall = resolveRelevantAutoMemoryPromptForQuery( + projectRoot, + 'what is the saved scanbench codeword for deployment', + { + config: { + getSessionId: () => 'session-scan-bench', + getModel: () => 'qwen3-coder-plus', + } as Config, + onFastResult: () => { + elapsed = performance.now() - startedAt; + }, + }, + ); + + // Let the pending recall settle so it does not leak into the next sample. + await recall; + return elapsed; +} + +describe('auto-memory recall scan latency', () => { + beforeAll(async () => { + tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'recall-scan-bench-')); + // The selector stands in for the network round trip: it must not settle + // before the fast callback, or the measurement would race it. Returning + // an empty selection keeps recall finishing promptly after that. + vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockResolvedValue([]); + for (const topicCount of TOPIC_COUNTS) { + projectRootByCount.set(topicCount, await buildMemoryTree(topicCount)); + } + }, 120_000); + + afterAll(async () => { + if (tempDir) { + await fs.rm(tempDir, { recursive: true, force: true }); + } + }); + + it('publishes the fast result well inside the initial budget', async () => { + const rows: Array<[number, number, number]> = []; + + for (const topicCount of TOPIC_COUNTS) { + const projectRoot = projectRootByCount.get(topicCount)!; + // Warm the page cache so the first sample does not report cold I/O as + // the steady-state cost. + await measureTimeToFastResultMs(projectRoot); + + const samples: number[] = []; + for (let i = 0; i < REPEATS; i += 1) { + samples.push(await measureTimeToFastResultMs(projectRoot)); + } + samples.sort((a, b) => a - b); + const median = samples[Math.floor(samples.length / 2)]; + const worst = samples[samples.length - 1]; + rows.push([topicCount, median, worst]); + + expect(Number.isFinite(median)).toBe(true); + } + + const [smallest] = rows; + // The ordinary case must leave the rest of the budget to spare. Loose + // because CI is shared; the table is what carries the detail. + expect(smallest[0]).toBe(TOPIC_COUNTS[0]); + expect(smallest[1]).toBeLessThan(INITIAL_BUDGET_MS / 2); + + console.log( + [ + '', + 'Scan gate — time from recall start to fast result (single project scope)', + `initial budget: ${INITIAL_BUDGET_MS} ms`, + '', + `| topics | median | worst of ${REPEATS} | share of budget | fast result inside budget? |`, + '| --- | --- | --- | --- | --- |', + ...rows.map( + ([topicCount, median, worst]) => + `| ${topicCount} | ${median.toFixed(1)} ms | ${worst.toFixed(1)} ms | ${((median / INITIAL_BUDGET_MS) * 100).toFixed(1)}% | ${worst < INITIAL_BUDGET_MS ? 'yes' : 'no'} |`, + ), + '', + 'The fast result is only available once this scan completes, so this is', + 'the real precondition for the fast path delivering anything — not the', + 'scoring cost, which is microseconds.', + '', + 'Where a row reads "no", the turn spends the whole budget and still', + 'delivers nothing, which is worse than the zero-wait behaviour this', + 'branch replaced. That is why the wait ends on the fast result rather', + 'than always running to the ceiling: it removes the cost for every tree', + 'small enough to scan in time, and bounds it for the rest.', + ].join('\n'), + ); + }, 120_000); +}); diff --git a/packages/core/src/memory/recall.test.ts b/packages/core/src/memory/recall.test.ts index 1c76852af0..20309f494f 100644 --- a/packages/core/src/memory/recall.test.ts +++ b/packages/core/src/memory/recall.test.ts @@ -340,6 +340,57 @@ describe('auto-memory relevant recall', () => { ); }); + it('tokenizes alphabetic scripts outside ASCII and CJK', () => { + // `[a-z0-9]{3,}` produced no tokens at all for these, so the + // deterministic path was unconditionally silent — no fast result, and a + // silent selector-failure fallback. + const cyrillic = memoryDoc( + 'ru.md', + 'project', + 'Процесс развёртывания', + '', + '', + ); + const greek = memoryDoc('el.md', 'reference', 'Ρύθμιση σύνδεσης', '', ''); + const accented = memoryDoc('fr.md', 'project', 'Démarrage à froid', '', ''); + const docs = [cyrillic, greek, accented]; + + expect( + selectRelevantAutoMemoryDocuments('развёртывания', docs)[0]?.filename, + ).toBe('ru.md'); + expect( + selectRelevantAutoMemoryDocuments('σύνδεσης', docs)[0]?.filename, + ).toBe('el.md'); + expect( + selectRelevantAutoMemoryDocuments('démarrage', docs)[0]?.filename, + ).toBe('fr.md'); + }); + + it('does not let a Latin run swallow the CJK that follows it', () => { + // `\p{L}` also matches Han, so a naive alphabetic class would tokenize + // `abc漢字` as one run and stop matching either half on its own. + const latin = memoryDoc('latin.md', 'reference', 'abc', '', ''); + const han = memoryDoc('han.md', 'reference', '漢字', '', ''); + + expect( + selectRelevantAutoMemoryDocuments('abc漢字', [latin, han]).map( + (doc) => doc.filename, + ), + ).toEqual(['latin.md', 'han.md']); + }); + + it('still ignores runs shorter than three characters', () => { + const doc = memoryDoc('go.md', 'reference', 'go go go', '', ''); + + expect(selectRelevantAutoMemoryDocuments('go', [doc])).toEqual([]); + // Two Cyrillic letters are below the threshold for the same reason. + expect( + selectRelevantAutoMemoryDocuments('до', [ + memoryDoc('ru.md', 'reference', 'до свидания', '', ''), + ]), + ).toEqual([]); + }); + it('breaks score ties by recency, not by document type', () => { // Every type carries the same title, so the only thing separating these // documents is the tie-break. An alphabetical type comparison orders them diff --git a/packages/core/src/memory/recall.ts b/packages/core/src/memory/recall.ts index cea468a75d..625d53e8b6 100644 --- a/packages/core/src/memory/recall.ts +++ b/packages/core/src/memory/recall.ts @@ -72,8 +72,35 @@ const TYPE_KEYWORDS: Record = { reference: ['reference', 'dashboard', 'ticket', 'docs', 'doc', 'link'], }; -const RECALL_TOKEN_RUN = - /[a-z0-9]{3,}|[\p{Script=Han}\p{Script=Hiragana}\p{Script_Extensions=Katakana}\p{Script=Hangul}]+/gu; +/** + * Scripts tokenized as code-point bigrams because they are written without + * word separators, so a whole run is one unsegmentable token. + */ +const CJK_CLASS = + '[\\p{Script=Han}\\p{Script=Hiragana}\\p{Script_Extensions=Katakana}\\p{Script=Hangul}]'; + +/** + * One token run: either a CJK run (bigram-tokenized below) or a run of at + * least three non-CJK letters, marks, and digits (kept whole). + * + * The alphabetic alternative is `\p{L}`-based rather than `[a-z0-9]`, so + * Cyrillic, Greek, Arabic, and accented Latin produce tokens instead of + * silently producing none. It excludes CJK per character rather than relying + * on alternation order: `\p{L}` also matches Han, so a plain class would let + * a run starting in Latin swallow the CJK that follows it and turn + * `abc漢字` into one token. + * + * Scripts written without spaces and outside the CJK set (Thai, Khmer, Lao) + * still collapse into a single long token. That is no worse than the previous + * behaviour of producing nothing, but it is not segmentation. + */ +const RECALL_TOKEN_RUN = new RegExp( + `${CJK_CLASS}+|(?!${CJK_CLASS})[\\p{L}\\p{N}](?:(?!${CJK_CLASS})[\\p{L}\\p{M}\\p{N}]){2,}`, + 'gu', +); + +/** Whether a matched run is CJK, and therefore bigram-tokenized. */ +const CJK_RUN_START = new RegExp(`^${CJK_CLASS}`, 'u'); function normalizeRecallText(text: string): string { return text.normalize('NFKC').toLowerCase(); @@ -103,14 +130,14 @@ function tokenize(text: string): string[] { for (const match of normalized.matchAll(RECALL_TOKEN_RUN)) { const run = match[0]; - if (run.charCodeAt(0) <= 0x7f) { - addToken(run); - } else { + if (CJK_RUN_START.test(run)) { let previous = ''; for (const codePoint of run) { if (previous) addToken(previous + codePoint); previous = codePoint; } + } else { + addToken(run); } } @@ -146,35 +173,48 @@ function toolAliases(toolName: string): string[] { ); } -function isActiveToolUsageMemory( - doc: ScannedAutoMemoryDocument, +/** + * Build the active-tool noise predicate once per recall rather than deriving + * it per document. The alias set depends only on `recentTools`, so computing + * it inside the per-document filter re-derived up to + * `MAX_RECENT_TOOL_NAMES_FOR_MEMORY` alias lists for every scanned document — + * which recall now does over an uncapped pool. + * + * Returns a predicate rather than a boolean so both filter sites share the + * hoisting; a `recentTools`-free recall short-circuits to a constant `false`. + */ +function createActiveToolUsageFilter( recentTools: readonly string[], -): boolean { +): (doc: ScannedAutoMemoryDocument) => boolean { if (recentTools.length === 0) { - return false; + return () => false; } - const haystack = [doc.title, doc.description, normalizeBody(doc.body)] - .join(' ') - .toLowerCase(); - const namesActiveTool = recentTools.some((toolName) => - toolAliases(toolName).some((alias) => haystack.includes(alias)), - ); - if (!namesActiveTool) { - return false; + const aliases = Array.from(new Set(recentTools.flatMap(toolAliases))); + if (aliases.length === 0) { + return () => false; } - if ( - DURABLE_ACTIVE_TOOL_MEMORY_MARKERS.some((marker) => + return (doc) => { + const haystack = [doc.title, doc.description, normalizeBody(doc.body)] + .join(' ') + .toLowerCase(); + if (!aliases.some((alias) => haystack.includes(alias))) { + return false; + } + + if ( + DURABLE_ACTIVE_TOOL_MEMORY_MARKERS.some((marker) => + haystack.includes(marker), + ) + ) { + return false; + } + + return ACTIVE_TOOL_USAGE_MEMORY_MARKERS.some((marker) => haystack.includes(marker), - ) - ) { - return false; - } - - return ACTIVE_TOOL_USAGE_MEMORY_MARKERS.some((marker) => - haystack.includes(marker), - ); + ); + }; } function scoreDocument( @@ -246,9 +286,8 @@ function selectModelCandidateDocuments( modelCandidates: ScannedAutoMemoryDocument[]; fallbackDocs: ScannedAutoMemoryDocument[]; } { - const eligible = docs.filter( - (doc) => !isActiveToolUsageMemory(doc, recentTools), - ); + const isActiveToolNoise = createActiveToolUsageFilter(recentTools); + const eligible = docs.filter((doc) => !isActiveToolNoise(doc)); const lexical = selectRelevantAutoMemoryDocuments( query, eligible, @@ -498,13 +537,14 @@ export async function resolveRelevantAutoMemoryPromptForQuery( }; } + const isActiveToolNoise = createActiveToolUsageFilter( + options.recentTools ?? [], + ); const selectedDocs = fallbackDocs ?? selectRelevantAutoMemoryDocuments( query, - docs.filter( - (doc) => !isActiveToolUsageMemory(doc, options.recentTools ?? []), - ), + docs.filter((doc) => !isActiveToolNoise(doc)), limit, ); const strategy: RelevantAutoMemoryPromptResult['strategy'] =