diff --git a/docs/design/2026-08-08-native-memory-recall-reliability.md b/docs/design/2026-08-08-native-memory-recall-reliability.md index 7693f8c3b5..1616fcb93d 100644 --- a/docs/design/2026-08-08-native-memory-recall-reliability.md +++ b/docs/design/2026-08-08-native-memory-recall-reliability.md @@ -27,7 +27,9 @@ Keep a single recall lifecycle and model-primary selection. Add one deterministic delivery stage in front of it — not the two-stage shared-scan Fast/Refined architecture originally proposed in RFC #7040. -- Give user-query recall a fixed 100 ms initial wait budget. +- Give user-query recall a 100 ms initial wait **ceiling**, not a fixed cost. + The wait ends on whichever comes first: recall settling, the deterministic + result being published, cancellation, or the ceiling. - Deliver a result that settles inside the budget in the initial prompt. - If the budget expires and the deterministic candidate pass found relevant documents, deliver that bounded result instead of nothing. @@ -52,6 +54,37 @@ The 100 ms budget stays internal, per RFC #7040's direction of a small fixed internal budget determined by benchmark rather than exposed as public configuration; telemetry can show whether a later change is justified. +### The budget is a ceiling because the scan, not the selector, decides + +The fast result is published once recall has enumerated, read, and parsed the +memory tree — and this design removed the 200-document cap for recall, so that +scan grows with the tree. `recall-scan-latency.test.ts` measures the wall-clock +time from the recall call to that publication against a real temporary tree: + +| topics | median | share of the 100 ms budget | +| ------ | ------- | -------------------------- | +| 200 | ~29 ms | ~29% | +| 500 | ~70 ms | ~70% | +| 1000 | ~130 ms | ~130% | + +Two conclusions follow, and neither is visible in the deterministic _scoring_ +cost, which is microseconds. + +First, for any tree small enough to scan in time — which is the ordinary case, +where a user holds tens of topics rather than hundreds — the fast result is in +hand long before the ceiling. Spending the remainder waits for a model selector +that this design already assumes will miss the budget, so it is close to pure +added latency on every user turn. The wait therefore ends on the fast result. +The preference order is unchanged: whatever ends the wait, a settled recall is +still delivered in preference to the fast result. + +Second, past roughly a thousand topics the scan alone exceeds the ceiling. Such +a turn spends the whole budget and still delivers nothing, which is worse than +the zero-wait behaviour this design replaced. Ending the wait early does not fix +that case; it bounds it and removes the cost everywhere else. A persistent +catalog is the actual fix and remains out of scope, per +`2026-08-09-bounded-memory-recall-candidates.md`. + ### `MAX_RELEVANT_DOCS` is per delivery, not per turn `MAX_RELEVANT_DOCS = 5` bounds one prompt. It does not bound a turn. A turn @@ -98,7 +131,12 @@ Improve the deterministic scorer, which now serves both the fast path and the selector-failure fallback: - normalize query and document text with Unicode NFKC; -- keep ASCII alphanumeric tokens of at least three characters; +- keep runs of at least three non-CJK letters, marks, and digits as whole + tokens. `\p{L}`-based rather than `[a-z0-9]`, so Cyrillic, Greek, Arabic, + and accented Latin produce tokens instead of none. CJK is excluded per + character rather than by alternation order, because `\p{L}` also matches + Han and a Latin-initial run would otherwise swallow the CJK after it and + turn `abc漢字` into a single token; - generate Unicode code-point bigrams for Han, Hiragana, Katakana, and Hangul runs; - ignore isolated CJK characters; @@ -128,10 +166,17 @@ selector-failure fallback: ## Verification Recall quality is measured in `packages/core/src/memory/recall-eval.test.ts` -against a 48-case labeled corpus, scored both by the shipped scorer and by a -frozen copy of the pre-change one so "no regression" is reproducible rather -than asserted. Delivery is measured separately in `recall-delivery-eval.test.ts`, -because a correct selection that never reaches the model is worth nothing. +against a 51-case, 25-document labeled corpus, scored both by the shipped +scorer and by a frozen copy of the pre-change one so "no regression" is +reproducible rather than asserted. Delivery is measured separately in +`recall-delivery-eval.test.ts`, because a correct selection that never reaches +the model is worth nothing, and scan latency in `recall-scan-latency.test.ts`, +because a correct selection that is not ready in time reaches nothing either. + +The eval prints the corpus size and the Recall@5 a query-blind random scorer +would achieve on it (5 of 25 documents, so 20%), and a test keeps that floor +at or below 25% with the measured result well clear of it. A small corpus +flatters every design; the floor is what makes the headline readable. - Recall settling inside the budget is delivered initially. - A budget expiry with deterministic candidates delivers that bounded result @@ -141,8 +186,17 @@ because a correct selection that never reaches the model is worth nothing. - A fast result never crosses a query boundary. - No-result queries stay silent under both designs. - A labeled set covers Chinese, English, Japanese, Korean, mixed text, - NFKC normalization, body-only matches, no-result queries, and answerable - queries that share no token with their document. + NFKC normalization, body-only matches, no-result queries, answerable + queries that share no token with their document, and alphabetic scripts + outside ASCII and CJK (Cyrillic, Greek, accented Latin). +- The fast result is published inside the initial ceiling for tree sizes a + user can plausibly reach, measured against a real temporary memory tree + rather than modelled. +- The initial wait ends as soon as the deterministic result is published and + does not run to the ceiling; a wait with nothing to deliver still runs to + the ceiling and then proceeds without memory. +- The active-tool alias set is derived once per recall rather than once per + scanned document. - Score ties are broken by recency rather than by document type, so a user-typed document is not pushed out of the two-document fast result by a tied feedback, project, or reference document. @@ -180,6 +234,14 @@ because a correct selection that never reaches the model is worth nothing. it, so requiring a lexical match did not create the gap, but it does keep the fast path silent there. This is why the headline tool-free delivery figure is 92.3% and not 100%: the residual 7.7% is exactly that slice. +- Past roughly a thousand topics in one scope, the memory-tree scan alone + exceeds the initial ceiling, so the turn spends the whole budget and still + delivers nothing. Ending the wait on the fast result bounds this rather than + removing it; the real fix is a persistent catalog, which is out of scope. +- Scripts written without word separators and outside the CJK set — Thai, + Khmer, Lao — now produce a token where they previously produced none, but + the token is the whole run. That is not segmentation, and such a query will + usually still match nothing. - Recall can see older documents outside the shared 200-document scanner cap, but non-recall callers, including Forget, keep the existing capped scanner. A broader manageability pass is separate from this recall-only change. diff --git a/docs/design/auto-memory/memory-system.md b/docs/design/auto-memory/memory-system.md index 002def9d60..47226d84a1 100644 --- a/docs/design/auto-memory/memory-system.md +++ b/docs/design/auto-memory/memory-system.md @@ -361,7 +361,7 @@ flowchart TD I -- 有文档 --> J[strategy: model] I -- 无文档 --> K[strategy: none\n仍然返回空] H -- "失败/异常" --> L[复用已计算的启发式排序] - F -- 否 --> M[tokenize query\nNFKC + ASCII token + CJK bigram\n最多 64 个 token] + F -- 否 --> M[tokenize query\nNFKC + 非 CJK 字母整串 + CJK bigram\n最多 64 个 token] M --> N[scoreDocument 打分\ntitle +4 / description +3 / body +1\n词法命中后类型加成最多 +2] N --> O[过滤 score=0 的文档\n按分数降序、mtime 降序、输入顺序排列\n取 Top 5] L --> O @@ -391,6 +391,16 @@ flowchart TD | query token 出现在 body 前 1200 字符 | +1（每个 token） | | 至少一次词法命中后，token 是类型特征关键词 | +1，整篇文档最多 +2 | +> **Tokenize 规则**：NFKC 归一化并转小写后，Han/Hiragana/Katakana/Hangul 连续片段 +> 按 code point bigram 切分（单字不产生 token）；其余至少 3 个字母、组合符或数字的 +> 连续片段整串保留。后者基于 `\p{L}` 而非 `[a-z0-9]`，因此西里尔、希腊、阿拉伯和 +> 带重音拉丁文都能产生 token。CJK 是**逐字符**排除的，不能只依赖正则分支顺序—— +> `\p{L}` 也匹配 Han，否则 `abc漢字` 会被并成一个 token。Thai/Khmer/Lao 这类 +> 无分词符又不在 CJK 集合内的文字，会整段变成一个 token：比之前完全没有 token 强， +> 但不是分词。 +> +> **同分排序**：按 mtime 降序，再按输入顺序（稳定排序），**不按 type**。 + **每种类型的特征关键词**： - `user`：user, preference, background, role, terse @@ -418,11 +428,12 @@ flowchart TD ```mermaid flowchart TD - A[UserQuery 到达\n启动 Recall Prefetch] --> B{100 ms 内\nRecall 是否完成?\nINITIAL_MEMORY_RECALL_WAIT_MS} - B -- 是 --> C{选中结果非空?} + A[UserQuery 到达\n启动 Recall Prefetch] --> B{等待结束\n以先到者为准:\nRecall 完成 / Fast 就绪 /\n取消 / 100 ms 上限} + B --> B1{Recall 是否完成?} + B1 -- 是 --> C{选中结果非空?} C -- 是 --> C1[注入首轮 Prompt\nphase: refined] C -- 否 --> C0[丢弃\nno_relevant_results] - B -- 否 --> D{是否有确定性\nFast 结果?} + B1 -- 否 --> D{是否有确定性\nFast 结果?} D -- 是 --> E[注入首轮 Prompt\nphase: fast\n最多 2 篇 MAX_FAST_RECALL_DOCS] D -- 否 --> F[首轮不注入] E --> G[Recall 继续运行] @@ -447,6 +458,15 @@ flowchart TD 用户级 Memory 最重要的场景。Fast 结果复用 `selectModelCandidateDocuments` 为 Model Manifest 已经算好的候选，不产生额外扫描或 I/O。 +**100 ms 是上限而不是固定开销**：Fast 结果在 Recall 扫完 Memory 树之后才发布， +所以真正决定它能否赶上的是**扫描耗时**，不是打分耗时（后者是微秒级）。 +`recall-scan-latency.test.ts` 在真实临时 Memory 树上实测：200 篇约 29 ms、 +500 篇约 70 ms、1000 篇约 130 ms。对能在预算内扫完的树（普通用户的常见情况）， +Fast 就绪后继续等待只是在等一个本设计已经假定赶不上的 Model Selector， +因此等待会在 Fast 就绪时立即结束。超过约 1000 篇时扫描本身就超预算， +该轮会付满 100 ms 且什么都投不到——提前结束等待只能把这种情况**限制住**， +消除不了它。 + **Fast 阶段的边界**：Fast 结果就是确定性结果，因此它只能解决**时机**问题， 解决不了**匹配**问题。与文档没有任何词面重叠的 Query 产生不了 Fast 结果， 这类 Query 在无工具回合仍然拿不到 Memory——只有 Model Selector 能覆盖它们， diff --git a/packages/core/src/core/client.test.ts b/packages/core/src/core/client.test.ts index 0b9a56784f..34b082b3f8 100644 --- a/packages/core/src/core/client.test.ts +++ b/packages/core/src/core/client.test.ts @@ -5156,10 +5156,9 @@ hello ), ); - // Held for the budget, then the fast result goes out instead of nothing. - await vi.advanceTimersByTimeAsync(99); - expect(mockTurnRunFn).not.toHaveBeenCalled(); - await vi.advanceTimersByTimeAsync(1); + // The deterministic result was already published, so the budget has + // nothing left to wait for and the request goes out without spending it. + await vi.advanceTimersByTimeAsync(0); await done; expect(mockTurnRunFn).toHaveBeenCalledWith( @@ -5171,6 +5170,58 @@ hello ); }); + it('ends the initial wait as soon as the deterministic result arrives', async () => { + vi.useFakeTimers(); + // Stands in for the memory-tree scan: the fast result is not ready when + // the wait begins, but lands well before the budget expires. + const SCAN_MS = 30; + mockMemoryManager.recall.mockImplementation((_root, _query, options) => { + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, SCAN_MS); + // Selector never settles — stands in for a slow round trip. + return new Promise(() => {}); + }); + + mockTurnRunFn.mockReturnValue( + (async function* () { + yield { type: 'content', value: 'Hello' }; + })(), + ); + client['chat'] = { + addHistory: vi.fn(), + getHistory: vi.fn().mockReturnValue([]), + } as unknown as GeminiChat; + + const done = fromAsync( + client.sendMessageStream( + [{ text: 'What do you know about me?' }], + new AbortController().signal, + 'prompt-id-fast-early-return', + ), + ); + + await vi.advanceTimersByTimeAsync(SCAN_MS - 1); + expect(mockTurnRunFn).not.toHaveBeenCalled(); + // The remaining ~70 ms of budget is never spent. + await vi.advanceTimersByTimeAsync(1); + expect(mockTurnRunFn).toHaveBeenCalledWith( + 'test-model', + expect.arrayContaining([ + expect.stringContaining('Fast deterministic result.'), + ]), + expect.any(AbortSignal), + ); + + await vi.advanceTimersByTimeAsync(100); + await done; + }); + it('still delivers the model-selected result at ToolResult after a fast initial delivery', async () => { vi.useFakeTimers(); let settleRecall: @@ -5507,12 +5558,18 @@ hello it('delivers no fast result when the turn is cancelled inside the initial window', async () => { vi.useFakeTimers(); const controller = new AbortController(); + // The fast result must still be in flight when the abort lands, + // otherwise the wait would already have ended on its arrival and there + // would be no window left to cancel inside. mockMemoryManager.recall.mockImplementation((_root, _query, options) => { - options.onFastResult?.({ - prompt: '## Relevant memory\n\nFast deterministic result.', - selectedDocs: [fastDoc('/m/fast.md', '- terse')], - strategy: 'heuristic', - }); + setTimeout(() => { + if (options.abortSignal?.aborted) return; + options.onFastResult?.({ + prompt: '## Relevant memory\n\nFast deterministic result.', + selectedDocs: [fastDoc('/m/fast.md', '- terse')], + strategy: 'heuristic', + }); + }, 80); return new Promise(() => {}); }); @@ -5737,9 +5794,11 @@ hello }); it('should hold the main request for exactly the initial recall budget when recall never settles', async () => { - // Recall never settles. Fake timers pin the budget contract: the + // Recall never settles and never publishes a deterministic result, so + // nothing can end the wait early. Fake timers pin the ceiling: the // request must still be blocked 1 ms inside the budget and proceed, - // without memory, the moment the budget expires. + // without memory, the moment the budget expires. This is also the shape + // of a memory tree whose scan is slower than the budget. vi.useFakeTimers(); mockMemoryManager.recall.mockReturnValue(new Promise(() => {})); diff --git a/packages/core/src/core/client.ts b/packages/core/src/core/client.ts index 3d6e5c966d..d39052fb64 100644 --- a/packages/core/src/core/client.ts +++ b/packages/core/src/core/client.ts @@ -304,6 +304,15 @@ function sameActiveGoalProjection( * 3. Aborted-and-discarded by every cleanup path (resetChat, * MaxSessionTurns, etc.) or replaced when a new UserQuery arrives. */ +/** + * Publication slot for recall's deterministic result, plus a one-shot + * listener for its arrival. + */ +type MemoryFastResultBox = { + current: RelevantAutoMemoryPromptResult | null; + onArrive?: () => void; +}; + type MemoryPrefetchHandle = { promise: Promise; /** Set by promise.finally(). null until the promise settles. */ @@ -320,8 +329,11 @@ type MemoryPrefetchHandle = { * Deterministic result published by recall before it blocks on the model * selector. A box rather than a plain field because recall can invoke the * callback before this handle object exists. + * + * `onArrive` lets the bounded initial wait stop as soon as there is + * something to deliver, instead of always spending the whole budget. */ - fastResultRef: { current: RelevantAutoMemoryPromptResult | null }; + fastResultRef: MemoryFastResultBox; /** True after the fast result was injected — prevents double-inject and double-log. */ fastDelivered: boolean; /** Paths injected by the fast phase, excluded from the later refined delivery. */ @@ -911,11 +923,32 @@ export class GeminiClient { return null; } - if (handle.settledAt === null && waitMs > 0) { + // `waitMs` is a ceiling, not a fixed cost. The wait ends on whichever + // comes first: recall settling, the deterministic result being published, + // cancellation, or the budget expiring. + // + // Ending on the fast result matters more than it looks. That result is + // published once recall has scanned the memory tree, which is milliseconds + // for an ordinary tree — while the model selector is a network round trip + // that this design already assumes will miss the budget. Spending the rest + // of the budget after the fast result is in hand therefore buys an + // outcome that almost never arrives, and charges every user turn for it. + // See `recall-scan-latency.test.ts` for the scan measurements. + // + // The preference order is unchanged: whatever ends the wait, the code + // below still prefers a settled recall over the fast result. + if ( + handle.settledAt === null && + handle.fastResultRef.current === null && + waitMs > 0 + ) { await new Promise((resolve) => { const finish = () => { clearTimeout(timer); handle.controller.signal.removeEventListener('abort', finish); + if (handle.fastResultRef.onArrive === finish) { + handle.fastResultRef.onArrive = undefined; + } resolve(); }; @@ -926,6 +959,7 @@ export class GeminiClient { handle.controller.signal.addEventListener('abort', finish, { once: true, }); + handle.fastResultRef.onArrive = finish; void handle.promise.then(finish, finish); } }); @@ -2862,9 +2896,7 @@ export class GeminiClient { } else { signal.addEventListener('abort', onParentAbort, { once: true }); } - const fastResultRef: { - current: RelevantAutoMemoryPromptResult | null; - } = { current: null }; + const fastResultRef: MemoryFastResultBox = { current: null }; const promise = this.config .getMemoryManager() .recall( @@ -2877,6 +2909,7 @@ export class GeminiClient { abortSignal: controller.signal, onFastResult: (result) => { fastResultRef.current = result; + fastResultRef.onArrive?.(); }, }, ) diff --git a/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json b/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json index 635e07fd4e..8adf013b54 100644 --- a/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json +++ b/packages/core/src/memory/__fixtures__/auto-memory-recall-eval.json @@ -153,6 +153,27 @@ "title": "Deployment 发布说明", "description": "Release 流程", "body": "Mixed-language deployment notes." + }, + { + "id": "ru-deploy", + "type": "project", + "title": "Процесс развёртывания", + "description": "Контрольный список релиза", + "body": "Перед выкатом проверить мониторинг и переключатели отката." + }, + { + "id": "el-auth", + "type": "reference", + "title": "Ρύθμιση ταυτοποίησης", + "description": "Επίλυση προβλημάτων σύνδεσης", + "body": "Έλεγχος λήξης συνεδρίας και δικαιωμάτων." + }, + { + "id": "fr-perf", + "type": "project", + "title": "Réduction du démarrage à froid", + "description": "Objectifs de performance trimestriels", + "body": "Mesurer la latence côté serveur avant toute optimisation." } ], "cases": [ @@ -479,6 +500,27 @@ "category": "semantic-no-lexical", "query": "首屏加载太久了", "relevantIds": ["zh-perf"] + }, + { + "id": "other-script-ru-deploy", + "category": "other-script", + "query": "процесс развёртывания", + "relevantIds": ["ru-deploy"], + "expectedTopId": "ru-deploy" + }, + { + "id": "other-script-el-auth", + "category": "other-script", + "query": "ρύθμιση ταυτοποίησης", + "relevantIds": ["el-auth"], + "expectedTopId": "el-auth" + }, + { + "id": "other-script-fr-perf", + "category": "other-script", + "query": "démarrage à froid", + "relevantIds": ["fr-perf"], + "expectedTopId": "fr-perf" } ] } diff --git a/packages/core/src/memory/recall-eval.test.ts b/packages/core/src/memory/recall-eval.test.ts index 5ee294aa36..d342613c2f 100644 --- a/packages/core/src/memory/recall-eval.test.ts +++ b/packages/core/src/memory/recall-eval.test.ts @@ -43,6 +43,7 @@ const categories = new Set([ 'nfkc', 'body-only', 'semantic-no-lexical', + 'other-script', 'no-result', ] as const); @@ -322,6 +323,15 @@ const isEnglish = (testCase: EvalCase) => testCase.category === 'english'; */ const isSemanticNoLexical = (testCase: EvalCase) => testCase.category === 'semantic-no-lexical'; +/** + * Alphabetic scripts outside ASCII and CJK — Cyrillic, Greek, and accented + * Latin here. The pre-change tokenizer kept only `[a-z0-9]{3,}` runs, so + * these queries produced no tokens at all and the deterministic path was + * unconditionally silent; the shipped tokenizer keeps whole non-CJK letter + * runs instead. + */ +const isOtherScript = (testCase: EvalCase) => + testCase.category === 'other-script'; const isCjk = (testCase: EvalCase) => testCase.category === 'chinese' || testCase.category === 'japanese' || @@ -334,11 +344,26 @@ function formatPercent(value: number | null): string { return value === null ? 'n/a' : `${(value * 100).toFixed(1)}%`; } +/** + * Expected Recall@5 of a scorer that ignores the query and returns five + * documents drawn uniformly at random from the corpus. Each labeled document + * has a `RECALL_AT / corpusSize` chance of being among them, so the expected + * per-case recall — and therefore the mean over any slice — is that ratio. + * + * Printed beside the measured columns because a small corpus flatters the + * headline: on a pool this size "100% Recall@5" is a much weaker statement + * than it reads, and a reader comparing designs needs the floor to calibrate + * against. It is exact rather than sampled, so the table stays deterministic. + */ +function randomBaselineRecallAt5(corpusSize: number): number { + return corpusSize === 0 ? 0 : Math.min(1, RECALL_AT / corpusSize); +} + describe('auto-memory recall evaluation', () => { it('loads a labeled corpus covering every required category', () => { const fixture = loadFixture(); expect(fixture.cases.length).toBeGreaterThanOrEqual(30); - expect(fixture.cases.length).toBeLessThanOrEqual(50); + expect(fixture.cases.length).toBeLessThanOrEqual(60); expect(new Set(fixture.cases.map((testCase) => testCase.category))).toEqual( categories, ); @@ -434,6 +459,7 @@ describe('auto-memory recall evaluation', () => { ['english', isEnglish], ['cjk', isCjk], ['mixed', isMixed], + ['other-script', isOtherScript], ['semantic-no-lexical', isSemanticNoLexical], ] as const; @@ -445,6 +471,14 @@ describe('auto-memory recall evaluation', () => { '| --- | --- | --- | --- |', ]; + const randomFloor = randomBaselineRecallAt5(fixture.docs.length); + lines.splice( + 2, + 0, + `corpus: ${fixture.docs.length} documents, ${fixture.cases.length} cases — a query-blind random scorer returning ${RECALL_AT} documents scores ${formatPercent(randomFloor)} Recall@5 on this pool`, + '', + ); + for (const [label, filter] of rows) { const before = evaluate( fixture, @@ -477,6 +511,24 @@ describe('auto-memory recall evaluation', () => { expect(lines.length).toBeGreaterThan(5); }); + /** + * Guards the headline against a corpus so small that Recall@5 is nearly + * free. This is a property of the fixture, not of the scorer: shrink the + * corpus far enough and every metric approaches 100% for any design. + */ + it('keeps the corpus large enough for Recall@5 to discriminate', () => { + const fixture = loadFixture(); + const randomFloor = randomBaselineRecallAt5(fixture.docs.length); + + expect(randomFloor).toBeLessThanOrEqual(0.25); + const after = evaluate( + fixture, + selectRelevantAutoMemoryDocuments, + (testCase) => !isSemanticNoLexical(testCase), + ); + expect(after.recallAt5!).toBeGreaterThan(randomFloor * 3); + }); + it('produces deterministic summaries', () => { const fixture = loadFixture(); expect(evaluate(fixture, selectRelevantAutoMemoryDocuments)).toEqual( diff --git a/packages/core/src/memory/recall-scan-latency.test.ts b/packages/core/src/memory/recall-scan-latency.test.ts new file mode 100644 index 0000000000..7f3672f53f --- /dev/null +++ b/packages/core/src/memory/recall-scan-latency.test.ts @@ -0,0 +1,182 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import * as fs from 'node:fs/promises'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import type { Config } from '../config/config.js'; +import { getAutoMemoryFilePath } from './paths.js'; +import { resolveRelevantAutoMemoryPromptForQuery } from './recall.js'; +import { selectRelevantAutoMemoryDocumentsByModel } from './relevanceSelector.js'; +import { ensureAutoMemoryScaffold } from './store.js'; + +/** + * Measures the part of the initial-turn budget nothing else measures. + * + * `recall-delivery-eval.test.ts` times the deterministic *scoring*, which is + * microseconds. That is not what decides whether the fast path delivers. The + * fast result is published from `onFastResult`, which fires only after recall + * has enumerated, read, and parsed every topic file — and this branch removed + * the 200-document cap for recall, so that scan grows with the memory tree. + * If the scan alone exceeds `INITIAL_MEMORY_RECALL_WAIT_MS`, the turn pays the + * full budget *and* delivers nothing, which is strictly worse than before. + * + * So this file measures wall-clock time from the recall call to the fast + * callback, against a real temporary memory tree, with the model selector + * mocked to hang the way a network round trip does. + * + * Timings are machine-dependent and CI is shared, so the assertions are + * deliberately loose; the printed table is the artifact worth reading. What + * is asserted is the structural claim: the fast result lands well inside the + * budget at memory-tree sizes users can plausibly reach. + */ + +vi.mock('./relevanceSelector.js', () => ({ + selectRelevantAutoMemoryDocumentsByModel: vi.fn(), +})); + +/** Mirrors INITIAL_MEMORY_RECALL_WAIT_MS in client.ts. */ +const INITIAL_BUDGET_MS = 100; +const TOPIC_COUNTS = [200, 500, 1000] as const; +const REPEATS = 5; + +let tempDir: string; +const projectRootByCount = new Map(); + +async function buildMemoryTree(topicCount: number): Promise { + const projectRoot = path.join(tempDir, `project-${topicCount}`); + await fs.mkdir(projectRoot, { recursive: true }); + await ensureAutoMemoryScaffold( + projectRoot, + new Date('2026-04-01T00:00:00.000Z'), + ); + + const referenceDir = path.dirname( + getAutoMemoryFilePath(projectRoot, 'reference/topic-0000.md'), + ); + await fs.mkdir(referenceDir, { recursive: true }); + + // Bodies are sized like real notes rather than one-liners: the scan reads + // and parses whole files, so a corpus of stubs would understate the cost. + const filler = 'Historical note about an unrelated subsystem. '.repeat(20); + await Promise.all( + Array.from({ length: topicCount }, (_, index) => + fs.writeFile( + path.join(referenceDir, `topic-${String(index).padStart(4, '0')}.md`), + [ + '---', + 'type: reference', + `name: Topic ${index}`, + `description: Reference note number ${index} about deployment history`, + '---', + '', + filler, + index === topicCount - 1 ? 'The saved codeword is SCANBENCH.' : '', + '', + ].join('\n'), + 'utf-8', + ), + ), + ); + + return projectRoot; +} + +/** Wall-clock ms from the recall call until the fast result is published. */ +async function measureTimeToFastResultMs(projectRoot: string): Promise { + let elapsed = Number.NaN; + const startedAt = performance.now(); + const recall = resolveRelevantAutoMemoryPromptForQuery( + projectRoot, + 'what is the saved scanbench codeword for deployment', + { + config: { + getSessionId: () => 'session-scan-bench', + getModel: () => 'qwen3-coder-plus', + } as Config, + onFastResult: () => { + elapsed = performance.now() - startedAt; + }, + }, + ); + + // Let the pending recall settle so it does not leak into the next sample. + await recall; + return elapsed; +} + +describe('auto-memory recall scan latency', () => { + beforeAll(async () => { + tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'recall-scan-bench-')); + // The selector stands in for the network round trip: it must not settle + // before the fast callback, or the measurement would race it. Returning + // an empty selection keeps recall finishing promptly after that. + vi.mocked(selectRelevantAutoMemoryDocumentsByModel).mockResolvedValue([]); + for (const topicCount of TOPIC_COUNTS) { + projectRootByCount.set(topicCount, await buildMemoryTree(topicCount)); + } + }, 120_000); + + afterAll(async () => { + if (tempDir) { + await fs.rm(tempDir, { recursive: true, force: true }); + } + }); + + it('publishes the fast result well inside the initial budget', async () => { + const rows: Array<[number, number, number]> = []; + + for (const topicCount of TOPIC_COUNTS) { + const projectRoot = projectRootByCount.get(topicCount)!; + // Warm the page cache so the first sample does not report cold I/O as + // the steady-state cost. + await measureTimeToFastResultMs(projectRoot); + + const samples: number[] = []; + for (let i = 0; i < REPEATS; i += 1) { + samples.push(await measureTimeToFastResultMs(projectRoot)); + } + samples.sort((a, b) => a - b); + const median = samples[Math.floor(samples.length / 2)]; + const worst = samples[samples.length - 1]; + rows.push([topicCount, median, worst]); + + expect(Number.isFinite(median)).toBe(true); + } + + const [smallest] = rows; + // The ordinary case must leave the rest of the budget to spare. Loose + // because CI is shared; the table is what carries the detail. + expect(smallest[0]).toBe(TOPIC_COUNTS[0]); + expect(smallest[1]).toBeLessThan(INITIAL_BUDGET_MS / 2); + + console.log( + [ + '', + 'Scan gate — time from recall start to fast result (single project scope)', + `initial budget: ${INITIAL_BUDGET_MS} ms`, + '', + `| topics | median | worst of ${REPEATS} | share of budget | fast result inside budget? |`, + '| --- | --- | --- | --- | --- |', + ...rows.map( + ([topicCount, median, worst]) => + `| ${topicCount} | ${median.toFixed(1)} ms | ${worst.toFixed(1)} ms | ${((median / INITIAL_BUDGET_MS) * 100).toFixed(1)}% | ${worst < INITIAL_BUDGET_MS ? 'yes' : 'no'} |`, + ), + '', + 'The fast result is only available once this scan completes, so this is', + 'the real precondition for the fast path delivering anything — not the', + 'scoring cost, which is microseconds.', + '', + 'Where a row reads "no", the turn spends the whole budget and still', + 'delivers nothing, which is worse than the zero-wait behaviour this', + 'branch replaced. That is why the wait ends on the fast result rather', + 'than always running to the ceiling: it removes the cost for every tree', + 'small enough to scan in time, and bounds it for the rest.', + ].join('\n'), + ); + }, 120_000); +}); diff --git a/packages/core/src/memory/recall.test.ts b/packages/core/src/memory/recall.test.ts index 1c76852af0..20309f494f 100644 --- a/packages/core/src/memory/recall.test.ts +++ b/packages/core/src/memory/recall.test.ts @@ -340,6 +340,57 @@ describe('auto-memory relevant recall', () => { ); }); + it('tokenizes alphabetic scripts outside ASCII and CJK', () => { + // `[a-z0-9]{3,}` produced no tokens at all for these, so the + // deterministic path was unconditionally silent — no fast result, and a + // silent selector-failure fallback. + const cyrillic = memoryDoc( + 'ru.md', + 'project', + 'Процесс развёртывания', + '', + '', + ); + const greek = memoryDoc('el.md', 'reference', 'Ρύθμιση σύνδεσης', '', ''); + const accented = memoryDoc('fr.md', 'project', 'Démarrage à froid', '', ''); + const docs = [cyrillic, greek, accented]; + + expect( + selectRelevantAutoMemoryDocuments('развёртывания', docs)[0]?.filename, + ).toBe('ru.md'); + expect( + selectRelevantAutoMemoryDocuments('σύνδεσης', docs)[0]?.filename, + ).toBe('el.md'); + expect( + selectRelevantAutoMemoryDocuments('démarrage', docs)[0]?.filename, + ).toBe('fr.md'); + }); + + it('does not let a Latin run swallow the CJK that follows it', () => { + // `\p{L}` also matches Han, so a naive alphabetic class would tokenize + // `abc漢字` as one run and stop matching either half on its own. + const latin = memoryDoc('latin.md', 'reference', 'abc', '', ''); + const han = memoryDoc('han.md', 'reference', '漢字', '', ''); + + expect( + selectRelevantAutoMemoryDocuments('abc漢字', [latin, han]).map( + (doc) => doc.filename, + ), + ).toEqual(['latin.md', 'han.md']); + }); + + it('still ignores runs shorter than three characters', () => { + const doc = memoryDoc('go.md', 'reference', 'go go go', '', ''); + + expect(selectRelevantAutoMemoryDocuments('go', [doc])).toEqual([]); + // Two Cyrillic letters are below the threshold for the same reason. + expect( + selectRelevantAutoMemoryDocuments('до', [ + memoryDoc('ru.md', 'reference', 'до свидания', '', ''), + ]), + ).toEqual([]); + }); + it('breaks score ties by recency, not by document type', () => { // Every type carries the same title, so the only thing separating these // documents is the tie-break. An alphabetical type comparison orders them diff --git a/packages/core/src/memory/recall.ts b/packages/core/src/memory/recall.ts index cea468a75d..625d53e8b6 100644 --- a/packages/core/src/memory/recall.ts +++ b/packages/core/src/memory/recall.ts @@ -72,8 +72,35 @@ const TYPE_KEYWORDS: Record = { reference: ['reference', 'dashboard', 'ticket', 'docs', 'doc', 'link'], }; -const RECALL_TOKEN_RUN = - /[a-z0-9]{3,}|[\p{Script=Han}\p{Script=Hiragana}\p{Script_Extensions=Katakana}\p{Script=Hangul}]+/gu; +/** + * Scripts tokenized as code-point bigrams because they are written without + * word separators, so a whole run is one unsegmentable token. + */ +const CJK_CLASS = + '[\\p{Script=Han}\\p{Script=Hiragana}\\p{Script_Extensions=Katakana}\\p{Script=Hangul}]'; + +/** + * One token run: either a CJK run (bigram-tokenized below) or a run of at + * least three non-CJK letters, marks, and digits (kept whole). + * + * The alphabetic alternative is `\p{L}`-based rather than `[a-z0-9]`, so + * Cyrillic, Greek, Arabic, and accented Latin produce tokens instead of + * silently producing none. It excludes CJK per character rather than relying + * on alternation order: `\p{L}` also matches Han, so a plain class would let + * a run starting in Latin swallow the CJK that follows it and turn + * `abc漢字` into one token. + * + * Scripts written without spaces and outside the CJK set (Thai, Khmer, Lao) + * still collapse into a single long token. That is no worse than the previous + * behaviour of producing nothing, but it is not segmentation. + */ +const RECALL_TOKEN_RUN = new RegExp( + `${CJK_CLASS}+|(?!${CJK_CLASS})[\\p{L}\\p{N}](?:(?!${CJK_CLASS})[\\p{L}\\p{M}\\p{N}]){2,}`, + 'gu', +); + +/** Whether a matched run is CJK, and therefore bigram-tokenized. */ +const CJK_RUN_START = new RegExp(`^${CJK_CLASS}`, 'u'); function normalizeRecallText(text: string): string { return text.normalize('NFKC').toLowerCase(); @@ -103,14 +130,14 @@ function tokenize(text: string): string[] { for (const match of normalized.matchAll(RECALL_TOKEN_RUN)) { const run = match[0]; - if (run.charCodeAt(0) <= 0x7f) { - addToken(run); - } else { + if (CJK_RUN_START.test(run)) { let previous = ''; for (const codePoint of run) { if (previous) addToken(previous + codePoint); previous = codePoint; } + } else { + addToken(run); } } @@ -146,35 +173,48 @@ function toolAliases(toolName: string): string[] { ); } -function isActiveToolUsageMemory( - doc: ScannedAutoMemoryDocument, +/** + * Build the active-tool noise predicate once per recall rather than deriving + * it per document. The alias set depends only on `recentTools`, so computing + * it inside the per-document filter re-derived up to + * `MAX_RECENT_TOOL_NAMES_FOR_MEMORY` alias lists for every scanned document — + * which recall now does over an uncapped pool. + * + * Returns a predicate rather than a boolean so both filter sites share the + * hoisting; a `recentTools`-free recall short-circuits to a constant `false`. + */ +function createActiveToolUsageFilter( recentTools: readonly string[], -): boolean { +): (doc: ScannedAutoMemoryDocument) => boolean { if (recentTools.length === 0) { - return false; + return () => false; } - const haystack = [doc.title, doc.description, normalizeBody(doc.body)] - .join(' ') - .toLowerCase(); - const namesActiveTool = recentTools.some((toolName) => - toolAliases(toolName).some((alias) => haystack.includes(alias)), - ); - if (!namesActiveTool) { - return false; + const aliases = Array.from(new Set(recentTools.flatMap(toolAliases))); + if (aliases.length === 0) { + return () => false; } - if ( - DURABLE_ACTIVE_TOOL_MEMORY_MARKERS.some((marker) => + return (doc) => { + const haystack = [doc.title, doc.description, normalizeBody(doc.body)] + .join(' ') + .toLowerCase(); + if (!aliases.some((alias) => haystack.includes(alias))) { + return false; + } + + if ( + DURABLE_ACTIVE_TOOL_MEMORY_MARKERS.some((marker) => + haystack.includes(marker), + ) + ) { + return false; + } + + return ACTIVE_TOOL_USAGE_MEMORY_MARKERS.some((marker) => haystack.includes(marker), - ) - ) { - return false; - } - - return ACTIVE_TOOL_USAGE_MEMORY_MARKERS.some((marker) => - haystack.includes(marker), - ); + ); + }; } function scoreDocument( @@ -246,9 +286,8 @@ function selectModelCandidateDocuments( modelCandidates: ScannedAutoMemoryDocument[]; fallbackDocs: ScannedAutoMemoryDocument[]; } { - const eligible = docs.filter( - (doc) => !isActiveToolUsageMemory(doc, recentTools), - ); + const isActiveToolNoise = createActiveToolUsageFilter(recentTools); + const eligible = docs.filter((doc) => !isActiveToolNoise(doc)); const lexical = selectRelevantAutoMemoryDocuments( query, eligible, @@ -498,13 +537,14 @@ export async function resolveRelevantAutoMemoryPromptForQuery( }; } + const isActiveToolNoise = createActiveToolUsageFilter( + options.recentTools ?? [], + ); const selectedDocs = fallbackDocs ?? selectRelevantAutoMemoryDocuments( query, - docs.filter( - (doc) => !isActiveToolUsageMemory(doc, options.recentTools ?? []), - ), + docs.filter((doc) => !isActiveToolNoise(doc)), limit, ); const strategy: RelevantAutoMemoryPromptResult['strategy'] =