* refactor(core): rewrite compression prompt to 9-section claude-code-style format
Replaces the <state_snapshot> XML template with a numbered 9-section
structure that mandates verbatim preservation of user messages, including
the historical chronological list (section 6). The new format is
designed to pair with post-compact file/image restoration (separate work)
so the agent can resume long single-turn tasks without losing intent.
* refactor(core): align compaction trigger string with new 9-section prompt
The user-turn trigger injected after the system prompt still said
'generate the <state_snapshot>' from the old XML prompt era. Updated to
'produce the 9-section summary' to match Task 1's new prompt format.
Also tightens the prompt test to assert the specific user-message
verbatim mandate (not just the word 'verbatim' anywhere) so a future
regression that drops the mandate won't silently pass.
* feat(core): add postCompactAttachments module with file path extractor
extractRecentFilePaths walks history newest-first and returns the top N
unique file paths touched by read_file/write_file/edit/replace tool calls.
Pure function, no side effects, no state cache — readiness for the next
compaction-rewrite tasks.
* refactor(core): simplify extractRecentFilePaths internals
Three small cleanups from code review:
- Map<string, number> -> Set<string> (the index value was never read)
- Guard against maxFiles <= 0 explicitly (avoids returning 1 result
when caller passes 0 as a 'disable' sentinel)
- Document 'replace' as a legacy alias for 'edit' so a future cleanup
pass does not delete it as apparent dead code
Adds one test covering the maxFiles=0 path.
* feat(core): add image extractor with source-tool metadata
extractRecentImages walks history newest-first, collects up to N image
inlineData parts, and attributes each one to the model+functionCall that
preceded it (when one exists). Returns chronological order so callers
can render a meaningful 'last visual state ends here' strip.
* feat(core): add size-adaptive file reader for post-compact restore
readFileSizeAdaptive reads a file and returns one of: embed (full content
for files ≤ maxTokens × 4 chars), reference (path-only for large files),
missing (deleted since last touch), or binary (non-text content). The
embed/reference distinction mirrors claude-code's compact_file_reference
vs file attachment behavior, but without introducing new message types.
* refactor(core): harden readFileSizeAdaptive size accounting
Three corrections from code review:
- Import CHARS_PER_TOKEN from tokenEstimation.ts (canonical) instead of
redeclaring locally, preventing silent drift between modules.
- Compare decoded character length, not raw byte length, against the
cap. Otherwise a 10k-char Chinese file would be ~30k bytes and would
be mis-classified as 'reference' despite fitting the budget.
- Rename FileReadResult -> FileEmbedResult to avoid a name collision
with the unrelated FileReadResult interface in fileUtils.ts.
Adds a CJK-text test that catches the byte/char regression.
* feat(core): add file restoration block composer
buildFileRestorationBlocks reads each candidate file, classifies it as
embed/reference/missing/binary, and emits one consolidated reference
block (path-only list) plus one user message per embedded small file.
Total embed size is capped at POST_COMPACT_TOKEN_BUDGET; over-budget
files downgrade to reference.
* test(core): make budget test actually exercise the downgrade path
The previous version of this test wrote 3 files totalling 9k chars
against a 200k char budget. The assertions trivially passed regardless
of whether the budget check existed in the implementation.
The new version writes 11 files of 20k chars (each at the per-file cap)
so the budget is exhausted by the 10th and the 11th must downgrade
from embed to reference. Asserts both: file 11 appears in the reference
block, and file 11's content does NOT appear in any embed block.
* feat(core): add image restoration block composer
buildImageRestorationBlock emits a single user message whose first part
is a metadata header (turn index + source tool name + args per image),
followed by the inlineData parts themselves. Handles user-paste images
(no source tool) by labeling them as 'user-provided'.
* feat(core): add composePostCompactHistory orchestrator
Assembles the full post-compact history in order:
summary → model ack → file references → file embeds → image block.
Each section is built by the per-concern extractors and builders added
in previous tasks. This is the single integration point that
chatCompressionService.compress() will call once the wire-up task lands.
* feat(core)!: rewrite compress() to claude-code-style full-history model
Replaces the split-point + tail-preservation model with full-history
compression + composePostCompactHistory. The entire curated history is
sent to the summary side-query, and the post-compact history is
assembled by the new composer (summary + ack + file restores + image
restore).
BREAKING: the previously-exported findCompressSplitPoint,
splitPointRetainingTrailingPairs, COMPRESSION_PRESERVE_THRESHOLD, and
TOOL_ROUND_RETAIN_COUNT will be removed in the next commit. Tests that
exercise them remain failing temporarily.
* chore(core): remove obsolete split-point compression infrastructure
Deletes findCompressSplitPoint, splitPointRetainingTrailingPairs,
COMPRESSION_PRESERVE_THRESHOLD, MIN_COMPRESSION_FRACTION, and
TOOL_ROUND_RETAIN_COUNT, plus the tests that exercised them. The new
behavior is covered by composePostCompactHistory and its unit tests.
Also cleans up:
- Stale orphan-strip comment in compress() that described the deleted
manual-trigger orphan-funcCall handling.
- TEST_ONLY.COMPRESSION_PRESERVE_THRESHOLD hatch in client.ts.
- Docstring references in config.ts and compactionInputSlimming.ts.
* test(core): add single-turn computer-use compaction regression
Reproduces the scenario the rewrite targets: one user prompt kicks off
many screenshot tool calls. Asserts that (a) the user prompt is carried
into the summary verbatim and (b) the 3 most recent screenshots are
restored as an image block with source-tool metadata. This is the canary
test for the computer-use UX claim made in the design discussion.
* docs(core): remove stale "split point" references in tokenEstimation comments
Aligns the docstrings with the new compose-based compression flow. The
"split point" and "splitter" concepts no longer exist after the rewrite.
* fix(core): iterate parts reverse so parallel tool calls keep the last N
Real-session E2E surfaced a bug: a model that issues N parallel ReadFile
calls puts all N functionCall parts in ONE model+fc content. The
extractor's outer history walk is newest-first, but the inner parts
walk was forward — so for a 6-parallel batch hitting the cap of 5,
the FIRST 5 parts won and the actually-most-recent (last-listed) file
was dropped.
Fix: walk parts in reverse within each content. Applied symmetrically
to extractRecentImages (same shape, even rarer trigger).
Adds a regression test that hits a 6-parallel batch.
* fix(core): code-review fixes — fence escape, path sanitize, alias removal
- CommonMark-safe fence in file embed blocks. The old 3-backtick fence
closed prematurely when a file's content contained a triple-backtick
run (Markdown, CLAUDE.md, JSDoc with code examples) — leaking the
remainder as unfenced text. Now uses a fence one longer than the
longest backtick run in the content.
- Strip control characters (\r, \n, \t) from file paths before
rendering into attachment markdown. Paths come from model-controlled
history; a \n could inject markdown structure. The actual path stays
intact for tool calls — only the displayed string is sanitized.
- Remove the historyForCompression alias for curatedHistory in
compress(). The alias was added as a comment anchor during the
rewrite but didn't carry semantic information.
* refactor(core): rewrite compression prompt to <state_snapshot> XML with 9 claude-aligned sections
Replaces the 9-section numbered-text prompt with qwen-code's original
<state_snapshot> XML envelope, but with the 9 inner section tags
content-aligned to claude-code:
<primary_request_and_intent>
<key_technical_concepts>
<files_and_code_sections>
<errors_and_fixes>
<problem_solving>
<all_user_messages>
<pending_tasks>
<current_work>
<next_step>
Also:
- <scratchpad> -> <analysis>, stripped by postProcessSummary (saves
~600-800 tokens of CoT noise per compaction).
- "Resume directly..." trailer moved out of the prompt body and into
postProcessSummary (no longer re-generated by the model every
compaction; lives once in code with our own wording).
- Section 6 verbatim-policed mandate relaxed to "chronological, include
short messages like 'ok' / 'continue'" — matches claude-code intent
without forcing the model to literally copy long user messages.
E2E (qwen3.6-plus, 6 substantial .ts files + thorough analysis):
raw history 6508 -> summary 1513 (after strip ~947), 38% history
compression. Overall context 24642 -> 20647 reported (-16%), with
another ~664 tokens actually saved by the post-strip but not
reflected in the conservative token-math heuristic.
* docs(core): code-review polish on XML prompt rewrite
Four small follow-ups from review of 641a0eadd:
- prompts.ts: rewrite getCompressionPrompt's stale JSDoc — it still
described the deleted 9-section numbered-text format and the
verbatim mandate that was relaxed.
- chatCompressionService.ts: clarify the token-math comment so it's
obvious the ~1000 token deduction covers the full compression
system prompt + kick-off user turn (not any single instruction)
and that newTokenCount slightly over-counts because <analysis>
gets stripped by postProcessSummary downstream.
- postCompactAttachments.ts: add a NOTE comment on the <analysis>
strip regex covering its strict-tag-match assumption and
multi-block / non-greedy semantics.
- postCompactAttachments.test.ts: replace the four lazy
`await import('./postCompactAttachments.js')` calls inside the
postProcessSummary describe block with one top-level static import
— consistent with how every other describe in the file imports.
* docs(core): drop stale duplicate sentence left in token-math comment
* fix(core): address wenshao review on PR #4599 (correctness + security + ergonomics)
Seven follow-ups from wenshao's review of the compaction rewrite.
Critical:
- newTokenCount now includes restoration-block tokens via
estimateContentChars over extraHistory[2..]. Previously the formula
only counted side-query output, so up to 5 × 5K (files) + 3 × image
tokens were missing — letting the inflation guard miss and the
cheap-gate under-estimate the next prompt size (Finding 1).
- composePostCompactHistory now merges every file restoration block
and the image block into a single user Content following the model
ack. The previous output had consecutive user roles, which
geminiChat.test.ts:6289 enforces against and Gemini providers
reject with 400 "consecutive same-role content" (Finding 2).
- Preserve a trailing model+functionCall through compaction so a
pending functionResponse (sitting in sendMessageStream's
pendingUserMessage) has a matching call. Without this, hard-rescue
auto-compaction mid tool-use loop produces a user+functionResponse
with no preceding model+functionCall → API 400. This restores the
protection the split-point in-flight fallback used to provide.
When the funcCall lands without attachments it folds into the
ack's own model Content to avoid model→model adjacency (Finding 3).
- composePostCompactHistory now takes an optional workspaceRoot and
silently skips file paths that resolve outside it.
extractRecentFilePaths picks up paths from model functionCall args
regardless of whether the tool execution succeeded; without a
boundary check, an adversarial model that issued
read_file('/etc/passwd') — denied by the permission system —
would still have its path extracted and re-read into the next
prompt. compress() passes config.getTargetDir() as the boundary
(Finding 4).
Suggestions:
- composePostCompactHistory + buildFileRestorationBlocks +
readFileSizeAdaptive all take optional AbortSignal and short-
circuit / pass it to readFile's { signal } option. Cancelled
compactions stop on the next file read (Finding 5).
- postProcessSummary fallback no longer re-injects the raw
<analysis> block when the strip leaves nothing. The new
stripAnalysisBlock helper runs the closed-tag strip AND an
unclosed-tag strip (handles 'model ran out of output tokens
before closing'). If both leave nothing, postProcessSummary
emits '[Summary unavailable]' rather than leaking scratchpad
(Finding 6).
- firePostCompactEvent now receives stripAnalysisBlock(summary) so
hook consumers see the same text that lands in history. The
resume trailer stays out of the hook payload — that's wrapper
decoration for the next agent turn, not state for consumers
(Finding 8a).
Docs:
- Update the geminiChat.ts comment around `trigger: 'auto'` to
describe what the trigger actually does post-refactor (hook event
categorization) rather than the deleted manual-only orphan-strip
it used to guard against (Finding 8b).
Regression tests cover all six fixable code-path changes
(role alternation, trailing funcCall preservation, workspace
boundary, abort propagation, closed-tag fallback strip, unclosed-tag
fallback strip).
* fix(core): add getTargetDir to geminiChat auto-compression test mock
The R3.4 end-to-end auto-compression test drives the real
ChatCompressionService, which reads config.getTargetDir() for the
post-compact file-restoration workspace boundary. The geminiChat mock
config lacked getTargetDir, so the test threw "config.getTargetDir is
not a function" on CI. Add the mock to unblock the failing Test jobs.
* feat(core): configurable compaction retention + computer-use screenshot trigger
Add four env-overridable chatCompression settings (priority env >
settings > default):
- maxRecentFilesToRetain (QWEN_COMPACT_MAX_RECENT_FILES, default 5)
- maxRecentImagesToRetain (QWEN_COMPACT_MAX_RECENT_IMAGES, default 3)
- enableScreenshotTrigger (QWEN_COMPACT_SCREENSHOT_TRIGGER, default true)
- screenshotTriggerThreshold(QWEN_COMPACT_SCREENSHOT_THRESHOLD, default 50)
The screenshot trigger fires auto-compaction once tool-returned images
accumulate to the threshold even when token usage is below the auto tier,
so computer-use sessions don't drown the model in stale screenshots. It
counts only images nested in functionResponse.parts (tool results), not
user pastes, and runs only in the would-be-NOOP path when enabled.
Fix a latent bug surfaced while wiring the trigger: extractRecentImages
only inspected top-level inlineData parts, but convertToFunctionResponse
nests tool media under functionResponse.parts — so post-compact
restoration recovered ZERO tool screenshots in real sessions, while unit
tests stayed green against a fabricated top-level shape. It now walks both
shapes; the image counter and tests use the real nested shape.
Remove the now-defunct contextPercentageThreshold deprecation warning (the
field was already dropped from ChatCompressionSettings) and its tests, and
document the four new settings.
* test(core): assert screenshot trigger can't re-fire post-compaction; fix misleading docs
Code-review follow-up. The screenshot trigger counts only images nested in
functionResponse.parts. Compaction replaces those with the summary and
re-embeds survivors as TOP-LEVEL parts in the restoration block, which the
counter ignores — so the tool-image count always resets to ~0 and the
trigger cannot immediately re-fire, independent of maxRecentImages.
The resolveCompactionTuning JSDoc and the settings.md note previously warned
of a non-existent "maxRecentImages near threshold => compact every turn"
loop. Correct both, and add a regression test asserting
countToolResponseImages() is 0 on composePostCompactHistory output.
* fix(core): guard readFileSizeAdaptive against multi-GB reads; cover composer 4-entry branch
wenshao review round 2 on PR #4599.
- readFileSizeAdaptive now stats the file first and short-circuits to a
reference when its byte size exceeds maxChars*4 (the safe UTF-8 upper
bound — a file larger than that cannot fit within maxChars chars). This
stops a multi-GB file the agent previously touched from being slurped
into a Buffer and exhausting the heap mid-compaction, exactly when we're
trying to reduce memory. A large binary file now references rather than
reading to binary-detect.
- Add a test for composePostCompactHistory's 4-entry branch (attachments +
trailing model+functionCall) producing [user(summary), model(ack),
user(attachments), model(fc)]. This is the common mid-tool-loop
compaction case; a model->model adjacency here is a provider 400. Prior
tests only covered the 2-entry fold (no attachments) and 3-entry (no
trailing fc) shapes.
* fix(core): resolve symlinks in workspace boundary; guard compose against throws
wenshao review round 3 on PR #4599 (two Criticals).
- isInsideWorkspace now resolves symlinks via realpathSync (safeRealpath,
with a lexical fallback for non-existent paths). A symlink living inside
the workspace but pointing outside (e.g. workspace/.env -> ~/.ssh/id_rsa)
previously passed the lexical boundary check and had its target read and
embedded into the post-compact history sent to the provider. Added a
RED-verified security regression test (secret embedded under the old
lexical check; rejected under realpath).
- Wrap composePostCompactHistory in try/catch inside compress(). The
summary side-query has already succeeded at that point, so a
restoration-assembly throw (disk I/O / malformed history) previously
escaped to sendMessageStream, crashing the active turn AND bypassing the
COMPRESSION_FAILED breaker. It now degrades to summary + ack.
* fix(core): close 4 compaction Criticals from review round 4
wenshao review round 4 on PR #4599.
- isSummaryEmpty now checks the STRIPPED summary: a response that is only an
<analysis> block (no <state_snapshot>) strips to empty, so it takes the
COMPRESSION_FAILED_EMPTY_SUMMARY path instead of "succeeding" with
`[Summary unavailable]` as the agent's only context (silent amnesia).
- Manual /compress strips a trailing ORPHANED model+functionCall before
composing — it has no pending functionResponse, so preserving it would
emit model[fc] then the next user text turn -> API 400. Auto-compaction
still keeps it (the pending response pairs with it).
- The restoration-failure catch fallback now folds a trailing
model+functionCall into the ack turn, so a pending functionResponse
(auto mid-tool-loop) keeps its matching call even on the degraded path.
- extractRecentFilePaths skips file paths whose tool call FAILED (an error
functionResponse), so a denied read_file is never re-read off disk during
compaction — closing a permission-bypass side channel.
RED-verified regression tests for the empty-summary, orphan-strip, and
permission-bypass fixes. Corrected the postProcessSummary comment.
* test(core): cover composePostCompactHistory catch-fallback; document fold text drop
wenshao review round 5 on PR #4599.
- Regression test for the restoration-failure catch fallback: mock
composePostCompactHistory to reject and assert compaction still returns
COMPRESSED (no escape to sendMessageStream / breaker bypass) with the
trailing functionCall folded into the ack and the trailing text dropped.
- Document that the fold branch intentionally keeps only functionCall parts
(the trailing turn's text is already captured in the summary); the
asymmetry with the with-attachments branch is deliberate.
365409366d