From 2eb5cd6df558c99e978949fe85ea548f18357f4d Mon Sep 17 00:00:00 2001 From: jinye Date: Fri, 7 Aug 2026 14:10:37 +0800 Subject: [PATCH] feat(serve): observe daemon and child memory against real denominators (#8423) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(serve): observe daemon memory pressure against a real denominator The daemon samples its own RSS and heap but has nothing to divide them by, so nothing in `/daemon/status` says whether a figure is fine or nearly fatal. #8245 landed the denominator (`limits.memory`); this turns it into a reading. `runtime.memory.pressure` reports `level`, `ratio`, `source`, and the six raw figures behind them. The level is the worse of two independent ratios, because the two failure modes are independent: a container dies by RSS against its cgroup limit, while a process on a large host can exhaust V8's heap long before RSS is a meaningful fraction of the machine. Reporting only one hides whichever failure the deployment is actually heading for. `source` names which ratio produced the level, and `unknown` says the daemon could not measure itself — which a consumer must not read as healthy. The denominator is `availableMemoryMb`, not `effectiveBudgetMb`: pressure asks how close this process is to being killed, and what kills it is the cgroup limit or host memory. An operator's budget is a policy number, so classifying against it would report `critical` for a daemon in no danger. `--memory-pressure-mode` is `off | observe`, default `observe`. Both modes report every figure; only `observe` also raises the `daemon_memory_pressure` warning, so `off` leaves the top-level `status` rollup untouched — the thresholds are inherited from an interactive-CLI monitor and are not yet calibrated for a long-running daemon, and a deployment that alerts on `status` needs the reading without the verdict. There is deliberately no `enforce`: nothing here remediates, and a value a caller can pass but never use is a dead switch. It arrives with the enforcement. Scope is the daemon root process only. `childRssCoverage` still reads `primary_only` and says so on the wire; aggregate child RSS and channel workers are separate measurements and land separately. Severity is `warning` at every level including `critical`, because `error` would make `rollupStatus` return `error` for the whole daemon — too strong a claim to stake on uncalibrated thresholds. Refs #8051. Co-Authored-By: Claude Opus 5 * feat(serve): report aggregate ACP child RSS, not just the primary's (#8462) * test(serve): close the under-determined assertions review probed The automated review mutation-probed this diff and found several assertions that were live but under-determined — each mutant it names kept the whole suite green. All confirmed locally, and all now fail: - Deleting `level !== 'normal'` from the issue gate raised daemon_memory_pressure on a healthy daemon and flipped top-level status to warning on every response — the exact false positive `--memory-pressure-mode off` exists to opt out of. Now covered on both sides: nothing raised at a realistic denominator, exactly one warning at a denominator sized to land this process in `soft`. - Summing children over `list()` instead of `listManaged()` dropped a draining-but-process-holding workspace while `activeAcpChildren` still counted it. The draining bridge now reports RSS, so the byte count can only come from that child. - The message's denominator ternary had no coverage; inverting it sent an operator hunting RSS growth during a heap-driven incident. - A truthiness guard on `ageMs` turned a measured-fresh reading (age exactly 0, when a status read lands in the sampler's millisecond) into `null`, which the field's own docs say never means fresh. - The multi-contributor age test listed ages ascending, so a plain-overwrite accumulator produced the same answer as Math.max. Reordered descending, which kills last-wins and first-wins both. Two declaration-only hunks — the issue-code union member and the `pressure` field — were guarded by tsc alone, which vitest does not run. Both are now pinned at runtime by asserting the code string and the full key set. Also fixes a real display defect: `toFixed(0)` renders a ratio of 0.795 as "hard at 80%", and 80% is critical's documented threshold. One decimal, so the number and the level cannot contradict each other. And corrects a JSDoc claim of mine that was simply wrong: `pressure` is absent not only for direct-embed but on the bootstrap /daemon/status route, which omits runtime.memory wholesale even though the budget is resolved — and that window is not just startup, since a daemon whose runtime fails to start serves the bootstrap app for its lifetime. Co-Authored-By: Claude Opus 5 * refactor(serve): model a per-child heap partition of the daemon budget (#8508) * feat(serve): add the child-heap admission primitives, unwired Groundwork for #8182 step 2. Nothing calls any of this yet, so no child is sized differently and no spawn is refused. `ProcessRegistry.committedProcessCount` counts attached children plus reservations that have not attached. That is the figure admission has to key on: `reserve()` inserts its token synchronously before `spawn()`, so two racing spawns each see the other, while neither appears in `activeProcessCount` until its child attaches. A child leaves the count on exit rather than when `terminate()` starts, so a channel swap counts twice while the old process winds down — deliberate, since its memory is still resident. `getAcpMemoryArgs(explicitMb?)` takes an optional share that bypasses both the module cache and the raise-only guard. Both bypasses are load-bearing. The cache, because the share depends on how many children are live now rather than on the host. The guard, because a budget-derived share is normally *below* the daemon's own heap limit, so routing it through `targetMB > currentLimitMB` would drop the flag, silently restore the overcommit, and leave every test green — the trap against a multi-GB runner, and mutation-checking it by reinstating the guard fails two tests. `createChildHeapPolicy` holds the mode, the budget, and the would-be refusal counter, and answers `decide(concurrentChildren)`. The refusal is derived from the unclamped quotient, not from `recommendedChildShareMb`, because that function clamps *up* to the 512 MB floor: past the point where the pool stops covering the count its answer saturates and can no longer distinguish "barely does not fit" from "wildly does not fit". `ChildHeapPoolExhaustedError` with both transport mappings — REST 503 with Retry-After, ACP `child_heap_pool_exhausted` — added together, since the two mappings are hand-written and drift silently otherwise. Refusing at spawn rather than at registration is the correction #8182 demands: registration allocates nothing, so this surfaces as "no new session in this workspace right now", which is true and retryable. Refs #8182. Co-Authored-By: Claude Opus 5 * feat(serve): size each ACP child by concurrently live children Wires the primitives from the previous commit into the spawn path, behind `--child-heap-mode off | observe | enforce`, default `observe`. Under `enforce` a child's `--max-old-space-size` is a share of the child pool divided by the children concurrently committed at the moment it spawns — read from the shared ProcessRegistry after `reserve()`, so two racing spawns each see the other. When the pool cannot cover another child at the 512 MB floor the spawn is refused with ChildHeapPoolExhaustedError, which is what turns a per-child ceiling into an aggregate bound: concurrent children can never exceed pool/512. Keyed on concurrency, never on registrations. A dormant workspace has no child, so it costs nothing — the specific correction #8182 records against the withdrawn proposal, which would have shrunk a lone live child to 614 MB because of 24 idle registrations. Default `observe` computes the share and the admission decision and applies neither, counting the refusals that would have happened. The divisor has never been checked against a real multi-workspace deployment, and a non-zero count is how an operator learns enforcement would have broken them without being broken. It also catches the case worth worrying about: a channel swap counts the dying child alongside its replacement, so on a saturated pool enforcement could refuse a restart and leave that workspace with no child at all. Excluding terminating children would authorise real overcommit to dodge a hypothetical refusal, so the count reports it instead. Ceilings already granted are not revisited — V8 cannot lower them — so granted ceilings transiently exceed the pool. Acceptable: the flag is a ceiling, not a reservation, and a workspace with no live sessions has no child and picks up the current share on its next spawn. `limits.memory.enforced` stops being a required literal `false`. #8245 made it one so a client could never mistake that namespace for enforcement that had not shipped; it has now, so the field is a boolean derived from the mode — and stays `false` under `observe`, which applies nothing. Refs #8182. Co-Authored-By: Claude Opus 5 * docs(serve): correct the claims child-heap enforcement makes false Two sentences in the protocol doc described the memory section as unconditionally observational: "a required `enforced: false`", and "no child spawn argument derives from these values, and no request is refused on their basis". Both are false under `--child-heap-mode enforce`, so both are rewritten rather than left to rot — `enforced` is now documented as the boolean that answers exactly this, and the refusal is documented with its wire shape on both transports. Also documents `childHeap.refusals` as the calibration signal, since a would-be-refusal count is useless if operators do not know to read it before switching to `enforce`; the flag row in the three operator docs; and the design doc's Part 1, which listed applying a share as a compatibility risk without recording how that was resolved. The end-to-end test asserts the policy reaches a real booted daemon's status with `enforced: false` under the default mode — the wire type in that test is a hand-written mirror, so its `enforced: false` literal had to widen too, which is the check that caught the type not being widened everywhere. Refs #8182. Co-Authored-By: Claude Opus 5 * test(serve): cover both branches of the enforced tripwire `enforced` was only ever asserted false — the unit tests build no policy and the end-to-end daemon runs the default `observe` mode, so the branch that makes the field worth having was untested. Hardcoding it back to `false` passed everything. Also pins `childHeap: null` as distinct from a policy in `off` mode: the first says no policy exists (direct-embed, or the bootstrap window before the runtime is built), the second says one exists and computes nothing. Refs #8182. Co-Authored-By: Claude Opus 5 * fix(serve): partition the child pool so granted ceilings stay inside it Review was right that the previous design did not deliver the aggregate bound it claimed. Sizing each child by the count live at *its* spawn bounds the child count but not the memory: V8 cannot lower a running child's ceiling, so grants accumulate as P + P/2 + P/3 + ... = P x H(n). Reproduced exactly — 9557 MB authorised against a 3687 MB pool at seven children on an 8 GB host, and 61355 MB against 15360 MB at the limit on 32 GB. That is 2.6x and 4x the pool, which is what the policy exists to prevent. Grant accounting alone does not fix it: the first child would take the whole pool and the second would be refused immediately. Keeping the invariant requires early children not to receive the whole pool, so the ceiling is now a fixed partition — childPoolMb / maxConcurrentChildren, constant for every child, with maxConcurrentChildren itself derived from the pool and capped at MAX_DAEMON_WORKSPACES. The sum is then n x ceiling <= pool by construction, with no ledger of outstanding grants and no dependence on arrival order. Tested as an invariant across four host sizes: fill the daemon to its admission limit and the authorised total still fits. The cost is deliberate and now documented rather than hidden: a lone workspace on a 32 GB host gets 614 MB rather than the pool, because any child may still be running when the house fills. An 8 GB host admits seven concurrent children at 526 MB each. Also from review: - The policy is no longer built for an injected `deps.bridge`. That bridge carries its own channel and never reaches the factory the policy rides on, so status could report `enforced: true` while nothing was being sized. - Both transport mappings now have direct tests. They are hand-written beside each other and drift silently; the spawn-policy tests cannot catch a wire regression. - Swept the "does not size any child" claim, which enforce makes false, out of the CLI help text, ServeOptions docs, the two operator tables, and the e2e header comment. The 17-configuration table realigns wholesale because that cell was its widest — whitespace only. Refs #8182. Co-Authored-By: Claude Opus 5 * refactor(serve): model the child heap partition, defer applying it Review established that the refusal counter cannot tell an operator whether enforcement is safe, and that is the ground the enforcing mode stood on. While observing, children run on the host-derived ceiling (16384 MB on a 32 GB host), so a workload needing 2 GB of old space is healthy with zero refusals and OOMs the moment a 614 MB partition is applied. The counter measures admission pressure, not ceiling adequacy. Rather than ship a switch with no safe way to decide when to turn it on, `enforce` is removed. `--child-heap-mode` is `off | observe`, and the mode that would apply the partition arrives with the measurement that justifies it: peak old-space per child, compared against the modeled ceiling. That is a real measurement chain — the child reports rss and cpu today, and `--max-old-space-size` bounds old space specifically, so neither rss nor heapUsed answers the question. With nothing applying the partition, the machinery that existed only to apply it goes too rather than shipping unreachable: `getAcpMemoryArgs(explicitMb?)`, `ChildHeapPoolExhaustedError` and both transport mappings, and `limits.memory.enforced` reverts to the required literal `false` it was before. The spawn path is untouched again; the factory asks the policy what it would decide purely so the count is real. Also fixes the zero-pool defect review found, which the removed clamp caused: forcing at least one admissible child on a 512 MB host — where the root reserve consumes the whole 256 MB budget — produced a ceiling of 0, and `--max-old-space-size=0` is V8's *default* heap, not a zero ceiling. A pool that cannot cover one child at the floor now reports `maxConcurrentChildren: 0` and `perChildCeilingMb: null`, and the test that enshrined the old behaviour is inverted. Status now publishes `maxConcurrentChildren` and `perChildCeilingMb`, so an operator can judge the partition against their own workload — the substitute for a counter that cannot judge it for them. Every claim that a zero refusal count means the partition is safe to apply is removed from the flag help, the operator docs, the protocol doc, and the design doc. Refs #8182. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 * fix(serve): repair the child-heap assertion and the reservation leak Three findings review raised against #8508 after the partition became observation-only, all still live on this branch now that it has merged. The status assertion in `run-qwen-serve.test.ts` failed on head: it used `toEqual` against `{ mode, refusals }` while the wire also carries `maxConcurrentChildren` and `perChildCeilingMb`, so the suite was red at 217 passed / 1 failed. The local type restating the wire shape was short the same two fields. Both are filled in, and the assertion stays `toEqual` so an unannounced field still fails it — the two derived figures get matchers because this suite boots a real daemon and the pool follows the machine. What they have to satisfy is now pinned separately: a fixed ceiling times the number admitted must fit inside the pool it partitions, which is the whole reason the partition bounds anything. `decide()` and `getAcpMemoryArgs()` ran between `reserve()` and the `try` that cancels the reservation. `childHeapPolicy` is a public `createSpawnChannelFactory` option, so `decide()` is caller code and may throw; the spawn then rejected with the token held for the process lifetime, inflating `committedProcessCount` for every later spawn. Both calls move inside the `try`. The regression test is mutation-verified — reverting the move gives `expected 1 to be +0`. `ServeOptions.memoryBudgetMb` still promised a `childHeapMode: 'enforce'` that sizes children and refuses spawns. No such mode exists. Co-Authored-By: Claude Opus 5 * fix(serve): report no child-heap partition under `off` `snapshot()` returned `maxConcurrentChildren` and `perChildCeilingMb` unconditionally, so a daemon run with `--child-heap-mode off` still published a partition — 7 children at 526 MB on an 8 GB host — under a mode whose documentation says "do not model it". Review raised it, and it mattered more than it looked: with `enforce` gone, `off` and `observe` differed only in whether `refusals` incremented, so nothing on the wire distinguished a model that was switched off from one in force. Both figures are now `null` under `off`, which required widening `maxConcurrentChildren` to `number | null` in the daemon type and the SDK mirror. `null` rather than `0`: zero is already the computed answer for a pool too small to host one child at the 512 MB floor, and collapsing the two would tell an operator who disabled the model that their host cannot run anything. That leaves three distinguishable states — no policy at all (`childHeap: null`), a policy modeling nothing (`mode: 'off'` with null figures), and a live model — and each now has a test. The `off` unit test previously asserted only `refusals`, so its name ("models nothing at all when off") promised more than it checked. It now covers the figures, with a sibling test pinning 7 / 526 under `observe` on the same budget so nulling them unconditionally cannot satisfy both. Mutation-verified in both directions. Co-Authored-By: Claude Opus 5 * fix(serve): never model a child heap ceiling below the documented minimum `perChildCeilingMb` is `min(floor(pool / maxConcurrentChildren), legacyChildCeilingMb)`. The first term is at least `MIN_CHILD_HEAP_MB` by construction; the second is `floor(available / 2)` and is not, so the `Math.min` could publish a ceiling *below* the `minChildHeapMb` sitting beside it in the same snapshot: avail=768 --memory-budget-mb 1024 pool=512 legacyCeil=384 perChild=384 avail=1023 --memory-budget-mb 1024 pool=767 legacyCeil=511 perChild=511 Unreachable from a derived budget — the pool reaches 0 first — but an explicit budget has a floor of 1024 while available memory does not, and `docs/users/qwen-serve.md` tells operators on exactly these hosts to pass that flag. The documented remedy is what reaches the band. Refuse the model rather than shrink under the floor, with `maxConcurrentChildren` zeroed in lockstep: a ceiling no child may run at is not a partition, and "one child fits" beside a null ceiling is the same contradiction from the other side. Nothing is applied today so the impact was a wrong published figure, but this is the number the partition asks to be judged by and the one an `enforce` mode would hand to `--max-old-space-size`. The existing matrix resolves derived budgets only, which is why the mutation sweep came back clean; add the `budgetMb` axis, asserting in each case the shape that makes it reachable, and pin the inclusive boundary (1024/1024 -> one child at 512) so nulling unconditionally cannot pass instead. Also, in the same review pass: - Split usable-gauge handling into numerator and denominator. Coercing an unusable numerator to 0 published `rssBytes: 0, rssRatio: 0, level: 'normal', source: 'rss'` — a daemon that measured nothing, indistinguishable from an idle one, which is the confusion `source: 'unknown'` and `sampled: 0` exist to prevent everywhere else here. An unusable numerator now retires its own side. Zero stays a reading for a numerator and not for a denominator. - Document that `rssRatio` divides by host total under `availableMemorySource: 'host'`, so it is a lower bound on real pressure there — a denominator problem no threshold calibration addresses. - Document that `refusals` counts channel swaps at full occupancy (the terminating child is counted until it exits) and equals the total spawn count on a host too small to model a partition. Deliberately not fixed by giving the comparison swap headroom, which would admit a 26th ceiling against a 25-child pool. - Keep the sampler's rejection handler as a documented backstop — the shipped `refreshChildResource` never rejects, but it is an optional `async` interface member, so a foreign implementation throwing early would otherwise surface as an unhandled rejection — and give it the workspace so it is attributable across the fan-out. - Test hygiene: drop a duplicated `enforced` assertion; replace a host- dependent `expect.any(Number)` with a key-set pin plus a branch, since a small host now legitimately reports no partition; use `vi.spyOn(Date, 'now')` over direct assignment; reuse the exported `ChildHeapMode` on the child-heap side, leaving the independent `memoryPressureMode` switch alone. Reported by @wenshao. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 Co-authored-by: Shaojin Wen --- ...daemon-capacity-model-and-memory-bounds.md | 16 +- docs/developers/daemon/17-configuration.md | 74 +-- docs/developers/daemon/19-observability.md | 40 ++ .../daemon/20-quickstart-operations.md | 4 +- docs/developers/qwen-serve-protocol.md | 6 +- docs/users/qwen-serve.md | 2 + packages/acp-bridge/package.json | 4 + packages/acp-bridge/src/bridge.test.ts | 38 ++ packages/acp-bridge/src/bridge.ts | 9 +- packages/acp-bridge/src/bridgeTypes.ts | 9 +- .../acp-bridge/src/child-heap-policy.test.ts | 175 ++++++++ packages/acp-bridge/src/child-heap-policy.ts | 174 +++++++ .../acp-bridge/src/process-registry.test.ts | 40 ++ packages/acp-bridge/src/process-registry.ts | 16 + packages/acp-bridge/src/spawnChannel.test.ts | 73 +++ packages/acp-bridge/src/spawnChannel.ts | 26 +- packages/cli/src/commands/serve.test.ts | 69 +++ packages/cli/src/commands/serve.ts | 36 +- .../src/serve/daemon-memory-pressure.test.ts | 185 ++++++++ .../cli/src/serve/daemon-memory-pressure.ts | 170 +++++++ packages/cli/src/serve/daemon-status.test.ts | 425 +++++++++++++++++- packages/cli/src/serve/daemon-status.ts | 234 +++++++++- packages/cli/src/serve/fast-path.test.ts | 36 ++ packages/cli/src/serve/fast-path.ts | 30 ++ .../cli/src/serve/routes/daemon-status.ts | 3 + packages/cli/src/serve/run-qwen-serve.test.ts | 109 ++++- packages/cli/src/serve/run-qwen-serve.ts | 68 ++- packages/cli/src/serve/server.ts | 3 + packages/cli/src/serve/types.ts | 39 +- packages/sdk-typescript/src/daemon/types.ts | 86 +++- 30 files changed, 2121 insertions(+), 78 deletions(-) create mode 100644 packages/acp-bridge/src/child-heap-policy.test.ts create mode 100644 packages/acp-bridge/src/child-heap-policy.ts create mode 100644 packages/cli/src/serve/daemon-memory-pressure.test.ts create mode 100644 packages/cli/src/serve/daemon-memory-pressure.ts diff --git a/docs/design/2026-07-31-daemon-capacity-model-and-memory-bounds.md b/docs/design/2026-07-31-daemon-capacity-model-and-memory-bounds.md index 3f78a77379..7af90f826d 100644 --- a/docs/design/2026-07-31-daemon-capacity-model-and-memory-bounds.md +++ b/docs/design/2026-07-31-daemon-capacity-model-and-memory-bounds.md @@ -113,11 +113,23 @@ The real control is admission at spawn time keyed on concurrently live children, - **It must never raise a ceiling.** Clamping to `legacyChildCeilingMb` is what makes the policy safe to apply unconditionally; without it the minimum-budget constant and an over-large explicit flag both inflate the share. - **The spawn path has a trap.** `getAcpMemoryArgs()` emits `--max-old-space-size` only when its computed target exceeds the _spawning daemon's own_ `heap_size_limit` (`spawnChannel.ts:27-34`). A budget-derived share is normally below that, so a naive change is silently dropped and the overcommit returns. The regression test must assert the flag survives a value below the test process's own limit. +**Modeled, not yet applied, as `--child-heap-mode`.** A first attempt sized each child by the count live at _its_ spawn; review showed that bounds the child count but not the memory, since V8 cannot lower a running child's ceiling and grants accumulate as P x H(n) — 2.6x the pool at seven children on 8 GB. The model is now a fixed partition: one constant ceiling for every child, with admission capped so the total stays inside the pool by construction. + +Applying it is deliberately deferred. The compatibility point above is why: enforcing changes child GC and OOM behaviour, and nothing yet tells an operator beforehand whether their workload fits the ceiling. The refusal count cannot — children run on the host-derived ceiling while observing, so it measures admission pressure, not ceiling adequacy. The enforcing mode ships with the measurement that justifies it: peak old-space per child, compared against the modeled ceiling. + ### Part 2 — Observe, with a denominator, before enforcing -The existing five-second sampler gains the effective memory limit, `v8.getHeapStatistics().heap_size_limit`, and aggregate child RSS across **all** workspace children and channel workers rather than the primary alone. Status gains `runtime.memory { level, ratio, source }` and two codes on the closed issue union at `daemon-status.ts:70-85`. +This part splits by what each piece measures, because the denominators are independent and the cheap one is worth landing first. -The mode flag follows the established `--mcp-client-budget` / `--mcp-budget-mode` idiom: `off | warn | enforce`, defaulting to `warn` when a budget is set, with `enforce` rejected at boot until a later change earns it. Nothing in this part remediates. +**Ships first — the daemon root against its own two limits.** Status gains `runtime.memory.pressure`, carrying `level`, `ratio`, `source`, and the six raw figures the ratios come from, plus one code — `daemon_memory_pressure` — on the closed issue union. `source` names which denominator produced the level: RSS against detected cgroup/host memory, or V8 heap used against `getHeapStatistics().heap_size_limit`. Both are needed, because a container dies by the first and a process on a large host exhausts the second long before RSS is a meaningful fraction of the machine. It reads `process.memoryUsage()` where the status response is built rather than extending the sampler: the reading is wanted per status request, not per five-second tick, and the sampler's ring is a separate consumer that can be fed once there is something to trend. + +**Landed second — aggregate child RSS.** The feared second failure mode (a child that exits mid-poll) turned out to need no new mechanism: the per-bridge cache already drops a reading when the channel dies, and the same `isChannelLive()` predicate removes that workspace from both the sum and the count in one synchronous pass, so numerator and denominator stay consistent. `childRssCoverage` now reads `active_children` and `runtime.memory.children` carries the sum with a `sampled` count beside it. + +**Still deferred — the rest of the tree.** Channel workers report no RSS at all today, so covering them means building the reporting path first; the children's own MCP descendants are invisible for the same reason, since each child self-reports only its own process. Neither figure is process-tree memory, and the response says so. + +An earlier draft of this section also promised a second issue code for a stale observation. It is not in the first change: nothing in it can produce a stale reading, since the figures are sampled synchronously as the response is built. A code with no reachable producer is an unwritable test and a contract clients would handle for nothing. It arrives with the polled measurements that can actually go stale. + +The mode flag exposes only `off | observe`, defaulting to `observe`. An earlier draft borrowed the `--mcp-budget-mode` triple and offered `enforce` with a boot-time rejection, which is a dead switch: a value a caller can pass but never use. The enforcing value arrives with the enforcement. Nothing in this part remediates — no forced GC, no eviction, no session closure, no process termination. This is deliberately promoted ahead of the byte-cap work. It is the only piece whose value does not depend on the rest of the design being correct, and every limit chosen later should be calibrated against its data rather than guessed. #8093's limit table is a weaker argument for this ordering than it first appears, and the weaker form is the honest one: `prompt: 384 MiB` is exactly `normalAdmissionBytes` and therefore redundant, but the 256 MiB categories are _not_ dead — a single category reaching 256 MiB binds well before total normal usage reaches the 384 MiB ceiling. The problem with the table is simply that the constants are uncalibrated, which is what observation fixes. diff --git a/docs/developers/daemon/17-configuration.md b/docs/developers/daemon/17-configuration.md index dd5ac43c7c..3cb294d54e 100644 --- a/docs/developers/daemon/17-configuration.md +++ b/docs/developers/daemon/17-configuration.md @@ -6,42 +6,44 @@ This page collects every setting that affects the `qwen serve` daemon and its ad ## CLI flags (`qwen serve`) -| Flag | Type | Default | Effect | -| --------------------------------------- | ---------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `--hostname ` | string | `127.0.0.1` | Bind address. Loopback values: `127.0.0.1`, `localhost`, `::1`, `[::1]`. Non-loopback requires a bearer token at boot. `host:port` input is rejected with guidance to use `--port`. | -| `--port ` | number | `4170` | Listen port; `0` means ephemeral. | -| `--token ` | string | env | Bearer token. Overrides `QWEN_SERVER_TOKEN` and is trimmed at boot. It appears in the process command line, so prefer env in deployments. | -| `--require-auth` | boolean | `false` | Extends bearer auth to loopback and `/health`; boot refuses to start without a token. | -| `--workspace ` | absolute path / repeatable | `process.cwd()` | Startup workspace runtime; repeat to register additional isolated runtimes. The first is primary. Every value must be absolute and a directory; canonicalized at boot. | -| `--memory-project-scope ` | `git-root` / `workspace` | `git-root` | Project-memory partitioning. `git-root` shares memory among workspaces at the same Git root; `workspace` isolates by exact workspace directory. Overrides `QWEN_CODE_MEMORY_PROJECT_SCOPE`. | -| `--max-sessions ` | number | `32` | Per-workspace active session cap. `0` / `Infinity` means unlimited; `NaN` / negative values throw. | -| `--max-total-sessions ` | number | derived for multiple startup/restored workspaces | Daemon-wide active session cap. When omitted, a finite default is derived once from the per-workspace cap and startup/restored workspace count. `0` / `Infinity` means unlimited. | -| `--max-pending-prompts-per-session ` | number | `5` | Accepted but pending/running prompt cap per session. Excess prompt returns 503. `0` / `Infinity` means unlimited; negative or non-integer values throw. | -| `--max-connections ` | number | `256` | HTTP listener `server.maxConnections`; `0` / `Infinity` means unlimited. | -| `--enable-session-shell` | boolean | `false` | Enables direct `POST /session/:id/shell` execution. Requires bearer token, and every call must carry a session-bound `X-Qwen-Client-Id`. | -| `--event-ring-size ` | number | `8000` | Per-session SSE replay ring; soft cap is `1_000_000`. | -| `--compacted-replay-max-bytes ` | positive integer | `4194304` | Byte cap for the bounded in-memory replay snapshot returned by `POST /session/:id/load`; hard cap is `268435456`. | -| `--memory-budget-mb ` | integer in `[1024, 1048576]` | 50% of cgroup-constrained or host memory, capped at the flag maximum (1048576 MB) | Total memory budget for the daemon process tree, capped at resolved available memory. Observed and reported under `limits.memory` in daemon status; it does not size any child process. Boot rejects out-of-range values. | -| `--http-bridge` | boolean | `true` | Stage 1 bridge mode. `--no-http-bridge` still falls back to http-bridge and prints to stderr. | -| `--mcp-client-budget ` | positive integer | unset | Sets `WorkspaceMcpBudget.clientBudget` and forwards it to the ACP child through `childEnvOverrides`. | -| `--mcp-budget-mode ` | `off` / `warn` / `enforce` | `warn` when budget is set, otherwise `off` | Sets `WorkspaceMcpBudget.mode`; `enforce` requires `--mcp-client-budget`. | -| `--external-tool-guard-mode ` | `off` / `required` | `off` | Enables the managed ACP external pre-execution Guard. `required` fails startup unless its loopback provider completes the v1 handshake. | -| `--external-tool-guard-endpoint ` | loopback HTTP(S) origin | unset | Provider origin used only in `required` mode. It must be origin-only and use `127.0.0.1`, `localhost`, or `::1`; paths, credentials, redirects, and proxy routing are rejected. | -| `--external-tool-guard-timeout-ms ` | integer `100..30000` | `3000` | Per-handshake and per-prepare deadline. A timeout fails startup during the handshake or fails the invocation closed during a turn. | -| `--allow-origin ` | repeatable string | unset | Cross-origin allowlist that replaces the default CORS denial. `*` allows any origin but requires a token. | -| `--allow-private-auth-base-url` | boolean | `false` | Allows `/workspace/auth/provider` to install localhost / private-network auth provider `baseUrl`; use only in trusted local development. | -| `--prompt-deadline-ms ` | positive integer | unset | Server-side prompt wallclock limit in ms. Timeout aborts and returns an error. | -| `--writer-idle-timeout-ms ` | positive integer | unset | Per-SSE-connection idle timeout in ms. The daemon closes the SSE connection when no event is sent for this duration. | -| `--channel-idle-timeout-ms ` | non-negative integer | `0` | How long to keep the ACP child alive after the last session closes. `0` means reclaim immediately. | -| `--initialize-timeout-ms ` | positive integer | `10000` | ACP child request timeout, including the initialize handshake (ms). | -| `--session-reap-interval-ms ` | non-negative integer | `60000` | Session reaper scan interval; `0` disables it. | -| `--session-idle-timeout-ms ` | non-negative integer | `1800000` | Disconnected-session idle reaping time; `0` disables it. | -| `--rate-limit` / `--no-rate-limit` | boolean | env / off | Enables per-tier HTTP rate limiting for prompt, mutation, and read routes. | -| `--rate-limit-prompt ` | positive integer | `10` | Prompt request limit per window; requires rate limiting to be enabled. | -| `--rate-limit-mutation ` | positive integer | `30` | Mutation request limit per window; requires rate limiting to be enabled. | -| `--rate-limit-read ` | positive integer | `120` | Read request limit per window; requires rate limiting to be enabled. | -| `--rate-limit-window-ms ` | integer `>= 1000` | `60000` | Rate limit window length; requires rate limiting to be enabled. | -| no flag | - | - | `QWEN_SERVE_NO_MCP_POOL=1` fully disables the pool. | +| Flag | Type | Default | Effect | +| --------------------------------------- | ---------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--hostname ` | string | `127.0.0.1` | Bind address. Loopback values: `127.0.0.1`, `localhost`, `::1`, `[::1]`. Non-loopback requires a bearer token at boot. `host:port` input is rejected with guidance to use `--port`. | +| `--port ` | number | `4170` | Listen port; `0` means ephemeral. | +| `--token ` | string | env | Bearer token. Overrides `QWEN_SERVER_TOKEN` and is trimmed at boot. It appears in the process command line, so prefer env in deployments. | +| `--require-auth` | boolean | `false` | Extends bearer auth to loopback and `/health`; boot refuses to start without a token. | +| `--workspace ` | absolute path / repeatable | `process.cwd()` | Startup workspace runtime; repeat to register additional isolated runtimes. The first is primary. Every value must be absolute and a directory; canonicalized at boot. | +| `--memory-project-scope ` | `git-root` / `workspace` | `git-root` | Project-memory partitioning. `git-root` shares memory among workspaces at the same Git root; `workspace` isolates by exact workspace directory. Overrides `QWEN_CODE_MEMORY_PROJECT_SCOPE`. | +| `--max-sessions ` | number | `32` | Per-workspace active session cap. `0` / `Infinity` means unlimited; `NaN` / negative values throw. | +| `--max-total-sessions ` | number | derived for multiple startup/restored workspaces | Daemon-wide active session cap. When omitted, a finite default is derived once from the per-workspace cap and startup/restored workspace count. `0` / `Infinity` means unlimited. | +| `--max-pending-prompts-per-session ` | number | `5` | Accepted but pending/running prompt cap per session. Excess prompt returns 503. `0` / `Infinity` means unlimited; negative or non-integer values throw. | +| `--max-connections ` | number | `256` | HTTP listener `server.maxConnections`; `0` / `Infinity` means unlimited. | +| `--enable-session-shell` | boolean | `false` | Enables direct `POST /session/:id/shell` execution. Requires bearer token, and every call must carry a session-bound `X-Qwen-Client-Id`. | +| `--event-ring-size ` | number | `8000` | Per-session SSE replay ring; soft cap is `1_000_000`. | +| `--compacted-replay-max-bytes ` | positive integer | `4194304` | Byte cap for the bounded in-memory replay snapshot returned by `POST /session/:id/load`; hard cap is `268435456`. | +| `--memory-budget-mb ` | integer in `[1024, 1048576]` | 50% of cgroup-constrained or host memory, capped at the flag maximum (1048576 MB) | Total memory budget for the daemon process tree, capped at resolved available memory. Observed and reported under `limits.memory` in daemon status; it does not size any child process. Boot rejects out-of-range values. | +| `--memory-pressure-mode ` | `off` \| `observe` | `observe` | Whether the daemon derives a memory-pressure level from its own RSS and V8 heap. Both modes report `runtime.memory.pressure`; only `observe` raises `daemon_memory_pressure`. Root process only; no remediation. | +| `--child-heap-mode ` | `off` \| `observe` | `observe` | Whether the daemon models a per-child heap partition of the budget. `observe` reports it and counts spawns past it; nothing is applied. `off` publishes no partition at all — `maxConcurrentChildren` and `perChildCeilingMb` are both `null`. | +| `--http-bridge` | boolean | `true` | Stage 1 bridge mode. `--no-http-bridge` still falls back to http-bridge and prints to stderr. | +| `--mcp-client-budget ` | positive integer | unset | Sets `WorkspaceMcpBudget.clientBudget` and forwards it to the ACP child through `childEnvOverrides`. | +| `--mcp-budget-mode ` | `off` / `warn` / `enforce` | `warn` when budget is set, otherwise `off` | Sets `WorkspaceMcpBudget.mode`; `enforce` requires `--mcp-client-budget`. | +| `--external-tool-guard-mode ` | `off` / `required` | `off` | Enables the managed ACP external pre-execution Guard. `required` fails startup unless its loopback provider completes the v1 handshake. | +| `--external-tool-guard-endpoint ` | loopback HTTP(S) origin | unset | Provider origin used only in `required` mode. It must be origin-only and use `127.0.0.1`, `localhost`, or `::1`; paths, credentials, redirects, and proxy routing are rejected. | +| `--external-tool-guard-timeout-ms ` | integer `100..30000` | `3000` | Per-handshake and per-prepare deadline. A timeout fails startup during the handshake or fails the invocation closed during a turn. | +| `--allow-origin ` | repeatable string | unset | Cross-origin allowlist that replaces the default CORS denial. `*` allows any origin but requires a token. | +| `--allow-private-auth-base-url` | boolean | `false` | Allows `/workspace/auth/provider` to install localhost / private-network auth provider `baseUrl`; use only in trusted local development. | +| `--prompt-deadline-ms ` | positive integer | unset | Server-side prompt wallclock limit in ms. Timeout aborts and returns an error. | +| `--writer-idle-timeout-ms ` | positive integer | unset | Per-SSE-connection idle timeout in ms. The daemon closes the SSE connection when no event is sent for this duration. | +| `--channel-idle-timeout-ms ` | non-negative integer | `0` | How long to keep the ACP child alive after the last session closes. `0` means reclaim immediately. | +| `--initialize-timeout-ms ` | positive integer | `10000` | ACP child request timeout, including the initialize handshake (ms). | +| `--session-reap-interval-ms ` | non-negative integer | `60000` | Session reaper scan interval; `0` disables it. | +| `--session-idle-timeout-ms ` | non-negative integer | `1800000` | Disconnected-session idle reaping time; `0` disables it. | +| `--rate-limit` / `--no-rate-limit` | boolean | env / off | Enables per-tier HTTP rate limiting for prompt, mutation, and read routes. | +| `--rate-limit-prompt ` | positive integer | `10` | Prompt request limit per window; requires rate limiting to be enabled. | +| `--rate-limit-mutation ` | positive integer | `30` | Mutation request limit per window; requires rate limiting to be enabled. | +| `--rate-limit-read ` | positive integer | `120` | Read request limit per window; requires rate limiting to be enabled. | +| `--rate-limit-window-ms ` | integer `>= 1000` | `60000` | Rate limit window length; requires rate limiting to be enabled. | +| no flag | - | - | `QWEN_SERVE_NO_MCP_POOL=1` fully disables the pool. | ## Environment variables diff --git a/docs/developers/daemon/19-observability.md b/docs/developers/daemon/19-observability.md index 5853214648..99a4f892b0 100644 --- a/docs/developers/daemon/19-observability.md +++ b/docs/developers/daemon/19-observability.md @@ -145,6 +145,46 @@ New OTel metric names: - `qwen-code.daemon.prompt.queue_wait`, histogram in milliseconds. - `qwen-code.daemon.pipe.message_bytes`, histogram in bytes with `direction=inbound|outbound`. +### 11. Is the daemon under memory pressure? + +```bash +curl -s 'http://127.0.0.1:4170/daemon/status' | \ + jq '.runtime.memory.pressure' +``` + +`level` is `normal` / `soft` / `hard` / `critical`, classified from `ratio` — +the worse of `rssRatio` (RSS against detected cgroup/host memory, which is what +the OOM killer watches) and `heapRatio` (V8 heap used against this process's +`heap_size_limit` — the whole heap, not only the old space that +`--max-old-space-size` names). `source` says which one produced it. Check `source` before acting: +`unknown` means the daemon could measure neither side, so `normal` there is the +absence of a reading, not evidence of health. A side is only reported when both +its numerator and its denominator were usable, so `source` is also what tells a +zero `rssBytes` / `heapUsedBytes` apart from a real one. + +**`rssRatio` is only as good as its denominator, and +`limits.memory.availableMemorySource` is what grades it.** Under a cgroup +(`constrained`) it is exactly the limit the OOM killer enforces, so the ratio +means what it says. On bare metal (`host`) it is the size of the whole machine, +while the daemon actually dies when the _machine_ runs out — which depends on +every other process on the box. A daemon holding 20% of a 64 GB host beside a +55 GB neighbour reports `level: normal, source: rss` right up until it is +killed. Under `source: 'host'`, read `rssRatio` as a **lower bound** on real +pressure. This is separate from the thresholds being uncalibrated: no threshold +choice fixes a denominator that is measuring the wrong thing. + +Two further things this does **not** cover. It is the daemon **root** process only, so +a daemon whose `qwen --acp` children are the ones growing can report `normal` +throughout — read `runtime.memory.children` beside it, which sums the live +children's own RSS (and says via `sampled` how many actually reported). +And nothing remediates: leaving `normal` raises a `daemon_memory_pressure` +warning and changes no behaviour. + +Under `--memory-pressure-mode off` every figure above is still reported and the +issue is not raised, so the top-level `status` stays whatever it would have +been. Use `off` while calibrating thresholds against a real workload, or if you +alert on `status` and do not want an uncalibrated signal moving it. + ## Flow ### Typical triage flow diff --git a/docs/developers/daemon/20-quickstart-operations.md b/docs/developers/daemon/20-quickstart-operations.md index 6f0cf433e7..2db30e65b5 100644 --- a/docs/developers/daemon/20-quickstart-operations.md +++ b/docs/developers/daemon/20-quickstart-operations.md @@ -80,7 +80,9 @@ The CLI is defined in **`packages/cli/src/commands/serve.ts`**: | `--token ` | string | env / none | Non-loopback and `--require-auth` | Bearer token; trimmed once. **It appears in `/proc//cmdline`, so prefer `QWEN_SERVER_TOKEN`**. Boot stderr also warns about this. | | `--max-sessions ` | number | `32` | - | Per-workspace active session cap. Excess spawn returns 503. `0` means unlimited. `NaN` / negative values throw. | | `--max-total-sessions ` | number | derived for multiple startup/restored workspaces | - | Daemon-wide active session cap. When omitted, a finite default is derived once from the per-workspace cap and startup/restored workspace count; dynamic registration does not recompute it. `0` means unlimited. | -| `--memory-budget-mb ` | integer in `[1024, 1048576]` | 50% of cgroup/host memory | Observation only | Total memory budget for the daemon process tree, capped at resolved available memory. Reported under `limits.memory`; does not size any child. | +| `--memory-budget-mb ` | integer in `[1024, 1048576]` | 50% of cgroup/host memory | Observation only | Total memory budget for the daemon process tree, capped at resolved available memory. Reported under `limits.memory`; modeled into a partition that nothing applies. | +| `--memory-pressure-mode ` | `off` \| `observe` | `observe` | Observation only | Reports `runtime.memory.pressure` in both modes; only `observe` raises the `daemon_memory_pressure` issue. Root process only. | +| `--child-heap-mode ` | `off` \| `observe` | `observe` | Observation only | Under `observe`, reports the modeled partition under `limits.memory.childHeap`; applies nothing and refuses nothing. Under `off`, that block's two figures are `null`. | | `--max-pending-prompts-per-session ` | number | `5` | - | Accepted but pending/running prompt cap per session. Excess prompt returns 503. `0` / `Infinity` means unlimited. Negative or non-integer values throw. | | `--workspace ` | string / repeatable | `process.cwd()` | - | Startup workspace runtime; repeat to register additional isolated runtimes. The first is primary. Each value **must be an absolute path, must exist, and must be a directory**. Boot canonicalizes every value via `canonicalizeWorkspace`. `POST /session` with a mismatched `cwd` returns `400 workspace_mismatch`. | | `--max-connections ` | number | `256` | - | Listener-level `server.maxConnections`. `0` / `Infinity` means unlimited. `NaN` / negative values fail boot to avoid fail-open behavior. | diff --git a/docs/developers/qwen-serve-protocol.md b/docs/developers/qwen-serve-protocol.md index 3ea52f19ee..f20e830538 100644 --- a/docs/developers/qwen-serve-protocol.md +++ b/docs/developers/qwen-serve-protocol.md @@ -620,7 +620,11 @@ runtime routes return `503`. `runtime.activity` reports daemon-wide prompt activity. `activePrompts` counts sessions with an in-flight prompt. `pendingPrompts` counts all accepted prompts that have not settled yet, including the running prompt and FIFO-waiting prompts. `queuedPrompts` counts FIFO-waiting prompts that have been accepted but not dispatched. `lastActivityAt` is the ISO 8601 timestamp of the last prompt start/end or session spawn; `null` when the daemon has never processed any activity since boot. `idleSinceMs` is computed from `lastActivityAt` at response generation time. -`limits.memory` is additive and reports the daemon's resolved memory figures: a required `enforced: false`, `configuredBudgetMb`, `effectiveBudgetMb` (the configured value capped at resolved cgroup/host memory), `budgetSource` (`flag` / `derived`), `availableMemoryMb`, `availableMemorySource` (`constrained` / `host`), `insufficientMemory`, and a `modeled` object holding `rootReserveMb`, `childPoolMb`, `minChildHeapMb`, `maxChildHeapMb`, and `legacyChildCeilingMb` (a conservative model of the ceiling an ACP child receives today, which can sit below the real figure). `runtime.memory` additionally reports `registeredWorkspaces` (the registration count — non-removed workspace entries, including draining, transitioning, or blocked ones; not a live-child count), `activeAcpChildren` (daemon-managed ACP children with a live, non-dying channel — includes transitioning or blocked entries, but excludes a workspace whose kill has started even if the child has not exited; not channel workers, MCP descendants, or unattached spawn reservations), `childRssCoverage` (`primary_only` today), and a `modeled` object holding `recommendedShareAtRegisteredMb` (`null` when no workspace is registered) and `recommendedShareAtActiveMb` (`null` when no child is active). Each share is capped at the legacy child ceiling, and floored at the minimum child heap only when the ceiling allows — on a small host the ceiling sits below the floor, so share × count can exceed the child pool. Read a share as advisory, not a partition of the pool. All of it is observation: no child spawn argument derives from these values, and no request is refused on their basis. On the normal `runQwenServe` path the budget is resolved before the bootstrap app is created, so `limits.memory` is already populated during the bootstrap window. It is `null` only on paths that resolve no budget (such as direct-embed bypassing `runQwenServeImpl`). The SDK type allows `null`, so correct clients cope. +`limits.memory` is additive and reports the daemon's resolved memory figures: a required `enforced: false`, a `childHeap` object (`mode`; `maxConcurrentChildren` and `perChildCeilingMb`, both `null` under `mode: 'off'`, which models nothing — and `perChildCeilingMb` additionally `null` wherever no partition can be modeled within `modeled.minChildHeapMb` — either the pool cannot cover one child at that floor, or the ceiling would land under it once capped at `modeled.legacyChildCeilingMb`, which is `floor(available / 2)` and so drops under the floor on a host below 1024 MB. It is never 0, and `maxConcurrentChildren` is `0` in those cases, since a host that models no partition is a computed answer rather than an absent model; and `refusals`, the spawns that would have exceeded the modeled limit), `configuredBudgetMb`, `effectiveBudgetMb` (the configured value capped at resolved cgroup/host memory), `budgetSource` (`flag` / `derived`), `availableMemoryMb`, `availableMemorySource` (`constrained` / `host`), `insufficientMemory`, and a `modeled` object holding `rootReserveMb`, `childPoolMb`, `minChildHeapMb`, `maxChildHeapMb`, and `legacyChildCeilingMb` (a conservative model of the ceiling an ACP child receives today, which can sit below the real figure). `runtime.memory` additionally reports `registeredWorkspaces` (the registration count — non-removed workspace entries, including draining, transitioning, or blocked ones; not a live-child count), `activeAcpChildren` (daemon-managed ACP children with a live, non-dying channel — includes transitioning or blocked entries, but excludes a workspace whose kill has started even if the child has not exited; not channel workers, MCP descendants, or unattached spawn reservations), `childRssCoverage` (`active_children` — every ACP child with a live channel, which is the set `activeAcpChildren` counts; older daemons send `primary_only`), a `children` object described below, and a `modeled` object holding `recommendedShareAtRegisteredMb` (`null` when no workspace is registered) and `recommendedShareAtActiveMb` (`null` when no child is active). Each share is capped at the legacy child ceiling, and floored at the minimum child heap only when the ceiling allows — on a small host the ceiling sits below the floor, so share × count can exceed the child pool. Read a share as advisory, not a partition of the pool. All of it is observation: no child spawn argument derives from these values, and no request is refused on their basis. `childHeap` models a fixed partition of `modeled.childPoolMb` — every child would receive the same `perChildCeilingMb`, so the modeled total stays inside the pool rather than accumulating as a per-spawn share would. Read `refusals` as admission pressure only: a count of 0 does **not** mean the partition is safe to apply, because children run on the much larger host-derived ceiling, so a workload needing more old space than `perChildCeilingMb` is healthy here and would only fail once the partition were applied. Two further reasons a nonzero count need not mean capacity pressure: the admission decision counts a terminating child until it exits, so on a daemon already at `maxConcurrentChildren` every channel replacement books a refusal during the overlap window; and on a host too small to model a partition `maxConcurrentChildren` is `0`, so `refusals` equals the total ACP spawn count, with `insufficientMemory` as the field that explains it. On the normal `runQwenServe` path the budget is resolved before the bootstrap app is created, so `limits.memory` is already populated during the bootstrap window. It is `null` only on paths that resolve no budget (such as direct-embed bypassing `runQwenServeImpl`). The SDK type allows `null`, so correct clients cope. + +`runtime.memory.children` is additive within that block and reports aggregate RSS across the children `childRssCoverage` names: `rssBytes` (their summed self-reported RSS), `sampled` (how many produced a reading), and `oldestReadingAgeMs` (the age of the oldest reading in the sum, so a caller can tell how far apart its parts were taken). The denominator for `sampled` is the sibling `activeAcpChildren`, not repeated inside the block; when `sampled` is lower, `rssBytes` is a floor rather than a total. Sampling is gated on an active SSE/WS watcher, so a status request against a daemon nobody is streaming from reports `sampled: 0` even with live children — `activeAcpChildren` beside it makes that gap visible, and `rssBytes: 0` with `sampled: 0` never means a measured zero. `oldestReadingAgeMs` is `null` when nothing was sampled and also when every contributor is a bridge predating the field, so it never means "fresh". Read the sum as an over-count and an under-count at once: summing per-process RSS double-counts pages the children share, while each child reports only its own process, so its MCP descendants and every channel worker are missing. It is not the daemon tree's memory. The field is optional in the SDK mirror because daemons reporting `primary_only` never send it. + +`runtime.memory.pressure` is additive within that block and reports the daemon root's own memory pressure: `mode` (`off` / `observe`), `level` (`normal` / `soft` / `hard` / `critical`), `source` (`rss` / `heap` / `unknown`), `ratio`, and the six raw figures the ratios come from — `rssBytes`, `rssRatio`, `availableBytes`, `heapUsedBytes`, `heapRatio`, `heapLimitBytes`. `ratio` is the larger of `rssRatio` and `heapRatio`, and `source` names which one it was; ties are reported as `rss`. `availableBytes` is `limits.memory.availableMemoryMb` in bytes — deliberately the detected cgroup/host figure rather than `effectiveBudgetMb`, because what ends the process is the real limit, not an operator's policy number. `source: "unknown"` means neither denominator was measurable and must not be read as healthy; `level` is `normal` in that case only because there is nothing to classify. The figures cover the daemon **root process only**: they are this process's own `memoryUsage()`, so children growing does not move them. `runtime.memory.children` reports those separately, and neither figure is process-tree memory. Both modes report the whole block; only `observe` additionally raises the path-free `daemon_memory_pressure` warning into the status rollup, so `off` leaves the top-level `status` unchanged. Nothing remediates in either mode. The field is optional in the SDK mirror because daemons that shipped `runtime.memory` before it exists send the block without it. `limits.maxTotalSessions` is additive. `null` means the effective daemon-wide fresh-session cap is disabled. When several startup/restored workspaces are present, `--max-total-sessions` is omitted, and `maxSessionsPerWorkspace` is finite, the daemon derives the effective total cap once as `maxSessionsPerWorkspace * startupWorkspaceCount`; later dynamic registration does not recompute it. When set, it limits fresh session creation across the daemon and reports total-limit failures with the existing `session_limit_exceeded` error shape plus `scope: "total"`. diff --git a/docs/users/qwen-serve.md b/docs/users/qwen-serve.md index 8f4b872f3f..54bd53ff23 100644 --- a/docs/users/qwen-serve.md +++ b/docs/users/qwen-serve.md @@ -396,6 +396,8 @@ Notes: | `--channel ` | — | Experimental daemon-managed channel worker. Repeat the flag to select multiple configured channels, or pass `all` to start every configured channel. `all` cannot be combined with named channels. Selected channel `cwd` values must resolve to a registered workspace; a multi-workspace daemon runs one worker per owning workspace. The worker is owned by `qwen serve`; stop the daemon to stop serve-managed channels. | | `--max-connections ` | `256` | Listener-level TCP connection cap (`server.maxConnections`). Bounds raw socket count irrespective of session count — slow / phantom SSE clients get rejected at accept time once full. Raise alongside `--max-sessions` if your deployment expects many SSE subscribers per session. | | `--memory-budget-mb ` | 50% of cgroup/host | Total memory budget in MB for the whole daemon process tree. When unset, derived as 50% of the cgroup limit or host memory; either way the effective value is capped at resolved available memory, and both the configured and effective figures are reported. Currently observation only — it does not change how any `qwen --acp` child is sized. Resolved figures appear under `limits.memory` in `GET /daemon/status`, alongside registered and live child counts and advisory per-child shares under `runtime.memory`. A host too small for the minimum reports `insufficientMemory` rather than being clamped upward; because the derived fraction is 50%, any host under ~2 GB trips this. Pass an explicit `--memory-budget-mb 1024` on such a host to override the derived figure (the flag still requires at least 1024 MB of available memory to clear the warning). Must be an integer in `[1024, 1048576]`. | +| `--memory-pressure-mode ` | `observe` | Whether the daemon turns its own memory reading into a verdict. `observe` (default) reports the pressure level under `runtime.memory.pressure` in `GET /daemon/status` and raises a `daemon_memory_pressure` issue — a `warning`, so the overall `status` leaves `ok` — whenever the level leaves `normal`. `off` still reports every figure, including the level, but raises no issue, so the overall `status` is unchanged; use it while calibrating, or if you alert on the top-level status. The level is the worse of two ratios: RSS against available memory (what the cgroup OOM killer watches) and V8 heap used against this process's heap ceiling. It covers the daemon root process only; compare it against `runtime.memory.children.rssBytes` for the children. Nothing remediates in either mode. One of `off`, `observe`. | +| `--child-heap-mode ` | `observe` | Whether the daemon models a per-child heap partition of `--memory-budget-mb`. `observe` (default) reports what it would apply — `limits.memory.childHeap.perChildCeilingMb` and `maxConcurrentChildren` — and counts spawns that would have exceeded the limit. **Nothing is applied**: no child is sized from the budget and no spawn is refused. `off` models nothing, and says so on the wire: `maxConcurrentChildren` and `perChildCeilingMb` are both `null` rather than carrying a partition you switched off. A refusal count of 0 does **not** mean the partition would be safe to apply: children still run on the much larger host-derived ceiling, so a workload needing more old space than the modeled ceiling looks perfectly healthy here. Applying the partition ships with the measurement that can answer that. | | `--event-ring-size ` | `8000` | Per-session SSE replay ring depth (#3803 §02 target). Sets the backlog available to `GET /session/:id/events` with `Last-Event-ID: N`. Larger = more reconnect headroom at the cost of a few hundred KB extra RAM per session. SDK clients can additionally request a larger per-subscriber backlog cap on a specific subscription via `?maxQueued=N` (range `[16, 2048]`, default 256). Daemons also emit a non-terminal `slow_client_warning` SSE frame at 75% queue fill so clients can drain / reconnect before getting evicted. Pre-flight `caps.features.slow_client_warning`. | | `--compacted-replay-max-bytes ` | `4194304` | Per-live-session byte cap for the retained replay events in the bounded snapshot returned by `POST /session/:id/load`. The cap applies to `compactedReplay`; the current in-flight `liveJournal` is separately capped by `--max-journal-events` and `--max-journal-bytes`. Values must be positive safe integers; invalid values fail at boot, and the hard ceiling is 256 MiB. When older retained replay is dropped, the snapshot begins with `history_truncated`. This does not limit the on-disk transcript. | | `--max-journal-events ` | `10000` | Per-session cap on the number of raw events retained in the in-flight live journal (the current unfinished turn). When exceeded, the oldest journal entries are dropped and a `history_truncated` marker is prepended. Must be a positive safe integer. | diff --git a/packages/acp-bridge/package.json b/packages/acp-bridge/package.json index 9d332c330f..1af01ce141 100644 --- a/packages/acp-bridge/package.json +++ b/packages/acp-bridge/package.json @@ -103,6 +103,10 @@ "types": "./dist/daemon-memory-budget.d.ts", "import": "./dist/daemon-memory-budget.js" }, + "./childHeapPolicy": { + "types": "./dist/child-heap-policy.d.ts", + "import": "./dist/child-heap-policy.js" + }, "./channelControlTimeouts": { "types": "./dist/channel-control-timeouts.d.ts", "import": "./dist/channel-control-timeouts.js" diff --git a/packages/acp-bridge/src/bridge.test.ts b/packages/acp-bridge/src/bridge.test.ts index 59b358d624..db841e5721 100644 --- a/packages/acp-bridge/src/bridge.test.ts +++ b/packages/acp-bridge/src/bridge.test.ts @@ -19924,4 +19924,42 @@ describe('createAcpSessionBridge — child-resource refresh', () => { await bridge.shutdown(); } }); + + it('ages the snapshot, and drops it entirely once past the staleness window', async () => { + const handle = makeChannel({ + extMethodImpl: async (method) => + method === SERVE_STATUS_EXT_METHODS.workspaceResource + ? { rssBytes: 4096, cpuPercent: 7 } + : {}, + }); + const bridge = makeBridge({ channelFactory: async () => handle.channel }); + try { + await bridge.spawnOrAttach({ workspaceCwd: WS_A }); + await bridge.refreshChildResource!(); + + const fresh = bridge.getChildResourceSnapshot!(); + expect(fresh).toMatchObject({ rssBytes: 4096, cpuPercent: 7 }); + // A reading just taken is not in the future and not already expired. + expect(fresh!.ageMs).toBeGreaterThanOrEqual(0); + expect(fresh!.ageMs).toBeLessThan(30_000); + + // Walk the clock past the cliff. `STALE_CHILD_RESOURCE_MS` is a const + // inside the bridge factory closure, not an export, so drive the + // boundary with the clock rather than exporting it for a test. + // `vi.spyOn` rather than assigning `Date.now` directly: the restore is + // registered with the test runner, so it survives an assertion throwing + // between here and a `finally`, and it is what the rest of the repo uses. + const staleAt = Date.now() + 30_001; + const nowSpy = vi.spyOn(Date, 'now').mockReturnValue(staleAt); + try { + // Dropped, not returned-and-stale: a zombie child must not read as + // healthy just because its last good value is still in the cache. + expect(bridge.getChildResourceSnapshot!()).toBeUndefined(); + } finally { + nowSpy.mockRestore(); + } + } finally { + await bridge.shutdown(); + } + }); }); diff --git a/packages/acp-bridge/src/bridge.ts b/packages/acp-bridge/src/bridge.ts index b7b8b4dbdf..63bf942936 100644 --- a/packages/acp-bridge/src/bridge.ts +++ b/packages/acp-bridge/src/bridge.ts @@ -3577,19 +3577,24 @@ export function createAcpSessionBridge(opts: BridgeOptions): AcpSessionBridge { } }; const getChildResourceSnapshot = (): - | { rssBytes: number; cpuPercent: number } + | { rssBytes: number; cpuPercent: number; ageMs: number } | undefined => { const info = liveChannelInfo(); if (!info || info.childResourceAt === undefined) return undefined; // Staleness: a child that goes unresponsive without a channel swap would // otherwise show its last-good rss/cpu forever (a zombie looking healthy). // Drop the reading once it ages past the window so the chart reads 0. - if (Date.now() - info.childResourceAt > STALE_CHILD_RESOURCE_MS) { + const ageMs = Date.now() - info.childResourceAt; + if (ageMs > STALE_CHILD_RESOURCE_MS) { return undefined; } return { rssBytes: info.childRssBytes ?? 0, cpuPercent: info.childCpuPercent ?? 0, + // Bounded by the guard above, so a caller summing several children's + // readings can say how far apart they were taken. Without it a sum of + // readings up to `STALE_CHILD_RESOURCE_MS` apart looks instantaneous. + ageMs, }; }; diff --git a/packages/acp-bridge/src/bridgeTypes.ts b/packages/acp-bridge/src/bridgeTypes.ts index c5405e2010..7893eb8ae0 100644 --- a/packages/acp-bridge/src/bridgeTypes.ts +++ b/packages/acp-bridge/src/bridgeTypes.ts @@ -1707,7 +1707,14 @@ export interface AcpSessionBridge { * live. Synchronous cache read for the metrics sampler. Optional — see * {@link pendingPromptTotal}. */ getChildResourceSnapshot?(): - | { rssBytes: number; cpuPercent: number } + | { + rssBytes: number; + cpuPercent: number; + /** How old this reading is, in ms. Absent on bridges predating the + * field — see {@link pendingPromptTotal} — so a caller aggregating + * several children must treat it as unknown rather than as fresh. */ + ageMs?: number; + } | undefined; /** Poll the live child's resource extMethod and refresh the cache that * {@link getChildResourceSnapshot} reads. Fired fire-and-forget by the diff --git a/packages/acp-bridge/src/child-heap-policy.test.ts b/packages/acp-bridge/src/child-heap-policy.test.ts new file mode 100644 index 0000000000..372e72f03d --- /dev/null +++ b/packages/acp-bridge/src/child-heap-policy.test.ts @@ -0,0 +1,175 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from 'vitest'; +import { createChildHeapPolicy } from './child-heap-policy.js'; +import { + MIN_CHILD_HEAP_MB, + resolveDaemonMemoryBudget, +} from './daemon-memory-budget.js'; + +describe('createChildHeapPolicy', () => { + it.each([2_048, 8_192, 32_768, 262_144])( + 'models a partition whose total fits the pool (%i MB host)', + (availableMemoryMb) => { + // The invariant a per-spawn share could not hold: sizing each child by + // the count live at *its* spawn accumulates grants as P x H(n), because + // V8 cannot lower a running child's ceiling. A constant ceiling makes + // the total n x ceiling, which admission keeps inside the pool. + const b = resolveDaemonMemoryBudget({ availableMemoryMb }); + const { maxConcurrentChildren, perChildCeilingMb } = + createChildHeapPolicy({ budget: b, mode: 'observe' }).snapshot(); + + // `observe` always models, so both are numbers here — `off` has its own + // test. Assert that before the `!`s, so a regression that nulled them + // fails on a named assertion instead of on NaN arithmetic below. + expect(maxConcurrentChildren).not.toBeNull(); + expect(perChildCeilingMb).not.toBeNull(); + expect(maxConcurrentChildren!).toBeGreaterThan(0); + expect(perChildCeilingMb!).toBeGreaterThanOrEqual(MIN_CHILD_HEAP_MB); + expect(maxConcurrentChildren! * perChildCeilingMb!).toBeLessThanOrEqual( + b.childPoolMb, + ); + }, + ); + + it('admits no child when the pool cannot cover one, and offers no ceiling', () => { + // A 512 MB host derives a 256 MB budget whose root reserve consumes all of + // it, leaving a pool of 0. Clamping the count up to 1 here produced a + // ceiling of 0 — and `--max-old-space-size=0` is not a zero ceiling, it is + // V8's *default* heap, so that would have modelled gigabytes against an + // empty pool. + const empty = resolveDaemonMemoryBudget({ availableMemoryMb: 512 }); + expect(empty.childPoolMb).toBe(0); + expect( + createChildHeapPolicy({ budget: empty, mode: 'observe' }).snapshot(), + ).toMatchObject({ maxConcurrentChildren: 0, perChildCeilingMb: null }); + }); + + it('never models a ceiling below the documented minimum', () => { + // A 1024 MB host leaves a 256 MB pool — under the 512 MB floor, so still + // no admissible child rather than one child at half the minimum. + const small = resolveDaemonMemoryBudget({ availableMemoryMb: 1_024 }); + expect(small.childPoolMb).toBeLessThan(MIN_CHILD_HEAP_MB); + const snap = createChildHeapPolicy({ + budget: small, + mode: 'observe', + }).snapshot(); + expect(snap.maxConcurrentChildren).toBe(0); + expect(snap.perChildCeilingMb).toBeNull(); + }); + + // Every case above resolves a *derived* budget, where the pool reaches 0 + // before the legacy ceiling can drop under the floor. An explicit budget + // separates the two: `--memory-budget-mb` has a floor of 1024 while the + // legacy ceiling is `floor(available / 2)`, so on a host under 1024 MB the + // pool clears 512 and the cap does not. `docs/users/qwen-serve.md` tells + // operators on exactly these hosts to pass that flag, so this band is the + // documented remedy rather than a contrived input. + it.each([768, 900, 1_000, 1_023])( + 'models no partition when the capped ceiling would fall under the floor (%i MB host, explicit budget)', + (availableMemoryMb) => { + const budget = resolveDaemonMemoryBudget({ + availableMemoryMb, + budgetMb: 1_024, + }); + // The shape that makes this reachable: pool clears the floor, cap does + // not. Asserted so a change to either derivation retires this test + // loudly instead of leaving it passing vacuously. + expect(budget.childPoolMb).toBeGreaterThanOrEqual(MIN_CHILD_HEAP_MB); + expect(budget.legacyChildCeilingMb).toBeLessThan(MIN_CHILD_HEAP_MB); + + const snap = createChildHeapPolicy({ + budget, + mode: 'observe', + }).snapshot(); + expect(snap.perChildCeilingMb).toBeNull(); + expect(snap.maxConcurrentChildren).toBe(0); + // The contradiction this prevents: a ceiling published next to a + // `minChildHeapMb` it sits below, in the same snapshot. + expect(snap.minChildHeapMb).toBe(MIN_CHILD_HEAP_MB); + }, + ); + + it('still models the partition at the first budget the floor allows', () => { + // 1024 MB available is where the legacy ceiling reaches exactly 512, so + // the boundary is inclusive. Without this, refusing unconditionally would + // satisfy the band test above and lose the feature on small hosts. + const budget = resolveDaemonMemoryBudget({ + availableMemoryMb: 1_024, + budgetMb: 1_024, + }); + expect( + createChildHeapPolicy({ budget, mode: 'observe' }).snapshot(), + ).toMatchObject({ maxConcurrentChildren: 1, perChildCeilingMb: 512 }); + }); + + it('sizes an 8 GB host for seven children, and a large host by the workspace cap', () => { + // Pinned: these are the numbers an operator plans against. The large host + // divides by MAX_DAEMON_WORKSPACES rather than pool/512, so the ceiling is + // 614 MB and not the floor. + expect( + createChildHeapPolicy({ + budget: resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }), + mode: 'observe', + }).snapshot(), + ).toMatchObject({ maxConcurrentChildren: 7, perChildCeilingMb: 526 }); + + expect( + createChildHeapPolicy({ + budget: resolveDaemonMemoryBudget({ availableMemoryMb: 32_768 }), + mode: 'observe', + }).snapshot(), + ).toMatchObject({ maxConcurrentChildren: 25, perChildCeilingMb: 614 }); + }); + + it('counts spawns past the modeled limit', () => { + const b = resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }); + const policy = createChildHeapPolicy({ budget: b, mode: 'observe' }); + // Non-null because the mode is `observe`; `off` publishes no limit. + const limit = policy.snapshot().maxConcurrentChildren!; + + expect(policy.decide(1).refuse).toBe(false); + expect(policy.decide(limit).refuse).toBe(false); + expect(policy.snapshot().refusals).toBe(0); + + expect(policy.decide(limit + 1).refuse).toBe(true); + policy.decide(limit + 9); + expect(policy.snapshot().refusals).toBe(2); + }); + + it('models nothing at all when off', () => { + const off = createChildHeapPolicy({ + budget: resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }), + mode: 'off', + }); + // An off policy must never accrue refusals, or the counter would report on + // a daemon that modelled nothing. + expect(off.decide(9_999).refuse).toBe(false); + expect(off.snapshot().refusals).toBe(0); + + // And it must publish no partition. `null`, not `0` — this same 8 GB + // budget models 7 children at 526 MB under `observe`, so reporting those + // figures here would hand an operator a partition they switched off with + // nothing on the wire marking it inert. Zero is a different claim: it is + // the computed answer for a pool too small to host one child. + expect(off.snapshot().maxConcurrentChildren).toBeNull(); + expect(off.snapshot().perChildCeilingMb).toBeNull(); + }); + + it('models the partition under observe on the same budget', () => { + // The other half of the assertion above: without this, nulling the + // figures unconditionally would satisfy the `off` test and lose the + // feature. + const observe = createChildHeapPolicy({ + budget: resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }), + mode: 'observe', + }).snapshot(); + + expect(observe.maxConcurrentChildren).toBe(7); + expect(observe.perChildCeilingMb).toBe(526); + }); +}); diff --git a/packages/acp-bridge/src/child-heap-policy.ts b/packages/acp-bridge/src/child-heap-policy.ts new file mode 100644 index 0000000000..f90a0731a4 --- /dev/null +++ b/packages/acp-bridge/src/child-heap-policy.ts @@ -0,0 +1,174 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + MIN_CHILD_HEAP_MB, + type DaemonMemoryBudget, +} from './daemon-memory-budget.js'; +import { MAX_DAEMON_WORKSPACES } from './channel-control-timeouts.js'; + +/** + * Whether the daemon models a per-child heap partition. + * + * `off` — do not model it. + * + * `observe` — compute the partition and count the spawns it would have + * refused. Nothing is applied: no child receives a derived + * `--max-old-space-size`, and no spawn is refused. + * + * There is deliberately no `enforce` yet. Applying the partition needs a way + * to tell an operator beforehand whether their workload fits it, and that + * observation does not exist: `refusals` below counts admission pressure, not + * whether a child would have survived the ceiling. Enforcing on a signal that + * cannot answer the question it is being read for is how a healthy daemon gets + * switched into an OOM loop. The enforcing mode ships with the measurement + * that justifies it — peak old-space per child, compared against + * `perChildCeilingMb`. + */ +export type ChildHeapMode = 'off' | 'observe'; + +export interface ChildHeapPolicySnapshot { + mode: ChildHeapMode; + childPoolMb: number; + minChildHeapMb: number; + /** + * Children the pool could host concurrently under the modeled partition. + * **0** when no partition can be modeled at all — either the pool cannot + * cover one child at `minChildHeapMb`, or the ceiling would land under that + * floor once capped at today's host-derived one. Both are real states on a + * small host, and neither is the same as 1. + * + * `null` under `off`, which models nothing. That is a different statement + * from `0`: zero is a computed answer meaning "this pool hosts no child", + * while null means no partition was computed at all. Collapsing them would + * make an operator who disabled the model read it as a host too small to + * run anything. + */ + maxConcurrentChildren: number | null; + /** + * The ceiling every child would receive. Never 0 and never below + * `minChildHeapMb` — `null` instead, under `off` and on any host where the + * partition cannot be modeled within that floor. A zero would be worse than + * useless: `--max-old-space-size=0` means *V8's default heap*, so emitting + * one would authorise gigabytes against an empty pool. + */ + perChildCeilingMb: number | null; + /** + * Spawns that would have been refused for exceeding + * `maxConcurrentChildren`. + * + * Read it as admission pressure and nothing more. In particular a count of + * 0 does **not** mean the partition is safe to apply: children currently + * run on the far larger host-derived ceiling, so a workload needing more + * old space than `perChildCeilingMb` is perfectly healthy here and would + * only fail once the partition were applied. + * + * Two ways it counts something other than capacity pressure, both by + * construction: + * + * - **Channel swaps at full occupancy.** The decision reads + * `committedProcessCount`, which counts a terminating child until it + * actually exits — deliberately, since its memory is still resident. So a + * replacement spawned before the old child exits transiently makes the + * count one higher than steady state. Where `MAX_DAEMON_WORKSPACES` is the + * binding term (`childPoolMb >= 12800`, i.e. a ~32 GB host and up), a + * daemon at 25 live children books a refusal on every channel replacement, + * with no memory pressure involved. Do not net this out by giving the + * comparison swap headroom: that would admit a 26th ceiling against a + * 25-child pool, trading a metric artifact for real overcommit. + * - **Hosts too small to model a partition.** `maxConcurrentChildren` is 0 + * there, so this equals the total ACP spawn count. Correct by the + * definition and alarming to read; `insufficientMemory` on the budget is + * the field that says why. + */ + refusals: number; +} + +export interface ChildHeapPolicy { + /** + * @param concurrentChildren Children already committed *including this one* + * — `ProcessRegistry.committedProcessCount` taken after `reserve()`. + */ + decide(concurrentChildren: number): { refuse: boolean }; + snapshot(): ChildHeapPolicySnapshot; +} + +export function createChildHeapPolicy(options: { + budget: DaemonMemoryBudget; + mode: ChildHeapMode; +}): ChildHeapPolicy { + const { budget, mode } = options; + let refusals = 0; + + // A FIXED partition, not a share of the pool divided by the children live + // at this instant. A per-spawn share bounds the child *count* but not the + // memory: V8 cannot lower a running child's ceiling, so grants accumulate + // as P + P/2 + P/3 + ... = P x H(n) — 2.6x the pool at seven children on an + // 8 GB host. Holding the ceiling constant makes the total n * ceiling, and + // admitting at most `maxConcurrentChildren` keeps that inside the pool by + // construction, with no ledger and no dependence on arrival order. + // + // Not clamped to a minimum of one. A pool below `MIN_CHILD_HEAP_MB` hosts + // no child at all, and saying "1" there produced a ceiling of 0 — which V8 + // reads as its *default* heap, roughly 4 GB, against a pool of nothing. + const admissible = Math.min( + Math.floor(budget.childPoolMb / MIN_CHILD_HEAP_MB), + MAX_DAEMON_WORKSPACES, + ); + // `floor(pool / admissible) >= MIN_CHILD_HEAP_MB` by construction, but the + // legacy cap is `floor(available / 2)` and is under the floor whenever + // available memory is below 1024 MB. The `Math.min` lets it win, so the + // partition could publish a ceiling *below* the `minChildHeapMb` sitting + // next to it in the same snapshot — a host with 768 MB available and an + // explicit `--memory-budget-mb 1024` modeled one child at 384 MB. Not + // reachable from a derived budget (the pool hits 0 first), but the docs tell + // operators on exactly those hosts to pass that flag, so the documented + // remedy is what reaches the band. + // + // Refuse the model rather than shrink under the floor: a ceiling the module + // says no child may run at is not a partition, and this is the figure a + // future `enforce` would hand to `--max-old-space-size`. Such a host already + // reports `insufficientMemory`, which is where an operator should be reading + // it from. + const rawCeilingMb = + admissible > 0 + ? Math.min( + Math.floor(budget.childPoolMb / admissible), + budget.legacyChildCeilingMb, + ) + : null; + const modelable = rawCeilingMb !== null && rawCeilingMb >= MIN_CHILD_HEAP_MB; + // Kept in lockstep: publishing "one child fits" beside a null ceiling would + // be the same contradiction from the other side. + const maxConcurrentChildren = modelable ? admissible : 0; + const perChildCeilingMb = modelable ? rawCeilingMb : null; + + return { + decide(concurrentChildren) { + if (mode === 'off') return { refuse: false }; + const refuse = concurrentChildren > maxConcurrentChildren; + if (refuse) refusals += 1; + return { refuse }; + }, + + snapshot() { + // `off` publishes no partition. The figures are computed above either + // way — the arithmetic is free and the code stays branchless — but + // reporting them under a mode documented as "do not model it" would + // hand an operator a 7-child / 526 MB partition they switched off, with + // nothing on the wire saying it is inert. + const modeled = mode !== 'off'; + return { + mode, + childPoolMb: budget.childPoolMb, + minChildHeapMb: MIN_CHILD_HEAP_MB, + maxConcurrentChildren: modeled ? maxConcurrentChildren : null, + perChildCeilingMb: modeled ? perChildCeilingMb : null, + refusals, + }; + }, + }; +} diff --git a/packages/acp-bridge/src/process-registry.test.ts b/packages/acp-bridge/src/process-registry.test.ts index d9f80a668c..cc8ed4da42 100644 --- a/packages/acp-bridge/src/process-registry.test.ts +++ b/packages/acp-bridge/src/process-registry.test.ts @@ -23,6 +23,46 @@ afterEach(() => { }); describe('ProcessRegistry', () => { + it('counts unattached reservations, which is what admission must key on', () => { + const registry = new ProcessRegistry(); + expect(registry.committedProcessCount).toBe(0); + + // Two spawns racing: both reserve before either attaches. This is the + // invariant an admission check rests on — `activeProcessCount` shows + // neither of them yet, so keying off it would let both through. + const first = registry.reserve(); + const second = registry.reserve(); + expect(registry.activeProcessCount).toBe(0); + expect(registry.committedProcessCount).toBe(2); + + first.attach(fakeChild(1)); + expect(registry.committedProcessCount).toBe(2); + + // A cancelled reservation releases its slot; leaking it would inflate the + // count for every later spawn. + second.cancel(); + expect(registry.committedProcessCount).toBe(1); + second.cancel(); + expect(registry.committedProcessCount).toBe(1); + }); + + it('releases a committed slot on exit, not when terminate starts', async () => { + const registry = new ProcessRegistry(); + const child = fakeChild(4321); + const tracked = registry.reserve().attach(child); + expect(registry.committedProcessCount).toBe(1); + + // Winding down still occupies the pool: the process is alive and its + // memory is still resident, so a swap legitimately counts twice. + const terminating = tracked.terminate(); + await Promise.resolve(); + expect(registry.committedProcessCount).toBe(1); + + child.emit('exit', 0, null); + await terminating; + expect(registry.committedProcessCount).toBe(0); + }); + it('classifies an error without a pid as no process', async () => { const registry = new ProcessRegistry(); const child = fakeChild(undefined); diff --git a/packages/acp-bridge/src/process-registry.ts b/packages/acp-bridge/src/process-registry.ts index df6d0c71a8..4191a3f1ac 100644 --- a/packages/acp-bridge/src/process-registry.ts +++ b/packages/acp-bridge/src/process-registry.ts @@ -79,6 +79,22 @@ export class ProcessRegistry { get activeProcessCount(): number { return this.children.size; } + + /** + * Children this registry has committed to: attached ones plus reservations + * that have not attached yet. Larger than {@link activeProcessCount}, and + * the right figure for admission — `reserve()` inserts its token + * synchronously before `spawn()`, so two racing spawns each see the other + * here, while neither is visible in `activeProcessCount` until its child is + * attached. + * + * A child leaves this count when it *exits*, not when `terminate()` starts, + * so a channel swap is counted twice while the old process is still winding + * down. That is deliberate: its memory is still resident. + */ + get committedProcessCount(): number { + return this.children.size + this.reservations.size; + } } class TrackedChild implements TrackedChildProcess { diff --git a/packages/acp-bridge/src/spawnChannel.test.ts b/packages/acp-bridge/src/spawnChannel.test.ts index 06d982e5c6..59156a68d6 100644 --- a/packages/acp-bridge/src/spawnChannel.test.ts +++ b/packages/acp-bridge/src/spawnChannel.test.ts @@ -35,6 +35,9 @@ import { EventEmitter } from 'node:events'; import type { ChildProcess } from 'node:child_process'; import { PassThrough } from 'node:stream'; +import { ProcessRegistry } from './process-registry.js'; +import { createChildHeapPolicy } from './child-heap-policy.js'; +import { resolveDaemonMemoryBudget } from './daemon-memory-budget.js'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; const mockSpawn = vi.hoisted(() => vi.fn()); @@ -245,6 +248,76 @@ describe('createSpawnChannelFactory env policy', () => { }); }); +describe('createSpawnChannelFactory child-heap observation', () => { + const originalArgv1 = process.argv[1]; + const budget = resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }); + + beforeEach(() => { + mockSpawn.mockReset(); + mockSpawn.mockReturnValue(createFakeChildProcess()); + process.argv[1] = '/tmp/qwen.js'; + process.env['QWEN_CLI_ENTRY'] = '/tmp/qwen.js'; + }); + afterEach(() => { + process.argv[1] = originalArgv1; + delete process.env['QWEN_CLI_ENTRY']; + }); + + it('leaves argv byte-identical while counting what it would have refused', async () => { + const policy = createChildHeapPolicy({ budget, mode: 'observe' }); + const registry = new ProcessRegistry(); + const factory = createSpawnChannelFactory({ + processRegistry: registry, + childHeapPolicy: policy, + }); + // Non-null because the mode is `observe`; `off` publishes no limit. + const limit = policy.snapshot().maxConcurrentChildren!; + + for (let i = 0; i < limit + 2; i++) await factory(`/tmp/w${i}`); + const observed = mockSpawn.mock.calls.at(-1)?.[1] as string[]; + + mockSpawn.mockClear(); + await createSpawnChannelFactory({ processRegistry: new ProcessRegistry() })( + '/tmp/w0', + ); + const bare = mockSpawn.mock.calls[0]?.[1] as string[]; + + // Nothing applied: passing a derived --max-old-space-size would change the + // child's GC and OOM behaviour, which an observing mode may not do. + expect(observed).toEqual(bare); + // Every spawn still went through — and the two past the modeled limit are + // counted, which is the whole product of this mode. + expect(registry.committedProcessCount).toBe(limit + 2); + expect(policy.snapshot().refusals).toBe(2); + }); + + it('releases the reservation when a supplied policy throws', async () => { + // `childHeapPolicy` is a public factory option, so `decide()` is caller + // code and may throw. The reservation is taken before it runs; if the + // throw escapes without cancelling, the token is held for the process + // lifetime and every later spawn sees an inflated committed count. + const registry = new ProcessRegistry(); + const factory = createSpawnChannelFactory({ + processRegistry: registry, + childHeapPolicy: { + decide: () => { + throw new Error('policy exploded'); + }, + snapshot: () => { + throw new Error('unused'); + }, + }, + }); + + await expect(factory('/tmp/w0')).rejects.toThrow('policy exploded'); + + // Nothing was spawned, so nothing may remain committed. A leak shows up + // here as 1 — the reservation that outlived its own spawn. + expect(registry.committedProcessCount).toBe(0); + expect(mockSpawn).not.toHaveBeenCalled(); + }); +}); + describe('createStderrForwarder', () => { it('calls onDiagnosticLine for each complete line', () => { const captured: Array<{ line: string; level?: string }> = []; diff --git a/packages/acp-bridge/src/spawnChannel.ts b/packages/acp-bridge/src/spawnChannel.ts index 2d9eaadbbf..6ad769052c 100644 --- a/packages/acp-bridge/src/spawnChannel.ts +++ b/packages/acp-bridge/src/spawnChannel.ts @@ -14,6 +14,7 @@ import { ndJsonStream, type NdJsonStreamHooks } from './ndJsonStream.js'; import { MissingCliEntryError } from './status.js'; import { EXTERNAL_TOOL_GUARD_TOKEN_ENV } from './externalToolGuard.js'; import { ProcessRegistry } from './process-registry.js'; +import type { ChildHeapPolicy } from './child-heap-policy.js'; let cachedMemoryArgs: string[] | undefined; export function getAcpMemoryArgs(): string[] { @@ -108,6 +109,17 @@ export interface SpawnChannelFactoryOptions { pipeHooks?: NdJsonStreamHooks; sourceEnv?: Readonly; processRegistry?: ProcessRegistry; + /** + * Daemon child-heap policy. Only meaningful together with a **shared** + * `processRegistry`: the factory otherwise builds its own, every spawn sees + * a concurrent count of 1, and each child is handed the whole pool — the + * current overcommit, now with a policy object attesting to it. All three + * daemon factories pass the same registry. + * + * Omitted by every single-child caller (interactive CLI, IDE companion, + * direct-embed), which keeps the host-derived ceiling. + */ + childHeapPolicy?: ChildHeapPolicy; } /** @@ -136,13 +148,25 @@ export function createSpawnChannelFactory( ); childEnv['QWEN_CODE_NO_RELAUNCH'] = 'true'; - const memoryArgs = getAcpMemoryArgs(); const execArgs = process.execArgv.filter( (a) => !/^--inspect(-brk)?($|=)/.test(a), ); + // Reserve BEFORE deciding: the reservation is what makes this spawn + // visible to any other spawn racing it, so the count below includes this + // child and two concurrent spawns cannot both be told they are alone. const reservation = processRegistry.reserve(); let child; + // Everything between `reserve()` and `attach()` belongs inside this try. + // `childHeapPolicy` is a public `createSpawnChannelFactory` option, so an + // externally supplied `decide()` can throw; outside the try that would + // reject the spawn while leaving the reservation held forever, inflating + // `committedProcessCount` for every later spawn. try { + // Observation only: the policy is asked what it *would* decide so the + // refusal count is real, but nothing here acts on the answer — no + // derived ceiling reaches the child and no spawn is refused. + options.childHeapPolicy?.decide(processRegistry.committedProcessCount); + const memoryArgs = getAcpMemoryArgs(); child = spawn( process.execPath, [ diff --git a/packages/cli/src/commands/serve.test.ts b/packages/cli/src/commands/serve.test.ts index c695cd6461..8101ef4736 100644 --- a/packages/cli/src/commands/serve.test.ts +++ b/packages/cli/src/commands/serve.test.ts @@ -229,6 +229,9 @@ describe('serve rate limit env parsing', () => { }); } + // Call this at most once per test: it waits on `toHaveBeenCalled()`, which a + // previous call in the same test already satisfies, so a second invocation + // returns before its own args land and assertions read the first call's. async function startServeHandlerWithArgs(args: string) { const handler = serveCommand.handler; if (!handler) throw new Error('serve handler missing'); @@ -384,6 +387,72 @@ describe('serve rate limit env parsing', () => { expect(process.env['QWEN_CODE_EXTERNAL_TOOL_GUARD_TOKEN']).toBeUndefined(); }); + it('passes --memory-pressure-mode to runQwenServe', async () => { + // Without this, deleting the `memoryPressureMode` line in the handler + // leaves every other suite green: the fast path parses the flag in its + // own module, and the status builder supplies the same default itself. + mockRunQwenServe.mockResolvedValueOnce({ + url: 'http://127.0.0.1:4170/', + webShellMounted: false, + }); + + await startServeHandlerWithArgs('--no-web --memory-pressure-mode off'); + + expect(mockRunQwenServe).toHaveBeenCalledWith( + expect.objectContaining({ memoryPressureMode: 'off' }), + ); + }); + + it('passes --child-heap-mode to runQwenServe', async () => { + mockRunQwenServe.mockResolvedValueOnce({ + url: 'http://127.0.0.1:4170/', + webShellMounted: false, + }); + + await startServeHandlerWithArgs('--no-web --child-heap-mode off'); + + expect(mockRunQwenServe).toHaveBeenCalledWith( + expect.objectContaining({ childHeapMode: 'off' }), + ); + }); + + it('defaults the child heap mode to observe, and rejects enforce outright', async () => { + mockRunQwenServe.mockResolvedValueOnce({ + url: 'http://127.0.0.1:4170/', + webShellMounted: false, + }); + + await startServeHandlerWithArgs('--no-web'); + + expect(mockRunQwenServe).toHaveBeenCalledWith( + expect.objectContaining({ childHeapMode: 'observe' }), + ); + // `enforce` is not a value yet, and boot must say so rather than accept + // it: applying the partition needs an observation this daemon cannot make. + expect(() => buildParser().parseSync('--child-heap-mode enforce')).toThrow( + /Invalid values/, + ); + }); + + it('defaults the memory pressure mode to observe', async () => { + mockRunQwenServe.mockResolvedValueOnce({ + url: 'http://127.0.0.1:4170/', + webShellMounted: false, + }); + + await startServeHandlerWithArgs('--no-web'); + + expect(mockRunQwenServe).toHaveBeenCalledWith( + expect.objectContaining({ memoryPressureMode: 'observe' }), + ); + }); + + it('rejects a memory pressure mode outside the choices', () => { + expect(() => + buildParser().parseSync('--memory-pressure-mode enforce'), + ).toThrow(/Invalid values/); + }); + it('passes --channel all as an all-channel selection', async () => { mockRunQwenServe.mockResolvedValueOnce({ url: 'http://127.0.0.1:4170/', diff --git a/packages/cli/src/commands/serve.ts b/packages/cli/src/commands/serve.ts index 7b2bf609cd..65baf959a7 100644 --- a/packages/cli/src/commands/serve.ts +++ b/packages/cli/src/commands/serve.ts @@ -20,6 +20,7 @@ import { DEFAULT_MAX_JOURNAL_EVENTS, } from '@qwen-code/acp-bridge/replayWindowLimits'; import { EXTERNAL_TOOL_GUARD_TOKEN_ENV } from '@qwen-code/acp-bridge/externalToolGuard'; +import type { ChildHeapMode } from '@qwen-code/acp-bridge/childHeapPolicy'; import { isValidMemoryBudgetMb, memoryBudgetRangeError, @@ -129,6 +130,8 @@ interface ServeArgs { 'http-bridge': boolean; 'mcp-client-budget'?: number; 'memory-budget-mb'?: number; + 'memory-pressure-mode'?: 'off' | 'observe'; + 'child-heap-mode'?: ChildHeapMode; 'mcp-budget-mode'?: 'enforce' | 'warn' | 'off'; 'allow-origin'?: string[]; 'allow-private-auth-base-url': boolean; @@ -334,9 +337,38 @@ export const serveCommand: CommandModule = { 'derived as 50% of cgroup-constrained ' + 'or host memory, and capped at the resolved available memory either ' + 'way. Currently observed and reported under `limits.memory` in daemon ' + - 'status; it does not yet size any child process. Must be an integer ' + + 'status, and modeled into a per-child partition reported under ' + + '`limits.memory.childHeap`. Nothing applies it: no child is sized ' + + 'from this budget. Must be an integer ' + 'in [1024, 1048576].', }) + .option('memory-pressure-mode', { + choices: ['off', 'observe'] as const, + default: 'observe' as const, + description: + 'Whether the daemon derives a memory-pressure level from its own ' + + 'RSS and V8 heap. `observe` (default) reports the level in daemon ' + + 'status and raises a status issue when it leaves normal. `off` ' + + 'still reports the underlying figures but raises no issue, so the ' + + 'overall status rollup is unchanged — use it while calibrating, or ' + + 'if you alert on the top-level status. Nothing remediates in ' + + 'either mode.', + }) + .option('child-heap-mode', { + choices: ['off', 'observe'] as const, + default: 'observe' as const, + description: + 'Whether the daemon models a per-child heap partition of the ' + + 'memory budget. `observe` (default) reports the partition it would ' + + 'apply — `limits.memory.childHeap.perChildCeilingMb` and ' + + '`maxConcurrentChildren` — and counts spawns that would have ' + + 'exceeded it. Nothing is applied: no child is sized from the ' + + 'budget and no spawn is refused. `off` models nothing. Note a ' + + 'refusal count of 0 does NOT mean the partition would be safe to ' + + 'apply; children still run on the much larger host-derived ' + + 'ceiling, so a workload needing more old space than the modeled ' + + 'ceiling looks healthy here.', + }) .option('mcp-client-budget', { type: 'number', description: @@ -669,6 +701,8 @@ export const serveCommand: CommandModule = { mcpClientBudget, mcpBudgetMode: resolvedMcpMode, ...(memoryBudgetMb !== undefined ? { memoryBudgetMb } : {}), + memoryPressureMode: argv['memory-pressure-mode'], + childHeapMode: argv['child-heap-mode'], ...(argv['allow-origin'] && argv['allow-origin'].length > 0 ? { allowOrigins: argv['allow-origin'] } : {}), diff --git a/packages/cli/src/serve/daemon-memory-pressure.test.ts b/packages/cli/src/serve/daemon-memory-pressure.test.ts new file mode 100644 index 0000000000..e56daebd08 --- /dev/null +++ b/packages/cli/src/serve/daemon-memory-pressure.test.ts @@ -0,0 +1,185 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { getHeapStatistics } from 'node:v8'; +import { describe, expect, it } from 'vitest'; +import { + computeDaemonMemoryPressure, + CRITICAL_PRESSURE_RATIO, + HARD_PRESSURE_RATIO, + SOFT_PRESSURE_RATIO, +} from './daemon-memory-pressure.js'; + +const GB = 1024 * 1024 * 1024; + +// Byte counts are exact multiples of the denominator so the ratio the function +// computes is exactly the one under test — rounding between the two would put +// a boundary case on the wrong side of its threshold without failing. +const AVAILABLE = 8 * GB; +function pressureAtBytes(rssBytes: number) { + return computeDaemonMemoryPressure({ + rssBytes, + heapUsedBytes: 0, + availableBytes: AVAILABLE, + heapLimitBytes: 4 * GB, + }); +} + +describe('computeDaemonMemoryPressure', () => { + it.each([ + [0, 'normal'], + [SOFT_PRESSURE_RATIO * AVAILABLE - 1, 'normal'], + [SOFT_PRESSURE_RATIO * AVAILABLE, 'soft'], + [HARD_PRESSURE_RATIO * AVAILABLE - 1, 'soft'], + [HARD_PRESSURE_RATIO * AVAILABLE, 'hard'], + [CRITICAL_PRESSURE_RATIO * AVAILABLE - 1, 'hard'], + [CRITICAL_PRESSURE_RATIO * AVAILABLE, 'critical'], + [AVAILABLE * 1.5, 'critical'], + ])('classifies %p rss bytes as %s', (rssBytes, expected) => { + expect(pressureAtBytes(rssBytes).level).toBe(expected); + }); + + it('reports the worse of the two denominators, and which one it was', () => { + // Heap is the one in trouble; RSS against the cgroup limit looks fine. + const heapBound = computeDaemonMemoryPressure({ + rssBytes: 1 * GB, + heapUsedBytes: Math.round(3.6 * GB), + availableBytes: 32 * GB, + heapLimitBytes: 4 * GB, + }); + expect(heapBound).toMatchObject({ source: 'heap', level: 'critical' }); + expect(heapBound.ratio).toBeCloseTo(0.9, 5); + + // And the reverse: a container near its cgroup limit with a small heap. + const rssBound = computeDaemonMemoryPressure({ + rssBytes: Math.round(3.5 * GB), + heapUsedBytes: Math.round(0.2 * GB), + availableBytes: 4 * GB, + heapLimitBytes: 4 * GB, + }); + expect(rssBound).toMatchObject({ source: 'rss', level: 'critical' }); + }); + + it('treats an unknown denominator as no pressure rather than dividing by zero', () => { + // `availableBytes` of 0 means detection failed, which is not the same as + // "the machine has no memory left". + const noLimit = computeDaemonMemoryPressure({ + rssBytes: 4 * GB, + heapUsedBytes: 1 * GB, + availableBytes: 0, + heapLimitBytes: 4 * GB, + }); + expect(noLimit.rssRatio).toBe(0); + expect(Number.isFinite(noLimit.ratio)).toBe(true); + expect(noLimit.source).toBe('heap'); + + // Neither denominator usable: say so rather than reporting a source that + // was never measured. `unknown` with level `normal` is the honest pair — + // a consumer can see the reading is not evidence of health. + const neither = computeDaemonMemoryPressure({ + rssBytes: 4 * GB, + heapUsedBytes: 1 * GB, + availableBytes: 0, + heapLimitBytes: 0, + }); + expect(neither).toMatchObject({ + ratio: 0, + level: 'normal', + source: 'unknown', + }); + }); + + it.each([Number.NaN, Number.POSITIVE_INFINITY, -1])( + 'treats a %p gauge as unmeasured rather than classifying on it', + (bad) => { + // daemon-metrics-ring sanitizes non-finite gauges to 0 before storing + // them, so these do reach callers in this codebase. + // + // `source: 'unknown'` is the assertion that matters. Without it this + // case passes just as well when an unusable *numerator* is coerced to 0 + // and divided anyway — which publishes `level: 'normal', source: 'rss'` + // for a daemon that measured nothing, the one reading this module exists + // to make impossible. + expect( + computeDaemonMemoryPressure({ + rssBytes: bad, + heapUsedBytes: bad, + availableBytes: 8 * GB, + heapLimitBytes: 4 * GB, + }), + ).toMatchObject({ ratio: 0, level: 'normal', source: 'unknown' }); + + // One bad numerator retires only its own side; the other still reports. + expect( + computeDaemonMemoryPressure({ + rssBytes: bad, + heapUsedBytes: 3 * GB, + availableBytes: 8 * GB, + heapLimitBytes: 4 * GB, + }), + ).toMatchObject({ source: 'heap', ratio: 0.75, rssRatio: 0 }); + + expect( + computeDaemonMemoryPressure({ + rssBytes: 4 * GB, + heapUsedBytes: 0, + availableBytes: bad, + heapLimitBytes: 4 * GB, + }), + ).toMatchObject({ source: 'heap' }); + }, + ); + + it('treats a zero numerator as a reading, not as an unusable gauge', () => { + // The asymmetry the two helpers encode: 0 bytes used is merely implausible, + // while dividing by 0 bytes available is undefined. Retiring a zero + // numerator would make an idle daemon indistinguishable from an + // unmeasurable one — the same collapse from the opposite direction. + expect( + computeDaemonMemoryPressure({ + rssBytes: 0, + heapUsedBytes: 0, + availableBytes: 8 * GB, + heapLimitBytes: 4 * GB, + }), + ).toMatchObject({ ratio: 0, level: 'normal', source: 'rss' }); + }); + + it("falls back to this process's real V8 ceiling when none is given", () => { + // The only production caller omits `heapLimitBytes`, so the documented + // default is the path that actually runs. Every other test here injects a + // limit, which would leave it asserted only by the end-to-end boot test. + const result = computeDaemonMemoryPressure({ + rssBytes: 1 * GB, + heapUsedBytes: 1 * GB, + availableBytes: 8 * GB, + }); + + expect(result.heapLimitBytes).toBe(getHeapStatistics().heap_size_limit); + expect(result.heapRatio).toBeCloseTo( + (1 * GB) / getHeapStatistics().heap_size_limit, + 10, + ); + }); + + it('carries both raw figures so a reader can check the arithmetic', () => { + expect( + computeDaemonMemoryPressure({ + rssBytes: 2 * GB, + heapUsedBytes: 1 * GB, + availableBytes: 8 * GB, + heapLimitBytes: 4 * GB, + }), + ).toMatchObject({ + rssBytes: 2 * GB, + rssRatio: 0.25, + availableBytes: 8 * GB, + heapUsedBytes: 1 * GB, + heapRatio: 0.25, + heapLimitBytes: 4 * GB, + }); + }); +}); diff --git a/packages/cli/src/serve/daemon-memory-pressure.ts b/packages/cli/src/serve/daemon-memory-pressure.ts new file mode 100644 index 0000000000..f7eaf15b9c --- /dev/null +++ b/packages/cli/src/serve/daemon-memory-pressure.ts @@ -0,0 +1,170 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { getHeapStatistics } from 'node:v8'; + +/** + * Thresholds mirror `MemoryPressureMonitor` in core + * (`packages/core/src/services/memoryPressureMonitor.ts`), which is the + * established contract for what these words mean in this codebase. They are + * duplicated rather than imported: the monitor is only constructed by + * `Config.initialize()`, which the daemon never calls, and its limit + * resolution is a private method. Consolidating means extracting from + * `packages/core/src/services/**`, which is a maintainer-gated change and + * belongs in its own review. + */ +export const SOFT_PRESSURE_RATIO = 0.5; +export const HARD_PRESSURE_RATIO = 0.65; +export const CRITICAL_PRESSURE_RATIO = 0.8; + +export type DaemonMemoryPressureLevel = 'normal' | 'soft' | 'hard' | 'critical'; + +/** + * Which side produced the reported ratio, or `unknown` when neither was + * usable. `unknown` is not the same as `normal`: it says the daemon could not + * measure its own pressure, which a consumer must not read as "fine". + * + * A side is usable only when *both* its numerator and its denominator are, so + * this is also the field that says a reported `rssBytes: 0` / `heapUsedBytes: + * 0` is a placeholder rather than a reading. + */ +export type DaemonMemoryPressureSource = 'rss' | 'heap' | 'unknown'; + +export interface DaemonMemoryPressure { + level: DaemonMemoryPressureLevel; + /** The larger of the two ratios below, which is what `level` classifies. */ + ratio: number; + source: DaemonMemoryPressureSource; + rssBytes: number; + rssRatio: number; + /** + * Bytes the daemon tree may use: the cgroup limit, else host memory. + * + * Those two are not equally trustworthy as a denominator, and + * `limits.memory.availableMemorySource` is what tells them apart. Under a + * cgroup (`constrained`) this is exactly the number the OOM killer watches, + * so `rssRatio` is the real thing. On bare metal (`host`) it is the size of + * the machine, and the daemon is killed when the *machine* runs out — which + * depends on every other process on the box. A daemon at 20% of a 64 GB host + * beside a 55 GB neighbour reports `normal` right up until it dies, so under + * `source: 'host'` read `rssRatio` as a lower bound on real pressure rather + * than as a measurement of it. This is a denominator caveat and is not + * covered by the separate one about the thresholds being uncalibrated. + */ + availableBytes: number; + heapUsedBytes: number; + heapRatio: number; + /** + * V8's `heap_size_limit` for this process — the whole heap, not only the old + * space `--max-old-space-size` names, which is what `heapUsedBytes` measures + * against. + */ + heapLimitBytes: number; +} + +function classify(ratio: number): DaemonMemoryPressureLevel { + if (ratio >= CRITICAL_PRESSURE_RATIO) return 'critical'; + if (ratio >= HARD_PRESSURE_RATIO) return 'hard'; + if (ratio >= SOFT_PRESSURE_RATIO) return 'soft'; + return 'normal'; +} + +/** + * Classifies the daemon root's memory pressure against two independent + * denominators, and reports the worse one. + * + * Both are needed: a container usually dies by RSS against its cgroup limit, + * while a process on a large host can exhaust V8's old space long before RSS + * is a meaningful fraction of the machine. Reporting only one hides whichever + * failure the deployment is actually heading for. + * + * This covers the daemon root process only. Aggregate child RSS is a separate + * measurement — the sampler currently reads the primary ACP child alone — so + * this figure must not be read as process-tree pressure. + */ +export function computeDaemonMemoryPressure(input: { + /** Bytes, to match the sampler's gauges. Callers holding MB must convert. */ + rssBytes: number; + heapUsedBytes: number; + /** + * Bytes. `DaemonMemoryBudget.availableMemoryMb` is in **megabytes** — the + * one place these two modules meet, and the one place a factor of 1024² can + * hide, since either unit produces a plausible-looking ratio. + */ + availableBytes: number; + /** Test seam; defaults to this process's real V8 ceiling. */ + heapLimitBytes?: number; +}): DaemonMemoryPressure { + const heapLimitBytes = usableDenominator( + input.heapLimitBytes ?? getHeapStatistics().heap_size_limit, + ); + // A denominator that is absent, zero, or non-finite means "not measurable", + // not "infinitely bad". The sampler already sanitizes non-finite gauges to 0 + // (`finiteGauge` in daemon-metrics-ring), so a 0 arriving here is a real + // possibility rather than a defensive hypothetical. + const availableBytes = usableDenominator(input.availableBytes); + // Numerators need the same treatment, and they need it separately. Coercing + // an unusable numerator to 0 the way a denominator is coerced would publish + // `rssBytes: 0, rssRatio: 0, level: 'normal', source: 'rss'` — a claim that + // the daemon measured itself using almost nothing, indistinguishable on the + // wire from a genuinely idle daemon. That is the exact confusion `source: + // 'unknown'` and `sampled: 0` exist to prevent everywhere else here, so an + // unusable numerator retires its side instead, and `source` reports which + // side actually produced the ratio. Unreachable from the sole production + // caller (`process.memoryUsage()` cannot return NaN), but this is a public + // wire contract and the docs invite readers to check the ratios by hand. + // + // Zero is a reading for a numerator and not for a denominator: a daemon + // using no memory is merely implausible, while dividing by nothing is + // undefined. Hence the two helpers rather than one. + const rss = usableNumerator(input.rssBytes); + const heapUsed = usableNumerator(input.heapUsedBytes); + const rssBytes = rss ?? 0; + const heapUsedBytes = heapUsed ?? 0; + + const rssMeasured = availableBytes > 0 && rss !== null; + const heapMeasured = heapLimitBytes > 0 && heapUsed !== null; + const rssRatio = rssMeasured ? rssBytes / availableBytes : 0; + const heapRatio = heapMeasured ? heapUsedBytes / heapLimitBytes : 0; + + let source: DaemonMemoryPressureSource; + if (!rssMeasured && !heapMeasured) source = 'unknown'; + else if (!heapMeasured) source = 'rss'; + else if (!rssMeasured) source = 'heap'; + // Ties go to RSS. Arbitrary but deterministic, and it only arises when the + // two ratios are equal — in which case either name describes the same + // number, and `level` is unaffected either way. + else source = rssRatio >= heapRatio ? 'rss' : 'heap'; + + const ratio = Math.max(rssRatio, heapRatio); + return { + level: classify(ratio), + ratio, + source, + rssBytes, + rssRatio, + availableBytes, + heapUsedBytes, + heapRatio, + heapLimitBytes, + }; +} + +/** + * Coerces a divisor to a usable positive number; NaN/Infinity/<=0 become 0, + * which every caller reads as "this denominator is not measurable". + */ +function usableDenominator(value: number): number { + return Number.isFinite(value) && value > 0 ? value : 0; +} + +/** + * A measured gauge, or `null` when it is not one. Unlike a denominator, 0 is a + * legitimate reading here, so only non-finite and negative values are retired. + */ +function usableNumerator(value: number): number | null { + return Number.isFinite(value) && value >= 0 ? value : null; +} diff --git a/packages/cli/src/serve/daemon-status.test.ts b/packages/cli/src/serve/daemon-status.test.ts index 36f6f59cd9..ba6bb49536 100644 --- a/packages/cli/src/serve/daemon-status.test.ts +++ b/packages/cli/src/serve/daemon-status.test.ts @@ -25,6 +25,7 @@ import type { ChannelWorkerSnapshot } from './channel-worker-supervisor.js'; import type { RateLimiterInstance, RateLimitTier } from './rate-limit.js'; import type { DaemonWorkspaceService } from './workspace-service/index.js'; import type { DaemonLogger } from './daemon-logger.js'; +import { createChildHeapPolicy } from '@qwen-code/acp-bridge/childHeapPolicy'; import { resolveDaemonMemoryBudget } from '@qwen-code/acp-bridge/daemonMemoryBudget'; const BASE_WORKSPACE = '/work/status'; @@ -130,6 +131,69 @@ describe('buildDaemonStatusResponse', () => { expect(response.limits.maxTotalSessions).toBe(50); }); + it('reports the modeled partition without claiming it is applied', async () => { + const budget = resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }); + const options = makeOptions(); + options.opts.daemonMemoryBudget = budget; + const policy = createChildHeapPolicy({ budget, mode: 'observe' }); + policy.decide(10_000); // one would-be refusal, so the counter is not trivially 0 + options.getChildHeapPolicySnapshot = () => policy.snapshot(); + + const response = await buildDaemonStatusResponse('summary', options); + + // The figures an operator needs to judge the partition for themselves — + // publishing them is the substitute for a refusal count that cannot say + // whether the ceiling would fit their workload. + expect(response.limits.memory).toMatchObject({ + enforced: false, + childHeap: { + mode: 'observe', + maxConcurrentChildren: 7, + perChildCeilingMb: 526, + refusals: 1, + }, + }); + }); + + it('reports no child-heap policy as null rather than as a disabled one', async () => { + // Direct-embed and the bootstrap window build no policy. `null` says + // "there is no policy", which a client must not read as "mode off". + const options = makeOptions(); + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 8_192, + }); + const response = await buildDaemonStatusResponse('summary', options); + + expect(response.limits.memory).toMatchObject({ + enforced: false, + childHeap: null, + }); + }); + + it('reports an off policy as present but modeling nothing', async () => { + // The third state, and the reason the two above are not enough: a policy + // exists and its mode is visible, but it published no partition. On this + // same budget `observe` reports 7 children at 526 MB, so carrying those + // figures here would show an operator a partition they turned off. + const options = makeOptions(); + const budget = resolveDaemonMemoryBudget({ availableMemoryMb: 8_192 }); + options.opts.daemonMemoryBudget = budget; + const policy = createChildHeapPolicy({ budget, mode: 'off' }); + options.getChildHeapPolicySnapshot = () => policy.snapshot(); + + const response = await buildDaemonStatusResponse('summary', options); + + expect(response.limits.memory).toMatchObject({ + enforced: false, + childHeap: { + mode: 'off', + maxConcurrentChildren: null, + perChildCeilingMb: null, + refusals: 0, + }, + }); + }); + it('reports the resolved memory budget in daemon status limits', () => { const options = makeOptions(); options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ @@ -181,11 +245,17 @@ describe('buildDaemonStatusResponse', () => { expect(response.runtime.memory).toEqual({ registeredWorkspaces: 1, activeAcpChildren: 1, - childRssCoverage: 'primary_only', + childRssCoverage: 'active_children', + // No registry in this case, so there are no bridges to enumerate — the + // honest reading is "nothing measured", not "no children". + children: { rssBytes: 0, sampled: 0, oldestReadingAgeMs: null }, modeled: { recommendedShareAtRegisteredMb: 15_360, recommendedShareAtActiveMb: 15_360, }, + // Real process figures; asserted for shape here and for arithmetic in + // the dedicated pressure tests. `toEqual` still fails on an extra key. + pressure: expect.objectContaining({ mode: 'observe' }), }); }); @@ -221,11 +291,15 @@ describe('buildDaemonStatusResponse', () => { expect(response.runtime.memory).toEqual({ registeredWorkspaces: 0, activeAcpChildren: 0, - childRssCoverage: 'primary_only', + childRssCoverage: 'active_children', + // No registry in this case, so there are no bridges to enumerate — the + // honest reading is "nothing measured", not "no children". + children: { rssBytes: 0, sampled: 0, oldestReadingAgeMs: null }, modeled: { recommendedShareAtRegisteredMb: null, recommendedShareAtActiveMb: null, }, + pressure: expect.objectContaining({ mode: 'observe' }), }); }); @@ -283,6 +357,200 @@ describe('buildDaemonStatusResponse', () => { }); }); + it('sums only the children that actually reported, and says how many did', async () => { + // Every state a live child can be in, in one response. `sampled` is what + // separates "measured and small" from "never measured": without it, the + // three unreported children below are indistinguishable from children + // using no memory. + const liveWith = (rssBytes: number, ageMs: number) => + ({ + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + getChildResourceSnapshot: () => ({ rssBytes, cpuPercent: 1, ageMs }), + lastActivityAt: null, + }) as unknown as AcpSessionBridge; + const liveUnpolled = { + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + // Live, but stale or never polled — the hook exists and returns nothing. + getChildResourceSnapshot: () => undefined, + lastActivityAt: null, + } as unknown as AcpSessionBridge; + const liveOlderContract = { + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + // An injected bridge predating the hook entirely. + lastActivityAt: null, + } as unknown as AcpSessionBridge; + const dormant = { + getDaemonStatusSnapshot: () => ({ + ...BASE_BRIDGE_SNAPSHOT, + channelLive: false, + }), + isChannelLive: () => false, + // Deliberately unfaithful: the real hook self-gates on a live channel + // and would return undefined here. This stub does not, so the test + // fails unless the sum gates on `isChannelLive` itself. + getChildResourceSnapshot: () => ({ + rssBytes: 999, + cpuPercent: 1, + ageMs: 1, + }), + lastActivityAt: null, + } as unknown as AcpSessionBridge; + + const runtimes = [ + // Descending age on purpose: with the oldest reading enumerated FIRST, + // a plain-overwrite accumulator yields 1_000 and fails. Ascending order + // would let "last contributor wins" pass with the same expectation. + { + workspaceId: 'a', + workspaceCwd: BASE_WORKSPACE, + bridge: liveWith(100, 9_000), + }, + { + workspaceId: 'b', + workspaceCwd: '/work/b', + bridge: liveWith(200, 1_000), + }, + { workspaceId: 'c', workspaceCwd: '/work/c', bridge: liveUnpolled }, + { workspaceId: 'd', workspaceCwd: '/work/d', bridge: liveOlderContract }, + { workspaceId: 'e', workspaceCwd: '/work/e', bridge: dormant }, + ]; + const options = makeOptions(); + options.bridge = runtimes[0].bridge; + options.workspaceRegistry = { + primary: { workspaceCwd: BASE_WORKSPACE, bridge: runtimes[0].bridge }, + list: () => runtimes, + listManaged: () => runtimes, + listEntries: () => runtimes.map(() => ({})), + } as unknown as BuildDaemonStatusOptions['workspaceRegistry']; + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 32_768, + }); + + const response = await buildDaemonStatusResponse('summary', options); + + expect(response.runtime.memory?.childRssCoverage).toBe('active_children'); + // Four children are live; only two reported. The dormant one is excluded + // from both figures even though its stub would have returned a value. + expect(response.runtime.memory?.activeAcpChildren).toBe(4); + expect(response.runtime.memory?.children).toEqual({ + rssBytes: 300, + sampled: 2, + // The oldest contributor, not the newest: the sum spans this much time. + oldestReadingAgeMs: 9_000, + }); + // The gap is visible without the client having to know it exists. + expect(response.runtime.memory!.children.sampled).toBeLessThan( + response.runtime.memory!.activeAcpChildren, + ); + }); + + it('reports a zero age as zero, and ages a mixed-contract sum by the ones that can', async () => { + // `ageMs` is exactly 0 when a status read lands in the same millisecond as + // the sampler's stamp. A truthiness guard, or a trailing `|| null`, turns + // that measured-fresh reading into `null` — which the field's own docs say + // never means fresh. + const freshBridge = { + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + getChildResourceSnapshot: () => ({ + rssBytes: 100, + cpuPercent: 1, + ageMs: 0, + }), + lastActivityAt: null, + } as unknown as AcpSessionBridge; + const withRegistry = (bridges: AcpSessionBridge[]) => { + const runtimes = bridges.map((bridge, i) => ({ + workspaceId: `w${i}`, + workspaceCwd: i === 0 ? BASE_WORKSPACE : `/work/w${i}`, + bridge, + })); + const options = makeOptions(); + options.bridge = bridges[0]; + options.workspaceRegistry = { + primary: { workspaceCwd: BASE_WORKSPACE, bridge: bridges[0] }, + list: () => runtimes, + listManaged: () => runtimes, + listEntries: () => runtimes.map(() => ({})), + } as unknown as BuildDaemonStatusOptions['workspaceRegistry']; + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 32_768, + }); + return options; + }; + + const fresh = await buildDaemonStatusResponse( + 'summary', + withRegistry([freshBridge]), + ); + expect(fresh.runtime.memory?.children.oldestReadingAgeMs).toBe(0); + + // Mixing an age-carrying contributor with a pre-`ageMs` one must age the + // sum by the ones that can report, not reset the whole thing to null. + const olderContract = { + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + getChildResourceSnapshot: () => ({ rssBytes: 50, cpuPercent: 1 }), + lastActivityAt: null, + } as unknown as AcpSessionBridge; + const aged = { + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + getChildResourceSnapshot: () => ({ + rssBytes: 70, + cpuPercent: 1, + ageMs: 5_000, + }), + lastActivityAt: null, + } as unknown as AcpSessionBridge; + const mixed = await buildDaemonStatusResponse( + 'summary', + withRegistry([olderContract, aged]), + ); + expect(mixed.runtime.memory?.children).toMatchObject({ + rssBytes: 120, + sampled: 2, + oldestReadingAgeMs: 5_000, + }); + }); + + it('counts a child whose bridge predates ageMs, but cannot age the sum', async () => { + // Distinct from a missing hook: the hook is present and returns a reading, + // it just carries no age. That child must still contribute memory, and + // `null` must not be mistaken for "sampled nothing". + const preAgeMs = { + getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, + isChannelLive: () => true, + getChildResourceSnapshot: () => ({ rssBytes: 512, cpuPercent: 3 }), + lastActivityAt: null, + } as unknown as AcpSessionBridge; + const runtimes = [ + { workspaceId: 'a', workspaceCwd: BASE_WORKSPACE, bridge: preAgeMs }, + ]; + const options = makeOptions(); + options.bridge = preAgeMs; + options.workspaceRegistry = { + primary: { workspaceCwd: BASE_WORKSPACE, bridge: preAgeMs }, + list: () => runtimes, + listManaged: () => runtimes, + listEntries: () => [{}], + } as unknown as BuildDaemonStatusOptions['workspaceRegistry']; + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 32_768, + }); + + const response = await buildDaemonStatusResponse('summary', options); + + expect(response.runtime.memory?.children).toEqual({ + rssBytes: 512, + sampled: 1, + oldestReadingAgeMs: null, + }); + }); + it('counts a draining workspace that still holds a live child', async () => { // A workspace mid-drain (or mid-replacement, or blocked) is dropped by // list() — active-state only — yet its ACP child is still alive. The live @@ -297,6 +565,15 @@ describe('buildDaemonStatusResponse', () => { const drainingBridge = { getDaemonStatusSnapshot: () => BASE_BRIDGE_SNAPSHOT, // channelLive: true isChannelLive: () => true, + // Reports RSS too: `list()` is active-state only and would drop this + // draining-but-process-holding workspace, so summing over it instead of + // `listManaged()` under-reports child RSS in exactly the drain window + // while `activeAcpChildren` still counts the child. + getChildResourceSnapshot: () => ({ + rssBytes: 4_096, + cpuPercent: 1, + ageMs: 10, + }), lastActivityAt: null, } as unknown as AcpSessionBridge; const primaryRuntime = { @@ -329,6 +606,15 @@ describe('buildDaemonStatusResponse', () => { registeredWorkspaces: 2, activeAcpChildren: 2, }); + // Only the draining bridge reports a reading, so this byte count can come + // from nowhere else: it pins that the sum enumerates the process-holding + // set (`listManaged()`), not the active-state set (`list()`), which would + // drop this child and leave `sampled` at 0 — a silent under-report + // confined to the drain window, while `activeAcpChildren` still counts it. + expect(response.runtime.memory?.children).toMatchObject({ + rssBytes: 4_096, + sampled: 1, + }); }); it('counts a single workspace on the external-bridge path', async () => { @@ -346,6 +632,141 @@ describe('buildDaemonStatusResponse', () => { }); }); + it('reports pressure figures in both modes, but only observe raises an issue', async () => { + // A 1 MiB denominator puts this test process far past `critical`, which is + // the only way to exercise the gate: at a realistic denominator the level + // is `normal` and no issue is raised in either mode, so the assertion + // below would hold even with the gate deleted. + const responses = await Promise.all( + (['off', 'observe'] as const).map((mode) => { + const options = makeOptions(); + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 1, + }); + options.opts.memoryPressureMode = mode; + return buildDaemonStatusResponse('summary', options); + }), + ); + const [offResponse, observeResponse] = responses; + + // Both report the reading — that is the point of `off` still observing. + for (const response of responses) { + expect(response.runtime.memory?.pressure.level).toBe('critical'); + expect(response.runtime.memory?.pressure.availableBytes).toBe( + 1024 * 1024, + ); + } + expect(offResponse.runtime.memory?.pressure.mode).toBe('off'); + expect(observeResponse.runtime.memory?.pressure.mode).toBe('observe'); + + // Only `observe` turns it into a verdict. + const pressureIssues = (r: (typeof responses)[number]) => + r.issues.filter((issue) => issue.code === 'daemon_memory_pressure'); + expect(pressureIssues(offResponse)).toHaveLength(0); + expect(pressureIssues(observeResponse)).toHaveLength(1); + // Warning, not error, while the thresholds are uncalibrated. + expect(pressureIssues(observeResponse)[0].severity).toBe('warning'); + // Pin the wire strings at runtime. Both the issue-code union member and + // the `pressure` field declaration are otherwise guarded only by tsc, + // which vitest does not run — a rename would ship green. + expect(pressureIssues(observeResponse)[0].code).toBe( + 'daemon_memory_pressure', + ); + expect( + Object.keys(observeResponse.runtime.memory!.pressure).sort(), + ).toEqual([ + 'availableBytes', + 'heapLimitBytes', + 'heapRatio', + 'heapUsedBytes', + 'level', + 'mode', + 'ratio', + 'rssBytes', + 'rssRatio', + 'source', + ]); + // The denominator named in the message follows `source`; inverting the + // ternary is otherwise invisible, and would send an operator hunting RSS + // growth during a heap-driven incident. + expect(pressureIssues(observeResponse)[0].message).toContain( + 'of available memory', + ); + // Both halves of the documented contract: the issue reaches the rollup in + // `observe` (a severity or list that bypassed it would leave this `ok`), + // and `off` leaves the rollup exactly where it was. Same input, so the + // only difference between these two is the mode. + expect(observeResponse.status).not.toBe('ok'); + expect(offResponse.status).toBe('ok'); + }); + + it('raises nothing on a healthy daemon, and a warning once pressure leaves normal', async () => { + // The `level !== 'normal'` half of the gate. Without this, deleting that + // clause keeps the whole suite green while every /daemon/status response + // on a healthy daemon carries a daemon_memory_pressure warning and a + // top-level `warning` status — the exact false positive `off` exists for. + const healthy = makeOptions(); + healthy.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 1_048_576, + }); + const healthyResponse = await buildDaemonStatusResponse('summary', healthy); + expect(healthyResponse.runtime.memory?.pressure.level).toBe('normal'); + expect( + healthyResponse.issues.filter( + (issue) => issue.code === 'daemon_memory_pressure', + ), + ).toHaveLength(0); + expect(healthyResponse.status).toBe('ok'); + + // And the other side of the same clause: a denominator sized so this + // process lands between the soft and hard thresholds must raise exactly + // one warning. Tightening the gate to `=== 'critical'` fails here. + const rss = process.memoryUsage().rss; + const softOptions = makeOptions(); + softOptions.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + // Target ~57% — comfortably inside [0.5, 0.65) at either rounding edge. + availableMemoryMb: Math.ceil(rss / 0.57 / (1024 * 1024)), + }); + const softResponse = await buildDaemonStatusResponse( + 'summary', + softOptions, + ); + expect(softResponse.runtime.memory?.pressure.level).toBe('soft'); + expect( + softResponse.issues.filter( + (issue) => issue.code === 'daemon_memory_pressure', + ), + ).toHaveLength(1); + }); + + it('defaults to observe when no mode was configured', async () => { + const options = makeOptions(); + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 32_768, + }); + const response = await buildDaemonStatusResponse('summary', options); + + expect(response.runtime.memory?.pressure.mode).toBe('observe'); + }); + + it('converts the budget from megabytes when computing the ratio', async () => { + // The budget module speaks MB and the pressure module speaks bytes. A + // missing 1024x here still yields a plausible-looking ratio, so assert the + // denominator directly rather than trusting the level. + const options = makeOptions(); + options.opts.daemonMemoryBudget = resolveDaemonMemoryBudget({ + availableMemoryMb: 4_096, + }); + const response = await buildDaemonStatusResponse('summary', options); + + const pressure = response.runtime.memory!.pressure; + expect(pressure.availableBytes).toBe(4_096 * 1024 * 1024); + expect(pressure.rssRatio).toBeCloseTo( + pressure.rssBytes / (4_096 * 1024 * 1024), + 10, + ); + }); + it('omits memory reporting when no budget was resolved', async () => { const response = await buildDaemonStatusResponse('summary', makeOptions()); diff --git a/packages/cli/src/serve/daemon-status.ts b/packages/cli/src/serve/daemon-status.ts index 6b0249bb91..b36d012897 100644 --- a/packages/cli/src/serve/daemon-status.ts +++ b/packages/cli/src/serve/daemon-status.ts @@ -23,6 +23,14 @@ import { recommendedChildShareMb, type DaemonMemoryBudget, } from '@qwen-code/acp-bridge/daemonMemoryBudget'; +import type { + ChildHeapMode, + ChildHeapPolicySnapshot, +} from '@qwen-code/acp-bridge/childHeapPolicy'; +import { + computeDaemonMemoryPressure, + type DaemonMemoryPressure, +} from './daemon-memory-pressure.js'; import { isLoopbackBind } from './loopback-binds.js'; import type { RateLimiterInstance, RateLimitTier } from './rate-limit.js'; import type { ServeOptions } from './types.js'; @@ -88,7 +96,8 @@ export interface DaemonStatusIssue { | 'channel_worker_partial_connect' | 'daemon_runtime_starting' | 'daemon_runtime_failed' - | 'daemon_log_degraded'; + | 'daemon_log_degraded' + | 'daemon_memory_pressure'; severity: IssueSeverity; message: string; section?: string; @@ -121,6 +130,8 @@ export interface BuildDaemonStatusOptions { getPerfSnapshot?: () => DaemonPerfSnapshot; getMetricsSeries?: () => DaemonMetricsBucket[]; getTotalSessionAdmissionSnapshot?: () => TotalSessionAdmissionSnapshot; + /** Returns undefined when no policy was built — direct-embed, or no budget. */ + getChildHeapPolicySnapshot?: () => ChildHeapPolicySnapshot | undefined; } interface DaemonStatusSection { @@ -191,6 +202,39 @@ export interface DaemonStatusMemoryLimits { * namespace for enforcement that has not shipped. */ enforced: false; + /** + * The per-child heap partition the daemon models but does not apply. + * `null` when no policy was built. + */ + childHeap: { + mode: ChildHeapMode; + /** + * Children the pool could host at once. 0 when no partition can be + * modeled — either the pool cannot cover one child at the minimum heap, + * or the ceiling would land under that minimum once capped at today's + * host-derived one. `null` under `off`, which models nothing and so is + * not the same claim as a pool that hosts zero children. + */ + maxConcurrentChildren: number | null; + /** + * What each would receive. Never 0 and never below + * `modeled.minChildHeapMb`; `null` instead, both under `off` and wherever + * the partition cannot be modeled within that floor. + */ + perChildCeilingMb: number | null; + /** + * Spawns that would have exceeded `maxConcurrentChildren`. Admission + * pressure only: 0 does **not** mean the partition is safe to apply, + * because children still run on the much larger host-derived ceiling. + * + * Two known sources of counts that are not capacity pressure: a channel + * swap on a daemon already at `maxConcurrentChildren` books one, because + * the terminating child is counted until it exits; and on a host too + * small to model a partition this equals the total ACP spawn count, with + * `insufficientMemory` as the field that says why. + */ + refusals: number; + } | null; /** What was asked for: the flag value, or half of available memory. */ configuredBudgetMb: number; /** `configured` capped at resolved cgroup/host memory. */ @@ -221,10 +265,19 @@ export interface DaemonStatusMemoryLimits { export function toDaemonStatusMemoryLimits( budget: DaemonMemoryBudget | undefined, + childHeap?: ChildHeapPolicySnapshot, ): DaemonStatusMemoryLimits | null { if (!budget) return null; return { enforced: false, + childHeap: childHeap + ? { + mode: childHeap.mode, + maxConcurrentChildren: childHeap.maxConcurrentChildren, + perChildCeilingMb: childHeap.perChildCeilingMb, + refusals: childHeap.refusals, + } + : null, configuredBudgetMb: budget.configuredBudgetMb, effectiveBudgetMb: budget.effectiveBudgetMb, budgetSource: budget.budgetSource, @@ -317,14 +370,46 @@ interface DaemonStatusRuntimeMemory { */ activeAcpChildren: number; /** - * Which children the daemon's RSS sampling actually covers. Only the primary - * ACP child is sampled today, so this section must not be read as - * process-tree observation. Sampling is gated on an active SSE/WS watcher; - * when no client is observing, childRssBytes reads 0 even for the primary. - * The drop is not instant: after the last watcher detaches, the last sampled - * value persists until it ages out of the staleness window (~30s). + * Which children the daemon's RSS sampling covers: every ACP child with a + * live channel, i.e. the same set `activeAcpChildren` counts. Still not + * process-tree observation — channel workers and the children's own MCP + * descendants report nothing (see `children`). + * + * Sampling is gated on an active SSE/WS watcher; with no client observing, + * `children.sampled` falls to 0 even though children are live. The drop is + * not instant: after the last watcher detaches, each reading persists until + * it ages out of the staleness window (~30s). */ - childRssCoverage: 'primary_only'; + childRssCoverage: 'active_children'; + /** + * Aggregate RSS across the children `childRssCoverage` names. + * + * Read it as a floor and an over-count at the same time. Over, because + * summing per-process RSS double-counts pages the children share (the node + * binary, libc). Under, because each child reports only its own process — + * MCP servers it spawned are invisible here, and channel workers have no + * reporting path at all. It is not "the daemon tree's memory". + */ + children: { + /** + * Sum over children that produced a reading. When `sampled` is below the + * sibling `activeAcpChildren`, this is a floor rather than a total. + */ + rssBytes: number; + /** + * How many children contributed. The denominator is `activeAcpChildren`, + * deliberately not repeated here. 0 with live children means nothing was + * measured — either no watcher is gating the sampler open, or the daemon + * was built without a workspace registry to enumerate. + */ + sampled: number; + /** + * Age of the oldest reading in the sum, so a caller can tell how far apart + * its parts were taken. `null` when nothing was sampled — and also when + * every contributor predates the field, so `null` never means "fresh". + */ + oldestReadingAgeMs: number | null; + }; /** * Modeled per-child shares. Advisory; nothing applies them. Each is capped * at the legacy child ceiling, and floored at the minimum child heap only @@ -338,6 +423,33 @@ interface DaemonStatusRuntimeMemory { /** `null` when no ACP child is active — there is no share to divide. */ recommendedShareAtActiveMb: number | null; }; + /** + * The daemon root's own memory pressure. Reported in both modes; only + * `observe` also raises a status issue from it. Covers the root process + * alone: these figures are `process.memoryUsage()` of this process, so a + * daemon whose children are the ones growing still reports `normal`. + * Compare against `children.rssBytes` to see that gap. + * + * The computed shape is referenced rather than restated so the two cannot + * drift: a field added or renamed in `daemon-memory-pressure.ts` would not + * be caught by a hand copy, since spreading an object with an extra property + * is not an excess-property error. `availableBytes` is the same figure as + * `limits.memory.availableMemoryMb`, repeated here in bytes so the ratio can + * be checked without cross-referencing. + * + * Nested here rather than at `runtime`, so it is absent whenever no budget + * resolved — even though the heap half of the signal needs no budget. That + * reaches direct-embed callers, and also the bootstrap `/daemon/status` + * route — which omits `runtime.memory` wholesale even though the budget is + * resolved before the bootstrap app exists, so `limits.memory` is populated + * there while `pressure` is not. That window is not only startup: a daemon + * whose runtime fails to start keeps serving the bootstrap app for its + * lifetime, which is exactly when the reading would explain the most. Do + * not write a client against "budget resolved implies pressure present". + * Hoisting it out would restructure the block for a path that does not need + * the reading. + */ + pressure: DaemonMemoryPressure & { mode: 'off' | 'observe' }; } export interface DaemonPipeStatsSnapshot { @@ -476,10 +588,57 @@ export async function buildDaemonStatusResponse( const registeredWorkspaceCount = input.workspaceRegistry ? input.workspaceRegistry.listEntries().length : workspaceSnapshots.length; + // Summed in the SAME synchronous pass that produced `activeAcpChildCount` + // above, over the same array. Keep it that way: an `await` slipped between + // them would not break `sampled <= activeAcpChildren` — a child that dies + // drops out of the sum, and one that starts has no cached reading yet — it + // would instead make the two figures describe different instants, so the + // gap between them would quietly absorb children that came or went while + // the response was being built. That gap is the entire reason `sampled` is + // reported, and no assertion would catch it going wrong. + let childRssBytesTotal = 0; + let childRssSampled = 0; + let oldestChildReadingAgeMs: number | null = null; + for (const runtime of managedRuntimes ?? []) { + // Gate on the same predicate `activeAcpChildCount` used, rather than + // trusting `getChildResourceSnapshot` to return nothing for a dead + // channel. It does today, but that is another package's internal, and + // leaning on it would make `sampled <= activeAcpChildren` — the one + // thing this block promises — hold by coincidence instead of by + // construction. + if (!runtime.bridge.isChannelLive()) continue; + const snapshot = runtime.bridge.getChildResourceSnapshot?.(); + if (!snapshot) continue; + childRssBytesTotal += snapshot.rssBytes; + childRssSampled += 1; + // Absent on bridges predating the field; such a child still counts + // toward the sum, it just cannot say how old its reading is. + if (snapshot.ageMs !== undefined) { + oldestChildReadingAgeMs = Math.max( + oldestChildReadingAgeMs ?? 0, + snapshot.ageMs, + ); + } + } + const pressureMode = input.opts.memoryPressureMode ?? 'observe'; + // One reading for the two figures of a single ratio. Reading twice would + // divide an rss and a heapUsed sampled at different instants. + // + // Deliberately not shared with `runtime.process` further down: a + // `detail=full` request awaits the workspace sections between here and + // there, so reusing this snapshot would silently change which instant that + // pre-existing field reports. A second syscall is cheaper than a semantics + // change to a field this PR is not about. + const pressureMemory = process.memoryUsage(); runtimeMemory = { registeredWorkspaces: registeredWorkspaceCount, activeAcpChildren: activeAcpChildCount, - childRssCoverage: 'primary_only', + childRssCoverage: 'active_children', + children: { + rssBytes: childRssBytesTotal, + sampled: childRssSampled, + oldestReadingAgeMs: oldestChildReadingAgeMs, + }, modeled: { recommendedShareAtRegisteredMb: registeredWorkspaceCount > 0 @@ -490,6 +649,22 @@ export async function buildDaemonStatusResponse( ? recommendedChildShareMb(memoryBudget, activeAcpChildCount) : null, }, + pressure: { + ...computeDaemonMemoryPressure({ + rssBytes: pressureMemory.rss, + heapUsedBytes: pressureMemory.heapUsed, + // `availableMemoryMb`, not `effectiveBudgetMb`: pressure asks how + // close this process is to being killed, and what kills it is the + // cgroup limit or host memory. An operator's budget is a policy + // number — exceeding it is not fatal, so classifying against it + // would report `critical` for a daemon in no danger. + // Note the unit change: the budget carries megabytes. + availableBytes: memoryBudget.availableMemoryMb * 1024 * 1024, + }), + // After the spread, so the flag stays authoritative if the computed + // shape ever grows a field of this name. + mode: pressureMode, + }, }; } const aggregatedLastActivity = workspaceSnapshots.reduce( @@ -559,6 +734,42 @@ export async function buildDaemonStatusResponse( totalAdmissionSnapshot, workspaceSnapshots, ); + // Only `observe` turns the level into an issue. `off` still reported the + // figures above; what it withholds is the effect on `rollupStatus`, which + // any one issue flips from `ok` to `warning`. The thresholds are inherited + // from an interactive-CLI monitor and are not yet calibrated for a + // long-running daemon, so a deployment that alerts on the top-level status + // needs a way to take the reading without the verdict. + if ( + runtimeMemory && + runtimeMemory.pressure.mode === 'observe' && + runtimeMemory.pressure.level !== 'normal' + ) { + const { level, ratio, source } = runtimeMemory.pressure; + issues.push({ + code: 'daemon_memory_pressure', + // `warning` at every level, including `critical`. An `error` severity + // makes `rollupStatus` return `error` for the whole daemon, which is a + // strong claim to stake on thresholds borrowed from an interactive-CLI + // monitor and not yet calibrated here. The level itself is reported in + // `runtime.memory.pressure`, so nothing is lost by keeping the rollup + // at `warning` until the numbers have been checked against real + // deployments — which is what this phase is for. + severity: 'warning', + // Name the denominator, not the numerator: "% of the rss limit" would + // call the measured value a limit. `section` is omitted because every + // other use of it names a workspace status section, and this is a + // daemon-level concern — the same reason `daemon_log_degraded` omits it. + // One decimal, not zero: at 0 decimals a ratio of 0.795 rounds to "80%" + // while `level` still reads `hard`, and 80% is critical's documented + // threshold. An oncall engineer comparing the two sees a contradiction + // in the one feature whose whole purpose is trustworthy triage. + message: + `Daemon memory pressure is ${level} at ` + + `${(ratio * 100).toFixed(1)}% of ` + + `${source === 'heap' ? 'the V8 heap limit' : 'available memory'}.`, + }); + } if (daemonLogStatus?.health === 'degraded') { issues.push({ code: 'daemon_log_degraded', @@ -638,7 +849,10 @@ export async function buildDaemonStatusResponse( channelIdleTimeoutMs: bridgeSnapshot.limits.channelIdleTimeoutMs, sessionIdleTimeoutMs: bridgeSnapshot.limits.sessionIdleTimeoutMs, acpConnectionCap: acpSnapshot?.connectionCap ?? null, - memory: toDaemonStatusMemoryLimits(memoryBudget), + memory: toDaemonStatusMemoryLimits( + memoryBudget, + input.getChildHeapPolicySnapshot?.(), + ), }, ...(workspaceRuntimes && workspaceRuntimes.length > 1 ? { diff --git a/packages/cli/src/serve/fast-path.test.ts b/packages/cli/src/serve/fast-path.test.ts index 56145a36bb..7496aff9eb 100644 --- a/packages/cli/src/serve/fast-path.test.ts +++ b/packages/cli/src/serve/fast-path.test.ts @@ -686,6 +686,8 @@ describe('serve fast path argument parsing', () => { ['open', ['--open']], ['http-bridge', ['--no-http-bridge']], ['memory-budget-mb', ['--memory-budget-mb', '8192']], + ['memory-pressure-mode', ['--memory-pressure-mode', 'observe']], + ['child-heap-mode', ['--child-heap-mode', 'observe']], ['mcp-client-budget', ['--mcp-client-budget', '10']], ['mcp-budget-mode', ['--mcp-budget-mode', 'warn']], ['allow-origin', ['--allow-origin', 'http://localhost:3000']], @@ -765,6 +767,40 @@ describe('serve fast path argument parsing', () => { ); }); + it('parses --memory-pressure-mode and falls back on an unknown value', () => { + for (const argv of [ + ['serve', '--memory-pressure-mode', 'off'], + ['serve', '--memory-pressure-mode=off'], + ]) { + expect(parseServeFastPathArgs(argv)).toMatchObject({ + kind: 'serve', + options: { memoryPressureMode: 'off' }, + }); + } + // An out-of-range choice defers to yargs rather than the fast path + // inventing a second wording for the same error. + expect( + parseServeFastPathArgs(['serve', '--memory-pressure-mode', 'enforce']), + ).toEqual({ kind: 'fallback' }); + }); + + it('parses --child-heap-mode and falls back on an unknown value', () => { + for (const argv of [ + ['serve', '--child-heap-mode', 'off'], + ['serve', '--child-heap-mode=off'], + ]) { + expect(parseServeFastPathArgs(argv)).toMatchObject({ + kind: 'serve', + options: { childHeapMode: 'off' }, + }); + } + // `enforce` is deliberately not a value yet, so it is the sample worth + // pinning: the fast path must defer to yargs rather than smuggle it in. + expect( + parseServeFastPathArgs(['serve', '--child-heap-mode', 'enforce']), + ).toEqual({ kind: 'fallback' }); + }); + it('parses --memory-budget-mb on the fast path in both spellings', () => { for (const argv of [ ['serve', '--memory-budget-mb', '8192'], diff --git a/packages/cli/src/serve/fast-path.ts b/packages/cli/src/serve/fast-path.ts index bbcf816513..e7ba58dd6c 100644 --- a/packages/cli/src/serve/fast-path.ts +++ b/packages/cli/src/serve/fast-path.ts @@ -405,6 +405,36 @@ export function parseServeFastPathArgs( continue; } + if (flag === 'memory-pressure-mode') { + const read = readOptionValue(argv, i, inlineValue); + if (!read) return { kind: 'fallback' }; + i = read.nextIndex; + // Unlike mcp-budget-mode, which captures the raw value and validates it + // later, an out-of-range value here falls back to the full yargs path: + // its `choices` already owns the error message, and letting an unknown + // string through would put a value in `ServeOptions` that its own type + // says cannot exist. + if (read.value !== 'off' && read.value !== 'observe') { + return { kind: 'fallback' }; + } + options.memoryPressureMode = read.value; + continue; + } + + if (flag === 'child-heap-mode') { + const read = readOptionValue(argv, i, inlineValue); + if (!read) return { kind: 'fallback' }; + i = read.nextIndex; + // Same reasoning as memory-pressure-mode: yargs `choices` already owns + // the error message for a bad value, and letting an unknown string past + // here would put a value in `ServeOptions` its own type forbids. + if (read.value !== 'off' && read.value !== 'observe') { + return { kind: 'fallback' }; + } + options.childHeapMode = read.value; + continue; + } + if (flag === 'mcp-budget-mode') { const read = readOptionValue(argv, i, inlineValue); if (!read) return { kind: 'fallback' }; diff --git a/packages/cli/src/serve/routes/daemon-status.ts b/packages/cli/src/serve/routes/daemon-status.ts index 2c4bc19943..252e89a498 100644 --- a/packages/cli/src/serve/routes/daemon-status.ts +++ b/packages/cli/src/serve/routes/daemon-status.ts @@ -28,6 +28,7 @@ import type { DaemonWorkspaceService } from '../workspace-service/index.js'; import { getServeProtocolVersions } from '../capabilities.js'; import type { TotalSessionAdmissionSnapshot } from '../total-session-admission.js'; import type { WorkspaceRegistry } from '../workspace-registry.js'; +import type { ChildHeapPolicySnapshot } from '@qwen-code/acp-bridge/childHeapPolicy'; interface RegisterDaemonStatusRoutesDeps { opts: ServeOptions; @@ -52,6 +53,7 @@ interface RegisterDaemonStatusRoutesDeps { getPerfSnapshot?: () => DaemonPerfSnapshot; getMetricsSeries?: () => DaemonMetricsBucket[]; getTotalSessionAdmissionSnapshot?: () => TotalSessionAdmissionSnapshot; + getChildHeapPolicySnapshot?: () => ChildHeapPolicySnapshot | undefined; } export function registerDaemonStatusRoutes( @@ -92,6 +94,7 @@ export function registerDaemonStatusRoutes( getMetricsSeries: deps.getMetricsSeries, getTotalSessionAdmissionSnapshot: deps.getTotalSessionAdmissionSnapshot, + getChildHeapPolicySnapshot: deps.getChildHeapPolicySnapshot, }), ); } catch (err) { diff --git a/packages/cli/src/serve/run-qwen-serve.test.ts b/packages/cli/src/serve/run-qwen-serve.test.ts index bc1d761fbf..47986a5fff 100644 --- a/packages/cli/src/serve/run-qwen-serve.test.ts +++ b/packages/cli/src/serve/run-qwen-serve.test.ts @@ -1996,7 +1996,8 @@ describe('runQwenServe permissionResponseTimeoutMs validation', () => { }); /** - * The budget is resolved at boot and reported; it does not size any child yet. + * The budget is resolved at boot and reported. Whether it also sizes a child + * depends on `childHeapMode`, which defaults to `observe` and sizes nothing. * The only boot-time behavior is rejecting an out-of-range flag value. */ describe('runQwenServe memory budget', () => { @@ -2039,7 +2040,13 @@ describe('runQwenServe memory budget', () => { const body = (await res.json()) as { limits: { memory: { - enforced: false; + enforced: boolean; + childHeap: { + mode: string; + maxConcurrentChildren: number; + perChildCeilingMb: number | null; + refusals: number; + } | null; configuredBudgetMb: number; effectiveBudgetMb: number; budgetSource: string; @@ -2059,18 +2066,74 @@ describe('runQwenServe memory budget', () => { registeredWorkspaces: number; activeAcpChildren: number; childRssCoverage: string; + children: { + rssBytes: number; + sampled: number; + oldestReadingAgeMs: number | null; + }; modeled: { recommendedShareAtRegisteredMb: number; recommendedShareAtActiveMb: number | null; }; + // Restated rather than imported on purpose: this shape is the + // wire contract, and casting to the internal type would make the + // assertions below accept whatever that type happens to say. + pressure: { + mode: string; + level: string; + source: string; + ratio: number; + rssBytes: number; + rssRatio: number; + availableBytes: number; + heapUsedBytes: number; + heapRatio: number; + heapLimitBytes: number; + }; }; }; }; const memory = body.limits.memory; expect(memory).not.toBeNull(); - // Nothing in this section is applied, and the wire says so. + // The child-heap policy reached status on a daemon that really booted. + // Default is `observe`, so it computed a share and applied nothing — + // `enforced` has to stay false or the field means "the feature exists" + // rather than "children are being sized by this". expect(memory?.enforced).toBe(false); + // Pin the key set rather than the values, so an unannounced field added + // to the wire still fails here. `toEqual` on the whole object was the + // other option and it does not survive this suite booting a real daemon: + // both derived figures follow the host's pool, and on a runner with + // under ~1 GB available the model correctly publishes no partition at + // all — so a matcher asserting `any(Number)` would fail on exactly the + // host where the code is doing the right thing. + expect(Object.keys(memory?.childHeap ?? {}).sort()).toEqual([ + 'maxConcurrentChildren', + 'mode', + 'perChildCeilingMb', + 'refusals', + ]); + expect(memory?.childHeap?.mode).toBe('observe'); + expect(memory?.childHeap?.refusals).toBe(0); + // Whichever branch this host took, the two figures agree with each + // other. The arithmetic itself is pinned exhaustively in + // `child-heap-policy.test.ts`; what this asserts is that a real daemon + // put a self-consistent pair on the wire. + if (memory?.childHeap?.perChildCeilingMb === null) { + expect(memory?.childHeap?.maxConcurrentChildren).toBe(0); + } else { + // A fixed grant handed to every admitted child must total no more than + // the pool it partitions. That product is the whole reason the + // partition is a bound rather than a per-spawn share. + expect(memory?.childHeap?.maxConcurrentChildren ?? 0).toBeGreaterThan( + 0, + ); + expect( + (memory?.childHeap?.maxConcurrentChildren ?? 0) * + (memory?.childHeap?.perChildCeilingMb ?? 0), + ).toBeLessThanOrEqual(memory?.modeled.childPoolMb ?? 0); + } expect(memory?.configuredBudgetMb).toBe(4096); expect(memory?.budgetSource).toBe('flag'); // The invariant that motivates separating configured from effective: @@ -2090,12 +2153,46 @@ describe('runQwenServe memory budget', () => { const runtimeMemory = body.runtime.memory; expect(runtimeMemory?.registeredWorkspaces).toBe(1); - // Sampling still covers only the primary child; say so rather than let - // the section imply process-tree observation. - expect(runtimeMemory?.childRssCoverage).toBe('primary_only'); + // Sampling now covers every live child; it still is not process-tree + // observation, which `children`'s own docs spell out. + expect(runtimeMemory?.childRssCoverage).toBe('active_children'); expect( runtimeMemory?.modeled.recommendedShareAtRegisteredMb, ).toBeGreaterThan(0); + + // Pressure, from a daemon that actually booted. Every other test for it + // calls the status builder directly, so nothing else would notice the + // reading failing to reach a live response. + const pressure = runtimeMemory?.pressure; + expect(pressure?.mode).toBe('observe'); + // A real process against a real denominator: assert the invariants + // rather than a level, which depends on the host running the test. + expect(pressure?.rssBytes).toBeGreaterThan(0); + expect(pressure?.heapLimitBytes).toBeGreaterThan(0); + expect(pressure?.availableBytes).toBe( + (memory?.availableMemoryMb ?? 0) * 1024 * 1024, + ); + expect(pressure?.ratio).toBe( + Math.max(pressure?.rssRatio ?? 0, pressure?.heapRatio ?? 0), + ); + expect(pressure?.source).not.toBe('unknown'); + + // Aggregate child RSS. This test opens no SSE/WS stream, so the + // sampler's watch gate never fires and nothing is polled — assert the + // invariants that hold regardless rather than a non-zero sum, which + // only a streaming client would produce. + const children = runtimeMemory?.children; + expect(children?.sampled).toBeLessThanOrEqual( + runtimeMemory?.activeAcpChildren ?? 0, + ); + // Nothing sampled must read as nothing summed and no age — never as a + // measured zero. + if (children?.sampled === 0) { + expect(children.rssBytes).toBe(0); + expect(children.oldestReadingAgeMs).toBeNull(); + } else { + expect(children?.rssBytes).toBeGreaterThan(0); + } } finally { await handle.close(); } diff --git a/packages/cli/src/serve/run-qwen-serve.ts b/packages/cli/src/serve/run-qwen-serve.ts index 275e58081c..ae3c849719 100644 --- a/packages/cli/src/serve/run-qwen-serve.ts +++ b/packages/cli/src/serve/run-qwen-serve.ts @@ -45,6 +45,10 @@ import { formatMemoryBudgetStderr, resolveDaemonMemoryBudget, } from '@qwen-code/acp-bridge/daemonMemoryBudget'; +import { + createChildHeapPolicy, + type ChildHeapPolicy, +} from '@qwen-code/acp-bridge/childHeapPolicy'; import { canonicalizeWorkspace, translateAndCheckAbsoluteWorkspacePath, @@ -1636,6 +1640,9 @@ function createBootstrapServeApp(input: { channelIdleTimeoutMs: channelIdleTimeoutMs(opts.channelIdleTimeoutMs), sessionIdleTimeoutMs: sessionIdleTimeoutMs(opts.sessionIdleTimeoutMs), acpConnectionCap: null, + // No child-heap policy during bootstrap: it is built with the + // runtime, so `enforced` is correctly false and `childHeap` null in + // this window even when the flag says `enforce`. memory: toDaemonStatusMemoryLimits(opts.daemonMemoryBudget), }, capabilities: { @@ -2860,6 +2867,9 @@ async function runQwenServeImpl( killAllSync(): void; } | undefined; + // Held for daemon status: `observe` mode's whole product is the would-be + // refusal count, which is useless unless it can be read back out. + let managedChildHeapPolicy: ChildHeapPolicy | undefined; const internalRuntimeBridgesForCleanup: AcpSessionBridge[] = []; let daemonEventLoopMonitor: | ReturnType @@ -3572,6 +3582,21 @@ async function runQwenServeImpl( workspaceTrustOperationGate.runExclusive('runtime-topology', operation); const processRegistry = new runtime.ProcessRegistry(); managedProcessRegistry = processRegistry; + // One policy for the whole daemon, beside the one registry it reads. Both + // must be shared: a per-factory registry would report a concurrent count + // of 1 on every spawn and hand each child the entire pool. + // Not built for an injected bridge: `deps.bridge` brings its own channel + // and never goes through the factory this policy rides on, so a policy + // here would size nothing while `limits.memory.enforced` claimed + // otherwise — a status field asserting enforcement that is not happening. + const childHeapPolicy: ChildHeapPolicy | undefined = + opts.daemonMemoryBudget && !deps.bridge + ? createChildHeapPolicy({ + budget: opts.daemonMemoryBudget, + mode: opts.childHeapMode ?? 'observe', + }) + : undefined; + managedChildHeapPolicy = childHeapPolicy; const fsFactory = runtime.resolveBridgeFsFactory({ // Secondary roots share a write-capable factory only after their own // folder trust check passes; untrusted secondary roots stay outside. @@ -3595,6 +3620,7 @@ async function runQwenServeImpl( }); const channelFactory = runtime.createSpawnChannelFactory({ processRegistry, + childHeapPolicy, sourceEnv: runtimeEffectiveEnv, onDiagnosticLine: diagnosticSink, pipeHooks: { @@ -4211,6 +4237,7 @@ async function runQwenServeImpl( }); const secondaryChannelFactory = runtime.createSpawnChannelFactory({ processRegistry, + childHeapPolicy, sourceEnv: secondaryEnv.effectiveEnv, onDiagnosticLine: diagnosticSink, pipeHooks: { @@ -4537,19 +4564,44 @@ async function runQwenServeImpl( primaryEntry.state === 'active' ? primaryEntry.current?.runtime.bridge : undefined; + // The ring's `childRssBytes` gauge stays the PRIMARY child's reading — + // its published meaning is "ACP child process RSS", singular. The + // aggregate across every workspace is reported separately, under + // `runtime.memory.children` in daemon status. const child = primaryRuntimeBridge?.getChildResourceSnapshot?.(); // Only poll the child's resources when someone is watching: the // staleness guard already drops the reading to 0 when idle, so gating // avoids a 5s RPC round-trip (pipe + child CPU) for a chart nobody has // open. if (runtime.getActiveSseCount() > 0 || (acp?.wsStreams ?? 0) > 0) { - void primaryRuntimeBridge?.refreshChildResource?.().catch((err) => { - daemonLog.warn( - `ACP child resource refresh failed: ${ - err instanceof Error ? err.message : String(err) - }`, - ); - }); + // Refresh EVERY managed workspace, not just the primary: the caches + // this warms are what `runtime.memory.children` sums, and a child + // nobody refreshed reads as unmeasured there. No `isChannelLive` + // filter is needed — `refreshChildResource` already no-ops without a + // live channel — and no concurrency limit is added, because the call + // is single-flight per bridge and the number of bridges is capped by + // MAX_DAEMON_WORKSPACES. + for (const managed of workspaceRegistry.listManaged()) { + // The shipped bridge's `refreshChildResource` never rejects: it + // catches the RPC failure itself, keeps the last good cache, and + // tees the reason to the serve debug log — which is why this + // handler has never fired and why the fan-out cannot turn it into + // 25 warnings a tick. It stays as a backstop rather than being + // deleted, because the method is an optional interface member and + // an `async` one, so any other implementation throwing before its + // own try block would surface here as an unhandled rejection and + // take the daemon down. + // + // Carrying the workspace matters for exactly that case: an + // unattributable warning repeated across a 25-workspace fan-out is + // the shape that is impossible to act on. + void managed.bridge.refreshChildResource?.().catch((err) => { + daemonLog.warn('ACP child resource refresh failed', { + workspaceId: managed.workspaceId, + error: err instanceof Error ? err.message : String(err), + }); + }); + } } metricsRing.sample(nowMs, { cpuPercent, @@ -4721,6 +4773,7 @@ async function runQwenServeImpl( : wsFsFactory; const wsChannelFactory = runtime.createSpawnChannelFactory({ processRegistry, + childHeapPolicy, sourceEnv: wsEnv.effectiveEnv, onDiagnosticLine: diagnosticSink, pipeHooks: { @@ -5497,6 +5550,7 @@ async function runQwenServeImpl( }), getMetricsSeries: () => metricsRing.snapshot(), getTotalSessionAdmissionSnapshot: totalSessionAdmission.snapshot, + getChildHeapPolicySnapshot: () => managedChildHeapPolicy?.snapshot(), recordDaemonRequest: (durationMs, statusCode) => metricsRing.recordRequest(durationMs, statusCode), workspace: workspaceService, diff --git a/packages/cli/src/serve/server.ts b/packages/cli/src/serve/server.ts index 8ea44491a7..c817a8461e 100644 --- a/packages/cli/src/serve/server.ts +++ b/packages/cli/src/serve/server.ts @@ -277,6 +277,7 @@ import { resolveLiveProviderCredential, type LiveProviderCredential, } from './live/provider-credentials.js'; +import type { ChildHeapPolicySnapshot } from '@qwen-code/acp-bridge/childHeapPolicy'; export { createDefaultFsAuditEmit, @@ -503,6 +504,7 @@ export interface ServeAppDeps { /** Rolling metrics series for the Daemon Status charts (oldest→newest). */ getMetricsSeries?: () => DaemonMetricsBucket[]; getTotalSessionAdmissionSnapshot?: () => TotalSessionAdmissionSnapshot; + getChildHeapPolicySnapshot?: () => ChildHeapPolicySnapshot | undefined; /** * Sink fed one (durationMs, statusCode) per matched daemon HTTP request, so * the metrics ring can bucket request rate and latency for the charts. @@ -1723,6 +1725,7 @@ export function createServeApp( getMetricsSeries: deps.getMetricsSeries, getTotalSessionAdmissionSnapshot: deps.getTotalSessionAdmissionSnapshot ?? totalSessionAdmission?.snapshot, + getChildHeapPolicySnapshot: deps.getChildHeapPolicySnapshot, }); if (liveVoiceEnabled) { diff --git a/packages/cli/src/serve/types.ts b/packages/cli/src/serve/types.ts index 031c825155..e2b16d410d 100644 --- a/packages/cli/src/serve/types.ts +++ b/packages/cli/src/serve/types.ts @@ -14,6 +14,11 @@ import { // are compiler-flagged here. import type { PermissionPolicy } from '@qwen-code/acp-bridge'; import type { DaemonMemoryBudget } from '@qwen-code/acp-bridge/daemonMemoryBudget'; +// Type-only, so it is erased before the serve fast-path bundle closure check +// ever sees it. Reused only for the child-heap knob: `memoryPressureMode` +// happens to share the same two values today but is an independent switch, and +// aliasing them would couple whichever one gains `enforce` first to the other. +import type { ChildHeapMode } from '@qwen-code/acp-bridge/childHeapPolicy'; import type { AuthType, InputModalities, @@ -227,10 +232,40 @@ export interface ServeOptions { /** * Total memory budget in MB for the whole daemon process tree — the root * plus every `qwen --acp` child it spawns. When unset, derived as half of - * the cgroup-constrained or host memory. Currently observed and reported - * only; it does not yet size any child. + * the cgroup-constrained or host memory. + * + * Observed and reported only. No child is sized from it and no spawn is + * refused on its basis: `childHeapMode: 'observe'` models a partition of it + * and publishes the model, but there is no mode that applies one. Sizing + * children arrives with the peak old-space measurement that can tell an + * operator beforehand whether their workload fits the partition. */ memoryBudgetMb?: number; + /** + * Whether the daemon derives and acts on a memory-pressure level. + * + * `observe` (default) reports the level alongside the raw figures and raises + * a status issue when it leaves `normal`. `off` still reports the figures — + * the point of this phase is to gather data, including from deployments that + * do not want the signal acting on them yet — but raises no issue, so the + * daemon's overall `status` rollup is unchanged. + * + * There is deliberately no `enforce`: nothing here remediates, and a value a + * caller can pass but never use is a dead switch. It arrives with the + * enforcement. + */ + memoryPressureMode?: 'off' | 'observe'; + /** + * Whether the daemon models a per-child heap partition of the budget. + * + * `observe` (default) computes the partition and counts the spawns it would + * have refused; nothing is applied. There is no `enforce` yet — applying it + * needs a way to tell an operator in advance whether their workload fits + * the ceiling, and `refusals` cannot answer that: it counts admission + * pressure, while children still run on the far larger host-derived + * ceiling. `off` models nothing. + */ + childHeapMode?: ChildHeapMode; /** * Resolved at boot by `runQwenServe`. Not an operator input, and not * consumed by any spawn path — it is reported under `limits.memory` on diff --git a/packages/sdk-typescript/src/daemon/types.ts b/packages/sdk-typescript/src/daemon/types.ts index d2ec380358..4880d16a6b 100644 --- a/packages/sdk-typescript/src/daemon/types.ts +++ b/packages/sdk-typescript/src/daemon/types.ts @@ -629,8 +629,32 @@ export interface DaemonStatusReport { * none. */ memory?: { - /** Always false: nothing in this section is applied to a process. */ + /** False, and required: nothing in this section is applied to a process. */ enforced: false; + /** + * The per-child heap partition the daemon models but does not apply. + * `null` when no policy was built; absent on daemons predating it. + */ + childHeap?: { + mode: 'off' | 'observe'; + /** + * `null` under `off`, which models nothing — distinct from `0`, + * a computed answer meaning the pool hosts no child. + */ + maxConcurrentChildren: number | null; + /** + * Never 0 and never below `modeled.minChildHeapMb`; `null` instead, + * under `off` and wherever no partition fits within that floor. + */ + perChildCeilingMb: number | null; + /** + * Admission pressure only. 0 does not mean the partition is safe to + * apply: children still run on the host-derived ceiling. A channel + * swap at full occupancy also books one, and on a host too small to + * model a partition this equals the total ACP spawn count. + */ + refusals: number; + } | null; configuredBudgetMb: number; effectiveBudgetMb: number; budgetSource: 'flag' | 'derived'; @@ -703,18 +727,68 @@ export interface DaemonStatusReport { */ activeAcpChildren: number; /** - * Which children the daemon's RSS sampling covers. Only the primary ACP - * child is sampled, and only while an SSE/WS watcher is active; when no - * client is observing, childRssBytes reads 0. After the last watcher - * detaches, the last sampled value persists until it ages out (~30s). + * Which children the daemon's RSS sampling covers, and only while an + * SSE/WS watcher is active; with no client observing, nothing is + * sampled. After the last watcher detaches, each reading persists until + * it ages out (~30s). + * + * A union, unlike the daemon's own type: `primary_only` is what daemons + * before the aggregate send, and this mirror describes every version. */ - childRssCoverage: 'primary_only'; + childRssCoverage: 'primary_only' | 'active_children'; + /** + * Aggregate RSS across the children `childRssCoverage` names. Both an + * over-count (summed per-process RSS double-counts shared pages) and a + * floor (each child reports only its own process, so its MCP descendants + * and all channel workers are missing). Not the daemon tree's memory. + * + * Optional because it is additive within an existing block: a daemon + * that shipped `runtime.memory` before it exists sends the block without + * it, and a daemon reporting `primary_only` never sends it at all. + */ + children?: { + /** A floor rather than a total whenever `sampled < activeAcpChildren`. */ + rssBytes: number; + /** Contributors. The denominator is the sibling `activeAcpChildren`. */ + sampled: number; + /** + * Age of the oldest reading in the sum. `null` when nothing was + * sampled, and also when every contributor predates the field — so + * `null` never means "fresh". + */ + oldestReadingAgeMs: number | null; + }; modeled: { /** `null` when no workspace is registered. */ recommendedShareAtRegisteredMb: number | null; /** `null` when no ACP child is active. */ recommendedShareAtActiveMb: number | null; }; + /** + * The daemon root's own memory pressure. Reported in both modes; only + * `observe` also raises a status issue from it, so `off` leaves the + * top-level `status` rollup unaffected. Root process only: these are + * this process's own figures, so children growing does not move them — + * compare against `children.rssBytes` for that. + * + * Optional because it is additive *within* an existing block: a daemon + * that shipped `runtime.memory` before this field exists and sends the + * block without it. Typing it as required would make this mirror lie + * about those daemons. + */ + pressure?: { + mode: 'off' | 'observe'; + level: 'normal' | 'soft' | 'hard' | 'critical'; + /** `unknown` means neither denominator was usable, not that all is well. */ + source: 'rss' | 'heap' | 'unknown'; + ratio: number; + rssBytes: number; + rssRatio: number; + availableBytes: number; + heapUsedBytes: number; + heapRatio: number; + heapLimitBytes: number; + }; }; /** Optional daemon-process performance counters. */ perf?: {