* fix(serve): allow bounded reads of large text files
* fix(serve): bound large-text reads by scan cost, not by which knob was set
Follow-up to the bounded large-text read path. Three changes:
Gate on any explicit window argument, not on `limit`. Gating on `limit`
had the cost model backwards in both directions: `{ line: 900_000_000,
limit: 20 }` was admitted despite walking the whole file, while
`{ maxBytes: 4096 }` — satisfiable from the first 4 KiB — was refused. A
read with no window argument at all still fails, since a caller that
believes it holds the whole file may write it back truncated.
Add MAX_TEXT_SCAN_BYTES (8 MiB). MAX_READ_BYTES caps what a read
returns; nothing capped what it cost. Line offsets are resolved by
scanning from byte 0, so a query param could turn into an
uninterruptible multi-second scan of an arbitrarily large file — and on
Windows hold a read handle for that span, blocking renames and deletes.
Past the budget the read is refused with `file_too_large` pointing at
readBytes, which reaches any offset in O(1).
Tolerate appends on streamed windows. Requiring whole-file size/mtime
stability after reading a prefix rejected reads whose returned bytes
were still valid, and the case it rejected — tailing a live log — is the
one this path exists for. Streamed windows now assert inode identity
plus "did not shrink"; truncation and replacement are still rejected.
Also: non-UTF-8 large text now returns `binary_file` rather than
`file_too_large`, so a client retrying on 413 with a smaller window
can't loop forever; and `readFileWithLineAndLimit` throws instead of
silently ignoring a caller-supplied `fileHandle` on the by-path
fallback.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(core): thread the descriptor instead of forking text-read helpers
PR #7947 pinned large-text reads to one inode by threading a caller-owned
FileHandle into readTextRange as an optional field, plus a second field,
forceStreaming, to suppress the buffering fast path. Two optional fields
produced four combinations: one meaningful, one used by a single test, one
unreachable, and — in readFileWithLineAndLimit — one that silently fell
through to a by-path read, defeating the reason the caller opened a handle.
Unify the two encoding detectors. detectFileEncoding now takes a path or a
borrowed handle, so detectFileHandleEncoding is deleted along with the
message discrepancy between them: an encoding iconv-lite cannot load now
raises LargeNonUtf8TextError naming that encoding rather than deferring to
the decoder's generic invalid-utf8 variant. Both still refuse the file, and
the Serve boundary maps both to binary_file.
Split the reader into readTextRange (path) and readTextRangeFromHandle
(always streams, both byte bounds required). The unreachable combination and
its untested readFileHandleBuffer are gone, and with no fileHandle parameter
left for readFileWithLineAndLimit to ignore, the RangeError guarding that
fallthrough is deleted too — the trap can no longer be expressed.
CoreReadTextFileHandleRequest drops its required stats field. Nothing
downstream read it, and because the ACP request type it extends permits
extra properties, TypeScript accepted the dead argument silently.
readFileHandleChunks becomes chunksFromHandle(fh, from) — the one seam
byte-cursor text paging needs.
No observable change at the Serve boundary: its 222 tests pass unmodified.
Two fileSystemService tests were deleted rather than repaired; they asserted
the arguments readFileWithLineAndLimit received, which is nothing once the
handle path stops calling it. Their coverage lives in read-text-range.test.ts
against real files and in workspace-file-system.test.ts at the real boundary.
258 production lines in core, net -71 overall.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(core): make CoreReadTextFileHandleRequest standalone
Self-audit follow-up to f55c867a. Two fields survived the reshape that the
handle path never reads:
- `stats` was documented as required ("must pass the Stats captured from that
handle") and nothing downstream read it. The handle path always streams, so
it never needs a size to choose a strategy, and the encoding probe does its
own fstat.
- `path` became dead once readTextRangeFromHandle replaced the path-plus-handle
call. Errors are labelled with the path by the Serve boundary that owns it.
Neither was caught by the compiler: the ACP ReadTextFileRequest the type
derived from permits extra properties, so the CLI kept passing both silently.
That is the argument for declaring the type standalone rather than Omit-ing
four of six inherited fields and quietly re-admitting the rest.
Also record the second behaviour delta of the detector merge in the design
doc: detectFileEncoding catches I/O errors and falls back to 'utf-8', where
detectFileHandleEncoding let them propagate. The failure is not lost — a handle
that fails the 8 KiB probe fails the streaming read immediately after — but a
different call now reports it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(serve): page large text files by byte cursor
Line offsets address a byte stream, so `readText` resolves them by scanning
from byte 0. Paging a large log that way is O(n^2) across pages, and past
MAX_TEXT_SCAN_BYTES (8 MiB) a deep page is refused outright — agents had no
O(1) path short of dropping to GET /file/bytes and splitting lines themselves,
losing encoding handling, multibyte safety, and the binary_file refusal.
A response that leaves content behind now returns `hasMore`, and where a file
byte offset is derivable, an opaque `nextCursor`. Passing it back as `cursor`
resumes in O(1). Page 1 is an ordinary `limit` read, so clients never compute
byte offsets themselves, and a paging loop does not break when a file happens
to be small.
The cursor is unsigned base64url JSON carrying {off, size, dev, ino}, matching
encodeOrganizedCursor rather than the HMAC-signed transcript codec: the path is
re-resolved through the workspace boundary on every request, so a forged cursor
can only move the offset within a file the caller may already read — what
GET /file/bytes?offset= allows today. What the payload is for is staleness:
a replaced or truncated file yields hash_mismatch instead of bytes from the
wrong place, while an append leaves an outstanding cursor valid — the case the
feature exists for.
Every minted cursor points at the start of a line. When a single line exceeds
maxOutputBytes the reader emits a truncated prefix and skips to the next line
rather than resuming mid-line, because a mid-line cursor makes the following
page snap forward and silently drop the rest of that line at the seam. Windows
cut mid-line by a byte cap therefore report hasMore with no cursor, as do
non-UTF-8 snapshot reads whose decoded text is a UTF-8 re-encoding with no
mapping back to file offsets. That is why hasMore is a field rather than a
restatement of nextCursor.
Cursor reads branch before the size check, not by widening the window gate:
a cursor read of a file under MAX_READ_BYTES would otherwise land on the
snapshot path, which knows only line/limit, and silently return line 0.
Adds the workspace_file_read_cursor capability, per the convention that new
behavior gets a new tag, and retargets the scan-budget hint at cursor paging.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(core): advance UTF-8 cursors after truncation
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
* docs(serve): clarify cursor bootstrap limits
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
* fix(sdk): raise daemon browser bundle budget
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
* test(serve): cover ACP cursor dispatch and cursor binary_file mapping (#8002)
* fix(core): only set sawCrlf for emitted lines in cursor paging (#8002)
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: Qwen Code Bot <qwen-code-bot@users.noreply.github.com>
22 KiB
Workspace File System Boundary
Overview
The daemon never lets HTTP routes or ACP-side agent calls touch the host filesystem directly. Every read, write, list, glob, and stat goes through the WorkspaceFileSystem boundary (packages/cli/src/serve/fs/), which provides:
- Path resolution — canonicalize paths and reject anything escaping the bound workspace, including via symlinks.
- Trust gating — refuse writes when the workspace is not trusted (
untrusted_workspace). - Size & content policy — full-snapshot/output cap (
MAX_READ_BYTES = 256 KiB), large-text windows bounded in both output and scan cost (MAX_TEXT_SCAN_BYTES = 8 MiB), write cap (MAX_WRITE_BYTES = 5 MiB), binary detection. - Atomicity — write-then-rename with target mode preservation and
0o600default for new files. - Audit — every access / denial emits a structured event for
PermissionAuditRing/ monitoring. - Typed errors — closed
FsErrorKindunion mapped to HTTP statuses.
The HTTP file routes (GET /file, GET /file/bytes, POST /file/write, POST /file/edit, GET /list, GET /glob, GET /stat) and the ACP-side BridgeFileSystem adapter (so agent-driven readTextFile / writeTextFile calls get the same gates) both go through this boundary.
Responsibilities
- Resolve user-supplied paths into branded
ResolvedPathvalues that the rest of the boundary can safely use. - Refuse paths outside the bound workspace (
path_outside_workspace) and paths whose target is a symlink (symlink_escape). - Refuse full-snapshot reads above
MAX_READ_BYTES, while allowing explicit windows with output capped atMAX_READ_BYTESand scan cost capped atMAX_TEXT_SCAN_BYTES; refuse writes aboveMAX_WRITE_BYTESand binary files (binary_file). - Refuse writes/edits when the workspace is untrusted (
untrusted_workspace) — gated byassertTrustedForIntent(trusted, intent). - Honor
.gitignore/.qwenignorepatterns viashouldIgnore. - Perform atomic write-then-rename with target mode preservation; default new file mode is
0o600. - Emit
fs.access/fs.deniedaudit events on every operation. - Map every failure to a
FsErrorwith kind and HTTP status; route handlers serialize them uniformly.
Architecture
Module layout
| File | Purpose |
|---|---|
paths.ts |
canonicalizeWorkspace, resolveWithinWorkspace, hasSuspiciousPathPattern, branded ResolvedPath, Intent union (read | write | list | stat | glob). |
policy.ts |
MAX_READ_BYTES, MAX_TEXT_SCAN_BYTES, MAX_WRITE_BYTES, BINARY_PROBE_BYTES, assertTrustedForIntent, detectBinary, enforceReadBytesSize, enforceReadSize, enforceWriteSize, shouldIgnore. |
audit.ts |
FS_ACCESS_EVENT_TYPE, FS_DENIED_EVENT_TYPE, createAuditPublisher, audit payload types. |
errors.ts |
FsError class, isFsError, FsErrorKind union (14 kinds), FsErrorStatus union (400 / 403 / 404 / 409 / 413 / 422 / 500 / 503). |
workspace-file-system.ts |
createWorkspaceFileSystemFactory, WorkspaceFileSystem (the orchestrator that reads/writes/lists), WriteMode, ContentHash, FsEntry, FsStat, ListOptions, GlobOptions, ReadTextOptions, ReadBytesOptions, WriteTextAtomicOptions. |
FsErrorKind taxonomy
| Kind | Default HTTP | Meaning |
|---|---|---|
path_outside_workspace |
400 | Resolved path is outside the bound workspace. |
symlink_escape |
400 | Target is a symlink (rejected per the conservative PR 18 + PR 20 posture). |
path_not_found |
404 | ENOENT. |
binary_file |
422 | Content sniffed binary on a text route, or large text in an encoding the text route cannot decode. |
file_too_large |
413 | Windowless/full-snapshot text above MAX_READ_BYTES, a line offset beyond MAX_TEXT_SCAN_BYTES, or a write above MAX_WRITE_BYTES. |
hash_mismatch |
409 | Optimistic-concurrency expectedSha256 failed, or the file changed during a stable read. |
file_already_exists |
409 | mode: 'create' against an existing file. |
text_not_found |
422 | POST /file/edit's search string wasn't in the file. |
ambiguous_text_match |
422 | Multiple matches when exactly one was required. |
untrusted_workspace |
403 | Write attempted in an untrusted workspace. |
permission_denied |
403 | OS-level EACCES / EPERM. |
io_error |
503 | ENOSPC / EIO / EBUSY / ETXTBSY / ENAMETOOLONG / EMFILE / ENFILE. Distinct from permission_denied so monitoring pipelines do not page security responders for "disk full". |
internal_error |
500 | Non-errno error that reaches the boundary (TypeError, programmer bug). |
parse_error |
400 / 422 | Request-body parse error (400) or service-level invariant breach (422). |
BridgeFileSystem (the ACP-side adapter)
packages/acp-bridge/src/bridgeFileSystem.ts defines:
interface BridgeFileSystem {
readText(params: ReadTextFileRequest): Promise<ReadTextFileResponse>;
writeText(params: WriteTextFileRequest): Promise<WriteTextFileResponse>;
}
This is the injection point for ACP readTextFile / writeTextFile. Bridge tests and Mode A embedded callers can omit it on BridgeOptions; BridgeClient falls back to its inline fs.readFile / fs.writeFile proxy (preserves pre-F1 behavior). Production qwen serve wires BridgeFileSystem through createBridgeFileSystemAdapter(fsFactory) (packages/cli/src/serve/bridge-file-system-adapter.ts) so agent-side ACP writes pick up the same TOCTOU, symlink, trust-gate, and audit gates the HTTP routes use.
Two defensive properties the adapter MUST preserve (because the inline proxy is fully bypassed when the adapter is injected):
- Reject non-regular files — sockets / pipes / char devices / procfs / sysfs entries can stream unbounded data despite
stats.size === 0. The inline path throws withdescribeStatKind(stats)in the message. - Avoid unbounded full-file buffering. The inline fallback caps a buffered read at
READ_FILE_SIZE_CAP = 100 MiB. The injected adapter instead applies the stricter WorkspaceFileSystem contract: full snapshots stop at 256 KiB, while larger UTF-8 files require a finitelimitand are streamed from an inode-bound handle with at most 256 KiB returned. It must not read an entire 500 MB log merely to return{ line: 1, limit: 10 }.
The adapter goes further: it uses WorkspaceFileSystem.writeTextOverwrite (PR 18 primitive) for atomic temporary-file-and-rename writes with mode preservation, 0o600 default, and symlink rejection inside a per-path lock. This is a divergence from the pre-F1 inline proxy which resolved symlinks and wrote through to their target — agents that relied on writing through symlinked dotfiles now have to address the resolved path directly.
FsError preservation over the ACP wire
When the BridgeFileSystem adapter throws an FsError (kind: 'untrusted_workspace' / 'symlink_escape' / 'file_too_large' / etc.), the ACP SDK's default RPC error path serializes only error.message as a generic -32603 "Internal error" — kind / status / hint are stripped. The downstream agent RPC client would then have to regex-match the human-readable message to dispatch typed UI (auth retry vs file picker vs proxy hint).
BridgeClient.writeTextFile and BridgeClient.readTextFile install a thin guard (packages/acp-bridge/src/bridgeClient.ts) that catches FsError-shaped throws and rethrows them as ACP RequestError:
function isFsErrorShape(err: unknown): err is FsErrorShape {
return (
err instanceof Error &&
err.name === 'FsError' &&
typeof (err as { kind?: unknown }).kind === 'string'
);
}
function preserveFsErrorOverAcp(err: unknown): never {
if (isFsErrorShape(err)) {
throw new RequestError(-32603, err.message, {
errorKind: err.kind,
...(err.hint !== undefined ? { hint: err.hint } : {}),
...(err.status !== undefined ? { status: err.status } : {}),
});
}
throw err;
}
The agent's RPC client now receives data.errorKind (the closed FsErrorKind value) plus the optional data.hint and data.status, so SDK consumers branch on the typed enum instead of regex-matching the message.
Two design notes:
- Duck typing over import —
FsErrorlives inpackages/cli/src/serve/fs/errors.tswhileBridgeClientlives inpackages/acp-bridge. A directimport { FsError }would invert the dependency. The duck check (name === 'FsError'+kind: string) mirrors whatmapDomainErrorToErrorKind(status.ts) already does forTrustGateError/SkillErrorfor the same cross-package bundling reason. - JSON-RPC code stays at -32603 — the bridge cannot reliably map
FsError.kindto a JSON-RPC error code shape, so the structureddatafield carries the semantic information for SDK consumers. The wire status code (-32603"internal error") is unchanged; clients route ondata.errorKind.
Trust gate
assertTrustedForIntent(trusted, intent) consumes the trust boolean injected by
the caller; the policy layer does not read Config.isTrustedFolder() directly.
Read / list / stat / glob are always allowed (trust is only for writes). Write
intents in untrusted workspaces throw
FsError('untrusted_workspace', ..., status: 403). The trust signal flows in
via WorkspaceFileSystemFactoryDeps.trusted: boolean — runQwenServe passes
true because the operator booted the daemon against a workspace they
implicitly trust; createServeApp (direct embed without runQwenServe)
defaults to false and warns once per process (see
02-serve-runtime.md).
Workflow
Read
sequenceDiagram
autonumber
participant R as HTTP route OR BridgeFileSystem.readText
participant FS as WorkspaceFileSystem
participant POL as policy.ts
participant FSP as node:fs
R->>FS: readText(ctx, path, opts)
FS->>FS: resolveWithinWorkspace(path) → ResolvedPath OR throw
FS->>FSP: stat(path)
FSP-->>FS: stats
FS->>FS: reject if not regular file (describeStatKind)
alt cursor supplied
FS->>FSP: open stable FileHandle
FS->>FS: validate cursor {dev,ino,size}; seek to the byte offset
FS->>FS: return whole lines; emit the next cursor
else file <= 256 KiB
FS->>FSP: open + read stable full snapshot
FSP-->>FS: buffer
FS->>FS: hash full snapshot; apply line/output limits
else file > 256 KiB AND an explicit window arg
FS->>FSP: open stable FileHandle
FS->>FS: stream requested lines from the same inode
FS->>FS: cap output at 256 KiB and scan at 8 MiB; omit full-file hash
else windowless large read
FS-->>R: file_too_large
end
FS->>POL: detectBinary(sample)
POL-->>FS: isBinary?
FS->>FS: reject if binary
FS->>FS: shouldIgnore? → annotate meta.matchedIgnore
FS->>FS: audit fs.access
FS-->>R: { content, optional sha256, truncated?, meta }
readText does not skip or reject reads because of ignore rules. It reads the
file normally and records the matching ignore classification in
meta.matchedIgnore. list and glob filter ignored results only when
includeIgnored is not enabled.
Write
sequenceDiagram
autonumber
participant R as POST /file/write OR ACP writeText
participant FS as WorkspaceFileSystem
participant POL as policy.ts
participant FSP as node:fs
R->>FS: writeTextAtomic(ctx, path, content, opts)
FS->>FS: assertTrustedForIntent(trusted, 'write') → throw untrusted_workspace OR ok
FS->>FS: resolveWithinWorkspace(path)
FS->>POL: enforceWriteSize(content) → throw file_too_large OR ok
FS->>FSP: lstat(path) → reject symlink
FS->>FS: acquire per-path lock
FS->>FSP: stat(existing?) → capture target mode (default 0o600)
FS->>FSP: writeFile(tmpPath, content, {mode})
FS->>FSP: rename(tmpPath, path) (atomic)
FS->>FS: audit fs.access (write)
FS-->>R: { sha256, mode, bytesWritten }
The atomic write-then-rename ensures a SIGKILL / OOM mid-write does NOT leave the target truncated. mode: 'create' aborts with file_already_exists on lstat; mode: 'overwrite' proceeds; expectedSha256 arms optimistic-concurrency (hash_mismatch on mismatch).
POST /file/edit (single text replacement)
Adds two failure modes on top of write:
text_not_found(422) — search string not in the file.ambiguous_text_match(422) — multiple matches when exactly one was required (the route's contract).
Audit fan-out
flowchart LR
A["WorkspaceFileSystem op succeeds OR fails"] --> P["createAuditPublisher → emit FS_ACCESS_EVENT_TYPE / FS_DENIED_EVENT_TYPE"]
P --> AR["PermissionAuditRing (512 entries, FIFO)"]
P --> MON["future: external monitoring sink"]
FS_ACCESS_EVENT_TYPE / FS_DENIED_EVENT_TYPE carry context (ctx), path, intent, outcome, errorKind?, bytesRead/written, sha256?.
State & Lifecycle
- The factory is built once at daemon boot (
runQwenServe→resolveBridgeFsFactory→ adapter). - Each request constructs a
RequestContextand invokes the factory's orchestrator for that call only — no long-lived per-file state. - Per-path locks live only for the duration of the write operation (no cross-call locking; concurrent writes to the same path race on the lock and serialize).
- Audit ring is owned by
runQwenServeand shared with the permission audit publisher.
Dependencies
@qwen-code/qwen-code-core—Ignore,isBinaryFile,Config.isTrustedFolder().node:fs,node:path,node:crypto.@qwen-code/acp-bridge—BridgeFileSystemcontract on the ACP side.- HTTP routes:
packages/cli/src/serve/routes/workspace-file-read.ts,workspace-file-write.ts.
Configuration
| Source | Knob | Effect |
|---|---|---|
WorkspaceFileSystemFactoryDeps.trusted: boolean |
Constructor input | Whether writes are allowed; defaults to true from runQwenServe, false from createServeApp (with warning). |
| Constant | MAX_READ_BYTES = 256 KiB |
Full-snapshot and returned-text cap; larger text requires an explicit window argument. |
| Constant | MAX_TEXT_SCAN_BYTES = 8 MiB |
Bytes a large-text read may scan to locate a line offset; past it, file_too_large. |
| Constant | MAX_WRITE_BYTES = 5 MiB |
Write cap; sized below express.json({ limit: '10mb' }). |
| Constant | BINARY_PROBE_BYTES = 4096 |
Sample size for content-based binary detection. |
| Capability tags | workspace_file_read, workspace_file_bytes, workspace_file_write |
See 11-capabilities-versioning.md. |
| Workspace files | .gitignore, .qwenignore |
Ignored paths surface as ignored: true from shouldIgnore. |
Caveats & Known Limits
- Symlinks are rejected, not followed. This is a divergence from the pre-F1 inline
BridgeClient.writeTextFileproxy which resolved symlinks. Agents writing through symlinked dotfiles need to address the resolved path directly. io_errorvspermission_deniedare distinct. Do not conflate them. Monitoring pipelines key onerrorKindfor alerting — folding ENOSPC into permission_denied would page security responders fordf -hproblems.- New file mode defaults to
0o600, not umask defaults. The write syscall'smodearg bypasses umask. Agents writing public files should explicitly pass a mode override. createServeAppdefaulttrusted: falsesilently rejects ACP writes withuntrusted_workspacefor embedders that do not inject a customfsFactoryorbridge. A one-time stderr warning fires the first time; further callers see no reminder. See02-serve-runtime.md.- Large text requires an explicit window argument, any of
line/limit/maxBytes. A read with none of them staysfile_too_large, because a caller that believes it holds the whole file may write it back truncated. Windows stream from an inode-bound handle and never return more thanMAX_READ_BYTES. MAX_READ_BYTEScaps what a read returns;MAX_TEXT_SCAN_BYTEScaps what it costs. Line offsets are resolved by scanning from byte 0, so{ line: 900_000_000, limit: 20 }returns almost nothing and still walks the file. Past 8 MiB of scanning the read is refused withfile_too_largepointing atreadBytes, which reaches any offset in O(1).- Streamed windows tolerate appends, not truncation. The full-snapshot path can demand byte-for-byte stability because it returns the whole file; a prefix window cannot, or every read of a live log fails. The streamed path asserts inode identity plus "did not shrink", so appends pass and truncation / replacement are still rejected.
sizeBytesreports the size atopen, describing the snapshot the window was cut from. - Large partial reads omit the full-file hash.
originalLineCountis omitted when streaming stops before EOF. - Paging is by byte cursor, not by line. A read that leaves content behind returns
hasMoreand, where a byte offset is derivable, an opaquenextCursor. Resuming from it is O(1); resuming bylinere-scans from byte 0 and is refused pastMAX_TEXT_SCAN_BYTES. The cursor carries{dev, ino, size}, so a replaced or truncated file yieldshash_mismatchrather than bytes from the wrong place, while an append leaves it valid. Non-UTF-8 snapshot reads reporthasMorebut no cursor — their decoded text is a UTF-8 re-encoding whose lengths do not map back to file offsets. BridgeFileSystemadapter MUST replicate both inline-proxy gates (non-regular-file refusal + bounded buffering/streaming). The inline path is fully bypassed when the adapter is injected.
References
packages/cli/src/serve/fs/index.ts(barrel)packages/cli/src/serve/fs/paths.tspackages/cli/src/serve/fs/policy.tspackages/cli/src/serve/fs/errors.tspackages/cli/src/serve/fs/audit.tspackages/cli/src/serve/fs/workspace-file-system.tspackages/cli/src/serve/bridge-file-system-adapter.tspackages/acp-bridge/src/bridgeFileSystem.ts- HTTP route reference:
../qwen-serve-protocol.md.