fix(dsh): cap zstd decode, coerce usage fields, scope the snap read

Security audit follow-ups on the DeepSeek Harness provider.

- **Decompression bomb.** Every zstd frame was decoded with no output bound, so
  a 16 KB crafted log expanded to ~916 MB of RSS (a 65 KB one declares 2 GB).
  Each frame now decodes under a 64 MB per-call cap, and the caps chain into a
  running per-file budget of MAX_SESSION_FILE_BYTES: a frame is given only the
  bytes the file has left, so node throws ERR_BUFFER_TOO_LARGE without
  allocating past the cap. The throw propagates out of the existing skip path,
  which discards the whole file rather than counting the frames read before the
  bomb, so a crafted tail cannot poison a partial total. The discovery header
  read takes the same per-frame cap. Measured on a 65 KB / 2 GB bomb: 916 MB
  -> 67 MB peak, zero calls emitted, one notice.
  Lines are still materialized eagerly; the byte budget bounds that, and making
  the read lazy would change readEventLines' contract for no further bound.
- **Usage type confusion.** Token fields were read with `?? 0` and never
  type-checked, so a string or array inputTokens flowed into the global totals
  and the persisted cache, where `0 + [1, 2]` becomes "01,2". They now go
  through numberOrZero (copilot.ts semantics: finite, positive, else 0).
  All-zero calls are still skipped.
- **Snap over-scope.** The personal-files read entry is `$HOME/.dsh/sessions`
  rather than all of `$HOME/.dsh`; the provider reads nothing else.
- **Third-party notice.** scanZstdFrames is transcribed from
  @deepseek-ai/dsh-session-persistence-jsonl. The published npm package is
  BSD-3-Clause (Copyright (c) 2026, DeepSeek) while the monorepo source
  declares MIT for the same package; THIRD_PARTY_NOTICES.md reproduces the
  stricter of the two and ships via package.json `files`.
This commit is contained in:
iamtoruk 2026-08-17 17:28:41 -07:00
parent ffd9213126
commit 09965f93ae
5 changed files with 168 additions and 15 deletions

51
THIRD_PARTY_NOTICES.md Normal file
View file

@ -0,0 +1,51 @@
# Third-party notices
CodeBurn is MIT licensed (see `LICENSE`). It also contains code derived from the
projects below, which carry their own terms. Each notice is reproduced here as
those terms require.
---
## @deepseek-ai/dsh-session-persistence-jsonl
`scanZstdFrames` in `src/providers/dsh.ts` is a transcription of the function of
the same name in this package (`src/zstd.ts`), which is what lets CodeBurn read
a DeepSeek Harness session log without depending on the harness itself. No other
part of the package is used.
Upstream declares two different licenses for this package: the published npm
package (0.0.1-rc.1) ships a BSD 3-Clause `LICENSE` and declares
`"license": "BSD-3-Clause"`, while the monorepo source it is built from
(`deepseek-ai/deepseek-harness`, `packages/session/session-persistence-jsonl`)
declares MIT. The stricter of the two is reproduced below.
```
BSD 3-Clause License
Copyright (c) 2026, DeepSeek
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
```

View file

@ -169,7 +169,7 @@
"$HOME/.copilot",
"$HOME/.cursor",
"$HOME/.deepseek",
"$HOME/.dsh",
"$HOME/.dsh/sessions",
"$HOME/.factory",
"$HOME/.forge",
"$HOME/.gemini",

View file

@ -9,6 +9,7 @@
},
"files": [
"dist",
"THIRD_PARTY_NOTICES.md",
"!dist/parse-worker.js.map"
],
"scripts": {

View file

@ -15,17 +15,25 @@ import type { ProbeRoot, Provider, SessionSource, SessionParser, ParsedProviderC
// so node:zlib's one-shot zstdDecompressSync (which decodes a single frame)
// must be driven frame-by-frame behind a structural frame-boundary scan. The
// scan below is a port of scanZstdFrames from the official
// @deepseek-ai/dsh-session-persistence-jsonl package.
// @deepseek-ai/dsh-session-persistence-jsonl package, which is third-party code
// under its own license - see THIRD_PARTY_NOTICES.md.
// zstd landed in node:zlib in 22.15 / 23.8; the package floor is lower, so the
// provider degrades with a notice instead of assuming the export exists.
const zstdDecompress = (zlib as { zstdDecompressSync?: (buf: Buffer) => Buffer }).zstdDecompressSync
const zstdDecompress = (zlib as { zstdDecompressSync?: (buf: Buffer, opts?: { maxOutputLength?: number }) => Buffer }).zstdDecompressSync
const ZSTD_MAGIC = 0xfd2fb528
// SESSION_FORMAT_VERSION in @deepseek-ai/dsh-session. DSH refuses to load a log
// stamped with any other version, and a bump means an event's meaning changed,
// so a foreign version is skipped rather than read with today's assumptions.
// A zstd frame's declared content size is attacker-controlled, so a few KB of
// crafted input can expand to gigabytes. Every decode is capped: no single
// frame may exceed this, and no file may decode to more than it would have been
// allowed to occupy uncompressed (MAX_SESSION_FILE_BYTES). Overflow throws, and
// the caller skips the WHOLE file rather than counting the frames it got to.
const MAX_FRAME_DECODED_BYTES = 64 * 1024 * 1024
const SESSION_FORMAT_VERSION = 0
const MIN_REASONABLE_TIMESTAMP_MS = 1_000_000_000_000
@ -164,6 +172,13 @@ function mapToolName(raw: string): string {
return toolNameMap[raw] ?? raw
}
// Usage fields are whatever the JSON held. A string or array would flow
// straight into the global token totals and the persisted cache, where
// `0 + [1, 2]` silently becomes "01,2". Same semantics as copilot.ts.
function numberOrZero(raw: unknown): number {
return typeof raw === 'number' && Number.isFinite(raw) && raw > 0 ? raw : 0
}
// A log stamped with a version this parser was not written against is skipped
// whole: a bump means an event's meaning changed, so reading it with today's
// assumptions would report confident wrong numbers.
@ -198,12 +213,24 @@ function projectFromCwd(cwd: string, fallback: string): string {
}
// Decode every complete frame and yield its JSONL lines. A torn final frame is
// ignored; a structurally corrupt file throws for the caller to report.
function* readZstdLines(buffer: Buffer, maxFrames = Number.POSITIVE_INFINITY): Generator<string> {
// ignored; a structurally corrupt file, or one that decodes past `budget`,
// throws for the caller to report. Exported for the decode-budget test.
export function* readZstdLines(
buffer: Buffer,
maxFrames = Number.POSITIVE_INFINITY,
budget = MAX_SESSION_FILE_BYTES,
): Generator<string> {
const { frames } = scanZstdFrames(buffer, maxFrames)
let remaining = budget
for (const frame of frames) {
const text = zstdDecompress!(buffer.subarray(frame.start, frame.end)).toString('utf-8')
for (const line of text.split('\n')) {
if (remaining <= 0) throw new Error(`decodes past the ${budget}-byte cap`)
// node throws ERR_BUFFER_TOO_LARGE without allocating past the cap, so the
// per-frame limit doubles as the running budget for the frames after it.
const decoded = zstdDecompress!(buffer.subarray(frame.start, frame.end), {
maxOutputLength: Math.min(remaining, MAX_FRAME_DECODED_BYTES),
})
remaining -= decoded.length
for (const line of decoded.toString('utf-8').split('\n')) {
if (line.trim()) yield line
}
}
@ -276,7 +303,9 @@ async function readSessionHeader(filePath: string): Promise<DshEvent | null> {
return null
}
}
const text = zstdDecompress(head.subarray(frames[0]!.start, frames[0]!.end)).toString('utf-8')
const text = zstdDecompress(head.subarray(frames[0]!.start, frames[0]!.end), {
maxOutputLength: MAX_FRAME_DECODED_BYTES,
}).toString('utf-8')
return text.split('\n').find(l => l.trim()) ?? null
}
const content = await readSessionFile(filePath)
@ -486,11 +515,11 @@ function createParser(source: SessionSource, seenKeys: Set<string>): SessionPars
for (const key of sortedKeys) {
const bucket = buckets.get(key)!
const input = bucket.usage.inputTokens ?? 0
const output = bucket.usage.outputTokens ?? 0
const cacheRead = bucket.usage.cacheReadTokens ?? 0
const cacheWrite = bucket.usage.cacheWriteTokens ?? 0
const reasoning = bucket.usage.reasoningTokens ?? 0
const input = numberOrZero(bucket.usage.inputTokens)
const output = numberOrZero(bucket.usage.outputTokens)
const cacheRead = numberOrZero(bucket.usage.cacheReadTokens)
const cacheWrite = numberOrZero(bucket.usage.cacheWriteTokens)
const reasoning = numberOrZero(bucket.usage.reasoningTokens)
if (input + output + cacheRead + cacheWrite + reasoning === 0) continue
const dedupKey = `dsh:${sessionId || source.path}:${key}`

View file

@ -1,10 +1,10 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { mkdtemp, mkdir, writeFile, readFile, rm } from 'fs/promises'
import { mkdtemp, mkdir, writeFile, readFile, rm, stat } from 'fs/promises'
import { join } from 'path'
import { homedir, tmpdir } from 'os'
import zlib from 'zlib'
import { createDshProvider } from '../../src/providers/dsh.js'
import { createDshProvider, readZstdLines } from '../../src/providers/dsh.js'
import { calculateCost } from '../../src/models.js'
import type { ParsedProviderCall } from '../../src/providers/types.js'
@ -546,3 +546,75 @@ describe('dsh provider - real log, real container', () => {
expect(framed).toHaveLength(2)
})
})
describe('dsh provider - hostile input', () => {
itZstd('skips a session whose frames decompress to far more than the file cap', async () => {
const dir = join(tmpDir, 'sessions', '--home-u-proj--', 'session-bomb')
await mkdir(dir, { recursive: true })
const filePath = join(dir, 'session.jsonl.zstd')
// 200 MB of zeros compresses to a few KB. Uncapped this decoded to ~916 MB
// of RSS for a 16 KB file; the per-frame cap now rejects it without
// allocating past the cap.
const bomb = zstdCompress!(Buffer.alloc(200 * 1024 * 1024))
const good = zstdCompress!(Buffer.from(
sessionHeader({ id: 'session-bomb', cwd: '/home/u/proj' }) + '\n'
+ chunkUsage(1, 1, { inputTokens: 100, outputTokens: 10 }, 1786707340000) + '\n',
'utf-8',
))
await writeFile(filePath, Buffer.concat([good, bomb]))
expect((await stat(filePath)).size).toBeLessThan(64 * 1024)
// The whole file is skipped: the frames read before the bomb are not
// counted, so a crafted tail cannot poison a partial total.
expect(await parseAll(createDshProvider(tmpDir), filePath)).toEqual([])
})
itZstd('stops decoding once the frames exceed the running budget', async () => {
const frame = zstdCompress!(Buffer.from('{"type":"turn/start","seq":0,"time":1,"data":{"turn":1}}\n', 'utf-8'))
const buffer = Buffer.concat([frame, frame, frame])
expect([...readZstdLines(buffer, Number.POSITIVE_INFINITY, 4096)]).toHaveLength(3)
// A budget under two frames' plaintext stops at the frame that overruns it.
expect(() => [...readZstdLines(buffer, Number.POSITIVE_INFINITY, 60)]).toThrow()
})
it('coerces non-numeric usage fields instead of poisoning the totals', async () => {
const filePath = await writePlainSession('--home-u-proj--', 'session-poison', [
sessionHeader({ id: 'session-poison', cwd: '/home/u/proj' }),
JSON.stringify({
type: 'assistant/message', seq: 1, time: 1786707340000,
data: {
turn: 1, step: 1, message: { role: 'assistant', content: [] },
usage: { inputTokens: '999', outputTokens: [1, 2], reasoningTokens: 1e308 * 10, cacheReadTokens: -5, cacheWriteTokens: 7 },
},
}),
])
const calls = await parseAll(createDshProvider(tmpDir), filePath)
expect(calls).toHaveLength(1)
// Only the one genuinely numeric field survives; every other shape is 0.
expect(calls[0]).toMatchObject({
inputTokens: 0,
outputTokens: 0,
reasoningTokens: 0,
cacheReadInputTokens: 0,
cacheCreationInputTokens: 7,
})
for (const value of [calls[0]!.inputTokens, calls[0]!.outputTokens, calls[0]!.costUSD]) {
expect(typeof value).toBe('number')
expect(Number.isFinite(value)).toBe(true)
}
})
it('still skips a call whose usage is all non-numeric', async () => {
const filePath = await writePlainSession('--home-u-proj--', 'session-poison-zero', [
sessionHeader({ id: 'session-poison-zero', cwd: '/home/u/proj' }),
JSON.stringify({
type: 'assistant/message', seq: 1, time: 1786707340000,
data: { turn: 1, step: 1, message: { role: 'assistant', content: [] }, usage: { inputTokens: '999', outputTokens: [1, 2] } },
}),
])
expect(await parseAll(createDshProvider(tmpDir), filePath)).toEqual([])
})
})