docs(dsh): finish the provider registration checklist

docs/providers/NEW_PROVIDER.md items the PR had not reached yet, plus the two
surfaces that are functional rather than cosmetic:

- docs/providers/dsh.md and its row in the provider index, documenting the
  storage layout, the JSONL-backend-only scope (the opt-in SQLite persistence
  backend is not read), and that DSH is a developer preview whose format
  version 0 implies no compatibility.
- CHANGELOG entry under Unreleased.
- README provider count 40 -> 41 and a data-locations row.
- app/package.json: $HOME/.dsh in the snap personal-files allowlist, without
  which the Linux snap build cannot read DSH sessions at all.
- UsageDataChangeGuard: the DSH sessions root, without which the menubar never
  notices a new session and does not refresh.
- Bumps the dsh parse version, since the parser's attribution changed.
This commit is contained in:
iamtoruk 2026-08-17 10:59:58 -07:00
parent 4fa16a2293
commit d9a9486b6d
8 changed files with 86 additions and 6 deletions

View file

@ -2,6 +2,9 @@
## Unreleased
### Added (CLI)
- **DeepSeek Harness (`dsh`) is now a supported provider.** Reads DeepSeek's open-source agent harness from `~/.dsh/sessions` (`DSH_HOME` relocates the root), both the default zstd logs and the uncompressed `session.jsonl` variant. A `.zstd` log is a concatenation of independent zstd frames, one per write batch, so it is decoded frame by frame behind a structural frame scan and a torn trailing frame from a crashed writer is ignored rather than failing the file (needs Node 22.15+ for `zlib` zstd; below that dsh is skipped with a notice instead of counted as $0). One call per `(turn, step)`, with the step's final `assistant/message` usage superseding the streamed `assistant/chunk` sample of the same call rather than adding to it, the model taken from the message that served the step, and reasoning tokens billed at the output rate. DSH records tokens but no cost, so calls are priced from the shared tables. The events a forked session replays from its parent are skipped, since codeburn already counts the parent's own log. The session format is pinned at version 0 upstream with no compatibility implied, so a log stamped with any other version is skipped with a notice instead of read under today's assumptions.
### Changed
- **Codex rollouts parse across worker threads too, and the workload gate now takes bytes or files.** Codex is the bigger half of a real cold parse — a 4 GB rollout corpus against 1.8 GB of Claude sessions — and it was still decoding one file at a time. A whole-file rollout decode now runs on the same pool, against an empty dedup set, and comes back with the calls, the dedup keys it claimed, and the codex-cache entry it would have written; the parent installs all three in the serial loop's order, so `codex-results.json` and every payload come out byte-identical to a serial run. Cross-file state stays where it was: a forked rollout replaying its parent's token_count history collides on the parent's keys and is re-parsed in-process, and no worker ever touches the cache module's per-directory state. Files the Codex cache can serve exactly or resume into from a byte offset never reach a worker — they read a few KB and the resume state belongs to the parent. The workload gate is now pending BYTES alone (200 MB), not file count: 250 pending files holding under a megabyte between them spawned threads that made the run ~5% slower, while a few hundred huge rollouts were being turned away. The count takes `max(pendingFiles / 50, pendingBytes / 200 MB)`, and the per-thread memory budget is derived per parse as `clamp(256 MB, 2 × average pending file + 128 MB, 1 GB)` rather than a flat 256 MB — a 260 MB rollout peaks near 430 MB in its worker and scales linearly with the pool, so the flat figure over-subscribed exactly the workload this adds. The decision is per provider, and at most one pool is alive at a time.
- **A large cold Claude parse now runs across worker threads.** Reading, decoding and line-parsing a session JSONL is per-file work that never touches anything shared, so it moves onto `worker_threads`; each worker ships its parsed turns back as a JSON string and the parent installs them in the exact order the serial loop would. Everything with cross-file state — the streaming-message dedup, canonical project paths, spawn links, PR correlation, progress saves — stays on the main thread, and a file whose message ids were already claimed by an earlier file (or whose worker failed) is simply re-parsed in-process, so the session cache and every payload are identical either way. On a 6 GB corpus a cold `status` drops from 27.5s to 14.8s with peak RSS up 2.27 GB → 2.52 GB. Threads only engage for a genuinely large cold parse: never with under 200 MB behind the pending whole-file re-parses, 2 or fewer cores, or under 4 GB of available memory — so warm and incremental runs are untouched and spawn nothing. Otherwise the count is `min(cores - 1, min(0.25 × available, 2 GB) / 256 MB, pendingFiles / 50)`, where available is `process.availableMemory()` (cgroup-aware in containers) rather than free memory, which on macOS reports free pages and would switch the feature on and off between runs. `CODEBURN_PARSE_WORKERS=0` forces the serial parse and `CODEBURN_PARSE_WORKERS=N` forces N (capped at the core count), both bypassing every gate; `CODEBURN_VERBOSE=1` prints the resolved count and why.

View file

@ -25,7 +25,7 @@
<a href="https://github.com/sponsors/iamtoruk"><img src="https://img.shields.io/badge/sponsor-♥-F97316?logo=github" alt="Sponsor" /></a>
</p>
<p align="center">If CodeBurn shows you something your bill never did, <a href="https://github.com/getagentseal/codeburn/stargazers">star the repo</a> so other developers find it, and consider <a href="https://github.com/sponsors/iamtoruk">sponsoring</a> to keep 40 integrations honest.</p>
<p align="center">If CodeBurn shows you something your bill never did, <a href="https://github.com/getagentseal/codeburn/stargazers">star the repo</a> so other developers find it, and consider <a href="https://github.com/sponsors/iamtoruk">sponsoring</a> to keep 41 integrations honest.</p>
<table align="center">
<tr>
@ -61,11 +61,11 @@
<p align="center"><em>Four surfaces, one source of truth: everything reads the session files already on your disk.</em></p>
**CodeBurn is a free, open-source, local-first tool that tracks AI coding token usage and cost across 40 tools and agents (Claude Code, Cursor, Codex, Gemini, Grok and more), broken down by model, project, and task.**
**CodeBurn is a free, open-source, local-first tool that tracks AI coding token usage and cost across 41 tools and agents (Claude Code, Cursor, Codex, Gemini, Grok and more), broken down by model, project, and task.**
You pay for Claude, Codex, Cursor, and a stack of other AI tools. The bill tells you the total. It never tells you that half of it went to conversation instead of code, or that an expensive model burned your budget on work a cheaper one would have one-shot.
CodeBurn does. It reads the session files your tools already write to disk and breaks down every token and dollar by **task, model, tool, and project**, across **40 AI tools**.
CodeBurn does. It reads the session files your tools already write to disk and breaks down every token and dollar by **task, model, tool, and project**, across **41 AI tools**.
Everything runs locally. No wrapper, no proxy, no API keys, nothing leaves your machine. Pricing comes from [LiteLLM](https://github.com/BerriAI/litellm), refreshed daily.
@ -683,6 +683,7 @@ These are starting points, not verdicts. A 60% cache hit on a single experimenta
| **Cline / Roo Code / KiloCode** | VS Code `globalStorage` across VS Code, VS Code Insiders, and VSCodium (Cline at `saoudrizwan.claude-dev`, plus `~/.cline/data`) | Cline-family agents. CodeBurn reads `ui_messages.json` from each task directory, extracting token counts from `type: "say"` entries with `say: "api_req_started"`. |
| **Cline CLI** | `~/.cline/data/sessions/<session-id>/` (honors `CLINE_SESSION_DATA_DIR`, `CLINE_DATA_DIR`, `CLINE_DIR`) | The Cline command-line agent, whose layout is unrelated to the VS Code extension's. Reads `<session-id>.json` for session metadata and the rolled-up `usage`, and `<session-id>.messages.json` for the per-message `metrics` block (input, output, cacheRead, cacheWrite, cost) that becomes one call each. |
| **CodeWhale** | `~/.codewhale/sessions/*.json` plus unmigrated legacy `~/.deepseek/sessions/*.json`; `$CODEWHALE_HOME/sessions` is an exact override | Emits one cumulative record per saved session. CodeWhale exposes only `total_tokens`, so CodeBurn preserves that aggregate in the input column rather than inventing an input/output split. Cost is the exact stored parent-session plus subagent USD total; model pricing is used only when the cost snapshot is absent. Tool blocks, shell commands, skills, and subagent types are retained. |
| **DeepSeek Harness** (`dsh`) | `~/.dsh/sessions/--<slug>--/<session-id>/session.jsonl.zstd` (or `session.jsonl` when compression is off); `DSH_HOME` relocates the root | DeepSeek's open-source agent harness, unrelated to the CodeWhale desktop app. The `.zstd` log is a concatenation of independent zstd frames (one per write batch), decoded frame by frame; needs Node 22.15+. One call per `(turn, step)`, with usage from the step's `assistant/message` (the streamed `assistant/chunk` sample is a draft of the same call, never a second one). DSH records tokens but no cost, so calls are priced from the shared tables with reasoning billed at the output rate. |
| **IBM Bob** | `User/globalStorage/ibm.bob-code/tasks/<task-id>/` (GA `IBM Bob` and preview `Bob-IDE` app folders) | Reads `ui_messages.json` for API request token/cost records and `api_conversation_history.json` for the selected model. |
| **Kimi Code CLI** | `$KIMI_SHARE_DIR/sessions/<workdir-hash>/<session-id>/` or `~/.kimi/sessions/<workdir-hash>/<session-id>/` | Reads `wire.jsonl` `StatusUpdate.token_usage` records, mapping `input_other`, `input_cache_read`, `input_cache_creation`, and `output` into the standard token columns; includes subagents under each session's `subagents/` folder. |
| **LingTai TUI** | `~/.lingtai/<agent>/logs/token_ledger.jsonl` plus project homes from `~/.lingtai-tui/registry.jsonl` (`<project>/.lingtai/<agent>/logs/token_ledger.jsonl`); honors `LINGTAI_HOME` / `LINGTAI_TUI_HOME` | Reads LingTai's append-only token ledger, mapping `input - cached` to fresh input, `cached` to cache reads, `output` to output, and `thinking` to reasoning. Nested daemon ledgers are skipped because parent ledgers already mirror daemon usage with `source`/`run_id` tags. |
@ -722,12 +723,12 @@ CodeBurn deduplicates messages (by API message ID for Claude, by cumulative toke
CodeBurn is free, runs entirely on your machine, and exists to cut your AI bill. If it has already saved you more than a sponsorship costs, consider sending a little of that back.
Keeping 40 integrations accurate is constant work. The tools underneath change every week: Cursor reshapes its database, Claude moves a config path, new models ship at new prices. Sponsorship keeps CodeBurn current with all of it, so the numbers you see are always the real ones.
Keeping 41 integrations accurate is constant work. The tools underneath change every week: Cursor reshapes its database, Claude moves a config path, new models ship at new prices. Sponsorship keeps CodeBurn current with all of it, so the numbers you see are always the real ones.
Where your sponsorship goes:
- **Honest numbers.** New models and price changes are mapped quickly, so your cost is the real cost, not a guess.
- **More tools.** Every one of the 40 providers started as a single file. Sponsorship funds the next one.
- **More tools.** Every one of the 41 providers started as a single file. Sponsorship funds the next one.
- **Fast fixes.** When a vendor breaks something, paid time is what gets it patched now instead of someday.
Sponsoring as a team or company? Your logo lands right here, in front of every developer who opens the repo. The first sponsor gets it to themselves until the next one shows up.

View file

@ -169,6 +169,7 @@
"$HOME/.copilot",
"$HOME/.cursor",
"$HOME/.deepseek",
"$HOME/.dsh",
"$HOME/.factory",
"$HOME/.forge",
"$HOME/.gemini",

View file

@ -191,7 +191,7 @@ type Provider = {
`src/providers/index.ts` registers providers across two tiers:
- **Eager**: `claude`, `cline`, `codewhale`, `codebuff`, `codex`, `copilot`, `devin`, `droid`, `gemini`, `hermes`, `ibm-bob`, `kilo-code`, `kiro`, `kimi`, `lingtai-tui`, `mistral-vibe`, `mux`, `openclaw`, `open-design`, `pi`, `omp`, `qwen`, `roo-code`, `zerostack`, `grok`. Imported at module load.
- **Eager**: `claude`, `cline`, `codewhale`, `codebuff`, `codex`, `copilot`, `devin`, `droid`, `dsh`, `gemini`, `hermes`, `ibm-bob`, `kilo-code`, `kiro`, `kimi`, `lingtai-tui`, `mistral-vibe`, `mux`, `openclaw`, `open-design`, `pi`, `omp`, `qwen`, `roo-code`, `zerostack`, `grok`. Imported at module load.
- **Lazy**: `antigravity`, `forge`, `goose`, `cursor`, `opencode`, `cursor-agent`, `crush`, `warp`, `vercel-gateway`, `zcode`, `zed`. Imported via dynamic `import()` so the heavy dependencies (SQLite, protobuf, network clients) do not touch users who do not have those tools installed.
Both lists hit the same `getAllProviders()` aggregator. A failed lazy import is silent and excludes that provider from the run.

View file

@ -18,6 +18,7 @@ For the architectural picture, see `../architecture.md`.
| [Copilot](copilot.md) | JSONL + SQLite (OTel) + Nitrite .db (JetBrains) | `src/providers/copilot.ts` | `tests/providers/copilot.test.ts` |
| [Devin](devin.md) | JSON + SQLite enrichment | `src/providers/devin.ts` | `tests/providers/devin.test.ts` |
| [Droid](droid.md) | JSONL | `src/providers/droid.ts` | `tests/providers/droid.test.ts` |
| [DeepSeek Harness](dsh.md) | JSONL (zstd frames) | `src/providers/dsh.ts` | `tests/providers/dsh.test.ts` |
| [Gemini](gemini.md) | JSON / JSONL | `src/providers/gemini.ts` | none |
| [Hermes Agent](hermes.md) | SQLite | `src/providers/hermes.ts` | `tests/providers/hermes.test.ts` |
| [IBM Bob](ibm-bob.md) | JSON | `src/providers/ibm-bob.ts` | `tests/providers/ibm-bob.test.ts` |

71
docs/providers/dsh.md Normal file
View file

@ -0,0 +1,71 @@
# DeepSeek Harness (dsh)
DeepSeek's open-source agent harness (`dsh`, npm `@deepseek-ai/dsh`). Unrelated to the [CodeWhale](codewhale.md) provider, which reads the DeepSeek desktop app.
- **Source:** `src/providers/dsh.ts`
- **Loading:** eager (`src/providers/index.ts`)
- **Test:** `tests/providers/dsh.test.ts`
## Where it reads from
| Level | Env var | Default |
|---|---|---|
| sessions | — | `<root>/sessions` |
| root | `DSH_HOME` | `~/.dsh` |
An empty `DSH_HOME` is treated as unset. `probeRoots()` reports the resolved sessions dir, so `codeburn doctor` distinguishes "dsh not installed" from "`DSH_HOME` pointing somewhere empty".
## Storage format
```
sessions/--<slugified-cwd>--/<session-id>/
session.jsonl.zstd default (compression: zstd)
session.jsonl when compression: none
```
Both variants are read; a session directory never holds both. The log is append-only JSONL whose first line is the session header:
```jsonc
{ "type": "session", "version": 0, "id": "...", "createdAt": 1783352050748,
"cwd": "/home/u/proj", "parentSession": "...", "seedLength": 3, "delegationDepth": 0 }
```
`cwd` becomes `projectPath` / `workingDirectory` (git-repo attribution) and its last segment the project name.
Every later line is one event `{ type, seq, time, data }`. The parser reads:
| Event | Used for |
|---|---|
| `turn/start` | current turn number |
| `user/message` | the turn's preview, when `data.source.kind === 'user'` |
| `request/header` | `data.header.config.model` — the model for steps that follow |
| `assistant/chunk` with `chunk.type === 'usage'` | streamed usage sample for `(turn, step)` |
| `assistant/message` | final usage for `(turn, step)`, plus `data.message.source.model` |
| `tool/call` | tool names, bash commands, skill names |
One parsed call per `(turn, step)` — one model call and the tools it requested. Dedup key: `dsh:<sessionId>:<turn>:<step>`.
`.zstd` logs are a concatenation of **independent** zstd frames, one per write batch, so they are decoded frame by frame behind a structural frame scan ported from `@deepseek-ai/dsh-session-persistence-jsonl`. Needs Node 22.15+ for `zlib.zstdDecompressSync`; below that dsh is skipped with a notice instead of counted as $0.
## Caching
None at the provider level; the log file is the cached source path and the normal parser/cache layers apply. Cache invalidates on `DSH_HOME` (`PROVIDER_ENV_VARS`) and on parser changes (`PROVIDER_PARSE_VERSIONS`).
## Quirks
- **DSH is a developer preview.** `SESSION_FORMAT_VERSION` is pinned at `0` with "no compatibility implied" upstream, and breaking changes are expected. The parser reads version `0` only and skips a log stamped with anything else, with a notice — reading a bumped format under today's assumptions would report confident wrong numbers. **A version bump upstream means this parser needs updating, not just relaxing the check.**
- **The JSONL backend only.** DSH also ships an opt-in SQLite persistence backend (`@deepseek-ai/dsh-session-persistence-sqlite`); it is not the default and is not read.
- **DSH records tokens, never dollars.** `usage` is `{ inputTokens, outputTokens, cacheReadTokens?, cacheWriteTokens?, reasoningTokens? }` with no cost field, so every call is priced from the shared tables. Reasoning bills at the output rate (same as Gemini and Hermes): `outputTokens + reasoningTokens` goes into `calculateCost`, while the two stay separate on the emitted call. Tokens are the provider's own exact counts, so `costIsEstimated` stays false.
- **`assistant/message` usage wins over the `assistant/chunk` sample** for the same `(turn, step)` — the two are adjacent reports of one API call, not two calls. A late chunk never overwrites a final report, so the two are never summed.
- **The model comes from the message, not the request.** `data.message.source.model` is what actually served the step; `request/header` only describes the request DSH was about to make, and is the fallback when a message names no model. The `provider` field there (`deepseek-official`) is the upstream LLM route, not the tool — the codeburn provider name is always `dsh`.
- **A forked session's log replays its parent's events.** The header's `parentSession` + `seedLength` mark that prefix; codeburn parses the parent's own log as its own session, so events with `seq < seedLength` are skipped to avoid billing the same calls twice.
- **`user/message` also carries agent-injected context** (runtime snapshots, skill bodies, file-change notices) under `source.kind: 'plugin'`. Only `kind: 'user'` messages become the preview.
- **Delta chunks are packed.** Runs of streamed deltas are stored as `text-chunks` / `reasoning-chunks` / `tool-call-chunks` storage rows rather than one event per line. They carry no usage and no tool identity the `tool/call` event lacks, so they are ignored — as is any event type the parser does not know.
- **A torn final zstd frame is ignored.** A crashed writer leaves an incomplete trailing frame; the complete frames before it parse normally. A structurally corrupt file is skipped whole with a notice rather than throwing.
## When fixing a bug here
1. Reproduce with a minimal session dir: `sessions/--proj--/<id>/session.jsonl` (uncompressed is easiest to hand-write).
2. `tests/fixtures/dsh/bash-tool-turn.jsonl` is the upstream `examples/acp-agent/tests/snapshots/bash-tool-turn/session.jsonl` snapshot with its template placeholders filled in — refresh it from the DSH repo when the format moves.
3. Run `tests/providers/dsh.test.ts`.
4. `.zstd` fixtures must compress **each batch separately**; one `zstdCompressSync` over the whole file is a single-frame layout DSH never writes.

View file

@ -72,6 +72,8 @@ enum UsageDataChangeGuard {
add(expand(environment["CODEWHALE_HOME"] ?? path(homeDirectory, ".codewhale"), homeDirectory: homeDirectory), scanFirstLevelDirectories: false)
add(path(homeDirectory, ".deepseek", "sessions"), scanFirstLevelDirectories: false)
add(path(homeDirectory, ".cline", "data"), scanFirstLevelDirectories: false)
let dshHome = expand(environment["DSH_HOME"] ?? path(homeDirectory, ".dsh"), homeDirectory: homeDirectory)
add(path(dshHome, "sessions"))
add(expand(environment["CODEBUFF_DATA_DIR"] ?? path(xdgConfig, "manicode"), homeDirectory: homeDirectory), scanFirstLevelDirectories: false)
let factoryHome = expand(environment["FACTORY_DIR"] ?? path(homeDirectory, ".factory"), homeDirectory: homeDirectory)
add(path(factoryHome, "sessions"), scanFirstLevelDirectories: false)

View file

@ -33,6 +33,7 @@
"pi",
"codebuff",
"codewhale",
"dsh",
"ai-coding",
"token-usage",
"cost-tracking",