mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-10-19 18:23:41 +00:00
devin: price sessions from tokens, drop the ACU rate (#1469)
Devin transcripts carry no committed_acu_cost, so the ACU path priced every session at zero and devin.acuUsdRate hid them all until it was set by hand. Cost now comes from each step's tokens and its generation_model through the shared pricing tables, with the cached share carved out of OpenAI-style prompt_tokens; swe-2-high stays unpriced at $0. Task text is the user turn, then the sessions.db title, then prompt_history. The menubar ACU field, its config store and the config key are gone. Separate commit: gpt-5-3-codex ids resolve to the gpt-5.3-codex pricing row instead of the base gpt-5 row.
This commit is contained in:
Resham Joshi
GitHub
parent
2371760b27
commit
b162dcfcef
11 changed files with 295 additions and 491 deletions
|
|
@ -22,10 +22,12 @@
|
|||
- **The Capacity Dock gauge can report the short usage window instead of the weekly one, without expanding the dock.** The resting rail shows one number per provider, and that number was always the weekly (else monthly) billing window. Clicking the provider already resting in the rail now switches its gauge to the provider's short rolling window — Claude's 5-hour limit, Codex's 5-hour or daily window, any `Hourly`, `Daily` or session row an adapter reports — and clicking again switches back. The choice is stored per provider under `CodeBurnCapacityDockGlanceWindows`, so Claude can sit on its 5-hour window while Codex stays weekly, and it survives relaunch. A per-model row such as `Weekly · Opus` is never read as a short window, a provider that reports only one window keeps the plain click-to-pin behaviour, and a stored horizon the provider stops reporting falls back to the window it does report rather than blanking the gauge to `--`. The rail's geometry is untouched, VoiceOver and keyboard users get the switch as a named action on the provider cell with the window named in the cell's value, and Escape or a click outside still unpins the dock. (#1243)
|
||||
|
||||
### Changed
|
||||
- **Devin is priced from tokens, like every other provider, and no longer needs to be configured before it appears.** Devin's transcripts stopped carrying `committed_acu_cost` (it is null in current builds), so a provider that converted ACUs to dollars had nothing to convert and hid every session until a `devin.acuUsdRate` was set by hand. Cost now comes from each step's own `prompt_tokens`/`completion_tokens`/`cached_tokens` and the model that produced it (`generation_model` per step, falling back to the session's model), priced through the same tables as Claude Code and Codex. Devin reports OpenAI-style prompt tokens with the cached share counted inside them, so that share is subtracted from input rather than billed twice. A model the tables do not carry (today `swe-2-high`) costs $0 and warns once, rather than being priced off a neighbour. The `devin.acuUsdRate` config key is gone, along with the ACU Conversion field in the menubar's Settings → Devin tab that wrote it; the key is ignored if still present in `~/.config/codeburn/config.json`, and the Devin tab is now the standard provider connection pane. Task text comes from the session's own user turns, which real transcripts mark with `source: "user"` rather than `is_user_input`, falling back to the `sessions.db` title and then to the session's first `prompt_history` prompt.
|
||||
- **Quota colours use one set of bands everywhere: green under 60%, yellow to 70%, orange to 95%, red from 95%.** The flame, the popover rows and the Capacity Dock rings each had their own thresholds; the dock now reads the same severity the flame does.
|
||||
- **The Capacity Dock card and the popover quota rows no longer print a pace verdict under each window.** The "On pace" / "Lasts until reset" caption from #1314 added a third line to every window cell and made the card read as clutter; the cells are back to percent, label and countdown, up to three windows share one row again with a fourth wrapping to a second, and the pace math still feeds the tooltip and the agent-tab card.
|
||||
|
||||
### Fixed
|
||||
- **A Devin step run on `gpt-5-3-codex-xhigh` is priced at the Codex row it names, not the base `gpt-5` one.** Devin writes the model's minor version with a dash, and that spelling has no pricing row, so lookup fell through to `gpt-5` and undercharged every Codex-backed step by $0.50 per million input tokens. The id is now normalised to `gpt-5.3-codex-xhigh` for pricing, the same rewrite the display name already used.
|
||||
- **Model tables add up to the headline above them.** `codeburn report --format json` took its totals from the durable daily cache but built its model rows from surviving sessions, so on a long period the rows summed to less than the headline; they now come from the same days the headline and the daily rows do. Days recorded before CodeBurn tracked usage per model kept their cost and calls but had nothing to attribute them to, and that money simply went missing from every model table. It now shows as a row named "Unknown (carried)" — the same figure, named instead of silently dropped.
|
||||
- **OpenCode 2.x sessions show up again.** OpenCode 2.x (mainline since 2.0.3) writes sessions to a new `session_v2` table and messages to `session_message`, tagged by type and ordered by sequence, while the old `session`/`message`/`part` tables freeze at upgrade — so every session created after the upgrade was invisible. CodeBurn now branches per database on what actually exists: when the 2.x tables are there they are read and the frozen legacy tables are ignored, otherwise the legacy path runs exactly as before. Compaction requests keep their own cost and tokens counted, as they were on 1.x. (#1293)
|
||||
- **The Overview model table lists every model with usage in the period instead of stopping at twenty.** The period-scoped model list the table reads is no longer cut at 20 rows by the payload: on a machine with 47 models every local and free model (qwen, llama, deepseek, gemini, grok) had dropped out of the table — exactly the rows whose $0 cost ranks them last — while the same change was busy making those models visible in the Models tables. Every consumer that wants fewer rows already slices its own (the desktop hero takes 8, MCP tables take 5 or the caller's limit) and the macOS menubar reads a per-session list, so nothing else renders more. Two smaller items from the same review: the menubar Models row's compact token count gains a billions rung, so a period cache-read total past a billion reads `12.3B` instead of `12345.7M`, with the exact value still in the row's accessibility text, and the CLI tables gain the same rung through `formatTokens` (a `codeburn models` totals row now reads `29.9B` instead of `29870.8M`); and the desktop Models tables stop dimming the calls column on unpriced rows — calls are observed usage like the token columns, not a pricing artifact — while cost/saved keep their dashes and the add-alias affordance. In the shipped app unpriced rows do not reach that table yet: `codeburn models` defaults to `minCost: 0.01` and drops rows whose cost and savings are both under $0.01, and the desktop's `codeburn:getModels` passes no `--min-cost`/`--unpriced`, so the change only takes effect once that filter is addressed (tracked separately). Status snapshot revision 9 invalidates older cached payloads once, so a warm snapshot cannot keep serving the truncated 20-row list; daily and session cache formats are unchanged. (#1318)
|
||||
|
|
|
|||
|
|
@ -32,30 +32,8 @@ or billing.
|
|||
|
||||
## Configuration
|
||||
|
||||
Devin reports spend in ACUs. CodeBurn reports provider cost through `costUSD`,
|
||||
so Devin stays disabled until a positive finite ACU-to-USD rate is configured:
|
||||
|
||||
```json
|
||||
{
|
||||
"devin": {
|
||||
"acuUsdRate": 2.25
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The config file is:
|
||||
|
||||
```text
|
||||
~/.config/codeburn/config.json
|
||||
```
|
||||
|
||||
The macOS Settings window writes this value from the Devin tab. There is no
|
||||
environment-variable override and no default rate. Do not hardcode a universal
|
||||
ACU price; Devin ACU pricing is account/contract dependent.
|
||||
|
||||
When the rate is missing or invalid, `discoverSessions()` returns `[]` and the
|
||||
parser yields no calls. Devin remains registered as a provider, but it does not
|
||||
appear in CLI/UI results until configured.
|
||||
None. Devin is priced from per-step tokens like every other provider, so it
|
||||
needs no config to appear in CLI/UI results.
|
||||
|
||||
## Storage format
|
||||
|
||||
|
|
@ -69,9 +47,7 @@ backend/permission info), `final_metrics`, and `steps[]`.
|
|||
|
||||
Steps now support two metric sources. The parser checks `step.metrics` first
|
||||
(the standard ATIF location) and falls back to `step.metadata.metrics` (the
|
||||
legacy Devin location). Similarly, ACU cost is read from
|
||||
`step.metadata.committed_acu_cost` first, falling back to
|
||||
`step.extra.committed_acu_cost`.
|
||||
legacy Devin location). `committed_acu_cost` is ignored wherever it appears.
|
||||
|
||||
Messages can be a plain string or an array of `ContentPart` objects (text or
|
||||
image), following the ATIF v1.6+ multimodal content model. The parser
|
||||
|
|
@ -80,7 +56,6 @@ normalises both forms when extracting user messages.
|
|||
Each counted step can provide:
|
||||
|
||||
- `step_id`
|
||||
- `metadata.committed_acu_cost` (or `extra.committed_acu_cost`)
|
||||
- `metrics.prompt_tokens` (or `metadata.metrics.input_tokens`)
|
||||
- `metrics.completion_tokens` (or `metadata.metrics.output_tokens`)
|
||||
- `metrics.extra.cache_creation_input_tokens` (or `metadata.metrics.cache_creation_tokens`)
|
||||
|
|
@ -91,24 +66,29 @@ Each counted step can provide:
|
|||
- `tool_calls[].function_name`
|
||||
- `observation.results[]` (tool output; not parsed for usage)
|
||||
|
||||
User-input steps (`metadata.is_user_input === true`) are skipped. Non-user
|
||||
steps are included only if they have positive ACU usage or positive token usage.
|
||||
User steps (`source === "user"` or `metadata.is_user_input === true`) are skipped. Non-user
|
||||
steps are included only if they have positive token usage.
|
||||
|
||||
## Pricing
|
||||
|
||||
ACU cost is per step, not cumulative. The provider reads
|
||||
`metadata.committed_acu_cost` first, falling back to
|
||||
`extra.committed_acu_cost`, then converts with:
|
||||
Per step, from tokens and the model id, through `calculateCost()` — the same
|
||||
pricing tables every other provider uses. The pricing id is
|
||||
`metadata.generation_model` (or `extra.generation_model`), falling back to
|
||||
`step.model_name`, `agent.model_name`, then `sessions.model`; a `MODEL_*`
|
||||
placeholder is never used for pricing. `gpt-5-3-codex` style ids are rewritten
|
||||
to `gpt-5.3-codex` before lookup, so they hit their own row instead of
|
||||
collapsing to the base `gpt-5` price.
|
||||
|
||||
```text
|
||||
costUSD = committed_acu_cost * devin.acuUsdRate
|
||||
```
|
||||
Devin reports OpenAI-style `prompt_tokens` with the cached tokens counted
|
||||
inside it, so the cached share is subtracted from input rather than billed at
|
||||
the full input rate twice.
|
||||
|
||||
Token-only steps are still included when they have positive token metrics, but
|
||||
their `costUSD` is `0` if `committed_acu_cost` is absent from both locations.
|
||||
A model with no pricing row (today: `swe-2-high`) costs `$0` and warns once,
|
||||
like any other unpriced model. It is never silently priced off a neighbouring
|
||||
row.
|
||||
|
||||
`src/parser.ts` preserves Devin's provider-supplied `costUSD` instead of
|
||||
re-pricing it through LiteLLM.
|
||||
re-pricing it, because the call carries Devin's display model name.
|
||||
|
||||
## sessions.db enrichment
|
||||
|
||||
|
|
@ -119,12 +99,13 @@ The provider currently reads these columns from `sessions`:
|
|||
| `id` | join key with transcript `session_id` during parsing; discovery uses the transcript filename before `.json` |
|
||||
| `working_directory` | `projectPath` and derived project name |
|
||||
| `model` | model fallback |
|
||||
| `title` | project name fallback |
|
||||
| `title` | task text; project name fallback (falls back to the session's first `prompt_history` row when empty) |
|
||||
| `created_at` | timestamp fallback |
|
||||
| `last_activity_at` | preferred session timestamp fallback |
|
||||
| `hidden` | skip hidden sessions |
|
||||
|
||||
`message_nodes`, `prompt_history`, and `tool_call_state` are not parsed yet.
|
||||
`prompt_history` supplies the task text for a session Devin has not titled yet.
|
||||
`message_nodes` and `tool_call_state` are not parsed.
|
||||
|
||||
## Timestamps
|
||||
|
||||
|
|
@ -139,7 +120,7 @@ Numeric normalization is only applied to `sessions.db` timestamps:
|
|||
|
||||
## Model Resolution
|
||||
|
||||
Model names resolve in this order:
|
||||
Model names (display and pricing) resolve in this order:
|
||||
|
||||
1. `step.metadata.generation_model`
|
||||
2. `step.model_name`
|
||||
|
|
@ -164,28 +145,29 @@ The provider name is part of the key via the `devin:` prefix.
|
|||
## Quirks
|
||||
|
||||
- The transcript directory has usage; `sessions.db` is enrichment only.
|
||||
- `committed_acu_cost` is per-generation/per-step ACU usage. Never treat it as cumulative. It can appear in `metadata` (legacy) or `extra` (ATIF v1.7); the provider checks both.
|
||||
- `committed_acu_cost` is null in current Devin builds and is ignored; cost comes from tokens.
|
||||
- Token metrics can live in `step.metrics` (standard ATIF) or `step.metadata.metrics` (legacy Devin). The provider checks `step.metrics` first, falling back to `metadata`.
|
||||
- Step messages can be a plain string or an array of `ContentPart` objects (text/image). The parser normalises both when extracting user messages.
|
||||
- There is no default ACU-to-USD rate. Missing config intentionally hides Devin.
|
||||
- Real transcripts mark the user turn with `source: "user"`; older ones set `metadata.is_user_input`. Either marks a step as the user's, so it is skipped and used as task text.
|
||||
- Hidden sessions from `sessions.db` are skipped in discovery and parsing.
|
||||
- Tool names come directly from `tool_calls[].function_name`; the provider assumes valid ATIF tool-call records.
|
||||
- If SQLite is unavailable or `sessions.db` cannot be opened, the provider still parses transcripts without enrichment.
|
||||
|
||||
## When fixing a bug here
|
||||
|
||||
1. First check whether `~/.config/codeburn/config.json` contains a valid
|
||||
`devin.acuUsdRate`. Without it, no Devin sessions should appear.
|
||||
2. For usage total bugs, compare against (ACU cost can live in `metadata` or `extra`):
|
||||
1. For usage total bugs, compare against the transcript's own metrics:
|
||||
|
||||
```bash
|
||||
jq '[.steps[] | select(.metadata.is_user_input != true) | (.metadata.committed_acu_cost // .extra.committed_acu_cost // 0)] | add' ~/.local/share/devin/cli/transcripts/<session>.json
|
||||
jq '.final_metrics, [.steps[] | select(.metrics.prompt_tokens) | .metrics.prompt_tokens] | add' ~/.local/share/devin/cli/transcripts/<session>.json
|
||||
```
|
||||
|
||||
2. For cost bugs, check which pricing row the step's `generation_model` hits;
|
||||
an unpriced model costs $0 by design.
|
||||
|
||||
3. If project/model/timestamp metadata is wrong, inspect `sessions.db`, not the transcript.
|
||||
4. If a hidden session appears, check the `hidden` column. Discovery can only
|
||||
hide sessions whose transcript filename matches `sessions.id`; parsing uses
|
||||
the transcript `session_id` when present.
|
||||
5. Run `tests/providers/devin.test.ts` after parser changes. It covers ACU conversion, disabled-until-configured behavior, timestamp parsing, deduplication, hidden sessions, `sessions.db` enrichment, ATIF v1.7 multimodal messages, `step.metrics` vs `metadata.metrics` priority, and `extra.committed_acu_cost` fallback.
|
||||
5. Run `tests/providers/devin.test.ts` after parser changes. It covers token pricing against the pricing tables, timestamp parsing, deduplication, hidden sessions, `sessions.db` enrichment, ATIF v1.7 multimodal messages, and `step.metrics` vs `metadata.metrics` priority.
|
||||
|
||||
[atif]: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md
|
||||
|
|
|
|||
|
|
@ -204,74 +204,3 @@ enum CLICurrencyConfig {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct CodeburnCLIConfigStore {
|
||||
let homeDirectory: String
|
||||
|
||||
init(homeDirectory: String = NSHomeDirectory()) {
|
||||
self.homeDirectory = homeDirectory
|
||||
}
|
||||
|
||||
private var configDir: String {
|
||||
(homeDirectory as NSString).appendingPathComponent(".config/codeburn")
|
||||
}
|
||||
private var configPath: String {
|
||||
(configDir as NSString).appendingPathComponent("config.json")
|
||||
}
|
||||
private var lockPath: String {
|
||||
(configDir as NSString).appendingPathComponent(".config.lock")
|
||||
}
|
||||
|
||||
func loadDevinAcuUsdRate() -> Double? {
|
||||
guard
|
||||
let data = try? SafeFile.read(from: configPath),
|
||||
let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let devin = json["devin"] as? [String: Any],
|
||||
let rate = devin["acuUsdRate"] as? Double,
|
||||
rate.isFinite,
|
||||
rate > 0
|
||||
else {
|
||||
return nil
|
||||
}
|
||||
return rate
|
||||
}
|
||||
|
||||
func persistDevinAcuUsdRate(_ rate: Double) {
|
||||
guard rate.isFinite, rate > 0 else { return }
|
||||
do {
|
||||
try SafeFile.withExclusiveLock(at: lockPath) {
|
||||
var existing: [String: Any] = [:]
|
||||
if let data = try? SafeFile.read(from: configPath),
|
||||
let parsed = try? JSONSerialization.jsonObject(with: data) as? [String: Any] {
|
||||
existing = parsed
|
||||
}
|
||||
|
||||
var devin = existing["devin"] as? [String: Any] ?? [:]
|
||||
devin["acuUsdRate"] = rate
|
||||
existing["devin"] = devin
|
||||
|
||||
guard let data = try? JSONSerialization.data(
|
||||
withJSONObject: existing,
|
||||
options: [.prettyPrinted, .sortedKeys]
|
||||
) else {
|
||||
return
|
||||
}
|
||||
try SafeFile.write(data, to: configPath, mode: 0o600)
|
||||
}
|
||||
} catch {
|
||||
NSLog("CodeBurn: failed to persist Devin ACU config: \(error)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
enum CLIDevinConfig {
|
||||
private static let store = CodeburnCLIConfigStore()
|
||||
|
||||
static func loadAcuUsdRate() -> Double? {
|
||||
store.loadDevinAcuUsdRate()
|
||||
}
|
||||
|
||||
static func persistAcuUsdRate(_ rate: Double) {
|
||||
store.persistDevinAcuUsdRate(rate)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -618,11 +618,6 @@
|
|||
"Automatic" = "Automatic";
|
||||
"CLI" = "CLI";
|
||||
"API" = "API";
|
||||
"ACU Conversion" = "ACU Conversion";
|
||||
"USD per ACU" = "USD per ACU";
|
||||
"Save" = "Save";
|
||||
"CodeBurn reads Devin ACU usage from local transcripts only after this rate is configured, then multiplies each step by the rate before reporting cost." = "CodeBurn reads Devin ACU usage from local transcripts only after this rate is configured, then multiplies each step by the rate before reporting cost.";
|
||||
"Saved. Refresh CodeBurn to recalculate Devin cost." = "Saved. Refresh CodeBurn to recalculate Devin cost.";
|
||||
"Version %@" = "Version %@";
|
||||
"%@ is available. Choose Check for Updates in the CodeBurn menu to install it." = "%@ is available. Choose Check for Updates in the CodeBurn menu to install it.";
|
||||
"GitHub" = "GitHub";
|
||||
|
|
|
|||
|
|
@ -610,11 +610,6 @@
|
|||
"Automatic" = "自动";
|
||||
"CLI" = "命令行工具";
|
||||
"API" = "API";
|
||||
"ACU Conversion" = "ACU 换算";
|
||||
"USD per ACU" = "每 ACU 美元";
|
||||
"Save" = "存储";
|
||||
"CodeBurn reads Devin ACU usage from local transcripts only after this rate is configured, then multiplies each step by the rate before reporting cost." = "只有在配置了此汇率之后，CodeBurn 才会从本地记录中读取 Devin 的 ACU 用量，并在报告花费前将每一步乘以该汇率。";
|
||||
"Saved. Refresh CodeBurn to recalculate Devin cost." = "已存储。刷新 CodeBurn 以重新计算 Devin 花费。";
|
||||
"Version %@" = "版本 %@";
|
||||
"%@ is available. Choose Check for Updates in the CodeBurn menu to install it." = "%@ 已发布。在 CodeBurn 菜单中选择“检查更新”即可安装。";
|
||||
"GitHub" = "GitHub";
|
||||
|
|
|
|||
|
|
@ -48,8 +48,6 @@ struct SettingsView: View {
|
|||
case CapacityDockProvider.gemini.id: store.geminiLoadState == .loaded
|
||||
case CapacityDockProvider.copilot.id: store.copilotLoadState == .loaded
|
||||
case CapacityDockProvider.antigravity.id: store.antigravityLoadState == .loaded
|
||||
case "devin":
|
||||
store.capacityDockProviderIsConnected(provider) || CLIDevinConfig.loadAcuUsdRate() != nil
|
||||
default: store.capacityDockProviderIsConnected(provider)
|
||||
}
|
||||
}
|
||||
|
|
@ -150,7 +148,6 @@ struct SettingsView: View {
|
|||
case "claude": ClaudeSettingsTab()
|
||||
case "codex": CodexSettingsTab()
|
||||
case "kimi": KimiSettingsTab()
|
||||
case "devin": DevinSettingsTab()
|
||||
case "gemini": GeminiSettingsTab()
|
||||
case "copilot": CopilotSettingsTab()
|
||||
case "antigravity": AntigravitySettingsTab()
|
||||
|
|
@ -2194,83 +2191,6 @@ private struct GenericProviderConnectionSections: View {
|
|||
|
||||
}
|
||||
|
||||
// MARK: - Devin
|
||||
|
||||
private struct DevinSettingsTab: View {
|
||||
@State private var rateText: String = ""
|
||||
@State private var statusText: String = ""
|
||||
|
||||
private var parsedRate: Double? {
|
||||
let trimmed = rateText.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard let value = Double(trimmed), value.isFinite, value > 0 else { return nil }
|
||||
return value
|
||||
}
|
||||
|
||||
var body: some View {
|
||||
Form {
|
||||
GenericProviderConnectionSections(provider: CapacityDockProvider(rawValue: "devin")!)
|
||||
|
||||
Section(L("ACU Conversion")) {
|
||||
HStack(alignment: .center, spacing: 10) {
|
||||
Text(L("USD per ACU"))
|
||||
Spacer()
|
||||
TextField("", text: $rateText)
|
||||
.textFieldStyle(.roundedBorder)
|
||||
.multilineTextAlignment(.trailing)
|
||||
.frame(width: 96)
|
||||
.accessibilityLabel(L("USD per ACU"))
|
||||
Text("USD")
|
||||
.foregroundStyle(.secondary)
|
||||
.frame(width: 36, alignment: .leading)
|
||||
}
|
||||
|
||||
Button(L("Save")) {
|
||||
saveRate()
|
||||
}
|
||||
.buttonStyle(.borderedProminent)
|
||||
.disabled(parsedRate == nil)
|
||||
|
||||
if !statusText.isEmpty {
|
||||
Text(statusText)
|
||||
.font(.system(size: 11))
|
||||
.foregroundStyle(.secondary)
|
||||
}
|
||||
}
|
||||
|
||||
Section {
|
||||
Text(L("CodeBurn reads Devin ACU usage from local transcripts only after this rate is configured, then multiplies each step by the rate before reporting cost."))
|
||||
.font(.system(size: 11))
|
||||
.foregroundStyle(.secondary)
|
||||
} header: {
|
||||
Text(L("How it works"))
|
||||
}
|
||||
}
|
||||
.formStyle(.grouped)
|
||||
.padding()
|
||||
.onAppear {
|
||||
if let rate = CLIDevinConfig.loadAcuUsdRate() {
|
||||
rateText = Self.format(rate)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func saveRate() {
|
||||
guard let rate = parsedRate else { return }
|
||||
CLIDevinConfig.persistAcuUsdRate(rate)
|
||||
rateText = Self.format(rate)
|
||||
statusText = L("Saved. Refresh CodeBurn to recalculate Devin cost.")
|
||||
}
|
||||
|
||||
private static func format(_ value: Double) -> String {
|
||||
let formatter = NumberFormatter()
|
||||
formatter.locale = Locale(identifier: "en_US_POSIX")
|
||||
formatter.numberStyle = .decimal
|
||||
formatter.minimumFractionDigits = 0
|
||||
formatter.maximumFractionDigits = 6
|
||||
return formatter.string(from: NSNumber(value: value)) ?? String(value)
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - About
|
||||
|
||||
private struct AboutSettingsTab: View {
|
||||
|
|
|
|||
|
|
@ -1,97 +0,0 @@
|
|||
import Foundation
|
||||
import Testing
|
||||
@testable import CodeBurnMenubar
|
||||
|
||||
@Suite("CLI Devin config", .serialized)
|
||||
struct CLIDevinConfigTests {
|
||||
private func withTemporaryStore(_ body: (URL, CodeburnCLIConfigStore) throws -> Void) throws {
|
||||
let root = FileManager.default.temporaryDirectory
|
||||
.appendingPathComponent("codeburn-devin-config-\(UUID().uuidString)", isDirectory: true)
|
||||
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
|
||||
defer {
|
||||
try? FileManager.default.removeItem(at: root)
|
||||
}
|
||||
try body(root, CodeburnCLIConfigStore(homeDirectory: root.path))
|
||||
}
|
||||
|
||||
private func configURL(in home: URL) -> URL {
|
||||
home
|
||||
.appendingPathComponent(".config", isDirectory: true)
|
||||
.appendingPathComponent("codeburn", isDirectory: true)
|
||||
.appendingPathComponent("config.json")
|
||||
}
|
||||
|
||||
private func writeConfig(_ object: [String: Any], in home: URL) throws {
|
||||
let url = configURL(in: home)
|
||||
try FileManager.default.createDirectory(
|
||||
at: url.deletingLastPathComponent(),
|
||||
withIntermediateDirectories: true
|
||||
)
|
||||
let data = try JSONSerialization.data(withJSONObject: object, options: [.prettyPrinted, .sortedKeys])
|
||||
try data.write(to: url)
|
||||
}
|
||||
|
||||
private func readConfig(in home: URL) throws -> [String: Any] {
|
||||
let data = try Data(contentsOf: configURL(in: home))
|
||||
return try #require(JSONSerialization.jsonObject(with: data) as? [String: Any])
|
||||
}
|
||||
|
||||
@Test("missing config has no ACU rate")
|
||||
func missingConfigHasNoRate() throws {
|
||||
try withTemporaryStore { _, store in
|
||||
#expect(store.loadDevinAcuUsdRate() == nil)
|
||||
}
|
||||
}
|
||||
|
||||
@Test("persists and loads ACU rate")
|
||||
func persistsAndLoadsRate() throws {
|
||||
try withTemporaryStore { _, store in
|
||||
store.persistDevinAcuUsdRate(2.25)
|
||||
|
||||
#expect(store.loadDevinAcuUsdRate() == 2.25)
|
||||
}
|
||||
}
|
||||
|
||||
@Test("preserves existing config while adding Devin rate")
|
||||
func preservesExistingConfig() throws {
|
||||
try withTemporaryStore { home, store in
|
||||
try writeConfig([
|
||||
"currency": [
|
||||
"code": "EUR",
|
||||
"symbol": "\u{20AC}"
|
||||
]
|
||||
], in: home)
|
||||
|
||||
store.persistDevinAcuUsdRate(3.5)
|
||||
|
||||
let json = try readConfig(in: home)
|
||||
let currency = try #require(json["currency"] as? [String: Any])
|
||||
let devin = try #require(json["devin"] as? [String: Any])
|
||||
#expect(currency["code"] as? String == "EUR")
|
||||
#expect(devin["acuUsdRate"] as? Double == 3.5)
|
||||
}
|
||||
}
|
||||
|
||||
@Test("ignores invalid rates")
|
||||
func ignoresInvalidRates() throws {
|
||||
try withTemporaryStore { _, store in
|
||||
store.persistDevinAcuUsdRate(1.75)
|
||||
store.persistDevinAcuUsdRate(0)
|
||||
store.persistDevinAcuUsdRate(-2)
|
||||
store.persistDevinAcuUsdRate(.infinity)
|
||||
|
||||
#expect(store.loadDevinAcuUsdRate() == 1.75)
|
||||
}
|
||||
}
|
||||
|
||||
@Test("loads only positive finite numeric rates")
|
||||
func loadsOnlyPositiveFiniteNumericRates() throws {
|
||||
try withTemporaryStore { home, store in
|
||||
try writeConfig(["devin": ["acuUsdRate": 0]], in: home)
|
||||
#expect(store.loadDevinAcuUsdRate() == nil)
|
||||
|
||||
try writeConfig(["devin": ["acuUsdRate": "2.25"]], in: home)
|
||||
#expect(store.loadDevinAcuUsdRate() == nil)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -27,9 +27,6 @@ export type CodeburnConfig = {
|
|||
code: string
|
||||
symbol?: string
|
||||
}
|
||||
devin?: {
|
||||
acuUsdRate?: number
|
||||
}
|
||||
plan?: Plan
|
||||
plans?: PlanConfigMap
|
||||
modelAliases?: Record<string, string>
|
||||
|
|
|
|||
|
|
@ -2,9 +2,8 @@ import { readdir, stat } from "fs/promises";
|
|||
import { basename, join } from "path";
|
||||
import { homedir } from "os";
|
||||
|
||||
import { getShortModelName } from "../models.js";
|
||||
import { calculateCost, getShortModelName } from "../models.js";
|
||||
import { openDatabase } from "../sqlite.js";
|
||||
import { readConfig } from "../config.js";
|
||||
import type {
|
||||
ProbeRoot,
|
||||
Provider,
|
||||
|
|
@ -51,7 +50,6 @@ type ToolCall = {
|
|||
|
||||
type DevinMetadata = {
|
||||
created_at?: string;
|
||||
committed_acu_cost?: number;
|
||||
generation_model?: string;
|
||||
is_user_input?: boolean;
|
||||
num_tokens?: number;
|
||||
|
|
@ -110,7 +108,6 @@ type DevinTelemetry = {
|
|||
};
|
||||
|
||||
type DevinStepExtra = {
|
||||
committed_acu_cost?: number;
|
||||
generation_model?: string;
|
||||
telemetry?: DevinTelemetry;
|
||||
};
|
||||
|
|
@ -152,7 +149,6 @@ type DevinSessionMetadata = {
|
|||
};
|
||||
|
||||
type DevinUsage = {
|
||||
committedAcuCost: number;
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
cacheCreationInputTokens: number;
|
||||
|
|
@ -187,15 +183,6 @@ function parseNumericTimestamp(value: number): string {
|
|||
return new Date(millis).toISOString();
|
||||
}
|
||||
|
||||
function getCommittedAcuCost(step: DevinStep): number {
|
||||
const acuCost = [
|
||||
step.metadata?.committed_acu_cost,
|
||||
step.extra?.committed_acu_cost,
|
||||
].filter((cost) => isPositiveNumber(cost));
|
||||
|
||||
return acuCost.shift() || 0;
|
||||
}
|
||||
|
||||
function hasAnyTokenField(
|
||||
metrics: Metrics<DevinMetricsExtra> | null | undefined,
|
||||
): boolean {
|
||||
|
|
@ -239,11 +226,9 @@ function getDevinMetricsFromMetadata(
|
|||
}
|
||||
|
||||
function getUsage(step: DevinStep): DevinUsage | null {
|
||||
const committedAcuCost = getCommittedAcuCost(step);
|
||||
const metrics = getMetricsFromStep(step);
|
||||
|
||||
const hasAnyUsage = [
|
||||
committedAcuCost,
|
||||
metrics?.prompt_tokens,
|
||||
metrics?.completion_tokens,
|
||||
metrics?.extra?.cache_creation_input_tokens,
|
||||
|
|
@ -252,14 +237,18 @@ function getUsage(step: DevinStep): DevinUsage | null {
|
|||
|
||||
if (!hasAnyUsage) return null;
|
||||
|
||||
const cacheReadInputTokens = safeNumber(metrics?.cached_tokens);
|
||||
|
||||
return {
|
||||
committedAcuCost,
|
||||
inputTokens: safeNumber(metrics?.prompt_tokens),
|
||||
// Devin reports OpenAI-style prompt_tokens, with the cached tokens counted
|
||||
// inside it; Anthropic semantics (what calculateCost expects) keep them
|
||||
// apart, so the cached share is carved out instead of billed twice.
|
||||
inputTokens: Math.max(0, safeNumber(metrics?.prompt_tokens) - cacheReadInputTokens),
|
||||
outputTokens: safeNumber(metrics?.completion_tokens),
|
||||
cacheCreationInputTokens: safeNumber(
|
||||
metrics?.extra?.cache_creation_input_tokens,
|
||||
),
|
||||
cacheReadInputTokens: safeNumber(metrics?.cached_tokens),
|
||||
cacheReadInputTokens,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -337,6 +326,13 @@ function getFriendlyGptName(model: string): string {
|
|||
return reconstructed;
|
||||
}
|
||||
|
||||
// Devin minor versions are always a single digit (gpt-5-3-codex). Restrict the
|
||||
// dash-to-dot rewrite to a single-digit minor at a token boundary so a dated
|
||||
// snapshot like gpt-4-1106-preview is not misread as version 4.1106.
|
||||
function normalizeDevinGptId(model: string): string {
|
||||
return model.replace(/^gpt-(\d+)-(\d)(?=-|$)/, "gpt-$1.$2");
|
||||
}
|
||||
|
||||
function getDevinDisplayModelName(
|
||||
generationModel: string | undefined,
|
||||
modelName: string,
|
||||
|
|
@ -346,10 +342,7 @@ function getDevinDisplayModelName(
|
|||
}
|
||||
|
||||
if (generationModel.startsWith("gpt-")) {
|
||||
// Devin minor versions are always a single digit (gpt-5-3-codex). Restrict
|
||||
// the dash-to-dot rewrite to a single-digit minor at a token boundary so a
|
||||
// dated snapshot like gpt-4-1106-preview is not misread as version 4.1106.
|
||||
const normalized = generationModel.replace(/^gpt-(\d+)-(\d)(?=-|$)/, "gpt-$1.$2");
|
||||
const normalized = normalizeDevinGptId(generationModel);
|
||||
const effortMatch = normalized.match(/-([^-]+)$/);
|
||||
const effort = effortMatch && DEVIN_EFFORT_TIERS.has(effortMatch[1]!)
|
||||
? effortMatch[1]
|
||||
|
|
@ -362,11 +355,11 @@ function getDevinDisplayModelName(
|
|||
return getShortModelName(generationModel);
|
||||
}
|
||||
|
||||
function getModelName(
|
||||
function getModels(
|
||||
transcript: DevinAgentTrajectory,
|
||||
step: DevinStep,
|
||||
session: DevinSessionMetadata | null,
|
||||
): string {
|
||||
): { pricingModel: string; displayModel: string } {
|
||||
const generationModel = firstPresentString(
|
||||
step.metadata?.generation_model,
|
||||
step.extra?.generation_model,
|
||||
|
|
@ -377,7 +370,15 @@ function getModelName(
|
|||
session?.model,
|
||||
) ?? DEFAULT_MODEL_NAME;
|
||||
|
||||
return getDevinDisplayModelName(generationModel, modelName);
|
||||
const pricingModel =
|
||||
!generationModel || /^MODEL_/.test(generationModel)
|
||||
? modelName
|
||||
: normalizeDevinGptId(generationModel);
|
||||
|
||||
return {
|
||||
pricingModel,
|
||||
displayModel: getDevinDisplayModelName(generationModel, modelName),
|
||||
};
|
||||
}
|
||||
|
||||
function getToolNames(step: DevinStep): string[] {
|
||||
|
|
@ -399,13 +400,19 @@ function normalizeStepMessage(message: string | Array<ContentPart>): string {
|
|||
return message.trim();
|
||||
}
|
||||
|
||||
// Real transcripts mark the user's turn with source "user" and leave
|
||||
// is_user_input unset; older ones do the opposite.
|
||||
function isUserStep(step: DevinStep): boolean {
|
||||
return step.metadata?.is_user_input === true || step.source === "user";
|
||||
}
|
||||
|
||||
function getFirstUserMessageBeforeStep(
|
||||
steps: DevinStep[],
|
||||
index: number,
|
||||
): string | null {
|
||||
for (let i = index - 1; i >= 0; i--) {
|
||||
const step = steps[i];
|
||||
if (!step?.metadata?.is_user_input) continue;
|
||||
if (!step || !isUserStep(step)) continue;
|
||||
const message = step.message
|
||||
? normalizeStepMessage(step.message)
|
||||
: undefined;
|
||||
|
|
@ -414,6 +421,25 @@ function getFirstUserMessageBeforeStep(
|
|||
return null;
|
||||
}
|
||||
|
||||
// Devin only fills sessions.title once it has summarised the session, so an
|
||||
// unsummarised session falls back to what the user actually typed first.
|
||||
function loadFirstPrompts(db: ReturnType<typeof openDatabase>): Map<string, string> {
|
||||
const prompts = new Map<string, string>();
|
||||
try {
|
||||
const rows = db.query<{ session_id: string; content: string }>(
|
||||
`SELECT session_id, content FROM prompt_history ORDER BY id`,
|
||||
);
|
||||
for (const row of rows) {
|
||||
const content = row.content?.trim();
|
||||
if (!row.session_id || !content || prompts.has(row.session_id)) continue;
|
||||
prompts.set(row.session_id, content);
|
||||
}
|
||||
} catch {
|
||||
// Older Devin builds have no prompt_history table.
|
||||
}
|
||||
return prompts;
|
||||
}
|
||||
|
||||
function loadSessionMetadata(
|
||||
dbPath: string,
|
||||
): Map<string, DevinSessionMetadata> {
|
||||
|
|
@ -433,13 +459,14 @@ function loadSessionMetadata(
|
|||
`SELECT id, working_directory, model, title, created_at, last_activity_at, hidden
|
||||
FROM sessions`,
|
||||
);
|
||||
const firstPrompts = loadFirstPrompts(db);
|
||||
for (const row of rows) {
|
||||
if (!row.id) continue;
|
||||
sessions.set(row.id, {
|
||||
id: row.id,
|
||||
workingDirectory: row.working_directory,
|
||||
model: row.model,
|
||||
title: row.title ?? undefined,
|
||||
title: row.title?.trim() || firstPrompts.get(row.id),
|
||||
createdAt: parseNumericTimestamp(row.created_at),
|
||||
lastActivityAt: parseNumericTimestamp(row.last_activity_at),
|
||||
hidden: !!row.hidden,
|
||||
|
|
@ -453,11 +480,6 @@ function loadSessionMetadata(
|
|||
return sessions;
|
||||
}
|
||||
|
||||
async function getCostFactor(): Promise<number | null> {
|
||||
const configRate = (await readConfig()).devin?.acuUsdRate;
|
||||
return isPositiveNumber(configRate) ? configRate : null;
|
||||
}
|
||||
|
||||
class DevinSessionParser implements SessionParser {
|
||||
constructor(
|
||||
private source: SessionSource,
|
||||
|
|
@ -478,12 +500,10 @@ class DevinSessionParser implements SessionParser {
|
|||
|
||||
const project = getProjectName(this.source, session);
|
||||
const projectPath = getProjectPath(session);
|
||||
const costFactor = await getCostFactor();
|
||||
if (costFactor === null) return;
|
||||
|
||||
for (let index = 0; index < transcript.steps.length; index++) {
|
||||
const step = transcript.steps[index];
|
||||
if (step.metadata?.is_user_input) continue;
|
||||
if (isUserStep(step)) continue;
|
||||
|
||||
const usage = getUsage(step);
|
||||
if (!usage) continue;
|
||||
|
|
@ -495,14 +515,16 @@ class DevinSessionParser implements SessionParser {
|
|||
if (this.seenKeys.has(deduplicationKey)) continue;
|
||||
this.seenKeys.add(deduplicationKey);
|
||||
|
||||
const model = getModelName(transcript, step, session);
|
||||
const { pricingModel, displayModel } = getModels(transcript, step, session);
|
||||
const tools = getToolNames(step);
|
||||
const userMessage =
|
||||
getFirstUserMessageBeforeStep(transcript.steps, index) ?? "";
|
||||
getFirstUserMessageBeforeStep(transcript.steps, index) ??
|
||||
session?.title ??
|
||||
"";
|
||||
|
||||
yield {
|
||||
provider: DEVIN_PROVIDER_NAME,
|
||||
model,
|
||||
model: displayModel,
|
||||
inputTokens: usage.inputTokens,
|
||||
outputTokens: usage.outputTokens,
|
||||
cacheCreationInputTokens: usage.cacheCreationInputTokens,
|
||||
|
|
@ -510,7 +532,14 @@ class DevinSessionParser implements SessionParser {
|
|||
cachedInputTokens: usage.cacheReadInputTokens,
|
||||
reasoningTokens: 0,
|
||||
webSearchRequests: 0,
|
||||
costUSD: usage.committedAcuCost * costFactor,
|
||||
costUSD: calculateCost(
|
||||
pricingModel,
|
||||
usage.inputTokens,
|
||||
usage.outputTokens,
|
||||
usage.cacheCreationInputTokens,
|
||||
usage.cacheReadInputTokens,
|
||||
0,
|
||||
),
|
||||
tools,
|
||||
bashCommands: [],
|
||||
timestamp,
|
||||
|
|
@ -570,8 +599,6 @@ export function createDevinProvider(cliDir?: string): Provider {
|
|||
},
|
||||
|
||||
async discoverSessions(): Promise<SessionSource[]> {
|
||||
if ((await getCostFactor()) === null) return [];
|
||||
|
||||
const entries = await readdir(transcriptsDir).catch(() => []);
|
||||
const metadata = getSessionMetadata();
|
||||
const sources: SessionSource[] = [];
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@ import { spawnSync } from 'node:child_process'
|
|||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { calculateCost } from '../src/models.js'
|
||||
|
||||
function runCli(args: string[], home: string) {
|
||||
return spawnSync(process.execPath, ['--import', 'tsx', 'src/cli.ts', ...args], {
|
||||
cwd: process.cwd(),
|
||||
|
|
@ -27,11 +29,6 @@ describe('codeburn report Devin model variants', () => {
|
|||
it('keeps friendly Devin effort-tier names in JSON model rows and efficiency rows', async () => {
|
||||
const home = await mkdtemp(join(tmpdir(), 'codeburn-cli-devin-models-'))
|
||||
try {
|
||||
await mkdir(join(home, '.config', 'codeburn'), { recursive: true })
|
||||
await writeFile(join(home, '.config', 'codeburn', 'config.json'), JSON.stringify({
|
||||
devin: { acuUsdRate: 1 },
|
||||
}))
|
||||
|
||||
const transcriptsDir = join(home, '.local', 'share', 'devin', 'cli', 'transcripts')
|
||||
await mkdir(transcriptsDir, { recursive: true })
|
||||
await writeFile(join(transcriptsDir, 'session-487.json'), JSON.stringify({
|
||||
|
|
@ -50,7 +47,6 @@ describe('codeburn report Devin model variants', () => {
|
|||
tool_calls: [{ function_name: 'Edit' }],
|
||||
metadata: {
|
||||
created_at: '2026-04-10T09:01:00.000Z',
|
||||
committed_acu_cost: 0.25,
|
||||
generation_model: 'gpt-5-3-codex-xhigh',
|
||||
metrics: { input_tokens: 100, output_tokens: 25 },
|
||||
},
|
||||
|
|
@ -86,10 +82,10 @@ describe('codeburn report Devin model variants', () => {
|
|||
expect(report.models[0]).toMatchObject({
|
||||
name: 'GPT-5.3 Codex (xhigh)',
|
||||
calls: 1,
|
||||
cost: 0.25,
|
||||
cost: calculateCost('gpt-5.3-codex-xhigh', 100, 25, 0, 0, 0),
|
||||
editTurns: 1,
|
||||
oneShotTurns: 1,
|
||||
costPerEdit: 0.25,
|
||||
costPerEdit: calculateCost('gpt-5.3-codex-xhigh', 100, 25, 0, 0, 0),
|
||||
})
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true })
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import { join } from 'path'
|
|||
import { tmpdir } from 'os'
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
|
||||
import { calculateCost, getModelCosts } from '../../src/models.js'
|
||||
import { isSqliteAvailable } from '../../src/sqlite.js'
|
||||
import { createDevinProvider } from '../../src/providers/devin.js'
|
||||
import type { ParsedProviderCall } from '../../src/providers/types.js'
|
||||
|
|
@ -19,13 +20,6 @@ afterEach(async () => {
|
|||
await rm(tmpDir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
async function configureDevinRate(rate = 1): Promise<void> {
|
||||
await mkdir(join(tmpDir, '.config', 'codeburn'), { recursive: true })
|
||||
await writeFile(join(tmpDir, '.config', 'codeburn', 'config.json'), JSON.stringify({
|
||||
devin: { acuUsdRate: rate },
|
||||
}))
|
||||
}
|
||||
|
||||
async function writeTranscript(name: string, transcript: unknown): Promise<string> {
|
||||
const transcriptsDir = join(tmpDir, 'transcripts')
|
||||
await mkdir(transcriptsDir, { recursive: true })
|
||||
|
|
@ -70,9 +64,44 @@ function createSessionsDb(): void {
|
|||
db.close()
|
||||
}
|
||||
|
||||
function createRealSessionsDb(): void {
|
||||
const { DatabaseSync: Database } = require('node:sqlite')
|
||||
const db = new Database(join(tmpDir, 'sessions.db'))
|
||||
db.exec(`
|
||||
CREATE TABLE sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
working_directory TEXT,
|
||||
backend_type TEXT,
|
||||
model TEXT,
|
||||
agent_mode TEXT,
|
||||
created_at INTEGER,
|
||||
last_activity_at INTEGER,
|
||||
title TEXT,
|
||||
workspace_dirs TEXT,
|
||||
hidden INTEGER NOT NULL DEFAULT 0,
|
||||
metadata TEXT
|
||||
);
|
||||
CREATE TABLE prompt_history (
|
||||
id INTEGER PRIMARY KEY,
|
||||
content TEXT,
|
||||
timestamp INTEGER,
|
||||
session_id TEXT,
|
||||
is_shell INTEGER
|
||||
);
|
||||
`)
|
||||
const insert = db.prepare(`
|
||||
INSERT INTO sessions (id, working_directory, model, created_at, last_activity_at, title, hidden)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
`)
|
||||
insert.run('dandy-edam', '/Users/example/work/codeburn', 'kimi-k3-high', 1_789_645_675, 1_789_646_740, 'Research Code Burn platform and Devin support status', 0)
|
||||
insert.run('chrome-tricorne', '/Users/example/work/scratch', 'swe-2-high', 1_789_645_588, 1_789_645_815, '', 0)
|
||||
db.prepare('INSERT INTO prompt_history (id, content, timestamp, session_id, is_shell) VALUES (?, ?, ?, ?, ?)')
|
||||
.run(1, 'fix the failing ledger tests', 1_789_645_588, 'chrome-tricorne', 0)
|
||||
db.close()
|
||||
}
|
||||
|
||||
describe('devin provider', () => {
|
||||
it('discovers Devin CLI transcript json files', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('glimmer-platinum.json', { steps: [] })
|
||||
await writeFile(join(tmpDir, 'transcripts', 'ignore.txt'), '{}')
|
||||
|
||||
|
|
@ -84,19 +113,7 @@ describe('devin provider', () => {
|
|||
])
|
||||
})
|
||||
|
||||
it('stays disabled until the Devin ACU rate is configured', async () => {
|
||||
await writeTranscript('glimmer-platinum.json', {
|
||||
session_id: 'session-123',
|
||||
steps: [{ step_id: 's1', metadata: { committed_acu_cost: 0.5 } }],
|
||||
})
|
||||
|
||||
const provider = createDevinProvider(tmpDir)
|
||||
expect(await provider.discoverSessions()).toEqual([])
|
||||
expect(await parseTranscript(join(tmpDir, 'transcripts', 'glimmer-platinum.json'))).toEqual([])
|
||||
})
|
||||
|
||||
it('parses per-step ACUs, tokens, tools, and model resolution', async () => {
|
||||
await configureDevinRate()
|
||||
it('parses per-step tokens, cost, tools, and model resolution', async () => {
|
||||
const filePath = await writeTranscript('glimmer-platinum.json', {
|
||||
schema_version: '1',
|
||||
session_id: 'session-123',
|
||||
|
|
@ -112,7 +129,6 @@ describe('devin provider', () => {
|
|||
model_name: 'step-model',
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.02076149918138981,
|
||||
generation_model: 'claude-opus-4-6',
|
||||
metrics: {
|
||||
input_tokens: 100,
|
||||
|
|
@ -128,7 +144,6 @@ describe('devin provider', () => {
|
|||
model_name: 'claude-sonnet-4-6',
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:02.000Z',
|
||||
committed_acu_cost: 0.005421000067144632,
|
||||
metrics: { input_tokens: 1 },
|
||||
},
|
||||
tool_calls: [{ function_name: 'str_replace' }],
|
||||
|
|
@ -139,16 +154,19 @@ describe('devin provider', () => {
|
|||
const calls = await parseTranscript(filePath)
|
||||
|
||||
expect(calls).toHaveLength(2)
|
||||
expect(calls.reduce((sum, call) => sum + call.costUSD, 0)).toBeCloseTo(0.026182499248534442, 15)
|
||||
expect(calls.reduce((sum, call) => sum + call.costUSD, 0)).toBeCloseTo(
|
||||
calculateCost('claude-opus-4-6', 95, 20, 10, 5, 0) + calculateCost('claude-sonnet-4-6', 1, 0, 0, 0, 0),
|
||||
15,
|
||||
)
|
||||
expect(calls[0]).toMatchObject({
|
||||
provider: 'devin',
|
||||
model: 'Opus 4.6',
|
||||
inputTokens: 100,
|
||||
inputTokens: 95,
|
||||
outputTokens: 20,
|
||||
cacheCreationInputTokens: 10,
|
||||
cacheReadInputTokens: 5,
|
||||
cachedInputTokens: 5,
|
||||
costUSD: 0.02076149918138981,
|
||||
costUSD: calculateCost('claude-opus-4-6', 95, 20, 10, 5, 0),
|
||||
tools: ['read_file'],
|
||||
timestamp: '2027-01-15T08:00:01.000Z',
|
||||
deduplicationKey: 'devin:session-123:2',
|
||||
|
|
@ -164,7 +182,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('renders Devin generation_model variants as friendly display names with effort tiers', async () => {
|
||||
await configureDevinRate()
|
||||
const cases = [
|
||||
{
|
||||
schema: '1.4',
|
||||
|
|
@ -249,7 +266,6 @@ describe('devin provider', () => {
|
|||
const row = cases[index]!
|
||||
const metadata: Record<string, unknown> = {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 1 },
|
||||
}
|
||||
const extra: Record<string, unknown> = {}
|
||||
|
|
@ -280,6 +296,29 @@ describe('devin provider', () => {
|
|||
}
|
||||
})
|
||||
|
||||
it('prices a dashed gpt generation_model at its own row, not the base gpt-5 row', async () => {
|
||||
const filePath = await writeTranscript('codex-id.json', {
|
||||
session_id: 'codex-id-session',
|
||||
steps: [
|
||||
{
|
||||
step_id: 1,
|
||||
source: 'agent',
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:00.000Z',
|
||||
generation_model: 'gpt-5-3-codex-xhigh',
|
||||
metrics: { input_tokens: 1_000_000 },
|
||||
},
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const calls = await parseTranscript(filePath)
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.costUSD).toBe(calculateCost('gpt-5.3-codex-xhigh', 1_000_000, 0, 0, 0, 0))
|
||||
expect(calls[0]!.costUSD).not.toBe(calculateCost('gpt-5-3-codex-xhigh', 1_000_000, 0, 0, 0, 0))
|
||||
})
|
||||
|
||||
it('leaves already-friendly Devin model display names unchanged', () => {
|
||||
const provider = createDevinProvider(tmpDir)
|
||||
|
||||
|
|
@ -287,7 +326,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('includes token-only steps and skips user-input or empty steps', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('token-only.json', {
|
||||
session_id: 'token-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -296,7 +334,6 @@ describe('devin provider', () => {
|
|||
step_id: 'user-cost',
|
||||
metadata: {
|
||||
is_user_input: true,
|
||||
committed_acu_cost: 99,
|
||||
metrics: { input_tokens: 99 },
|
||||
},
|
||||
},
|
||||
|
|
@ -319,38 +356,21 @@ describe('devin provider', () => {
|
|||
expect(calls[0]!.costUSD).toBe(0)
|
||||
})
|
||||
|
||||
it('converts ACUs to costUSD using the configured Devin rate', async () => {
|
||||
await configureDevinRate(2.5)
|
||||
const filePath = await writeTranscript('configured-rate.json', {
|
||||
session_id: 'configured-rate',
|
||||
agent: { model_name: 'agent-model' },
|
||||
steps: [
|
||||
{ step_id: 's1', metadata: { committed_acu_cost: 0.4 } },
|
||||
],
|
||||
})
|
||||
|
||||
const calls = await parseTranscript(filePath)
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.costUSD).toBeCloseTo(1, 12)
|
||||
})
|
||||
|
||||
it('falls back to filename session id and deduplicates by step id', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('fallback-session.json', {
|
||||
steps: [
|
||||
{
|
||||
step_id: 1,
|
||||
metadata: {
|
||||
request_id: 'req-1',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 10 },
|
||||
},
|
||||
},
|
||||
{
|
||||
step_id: 2,
|
||||
metadata: {
|
||||
created_at: '2026-06-05T10:00:00.000Z',
|
||||
committed_acu_cost: 0.2,
|
||||
metrics: { input_tokens: 20 },
|
||||
},
|
||||
},
|
||||
],
|
||||
|
|
@ -367,7 +387,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('extracts user message from ContentPart[] messages (ATIF v1.7 multimodal)', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('content-parts.json', {
|
||||
session_id: 'cp-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -384,7 +403,6 @@ describe('devin provider', () => {
|
|||
step_id: 2,
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 50 },
|
||||
},
|
||||
},
|
||||
|
|
@ -398,7 +416,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('parses ATIF v1.7 transcripts with agent.extra, final_metrics, and observations', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('atif-v17.json', {
|
||||
schema_version: '1.7',
|
||||
session_id: 'v17-session',
|
||||
|
|
@ -430,13 +447,11 @@ describe('devin provider', () => {
|
|||
results: [{ source_call_id: 'tc1', content: 'file contents here' }],
|
||||
},
|
||||
extra: {
|
||||
committed_acu_cost: 0.15,
|
||||
generation_model: 'claude-sonnet-4-6',
|
||||
telemetry: { source: 'devin-cli', operation: 'generate' },
|
||||
},
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.15,
|
||||
metrics: { input_tokens: 200, output_tokens: 50, cache_creation_tokens: 20, cache_read_tokens: 10 },
|
||||
},
|
||||
},
|
||||
|
|
@ -449,11 +464,11 @@ describe('devin provider', () => {
|
|||
expect(calls[0]).toMatchObject({
|
||||
provider: 'devin',
|
||||
model: 'Sonnet 4.6',
|
||||
inputTokens: 200,
|
||||
inputTokens: 190,
|
||||
outputTokens: 50,
|
||||
cacheCreationInputTokens: 20,
|
||||
cacheReadInputTokens: 10,
|
||||
costUSD: 0.15,
|
||||
costUSD: calculateCost('claude-sonnet-4-6', 190, 50, 20, 10, 0),
|
||||
tools: ['read_file'],
|
||||
userMessage: 'fix the bug',
|
||||
sessionId: 'v17-session',
|
||||
|
|
@ -461,7 +476,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('handles plain string user messages alongside ContentPart[] messages', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('mixed-messages.json', {
|
||||
session_id: 'mixed-msg-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -475,7 +489,6 @@ describe('devin provider', () => {
|
|||
step_id: 2,
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 50 },
|
||||
},
|
||||
},
|
||||
|
|
@ -488,7 +501,6 @@ describe('devin provider', () => {
|
|||
step_id: 4,
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:03.000Z',
|
||||
committed_acu_cost: 0.2,
|
||||
metrics: { input_tokens: 100 },
|
||||
},
|
||||
},
|
||||
|
|
@ -502,55 +514,7 @@ describe('devin provider', () => {
|
|||
expect(calls[1]!.userMessage).toBe('multimodal user message')
|
||||
})
|
||||
|
||||
it('reads ACU cost from step.extra when metadata.committed_acu_cost is absent', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('extra-acu.json', {
|
||||
session_id: 'extra-acu-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
steps: [
|
||||
{
|
||||
step_id: 1,
|
||||
extra: { committed_acu_cost: 0.3 },
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:00.000Z',
|
||||
metrics: { input_tokens: 10 },
|
||||
},
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const calls = await parseTranscript(filePath)
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.costUSD).toBeCloseTo(0.3, 12)
|
||||
})
|
||||
|
||||
it('prefers metadata.committed_acu_cost over extra.committed_acu_cost', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('acu-priority.json', {
|
||||
session_id: 'acu-priority-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
steps: [
|
||||
{
|
||||
step_id: 1,
|
||||
extra: { committed_acu_cost: 0.99 },
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:00.000Z',
|
||||
committed_acu_cost: 0.11,
|
||||
metrics: { input_tokens: 10 },
|
||||
},
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const calls = await parseTranscript(filePath)
|
||||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]!.costUSD).toBeCloseTo(0.11, 12)
|
||||
})
|
||||
|
||||
it('reads tokens from step.metrics when metadata.metrics is absent', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('step-metrics.json', {
|
||||
session_id: 'step-metrics-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -563,7 +527,6 @@ describe('devin provider', () => {
|
|||
cached_tokens: 15,
|
||||
extra: { cache_creation_input_tokens: 25 },
|
||||
},
|
||||
extra: { committed_acu_cost: 0.2 },
|
||||
metadata: { created_at: '2027-01-15T08:00:00.000Z' },
|
||||
},
|
||||
],
|
||||
|
|
@ -573,17 +536,16 @@ describe('devin provider', () => {
|
|||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({
|
||||
inputTokens: 300,
|
||||
inputTokens: 285,
|
||||
outputTokens: 75,
|
||||
cacheCreationInputTokens: 25,
|
||||
cacheReadInputTokens: 15,
|
||||
cachedInputTokens: 15,
|
||||
costUSD: 0.2,
|
||||
costUSD: 0,
|
||||
})
|
||||
})
|
||||
|
||||
it('prefers step.metrics over metadata.metrics when both are present', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('metrics-priority.json', {
|
||||
session_id: 'metrics-priority-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -598,7 +560,6 @@ describe('devin provider', () => {
|
|||
},
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:00.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: {
|
||||
input_tokens: 1,
|
||||
output_tokens: 1,
|
||||
|
|
@ -614,7 +575,7 @@ describe('devin provider', () => {
|
|||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({
|
||||
inputTokens: 500,
|
||||
inputTokens: 480,
|
||||
outputTokens: 100,
|
||||
cacheCreationInputTokens: 30,
|
||||
cacheReadInputTokens: 20,
|
||||
|
|
@ -623,7 +584,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('handles observation results with ContentPart[] content', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('observation-content-parts.json', {
|
||||
session_id: 'obs-cp-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -649,7 +609,6 @@ describe('devin provider', () => {
|
|||
},
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 100, output_tokens: 30 },
|
||||
},
|
||||
},
|
||||
|
|
@ -661,14 +620,13 @@ describe('devin provider', () => {
|
|||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({
|
||||
tools: ['read_file'],
|
||||
costUSD: 0.1,
|
||||
costUSD: 0,
|
||||
inputTokens: 100,
|
||||
outputTokens: 30,
|
||||
})
|
||||
})
|
||||
|
||||
it('falls back to metadata.metrics when step.metrics is present but empty', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('empty-step-metrics.json', {
|
||||
session_id: 'empty-metrics-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -678,7 +636,6 @@ describe('devin provider', () => {
|
|||
metrics: {},
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:00.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 80, output_tokens: 20, cache_read_tokens: 5 },
|
||||
},
|
||||
},
|
||||
|
|
@ -689,15 +646,14 @@ describe('devin provider', () => {
|
|||
|
||||
expect(calls).toHaveLength(1)
|
||||
expect(calls[0]).toMatchObject({
|
||||
inputTokens: 80,
|
||||
inputTokens: 75,
|
||||
outputTokens: 20,
|
||||
cacheReadInputTokens: 5,
|
||||
costUSD: 0.1,
|
||||
costUSD: 0,
|
||||
})
|
||||
})
|
||||
|
||||
it('normalizes an image-only ContentPart[] user message to its path', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('image-only.json', {
|
||||
session_id: 'image-only-session',
|
||||
agent: { model_name: 'agent-model' },
|
||||
|
|
@ -713,7 +669,6 @@ describe('devin provider', () => {
|
|||
step_id: 2,
|
||||
metadata: {
|
||||
created_at: '2027-01-15T08:00:01.000Z',
|
||||
committed_acu_cost: 0.1,
|
||||
metrics: { input_tokens: 40 },
|
||||
},
|
||||
},
|
||||
|
|
@ -727,7 +682,6 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('ignores array-root and malformed transcripts', async () => {
|
||||
await configureDevinRate()
|
||||
const arrayPath = await writeTranscript('array.json', [])
|
||||
const malformedPath = join(tmpDir, 'transcripts', 'bad.json')
|
||||
await writeFile(malformedPath, '{')
|
||||
|
|
@ -737,10 +691,9 @@ describe('devin provider', () => {
|
|||
})
|
||||
|
||||
it('deduplicates calls with a shared seen key set', async () => {
|
||||
await configureDevinRate()
|
||||
const filePath = await writeTranscript('dupe.json', {
|
||||
session_id: 'dupe-session',
|
||||
steps: [{ step_id: 's1', metadata: { committed_acu_cost: 0.5 } }],
|
||||
steps: [{ step_id: 's1', metadata: { metrics: { input_tokens: 10 } } }],
|
||||
})
|
||||
const provider = createDevinProvider(tmpDir)
|
||||
const seenKeys = new Set<string>()
|
||||
|
|
@ -760,7 +713,6 @@ const skipUnlessSqlite = isSqliteAvailable() ? describe : describe.skip
|
|||
|
||||
skipUnlessSqlite('devin provider sessions.db enrichment', () => {
|
||||
it('uses sessions.db to enrich project, projectPath, model, and timestamp fallbacks', async () => {
|
||||
await configureDevinRate()
|
||||
createSessionsDb()
|
||||
const filePath = await writeTranscript('db-session.json', {
|
||||
session_id: 'db-session',
|
||||
|
|
@ -768,7 +720,6 @@ skipUnlessSqlite('devin provider sessions.db enrichment', () => {
|
|||
{
|
||||
step_id: 's1',
|
||||
metadata: {
|
||||
committed_acu_cost: 0.25,
|
||||
metrics: { input_tokens: 10 },
|
||||
},
|
||||
},
|
||||
|
|
@ -783,12 +734,11 @@ skipUnlessSqlite('devin provider sessions.db enrichment', () => {
|
|||
project: 'codeburn',
|
||||
projectPath: '/Users/example/work/codeburn',
|
||||
timestamp: '2027-01-15T08:00:10.000Z',
|
||||
costUSD: 0.25,
|
||||
costUSD: calculateCost('claude-sonnet-4-6', 10, 0, 0, 0, 0),
|
||||
})
|
||||
})
|
||||
|
||||
it('uses sessions.db project labels during discovery when transcript filename matches the session id', async () => {
|
||||
await configureDevinRate()
|
||||
createSessionsDb()
|
||||
const filePath = await writeTranscript('db-session.json', { session_id: 'db-session', steps: [] })
|
||||
|
||||
|
|
@ -801,11 +751,10 @@ skipUnlessSqlite('devin provider sessions.db enrichment', () => {
|
|||
})
|
||||
|
||||
it('skips sessions hidden in sessions.db', async () => {
|
||||
await configureDevinRate()
|
||||
createSessionsDb()
|
||||
await writeTranscript('hidden-session.json', {
|
||||
session_id: 'hidden-session',
|
||||
steps: [{ step_id: 's1', metadata: { committed_acu_cost: 0.25 } }],
|
||||
steps: [{ step_id: 's1', metadata: { metrics: { input_tokens: 10 } } }],
|
||||
})
|
||||
|
||||
const provider = createDevinProvider(tmpDir)
|
||||
|
|
@ -814,4 +763,113 @@ skipUnlessSqlite('devin provider sessions.db enrichment', () => {
|
|||
const calls = await parseTranscript(join(tmpDir, 'transcripts', 'hidden-session.json'))
|
||||
expect(calls).toEqual([])
|
||||
})
|
||||
it('prices a real-shape transcript per step from tokens with no config at all', async () => {
|
||||
createRealSessionsDb()
|
||||
// Real dandy-edam step metrics: [generation_model, prompt, completion, cached].
|
||||
const steps = [
|
||||
['swe-2-high', 18009, 216, 10752],
|
||||
['swe-2-high', 19710, 142, 17920],
|
||||
['swe-2-high', 20018, 59, 19456],
|
||||
['swe-2-high', 25876, 170, 19968],
|
||||
['swe-2-high', 27719, 144, 25600],
|
||||
['swe-2-high', 31464, 510, 27648],
|
||||
['swe-2-high', 35889, 227, 31744],
|
||||
['swe-2-high', 36355, 303, 35840],
|
||||
['kimi-k3-high', 38273, 1041, 0],
|
||||
['kimi-k3-high', 39349, 444, 35328],
|
||||
['kimi-k3-high', 41494, 269, 36864],
|
||||
['kimi-k3-high', 43569, 121, 39936],
|
||||
['kimi-k3-high', 44174, 781, 41472],
|
||||
['kimi-k3-high', 45003, 848, 41472],
|
||||
['kimi-k3-high', 46676, 93, 43008],
|
||||
['kimi-k3-high', 47578, 133, 44544],
|
||||
['kimi-k3-high', 49263, 966, 44544],
|
||||
] as const
|
||||
|
||||
const filePath = await writeTranscript('dandy-edam.json', {
|
||||
schema_version: 'ATIF-v1.7',
|
||||
session_id: 'dandy-edam',
|
||||
agent: { name: 'devin', version: '3000.10.31', model_name: 'Kimi K3 High' },
|
||||
final_metrics: {
|
||||
total_prompt_tokens: 610_419,
|
||||
total_completion_tokens: 6467,
|
||||
total_cached_tokens: 516_096,
|
||||
total_steps: 28,
|
||||
},
|
||||
steps: [
|
||||
{ step_id: 1, source: 'system', message: 'You are Devin.' },
|
||||
{ step_id: 2, source: 'user', message: 'do we support devin in codeburn' },
|
||||
...steps.map(([model, prompt, completion, cached], index) => ({
|
||||
step_id: index + 3,
|
||||
source: 'agent',
|
||||
message: 'working',
|
||||
tool_calls: [{ tool_call_id: `tc${index}`, function_name: 'shell', arguments: {} }],
|
||||
extra: { generation_model: model },
|
||||
metrics: {
|
||||
prompt_tokens: prompt,
|
||||
completion_tokens: completion,
|
||||
...(cached ? { cached_tokens: cached } : {}),
|
||||
},
|
||||
})),
|
||||
],
|
||||
})
|
||||
|
||||
const provider = createDevinProvider(tmpDir)
|
||||
// chrome-tricorne is in sessions.db but has no transcript on disk.
|
||||
expect(await provider.discoverSessions()).toEqual([
|
||||
{ path: filePath, project: 'codeburn', provider: 'devin' },
|
||||
])
|
||||
|
||||
const calls = await parseTranscript(filePath)
|
||||
|
||||
expect(calls).toHaveLength(17)
|
||||
expect(calls.map(call => call.model)).toEqual([
|
||||
...Array(8).fill('swe-2-high'),
|
||||
...Array(9).fill('Kimi K3'),
|
||||
])
|
||||
const sum = (pick: (call: ParsedProviderCall) => number) => calls.reduce((total, call) => total + pick(call), 0)
|
||||
expect(sum(c => c.inputTokens) + sum(c => c.cacheReadInputTokens)).toBe(610_419)
|
||||
expect(sum(c => c.outputTokens)).toBe(6467)
|
||||
expect(sum(c => c.cacheReadInputTokens)).toBe(516_096)
|
||||
|
||||
const kimiRates = getModelCosts('kimi-k3')!
|
||||
const expectedKimiCost = steps
|
||||
.filter(([model]) => model === 'kimi-k3-high')
|
||||
.reduce((total, [, prompt, completion, cached]) =>
|
||||
total
|
||||
+ (prompt - cached) * kimiRates.inputCostPerToken
|
||||
+ completion * kimiRates.outputCostPerToken
|
||||
+ cached * kimiRates.cacheReadCostPerToken, 0)
|
||||
|
||||
expect(expectedKimiCost).toBeGreaterThan(0)
|
||||
expect(sum(c => (c.model === 'Kimi K3' ? c.costUSD : 0))).toBeCloseTo(expectedKimiCost, 12)
|
||||
// swe-2-high has no pricing table entry: unpriced, never silently priced.
|
||||
expect(getModelCosts('swe-2-high')).toBeNull()
|
||||
expect(calls.filter(c => c.model === 'swe-2-high').every(c => c.costUSD === 0)).toBe(true)
|
||||
|
||||
expect(calls[0]).toMatchObject({
|
||||
project: 'codeburn',
|
||||
projectPath: '/Users/example/work/codeburn',
|
||||
sessionId: 'dandy-edam',
|
||||
tools: ['shell'],
|
||||
// The transcript's own user turn, marked source: 'user', beats the title.
|
||||
userMessage: 'do we support devin in codeburn',
|
||||
})
|
||||
})
|
||||
|
||||
it('falls back to the sessions.db title, then prompt_history, for task text', async () => {
|
||||
createRealSessionsDb()
|
||||
const untitled = await writeTranscript('chrome-tricorne.json', {
|
||||
session_id: 'chrome-tricorne',
|
||||
steps: [{ step_id: 1, source: 'agent', metrics: { prompt_tokens: 10, completion_tokens: 2 } }],
|
||||
})
|
||||
const titled = await writeTranscript('dandy-edam.json', {
|
||||
session_id: 'dandy-edam',
|
||||
steps: [{ step_id: 1, source: 'agent', metrics: { prompt_tokens: 10, completion_tokens: 2 } }],
|
||||
})
|
||||
|
||||
expect((await parseTranscript(titled))[0]!.userMessage).toBe('Research Code Burn platform and Devin support status')
|
||||
expect((await parseTranscript(untitled))[0]!.userMessage).toBe('fix the failing ledger tests')
|
||||
})
|
||||
})
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue