Commit graph

9 commits

Author SHA1 Message Date
Aditya Vikram Singh
5ba787346b fix(compare): honor --model-a/--model-b in the TUI and resolve display names
The TUI comparison picker silently ignored --model-a/--model-b (only the
--format json path used them), and both paths required the canonical model
id verbatim with no way to pass the display name the picker itself shows
(e.g. "Opus 4.8" for claude-opus-4-8).

Adds findModelStat, a shared lookup that matches the canonical id first and
falls back to getShortModelName's existing canonical -> display mapping
(case-insensitive) rather than a new alias table. Both the JSON path and
renderCompare/CompareView now use it; renderCompare resolves --model-a/
--model-b up front and seeds CompareView's picked-models state so results
load immediately instead of showing the picker. Also validates "both or
neither" for the TUI (default) format, matching the JSON path.

Fixes getagentseal/codeburn#767 (item 1).
2026-07-29 22:58:05 +05:30
iamtoruk
44d2449d30 fix(cli): compare cache-hit rate excludes cache writes
Compare defined cache-hit as read/(input+read+write) while menubar-json
and every other surface use read/(input+read), so the same model showed
two different numbers depending on screen. Standardize on the latter;
TUI compare output changes intentionally.
2026-07-16 01:14:41 -07:00
iamtoruk
fc1920d4f1 feat(cli): add compare and sessions JSON emitters
compare --format json: list mode (aggregateModelStats for pickers) +
full mode via --model-a/--model-b, emitting
{period, modelA, modelB, metrics, categories, workingStyle}.
Reuses computeComparison/computeCategoryComparison/computeWorkingStyle/
scanSelfCorrections; TUI path unchanged.

sessions --format json: new src/sessions-report.ts, uncapped flat
SessionRow[] (id, project, provider, models, cost, tokens, turns,
duration) via aggregateSessions; table + json formats.

Enables the desktop app's Sessions + Compare screens (Phase 3b).
2026-07-11 12:16:45 -07:00
voidborne-d
c16b21ec50 fix(classifier): surface skill name as subCategory for general turns (#203)
Turns whose only assistant tool is `Skill` collapse to category `general`
because `classifyByToolPattern` returns `'general'` and `refineByKeywords`
only operates on `coding`/`exploration`. In environments that lean on Claude
Code skills, the per-activity dashboard column flattens — every `/init`,
`/review`, `/security-review`, `/claude-api`, plus user-defined skills, all
land in `general` with no signal about which workflow ran.

Implements Option A from the issue:

- `ParsedApiCall.skills: string[]` populated in the Anthropic-path parser
  via a new `extractSkillNames` helper that reads `input.skill || input.name`
  from each `Skill` ToolUseBlock (mirrors `detectGhostSkills` extraction at
  optimize.ts:765 so the two stay in sync).
- `ClassifiedTurn.subCategory?: string` set to the first skill name when the
  resolved category is `general` AND any skill identifier was extracted.
  Top-level category stays `general` — existing aggregations, exports, and
  category-keyed code paths unchanged.
- `SessionSummary.skillBreakdown: Record<string, {turns,costUSD,editTurns,
  oneShotTurns}>` populated in the same per-turn loop that builds
  `categoryBreakdown`. Provider sessions (Codex/Cursor/etc.) keep `skills:
  []` — they don't expose the Skill tool surface today.
- Dashboard `ActivityBreakdown` renders top-N skill sub-rows beneath the
  `general` row when present (indented `/skill-name`, dimmed). Other
  categories render exactly as before; if no skills were invoked, the panel
  is byte-identical to current output.

Existing 419 tests still pass. New `tests/classifier.test.ts` adds 8 cases:
single skill via `input.skill`, single via `input.name`, first-wins for
multi-skill turns, aggregation across multiple assistant calls in one turn,
no-name fallback (`subCategory` stays undefined), `Skill+Edit` promoting to
`coding` and dropping subCategory, non-Skill general turns, and a legacy
ParsedApiCall shape with `skills` field absent (forward-compat). Pre-fix
verification by stashing the source change reproduces 4/8 failures with the
exact "expected 'init', received undefined" diff; restoring → 8/8 pass.

Closes #203.

🤖 AI assistance disclosure: assistant-scaffolded by Claude (Opus 4.7);
author of record reviewed every line, ran the full vitest suite locally
(`npm test` → 32 files / 427 tests pass), `npx tsc --noEmit` clean, and
`npm run build` produces a clean ESM bundle.
2026-05-04 06:26:45 +08:00
iamtoruk
bd43b15342 feat(compare): model comparison with planning rate fix
5-section compare view: Performance (one-shot, retry, self-correction),
Efficiency (cost/call, cost/edit, output/call, cache hit), Category
Head-to-Head bar charts, Working Style, and Context.

Planning rate now detects TaskCreate/TaskUpdate/TodoWrite instead of
only EnterPlanMode (which was never used, showing 0% for all models).
Validated against raw JSONL with zero false positives.

Responsive side-by-side layout at 90+ cols. Self-correction scanner
with compact file skipping and model+timestamp dedup. 274 tests.
2026-04-19 08:34:49 -07:00
iamtoruk
fb24eea186 fix(compare): refine self-correction patterns, skip compact files, deduplicate
Remove high-false-positive patterns (I'm sorry, I should have, sorry for).
Add precise patterns (you're right I, that was incorrect, let me correct).
Skip compact JSONL files that replay compressed context.
Deduplicate by model+timestamp to prevent double-counting.
Fix test timestamps to work with deduplication.
2026-04-19 07:14:02 -07:00
iamtoruk
3cb9a7a7bc feat(compare): add self-correction JSONL scanner
Adds scanSelfCorrections() which reads raw .jsonl session files (including subagent dirs) and counts per-model self-correction patterns for use in the model comparison metrics.
2026-04-19 05:25:31 -07:00
iamtoruk
ac9afffed5 feat(compare): add computeComparison with normalized metrics 2026-04-19 05:22:34 -07:00
iamtoruk
9d119bfe40 feat(compare): add ModelStats type and aggregateModelStats 2026-04-19 05:20:37 -07:00