Merge 68488e8e66 into 1aa05bdbbe

docs/providers/devin.md

@@ -4,7 +4,7 @@ Cognition Devin CLI local usage tracking.
 
 - **Source:** `src/providers/devin.ts`
 - **Loading:** eager (`src/providers/index.ts`)
-- **Test:** `tests/providers/devin.test.ts` (336 lines)
+- **Test:** `tests/providers/devin.test.ts`
 
 ## Where it reads from
 
@@ -59,39 +59,53 @@ appear in CLI/UI results until configured.
 
 ## Storage format
 
-Transcript root is a JSON object following the [ATIF-v1.4 trajectory schema][atif],
-with Devin-specific additions such as per-step `metadata`. The parser does not
-validate `schema_version`; it only requires a parseable object with `steps[]`.
+Transcript root is a JSON object following the [ATIF-v1.7 trajectory schema][atif],
+with Devin-specific additions such as per-step `metadata` and `extra`. The
+parser does not validate `schema_version`; it only requires a parseable object
+with `steps[]`.
 
-Core fields include `session_id`, `agent.model_name`, and `steps[]`.
+Core fields include `session_id`, `agent.model_name`, `agent.extra` (Devin
+backend/permission info), `final_metrics`, and `steps[]`.
+
+Steps now support two metric sources. The parser checks `step.metrics` first
+(the standard ATIF location) and falls back to `step.metadata.metrics` (the
+legacy Devin location). Similarly, ACU cost is read from
+`step.metadata.committed_acu_cost` first, falling back to
+`step.extra.committed_acu_cost`.
+
+Messages can be a plain string or an array of `ContentPart` objects (text or
+image), following the ATIF v1.6+ multimodal content model. The parser
+normalises both forms when extracting user messages.
 
 Each counted step can provide:
 
 - `step_id`
-- `metadata.committed_acu_cost`
-- `metadata.metrics.input_tokens`
-- `metadata.metrics.output_tokens`
-- `metadata.metrics.cache_creation_tokens`
-- `metadata.metrics.cache_read_tokens`
+- `metadata.committed_acu_cost` (or `extra.committed_acu_cost`)
+- `metrics.prompt_tokens` (or `metadata.metrics.input_tokens`)
+- `metrics.completion_tokens` (or `metadata.metrics.output_tokens`)
+- `metrics.extra.cache_creation_input_tokens` (or `metadata.metrics.cache_creation_tokens`)
+- `metrics.cached_tokens` (or `metadata.metrics.cache_read_tokens`)
 - `metadata.created_at`
-- `metadata.generation_model`
+- `metadata.generation_model` (or `extra.generation_model`)
 - `metadata.request_id`
 - `tool_calls[].function_name`
+- `observation.results[]` (tool output; not parsed for usage)
 
 User-input steps (`metadata.is_user_input === true`) are skipped. Non-user
 steps are included only if they have positive ACU usage or positive token usage.
 
 ## Pricing
 
-`metadata.committed_acu_cost` is per step, not cumulative. The provider converts
-each step with:
+ACU cost is per step, not cumulative. The provider reads
+`metadata.committed_acu_cost` first, falling back to
+`extra.committed_acu_cost`, then converts with:
 
 ```text
 costUSD = committed_acu_cost * devin.acuUsdRate
 ```
 
 Token-only steps are still included when they have positive token metrics, but
-their `costUSD` is `0` if `committed_acu_cost` is absent.
+their `costUSD` is `0` if `committed_acu_cost` is absent from both locations.
 
 `src/parser.ts` preserves Devin's provider-supplied `costUSD` instead of
 re-pricing it through LiteLLM.
@@ -150,7 +164,9 @@ The provider name is part of the key via the `devin:` prefix.
 ## Quirks
 
 - The transcript directory has usage; `sessions.db` is enrichment only.
-- `committed_acu_cost` is per-generation/per-step ACU usage. Never treat it as cumulative.
+- `committed_acu_cost` is per-generation/per-step ACU usage. Never treat it as cumulative. It can appear in `metadata` (legacy) or `extra` (ATIF v1.7); the provider checks both.
+- Token metrics can live in `step.metrics` (standard ATIF) or `step.metadata.metrics` (legacy Devin). The provider checks `step.metrics` first, falling back to `metadata`.
+- Step messages can be a plain string or an array of `ContentPart` objects (text/image). The parser normalises both when extracting user messages.
 - There is no default ACU-to-USD rate. Missing config intentionally hides Devin.
 - Hidden sessions from `sessions.db` are skipped in discovery and parsing.
 - Tool names come directly from `tool_calls[].function_name`; the provider assumes valid ATIF tool-call records.
@@ -160,16 +176,16 @@ The provider name is part of the key via the `devin:` prefix.
 
 1. First check whether `~/.config/codeburn/config.json` contains a valid
    `devin.acuUsdRate`. Without it, no Devin sessions should appear.
-2. For usage total bugs, compare against:
+2. For usage total bugs, compare against (ACU cost can live in `metadata` or `extra`):
 
    ```bash
-   jq '[.steps[] | select(.metadata.committed_acu_cost != null) | .metadata.committed_acu_cost] | add' ~/.local/share/devin/cli/transcripts/<session>.json
+   jq '[.steps[] | select(.metadata.is_user_input != true) | (.metadata.committed_acu_cost // .extra.committed_acu_cost // 0)] | add' ~/.local/share/devin/cli/transcripts/<session>.json
    ```
 
 3. If project/model/timestamp metadata is wrong, inspect `sessions.db`, not the transcript.
 4. If a hidden session appears, check the `hidden` column. Discovery can only
    hide sessions whose transcript filename matches `sessions.id`; parsing uses
    the transcript `session_id` when present.
-5. Run `tests/providers/devin.test.ts` after parser changes. It covers ACU conversion, disabled-until-configured behavior, timestamp parsing, deduplication, hidden sessions, and `sessions.db` enrichment.
+5. Run `tests/providers/devin.test.ts` after parser changes. It covers ACU conversion, disabled-until-configured behavior, timestamp parsing, deduplication, hidden sessions, `sessions.db` enrichment, ATIF v1.7 multimodal messages, `step.metrics` vs `metadata.metrics` priority, and `extra.committed_acu_cost` fallback.
 
 [atif]: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md

src/providers/devin.ts

@@ -13,18 +13,45 @@ import type {
 } from "./types.js";
 import { readSessionFile } from "../fs-utils.js";
 import { isPositiveNumber, safeNumber } from "../parser.js";
+import { JsonSchemaType } from "@modelcontextprotocol/sdk/validation";
 
-type AgentTrajectory<T extends Step> = {
+type AgentTrajectory<StepType extends Step = Step, AgentExtra = unknown> = {
   schema_version: string;
   session_id?: string;
-  agent: Agent;
-  steps: T[];
+  agent: Agent<AgentExtra>;
+  steps: StepType[];
+  final_metrics?: FinalMetrics;
 };
 
-type Agent = {
+type FinalMetrics = {
+  total_prompt_tokens?: number;
+  total_completion_tokens?: number;
+  total_cached_tokens?: number;
+  total_steps?: number;
+};
+
+type DevinAgentExtra = {
+  backend?: string;
+  permission_mode?: string;
+};
+
+type Agent<Extra = unknown> = {
   name: string;
   version: string;
   model_name?: string;
+  tool_definitions?: ToolDefinition;
+  extra?: Extra;
+};
+
+type ToolDefinition = {
+  type: "function";
+  function: FunctionDefinition;
+};
+
+type FunctionDefinition = {
+  name: string;
+  description: string;
+  parameters: JsonSchemaType;
 };
 
 type ToolCall = {
@@ -53,17 +80,83 @@ type DevinMetadata = {
   };
 };
 
-type Step = {
-  step_id: number;
-  source: string;
-  model_name?: string;
-  message: string;
-  tool_calls?: Array<ToolCall>;
+type ContentPart = ContentPartText | ContentPartImage;
+
+type ContentPartText = {
+  type: "text";
+  text: string;
 };
 
-type DevinStep = Step & { metadata?: DevinMetadata };
+type ContentPartImage = {
+  type: "image";
+  source: ImageSource;
+};
 
-type DevinAgentTrajectory = AgentTrajectory<DevinStep>;
+function isTextContentPart(
+  contentPart: ContentPart,
+): contentPart is ContentPartText {
+  return "text" in contentPart;
+}
+
+function isImageContentPart(
+  contentPart: ContentPart,
+): contentPart is ContentPartImage {
+  return "image" in contentPart;
+}
+
+type ImageSource = {
+  media_type: string;
+  path: string;
+};
+
+type Step<StepExtra = unknown, MetricsExtra = unknown> = {
+  step_id: number;
+  timestamp?: string;
+  source: string;
+  model_name?: string;
+  message: string | Array<ContentPart>;
+  tool_calls?: Array<ToolCall>;
+  extra?: StepExtra;
+  observation?: Observation;
+  metrics?: Metrics<MetricsExtra>;
+};
+
+type DevinTelemetry = {
+  source?: string;
+  operation?: string;
+};
+
+type DevinStepExtra = {
+  committed_acu_cost?: number;
+  generation_model?: string;
+  telemetry?: DevinTelemetry;
+};
+
+type Observation = {
+  results: Array<ObservationResult>;
+};
+
+type ObservationResult = {
+  source_call_id?: string;
+  content?: string | Array<ContentPart>;
+};
+
+type Metrics<Extra = unknown> = {
+  prompt_tokens?: number;
+  completion_tokens?: number;
+  cached_tokens?: number;
+  extra?: Extra;
+};
+
+type DevinMetricsExtra = {
+  cache_creation_input_tokens?: number;
+};
+
+type DevinStep = Step<DevinStepExtra, DevinMetricsExtra> & {
+  metadata?: DevinMetadata;
+};
+
+type DevinAgentTrajectory = AgentTrajectory<DevinStep, DevinAgentExtra>;
 
 type DevinSessionMetadata = {
   id: string;
@@ -110,28 +203,62 @@ function parseNumericTimestamp(value: number): string {
   return new Date(millis).toISOString();
 }
 
-function getUsage(
-  metadata: DevinMetadata | undefined | null,
-): DevinUsage | null {
-  if (!metadata) return null;
-  const metrics = metadata.metrics;
+function getCommittedAcuCost(step: DevinStep): number {
+  const acuCost = [
+    step.metadata?.committed_acu_cost,
+    step.extra?.committed_acu_cost,
+  ].filter((cost) => isPositiveNumber(cost));
+
+  return acuCost.shift() || 0;
+}
+
+function getMetricsFromStep(
+  step: DevinStep,
+): Metrics<DevinMetricsExtra> | null {
+  let metrics = step.metrics || null;
+
+  if (!metrics && step.metadata) {
+    metrics = getDevinMetricsFromMetadata(step.metadata);
+  }
+
+  return metrics;
+}
+
+function getDevinMetricsFromMetadata(
+  metadata: DevinMetadata,
+): Metrics<DevinMetricsExtra> {
+  return {
+    prompt_tokens: metadata.metrics?.input_tokens,
+    completion_tokens: metadata.metrics?.output_tokens,
+    cached_tokens: metadata.metrics?.cache_read_tokens,
+    extra: {
+      cache_creation_input_tokens: metadata.metrics?.cache_creation_tokens,
+    },
+  };
+}
+
+function getUsage(step: DevinStep): DevinUsage | null {
+  const committedAcuCost = getCommittedAcuCost(step);
+  const metrics = getMetricsFromStep(step);
 
   const hasAnyUsage = [
-    metadata.committed_acu_cost,
-    metrics?.input_tokens,
-    metrics?.output_tokens,
-    metrics?.cache_creation_tokens,
-    metrics?.cache_read_tokens,
+    committedAcuCost,
+    metrics?.prompt_tokens,
+    metrics?.completion_tokens,
+    metrics?.extra?.cache_creation_input_tokens,
+    metrics?.cached_tokens,
   ].some((x) => isPositiveNumber(x));
 
   if (!hasAnyUsage) return null;
 
   return {
-    committedAcuCost: safeNumber(metadata.committed_acu_cost),
-    inputTokens: safeNumber(metrics?.input_tokens),
-    outputTokens: safeNumber(metrics?.output_tokens),
-    cacheCreationInputTokens: safeNumber(metrics?.cache_creation_tokens),
-    cacheReadInputTokens: safeNumber(metrics?.cache_read_tokens),
+    committedAcuCost,
+    inputTokens: safeNumber(metrics?.prompt_tokens),
+    outputTokens: safeNumber(metrics?.completion_tokens),
+    cacheCreationInputTokens: safeNumber(
+      metrics?.extra?.cache_creation_input_tokens,
+    ),
+    cacheReadInputTokens: safeNumber(metrics?.cached_tokens),
   };
 }
 
@@ -198,6 +325,21 @@ function getToolNames(step: DevinStep): string[] {
   return (step.tool_calls ?? []).map((call) => call.function_name);
 }
 
+function normalizeContentPartMessage(contentPart: ContentPart) {
+  if (isTextContentPart(contentPart)) {
+    return contentPart.text;
+  } else {
+    return contentPart.source.path;
+  }
+}
+
+function normalizeStepMessage(message: string | Array<ContentPart>): string {
+  if (Array.isArray(message)) {
+    return message.map((x) => normalizeContentPartMessage(x).trim()).join(" ");
+  }
+  return message.trim();
+}
+
 function getFirstUserMessageBeforeStep(
   steps: DevinStep[],
   index: number,
@@ -205,7 +347,9 @@ function getFirstUserMessageBeforeStep(
   for (let i = index - 1; i >= 0; i--) {
     const step = steps[i];
     if (!step?.metadata?.is_user_input) continue;
-    const message = step.message?.trim();
+    const message = step.message
+      ? normalizeStepMessage(step.message)
+      : undefined;
     if (message) return message;
   }
   return null;
@@ -282,7 +426,7 @@ class DevinSessionParser implements SessionParser {
       const step = transcript.steps[index];
       if (step.metadata?.is_user_input) continue;
 
-      const usage = getUsage(step.metadata);
+      const usage = getUsage(step);
       if (!usage) continue;
 
       const timestamp = getTimestamp(step, session) ?? "";

tests/providers/devin.test.ts

@@ -242,6 +242,307 @@ describe('devin provider', () => {
     ])
   })
 
+  it('extracts user message from ContentPart[] messages (ATIF v1.7 multimodal)', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('content-parts.json', {
+      session_id: 'cp-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          message: [
+            { type: 'text', text: 'look at this screenshot' },
+            { type: 'image', source: { media_type: 'image/png', path: '/tmp/screenshot.png' } },
+          ],
+          metadata: { is_user_input: true, created_at: '2027-01-15T08:00:00.000Z' },
+        },
+        {
+          step_id: 2,
+          metadata: {
+            created_at: '2027-01-15T08:00:01.000Z',
+            committed_acu_cost: 0.1,
+            metrics: { input_tokens: 50 },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]!.userMessage).toBe('look at this screenshot /tmp/screenshot.png')
+  })
+
+  it('parses ATIF v1.7 transcripts with agent.extra, final_metrics, and observations', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('atif-v17.json', {
+      schema_version: '1.7',
+      session_id: 'v17-session',
+      agent: {
+        name: 'devin',
+        version: '2.0',
+        model_name: 'claude-sonnet-4-6',
+        extra: { backend: 'cloud', permission_mode: 'auto' },
+      },
+      final_metrics: {
+        total_prompt_tokens: 500,
+        total_completion_tokens: 200,
+        total_cached_tokens: 50,
+        total_steps: 2,
+      },
+      steps: [
+        {
+          step_id: 1,
+          message: 'fix the bug',
+          metadata: { is_user_input: true, created_at: '2027-01-15T08:00:00.000Z' },
+        },
+        {
+          step_id: 2,
+          source: 'assistant',
+          model_name: 'claude-sonnet-4-6',
+          message: 'I will read the file first',
+          tool_calls: [{ tool_call_id: 'tc1', function_name: 'read_file', arguments: { path: 'src/main.ts' } }],
+          observation: {
+            results: [{ source_call_id: 'tc1', content: 'file contents here' }],
+          },
+          extra: {
+            committed_acu_cost: 0.15,
+            generation_model: 'claude-sonnet-4-6',
+            telemetry: { source: 'devin-cli', operation: 'generate' },
+          },
+          metadata: {
+            created_at: '2027-01-15T08:00:01.000Z',
+            committed_acu_cost: 0.15,
+            metrics: { input_tokens: 200, output_tokens: 50, cache_creation_tokens: 20, cache_read_tokens: 10 },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]).toMatchObject({
+      provider: 'devin',
+      model: 'claude-sonnet-4-6',
+      inputTokens: 200,
+      outputTokens: 50,
+      cacheCreationInputTokens: 20,
+      cacheReadInputTokens: 10,
+      costUSD: 0.15,
+      tools: ['read_file'],
+      userMessage: 'fix the bug',
+      sessionId: 'v17-session',
+    })
+  })
+
+  it('handles plain string user messages alongside ContentPart[] messages', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('mixed-messages.json', {
+      session_id: 'mixed-msg-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          message: 'plain text user message',
+          metadata: { is_user_input: true, created_at: '2027-01-15T08:00:00.000Z' },
+        },
+        {
+          step_id: 2,
+          metadata: {
+            created_at: '2027-01-15T08:00:01.000Z',
+            committed_acu_cost: 0.1,
+            metrics: { input_tokens: 50 },
+          },
+        },
+        {
+          step_id: 3,
+          message: [{ type: 'text', text: 'multimodal user message' }],
+          metadata: { is_user_input: true, created_at: '2027-01-15T08:00:02.000Z' },
+        },
+        {
+          step_id: 4,
+          metadata: {
+            created_at: '2027-01-15T08:00:03.000Z',
+            committed_acu_cost: 0.2,
+            metrics: { input_tokens: 100 },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(2)
+    expect(calls[0]!.userMessage).toBe('plain text user message')
+    expect(calls[1]!.userMessage).toBe('multimodal user message')
+  })
+
+  it('reads ACU cost from step.extra when metadata.committed_acu_cost is absent', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('extra-acu.json', {
+      session_id: 'extra-acu-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          extra: { committed_acu_cost: 0.3 },
+          metadata: {
+            created_at: '2027-01-15T08:00:00.000Z',
+            metrics: { input_tokens: 10 },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]!.costUSD).toBeCloseTo(0.3, 12)
+  })
+
+  it('prefers metadata.committed_acu_cost over extra.committed_acu_cost', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('acu-priority.json', {
+      session_id: 'acu-priority-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          extra: { committed_acu_cost: 0.99 },
+          metadata: {
+            created_at: '2027-01-15T08:00:00.000Z',
+            committed_acu_cost: 0.11,
+            metrics: { input_tokens: 10 },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]!.costUSD).toBeCloseTo(0.11, 12)
+  })
+
+  it('reads tokens from step.metrics when metadata.metrics is absent', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('step-metrics.json', {
+      session_id: 'step-metrics-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          metrics: {
+            prompt_tokens: 300,
+            completion_tokens: 75,
+            cached_tokens: 15,
+            extra: { cache_creation_input_tokens: 25 },
+          },
+          extra: { committed_acu_cost: 0.2 },
+          metadata: { created_at: '2027-01-15T08:00:00.000Z' },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]).toMatchObject({
+      inputTokens: 300,
+      outputTokens: 75,
+      cacheCreationInputTokens: 25,
+      cacheReadInputTokens: 15,
+      cachedInputTokens: 15,
+      costUSD: 0.2,
+    })
+  })
+
+  it('prefers step.metrics over metadata.metrics when both are present', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('metrics-priority.json', {
+      session_id: 'metrics-priority-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          metrics: {
+            prompt_tokens: 500,
+            completion_tokens: 100,
+            cached_tokens: 20,
+            extra: { cache_creation_input_tokens: 30 },
+          },
+          metadata: {
+            created_at: '2027-01-15T08:00:00.000Z',
+            committed_acu_cost: 0.1,
+            metrics: {
+              input_tokens: 1,
+              output_tokens: 1,
+              cache_creation_tokens: 1,
+              cache_read_tokens: 1,
+            },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]).toMatchObject({
+      inputTokens: 500,
+      outputTokens: 100,
+      cacheCreationInputTokens: 30,
+      cacheReadInputTokens: 20,
+      cachedInputTokens: 20,
+    })
+  })
+
+  it('handles observation results with ContentPart[] content', async () => {
+    await configureDevinRate()
+    const filePath = await writeTranscript('observation-content-parts.json', {
+      session_id: 'obs-cp-session',
+      agent: { model_name: 'agent-model' },
+      steps: [
+        {
+          step_id: 1,
+          message: 'check the image',
+          metadata: { is_user_input: true, created_at: '2027-01-15T08:00:00.000Z' },
+        },
+        {
+          step_id: 2,
+          source: 'assistant',
+          message: 'reading file',
+          tool_calls: [{ tool_call_id: 'tc1', function_name: 'read_file', arguments: {} }],
+          observation: {
+            results: [{
+              source_call_id: 'tc1',
+              content: [
+                { type: 'text', text: 'file output here' },
+                { type: 'image', source: { media_type: 'image/png', path: '/tmp/output.png' } },
+              ],
+            }],
+          },
+          metadata: {
+            created_at: '2027-01-15T08:00:01.000Z',
+            committed_acu_cost: 0.1,
+            metrics: { input_tokens: 100, output_tokens: 30 },
+          },
+        },
+      ],
+    })
+
+    const calls = await parseTranscript(filePath)
+
+    expect(calls).toHaveLength(1)
+    expect(calls[0]).toMatchObject({
+      tools: ['read_file'],
+      costUSD: 0.1,
+      inputTokens: 100,
+      outputTokens: 30,
+    })
+  })
+
   it('ignores array-root and malformed transcripts', async () => {
     await configureDevinRate()
     const arrayPath = await writeTranscript('array.json', [])