mirror of
https://github.com/AgentSeal/codeburn.git
synced 2026-07-10 01:29:41 +00:00
refactor(copilot): apply review fixes to OTel parsing PR #477
Maintainer review fixes on top of the OTel cache-token work: - Remove all DEBUG_OTEL console.warn scaffolding from parser.ts and copilot.ts (gated but unlike the rest of the codebase). - Parameterize the spans IN (...) query instead of string-interpolating trace IDs. - Fold the per-chat-span metadata query into the trace-span query to drop the N+1 (one query per chat span -> one per conversation). - Guard epochToISO against null/NaN/0 so a malformed start_time_ms row no longer throws on new Date(NaN).toISOString(). - Remove dead code: parseSpanAttributes, OTelSpanRow, and the unreachable `if (!db) return`; drop unused catch bindings. - Note in parseProviderSources that the non-durable append path assumes unique source paths. - Document the OTel source in docs/providers/copilot.md: Node 22+ requirement, durable-cache monotonic totals, and the one-time parse-version cache reset on upgrade.
This commit is contained in:
AgentSeal
parent
d68a214c76
commit
9917a8a181
3 changed files with 51 additions and 118 deletions
|
|
@ -8,18 +8,38 @@ GitHub Copilot Chat (CLI and VS Code extension transcripts).
|
|||
|
||||
## Where it reads from
|
||||
|
||||
Two locations. Both are walked on every run; results merge.
|
||||
Two JSONL locations plus an optional OpenTelemetry SQLite source (see below). All
|
||||
discovered sources are walked on every run; results merge and dedupe.
|
||||
|
||||
1. **Legacy CLI sessions:** `~/.copilot/session-state/`
|
||||
2. **VS Code transcripts:** `~/Library/Application Support/Code/User/workspaceStorage/<hash>/GitHub.copilot-chat/transcripts/` and equivalents on Windows / Linux
|
||||
3. **OTel SQLite store:** VS Code Copilot Chat's `agent-traces.db` (see the OTel section). Preferred when present because it carries full input / output / cache token counts; the JSONL sources only record output tokens.
|
||||
|
||||
## Storage format
|
||||
|
||||
JSONL in both locations, but the schemas differ. The parser switches by detecting which schema the first event uses (`copilot.ts:83-159` for legacy, `copilot.ts:215-293` for transcripts).
|
||||
JSONL in the first two locations (schemas differ; the parser switches by detecting which schema the first event uses), and a SQLite DB for the OTel source.
|
||||
|
||||
## OpenTelemetry (OTel) source
|
||||
|
||||
When VS Code Copilot Chat's `agent-traces.db` exists, the parser reads per-LLM-call token
|
||||
breakdowns (input, output, cache-read, cache-creation) from it, which the JSONL sources do
|
||||
not record. Discovery is skipped with `CODEBURN_COPILOT_DISABLE_OTEL=1`, and the DB path
|
||||
can be overridden with `CODEBURN_COPILOT_OTEL_DB`.
|
||||
|
||||
- **Requires Node 22+.** The OTel source uses the built-in `node:sqlite` module (the same
|
||||
backend as Cursor / OpenCode). On Node 20, or if the DB is missing / locked / corrupt /
|
||||
wrong-schema, OTel is skipped and the JSONL/transcript sources are used as a fallback.
|
||||
- **Durable cache (monotonic totals).** Copilot is marked `durableSources`: OTel-derived
|
||||
cache entries are never evicted when VS Code prunes old spans from the DB, so
|
||||
month-to-date totals do not drop as the DB rotates. Entries age out after 90 days.
|
||||
- **Upgrade note.** The first run after upgrading to the OTel version bumps the copilot
|
||||
parse version, which discards the prior copilot cache. Spans already pruned from the DB
|
||||
before the upgrade cannot be recovered, so monotonicity starts from the upgrade point,
|
||||
not retroactively.
|
||||
|
||||
## Caching
|
||||
|
||||
None at the provider level.
|
||||
None for the JSONL sources. The OTel source uses a durable cache (see above).
|
||||
|
||||
## Deduplication
|
||||
|
||||
|
|
|
|||
|
|
@ -1325,10 +1325,6 @@ function buildSessionSummary(
|
|||
totalCacheWrite += call.usage.cacheCreationInputTokens
|
||||
apiCalls++
|
||||
|
||||
if (process.env['DEBUG_OTEL'] && (call.usage.cacheReadInputTokens > 0 || call.usage.cacheCreationInputTokens > 0)) {
|
||||
console.warn(`[Aggregate] Model=${call.model}, cache_read=${call.usage.cacheReadInputTokens}, cache_write=${call.usage.cacheCreationInputTokens}`)
|
||||
}
|
||||
|
||||
const modelKey = getShortModelName(call.model)
|
||||
if (!modelBreakdown[modelKey]) {
|
||||
modelBreakdown[modelKey] = {
|
||||
|
|
@ -2033,18 +2029,9 @@ async function parseProviderSources(
|
|||
for await (const call of parser.parse()) {
|
||||
providerCalls.push(call)
|
||||
}
|
||||
if (process.env['DEBUG_OTEL']) {
|
||||
console.warn(`[Parse] File ${source.path.substring(0, 40)}: Collected ${providerCalls.length} provider calls, ${providerCalls.filter(c => c.cacheReadInputTokens > 0 || c.cacheCreationInputTokens > 0).length} with cache tokens`)
|
||||
}
|
||||
if (process.env['DEBUG_OTEL'] && providerCalls.some(c => c.cacheReadInputTokens > 0 || c.cacheCreationInputTokens > 0)) {
|
||||
console.warn(`[Parse] After conversion: turns to be created...`)
|
||||
}
|
||||
const canonicalCalls = await Promise.all(providerCalls.map(canonicalizeProviderCallProject))
|
||||
const turns = providerCallsToCachedTurns(canonicalCalls)
|
||||
if (process.env['DEBUG_OTEL'] && providerCalls.some(c => c.cacheReadInputTokens > 0 || c.cacheCreationInputTokens > 0)) {
|
||||
console.warn(`[Parse] After conversion: ${turns.length} turns, calls with cache: ${turns.flatMap(t => t.calls).filter(c => c.usage.cacheReadInputTokens > 0 || c.usage.cacheCreationInputTokens > 0).length}`)
|
||||
}
|
||||
|
||||
|
||||
// Store/merge parsed turns into the cache.
|
||||
// Durable providers use a union-by-deduplicationKey merge: existing turns
|
||||
// are NEVER deleted (preserves data for spans pruned from the DB), and
|
||||
|
|
@ -2061,24 +2048,17 @@ async function parseProviderSources(
|
|||
)
|
||||
existingEntry.turns = [...existingEntry.turns, ...newTurns]
|
||||
existingEntry.fingerprint = fp
|
||||
if (process.env['DEBUG_OTEL']) {
|
||||
console.warn(`[Parse] Durable union-merge for ${source.path.substring(0, 40)}: kept ${existingEntry.turns.length - newTurns.length} existing, added ${newTurns.length} new turns`)
|
||||
}
|
||||
} else {
|
||||
section.files[source.path] = { fingerprint: fp, mcpInventory: [], turns }
|
||||
}
|
||||
} else {
|
||||
// Non-durable: overwrite (clearedPaths already deleted stale entry above)
|
||||
// or append when multiple sources map to the same path.
|
||||
// or append when multiple sources map to the same path. NOTE: the append
|
||||
// path assumes discoverSessions yields a unique path per source, which all
|
||||
// current providers do; it only fires for same-path multi-source providers.
|
||||
const existingCacheEntry = section.files[source.path]
|
||||
if (process.env['DEBUG_OTEL']) {
|
||||
console.warn(`[Parse] Cache entry exists for path: ${!!existingCacheEntry}, turns to merge: ${turns.length}`)
|
||||
}
|
||||
if (existingCacheEntry) {
|
||||
existingCacheEntry.turns = [...existingCacheEntry.turns, ...turns]
|
||||
if (process.env['DEBUG_OTEL']) {
|
||||
console.warn(`[Parse] Merged with existing cache entry for ${source.path.substring(0, 40)}, now has ${existingCacheEntry.turns.length} turns total`)
|
||||
}
|
||||
} else {
|
||||
section.files[source.path] = { fingerprint: fp, mcpInventory: [], turns }
|
||||
}
|
||||
|
|
@ -2147,31 +2127,13 @@ async function parseProviderSources(
|
|||
// Uses seenKeys (shared across providers) for cross-provider dedup.
|
||||
const sessionMap = new Map<string, { project: string; projectPath?: string; turns: ClassifiedTurn[] }>()
|
||||
|
||||
if (process.env['DEBUG_OTEL']) {
|
||||
const totalCacheCalls = sources.flatMap(s => section.files[s.path]?.turns ?? []).flatMap(t => t.calls).filter(c => c.usage.cacheReadInputTokens > 0 || c.usage.cacheCreationInputTokens > 0)
|
||||
console.warn(`[SessionMap] Starting with ${sources.length} sources, ${totalCacheCalls.length} calls with cache tokens`)
|
||||
|
||||
const filesInfo = Object.entries(section.files).map(([path, file]) => ({
|
||||
path: path.substring(0, 50),
|
||||
turns: file.turns.length,
|
||||
totalCalls: file.turns.flatMap(t => t.calls).length,
|
||||
cacheTokenCalls: file.turns.flatMap(t => t.calls).filter(c => c.usage.cacheReadInputTokens > 0 || c.usage.cacheCreationInputTokens > 0).length
|
||||
}))
|
||||
console.warn(`[SessionMap] Cache files: ${JSON.stringify(filesInfo)}`)
|
||||
}
|
||||
|
||||
for (const source of sources) {
|
||||
const cachedFile = section.files[source.path]
|
||||
if (!cachedFile) continue
|
||||
|
||||
for (const turn of cachedFile.turns) {
|
||||
const hasDup = turn.calls.some(c => seenKeys.has(c.deduplicationKey))
|
||||
if (hasDup) {
|
||||
if (process.env['DEBUG_OTEL'] && turn.calls.some(c => c.usage.cacheReadInputTokens > 0 || c.usage.cacheCreationInputTokens > 0)) {
|
||||
console.warn(`[SessionMap] Skipping turn with cache tokens due to dedup: ${turn.calls.map(c => c.deduplicationKey).join(',')}`)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if (hasDup) continue
|
||||
|
||||
for (const c of turn.calls) seenKeys.add(c.deduplicationKey)
|
||||
|
||||
|
|
@ -2186,10 +2148,6 @@ async function parseProviderSources(
|
|||
const project = turn.calls[0]?.project ?? source.project
|
||||
const key = `${providerName}:${turn.sessionId}:${project}`
|
||||
|
||||
if (process.env['DEBUG_OTEL'] && turn.calls.some(c => c.usage.cacheReadInputTokens > 0 || c.usage.cacheCreationInputTokens > 0)) {
|
||||
console.warn(`[SessionMap] Adding turn with cache tokens to sessionMap key=${key}`)
|
||||
}
|
||||
|
||||
const existing = sessionMap.get(key)
|
||||
if (existing) {
|
||||
existing.turns.push(classified)
|
||||
|
|
|
|||
|
|
@ -184,19 +184,6 @@ type CopilotEvent =
|
|||
// The Copilot Budget extension reads from this same DB and uses per-span
|
||||
// token counts, confirming this schema is stable enough to depend on.
|
||||
|
||||
interface OTelSpanRow {
|
||||
trace_id: string
|
||||
span_id: string
|
||||
parent_span_id: string | null
|
||||
name: string // e.g. "chat gpt-4o" or "invoke_agent copilot"
|
||||
start_time: number // nanosecond epoch or millisecond epoch
|
||||
end_time: number
|
||||
attributes: string | null // JSON blob of all OTel attributes
|
||||
// Some DB versions may use separate columns instead of a JSON blob.
|
||||
// We try JSON parsing first, then fall back to individual column queries.
|
||||
[key: string]: unknown
|
||||
}
|
||||
|
||||
// Parsed attribute bag from a span
|
||||
interface SpanAttributes {
|
||||
'gen_ai.operation.name'?: string
|
||||
|
|
@ -285,19 +272,6 @@ function parseCwd(yaml: string): string | null {
|
|||
return raw || null
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the JSON attributes blob from an OTel span row.
|
||||
* Returns an empty object if parsing fails.
|
||||
*/
|
||||
function parseSpanAttributes(raw: string | null): SpanAttributes {
|
||||
if (!raw) return {}
|
||||
try {
|
||||
return JSON.parse(raw) as SpanAttributes
|
||||
} catch {
|
||||
return {}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Load span attributes from the span_attributes table (key-value pairs).
|
||||
* This handles the modern VS Code Copilot Chat schema where attributes
|
||||
|
|
@ -327,8 +301,7 @@ function loadSpanAttributesFromTable(
|
|||
}
|
||||
}
|
||||
return attrs
|
||||
} catch (e) {
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`loadSpanAttributesFromTable error for span ${spanId}:`, e)
|
||||
} catch {
|
||||
return {}
|
||||
}
|
||||
}
|
||||
|
|
@ -338,6 +311,9 @@ function loadSpanAttributesFromTable(
|
|||
* The OTel spec uses nanoseconds, but some implementations use milliseconds.
|
||||
*/
|
||||
function epochToISO(epoch: number): string {
|
||||
// Guard malformed rows: new Date(NaN).toISOString() throws. Fall back to the
|
||||
// epoch (1970) so a bad timestamp is excluded from period totals, not crashing.
|
||||
if (!Number.isFinite(epoch) || epoch <= 0) return new Date(0).toISOString()
|
||||
// If the value looks like nanoseconds (> 1e15), convert to ms
|
||||
const ms = epoch > 1e15 ? Math.floor(epoch / 1e6) : epoch > 1e12 ? epoch : epoch * 1000
|
||||
return new Date(ms).toISOString()
|
||||
|
|
@ -529,7 +505,6 @@ function createOtelParser(
|
|||
|
||||
// One DB open handles ALL conversations — avoids N opens for N conversations.
|
||||
const db = openDatabase(source.path)
|
||||
if (!db) return
|
||||
|
||||
try {
|
||||
// ---------------------------------------------------------------
|
||||
|
|
@ -554,8 +529,6 @@ function createOtelParser(
|
|||
ORDER BY min_start DESC`
|
||||
)
|
||||
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] Found ${conversationRows.length} conversations in DB`)
|
||||
|
||||
for (const convRow of conversationRows) {
|
||||
const conversationId = convRow.conversation_id
|
||||
if (!conversationId) continue
|
||||
|
|
@ -578,8 +551,6 @@ function createOtelParser(
|
|||
[conversationId]
|
||||
)
|
||||
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] Found ${spanIdRows.length} spans for conversation ${conversationId}`)
|
||||
|
||||
// Collect trace IDs and span IDs belonging to this conversation
|
||||
const traceIds = new Set<string>()
|
||||
for (const row of spanIdRows) {
|
||||
|
|
@ -587,24 +558,33 @@ function createOtelParser(
|
|||
}
|
||||
|
||||
if (traceIds.size === 0) {
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] No trace IDs found for conversation`)
|
||||
continue
|
||||
}
|
||||
|
||||
// Now query all spans within those traces to find chat and tool spans
|
||||
const traceIdList = [...traceIds].map((id) => `'${id.replace(/'/g, "''")}'`).join(',')
|
||||
const traceSpans = db.query<{ span_id: string; trace_id: string; operation_name: string | null }>(
|
||||
`SELECT span_id, trace_id, operation_name FROM spans WHERE trace_id IN (${traceIdList})`
|
||||
// Now query all spans within those traces to find chat and tool spans.
|
||||
// Pull the metadata columns in the same query so we don't re-query the
|
||||
// spans table once per chat span below (avoids an N+1).
|
||||
const traceIdArr = [...traceIds]
|
||||
const tracePlaceholders = traceIdArr.map(() => '?').join(',')
|
||||
const traceSpans = db.query<{
|
||||
span_id: string
|
||||
trace_id: string
|
||||
operation_name: string | null
|
||||
start_time_ms: number
|
||||
response_model: string | null
|
||||
}>(
|
||||
`SELECT span_id, trace_id, operation_name, start_time_ms, response_model FROM spans WHERE trace_id IN (${tracePlaceholders})`,
|
||||
traceIdArr
|
||||
)
|
||||
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] Found ${traceSpans.length} total spans across ${traceIds.size} traces`)
|
||||
|
||||
// Collect tool names from execute_tool spans for each trace
|
||||
const toolsByTrace = new Map<string, string[]>()
|
||||
const chatSpanIds: string[] = []
|
||||
const spanMetaById = new Map<string, { trace_id: string; start_time_ms: number; response_model: string | null }>()
|
||||
|
||||
for (const span of traceSpans) {
|
||||
const opName = span.operation_name || ''
|
||||
spanMetaById.set(span.span_id, span)
|
||||
|
||||
if (opName === 'chat') {
|
||||
chatSpanIds.push(span.span_id)
|
||||
|
|
@ -622,22 +602,12 @@ function createOtelParser(
|
|||
}
|
||||
}
|
||||
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] Found ${chatSpanIds.length} chat spans`)
|
||||
|
||||
// Yield one ParsedProviderCall per chat span
|
||||
for (const spanId of chatSpanIds) {
|
||||
const attrs = loadSpanAttributesFromTable(db, spanId)
|
||||
|
||||
// Get span metadata from the spans table
|
||||
const spanMetadata = db.query<{ trace_id: string; start_time_ms: number; response_model: string | null }>(
|
||||
`SELECT trace_id, start_time_ms, response_model FROM spans WHERE span_id = ?`,
|
||||
[spanId]
|
||||
)?.[0]
|
||||
|
||||
if (!spanMetadata) {
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] No metadata for span ${spanId}`)
|
||||
continue
|
||||
}
|
||||
const spanMetadata = spanMetaById.get(spanId)
|
||||
if (!spanMetadata) continue
|
||||
|
||||
const model =
|
||||
(attrs['gen_ai.response.model'] as string | undefined) ??
|
||||
|
|
@ -650,10 +620,7 @@ function createOtelParser(
|
|||
const cacheReadTokens = Number(attrs['gen_ai.usage.cache_read.input_tokens'] ?? 0)
|
||||
const cacheCreationTokens = Number(attrs['gen_ai.usage.cache_creation.input_tokens'] ?? 0)
|
||||
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] Span ${spanId.substring(0, 8)}: model=${model}, input=${inputTokens}, output=${outputTokens}, cache_read=${cacheReadTokens}, cache_creation=${cacheCreationTokens}`)
|
||||
|
||||
if (inputTokens === 0 && outputTokens === 0) {
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[OTel] Skipping span with 0 tokens`)
|
||||
continue
|
||||
}
|
||||
|
||||
|
|
@ -935,17 +902,12 @@ export function createCopilotProvider(sessionStateDir?: string, workspaceStorage
|
|||
if (!disableOtel) {
|
||||
const dbPath = getAgentTracesDbPath()
|
||||
if (dbPath) {
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] Discovering OTel sessions from ${dbPath}`)
|
||||
try {
|
||||
const otelSources = await discoverOtelSessions(dbPath)
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] Got ${otelSources.length} OTel sources`)
|
||||
sources.push(...otelSources)
|
||||
} catch (e) {
|
||||
} catch {
|
||||
// OTel discovery failed — fall through to JSONL
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] OTel discovery error:`, e)
|
||||
}
|
||||
} else {
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] No OTel DB path found`)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -953,7 +915,6 @@ export function createCopilotProvider(sessionStateDir?: string, workspaceStorage
|
|||
try {
|
||||
const jsonlDir = getCopilotSessionStateDir(sessionStateDir)
|
||||
const jsonlSources = await discoverJsonlSessions(jsonlDir)
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] Got ${jsonlSources.length} JSONL sources`)
|
||||
sources.push(...jsonlSources)
|
||||
} catch {
|
||||
// JSONL discovery failed
|
||||
|
|
@ -962,13 +923,11 @@ export function createCopilotProvider(sessionStateDir?: string, workspaceStorage
|
|||
// 3. Discover VS Code workspace transcript sessions
|
||||
try {
|
||||
const transcriptSources = await discoverTranscriptSessions(getWsDirs())
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] Got ${transcriptSources.length} transcript sources`)
|
||||
sources.push(...transcriptSources)
|
||||
} catch {
|
||||
// Transcript discovery failed
|
||||
}
|
||||
|
||||
if (process.env['DEBUG_OTEL']) console.warn(`[Provider] Total sources: ${sources.length}`)
|
||||
return sources
|
||||
},
|
||||
|
||||
|
|
@ -980,10 +939,6 @@ export function createCopilotProvider(sessionStateDir?: string, workspaceStorage
|
|||
// The dedup key set (seenKeys) is shared across both parsers,
|
||||
// so if OTel already yielded a span, the JSONL parser will skip
|
||||
// the matching assistant.message (and vice versa).
|
||||
if (process.env['DEBUG_OTEL']) {
|
||||
const isOtel = isOtelSource(source)
|
||||
console.warn(`[Provider] Creating ${isOtel ? 'OTel' : 'JSONL'} parser for source: ${source.path}`)
|
||||
}
|
||||
if (isOtelSource(source)) {
|
||||
return createOtelParser(source, seenKeys)
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue