qwen-code/docs/developers/development/telemetry.md
jinye 62ed44e1f3
Some checks are pending
Qwen Code CI / Classify PR (push) Waiting to run
Qwen Code CI / Lint (push) Blocked by required conditions
Qwen Code CI / Test (macos-latest, Node 22.x) (push) Blocked by required conditions
Qwen Code CI / Test (ubuntu-latest, Node 22.x) (push) Blocked by required conditions
Qwen Code CI / Test (windows-latest, Node 22.x) (push) Blocked by required conditions
Qwen Code CI / Post Coverage Comment (push) Blocked by required conditions
Qwen Code CI / CodeQL (push) Blocked by required conditions
E2E Tests / E2E Test (Linux) - sandbox:docker (push) Waiting to run
E2E Tests / E2E Test (Linux) - sandbox:none (push) Waiting to run
E2E Tests / E2E Test - macOS (push) Waiting to run
feat(telemetry): client-side HTTP span + opt-in W3C traceparent propagation (#4384) (#4390)
* feat(telemetry): propagate W3C traceparent on outbound LLM requests

Part 1 of #4384 (sub-issue of #3731 P3 deeper observability).

Today qwen-code's only OTel instrumentation is `HttpInstrumentation`,
which only patches Node's `http`/`https` modules. The `openai` and
`@google/genai` SDKs use `globalThis.fetch` (undici), so outbound LLM
requests carry no `traceparent` header and trace context dies at the
qwen-code process boundary.

Adds `@opentelemetry/instrumentation-undici@0.14.0` (peer-compatible
with the installed `@opentelemetry/instrumentation@0.203.0`) and wires
it into `initializeTelemetry()` next to the existing
`HttpInstrumentation`. Default propagator (W3C tracecontext + baggage)
remains unchanged — no explicit `textMapPropagator` needed.

`ignoreRequestHook` skips OTLP exporter endpoints to avoid the
classic feedback loop (OTel SDK uses fetch to upload OTLP data; without
the hook each upload would create a span that gets uploaded, infinitely).
Configured `otlpEndpoint` / per-signal endpoints are stripped of trailing
slash and query string for robust prefix matching against undici's
`request.origin + request.path`.

Outbound LLM calls now also produce a client-side HTTP span (separating
network TTFB / transfer time from the existing `api.generateContent`
total-duration span).

Design doc: docs/design/telemetry-outbound-propagation-design.md
(Part A — traceparent; Part B — session id header — lands in a
follow-up PR per the design's split rationale.)

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* fix(telemetry): harden OTLP feedback-loop guard + slim lockfile diff

Review feedback on #4390:

1. CI was failing on npm ci because the lockfile was generated with npm 11
   locally (it sprinkles `peer: true` annotations npm 10 reads differently
   and rejects). Regenerated with npm 10 (matching CI's Node 22.x default),
   so the diff vs main is now 18 lines (the actual instrumentation-undici
   entry) instead of 105 lines of npm-version drift noise.

2. (Copilot inline at sdk.ts:330) `otlpUrlPrefixes` was derived from raw
   Config strings, so a settings.json `"otlpEndpoint": "\"http://...\""`
   (quoted) or trailing `#fragment` would silently miss the prefix match
   and reintroduce the feedback loop the hook exists to prevent. Replaced
   the regex-based suffix trim with a WHATWG URL parser:
   - strips ?query, #fragment, trailing slash
   - trims symmetric ASCII quotes a user may have placed in settings.json
   - falls back to safe suffix trimming if URL parsing fails (misconfigured
     endpoint still gets SOME protection)

3. (CodeQL inline) Replaced the `/\?.*$/` regex in ignoreRequestHook with
   `indexOf('?')`/`indexOf('#')` slicing for ReDoS hygiene. The regex was
   linear in practice but flagged as polynomial — using indexOf removes
   the ambiguity and is arguably simpler.

Added 3 tests in sdk.test.ts covering the new normalizations (#fragment
on incoming path, quoted endpoint, #fragment on configured endpoint).

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* feat(telemetry): propagate X-Qwen-Code-Session-Id on outbound LLM requests

Part 2 of #4384. Stacks on top of PR #4390 (traceparent via undici).

Adds a product-namespaced HTTP header X-Qwen-Code-Session-Id to every
outbound LLM request when telemetry is enabled, so server-side ingestion
can correlate observed requests with qwen-code session metric/log records.
Pattern matched from claude-code (X-Claude-Code-Session-Id, verified at
src/services/api/client.ts:108 in their open-source repo).

Critical design decision (design doc section 4.3): the OpenAI / Anthropic
providers use a per-request fetch wrapper rather than the SDK defaultHeaders
option, because content-generator SDK clients are constructed once and NOT
recreated on /clear-triggered session resets (Config.resetSession updates
this.sessionId but the contentGenerator keeps using the stale header value).
Reading config.getSessionId() from inside the wrapper at request time gives
the live value.

Gemini provider uses static httpOptions.headers — @google/genai HttpOptions
interface does not expose a fetch hook (only headers, baseUrl, apiVersion,
timeout, extraParams). This is a known limitation: after session reset,
Gemini X-Qwen-Code-Session-Id stays stale until the contentGenerator is
recreated. Documented in telemetry.md and the design doc section 8.6;
spans/logs continue to carry the live session id for trace/log correlation.
Lazy-invalidate fix is a follow-up sub-issue.

Header is omitted when telemetry is disabled OR when getSessionId returns
an empty string (some HTTP middleware rejects empty header values).

Integration sites:
- packages/core/src/core/openaiContentGenerator/provider/default.ts
  (base class — automatically covered by deepseek/minimax/mistral/
  modelscope/openrouter subclasses; openrouter calls super.buildHeaders)
- packages/core/src/core/openaiContentGenerator/provider/dashscope.ts
  (overrides buildClient — must be touched separately; QwenContentGenerator
  inherits via this provider)
- packages/core/src/core/anthropicContentGenerator/anthropicContentGenerator.ts
- packages/core/src/core/geminiContentGenerator/index.ts (factory function,
  not the GeminiContentGenerator class — no signature change)

End-to-end verification (local HTTP server in tmux):
  PASS: traceparent + X-Qwen-Code-Session-Id on every LLM request
  PASS: session id refreshes after simulated /clear (staleness regression
        guarded by llm-correlation-fetch.test.ts)
  PASS: OTLP upload traffic not traced (no feedback loop — PR A
        ignoreRequestHook working)

Robot generated with Qwen Code https://github.com/QwenLM/qwen-code

* fix(telemetry): R2 review fixes — critical correctness + tsc + boundary safety

Adopts 7 review findings from wenshao on #4390 (+ duplicates from now-closed
#4393). Critical bugs first, polish second.

CRITICAL:

1. tsc TS2322 — wrapper return type incompatible with Anthropic SDK Fetch.
   `typeof fetch` (Node WHATWG, 2 overloads) is not structurally assignable
   to Anthropic's narrower `Fetch = (input: RequestInfo, init?) => ...`,
   even though they're call-compatible at runtime. Make wrapper generic
   `<TFetch extends FetchLikeLoose>` so callers preserve their exact fetch
   signature; cast the Anthropic call site through `unknown` with a comment
   explaining why.

2. tsc TS2352 / TS2493 — `baseFetch.mock.calls[0]![1] as RequestInit` was
   out-of-bounds when wrapped was called with no init arg. Replaced with a
   `makeFetchMock()` helper returning typed accessors.

3. normalizeOtlpPrefix catch fallback was DANGEROUS — a config of `"http"`
   produced prefix `"http"` which `startsWith`-matched every outbound HTTP
   request → silently disabled ALL instrumentation (no client spans, no
   correlation header — defeats the entire feature). Fixed: catch returns
   undefined + diag.warn. Misconfigured endpoint loses its feedback-loop
   guard (acceptable) instead of disabling all guards (catastrophic).

4. `url.startsWith(prefix)` matching was NOT boundary-safe — port collision
   (`:4318` matches `:43180`), hostname suffix collision (`otlp.example.com`
   matches `otlp.example.com.evil.net`), path-segment collision (`/v1`
   matches `/v1foo/x`). Replaced with origin-equality + path-prefix +
   boundary-char check (next char must be `/`, `?`, `#`, or end-of-string).

5. HttpInstrumentation also lacked the OTLP feedback-loop guard. The OTLP
   HTTP exporter (`@opentelemetry/exporter-trace-otlp-http`) uses node:http
   (patched by HttpInstrumentation, NOT undici). Without this, every OTLP
   upload batch creates a parasitic client span → feedback loop. Added
   `ignoreOutgoingRequestHook` that reuses the same `matchesOtlpPrefix` /
   `stripPathSuffix` helpers as the undici instrumentation.

SAFETY:

6. Request input + undefined init dropped the Request's own headers
   (Authorization etc.) because `new Headers(undefined)` → `{...init, headers}`
   replaced them with just our session header. Fix: when input is a Request
   and init.headers is unset, seed from input.headers before adding ours.

7. Wrapped fetch had no try/catch — a throwing Config getter or Headers
   constructor would propagate as TypeError and break the LLM request path.
   Wrapped header construction in try/catch; on failure, fall through to
   baseFetch with original init (no header) + diag.warn. Telemetry must
   never break the model call.

COVERAGE:

- 3 new sdk.test.ts boundary tests (port/host/path)
- 1 new sdk.test.ts normalizeOtlpPrefix catch-branch coverage
- 1 new sdk.test.ts HttpInstrumentation OTLP guard test
- 1 new sdk.test.ts proxy-mode wrapped-fetch test (default.test.ts)
- 1 new anthropic test asserting wrapped fetch installed on Anthropic SDK
- 2 new llm-correlation-fetch.test.ts (Request-headers preservation + try/catch fall-through)

All 668 tests pass (1 pre-existing Anthropic User-Agent failure on main is
unrelated). tsc clean.

Declined: #10 DRY-refactor of baseFetch extraction across 3 sites — the
duplication was pre-existing (default/dashscope buildClient was already
near-identical), refactoring is a separate cleanup PR not gated by this
feature. Will reply on the thread.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* chore(deps): allow patch updates for @opentelemetry/instrumentation-undici

Switch from exact pin `0.14.0` to `^0.14.0` for consistency with the rest
of the `@opentelemetry/*` deps in this block (all carated).

For 0.x semver, npm treats `^0.14.0` as `>=0.14.0 <0.15.0`, so patch
updates within the 0.14.x line — which are tied to the same
`@opentelemetry/instrumentation@0.203.x` peer — flow in via `npm update`
without requiring a manual package.json edit. A bump across the 0.x
minor (e.g. 0.15.x) would shift the instrumentation peer compatibility
and still requires explicit attention, which the caret correctly blocks.

Per review feedback on #4390 (wenshao).

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* test(telemetry): stub getTelemetryEnabled + getSessionId in Gemini factory tests

The X-Qwen-Code-Session-Id commit added a `staticCorrelationHeaders(gcConfig)`
call inside the Gemini content generator factory. That helper reads
`gcConfig.getTelemetryEnabled()` and `gcConfig.getSessionId()` per request.

Both pre-existing Gemini tests in `contentGenerator.test.ts` build a minimal
partial Config stub via `as unknown as Config` and only stub the methods the
factory used to need. The new call path now hits the unstubbed methods at
runtime, surfacing as `TypeError: config.getTelemetryEnabled is not a function`
on all three CI platforms.

Add the two missing stubs to both test cases. The Gemini factory continues
to ignore the values when telemetry is off — these stubs only have to exist,
not return anything in particular.

Local check ran the full test suite for the four directories `/loop` covers
plus `src/core/contentGenerator.test.ts` itself; all green. Also re-ran the
other test files that build partial Config mocks via the same idiom
(`client.test.ts`, `config.test.ts`, `nextSpeakerChecker.test.ts`,
`content-generator-config.test.ts`) — none exercise the new code path.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* fix(telemetry): R3 review fixes — port + protocol + quote + safety

Four issues found by wenshao reviewing the R2 boundary-safety pass on PR
#4390. All four close gaps where the OTLP feedback-loop guard or the
correlation-header path could fail silently.

1. **Port normalization mismatch** (sdk.ts ignoreOutgoingRequestHook):
   `normalizeOtlpPrefix` builds prefixes via `URL.origin`, which strips
   default ports (`:80` for http, `:443` for https). The hook reconstructed
   request origin manually as `${proto}://${host}${portPart}`, keeping the
   port. Result: prefix `http://collector` (no explicit port) didn't match
   a request to `http://collector:80/v1/traces` because their `.origin`
   differed → guard bypassed → feedback loop. Now the reconstructed origin
   is also routed through `URL` so both sides apply the same default-port
   stripping.

2. **HTTPS proto silent fallback** (sdk.ts ignoreOutgoingRequestHook):
   The `(req.protocol && ...) || 'http'` fallback would silently mis-bucket
   HTTPS requests as HTTP when `req.protocol` was unset, so HTTPS OTLP
   endpoints couldn't match their prefix. Changed to fail open: when proto
   can't be determined, return false (request gets instrumented). Worst
   case is a parasitic client span — observable, recoverable — versus the
   previous unbounded silent feedback loop. Picked fail-open over the bot's
   port-based heuristic because non-standard HTTPS ports break the
   heuristic but not fail-open.

3. **Quote-stripping divergence** (sdk.ts normalizeOtlpPrefix):
   `parseOtlpEndpoint` (line 109) uses `/^["']|["']$/g` which strips
   asymmetric leading/trailing quotes; `normalizeOtlpPrefix` previously
   only stripped symmetric pairs. A settings.json typo like `"value'` would
   let the exporter connect (parseOtlpEndpoint trims) but leave the guard
   returning `undefined` (normalizeOtlpPrefix rejected) → parasitic loop.
   Aligned `normalizeOtlpPrefix` to the same lenient regex.

4. **`staticCorrelationHeaders` missing try/catch** (llm-correlation-fetch.ts):
   `wrapFetchWithCorrelation` already catches all internal exceptions and
   falls through to baseFetch — same "telemetry must never break LLM path"
   contract was missing on the static-headers helper. A throw here would
   propagate up to the Gemini content-generator factory and crash
   content-generator init for the whole session. Wrapped the body in
   try/catch with `diag.warn` fall-through to `{}`.

Tests: added 4 regression tests covering each scenario:
- default-port HTTP request matched against portless prefix (1)
- hook returns false when req.protocol missing on https endpoint (2)
- asymmetric-quoted endpoint normalizes for guard parity (3)
- staticCorrelationHeaders returns {} when config getter throws (4)

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* docs(telemetry): fix misleading "BOTH" wording in wrapFetchWithCorrelation

The comment described the header-seeding logic as merging "BOTH the
init.headers AND the Request's own headers", but the two branches are
mutually exclusive — `new Headers(init?.headers)` runs unconditionally
(empty Headers when init.headers is undefined), and the Request-headers
copy only runs when init.headers is undefined. So in practice it's
either-or, not BOTH.

Reworded to match the actual logic per #4390 review feedback (wenshao).
Behavior unchanged.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* fix(telemetry): strip port from req.host fallback + document undici scope

Two issues found by wenshao reviewing the R3 boundary-safety fixes on PR
#4390.

1. **`req.host` may already include `:port`** (sdk.ts ignoreOutgoingRequestHook):
   When `req.hostname` is absent and `req.host` is the fallback, the value
   may already be `"collector:4318"`. Naively appending `:${req.port}`
   produced `"http://collector:4318:4318"` → `new URL()` rejects → catch
   returns false → silent guard bypass for that request. Currently
   unreachable because `@opentelemetry/otlp-exporter-base` always sets
   `hostname` from WHATWG URL parsing, but the fallback exists in the
   code and must be correct — a future OTLP transport that emits `host`
   without `hostname` would silently trigger the feedback loop. Strip the
   port when falling back; bracketed IPv6 literals like `"[::1]:443"`
   keep their bracketed host intact.

2. **Undici scope honesty** (telemetry.md):
   Previous docs framed the propagation as "outbound LLM requests", but
   `UndiciInstrumentation` actually patches `globalThis.fetch` for the
   whole process — `WebFetch`, MCP clients, IDE extension calls all get
   spans + `traceparent` injection too. Added a "Scope: all fetch() calls,
   not just LLM" subsection covering: (a) trace ID leakage to third-party
   URLs (the user-supplied destinations of `WebFetch` see our trace ID;
   not secret per W3C but worth knowing); (b) non-LLM span volume
   inflating OTLP batches with a workaround tip. Per-destination scoping
   toggle deferred as a follow-up — out of scope for this PR.

Added regression test for the host:port-fallback path. Test exercises
the previously broken combination (hostname absent, host carries port)
through the existing test harness.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* feat(telemetry): scope X-Qwen-Code-Session-Id to first-party hosts by default

Address LaZzyMan's REQUEST_CHANGES review of PR #4390.

The original design injected `X-Qwen-Code-Session-Id` on every outbound
LLM request gated only by `telemetry.enabled`. Review caught that this
broadcasts a stable cross-request client identifier to every configured
third-party provider (OpenAI, Anthropic, OpenRouter, MiniMax, ModelScope,
Mistral, vanilla Gemini, ...), which the claude-code precedent does NOT
justify — claude-code is a first-party Anthropic→Anthropic flow; qwen-code
is an open-source CLI connecting to many providers.

Fix: add a host allowlist with a deliberately narrow default. The header
is now only attached to destinations whose hostname matches:

  dashscope.aliyuncs.com
  dashscope-intl.aliyuncs.com
  *.dashscope.aliyuncs.com
  *.dashscope-intl.aliyuncs.com
  *.alibaba-inc.com
  *.aliyun-inc.com

This is exactly the set where the LLM provider, the upstream telemetry
backend (ARMS Tracing), and qwen-code itself are the same legal entity —
mirroring the first-party claude-code pattern and preserving the real
product value (server-side trace stitching against DashScope) without
exposing the session id to third parties.

Operators with broader correlation requirements override via:

  "telemetry": {
    "sessionIdHeaderHosts": ["*"]                          // restore broadcast
    "sessionIdHeaderHosts": []                              // fully disable
    "sessionIdHeaderHosts": ["api.example.com", "*.foo"]    // custom allowlist
  }

Implementation:

- NEW `telemetry/trusted-llm-hosts.ts`: `DEFAULT_SESSION_ID_HEADER_HOSTS`
  + `matchesTrustedHost(hostname, patterns)` + `extractRequestHost(input)`.
  Pattern syntax is intentionally tiny (bare hostname OR `*.suffix`,
  dot-anchored to reject `evil-alibaba-inc.com` style attacks). Unit-tested
  in dedicated test file including TLD/sub-domain attack vectors.

- `wrapFetchWithCorrelation` (openai + anthropic providers): resolves the
  allowlist at wrap time (Config snapshot), inspects each request's
  destination URL inside `correlationFetch`, falls through to baseFetch
  for non-trusted destinations. Wildcard escape hatch via `["*"]`.

- `staticCorrelationHeaders` (Gemini factory): now takes an optional
  `destinationUrl` and applies the same host gate. The Gemini SDK default
  endpoint `generativelanguage.googleapis.com` is NOT on the default
  allowlist, so vanilla Gemini calls receive no header — matching the
  "first-party only" scope. Operators who put the Gemini SDK on a
  DashScope-compatible endpoint via `baseUrl` get the header naturally.

- `Config.getTelemetrySessionIdHeaderHosts()` getter +
  `TelemetrySettings.sessionIdHeaderHosts` interface field + JSON schema
  entry in `settingsSchema.ts`. Wired through `resolveTelemetrySettings`.

- Defensive optional-chaining + try/catch on the Config getter call at
  wrap time so partial test mocks (or pre-getter Config implementations)
  fall back to the default allowlist rather than crashing buildClient.

Tests: 12 new cases covering host match/skip on default allowlist,
sub-domain handling, TLD-suffix attack rejection, `["*"]` broadcast
override, `[]` full-disable, custom operator allowlist, unparseable
destination (fail closed), and the three Gemini factory paths
(googleapis.com default → omit; DashScope `baseUrl` → inject; custom
allowlist → inject).

Docs updated in `docs/developers/development/telemetry.md` Session
correlation header section, including override examples and the new
Gemini host-gate semantics.

Closes the LaZzyMan REQUEST_CHANGES blocker. The cross-vendor
fingerprint-broadcast failure mode is now opt-in rather than default,
restoring the first-party-only semantics that make the claude-code
precedent applicable.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* fix(telemetry): R5 review fixups — Vertex destination + ["*"] trim + docs

Self-review pass on commit 1c8528a56 (host-scoped session-id header):

1. **Vertex AI destination guessing**
   (geminiContentGenerator/index.ts)
   `@google/genai` routes to `{region}-aiplatform.googleapis.com` (not
   `generativelanguage.googleapis.com`) when `vertexai: true` and no
   `baseUrl`. The previous "guess generativelanguage" default would have
   mis-bucketed Vertex traffic under any operator-supplied allowlist that
   covered the public Gemini endpoint but not the Vertex one. Today
   invisible (both off the default allowlist), but a latent gotcha for
   operators tuning `telemetry.sessionIdHeaderHosts`.
   Fix: pass `undefined` when `config.baseUrl` is unset (fail-closed —
   no header). Operators who want correlation against Google endpoints
   must set `baseUrl` explicitly, which is also the SDK's input for
   destination resolution.

2. **`["*"]` broadcast escape hatch tolerates whitespace**
   (llm-correlation-fetch.ts)
   `[" * "]` (a settings.json hand-edit with a stray space) previously
   silently fell back to "no host matches" — the opposite of operator
   intent. Now `.trim()` before comparing, so common whitespace mistakes
   still trigger broadcast.

3. **Doc note on wrap-time allowlist snapshot**
   (llm-correlation-fetch.ts JSDoc)
   The session id is read live per-request, but `trustedHosts` is
   snapshotted once at `wrapFetchWithCorrelation` call time. Spell this
   out in the JSDoc so a future maintainer doesn't read the live
   `getSessionId()` and assume the allowlist is the same shape.

4. **Defensive test coverage**
   (trusted-llm-hosts.test.ts, llm-correlation-fetch.test.ts)
   Added: extractRequestHost with explicit port / userinfo / query /
   fragment / IPv6 bracket form. Whitespace `[" * "]` broadcast test.
   IPv6 case documents the "bracketed → never matches" behavior is
   intentional fail-closed for the named-host allowlist scope.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* chore: regenerate settings.schema.json for sessionIdHeaderHosts

Lint check `Check settings schema is up-to-date` failed because the
checked-in `packages/vscode-ide-companion/schemas/settings.schema.json`
wasn't regenerated after adding `telemetry.sessionIdHeaderHosts` to
`settingsSchema.ts` in commit 1c8528a56. Regenerated via
`npm run generate:settings-schema`.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* docs(design): update telemetry-outbound-propagation design for R3 host-allowlist scoping

Adds a "修订历史" header table at the top and a new §11 "R3 修订 —
Host-Allowlist Scoping for X-Qwen-Code-Session-Id" capturing what changed
after LaZzyMan's REQUEST_CHANGES review, why, and how. Inline pointers
added at §3.1, §3.2, §4.3, §4.4, §9 (claude-code comparison table) to
point readers at §11 — original prose preserved as a record of the
decision path rather than rewritten in place.

Concretely §11 covers:
- The three-step LazzyMan critique and why R1's "broadcast to all
  providers" was structurally wrong for an open-source multi-provider CLI
- The default allowlist (`DEFAULT_SESSION_ID_HEADER_HOSTS`) and its
  semantic alignment with the DashScope provider detector
- Pattern grammar (bare hostname / `*.suffix` dot-anchored), the
  TLD-suffix attack vectors it rejects, why no regex / port-aware globbing
- `wrapFetchWithCorrelation` host gate, wrap-time vs request-time
  semantics, `[" * "]` whitespace tolerance
- `staticCorrelationHeaders` `destinationUrl` parameter, Gemini factory's
  fail-closed treatment of unset `baseUrl` (avoids the Vertex
  vs `generativelanguage.googleapis.com` ambiguity)
- All R3 file changes mapped to the original §5 file-change list
- Mapping of LazzyMan's three concerns to R3's responses
- §10 future-work additions: `traceparent` per-destination toggle,
  `X-Qwen-Code-Request-Id`, IPv6 allowlist syntax

No code changes; documentation only.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* fix(telemetry): defensive allowlist normalization + positive proxy test

Three issues found by wenshao reviewing the R3 host-allowlist scoping.

1. **[Critical] `broadcastAll` outside safety try/catch**
   (llm-correlation-fetch.ts wrapFetchWithCorrelation)
   The try/catch only fires when `getTelemetrySessionIdHeaderHosts()`
   throws. If it returns a malformed value — a bare string (settings.json
   typo `"sessionIdHeaderHosts": "host"` instead of `["host"]`), an array
   containing `null`/`undefined`/number entries, or whitespace-padded
   entries — `.some((p) => p.trim() === '*')` throws TypeError at
   buildClient time, bricking the LLM session before the first prompt.
   `staticCorrelationHeaders` already handled this via its end-to-end
   try/catch but the sister helper diverged. Settings loader does no
   runtime schema validation so this is reachable via a single typo.

   Fix: normalize the allowlist at wrap time:
     1. catch a throwing getter (existing)
     2. reject non-array → default allowlist (NEW — bare string typo)
     3. filter out non-string elements (NEW — [null, ...] typo)
     4. trim every surviving entry uniformly (NEW — see #2 below)
   Then `trustedHosts.includes('*')` instead of `.some((p) => p.trim() === '*')`,
   since patterns are already pre-trimmed.

2. **Trim asymmetry between `*` detection and host-pattern match**
   (llm-correlation-fetch.ts)
   `[" * "]` was tolerated (trimmed before `===` compare) but
   `[" dashscope.aliyuncs.com "]` silently never matched. The
   normalization above fixes this by trimming uniformly upstream.

3. **Proxy fetch test: only negative assertions**
   (openaiContentGenerator/provider/default.test.ts)
   The test asserted `callArg.fetch !== proxyFetch` and `!== globalThis.fetch`
   but both passed for ANY wrapper, including a buggy one that
   accidentally wraps globalThis.fetch instead of proxyFetch. Added a
   positive assertion: call the wrapped fetch and verify proxyFetch was
   the delegation target.

Tests: 4 new cases — whitespace-padded host pattern, bare-string
malformed config (both wrapper and static), null/number-containing
array malformed config (both wrapper and static), positive proxy fetch
delegation. All pass; pre-existing Anthropic User-Agent failure
unrelated.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* refactor(telemetry): split outbound correlation out of telemetry scope (R4)

Address LaZzyMan round-8 follow-up review on PR #4390: even though R3's
host allowlist made the default behavior safe, the meta-architectural
concern remains: telemetry's namespace and consent flow shouldn't quietly
extend to wire-level behavior aimed at third-party LLM provider request
streams. The recipient sets differ; the consent decisions differ; they
deserve separate namespaces, separate threat models, separate PRs.

This commit (called R4 in the design doc) collapses the PR scope so it
lands ONLY telemetry observability work:

REMOVED from this PR:
  - packages/core/src/telemetry/llm-correlation-fetch.ts(.test.ts)
  - packages/core/src/telemetry/trusted-llm-hosts.ts(.test.ts)
  - telemetry.sessionIdHeaderHosts setting + Config getter +
    resolveTelemetrySettings wiring + settingsSchema entry
  - wrapFetchWithCorrelation usage from four provider construction
    points (default.ts, dashscope.ts, anthropicContentGenerator.ts,
    geminiContentGenerator/index.ts)
  - All session-id provider tests across the four providers + the
    contentGenerator.test.ts mock stub
  - "Session correlation header" section in telemetry.md

ADDED:
  - OutboundCorrelationSettings interface in packages/core/src/config/config.ts,
    standalone top-level namespace separate from TelemetrySettings —
    SECURITY-RELEVANT label, all defaults off
  - Config.getOutboundCorrelationPropagateTraceContext() getter
  - outboundCorrelation top-level entry in settingsSchema.ts with
    propagateTraceContext: { default: false } and explicit
    SECURITY-RELEVANT framing in the description
  - CLI config-load pipeline passes settings.outboundCorrelation into
    ConfigParameters
  - NOOP_PROPAGATOR (TextMapPropagator no-op) in sdk.ts, conditionally
    installed on NodeSDK when propagateTraceContext is false (default).
    When true, omits textMapPropagator from NodeSDK options so the SDK
    keeps its default W3C composite propagator
  - 2 new sdk.test.ts cases covering the propagator gate behavior

UNCHANGED:
  - UndiciInstrumentation registration + OTLP feedback-loop guard +
    HttpInstrumentation OTLP guard from R2/R3 stay intact — they are
    pure telemetry (client HTTP spans into the operator's own OTLP
    collector), no wire-level data egress
  - Documentation rewrites telemetry.md to split "client-side HTTP
    span on outbound fetch" (telemetry) from a new "Outbound
    correlation (SECURITY-RELEVANT)" top-level section
  - design doc gets R4 revision row + new §12 "R4 Scope Conflation
    Split" capturing the rationale and follow-up PR outline

The session-id apparatus (R3 code) lives in git history at commits
1c8528a56 / cb162e716 / 7a1b4f8d0 / 40e1efc1f / 106598ca2; the
follow-up PR can cherry-pick or restore those files under the new
outboundCorrelation.* namespace as LazzyMan suggested.

Vscode-ide-companion settings.schema.json regenerated.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* docs(telemetry): disclose telemetry.enabled dependency on propagateTraceContext

Self-review pass on R4 commit 9bdd3bd6f flagged one footgun: both
`docs/developers/development/telemetry.md` and the settingsSchema.ts
description for `outboundCorrelation.propagateTraceContext` describe
the toggle's behavior without noting that the flag is a silent no-op
when `telemetry.enabled` is false. An operator who sets only
`outboundCorrelation.propagateTraceContext: true` and forgets the
telemetry switch gets zero behavior change — no error, no warning, no
traceparent.

Fix: add the dependency disclosure to both surfaces, plus a JSON
example showing both flags wired together for the ARMS+DashScope
cross-process trace continuation use case.

Also fix a minor comment accuracy nit at `sdk.test.ts:683`: said the
SDK installs `W3CTraceContextPropagator` instance when opt-in is true,
but the actual default is `CompositePropagator(W3CTraceContextPropagator
+ W3CBaggagePropagator)` per `@opentelemetry/sdk-node` source.

Vscode-ide-companion settings.schema.json regenerated to reflect the
expanded description string.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* test(config): cover getOutboundCorrelationPropagateTraceContext defaults

R4 (commit 9bdd3bd6f) added the getter but the test file didn't grow a
corresponding describe block — sibling telemetry getters all have unit
tests but this new one was missed.

Add 4 cases covering the security-relevant default-to-false invariant
and explicit-set behavior:
- omitted outboundCorrelation → false
- empty outboundCorrelation: {} → false (the `?? false` collapse on the
  getter, complementing the same on the constructor)
- explicit true → true
- explicit false → false

PR #4390 review (wenshao).

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* docs(design): reflect post-R4 polish commits in §12

R4 (commit 9bdd3bd6f) was followed by two polish commits that the
design doc §12 didn't track:

- 0be0df270 (docs): telemetry.enabled dependency disclosure on
  propagateTraceContext — added to telemetry.md + settingsSchema
  description because a self-review pass identified the silent-no-op
  footgun (operator sets propagateTraceContext: true but forgets
  telemetry.enabled: true, sees zero behavior change with no error).
- c0352fd5b (test): 4 config.test.ts cases covering the
  getOutboundCorrelationPropagateTraceContext default-false invariant
  (omitted / {} / explicit true / explicit false) — wenshao review
  flagged the test gap.

Updates §12.4 with a new "Hidden dependency: telemetry.enabled" sub-
section explaining the gating relationship and pointing forward at the
follow-up PR (future outboundCorrelation.* settings inherit the same
dependency). Updates §12.5 implementation table to add the
config.test.ts row and clarify the telemetry.md / vscode-schema rows
were touched again in the polish pass.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)

* refactor: simplify post-R4 polish per /simplify review

/simplify review pass on commits 0be0df270 + c0352fd5b + 62cf6b4ee
flagged 4 concerns. Fix all 4:

1. **settingsSchema.ts description**: footgun warning ("Depends on
   telemetry.enabled: true") was at char 600+ of a 650-char description.
   VS Code settings UI truncates to ~300 chars inline → the most
   important warning was hidden in the most-glanced view. Hoist to first
   sentence ("Requires telemetry.enabled: true.").

2. **config.test.ts**: drop the task-narration comment
   ("PR #4390 R4: keep wire-level toggle out of telemetry namespace.")
   that just restated the change context. The remaining 2-line comment
   explaining WHY default-to-false is security-relevant survives.

3. **config.test.ts**: collapse 4 separate `it()` blocks into a single
   `it.each([...])` covering the same 4 precondition × expectation
   combinations. Removes boilerplate (`new Config({...baseParams, ...})`
   repeated 4×) without losing assertion power; case-3 ("explicit false")
   was a weak duplicate of case-2 ("empty object") since both hit the
   same `?? false` branch, but keeping all 4 in the parametric table
   documents intent more clearly than dropping case-3.

4. **design doc §12.4 + §12.5**: strip specific commit SHAs
   (`0be0df270`, `c0352fd5b`) — design docs should be evergreen, not
   doubled-up commit logs (those live in `git log`). Keep the design
   intent ("two panels both document the dependency" / "test block
   added") without naming the specific commits.

Regenerated vscode-ide-companion/schemas/settings.schema.json to
reflect the hoisted description sentence.

🤖 Generated with [Qwen Code](https://github.com/QwenLM/qwen-code)
2026-05-25 22:16:54 +08:00

644 lines
29 KiB
Markdown

# Observability with OpenTelemetry
Learn how to enable and setup OpenTelemetry for Qwen Code.
- [Observability with OpenTelemetry](#observability-with-opentelemetry)
- [Key Benefits](#key-benefits)
- [OpenTelemetry Integration](#opentelemetry-integration)
- [Configuration](#configuration)
- [Aliyun Telemetry](#aliyun-telemetry)
- [Manual OTLP Export](#manual-otlp-export)
- [Local Telemetry](#local-telemetry)
- [File-based Output (Recommended)](#file-based-output-recommended)
- [Collector-Based Export (Advanced)](#collector-based-export-advanced)
- [Logs and Metrics](#logs-and-metrics)
- [Logs](#logs)
- [Metrics](#metrics)
## Key Benefits
- **🔍 Usage Analytics**: Understand interaction patterns and feature adoption
across your team
- **⚡ Performance Monitoring**: Track response times, token consumption, and
resource utilization
- **🐛 Real-time Debugging**: Identify bottlenecks, failures, and error patterns
as they occur
- **📊 Workflow Optimization**: Make informed decisions to improve
configurations and processes
- **🏢 Enterprise Governance**: Monitor usage across teams, track costs, ensure
compliance, and integrate with existing monitoring infrastructure
## OpenTelemetry Integration
Built on **[OpenTelemetry]** — the vendor-neutral, industry-standard
observability framework — Qwen Code's observability system provides:
- **Universal Compatibility**: Export to any OpenTelemetry backend (Aliyun,
Jaeger, Prometheus, Datadog, etc.)
- **Standardized Data**: Use consistent formats and collection methods across
your toolchain
- **Future-Proof Integration**: Connect with existing and future observability
infrastructure
- **No Vendor Lock-in**: Switch between backends without changing your
instrumentation
[OpenTelemetry]: https://opentelemetry.io/
[aliyun-opentelemetry-overview]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/product-overview/what-is-tracing-analysis
[aliyun-opentelemetry-get-started]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/before-you-begin
[aliyun-opentelemetry-console-cn]: https://trace.console.aliyun.com
[aliyun-opentelemetry-console-cn-legacy]: https://tracing.console.aliyun.com
[aliyun-opentelemetry-console-intl]: https://arms.console.alibabacloud.com
## Configuration
All telemetry behavior is controlled through your `.qwen/settings.json` file.
These settings can be overridden by environment variables or CLI flags.
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
| -------------------------------- | -------------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ----------------------- |
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing — set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
**Note on boolean environment variables:** For the boolean settings (`enabled`,
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
corresponding environment variable to `true` or `1` will enable the feature. Any
other value will disable it.
**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
two things happen:
1. **Native span attributes (`qwen-code.interaction`, `api.generateContent*`,
`tool.<name>`)** carry verbatim conversation content:
- User prompts (`new_context`)
- System prompts (`system_prompt` — full text once per session, deduped by
SHA-256 hash; subsequent spans only carry `system_prompt_hash` +
`system_prompt_preview` + `system_prompt_length`)
- Tool schemas (emitted as `tool_schema` events, also hash-deduped)
- Tool inputs (`tool_input`) and tool results (`tool_result`)
- Model output (`response.model_output`)
Each value is truncated at 60 KB; `*_truncated` and `*_original_length`
flags surface when truncation occurs.
2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
logs endpoint) keep their existing `prompt`, `function_args`, and
`response_text` fields, instead of being dropped.
⚠️ **Security warning:** enabling this flag streams full conversation history,
file contents read by `read_file`, shell commands and their output (including
secrets in env vars or arguments), and model responses to the configured OTLP
backend. Treat the backend as a privileged data sink. The flag defaults to
`false`.
**Cost / payload size:** A heavy turn (60 KB system prompt + 10 tool calls,
each up to 60 KB input + 60 KB result, plus 60 KB model output) can produce up
to ~1.5 MB of attribute payload before OTLP compression. When pointing tools
that read large files (`read_file`, etc.) at long-running sessions, monitor
exporter throughput.
This setting does not disable sensitive data in OTel logs or other telemetry
sinks; non-internal API response telemetry can populate `response_text`, so
OTel logs, UI telemetry, and chat recording may receive response text
independently of this setting. QwenLogger does not include `response_text`.
**HTTP OTLP signal routing:** When using HTTP protocol (`otlpProtocol: "http"`),
Qwen Code automatically appends signal-specific paths (`/v1/traces`, `/v1/logs`,
`/v1/metrics`) to the base `otlpEndpoint`. For example, `http://collector:4318`
becomes `http://collector:4318/v1/traces` for traces. If the URL already ends
with a signal path, it is used as-is. Per-signal endpoint overrides
(`otlpTracesEndpoint`, etc.) take precedence over the base endpoint and are used
verbatim. gRPC protocol uses service-based routing and does not append paths.
The per-signal endpoint environment variables also accept the standard
OpenTelemetry names: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
For detailed information about all configuration options, see the
[Configuration Guide](./cli/configuration.md).
### Resource attributes
Resource attributes are static key-value pairs attached to every span, log,
and metric exported via OTLP. Use them to slice telemetry by team, environment,
deployment region, or any other dimension your backend cares about.
Two sources, merged in priority order (lowest → highest):
1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
key conflict)
`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides
`service.name` from any other source (per the OpenTelemetry spec).
#### Examples
**Slice all telemetry by team / environment:**
```bash
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
```
**Route to a per-tenant collector via `service.name`:**
```bash
export OTEL_SERVICE_NAME=qwen-code-ci
```
**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**
```json
{
"telemetry": {
"resourceAttributes": {
"deployment.environment": "production",
"service.namespace": "engineering-tooling"
}
}
}
```
```bash
# Add a one-off tag without touching settings:
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
```
#### Reserved keys
Some keys are runtime-controlled and cannot be overridden:
- `service.version` — always set to the running CLI version. Setting it from
any source is silently dropped with a warning.
- `session.id` — runtime-injected per session. User-provided values from
either env or settings are dropped with a warning. The reason is that
Resource attributes auto-attach to every metric data point; allowing user
override would bypass [Cardinality controls](#cardinality-controls) below.
Spans and logs always carry `session.id`.
`service.name` is **not** reserved; it follows the precedence chain above.
#### Format
`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
the wrong boundary and the second half is dropped as malformed). Malformed
pairs are skipped with a warning rather than failing telemetry startup.
#### Troubleshooting: when a user-provided attribute appears not to take effect
Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
settings values, and invalid percent-encoding are all silently dropped with a
warning logged via the OpenTelemetry diagnostics channel. That channel routes
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
behavior can look like silent failure.
If a custom resource attribute isn't appearing on exported telemetry:
1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
(non-string settings value).
2. Verify the env var is set in the qwen-code process's environment (not just
your shell) and that values are percent-encoded.
3. Confirm `telemetry.enabled` is `true` — telemetry init only runs if enabled.
### Cardinality controls
Metrics are aggregated by attribute set at the backend — every distinct
combination of attribute values produces a new time series. Attaching a
high-cardinality field like `session.id` to a metric causes time-series fan-out
proportional to the number of sessions, which quickly exhausts metric backend
storage.
To prevent this, Qwen Code keeps high-cardinality attributes off metric data
points by default. Spans and logs are per-event and unaffected, so they
continue to carry `session.id` for trace and log correlation.
#### `telemetry.metrics.includeSessionId` (default: `false`)
Setting this to `true` (via settings or
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
every metric data point.
⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a
fleet will blow up metric storage. Recommended only for short-term debugging.
For long-term session correlation, query trace or log backends instead.
#### Migration from earlier versions
Prior to this release, `session.id` was attached to metrics by default. If
your Prometheus queries / Grafana dashboards / alert rules reference
`session_id` on a metric, you have two options:
**Option A** — restore the previous behavior for short-term debugging:
```bash
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
```
or:
```json
{
"telemetry": {
"metrics": { "includeSessionId": true }
}
}
```
**Option B (recommended)** — move session-level analysis off metrics. Spans
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
cardinality pressure.
### Client-side HTTP span on outbound fetch
When telemetry is enabled, Qwen Code registers `UndiciInstrumentation`
which creates a client-side HTTP span for every outbound `fetch()`
request originated by the process — including the LLM SDKs (`openai`,
`@google/genai`, `@anthropic-ai/sdk`), the MCP StreamableHTTP client, the
`WebFetch` tool, and any IDE-extension out-of-process calls. The span
lets you see network latency (TTFB / response body transfer) separately
from upstream model processing time, which the existing
`api.generateContent` span alone can't distinguish.
These spans go to your **own** OTLP collector (or file outfile) just like
the rest of the telemetry — they do not affect what is written onto the
outbound HTTP request itself. Whether the W3C `traceparent` header is
also written into the outgoing request stream is controlled by a
**separate, security-relevant setting** documented in
[outbound correlation](#outbound-correlation-security-relevant) below.
**Feedback-loop avoidance.** OTel SDK uses `fetch` internally to upload OTLP
data. Without protection, instrumenting `fetch` would trace those uploads,
which would themselves be uploaded, causing an infinite loop. Qwen Code's
undici instrumentation is configured with an `ignoreRequestHook` that skips
URLs matching the configured `telemetry.otlpEndpoint` /
`telemetry.otlpTracesEndpoint` / `telemetry.otlpLogsEndpoint` /
`telemetry.otlpMetricsEndpoint` prefixes. In file-outfile mode there are no
outbound HTTP uploads, so the hook is a no-op.
## Outbound correlation (SECURITY-RELEVANT)
These settings live in a **separate top-level namespace** from `telemetry.*`
on purpose: telemetry controls data flow into the operator's own
observability backend, while `outboundCorrelation.*` controls what
client-side correlation data qwen-code writes **into outbound LLM API
request streams** that reach third-party LLM provider endpoints
(DashScope, OpenAI, Anthropic, etc.). Different recipients, different
consent decision. **All values default to off.** See PR #4390 review
discussion for the framing rationale.
### `outboundCorrelation.propagateTraceContext`
```jsonc
"outboundCorrelation": {
"propagateTraceContext": false // default
}
```
When `false` (default), Qwen Code installs a no-op `TextMapPropagator` on
the OTel SDK. UndiciInstrumentation still creates client HTTP spans for
your OTLP collector, but `propagation.inject()` is a no-op so **no
`traceparent` is written onto outbound requests**. Trace IDs stay
internal to the operator's collector.
When `true`, the SDK's default W3C composite propagator
(`tracecontext` + `baggage`) is installed and the standard `traceparent`
header is written on every outbound `fetch`:
```
traceparent: 00-<32-hex traceId>-<16-hex parentSpanId>-<01-sampled | 00-not-sampled>
```
Opt in only when the LLM provider also reports into your OTel collector
for cross-process trace stitching — e.g. ARMS Tracing serving DashScope.
For most operators the value is `false`; cross-vendor trace continuation
is niche.
**Depends on `telemetry.enabled: true`.** The OTel SDK only initializes
when telemetry is enabled, so `propagateTraceContext` only takes effect
in that state. Setting it to `true` while telemetry is disabled is a
silent no-op — no SDK, no propagator, no `traceparent` on the wire.
Verify both flags when wiring an ARMS+DashScope correlation setup:
```jsonc
{
"telemetry": {
"enabled": true,
"otlpTracesEndpoint": "http://tracing-analysis-...",
},
"outboundCorrelation": {
"propagateTraceContext": true,
},
}
```
### Other outbound correlation headers
`X-Qwen-Code-Session-Id` and `X-Qwen-Code-Request-Id` are **not part of
this PR**. They will be designed and proposed in their own follow-up
PR(s) under the same `outboundCorrelation.*` namespace, each with its
own threat model and operator-consent flow. PR #4390 review (LaZzyMan)
established the principle: "telemetry's scope of work doesn't include
sending identifiers to LLM providers"; correlation-header work moves to
its own design discussion rather than landing under telemetry.
## Aliyun Telemetry
### Manual OTLP Export
To view Qwen Code telemetry in Alibaba Cloud Managed Service for
OpenTelemetry, configure Qwen Code to export to the OTLP endpoint
provided by ARMS.
Setting `"target": "gcp"` alone does not configure the export
destination. If `otlpEndpoint` is not set, Qwen Code still defaults to
`http://localhost:4317`. If `outfile` is set, it overrides
`otlpEndpoint` and telemetry is written to the file instead of being
sent to Alibaba Cloud.
1. Enable telemetry in your `.qwen/settings.json` and set the OTLP
endpoint:
**Option A: gRPC protocol** (standard OTLP endpoint):
```json
{
"telemetry": {
"enabled": true,
"target": "gcp",
"otlpEndpoint": "https://<your-otlp-endpoint>",
"otlpProtocol": "grpc"
}
}
```
**Option B: HTTP protocol with per-signal endpoints** (for backends
that use non-standard paths, e.g., `/api/otlp/traces` instead of
`/v1/traces`):
```json
{
"telemetry": {
"enabled": true,
"otlpProtocol": "http",
"otlpTracesEndpoint": "http://<host>/<token>/api/otlp/traces",
"otlpLogsEndpoint": "http://<host>/<token>/api/otlp/logs",
"otlpMetricsEndpoint": "http://<host>/<token>/api/otlp/metrics"
}
}
```
> **Note:** When using HTTP protocol with only `otlpEndpoint` (no
> per-signal overrides), Qwen Code appends standard OTLP paths
> (`/v1/traces`, `/v1/logs`, `/v1/metrics`) to the base URL. If your
> backend uses different paths, use per-signal endpoint overrides as
> shown in Option B.
2. If your Alibaba Cloud endpoint requires authentication, provide OTLP
headers through standard OpenTelemetry environment variables such as
`OTEL_EXPORTER_OTLP_HEADERS` (or the signal-specific variants). Qwen
Code does not currently expose OTLP auth headers directly in
`.qwen/settings.json`.
3. Run Qwen Code and send prompts.
4. View telemetry in Managed Service for OpenTelemetry:
- Product overview:
[What is Managed Service for OpenTelemetry?][aliyun-opentelemetry-overview]
- Getting started:
[Get started with Managed Service for OpenTelemetry][aliyun-opentelemetry-get-started]
- Console entry points:
- China mainland:
[trace.console.aliyun.com][aliyun-opentelemetry-console-cn]
(legacy console:
[tracing.console.aliyun.com][aliyun-opentelemetry-console-cn-legacy])
- International:
[arms.console.alibabacloud.com][aliyun-opentelemetry-console-intl]
- In the console, use `Applications` to inspect traces and service
topology.
- To locate the OTLP endpoint and access information:
- **New console** (`trace.console.aliyun.com` or international):
navigate to `Integration Center`.
- **Legacy console** (`tracing.console.aliyun.com`): navigate to
`Cluster Configurations` → `Access point information`.
## Local Telemetry
For local development and debugging, you can capture telemetry data locally:
### File-based Output (Recommended)
1. Enable telemetry in your `.qwen/settings.json`:
```json
{
"telemetry": {
"enabled": true,
"outfile": ".qwen/telemetry.log"
}
}
```
> **Note:** When `outfile` is set, OTLP export is automatically disabled.
> The `target` and `otlpEndpoint` settings are not needed for file-only
> output and can be safely omitted from your config.
2. Run Qwen Code and send prompts.
3. View logs and metrics in the specified file (e.g., `.qwen/telemetry.log`).
### Collector-Based Export (Advanced)
1. Run the automation script:
```bash
npm run telemetry -- --target=local
```
This will:
- Download and start Jaeger and OTEL collector
- Configure your workspace for local telemetry
- Provide a Jaeger UI at http://localhost:16686
- Save logs/metrics to `~/.qwen/tmp/<projectHash>/otel/collector.log`
- Stop collector on exit (e.g. `Ctrl+C`)
2. Run Qwen Code and send prompts.
3. View traces at http://localhost:16686 and logs/metrics in the collector log
file.
## Logs and Metrics
The following section describes the structure of logs and metrics generated for
Qwen Code.
- A `sessionId` is included as a common attribute on all logs and metrics.
### Logs
Logs are timestamped records of specific events. The following events are logged for Qwen Code:
- `qwen-code.config`: This event occurs once at startup with the CLI's configuration.
- **Attributes**:
- `model` (string)
- `sandbox_enabled` (boolean)
- `core_tools_enabled` (string)
- `approval_mode` (string)
- `file_filtering_respect_git_ignore` (boolean)
- `debug_mode` (boolean)
- `truncate_tool_output_threshold` (number)
- `truncate_tool_output_lines` (number)
- `hooks` (string, comma-separated hook event types, omitted if hooks disabled)
- `ide_enabled` (boolean)
- `interactive_shell_enabled` (boolean)
- `mcp_servers` (string)
- `output_format` (string: "text" or "json")
- `qwen-code.user_prompt`: This event occurs when a user submits a prompt.
- **Attributes**:
- `prompt_length` (int)
- `prompt_id` (string)
- `prompt` (string, this attribute is excluded if `log_prompts_enabled` is
configured to be `false`)
- `auth_type` (string)
- `qwen-code.tool_call`: This event occurs for each function call.
- **Attributes**:
- `function_name`
- `function_args`
- `duration_ms`
- `success` (boolean)
- `decision` (string: "accept", "reject", "auto_accept", or "modify", if
applicable)
- `error` (if applicable)
- `error_type` (if applicable)
- `content_length` (int, if applicable)
- `metadata` (if applicable, dictionary of string -> any)
- `qwen-code.file_operation`: This event occurs for each file operation.
- **Attributes**:
- `tool_name` (string)
- `operation` (string: "create", "read", "update")
- `lines` (int, if applicable)
- `mimetype` (string, if applicable)
- `extension` (string, if applicable)
- `programming_language` (string, if applicable)
- `diff_stat` (json string, if applicable): A JSON string with the following members:
- `ai_added_lines` (int)
- `ai_removed_lines` (int)
- `user_added_lines` (int)
- `user_removed_lines` (int)
- `qwen-code.api_request`: This event occurs when making a request to Qwen API.
- **Attributes**:
- `model`
- `request_text` (if applicable)
- `qwen-code.api_error`: This event occurs if the API request fails.
- **Attributes**:
- `model`
- `error`
- `error_type`
- `status_code`
- `duration_ms`
- `auth_type`
- `qwen-code.api_response`: This event occurs upon receiving a response from Qwen API.
- **Attributes**:
- `model`
- `status_code`
- `duration_ms`
- `error` (optional)
- `input_token_count`
- `output_token_count`
- `cached_content_token_count`
- `thoughts_token_count`
- `response_text` (if applicable)
- `auth_type`
- `qwen-code.tool_output_truncated`: This event occurs when the output of a tool call is too large and gets truncated.
- **Attributes**:
- `tool_name` (string)
- `original_content_length` (int)
- `truncated_content_length` (int)
- `threshold` (int)
- `lines` (int)
- `prompt_id` (string)
- `qwen-code.malformed_json_response`: This event occurs when a `generateJson` response from Qwen API cannot be parsed as a json.
- **Attributes**:
- `model`
- `qwen-code.flash_fallback`: This event occurs when Qwen Code switches to flash as fallback.
- **Attributes**:
- `auth_type`
- `qwen-code.slash_command`: This event occurs when a user executes a slash command.
- **Attributes**:
- `command` (string)
- `subcommand` (string, if applicable)
- `qwen-code.extension_enable`: This event occurs when an extension is enabled
- `qwen-code.extension_install`: This event occurs when an extension is installed
- **Attributes**:
- `extension_name` (string)
- `extension_version` (string)
- `extension_source` (string)
- `status` (string)
- `qwen-code.extension_uninstall`: This event occurs when an extension is uninstalled
### Metrics
Metrics are numerical measurements of behavior over time. The following metrics are collected for Qwen Code (metric names remain `qwen-code.*` for compatibility):
- `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup.
- `qwen-code.tool.call.count` (Counter, Int): Counts tool calls.
- **Attributes**:
- `function_name`
- `success` (boolean)
- `decision` (string: "accept", "reject", or "modify", if applicable)
- `tool_type` (string: "mcp", or "native", if applicable)
- `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency.
- **Attributes**:
- `function_name`
- `decision` (string: "accept", "reject", or "modify", if applicable)
- `qwen-code.api.request.count` (Counter, Int): Counts all API requests.
- **Attributes**:
- `model`
- `status_code`
- `error_type` (if applicable)
- `qwen-code.api.request.latency` (Histogram, ms): Measures API request latency.
- **Attributes**:
- `model`
- `qwen-code.token.usage` (Counter, Int): Counts the number of tokens used.
- **Attributes**:
- `model`
- `type` (string: "input", "output", "thought", or "cache")
- `qwen-code.file.operation.count` (Counter, Int): Counts file operations.
- **Attributes**:
- `operation` (string: "create", "read", "update"): The type of file operation.
- `lines` (Int, if applicable): Number of lines in the file.
- `mimetype` (string, if applicable): Mimetype of the file.
- `extension` (string, if applicable): File extension of the file.
- `model_added_lines` (Int, if applicable): Number of lines added/changed by the model.
- `model_removed_lines` (Int, if applicable): Number of lines removed/changed by the model.
- `user_added_lines` (Int, if applicable): Number of lines added/changed by user in AI proposed changes.
- `user_removed_lines` (Int, if applicable): Number of lines removed/changed by user in AI proposed changes.
- `programming_language` (string, if applicable): The programming language of the file.
- `qwen-code.chat_compression` (Counter, Int): Counts chat compression operations
- **Attributes**:
- `tokens_before`: (Int): Number of tokens in context prior to compression
- `tokens_after`: (Int): Number of tokens in context after compression