qwen-code/docs/developers/development/telemetry.md
ChiGao d96f264de7
Some checks are pending
E2E Tests / E2E Test (Linux) - sandbox:docker - shard 1/3 (push) Waiting to run
E2E Tests / E2E Test (Linux) - sandbox:docker - shard 2/3 (push) Waiting to run
E2E Tests / E2E Test (Linux) - sandbox:docker - shard 3/3 (push) Waiting to run
E2E Tests / E2E Test (Linux) - sandbox:none - shard 1/3 (push) Waiting to run
E2E Tests / E2E Test (Linux) - sandbox:none - shard 2/3 (push) Waiting to run
E2E Tests / E2E Test (Linux) - sandbox:none - shard 3/3 (push) Waiting to run
E2E Tests / E2E Test - macOS - shard 1/2 (push) Waiting to run
E2E Tests / E2E Test - macOS - shard 2/2 (push) Waiting to run
E2E Tests / channel-plugin E2E (nightly) (push) Waiting to run
E2E Tests / cron-interactive E2E (nightly) (push) Waiting to run
E2E Tests / web-shell Browser Regression (push) Waiting to run
npm cache producer / Save npm cache (push) Waiting to run
SDK Java / ubuntu-latest / Java 11 (push) Waiting to run
SDK Java / ubuntu-latest / Java 17 (push) Waiting to run
SDK Java / macos-latest / Java 21 (push) Waiting to run
SDK Java / ubuntu-latest / Java 21 (push) Waiting to run
SDK Java / windows-latest / Java 21 (push) Waiting to run
SDK Java / Real daemon E2E / Java 11 (push) Waiting to run
Security Checks / Dependency CVE audit (push) Waiting to run
Security Checks / Secret scan (TruffleHog) (push) Waiting to run
feat(telemetry): link daemon HTTP request spans to inbound W3C traceparent (#9391)
* feat(telemetry): link daemon HTTP request spans to inbound W3C traceparent

The daemon HTTP surface records a request span per request, but every span
starts a new trace: a caller forwarding the standard W3C traceparent header
(OTel-instrumented clients, proxies, gateways) gets no linkage back to its
own trace.

Extract traceparent/tracestate from inbound request headers in the daemon
telemetry middleware and parent the request span to that remote context.
Extraction reuses the same path as the existing JSON-RPC _meta extraction
(global propagator first, strict manual fallback so behavior is identical
without a registered SDK) and fails closed: requests without a valid header
keep the exact current span shape.

* fix(telemetry): guard inbound traceparent sampling and align W3C fallback

- Force TraceFlags.SAMPLED on inbound HTTP parents via the existing
  shouldForceSampled() matrix: an unsampled remote parent under the
  default parentbased_always_on sampler silently dropped the request
  span, the whole next() subtree, and the session-subprocess spans
  forwarded via _meta (review C1).
- Replace the hand-rolled manual fallback parser with a direct
  W3CTraceContextPropagator instance so acceptance rules (future
  versions, tracestate, all-zero ids, version-00 extension field)
  match the registered path with or without an initialized SDK.
- Gate middleware extraction behind isTelemetrySdkInitialized() to
  skip the hot-path parse when telemetry is off, and emit a debug
  daemon log when a present-but-invalid traceparent header is
  rejected.
- Re-export DaemonRequestSpanOptions from the core barrel and add a
  type-level guard so the parentContext field cannot silently
  disappear (vitest alone cannot catch its removal).

* chore(vscode): regenerate companion NOTICES.txt for @opentelemetry/core

* fix(telemetry): lazy-load OTel core fallback propagator behind SDK init

Address review feedback on the inbound traceparent linkage:

- Keep @opentelemetry/core out of the static graph. The module-level
  W3CTraceContextPropagator in daemon-tracing.ts pulled the CJS barrel
  (bot-measured +65,046 bytes) into every closure loading that module,
  including telemetry-off deployments. daemon-tracing.ts now keeps only a
  holder + setter (setDaemonFallbackPropagator, typed against
  @opentelemetry/api — type imports stay free at runtime); the lazy
  sdk-impl.ts chunk, whose closure already contains @opentelemetry/core via
  sdk-node/resources, constructs and injects the W3C instance on the
  successful SDK assembly path. Until injection, extraction returns no
  parent context: the HTTP edge is already gated on
  isTelemetrySdkInitialized (nothing changes when telemetry is off), and
  the _meta edge's consumers (withDaemonSpan / withInteractionSpan)
  short-circuit on the same flag, so an unresolved pre-init parent never
  had an observable effect.
- Add the mutation-verified fail-closed test for the header-extraction
  try/catch in daemonTelemetryMiddleware: a throwing extractor leaves the
  request settling normally (recordDaemonHttpRequest still fires once)
  with no parentContext on the span options.
- Record the rejected traceparent value (truncated to 128 chars) as
  http.request.header.traceparent on the invalid-header breadcrumb —
  traceparent only carries trace-id/span-id/flags, so this is
  privacy-safe and makes broken cross-service joins diagnosable.

Also document why the _meta extraction path deliberately skips
shouldForceSampled (trusted in-process bridge vs external HTTP input).

* feat(telemetry): carry inbound trace id into daemon access log with telemetry off

Telemetry off (the default) left daemon logs without any trace id: with no
request span, the log trace prefix never fires, so a caller forwarding W3C
traceparent could not be joined to its daemon log lines.

The middleware now parses the header with a plain regex
(extractInboundTraceId — same shape/all-zero/ff rejections as the W3C
propagator, no OTel machinery) and stores the trace id on the per-response
telemetry context. The access log emits it as the camelCase traceId field
of "request completed", keeping the log-based join alive with no telemetry
config and no trace backend. With telemetry on nothing changes: the request
span already carries the caller's trace id into the log prefix.

* fix(telemetry): unify _meta/HTTP sampling and repair build export

- Export extractInboundTraceId from the core barrel: the previous commit
  exported it from daemon-tracing.ts only, so downstream package builds
  failed with TS2305.
- extractDaemonTraceContext now applies the same shouldForceSampled()
  matrix as the HTTP edge: the _meta path is also reachable from direct
  ACP clients (acpAgent newSession/loadSession/unstable_resumeSession
  and Session.prompt pass caller-controlled _meta), so an external
  sampled=0 parent no longer silences daemon spans there either. The
  in-process bridge is unaffected (its injected values are already
  SAMPLED).
- The rejected-header breadcrumb now goes through sanitizeLogText so a
  crafted traceparent cannot forge log line structure with control
  characters.
- Add the sdk-impl wiring test: after initializeTelemetry the injected
  W3C fallback propagator resolves inbound HTTP parents.

* fix(telemetry): align log-path traceparent parsing and emit traceId in both modes

- extractInboundTraceId now mirrors the vendored W3C propagator's
  acceptance exactly: single optional leading/trailing whitespace and
  trailing extension fields above version 00 (version 00 must stay
  four fields). Previously the strict four-field anchor made the two
  paths disagree on the same forward-compatible header, silently
  dropping the access-log traceId for exactly the callers the
  propagator path supports.
- The camelCase traceId access-log field is now captured whenever a
  valid header parses, regardless of telemetry mode, so one saved log
  query / alert shape works for every deployment; with telemetry on the
  snake_case span prefix carries the same id redundantly.

* fix(telemetry): move inbound trace id getter out of the middleware module

52d572c0f2 made the access log statically import the telemetry
middleware module to read the captured inbound trace id. The access log
sits inside the serve fast-path pre-listen closure (run-qwen-serve
imports it directly), so the middleware's core-barrel import graph came
along for the ride and check-serve-fast-path-bundle started failing:
the 5.6MB core chunk (shell tool, glob, chokidar, @iarna/toml, fzf)
became statically reachable from run-qwen-serve.

Move the response-context symbol, its type, and the
getDaemonTelemetryInboundTraceId getter into a new import-light
telemetry-context.ts; the middleware imports the symbol from there and
re-exports the getter, so the access log no longer links against the
telemetry module at all.

* fix(telemetry): capture inbound trace id pre-auth under a dedicated symbol

* test(telemetry): pin the trace id seam through the context module getter

---------

Co-authored-by: 秦奇 <gary.gq@alibaba-inc.com>
2026-08-19 12:21:49 +00:00

1004 lines
58 KiB
Markdown

# Observability with OpenTelemetry
Learn how to enable and setup OpenTelemetry for Qwen Code.
- [Observability with OpenTelemetry](#observability-with-opentelemetry)
- [Key Benefits](#key-benefits)
- [OpenTelemetry Integration](#opentelemetry-integration)
- [Configuration](#configuration)
- [Aliyun Telemetry](#aliyun-telemetry)
- [Manual OTLP Export](#manual-otlp-export)
- [Local Telemetry](#local-telemetry)
- [File-based Output (Recommended)](#file-based-output-recommended)
- [Collector-Based Export (Advanced)](#collector-based-export-advanced)
- [Logs and Metrics](#logs-and-metrics)
- [Logs](#logs)
- [Metrics](#metrics)
- [Daemon Metrics](#daemon-metrics)
- [Spans](#spans)
- [Resource Metrics](#resource-metrics)
- [Performance Monitoring (Reserved)](#performance-monitoring-reserved)
## Migration Notes
- `tool_output_truncated` was renamed to `qwen-code.tool_output_truncated` for namespace consistency — downstream consumers filtering on the old name should update their queries.
- The `tool.call.latency` histogram documentation previously listed a `decision` attribute — this was never set on the histogram (only `function_name` is recorded). The `tool.call.count` counter continues to include `decision`.
- The `qwen-code.file_operation` log event and `file.operation.count` metric documentation previously listed diff-stat attributes (`model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`) — these were never set on either. Diff-stat data is available via the `tool_call` log event's `metadata` attribute.
## Key Benefits
- **🔍 Usage Analytics**: Understand interaction patterns and feature adoption
across your team
- **⚡ Performance Monitoring**: Track response times, token consumption, and
resource utilization
- **🐛 Real-time Debugging**: Identify bottlenecks, failures, and error patterns
as they occur
- **📊 Workflow Optimization**: Make informed decisions to improve
configurations and processes
- **🏢 Enterprise Governance**: Monitor usage across teams, track costs, ensure
compliance, and integrate with existing monitoring infrastructure
## OpenTelemetry Integration
Built on **[OpenTelemetry]** — the vendor-neutral, industry-standard
observability framework — Qwen Code's observability system provides:
- **Universal Compatibility**: Export to any OpenTelemetry backend (Aliyun,
Jaeger, Prometheus, Datadog, etc.)
- **Standardized Data**: Use consistent formats and collection methods across
your toolchain
- **Future-Proof Integration**: Connect with existing and future observability
infrastructure
- **No Vendor Lock-in**: Switch between backends without changing your
instrumentation
[OpenTelemetry]: https://opentelemetry.io/
[aliyun-opentelemetry-overview]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/product-overview/what-is-tracing-analysis
[aliyun-opentelemetry-get-started]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/before-you-begin
[aliyun-opentelemetry-console-cn]: https://trace.console.aliyun.com
[aliyun-opentelemetry-console-cn-legacy]: https://tracing.console.aliyun.com
[aliyun-opentelemetry-console-intl]: https://arms.console.alibabacloud.com
## Configuration
All telemetry behavior is controlled through your `.qwen/settings.json` file.
These settings can be overridden by environment variables or CLI flags.
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
| --------------------------------- | ---------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------- |
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing — set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `userId` | `QWEN_TELEMETRY_USER_ID` | - | Stable end-user identifier written to GenAI spans as the ARMS extension `gen_ai.user.id`; prefer a pseudonymous value | string | - |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include standard GenAI messages, instructions, tool definitions, tool arguments, and successful tool results as native span attributes | `true`/`false` | `false` |
| `sensitiveSpanAttributeMaxLength` | `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | - | Maximum compact JSON string length for each sensitive native span attribute. Set lower if your backend rejects large attributes. | `1..104857600` | `1048576` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
**Note on boolean environment variables:** For the boolean settings (`enabled`,
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
corresponding environment variable to `true` or `1` will enable the feature. Any
other value will disable it.
**Note on integer environment variables:** `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH`
must be a positive integer when set. Invalid values fail telemetry configuration
resolution instead of silently falling back.
`gen_ai.tool.description` is non-sensitive static registry metadata and is
emitted independently of `includeSensitiveSpanAttributes`. This includes
descriptions supplied by MCP servers and other workspace tool providers. The
value is limited to 4096 UTF-16 code units and never includes dynamic invocation
details.
**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
two things happen:
1. **Native span attributes** carry standard OpenTelemetry GenAI JSON:
- Main-agent and LLM input messages (`gen_ai.input.messages`)
- System instructions (`gen_ai.system_instructions`)
- Tool definitions (`gen_ai.tool.definitions`)
- Main-agent and LLM output messages (`gen_ai.output.messages`)
- Final executed tool arguments (`gen_ai.tool.call.arguments`)
- Successful tool results (`gen_ai.tool.call.result`)
- Interaction spans retain the compatibility `new_context` attribute.
Main-agent input is one original user-text projection before context
expansion, and main-agent output is one final user-visible answer after all
tool and continuation work settles. LLM values still come from provider-final
SDK request objects and raw provider responses, so their input can include
history, expanded files, system instructions, and tool results, and their
output can include every provider candidate. Tool values come from the final
invocation parameters and successful model-facing result. Each
standard GenAI value is compact JSON and must be complete and schema-valid.
A value that is invalid, cyclic, or longer than
`sensitiveSpanAttributeMaxLength` is omitted as a whole; JSON is never
truncated and no preview, hash, or truncation metadata is emitted. The
interaction-specific `new_context` attribute retains its existing
truncation behavior. The default maximum is 1 MiB (`1048576`) per attribute
and the accepted range is `1..104857600` (100 MiB). The limit is measured as
JavaScript string length rather than UTF-8 bytes. Non-ASCII content can
therefore occupy more bytes after OTLP export.
2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
logs endpoint) keep their existing `prompt`, `function_args`, and
`response_text` fields, instead of being dropped.
⚠️ **Security warning:** enabling this flag streams full conversation history,
file contents read by `read_file`, shell commands and their output (including
secrets in env vars or arguments), and model responses to the configured OTLP
backend. Treat the backend as a privileged data sink. The flag defaults to
`false`.
**Cost / payload size:** At the default limit, one LLM span can carry at most about 4 MiB across input, output, system instructions, and tool definitions; one Tool span can carry about 2 MiB across arguments and result; and one interaction can carry about 3 MiB across Agent input, Agent output, and compatibility `new_context`. This is Qwen Code's application-side cap, not a guarantee that every collector or backend accepts a single attribute that large. If spans are rejected or dropped, lower `sensitiveSpanAttributeMaxLength` (for example, to `61440`) and monitor exporter throughput.
This setting does not disable sensitive data in OTel logs or other telemetry
sinks; non-internal API response telemetry can populate `response_text`, so
OTel logs, UI telemetry, and chat recording may receive response text
independently of this setting. QwenLogger does not include `response_text`.
**HTTP OTLP signal routing:** When using HTTP protocol (`otlpProtocol: "http"`),
Qwen Code automatically appends signal-specific paths (`/v1/traces`, `/v1/logs`,
`/v1/metrics`) to the base `otlpEndpoint`. For example, `http://collector:4318`
becomes `http://collector:4318/v1/traces` for traces. If the URL already ends
with a signal path, it is used as-is. Per-signal endpoint overrides
(`otlpTracesEndpoint`, etc.) take precedence over the base endpoint and are used
verbatim. gRPC protocol uses service-based routing and does not append paths.
The per-signal endpoint environment variables also accept the standard
OpenTelemetry names: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
**End-user identity:** `telemetry.userId` and
`QWEN_TELEMETRY_USER_ID` are explicit opt-ins for the ARMS span attribute
`gen_ai.user.id`. The environment variable takes precedence after both values
are trimmed; a blank environment value falls back to settings. The identifier
is written only to interaction, LLM, Tool, and Agent spans. It is not a Resource
attribute, log or metric attribute, outbound Baggage value, or current
OpenTelemetry GenAI standard field. Prefer a stable pseudonymous identifier.
The value is resolved at startup, so configuration changes require a restart.
Do not configure a process-wide value on a daemon or channel instance serving
multiple end users.
For detailed information about all configuration options, see the
[Configuration Guide](../../users/configuration/settings.md).
### Resource attributes
Resource attributes are static key-value pairs attached to every span, log,
and metric exported via OTLP. Use them to slice telemetry by team, environment,
deployment region, or any other dimension your backend cares about.
Two sources, merged in priority order (lowest → highest):
1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
key conflict)
`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides
`service.name` from any other source (per the OpenTelemetry spec).
#### Examples
**Slice all telemetry by team / environment:**
```bash
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
```
**Route to a per-tenant collector via `service.name`:**
```bash
export OTEL_SERVICE_NAME=qwen-code-ci
```
**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**
```json
{
"telemetry": {
"resourceAttributes": {
"deployment.environment": "production",
"service.namespace": "engineering-tooling"
}
}
}
```
```bash
# Add a one-off tag without touching settings:
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
```
#### Reserved keys
Some keys are runtime-controlled and cannot be overridden:
- `service.version` — always set to the running CLI version. Setting it from
any source is silently dropped with a warning.
- `session.id` — runtime-injected per session. User-provided values from
either env or settings are dropped with a warning. The reason is that
Resource attributes auto-attach to every metric data point; allowing user
override would bypass [Cardinality controls](#cardinality-controls) below.
Spans and logs always carry `session.id`.
`service.name` is **not** reserved; it follows the precedence chain above.
#### Format
`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
the wrong boundary and the second half is dropped as malformed). Malformed
pairs are skipped with a warning rather than failing telemetry startup.
#### Troubleshooting: when a user-provided attribute appears not to take effect
Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
settings values, and invalid percent-encoding are all silently dropped with a
warning logged via the OpenTelemetry diagnostics channel. That channel routes
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
behavior can look like silent failure.
If a custom resource attribute isn't appearing on exported telemetry:
1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
(non-string settings value).
2. Verify the env var is set in the qwen-code process's environment (not just
your shell) and that values are percent-encoded.
3. Confirm `telemetry.enabled` is `true` — telemetry init only runs if enabled.
### Cardinality controls
Metrics are aggregated by attribute set at the backend — every distinct
combination of attribute values produces a new time series. Attaching a
high-cardinality field like `session.id` to a metric causes time-series fan-out
proportional to the number of sessions, which quickly exhausts metric backend
storage.
To prevent this, Qwen Code keeps high-cardinality attributes off metric data
points by default. Spans and logs are per-event and unaffected, so they
continue to carry `session.id` for trace and log correlation.
#### `telemetry.metrics.includeSessionId` (default: `false`)
Setting this to `true` (via settings or
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
every metric data point.
⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a
fleet will blow up metric storage. Recommended only for short-term debugging.
For long-term session correlation, query trace or log backends instead.
#### Migration from earlier versions
Prior to this release, `session.id` was attached to metrics by default. If
your Prometheus queries / Grafana dashboards / alert rules reference
`session_id` on a metric, you have two options:
**Option A** — restore the previous behavior for short-term debugging:
```bash
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
```
or:
```json
{
"telemetry": {
"metrics": { "includeSessionId": true }
}
}
```
**Option B (recommended)** — move session-level analysis off metrics. Spans
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
cardinality pressure.
### Client-side HTTP span on outbound fetch
When telemetry is enabled, Qwen Code registers `UndiciInstrumentation`
which creates a client-side HTTP span for every outbound `fetch()`
request originated by the process — including the LLM SDKs (`openai`,
`@google/genai`, `@anthropic-ai/sdk`), the MCP StreamableHTTP client, the
`WebFetch` tool, and any IDE-extension out-of-process calls. The span
lets you see network latency (TTFB / response body transfer) separately
from upstream model processing time, which the existing
`api.generateContent` span alone can't distinguish.
These spans go to your **own** OTLP collector (or file outfile) just like
the rest of the telemetry — they do not affect what is written onto the
outbound HTTP request itself. Whether the W3C `traceparent` header is
also written into the outgoing request stream is controlled by a
**separate, security-relevant setting** documented in
[outbound correlation](#outbound-correlation-security-relevant) below.
**Feedback-loop avoidance.** OTel SDK uses `fetch` internally to upload OTLP
data. Without protection, instrumenting `fetch` would trace those uploads,
which would themselves be uploaded, causing an infinite loop. Qwen Code's
undici instrumentation is configured with an `ignoreRequestHook` that skips
URLs matching the configured `telemetry.otlpEndpoint` /
`telemetry.otlpTracesEndpoint` / `telemetry.otlpLogsEndpoint` /
`telemetry.otlpMetricsEndpoint` prefixes. In file-outfile mode there are no
outbound HTTP uploads, so the hook is a no-op.
## Outbound correlation (SECURITY-RELEVANT)
These settings live in a **separate top-level namespace** from `telemetry.*`
on purpose: telemetry controls data flow into the operator's own
observability backend, while `outboundCorrelation.*` controls what
client-side correlation data qwen-code writes **into outbound LLM API
request streams** that reach third-party LLM provider endpoints
(DashScope, OpenAI, Anthropic, etc.). Different recipients, different
consent decision. **All values default to off.** See PR #4390 review
discussion for the framing rationale.
### `outboundCorrelation.propagateTraceContext`
```jsonc
"outboundCorrelation": {
"propagateTraceContext": false // default
}
```
When `false` (default), Qwen Code installs a no-op `TextMapPropagator` on
the OTel SDK. UndiciInstrumentation still creates client HTTP spans for
your OTLP collector, but `propagation.inject()` is a no-op so **no
`traceparent` is written onto outbound requests**. Trace IDs stay
internal to the operator's collector.
When `true`, the SDK's default W3C composite propagator
(`tracecontext` + `baggage`) is installed and the standard `traceparent`
header is written on every outbound `fetch`:
```
traceparent: 00-<32-hex traceId>-<16-hex parentSpanId>-<01-sampled | 00-not-sampled>
```
Additionally, `TRACEPARENT` and `TRACESTATE` environment variables are set in
shell child processes (Bash tool, hooks, monitor) so that spawned commands can
participate in the same distributed trace.
Opt in only when the LLM provider also reports into your OTel collector
for cross-process trace stitching — e.g. ARMS Tracing serving DashScope.
For most operators the value is `false`; cross-vendor trace continuation
is niche.
**Depends on `telemetry.enabled: true`.** The OTel SDK only initializes
when telemetry is enabled, so `propagateTraceContext` only takes effect
in that state. Setting it to `true` while telemetry is disabled is a
silent no-op — no SDK, no propagator, no `traceparent` on the wire.
Verify both flags when wiring an ARMS+DashScope correlation setup:
```jsonc
{
"telemetry": {
"enabled": true,
"otlpTracesEndpoint": "http://tracing-analysis-...",
},
"outboundCorrelation": {
"propagateTraceContext": true,
},
}
```
### Other outbound correlation headers
`X-Qwen-Code-Session-Id` and `X-Qwen-Code-Request-Id` are **not part of
this PR**. They will be designed and proposed in their own follow-up
PR(s) under the same `outboundCorrelation.*` namespace, each with its
own threat model and operator-consent flow. PR #4390 review (LaZzyMan)
established the principle: "telemetry's scope of work doesn't include
sending identifiers to LLM providers"; correlation-header work moves to
its own design discussion rather than landing under telemetry.
## Inbound correlation (daemon HTTP API)
The daemon HTTP API accepts the standard W3C `traceparent` header on every
request. Two consumers read it independently:
- **Request span re-parenting (telemetry enabled).** When the telemetry SDK
is initialized, a valid header is extracted as the request span's remote
parent, so daemon spans attach under the caller's trace instead of
starting a new one. The `_meta` forwarding path reads the same parent
chain, so session subprocess spans forwarded through a daemon request
inherit it too.
- **Access-log `traceId` field (both modes).** A dedicated pre-auth capture
middleware parses the header on every request — including ones
short-circuited at auth (401), the rate limiter (429), the JSON body
parser (400), or never matched by any route (404) — and the access log
emits the caller trace id as a camelCase `traceId` field. With telemetry
disabled this field is the only join between a daemon log line and the
caller's logs (or trace backend), so one saved query works for both modes
with no telemetry configuration.
An invalid-but-present header is rejected (the span stays parentless) and
leaves a rate-limited DEBUG breadcrumb
(`qwen-code.daemon.traceparent.invalid`) recording the rejected value, so a
broken cross-service join is diagnosable from daemon logs alone.
### Forced sampling under inbound parents
Under the default `parentbased_always_on` sampler (and other parentbased
defaults), a remote parent's `sampled=0` flag is a head-based decision on
the caller's side, not a request to drop daemon telemetry, so extraction
forces the SAMPLED flag on inbound parents. The only opt-out is
`OTEL_TRACES_SAMPLER=parentbased_always_off`, which honors the caller's
flags — note it also disables root-span sampling for the whole daemon, not
just inbound-linked requests.
**Warning:** a constant `traceparent` (e.g. hardcoded in a load-test
client) re-parents every daemon request into one single trace; generate a
fresh header per request.
## Aliyun Telemetry
### Manual OTLP Export
To view Qwen Code telemetry in Alibaba Cloud Managed Service for
OpenTelemetry, configure Qwen Code to export to the OTLP endpoint
provided by ARMS.
Setting `"target": "gcp"` alone does not configure the export
destination. If `otlpEndpoint` is not set, Qwen Code still defaults to
`http://localhost:4317`. If `outfile` is set, it overrides
`otlpEndpoint` and telemetry is written to the file instead of being
sent to Alibaba Cloud.
1. Enable telemetry in your `.qwen/settings.json` and set the OTLP
endpoint:
**Option A: gRPC protocol** (standard OTLP endpoint):
```json
{
"telemetry": {
"enabled": true,
"target": "gcp",
"otlpEndpoint": "https://<your-otlp-endpoint>",
"otlpProtocol": "grpc"
}
}
```
**Option B: HTTP protocol with per-signal endpoints** (for backends
that use non-standard paths, e.g., `/api/otlp/traces` instead of
`/v1/traces`):
```json
{
"telemetry": {
"enabled": true,
"otlpProtocol": "http",
"otlpTracesEndpoint": "http://<host>/<token>/api/otlp/traces",
"otlpLogsEndpoint": "http://<host>/<token>/api/otlp/logs",
"otlpMetricsEndpoint": "http://<host>/<token>/api/otlp/metrics"
}
}
```
> **Note:** When using HTTP protocol with only `otlpEndpoint` (no
> per-signal overrides), Qwen Code appends standard OTLP paths
> (`/v1/traces`, `/v1/logs`, `/v1/metrics`) to the base URL. If your
> backend uses different paths, use per-signal endpoint overrides as
> shown in Option B.
To populate ARMS Session Analysis `User ID`, add a stable pseudonymous
identity as a span-level setting:
```json
{
"telemetry": {
"userId": "user-079458",
"resourceAttributes": {
"acs.arms.service.feature": "genai_app"
}
}
}
```
For container deployments, set
`QWEN_TELEMETRY_USER_ID=user-079458` instead. A custom
`telemetry.resourceAttributes.user.id` remains an unrelated Resource
dimension and does not populate ARMS Session Analysis; remove it when
migrating to the span-level setting.
2. If your Alibaba Cloud endpoint requires authentication, provide OTLP
headers through standard OpenTelemetry environment variables such as
`OTEL_EXPORTER_OTLP_HEADERS` (or the signal-specific variants). Qwen
Code does not currently expose OTLP auth headers directly in
`.qwen/settings.json`.
3. Run Qwen Code and send prompts.
4. View telemetry in Managed Service for OpenTelemetry:
- Product overview:
[What is Managed Service for OpenTelemetry?][aliyun-opentelemetry-overview]
- Getting started:
[Get started with Managed Service for OpenTelemetry][aliyun-opentelemetry-get-started]
- Console entry points:
- China mainland:
[trace.console.aliyun.com][aliyun-opentelemetry-console-cn]
(legacy console:
[tracing.console.aliyun.com][aliyun-opentelemetry-console-cn-legacy])
- International:
[arms.console.alibabacloud.com][aliyun-opentelemetry-console-intl]
- In the console, use `Applications` to inspect traces and service
topology.
- To locate the OTLP endpoint and access information:
- **New console** (`trace.console.aliyun.com` or international):
navigate to `Integration Center`.
- **Legacy console** (`tracing.console.aliyun.com`): navigate to
`Cluster Configurations` → `Access point information`.
## Local Telemetry
For local development and debugging, you can capture telemetry data locally:
### File-based Output (Recommended)
1. Enable telemetry in your `.qwen/settings.json`:
```json
{
"telemetry": {
"enabled": true,
"outfile": ".qwen/telemetry.log"
}
}
```
> **Note:** When `outfile` is set, OTLP export is automatically disabled.
> The `target` and `otlpEndpoint` settings are not needed for file-only
> output and can be safely omitted from your config.
2. Run Qwen Code and send prompts.
3. View logs and metrics in the specified file (e.g., `.qwen/telemetry.log`).
### Collector-Based Export (Advanced)
1. Run the automation script:
```bash
npm run telemetry -- --target=local
```
This will:
- Download and start Jaeger and OTEL collector
- Configure your workspace for local telemetry
- Provide a Jaeger UI at http://localhost:16686
- Save logs/metrics to `~/.qwen/tmp/<projectHash>/otel/collector.log`
- Stop collector on exit (e.g. `Ctrl+C`)
2. Run Qwen Code and send prompts.
3. View traces at http://localhost:16686 and logs/metrics in the collector log
file.
## Logs and Metrics
The following section describes the structure of logs, metrics, and spans
generated for Qwen Code.
- A `sessionId` is included as a common attribute on all logs and metrics.
### Logs
Logs are timestamped records of specific events. All log records automatically include `event.name` and `event.timestamp` attributes.
The following events are logged:
#### Core Session Events
- `qwen-code.config`: Emitted once at startup with CLI configuration.
- **Attributes**: `model`, `sandbox_enabled`, `core_tools_enabled`, `approval_mode`, `file_filtering_respect_git_ignore`, `debug_mode`, `truncate_tool_output_threshold`, `truncate_tool_output_lines`, `hooks` (comma-separated, omitted if disabled), `ide_enabled`, `interactive_shell_enabled`, `mcp_servers`, `mcp_servers_count`, `mcp_tools`, `mcp_tools_count`, `output_format`, `skills`, `subagents`
- `session.start`: A session begins. Emitted after telemetry initialization at startup and again on every session switch; lifecycle semantics are described in the Spans section.
- **Attributes**: `session.id` (string), `session.previous_id` (string, present only when this start continues a persisted conversation under a new session id)
- `session.end`: A session ends. Emitted before a session switch replaces the current session, and at telemetry shutdown.
- **Attributes**: `session.id` (string)
- `qwen-code.user_prompt`: User submits a prompt.
- **Attributes**: `prompt_length` (int), `prompt_id` (string), `prompt` (string, excluded if `log_prompts_enabled` is false), `auth_type` (string)
- `qwen-code.user_retry`: User retries the last prompt.
- **Attributes**: `prompt_id` (string)
- `qwen-code.conversation_finished`: A conversation turn sequence completes.
- **Attributes**: `approvalMode` (string), `turnCount` (int)
- `qwen-code.user_feedback`: User submits session feedback.
- **Attributes**: `session_id` (string), `rating` (int: 1=bad, 2=fine, 3=good), `model` (string), `approval_mode` (string), `prompt_id` (string, optional)
#### Tool Events
- `qwen-code.tool_call`: Each function/tool call. Terminal events are normalized so `status` is authoritative: success and cancelled events omit error fields, while error events always have a non-empty `error_type` (`unknown` when the producer did not classify the error). Blank tool names are emitted as `unknown_tool`. A missing `execution_status` is normalized to `unknown` and is never inferred from the terminal `status`.
- **Attributes**: `function_name` (string), `function_args` (object), `call_id` (string, optional), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `execution_status` (string: "not_started", "success", "error", "cancelled", or "unknown"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, present for error events), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`)
- `qwen-code.file_operation`: Each file operation.
- **Attributes**: `tool_name` (string), `operation` (string: "create", "read", "update"), `lines` (int, optional), `mimetype` (string, optional), `extension` (string, optional), `programming_language` (string, optional)
- `qwen-code.tool_output_truncated`: Tool output exceeded size threshold.
- **Attributes**: `tool_name` (string), `original_content_length` (int), `truncated_content_length` (int), `threshold` (int), `lines` (int), `prompt_id` (string)
#### API Events
- `qwen-code.api_request`: Outgoing request to the LLM API.
- **Attributes**: `model` (string), `prompt_id` (string), `request_text` (string, optional), `subagent_name` (string, optional)
- `qwen-code.api_response`: Response received from LLM API.
- **Attributes**: `response_id` (string), `model` (string), `status_code` (int/string, optional), `duration_ms` (int), `input_token_count` (int), `output_token_count` (int), `cached_content_token_count` (int), `thoughts_token_count` (int), `total_token_count` (int), `prompt_id` (string), `auth_type` (string, optional), `response_text` (string, optional), `subagent_name` (string, optional)
- `qwen-code.api_error`: API request failed.
- **Attributes**: `model` (string), `prompt_id` (string), `duration_ms` (int), `error_message` (string), `response_id` (string, optional), `auth_type` (string, optional), `error_type` (string, optional), `status_code` (int/string, optional), `subagent_name` (string, optional)
Additionally, OTel-standard aliases (`http.status_code`, `error.message`, `model_name`, `duration`) are emitted for compatibility.
- `qwen-code.api_cancel`: API request cancelled by user.
- **Attributes**: `model` (string), `prompt_id` (string), `auth_type` (string, optional), `loop_wakeups_cancelled` (int, optional)
- `qwen-code.api_retry`: HTTP-status retry (429/5xx) at an LLM call site. Distinct from `chat.content_retry` which handles `InvalidStreamError` retries on a separate budget.
- **Attributes**: `model` (string), `prompt_id` (string, optional), `attempt_number` (int), `error_type` (string, optional), `error_message` (string), `status_code` (int/string, optional), `retry_delay_ms` (int), `duration_ms` (int, equals retry_delay_ms — backoff sleep, not HTTP round-trip; for attempt duration see the qwen-code.llm_request span), `subagent_name` (string, optional)
- `qwen-code.malformed_json_response`: `generateJson` response couldn't be parsed.
- **Attributes**: `model` (string)
- `qwen-code.flash_fallback`: Switched to flash model as fallback.
- **Attributes**: `auth_type` (string)
- `qwen-code.ripgrep_fallback`: Switched to grep as fallback.
- **Attributes**: `use_ripgrep` (boolean), `use_builtin_ripgrep` (boolean), `error` (string, optional)
#### Resilience Events
- `qwen-code.chat.content_retry`: Content-error retry (e.g. empty stream).
- **Attributes**: `attempt_number` (int), `error_type` (string), `retry_delay_ms` (int), `model` (string)
- `qwen-code.chat.content_retry_failure`: All content retries exhausted.
- **Attributes**: `total_attempts` (int), `final_error_type` (string), `total_duration_ms` (int, optional), `model` (string)
- `qwen-code.chat.invalid_chunk`: Invalid chunk received from stream.
- **Attributes**: `error.message` (string, optional)
#### Command & Extension Events
- `qwen-code.slash_command`: User executes a slash command.
- **Attributes**: `command` (string), `subcommand` (string, optional), `status` (string: "success" or "error", optional)
- `qwen-code.slash_command.model`: User switches model via `/model` command.
- **Attributes**: `model_name` (string)
- `qwen-code.skill_launch`: A skill is launched.
- **Attributes**: `skill_name` (string), `success` (boolean), `prompt_id` (string)
- `qwen-code.extension_install`: Extension installed.
- **Attributes**: `extension_name` (string), `extension_version` (string), `extension_source` (string), `status` (string: "success"/"error")
- `qwen-code.extension_uninstall`: Extension uninstalled.
- **Attributes**: `extension_name` (string), `status` (string)
- `qwen-code.extension_enable`: Extension enabled.
- **Attributes**: `extension_name` (string), `setting_scope` (string)
- `qwen-code.extension_disable`: Extension disabled.
- **Attributes**: `extension_name` (string), `setting_scope` (string)
- `qwen-code.extension_update`: Extension updated.
- **Attributes**: `extension_name` (string), `extension_id` (string), `extension_previous_version` (string), `extension_version` (string), `extension_source` (string), `status` (string: "success"/"error")
- `qwen-code.ide_connection`: IDE connection event.
- **Attributes**: `connection_type` (string: "start" or "session")
- `qwen-code.auth`: Authentication event.
- **Attributes**: `auth_type` (string), `action_type` ("auto", "manual", "coding-plan"), `status` ("success", "error", "cancelled"), `error_message` (optional)
#### Subagent Events
- `qwen-code.subagent_execution`: Subagent lifecycle event.
- **Attributes**: `subagent_name` (string), `status` ("started", "completed", "failed", "cancelled"), `terminate_reason` (optional), `result` (optional), `execution_summary` (optional)
#### Arena Events
- `qwen-code.arena_session_started`: Arena session begins.
- **Attributes**: `arena_session_id` (string), `model_ids` (JSON string array), `task_length` (int)
- `qwen-code.arena_agent_completed`: An arena agent finishes.
- **Attributes**: `arena_session_id` (string), `agent_session_id` (string), `agent_model_id` (string), `status` (string: "completed"/"failed"/"cancelled"), `duration_ms` (int), `rounds` (int), `total_tokens` (int), `input_tokens` (int), `output_tokens` (int), `tool_calls` (int), `successful_tool_calls` (int), `failed_tool_calls` (int)
- `qwen-code.arena_session_ended`: Arena session completes.
- **Attributes**: `arena_session_id` (string), `status` (string: "selected"/"discarded"/"failed"/"cancelled"), `duration_ms` (int), `display_backend` (string, optional), `agent_count` (int), `completed_agents` (int), `failed_agents` (int), `cancelled_agents` (int), `winner_model_id` (string, optional)
#### Workflow Events
- `qwen-code.workflow_keyword`: Workflow keyword trigger fired.
- `qwen-code.workflow_run`: Workflow run reached terminal state.
- **Attributes**: `status` (string), `agents_dispatched` (int), `agents_completed` (int), `phase_count` (int), `tokens_spent` (int), `duration_ms` (int)
#### Auto-Memory Events
- `qwen-code.memory.extract`: Memory extraction run completed.
- **Attributes**: `trigger` ("auto"/"manual"), `status` ("completed"/"skipped"/"failed"), `skipped_reason` (optional), `patches_count` (int), `touched_topics` (string), `duration_ms` (int)
- `qwen-code.memory.dream`: Memory consolidation (dream) run completed.
- **Attributes**: `trigger` ("auto"/"manual"), `status` ("updated"/"noop"/"failed"/"cancelled"), `deduped_entries` (int), `touched_topics_count` (int), `touched_topics` (string), `duration_ms` (int)
- `qwen-code.memory.recall`: Memory recall operation completed.
- **Attributes**: `query_length` (int), `docs_scanned` (int), `docs_selected` (int), `strategy` ("none"/"heuristic"/"model"), `duration_ms` (int)
#### Prompt Suggestion & Speculation Events
- `qwen-code.prompt_suggestion`: Prompt suggestion outcome.
- **Attributes**: `outcome` ("accepted"/"ignored"/"suppressed"), `prompt_id` (optional), `accept_method` ("tab"/"enter"/"right", optional), `accept_source` ("live"/"fallback", optional), `time_to_accept_ms` (optional), `time_to_ignore_ms` (optional), `time_to_first_keystroke_ms` (optional), `suggestion_length` (optional), `similarity` (optional), `was_focused_when_shown` (optional), `reason` (optional)
- `qwen-code.speculation`: Speculative execution outcome.
- **Attributes**: `outcome` ("accepted"/"aborted"/"failed"), `turns_used` (int), `files_written` (int), `tool_use_count` (int), `duration_ms` (int), `boundary_type` (optional), `had_pipelined_suggestion` (boolean)
#### Other Events
- `qwen-code.chat_compression`: Chat context compressed.
- **Attributes**: `tokens_before` (int), `tokens_after` (int), `compression_input_token_count` (int, optional), `compression_output_token_count` (int, optional)
- `qwen-code.next_speaker_check`: Next speaker determination.
- **Attributes**: `prompt_id` (string), `finish_reason` (string), `result` (string)
- `loop_detected`: Loop detected during agent execution. _(Note: emitted without `qwen-code.` prefix — pre-existing inconsistency.)_
- **Attributes**: `loop_type` (string), `prompt_id` (string)
- `kitty_sequence_overflow`: Kitty graphics protocol sequence exceeded buffer size. _(Note: emitted without `qwen-code.` prefix — pre-existing inconsistency.)_
- **Attributes**: `sequence_length` (int), `truncated_sequence` (string, first 20 chars)
### Metrics
Metrics are numerical measurements of behavior over time. Metric names use the `qwen-code.*` prefix.
#### Core Metrics
- `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup.
- `qwen-code.tool.call.count` (Counter, Int): Counts tool calls.
- **Attributes**: `function_name`, `status` ("success"/"error"/"cancelled"), `success` (boolean, retained for compatibility), `decision` ("accept"/"reject"/"auto_accept"/"modify", optional), `tool_type` ("mcp"/"native", optional)
- `qwen-code.tool.execution.count` (Counter, Int): Counts tool execution outcomes. Deliberately carries no `function_name` dimension to stay low-cardinality, so an execution-failure rate cannot be attributed to a specific tool without dropping to the `qwen-code.tool_call` logs; exclude `unknown`, `not_started`, and `cancelled` when computing execution-failure ratios (denominator is `success` + `error`).
- **Attributes**: `execution_status` ("not_started"/"success"/"error"/"cancelled"/"unknown"), `tool_type` ("mcp"/"native"), plus globally configured common metric attributes such as the opt-in `session.id`
- `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency.
- **Attributes**: `function_name` (string)
- `qwen-code.api.request.count` (Counter, Int): Counts all API requests.
- **Attributes**: `model`, `status_code`, `error_type` (optional)
- `qwen-code.api.request.latency` (Histogram, ms): Measures API request latency.
- **Attributes**: `model` (string)
- `qwen-code.token.usage` (Counter, Int): Counts tokens used.
- **Attributes**: `model`, `type` ("input"/"output"/"thought"/"cache")
- `qwen-code.file.operation.count` (Counter, Int): Counts file operations.
- **Attributes**: `operation` ("create"/"read"/"update"), `lines` (optional), `mimetype` (optional), `extension` (optional), `programming_language` (optional)
- `qwen-code.chat_compression` (Counter, Int): Counts chat compression operations.
- **Attributes**: `tokens_before` (int), `tokens_after` (int)
- `qwen-code.slash_command.model.call_count` (Counter, Int): Counts model slash command calls.
- **Attributes**: `slash_command.model.model_name` (string)
- `qwen-code.subagent.execution.count` (Counter, Int): Counts subagent execution events.
- **Attributes**: `subagent_name`, `status` ("started"/"completed"/"failed"/"cancelled"), `terminate_reason` (optional)
#### Resilience Metrics
- `qwen-code.api.retry.count` (Counter, Int): HTTP-status retries (429/5xx) at LLM call sites.
- **Attributes**: `model` (string)
- `qwen-code.chat.content_retry.count` (Counter, Int): Retries due to content errors.
- `qwen-code.chat.content_retry_failure.count` (Counter, Int): All content retries exhausted.
- `qwen-code.chat.invalid_chunk.count` (Counter, Int): Invalid chunks from stream.
#### Arena Metrics
- `qwen-code.arena.session.count` (Counter, Int): Arena sessions by status.
- **Attributes**: `status`, `display_backend` (optional)
- `qwen-code.arena.session.duration` (Histogram, ms): Arena session duration.
- **Attributes**: `status`
- `qwen-code.arena.agent.count` (Counter, Int): Arena agent completions.
- **Attributes**: `status`, `model_id`
- `qwen-code.arena.agent.duration` (Histogram, ms): Arena agent execution duration.
- **Attributes**: `model_id`
- `qwen-code.arena.agent.tokens` (Counter, Int): Token usage by arena agents.
- **Attributes**: `model_id`, `type` ("input"/"output")
- `qwen-code.arena.result.selected` (Counter, Int): Arena result selections.
- **Attributes**: `model_id`
#### Auto-Memory Metrics
- `qwen-code.memory.extract.count` (Counter, Int): Auto-memory extraction runs.
- **Attributes**: `trigger` ("auto"/"manual"), `status`
- `qwen-code.memory.extract.duration` (Histogram, ms): Extraction duration.
- **Attributes**: `trigger`, `status`
- `qwen-code.memory.dream.count` (Counter, Int): Auto-memory dream runs.
- **Attributes**: `trigger` ("auto"/"manual"), `status`
- `qwen-code.memory.dream.duration` (Histogram, ms): Dream run duration.
- **Attributes**: `trigger`, `status`
- `qwen-code.memory.recall.count` (Counter, Int): Auto-memory recall operations.
- **Attributes**: `strategy` ("none"/"heuristic"/"model")
- `qwen-code.memory.recall.duration` (Histogram, ms): Recall duration.
- **Attributes**: `strategy`
#### API Request Breakdown
- `qwen-code.api.request.breakdown` (Histogram, ms): API request time breakdown by phase.
- **Attributes**: `model`, `phase` ("request_preparation"/"network_latency"/"response_processing"/"token_processing")
### Daemon Metrics
The daemon process (long-running HTTP server mode) exposes its own metrics.
> **Note:** The three Observable Gauges (`daemon.session.active`, `daemon.sse.active`, `daemon.process.heap_used`) are callback-based metrics updated at each collection interval; `registerDaemonGaugeCallbacks()` must be invoked during daemon initialization to register the observation callbacks.
#### HTTP
- `qwen-code.daemon.http.request.count` (Counter, Int): Request count by route and status class.
- **Attributes**: `route`, `status_class` ("2xx"/"4xx"/"5xx")
- `qwen-code.daemon.http.request.duration` (Histogram, ms): Request duration.
- **Attributes**: `route`
- **Buckets**: 1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000
#### Sessions
- `qwen-code.daemon.session.active` (ObservableGauge, Int): Current active sessions.
- `qwen-code.daemon.session.lifecycle` (Counter, Int): Session lifecycle events.
- **Attributes**: `action` ("spawn"/"close"/"die")
#### Channels
- `qwen-code.daemon.channel.lifecycle` (Counter, Int): ACP channel lifecycle events.
- **Attributes**: `action` ("spawn"/"exit"), `expected` (boolean, optional)
#### Prompts
- `qwen-code.daemon.prompt.queue_wait` (Histogram, ms): Prompt FIFO queue wait time.
- **Buckets**: 1, 5, 10, 50, 100, 500, 1000, 5000, 10000, 30000, 60000
- `qwen-code.daemon.prompt.duration` (Histogram, ms): End-to-end prompt duration.
- **Buckets**: 100, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000
#### Errors
- `qwen-code.daemon.bridge.error.count` (Counter, Int): Bridge errors by type.
- **Attributes**: `error_type` (known class name or "unknown")
- `qwen-code.daemon.cancel.count` (Counter, Int): Cancel request count.
#### Resources
- `qwen-code.daemon.sse.active` (ObservableGauge, Int): Active SSE connections.
- `qwen-code.daemon.process.heap_used` (ObservableGauge, Int, bytes): Heap memory usage.
### Spans
Distributed tracing spans form a tree rooted at `qwen-code.interaction`. In the CLI, each interaction is a trace root with its own `traceId`; ACP and daemon paths may inherit an inbound parent context. Cross-prompt correlation uses the `session.id` attribute.
Session lifecycle is also exported through the OpenTelemetry General Session
semantic conventions. When the OTel logs pipeline is enabled, Qwen Code emits
`session.start` and `session.end` log events with the required `session.id`
attribute (cataloged under Core Session Events above). A resumed persisted
conversation includes `session.previous_id` on its `session.start` event only
when the resumed session id differs from the current one; cold-start
resumptions (`--resume`, `--continue`, `--fork-session`) do not carry it.
`/clear` and other replacement flows intentionally do not claim continuation
because they discard the previous conversation.
The existing Qwen-specific `qwen-code.config`/`cli_config` and RUM
`session_start` records remain available for compatibility. GenAI request
spans continue to use `gen_ai.conversation.id` for the same owning session ID.
- `qwen-code.interaction`: Main-agent invocation span. It covers all LLM requests, tool approval/execution, and continuations for one logical prompt. User queries, retries, cron prompts, notifications, teammate messages, and Goal turns create invocations; tool results, hooks, and steering reuse the exact active prompt ID.
- **GenAI attributes**: `gen_ai.operation.name` (`invoke_agent`), `gen_ai.agent.name` (`qwen-code`), `gen_ai.conversation.id`, optional `gen_ai.output.type` (`json` only with a configured JSON Schema), sensitive `gen_ai.input.messages`, sensitive `gen_ai.output.messages`, and optional ARMS extension `gen_ai.user.id`
- **Compatibility attributes**: `session.id`, `qwen-code.prompt_id`, `qwen-code.message_type`, `qwen-code.model`, `qwen-code.approval_mode`, `interaction.sequence`, `interaction.duration_ms`, `qwen-code.turn_status` ("ok"/"error"/"cancelled")
- `gen_ai.request.model` is intentionally omitted because the agent supports overrides, fallback, and dynamic model selection. `gen_ai.provider.name` and agent ID/version/description are also omitted.
- Agent input is one original user prompt, not the expanded model request. Agent output is one final user-visible text projection; structured JSON uses compact JSON text with `finish_reason=tool_call`. Both are omitted unless sensitive span attributes are enabled and the complete JSON fits the per-attribute limit.
- `qwen-code.llm_request`: Wraps a single LLM API call.
- **GenAI attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, optional ARMS extension `gen_ai.user.id`, `gen_ai.request.model`, `gen_ai.request.stream`, `gen_ai.request.choice.count`, `gen_ai.request.max_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.frequency_penalty`, `gen_ai.request.presence_penalty`, `gen_ai.request.stop_sequences`, optional `gen_ai.output.type`, `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`, `gen_ai.response.time_to_first_chunk`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`
- **Compatibility attributes**: `session.id`, `qwen-code.prompt_id`, `llm_request.context` ("subagent"/"interaction"/"standalone"), `duration_ms`, `ttft_ms`, `request_setup_ms`, `attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`, `success`, `error`, `finish_reason`, `thoughts_token_count`, `subagent_name`, `error_type`, `error_status_code`
- Standard response fields come from the provider response. Standard token fields are emitted only for provider-reported non-negative safe integers. If the provider reports only a total token count, input/output usage is omitted rather than estimated.
- Standard request-parameter fields come from the first provider-final SDK request object after adapter defaults, overrides, unsupported-field removal, and output-window clamps. Qwen Code does not infer SDK or server defaults.
- Streaming requests emit `gen_ai.request.stream=true`. `gen_ai.response.time_to_first_chunk` measures seconds from the provider call to the first normalized response yielded by the provider adapter, which may differ from the first raw network frame. Non-streaming requests omit both standard streaming attributes because an absent `gen_ai.request.stream` means non-streaming in the semantic convention.
- `qwen-code.tool`: Wraps the full tool lifecycle (approval wait + execution).
- **Attributes**: `session.id`, optional ARMS extension `gen_ai.user.id`, `gen_ai.operation.name` (`execute_tool`), optional inherited `gen_ai.agent.name`, `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`, `tool.call_id`, `duration_ms`, `success`, `error`, `error.type` on failure, `tool.failure_kind` (string, optional — the specific failure reason, e.g. "cancelled", "tool_error", "tool_exception", "timeout", "permission_denied", "pre_hook_blocked")
- `qwen-code.tool.execution`: Wraps the tool execution phase (after approval). Emitted only for attempted executions.
- **Attributes**: `session.id`, `gen_ai.tool.name` (optional), `tool.call_id` (optional), `duration_ms`, `success`, `error`, `execution_status` ("success"/"error"/"cancelled"), `error_type`, `error.type`
- `qwen-code.tool.blocked_on_user`: Time a tool spends waiting on user approval.
- **Attributes**: `session.id`, `tool.name`, `tool.call_id`, `duration_ms`, `decision` ("proceed_once"/"proceed_always"/"cancel"/"aborted"/"auto_approved"/"error"), `source` ("cli"/"ide"/"hook"/"auto"/"system")
- `qwen-code.hook`: Wraps each pre/post-tool-use hook fire site.
- **Attributes**: `session.id`, `hook_event` ("PreToolUse"/"PostToolUse"/"PostToolUseFailure"/"PostToolBatch"), `tool.name`, `tool.use_id` (optional), `is_interrupt` (boolean, optional), `duration_ms`, `success`, `should_proceed` (optional), `should_stop` (optional), `block_type` (optional), `error` (optional)
- `qwen-code.subagent`: Wraps a single subagent invocation.
- **Attributes**: `gen_ai.operation.name` (`invoke_agent`), `gen_ai.agent.name`, `gen_ai.agent.description`, `gen_ai.conversation.id`, optional ARMS extension `gen_ai.user.id`, optional `gen_ai.request.model`, `qwen-code.subagent.id`, `qwen-code.subagent.name`, `qwen-code.subagent.invocation_kind` ("foreground"/"fork"/"background"), `qwen-code.subagent.is_built_in`, `qwen-code.subagent.depth`, `qwen-code.subagent.status`, `qwen-code.subagent.terminate_reason`, `qwen-code.subagent.duration_ms`
Successful and cancelled GenAI spans leave `SpanStatus` as `UNSET`. Failures set `ERROR`, a bounded status description, and low-cardinality `error.type`.
#### GenAI field migration and ARMS recognition
LLM spans now use standard `gen_ai.request.*`, `gen_ai.response.*`, and `gen_ai.usage.*` fields without exact-equivalent private aliases. Request sampling attributes are written only under their standard names; no bare `temperature`, `top_p`, `max_tokens`, penalty, choice-count, or stop-sequence aliases are emitted. Tool spans similarly use `gen_ai.tool.name` without `tool.name`; blocked-on-user and hook spans keep `tool.name` because they are not GenAI Tool spans. The invalid aliases `gen_ai.usage.cached_tokens`, `gen_ai.server.time_to_first_token`, and `gen_ai.usage.reasoning_tokens` are no longer emitted. Use `gen_ai.usage.cache_read.input_tokens` for provider-reported cache reads and `gen_ai.response.time_to_first_chunk` for standard streaming latency. The private `ttft_ms` Span attribute remains available for first-user-visible-output latency and continues driving `/stats`, `sampling_ms`, and output-token throughput; `gen_ai.response.time_to_first_chunk` is an independent standard attribute measuring first normalized chunk latency. The full version-pinned contract and deferred fields are documented in [GenAI and ARMS field alignment](../../design/gen-ai-arms-field-alignment.md).
To make ARMS recognize exported spans as a GenAI application, configure its resource feature explicitly:
```json
{
"telemetry": {
"resourceAttributes": {
"acs.arms.service.feature": "genai_app"
}
}
}
```
Qwen Code does not inject this ARMS-specific resource attribute or `gen_ai.span.kind`. ARMS can infer LLM, Tool, and Agent roles from `gen_ai.operation.name`.
- `qwen-code.daemon.request`: Wraps a daemon HTTP request.
- **Attributes**: `http.request.method`, `http.route`, `qwen-code.daemon.operation`, `session.id`, `http.response.status_code`
- `qwen-code.daemon.bridge`: Wraps daemon bridge operations.
- **Attributes**: `qwen-code.daemon.operation`
#### Resource Metrics
- `qwen-code.memory.usage` (Histogram, bytes): Memory usage. Recorded by the memory-pressure monitor when telemetry is enabled.
- **Attributes**: `memory_type` (string: "heap_used"/"rss")
- `qwen-code.cpu.usage` (Histogram, percent): CPU usage percentage. Recorded by the memory-pressure monitor when telemetry is enabled.
- **Attributes**: (none)
### Performance Monitoring (Reserved)
The following metrics are defined but **not yet enabled in production**. They will be activated behind a dedicated performance monitoring config flag.
- `qwen-code.startup.duration` (Histogram, ms): CLI startup time by phase.
- **Attributes**: `phase` (string)
- `qwen-code.tool.queue.depth` (Histogram, count): Tools in execution queue.
- `qwen-code.tool.execution.breakdown` (Histogram, ms): Tool execution time by phase.
- **Attributes**: `function_name`, `phase` ("validation"/"preparation"/"execution"/"result_processing")
- `qwen-code.token.efficiency` (Histogram, ratio): Token efficiency metrics.
- **Attributes**: `model`, `metric`, `context` (optional)
- `qwen-code.performance.score` (Histogram, score): Composite performance score (0-100).
- **Attributes**: `category`, `baseline` (optional)
- `qwen-code.performance.regression` (Counter, Int): Regression detection events.
- **Attributes**: `metric`, `severity` ("low"/"medium"/"high"), `current_value`, `baseline_value`
- `qwen-code.performance.regression.percentage_change` (Histogram, percent): Percentage change vs baseline.
- **Attributes**: `metric`, `severity`, `current_value`, `baseline_value`
- `qwen-code.performance.baseline.comparison` (Histogram, percent): Performance vs baseline.
- **Attributes**: `metric`, `category`, `current_value`, `baseline_value`