mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-08-07 07:45:54 +00:00
* feat(telemetry): track tool execution outcomes Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(telemetry): address execution-status review feedback (#8180) - Update stale nonInteractiveToolExecutor expectations for executionStatus (red CI) - Scope the cancelled span-status short-circuit to tool_call events so other cancelled events carrying an error keep ERROR status - Record loop-detection skips as UNKNOWN, not EXECUTION_DENIED, keeping the denial metric accurate - Assert execution_status on the resolved-with-error PostToolBatch path - Raise tool-call observer-failure logging from warn to error - Clarify subagent-projection exclusion and JSONL compatibility in design doc * fix(core): address review findings 3-6 on tool execution status (#8180) - recordToolExecutionMetrics now merges common attributes (session.id opt-in) like every other counter in metrics.ts - Lift TOOL_FAILURE_KIND_ATTRIBUTE / TOOL_FAILURE_KIND_CANCELLED into telemetry/constants.ts so coreToolScheduler and session-tracing share one definition - Add debugLogger to runToolTelemetrySink catch (was silent) - Replace delete-based absence in withPostToolBatchStop with conditional spread * fix(core): address remaining review findings on tool execution status (#8180) - Pass the frozen executionStatus variable instead of the literal 'success' in the post-hook-stop error response, keeping the frozen value the single source of truth (finding 4) - Force-finalize the deferred PostToolBatch parent span in the abort drain, since that terminal path cancels the batch hook that otherwise owns the span; documents the invariant at the call site (finding 6) - Comment the loop-detection guard so the permission-cancellation exclusion from invalid-param loop detection is explicit (finding 9) - Rename the design doc to the dated docs/design convention and note the schedule()/handleConfirmationResponse() resolution contract change for embedders (finding 2, doc convention) * docs(core): note schedule() resolution contract in tool execution status design (#8180) Record the embedder-facing behavior change that schedule() and handleConfirmationResponse() resolve with a terminal error call rather than rejecting, so a failing tool no longer aborts its siblings. * fix(telemetry): address review feedback for tool execution status (#8180) - Document the new tool_call attributes (call_id, execution_status), the qwen-code.tool.execution.count metric, the tool.execution span attributes, and the tool.failure_kind=cancelled span field in telemetry.md. - Pass ToolErrorType explicitly at loop-detection skip sites instead of inferring it from the skip message string, so copy edits cannot silently reclassify loop skips as approval denials. - Simplify withPostToolBatchStop response construction (drop the destructure-and-reattach used to preserve a missing execution status). - Add a debug breadcrumb when a PostToolBatch stop has no span to attach to, and a one-time warning when PostToolBatch hook detection fails open. - Drop the try/catch wrapping the pure isTelemetrySdkInitialized getter. - Clarify the design doc invalid-combination wording and note that the execution-failure SLI cannot be attributed to a specific tool. - Add a regression test pinning that schedule() resolves (not rejects) when a tool execution throws. * fix(telemetry): address round-7 review feedback for tool execution status (#8180) * fix(telemetry): restore type-safety fallback for executionErrorType (#8180) * fix(telemetry): align tool execution failure outcomes Keep Core and ACP cancellation arbitration consistent, preserve structured post-processing errors, and restore QwenLogger MCP metadata privacy. Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> * fix(core): address review suggestions for tool execution status (#8180) * test(core,cli): strengthen test-efficacy for tool execution status (#8180) * fix(core): address review suggestions for tool execution status (#8180) - Improve post-processing cancellation message to indicate the tool had already completed, preventing silent model redo of completed work - Remove dead !isExecutionTimeout conjunct in Session.ts PostToolUse cancellation check (unreachable: timeout always sets toolResult.error) - Replace construct-then-delete with destructuring in withPostToolBatchStop - Move all failure-kind constants to telemetry/constants.ts so the full documented vocabulary lives in one place - Re-export StructuredToolError from tool-error.ts instead of importing from the unrelated priorReadEnforcement module - Add JSDoc to normalizeToolCallEvent documenting key-absent semantics - Add ordering-safety comment to createParentAbortRace microtask guarantee - Document endToolExecutionSpan not_started guard as defence-in-depth - Document PostToolBatch span leak window in finalizeToolSpan - Add design doc note about hand-placed cancellation check invariant - Add test for unknown execution_status normalization path - Revert unrelated generate-notices.js formatting change * fix(core): address review feedback for tool execution status (#8180) - Gate cancel message on executionThrew so the model sees 'User cancelled tool execution.' when execute() rejected under abort, reserving 'already completed' wording for post-processing cancels - Move StructuredToolError into tool-error.ts to break the tool-error ↔ priorReadEnforcement module cycle - Revert unrelated Prettier reformat in generate-notices.js * test(core): pin both tool cancellation notices; extract them as constants afd349ca gated the cancel message on executionThrew but left the two wordings as bare literals at four sites and added no test. That is the exact shape the bug had: it was introduced by editing one literal and missing the others. Extract TOOL_CANCELLED_{BEFORE,AFTER}_COMPLETION_MESSAGE so the four sites cannot drift, and add regression tests for both paths — a tool interrupted mid-flight (execute() rejected under abort) must report "User cancelled tool execution.", while a cancel after execute() returned must report that the output was discarded. The mid-flight test fails against the pre-afd349ca behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(core): keep MCP reconnect for a timeout on a dead transport Classifying every `-32001` as EXECUTION_TIMEOUT skips handleReconnectOnError, which previously recovered one real case: the transport dies mid-request, the SDK request times out because no response will ever arrive, and the server is already recorded DISCONNECTED. That reconnected and retried; now it hard-fails and the user has to retry by hand. Divert back to the reconnect path only on positive evidence the transport is dead. Note that getMCPServerStatus() reports DISCONNECTED for servers it has never seen, so the guard checks for a *recorded* DISCONNECTED — the naive comparison misroutes every timeout from a server whose status was never registered, which broke four existing timeout tests when tried. A timeout on a healthy server is still EXECUTION_TIMEOUT: retrying it after a reconnect would just double the wait. The client-side idle timeout keeps classifying unconditionally; it is our own timer, not a transport signal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(core): address blocking review feedback for tool execution status (#8180) Two blocking items from the maintainer review: 1. Post-processing cancellations dropped persistedOutputFiles (and visionBridgeNotice) along with the model-visible output, orphaning files the tool had already spilled to disk. createCancelledResponse now carries both, and every cancelAfterPostProcessing site passes what it has; the settle-then-abort and hook-stop paths do the same. 2. A -32001 that lands while the parent signal is aborted is the SDK's abort rejection or a timeout that raced with a cancel; classifying it EXECUTION_TIMEOUT would count user cancels against the timeout SLI. isExecutionTimeoutFailure now defers to the abort in both catch blocks, regardless of which side settled the race first. The two tests that pinned the opposite timeout-wins ordering are updated to the abort-wins semantics the review asked for. Co-Authored-By: Qwen Code <noreply@alibaba-inc.com> --------- Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com> Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com> Co-authored-by: qwen-code-dev-bot <qwen-code-dev@service.alibaba.com> Co-authored-by: qwen-code-dev-bot <qwen-code-dev-bot@users.noreply.github.com> Co-authored-by: qwen-code-bot <qwen-code-bot@users.noreply.github.com> Co-authored-by: qwen-code-ci-bot <253268222+qwen-code-ci-bot@users.noreply.github.com> Co-authored-by: Qwen Code Autofix <qwen-code-autofix@users.noreply.github.com> Co-authored-by: qwen-code-ci-bot <qwen-code-ci-bot@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Qwen Code <noreply@alibaba-inc.com>
943 lines
53 KiB
Markdown
943 lines
53 KiB
Markdown
# Observability with OpenTelemetry
|
|
|
|
Learn how to enable and setup OpenTelemetry for Qwen Code.
|
|
|
|
- [Observability with OpenTelemetry](#observability-with-opentelemetry)
|
|
- [Key Benefits](#key-benefits)
|
|
- [OpenTelemetry Integration](#opentelemetry-integration)
|
|
- [Configuration](#configuration)
|
|
- [Aliyun Telemetry](#aliyun-telemetry)
|
|
- [Manual OTLP Export](#manual-otlp-export)
|
|
- [Local Telemetry](#local-telemetry)
|
|
- [File-based Output (Recommended)](#file-based-output-recommended)
|
|
- [Collector-Based Export (Advanced)](#collector-based-export-advanced)
|
|
- [Logs and Metrics](#logs-and-metrics)
|
|
- [Logs](#logs)
|
|
- [Metrics](#metrics)
|
|
- [Daemon Metrics](#daemon-metrics)
|
|
- [Spans](#spans)
|
|
- [Resource Metrics](#resource-metrics)
|
|
- [Performance Monitoring (Reserved)](#performance-monitoring-reserved)
|
|
|
|
## Migration Notes
|
|
|
|
- `tool_output_truncated` was renamed to `qwen-code.tool_output_truncated` for namespace consistency — downstream consumers filtering on the old name should update their queries.
|
|
|
|
- The `tool.call.latency` histogram documentation previously listed a `decision` attribute — this was never set on the histogram (only `function_name` is recorded). The `tool.call.count` counter continues to include `decision`.
|
|
|
|
- The `qwen-code.file_operation` log event and `file.operation.count` metric documentation previously listed diff-stat attributes (`model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`) — these were never set on either. Diff-stat data is available via the `tool_call` log event's `metadata` attribute.
|
|
|
|
## Key Benefits
|
|
|
|
- **🔍 Usage Analytics**: Understand interaction patterns and feature adoption
|
|
across your team
|
|
- **⚡ Performance Monitoring**: Track response times, token consumption, and
|
|
resource utilization
|
|
- **🐛 Real-time Debugging**: Identify bottlenecks, failures, and error patterns
|
|
as they occur
|
|
- **📊 Workflow Optimization**: Make informed decisions to improve
|
|
configurations and processes
|
|
- **🏢 Enterprise Governance**: Monitor usage across teams, track costs, ensure
|
|
compliance, and integrate with existing monitoring infrastructure
|
|
|
|
## OpenTelemetry Integration
|
|
|
|
Built on **[OpenTelemetry]** — the vendor-neutral, industry-standard
|
|
observability framework — Qwen Code's observability system provides:
|
|
|
|
- **Universal Compatibility**: Export to any OpenTelemetry backend (Aliyun,
|
|
Jaeger, Prometheus, Datadog, etc.)
|
|
- **Standardized Data**: Use consistent formats and collection methods across
|
|
your toolchain
|
|
- **Future-Proof Integration**: Connect with existing and future observability
|
|
infrastructure
|
|
- **No Vendor Lock-in**: Switch between backends without changing your
|
|
instrumentation
|
|
|
|
[OpenTelemetry]: https://opentelemetry.io/
|
|
[aliyun-opentelemetry-overview]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/product-overview/what-is-tracing-analysis
|
|
[aliyun-opentelemetry-get-started]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/before-you-begin
|
|
[aliyun-opentelemetry-console-cn]: https://trace.console.aliyun.com
|
|
[aliyun-opentelemetry-console-cn-legacy]: https://tracing.console.aliyun.com
|
|
[aliyun-opentelemetry-console-intl]: https://arms.console.alibabacloud.com
|
|
|
|
## Configuration
|
|
|
|
All telemetry behavior is controlled through your `.qwen/settings.json` file.
|
|
These settings can be overridden by environment variables or CLI flags.
|
|
|
|
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
|
|
| --------------------------------- | ---------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------- |
|
|
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
|
|
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing — set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
|
|
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
|
|
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
|
|
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
|
|
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
|
|
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
|
|
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
|
|
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
|
|
| `userId` | `QWEN_TELEMETRY_USER_ID` | - | Stable end-user identifier written to GenAI spans as the ARMS extension `gen_ai.user.id`; prefer a pseudonymous value | string | - |
|
|
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include standard GenAI messages, instructions, tool definitions, tool arguments, and successful tool results as native span attributes | `true`/`false` | `false` |
|
|
| `sensitiveSpanAttributeMaxLength` | `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | - | Maximum compact JSON string length for each sensitive native span attribute. Set lower if your backend rejects large attributes. | `1..104857600` | `1048576` |
|
|
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
|
|
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
|
|
|
|
**Note on boolean environment variables:** For the boolean settings (`enabled`,
|
|
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
|
|
corresponding environment variable to `true` or `1` will enable the feature. Any
|
|
other value will disable it.
|
|
|
|
**Note on integer environment variables:** `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH`
|
|
must be a positive integer when set. Invalid values fail telemetry configuration
|
|
resolution instead of silently falling back.
|
|
|
|
`gen_ai.tool.description` is non-sensitive static registry metadata and is
|
|
emitted independently of `includeSensitiveSpanAttributes`. This includes
|
|
descriptions supplied by MCP servers and other workspace tool providers. The
|
|
value is limited to 4096 UTF-16 code units and never includes dynamic invocation
|
|
details.
|
|
|
|
**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
|
|
two things happen:
|
|
|
|
1. **Native span attributes** carry standard OpenTelemetry GenAI JSON:
|
|
- LLM input messages (`gen_ai.input.messages`)
|
|
- System instructions (`gen_ai.system_instructions`)
|
|
- Tool definitions (`gen_ai.tool.definitions`)
|
|
- LLM output messages (`gen_ai.output.messages`)
|
|
- Final executed tool arguments (`gen_ai.tool.call.arguments`)
|
|
- Successful tool results (`gen_ai.tool.call.result`)
|
|
- Interaction spans continue to use `new_context` because they are not GenAI
|
|
inference spans.
|
|
|
|
LLM values come from provider-final SDK request objects and raw provider
|
|
responses, not the original logical configuration. Tool values come from
|
|
the final invocation parameters and successful model-facing result. Each
|
|
standard GenAI value is compact JSON and must be complete and schema-valid.
|
|
A value that is invalid, cyclic, or longer than
|
|
`sensitiveSpanAttributeMaxLength` is omitted as a whole; JSON is never
|
|
truncated and no preview, hash, or truncation metadata is emitted. The
|
|
interaction-specific `new_context` attribute retains its existing
|
|
truncation behavior. The default maximum is 1 MiB (`1048576`) per attribute
|
|
and the accepted range is `1..104857600` (100 MiB). The limit is measured as
|
|
JavaScript string length rather than UTF-8 bytes. Non-ASCII content can
|
|
therefore occupy more bytes after OTLP export.
|
|
|
|
2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
|
|
logs endpoint) keep their existing `prompt`, `function_args`, and
|
|
`response_text` fields, instead of being dropped.
|
|
|
|
⚠️ **Security warning:** enabling this flag streams full conversation history,
|
|
file contents read by `read_file`, shell commands and their output (including
|
|
secrets in env vars or arguments), and model responses to the configured OTLP
|
|
backend. Treat the backend as a privileged data sink. The flag defaults to
|
|
`false`.
|
|
|
|
**Cost / payload size:** At the default limit, one LLM span can carry at most
|
|
about 4 MiB across input, output, system instructions, and tool definitions;
|
|
one Tool span can carry about 2 MiB across arguments and result. This is Qwen
|
|
Code's application-side cap, not a guarantee that every collector or backend
|
|
accepts a single attribute that large. If spans are rejected or dropped, lower
|
|
`sensitiveSpanAttributeMaxLength` (for example, to `61440`) and monitor exporter
|
|
throughput.
|
|
|
|
This setting does not disable sensitive data in OTel logs or other telemetry
|
|
sinks; non-internal API response telemetry can populate `response_text`, so
|
|
OTel logs, UI telemetry, and chat recording may receive response text
|
|
independently of this setting. QwenLogger does not include `response_text`.
|
|
|
|
**HTTP OTLP signal routing:** When using HTTP protocol (`otlpProtocol: "http"`),
|
|
Qwen Code automatically appends signal-specific paths (`/v1/traces`, `/v1/logs`,
|
|
`/v1/metrics`) to the base `otlpEndpoint`. For example, `http://collector:4318`
|
|
becomes `http://collector:4318/v1/traces` for traces. If the URL already ends
|
|
with a signal path, it is used as-is. Per-signal endpoint overrides
|
|
(`otlpTracesEndpoint`, etc.) take precedence over the base endpoint and are used
|
|
verbatim. gRPC protocol uses service-based routing and does not append paths.
|
|
|
|
The per-signal endpoint environment variables also accept the standard
|
|
OpenTelemetry names: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
|
|
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
|
|
The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
|
|
|
|
**End-user identity:** `telemetry.userId` and
|
|
`QWEN_TELEMETRY_USER_ID` are explicit opt-ins for the ARMS span attribute
|
|
`gen_ai.user.id`. The environment variable takes precedence after both values
|
|
are trimmed; a blank environment value falls back to settings. The identifier
|
|
is written only to interaction, LLM, Tool, and Agent spans. It is not a Resource
|
|
attribute, log or metric attribute, outbound Baggage value, or current
|
|
OpenTelemetry GenAI standard field. Prefer a stable pseudonymous identifier.
|
|
The value is resolved at startup, so configuration changes require a restart.
|
|
Do not configure a process-wide value on a daemon or channel instance serving
|
|
multiple end users.
|
|
|
|
For detailed information about all configuration options, see the
|
|
[Configuration Guide](../../users/configuration/settings.md).
|
|
|
|
### Resource attributes
|
|
|
|
Resource attributes are static key-value pairs attached to every span, log,
|
|
and metric exported via OTLP. Use them to slice telemetry by team, environment,
|
|
deployment region, or any other dimension your backend cares about.
|
|
|
|
Two sources, merged in priority order (lowest → highest):
|
|
|
|
1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
|
|
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
|
|
key conflict)
|
|
|
|
`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides
|
|
`service.name` from any other source (per the OpenTelemetry spec).
|
|
|
|
#### Examples
|
|
|
|
**Slice all telemetry by team / environment:**
|
|
|
|
```bash
|
|
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
|
|
```
|
|
|
|
**Route to a per-tenant collector via `service.name`:**
|
|
|
|
```bash
|
|
export OTEL_SERVICE_NAME=qwen-code-ci
|
|
```
|
|
|
|
**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"resourceAttributes": {
|
|
"deployment.environment": "production",
|
|
"service.namespace": "engineering-tooling"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
```bash
|
|
# Add a one-off tag without touching settings:
|
|
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
|
|
```
|
|
|
|
#### Reserved keys
|
|
|
|
Some keys are runtime-controlled and cannot be overridden:
|
|
|
|
- `service.version` — always set to the running CLI version. Setting it from
|
|
any source is silently dropped with a warning.
|
|
- `session.id` — runtime-injected per session. User-provided values from
|
|
either env or settings are dropped with a warning. The reason is that
|
|
Resource attributes auto-attach to every metric data point; allowing user
|
|
override would bypass [Cardinality controls](#cardinality-controls) below.
|
|
Spans and logs always carry `session.id`.
|
|
|
|
`service.name` is **not** reserved; it follows the precedence chain above.
|
|
|
|
#### Format
|
|
|
|
`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
|
|
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
|
|
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
|
|
the wrong boundary and the second half is dropped as malformed). Malformed
|
|
pairs are skipped with a warning rather than failing telemetry startup.
|
|
|
|
#### Troubleshooting: when a user-provided attribute appears not to take effect
|
|
|
|
Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
|
|
settings values, and invalid percent-encoding are all silently dropped with a
|
|
warning logged via the OpenTelemetry diagnostics channel. That channel routes
|
|
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
|
|
behavior can look like silent failure.
|
|
|
|
If a custom resource attribute isn't appearing on exported telemetry:
|
|
|
|
1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
|
|
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
|
|
(non-string settings value).
|
|
2. Verify the env var is set in the qwen-code process's environment (not just
|
|
your shell) and that values are percent-encoded.
|
|
3. Confirm `telemetry.enabled` is `true` — telemetry init only runs if enabled.
|
|
|
|
### Cardinality controls
|
|
|
|
Metrics are aggregated by attribute set at the backend — every distinct
|
|
combination of attribute values produces a new time series. Attaching a
|
|
high-cardinality field like `session.id` to a metric causes time-series fan-out
|
|
proportional to the number of sessions, which quickly exhausts metric backend
|
|
storage.
|
|
|
|
To prevent this, Qwen Code keeps high-cardinality attributes off metric data
|
|
points by default. Spans and logs are per-event and unaffected, so they
|
|
continue to carry `session.id` for trace and log correlation.
|
|
|
|
#### `telemetry.metrics.includeSessionId` (default: `false`)
|
|
|
|
Setting this to `true` (via settings or
|
|
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
|
|
every metric data point.
|
|
|
|
⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a
|
|
fleet will blow up metric storage. Recommended only for short-term debugging.
|
|
For long-term session correlation, query trace or log backends instead.
|
|
|
|
#### Migration from earlier versions
|
|
|
|
Prior to this release, `session.id` was attached to metrics by default. If
|
|
your Prometheus queries / Grafana dashboards / alert rules reference
|
|
`session_id` on a metric, you have two options:
|
|
|
|
**Option A** — restore the previous behavior for short-term debugging:
|
|
|
|
```bash
|
|
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
|
|
```
|
|
|
|
or:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"metrics": { "includeSessionId": true }
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B (recommended)** — move session-level analysis off metrics. Spans
|
|
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
|
|
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
|
|
cardinality pressure.
|
|
|
|
### Client-side HTTP span on outbound fetch
|
|
|
|
When telemetry is enabled, Qwen Code registers `UndiciInstrumentation`
|
|
which creates a client-side HTTP span for every outbound `fetch()`
|
|
request originated by the process — including the LLM SDKs (`openai`,
|
|
`@google/genai`, `@anthropic-ai/sdk`), the MCP StreamableHTTP client, the
|
|
`WebFetch` tool, and any IDE-extension out-of-process calls. The span
|
|
lets you see network latency (TTFB / response body transfer) separately
|
|
from upstream model processing time, which the existing
|
|
`api.generateContent` span alone can't distinguish.
|
|
|
|
These spans go to your **own** OTLP collector (or file outfile) just like
|
|
the rest of the telemetry — they do not affect what is written onto the
|
|
outbound HTTP request itself. Whether the W3C `traceparent` header is
|
|
also written into the outgoing request stream is controlled by a
|
|
**separate, security-relevant setting** documented in
|
|
[outbound correlation](#outbound-correlation-security-relevant) below.
|
|
|
|
**Feedback-loop avoidance.** OTel SDK uses `fetch` internally to upload OTLP
|
|
data. Without protection, instrumenting `fetch` would trace those uploads,
|
|
which would themselves be uploaded, causing an infinite loop. Qwen Code's
|
|
undici instrumentation is configured with an `ignoreRequestHook` that skips
|
|
URLs matching the configured `telemetry.otlpEndpoint` /
|
|
`telemetry.otlpTracesEndpoint` / `telemetry.otlpLogsEndpoint` /
|
|
`telemetry.otlpMetricsEndpoint` prefixes. In file-outfile mode there are no
|
|
outbound HTTP uploads, so the hook is a no-op.
|
|
|
|
## Outbound correlation (SECURITY-RELEVANT)
|
|
|
|
These settings live in a **separate top-level namespace** from `telemetry.*`
|
|
on purpose: telemetry controls data flow into the operator's own
|
|
observability backend, while `outboundCorrelation.*` controls what
|
|
client-side correlation data qwen-code writes **into outbound LLM API
|
|
request streams** that reach third-party LLM provider endpoints
|
|
(DashScope, OpenAI, Anthropic, etc.). Different recipients, different
|
|
consent decision. **All values default to off.** See PR #4390 review
|
|
discussion for the framing rationale.
|
|
|
|
### `outboundCorrelation.propagateTraceContext`
|
|
|
|
```jsonc
|
|
"outboundCorrelation": {
|
|
"propagateTraceContext": false // default
|
|
}
|
|
```
|
|
|
|
When `false` (default), Qwen Code installs a no-op `TextMapPropagator` on
|
|
the OTel SDK. UndiciInstrumentation still creates client HTTP spans for
|
|
your OTLP collector, but `propagation.inject()` is a no-op so **no
|
|
`traceparent` is written onto outbound requests**. Trace IDs stay
|
|
internal to the operator's collector.
|
|
|
|
When `true`, the SDK's default W3C composite propagator
|
|
(`tracecontext` + `baggage`) is installed and the standard `traceparent`
|
|
header is written on every outbound `fetch`:
|
|
|
|
```
|
|
traceparent: 00-<32-hex traceId>-<16-hex parentSpanId>-<01-sampled | 00-not-sampled>
|
|
```
|
|
|
|
Additionally, `TRACEPARENT` and `TRACESTATE` environment variables are set in
|
|
shell child processes (Bash tool, hooks, monitor) so that spawned commands can
|
|
participate in the same distributed trace.
|
|
|
|
Opt in only when the LLM provider also reports into your OTel collector
|
|
for cross-process trace stitching — e.g. ARMS Tracing serving DashScope.
|
|
For most operators the value is `false`; cross-vendor trace continuation
|
|
is niche.
|
|
|
|
**Depends on `telemetry.enabled: true`.** The OTel SDK only initializes
|
|
when telemetry is enabled, so `propagateTraceContext` only takes effect
|
|
in that state. Setting it to `true` while telemetry is disabled is a
|
|
silent no-op — no SDK, no propagator, no `traceparent` on the wire.
|
|
Verify both flags when wiring an ARMS+DashScope correlation setup:
|
|
|
|
```jsonc
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"otlpTracesEndpoint": "http://tracing-analysis-...",
|
|
},
|
|
"outboundCorrelation": {
|
|
"propagateTraceContext": true,
|
|
},
|
|
}
|
|
```
|
|
|
|
### Other outbound correlation headers
|
|
|
|
`X-Qwen-Code-Session-Id` and `X-Qwen-Code-Request-Id` are **not part of
|
|
this PR**. They will be designed and proposed in their own follow-up
|
|
PR(s) under the same `outboundCorrelation.*` namespace, each with its
|
|
own threat model and operator-consent flow. PR #4390 review (LaZzyMan)
|
|
established the principle: "telemetry's scope of work doesn't include
|
|
sending identifiers to LLM providers"; correlation-header work moves to
|
|
its own design discussion rather than landing under telemetry.
|
|
|
|
## Aliyun Telemetry
|
|
|
|
### Manual OTLP Export
|
|
|
|
To view Qwen Code telemetry in Alibaba Cloud Managed Service for
|
|
OpenTelemetry, configure Qwen Code to export to the OTLP endpoint
|
|
provided by ARMS.
|
|
|
|
Setting `"target": "gcp"` alone does not configure the export
|
|
destination. If `otlpEndpoint` is not set, Qwen Code still defaults to
|
|
`http://localhost:4317`. If `outfile` is set, it overrides
|
|
`otlpEndpoint` and telemetry is written to the file instead of being
|
|
sent to Alibaba Cloud.
|
|
|
|
1. Enable telemetry in your `.qwen/settings.json` and set the OTLP
|
|
endpoint:
|
|
|
|
**Option A: gRPC protocol** (standard OTLP endpoint):
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"target": "gcp",
|
|
"otlpEndpoint": "https://<your-otlp-endpoint>",
|
|
"otlpProtocol": "grpc"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B: HTTP protocol with per-signal endpoints** (for backends
|
|
that use non-standard paths, e.g., `/api/otlp/traces` instead of
|
|
`/v1/traces`):
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"otlpProtocol": "http",
|
|
"otlpTracesEndpoint": "http://<host>/<token>/api/otlp/traces",
|
|
"otlpLogsEndpoint": "http://<host>/<token>/api/otlp/logs",
|
|
"otlpMetricsEndpoint": "http://<host>/<token>/api/otlp/metrics"
|
|
}
|
|
}
|
|
```
|
|
|
|
> **Note:** When using HTTP protocol with only `otlpEndpoint` (no
|
|
> per-signal overrides), Qwen Code appends standard OTLP paths
|
|
> (`/v1/traces`, `/v1/logs`, `/v1/metrics`) to the base URL. If your
|
|
> backend uses different paths, use per-signal endpoint overrides as
|
|
> shown in Option B.
|
|
|
|
To populate ARMS Session Analysis `User ID`, add a stable pseudonymous
|
|
identity as a span-level setting:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"userId": "user-079458",
|
|
"resourceAttributes": {
|
|
"acs.arms.service.feature": "genai_app"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
For container deployments, set
|
|
`QWEN_TELEMETRY_USER_ID=user-079458` instead. A custom
|
|
`telemetry.resourceAttributes.user.id` remains an unrelated Resource
|
|
dimension and does not populate ARMS Session Analysis; remove it when
|
|
migrating to the span-level setting.
|
|
|
|
2. If your Alibaba Cloud endpoint requires authentication, provide OTLP
|
|
headers through standard OpenTelemetry environment variables such as
|
|
`OTEL_EXPORTER_OTLP_HEADERS` (or the signal-specific variants). Qwen
|
|
Code does not currently expose OTLP auth headers directly in
|
|
`.qwen/settings.json`.
|
|
3. Run Qwen Code and send prompts.
|
|
4. View telemetry in Managed Service for OpenTelemetry:
|
|
- Product overview:
|
|
[What is Managed Service for OpenTelemetry?][aliyun-opentelemetry-overview]
|
|
- Getting started:
|
|
[Get started with Managed Service for OpenTelemetry][aliyun-opentelemetry-get-started]
|
|
- Console entry points:
|
|
- China mainland:
|
|
[trace.console.aliyun.com][aliyun-opentelemetry-console-cn]
|
|
(legacy console:
|
|
[tracing.console.aliyun.com][aliyun-opentelemetry-console-cn-legacy])
|
|
- International:
|
|
[arms.console.alibabacloud.com][aliyun-opentelemetry-console-intl]
|
|
- In the console, use `Applications` to inspect traces and service
|
|
topology.
|
|
- To locate the OTLP endpoint and access information:
|
|
- **New console** (`trace.console.aliyun.com` or international):
|
|
navigate to `Integration Center`.
|
|
- **Legacy console** (`tracing.console.aliyun.com`): navigate to
|
|
`Cluster Configurations` → `Access point information`.
|
|
|
|
## Local Telemetry
|
|
|
|
For local development and debugging, you can capture telemetry data locally:
|
|
|
|
### File-based Output (Recommended)
|
|
|
|
1. Enable telemetry in your `.qwen/settings.json`:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"outfile": ".qwen/telemetry.log"
|
|
}
|
|
}
|
|
```
|
|
|
|
> **Note:** When `outfile` is set, OTLP export is automatically disabled.
|
|
> The `target` and `otlpEndpoint` settings are not needed for file-only
|
|
> output and can be safely omitted from your config.
|
|
|
|
2. Run Qwen Code and send prompts.
|
|
3. View logs and metrics in the specified file (e.g., `.qwen/telemetry.log`).
|
|
|
|
### Collector-Based Export (Advanced)
|
|
|
|
1. Run the automation script:
|
|
```bash
|
|
npm run telemetry -- --target=local
|
|
```
|
|
This will:
|
|
- Download and start Jaeger and OTEL collector
|
|
- Configure your workspace for local telemetry
|
|
- Provide a Jaeger UI at http://localhost:16686
|
|
- Save logs/metrics to `~/.qwen/tmp/<projectHash>/otel/collector.log`
|
|
- Stop collector on exit (e.g. `Ctrl+C`)
|
|
2. Run Qwen Code and send prompts.
|
|
3. View traces at http://localhost:16686 and logs/metrics in the collector log
|
|
file.
|
|
|
|
## Logs and Metrics
|
|
|
|
The following section describes the structure of logs, metrics, and spans
|
|
generated for Qwen Code.
|
|
|
|
- A `sessionId` is included as a common attribute on all logs and metrics.
|
|
|
|
### Logs
|
|
|
|
Logs are timestamped records of specific events. All log records automatically include `event.name` and `event.timestamp` attributes.
|
|
|
|
The following events are logged:
|
|
|
|
#### Core Session Events
|
|
|
|
- `qwen-code.config`: Emitted once at startup with CLI configuration.
|
|
- **Attributes**: `model`, `sandbox_enabled`, `core_tools_enabled`, `approval_mode`, `file_filtering_respect_git_ignore`, `debug_mode`, `truncate_tool_output_threshold`, `truncate_tool_output_lines`, `hooks` (comma-separated, omitted if disabled), `ide_enabled`, `interactive_shell_enabled`, `mcp_servers`, `mcp_servers_count`, `mcp_tools`, `mcp_tools_count`, `output_format`, `skills`, `subagents`
|
|
|
|
- `qwen-code.user_prompt`: User submits a prompt.
|
|
- **Attributes**: `prompt_length` (int), `prompt_id` (string), `prompt` (string, excluded if `log_prompts_enabled` is false), `auth_type` (string)
|
|
|
|
- `qwen-code.user_retry`: User retries the last prompt.
|
|
- **Attributes**: `prompt_id` (string)
|
|
|
|
- `qwen-code.conversation_finished`: A conversation turn sequence completes.
|
|
- **Attributes**: `approvalMode` (string), `turnCount` (int)
|
|
|
|
- `qwen-code.user_feedback`: User submits session feedback.
|
|
- **Attributes**: `session_id` (string), `rating` (int: 1=bad, 2=fine, 3=good), `model` (string), `approval_mode` (string), `prompt_id` (string, optional)
|
|
|
|
#### Tool Events
|
|
|
|
- `qwen-code.tool_call`: Each function/tool call. Terminal events are normalized so `status` is authoritative: success and cancelled events omit error fields, while error events always have a non-empty `error_type` (`unknown` when the producer did not classify the error). Blank tool names are emitted as `unknown_tool`. A missing `execution_status` is normalized to `unknown` and is never inferred from the terminal `status`.
|
|
- **Attributes**: `function_name` (string), `function_args` (object), `call_id` (string, optional), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `execution_status` (string: "not_started", "success", "error", "cancelled", or "unknown"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, present for error events), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`)
|
|
|
|
- `qwen-code.file_operation`: Each file operation.
|
|
- **Attributes**: `tool_name` (string), `operation` (string: "create", "read", "update"), `lines` (int, optional), `mimetype` (string, optional), `extension` (string, optional), `programming_language` (string, optional)
|
|
|
|
- `qwen-code.tool_output_truncated`: Tool output exceeded size threshold.
|
|
- **Attributes**: `tool_name` (string), `original_content_length` (int), `truncated_content_length` (int), `threshold` (int), `lines` (int), `prompt_id` (string)
|
|
|
|
#### API Events
|
|
|
|
- `qwen-code.api_request`: Outgoing request to the LLM API.
|
|
- **Attributes**: `model` (string), `prompt_id` (string), `request_text` (string, optional), `subagent_name` (string, optional)
|
|
|
|
- `qwen-code.api_response`: Response received from LLM API.
|
|
- **Attributes**: `response_id` (string), `model` (string), `status_code` (int/string, optional), `duration_ms` (int), `input_token_count` (int), `output_token_count` (int), `cached_content_token_count` (int), `thoughts_token_count` (int), `total_token_count` (int), `prompt_id` (string), `auth_type` (string, optional), `response_text` (string, optional), `subagent_name` (string, optional)
|
|
|
|
- `qwen-code.api_error`: API request failed.
|
|
- **Attributes**: `model` (string), `prompt_id` (string), `duration_ms` (int), `error_message` (string), `response_id` (string, optional), `auth_type` (string, optional), `error_type` (string, optional), `status_code` (int/string, optional), `subagent_name` (string, optional)
|
|
|
|
Additionally, OTel-standard aliases (`http.status_code`, `error.message`, `model_name`, `duration`) are emitted for compatibility.
|
|
|
|
- `qwen-code.api_cancel`: API request cancelled by user.
|
|
- **Attributes**: `model` (string), `prompt_id` (string), `auth_type` (string, optional), `loop_wakeups_cancelled` (int, optional)
|
|
|
|
- `qwen-code.api_retry`: HTTP-status retry (429/5xx) at an LLM call site. Distinct from `chat.content_retry` which handles `InvalidStreamError` retries on a separate budget.
|
|
- **Attributes**: `model` (string), `prompt_id` (string, optional), `attempt_number` (int), `error_type` (string, optional), `error_message` (string), `status_code` (int/string, optional), `retry_delay_ms` (int), `duration_ms` (int, equals retry_delay_ms — backoff sleep, not HTTP round-trip; for attempt duration see the qwen-code.llm_request span), `subagent_name` (string, optional)
|
|
|
|
- `qwen-code.malformed_json_response`: `generateJson` response couldn't be parsed.
|
|
- **Attributes**: `model` (string)
|
|
|
|
- `qwen-code.flash_fallback`: Switched to flash model as fallback.
|
|
- **Attributes**: `auth_type` (string)
|
|
|
|
- `qwen-code.ripgrep_fallback`: Switched to grep as fallback.
|
|
- **Attributes**: `use_ripgrep` (boolean), `use_builtin_ripgrep` (boolean), `error` (string, optional)
|
|
|
|
#### Resilience Events
|
|
|
|
- `qwen-code.chat.content_retry`: Content-error retry (e.g. empty stream).
|
|
- **Attributes**: `attempt_number` (int), `error_type` (string), `retry_delay_ms` (int), `model` (string)
|
|
|
|
- `qwen-code.chat.content_retry_failure`: All content retries exhausted.
|
|
- **Attributes**: `total_attempts` (int), `final_error_type` (string), `total_duration_ms` (int, optional), `model` (string)
|
|
|
|
- `qwen-code.chat.invalid_chunk`: Invalid chunk received from stream.
|
|
- **Attributes**: `error.message` (string, optional)
|
|
|
|
#### Command & Extension Events
|
|
|
|
- `qwen-code.slash_command`: User executes a slash command.
|
|
- **Attributes**: `command` (string), `subcommand` (string, optional), `status` (string: "success" or "error", optional)
|
|
|
|
- `qwen-code.slash_command.model`: User switches model via `/model` command.
|
|
- **Attributes**: `model_name` (string)
|
|
|
|
- `qwen-code.skill_launch`: A skill is launched.
|
|
- **Attributes**: `skill_name` (string), `success` (boolean), `prompt_id` (string)
|
|
|
|
- `qwen-code.extension_install`: Extension installed.
|
|
- **Attributes**: `extension_name` (string), `extension_version` (string), `extension_source` (string), `status` (string: "success"/"error")
|
|
|
|
- `qwen-code.extension_uninstall`: Extension uninstalled.
|
|
- **Attributes**: `extension_name` (string), `status` (string)
|
|
|
|
- `qwen-code.extension_enable`: Extension enabled.
|
|
- **Attributes**: `extension_name` (string), `setting_scope` (string)
|
|
|
|
- `qwen-code.extension_disable`: Extension disabled.
|
|
- **Attributes**: `extension_name` (string), `setting_scope` (string)
|
|
|
|
- `qwen-code.extension_update`: Extension updated.
|
|
- **Attributes**: `extension_name` (string), `extension_id` (string), `extension_previous_version` (string), `extension_version` (string), `extension_source` (string), `status` (string: "success"/"error")
|
|
|
|
- `qwen-code.ide_connection`: IDE connection event.
|
|
- **Attributes**: `connection_type` (string: "start" or "session")
|
|
|
|
- `qwen-code.auth`: Authentication event.
|
|
- **Attributes**: `auth_type` (string), `action_type` ("auto", "manual", "coding-plan"), `status` ("success", "error", "cancelled"), `error_message` (optional)
|
|
|
|
#### Subagent Events
|
|
|
|
- `qwen-code.subagent_execution`: Subagent lifecycle event.
|
|
- **Attributes**: `subagent_name` (string), `status` ("started", "completed", "failed", "cancelled"), `terminate_reason` (optional), `result` (optional), `execution_summary` (optional)
|
|
|
|
#### Arena Events
|
|
|
|
- `qwen-code.arena_session_started`: Arena session begins.
|
|
- **Attributes**: `arena_session_id` (string), `model_ids` (JSON string array), `task_length` (int)
|
|
|
|
- `qwen-code.arena_agent_completed`: An arena agent finishes.
|
|
- **Attributes**: `arena_session_id` (string), `agent_session_id` (string), `agent_model_id` (string), `status` (string: "completed"/"failed"/"cancelled"), `duration_ms` (int), `rounds` (int), `total_tokens` (int), `input_tokens` (int), `output_tokens` (int), `tool_calls` (int), `successful_tool_calls` (int), `failed_tool_calls` (int)
|
|
|
|
- `qwen-code.arena_session_ended`: Arena session completes.
|
|
- **Attributes**: `arena_session_id` (string), `status` (string: "selected"/"discarded"/"failed"/"cancelled"), `duration_ms` (int), `display_backend` (string, optional), `agent_count` (int), `completed_agents` (int), `failed_agents` (int), `cancelled_agents` (int), `winner_model_id` (string, optional)
|
|
|
|
#### Workflow Events
|
|
|
|
- `qwen-code.workflow_keyword`: Workflow keyword trigger fired.
|
|
|
|
- `qwen-code.workflow_run`: Workflow run reached terminal state.
|
|
- **Attributes**: `status` (string), `agents_dispatched` (int), `agents_completed` (int), `phase_count` (int), `tokens_spent` (int), `duration_ms` (int)
|
|
|
|
#### Auto-Memory Events
|
|
|
|
- `qwen-code.memory.extract`: Memory extraction run completed.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status` ("completed"/"skipped"/"failed"), `skipped_reason` (optional), `patches_count` (int), `touched_topics` (string), `duration_ms` (int)
|
|
|
|
- `qwen-code.memory.dream`: Memory consolidation (dream) run completed.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status` ("updated"/"noop"/"failed"/"cancelled"), `deduped_entries` (int), `touched_topics_count` (int), `touched_topics` (string), `duration_ms` (int)
|
|
|
|
- `qwen-code.memory.recall`: Memory recall operation completed.
|
|
- **Attributes**: `query_length` (int), `docs_scanned` (int), `docs_selected` (int), `strategy` ("none"/"heuristic"/"model"), `duration_ms` (int)
|
|
|
|
#### Prompt Suggestion & Speculation Events
|
|
|
|
- `qwen-code.prompt_suggestion`: Prompt suggestion outcome.
|
|
- **Attributes**: `outcome` ("accepted"/"ignored"/"suppressed"), `prompt_id` (optional), `accept_method` ("tab"/"enter"/"right", optional), `accept_source` ("live"/"fallback", optional), `time_to_accept_ms` (optional), `time_to_ignore_ms` (optional), `time_to_first_keystroke_ms` (optional), `suggestion_length` (optional), `similarity` (optional), `was_focused_when_shown` (optional), `reason` (optional)
|
|
|
|
- `qwen-code.speculation`: Speculative execution outcome.
|
|
- **Attributes**: `outcome` ("accepted"/"aborted"/"failed"), `turns_used` (int), `files_written` (int), `tool_use_count` (int), `duration_ms` (int), `boundary_type` (optional), `had_pipelined_suggestion` (boolean)
|
|
|
|
#### Other Events
|
|
|
|
- `qwen-code.chat_compression`: Chat context compressed.
|
|
- **Attributes**: `tokens_before` (int), `tokens_after` (int), `compression_input_token_count` (int, optional), `compression_output_token_count` (int, optional)
|
|
|
|
- `qwen-code.next_speaker_check`: Next speaker determination.
|
|
- **Attributes**: `prompt_id` (string), `finish_reason` (string), `result` (string)
|
|
|
|
- `loop_detected`: Loop detected during agent execution. _(Note: emitted without `qwen-code.` prefix — pre-existing inconsistency.)_
|
|
- **Attributes**: `loop_type` (string), `prompt_id` (string)
|
|
|
|
- `kitty_sequence_overflow`: Kitty graphics protocol sequence exceeded buffer size. _(Note: emitted without `qwen-code.` prefix — pre-existing inconsistency.)_
|
|
- **Attributes**: `sequence_length` (int), `truncated_sequence` (string, first 20 chars)
|
|
|
|
### Metrics
|
|
|
|
Metrics are numerical measurements of behavior over time. Metric names use the `qwen-code.*` prefix.
|
|
|
|
#### Core Metrics
|
|
|
|
- `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup.
|
|
|
|
- `qwen-code.tool.call.count` (Counter, Int): Counts tool calls.
|
|
- **Attributes**: `function_name`, `status` ("success"/"error"/"cancelled"), `success` (boolean, retained for compatibility), `decision` ("accept"/"reject"/"auto_accept"/"modify", optional), `tool_type` ("mcp"/"native", optional)
|
|
|
|
- `qwen-code.tool.execution.count` (Counter, Int): Counts tool execution outcomes. Deliberately carries no `function_name` dimension to stay low-cardinality, so an execution-failure rate cannot be attributed to a specific tool without dropping to the `qwen-code.tool_call` logs; exclude `unknown`, `not_started`, and `cancelled` when computing execution-failure ratios (denominator is `success` + `error`).
|
|
- **Attributes**: `execution_status` ("not_started"/"success"/"error"/"cancelled"/"unknown"), `tool_type` ("mcp"/"native"), plus globally configured common metric attributes such as the opt-in `session.id`
|
|
|
|
- `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency.
|
|
- **Attributes**: `function_name` (string)
|
|
|
|
- `qwen-code.api.request.count` (Counter, Int): Counts all API requests.
|
|
- **Attributes**: `model`, `status_code`, `error_type` (optional)
|
|
|
|
- `qwen-code.api.request.latency` (Histogram, ms): Measures API request latency.
|
|
- **Attributes**: `model` (string)
|
|
|
|
- `qwen-code.token.usage` (Counter, Int): Counts tokens used.
|
|
- **Attributes**: `model`, `type` ("input"/"output"/"thought"/"cache")
|
|
|
|
- `qwen-code.file.operation.count` (Counter, Int): Counts file operations.
|
|
- **Attributes**: `operation` ("create"/"read"/"update"), `lines` (optional), `mimetype` (optional), `extension` (optional), `programming_language` (optional)
|
|
|
|
- `qwen-code.chat_compression` (Counter, Int): Counts chat compression operations.
|
|
- **Attributes**: `tokens_before` (int), `tokens_after` (int)
|
|
|
|
- `qwen-code.slash_command.model.call_count` (Counter, Int): Counts model slash command calls.
|
|
- **Attributes**: `slash_command.model.model_name` (string)
|
|
|
|
- `qwen-code.subagent.execution.count` (Counter, Int): Counts subagent execution events.
|
|
- **Attributes**: `subagent_name`, `status` ("started"/"completed"/"failed"/"cancelled"), `terminate_reason` (optional)
|
|
|
|
#### Resilience Metrics
|
|
|
|
- `qwen-code.api.retry.count` (Counter, Int): HTTP-status retries (429/5xx) at LLM call sites.
|
|
- **Attributes**: `model` (string)
|
|
|
|
- `qwen-code.chat.content_retry.count` (Counter, Int): Retries due to content errors.
|
|
|
|
- `qwen-code.chat.content_retry_failure.count` (Counter, Int): All content retries exhausted.
|
|
|
|
- `qwen-code.chat.invalid_chunk.count` (Counter, Int): Invalid chunks from stream.
|
|
|
|
#### Arena Metrics
|
|
|
|
- `qwen-code.arena.session.count` (Counter, Int): Arena sessions by status.
|
|
- **Attributes**: `status`, `display_backend` (optional)
|
|
|
|
- `qwen-code.arena.session.duration` (Histogram, ms): Arena session duration.
|
|
- **Attributes**: `status`
|
|
|
|
- `qwen-code.arena.agent.count` (Counter, Int): Arena agent completions.
|
|
- **Attributes**: `status`, `model_id`
|
|
|
|
- `qwen-code.arena.agent.duration` (Histogram, ms): Arena agent execution duration.
|
|
- **Attributes**: `model_id`
|
|
|
|
- `qwen-code.arena.agent.tokens` (Counter, Int): Token usage by arena agents.
|
|
- **Attributes**: `model_id`, `type` ("input"/"output")
|
|
|
|
- `qwen-code.arena.result.selected` (Counter, Int): Arena result selections.
|
|
- **Attributes**: `model_id`
|
|
|
|
#### Auto-Memory Metrics
|
|
|
|
- `qwen-code.memory.extract.count` (Counter, Int): Auto-memory extraction runs.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status`
|
|
|
|
- `qwen-code.memory.extract.duration` (Histogram, ms): Extraction duration.
|
|
- **Attributes**: `trigger`, `status`
|
|
|
|
- `qwen-code.memory.dream.count` (Counter, Int): Auto-memory dream runs.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status`
|
|
|
|
- `qwen-code.memory.dream.duration` (Histogram, ms): Dream run duration.
|
|
- **Attributes**: `trigger`, `status`
|
|
|
|
- `qwen-code.memory.recall.count` (Counter, Int): Auto-memory recall operations.
|
|
- **Attributes**: `strategy` ("none"/"heuristic"/"model")
|
|
|
|
- `qwen-code.memory.recall.duration` (Histogram, ms): Recall duration.
|
|
- **Attributes**: `strategy`
|
|
|
|
#### API Request Breakdown
|
|
|
|
- `qwen-code.api.request.breakdown` (Histogram, ms): API request time breakdown by phase.
|
|
- **Attributes**: `model`, `phase` ("request_preparation"/"network_latency"/"response_processing"/"token_processing")
|
|
|
|
### Daemon Metrics
|
|
|
|
The daemon process (long-running HTTP server mode) exposes its own metrics.
|
|
|
|
> **Note:** The three Observable Gauges (`daemon.session.active`, `daemon.sse.active`, `daemon.process.heap_used`) are callback-based metrics updated at each collection interval; `registerDaemonGaugeCallbacks()` must be invoked during daemon initialization to register the observation callbacks.
|
|
|
|
#### HTTP
|
|
|
|
- `qwen-code.daemon.http.request.count` (Counter, Int): Request count by route and status class.
|
|
- **Attributes**: `route`, `status_class` ("2xx"/"4xx"/"5xx")
|
|
|
|
- `qwen-code.daemon.http.request.duration` (Histogram, ms): Request duration.
|
|
- **Attributes**: `route`
|
|
- **Buckets**: 1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000
|
|
|
|
#### Sessions
|
|
|
|
- `qwen-code.daemon.session.active` (ObservableGauge, Int): Current active sessions.
|
|
|
|
- `qwen-code.daemon.session.lifecycle` (Counter, Int): Session lifecycle events.
|
|
- **Attributes**: `action` ("spawn"/"close"/"die")
|
|
|
|
#### Channels
|
|
|
|
- `qwen-code.daemon.channel.lifecycle` (Counter, Int): ACP channel lifecycle events.
|
|
- **Attributes**: `action` ("spawn"/"exit"), `expected` (boolean, optional)
|
|
|
|
#### Prompts
|
|
|
|
- `qwen-code.daemon.prompt.queue_wait` (Histogram, ms): Prompt FIFO queue wait time.
|
|
- **Buckets**: 1, 5, 10, 50, 100, 500, 1000, 5000, 10000, 30000, 60000
|
|
|
|
- `qwen-code.daemon.prompt.duration` (Histogram, ms): End-to-end prompt duration.
|
|
- **Buckets**: 100, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000
|
|
|
|
#### Errors
|
|
|
|
- `qwen-code.daemon.bridge.error.count` (Counter, Int): Bridge errors by type.
|
|
- **Attributes**: `error_type` (known class name or "unknown")
|
|
|
|
- `qwen-code.daemon.cancel.count` (Counter, Int): Cancel request count.
|
|
|
|
#### Resources
|
|
|
|
- `qwen-code.daemon.sse.active` (ObservableGauge, Int): Active SSE connections.
|
|
|
|
- `qwen-code.daemon.process.heap_used` (ObservableGauge, Int, bytes): Heap memory usage.
|
|
|
|
### Spans
|
|
|
|
Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each interaction is a trace root with its own `traceId`; cross-prompt correlation uses the `session.id` attribute.
|
|
|
|
- `qwen-code.interaction`: Root span for each user prompt turn.
|
|
- **Attributes**: `session.id`, optional ARMS extension `gen_ai.user.id`, `qwen-code.prompt_id`, `qwen-code.message_type`, `qwen-code.model`, `qwen-code.approval_mode`, `interaction.sequence`, `interaction.duration_ms`, `qwen-code.turn_status` ("ok"/"error"/"cancelled")
|
|
|
|
- `qwen-code.llm_request`: Wraps a single LLM API call.
|
|
- **GenAI attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.conversation.id`, optional ARMS extension `gen_ai.user.id`, `gen_ai.request.model`, `gen_ai.request.stream`, `gen_ai.request.choice.count`, `gen_ai.request.max_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.frequency_penalty`, `gen_ai.request.presence_penalty`, `gen_ai.request.stop_sequences`, optional `gen_ai.output.type`, `gen_ai.response.id`, `gen_ai.response.model`, `gen_ai.response.finish_reasons`, `gen_ai.response.time_to_first_chunk`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_creation.input_tokens`
|
|
- **Compatibility attributes**: `session.id`, `qwen-code.prompt_id`, `llm_request.context` ("subagent"/"interaction"/"standalone"), `duration_ms`, `ttft_ms`, `request_setup_ms`, `attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`, `success`, `error`, `finish_reason`, `thoughts_token_count`, `subagent_name`, `error_type`, `error_status_code`
|
|
- Standard response fields come from the provider response. Standard token fields are emitted only for provider-reported non-negative safe integers. If the provider reports only a total token count, input/output usage is omitted rather than estimated.
|
|
- Standard request-parameter fields come from the first provider-final SDK request object after adapter defaults, overrides, unsupported-field removal, and output-window clamps. Qwen Code does not infer SDK or server defaults.
|
|
- Streaming requests emit `gen_ai.request.stream=true`. `gen_ai.response.time_to_first_chunk` measures seconds from the provider call to the first normalized response yielded by the provider adapter, which may differ from the first raw network frame. Non-streaming requests omit both standard streaming attributes because an absent `gen_ai.request.stream` means non-streaming in the semantic convention.
|
|
|
|
- `qwen-code.tool`: Wraps the full tool lifecycle (approval wait + execution).
|
|
- **Attributes**: `session.id`, optional ARMS extension `gen_ai.user.id`, `gen_ai.operation.name` (`execute_tool`), `gen_ai.tool.name`, `gen_ai.tool.type` (`function`), `gen_ai.tool.call.id`, `tool.call_id`, `duration_ms`, `success`, `error`, `tool.failure_kind` (string, optional — the specific failure reason, e.g. "cancelled", "tool_error", "tool_exception", "timeout", "permission_denied", "pre_hook_blocked")
|
|
|
|
- `qwen-code.tool.execution`: Wraps the tool execution phase (after approval). Emitted only for attempted executions.
|
|
- **Attributes**: `session.id`, `gen_ai.tool.name` (optional), `tool.call_id` (optional), `duration_ms`, `success`, `error`, `execution_status` ("success"/"error"/"cancelled"), `error_type`, `error.type`
|
|
|
|
- `qwen-code.tool.blocked_on_user`: Time a tool spends waiting on user approval.
|
|
- **Attributes**: `session.id`, `tool.name`, `tool.call_id`, `duration_ms`, `decision` ("proceed_once"/"proceed_always"/"cancel"/"aborted"/"auto_approved"/"error"), `source` ("cli"/"ide"/"hook"/"auto"/"system")
|
|
|
|
- `qwen-code.hook`: Wraps each pre/post-tool-use hook fire site.
|
|
- **Attributes**: `session.id`, `hook_event` ("PreToolUse"/"PostToolUse"/"PostToolUseFailure"/"PostToolBatch"), `tool.name`, `tool.use_id` (optional), `is_interrupt` (boolean, optional), `duration_ms`, `success`, `should_proceed` (optional), `should_stop` (optional), `block_type` (optional), `error` (optional)
|
|
|
|
- `qwen-code.subagent`: Wraps a single subagent invocation.
|
|
- **Attributes**: `gen_ai.operation.name` (`invoke_agent`), `gen_ai.agent.name`, `gen_ai.agent.description`, `gen_ai.conversation.id`, optional ARMS extension `gen_ai.user.id`, optional `gen_ai.request.model`, `qwen-code.subagent.id`, `qwen-code.subagent.name`, `qwen-code.subagent.invocation_kind` ("foreground"/"fork"/"background"), `qwen-code.subagent.is_built_in`, `qwen-code.subagent.depth`, `qwen-code.subagent.status`, `qwen-code.subagent.terminate_reason`, `qwen-code.subagent.duration_ms`
|
|
|
|
#### GenAI field migration and ARMS recognition
|
|
|
|
LLM spans now use standard `gen_ai.request.*`, `gen_ai.response.*`, and `gen_ai.usage.*` fields without exact-equivalent private aliases. Request sampling attributes are written only under their standard names; no bare `temperature`, `top_p`, `max_tokens`, penalty, choice-count, or stop-sequence aliases are emitted. Tool spans similarly use `gen_ai.tool.name` without `tool.name`; blocked-on-user and hook spans keep `tool.name` because they are not GenAI Tool spans. The invalid aliases `gen_ai.usage.cached_tokens`, `gen_ai.server.time_to_first_token`, and `gen_ai.usage.reasoning_tokens` are no longer emitted. Use `gen_ai.usage.cache_read.input_tokens` for provider-reported cache reads and `gen_ai.response.time_to_first_chunk` for standard streaming latency. The private `ttft_ms` Span attribute remains available for first-user-visible-output latency and continues driving `/stats`, `sampling_ms`, and output-token throughput; `gen_ai.response.time_to_first_chunk` is an independent standard attribute measuring first normalized chunk latency. The full version-pinned contract and deferred fields are documented in [GenAI and ARMS field alignment](../../design/gen-ai-arms-field-alignment.md).
|
|
|
|
To make ARMS recognize exported spans as a GenAI application, configure its resource feature explicitly:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"resourceAttributes": {
|
|
"acs.arms.service.feature": "genai_app"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Qwen Code does not inject this ARMS-specific resource attribute or `gen_ai.span.kind`. ARMS can infer LLM, Tool, and Agent roles from `gen_ai.operation.name`.
|
|
|
|
- `qwen-code.daemon.request`: Wraps a daemon HTTP request.
|
|
- **Attributes**: `http.request.method`, `http.route`, `qwen-code.daemon.operation`, `session.id`, `http.response.status_code`
|
|
|
|
- `qwen-code.daemon.bridge`: Wraps daemon bridge operations.
|
|
- **Attributes**: `qwen-code.daemon.operation`
|
|
|
|
#### Resource Metrics
|
|
|
|
- `qwen-code.memory.usage` (Histogram, bytes): Memory usage. Recorded by the memory-pressure monitor when telemetry is enabled.
|
|
- **Attributes**: `memory_type` (string: "heap_used"/"rss")
|
|
|
|
- `qwen-code.cpu.usage` (Histogram, percent): CPU usage percentage. Recorded by the memory-pressure monitor when telemetry is enabled.
|
|
- **Attributes**: (none)
|
|
|
|
### Performance Monitoring (Reserved)
|
|
|
|
The following metrics are defined but **not yet enabled in production**. They will be activated behind a dedicated performance monitoring config flag.
|
|
|
|
- `qwen-code.startup.duration` (Histogram, ms): CLI startup time by phase.
|
|
- **Attributes**: `phase` (string)
|
|
|
|
- `qwen-code.tool.queue.depth` (Histogram, count): Tools in execution queue.
|
|
|
|
- `qwen-code.tool.execution.breakdown` (Histogram, ms): Tool execution time by phase.
|
|
- **Attributes**: `function_name`, `phase` ("validation"/"preparation"/"execution"/"result_processing")
|
|
|
|
- `qwen-code.token.efficiency` (Histogram, ratio): Token efficiency metrics.
|
|
- **Attributes**: `model`, `metric`, `context` (optional)
|
|
|
|
- `qwen-code.performance.score` (Histogram, score): Composite performance score (0-100).
|
|
- **Attributes**: `category`, `baseline` (optional)
|
|
|
|
- `qwen-code.performance.regression` (Counter, Int): Regression detection events.
|
|
- **Attributes**: `metric`, `severity` ("low"/"medium"/"high"), `current_value`, `baseline_value`
|
|
|
|
- `qwen-code.performance.regression.percentage_change` (Histogram, percent): Percentage change vs baseline.
|
|
- **Attributes**: `metric`, `severity`, `current_value`, `baseline_value`
|
|
|
|
- `qwen-code.performance.baseline.comparison` (Histogram, percent): Performance vs baseline.
|
|
- **Attributes**: `metric`, `category`, `current_value`, `baseline_value`
|