mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-07-19 22:14:08 +00:00
988a5021e8
* docs(daemon): update developer docs for recent daemon PRs - Add Last-Event-ID client reconnect guide (10-event-bus, 13-sdk-daemon-client) - Add cross-connection vote routing section (04-permission-mediation) - Add new capability tags: daemon_status, workspace_permissions, workspace_trust, workspace_github_setup, workspace_voice, workspace_voice_transcription, voice_transcribe (11-capabilities-versioning) - Add new event types: trust_change_requested, github_setup_completed, extensions_changed, mid_turn_message_injected (09-event-schema) - Fix _meta.serverTimestamp source description (09-event-schema, 10-event-bus) - Fix async function* syntax in SDK example (13-sdk-daemon-client) - Sync event/capability counts across all docs (43->47 events, 67->75 tags) * docs(daemon): add workspace remember design doc (PR #5884) Design document for the sessionless workspace remember API proposed in PR #5884. Covers API endpoints, task lifecycle, implementation details, events, error handling, and SDK integration. Status: Proposed (not yet merged). * docs(telemetry): comprehensive update to match current implementation Added 34 undocumented events, 17 metrics, 11 daemon metrics, 2 spans. Fixed diff_stat attribute schema (was documented as JSON string, actually individual attributes). Added Performance Monitoring reserved section. Standardized attribute annotations with type and optionality markers. * fix(telemetry): extract EVENT_TOOL_OUTPUT_TRUNCATED constant and add qwen-code prefix Extracted hardcoded event name to constant for consistency with other telemetry events. Added standard qwen-code. namespace prefix. Updated test assertion to match the new prefixed event name. * qwen: address PR review feedback (#5960) * qwen: address PR review feedback (#5960) * qwen: address PR review feedback (#5960) * qwen: address PR review feedback (#5960) * qwen: address PR review feedback (#5960) * qwen: address PR review feedback (#5960) * docs: address PR review feedback (#5960) * docs: address PR review feedback (#5960)
881 lines
47 KiB
Markdown
881 lines
47 KiB
Markdown
# Observability with OpenTelemetry
|
|
|
|
Learn how to enable and setup OpenTelemetry for Qwen Code.
|
|
|
|
- [Observability with OpenTelemetry](#observability-with-opentelemetry)
|
|
- [Key Benefits](#key-benefits)
|
|
- [OpenTelemetry Integration](#opentelemetry-integration)
|
|
- [Configuration](#configuration)
|
|
- [Aliyun Telemetry](#aliyun-telemetry)
|
|
- [Manual OTLP Export](#manual-otlp-export)
|
|
- [Local Telemetry](#local-telemetry)
|
|
- [File-based Output (Recommended)](#file-based-output-recommended)
|
|
- [Collector-Based Export (Advanced)](#collector-based-export-advanced)
|
|
- [Logs and Metrics](#logs-and-metrics)
|
|
- [Logs](#logs)
|
|
- [Metrics](#metrics)
|
|
- [Daemon Metrics](#daemon-metrics)
|
|
- [Spans](#spans)
|
|
- [Resource Metrics](#resource-metrics)
|
|
- [Performance Monitoring (Reserved)](#performance-monitoring-reserved)
|
|
|
|
## Migration Notes
|
|
|
|
- `tool_output_truncated` was renamed to `qwen-code.tool_output_truncated` for namespace consistency — downstream consumers filtering on the old name should update their queries.
|
|
|
|
- The `tool.call.latency` histogram documentation previously listed a `decision` attribute — this was never set on the histogram (only `function_name` is recorded). The `tool.call.count` counter continues to include `decision`.
|
|
|
|
- The `qwen-code.file_operation` log event and `file.operation.count` metric documentation previously listed diff-stat attributes (`model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`) — these were never set on either. Diff-stat data is available via the `tool_call` log event's `metadata` attribute.
|
|
|
|
## Key Benefits
|
|
|
|
- **🔍 Usage Analytics**: Understand interaction patterns and feature adoption
|
|
across your team
|
|
- **⚡ Performance Monitoring**: Track response times, token consumption, and
|
|
resource utilization
|
|
- **🐛 Real-time Debugging**: Identify bottlenecks, failures, and error patterns
|
|
as they occur
|
|
- **📊 Workflow Optimization**: Make informed decisions to improve
|
|
configurations and processes
|
|
- **🏢 Enterprise Governance**: Monitor usage across teams, track costs, ensure
|
|
compliance, and integrate with existing monitoring infrastructure
|
|
|
|
## OpenTelemetry Integration
|
|
|
|
Built on **[OpenTelemetry]** — the vendor-neutral, industry-standard
|
|
observability framework — Qwen Code's observability system provides:
|
|
|
|
- **Universal Compatibility**: Export to any OpenTelemetry backend (Aliyun,
|
|
Jaeger, Prometheus, Datadog, etc.)
|
|
- **Standardized Data**: Use consistent formats and collection methods across
|
|
your toolchain
|
|
- **Future-Proof Integration**: Connect with existing and future observability
|
|
infrastructure
|
|
- **No Vendor Lock-in**: Switch between backends without changing your
|
|
instrumentation
|
|
|
|
[OpenTelemetry]: https://opentelemetry.io/
|
|
[aliyun-opentelemetry-overview]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/product-overview/what-is-tracing-analysis
|
|
[aliyun-opentelemetry-get-started]: https://www.alibabacloud.com/help/en/arms/tracing-analysis/before-you-begin
|
|
[aliyun-opentelemetry-console-cn]: https://trace.console.aliyun.com
|
|
[aliyun-opentelemetry-console-cn-legacy]: https://tracing.console.aliyun.com
|
|
[aliyun-opentelemetry-console-intl]: https://arms.console.alibabacloud.com
|
|
|
|
## Configuration
|
|
|
|
All telemetry behavior is controlled through your `.qwen/settings.json` file.
|
|
These settings can be overridden by environment variables or CLI flags.
|
|
|
|
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
|
|
| --------------------------------- | ---------------------------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------- |
|
|
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
|
|
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing — set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
|
|
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
|
|
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
|
|
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
|
|
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
|
|
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
|
|
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
|
|
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
|
|
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
|
|
| `sensitiveSpanAttributeMaxLength` | `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | - | Maximum JavaScript string length for each sensitive native span attribute content payload. Set lower if your backend rejects large attributes. | `1..104857600` | `1048576` |
|
|
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
|
|
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
|
|
|
|
**Note on boolean environment variables:** For the boolean settings (`enabled`,
|
|
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
|
|
corresponding environment variable to `true` or `1` will enable the feature. Any
|
|
other value will disable it.
|
|
|
|
**Note on integer environment variables:** `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH`
|
|
must be a positive integer when set. Invalid values fail telemetry configuration
|
|
resolution instead of silently falling back.
|
|
|
|
**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
|
|
two things happen:
|
|
|
|
1. **Native span attributes (`qwen-code.interaction`, `api.generateContent*`,
|
|
`tool.<name>`)** carry verbatim conversation content:
|
|
- User prompts (`new_context`)
|
|
- System prompts (`system_prompt` — full text once per session, deduped by
|
|
SHA-256 hash; subsequent spans only carry `system_prompt_hash` +
|
|
`system_prompt_preview` + `system_prompt_length`)
|
|
- Tool schemas (emitted as `tool_schema` events, also hash-deduped)
|
|
- Tool inputs (`tool_input`) and tool results (`tool_result`)
|
|
- Model output (`response.model_output`)
|
|
|
|
Each content payload is truncated at `sensitiveSpanAttributeMaxLength`
|
|
JavaScript string units. The default is 1 MiB (`1048576`), raised from the
|
|
previous 60 KiB default; set `61440` to preserve the old cap. The limit
|
|
must be between `1` and `104857600` (100 MiB). For labeled attributes, fixed
|
|
labels such as `[USER PROMPT]`, `[TOOL INPUT: ...]`, and
|
|
`[TOOL RESULT: ...]` count against the cap; the truncation marker also counts
|
|
against it. The limit is measured as JavaScript string length rather than
|
|
UTF-8 bytes. Non-ASCII content can therefore occupy more bytes after OTLP
|
|
export. For most payload types, truncation adds both `*_truncated` and
|
|
`*_original_length`. System prompts also set `system_prompt_truncated` when
|
|
truncated, but use the always-present `system_prompt_length` for the original
|
|
length.
|
|
|
|
2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
|
|
logs endpoint) keep their existing `prompt`, `function_args`, and
|
|
`response_text` fields, instead of being dropped.
|
|
|
|
⚠️ **Security warning:** enabling this flag streams full conversation history,
|
|
file contents read by `read_file`, shell commands and their output (including
|
|
secrets in env vars or arguments), and model responses to the configured OTLP
|
|
backend. Treat the backend as a privileged data sink. The flag defaults to
|
|
`false`.
|
|
|
|
**Cost / payload size:** A heavy turn at the default limit (1 MiB system prompt
|
|
plus 10 tool calls, each up to 1 MiB input + 1 MiB result, plus 1 MiB model
|
|
output) can produce up to ~22 MiB of attribute payload before OTLP compression,
|
|
plus up to 1 MiB per emitted tool schema in workspaces with large tool
|
|
definitions. This is Qwen Code's application-side cap, not a guarantee that
|
|
every collector or backend accepts a single attribute that large. If spans are
|
|
rejected or dropped, lower `sensitiveSpanAttributeMaxLength` (for example, to
|
|
`61440`) and monitor exporter throughput.
|
|
|
|
This setting does not disable sensitive data in OTel logs or other telemetry
|
|
sinks; non-internal API response telemetry can populate `response_text`, so
|
|
OTel logs, UI telemetry, and chat recording may receive response text
|
|
independently of this setting. QwenLogger does not include `response_text`.
|
|
|
|
**HTTP OTLP signal routing:** When using HTTP protocol (`otlpProtocol: "http"`),
|
|
Qwen Code automatically appends signal-specific paths (`/v1/traces`, `/v1/logs`,
|
|
`/v1/metrics`) to the base `otlpEndpoint`. For example, `http://collector:4318`
|
|
becomes `http://collector:4318/v1/traces` for traces. If the URL already ends
|
|
with a signal path, it is used as-is. Per-signal endpoint overrides
|
|
(`otlpTracesEndpoint`, etc.) take precedence over the base endpoint and are used
|
|
verbatim. gRPC protocol uses service-based routing and does not append paths.
|
|
|
|
The per-signal endpoint environment variables also accept the standard
|
|
OpenTelemetry names: `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
|
|
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
|
|
The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
|
|
|
|
For detailed information about all configuration options, see the
|
|
[Configuration Guide](../../users/configuration/settings.md).
|
|
|
|
### Resource attributes
|
|
|
|
Resource attributes are static key-value pairs attached to every span, log,
|
|
and metric exported via OTLP. Use them to slice telemetry by team, environment,
|
|
deployment region, or any other dimension your backend cares about.
|
|
|
|
Two sources, merged in priority order (lowest → highest):
|
|
|
|
1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
|
|
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
|
|
key conflict)
|
|
|
|
`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides
|
|
`service.name` from any other source (per the OpenTelemetry spec).
|
|
|
|
#### Examples
|
|
|
|
**Slice all telemetry by team / environment:**
|
|
|
|
```bash
|
|
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
|
|
```
|
|
|
|
**Route to a per-tenant collector via `service.name`:**
|
|
|
|
```bash
|
|
export OTEL_SERVICE_NAME=qwen-code-ci
|
|
```
|
|
|
|
**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"resourceAttributes": {
|
|
"deployment.environment": "production",
|
|
"service.namespace": "engineering-tooling"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
```bash
|
|
# Add a one-off tag without touching settings:
|
|
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
|
|
```
|
|
|
|
#### Reserved keys
|
|
|
|
Some keys are runtime-controlled and cannot be overridden:
|
|
|
|
- `service.version` — always set to the running CLI version. Setting it from
|
|
any source is silently dropped with a warning.
|
|
- `session.id` — runtime-injected per session. User-provided values from
|
|
either env or settings are dropped with a warning. The reason is that
|
|
Resource attributes auto-attach to every metric data point; allowing user
|
|
override would bypass [Cardinality controls](#cardinality-controls) below.
|
|
Spans and logs always carry `session.id`.
|
|
|
|
`service.name` is **not** reserved; it follows the precedence chain above.
|
|
|
|
#### Format
|
|
|
|
`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
|
|
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
|
|
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
|
|
the wrong boundary and the second half is dropped as malformed). Malformed
|
|
pairs are skipped with a warning rather than failing telemetry startup.
|
|
|
|
#### Troubleshooting: when a user-provided attribute appears not to take effect
|
|
|
|
Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
|
|
settings values, and invalid percent-encoding are all silently dropped with a
|
|
warning logged via the OpenTelemetry diagnostics channel. That channel routes
|
|
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
|
|
behavior can look like silent failure.
|
|
|
|
If a custom resource attribute isn't appearing on exported telemetry:
|
|
|
|
1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
|
|
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
|
|
(non-string settings value).
|
|
2. Verify the env var is set in the qwen-code process's environment (not just
|
|
your shell) and that values are percent-encoded.
|
|
3. Confirm `telemetry.enabled` is `true` — telemetry init only runs if enabled.
|
|
|
|
### Cardinality controls
|
|
|
|
Metrics are aggregated by attribute set at the backend — every distinct
|
|
combination of attribute values produces a new time series. Attaching a
|
|
high-cardinality field like `session.id` to a metric causes time-series fan-out
|
|
proportional to the number of sessions, which quickly exhausts metric backend
|
|
storage.
|
|
|
|
To prevent this, Qwen Code keeps high-cardinality attributes off metric data
|
|
points by default. Spans and logs are per-event and unaffected, so they
|
|
continue to carry `session.id` for trace and log correlation.
|
|
|
|
#### `telemetry.metrics.includeSessionId` (default: `false`)
|
|
|
|
Setting this to `true` (via settings or
|
|
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
|
|
every metric data point.
|
|
|
|
⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a
|
|
fleet will blow up metric storage. Recommended only for short-term debugging.
|
|
For long-term session correlation, query trace or log backends instead.
|
|
|
|
#### Migration from earlier versions
|
|
|
|
Prior to this release, `session.id` was attached to metrics by default. If
|
|
your Prometheus queries / Grafana dashboards / alert rules reference
|
|
`session_id` on a metric, you have two options:
|
|
|
|
**Option A** — restore the previous behavior for short-term debugging:
|
|
|
|
```bash
|
|
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
|
|
```
|
|
|
|
or:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"metrics": { "includeSessionId": true }
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B (recommended)** — move session-level analysis off metrics. Spans
|
|
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
|
|
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
|
|
cardinality pressure.
|
|
|
|
### Client-side HTTP span on outbound fetch
|
|
|
|
When telemetry is enabled, Qwen Code registers `UndiciInstrumentation`
|
|
which creates a client-side HTTP span for every outbound `fetch()`
|
|
request originated by the process — including the LLM SDKs (`openai`,
|
|
`@google/genai`, `@anthropic-ai/sdk`), the MCP StreamableHTTP client, the
|
|
`WebFetch` tool, and any IDE-extension out-of-process calls. The span
|
|
lets you see network latency (TTFB / response body transfer) separately
|
|
from upstream model processing time, which the existing
|
|
`api.generateContent` span alone can't distinguish.
|
|
|
|
These spans go to your **own** OTLP collector (or file outfile) just like
|
|
the rest of the telemetry — they do not affect what is written onto the
|
|
outbound HTTP request itself. Whether the W3C `traceparent` header is
|
|
also written into the outgoing request stream is controlled by a
|
|
**separate, security-relevant setting** documented in
|
|
[outbound correlation](#outbound-correlation-security-relevant) below.
|
|
|
|
**Feedback-loop avoidance.** OTel SDK uses `fetch` internally to upload OTLP
|
|
data. Without protection, instrumenting `fetch` would trace those uploads,
|
|
which would themselves be uploaded, causing an infinite loop. Qwen Code's
|
|
undici instrumentation is configured with an `ignoreRequestHook` that skips
|
|
URLs matching the configured `telemetry.otlpEndpoint` /
|
|
`telemetry.otlpTracesEndpoint` / `telemetry.otlpLogsEndpoint` /
|
|
`telemetry.otlpMetricsEndpoint` prefixes. In file-outfile mode there are no
|
|
outbound HTTP uploads, so the hook is a no-op.
|
|
|
|
## Outbound correlation (SECURITY-RELEVANT)
|
|
|
|
These settings live in a **separate top-level namespace** from `telemetry.*`
|
|
on purpose: telemetry controls data flow into the operator's own
|
|
observability backend, while `outboundCorrelation.*` controls what
|
|
client-side correlation data qwen-code writes **into outbound LLM API
|
|
request streams** that reach third-party LLM provider endpoints
|
|
(DashScope, OpenAI, Anthropic, etc.). Different recipients, different
|
|
consent decision. **All values default to off.** See PR #4390 review
|
|
discussion for the framing rationale.
|
|
|
|
### `outboundCorrelation.propagateTraceContext`
|
|
|
|
```jsonc
|
|
"outboundCorrelation": {
|
|
"propagateTraceContext": false // default
|
|
}
|
|
```
|
|
|
|
When `false` (default), Qwen Code installs a no-op `TextMapPropagator` on
|
|
the OTel SDK. UndiciInstrumentation still creates client HTTP spans for
|
|
your OTLP collector, but `propagation.inject()` is a no-op so **no
|
|
`traceparent` is written onto outbound requests**. Trace IDs stay
|
|
internal to the operator's collector.
|
|
|
|
When `true`, the SDK's default W3C composite propagator
|
|
(`tracecontext` + `baggage`) is installed and the standard `traceparent`
|
|
header is written on every outbound `fetch`:
|
|
|
|
```
|
|
traceparent: 00-<32-hex traceId>-<16-hex parentSpanId>-<01-sampled | 00-not-sampled>
|
|
```
|
|
|
|
Additionally, `TRACEPARENT` and `TRACESTATE` environment variables are set in
|
|
shell child processes (Bash tool, hooks, monitor) so that spawned commands can
|
|
participate in the same distributed trace.
|
|
|
|
Opt in only when the LLM provider also reports into your OTel collector
|
|
for cross-process trace stitching — e.g. ARMS Tracing serving DashScope.
|
|
For most operators the value is `false`; cross-vendor trace continuation
|
|
is niche.
|
|
|
|
**Depends on `telemetry.enabled: true`.** The OTel SDK only initializes
|
|
when telemetry is enabled, so `propagateTraceContext` only takes effect
|
|
in that state. Setting it to `true` while telemetry is disabled is a
|
|
silent no-op — no SDK, no propagator, no `traceparent` on the wire.
|
|
Verify both flags when wiring an ARMS+DashScope correlation setup:
|
|
|
|
```jsonc
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"otlpTracesEndpoint": "http://tracing-analysis-...",
|
|
},
|
|
"outboundCorrelation": {
|
|
"propagateTraceContext": true,
|
|
},
|
|
}
|
|
```
|
|
|
|
### Other outbound correlation headers
|
|
|
|
`X-Qwen-Code-Session-Id` and `X-Qwen-Code-Request-Id` are **not part of
|
|
this PR**. They will be designed and proposed in their own follow-up
|
|
PR(s) under the same `outboundCorrelation.*` namespace, each with its
|
|
own threat model and operator-consent flow. PR #4390 review (LaZzyMan)
|
|
established the principle: "telemetry's scope of work doesn't include
|
|
sending identifiers to LLM providers"; correlation-header work moves to
|
|
its own design discussion rather than landing under telemetry.
|
|
|
|
## Aliyun Telemetry
|
|
|
|
### Manual OTLP Export
|
|
|
|
To view Qwen Code telemetry in Alibaba Cloud Managed Service for
|
|
OpenTelemetry, configure Qwen Code to export to the OTLP endpoint
|
|
provided by ARMS.
|
|
|
|
Setting `"target": "gcp"` alone does not configure the export
|
|
destination. If `otlpEndpoint` is not set, Qwen Code still defaults to
|
|
`http://localhost:4317`. If `outfile` is set, it overrides
|
|
`otlpEndpoint` and telemetry is written to the file instead of being
|
|
sent to Alibaba Cloud.
|
|
|
|
1. Enable telemetry in your `.qwen/settings.json` and set the OTLP
|
|
endpoint:
|
|
|
|
**Option A: gRPC protocol** (standard OTLP endpoint):
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"target": "gcp",
|
|
"otlpEndpoint": "https://<your-otlp-endpoint>",
|
|
"otlpProtocol": "grpc"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B: HTTP protocol with per-signal endpoints** (for backends
|
|
that use non-standard paths, e.g., `/api/otlp/traces` instead of
|
|
`/v1/traces`):
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"otlpProtocol": "http",
|
|
"otlpTracesEndpoint": "http://<host>/<token>/api/otlp/traces",
|
|
"otlpLogsEndpoint": "http://<host>/<token>/api/otlp/logs",
|
|
"otlpMetricsEndpoint": "http://<host>/<token>/api/otlp/metrics"
|
|
}
|
|
}
|
|
```
|
|
|
|
> **Note:** When using HTTP protocol with only `otlpEndpoint` (no
|
|
> per-signal overrides), Qwen Code appends standard OTLP paths
|
|
> (`/v1/traces`, `/v1/logs`, `/v1/metrics`) to the base URL. If your
|
|
> backend uses different paths, use per-signal endpoint overrides as
|
|
> shown in Option B.
|
|
|
|
2. If your Alibaba Cloud endpoint requires authentication, provide OTLP
|
|
headers through standard OpenTelemetry environment variables such as
|
|
`OTEL_EXPORTER_OTLP_HEADERS` (or the signal-specific variants). Qwen
|
|
Code does not currently expose OTLP auth headers directly in
|
|
`.qwen/settings.json`.
|
|
3. Run Qwen Code and send prompts.
|
|
4. View telemetry in Managed Service for OpenTelemetry:
|
|
- Product overview:
|
|
[What is Managed Service for OpenTelemetry?][aliyun-opentelemetry-overview]
|
|
- Getting started:
|
|
[Get started with Managed Service for OpenTelemetry][aliyun-opentelemetry-get-started]
|
|
- Console entry points:
|
|
- China mainland:
|
|
[trace.console.aliyun.com][aliyun-opentelemetry-console-cn]
|
|
(legacy console:
|
|
[tracing.console.aliyun.com][aliyun-opentelemetry-console-cn-legacy])
|
|
- International:
|
|
[arms.console.alibabacloud.com][aliyun-opentelemetry-console-intl]
|
|
- In the console, use `Applications` to inspect traces and service
|
|
topology.
|
|
- To locate the OTLP endpoint and access information:
|
|
- **New console** (`trace.console.aliyun.com` or international):
|
|
navigate to `Integration Center`.
|
|
- **Legacy console** (`tracing.console.aliyun.com`): navigate to
|
|
`Cluster Configurations` → `Access point information`.
|
|
|
|
## Local Telemetry
|
|
|
|
For local development and debugging, you can capture telemetry data locally:
|
|
|
|
### File-based Output (Recommended)
|
|
|
|
1. Enable telemetry in your `.qwen/settings.json`:
|
|
|
|
```json
|
|
{
|
|
"telemetry": {
|
|
"enabled": true,
|
|
"outfile": ".qwen/telemetry.log"
|
|
}
|
|
}
|
|
```
|
|
|
|
> **Note:** When `outfile` is set, OTLP export is automatically disabled.
|
|
> The `target` and `otlpEndpoint` settings are not needed for file-only
|
|
> output and can be safely omitted from your config.
|
|
|
|
2. Run Qwen Code and send prompts.
|
|
3. View logs and metrics in the specified file (e.g., `.qwen/telemetry.log`).
|
|
|
|
### Collector-Based Export (Advanced)
|
|
|
|
1. Run the automation script:
|
|
```bash
|
|
npm run telemetry -- --target=local
|
|
```
|
|
This will:
|
|
- Download and start Jaeger and OTEL collector
|
|
- Configure your workspace for local telemetry
|
|
- Provide a Jaeger UI at http://localhost:16686
|
|
- Save logs/metrics to `~/.qwen/tmp/<projectHash>/otel/collector.log`
|
|
- Stop collector on exit (e.g. `Ctrl+C`)
|
|
2. Run Qwen Code and send prompts.
|
|
3. View traces at http://localhost:16686 and logs/metrics in the collector log
|
|
file.
|
|
|
|
## Logs and Metrics
|
|
|
|
The following section describes the structure of logs, metrics, and spans
|
|
generated for Qwen Code.
|
|
|
|
- A `sessionId` is included as a common attribute on all logs and metrics.
|
|
|
|
### Logs
|
|
|
|
Logs are timestamped records of specific events. All log records automatically include `event.name` and `event.timestamp` attributes.
|
|
|
|
The following events are logged:
|
|
|
|
#### Core Session Events
|
|
|
|
- `qwen-code.config`: Emitted once at startup with CLI configuration.
|
|
- **Attributes**: `model`, `sandbox_enabled`, `core_tools_enabled`, `approval_mode`, `file_filtering_respect_git_ignore`, `debug_mode`, `truncate_tool_output_threshold`, `truncate_tool_output_lines`, `hooks` (comma-separated, omitted if disabled), `ide_enabled`, `interactive_shell_enabled`, `mcp_servers`, `mcp_servers_count`, `mcp_tools`, `mcp_tools_count`, `output_format`, `skills`, `subagents`
|
|
|
|
- `qwen-code.user_prompt`: User submits a prompt.
|
|
- **Attributes**: `prompt_length` (int), `prompt_id` (string), `prompt` (string, excluded if `log_prompts_enabled` is false), `auth_type` (string)
|
|
|
|
- `qwen-code.user_retry`: User retries the last prompt.
|
|
- **Attributes**: `prompt_id` (string)
|
|
|
|
- `qwen-code.conversation_finished`: A conversation turn sequence completes.
|
|
- **Attributes**: `approvalMode` (string), `turnCount` (int)
|
|
|
|
- `qwen-code.user_feedback`: User submits session feedback.
|
|
- **Attributes**: `session_id` (string), `rating` (int: 1=bad, 2=fine, 3=good), `model` (string), `approval_mode` (string), `prompt_id` (string, optional)
|
|
|
|
#### Tool Events
|
|
|
|
- `qwen-code.tool_call`: Each function/tool call.
|
|
- **Attributes**: `function_name` (string), `function_args` (object), `duration_ms` (int), `status` (string: "success", "error", or "cancelled"), `success` (boolean), `decision` (string: "accept", "reject", "auto_accept", or "modify", optional), `error` (string, optional), `error_type` (string, optional), `prompt_id` (string), `response_id` (string, optional), `content_length` (int, optional), `tool_type` (string: "native" or "mcp"), `mcp_server_name` (string, optional), `metadata` (object, optional — for file-writing tools contains `model_added_lines`, `model_removed_lines`, `user_added_lines`, `user_removed_lines`, `model_added_chars`, `model_removed_chars`, `user_added_chars`, `user_removed_chars`)
|
|
|
|
- `qwen-code.file_operation`: Each file operation.
|
|
- **Attributes**: `tool_name` (string), `operation` (string: "create", "read", "update"), `lines` (int, optional), `mimetype` (string, optional), `extension` (string, optional), `programming_language` (string, optional)
|
|
|
|
- `qwen-code.tool_output_truncated`: Tool output exceeded size threshold.
|
|
- **Attributes**: `tool_name` (string), `original_content_length` (int), `truncated_content_length` (int), `threshold` (int), `lines` (int), `prompt_id` (string)
|
|
|
|
#### API Events
|
|
|
|
- `qwen-code.api_request`: Outgoing request to the LLM API.
|
|
- **Attributes**: `model` (string), `prompt_id` (string), `request_text` (string, optional), `subagent_name` (string, optional)
|
|
|
|
- `qwen-code.api_response`: Response received from LLM API.
|
|
- **Attributes**: `response_id` (string), `model` (string), `status_code` (int/string, optional), `duration_ms` (int), `input_token_count` (int), `output_token_count` (int), `cached_content_token_count` (int), `thoughts_token_count` (int), `total_token_count` (int), `prompt_id` (string), `auth_type` (string, optional), `response_text` (string, optional), `subagent_name` (string, optional)
|
|
|
|
- `qwen-code.api_error`: API request failed.
|
|
- **Attributes**: `model` (string), `prompt_id` (string), `duration_ms` (int), `error_message` (string), `response_id` (string, optional), `auth_type` (string, optional), `error_type` (string, optional), `status_code` (int/string, optional), `subagent_name` (string, optional)
|
|
|
|
Additionally, OTel-standard aliases (`http.status_code`, `error.message`, `model_name`, `duration`) are emitted for compatibility.
|
|
|
|
- `qwen-code.api_cancel`: API request cancelled by user.
|
|
- **Attributes**: `model` (string), `prompt_id` (string), `auth_type` (string, optional), `loop_wakeups_cancelled` (int, optional)
|
|
|
|
- `qwen-code.api_retry`: HTTP-status retry (429/5xx) at an LLM call site. Distinct from `chat.content_retry` which handles `InvalidStreamError` retries on a separate budget.
|
|
- **Attributes**: `model` (string), `prompt_id` (string, optional), `attempt_number` (int), `error_type` (string, optional), `error_message` (string), `status_code` (int/string, optional), `retry_delay_ms` (int), `duration_ms` (int, equals retry_delay_ms — backoff sleep, not HTTP round-trip; for attempt duration see the qwen-code.llm_request span), `subagent_name` (string, optional)
|
|
|
|
- `qwen-code.malformed_json_response`: `generateJson` response couldn't be parsed.
|
|
- **Attributes**: `model` (string)
|
|
|
|
- `qwen-code.flash_fallback`: Switched to flash model as fallback.
|
|
- **Attributes**: `auth_type` (string)
|
|
|
|
- `qwen-code.ripgrep_fallback`: Switched to grep as fallback.
|
|
- **Attributes**: `use_ripgrep` (boolean), `use_builtin_ripgrep` (boolean), `error` (string, optional)
|
|
|
|
#### Resilience Events
|
|
|
|
- `qwen-code.chat.content_retry`: Content-error retry (e.g. empty stream).
|
|
- **Attributes**: `attempt_number` (int), `error_type` (string), `retry_delay_ms` (int), `model` (string)
|
|
|
|
- `qwen-code.chat.content_retry_failure`: All content retries exhausted.
|
|
- **Attributes**: `total_attempts` (int), `final_error_type` (string), `total_duration_ms` (int, optional), `model` (string)
|
|
|
|
- `qwen-code.chat.invalid_chunk`: Invalid chunk received from stream.
|
|
- **Attributes**: `error.message` (string, optional)
|
|
|
|
#### Command & Extension Events
|
|
|
|
- `qwen-code.slash_command`: User executes a slash command.
|
|
- **Attributes**: `command` (string), `subcommand` (string, optional), `status` (string: "success" or "error", optional)
|
|
|
|
- `qwen-code.slash_command.model`: User switches model via `/model` command.
|
|
- **Attributes**: `model_name` (string)
|
|
|
|
- `qwen-code.skill_launch`: A skill is launched.
|
|
- **Attributes**: `skill_name` (string), `success` (boolean), `prompt_id` (string)
|
|
|
|
- `qwen-code.extension_install`: Extension installed.
|
|
- **Attributes**: `extension_name` (string), `extension_version` (string), `extension_source` (string), `status` (string: "success"/"error")
|
|
|
|
- `qwen-code.extension_uninstall`: Extension uninstalled.
|
|
- **Attributes**: `extension_name` (string), `status` (string)
|
|
|
|
- `qwen-code.extension_enable`: Extension enabled.
|
|
- **Attributes**: `extension_name` (string), `setting_scope` (string)
|
|
|
|
- `qwen-code.extension_disable`: Extension disabled.
|
|
- **Attributes**: `extension_name` (string), `setting_scope` (string)
|
|
|
|
- `qwen-code.extension_update`: Extension updated.
|
|
- **Attributes**: `extension_name` (string), `extension_id` (string), `extension_previous_version` (string), `extension_version` (string), `extension_source` (string), `status` (string: "success"/"error")
|
|
|
|
- `qwen-code.ide_connection`: IDE connection event.
|
|
- **Attributes**: `connection_type` (string: "start" or "session")
|
|
|
|
- `qwen-code.auth`: Authentication event.
|
|
- **Attributes**: `auth_type` (string), `action_type` ("auto", "manual", "coding-plan"), `status` ("success", "error", "cancelled"), `error_message` (optional)
|
|
|
|
#### Subagent Events
|
|
|
|
- `qwen-code.subagent_execution`: Subagent lifecycle event.
|
|
- **Attributes**: `subagent_name` (string), `status` ("started", "completed", "failed", "cancelled"), `terminate_reason` (optional), `result` (optional), `execution_summary` (optional)
|
|
|
|
#### Arena Events
|
|
|
|
- `qwen-code.arena_session_started`: Arena session begins.
|
|
- **Attributes**: `arena_session_id` (string), `model_ids` (JSON string array), `task_length` (int)
|
|
|
|
- `qwen-code.arena_agent_completed`: An arena agent finishes.
|
|
- **Attributes**: `arena_session_id` (string), `agent_session_id` (string), `agent_model_id` (string), `status` (string: "completed"/"failed"/"cancelled"), `duration_ms` (int), `rounds` (int), `total_tokens` (int), `input_tokens` (int), `output_tokens` (int), `tool_calls` (int), `successful_tool_calls` (int), `failed_tool_calls` (int)
|
|
|
|
- `qwen-code.arena_session_ended`: Arena session completes.
|
|
- **Attributes**: `arena_session_id` (string), `status` (string: "selected"/"discarded"/"failed"/"cancelled"), `duration_ms` (int), `display_backend` (string, optional), `agent_count` (int), `completed_agents` (int), `failed_agents` (int), `cancelled_agents` (int), `winner_model_id` (string, optional)
|
|
|
|
#### Workflow Events
|
|
|
|
- `qwen-code.workflow_keyword`: Workflow keyword trigger fired.
|
|
|
|
- `qwen-code.workflow_run`: Workflow run reached terminal state.
|
|
- **Attributes**: `status` (string), `agents_dispatched` (int), `agents_completed` (int), `phase_count` (int), `tokens_spent` (int), `duration_ms` (int)
|
|
|
|
#### Auto-Memory Events
|
|
|
|
- `qwen-code.memory.extract`: Memory extraction run completed.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status` ("completed"/"skipped"/"failed"), `skipped_reason` (optional), `patches_count` (int), `touched_topics` (string), `duration_ms` (int)
|
|
|
|
- `qwen-code.memory.dream`: Memory consolidation (dream) run completed.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status` ("updated"/"noop"/"failed"/"cancelled"), `deduped_entries` (int), `touched_topics_count` (int), `touched_topics` (string), `duration_ms` (int)
|
|
|
|
- `qwen-code.memory.recall`: Memory recall operation completed.
|
|
- **Attributes**: `query_length` (int), `docs_scanned` (int), `docs_selected` (int), `strategy` ("none"/"heuristic"/"model"), `duration_ms` (int)
|
|
|
|
#### Prompt Suggestion & Speculation Events
|
|
|
|
- `qwen-code.prompt_suggestion`: Prompt suggestion outcome.
|
|
- **Attributes**: `outcome` ("accepted"/"ignored"/"suppressed"), `prompt_id` (optional), `accept_method` ("tab"/"enter"/"right", optional), `accept_source` ("live"/"fallback", optional), `time_to_accept_ms` (optional), `time_to_ignore_ms` (optional), `time_to_first_keystroke_ms` (optional), `suggestion_length` (optional), `similarity` (optional), `was_focused_when_shown` (optional), `reason` (optional)
|
|
|
|
- `qwen-code.speculation`: Speculative execution outcome.
|
|
- **Attributes**: `outcome` ("accepted"/"aborted"/"failed"), `turns_used` (int), `files_written` (int), `tool_use_count` (int), `duration_ms` (int), `boundary_type` (optional), `had_pipelined_suggestion` (boolean)
|
|
|
|
#### Other Events
|
|
|
|
- `qwen-code.chat_compression`: Chat context compressed.
|
|
- **Attributes**: `tokens_before` (int), `tokens_after` (int), `compression_input_token_count` (int, optional), `compression_output_token_count` (int, optional)
|
|
|
|
- `qwen-code.next_speaker_check`: Next speaker determination.
|
|
- **Attributes**: `prompt_id` (string), `finish_reason` (string), `result` (string)
|
|
|
|
- `loop_detected`: Loop detected during agent execution. _(Note: emitted without `qwen-code.` prefix — pre-existing inconsistency.)_
|
|
- **Attributes**: `loop_type` (string), `prompt_id` (string)
|
|
|
|
- `kitty_sequence_overflow`: Kitty graphics protocol sequence exceeded buffer size. _(Note: emitted without `qwen-code.` prefix — pre-existing inconsistency.)_
|
|
- **Attributes**: `sequence_length` (int), `truncated_sequence` (string, first 20 chars)
|
|
|
|
### Metrics
|
|
|
|
Metrics are numerical measurements of behavior over time. Metric names use the `qwen-code.*` prefix.
|
|
|
|
#### Core Metrics
|
|
|
|
- `qwen-code.session.count` (Counter, Int): Incremented once per CLI startup.
|
|
|
|
- `qwen-code.tool.call.count` (Counter, Int): Counts tool calls.
|
|
- **Attributes**: `function_name`, `success` (boolean), `decision` ("accept"/"reject"/"auto_accept"/"modify", optional), `tool_type` ("mcp"/"native", optional)
|
|
|
|
- `qwen-code.tool.call.latency` (Histogram, ms): Measures tool call latency.
|
|
- **Attributes**: `function_name` (string)
|
|
|
|
- `qwen-code.api.request.count` (Counter, Int): Counts all API requests.
|
|
- **Attributes**: `model`, `status_code`, `error_type` (optional)
|
|
|
|
- `qwen-code.api.request.latency` (Histogram, ms): Measures API request latency.
|
|
- **Attributes**: `model` (string)
|
|
|
|
- `qwen-code.token.usage` (Counter, Int): Counts tokens used.
|
|
- **Attributes**: `model`, `type` ("input"/"output"/"thought"/"cache")
|
|
|
|
- `qwen-code.file.operation.count` (Counter, Int): Counts file operations.
|
|
- **Attributes**: `operation` ("create"/"read"/"update"), `lines` (optional), `mimetype` (optional), `extension` (optional), `programming_language` (optional)
|
|
|
|
- `qwen-code.chat_compression` (Counter, Int): Counts chat compression operations.
|
|
- **Attributes**: `tokens_before` (int), `tokens_after` (int)
|
|
|
|
- `qwen-code.slash_command.model.call_count` (Counter, Int): Counts model slash command calls.
|
|
- **Attributes**: `slash_command.model.model_name` (string)
|
|
|
|
- `qwen-code.subagent.execution.count` (Counter, Int): Counts subagent execution events.
|
|
- **Attributes**: `subagent_name`, `status` ("started"/"completed"/"failed"/"cancelled"), `terminate_reason` (optional)
|
|
|
|
#### Resilience Metrics
|
|
|
|
- `qwen-code.api.retry.count` (Counter, Int): HTTP-status retries (429/5xx) at LLM call sites.
|
|
- **Attributes**: `model` (string)
|
|
|
|
- `qwen-code.chat.content_retry.count` (Counter, Int): Retries due to content errors.
|
|
|
|
- `qwen-code.chat.content_retry_failure.count` (Counter, Int): All content retries exhausted.
|
|
|
|
- `qwen-code.chat.invalid_chunk.count` (Counter, Int): Invalid chunks from stream.
|
|
|
|
#### Arena Metrics
|
|
|
|
- `qwen-code.arena.session.count` (Counter, Int): Arena sessions by status.
|
|
- **Attributes**: `status`, `display_backend` (optional)
|
|
|
|
- `qwen-code.arena.session.duration` (Histogram, ms): Arena session duration.
|
|
- **Attributes**: `status`
|
|
|
|
- `qwen-code.arena.agent.count` (Counter, Int): Arena agent completions.
|
|
- **Attributes**: `status`, `model_id`
|
|
|
|
- `qwen-code.arena.agent.duration` (Histogram, ms): Arena agent execution duration.
|
|
- **Attributes**: `model_id`
|
|
|
|
- `qwen-code.arena.agent.tokens` (Counter, Int): Token usage by arena agents.
|
|
- **Attributes**: `model_id`, `type` ("input"/"output")
|
|
|
|
- `qwen-code.arena.result.selected` (Counter, Int): Arena result selections.
|
|
- **Attributes**: `model_id`
|
|
|
|
#### Auto-Memory Metrics
|
|
|
|
- `qwen-code.memory.extract.count` (Counter, Int): Auto-memory extraction runs.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status`
|
|
|
|
- `qwen-code.memory.extract.duration` (Histogram, ms): Extraction duration.
|
|
- **Attributes**: `trigger`, `status`
|
|
|
|
- `qwen-code.memory.dream.count` (Counter, Int): Auto-memory dream runs.
|
|
- **Attributes**: `trigger` ("auto"/"manual"), `status`
|
|
|
|
- `qwen-code.memory.dream.duration` (Histogram, ms): Dream run duration.
|
|
- **Attributes**: `trigger`, `status`
|
|
|
|
- `qwen-code.memory.recall.count` (Counter, Int): Auto-memory recall operations.
|
|
- **Attributes**: `strategy` ("none"/"heuristic"/"model")
|
|
|
|
- `qwen-code.memory.recall.duration` (Histogram, ms): Recall duration.
|
|
- **Attributes**: `strategy`
|
|
|
|
#### API Request Breakdown
|
|
|
|
- `qwen-code.api.request.breakdown` (Histogram, ms): API request time breakdown by phase.
|
|
- **Attributes**: `model`, `phase` ("request_preparation"/"network_latency"/"response_processing"/"token_processing")
|
|
|
|
### Daemon Metrics
|
|
|
|
The daemon process (long-running HTTP server mode) exposes its own metrics.
|
|
|
|
> **Note:** The three Observable Gauges (`daemon.session.active`, `daemon.sse.active`, `daemon.process.heap_used`) are callback-based metrics updated at each collection interval; `registerDaemonGaugeCallbacks()` must be invoked during daemon initialization to register the observation callbacks.
|
|
|
|
#### HTTP
|
|
|
|
- `qwen-code.daemon.http.request.count` (Counter, Int): Request count by route and status class.
|
|
- **Attributes**: `route`, `status_class` ("2xx"/"4xx"/"5xx")
|
|
|
|
- `qwen-code.daemon.http.request.duration` (Histogram, ms): Request duration.
|
|
- **Attributes**: `route`
|
|
- **Buckets**: 1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000
|
|
|
|
#### Sessions
|
|
|
|
- `qwen-code.daemon.session.active` (ObservableGauge, Int): Current active sessions.
|
|
|
|
- `qwen-code.daemon.session.lifecycle` (Counter, Int): Session lifecycle events.
|
|
- **Attributes**: `action` ("spawn"/"close"/"die")
|
|
|
|
#### Channels
|
|
|
|
- `qwen-code.daemon.channel.lifecycle` (Counter, Int): ACP channel lifecycle events.
|
|
- **Attributes**: `action` ("spawn"/"exit"), `expected` (boolean, optional)
|
|
|
|
#### Prompts
|
|
|
|
- `qwen-code.daemon.prompt.queue_wait` (Histogram, ms): Prompt FIFO queue wait time.
|
|
- **Buckets**: 1, 5, 10, 50, 100, 500, 1000, 5000, 10000, 30000, 60000
|
|
|
|
- `qwen-code.daemon.prompt.duration` (Histogram, ms): End-to-end prompt duration.
|
|
- **Buckets**: 100, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000
|
|
|
|
#### Errors
|
|
|
|
- `qwen-code.daemon.bridge.error.count` (Counter, Int): Bridge errors by type.
|
|
- **Attributes**: `error_type` (known class name or "unknown")
|
|
|
|
- `qwen-code.daemon.cancel.count` (Counter, Int): Cancel request count.
|
|
|
|
#### Resources
|
|
|
|
- `qwen-code.daemon.sse.active` (ObservableGauge, Int): Active SSE connections.
|
|
|
|
- `qwen-code.daemon.process.heap_used` (ObservableGauge, Int, bytes): Heap memory usage.
|
|
|
|
### Spans
|
|
|
|
Distributed tracing spans form a tree rooted at `qwen-code.interaction`. Each interaction is a trace root with its own `traceId`; cross-prompt correlation uses the `session.id` attribute.
|
|
|
|
- `qwen-code.interaction`: Root span for each user prompt turn.
|
|
- **Attributes**: `session.id`, `qwen-code.prompt_id`, `qwen-code.message_type`, `qwen-code.model`, `qwen-code.approval_mode`, `interaction.sequence`, `interaction.duration_ms`, `qwen-code.turn_status` ("ok"/"error"/"cancelled")
|
|
|
|
- `qwen-code.llm_request`: Wraps a single LLM API call.
|
|
- **Attributes**: `session.id`, `qwen-code.model`, `qwen-code.prompt_id`, `llm_request.context` ("subagent"/"interaction"/"standalone"), `gen_ai.request.model`, `duration_ms`, `input_tokens`, `output_tokens`, `cached_input_tokens`, `ttft_ms`, `request_setup_ms`, `attempt`, `retry_total_delay_ms`, `sampling_ms`, `output_tokens_per_second`, `success`, `error`, `response_id`, `finish_reason`, `thoughts_token_count`, `subagent_name`, `error_type`, `error_status_code`
|
|
|
|
- `qwen-code.tool`: Wraps the full tool lifecycle (approval wait + execution).
|
|
- **Attributes**: `session.id`, `tool.name`, `duration_ms`, `success`, `error`
|
|
|
|
- `qwen-code.tool.execution`: Wraps the tool execution phase (after approval).
|
|
- **Attributes**: `session.id`, `duration_ms`, `success`, `error`
|
|
|
|
- `qwen-code.tool.blocked_on_user`: Time a tool spends waiting on user approval.
|
|
- **Attributes**: `session.id`, `tool.name`, `tool.call_id`, `duration_ms`, `decision` ("proceed_once"/"proceed_always"/"cancel"/"aborted"/"auto_approved"/"error"), `source` ("cli"/"ide"/"hook"/"auto"/"system")
|
|
|
|
- `qwen-code.hook`: Wraps each pre/post-tool-use hook fire site.
|
|
- **Attributes**: `session.id`, `hook_event` ("PreToolUse"/"PostToolUse"/"PostToolUseFailure"/"PostToolBatch"), `tool.name`, `tool.use_id` (optional), `is_interrupt` (boolean, optional), `duration_ms`, `success`, `should_proceed` (optional), `should_stop` (optional), `block_type` (optional), `error` (optional)
|
|
|
|
- `qwen-code.subagent`: Wraps a single subagent invocation.
|
|
- **Attributes**: `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.conversation.id`, `qwen-code.subagent.id`, `qwen-code.subagent.name`, `qwen-code.subagent.invocation_kind` ("foreground"/"fork"/"background"), `qwen-code.subagent.is_built_in`, `qwen-code.subagent.depth`, `qwen-code.subagent.status`, `qwen-code.subagent.terminate_reason`, `qwen-code.subagent.duration_ms`
|
|
|
|
- `qwen-code.daemon.request`: Wraps a daemon HTTP request.
|
|
- **Attributes**: `http.request.method`, `http.route`, `qwen-code.daemon.operation`, `session.id`, `http.response.status_code`
|
|
|
|
- `qwen-code.daemon.bridge`: Wraps daemon bridge operations.
|
|
- **Attributes**: `qwen-code.daemon.operation`
|
|
|
|
#### Resource Metrics
|
|
|
|
- `qwen-code.memory.usage` (Histogram, bytes): Memory usage. Recorded by the memory-pressure monitor when telemetry is enabled.
|
|
- **Attributes**: `memory_type` (string: "heap_used"/"rss")
|
|
|
|
- `qwen-code.cpu.usage` (Histogram, percent): CPU usage percentage. Recorded by the memory-pressure monitor when telemetry is enabled.
|
|
- **Attributes**: (none)
|
|
|
|
### Performance Monitoring (Reserved)
|
|
|
|
The following metrics are defined but **not yet enabled in production**. They will be activated behind a dedicated performance monitoring config flag.
|
|
|
|
- `qwen-code.startup.duration` (Histogram, ms): CLI startup time by phase.
|
|
- **Attributes**: `phase` (string)
|
|
|
|
- `qwen-code.tool.queue.depth` (Histogram, count): Tools in execution queue.
|
|
|
|
- `qwen-code.tool.execution.breakdown` (Histogram, ms): Tool execution time by phase.
|
|
- **Attributes**: `function_name`, `phase` ("validation"/"preparation"/"execution"/"result_processing")
|
|
|
|
- `qwen-code.token.efficiency` (Histogram, ratio): Token efficiency metrics.
|
|
- **Attributes**: `model`, `metric`, `context` (optional)
|
|
|
|
- `qwen-code.performance.score` (Histogram, score): Composite performance score (0-100).
|
|
- **Attributes**: `category`, `baseline` (optional)
|
|
|
|
- `qwen-code.performance.regression` (Counter, Int): Regression detection events.
|
|
- **Attributes**: `metric`, `severity` ("low"/"medium"/"high"), `current_value`, `baseline_value`
|
|
|
|
- `qwen-code.performance.regression.percentage_change` (Histogram, percent): Percentage change vs baseline.
|
|
- **Attributes**: `metric`, `severity`, `current_value`, `baseline_value`
|
|
|
|
- `qwen-code.performance.baseline.comparison` (Histogram, percent): Performance vs baseline.
|
|
- **Attributes**: `metric`, `category`, `current_value`, `baseline_value`
|