vrr/cozystack
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(gpu-operator): clarify minimum required DCGM metrics

Browse source 
The previous wording implied that the entire custom DCGM CSV was
required by the recording rules. In fact only the profiling counters
(DCGM_FI_PROF_*) need to be added on top of the upstream defaults —
everything else the rules consume is already in default-counters.csv.

Add a Verification status block flagging that the minimum-set claim is
derived from the DCGM Exporter version pinned in the currently shipped
gpu-operator package and must be re-checked when that package moves to
a newer release.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
This commit is contained in:
Arsolitt 2026-04-18 06:15:58 +03:00
parent 0e20159bd9
commit 38c8a37cb5
No known key found for this signature in database
GPG key ID: 4D8302CE6A9247C4
1 changed files with 27 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

30
packages/system/gpu-operator/examples/README.md
View file

@ -21,9 +21,16 @@ Talos.
ConfigMap.
- [`dcgm-custom-metrics.yaml`](./dcgm-custom-metrics.yaml) — `ConfigMap`
with a DCGM metrics CSV that adds profiling, ECC, throttling and
energy counters on top of the upstream defaults. Required by the
recording rules in `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`
and by several panels in the `gpu/gpu-performance` dashboard.
energy counters on top of the upstream defaults. The CSV is the
superset needed for full dashboard coverage; the **recording rules
themselves** only require the profiling subset
(`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`, `DCGM_FI_PROF_GR_ENGINE_ACTIVE`)
on top of the upstream `default-counters.csv` — every other DCGM
series the rules consume (utilization, FB used/free, power,
temperature, energy) is already in the default set. The
`gpu/gpu-performance` dashboard additionally needs the throttle
counters (`DCGM_FI_DEV_POWER_VIOLATION`,
`DCGM_FI_DEV_THERMAL_VIOLATION`), which are not in the default set.
- [`nvidia-driver-compat.yaml`](./nvidia-driver-compat.yaml) — DaemonSet
that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc
tree into a path where the NVIDIA GPU Operator validator expects
@ -55,6 +62,23 @@ files into a directory the validator does inspect and creates the
[1]: https://github.com/NVIDIA/gpu-operator/issues/1687
## Verification status
> **Pending verification on an updated GPU Operator release.**
>
> The minimum-CSV claim above (only `DCGM_FI_PROF_*` is needed beyond
> the upstream default counters) is derived by cross-referencing
> `gpu-recording.rules.yaml` against the DCGM Exporter
> [`default-counters.csv`][default-csv] for the version pinned in the
> currently shipped `gpu-operator` package. The package in this branch
> is **not** the latest GPU Operator release; once we move to a newer
> version, the claim must be re-checked because the upstream default
> set occasionally adds or removes counters between releases. Until
> then, treat the CSV in `dcgm-custom-metrics.yaml` as a known-good
> superset rather than a minimal config.
[default-csv]: https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv
## How the dashboard and recording rules fit in
- `dashboards/gpu/gpu-performance.json` expects `DCGM_FI_*` metrics,