From 38c8a37cb585ff235c7d90d0824a457e7da93095 Mon Sep 17 00:00:00 2001 From: Arsolitt Date: Sat, 18 Apr 2026 06:15:58 +0300 Subject: [PATCH] docs(gpu-operator): clarify minimum required DCGM metrics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous wording implied that the entire custom DCGM CSV was required by the recording rules. In fact only the profiling counters (DCGM_FI_PROF_*) need to be added on top of the upstream defaults — everything else the rules consume is already in default-counters.csv. Add a Verification status block flagging that the minimum-set claim is derived from the DCGM Exporter version pinned in the currently shipped gpu-operator package and must be re-checked when that package moves to a newer release. Signed-off-by: Arsolitt --- .../system/gpu-operator/examples/README.md | 30 +++++++++++++++++-- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/packages/system/gpu-operator/examples/README.md b/packages/system/gpu-operator/examples/README.md index dac3703e..11c2bb09 100644 --- a/packages/system/gpu-operator/examples/README.md +++ b/packages/system/gpu-operator/examples/README.md @@ -21,9 +21,16 @@ Talos. ConfigMap. - [`dcgm-custom-metrics.yaml`](./dcgm-custom-metrics.yaml) — `ConfigMap` with a DCGM metrics CSV that adds profiling, ECC, throttling and - energy counters on top of the upstream defaults. Required by the - recording rules in `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml` - and by several panels in the `gpu/gpu-performance` dashboard. + energy counters on top of the upstream defaults. The CSV is the + superset needed for full dashboard coverage; the **recording rules + themselves** only require the profiling subset + (`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`, `DCGM_FI_PROF_GR_ENGINE_ACTIVE`) + on top of the upstream `default-counters.csv` — every other DCGM + series the rules consume (utilization, FB used/free, power, + temperature, energy) is already in the default set. The + `gpu/gpu-performance` dashboard additionally needs the throttle + counters (`DCGM_FI_DEV_POWER_VIOLATION`, + `DCGM_FI_DEV_THERMAL_VIOLATION`), which are not in the default set. - [`nvidia-driver-compat.yaml`](./nvidia-driver-compat.yaml) — DaemonSet that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a path where the NVIDIA GPU Operator validator expects @@ -55,6 +62,23 @@ files into a directory the validator does inspect and creates the [1]: https://github.com/NVIDIA/gpu-operator/issues/1687 +## Verification status + +> **Pending verification on an updated GPU Operator release.** +> +> The minimum-CSV claim above (only `DCGM_FI_PROF_*` is needed beyond +> the upstream default counters) is derived by cross-referencing +> `gpu-recording.rules.yaml` against the DCGM Exporter +> [`default-counters.csv`][default-csv] for the version pinned in the +> currently shipped `gpu-operator` package. The package in this branch +> is **not** the latest GPU Operator release; once we move to a newer +> version, the claim must be re-checked because the upstream default +> set occasionally adds or removes counters between releases. Until +> then, treat the CSV in `dcgm-custom-metrics.yaml` as a known-good +> superset rather than a minimal config. + +[default-csv]: https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/default-counters.csv + ## How the dashboard and recording rules fit in - `dashboards/gpu/gpu-performance.json` expects `DCGM_FI_*` metrics,