ccfec2ef62
gpu-fleet.json references DCGM_FI_DEV_POWER_MGMT_LIMIT for its "TDP vs draw" panel, but the custom DCGM Exporter CSV did not declare it, so the panel silently rendered "No data" on clusters using that config. Declare the counter, fix the dashboards table in the gpu-operator examples README, and add a bats test that cross-checks every DCGM_FI_* reference in tracked dashboards and recording rules against the union of the upstream default set (snapshotted under hack/) and the project's custom CSV. Signed-off-by: Arsolitt <arsolitt@gmail.com>
6.2 KiB
6.2 KiB
| 1 | # Snapshot of the upstream DCGM Exporter default-counters.csv used as a |
|---|---|
| 2 | # fixture by hack/check-gpu-recording-rules.bats. The test verifies that |
| 3 | # every DCGM_FI_* metric referenced by a tracked GPU dashboard is either |
| 4 | # declared here (upstream defaults) or in |
| 5 | # packages/system/gpu-operator/examples/dcgm-custom-metrics.yaml |
| 6 | # (the project's custom CSV). |
| 7 | # |
| 8 | # Source: https://github.com/NVIDIA/dcgm-exporter/blob/4.1.1-4.0.4/etc/default-counters.csv |
| 9 | # Pinned to the DCGM Exporter image tag shipped by the gpu-operator |
| 10 | # chart under packages/system/gpu-operator/charts/gpu-operator/values.yaml |
| 11 | # (dcgmExporter.version = 4.1.1-4.0.4-ubuntu22.04). When that image is |
| 12 | # repo and re-run `./hack/cozytest.sh hack/check-gpu-recording-rules.bats`. |
| 13 | # Format |
| 14 | # If line starts with a '#' it is considered a comment |
| 15 | # Clocks |
| 16 | # Temperature |
| 17 | # Power |
| 18 | # PCIE |
| 19 | # Utilization (the sample period varies depending on the product) |
| 20 | # Errors and violations |
| 21 | # Memory usage |
| 22 | # ECC |
| 23 | # Retired pages |
| 24 | # NVLink |
| 25 | # VGPU License status |
| 26 | # Remapped rows |
| 27 | # Static configuration information. These appear as labels on the other metrics |
| 28 | # Datacenter Profiling (DCP) metrics |
| 29 | # NOTE: supported on Nvidia datacenter Volta GPUs and newer |