Address review feedback from coderabbitai on packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml:119
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback from coderabbitai on packages/system/gpu-operator/examples/nvidia-driver-compat.yaml:77
Signed-off-by: Arsolitt <arsolitt@gmail.com>
DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.
Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The /1e9 divisor in gpu:{power,thermal}_throttle_fraction:rate5m was
derived empirically against DCGM 3.x on A10 — the counter documents
itself as microseconds but ticks in nanoseconds in practice. If a
future exporter release honors the documented units, pre-clamp values
would exceed 1.0 while clamp_max(..., 1) silently masks the drift,
plateauing every throttle fraction at 100% and making the panels
lie in unison.
Add a validation group that fires when the raw max/1e9 value exceeds
1.0 for 15m, so we notice and rescale to /1e6 before dashboards
silently mislead operators.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
pod:util_per_watt:avg5m divided two DCGM metrics without an explicit
on(...) clause, so the match used the intersection of their label
sets. If dcgm-exporter relabeling ever diverges between
DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_POWER_USAGE (e.g. a pod-mapping
label appears on one but not the other after a config change), the
entire result drops to empty silently. Pin the match to the labels we
group by so divergence becomes a missing side, not a missing rule.
Throttle fractions had a related shape problem: dcgm-exporter emits
one series per GPU for each pod-mapping combination. On a shared GPU
(restart races, MIG/MPS) the same physical counter appears under
multiple pod labels and downstream avg(...) panels get diluted by the
pod count. Fold duplicates with max by (Hostname, gpu, UUID) before
clamp_max so the fraction is tied to the physical GPU.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
kube-state-metrics keeps kube_pod_container_resource_requests series
for Failed/Succeeded pods until the apiserver garbage-collects them,
which could inflate :allocated beyond what tenants actually hold and
drive cluster:gpu_count:free negative.
Join the request metric against kube_pod_status_phase filtered to
Pending|Running — the canonical pattern from Kubernetes' own
container_resource recording rules — on both the cluster and namespace
aggregates. Add clamp_min(..., 0) on cluster:gpu_count:free as a
second line of defence against transient label drift between
kube-state-metrics and DCGM.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Drop the hardcoded metadata.namespace so the rule inherits the chart's
release namespace, and add an explicit empty params field on every
group for schema consistency. No behavior change.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The Pending GPU pods counter on gpu-quotas joined raw
kube_pod_container_resource_requests (per-container series) against
kube_pod_status_phase (per-pod series). Multi-container pods were
counted once per requesting container instead of once per pod, so the
widget over-reported whenever a Pending pod had more than one GPU
container. Collapse the requests to pod level before the join.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Pod-level panels on the efficiency dashboard and DCGM-level panels on
the performance dashboard ignored the $namespace template variable, so
changing it left the visualizations unchanged. Add the filter to each
query. Performance-side queries use the `$namespace|` empty-tolerant
form so host-level DCGM series without a namespace label remain
visible when a specific namespace is selected.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Explain why tolerations: [{operator: Exists}] is safe on the driver
compat DaemonSet: the nodeSelector already confines scheduling to GPU
nodes, so the blanket toleration only kicks in when those nodes carry
the dedicated=gpu / nvidia.com/gpu taints that the GPU Operator's
default policy and many deployments apply.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Clarify that the "Average utilization" panel on gpu-fleet reflects the
legacy NVML view (DCGM_FI_DEV_GPU_UTIL) rather than engine-active
profiling. For AI/LLM workloads the NVML number is optimistic; the
gpu-efficiency dashboard carries the profiling-based view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
gpu-fleet.json references DCGM_FI_DEV_POWER_MGMT_LIMIT for its
"TDP vs draw" panel, but the custom DCGM Exporter CSV did not declare
it, so the panel silently rendered "No data" on clusters using that
config. Declare the counter, fix the dashboards table in the
gpu-operator examples README, and add a bats test that cross-checks
every DCGM_FI_* reference in tracked dashboards and recording rules
against the union of the upstream default set (snapshotted under
hack/) and the project's custom CSV.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- namespace:gpu_count:sum — never consumed by any tracked dashboard;
the billable view is already covered by namespace:gpu_count:allocated,
and the admin view by cluster:gpu_count:allocated.
- namespace:energy_joules:sum — no panel integrates joules; kWh
readings on the tenant dashboard compute their own integrations
from namespace:power_watts:sum.
- pod:tensor_to_nvml_ratio:avg5m — interesting tenant signal in
theory, but not wired into any panel and carrying it just burns
cardinality on large fleets.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-fleet: cluster-wide admin view — inventory, capacity (total /
allocated / free), per-node utilization and power, throttling,
temperatures, XID errors.
- gpu-tenants: per-namespace view — live allocation, utilization,
tensor saturation, power, and 24h GPU-hours / kWh integrations for
billing inputs.
Register both under gpu/* in dashboards-infra.list so they ship as
GrafanaDashboard CRs and fall under the bats cross-check introduced
earlier on this branch.
Update examples/README to spell out which DCGM counters each of the
five gpu/* dashboards actually needs on top of the upstream default
CSV — gpu-performance needs profiling and throttling counters,
gpu-efficiency needs profiling, gpu-tenants needs only
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet
and gpu-quotas work on the default counter set alone.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align namespace:gpu_count:allocated with every other namespace:* rule
by filtering out cozy-*/kube-*. All other per-namespace rules
(gpu_util, tensor_active, fb_used_bytes, power_watts) already exclude
system namespaces, so the label set produced by :allocated diverged
from them — any dashboard variable or join that reads across these
rules could end up with a different namespace list depending on which
rule supplied the :allocated column.
Trade-off: per-namespace GPU accounting for system workloads is no
longer available through this rule. If it's ever needed, add a
dedicated system:gpu_count:allocated rather than widening this one —
the "billable tenant view" invariant is what the filter is protecting.
Cluster-level cluster:gpu_count:allocated intentionally keeps system
pods so it stays aligned with cluster:gpu_count:total and
cluster:gpu_count:free remains meaningful. As a consequence,
sum(namespace:gpu_count:allocated) no longer equals
cluster:gpu_count:allocated; the delta is system-pod GPU usage, which
is fine for the cluster-admin view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Catch dangling references at PR time: every recording-rule name used
inside a tracked GPU dashboard must exist in
packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml. The
first iteration of gpu-efficiency.json shipped panels keyed on
pod:tensor_saturation:avg5m without the rule defined; the test fails
on exactly that class of bug.
Scoped to dashboards listed under gpu/* in dashboards-infra.list, so
untracked drafts stay out of scope until they are registered. Reverse
direction (rule defined but unused) is intentionally NOT enforced —
some rules exist for ad-hoc PromQL or upcoming dashboards.
Auto-discovered by make bats-unit-tests via hack/cozytest.sh.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The previous wording implied that the entire custom DCGM CSV was
required by the recording rules. In fact only the profiling counters
(DCGM_FI_PROF_*) need to be added on top of the upstream defaults —
everything else the rules consume is already in default-counters.csv.
Add a Verification status block flagging that the minimum-set claim is
derived from the DCGM Exporter version pinned in the currently shipped
gpu-operator package and must be re-checked when that package moves to
a newer release.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Restrict the nvidia-driver-compat DaemonSet to nodes labelled
nvidia.com/gpu.present=true (NFD/GPU Operator label). Without the
nodeSelector it was scheduling onto every node — control-plane and
CPU-only workers included — burning a privileged pod slot per host
for no benefit.
Add resource requests and limits to the init and pause containers so
the DaemonSet stays within control-plane budget on small clusters.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Drop gpu.recording.30s group: per-GPU 30s aggregates had no consumers
in tracked dashboards, only burned cardinality.
- Drop namespace:gpu_allocated_count:gauge: identical expression to
namespace:gpu_count:sum under a different name.
- Reground :allocated on kube_pod_container_resource_requests so it
reflects what tenants requested (Pending+Running) rather than what
DCGM currently sees. namespace:gpu_count:sum stays DCGM-based and
represents actually-running pods; the gap between the two is the
signal admins want.
- Add namespace:gpu_count:allocated as the per-namespace counterpart.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Strip Grafana export boilerplate (__inputs, __elements, __requires,
default annotations, embedded datasource inputs) and tighten panel
layouts across the three GPU dashboards. All three continue to use
the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Revise gpu-performance and add two new dashboards, registered in
dashboards-infra.list:
- gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity
and workload efficiency signals.
- gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs.
used GPUs for tenant capacity planning.
All three dashboards use the $ds_prometheus template variable, per
the project convention.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add reference manifests (not templates) under
packages/system/gpu-operator/examples/ documenting one working
configuration for running CUDA workloads directly in pods on a Talos
cluster, with DCGM metrics that drive the gpu/gpu-performance
dashboard.
- values-native-talos.yaml: Cozystack Package values that disable the
sandbox path, enable the device plugin, and wire DCGM to the custom
metrics ConfigMap.
- dcgm-custom-metrics.yaml: ConfigMap extending the default DCGM CSV
with profiling, ECC, throttling and energy counters used by the
dashboard and recording rules.
- nvidia-driver-compat.yaml: DaemonSet that stages libnvidia-ml.so.1
and nvidia-smi from the Talos glibc tree into a location the
gpu-operator validator inspects. Workaround for
NVIDIA/gpu-operator#1687.
- README.md: explains why these are shipped as references rather than
first-class templates (sandbox vs native is a deployment choice),
and how the pieces connect.
The out-of-the-box values-talos.yaml still targets the sandbox (VFIO
passthrough) scenario. Operators who want native pod GPU workloads can
start from these references.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add the gpu/gpu-performance dashboard and register it in the infra
dashboard list. The dashboard provides:
- Cluster overview: total/allocated GPUs, average utilization,
aggregate power draw.
- Utilization: GPU util (NVML), tensor pipe active (realistic load
for LLM/AI workloads), graphics engine active, memory copy util.
- Memory: VRAM used/free per GPU.
- Power and temperature per GPU.
- Health: XID errors, power and thermal throttling.
The dashboard relies on DCGM_FI_* metrics plus the cluster:gpu_* and
namespace:gpu_* recording rules added to monitoring-agents.
The JSON follows the cozystack convention — Prometheus data source is
selected via the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
