DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.
Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The Pending GPU pods counter on gpu-quotas joined raw
kube_pod_container_resource_requests (per-container series) against
kube_pod_status_phase (per-pod series). Multi-container pods were
counted once per requesting container instead of once per pod, so the
widget over-reported whenever a Pending pod had more than one GPU
container. Collapse the requests to pod level before the join.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Pod-level panels on the efficiency dashboard and DCGM-level panels on
the performance dashboard ignored the $namespace template variable, so
changing it left the visualizations unchanged. Add the filter to each
query. Performance-side queries use the `$namespace|` empty-tolerant
form so host-level DCGM series without a namespace label remain
visible when a specific namespace is selected.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Clarify that the "Average utilization" panel on gpu-fleet reflects the
legacy NVML view (DCGM_FI_DEV_GPU_UTIL) rather than engine-active
profiling. For AI/LLM workloads the NVML number is optimistic; the
gpu-efficiency dashboard carries the profiling-based view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-fleet: cluster-wide admin view — inventory, capacity (total /
allocated / free), per-node utilization and power, throttling,
temperatures, XID errors.
- gpu-tenants: per-namespace view — live allocation, utilization,
tensor saturation, power, and 24h GPU-hours / kWh integrations for
billing inputs.
Register both under gpu/* in dashboards-infra.list so they ship as
GrafanaDashboard CRs and fall under the bats cross-check introduced
earlier on this branch.
Update examples/README to spell out which DCGM counters each of the
five gpu/* dashboards actually needs on top of the upstream default
CSV — gpu-performance needs profiling and throttling counters,
gpu-efficiency needs profiling, gpu-tenants needs only
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet
and gpu-quotas work on the default counter set alone.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Strip Grafana export boilerplate (__inputs, __elements, __requires,
default annotations, embedded datasource inputs) and tighten panel
layouts across the three GPU dashboards. All three continue to use
the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Revise gpu-performance and add two new dashboards, registered in
dashboards-infra.list:
- gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity
and workload efficiency signals.
- gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs.
used GPUs for tenant capacity planning.
All three dashboards use the $ds_prometheus template variable, per
the project convention.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add the gpu/gpu-performance dashboard and register it in the infra
dashboard list. The dashboard provides:
- Cluster overview: total/allocated GPUs, average utilization,
aggregate power draw.
- Utilization: GPU util (NVML), tensor pipe active (realistic load
for LLM/AI workloads), graphics engine active, memory copy util.
- Memory: VRAM used/free per GPU.
- Power and temperature per GPU.
- Health: XID errors, power and thermal throttling.
The dashboard relies on DCGM_FI_* metrics plus the cluster:gpu_* and
namespace:gpu_* recording rules added to monitoring-agents.
The JSON follows the cozystack convention — Prometheus data source is
selected via the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
