vrr/cozystack
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(monitoring): store pod:tensor_saturation as unitless ratio

Browse source 
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
This commit is contained in:
Arsolitt 2026-04-18 07:03:42 +03:00
parent ccfec2ef62
commit 2518e09d67
No known key found for this signature in database
GPG key ID: 4D8302CE6A9247C4
2 changed files with 12 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

4
dashboards/gpu/gpu-efficiency.json
View file

@ -35,7 +35,7 @@
"id": 2,
"targets": [
{
"expr": "avg(pod:tensor_saturation:avg5m{namespace=~\"$namespace\"})",
"expr": "avg(pod:tensor_saturation:avg5m{namespace=~\"$namespace\"}) * 100",
"refId": "A"
}
],
@ -363,7 +363,7 @@
"id": 21,
"targets": [
{
"expr": "topk(20, pod:tensor_saturation:avg5m)",
"expr": "topk(20, pod:tensor_saturation:avg5m * 100)",
"instant": true,
"range": false,
"format": "table",

16
packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml
View file

@ -64,20 +64,24 @@ spec:
rules:
# Tensor hardware saturation — the honest "am I using the GPU" metric
# for AI/LLM workloads. Unlike NVML, idle tensor cores are visible.
# Stored as a 0..1 ratio to stay consistent with namespace:tensor_active:avg
# and DCGM's native units. Consumers multiply by 100 at display time.
- record: pod:tensor_saturation:avg5m
expr: |
avg by (Hostname, gpu, UUID, namespace, pod) (
avg_over_time(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{namespace!="", namespace!~"cozy-.*|kube-.*"}[5m])
) * 100
)
# Power efficiency — utilization per watt, reveals unoptimized clients.
- record: pod:util_per_watt:avg5m
expr: |
avg_over_time(DCGM_FI_DEV_GPU_UTIL{namespace!="", namespace!~"cozy-.*|kube-.*"}[5m])
/
clamp_min(
avg_over_time(DCGM_FI_DEV_POWER_USAGE{namespace!="", namespace!~"cozy-.*|kube-.*"}[5m]),
1
avg by (Hostname, gpu, UUID, namespace, pod) (
avg_over_time(DCGM_FI_DEV_GPU_UTIL{namespace!="", namespace!~"cozy-.*|kube-.*"}[5m])
/
clamp_min(
avg_over_time(DCGM_FI_DEV_POWER_USAGE{namespace!="", namespace!~"cozy-.*|kube-.*"}[5m]),
1
)
)
# Fraction of time power-throttled (TDP cap) — 1.0 = fully throttled.