From dbed4992b08392c28bac8f0b5f917619b88cec65 Mon Sep 17 00:00:00 2001 From: Arsolitt Date: Sat, 18 Apr 2026 06:28:49 +0300 Subject: [PATCH] fix(monitoring): exclude system namespaces from namespace:gpu_count:allocated MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Align namespace:gpu_count:allocated with every other namespace:* rule by filtering out cozy-*/kube-*. All other per-namespace rules (gpu_util, tensor_active, fb_used_bytes, power_watts) already exclude system namespaces, so the label set produced by :allocated diverged from them — any dashboard variable or join that reads across these rules could end up with a different namespace list depending on which rule supplied the :allocated column. Trade-off: per-namespace GPU accounting for system workloads is no longer available through this rule. If it's ever needed, add a dedicated system:gpu_count:allocated rather than widening this one — the "billable tenant view" invariant is what the filter is protecting. Cluster-level cluster:gpu_count:allocated intentionally keeps system pods so it stays aligned with cluster:gpu_count:total and cluster:gpu_count:free remains meaningful. As a consequence, sum(namespace:gpu_count:allocated) no longer equals cluster:gpu_count:allocated; the delta is system-pod GPU usage, which is fine for the cluster-admin view. Signed-off-by: Arsolitt --- .../alerts/gpu-recording.rules.yaml | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml b/packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml index a54860b7..3a31f5a2 100644 --- a/packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml +++ b/packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml @@ -33,8 +33,27 @@ spec: # Kube-requested GPU count per namespace — billable view, includes # Pending pods. Use this for GPU-hour reporting via # sum_over_time(...[1h:1m])/60. + # + # Filters cozy-*/kube-* so the namespace set here matches every + # other namespace:* rule below. This keeps the $namespace variable + # in the tenant dashboard consistent with panels that read filtered + # rules — otherwise picking a system ns from the dropdown leaves + # half the panels empty while others show values. + # + # Trade-off: per-namespace GPU accounting for system workloads is + # no longer available here. If it's ever needed (e.g. operator + # pods pinning tenant GPUs), add a separate rule like + # system:gpu_count:allocated rather than widening this one — the + # "billable tenant view" invariant is relied on by tenants.json + # and keeping it narrow protects consumers from the asymmetry + # that triggered this filter in the first place. + # + # Note: sum(namespace:gpu_count:allocated) no longer equals + # cluster:gpu_count:allocated — the cluster-level rule intentionally + # keeps system pods to stay aligned with cluster:gpu_count:total + # when computing :free. - record: namespace:gpu_count:allocated - expr: sum by (namespace) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"}) + expr: sum by (namespace) (kube_pod_container_resource_requests{resource="nvidia_com_gpu", namespace!="", namespace!~"cozy-.*|kube-.*"}) - record: namespace:gpu_util:avg expr: avg by (namespace) (DCGM_FI_DEV_GPU_UTIL{namespace!="", namespace!~"cozy-.*|kube-.*"}) - record: namespace:tensor_active:avg