docs(gpu-operator): add native-pod Talos reference manifests
Add reference manifests (not templates) under packages/system/gpu-operator/examples/ documenting one working configuration for running CUDA workloads directly in pods on a Talos cluster, with DCGM metrics that drive the gpu/gpu-performance dashboard. - values-native-talos.yaml: Cozystack Package values that disable the sandbox path, enable the device plugin, and wire DCGM to the custom metrics ConfigMap. - dcgm-custom-metrics.yaml: ConfigMap extending the default DCGM CSV with profiling, ECC, throttling and energy counters used by the dashboard and recording rules. - nvidia-driver-compat.yaml: DaemonSet that stages libnvidia-ml.so.1 and nvidia-smi from the Talos glibc tree into a location the gpu-operator validator inspects. Workaround for NVIDIA/gpu-operator#1687. - README.md: explains why these are shipped as references rather than first-class templates (sandbox vs native is a deployment choice), and how the pieces connect. The out-of-the-box values-talos.yaml still targets the sandbox (VFIO passthrough) scenario. Operators who want native pod GPU workloads can start from these references. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com>
This commit is contained in:
Arsolitt
parent
d1d19e9978
commit
5d6654c6f4
4 changed files with 268 additions and 0 deletions
69
packages/system/gpu-operator/examples/README.md
Normal file
69
packages/system/gpu-operator/examples/README.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
# GPU operator — native pod workload on Talos (reference)
|
||||
|
||||
The files in this directory are **not** templates. They are reference
|
||||
artifacts that document one working configuration for running GPU
|
||||
workloads directly in pods on a Talos-based Cozystack cluster, together
|
||||
with the DCGM metrics needed by the `gpu/gpu-performance` Grafana
|
||||
dashboard.
|
||||
|
||||
The out-of-the-box `values-talos.yaml` for this package targets the
|
||||
sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here
|
||||
illustrate an alternative — running CUDA workloads in regular pods with
|
||||
the NVIDIA device plugin — and the workarounds it currently requires on
|
||||
Talos.
|
||||
|
||||
## Files
|
||||
|
||||
- [`values-native-talos.yaml`](./values-native-talos.yaml) — Cozystack
|
||||
`Package` values that disable sandbox workloads, enable the device
|
||||
plugin, point `hostPaths.driverInstallDir` at the staging location
|
||||
used by the compat DaemonSet, and wire DCGM to the custom metrics
|
||||
ConfigMap.
|
||||
- [`dcgm-custom-metrics.yaml`](./dcgm-custom-metrics.yaml) — `ConfigMap`
|
||||
with a DCGM metrics CSV that adds profiling, ECC, throttling and
|
||||
energy counters on top of the upstream defaults. Required by the
|
||||
recording rules in `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`
|
||||
and by several panels in the `gpu/gpu-performance` dashboard.
|
||||
- [`nvidia-driver-compat.yaml`](./nvidia-driver-compat.yaml) — DaemonSet
|
||||
that stages `libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc
|
||||
tree into a path where the NVIDIA GPU Operator validator expects
|
||||
them. See the "Why the compat DaemonSet exists" section below.
|
||||
|
||||
## Why these are reference, not templates
|
||||
|
||||
Shipping these as first-class templates would silently impose
|
||||
assumptions that do not hold for every user:
|
||||
|
||||
- Whether the NVIDIA Talos system extension is installed on the nodes.
|
||||
- Whether GPUs are exposed directly to pods or passed through to VMs.
|
||||
- The exact path the installed driver ends up at (depends on the
|
||||
extension version and Talos release).
|
||||
|
||||
The sandbox-oriented `values-talos.yaml` remains the default. Operators
|
||||
who want native pod GPU workloads can start from this directory and
|
||||
adapt as needed.
|
||||
|
||||
## Why the compat DaemonSet exists
|
||||
|
||||
The NVIDIA GPU Operator validator checks for `libnvidia-ml.so.1` and
|
||||
`bin/nvidia-smi` in the path given by `hostPaths.driverInstallDir`.
|
||||
Talos installs them under `/usr/local/glibc/usr/lib/` and
|
||||
`/usr/local/bin/`, which the validator does not look at. Until upstream
|
||||
addresses [NVIDIA/gpu-operator#1687][1], the DaemonSet copies those
|
||||
files into a directory the validator does inspect and creates the
|
||||
`.driver-ctr-ready` flag file so the validator proceeds.
|
||||
|
||||
[1]: https://github.com/NVIDIA/gpu-operator/issues/1687
|
||||
|
||||
## How the dashboard and recording rules fit in
|
||||
|
||||
- `dashboards/gpu/gpu-performance.json` expects `DCGM_FI_*` metrics,
|
||||
including profiling series (`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`,
|
||||
`DCGM_FI_PROF_GR_ENGINE_ACTIVE`) and throttling counters
|
||||
(`DCGM_FI_DEV_POWER_VIOLATION`, `DCGM_FI_DEV_THERMAL_VIOLATION`).
|
||||
These are only emitted when DCGM Exporter is started with the custom
|
||||
CSV in `dcgm-custom-metrics.yaml`.
|
||||
- `packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`
|
||||
precomputes cluster-wide and per-namespace aggregations used by the
|
||||
overview panels of the dashboard. The rules are safe to ship on any
|
||||
cluster — they evaluate to empty series when DCGM is not scraped.
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
# Custom DCGM Exporter metrics CSV. Referenced from
|
||||
# examples/values-native-talos.yaml via dcgmExporter.config.name.
|
||||
#
|
||||
# Extends the upstream default set with profiling counters, ECC, page
|
||||
# retirement, row remap, energy and throttling violations — everything
|
||||
# the gpu/gpu-performance dashboard and the GPU recording rules in
|
||||
# monitoring-agents expect.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: dcgm-custom-metrics
|
||||
namespace: cozy-gpu-operator
|
||||
data:
|
||||
dcgm-metrics.csv: |
|
||||
# Format
|
||||
# If line starts with a '#' it is considered a comment
|
||||
# DCGM FIELD, Prometheus metric type, help message
|
||||
|
||||
# Clocks
|
||||
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
|
||||
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
|
||||
|
||||
# Temperature
|
||||
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
|
||||
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
|
||||
|
||||
# Power
|
||||
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
|
||||
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).
|
||||
|
||||
# PCIE
|
||||
DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.
|
||||
|
||||
# Utilization (the sample period varies depending on the product)
|
||||
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
|
||||
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
|
||||
DCGM_FI_DEV_ENC_UTIL, gauge, Encoder utilization (in %).
|
||||
DCGM_FI_DEV_DEC_UTIL, gauge, Decoder utilization (in %).
|
||||
|
||||
# Errors and violations
|
||||
DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
|
||||
|
||||
# Memory usage
|
||||
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
|
||||
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
|
||||
DCGM_FI_DEV_FB_RESERVED, gauge, Framebuffer memory reserved (in MiB).
|
||||
|
||||
# ECC (supported on datacenter-class GPUs such as A10)
|
||||
DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, Total number of single-bit volatile ECC errors.
|
||||
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total number of double-bit volatile ECC errors.
|
||||
DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, counter, Total number of single-bit persistent ECC errors.
|
||||
DCGM_FI_DEV_ECC_DBE_AGG_TOTAL, counter, Total number of double-bit persistent ECC errors.
|
||||
|
||||
# Retired pages
|
||||
DCGM_FI_DEV_RETIRED_SBE, counter, Total number of retired pages due to single-bit errors.
|
||||
DCGM_FI_DEV_RETIRED_DBE, counter, Total number of retired pages due to double-bit errors.
|
||||
DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.
|
||||
|
||||
# Row remapping (not applicable to GDDR6 but left for datacenter GPUs)
|
||||
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors.
|
||||
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for correctable errors.
|
||||
DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, Whether remapping of rows has failed.
|
||||
|
||||
# Throttle / violation counters (crucial for SLA and tenant monitoring)
|
||||
DCGM_FI_DEV_POWER_VIOLATION, counter, Throttling duration due to power constraints (in us).
|
||||
DCGM_FI_DEV_THERMAL_VIOLATION, counter, Throttling duration due to thermal constraints (in us).
|
||||
DCGM_FI_DEV_SYNC_BOOST_VIOLATION, counter, Throttling duration due to sync-boost constraints (in us).
|
||||
DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in us).
|
||||
DCGM_FI_DEV_LOW_UTIL_VIOLATION, counter, Throttling duration due to low utilization (in us).
|
||||
DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in us).
|
||||
|
||||
# NVLink — enable only for GPUs that actually have NVLink (A10 has none).
|
||||
# DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, counter, Total number of NVLink bandwidth counters for all lanes.
|
||||
|
||||
# DCP (profiling) metrics — supported from Ampere onwards.
|
||||
DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, Ratio of time the graphics engine is active.
|
||||
DCGM_FI_PROF_SM_ACTIVE, gauge, The ratio of cycles an SM has at least 1 warp assigned.
|
||||
DCGM_FI_PROF_SM_OCCUPANCY, gauge, The ratio of number of warps resident on an SM.
|
||||
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
|
||||
DCGM_FI_PROF_DRAM_ACTIVE, gauge, Ratio of cycles the device memory interface is active sending or receiving data.
|
||||
DCGM_FI_PROF_PCIE_TX_BYTES, counter, The number of bytes of active PCIe tx (transmit) data including both header and payload.
|
||||
DCGM_FI_PROF_PCIE_RX_BYTES, counter, The number of bytes of active PCIe rx (read) data including both header and payload.
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
# Workaround for https://github.com/NVIDIA/gpu-operator/issues/1687
|
||||
#
|
||||
# On Talos, the NVIDIA system extension installs driver files under
|
||||
# /usr/local/glibc/usr/lib/ and /usr/local/bin/. The GPU Operator
|
||||
# validator only looks for libnvidia-ml.so.1 and bin/nvidia-smi under
|
||||
# the path configured as hostPaths.driverInstallDir. This DaemonSet
|
||||
# stages the required files into /var/nvidia-driver on each node and
|
||||
# creates the .driver-ctr-ready flag so the validator proceeds.
|
||||
#
|
||||
# Paired with examples/values-native-talos.yaml, which sets
|
||||
# hostPaths.driverInstallDir to /var/nvidia-driver.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: nvidia-driver-compat
|
||||
namespace: cozy-gpu-operator
|
||||
labels:
|
||||
app: nvidia-driver-compat
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nvidia-driver-compat
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nvidia-driver-compat
|
||||
spec:
|
||||
hostPID: true
|
||||
priorityClassName: system-node-critical
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
initContainers:
|
||||
- name: create-driver-tree
|
||||
image: busybox:1.37
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
set -e
|
||||
GLIBC_LIB="/host/usr/local/glibc/usr/lib"
|
||||
SRC_BIN="/host/usr/local/bin"
|
||||
DST="/host/var/nvidia-driver"
|
||||
|
||||
mkdir -p "$DST/bin"
|
||||
|
||||
if [ -f "$GLIBC_LIB/libnvidia-ml.so.1" ]; then
|
||||
cp -f "$GLIBC_LIB/libnvidia-ml.so.1" "$DST/libnvidia-ml.so.1"
|
||||
echo "Copied libnvidia-ml.so.1"
|
||||
fi
|
||||
|
||||
if [ -f "$SRC_BIN/nvidia-smi" ]; then
|
||||
cp -f "$SRC_BIN/nvidia-smi" "$DST/bin/nvidia-smi"
|
||||
chmod +x "$DST/bin/nvidia-smi"
|
||||
echo "Copied nvidia-smi"
|
||||
fi
|
||||
|
||||
mkdir -p /host/run/nvidia/validations
|
||||
touch /host/run/nvidia/validations/.driver-ctr-ready
|
||||
echo "Created driver-ctr-ready flag"
|
||||
echo "Done"
|
||||
securityContext:
|
||||
privileged: true
|
||||
volumeMounts:
|
||||
- name: host-root
|
||||
mountPath: /host
|
||||
containers:
|
||||
- name: pause
|
||||
image: registry.k8s.io/pause:3.10
|
||||
volumes:
|
||||
- name: host-root
|
||||
hostPath:
|
||||
path: /
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
# Cozystack Package values for running GPU workloads natively in pods
|
||||
# on Talos. This is the counterpart to values-talos.yaml, which targets
|
||||
# the sandbox (VFIO passthrough) scenario.
|
||||
#
|
||||
# Prerequisites:
|
||||
# - NVIDIA Talos system extension installed on GPU nodes.
|
||||
# - examples/nvidia-driver-compat.yaml deployed to stage driver files
|
||||
# where the gpu-operator validator looks for them.
|
||||
# - examples/dcgm-custom-metrics.yaml applied so DCGM Exporter exports
|
||||
# the full set of metrics used by the dashboard and recording rules.
|
||||
apiVersion: cozystack.io/v1alpha1
|
||||
kind: Package
|
||||
metadata:
|
||||
name: cozystack.gpu-operator
|
||||
spec:
|
||||
variant: default
|
||||
components:
|
||||
gpu-operator:
|
||||
values:
|
||||
gpu-operator:
|
||||
# The compat DaemonSet stages libnvidia-ml.so.1 and nvidia-smi
|
||||
# under /var/nvidia-driver. Point the validator at that path.
|
||||
hostPaths:
|
||||
driverInstallDir: "/var/nvidia-driver"
|
||||
# Disable the sandbox path — workloads run in pods, not VMs.
|
||||
sandboxWorkloads:
|
||||
enabled: false
|
||||
vfioManager:
|
||||
enabled: false
|
||||
# The Talos extension provides the driver and runtime hooks,
|
||||
# so the operator's own toolkit and driver components must be
|
||||
# switched off to avoid conflicting installations.
|
||||
toolkit:
|
||||
enabled: false
|
||||
devicePlugin:
|
||||
enabled: true
|
||||
# Export full set of DCGM metrics using the ConfigMap in
|
||||
# examples/dcgm-custom-metrics.yaml.
|
||||
dcgmExporter:
|
||||
serviceMonitor:
|
||||
enabled: true
|
||||
interval: "15s"
|
||||
honorLabels: true
|
||||
config:
|
||||
name: dcgm-custom-metrics
|
||||
Loading…
Add table
Add a link
Reference in a new issue