## What this PR does
Documentation: https://github.com/cozystack/website/pull/517
Integrates [HAMi](https://github.com/Project-HAMi/HAMi) v2.8.1 (CNCF
Sandbox) into Cozystack as a system-level package for fractional GPU
sharing in tenant Kubernetes clusters.
The integration covers three layers:
- **System chart** (`packages/system/hami/`): Vendors upstream HAMi Helm
chart with device plugin, scheduler extender, mutating webhook, and
RuntimeClass configuration. The broken DRA subchart was removed — it
renders resources even when disabled and references unpublished images.
- **Kubernetes app addon** (`packages/apps/kubernetes/`): HAMi exposed
as an optional toggle (`hami.enabled`). Automatically disables GPU
Operator's native device plugin when active to avoid conflicts. Enforces
hard dependency on GPU Operator.
- **Platform registration** (`packages/core/platform/`): HAMi declared
as PackageSource with gpu-operator dependency, included in the iaas
bundle.
**Known limitation**: HAMi-core relies on a private glibc symbol
(`_dl_sym`) removed in glibc 2.34, which breaks compute isolation on
modern container images (Ubuntu 22.04+, PyTorch/TensorFlow official
images). Alpine/musl is entirely incompatible. See the package README
for details and upstream issue references.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(hami): add HAMi GPU virtualization as an optional system package for fractional GPU sharing. Enables memory and compute isolation for NVIDIA GPUs across tenant workloads. Integrates with GPU Operator and can be enabled per-cluster via the hami.enabled toggle. Note: compute isolation requires glibc < 2.34 in workload containers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added HAMi GPU virtualization middleware as an optional cluster addon
to enable fractional GPU sharing.
* HAMi addon includes an enable toggle (default: disabled) and a
customizable Helm values override; when enabled it enforces GPU Operator
presence and adjusts GPU Operator values accordingly.
* New HelmRelease integration for deploying HAMi with dependency
ordering and conditional rendering.
* **Documentation**
* Added HAMi setup guide, compatibility notes, and updated cluster addon
configuration reference.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
In environments where control-plane nodes are not available (e.g., AWS
managed nodes, Cozy-in-Cozy tenant Kubernetes clusters), the DaemonSet
cannot schedule any pods because no nodes match this selector.
## What this PR does
This PR adds ability to replace DaemonSet by Deployment with custom
number of replicas for HA purpose. Default values saves current
behavior.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Optional Deployment mode for lineage-controller-webhook with
configurable replicas
* PodDisruptionBudget support when Deployment mode is enabled
* **Updates**
* Improved pod scheduling: node affinity replaces nodeSelector, optional
tolerations, and pod anti-affinity when using Deployment
* Clarified behavior for local API endpoint scheduling (only applies
when pod lands on nodes running an API server)
* **Documentation**
* Added README with usage examples and parameter details
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The `update:` recipe now reproduces the top-level vendoring overrides
(remove broken hami-dra subchart, clear Chart.yaml dependencies, strip
dra/hami-dra/podSecurityPolicy from upstream values.yaml) after
`helm pull`, so they no longer silently disappear on the next bump.
Template-level patches (DRA guards in scheduler/*, indent fix in
device-plugin/monitorservice.yaml) are documented with rationale and
commit references — they remain a manual step because automating them
would be fragile across upstream template restructures.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The HelmRelease at packages/apps/kubernetes/templates/helmreleases/hami.yaml
references chartRef.name 'cozystack-kubernetes-application-kubevirt-kubernetes-hami',
but that component was missing from the kubernetes-application PackageSource.
The HelmRelease would sit in Stalled: ArtifactNotFound at install time.
Add kubernetes-hami next to kubernetes-gpu-operator under variant kubevirt,
mirroring the existing pattern for every other tenant-cluster addon. The
standalone cozystack.hami PackageSource is retained — same shape as
gpu-operator, which is registered both standalone (for bundles/iaas) and
as a kubernetes-application component (for tenant-cluster HelmReleases).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Match the pattern used by every other tenant-cluster addon HelmRelease
(gpu-operator, cilium, fluxcd, ...): only render when the upstream etcd
namespace is ready, not just when the addon is enabled. Add values-ci.yaml
to the HAMi test suite so the new gate has the fixture it needs.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The HAMi-driven default of disabling gpu-operator's device plugin is
applied via valuesOverride and can be re-enabled by users for advanced
topologies (mixed HAMi / vanilla NVIDIA device plugin pools). Document
this explicitly so the README aligns with the existing template merge
order.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
HAMi#173 was closed as "not planned" and only suggests a typo fix
(< 2.3.0 → < 2.30); it does not establish the actual 2.34 boundary.
HAMi-core#174 (symbol-level cause) and HAMi#1190 (empirical per-glibc
behavior) cover the same ground accurately.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The auto-generated changelog only listed #2459/#2467 (velero-configmap
Role move). Five additional PRs were backported and merged into
release-1.3 between then and the v1.3.1 tag (2026-04-28):
- #2471/#2524 - fix(api): IDOR in TenantNamespace Get/Watch
- #2496/#2505 - feat(linstor): linstor-csi v1.10.6 (Protocol-C dual-attach)
- #2462/#2511 - fix(etcd): remove destructive post-upgrade hook
- #2421/#2491 - fix(kamaji): memory limits + startup probe
- #2498/#2518 - build(linstor): wire linstor-gui into root build target
Update the release date to match the actual tag (2026-04-28), rewrite
the intro paragraph, and add @kvaps to contributors.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The AI-generated v1.3.1 changelog (#2480) was generated from
git log v1.3.0..main rather than git log v1.3.0..v1.3.1, because
the workflow checked out main while the v1.3.1 tag points to
release-1.3. As a result the changelog included:
- 8 PRs that were merged to main but never shipped in v1.3.1
- 6 backport PRs that were merged to release-1.3 *after* v1.3.1
was tagged
- Both originals and their backports as separate duplicate entries
- A 2024 PR (#435) that has nothing to do with this range
- Generic "Documentation updates" placeholders for website entries
- Title duplicated as both the bold label and the description
(`* **fix(...): X**: fix(...): X (...)`)
- The cozystack-ci bot listed as a human contributor
The actual v1.3.1 release range (v1.3.0..v1.3.1) contains exactly
one user-facing change:
41bcb0be [Backport release-1.3] fix(backups): move
velero-configmap Role to velero chart (#2467)
which is the backport of #2459 (myasnikovdaniil) shipped via
#2467 (IvanHunters). This commit replaces the contents of
docs/changelogs/v1.3.1.md with that one entry, the matching
two-person contributors list, and the standard footer.
The workflow + docs fixes that prevent this regression for future
patch releases will land in a separate PR against main.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds GPU observability to Cozystack: five Grafana dashboards, a set of
VictoriaMetrics recording rules with a throttle-regression alert, and
reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter
on Talos when running GPU workloads directly in pods.
### Components
1. **Dashboards** — `dashboards/gpu/*.json`, registered in
`packages/system/monitoring/dashboards-infra.list`. All follow the
project convention: the Prometheus data source is selected through the
`$ds_prometheus` template variable.
- `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics
engine, memory copy), VRAM, power, temperature, and health (XID errors,
power/thermal throttling). All metrics are shown per-GPU, filterable by
the `$Hostname` selector.
- `gpu-fleet` — cluster-wide inventory, capacity, utilization, and
per-node power draw as percentage of TDP.
- `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh
accounting; utilization and power are shown per-node (DCGM cannot
attribute hardware metrics to namespaces).
- `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a
per-namespace requests/limits table. All counts derive from the
`namespace:gpu_count:allocated` recording rule, which excludes
`Failed`/`Succeeded` pod carryover.
- `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation,
util-per-watt, and per-GPU power/thermal throttle fractions.
2. **Recording rules and alert** —
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`.
Four rule groups:
- `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview
panels.
- `gpu.recording.node.1m` — per-node hardware aggregates (grouped by
`Hostname`) plus per-namespace allocation counts from kube resource
requests; filters `cozy-*`/`kube-*`.
- `gpu.recording.efficiency.1m` — per-GPU tensor saturation and
util-per-watt, plus per-GPU power/thermal throttle fractions bounded via
`clamp_max(..., 1)`.
- `gpu.recording.throttle.validation.5m` — houses the
`GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp
throttle fraction exceeds 1.0. This guards against DCGM counter-unit
drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10
with DCGM 3.x.
Rules are safe on clusters without DCGM — they evaluate to empty series
when no matching metrics are scraped.
3. **Reference manifests** under
`packages/system/gpu-operator/examples/`. These are **not** templates;
they document one working configuration:
- `values-native-talos.yaml` — Cozystack Package values for the
native-pod scenario.
- `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with
profiling, ECC, throttling, and energy counters used by the dashboards
and recording rules.
- `nvidia-driver-compat.yaml` — DaemonSet that stages
`libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a
location the GPU Operator validator inspects. Workaround for
[NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687).
Requires `pod-security.kubernetes.io/enforce: privileged` on the target
namespace when the cluster enforces baseline/restricted
PodSecurityStandards.
- `README.md` — explains the two deployment paths (sandbox vs. native),
which DCGM metrics each dashboard depends on, and how the pieces fit
together.
4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats`
enforces that every recording-rule reference in tracked dashboards
resolves, and that every DCGM metric referenced by the rules is declared
in either the upstream default CSV snapshot
(`hack/dcgm-default-counters.csv`) or the custom CSV shipped under
`examples/`.
### Why reference manifests, not templates
The out-of-the-box `packages/system/gpu-operator/values-talos.yaml`
targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via
VFIO, driver disabled on the host, device plugin disabled. That is the
sensible default for Cozystack today.
Running GPU workloads directly in pods on Talos is also possible but
requires a different configuration that makes assumptions not every user
shares:
- the NVIDIA Talos system extension must be installed on GPU nodes,
- the operator's own driver and toolkit components must be disabled
because the extension provides them,
- the validator's hardcoded paths need a workaround until
NVIDIA/gpu-operator#1687 lands upstream.
Shipping those as active templates would silently impose those
assumptions. Shipping them as reference files next to the package lets
operators opt in with eyes open and understand the moving parts before
applying them.
### Scope note
The dashboards and recording rules are useful independently of the
reference manifests — any cluster that scrapes DCGM Exporter (however it
is installed) will populate them. The reference manifests are
self-contained and do not change any default behavior of the
`gpu-operator` package.
### Screenshots
<img width="2824" height="1055" alt="2026-04-27_12-36-52"
src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a"
/>
<img width="2913" height="1255" alt="2026-04-27_12-36-39"
src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229"
/>
<img width="2932" height="1251" alt="2026-04-27_12-36-30"
src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23"
/>
<img width="2864" height="1222" alt="2026-04-27_12-36-00"
src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9"
/>
<img width="2958" height="1226" alt="image"
src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76"
/>
### Release note
```release-note
feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added five GPU dashboards for performance, efficiency, fleet overview,
quotas/allocation, and tenant usage (utilization, power, throttling,
allocation, billing).
* Added cluster recording rules and a throttling validation alert; new
Prometheus-derived metrics and per-GPU rankings/timeseries.
* **Documentation**
* Added GPU Operator examples and README documenting native Talos setup
and required DCGM counters.
* **Tests**
* Added validation tests ensuring dashboard ↔ recording-rule and DCGM
metric consistency.
* **Chores**
* Registered new GPU dashboards in infra listings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Tests need _namespace.etcd from values-ci.yaml after the merge
introduced an etcd-namespace guard on the gpu-operator HelmRelease
condition.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## Summary
Fixed two IDOR (Insecure Direct Object Reference) vulnerabilities in the
TenantNamespace API handlers that allowed authenticated users to access
metadata of tenant namespaces without proper authorization checks.
## Changes
### New optimized function: hasAccessToNamespace()
- Lists RoleBindings **only in the target namespace** instead of all
cluster RoleBindings
- Used by Get() and Watch() for single-namespace access checks
- Order of magnitude faster than the previous approach
### Get() handler
- Uses `hasAccessToNamespace()` for efficient authorization
- Returns `NotFound` instead of `Forbidden` to prevent tenant
enumeration
- Now correctly enforces RoleBinding-based access control
### Watch() handler
- Uses `hasAccessToNamespace()` for per-event authorization
- Logs authorization errors with klog for security audit
- Events for unauthorized namespaces are silently filtered out
- Ensures users only receive watch events for namespaces they have
access to
### Additional fixes
- Fixed ServiceAccount subject handling when namespace is empty
(defaults to RoleBinding namespace)
- Added proper error logging in Watch handler
## Performance Impact
| Handler | Before | After |
|---------|--------|-------|
| List() | List all RoleBindings × 1 | No change ✅ |
| Get() | List all RoleBindings × 1 | List RoleBindings in 1 namespace 🚀
|
| Watch() | List all RoleBindings × N events | List RoleBindings in 1
namespace × N events 🚀 |
**For Watch with 100 events:**
- Before: 100 × (all cluster RoleBindings) = catastrophic
- After: 100 × (1-5 RoleBindings in namespace) = fast + cached by
controller-runtime
## Security Impact
**Before**: Any authenticated user could:
- Read metadata (labels, annotations, creation time) of any `tenant-*`
namespace via `Get()`
- Stream all tenant namespace events via `Watch()`, including
creation/modification/deletion
**After**: Users can only access tenant namespaces they have explicit
RoleBindings for, matching the behavior of the `List()` handler.
## Testing
Manually verified:
- Users can only `get` their own tenant namespaces
- Users can only `watch` events for their own tenant namespaces
- Unauthorized access returns `NotFound` (not `Forbidden`) to prevent
enumeration
- `List()` behavior remains unchanged and consistent with `Get()` and
`Watch()`
- Authorization errors are logged for security audit
## Checklist
- [x] Code follows project style and conventions
- [x] Security vulnerability is fully mitigated
- [x] Authorization logic is consistent across List/Get/Watch handlers
- [x] Performance optimized per feedback from @timofei.larkin
- [x] Error logging added for security audit
- [ ] Unit tests added (can be done in follow-up PR)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enforced per-namespace access control for Get and Watch; inaccessible
namespaces return Forbidden or NotFound as appropriate
* Forwarded field/label selectors to upstream watches and added
defensive filtering to skip inaccessible events (logged)
* Improved ServiceAccount subject namespace fallback and
privileged-group bypass
* **Tests**
* Added security tests covering RoleBinding subjects, groups, privileged
bypasses, ServiceAccounts, and access-denied behaviors
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Add `serverName` field to bootstrap configuration to support backup
recovery when the Barman server name in `backup.info` differs from the
Kubernetes cluster name.
This fixes "no target backup found" errors during CloudNativePG recovery
operations.
## Problem
CloudNativePG forms the backup path as `destinationPath + "postgres-" +
oldName`, but searches for backups using the `server_name` field from
`backup.info`. When these values don't match (e.g., cluster name is
`grafana` but `server_name` is `cloud`), recovery fails with "no target
backup found".
## Solution
- Add optional `serverName` parameter to `bootstrap` configuration
- When specified, CloudNativePG uses this value to search for backups in
S3
- Falls back to `oldName` when `serverName` is not provided (backwards
compatible)
## Changes
- Add `ServerName` field to PostgreSQL CRD type definition
- Add conditional `serverName` to Cluster `externalClusters` template
- Update `values.yaml` and README.md with `serverName` documentation
- Regenerate `values.schema.json` and `postgres.yaml` CRD
## Test plan
- [ ] Deploy postgres cluster with backups enabled
- [ ] Create backup
- [ ] Create new postgres instance with `bootstrap.enabled=true`,
`bootstrap.oldName=<original-name>`, and
`bootstrap.serverName=<value-from-backup.info>`
- [ ] Verify recovery completes successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an optional bootstrap.serverName setting to specify the Barman
server name from the original cluster’s backup.info (used when that
server name differs from the Kubernetes cluster name).
* **Documentation**
* Clarified that bootstrap.oldName must match the serverName value
recorded in backup.info; updated docs and examples to reflect the new
bootstrap.serverName option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an opt-in `publishing.exposure=loadBalancer` mode for the
ingress-nginx Service as a migration path away from
`Service.spec.externalIPs`, which is deprecated upstream in Kubernetes
v1.36
([KEP-5707](https://github.com/kubernetes/enhancements/issues/5707),
[kubernetes#137293](https://github.com/kubernetes/kubernetes/pull/137293)).
The `AllowServiceExternalIPs` feature gate is expected to default to off
around v1.40 and the implementation to be removed around v1.43.
Stacked on top of #2464 (cilium v1.19.3 bump) — depends on it for the
`CiliumLoadBalancerIPPool` at `cilium.io/v2`.
### Behavior
- New platform value `publishing.exposure` — enum `externalIPs |
loadBalancer`, default `externalIPs` (current behavior unchanged on
upgrade).
- Plumbed through `cozystack-values` into each tenant's ingress
HelmRelease via the new `expose-mode` key.
- When `exposure=loadBalancer` and the current namespace matches
`publishing.ingressName`, the Service becomes `type: LoadBalancer` with
`externalTrafficPolicy: Local` and a `CiliumLoadBalancerIPPool`
announces the addresses from `publishing.externalIPs` via Cilium LB
IPAM.
- The pool uses a namespace-only `serviceSelector`
(`io.kubernetes.service.namespace: <ns>`) — any LoadBalancer Service in
the tenant namespace draws from it. See "Pool ownership" below.
- IPv4 addresses get `/32` CIDRs, IPv6 addresses get `/128`. Mixed
families supported. Pre-CIDR entries (`192.0.2.10/32`) are accepted
without double-suffixing.
- Unknown values, `loadBalancer` with empty externalIPs, and stray empty
entries from `publishing.externalIPs` are rejected at render time with
explicit error messages.
### Pool ownership
The `CiliumLoadBalancerIPPool` is rendered from
`packages/apps/tenant/templates/cilium-lb-pool.yaml`, not from the
ingress chart. The tenant chart is the per-tenant owner of cross-cutting
resources (Namespace, `cozystack-values` Secret, HelmReleases for
ingress and gateway), so one pool per tenant lives there.
Cilium LB IPAM rejects overlapping CIDRs across pools regardless of
`serviceSelector` — the last-added pool gets `cilium.io/PoolConflict`
and stops allocating. Keeping the pool in the ingress chart would
collide with the Gateway-API PR (#2470), which materialises its own
LoadBalancer Service from the same `publishing.externalIPs` range when
`tenant.spec.gateway=true`. Moving the pool to the tenant chart with a
namespace-only selector lets a single pool back both services
(ingress-nginx today, a Cilium Gateway Service once #2470 lands).
Only the ingress-loadBalancer signal is wired here
(`_cluster.expose-mode=loadBalancer` + `.Values.ingress=true` +
publishing tenant). #2470 rebases on top of this PR, drops its own
`packages/extra/gateway/templates/cilium-lb-pool.yaml`, and adds an OR
branch for `.Values.gateway` in the tenant template.
### Scope
Only the ingress-nginx Service is migrated by this setting. Other
cozystack components that still write `Service.spec.externalIPs`
directly (notably the `vpn` app at
`packages/apps/vpn/templates/service.yaml`) need separate follow-up
before Kubernetes v1.40.
### Tests
- `packages/extra/ingress/tests/exposure_test.yaml` — 10 helm-unittest
cases on the ingress-nginx Service: type / externalTrafficPolicy /
externalIPs assertion for both modes, unknown-mode rejection
(case-sensitive enum), empty-IPs failure, empty-entry filtering,
non-publishing-tenant fallback.
- `packages/apps/tenant/tests/exposure_test.yaml` — 9 cases on the pool
itself: IPv4, IPv6, mixed, pre-CIDR input, empty-entry filtering,
`ingress=false` in publishing tenant (no pool), non-publishing tenant
(no pool), empty externalIPs (no pool).
- Both suites are auto-discovered by `hack/helm-unit-tests.sh` via the
`test:` target in each package's Makefile.
### Caveats (copied from the inline `values.yaml` comment)
- `loadBalancer` mode uses `externalTrafficPolicy: Local`. The external
IP must already be routed to a node that hosts an ingress pod (floating
IP / upstream router / podAntiAffinity).
- Cilium does not announce the IP on its own unless L2 announcements or
BGP are enabled in the Cilium values (disabled by default in cozystack).
- Switching the value on a running cluster causes the ingress-nginx
Service to be recreated (`upgrade.force: true` on the HelmRelease +
Service kind change); expect a brief interruption of ingress traffic.
### Release note
```release-note
feat(ingress): add opt-in publishing.exposure=loadBalancer mode that uses type: LoadBalancer + CiliumLoadBalancerIPPool instead of the deprecated Service.spec.externalIPs. Default (externalIPs) preserves existing behavior.
```
## What this PR does
Closes a bootstrap race where the `cnpg-webhook-service` Service gets
its EndpointSlice populated and the data plane (kube-proxy / Cilium)
programmed a second or two after `helm install --wait` on
`cozy-postgres-operator` declares the HelmRelease Ready. Any HelmRelease
that `dependsOn: postgres-operator` and creates a
`postgresql.cnpg.io/v1` resource in that window (cozy-keycloak, tenant
Postgres apps) has kube-apiserver's call to `mcluster.cnpg.io` fail with
```
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
```
which fails the downstream release's install.
The fix is a post-install / post-upgrade Helm hook (ServiceAccount +
ClusterRole + ClusterRoleBinding + Job) that probes `/readyz` on the
webhook through the apiserver service proxy. Apiserver's service proxy
uses the same endpoint-resolution and apiserver-initiated pod dial as
the admission webhook path, so once `/readyz` answers through the proxy,
the subsequent admission call will also succeed. Helm `--wait` blocks
the install from completing until the Job exits 0, so the HelmRelease
Ready condition does not lie anymore.
Hardened per review:
- RBAC scoped to a single `services/proxy` resourceName
(`https:cnpg-webhook-service:webhook-server`) — the exact string the
apiserver URL path parser expects for the Service proxy subresource.
- A drift-guard helm-unittest test renders the vendored cnpg Service
template and fails if its `metadata.name` / `ports[0].name` diverge from
the literals in the hook, so a future `make update` that renames the
service forces this template to be updated in the same change.
- Image digest-pinned (`clastix/kubectl:v1.32@sha256:…`) with a
`renovate:` annotation; `imagePullPolicy: IfNotPresent` when
digest-pinned, `Always` when tag-only.
- Job runs under PSA-restricted-compatible securityContext (non-root,
seccomp RuntimeDefault, readOnlyRootFilesystem, drop ALL caps).
- `activeDeadlineSeconds` is derived from `maxAttempts × sleepSeconds +
60s` so a values override raising retries does not get silently cut by a
fixed deadline.
- `backoffLimit: 2` (configurable) so a transient pod-level failure
(image pull rate limit, OOM, CNI hiccup) does not fail the whole
HelmRelease.
- On timeout the Job prints the last `kubectl get --raw` stderr so the
operator can distinguish DNS / refused / 401 / TLS from the Job logs.
21 helm-unittest assertions cover ordering, RBAC/URL parity, subchart
drift, image policy, retry-loop bounds, securityContext, and the
deadline-scaling invariant. Wired via `make test` in
`packages/system/postgres-operator/Makefile`.
First surfaced on #2470 E2E run 24862782568 where
`cozy-keycloak/keycloak` failed with the exact signature above. Not tied
to that PR's branch — this is independently applicable to `main`.
### Release note
```release-note
fix(postgres-operator): add a post-install readiness gate that blocks the HelmRelease from reporting Ready until the cnpg admission webhook actually serves through the cluster Service, preventing "dial tcp <svc-ip>:443: connect: connection refused" on the first HelmRelease that depends on postgres-operator (keycloak, tenant Postgres apps).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added webhook readiness validation during install/upgrade to block
completion until the admission webhook is reachable.
* New configuration options for the readiness probe image and
retry/timeouts.
* **Tests**
* Introduced a comprehensive test suite validating rendered manifests,
probe behavior, RBAC, image rendering, and retry/timeout logic.
* **Chores**
* Added a test entry point to the project Makefile to run chart/unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Collapse the chart to a single Deployment shape that mirrors
cozystack-api: 2 replicas, soft (preferred) nodeAffinity to
node-role.kubernetes.io/control-plane via Exists, permissive
tolerations, soft podAntiAffinity on hostname, an unconditional
PodDisruptionBudget with maxUnavailable: 1, and a Service with
spec.trafficDistribution: PreferClose. The same shape works on
Talos / kubeadm / k3s and on managed Kubernetes / Cozy-in-Cozy
tenant clusters without per-distro overrides — fixing #2417 by
making the soft control-plane affinity gracefully fall back to
worker scheduling when no control-plane nodes are visible.
Drop the DaemonSet path entirely. The previous PR's deployment.enabled
toggle, the workload-kind switch in templates/workload.yaml, the
fail-guard for localK8sAPIEndpoint+nodeAffinity=[], the conditional
PDB, and the values-shape knobs for nodeAffinity and tolerations all
go away as a consequence. Override the standard Deployment fields
through the usual component-values mechanism if a non-default topology
is ever needed.
Mark localK8sAPIEndpoint.enabled deprecated and flip the default to
false. The flag injects KUBERNETES_SERVICE_HOST=status.hostIP, which
is only valid when the pod is actually scheduled on an apiserver-
bearing node. With the new soft control-plane affinity, the pod can
land off-control-plane and crash-loop. The latency motivation for the
flag is real but pending separate webhook performance work; once
addressed, the flag can be removed.
Revert all changes to packages/core/platform/images/migrations/migrations/20.
The earlier ds/...-or-deploy/... fallback was over-engineered: per
run-migrations.sh, migration 20 fires only when CURRENT < 20 (i.e.
direct upgrades from pre-0.37 to 1.3+), and that path is unsupported
anyway. The original ds/... rollout-status line is dead code on every
supported install and upgrade path.
Add helm-unittest coverage for: the default Deployment shape (replicas,
soft nodeAffinity, soft podAntiAffinity, tolerations); no env vars at
the default localK8sAPIEndpoint.enabled=false; PDB rendered
unconditionally with maxUnavailable: 1, including at replicas=1 (the
no-op case); replicas value drives spec.replicas; and that
localK8sAPIEndpoint.enabled=true does inject the env vars. The Service
test continues to assert trafficDistribution: PreferClose with no
internalTrafficPolicy.
Add a slim README documenting the topology, parameters, and the
deprecation note for localK8sAPIEndpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
- Hoist user identity extraction out of the Watch goroutine; reuse a
cached username and groups map across events instead of re-fetching
them per event. Watch now returns Unauthorized up front when no user
is present in the context, rather than failing silently per event.
- Switch the per-event access-check error log to structured klog.ErrorS
to comply with the project Go style guide.
- Strengthen TestGet_WithAccess to assert the concrete *TenantNamespace
type plus Name, Kind, and APIVersion, so type or metadata regressions
fail fast.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `linstor-gui` package (added in #2382) was never wired into the root
`Makefile`'s `build:` target, so the image is not built or published by
CI. `ghcr.io/cozystack/cozystack/linstor-gui` returns `NAME_UNKNOWN`
from the GHCR API, and `values.yaml` still pins `tag: 2.3.0` without a
digest because the per-package Makefile that rewrites the tag after a
successful push has never run in CI.
This PR adds the missing line so the next build publishes the image and
digest-pins `values.yaml` automatically.
### Release note
```release-note
build(linstor): include linstor-gui in root image build target so the image is built and published by CI (the chart previously referenced an image that did not exist in the registry)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Integrated the linstor-gui container image build into the main image
build workflow so the GUI image is produced as part of standard builds.
* Streamlined the GUI image build configuration to use consolidated
build arguments, improving consistency and maintainability of automated
image builds and metadata updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Bumps the vendored cozy-proxy chart and image tag from v0.2.0 to v0.3.0
by running `make update` in `packages/system/cozy-proxy`.
cozy-proxy v0.3.0 release:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
## Why
cozy-proxy v0.3.0 is the companion of #2501. The chart fix in #2501
finally honors the documented PortList semantics (only `externalPorts`
reachable, ICMP preserved by default), but only takes effect once
cozy-proxy v0.3.0 is in place — v0.2.0 has no port-aware logic, so the
chart fix silently no-ops on it. This PR ships that runtime side.
## Upgrade impact
For cozystack users this is a **bug fix, not a breaking change**.
cozy-proxy v0.3.0 introduces two contract changes upstream — (1)
label-only selector via `service.kubernetes.io/service-proxy-name:
cozy-proxy`, and (2) port-filter as the default ingress mode — both of
which are absorbed by the vm-instance chart (label landed in #2357,
`wholeIP` + `allowICMP` wired in #2501).
- **`externalMethod: WholeIP`** (default) — chart renders label +
`wholeIP: "true"` in both old and new versions. cozy-proxy keeps
passthrough. **No change.**
- **`externalMethod: PortList`** — previously the chart always rendered
`wholeIP: "true"` regardless of method, and cozy-proxy v0.2.0 had no
port-aware logic, so PortList was silently a no-op (all ports
reachable). After this bump + #2501, PortList behaves as documented:
only ports listed in `externalPorts` reach the VM. Users who configured
PortList but reached undeclared ports (relying on the silent no-op) will
see those ports become unreachable — that's the documented intent of
PortList finally taking effect. ICMP keeps working by default thanks to
`externalAllowICMP: true`.
- **DaemonSet rollover** — re-init of the nft table during cozy-proxy
pod restart is a brief blip on existing flows; same as any DaemonSet
upgrade.
Out-of-tree consumers using cozy-proxy directly (without the
`service-proxy-name` label, or relying on the absent-annotation
passthrough default) need to migrate per the upstream v0.3.0 release
notes — not a cozystack concern.
## Order of operations
This PR is functionally complete on its own (image bump), but the
user-facing PortList fix it enables only happens when #2501 also lands.
Recommended merge order:
1. #2501 first (chart adapts to the new contract)
2. This PR second (cozy-proxy DaemonSet picks up the new image)
The reverse order is also safe — cozy-proxy v0.3.0 with the unmodified
vm-instance chart still produces correct results for `externalMethod:
WholeIP` (most common case) and keeps PortList silently broken until
#2501 lands.
## Test plan
- [ ] `helm template packages/system/cozy-proxy` — image tag `v0.3.0`
rendered
- [ ] `make unit-tests` passes
- [ ] After #2501 lands, end-to-end: `externalMethod: PortList` with
`externalPorts: [22]` filters all other ports, ping works
- [ ] WholeIP-method VM Service unchanged after upgrade
## Companion PRs
- #2501 — vm-instance chart wires `wholeIP` / `allowICMP` per
`externalMethod`
- cozystack/cozy-proxy#13 (merged) — label-only selector + port-filter
default
- cozy-proxy v0.3.0 release notes —
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Released version 0.3.0 with updated Helm chart metadata and container
image configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `post-upgrade` Helm hook in `packages/extra/etcd` that was
`kubectl delete`ing the etcd TLS chain (`etcd-ca-tls`,
`etcd-peer-ca-tls`,
`etcd-client-tls`, `etcd-peer-tls`, `etcd-server-tls`) and then deleting
etcd pods on every chart upgrade.
The hook was gated by a semver compare of a stored
`etcd-deployed-version`
ConfigMap against `2.6.1`. That gate was written when chart versions
looked
like `2.6.0`, `2.6.1`, etc. Since commit `f871fbdb` ("Remove
versions_map
logic") all chart versions are stamped as `0.0.0+<git-hash>`, which per
semver is always `< 2.6.1`, so the gate always evaluates to "regenerate
certs" and the destructive hook fires on every upgrade.
On clusters running Kamaji-managed tenant control planes the consequence
is
severe: wiping the etcd CA triggers cert-manager to re-issue it with a
brand-new CA, but Kamaji's `datastore-certificate` Secrets still carry
the
old CA bundle mounted into tenant `kube-apiserver` pods. Those
apiservers
hit `x509: certificate signed by unknown authority` against
`etcd.<ns>.svc:2379` and go into CrashLoopBackOff until each tenant
`DataStore` is individually force-reconciled and the Deployments rolled.
Commit `47d81f70` ("Disabled private key rotation in CA certs") already
fixed the underlying `rotationPolicy: Always` issue the hook was written
to
paper over in March 2025, so the migration has no remaining use.
Changes:
- Delete `templates/hook/{job,role,rolebinding,serviceaccount}.yaml`
- Delete `templates/version.yaml` (only the hook read
`etcd-deployed-version`)
- Add `tests/no-post-upgrade-hook_test.yaml` as a regression guard — if
`templates/hook/job.yaml` or `templates/version.yaml` is ever brought
back,
`make unit-tests` fails
### Release note
```release-note
fix(etcd): remove destructive post-upgrade cert-regeneration hook. Previously the etcd chart ran a `post-upgrade` hook on every Helm upgrade that deleted etcd TLS Secrets and pods, causing cert-manager to re-issue the etcd CA. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 -> 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+<git-hash>` and the underlying `rotationPolicy: Always` issue was fixed in commit `47d81f70`. This is behavior removal, not cosmetic cleanup.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added Helm chart test suite verifying absence of post-upgrade hook Job
and version ConfigMap.
* **Chores**
* Removed post-upgrade hook Job and associated RBAC resources (Role,
RoleBinding, ServiceAccount).
* Removed version ConfigMap.
* Added `test` target to Makefile for running Helm chart unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from sircthulhu and CodeRabbit:
1. Return Forbidden instead of NotFound for unauthorized access
- Get() now returns 403 Forbidden to follow standard K8s RBAC behavior
- Previously returned 404 NotFound for security-by-obscurity
- Updated test expectations to match new behavior
2. Propagate field and label selectors in Watch handler
- Pass opts.FieldSelector and opts.LabelSelector to upstream Watch
- Add defensive filtering before authorization to prevent RBAC bypass
- Fixes potential issue with resourceNames restrictions
3. Refactor subject-matching logic to eliminate duplication
- Extract matchesSubject() helper for Group/User/ServiceAccount checks
- Remove duplicated code from filterAccessible and hasAccessToNamespace
- Consolidates ServiceAccount namespace fallback logic
All tests pass successfully.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
The `vm-instance` chart now drives the cozy-proxy `wholeIP` and
`allowICMP` annotations explicitly so that `externalMethod: PortList`
actually filters ingress traffic to declared ports while keeping
ping/PMTU functional.
- Render `networking.cozystack.io/wholeIP: "false"` when
`externalMethod: PortList` (was always `"true"`, which silently disabled
the PortList semantics in cozy-proxy).
- Add `externalAllowICMP` value (default `true`) propagated as
`networking.cozystack.io/allowICMP` when `externalMethod: PortList`.
Without this, cozy-proxy drops ICMP in port-filter mode (ping/PMTU
broken). Operators can set `externalAllowICMP: false` to opt out.
The changelog entry is intentionally **not** part of this PR — it will
be added in a dedicated `docs: add changelog for vX.Y.Z` commit at
release time, per project convention.
## Why
`externalMethod: PortList` is documented as filtering ingress traffic to
declared ports but has been non-functional on Cozystack v1.3.0 —
verified empirically on a 3-node Talos lab. Root cause was twofold:
chart always set `wholeIP: "true"`, and cozy-proxy v0.2.0 had no
port-aware logic. The cozy-proxy side was fixed in
cozystack/cozy-proxy#11 (merged) and cozystack/cozy-proxy#12 (allowICMP
follow-up); this PR completes the user-visible fix on the chart side.
## Companion PRs
- cozystack/cozy-proxy#11 (merged) — per-service ingress port filtering
- cozystack/cozy-proxy#12 (merged) — `allowICMP` annotation for
port-filter mode
## Test plan
- [x] Built cozy-proxy with the companion fix locally, deployed on a
3-node Talos lab (Cozystack v1.3.0)
- [x] `wholeIP: "false"` Service with `spec.ports: [22]`: only port 22
reachable from outside; ports 80/443/8080/9999 filtered
- [x] WholeIP-annotated Service unchanged: all listening ports reachable
- [x] Egress IP preservation works in both modes (TCP curl + UDP DNS)
- [x] `nft list table ip cozy_proxy` confirms expected ruleset
- [x] `helm template` renders the expected annotation matrix: `PortList`
default → `wholeIP=false, allowICMP=true`; `PortList` opt-out →
`allowICMP=false`; `WholeIP` → only `wholeIP=true`
- [x] `make unit-tests` passes locally
- [ ] CI unit tests
- [ ] CI E2E
## Backport
Suggesting `backport-v1.3` once merged.
## Release note
```release-note
[vm-instance] Make `externalMethod: PortList` actually filter ingress traffic to ports listed in `externalPorts`. New `externalAllowICMP` knob (default true) propagates the cozy-proxy `allowICMP` annotation to keep ping/PMTU functional in port-filter mode. Combined with cozy-proxy v0.3.0+, only listed ports plus ICMP are reachable from the VM's LoadBalancer IP.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `externalAllowICMP` configuration option to control ICMP traffic
acceptance for VM external access in PortList mode (enabled by default).
* **Documentation**
* Updated parameter documentation to include the new ICMP traffic
control setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pulls in the per-port filtering and allowICMP support that the companion
vm-instance chart fix in #2501 relies on. cozy-proxy v0.3.0 also tightens
the selector to the standard service.kubernetes.io/service-proxy-name=cozy-proxy
label and switches the default ingress mode to port-filter; both are
already covered by the vm-instance chart (label landed in #2357,
wholeIP/allowICMP wired explicitly in #2501), so VM workloads upgrade
transparently.
Out-of-tree consumers using cozy-proxy annotations directly (without the
label, or relying on the absent-annotation passthrough default) are
called out in the upstream v0.3.0 release notes:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Address review feedback from coderabbitai on packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml:119
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback from coderabbitai on packages/system/gpu-operator/examples/nvidia-driver-compat.yaml:77
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
- Adds a pre-commit hook and a dedicated GitHub Actions workflow that
run `make generate` at the repo root to catch drift in generated API
code (CRDs, DeepCopy, clients, RBAC) before it lands on main.
- Pre-commit hook is scoped to paths that actually affect codegen
(`api/`, `pkg/apis/`, `hack/update-codegen.sh`,
`hack/boilerplate.go.txt`) so unrelated commits are not slowed down.
- CI workflow (`.github/workflows/codegen-drift.yml`) installs Go from
`go.mod`, runs `make generate`, and fails on drift with an error
pointing contributors to the local fix.
- Includes one drift fix the check surfaced: `RestoreJobSpec.Options`
(added in #2437) had no `DeepCopyInto` handling — regenerated.
### Release note
```release-note
ci(api): add pre-commit hook and GitHub Actions workflow that verify generated API code (CRDs, deepcopy, clients, RBAC) is in sync with `make generate`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a CI workflow that checks generated code during PRs and fails
the build if generated artifacts diverge.
* Added a pre-commit hook that runs generation checks locally to prevent
committing outdated generated files.
* Fixed deep-copy behavior for backup restore job specs so nested
options are correctly duplicated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Add `externalAllowICMP` value (default true) propagated as
`networking.cozystack.io/allowICMP` annotation on the rendered Service
when `externalMethod: PortList`. The cozy-proxy companion (released as
part of cozystack/cozy-proxy#11 + #12) drops ICMP by default in
port-filter mode, which breaks ping and PMTU discovery; defaulting the
chart to "true" preserves user expectations while still allowing
operators to opt out by setting `externalAllowICMP: false`.
- Remove the v1.3.1.md changelog entry. Project convention is to add
changelogs in a dedicated "docs: add changelog for vX.Y.Z" commit at
release time, not as part of feature/fix PRs.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
## What this PR does
Bumps `linstor-csi` from v1.10.5 to v1.10.6 and ships an out-of-tree
patch that fixes live migration of KubeVirt VMs whose volumes sit on a
DRBD Protocol-A/B resource (e.g. a `replicated-async` StorageClass).
### Problem
DRBD requires Protocol C whenever `allow-two-primaries=yes` is enabled.
Operators commonly opt into Protocol A on a per-resource-group basis for
async / WAN replication, which silently breaks every subsequent live
migration of consumers of those volumes: `drbdadm adjust` rejects the
second-attach with `Protocol C required` (errno 139), KubeVirt's
evacuation loop retries indefinitely, and the VM stays pinned to the
source node.
### Change
- `LINSTOR_CSI_VERSION` 1.10.5 → 1.10.6 (Makefile + Dockerfile default).
- New patch `002-protocol-c-override-for-dual-attach.diff`: when
`Attach` installs `allow-two-primaries=yes` on the resource-definition
during a second attach, it also installs `DrbdOptions/Net/protocol=C` as
an override on the resource-definition. The override applies to every
connection (including diskless TieBreaker peers, where a per-pair
override would still leave one connection broken). It is tagged with
`Aux/csi-protocol-override=yes` so `Detach` removes only the override
this driver installed, leaving any operator-set `Protocol` property on
the resource-definition untouched.
- Existing patch `001-relocate-after-clone-restore.diff` regenerated
against v1.10.6 (context shift only, no logic change — the old patch
hunks no longer aligned cleanly).
### Verification
- `make image-linstor-csi` builds successfully on linux/amd64 with both
patches applied.
- End-to-end test on dev5 cluster (KubeVirt v1.6.3, 3-node Talos):
created a Protocol-A resource-group + StorageClass, provisioned a VM on
top, and triggered live migration. Migration succeeds in a single Attach
with the override installed during dual-attach and removed by Detach.
Reproducer (without the patch) is the well-known evacuation loop with
`(node) Failed to adjust DRBD resource ... Protocol C required`.
### Upstream
Upstreamed as draft PR piraeusdatastore/linstor-csi#435.
### Compatibility
- No behaviour change for resources already using Protocol C (the common
case).
- No behaviour change for resources never attached with
allow-two-primaries.
- Idempotent: re-running `Attach` is a no-op once the override is
installed.
- Operator-set Protocol overrides on the resource-definition are
preserved (gated by the Aux marker).
### Release note
```release-note
fix(linstor): live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with "Protocol C required" — linstor-csi now installs a Protocol=C override on the resource-definition during dual-attach and reverts it on detach.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic replica relocation after volume clone and snapshot restore
to improve placement and load distribution.
* Conditional DRBD protocol override to enable/clean up dual-attach
(two-primaries) scenarios more reliably.
* **Chores**
* Updated LINSTOR CSI default to v1.10.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Render `networking.cozystack.io/wholeIP: "false"` on the Service when
`externalMethod: PortList` is configured (was always `"true"` before).
Combined with cozy-proxy v0.3.0+ which adds per-port filtering for
"false"-annotated services, this makes `externalMethod: PortList`
behave as documented: only ports listed in `externalPorts` are
reachable from the LoadBalancer IP.
Backward-compatible: existing services with `externalMethod: WholeIP`
continue to set `wholeIP: "true"` and behave identically. cozy-proxy
versions older than v0.3.0 ignore Services with `wholeIP: "false"`,
which means PortList Services on older cozy-proxy will lose their
egress IP preservation — but that path was already non-functional
ingress-wise, so this is not a regression for users actually relying
on PortList.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Bump linstor-csi to v1.10.6 and add an out-of-tree patch that overrides
DrbdOptions/Net/protocol to C on the resource-definition whenever a
volume is attached to a second node with allow-two-primaries (the
live-migration flow), and reverts it on detach.
Without the patch, any KubeVirt live migration of a VM whose volume
sits on a Protocol-A/B resource (for example a "replicated-async"
StorageClass) ends up in a permanent evacuation loop: drbdadm adjust
on the satellite rejects allow-two-primaries with "Protocol C
required" (errno 139), the migration fails, and KubeVirt retries
until manual intervention.
The override is set at the resource-definition level so it covers
every connection, including diskless TieBreaker peers, and is tagged
with an Aux marker so Detach reverts only the override we installed.
The 001 relocate-after-clone patch was regenerated against v1.10.6
(context shift only, no logic change).
Verified end-to-end on dev5: live migration of a KubeVirt VM whose
PVC sits on a Protocol-A resource-group now succeeds in a single
Attach.
Patch is upstreamed as draft PR piraeusdatastore/linstor-csi#435.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Closes#2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.
Implements a defense-in-depth fix:
- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.
Tests:
- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.
All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).
Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.
### Release note
```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
* Chart resources now render conditionally based on etcd presence.
* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.
* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The per-package Makefile added in #2382 hardcoded buildx flags
(--provenance, --builder, --platform=linux/amd64,linux/arm64,
--push, --load, --label) instead of using the shared $(BUILDX_ARGS)
macro from hack/common-envs.mk.
This broke CI: the runner's default docker driver does not support
multi-platform builds, and the hardcoded multi-arch platform list
crashed `make build` with "Multi-platform build is not supported for
the docker driver."
Replace the hardcoded flags with $(BUILDX_ARGS) to match every other
package (e.g. linstor, dashboard, cilium). $(BUILDX_ARGS) injects
--push, --load, --label, --provenance=false, and only sets --builder
or --platform when the operator explicitly exports BUILDER/PLATFORM.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The linstor-gui package was added in #2382 with its own per-package
Makefile and Dockerfile, but the root Makefile's `build:` target was
not updated to invoke it. As a result `ghcr.io/cozystack/cozystack/
linstor-gui` has never been published (registry returns NAME_UNKNOWN)
and the chart's `image.tag` was never digest-pinned. Any cluster
deploying the chart hits ImagePullBackOff.
Wire the package into the root build alongside the other system
images. The next CI build will publish the image and the per-package
Makefile will rewrite values.yaml `image.repository`/`image.tag` to a
digest-pinned reference automatically.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozystack.io/status-beacon: "true" annotation had no consumer in
the chart, no documented contract, and no convention defined for other
charts to follow. It would have become accidental precedent for
contributors copying the pattern without understanding it.
The ConfigMap itself is self-explanatory: the name <release>-awaiting-etcd,
data.status: "awaiting-etcd", and the human-readable message in
data.message all surface the same operator signal via kubectl get cm.
Drop the annotation; keep the ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Operators in air-gapped or rate-limited environments cannot reach
docker.io and the bundled busybox digest pin gives them no escape
hatch. Add an optional images.waitForKubeconfig chart value that, when
set, replaces the helper's image reference with any registry path
kubelet can pull. Empty value falls back to images/busybox.tag, so the
prior digest-pinned default is preserved.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Wrap the values: block in a conditional so the key is omitted entirely
when no defaults or overrides produce content. Previously the template
always emitted values: null, triggering unnecessary FluxCD reconciliation.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Use nindent instead of indent with leading whitespace to prevent
broken YAML rendering when devicePlugin.service.labels is set.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
OnDelete requires manual pod deletion to apply updates. RollingUpdate
with maxUnavailable constraint matches upstream default and is consistent
with all other system packages.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Adopt CNCF/Kubernetes label conventions for issues and PRs and add
automated labeling.
**Canonical label file**: `.github/labels.yml`. Synced into the
repository by `.github/workflows/labels.yaml` (EndBug/label-sync@v2) on
push to `main`, weekly cron, and manual dispatch. UI-only label edits
are overwritten — propose changes via PR to this file.
### Label namespaces
Following the [Kubernetes label
scheme](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md):
- `kind/*` — issue or PR type (bug, feature, documentation, support,
cleanup, regression, flake, failing-test, api-change, breaking-change)
- `priority/*` — urgency (critical-urgent, important-soon,
important-longterm, backlog)
- `triage/*` — review state (needs-triage, accepted, needs-information,
not-reproducible, duplicate, unresolved)
- `lifecycle/*` — issue/PR lifecycle (active, frozen, stale, rotten)
- `area/*` — subsystem; 15 seeded plus `area/uncategorized` fallback.
Extensible — propose a new area when no existing one fits.
- `do-not-merge/*` — PR merge blockers (work-in-progress, hold)
Cozystack-specific labels preserved: `epic`, `community`, `help wanted`,
`good first issue`, `quality-of-life`, `upstream-issue`, `backport`,
`backport-previous`, `release`, `automated`, `debug`, `sponsored`,
`lgtm`, `ok-to-test`, `security/*`, `size:*`.
### `area/*` set
15 areas seeded by activity in open issues and PRs: `area/ai`,
`area/api`, `area/build`, `area/ci`, `area/dashboard`, `area/database`,
`area/extra`, `area/kubernetes`, `area/monitoring`, `area/networking`,
`area/platform`, `area/release`, `area/storage`, `area/testing`,
`area/virtualization`. Plus `area/uncategorized` as the auto-labeler
fallback.
### Migration safety
Existing labels are renamed via `aliases:` in `labels.yml`. GitHub
preserves the label ID, so all currently tagged issues and PRs keep
their tags under the new name without losing references:
| Old | New |
|---|---|
| `bug` | `kind/bug` |
| `enhancement` | `kind/feature` |
| `documentation` | `kind/documentation` |
| `question` | `kind/support` |
| `frozen` | `lifecycle/frozen` |
| `stale` | `lifecycle/stale` |
| `duplicate` | `triage/duplicate` |
| `do-not-merge` | `do-not-merge/work-in-progress` |
| `do not merge` | `do-not-merge/work-in-progress` |
`delete-other-labels: false` on the initial rollout. Generic
GitHub-default labels (`wontfix`, `invalid`) are preserved untouched and
will be removed in a follow-up cleanup PR. EndBug processes aliases
sequentially, so the second of the two `do-not-merge*` aliases hits a
name collision and logs a warning — the legacy label survives that one
sync and is cleaned up in the same follow-up.
### PR auto-labeling
`.github/workflows/pr-labeler.yaml` parses each PR title on `opened`,
`edited`, `reopened`, and `synchronize` and applies labels additively
(never removes):
- **type → `kind/*`**: feat, fix, docs, chore, refactor (others get no
kind)
- **scope → `area/*`**: scope mapping covers all current cozystack
components (full table in `docs/agents/contributing.md`)
- **`!` after type or `BREAKING CHANGE:` footer**: applies
`kind/breaking-change`
- **`[Backport release-1.x]` prefix**: stripped before parsing;
`area/release` and `backport` labels added
- **Composite scope** (`feat(platform, system, apps): …`): each part
mapped independently
- **Bracket fallback** (`[scope] description`): maps `area/*` but cannot
infer `kind/*`
- **Unmapped scope or non-conventional title**: applies
`area/uncategorized` for human review
### Schema validation
`.github/workflows/labels.yaml` runs a `validate` job on every PR
touching `labels.yml` or its workflow. Asserts:
- description ≤ 100 chars (GitHub REST API limit)
- color is 6-char hex without leading `#`
- unique top-level names
- aliases do not collide with top-level names
Sync runs only on push to main, weekly cron, and manual dispatch; PR
runs validate-only.
### Hardcoded label/title references updated
- `.github/ISSUE_TEMPLATE/bug_report.md`: `labels: 'bug'` → `labels:
'kind/bug'`
- `.github/workflows/tags.yaml`:
- changelog PR labels `['documentation', 'automated']` →
`['kind/documentation', 'automated']`
- release PR title `Release v${version}` → `chore(release): cut
v${version}` (so the auto-labeler applies `kind/cleanup` +
`area/release`)
- changelog PR title `docs: add changelog for v${version}` →
`docs(release): add changelog for v${version}` (so the auto-labeler
applies `kind/documentation` + `area/release`)
### Documentation
- `AGENTS.md`: Activation entry pointing agents to `labels.yml` and the
PR title auto-labeling rules. States explicitly that `area/*` accuracy
outweighs reuse — propose a new area when none fits, do not shoehorn
into a wrong one.
- `docs/agents/contributing.md`: PR Title Auto-Labeling section with
type→kind and scope→area tables.
### Out of scope (follow-up PRs)
- Removal of redundant labels (`wontfix`, `invalid`, plus the surviving
legacy `do not merge` if alias-rename collision keeps it)
- Org-wide sync from `cozystack/.github/labels.yml`
- Dosu bot configuration update for `lifecycle/stale` (requires
dashboard access)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automated PR labeling from Conventional Commits (type → kind/*, scope
→ area/*), plus a comprehensive namespaced label taxonomy.
* **Chores**
* Workflows to validate, sync, and auto-apply labels (including
scheduled/manual runs and validation checks).
* Added repository-wide label configuration and normalized bug label
metadata to namespaced form.
* Updated release PR titling/labeling conventions.
* **Documentation**
* Contributor and agent guidance on PR title conventions, label
mappings, and triage procedures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:160,173:
emit core.warning when a Conventional Commits type has no kind/*
mapping or a scope has no area/* mapping. Without the warning, typos
(e.g., "hotfix" instead of "fix") and recurring new scopes silently
fall through to area/uncategorized, masking that the mapping has
drifted.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:155:
Conventional Commits 1.0 spec item 16 treats BREAKING CHANGE: and
BREAKING-CHANGE: as synonymous footers. The hyphen form was silently
ignored before, so PRs that use it would miss kind/breaking-change.
https://www.conventionalcommits.org/en/v1.0.0/#specification
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:129:
defensive (pr.labels || []) avoids TypeError if the webhook payload
arrives without the labels field on edge cases like stripped edited
events.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.
Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Add an opt-in `upgradeCRDs` field to `ComponentInstall` in
`PackageSource`. The field maps directly to
`HelmRelease.Spec.Upgrade.CRDs` so a component author can declare how
Flux should handle CRDs from the chart's `crds/` directory when the
release is upgraded.
The helm-controller default on upgrade is `Skip`, which means CRDs added
by a chart bump never reach clusters that already have the release
installed — they must be applied manually with `kubectl apply --filename
charts/.../crds/`. This surfaces on every upgrade of an operator whose
CRD set expands between versions (etcd-operator, cnpg, kubevirt, kamaji,
etc.).
Setting `upgradeCRDs: CreateReplace` lets Flux apply new CRDs
declaratively with the chart.
Values are restricted to `Skip`, `Create`, `CreateReplace` via a
kubebuilder enum marker. Empty / unset preserves the existing
helm-controller default, so every current `PackageSource` keeps working
unchanged.
Migration is out of scope here — follow-ups will opt individual packages
in case-by-case.
### Relation to existing CRD management approach
The project convention (per #377) is to extract CRDs into a dedicated
Helm chart that reconciles ahead of the operator chart. This PR does not
replace that pattern — it complements it for charts that keep CRDs
inline under `charts/<name>/crds/` where extraction isn't practical
(vendored upstream charts with tightly coupled CRDs). Packages that
already split CRDs out can leave `upgradeCRDs` unset and keep using
their existing separate chart.
### Release note
```release-note
feat(operator): add opt-in `upgradeCRDs` field to `PackageSource` component `install` block to control how CRDs from the chart's `crds/` directory are applied on HelmRelease upgrades (`Skip` by default; use `CreateReplace` for operators whose CRD set expands between versions).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable CRD upgrade policy for component installs: Skip, Create,
CreateReplace — controls CRD handling during package upgrades and
preserves controller default when unset.
* **Documentation**
* Guidance on CRD upgrade semantics, advice to use CreateReplace for
specific operators, and warning about potential data-loss risks;
clarified contributor scope examples and PR template guidance.
* **Tests**
* Added tests validating CRD policy parsing and presence of the CRD
policy enum in the published schema.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Match the wording adopted in docs/agents/contributing.md so that human
contributors and AI agents see the same guidance in both places.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Scopes section was read as an exhaustive enumeration, which led to
review feedback flagging any scope outside the list as invalid. The
intent has always been that contributors pick the most specific scope
for the change and extend the list when a genuinely new area appears.
Reword the section accordingly and add operator as an example scope.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe when to set upgradeCRDs: CreateReplace (operators that evolve
their CRD set additively between versions) and the data-loss risk of
enabling it on operators that drop fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add an opt-in UpgradeCRDs field to ComponentInstall that maps to
HelmRelease.Spec.Upgrade.CRDs, allowing a PackageSource component to
declare how Flux should handle CRDs from the chart's crds/ directory
on upgrade.
The helm-controller default on upgrade is Skip, which means new CRDs
added between chart versions never reach existing clusters and must be
applied manually. Setting upgradeCRDs: CreateReplace makes Flux apply
new CRDs declaratively with the chart.
Allowed values are restricted to Skip, Create, CreateReplace via a
kubebuilder enum marker. Empty / unset preserves the existing Flux
default, so all existing PackageSource resources keep working.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:73:
add space before closing pipe in the type to kind mapping table for
consistency with other rows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on AGENTS.md:33:
expand bare labels.yml and pr-labeler.yaml to .github/labels.yml and
.github/workflows/pr-labeler.yaml for consistency with surrounding refs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:88:
… replaced with ... for compatibility across editors and tools.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on .github/labels.yml:242:
all hex color values use lowercase characters for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to retrigger Build that lost the OCI registry tag race
on the previous attempt.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to retrigger Build job that failed on Docker Hub anonymous
pull rate limit (429 Too Many Requests) for golang:1.26.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
- Increase kamaji controller memory limit from 500Mi to 512Mi
- Increase kamaji controller memory request from 100Mi to 256Mi
- Add startup probe with 60-second timeout (12 attempts × 5s periods)
- Increase readiness/liveness probe initialDelaySeconds from 5s/15s to
30s
## Problem
The kamaji controller was experiencing frequent CrashLoopBackOff due to
OOMKilled errors. Analysis showed:
- Container was being killed with exit code 137 (OOMKilled) after ~20-25
seconds of runtime
- Memory limit of 500Mi was insufficient for controller initialization
- Readiness probe was failing because it started too early (5s
initialDelay), before the controller finished leader election (~17s)
## Solution
**Memory increase:**
- Limit: 500Mi → 512Mi (based on production testing)
- Request: 100Mi → 256Mi (ensures adequate reservation)
**Startup probe:**
- Added to give controller up to 60 seconds to initialize without being
killed by liveness probe
- 12 attempts × 5s period = 60s maximum startup time
**Probe delays:**
- ReadinessProbe: 5s → 30s initialDelay (controller needs ~17s to
acquire leader lease)
- LivenessProbe: 15s → 30s initialDelay (aligned with readiness)
## Testing
Verified in production cluster:
- Controller runs stable with 0 restarts
- No more OOMKilled events
- Successfully creates kubeconfig secrets for tenant clusters
## Related Issues
Fixes tenant cluster components stuck in ContainerCreating due to
missing kubeconfig secrets (caused by crashing kamaji controller).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Introduced automated health checks using HTTP-based probes to monitor
service status during startup, continuous operation, and readiness to
handle traffic.
* Adjusted container memory resource allocation for enhanced stability
and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The pool is now rendered from packages/apps/tenant/templates/cilium-lb-pool.yaml instead of packages/extra/ingress/templates/cilium-lb-pool.yaml. Cilium LB IPAM forbids overlapping CIDRs across pools regardless of serviceSelector, so both the ingress-loadBalancer path and the upcoming per-tenant Gateway in #2470 cannot each own their own pool on the same publishing.externalIPs range. The tenant chart is the natural per-tenant owner — it already creates the Namespace, the cozystack-values Secret, and the HelmReleases for both ingress and gateway.
The new pool uses a namespace-only serviceSelector (io.kubernetes.service.namespace: <ns>), which matches any LoadBalancer Service in the tenant namespace. The metadata.name changed from <trim>-ingress to <trim>-exposure to reflect that the pool is not ingress-specific.
Only the ingress-loadBalancer signal is wired in this commit (_cluster.expose-mode=loadBalancer plus .Values.ingress=true on the publishing tenant). The gateway branch is added in #2470 on top of this commit — it rebases, drops its own packages/extra/gateway/templates/cilium-lb-pool.yaml, and adds an OR branch for .Values.gateway in the tenant template.
Pool-rendering unit tests moved from packages/extra/ingress/tests/ to packages/apps/tenant/tests/. The ingress chart tests keep the Service-level asserts. packages/apps/tenant/Makefile gains a test target so hack/helm-unit-tests.sh picks up the new suite.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Only emit values key in hami.yaml when valuesOverride has content,
matching gpu-operator pattern. Add test verifying empty valuesOverride
does not produce spec.values. Fix trailing whitespace and missing
newlines in vendored chart templates.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Unconditionally emit values in hami.yaml matching the project pattern.
Remove duplicate test case and add coverage for omitted valuesOverride key.
Delete dead PSP template and RBAC rules (policy/v1beta1 removed in K8s
1.25). Override kube-scheduler image registry to registry.k8s.io to avoid
Chinese registry for international users.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Remove broken hami-dra subchart dependency from vendored chart
(Chart.yaml, Chart.lock, values.yaml) and strip DRA condition guards
from all templates since the subchart was already deleted. Override
devicePlugin updateStrategy to OnDelete to prevent destructive rolling
updates of GPU workloads. Align gpu-operator template with project
pattern (unconditional values emission). Add nodeConfiguration format
documentation and test for conflicting valuesOverride scenario.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback from coderabbitai on
packages/system/postgres-operator/values.yaml:30.
The old comment's 5-minute ImagePullBackOff scenario conflicted with
the 180s activeDeadlineSeconds that the default maxAttempts/sleepSeconds
resolve to, so the numbers could not both be taken at face value.
Rewrite the comment to state the actual deadline math and frame the two
gates as an AND with activeDeadlineSeconds being the shorter one under
defaults, so readers understand why backoffLimit has little headroom
without an accompanying maxAttempts/sleepSeconds bump.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:115.
The wait container had no resource requests or limits, so schedulers
treated it as BestEffort and downstream quota enforcement had no signal.
Set small requests (10m CPU, 32Mi memory) and conservative limits
(100m CPU, 64Mi memory) matching the actual footprint of the kubectl
polling loop.
Add a matching unittest assertion so the values stay in sync if anyone
touches the template.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:83.
The Job targets the cnpg-webhook-service/services/proxy subresource, which
is namespaced and lives in the release namespace. A namespaced Role and
RoleBinding grant the exact permission needed without creating global
RBAC for a namespaced probe, which is the principle of least privilege.
Also update the kind assertions in tests/webhook-ready-hook_test.yaml so
the unittest suite tracks the new Role/RoleBinding objects.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Bumps the vendored `robotlb` chart to the latest upstream build.
Chart version remains `0.1.3`; the bundled `appVersion` moves from
`0.0.5` to `0.0.6`.
The new `robotlb` release adds RBAC permissions for
`discovery.k8s.io/endpointslices` (`get`, `list`, `watch`), which are
required to manage services backed by `EndpointSlice` — notably
KubeVirt-exposed workloads that do not publish classic `Endpoints`.
Notes:
- Upstream also replaced `replicas: {{ .Values.replicas }}` with a
hardcoded `replicas: 1` in `templates/deployment.yaml`. The
effective replica count is unchanged (we already set `1`), but the
value is no longer overridable via chart values. A minor cosmetic
reformat was applied to `templates/role.yaml`.
Closes#2256
### Release note
```release-note
chore(hetzner-robotlb): update robotlb to 0.0.6 — adds RBAC for EndpointSlices so services backed by EndpointSlice (e.g. KubeVirt) are supported.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Extended service account permissions to access Kubernetes endpoint
slices from the discovery API.
* **Bug Fixes**
* Deployment replica configuration now fixed to single instance.
* **Style**
* Improved YAML formatting in role template declarations.
* **Chores**
* Updated application version metadata to 0.0.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Refreshes the vendored Cilium chart in `packages/system/cilium` from
v1.19.1 to v1.19.3 via `make update`. Chart templates, values, CRDs and
the Cilium image reference are regenerated from upstream.
### Motivation
- **v1.19.2** ships a critical fix for cert-manager HTTP-01 Gateway API
challenges on hostnames that have both HTTP and HTTPS listeners
([cilium#44492](https://github.com/cilium/cilium/pull/44492), backport
[#44517](https://github.com/cilium/cilium/pull/44517)). Without this
fix, cert-manager cannot issue certificates via Gateway API when a
redirect HTTP listener and a TLS HTTPS listener share a hostname.
- **v1.19.3** is the latest stable patch release in the v1.19.x line (15
Apr 2026).
- This bump is a prerequisite for upcoming Gateway API work tracked
separately.
### Upstream changes pulled in
- Cilium Envoy bootstrap config, operator clusterrole, config template,
`values.schema.json` and the cilium-agent DaemonSet refreshed from
upstream.
- New `templates/ztunnel/` directory (DaemonSet, Secret, ServiceAccount)
added by upstream — not enabled by default in Cozystack values.
### Release note
```release-note
chore(cilium): bump to v1.19.3 (cert-manager HTTP-01 fix via cilium#44492)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added ztunnel encryption support with configurable deployment settings
* Added ConfigDriftDetection for monitoring ConfigMap changes
* Added endpoint policy update timeout configuration
* Added load balancer service topology support
* Extended Envoy circuit breaker configuration with connection and
request limits
* **Updates**
* Upgraded Cilium to v1.19.3
* Updated container images (Envoy, certgen, Hubble relay, clustermesh)
* Enhanced Cilium operator RBAC capabilities for managing ServiceImport
finalizers
* **Removals**
* Removed BIG TCP tunnel configuration option
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The cnpg admission webhook's controller pod passes readinessProbe as soon
as the local HTTPS server binds to :9443. The HelmRelease marks itself
Ready right after that via helm install --wait. But the Service
cnpg-webhook-service needs its EndpointSlice populated and the data plane
(kube-proxy / cilium) programmed before kube-apiserver can reach the
webhook through the Service ClusterIP. That gap is short but not zero,
and any HelmRelease that depends on postgres-operator (cozy-keycloak,
tenant Postgres apps) can fire its own install inside the window and hit
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
failed to call webhook: Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
which fails the install of the downstream release. Seen on cozystack/cozystack#2470
E2E run 24862782568.
Add a post-install,post-upgrade Helm hook Job that blocks the release
from reporting Ready until the webhook answers /readyz through the
apiserver service proxy. Apiserver proxy routes the call over the same
Service IP → EndpointSlice → pod path the admission webhook uses, so
once it responds, the webhook admission path is also working.
RBAC is minimal: a dedicated ServiceAccount with a ClusterRole that only
grants get on services/proxy scoped to https:cnpg-webhook-service:webhook-server.
The Job times out after 120s with 60 attempts at 2s intervals — longer
than any data-plane programming delay seen on E2E, but bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/extra/ingress/templates/cilium-lb-pool.yaml:19: if the operator
passes an externalIP already in CIDR form (192.0.2.10/32 or 2001:db8::1/128),
the template appended a second /32 or /128 suffix producing an invalid
CiliumLoadBalancerIPPool block. Guard the suffix append on the absence of
"/" in the input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add PackageSource for HAMi with dependency on gpu-operator.
Include HAMi in the iaas bundle as an optional package, allowing
host-level deployment alongside the existing kubernetes addon path.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The hami-dra subchart renders resources even when dra.enabled=false
because Helm dependency conditions don't work for vendored subcharts.
The subchart also contains duplicate YAML label keys
(app.kubernetes.io/component, app.kubernetes.io/name) and references
unpublished container images (v0.0.1-dev). DRA support can be
re-added when the upstream project stabilizes it.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Add generated-output dirs (pkg/generated, internal/crdinstall/manifests,
packages/system/*/definitions) and Makefile to the workflow paths: filter
so PRs that modify only generated artifacts still trigger the drift check.
- Mirror the same paths in the root pre-commit hook's files: regex so
manual edits to generated files or changes to the root generate target
re-run make generate through pre-commit.
- Switch drift detection in the workflow from `git diff --exit-code` to
`git status --porcelain` so new untracked files produced by make generate
(e.g. generated YAML/Go for a new API type) also fail the job; dump
`git diff --color=always` on failure for easier debugging.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Added comprehensive unit tests for authorization logic:
- hasAccessToNamespace() tests:
- User subject access (positive and negative cases)
- Group subject access
- ServiceAccount subject access
- ServiceAccount with empty namespace (defaults to RoleBinding ns)
- Privileged groups (system:masters, cozystack-cluster-admin)
- Get() handler tests:
- Returns namespace when user has access
- Returns NotFound when user lacks access (not Forbidden)
- Returns NotFound for non-tenant namespaces
All tests verify that authorization correctly enforces RoleBinding-based
access control and prevents IDOR vulnerabilities.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Adds explicit guidance to `docs/agents/contributing.md` about running
`make generate` in any touched package before committing. Pre-commit CI
runs `make generate` in every package and fails with exit code 123 on
any uncommitted generator output (regenerated `README.md`, reordered
`values.schema.json`, refreshed
`packages/system/<name>-rd/cozyrds/<name>.yaml`).
Recent PRs have tripped on this during review cycles. Documenting it in
the contributing checklist saves a round-trip.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Require regenerating and committing generated artifacts when
designated source files are edited.
* Added a section listing generated artifacts per package and concrete
regeneration/staging steps.
* Documented CI enforcement that detects unstaged generator output and
blocks PRs.
* Added guidance for rerunning generation after amended commits, plus
updated commit-scope guidance (now “not exhaustive”) and included
“agents” in allowed scopes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on docs/agents/contributing.md:11:
Scope linters kept flagging valid scopes like 'agents' as unknown because
the list read as exhaustive. Annotate it as examples (not exhaustive) and
add 'agents' to the Other group so both humans and review bots stop
tripping on scopes that are already in regular use across the repo
history.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:30:
Reword "likely to need regenerated" (regional construction) to
"likely needs to be regenerated" for standard technical prose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on docs/agents/contributing.md:26:
Replace the hard-coded packages/extra/<name> path in the example with
packages/<apps-or-extra>/<name> so the example matches the preceding
text that describes both apps and extra packages.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Service.spec.externalIPs is deprecated upstream in Kubernetes v1.36
(KEP-5707, kubernetes#137293). The AllowServiceExternalIPs feature gate
is expected to default to off around v1.40 and the implementation to
be removed around v1.43. For bare-metal installs that rely on
externalIPs today, cozystack needs a migration path.
This change adds an opt-in 'loadBalancer' exposure mode for the
ingress-nginx Service:
- New platform value 'publishing.exposure' (enum: externalIPs |
loadBalancer, default externalIPs). Plumbed through cozystack-values
into each tenant's ingress HelmRelease via the new 'expose-mode' key.
- Unknown values and loadBalancer with an empty externalIPs list fail
the chart render with explicit error messages, rather than silently
producing a broken Service.
- When exposure=loadBalancer and the current namespace matches
publishing.ingressName, the Service becomes type: LoadBalancer with
externalTrafficPolicy: Local.
- A new template renders a CiliumLoadBalancerIPPool whose blocks come
from publishing.externalIPs (IPv4 addresses get /32, IPv6 addresses
get /128) and whose serviceSelector uses Cilium's synthetic
io.kubernetes.service.namespace key combined with the standard
app.kubernetes.io/name: ingress-nginx label. No custom label is
written to the Service itself, avoiding cross-tenant collisions from
user-defined labels.
Default behaviour is unchanged: without opting in, the Service is
still ClusterIP + spec.externalIPs as today.
Scope: only ingress-nginx is migrated by this setting. Other cozystack
components that still write Service.spec.externalIPs directly (notably
the vpn app) must be migrated separately before the v1.40 feature gate
flip.
Tests: packages/extra/ingress/tests/exposure_test.yaml adds 13
helm-unittest cases covering both modes, IPv4/IPv6, empty-token
filtering, unknown-mode rejection, and the non-matching-namespace
fallback.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous image digest in values.yaml pointed at a single-arch
linux/arm64 manifest because 'make image' was run from an arm64 host
with the default buildx platform. Cozystack targets amd64 (Talos build
output, E2E runners, most real-world clusters) and also arm64 for
hybrid fleets, so Helm install would fail on amd64 nodes with 'no
matching manifest for linux/amd64 in the manifest list entries'
whenever somebody installed directly from this commit between merge
and the next release-tag CI rebuild.
Fix: rebuilt the image locally with
PLATFORM='linux/amd64,linux/arm64' make image from a buildx
docker-container driver, pushed the multi-arch manifest, and
refreshed values.yaml with:
- digest of the new multi-arch manifest list (verified via
'docker manifest inspect': amd64 sha256:e1977323..., arm64
sha256:8f5ab529...).
- tag bumped from 'latest' (emitted by the common-envs.mk settag
macro on a non-tagged checkout) to '1.19.3', matching the
established convention in every other packages/system/*/values.yaml
so reviewers and incident response have a human-readable version
anchor independent of digest chasing.
The Makefile is left untouched so the CI builder (which only uses the
default docker driver) keeps building single-arch for whatever
architecture it runs on; multi-arch is a responsibility of the
release-tag pipeline or an explicit local rebuild.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Built ghcr.io/cozystack/cozystack/cilium from the refreshed upstream
v1.19.3 base image and updated values.yaml with the new digest.
Previously values.yaml still pointed at the v1.19.1 cozystack rebuild
by digest while Chart.yaml and the Dockerfile were on v1.19.3 — with
chart default useDigest=true that would have silently pulled v1.19.1
until the next release-tag rebuild.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Vendored chart refreshed via make update in packages/system/cilium.
Motivation: v1.19.2 fixes a cert-manager HTTP-01 bug on hostnames with
both HTTP and HTTPS listeners (cilium#44492, backport PR #44517). This
is a prerequisite for upcoming Gateway API work.
v1.19.3 is the latest stable release in the v1.19.x line (15 Apr 2026).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pre-commit CI runs make generate in every package and fails with exit
123 on any uncommitted generator output. Add explicit guidance so
agents stage regenerated README.md, values.schema.json and
packages/system/<name>-rd artifacts alongside the hand edits.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pulls the latest robotlb chart (0.1.3) which ships robotlb 0.0.6.
The new appVersion adds RBAC permissions for discovery.k8s.io/endpointslices
needed to support EndpointSlice-based services such as KubeVirt.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
## What this PR does
Fixes the failures that blocked the v1.3.0 release pipeline (workflow
run [24765377017]) and closes the gap in `docs/agents/changelog.md` that
made them possible.
### Bug 1 — pathspec error in `Create changelog branch and commit`
After the Copilot step, the workflow runs `git checkout -b
"$CHANGELOG_BRANCH" origin/main`. Copilot CLI was invoked with
`--allow-all-tools` and had committed the generated changelog onto HEAD
of `main`, so the reset to `origin/main` deleted the tracked file, and
`git add "$CHANGELOG_FILE"` then failed with `fatal: pathspec ... did
not match any files`.
Fix (commit 2):
- Snapshot the file across the branch switch so the checkout cannot drop
it.
- `set -euo pipefail` so any failure surfaces loudly.
- Drop the dead "no changes to commit" soft-branch — the earlier
`check_changelog` step gates this step on the file being absent from
`origin/main`, so `git commit` must produce a diff; if it doesn't (e.g.
empty file), fail loud instead of pushing an empty branch.
- Fail-early file-existence check with `::error::` annotation, and
`VERSION` moved to step `env:`.
### Bug 2 — no timeout on `Generate changelog using AI`
The re-run hung in Copilot for 10+ minutes with zero log output. With no
`timeout-minutes`, a hung Copilot would hold a self-hosted runner for 6
hours (job default).
Fix (commit 2): `timeout-minutes: 30` (the prior successful run took ~26
minutes).
### Root-cause fix — scope the agent prompt
`docs/agents/changelog.md` is the actual prompt driving Copilot. It
ended with "Save the changelog" and gave no boundary, so an agent with
`--allow-all-tools` could reasonably interpret "done" as "commit, push,
open a PR".
Fix (commit 1):
- Add a "Scope and boundaries" section at the top stating the single
deliverable is `docs/changelogs/v<version>.md` and enumerating forbidden
operations (git commit / push / branch / tag / reset / merge / rebase,
PR creation, GitHub API writes, modifying any file other than the
changelog).
- Add an explicit "then exit" at the end of Step 9 with a back-reference
to the scope section.
- Scoped "unless the caller explicitly instructs otherwise" so
interactive IDE use stays flexible.
With the rules in the doc, CI and interactive callers share the same
boundary, and the workflow prompt becomes a one-line invocation
(`--prompt "Generate the release changelog for tag v${VERSION}. Follow
the instructions in @docs/agents/changelog.md exactly, including the
'Scope and boundaries' section at the top. ..."`).
### Files touched
- `docs/agents/changelog.md` — +16 / -0 (scope section + exit rule)
- `.github/workflows/tags.yaml` — +36 / -31 (timeout + rewritten commit
step + simplified prompt)
Other jobs in the workflow (`prepare-release`, `update-website-docs`)
are untouched.
### Release note
```release-note
NONE
```
[24765377017]:
https://github.com/cozystack/cozystack/actions/runs/24765377017
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved release automation reliability with stricter verification,
enforced error handling, and a timeout for changelog generation
* Ensured automated commits/pushes occur only after successful output
validation to prevent accidental repository mutations
* **Documentation**
* Clarified agent instructions to produce a single changelog file and
terminate, and to restrict the agent to read-only inspection during
generation
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
hack/update-codegen.sh sources kube_codegen.sh from the Go module
cache at ~/go/pkg/mod/k8s.io/code-generator@vX.Y.Z/, but the module
is not declared in go.mod so a fresh runner has nothing to source
from. Add a workflow step that parses the pinned version out of the
script and pulls the module into the cache before running
make generate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Post-release cleanup of `docs/changelogs/v1.3.0.md` so the notes match
what users actually experience in v1.3.0. No code changes.
- **Rewrite the postgres major-features entry** so author
(`@myasnikovdaniil`), PR (`#2369`), and description all line up with the
`17.7-standard-trixie` pin + migration-37 `imageName` rewrite that
actually shipped. The previous entry credited `#2304` with a description
matching a superseded `spec.version=v17` backfill approach.
- **Remove the duplicate `#2364` postgres bug-fix entry** — the same
work is now folded into the single major-features entry above, with
backport references to `#2309` (v1.2.1) and `#2364` (v1.2.2).
- **Remove the `[linstor-gui] Restrict to cozystack-cluster-admin group`
security entry.** The vulnerable state never shipped in a tagged
release, so there is nothing user-facing to announce. The
`cozystack-cluster-admin`-group restriction is already described in the
linstor-gui Feature Highlights section as part of the feature's day-one
shipping behavior.
### Release note
```release-note
[]
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated v1.3.0 changelog with clarified PostgreSQL system version
pinning details and removed redundant entries for improved documentation
clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The etcd chart shipped a `post-upgrade` Helm hook that `kubectl delete`d the
etcd TLS chain (`etcd-{ca,peer-ca,client,peer,server}-tls`) and then deleted
etcd pods on every chart upgrade, gated by a semver compare of an
`etcd-deployed-version` ConfigMap against `2.6.1`.
The hook was added as a one-shot migration for the chart `2.6.0 -> 2.6.1`
transition. Since commit f871fbdb ("Remove versions_map logic") all chart
versions are stamped as `0.0.0+<git-hash>`, which per semver is always
`< 2.6.1`. The gate therefore always resolves to "update certs", firing the
destructive hook on every etcd upgrade. On clusters running Kamaji-managed
tenant control planes this wipes the etcd CA, cert-manager re-issues it, and
tenant kube-apiservers hit `x509: certificate signed by unknown authority`
against `etcd.<ns>.svc:2379` until each tenant DataStore is manually
re-reconciled.
Commit 47d81f70 ("Disabled private key rotation in CA certs") already fixed
the underlying `rotationPolicy: Always` issue the migration was papering
over, so the hook has no remaining use. Remove the hook Job, its RBAC, the
version ConfigMap it read, and add a helm-unittest suite under
`packages/extra/etcd/tests/` that guards against re-introducing the hook
or the version ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the Generate changelog using AI step ran
Copilot with --allow-all-tools and GH_TOKEN set to the write-capable
installation token issued to the job (contents: write,
pull-requests: write on all cozystack/* repos). The scope rules in
docs/agents/changelog.md and the step prompt tell the agent not to
use those permissions, but nothing at the token layer prevented it.
Mint a second, read-only installation token from the same app
(same COZYSTACK_CI_APP_ID / COZYSTACK_CI_PRIVATE_KEY, scoped to
contents/pull-requests/metadata read) and pass that one to the AI
step instead. The write-capable token is still used by the checkout,
commit/push, and PR-creation steps that actually need it.
This is defense in depth: even if a future prompt change or agent
misbehavior ignored the scope rules, the token itself has no write
capability on any repository in the cozystack org. No new secret,
no new GitHub App install, no admin-side change — the RO token is
minted in the same workflow from the same app credentials.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the existing `[ -f ]` check catches
missing files but a zero-byte `docs/changelogs/v${VERSION}.md` would
still be staged and committed — `git add` + `git commit -s` on a new
empty file succeeds and produces a real commit, leaving the
downstream PR with no actual changelog content.
Add a `[ -s ]` guard after the existence check: if the Generate
changelog using AI step produces an empty file, emit a matching
`::error::` annotation and exit 1 before snapshotting.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the previous "do not write to refs, HEAD,
or remotes" wording contradicted the explicit allowance of `git fetch`
(which updates remote-tracking refs) and the mandatory cross-repo
checks in Step 6, which `cd` into `_repos/<repo>` and run
`git checkout`, `git pull`, etc.
Sharpen the scope paragraph:
- The git-write ban is now explicitly scoped to the cozystack working
tree — it bans writing to local branches, tags, or HEAD in that
repo, not "refs/HEAD/remotes" globally.
- `git fetch` is called out as expected.
- Local git operations inside disposable `_repos/` clones
(`git checkout`, `git pull`, etc.) are explicitly allowed, with
the remaining rules (no push, no PR creation, no API writes)
applying to any repository.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `backupstrategy-controller` chart declared a `Role` and
`RoleBinding` scoped to the `cozy-velero` namespace (for managing
`ResourceModifier` ConfigMaps consumed by Velero Restore). Because
`cozystack.velero` is an optional package, that namespace does not exist
in bundles that do not enable velero — and `backupstrategy-controller`
is a **default** package. Helm install aborted with:
```
namespaces "cozy-velero" not found
```
which blocked the entire
`cozy-backup-controller/backupstrategy-controller` HelmRelease on any
cluster where velero was not explicitly enabled (including the E2E
environment).
This PR moves that Role/RoleBinding into the velero chart
(`packages/system/velero/templates/backupstrategy-controller-rbac.yaml`),
so the permission grant only exists when velero is actually installed —
where it is useful. The RoleBinding subject points to the stable
`backupstrategy-controller` ServiceAccount in `cozy-backup-controller`.
### Release note
```release-note
fix(backups): moved the velero-namespaced ResourceModifier ConfigMap Role and RoleBinding from the backupstrategy-controller chart into the velero chart. This unblocks installs of backupstrategy-controller on bundles that do not enable velero (previously the HelmRelease failed with `namespaces "cozy-velero" not found`).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Reorganized RBAC configuration for the backup strategy controller by
consolidating namespace-scoped role definitions in the Velero chart
template
* Updated role bindings and permissions structure across system packages
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The backupstrategy-controller chart declared a Role/RoleBinding in the
cozy-velero namespace for ResourceModifier ConfigMap management. Because
velero is an optional package, that namespace does not exist in bundles
without velero, so Helm install aborted with "namespaces \"cozy-velero\"
not found" and blocked the default install of backupstrategy-controller.
Move the Role and RoleBinding into the velero chart so they are created
only when velero is actually installed. The RoleBinding subject points
to the backupstrategy-controller ServiceAccount in its fixed namespace
(cozy-backup-controller).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes to the generate-changelog job to fix the v1.3.0
release pipeline failure (run 24765377017) and make the job robust
to whatever state the Copilot step leaves behind.
1. Add `timeout-minutes: 30` to the Generate changelog using AI
step. On the v1.3.0 re-run the step hung silently for 10+
minutes; with no timeout a hung Copilot would hold a self-hosted
runner for up to 6 hours (job default). The previous successful
run took ~26 minutes, so 30 is a reasonable ceiling.
2. Replace the terse, ambiguous Copilot prompt with a one-liner
that invokes docs/agents/changelog.md directly. The "Scope and
boundaries" section added to that doc in the previous commit is
now the single source of truth for what the agent may and may
not do, so the workflow only needs to pass the version and
point at the relevant doc. VERSION is moved to step env: to
match GitHub's workflow-injection hardening guidance.
3. Rewrite the Create changelog branch and commit step:
- add `set -euo pipefail` so any failure is visible
- validate the file exists up front and fail loud with
`::error::` if not
- copy the file to a tempfile BEFORE `git checkout -b`, so the
checkout to `origin/main` cannot remove it (this is the fix
for the original pathspec error the v1.3.0 run hit)
- use `trap` to clean up the tempfile on any exit path
- move VERSION to env
- drop the dead "no changes to commit" branch: the check_changelog
step earlier in the job gates this step on the file being
absent from origin/main, so `git add` + `git commit` must
produce a diff. If they don't (e.g. Copilot emitted an empty
file), fail loud instead of pushing an empty branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The v1.3.0 release pipeline broke because Copilot, invoked by
.github/workflows/tags.yaml with --allow-all-tools, committed the
generated changelog onto HEAD of main on its own. The workflow's
next step — `git checkout -b ... origin/main` — then wiped the file,
and `git add` failed with a pathspec error.
The root cause is in this document. The checklist ends with "Save
the changelog", which an agent with broad tool access can reasonably
interpret as "also commit it, push it, and open a PR". There was no
explicit boundary.
Add a "Scope and boundaries" section at the top and an explicit
"then exit" at the end of Step 9:
- The single deliverable is docs/changelogs/v<version>.md.
- Forbidden by default: git commit / push / checkout (to switch
branches) / branch / tag / reset / merge / rebase; PR creation;
GitHub API writes (POST/PATCH/DELETE); modifying any file other
than the changelog.
- Read-only analysis (git log/show/fetch/diff, gh pr view, gh api
GET) remains expected.
- Auxiliary repo clones under _repos/ remain allowed for cross-repo
analysis per Step 6.
- Scoped "unless the caller explicitly instructs otherwise" so
interactive use with an IDE remains flexible.
With the rules in the doc, CI and interactive callers share the
same boundary; the workflow can invoke the doc with a one-line
prompt instead of re-stating the constraints every time.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Post-release cleanup of docs/changelogs/v1.3.0.md so the notes match
what users actually experience in the released v1.3.0:
- Rewrite the postgres major-features entry so author (myasnikovdaniil),
PR (#2369), and description all match the 17.7-standard-trixie pin +
migration-37 imageName rewrite that actually shipped. The previous
entry credited #2304 (superseded spec.version=v17 backfill approach).
- Remove the duplicate #2364 postgres bug-fix entry; the same work is
now folded into the single major-features entry above, with backport
references to #2309 (v1.2.1) and #2364 (v1.2.2).
- Remove the [linstor-gui] Restrict to cozystack-cluster-admin group
security entry. The vulnerable state never shipped in a tagged
release, so there is nothing user-facing to announce; the restriction
is already described in the linstor-gui Feature Highlights section
as part of the feature's day-one behavior.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This PR adds the changelog for release `v1.3.0`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.3.0.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Published Cozystack v1.3.0 release notes featuring LINSTOR scheduler
extender for storage-aware pod placement, managed LINSTOR GUI web
console, VM Default Images catalog, expanded observability with Events
dashboard and S3 metering, cross-namespace VMInstance backup/restore,
and various platform enhancements and bug fixes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
HAMi-core relies on _dl_sym (private glibc symbol removed in 2.34)
for CUDA interception. This breaks compute isolation on modern
container images using Ubuntu 22.04+ and makes Alpine/musl completely
incompatible.
Include upstream issue references and a compatibility matrix so users
can make informed decisions about base image selection.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add HAMi HelmRelease to the kubernetes app as a toggleable addon.
When enabled, GPU Operator's built-in device plugin is automatically
disabled via a values merge to avoid conflicts with HAMi's own
device plugin.
The HelmRelease fails fast if GPU Operator is not enabled, since HAMi
depends on it for driver management and container toolkit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add HAMi v2.8.1 (CNCF Sandbox) as a new system package for fractional
GPU sharing in tenant Kubernetes clusters. The chart enables workloads
to request specific amounts of GPU memory and compute cores instead of
claiming entire GPUs.
Vendored upstream chart includes device plugin, scheduler extender,
mutating webhook, and DRA subchart. Wrapper values configure nvidia
RuntimeClass and clear hardcoded node config from upstream defaults.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Harbor is a PaaS service (`category: PaaS`), not a tenant module. It is
not deployed automatically into tenant namespaces — there is no
corresponding manifest in `packages/apps/tenant/templates/`, unlike
actual tenant modules (monitoring, ingress, etcd, info, seaweedfs).
Two flags were incorrectly set on its `ApplicationDefinition`:
- `spec.dashboard.module: true` — caused Harbor to appear in the sidebar
"Modules" section and be excluded from its proper PaaS category.
- `spec.release.labels."internal.cozystack.io/tenantmodule": "true"` —
caused controllers (`pkg/registry/core/tenantmodule`, dashboard) to
treat the Harbor HelmRelease as a tenant module.
Both flags are removed so Harbor is listed correctly under PaaS and
handled as an ordinary managed application.
### Release note
```release-note
fix(harbor): remove incorrect tenant module flags from Harbor ApplicationDefinition so it appears under the PaaS category in the dashboard and is no longer treated as a tenant module
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Simplified internal application configuration settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps `packages/system/kubeovn` to `cozystack/kubeovn-chart` tag
`v1.15.10-cozy.1`, which:
1. Updates upstream kube-ovn from v1.15.3 to v1.15.10 (latest patch in
the v1.15 series).
2. Carries a patch over `pkg/controller/pod.go` that preserves a VM
LSP's port-group memberships when kubernetes GCs a completed
virt-launcher pod while another virt-launcher pod of the same VM is
still running.
Without the patch, the destination pod of a successful live migration
loses its security groups, network policies and node-scoped routing
after kubernetes cleans up the source pod, and only recovers after a
`kube-ovn-controller` restart. The buggy code path is identical between
v1.15.3, v1.15.10, release-1.15 HEAD and master HEAD — confirmed by diff
and by reproduction on a cozystack cluster.
- Upstream issue: kubeovn/kube-ovn#6665
- Upstream fix PR: kubeovn/kube-ovn#6666
- Chart carry: cozystack/kubeovn-chart#4
### Release note
```release-note
fix(kube-ovn): bump kube-ovn to v1.15.10 and carry upstream fix that keeps VM LSP port-group memberships when kubernetes GCs a completed virt-launcher pod (post-migration regression)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VpcEgressGateway CRD extended with optional resources
(claims/limits/requests) and bandwidth (ingress/egress) fields.
* Added a pre-upgrade hook to run compatibility steps before upgrades.
* CNI DaemonSet now optionally respects MTU when configured.
* **Bug Fixes**
* Updated Kube-OVN and NAT gateway container images to the latest
maintenance release.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Fixes an RBAC failure in `kube-ovn-plunger` that prevented it from
reconciling the `ovn-central` Deployment:
```
deployments.apps is forbidden: User "system:serviceaccount:cozy-kubeovn:kube-ovn-plunger"
cannot list resource "deployments" in API group "apps" at the cluster scope
```
Two causes addressed:
- The controller-runtime cache issued cluster-wide list/watch for
Deployments and Pods by default. The manager cache is now scoped to the
kube-ovn namespace, so requests go to the namespaced endpoints covered
by the Role.
- The Role's deployments rule used `resourceNames`, which does not apply
to list/watch in Kubernetes RBAC. The restriction is removed; the Role
is still namespace-scoped via RoleBinding to the `cozy-kubeovn`
namespace.
`--kube-ovn-namespace` and `--ovn-central-name` are now passed to the
binary from chart values so the previously unused value is wired up.
### Release note
```release-note
fix(kube-ovn): fix kubeovn-plunger RBAC forbidden error on ovn-central Deployment reconcile
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable parameters for KubeOVN namespace and OVN central
name in deployment configuration.
* **Improvements**
* Enhanced cache configuration with namespace-scoped resource management
capabilities.
* Updated RBAC permissions to remove resource-specific constraints,
enabling broader deployment access.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Options field was added to RestoreJobSpec without re-running
'make generate', leaving zz_generated.deepcopy.go out of sync.
Regenerated to include the missing DeepCopyInto handling for the
runtime.RawExtension pointer.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Run root 'make generate' as a pre-commit hook and as a dedicated
CI workflow so missed codegen updates (CRDs, deepcopy, clients, RBAC)
are caught instead of merging stale generated files.
Pre-commit hook is scoped to files that actually affect codegen
(api/, pkg/apis/, hack/update-codegen.sh, hack/boilerplate.go.txt)
so unrelated commits are not slowed down. CI job sets up Go from
go.mod, runs make generate, and fails on drift with a pointer to
the local fix.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Harbor is a PaaS service, not a tenant module. It is not deployed
automatically into tenant namespaces (no manifest in
packages/apps/tenant/templates/). Remove the misplaced
`dashboard.module: true` flag and
`internal.cozystack.io/tenantmodule: "true"` release label so Harbor
appears under the PaaS category in the dashboard and is not treated as
a tenant module by the controllers.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Pulls cozystack/kubeovn-chart v1.15.10-cozy.1, which bumps upstream
kube-ovn from v1.15.3 to v1.15.10 and carries a patch over
pkg/controller/pod.go that preserves a VM LSP's port-group memberships
when kubernetes GCs a completed virt-launcher pod while another
virt-launcher pod of the same VM is still running.
Without the patch, the destination pod of a successful live migration
loses its security groups, network policies and node-scoped routing
after kubernetes cleans up the migration source pod, and only recovers
after a kube-ovn-controller restart.
Upstream issue: kubeovn/kube-ovn#6665
Upstream fix PR: kubeovn/kube-ovn#6666
Chart carry: cozystack/kubeovn-chart#4
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The kubeovn-plunger controller-runtime cache attempted cluster-wide
list/watch on Deployments and Pods, which the namespace-scoped Role
cannot satisfy. Additionally, the deployments rule relied on
resourceNames, which does not restrict list/watch verbs and left the
permission effectively unusable.
Scope the manager cache to the kube-ovn namespace so list/watch hit
the namespaced API, and drop resourceNames from the deployments rule.
Wire --kube-ovn-namespace and --ovn-central-name through the
Deployment so values are actually consumed.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Update oldName and serverName field descriptions based on code review
feedback to avoid confusion about their actual roles:
- oldName: Remove misleading "(matches serverName in backup.info)"
text. This field represents the Kubernetes cluster resource name,
not the Barman server name.
- serverName: Provide clearer explanation that it's the S3 path prefix
(barmanObjectStore.serverName) used by the original cluster, and should
only be set when it differs from the Kubernetes resource name.
Updated in:
- values.yaml (source of truth for field documentation)
- types.go (Go API type comments)
- values.schema.json (JSON schema for validation)
- postgres.yaml (CRD with embedded OpenAPI schema)
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add serverName field to bootstrap configuration to explicitly specify
Barman server name from backup.info. This fixes "no target backup found"
errors when server_name in backup.info differs from Kubernetes cluster name.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
- Increase memory limit from 500Mi to 512Mi
- Increase memory request from 100Mi to 256Mi
- Add startup probe with 60s timeout (12 attempts × 5s)
- Increase readiness/liveness initialDelaySeconds from 5/15 to 30s
This fixes OOMKilled crashes observed in production where kamaji
controller was being killed due to insufficient memory during startup.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The /1e9 divisor in gpu:{power,thermal}_throttle_fraction:rate5m was
derived empirically against DCGM 3.x on A10 — the counter documents
itself as microseconds but ticks in nanoseconds in practice. If a
future exporter release honors the documented units, pre-clamp values
would exceed 1.0 while clamp_max(..., 1) silently masks the drift,
plateauing every throttle fraction at 100% and making the panels
lie in unison.
Add a validation group that fires when the raw max/1e9 value exceeds
1.0 for 15m, so we notice and rescale to /1e6 before dashboards
silently mislead operators.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
pod:util_per_watt:avg5m divided two DCGM metrics without an explicit
on(...) clause, so the match used the intersection of their label
sets. If dcgm-exporter relabeling ever diverges between
DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_POWER_USAGE (e.g. a pod-mapping
label appears on one but not the other after a config change), the
entire result drops to empty silently. Pin the match to the labels we
group by so divergence becomes a missing side, not a missing rule.
Throttle fractions had a related shape problem: dcgm-exporter emits
one series per GPU for each pod-mapping combination. On a shared GPU
(restart races, MIG/MPS) the same physical counter appears under
multiple pod labels and downstream avg(...) panels get diluted by the
pod count. Fold duplicates with max by (Hostname, gpu, UUID) before
clamp_max so the fraction is tied to the physical GPU.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
kube-state-metrics keeps kube_pod_container_resource_requests series
for Failed/Succeeded pods until the apiserver garbage-collects them,
which could inflate :allocated beyond what tenants actually hold and
drive cluster:gpu_count:free negative.
Join the request metric against kube_pod_status_phase filtered to
Pending|Running — the canonical pattern from Kubernetes' own
container_resource recording rules — on both the cluster and namespace
aggregates. Add clamp_min(..., 0) on cluster:gpu_count:free as a
second line of defence against transient label drift between
kube-state-metrics and DCGM.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Drop the hardcoded metadata.namespace so the rule inherits the chart's
release namespace, and add an explicit empty params field on every
group for schema consistency. No behavior change.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The Pending GPU pods counter on gpu-quotas joined raw
kube_pod_container_resource_requests (per-container series) against
kube_pod_status_phase (per-pod series). Multi-container pods were
counted once per requesting container instead of once per pod, so the
widget over-reported whenever a Pending pod had more than one GPU
container. Collapse the requests to pod level before the join.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Pod-level panels on the efficiency dashboard and DCGM-level panels on
the performance dashboard ignored the $namespace template variable, so
changing it left the visualizations unchanged. Add the filter to each
query. Performance-side queries use the `$namespace|` empty-tolerant
form so host-level DCGM series without a namespace label remain
visible when a specific namespace is selected.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Explain why tolerations: [{operator: Exists}] is safe on the driver
compat DaemonSet: the nodeSelector already confines scheduling to GPU
nodes, so the blanket toleration only kicks in when those nodes carry
the dedicated=gpu / nvidia.com/gpu taints that the GPU Operator's
default policy and many deployments apply.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Clarify that the "Average utilization" panel on gpu-fleet reflects the
legacy NVML view (DCGM_FI_DEV_GPU_UTIL) rather than engine-active
profiling. For AI/LLM workloads the NVML number is optimistic; the
gpu-efficiency dashboard carries the profiling-based view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
gpu-fleet.json references DCGM_FI_DEV_POWER_MGMT_LIMIT for its
"TDP vs draw" panel, but the custom DCGM Exporter CSV did not declare
it, so the panel silently rendered "No data" on clusters using that
config. Declare the counter, fix the dashboards table in the
gpu-operator examples README, and add a bats test that cross-checks
every DCGM_FI_* reference in tracked dashboards and recording rules
against the union of the upstream default set (snapshotted under
hack/) and the project's custom CSV.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- namespace:gpu_count:sum — never consumed by any tracked dashboard;
the billable view is already covered by namespace:gpu_count:allocated,
and the admin view by cluster:gpu_count:allocated.
- namespace:energy_joules:sum — no panel integrates joules; kWh
readings on the tenant dashboard compute their own integrations
from namespace:power_watts:sum.
- pod:tensor_to_nvml_ratio:avg5m — interesting tenant signal in
theory, but not wired into any panel and carrying it just burns
cardinality on large fleets.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-fleet: cluster-wide admin view — inventory, capacity (total /
allocated / free), per-node utilization and power, throttling,
temperatures, XID errors.
- gpu-tenants: per-namespace view — live allocation, utilization,
tensor saturation, power, and 24h GPU-hours / kWh integrations for
billing inputs.
Register both under gpu/* in dashboards-infra.list so they ship as
GrafanaDashboard CRs and fall under the bats cross-check introduced
earlier on this branch.
Update examples/README to spell out which DCGM counters each of the
five gpu/* dashboards actually needs on top of the upstream default
CSV — gpu-performance needs profiling and throttling counters,
gpu-efficiency needs profiling, gpu-tenants needs only
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet
and gpu-quotas work on the default counter set alone.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align namespace:gpu_count:allocated with every other namespace:* rule
by filtering out cozy-*/kube-*. All other per-namespace rules
(gpu_util, tensor_active, fb_used_bytes, power_watts) already exclude
system namespaces, so the label set produced by :allocated diverged
from them — any dashboard variable or join that reads across these
rules could end up with a different namespace list depending on which
rule supplied the :allocated column.
Trade-off: per-namespace GPU accounting for system workloads is no
longer available through this rule. If it's ever needed, add a
dedicated system:gpu_count:allocated rather than widening this one —
the "billable tenant view" invariant is what the filter is protecting.
Cluster-level cluster:gpu_count:allocated intentionally keeps system
pods so it stays aligned with cluster:gpu_count:total and
cluster:gpu_count:free remains meaningful. As a consequence,
sum(namespace:gpu_count:allocated) no longer equals
cluster:gpu_count:allocated; the delta is system-pod GPU usage, which
is fine for the cluster-admin view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Catch dangling references at PR time: every recording-rule name used
inside a tracked GPU dashboard must exist in
packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml. The
first iteration of gpu-efficiency.json shipped panels keyed on
pod:tensor_saturation:avg5m without the rule defined; the test fails
on exactly that class of bug.
Scoped to dashboards listed under gpu/* in dashboards-infra.list, so
untracked drafts stay out of scope until they are registered. Reverse
direction (rule defined but unused) is intentionally NOT enforced —
some rules exist for ad-hoc PromQL or upcoming dashboards.
Auto-discovered by make bats-unit-tests via hack/cozytest.sh.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The previous wording implied that the entire custom DCGM CSV was
required by the recording rules. In fact only the profiling counters
(DCGM_FI_PROF_*) need to be added on top of the upstream defaults —
everything else the rules consume is already in default-counters.csv.
Add a Verification status block flagging that the minimum-set claim is
derived from the DCGM Exporter version pinned in the currently shipped
gpu-operator package and must be re-checked when that package moves to
a newer release.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Restrict the nvidia-driver-compat DaemonSet to nodes labelled
nvidia.com/gpu.present=true (NFD/GPU Operator label). Without the
nodeSelector it was scheduling onto every node — control-plane and
CPU-only workers included — burning a privileged pod slot per host
for no benefit.
Add resource requests and limits to the init and pause containers so
the DaemonSet stays within control-plane budget on small clusters.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Drop gpu.recording.30s group: per-GPU 30s aggregates had no consumers
in tracked dashboards, only burned cardinality.
- Drop namespace:gpu_allocated_count:gauge: identical expression to
namespace:gpu_count:sum under a different name.
- Reground :allocated on kube_pod_container_resource_requests so it
reflects what tenants requested (Pending+Running) rather than what
DCGM currently sees. namespace:gpu_count:sum stays DCGM-based and
represents actually-running pods; the gap between the two is the
signal admins want.
- Add namespace:gpu_count:allocated as the per-namespace counterpart.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Strip Grafana export boilerplate (__inputs, __elements, __requires,
default annotations, embedded datasource inputs) and tighten panel
layouts across the three GPU dashboards. All three continue to use
the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Revise gpu-performance and add two new dashboards, registered in
dashboards-infra.list:
- gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity
and workload efficiency signals.
- gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs.
used GPUs for tenant capacity planning.
All three dashboards use the $ds_prometheus template variable, per
the project convention.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add reference manifests (not templates) under
packages/system/gpu-operator/examples/ documenting one working
configuration for running CUDA workloads directly in pods on a Talos
cluster, with DCGM metrics that drive the gpu/gpu-performance
dashboard.
- values-native-talos.yaml: Cozystack Package values that disable the
sandbox path, enable the device plugin, and wire DCGM to the custom
metrics ConfigMap.
- dcgm-custom-metrics.yaml: ConfigMap extending the default DCGM CSV
with profiling, ECC, throttling and energy counters used by the
dashboard and recording rules.
- nvidia-driver-compat.yaml: DaemonSet that stages libnvidia-ml.so.1
and nvidia-smi from the Talos glibc tree into a location the
gpu-operator validator inspects. Workaround for
NVIDIA/gpu-operator#1687.
- README.md: explains why these are shipped as references rather than
first-class templates (sandbox vs native is a deployment choice),
and how the pieces connect.
The out-of-the-box values-talos.yaml still targets the sandbox (VFIO
passthrough) scenario. Operators who want native pod GPU workloads can
start from these references.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add the gpu/gpu-performance dashboard and register it in the infra
dashboard list. The dashboard provides:
- Cluster overview: total/allocated GPUs, average utilization,
aggregate power draw.
- Utilization: GPU util (NVML), tensor pipe active (realistic load
for LLM/AI workloads), graphics engine active, memory copy util.
- Memory: VRAM used/free per GPU.
- Power and temperature per GPU.
- Health: XID errors, power and thermal throttling.
The dashboard relies on DCGM_FI_* metrics plus the cluster:gpu_* and
namespace:gpu_* recording rules added to monitoring-agents.
The JSON follows the cozystack convention — Prometheus data source is
selected via the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add VMRule with recording rules for DCGM metrics at three levels:
- gpu.recording.30s: per-GPU aggregates over 30s windows
- gpu.recording.cluster.1m: cluster-wide totals for overview panels
- gpu.recording.namespace.1m: per-namespace aggregates for tenant
reporting and GPU-hour calculations
The rules are safe to ship on clusters without DCGM — they evaluate to
empty series when no matching metrics are scraped.
Used by dashboards/gpu/gpu-performance.json.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The test body asserts .status.history[].status extraction, but the
test name still referenced the old installFailures counter (leftover
from when the guard used that field before switching to status.history
to avoid ClearFailures zeroing the counters on successful reconcile).
Address review feedback from coderabbitai on
hack/remediation-guard.bats:84: rename so grep for what the test
actually pins matches the test name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Log .status.history regardless of content so a silently empty result
(Flux field rename) shows up in CI logs, and treat empty history on
a Ready HelmRelease as a distinct failure. A Ready HR by definition
has at least one snapshot; anything else is a shape-drift signal.
- Replace the unquoted heredoc in remediation-guard.sh with a printf |
grep pipeline. printf %s treats statuses as literal payload (no $
expansion surprises for future callers), grep --quiet --extended-regexp
returns exit status the caller can forward directly.
- Share the etcd-absent values file between both invariant tests
(packages/apps/kubernetes/tests/values-ci-no-etcd.yaml) instead of
duplicating the --set block.
- Fix typo "override applied" -> "override is applied" in the
Kubernetes ApplicationDefinition.
- Add a coupling comment in the ApplicationDefinition annotation that
points at the wait-for-kubeconfig init deadline in _helpers.tpl, so
a future operator raising the HR timeout updates the init deadline
too.
- Clarify the per-annotation timeout comment in rest.go so it stops
implying the feature is Kubernetes-only (it is not - only today's
one user is).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pulls the release.cozystack.io/helm-install-timeout parsing out of
start.go into ParseHelmInstallTimeoutAnnotation in pkg/config. The
helper rejects units that time.ParseDuration accepts but Flux
helm-controller rejects (ns, us, µs): feeding one of those through
would cause the HelmRelease admission webhook to reject the object at
install time, giving a silent drop to flux defaults that is hard to
debug. Fail loudly at cozystack-api startup instead.
Adds a table-driven unit test covering: unset (empty), accepted units
ms/s/m/h, compound 2h30m, decimal 1.5m, and the rejected shapes
(bare digits, garbage, negative, ns/us/µs). The test lives in
pkg/config so it runs under the existing go-unit-tests make target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
17 child HelmReleases (cilium, coredns, csi, cert-manager,
metrics-server, ...) referenced *-admin-kubeconfig via
kubeConfig.secretRef and rendered even when _namespace.etcd was empty.
On an etcd-less tenant each one sat in NotReady forever because the
admin-kubeconfig Secret only exists after a KamajiControlPlane
reconciles, and KamajiControlPlane now only renders when etcd is set.
The outcome contradicted the "beacon only" contract claimed in the
soft-skip commit.
Extend the existing addon guards to also require _namespace.etcd, and
wrap the four unconditional HelmReleases (csi, metrics-server,
prometheus-operator-crds, volumesnapshot-crd) plus the always-on
cilium/coredns HR resources in the same gate. Add an invariant bats
test that renders the whole chart with etcd empty and asserts zero
HelmReleases reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The image-busybox Makefile target + images/busybox/Dockerfile wrapper
just rebuilt an upstream busybox digest as ghcr.io/cozystack/cozystack/
busybox. Payload is a one-shot sh loop run once per pod; the pinned
upstream digest is already immutable, so maintaining a private mirror
adds churn (rebuild on every release) for no real hardening benefit.
Delete the wrapper and reference docker.io/library/busybox:<digest>
directly from images/busybox.tag. Document the choice in _helpers.tpl.
Also drop the false coupling in the go table test: the "unrelated
kind without configured timeout" case used the real kind name Qdrant,
which tied the test to the Qdrant ApplicationDefinition for no reason.
Switch to a clearly fictional kind so a future Qdrant timeout override
does not break this assertion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
CI runs make unit-tests on every PR, which already covers helm
unittests and bats (hack/admin-kubeconfig-invariant.bats and
hack/remediation-guard.bats are both picked up by the existing
hack/*.bats glob). What was missing was any go test invocation.
Add a go-unit-tests target scoped to pkg/registry, pkg/config, and
pkg/cmd/server - the cozystack-api surface this repo actually owns
and tests in-tree. Running go test ./... pulls in generated-code
round-trip suites whose behavior is governed by generator tool
versions outside this repo's control; those are better exercised
from their own generator workflows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Flux helm-controller's ClearFailures() zeroes installFailures and
upgradeFailures on every successful reconciliation (see the upstream
HelmReleaseStatus method). The previous guard ran after the HelmRelease
was Ready, at which point the counters were always 0 - the assertion
was vacuous and would have passed against a reverted fix.
Switch to .status.history, which retains per-revision release
Snapshots that survive a subsequent successful reconciliation. A
remediation cycle leaves behind a Snapshot with status=uninstalled
(the install-remediation code path) or status=failed (Helm release
failure that remediation then uninstalled). Either one signals the
race actually fired.
Rewrite the bats unit tests to cover: empty history, deployed-only,
deployed+superseded (happy path - not detected), single failed,
single uninstalled, uninstalled-then-deployed, and deployed-then-failed
(all detected). The pinned-shape test feeds a realistic HR status
snippet through yq the same way run-kubernetes.sh does via kubectl
-o jsonpath.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The soft-skip wrap in cluster.yaml only silenced Cluster and
KamajiControlPlane rendering when _namespace.etcd is empty. The three
CP-side Deployments (cluster-autoscaler, kccm, kcsi-controller) still
rendered, their wait-for-kubeconfig init containers CrashLoopBackOff'd
forever (no KamajiControlPlane = no admin-kubeconfig Secret),
HelmRelease hit its 15m wait timeout and triggered the very install
remediation cycle the rest of this PR prevents. Self-contradiction.
Wrap each of the three Deployment templates in
{{- if .Values._namespace.etcd }}...{{- end }} so they render only
when there is a DataStore to back them. Add an invariant bats test
that renders the whole chart with etcd empty and asserts zero
Deployments reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
run-kubernetes.sh extracts .status.installFailures and
.status.upgradeFailures via kubectl -o jsonpath. If a future flux
release renames the counters, kubectl returns empty, the guard reports
no cycle, and e2e silently misses real remediation loops.
Add a bats unit test that feeds a pinned HelmRelease v2 status snippet
through the same jsonpath and asserts the extraction still yields the
expected values. Also leave a pointer comment in run-kubernetes.sh so a
future flux bump surfaces the version coupling in review.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-template unittests in packages/apps/kubernetes/tests/ assert
that cluster-autoscaler, kccm, and the csi controller each mount the
admin-kubeconfig Secret optional and carry the wait-for-kubeconfig
init. That locks in today's three Deployments by name - a fourth
Deployment that mounts the same Secret but forgets the guard would
slip past them.
Add a bats-unit test that renders the entire chart, enumerates every
Deployment whose spec mounts a Secret ending in -admin-kubeconfig, and
asserts optional:true plus wait-for-kubeconfig init on all of them.
Verified by temporarily removing optional:true from csi/deploy.yaml:
the test correctly flagged invariant-kcsi-controller as an offender.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the hardcoded r.kindName == "Kubernetes" switch in rest.go
with a config-driven path. The ApplicationDefinition CR now accepts a
release.cozystack.io/helm-install-timeout annotation that is parsed
at cozystack-api startup into config.ReleaseConfig.HelmInstallTimeout
and applied to both Install.Timeout and Upgrade.Timeout on the
rendered HelmRelease. Applications that leave the annotation unset
keep flux defaults so their failed installs remediate on the normal
cadence - only the Kubernetes kind carries the override and gets a
15m budget. New kinds with a similar race can opt in by setting the
same annotation; no rest.go patch needed.
Kubernetes-rd sets the annotation to 15m. Table-driven test in
rest_timeout_test.go covers three cases: Kubernetes with 15m, Qdrant
unset, and an arbitrary future kind with 20m - all of which pin the
Remediation.Retries == -1 contract as well.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The three control-plane-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) carried three copies of the same 20-line init
container. That already drifted: the CSI copy used 4-space nesting
while the other two used 2-space. Any future update to the image,
the deadline, or the poll script had to land in three places or
silently diverge.
Extract the block into a new kubernetes.waitForAdminKubeconfig helper
in templates/_helpers.tpl and include it at each call site. Tighten
the deadline from 20m to 10m so it stays strictly below the 15m
HelmRelease Install.Timeout and the CrashLoopBackOff surfaces in
dashboards before flux remediation can fire. Also clarify the wait
message so operators debugging a stuck init container do not chase
Kamaji for what is actually kubelet's optional-Secret refresh cadence.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A hard helm fail in cluster.yaml made every cold bootstrap racy: if
the parent Tenant chart had not yet populated _namespace.etcd in
cozystack-values when the Kubernetes HelmRelease first reconciled, the
fail fired, install.remediation triggered, installFailures incremented
and the new e2e remediation-guard flagged it as a bug. That directly
contradicts the race the rest of this PR is trying to close.
Replace fail with a graceful skip: render only a status-beacon
ConfigMap (test-awaiting-etcd) when etcd is empty, wrap all
CAPI/Kamaji resources in {{ if $etcd }}. The HelmRelease installs
successfully and goes Ready; flux retries on its 5m interval and picks
up the DataStore as soon as the Tenant chart finishes reconciling.
Update the helm unittest: positive test still asserts dataStoreName on
KamajiControlPlane; the negative test now asserts exactly one
ConfigMap document with status=awaiting-etcd, no Cluster / KCP /
KubevirtCluster / WorkloadMonitor rendered.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cap the wait-for-kubeconfig init container at 20m. If Kamaji genuinely
fails to produce the admin-kubeconfig Secret (misconfigured tenant,
etcd outage after the guard already passed, Kamaji crash-loop), the
pod now exits non-zero and goes CrashLoopBackOff so the failure is
visible in dashboards, instead of silently sleeping in Init forever
and leaving only the flux helm-wait timeout to surface the problem.
Reword the etcd DataStore guard to reference the parent Tenant
application's etcd flag (not .Values.etcd of the Kubernetes chart,
which is a different chart). Update the helm unittest errorPattern
accordingly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds an assertion that cluster.yaml renders successfully when
_namespace.etcd is set and produces a KamajiControlPlane whose
dataStoreName equals the tenant's etcd DataStore name. Without this
positive case a future edit that inverts or removes the existing etcd
guard would pass the suite as long as the negative case still fails.
Also adds documentSelector: kind=Deployment to the kccm and csi
controller assertions so the jsonpath filter operates on a single
document, matching the cluster-autoscaler case and removing
reliance on helm-unittest filter-vs-single-value coercion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds images/busybox/Dockerfile and an image-busybox Makefile target
that mirror the same pattern as the rest of this chart's images (the
Dockerfile pins the upstream busybox by digest; the Makefile target
builds and tags for ghcr.io/cozystack/cozystack/busybox the same way
cluster-autoscaler et al. are handled). Also wires it into the
umbrella image target so 'make image' rebuilds everything.
Until the first release build runs image-busybox and rewrites the
.tag to point at ghcr.io, the .tag keeps a fully-qualified
docker.io/library/busybox:1.37.0@sha256:... reference so pods do not
silently resolve the short name via the default registry and pulls
remain immutable by digest. The release workflow overwrites this
file the same way it does for the other images.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous change set Install.Timeout and Upgrade.Timeout to 15m
on every Application's parent HelmRelease, but the admin-kubeconfig
race documented in #2412 is specific to the Kubernetes kind: only its
parent chart creates CAPI/Kamaji resources whose admin-kubeconfig
Secret is asynchronously provisioned and mounted by Deployments in
the same chart. Other kinds (Qdrant, MongoDB, Postgres, ...) have no
such race and should not have their failed installs linger three times
longer before flux triggers remediation.
Gate the timeout on r.kindName == "Kubernetes". Rewrite
rest_timeout_test.go to cover both sides: Kubernetes must get a
>= 15m timeout, other kinds must keep the flux defaults. Both tests
also pin Install/Upgrade Remediation.Retries == -1 so a future edit
that removes unbounded remediation would show up as a red test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Shell && and || have equal precedence and left-to-right associativity,
so the previous guard parsed as (((A && B) || C) && D) and silently
passed on the canonical failure mode: install_failures=1 with an empty
upgrade_failures.
Extract the check into helmrelease_has_remediation_cycle() in a
dedicated helper sourced from run-kubernetes.sh, and add unit tests
under hack/remediation-guard.bats that pin the expected behavior for
every combination of empty, zero, and positive counter values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Before cleanup, inspect the parent HelmRelease installFailures and
upgradeFailures counters. A non-zero value means flux helm-controller
hit its wait timeout, ran install/upgrade remediation (uninstall),
and re-installed - the exact race condition this PR closes. Fail the
bats test in that case so the signal surfaces in CI instead of being
masked by a green retry.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When a Kubernetes tenant is created without a parent tenant that has
etcd enabled, .Values._namespace.etcd is empty and the rendered
KamajiControlPlane spec carries an empty dataStoreName. The Kamaji
admission webhook then rejects every TenantControlPlane create with
"tenant-root DataStore does not exist" and the control plane never
comes up.
Add a helm template-level guard that fails rendering with a
descriptive, actionable error message before the HelmRelease even
reaches the webhook. This also closes the narrow race where a
Kubernetes HelmRelease reconciles before the etcd HelmRelease has
created the DataStore CR - flux retries on its interval and picks up
the DataStore once it appears.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three Deployments in the Kubernetes app chart mount the tenant
admin-kubeconfig Secret directly as a volume: cluster-autoscaler,
kccm, and the kcsi controller. That Secret is provisioned
asynchronously by Kamaji after control-plane bootstrap, so on a fresh
install the pods used to hit FailedMount and the parent HelmRelease
ran out of its wait budget.
Mark the Secret volume optional and add a wait-for-kubeconfig
initContainer that polls the mounted path until the Secret appears.
Kubelet remounts the optional Secret within its sync period once
Kamaji publishes it, the init container exits, and the main container
starts cleanly. The Deployment becomes Available without the helm-wait
ever seeing a FailedMount.
Pins a busybox image for the init container via
images/busybox.tag (same format as the other pinned tags in this
chart).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds failing helm unittest suite for packages/apps/kubernetes covering:
- cluster-autoscaler, kccm, and csi controller Deployments mount the
admin-kubeconfig Secret with optional: true
- each of those Deployments has a wait-for-kubeconfig initContainer
that mounts the same kubeconfig path
- cluster.yaml renders a helm fail with a descriptive message when the
tenant has no etcd DataStore (empty _namespace.etcd)
Also wires up a test target in the chart Makefile so helm-unit-tests.sh
picks it up.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Parent HelmRelease created by cozystack-api for Kubernetes tenants
contains CAPI/Kamaji resources (Cluster, KamajiControlPlane,
MachineDeployment) that asynchronously provision the
*-admin-kubeconfig Secret. Three Deployments in the same chart
(cluster-autoscaler, kccm, kcsi-controller) mount that Secret
directly, so the helm-wait cannot complete until control-plane
bootstrap finishes.
Default flux helm-controller timeout is too short for a cold-node
first-tenant bootstrap (image pull + etcd bootstrap + apiserver Ready
+ admin-kubeconfig generation routinely exceed it). On timeout,
install.remediation triggers uninstall, which removes the Cluster CR
and restarts the cycle indefinitely.
Bumping Install.Timeout and Upgrade.Timeout to 15m gives realistic
bootstrap headroom while remaining bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds a failing unit test for convertApplicationToHelmRelease asserting
that Install.Timeout and Upgrade.Timeout are at least 15 minutes. The
default flux helm-controller timeout is too short to cover cold-start
Kamaji control-plane bootstrap (image pull + etcd bootstrap + apiserver
Ready + admin-kubeconfig Secret generation) and causes install
remediation loops.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 20:20:41 +03:00
221 changed files with 14262 additions and 453 deletions
copilot --prompt "prepare changelog file for tagged release v${{ steps.tag.outputs.version }}, use @docs/agents/changelog.md for it. Create the changelog file at docs/changelogs/v${{ steps.tag.outputs.version }}.md" \
copilot \
--prompt "Generate the release changelog for tag v${VERSION}. Follow the instructions in @docs/agents/changelog.md exactly, including the 'Scope and boundaries' section at the top. Your deliverable is the single file docs/changelogs/v${VERSION}.md — write it and exit; this workflow handles branching, committing, pushing, and opening the PR." \
- Action: Use labels defined there. Conventions follow the Kubernetes scheme — `kind/*` (type), `area/*` (subsystem), `priority/*` (urgency), `triage/*` (review state), `lifecycle/*` (auto-close), `do-not-merge/*` (PR blockers), `security/*` (severity)
- For `area/*`: accuracy outweighs reuse. If no existing `area/*` truly fits the change, propose a new one via PR (extend `.github/labels.yml` and the scope mapping in `.github/workflows/pr-labeler.yaml`) — do not shoehorn the change into a wrong area. `area/uncategorized` is the auto-labeler fallback; treat it as a signal to pick a fit, create a new area, or correct the PR title
- PR titles: a Conventional Commits header (`type(scope): description`, types from [`contributing.md`](./docs/agents/contributing.md)) auto-applies `kind/*` and `area/*` via `.github/workflows/pr-labeler.yaml`. Append `!` (or add a `BREAKING CHANGE:` footer) to apply `kind/breaking-change`
**Important rules:**
- ✅ **ONLY read the file if the task matches the documented process scope** - do not read files for tasks that don't match their purpose
- ✅ **ALWAYS read the file FIRST** before starting the task (when applicable)
// Timestamp (RFC3339) for point-in-time recovery; empty means latest.
// +kubebuilder:default:=""
RecoveryTimestring`json:"recoveryTime,omitempty"`
// Barman server name (S3 path prefix) used by the original cluster when writing backups. Set this only when the original cluster had an explicit barmanObjectStore.serverName that differed from its Kubernetes resource name.
// Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect.
// +kubebuilder:default:=true
ExternalAllowICMPbool`json:"externalAllowICMP"`
// Requested running state of the VirtualMachineInstance
"description":"Tensor saturation, util/watt and throttling — reveals inefficient GPU workloads",
"tags":[
"gpu",
"efficiency",
"finops"
],
"timezone":"browser",
"editable":true,
"graphTooltip":1,
"time":{
"from":"now-1h",
"to":"now"
},
"fiscalYearStartMonth":0,
"schemaVersion":42,
"panels":[
{
"type":"row",
"collapsed":false,
"title":"Overall efficiency metrics",
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":0
},
"id":1,
"panels":[]
},
{
"type":"stat",
"id":2,
"targets":[
{
"expr":"avg(gpu:tensor_saturation:avg5m) * 100",
"refId":"A"
}
],
"title":"Avg Tensor Saturation",
"description":"Mean tensor core saturation across all GPUs. \u003c10% means GPUs are used inefficiently (workloads could move to CPU or optimize their code). Cluster-wide — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":5,
"w":8,
"x":0,
"y":1
},
"repeatDirection":"h",
"options":{
"graphMode":"area",
"colorMode":"background",
"justifyMode":"auto",
"textMode":"value",
"wideLayout":true,
"showPercentChange":false,
"reduceOptions":{
"calcs":[
"lastNotNull"
]
},
"percentChangeColorMode":"standard",
"orientation":""
},
"fieldConfig":{
"defaults":{
"unit":"percent",
"min":0,
"max":100,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"red"
},
{
"value":10,
"color":"orange"
},
{
"value":30,
"color":"yellow"
},
{
"value":60,
"color":"green"
}
]
}
},
"overrides":[]
}
},
{
"type":"stat",
"id":3,
"targets":[
{
"expr":"avg(gpu:util_per_watt:avg5m)",
"refId":"A"
}
],
"title":"Avg Utilization per Watt",
"description":"NVML utilization % per watt across all GPUs. Higher value = more efficient workload. Cluster-wide — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":5,
"w":8,
"x":8,
"y":1
},
"repeatDirection":"h",
"options":{
"graphMode":"area",
"colorMode":"background",
"justifyMode":"auto",
"textMode":"value",
"wideLayout":true,
"showPercentChange":false,
"reduceOptions":{
"calcs":[
"lastNotNull"
]
},
"percentChangeColorMode":"standard",
"orientation":""
},
"fieldConfig":{
"defaults":{
"unit":"none",
"decimals":3,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"red"
},
{
"value":0.5,
"color":"yellow"
},
{
"value":1,
"color":"green"
}
]
}
},
"overrides":[]
}
},
{
"type":"stat",
"id":4,
"targets":[
{
"expr":"avg(gpu:power_throttle_fraction:rate5m)",
"refId":"A"
}
],
"title":"Avg Power Throttling",
"description":"Fraction of time GPUs hit the TDP cap and lose performance. \u003e5% means tenants underutilize billed FLOPS. Cluster-wide — rule aggregates by (Hostname, gpu, UUID) and drops namespace label.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":5,
"w":8,
"x":16,
"y":1
},
"repeatDirection":"h",
"options":{
"graphMode":"area",
"colorMode":"background",
"justifyMode":"auto",
"textMode":"value",
"wideLayout":true,
"showPercentChange":false,
"reduceOptions":{
"calcs":[
"lastNotNull"
]
},
"percentChangeColorMode":"standard",
"orientation":""
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"decimals":2,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"green"
},
{
"value":0.05,
"color":"yellow"
},
{
"value":0.2,
"color":"red"
}
]
}
},
"overrides":[]
}
},
{
"type":"row",
"collapsed":false,
"title":"NVML vs Tensor (mismatch detector)",
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":6
},
"id":10,
"panels":[]
},
{
"type":"timeseries",
"id":11,
"targets":[
{
"expr":"DCGM_FI_DEV_GPU_UTIL",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"NVML GPU Utilization",
"description":"Classic utilization metric. Shows activity of any engine (SM, copy, encoder). Cluster-wide — DCGM namespace is the exporter's own namespace, not the workload namespace.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":7
},
"repeatDirection":"h",
"options":{
"legend":{
"displayMode":"table",
"placement":"bottom",
"showLegend":false,
"calcs":[
"mean",
"max"
]
},
"tooltip":{
"mode":"multi",
"sort":""
}
},
"fieldConfig":{
"defaults":{
"unit":"percent",
"min":0,
"max":100,
"custom":{
"drawStyle":"line",
"lineInterpolation":"linear",
"fillOpacity":10,
"showPoints":"never"
}
},
"overrides":[]
}
},
{
"type":"timeseries",
"id":12,
"targets":[
{
"expr":"DCGM_FI_PROF_PIPE_TENSOR_ACTIVE * 100",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"Tensor Pipe Active",
"description":"Real tensor core load (HMMA). For AI/LLM inference it should be ≥20%, otherwise the workload is unoptimized. Cluster-wide — DCGM namespace is the exporter's own namespace, not the workload namespace.",
"description":"Which GPUs are exercising tensor cores and which are not. Sorted descending. Grouped by Hostname/gpu/UUID — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":16
},
"repeatDirection":"h",
"transformations":[
{
"id":"organize",
"options":{
"excludeByName":{
"DCGM_FI_DRIVER_VERSION":true,
"Time":true,
"__name__":true,
"cluster":true,
"container":true,
"device":true,
"endpoint":true,
"gpu_driver_version":true,
"instance":true,
"job":true,
"modelName":true,
"namespace":true,
"pci_bus_id":true,
"pod":true,
"prometheus":true,
"service":true,
"tenant":true,
"tier":true,
"uid":true,
"unit":true
},
"indexByName":{
"Hostname":0,
"UUID":2,
"Value":3,
"gpu":1
},
"renameByName":{
"Hostname":"Node",
"UUID":"UUID",
"Value":"Saturation",
"gpu":"GPU"
}
}
}
],
"options":{
"frameIndex":0,
"showHeader":true,
"showTypeIcons":false,
"sortBy":[
{
"displayName":"Saturation",
"desc":true
}
],
"footer":{
"show":false,
"reducer":[]
},
"cellHeight":"sm"
},
"fieldConfig":{
"defaults":{},
"overrides":[
{
"matcher":{
"id":"byName",
"options":"Saturation"
},
"properties":[
{
"id":"unit",
"value":"percent"
},
{
"id":"min",
"value":0
},
{
"id":"max",
"value":100
},
{
"id":"custom.cellOptions",
"value":{
"mode":"gradient",
"type":"gauge"
}
},
{
"id":"thresholds",
"value":{
"mode":"absolute",
"steps":[
{
"color":"red"
},
{
"color":"orange",
"value":10
},
{
"color":"yellow",
"value":30
},
{
"color":"green",
"value":60
}
]
}
}
]
}
]
}
},
{
"type":"table",
"id":22,
"targets":[
{
"expr":"topk(20, gpu:util_per_watt:avg5m)",
"instant":true,
"range":false,
"format":"table",
"refId":"A"
}
],
"title":"Utilization / Watt per GPU (5m avg)",
"description":"How efficiently each GPU spends watts. Low value = poor optimization. Grouped by Hostname/gpu/UUID — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":16
},
"repeatDirection":"h",
"transformations":[
{
"id":"organize",
"options":{
"excludeByName":{
"DCGM_FI_DRIVER_VERSION":true,
"Time":true,
"__name__":true,
"cluster":true,
"container":true,
"device":true,
"endpoint":true,
"gpu_driver_version":true,
"instance":true,
"job":true,
"modelName":true,
"namespace":true,
"pci_bus_id":true,
"pod":true,
"prometheus":true,
"service":true,
"tenant":true,
"tier":true,
"uid":true,
"unit":true
},
"indexByName":{
"Hostname":0,
"UUID":2,
"Value":3,
"gpu":1
},
"renameByName":{
"Hostname":"Node",
"UUID":"UUID",
"Value":"Util/Watt",
"gpu":"GPU"
}
}
}
],
"options":{
"frameIndex":0,
"showHeader":true,
"showTypeIcons":false,
"sortBy":[
{
"displayName":"Util/Watt",
"desc":true
}
],
"footer":{
"show":false,
"reducer":[]
},
"cellHeight":"sm"
},
"fieldConfig":{
"defaults":{},
"overrides":[
{
"matcher":{
"id":"byName",
"options":"Util/Watt"
},
"properties":[
{
"id":"unit",
"value":"none"
},
{
"id":"decimals",
"value":3
},
{
"id":"min",
"value":0
},
{
"id":"max",
"value":1.5
},
{
"id":"custom.cellOptions",
"value":{
"mode":"gradient",
"type":"gauge"
}
},
{
"id":"thresholds",
"value":{
"mode":"absolute",
"steps":[
{
"color":"red"
},
{
"color":"yellow",
"value":0.5
},
{
"color":"green",
"value":1
}
]
}
}
]
}
]
}
},
{
"type":"row",
"collapsed":false,
"title":"Throttling",
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":24
},
"id":30,
"panels":[]
},
{
"type":"timeseries",
"id":31,
"targets":[
{
"expr":"gpu:power_throttle_fraction:rate5m",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"Power throttle fraction per GPU",
"description":"Fraction of time the GPU was power-throttled. 1.0 = always throttled. Cluster-wide — rule aggregates by (Hostname, gpu, UUID) and drops namespace label.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":25
},
"repeatDirection":"h",
"options":{
"legend":{
"displayMode":"table",
"placement":"bottom",
"showLegend":false,
"calcs":[
"mean",
"max"
]
},
"tooltip":{
"mode":"multi",
"sort":""
}
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"min":0,
"max":1,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"green"
},
{
"value":0.05,
"color":"yellow"
},
{
"value":0.2,
"color":"red"
}
]
},
"custom":{
"drawStyle":"line",
"lineInterpolation":"linear",
"fillOpacity":25,
"showPoints":"never"
}
},
"overrides":[]
}
},
{
"type":"timeseries",
"id":32,
"targets":[
{
"expr":"gpu:thermal_throttle_fraction:rate5m",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"Thermal throttle fraction per GPU",
"description":"Fraction of time the GPU was thermal-throttled. Cluster-wide — rule aggregates by (Hostname, gpu, UUID) and drops namespace label.",
@ -6,6 +6,20 @@ This file contains detailed instructions for AI-powered IDE on how to generate c
Follow these instructions when the user explicitly asks to generate a changelog.
## Scope and boundaries
**Your single deliverable is the file `docs/changelogs/v<version>.md`.** Write the complete, verified changelog to that path. That is the entire task. Exit as soon as the file is written and verified against the checklist in Step 9.
Unless the caller explicitly instructs otherwise:
- **In the cozystack working tree**, do not run `git commit`, `git push`, `git checkout` (to switch branches), `git branch`, `git tag`, `git reset`, `git merge`, or `git rebase`. Do not write to local branches, tags, or HEAD. `git fetch` is expected and fine (see the read-only analysis list below).
- **Do not** push to any remote, open pull requests, or issue GitHub API write calls (POST / PATCH / DELETE) for any repository.
- In the cozystack working tree, the **only** file you create or modify is `docs/changelogs/v<version>.md`. Cloning auxiliary repositories under `_repos/` for cross-repo analysis (see Step 6) is fine; local git operations inside those disposable clones (`git checkout`, `git pull`, etc.) are allowed — just never push from them or open PRs against them.
The caller — a GitHub Actions workflow in CI, or a developer running you interactively — owns branching, committing, pushing, and PR creation. They will perform those actions after you exit. Do not pre-empt them even if the working tree looks ready.
Read-only analysis is expected and encouraged: `git log`, `git show`, `git fetch`, `git diff`, `gh pr view`, `gh api` GET requests, and reading any file in the repository.
## Required Tools
Before generating changelogs, ensure you have access to `gh` (GitHub CLI) tool, which is used to fetch commit and PR author information. The GitHub CLI is used to correctly identify PR authors from commits and pull requests.
@ -608,6 +622,8 @@ Create a new changelog file in the format matching previous versions:
**Save the changelog:**
Save the changelog to file `docs/changelogs/v<version>.md` according to the version for which the changelog is being generated.
**Then exit.** Do not commit, push, create a branch, or open a pull request — the caller handles all git and GitHub operations after you return. See the "Scope and boundaries" section at the top of this document.
### Important notes
- **After fetch with --force** local tags are up-to-date, use them for work
@ -8,6 +8,30 @@ Project-side conventions for commits, branches, and pull requests in Cozystack.
- [ ] Commit is signed off with `--signoff`
- [ ] Branch is rebased on `upstream/main` (no extra commits)
- [ ] PR body includes description and release note
- [ ] Ran `make generate` in every package whose `values.yaml`, `values.schema.json`, `Chart.yaml`, or `README.md` was touched, and committed the regenerated files
## Regenerate Artifacts Before Committing
Several files in each package are produced by `make generate` from `values.yaml` + `values.schema.json` and must stay in sync with the hand-edited sources:
- `packages/(apps|extra)/<name>/README.md` — regenerated by `cozyvalues-gen` (parameter table, formatting).
- `packages/(apps|extra)/<name>/values.schema.json` — `cozyvalues-gen` rewrites ordering and derived fields.
- `packages/system/<name>-rd/cozyrds/<name>.yaml` — produced by `hack/update-crd.sh`, which `make generate` invokes.
**Before committing edits to any of those sources**, run `make generate` inside the package and stage the full diff:
The repo's pre-commit CI job runs `make generate` in every package and then `git diff --exit-code`. Any unstaged generator output fails the job with exit code 123 and blocks the PR. Also rerun `make generate` after a `git commit --amend` if the amended change touched any of the sources above.
To locate packages a WIP branch likely needs to be regenerated:
**Scopes** (examples — not an exhaustive list; pick the most specific scope that describes the change, and introduce a new one if a genuinely new area needs its own):
`.github/workflows/pr-labeler.yaml` parses the PR title on `opened`, `edited`, `reopened`, and `synchronize` events and applies labels additively (never removes). The title is expected to follow Conventional Commits — same format as commit messages above.
Each component in a `PackageSource` may set `install.upgradeCRDs` to control how CRDs from the chart's `crds/` directory are handled on `HelmRelease` upgrades. Allowed values: `Skip` (default — helm-controller does not touch CRDs on upgrade), `Create` (create new CRDs only), `CreateReplace` (create new and overwrite existing).
Set `upgradeCRDs: CreateReplace` for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them.
Do **not** set `CreateReplace` blindly: it overwrites every CRD in `crds/` and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually.
Cozystack v1.3.0 brings **storage-aware pod scheduling** via a LINSTOR scheduler extender, a managed **LINSTOR GUI** web console with Keycloak SSO, a curated **VM Default Images** catalog for out-of-the-box virtual-machine provisioning, a new **WorkloadsReady / Events** observability surface with S3 bucket metering, and **cross-namespace VMInstance backup restore** with a full **RestoreJob dashboard** flow. The release also ships stricter tenant-name validation, VMInstance network-selector improvements, Keycloak theme injection and SMTP configuration, a host-runtime preflight check, and rolls up every fix from the v1.2.1 → v1.2.4 patch line.
> **Note:** Items marked *(backported to v1.2.x)* were also shipped in v1.2.1, v1.2.2, v1.2.3, or v1.2.4 patch releases.
## Feature Highlights
### Storage-Aware Scheduling via the LINSTOR Extender
The `cozystack-scheduler` now calls a **LINSTOR scheduler extender** for storage-locality-aware pod placement. When a pod declares both a `SchedulingClass` and LINSTOR-backed PVCs, the scheduler consults LINSTOR to prefer nodes where volume replicas already exist — reducing cross-node replication traffic and improving I/O latency for storage-heavy workloads such as databases, object stores, and VMs.
The integration builds on the existing `SchedulingClass` tenant workload placement system introduced in v1.2.0 and requires no tenant-side configuration — workloads simply benefit once a SchedulingClass is assigned. Administrators can mix storage locality with the existing data-center / hardware-generation constraints defined on SchedulingClass CRs ([**@lllamnyp**](https://github.com/lllamnyp) in #2330).
### LINSTOR GUI: Managed Web Console for Storage Administration
A new opt-in `linstor-gui` system package deploys **LINBIT's linstor-gui web UI** alongside the LINSTOR controller with mTLS client authentication, non-root security context, and a ClusterIP-only service by default. When OIDC is configured on the platform, an optional **Keycloak-protected Ingress** (via oauth2-proxy) exposes the UI for browser access. Access is restricted to members of the `cozystack-cluster-admin` Keycloak group, consistent with host-cluster admin RBAC, and the gatekeeper blocks in-app LINSTOR authentication setup at the nginx proxy layer so the managed configuration cannot be subverted through the UI.
Operators who prefer CLI access keep the existing `linstor` command; the GUI is strictly additive and stays disabled by default ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2382, #2390, #2415, #2419).
### VM Default Images: Out-of-the-Box VM Provisioning
The new `vm-default-images` package provides a curated set of **cluster-wide virtual-machine images** (Ubuntu, Debian, CentOS Stream, and others) as pre-populated DataVolumes, so tenants can provision VMs against well-known base images without first having to upload them. The package is opt-in via the `iaas` bundle and defaults to replicated storage for high availability. Migration 38 renames legacy `vm-image-*` DataVolumes to the new `vm-default-images-*` naming scheme, and the `vm-disk` chart gains a new "disk" source type for cloning from existing vm-disks in the same namespace ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2258).
### Application Observability: WorkloadsReady, Events, and S3 Bucket Metering
Applications now expose a **WorkloadsReady** condition on their status by querying associated WorkloadMonitor resources, giving operators a single place to check whether all underlying workloads (Deployments, StatefulSets, DaemonSets, PVCs) are healthy. The dashboard gains a new **Events tab** showing namespace-scoped Kubernetes events per application, with fallback to `.firstTimestamp` when `.eventTime` is absent. A long-standing bug where WorkloadMonitor's `Operational` status was never persisted is fixed in the same change ([**@lexfrei**](https://github.com/lexfrei) in #2356).
The WorkloadMonitor reconciler is extended to track **COSI BucketClaim** objects as first-class Workloads, and the bucket controller now queries SeaweedFS logical and physical bucket-size metrics from VictoriaMetrics via a namespace-scoped monitoring endpoint, enabling S3 billing integration on par with Pods and PVCs ([**@kitsunoff**](https://github.com/kitsunoff) in #2391). Workloads are also enriched with `workloads.cozystack.io/resource-preset` and source-object labels so downstream billing pipelines can correlate monitors with the tenant preset that produced them ([**@androndo**](https://github.com/androndo) in #2416).
### Cross-Namespace VM Backup Restore and RestoreJob Dashboard
The backup system now supports **restoring VMInstance backups into a different namespace** (cross-namespace copy restores) with IP/MAC preservation and safe rename semantics. In-place backup and restore flows for VMDisk and VMInstance are improved: HelmReleases and DataVolumes are properly handled, and Velero failure messages are propagated to the Application status. The backup status structure has been refactored to store underlying resources as a generic opaque JSON object, enabling arbitrary application-specific metadata without status-schema churn ([**@androndo**](https://github.com/androndo) in #2251, #2319, #2329).
The dashboard now ships a complete **RestoreJob experience**: list view, details page, create form, and sidebar entry, with a "Same as backup" fallback rendering when `spec.targetApplicationRef` is omitted. Non-CRD-backed sidebar factories (`kube-*`, `plan`, `backupjob`, `backup`, `restorejob`) are marked static so they pick up consistent managed-by labels across reconciles ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2437).
## Major Features and Improvements
* **[api] Reject tenant names with dashes at Create time**: Enforces alphanumeric-only naming for Tenants at the API level, preventing names with hyphens that would silently fail during Helm reconciliation. A corresponding regex tightening and regression test suite hardens the validation ([**@lexfrei**](https://github.com/lexfrei) in #2380).
* **[platform] Validate computed tenant namespace length**: Rejects Tenant creation when the computed ancestor-chain namespace would exceed the 63-character Kubernetes namespace limit, preventing opaque HelmRelease reconcile errors downstream ([**@lexfrei**](https://github.com/lexfrei) in #2376).
* **[vm-instance] Rename subnets to networks and add dropdown selector**: Renames the misleading `subnets` field to `networks` in VMInstance for clarity, adds a dropdown selector for available networks in the dashboard form, and includes migration 36 to copy existing `subnets` values. The old field remains supported for backward compatibility ([**@sircthulhu**](https://github.com/sircthulhu) in #2263).
* **[keycloak] Enable injecting themes**: Cozystack administrators can now inject custom Keycloak themes via `initContainers` for UI white-labeling and customization ([**@lllamnyp**](https://github.com/lllamnyp) in #2142).
* **[keycloak-configure] Add email verification and SMTP configuration**: Adds configurable Keycloak settings for user self-registration, email verification, and SMTP server configuration, enabling automated user onboarding flows ([**@BROngineer**](https://github.com/BROngineer) in #2318).
* **[postgres] Pin system PostgreSQL to 17.7-standard-trixie**: Pins the PostgreSQL image for system databases (Grafana, Alerta, Harbor, Keycloak, SeaweedFS) to `17.7-standard-trixie` across chart templates and `values.yaml`, and ships migration 37 to patch existing CNPG Cluster `imageName` fields to the same variant (handling unset, any PG 17 tag, and bare-version tags). This prevents CNPG from defaulting to PostgreSQL 18 and locks system databases to the trixie variant consistent with the monitoring stack requirements *(related backports shipped in v1.2.1 via #2309 and v1.2.2 via #2364)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2369).
* **[platform] Prevent installed packages deletion**: Adds the `helm.sh/resource-policy: keep` annotation to platform packages so disabling a package no longer triggers automatic Helm deletion, restoring the documented behavior where operators must explicitly delete a package *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2273).
* **[mariadb] Always enable replication for consistent service naming**: MariaDB now always enables replication, creating `-primary`/`-secondary` services even for single-replica instances. This fixes dashboard visibility and backup functionality for single-replica setups ([**@sircthulhu**](https://github.com/sircthulhu) in #2279).
* **[hack] Add host runtime preflight check**: New `check-host-runtime.sh` script and `make preflight` target that warns operators when a standalone containerd or docker runtime is running alongside the embedded k3s runtime, helping diagnose container-runtime conflicts early in an installation ([**@lexfrei**](https://github.com/lexfrei) in #2371).
* **[hack] Add check-readiness.sh diagnostic script**: A new diagnostic script for tracking platform reconciliation by checking readiness of Packages, ArtifactGenerators, ExternalArtifacts, and HelmReleases, with support for watch mode and continuous monitoring ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2294).
* **[platform] Add resourcePreset labels to WorkloadMonitor labels**: WorkloadMonitor labels with the `workloads.cozystack.io/` prefix are now propagated onto created Workloads; created Workloads always include the reserved `workloads.cozystack.io/monitor` label, and Helm app charts add `workloads.cozystack.io/resource-preset` metadata to WorkloadMonitor manifests, enabling downstream billing pipelines to correlate monitors with the tenant preset that produced them ([**@androndo**](https://github.com/androndo) in #2416).
## Bug Fixes
* **[platform] Migrate ACME HTTP-01 to ingressClassName API**: Switches ACME HTTP-01 issuance from the deprecated `acme.cert-manager.io/http01-ingress-class` annotation to the modern `ingressClassName` field on `ClusterIssuer` and solver pods. Previously, ClusterIssuers referenced a non-existent `nginx` class while each Ingress individually overrode it via annotation — producing `ingressClassName and class cannot be set at the same time` errors when tenants attempted to migrate to the modern field. The migration is atomic: both the ClusterIssuer and consuming Ingresses are updated together *(backported to v1.2.4)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2436).
* **[harbor] Remove incorrect tenant module flags**: Harbor is a PaaS service, not a tenant module. Incorrect `spec.dashboard.module: true` and `internal.cozystack.io/tenantmodule` flags caused Harbor to appear in the sidebar "Modules" section and be misclassified by controllers handling tenant modules. The flags are now removed so Harbor is displayed in its proper PaaS category and is no longer treated as a tenant-scoped HelmRelease ([**@kvaps**](https://github.com/kvaps) in #2444).
* **[kube-ovn] Resolve kubeovn-plunger RBAC forbidden on deployments**: Grants `kube-ovn-plunger` the RBAC needed to list Deployments so it can reconcile `ovn-central`, fixing `deployments.apps is forbidden` errors in `cozy-kubeovn` ([**@kvaps**](https://github.com/kvaps) in #2441).
* **[cilium] Opt-out of cri-containerd.apparmor.d for nsenter init containers**: Opts cilium-agent init containers out of the `cri-containerd.apparmor.d` AppArmor profile on non-Talos variants (`cilium-generic`, `kubeovn-cilium-generic`), fixing `Init:CrashLoopBackOff` on Ubuntu 22.04+ and Debian where the profile denies `nsenter` namespace entry. Talos variants are untouched as Talos does not load the AppArmor LSM *(backported to v1.2.2)* ([**@lexfrei**](https://github.com/lexfrei) in #2370).
* **[virtual-machine] Exclude external VM services from Cilium BPF LB**: Adds the `service.kubernetes.io/service-proxy-name: cozy-proxy` label to VM LoadBalancer services with `external: true`, telling Cilium to skip BPF processing entirely. Fixes inter-tenant connectivity via public LB IPs (Cilium's DNAT caused cross-tenant pod-to-pod flow classification, triggering CiliumClusterwideNetworkPolicy blocks) and restores WholeIP behavior on Cilium 1.19+ where wildcard service drop entries previously blocked traffic to LB IPs on undeclared ports *(backported to v1.2.2)* ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2357).
* **[monitoring] Fix infra dashboards missing in default variant**: Includes the `cozy-monitoring` namespace in the dashboard rendering condition, fixing infrastructure Grafana dashboards not rendering in the default platform variant (only the `tenant-root` namespace was previously checked) *(backported to v1.2.2)* ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2365).
* **[build] Filter git describe to match only v* tags**: Adds `--match 'v*'` to all `git describe` calls in `hack/common-envs.mk`, preventing the `api/apps/v1alpha1/vX.Y.Z` subtag from being picked up instead of the release tag and producing invalid Docker image tags *(backported to v1.2.2)* ([**@kvaps**](https://github.com/kvaps) in #2386).
* **[platform] Fix resource allocation ratios not propagated to packages**: Restores propagation of `cpuAllocationRatio`, `memoryAllocationRatio`, and `ephemeralStorageAllocationRatio` from `platform/values.yaml` to the `cozystack-values` Secret that managed applications and KubeVirt read, fixing a regression introduced in the bundle restructure that silently ignored operator-configured ratios *(backported to v1.2.1)* ([**@sircthulhu**](https://github.com/sircthulhu) in #2296).
* **[kubernetes] Set explicit ephemeral-storage on virt-launcher pods**: Sets explicit `domain.resources` ephemeral-storage on the VirtualMachine spec to prevent virt-launcher pods from being evicted because LimitRange defaults were too small for the actual emptyDisk capacity *(backported to v1.2.3)* ([**@kvaps**](https://github.com/kvaps) in #2317).
* **[multus] Pin master CNI to 05-cilium.conflist**: Prevents a boot-time race where multus could auto-detect kube-ovn's conflist instead of Cilium's, which would cause pods to bypass the Cilium chain entirely and lose their endpoint *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2315).
* **[multus] Build custom image with DEL cache fix**: Fixes sandbox cleanup deadlock when CNI ADD never completes, preventing stale sandbox name reservations from permanently blocking pod creation after a node disruption *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2313).
* **[linstor] Set verify-alg to crc32c**: Prevents DRBD connection failures on kernels where `crct10dif` is unavailable (e.g., Talos v1.12.6 with kernel 6.18.18) by setting the LINSTOR verify-alg controller default to `crc32c`*(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2303).
* **[linstor] Preserve TCP ports during toggle-disk operations**: Saves existing TCP ports into the `LayerPayload` before `removeLayerData()` deletes them, preventing DRBD resources from entering StandAlone state when a satellite misses the resulting update *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2292).
* **[linstor] Increase satellite startup probe failure threshold**: Raises the LINSTOR satellite `startupProbe``failureThreshold` from 3 to 30 (30s → 300s) in the `LinstorSatelliteConfiguration` pod template, giving satellites with slow storage initialization enough time to come up without being killed and restarted ([**@Arsolitt**](https://github.com/Arsolitt) in #2425).
## Security
* **docs: add SECURITY.md**: Adds vulnerability reporting procedures, disclosure expectations, and supported release lines ([**@kvaps**](https://github.com/kvaps) in #2230).
* **docs: add OpenSSF Best Practices badge to README**: Adds the OpenSSF Best Practices passing badge to the project README ([**@lexfrei**](https://github.com/lexfrei) in #2320).
## Dependencies & Version Updates
* **[kube-ovn] Bump kube-ovn to v1.15.10 with port-group regression fix**: Updates `packages/system/kubeovn` to upstream v1.15.10 (from v1.15.3) and carries a patch for `pkg/controller/pod.go` that preserves a VM LSP's port-group memberships when Kubernetes GCs a completed virt-launcher pod while another virt-launcher pod of the same VM is still running. Without the patch, the destination pod of a successful live migration lost its security groups, network policies, and node-scoped routing until `kube-ovn-controller` was restarted ([**@kvaps**](https://github.com/kvaps) in #2443).
* **[monitoring] Upgrade victoria-metrics-operator to v0.68.4**: Bumps the vendored `victoria-metrics-operator` Helm chart from 0.59.1 to 0.61.0 (operator appVersion v0.68.1 → v0.68.4), picking up upstream fixes for `VMPodScrape` port routing on VMAgent/VLAgent and `StatefulSet` pod deletion (not eviction) when `maxUnavailable=100%` ([**@lexfrei**](https://github.com/lexfrei) in #2426).
* **[linstor] Update piraeus-server to v1.33.2 with selected backports**: Bumps LINSTOR server from v1.33.1 to v1.33.2 with backported patches for stale bitmap adjust retry, LUKS2 header sizing, optimal I/O size detection, and the maintainer implementation. All patches verified against upstream v1.33.2 with `git apply --check` and `gradlew compileJava`*(backported to v1.2.2)* ([**@kvaps**](https://github.com/kvaps) in #2331).
* **[kamaji] Update to 26.3.5-edge, drop upstreamed patches**: Updates Kamaji from edge-26.2.4 to 26.3.5-edge and removes two patches accepted upstream. Adds configurable probe tuning and DataStore readiness conditions ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2260).
* **[talm] Release v0.23.0, v0.23.1, v0.24.0** (github.com/cozystack/talm): Migrates to the Talos v1.12 multi-document machine config format ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#116); renders templates online in `apply` to resolve lookups ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/talm#119); bumps dependencies and modernizes the codebase ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#124).
* **[ansible-cozystack] Release v1.2.1, v1.2.2, v1.2.4** (github.com/cozystack/ansible-cozystack): Exposes `publishing.externalIPs` and tenant-root ingress via role variables ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#30); adds a comprehensive node prerequisites audit ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#27); replaces `ansible.utils.ipaddr` with a stdlib-based test plugin ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#24); adds `v` prefix to collection version in requirements.yml examples ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#23); tracks installer releases v1.2.1 through v1.2.4 ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#20, #22, #29, #31, #32).
## Development, Testing, and CI/CD
* **[ci] Replace cozystack-bot PAT with cozystack-ci GitHub App**: Replaces the long-lived `cozystack-bot` personal access token with short-lived, scoped tokens from the `cozystack-ci` GitHub App across all release workflows (`tags.yaml`, `auto-release.yaml`, `pull-requests-release.yaml`), improving security and auditability of CI operations ([**@tym83**](https://github.com/tym83) in #2351; [**@kvaps**](https://github.com/kvaps) in #2383, #2392).
* **[ci] Add Gemini Code Assist and CodeRabbit configuration**: Adds repository-level configuration for AI code reviewers with ignore patterns for vendored/generated code and incremental review settings ([**@lexfrei**](https://github.com/lexfrei) in #2385).
* **[ci] Promote next/ trunk on new minor/major releases**: Updates `update-website-docs` in `tags.yaml` to match the new docs-versioning contract — the website repo replaces the old "pre-create `vX.Y/` draft directory" scheme with a permanent `content/en/docs/next/` trunk, and released version directories are promoted explicitly by the release workflow ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2433).
* **[tests] Fix Kafka E2E test timeout and retry race condition**: Increases Kafka E2E test timeout from 60s to 300s and fixes a retry race where `kubectl apply` could hit a still-deleting resource ([**@lexfrei**](https://github.com/lexfrei) in #2358).
* **docs: adopt Conventional Commits for commit and PR titles**: Standardizes commit and PR title format to `type(scope): description` across all contributing docs and the PR template ([**@lexfrei**](https://github.com/lexfrei) in #2395).
* **docs(ci): require screenshots for UI changes in PR template**: Adds a mandatory screenshots section to the PR template for UI-related changes ([**@kitsunoff**](https://github.com/kitsunoff) in #2407).
* **chore(maintenance): add @myasnikovdaniil to CODEOWNERS**: Adds @myasnikovdaniil to the default owners in `.github/CODEOWNERS` for automatic review requests ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2434).
## Documentation
* **[website] Add ApplicationDefinition naming convention reference**: Documents how `cozystack-api` resolves kinds to their backing definitions ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#478).
* **[website] Document Talos / talosctl / Cozystack version pairing**: Adds a version compatibility matrix for installation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#484).
* **[website] Document namespace layout and parent/child derivation**: Explains tenant namespace hierarchy and parent/child namespace derivation rules ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#479).
* **[website] Document the checkbox-then-edit-CR customization pattern for tenants**: Describes the workflow for customizing tenant settings via the CR after initial checkbox-based creation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#485).
* **[website] Add custom Keycloak themes documentation**: Covers the theme image contract, configuration, `imagePullSecrets`, and theme activation in the Keycloak admin console ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#463).
* **[website] Add bonding (LACP) configuration how-to guide**: Covers network bonding configuration for Cozystack installations ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#459).
* **[website] Improve registry mirrors for tenant Kubernetes in air-gapped guide**: Improves documentation for configuring registry mirrors in air-gapped environments ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#461).
* **[website] Rewrite guide for ApplicationDefinition API (external-apps)**: Comprehensive rewrite of the external apps guide using the ApplicationDefinition API with Minecraft server examples ([**@kitsunoff**](https://github.com/kitsunoff) in cozystack/website#488).
* **[website] Add documentation for Go types usage**: Guide for using generated Go types for Cozystack managed applications as a Go module ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#465).
* **[website] Update backup/restore documentation for VMI/VMDisk**: Updates backup documentation with VM instance and VM disk restore improvements ([**@androndo**](https://github.com/androndo) in cozystack/website#466).
* **[website] Refactor docs versions to major.minor variants**: Moves docs to major.minor versioning for the v1.x series ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#477).
* **[website] Trunk-based versioning with permanent next/ directory**: Replaces the old "pre-create `vX.Y/` draft directory" scheme with a permanent `content/en/docs/next/` trunk; released version directories are promoted explicitly by `hack/release_next.sh` on new minor/major releases, and routing between `next/` and `vX.Y/` is Makefile-driven ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#495).
* **[website] Add updated OpenAPI spec**: Updates the OpenAPI specification for managed applications reference ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#469).
* **[website] Add OpenAPI spec download to GitHub Pages build**: Fixes the GitHub Pages build to include the OpenAPI spec download ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#494).
* **[website] Add OSS Health pages and OpenSSF badge**: Adds OSS Health section with OpenSSF Scorecard and Best Practices badges to the website ([**@tym83**](https://github.com/tym83) in cozystack/website#470).
* **[website] Add Telemetry page under OSS Health section**: Adds the Telemetry page with initial data seeding to the OSS Health docs ([**@tym83**](https://github.com/tym83) in cozystack/website#471, cozystack/website#504).
* **[website] Blog: OSS Health section launch announcement**: Publishes the announcement blog post for the OSS Health section ([**@tym83**](https://github.com/tym83) in cozystack/website#474).
* **[website] Fix OpenSSF canonical status URL**: Changes the OpenSSF canonical status URL from pt-BR to en ([**@tym83**](https://github.com/tym83) in cozystack/website#475).
* **[website] Add CozySummit Virtual 2026 program announcement**: Publishes the CozySummit Virtual 2026 program announcement blog post ([**@tym83**](https://github.com/tym83) in cozystack/website#472).
* **[website] Add missing release announcements for v0.1–v0.41**: Backfills missing release announcement blog posts for historical Cozystack versions ([**@tym83**](https://github.com/tym83) in cozystack/website#468).
* **[website] Blog: managed PostgreSQL with synchronous replication**: Adds a post covering the managed PostgreSQL synchronous-replication feature ([**@tym83**](https://github.com/tym83) in cozystack/website#497).
* **[website] Blog taxonomies and client-side filter UI**: Registers article-type and topic taxonomies and adds a client-side filter on the blog list page ([**@tym83**](https://github.com/tym83) in cozystack/website#499).
* **[website] Add images frontmatter for social preview on existing posts**: Adds images frontmatter for social preview on existing blog posts ([**@tym83**](https://github.com/tym83) in cozystack/website#498).
* **[website] Fix broken links and stale anchors across v1 docs**: Fixes 14 broken links and stale talm anchors ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#486).
* **[website] Prefix bundle package names with cozystack. in v1 examples**: Corrects package naming in documentation examples ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#482).
* **[website] Finish isolated-field removal and document opt-in policy labels**: Removes the obsolete `isolated` field from tenant documentation and documents the new opt-in policy labels approach ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#481).
* **[website] Add --take-ownership flag and describe networking.* fields**: Documents the `--take-ownership` flag and `networking.*` fields in the installation guide ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#480).
* **[website] Fix KubeOVN MASTER_NODES example path and key in troubleshooting**: Corrects the MASTER_NODES example path and key ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#483).
* **[website] Add CLAUDE.md for AI agent guidance**: Adds a CLAUDE.md file describing the trunk-based docs architecture for AI agent guidance ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#489).
* **[website] Update /docs/v1/ redirect to latest v1.2**: Updates the `/docs/v1/` redirect target to point to the latest v1.2 docs on GitHub Pages ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#492).
* **[website] Remove nbykov from CODEOWNERS and CLAUDE.md**: Cleans up CODEOWNERS and CLAUDE.md entries ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#491).
* **[website] Add Ahrefs Analytics tracker**: Adds the Ahrefs Analytics tracker to the website ([**@tym83**](https://github.com/tym83) in cozystack/website#503).
* **[website] Add breathing room between navbar and hero on OSS Health**: Minor styling fix for the OSS Health section ([**@tym83**](https://github.com/tym83) in cozystack/website#500).
* **[website] Fix og social badge image and title**: Updates the social badge image and title ([**@tym83**](https://github.com/tym83) in cozystack/website#487).
* **[website] Update managed apps reference for v1.2.1**: Automated managed-apps reference update ([**@cozystack-bot**](https://github.com/cozystack-bot) in cozystack/website#464).
* **[external-apps-example] Replace MongoDB example with Minecraft apps**: Refactors the external apps example to use the ApplicationDefinition API with Minecraft server applications ([**@lexfrei**](https://github.com/lexfrei) in cozystack/external-apps-example#2).
* **docs: update README introductory description**: Refines the platform positioning and improves clarity on core capabilities in the main README ([**@tym83**](https://github.com/tym83) in #2409).
## Governance
* **Add Mattia Eleuteri ([@mattia-eleuteri](https://github.com/mattia-eleuteri)) as Maintainer**: CSI, Storage, Networking & Security ([**@tym83**](https://github.com/tym83) in #2345).
* **Add Matthieu Robin ([@matthieu-robin](https://github.com/matthieu-robin)) as Maintainer**: Managed applications, platform quality, and benchmarking ([**@tym83**](https://github.com/tym83) in #2346).
## Contributors
We'd like to thank all contributors who made this release possible:
Patch release covering a TenantNamespace IDOR fix in the API, a destructive `post-upgrade` hook removed from the etcd chart, kamaji controller stability, a `linstor-csi` bump that fixes live migration on Protocol-A/B DRBD resources, the missing `linstor-gui` build wiring, and a velero RBAC fix that unblocked installs on bundles without Velero.
## Security
* **fix(api): prevent IDOR in TenantNamespace Get and Watch handlers**: Two IDOR (Insecure Direct Object Reference) vulnerabilities allowed authenticated users to read TenantNamespace metadata they had no RoleBinding for. The `Get` and `Watch` handlers now go through a new `hasAccessToNamespace()` helper that lists RoleBindings scoped only to the target namespace (orders of magnitude faster than the previous all-cluster scan), returns `NotFound` instead of leaking existence on unauthorized access, and applies the same check on the `Watch` filter path. Includes regression tests for the unauthorized paths. ([**@IvanHunters**](https://github.com/IvanHunters) in #2471, backport #2524)
## Features
* **feat(linstor): bump linstor-csi to v1.10.6 with Protocol-C dual-attach fix**: Live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with `Protocol C required`. `linstor-csi` v1.10.6 now installs a `Protocol=C` override on the resource-definition during dual-attach and reverts it on detach, so `replicated-async` StorageClasses and other Protocol-A/B resource groups support live migration without manual `drbdadm adjust` intervention. ([**@kvaps**](https://github.com/kvaps) in #2496, backport #2505)
## Fixes
* **fix(backups): move velero-configmap Role to velero chart**: The `backupstrategy-controller` (a default package) declared a Role/RoleBinding scoped to the `cozy-velero` namespace for managing `ResourceModifier` ConfigMaps. On bundles where Velero was not enabled, that namespace did not exist and the HelmRelease failed with `namespaces "cozy-velero" not found`, blocking installation. The Role/RoleBinding now lives in the velero chart, so it is created only when velero is actually deployed. ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2459, backport #2467)
* **fix(etcd): remove destructive post-upgrade cert-regeneration hook**: The etcd chart ran a `post-upgrade` Helm hook on every upgrade that deleted etcd TLS Secrets (`etcd-ca-tls`, `etcd-peer-ca-tls`, `etcd-client-tls`, `etcd-peer-tls`, `etcd-server-tls`) and then deleted etcd pods, forcing cert-manager to re-issue the entire etcd CA chain. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 → 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+<git-hash>` (always `< 2.6.1` per semver) and after the underlying `rotationPolicy: Always` issue was fixed in `47d81f70`. The hook is now removed entirely. ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2462, backport #2511)
* **fix(kamaji): increase memory limits and add startup probe**: The kamaji controller frequently entered CrashLoopBackOff due to OOMKills (exit 137) within ~20–25 seconds of startup, with the readiness probe failing while the controller was still finishing initialization. Memory limit raised from 500Mi to 512Mi, request from 100Mi to 256Mi, and a 60-second startup probe (12 attempts × 5s periods) is added so the controller has room to boot before liveness/readiness probes engage. ([**@IvanHunters**](https://github.com/IvanHunters) in #2421, backport #2491)
## Build
* **build(linstor): include linstor-gui in root image build target**: The `linstor-gui` package (added in #2382) was never wired into the root `Makefile`'s `build:` target, so CI never built or published the image. `ghcr.io/cozystack/cozystack/linstor-gui` returned `NAME_UNKNOWN` and `values.yaml` stayed pinned to `tag: 2.3.0` without a digest. The missing build line is added so the next CI run publishes the image and the per-package `Makefile` digest-pins `values.yaml` automatically. ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2498, backport #2518)
## Contributors
Thanks to everyone who contributed to this patch release:
# Pinned to the DCGM Exporter image tag shipped by the gpu-operator
# chart under packages/system/gpu-operator/charts/gpu-operator/values.yaml
# (dcgmExporter.version = 4.1.1-4.0.4-ubuntu22.04). When that image is
# bumped, refresh this file from the matching tag in the NVIDIA/dcgm-exporter
# repo and re-run `./hack/cozytest.sh hack/check-gpu-recording-rules.bats`.
# Format
# If line starts with a '#' it is considered a comment
# DCGM FIELD, Prometheus metric type, help message
# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
# Power
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).
# PCIE
# DCGM_FI_PROF_PCIE_TX_BYTES, counter, Total number of bytes transmitted through PCIe TX via NVML.
# DCGM_FI_PROF_PCIE_RX_BYTES, counter, Total number of bytes received through PCIe RX via NVML.
DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.
# Utilization (the sample period varies depending on the product)
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
DCGM_FI_DEV_ENC_UTIL, gauge, Encoder utilization (in %).
DCGM_FI_DEV_DEC_UTIL , gauge, Decoder utilization (in %).
# Errors and violations
DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
# DCGM_FI_DEV_POWER_VIOLATION, counter, Throttling duration due to power constraints (in us).
# DCGM_FI_DEV_THERMAL_VIOLATION, counter, Throttling duration due to thermal constraints (in us).
# DCGM_FI_DEV_SYNC_BOOST_VIOLATION, counter, Throttling duration due to sync-boost constraints (in us).
# DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in us).
# DCGM_FI_DEV_LOW_UTIL_VIOLATION, counter, Throttling duration due to low utilization (in us).
# DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in us).
# Memory usage
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
# ECC
# DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, Total number of single-bit volatile ECC errors.
# DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total number of double-bit volatile ECC errors.
# DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, counter, Total number of single-bit persistent ECC errors.
# DCGM_FI_DEV_ECC_DBE_AGG_TOTAL, counter, Total number of double-bit persistent ECC errors.
# Retired pages
# DCGM_FI_DEV_RETIRED_SBE, counter, Total number of retired pages due to single-bit errors.
# DCGM_FI_DEV_RETIRED_DBE, counter, Total number of retired pages due to double-bit errors.
# DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.
# NVLink
# DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total number of NVLink flow-control CRC errors.
# DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL, counter, Total number of NVLink data CRC errors.
# DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL, counter, Total number of NVLink retries.
# DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, Total number of NVLink recovery errors.
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, counter, Total number of NVLink bandwidth counters for all lanes.
# DCGM_FI_DEV_NVLINK_BANDWIDTH_L0, counter, The number of bytes of active NVLink rx or tx data including both header and payload.
# VGPU License status
DCGM_FI_DEV_VGPU_LICENSE_STATUS, gauge, vGPU License status
# Remapped rows
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for correctable errors
DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, Whether remapping of rows has failed
# Static configuration information. These appear as labels on the other metrics
DCGM_FI_DRIVER_VERSION, label, Driver Version
# DCGM_FI_NVML_VERSION, label, NVML Version
# DCGM_FI_DEV_BRAND, label, Device Brand
# DCGM_FI_DEV_SERIAL, label, Device Serial Number
# DCGM_FI_DEV_OEM_INFOROM_VER, label, OEM inforom version
# DCGM_FI_DEV_ECC_INFOROM_VER, label, ECC inforom version
# DCGM_FI_DEV_POWER_INFOROM_VER, label, Power management object inforom version
# DCGM_FI_DEV_INFOROM_IMAGE_VER, label, Inforom image version
# DCGM_FI_DEV_VBIOS_VERSION, label, VBIOS version of the device
# Datacenter Profiling (DCP) metrics
# NOTE: supported on Nvidia datacenter Volta GPUs and newer
DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, Ratio of time the graphics engine is active.
# DCGM_FI_PROF_SM_ACTIVE, gauge, The ratio of cycles an SM has at least 1 warp assigned.
# DCGM_FI_PROF_SM_OCCUPANCY, gauge, The ratio of number of warps resident on an SM.
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE, gauge, Ratio of cycles the device memory interface is active sending or receiving data.
# DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles the fp64 pipes are active.
# DCGM_FI_PROF_PIPE_FP32_ACTIVE, gauge, Ratio of cycles the fp32 pipes are active.
# DCGM_FI_PROF_PIPE_FP16_ACTIVE, gauge, Ratio of cycles the fp16 pipes are active.
DCGM_FI_PROF_PCIE_TX_BYTES, gauge, The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
DCGM_FI_PROF_PCIE_RX_BYTES, gauge, The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
Can't render this file because it has a wrong number of fields in line 12.
| `bootstrap.enabled` | Whether to restore from a backup. | `bool` | `false` |
| `bootstrap.recoveryTime` | Timestamp (RFC3339) for point-in-time recovery; empty means latest. | `string` | `""` |
| `bootstrap.oldName` | Previous cluster name before deletion. | `string` | `""` |
| `bootstrap.serverName` | Barman server name (S3 path prefix) used by the original cluster when writing backups. Set this only when the original cluster had an explicit barmanObjectStore.serverName that differed from its Kubernetes resource name. | `string` | `""` |
"description":"Timestamp (RFC3339) for point-in-time recovery; empty means latest.",
"type":"string",
"default":""
},
"serverName":{
"description":"Barman server name (S3 path prefix) used by the original cluster when writing backups. Set this only when the original cluster had an explicit barmanObjectStore.serverName that differed from its Kubernetes resource name.",
## @field {bool} enabled - Whether to restore from a backup.
## @field {string} [recoveryTime] - Timestamp (RFC3339) for point-in-time recovery; empty means latest.
## @field {string} oldName - Previous cluster name before deletion.
## @field {string} [serverName] - Barman server name (S3 path prefix) used by the original cluster when writing backups. Set this only when the original cluster had an explicit barmanObjectStore.serverName that differed from its Kubernetes resource name.
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
| `gpus[i].name` | The name of the GPU resource to attach. | `string` | `""` |
| `cpuModel` | Model specifies the CPU model inside the VMI. List of available models https://github.com/libvirt/libvirt/tree/master/src/cpu_map | `string` | `""` |
| `resources` | Resource configuration for the virtual machine. | `object` | `{}` |
| `resources.cpu` | Number of CPU cores allocated. | `quantity` | `""` |
| `external` | Enable external access from outside the cluster. | `bool` | `false` |
| `externalMethod` | Method to pass through traffic to the VM. | `string` | `PortList` |
| `externalPorts` | Ports to forward from outside the cluster. | `[]int` | `[22]` |
| `externalAllowICMP` | Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect. | `bool` | `true` |
| `runStrategy` | Requested running state of the VirtualMachineInstance | `string` | `Always` |
| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
| `gpus[i].name` | The name of the GPU resource to attach. | `string` | `""` |
| `cpuModel` | Model specifies the CPU model inside the VMI. List of available models https://github.com/libvirt/libvirt/tree/master/src/cpu_map | `string` | `""` |
| `resources` | Resource configuration for the virtual machine. | `object` | `{}` |
| `resources.cpu` | Number of CPU cores allocated. | `quantity` | `""` |
"description":"Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect.",
"type":"boolean",
"default":true
},
"runStrategy":{
"description":"Requested running state of the VirtualMachineInstance",
## @param {bool} externalAllowICMP - Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect.
externalAllowICMP:true
## @enum {string} RunStrategy - Requested running state of the VirtualMachineInstance
| `resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` |
| `resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `micro` |
## Exposure mode
The ingress Service type is driven by the cluster-wide `publishing.exposure` value in the platform chart, not by any key in this package. Two modes exist:
- `externalIPs` (default) has three rendered shapes:
- Release namespace matches `publishing.ingressName` AND `publishing.externalIPs` is non-empty → Service is `ClusterIP` with `Service.spec.externalIPs` set from that list and `externalTrafficPolicy: Cluster`.
- Release namespace matches `publishing.ingressName` but `publishing.externalIPs` is empty → Service falls back to `type: LoadBalancer` with `externalTrafficPolicy: Local`.
- Release namespace does not match `publishing.ingressName` (non-root tenants) → Service is `type: LoadBalancer` with `externalTrafficPolicy: Local`.
`Service.spec.externalIPs` is deprecated upstream in Kubernetes v1.36 (KEP-5707); plan migration before v1.40.
- `loadBalancer` — Service is `type: LoadBalancer` with `externalTrafficPolicy: Local`, and a `CiliumLoadBalancerIPPool` makes the addresses in `publishing.externalIPs` allocatable via Cilium LB IPAM. Requires `publishing.externalIPs` to contain at least one non-empty address (render fails otherwise) and assumes the addresses are already routed to a cluster node (floating IP / upstream router). See the inline comment on `publishing.exposure` in the platform chart for full caveats, including the note that switching the value on a running cluster causes the ingress Service to be recreated.
This setting only migrates ingress-nginx away from `Service.spec.externalIPs`. Other cozystack components that use the same deprecated field (e.g. the `vpn` app) must be migrated separately before Kubernetes v1.40 flips the `AllowServiceExternalIPs` feature gate off.
Some files were not shown because too many files have changed in this diff
Show more
