## What this PR does
Documentation: https://github.com/cozystack/website/pull/517
Integrates [HAMi](https://github.com/Project-HAMi/HAMi) v2.8.1 (CNCF
Sandbox) into Cozystack as a system-level package for fractional GPU
sharing in tenant Kubernetes clusters.
The integration covers three layers:
- **System chart** (`packages/system/hami/`): Vendors upstream HAMi Helm
chart with device plugin, scheduler extender, mutating webhook, and
RuntimeClass configuration. The broken DRA subchart was removed — it
renders resources even when disabled and references unpublished images.
- **Kubernetes app addon** (`packages/apps/kubernetes/`): HAMi exposed
as an optional toggle (`hami.enabled`). Automatically disables GPU
Operator's native device plugin when active to avoid conflicts. Enforces
hard dependency on GPU Operator.
- **Platform registration** (`packages/core/platform/`): HAMi declared
as PackageSource with gpu-operator dependency, included in the iaas
bundle.
**Known limitation**: HAMi-core relies on a private glibc symbol
(`_dl_sym`) removed in glibc 2.34, which breaks compute isolation on
modern container images (Ubuntu 22.04+, PyTorch/TensorFlow official
images). Alpine/musl is entirely incompatible. See the package README
for details and upstream issue references.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(hami): add HAMi GPU virtualization as an optional system package for fractional GPU sharing. Enables memory and compute isolation for NVIDIA GPUs across tenant workloads. Integrates with GPU Operator and can be enabled per-cluster via the hami.enabled toggle. Note: compute isolation requires glibc < 2.34 in workload containers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added HAMi GPU virtualization middleware as an optional cluster addon
to enable fractional GPU sharing.
* HAMi addon includes an enable toggle (default: disabled) and a
customizable Helm values override; when enabled it enforces GPU Operator
presence and adjusts GPU Operator values accordingly.
* New HelmRelease integration for deploying HAMi with dependency
ordering and conditional rendering.
* **Documentation**
* Added HAMi setup guide, compatibility notes, and updated cluster addon
configuration reference.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
In environments where control-plane nodes are not available (e.g., AWS
managed nodes, Cozy-in-Cozy tenant Kubernetes clusters), the DaemonSet
cannot schedule any pods because no nodes match this selector.
## What this PR does
This PR adds ability to replace DaemonSet by Deployment with custom
number of replicas for HA purpose. Default values saves current
behavior.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Optional Deployment mode for lineage-controller-webhook with
configurable replicas
* PodDisruptionBudget support when Deployment mode is enabled
* **Updates**
* Improved pod scheduling: node affinity replaces nodeSelector, optional
tolerations, and pod anti-affinity when using Deployment
* Clarified behavior for local API endpoint scheduling (only applies
when pod lands on nodes running an API server)
* **Documentation**
* Added README with usage examples and parameter details
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The `update:` recipe now reproduces the top-level vendoring overrides
(remove broken hami-dra subchart, clear Chart.yaml dependencies, strip
dra/hami-dra/podSecurityPolicy from upstream values.yaml) after
`helm pull`, so they no longer silently disappear on the next bump.
Template-level patches (DRA guards in scheduler/*, indent fix in
device-plugin/monitorservice.yaml) are documented with rationale and
commit references — they remain a manual step because automating them
would be fragile across upstream template restructures.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The HelmRelease at packages/apps/kubernetes/templates/helmreleases/hami.yaml
references chartRef.name 'cozystack-kubernetes-application-kubevirt-kubernetes-hami',
but that component was missing from the kubernetes-application PackageSource.
The HelmRelease would sit in Stalled: ArtifactNotFound at install time.
Add kubernetes-hami next to kubernetes-gpu-operator under variant kubevirt,
mirroring the existing pattern for every other tenant-cluster addon. The
standalone cozystack.hami PackageSource is retained — same shape as
gpu-operator, which is registered both standalone (for bundles/iaas) and
as a kubernetes-application component (for tenant-cluster HelmReleases).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Match the pattern used by every other tenant-cluster addon HelmRelease
(gpu-operator, cilium, fluxcd, ...): only render when the upstream etcd
namespace is ready, not just when the addon is enabled. Add values-ci.yaml
to the HAMi test suite so the new gate has the fixture it needs.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The HAMi-driven default of disabling gpu-operator's device plugin is
applied via valuesOverride and can be re-enabled by users for advanced
topologies (mixed HAMi / vanilla NVIDIA device plugin pools). Document
this explicitly so the README aligns with the existing template merge
order.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
HAMi#173 was closed as "not planned" and only suggests a typo fix
(< 2.3.0 → < 2.30); it does not establish the actual 2.34 boundary.
HAMi-core#174 (symbol-level cause) and HAMi#1190 (empirical per-glibc
behavior) cover the same ground accurately.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The auto-generated changelog only listed #2459/#2467 (velero-configmap
Role move). Five additional PRs were backported and merged into
release-1.3 between then and the v1.3.1 tag (2026-04-28):
- #2471/#2524 - fix(api): IDOR in TenantNamespace Get/Watch
- #2496/#2505 - feat(linstor): linstor-csi v1.10.6 (Protocol-C dual-attach)
- #2462/#2511 - fix(etcd): remove destructive post-upgrade hook
- #2421/#2491 - fix(kamaji): memory limits + startup probe
- #2498/#2518 - build(linstor): wire linstor-gui into root build target
Update the release date to match the actual tag (2026-04-28), rewrite
the intro paragraph, and add @kvaps to contributors.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The AI-generated v1.3.1 changelog (#2480) was generated from
git log v1.3.0..main rather than git log v1.3.0..v1.3.1, because
the workflow checked out main while the v1.3.1 tag points to
release-1.3. As a result the changelog included:
- 8 PRs that were merged to main but never shipped in v1.3.1
- 6 backport PRs that were merged to release-1.3 *after* v1.3.1
was tagged
- Both originals and their backports as separate duplicate entries
- A 2024 PR (#435) that has nothing to do with this range
- Generic "Documentation updates" placeholders for website entries
- Title duplicated as both the bold label and the description
(`* **fix(...): X**: fix(...): X (...)`)
- The cozystack-ci bot listed as a human contributor
The actual v1.3.1 release range (v1.3.0..v1.3.1) contains exactly
one user-facing change:
41bcb0be [Backport release-1.3] fix(backups): move
velero-configmap Role to velero chart (#2467)
which is the backport of #2459 (myasnikovdaniil) shipped via
#2467 (IvanHunters). This commit replaces the contents of
docs/changelogs/v1.3.1.md with that one entry, the matching
two-person contributors list, and the standard footer.
The workflow + docs fixes that prevent this regression for future
patch releases will land in a separate PR against main.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds GPU observability to Cozystack: five Grafana dashboards, a set of
VictoriaMetrics recording rules with a throttle-regression alert, and
reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter
on Talos when running GPU workloads directly in pods.
### Components
1. **Dashboards** — `dashboards/gpu/*.json`, registered in
`packages/system/monitoring/dashboards-infra.list`. All follow the
project convention: the Prometheus data source is selected through the
`$ds_prometheus` template variable.
- `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics
engine, memory copy), VRAM, power, temperature, and health (XID errors,
power/thermal throttling). All metrics are shown per-GPU, filterable by
the `$Hostname` selector.
- `gpu-fleet` — cluster-wide inventory, capacity, utilization, and
per-node power draw as percentage of TDP.
- `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh
accounting; utilization and power are shown per-node (DCGM cannot
attribute hardware metrics to namespaces).
- `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a
per-namespace requests/limits table. All counts derive from the
`namespace:gpu_count:allocated` recording rule, which excludes
`Failed`/`Succeeded` pod carryover.
- `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation,
util-per-watt, and per-GPU power/thermal throttle fractions.
2. **Recording rules and alert** —
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`.
Four rule groups:
- `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview
panels.
- `gpu.recording.node.1m` — per-node hardware aggregates (grouped by
`Hostname`) plus per-namespace allocation counts from kube resource
requests; filters `cozy-*`/`kube-*`.
- `gpu.recording.efficiency.1m` — per-GPU tensor saturation and
util-per-watt, plus per-GPU power/thermal throttle fractions bounded via
`clamp_max(..., 1)`.
- `gpu.recording.throttle.validation.5m` — houses the
`GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp
throttle fraction exceeds 1.0. This guards against DCGM counter-unit
drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10
with DCGM 3.x.
Rules are safe on clusters without DCGM — they evaluate to empty series
when no matching metrics are scraped.
3. **Reference manifests** under
`packages/system/gpu-operator/examples/`. These are **not** templates;
they document one working configuration:
- `values-native-talos.yaml` — Cozystack Package values for the
native-pod scenario.
- `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with
profiling, ECC, throttling, and energy counters used by the dashboards
and recording rules.
- `nvidia-driver-compat.yaml` — DaemonSet that stages
`libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a
location the GPU Operator validator inspects. Workaround for
[NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687).
Requires `pod-security.kubernetes.io/enforce: privileged` on the target
namespace when the cluster enforces baseline/restricted
PodSecurityStandards.
- `README.md` — explains the two deployment paths (sandbox vs. native),
which DCGM metrics each dashboard depends on, and how the pieces fit
together.
4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats`
enforces that every recording-rule reference in tracked dashboards
resolves, and that every DCGM metric referenced by the rules is declared
in either the upstream default CSV snapshot
(`hack/dcgm-default-counters.csv`) or the custom CSV shipped under
`examples/`.
### Why reference manifests, not templates
The out-of-the-box `packages/system/gpu-operator/values-talos.yaml`
targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via
VFIO, driver disabled on the host, device plugin disabled. That is the
sensible default for Cozystack today.
Running GPU workloads directly in pods on Talos is also possible but
requires a different configuration that makes assumptions not every user
shares:
- the NVIDIA Talos system extension must be installed on GPU nodes,
- the operator's own driver and toolkit components must be disabled
because the extension provides them,
- the validator's hardcoded paths need a workaround until
NVIDIA/gpu-operator#1687 lands upstream.
Shipping those as active templates would silently impose those
assumptions. Shipping them as reference files next to the package lets
operators opt in with eyes open and understand the moving parts before
applying them.
### Scope note
The dashboards and recording rules are useful independently of the
reference manifests — any cluster that scrapes DCGM Exporter (however it
is installed) will populate them. The reference manifests are
self-contained and do not change any default behavior of the
`gpu-operator` package.
### Screenshots
<img width="2824" height="1055" alt="2026-04-27_12-36-52"
src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a"
/>
<img width="2913" height="1255" alt="2026-04-27_12-36-39"
src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229"
/>
<img width="2932" height="1251" alt="2026-04-27_12-36-30"
src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23"
/>
<img width="2864" height="1222" alt="2026-04-27_12-36-00"
src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9"
/>
<img width="2958" height="1226" alt="image"
src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76"
/>
### Release note
```release-note
feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added five GPU dashboards for performance, efficiency, fleet overview,
quotas/allocation, and tenant usage (utilization, power, throttling,
allocation, billing).
* Added cluster recording rules and a throttling validation alert; new
Prometheus-derived metrics and per-GPU rankings/timeseries.
* **Documentation**
* Added GPU Operator examples and README documenting native Talos setup
and required DCGM counters.
* **Tests**
* Added validation tests ensuring dashboard ↔ recording-rule and DCGM
metric consistency.
* **Chores**
* Registered new GPU dashboards in infra listings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Tests need _namespace.etcd from values-ci.yaml after the merge
introduced an etcd-namespace guard on the gpu-operator HelmRelease
condition.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## Summary
Fixed two IDOR (Insecure Direct Object Reference) vulnerabilities in the
TenantNamespace API handlers that allowed authenticated users to access
metadata of tenant namespaces without proper authorization checks.
## Changes
### New optimized function: hasAccessToNamespace()
- Lists RoleBindings **only in the target namespace** instead of all
cluster RoleBindings
- Used by Get() and Watch() for single-namespace access checks
- Order of magnitude faster than the previous approach
### Get() handler
- Uses `hasAccessToNamespace()` for efficient authorization
- Returns `NotFound` instead of `Forbidden` to prevent tenant
enumeration
- Now correctly enforces RoleBinding-based access control
### Watch() handler
- Uses `hasAccessToNamespace()` for per-event authorization
- Logs authorization errors with klog for security audit
- Events for unauthorized namespaces are silently filtered out
- Ensures users only receive watch events for namespaces they have
access to
### Additional fixes
- Fixed ServiceAccount subject handling when namespace is empty
(defaults to RoleBinding namespace)
- Added proper error logging in Watch handler
## Performance Impact
| Handler | Before | After |
|---------|--------|-------|
| List() | List all RoleBindings × 1 | No change ✅ |
| Get() | List all RoleBindings × 1 | List RoleBindings in 1 namespace 🚀
|
| Watch() | List all RoleBindings × N events | List RoleBindings in 1
namespace × N events 🚀 |
**For Watch with 100 events:**
- Before: 100 × (all cluster RoleBindings) = catastrophic
- After: 100 × (1-5 RoleBindings in namespace) = fast + cached by
controller-runtime
## Security Impact
**Before**: Any authenticated user could:
- Read metadata (labels, annotations, creation time) of any `tenant-*`
namespace via `Get()`
- Stream all tenant namespace events via `Watch()`, including
creation/modification/deletion
**After**: Users can only access tenant namespaces they have explicit
RoleBindings for, matching the behavior of the `List()` handler.
## Testing
Manually verified:
- Users can only `get` their own tenant namespaces
- Users can only `watch` events for their own tenant namespaces
- Unauthorized access returns `NotFound` (not `Forbidden`) to prevent
enumeration
- `List()` behavior remains unchanged and consistent with `Get()` and
`Watch()`
- Authorization errors are logged for security audit
## Checklist
- [x] Code follows project style and conventions
- [x] Security vulnerability is fully mitigated
- [x] Authorization logic is consistent across List/Get/Watch handlers
- [x] Performance optimized per feedback from @timofei.larkin
- [x] Error logging added for security audit
- [ ] Unit tests added (can be done in follow-up PR)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enforced per-namespace access control for Get and Watch; inaccessible
namespaces return Forbidden or NotFound as appropriate
* Forwarded field/label selectors to upstream watches and added
defensive filtering to skip inaccessible events (logged)
* Improved ServiceAccount subject namespace fallback and
privileged-group bypass
* **Tests**
* Added security tests covering RoleBinding subjects, groups, privileged
bypasses, ServiceAccounts, and access-denied behaviors
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Add `serverName` field to bootstrap configuration to support backup
recovery when the Barman server name in `backup.info` differs from the
Kubernetes cluster name.
This fixes "no target backup found" errors during CloudNativePG recovery
operations.
## Problem
CloudNativePG forms the backup path as `destinationPath + "postgres-" +
oldName`, but searches for backups using the `server_name` field from
`backup.info`. When these values don't match (e.g., cluster name is
`grafana` but `server_name` is `cloud`), recovery fails with "no target
backup found".
## Solution
- Add optional `serverName` parameter to `bootstrap` configuration
- When specified, CloudNativePG uses this value to search for backups in
S3
- Falls back to `oldName` when `serverName` is not provided (backwards
compatible)
## Changes
- Add `ServerName` field to PostgreSQL CRD type definition
- Add conditional `serverName` to Cluster `externalClusters` template
- Update `values.yaml` and README.md with `serverName` documentation
- Regenerate `values.schema.json` and `postgres.yaml` CRD
## Test plan
- [ ] Deploy postgres cluster with backups enabled
- [ ] Create backup
- [ ] Create new postgres instance with `bootstrap.enabled=true`,
`bootstrap.oldName=<original-name>`, and
`bootstrap.serverName=<value-from-backup.info>`
- [ ] Verify recovery completes successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an optional bootstrap.serverName setting to specify the Barman
server name from the original cluster’s backup.info (used when that
server name differs from the Kubernetes cluster name).
* **Documentation**
* Clarified that bootstrap.oldName must match the serverName value
recorded in backup.info; updated docs and examples to reflect the new
bootstrap.serverName option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an opt-in `publishing.exposure=loadBalancer` mode for the
ingress-nginx Service as a migration path away from
`Service.spec.externalIPs`, which is deprecated upstream in Kubernetes
v1.36
([KEP-5707](https://github.com/kubernetes/enhancements/issues/5707),
[kubernetes#137293](https://github.com/kubernetes/kubernetes/pull/137293)).
The `AllowServiceExternalIPs` feature gate is expected to default to off
around v1.40 and the implementation to be removed around v1.43.
Stacked on top of #2464 (cilium v1.19.3 bump) — depends on it for the
`CiliumLoadBalancerIPPool` at `cilium.io/v2`.
### Behavior
- New platform value `publishing.exposure` — enum `externalIPs |
loadBalancer`, default `externalIPs` (current behavior unchanged on
upgrade).
- Plumbed through `cozystack-values` into each tenant's ingress
HelmRelease via the new `expose-mode` key.
- When `exposure=loadBalancer` and the current namespace matches
`publishing.ingressName`, the Service becomes `type: LoadBalancer` with
`externalTrafficPolicy: Local` and a `CiliumLoadBalancerIPPool`
announces the addresses from `publishing.externalIPs` via Cilium LB
IPAM.
- The pool uses a namespace-only `serviceSelector`
(`io.kubernetes.service.namespace: <ns>`) — any LoadBalancer Service in
the tenant namespace draws from it. See "Pool ownership" below.
- IPv4 addresses get `/32` CIDRs, IPv6 addresses get `/128`. Mixed
families supported. Pre-CIDR entries (`192.0.2.10/32`) are accepted
without double-suffixing.
- Unknown values, `loadBalancer` with empty externalIPs, and stray empty
entries from `publishing.externalIPs` are rejected at render time with
explicit error messages.
### Pool ownership
The `CiliumLoadBalancerIPPool` is rendered from
`packages/apps/tenant/templates/cilium-lb-pool.yaml`, not from the
ingress chart. The tenant chart is the per-tenant owner of cross-cutting
resources (Namespace, `cozystack-values` Secret, HelmReleases for
ingress and gateway), so one pool per tenant lives there.
Cilium LB IPAM rejects overlapping CIDRs across pools regardless of
`serviceSelector` — the last-added pool gets `cilium.io/PoolConflict`
and stops allocating. Keeping the pool in the ingress chart would
collide with the Gateway-API PR (#2470), which materialises its own
LoadBalancer Service from the same `publishing.externalIPs` range when
`tenant.spec.gateway=true`. Moving the pool to the tenant chart with a
namespace-only selector lets a single pool back both services
(ingress-nginx today, a Cilium Gateway Service once #2470 lands).
Only the ingress-loadBalancer signal is wired here
(`_cluster.expose-mode=loadBalancer` + `.Values.ingress=true` +
publishing tenant). #2470 rebases on top of this PR, drops its own
`packages/extra/gateway/templates/cilium-lb-pool.yaml`, and adds an OR
branch for `.Values.gateway` in the tenant template.
### Scope
Only the ingress-nginx Service is migrated by this setting. Other
cozystack components that still write `Service.spec.externalIPs`
directly (notably the `vpn` app at
`packages/apps/vpn/templates/service.yaml`) need separate follow-up
before Kubernetes v1.40.
### Tests
- `packages/extra/ingress/tests/exposure_test.yaml` — 10 helm-unittest
cases on the ingress-nginx Service: type / externalTrafficPolicy /
externalIPs assertion for both modes, unknown-mode rejection
(case-sensitive enum), empty-IPs failure, empty-entry filtering,
non-publishing-tenant fallback.
- `packages/apps/tenant/tests/exposure_test.yaml` — 9 cases on the pool
itself: IPv4, IPv6, mixed, pre-CIDR input, empty-entry filtering,
`ingress=false` in publishing tenant (no pool), non-publishing tenant
(no pool), empty externalIPs (no pool).
- Both suites are auto-discovered by `hack/helm-unit-tests.sh` via the
`test:` target in each package's Makefile.
### Caveats (copied from the inline `values.yaml` comment)
- `loadBalancer` mode uses `externalTrafficPolicy: Local`. The external
IP must already be routed to a node that hosts an ingress pod (floating
IP / upstream router / podAntiAffinity).
- Cilium does not announce the IP on its own unless L2 announcements or
BGP are enabled in the Cilium values (disabled by default in cozystack).
- Switching the value on a running cluster causes the ingress-nginx
Service to be recreated (`upgrade.force: true` on the HelmRelease +
Service kind change); expect a brief interruption of ingress traffic.
### Release note
```release-note
feat(ingress): add opt-in publishing.exposure=loadBalancer mode that uses type: LoadBalancer + CiliumLoadBalancerIPPool instead of the deprecated Service.spec.externalIPs. Default (externalIPs) preserves existing behavior.
```
## What this PR does
Closes a bootstrap race where the `cnpg-webhook-service` Service gets
its EndpointSlice populated and the data plane (kube-proxy / Cilium)
programmed a second or two after `helm install --wait` on
`cozy-postgres-operator` declares the HelmRelease Ready. Any HelmRelease
that `dependsOn: postgres-operator` and creates a
`postgresql.cnpg.io/v1` resource in that window (cozy-keycloak, tenant
Postgres apps) has kube-apiserver's call to `mcluster.cnpg.io` fail with
```
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
```
which fails the downstream release's install.
The fix is a post-install / post-upgrade Helm hook (ServiceAccount +
ClusterRole + ClusterRoleBinding + Job) that probes `/readyz` on the
webhook through the apiserver service proxy. Apiserver's service proxy
uses the same endpoint-resolution and apiserver-initiated pod dial as
the admission webhook path, so once `/readyz` answers through the proxy,
the subsequent admission call will also succeed. Helm `--wait` blocks
the install from completing until the Job exits 0, so the HelmRelease
Ready condition does not lie anymore.
Hardened per review:
- RBAC scoped to a single `services/proxy` resourceName
(`https:cnpg-webhook-service:webhook-server`) — the exact string the
apiserver URL path parser expects for the Service proxy subresource.
- A drift-guard helm-unittest test renders the vendored cnpg Service
template and fails if its `metadata.name` / `ports[0].name` diverge from
the literals in the hook, so a future `make update` that renames the
service forces this template to be updated in the same change.
- Image digest-pinned (`clastix/kubectl:v1.32@sha256:…`) with a
`renovate:` annotation; `imagePullPolicy: IfNotPresent` when
digest-pinned, `Always` when tag-only.
- Job runs under PSA-restricted-compatible securityContext (non-root,
seccomp RuntimeDefault, readOnlyRootFilesystem, drop ALL caps).
- `activeDeadlineSeconds` is derived from `maxAttempts × sleepSeconds +
60s` so a values override raising retries does not get silently cut by a
fixed deadline.
- `backoffLimit: 2` (configurable) so a transient pod-level failure
(image pull rate limit, OOM, CNI hiccup) does not fail the whole
HelmRelease.
- On timeout the Job prints the last `kubectl get --raw` stderr so the
operator can distinguish DNS / refused / 401 / TLS from the Job logs.
21 helm-unittest assertions cover ordering, RBAC/URL parity, subchart
drift, image policy, retry-loop bounds, securityContext, and the
deadline-scaling invariant. Wired via `make test` in
`packages/system/postgres-operator/Makefile`.
First surfaced on #2470 E2E run 24862782568 where
`cozy-keycloak/keycloak` failed with the exact signature above. Not tied
to that PR's branch — this is independently applicable to `main`.
### Release note
```release-note
fix(postgres-operator): add a post-install readiness gate that blocks the HelmRelease from reporting Ready until the cnpg admission webhook actually serves through the cluster Service, preventing "dial tcp <svc-ip>:443: connect: connection refused" on the first HelmRelease that depends on postgres-operator (keycloak, tenant Postgres apps).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added webhook readiness validation during install/upgrade to block
completion until the admission webhook is reachable.
* New configuration options for the readiness probe image and
retry/timeouts.
* **Tests**
* Introduced a comprehensive test suite validating rendered manifests,
probe behavior, RBAC, image rendering, and retry/timeout logic.
* **Chores**
* Added a test entry point to the project Makefile to run chart/unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Collapse the chart to a single Deployment shape that mirrors
cozystack-api: 2 replicas, soft (preferred) nodeAffinity to
node-role.kubernetes.io/control-plane via Exists, permissive
tolerations, soft podAntiAffinity on hostname, an unconditional
PodDisruptionBudget with maxUnavailable: 1, and a Service with
spec.trafficDistribution: PreferClose. The same shape works on
Talos / kubeadm / k3s and on managed Kubernetes / Cozy-in-Cozy
tenant clusters without per-distro overrides — fixing #2417 by
making the soft control-plane affinity gracefully fall back to
worker scheduling when no control-plane nodes are visible.
Drop the DaemonSet path entirely. The previous PR's deployment.enabled
toggle, the workload-kind switch in templates/workload.yaml, the
fail-guard for localK8sAPIEndpoint+nodeAffinity=[], the conditional
PDB, and the values-shape knobs for nodeAffinity and tolerations all
go away as a consequence. Override the standard Deployment fields
through the usual component-values mechanism if a non-default topology
is ever needed.
Mark localK8sAPIEndpoint.enabled deprecated and flip the default to
false. The flag injects KUBERNETES_SERVICE_HOST=status.hostIP, which
is only valid when the pod is actually scheduled on an apiserver-
bearing node. With the new soft control-plane affinity, the pod can
land off-control-plane and crash-loop. The latency motivation for the
flag is real but pending separate webhook performance work; once
addressed, the flag can be removed.
Revert all changes to packages/core/platform/images/migrations/migrations/20.
The earlier ds/...-or-deploy/... fallback was over-engineered: per
run-migrations.sh, migration 20 fires only when CURRENT < 20 (i.e.
direct upgrades from pre-0.37 to 1.3+), and that path is unsupported
anyway. The original ds/... rollout-status line is dead code on every
supported install and upgrade path.
Add helm-unittest coverage for: the default Deployment shape (replicas,
soft nodeAffinity, soft podAntiAffinity, tolerations); no env vars at
the default localK8sAPIEndpoint.enabled=false; PDB rendered
unconditionally with maxUnavailable: 1, including at replicas=1 (the
no-op case); replicas value drives spec.replicas; and that
localK8sAPIEndpoint.enabled=true does inject the env vars. The Service
test continues to assert trafficDistribution: PreferClose with no
internalTrafficPolicy.
Add a slim README documenting the topology, parameters, and the
deprecation note for localK8sAPIEndpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
- Hoist user identity extraction out of the Watch goroutine; reuse a
cached username and groups map across events instead of re-fetching
them per event. Watch now returns Unauthorized up front when no user
is present in the context, rather than failing silently per event.
- Switch the per-event access-check error log to structured klog.ErrorS
to comply with the project Go style guide.
- Strengthen TestGet_WithAccess to assert the concrete *TenantNamespace
type plus Name, Kind, and APIVersion, so type or metadata regressions
fail fast.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `linstor-gui` package (added in #2382) was never wired into the root
`Makefile`'s `build:` target, so the image is not built or published by
CI. `ghcr.io/cozystack/cozystack/linstor-gui` returns `NAME_UNKNOWN`
from the GHCR API, and `values.yaml` still pins `tag: 2.3.0` without a
digest because the per-package Makefile that rewrites the tag after a
successful push has never run in CI.
This PR adds the missing line so the next build publishes the image and
digest-pins `values.yaml` automatically.
### Release note
```release-note
build(linstor): include linstor-gui in root image build target so the image is built and published by CI (the chart previously referenced an image that did not exist in the registry)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Integrated the linstor-gui container image build into the main image
build workflow so the GUI image is produced as part of standard builds.
* Streamlined the GUI image build configuration to use consolidated
build arguments, improving consistency and maintainability of automated
image builds and metadata updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Bumps the vendored cozy-proxy chart and image tag from v0.2.0 to v0.3.0
by running `make update` in `packages/system/cozy-proxy`.
cozy-proxy v0.3.0 release:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
## Why
cozy-proxy v0.3.0 is the companion of #2501. The chart fix in #2501
finally honors the documented PortList semantics (only `externalPorts`
reachable, ICMP preserved by default), but only takes effect once
cozy-proxy v0.3.0 is in place — v0.2.0 has no port-aware logic, so the
chart fix silently no-ops on it. This PR ships that runtime side.
## Upgrade impact
For cozystack users this is a **bug fix, not a breaking change**.
cozy-proxy v0.3.0 introduces two contract changes upstream — (1)
label-only selector via `service.kubernetes.io/service-proxy-name:
cozy-proxy`, and (2) port-filter as the default ingress mode — both of
which are absorbed by the vm-instance chart (label landed in #2357,
`wholeIP` + `allowICMP` wired in #2501).
- **`externalMethod: WholeIP`** (default) — chart renders label +
`wholeIP: "true"` in both old and new versions. cozy-proxy keeps
passthrough. **No change.**
- **`externalMethod: PortList`** — previously the chart always rendered
`wholeIP: "true"` regardless of method, and cozy-proxy v0.2.0 had no
port-aware logic, so PortList was silently a no-op (all ports
reachable). After this bump + #2501, PortList behaves as documented:
only ports listed in `externalPorts` reach the VM. Users who configured
PortList but reached undeclared ports (relying on the silent no-op) will
see those ports become unreachable — that's the documented intent of
PortList finally taking effect. ICMP keeps working by default thanks to
`externalAllowICMP: true`.
- **DaemonSet rollover** — re-init of the nft table during cozy-proxy
pod restart is a brief blip on existing flows; same as any DaemonSet
upgrade.
Out-of-tree consumers using cozy-proxy directly (without the
`service-proxy-name` label, or relying on the absent-annotation
passthrough default) need to migrate per the upstream v0.3.0 release
notes — not a cozystack concern.
## Order of operations
This PR is functionally complete on its own (image bump), but the
user-facing PortList fix it enables only happens when #2501 also lands.
Recommended merge order:
1. #2501 first (chart adapts to the new contract)
2. This PR second (cozy-proxy DaemonSet picks up the new image)
The reverse order is also safe — cozy-proxy v0.3.0 with the unmodified
vm-instance chart still produces correct results for `externalMethod:
WholeIP` (most common case) and keeps PortList silently broken until
#2501 lands.
## Test plan
- [ ] `helm template packages/system/cozy-proxy` — image tag `v0.3.0`
rendered
- [ ] `make unit-tests` passes
- [ ] After #2501 lands, end-to-end: `externalMethod: PortList` with
`externalPorts: [22]` filters all other ports, ping works
- [ ] WholeIP-method VM Service unchanged after upgrade
## Companion PRs
- #2501 — vm-instance chart wires `wholeIP` / `allowICMP` per
`externalMethod`
- cozystack/cozy-proxy#13 (merged) — label-only selector + port-filter
default
- cozy-proxy v0.3.0 release notes —
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Released version 0.3.0 with updated Helm chart metadata and container
image configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `post-upgrade` Helm hook in `packages/extra/etcd` that was
`kubectl delete`ing the etcd TLS chain (`etcd-ca-tls`,
`etcd-peer-ca-tls`,
`etcd-client-tls`, `etcd-peer-tls`, `etcd-server-tls`) and then deleting
etcd pods on every chart upgrade.
The hook was gated by a semver compare of a stored
`etcd-deployed-version`
ConfigMap against `2.6.1`. That gate was written when chart versions
looked
like `2.6.0`, `2.6.1`, etc. Since commit `f871fbdb` ("Remove
versions_map
logic") all chart versions are stamped as `0.0.0+<git-hash>`, which per
semver is always `< 2.6.1`, so the gate always evaluates to "regenerate
certs" and the destructive hook fires on every upgrade.
On clusters running Kamaji-managed tenant control planes the consequence
is
severe: wiping the etcd CA triggers cert-manager to re-issue it with a
brand-new CA, but Kamaji's `datastore-certificate` Secrets still carry
the
old CA bundle mounted into tenant `kube-apiserver` pods. Those
apiservers
hit `x509: certificate signed by unknown authority` against
`etcd.<ns>.svc:2379` and go into CrashLoopBackOff until each tenant
`DataStore` is individually force-reconciled and the Deployments rolled.
Commit `47d81f70` ("Disabled private key rotation in CA certs") already
fixed the underlying `rotationPolicy: Always` issue the hook was written
to
paper over in March 2025, so the migration has no remaining use.
Changes:
- Delete `templates/hook/{job,role,rolebinding,serviceaccount}.yaml`
- Delete `templates/version.yaml` (only the hook read
`etcd-deployed-version`)
- Add `tests/no-post-upgrade-hook_test.yaml` as a regression guard — if
`templates/hook/job.yaml` or `templates/version.yaml` is ever brought
back,
`make unit-tests` fails
### Release note
```release-note
fix(etcd): remove destructive post-upgrade cert-regeneration hook. Previously the etcd chart ran a `post-upgrade` hook on every Helm upgrade that deleted etcd TLS Secrets and pods, causing cert-manager to re-issue the etcd CA. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 -> 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+<git-hash>` and the underlying `rotationPolicy: Always` issue was fixed in commit `47d81f70`. This is behavior removal, not cosmetic cleanup.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added Helm chart test suite verifying absence of post-upgrade hook Job
and version ConfigMap.
* **Chores**
* Removed post-upgrade hook Job and associated RBAC resources (Role,
RoleBinding, ServiceAccount).
* Removed version ConfigMap.
* Added `test` target to Makefile for running Helm chart unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from sircthulhu and CodeRabbit:
1. Return Forbidden instead of NotFound for unauthorized access
- Get() now returns 403 Forbidden to follow standard K8s RBAC behavior
- Previously returned 404 NotFound for security-by-obscurity
- Updated test expectations to match new behavior
2. Propagate field and label selectors in Watch handler
- Pass opts.FieldSelector and opts.LabelSelector to upstream Watch
- Add defensive filtering before authorization to prevent RBAC bypass
- Fixes potential issue with resourceNames restrictions
3. Refactor subject-matching logic to eliminate duplication
- Extract matchesSubject() helper for Group/User/ServiceAccount checks
- Remove duplicated code from filterAccessible and hasAccessToNamespace
- Consolidates ServiceAccount namespace fallback logic
All tests pass successfully.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
The `vm-instance` chart now drives the cozy-proxy `wholeIP` and
`allowICMP` annotations explicitly so that `externalMethod: PortList`
actually filters ingress traffic to declared ports while keeping
ping/PMTU functional.
- Render `networking.cozystack.io/wholeIP: "false"` when
`externalMethod: PortList` (was always `"true"`, which silently disabled
the PortList semantics in cozy-proxy).
- Add `externalAllowICMP` value (default `true`) propagated as
`networking.cozystack.io/allowICMP` when `externalMethod: PortList`.
Without this, cozy-proxy drops ICMP in port-filter mode (ping/PMTU
broken). Operators can set `externalAllowICMP: false` to opt out.
The changelog entry is intentionally **not** part of this PR — it will
be added in a dedicated `docs: add changelog for vX.Y.Z` commit at
release time, per project convention.
## Why
`externalMethod: PortList` is documented as filtering ingress traffic to
declared ports but has been non-functional on Cozystack v1.3.0 —
verified empirically on a 3-node Talos lab. Root cause was twofold:
chart always set `wholeIP: "true"`, and cozy-proxy v0.2.0 had no
port-aware logic. The cozy-proxy side was fixed in
cozystack/cozy-proxy#11 (merged) and cozystack/cozy-proxy#12 (allowICMP
follow-up); this PR completes the user-visible fix on the chart side.
## Companion PRs
- cozystack/cozy-proxy#11 (merged) — per-service ingress port filtering
- cozystack/cozy-proxy#12 (merged) — `allowICMP` annotation for
port-filter mode
## Test plan
- [x] Built cozy-proxy with the companion fix locally, deployed on a
3-node Talos lab (Cozystack v1.3.0)
- [x] `wholeIP: "false"` Service with `spec.ports: [22]`: only port 22
reachable from outside; ports 80/443/8080/9999 filtered
- [x] WholeIP-annotated Service unchanged: all listening ports reachable
- [x] Egress IP preservation works in both modes (TCP curl + UDP DNS)
- [x] `nft list table ip cozy_proxy` confirms expected ruleset
- [x] `helm template` renders the expected annotation matrix: `PortList`
default → `wholeIP=false, allowICMP=true`; `PortList` opt-out →
`allowICMP=false`; `WholeIP` → only `wholeIP=true`
- [x] `make unit-tests` passes locally
- [ ] CI unit tests
- [ ] CI E2E
## Backport
Suggesting `backport-v1.3` once merged.
## Release note
```release-note
[vm-instance] Make `externalMethod: PortList` actually filter ingress traffic to ports listed in `externalPorts`. New `externalAllowICMP` knob (default true) propagates the cozy-proxy `allowICMP` annotation to keep ping/PMTU functional in port-filter mode. Combined with cozy-proxy v0.3.0+, only listed ports plus ICMP are reachable from the VM's LoadBalancer IP.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `externalAllowICMP` configuration option to control ICMP traffic
acceptance for VM external access in PortList mode (enabled by default).
* **Documentation**
* Updated parameter documentation to include the new ICMP traffic
control setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pulls in the per-port filtering and allowICMP support that the companion
vm-instance chart fix in #2501 relies on. cozy-proxy v0.3.0 also tightens
the selector to the standard service.kubernetes.io/service-proxy-name=cozy-proxy
label and switches the default ingress mode to port-filter; both are
already covered by the vm-instance chart (label landed in #2357,
wholeIP/allowICMP wired explicitly in #2501), so VM workloads upgrade
transparently.
Out-of-tree consumers using cozy-proxy annotations directly (without the
label, or relying on the absent-annotation passthrough default) are
called out in the upstream v0.3.0 release notes:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Address review feedback from coderabbitai on packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml:119
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback from coderabbitai on packages/system/gpu-operator/examples/nvidia-driver-compat.yaml:77
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
- Adds a pre-commit hook and a dedicated GitHub Actions workflow that
run `make generate` at the repo root to catch drift in generated API
code (CRDs, DeepCopy, clients, RBAC) before it lands on main.
- Pre-commit hook is scoped to paths that actually affect codegen
(`api/`, `pkg/apis/`, `hack/update-codegen.sh`,
`hack/boilerplate.go.txt`) so unrelated commits are not slowed down.
- CI workflow (`.github/workflows/codegen-drift.yml`) installs Go from
`go.mod`, runs `make generate`, and fails on drift with an error
pointing contributors to the local fix.
- Includes one drift fix the check surfaced: `RestoreJobSpec.Options`
(added in #2437) had no `DeepCopyInto` handling — regenerated.
### Release note
```release-note
ci(api): add pre-commit hook and GitHub Actions workflow that verify generated API code (CRDs, deepcopy, clients, RBAC) is in sync with `make generate`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a CI workflow that checks generated code during PRs and fails
the build if generated artifacts diverge.
* Added a pre-commit hook that runs generation checks locally to prevent
committing outdated generated files.
* Fixed deep-copy behavior for backup restore job specs so nested
options are correctly duplicated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Add `externalAllowICMP` value (default true) propagated as
`networking.cozystack.io/allowICMP` annotation on the rendered Service
when `externalMethod: PortList`. The cozy-proxy companion (released as
part of cozystack/cozy-proxy#11 + #12) drops ICMP by default in
port-filter mode, which breaks ping and PMTU discovery; defaulting the
chart to "true" preserves user expectations while still allowing
operators to opt out by setting `externalAllowICMP: false`.
- Remove the v1.3.1.md changelog entry. Project convention is to add
changelogs in a dedicated "docs: add changelog for vX.Y.Z" commit at
release time, not as part of feature/fix PRs.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
## What this PR does
Bumps `linstor-csi` from v1.10.5 to v1.10.6 and ships an out-of-tree
patch that fixes live migration of KubeVirt VMs whose volumes sit on a
DRBD Protocol-A/B resource (e.g. a `replicated-async` StorageClass).
### Problem
DRBD requires Protocol C whenever `allow-two-primaries=yes` is enabled.
Operators commonly opt into Protocol A on a per-resource-group basis for
async / WAN replication, which silently breaks every subsequent live
migration of consumers of those volumes: `drbdadm adjust` rejects the
second-attach with `Protocol C required` (errno 139), KubeVirt's
evacuation loop retries indefinitely, and the VM stays pinned to the
source node.
### Change
- `LINSTOR_CSI_VERSION` 1.10.5 → 1.10.6 (Makefile + Dockerfile default).
- New patch `002-protocol-c-override-for-dual-attach.diff`: when
`Attach` installs `allow-two-primaries=yes` on the resource-definition
during a second attach, it also installs `DrbdOptions/Net/protocol=C` as
an override on the resource-definition. The override applies to every
connection (including diskless TieBreaker peers, where a per-pair
override would still leave one connection broken). It is tagged with
`Aux/csi-protocol-override=yes` so `Detach` removes only the override
this driver installed, leaving any operator-set `Protocol` property on
the resource-definition untouched.
- Existing patch `001-relocate-after-clone-restore.diff` regenerated
against v1.10.6 (context shift only, no logic change — the old patch
hunks no longer aligned cleanly).
### Verification
- `make image-linstor-csi` builds successfully on linux/amd64 with both
patches applied.
- End-to-end test on dev5 cluster (KubeVirt v1.6.3, 3-node Talos):
created a Protocol-A resource-group + StorageClass, provisioned a VM on
top, and triggered live migration. Migration succeeds in a single Attach
with the override installed during dual-attach and removed by Detach.
Reproducer (without the patch) is the well-known evacuation loop with
`(node) Failed to adjust DRBD resource ... Protocol C required`.
### Upstream
Upstreamed as draft PR piraeusdatastore/linstor-csi#435.
### Compatibility
- No behaviour change for resources already using Protocol C (the common
case).
- No behaviour change for resources never attached with
allow-two-primaries.
- Idempotent: re-running `Attach` is a no-op once the override is
installed.
- Operator-set Protocol overrides on the resource-definition are
preserved (gated by the Aux marker).
### Release note
```release-note
fix(linstor): live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with "Protocol C required" — linstor-csi now installs a Protocol=C override on the resource-definition during dual-attach and reverts it on detach.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic replica relocation after volume clone and snapshot restore
to improve placement and load distribution.
* Conditional DRBD protocol override to enable/clean up dual-attach
(two-primaries) scenarios more reliably.
* **Chores**
* Updated LINSTOR CSI default to v1.10.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Render `networking.cozystack.io/wholeIP: "false"` on the Service when
`externalMethod: PortList` is configured (was always `"true"` before).
Combined with cozy-proxy v0.3.0+ which adds per-port filtering for
"false"-annotated services, this makes `externalMethod: PortList`
behave as documented: only ports listed in `externalPorts` are
reachable from the LoadBalancer IP.
Backward-compatible: existing services with `externalMethod: WholeIP`
continue to set `wholeIP: "true"` and behave identically. cozy-proxy
versions older than v0.3.0 ignore Services with `wholeIP: "false"`,
which means PortList Services on older cozy-proxy will lose their
egress IP preservation — but that path was already non-functional
ingress-wise, so this is not a regression for users actually relying
on PortList.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Bump linstor-csi to v1.10.6 and add an out-of-tree patch that overrides
DrbdOptions/Net/protocol to C on the resource-definition whenever a
volume is attached to a second node with allow-two-primaries (the
live-migration flow), and reverts it on detach.
Without the patch, any KubeVirt live migration of a VM whose volume
sits on a Protocol-A/B resource (for example a "replicated-async"
StorageClass) ends up in a permanent evacuation loop: drbdadm adjust
on the satellite rejects allow-two-primaries with "Protocol C
required" (errno 139), the migration fails, and KubeVirt retries
until manual intervention.
The override is set at the resource-definition level so it covers
every connection, including diskless TieBreaker peers, and is tagged
with an Aux marker so Detach reverts only the override we installed.
The 001 relocate-after-clone patch was regenerated against v1.10.6
(context shift only, no logic change).
Verified end-to-end on dev5: live migration of a KubeVirt VM whose
PVC sits on a Protocol-A resource-group now succeeds in a single
Attach.
Patch is upstreamed as draft PR piraeusdatastore/linstor-csi#435.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Closes#2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.
Implements a defense-in-depth fix:
- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.
Tests:
- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.
All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).
Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.
### Release note
```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
* Chart resources now render conditionally based on etcd presence.
* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.
* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The per-package Makefile added in #2382 hardcoded buildx flags
(--provenance, --builder, --platform=linux/amd64,linux/arm64,
--push, --load, --label) instead of using the shared $(BUILDX_ARGS)
macro from hack/common-envs.mk.
This broke CI: the runner's default docker driver does not support
multi-platform builds, and the hardcoded multi-arch platform list
crashed `make build` with "Multi-platform build is not supported for
the docker driver."
Replace the hardcoded flags with $(BUILDX_ARGS) to match every other
package (e.g. linstor, dashboard, cilium). $(BUILDX_ARGS) injects
--push, --load, --label, --provenance=false, and only sets --builder
or --platform when the operator explicitly exports BUILDER/PLATFORM.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The linstor-gui package was added in #2382 with its own per-package
Makefile and Dockerfile, but the root Makefile's `build:` target was
not updated to invoke it. As a result `ghcr.io/cozystack/cozystack/
linstor-gui` has never been published (registry returns NAME_UNKNOWN)
and the chart's `image.tag` was never digest-pinned. Any cluster
deploying the chart hits ImagePullBackOff.
Wire the package into the root build alongside the other system
images. The next CI build will publish the image and the per-package
Makefile will rewrite values.yaml `image.repository`/`image.tag` to a
digest-pinned reference automatically.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
