## Summary
Add `serverName` field to bootstrap configuration to support backup
recovery when the Barman server name in `backup.info` differs from the
Kubernetes cluster name.
This fixes "no target backup found" errors during CloudNativePG recovery
operations.
## Problem
CloudNativePG forms the backup path as `destinationPath + "postgres-" +
oldName`, but searches for backups using the `server_name` field from
`backup.info`. When these values don't match (e.g., cluster name is
`grafana` but `server_name` is `cloud`), recovery fails with "no target
backup found".
## Solution
- Add optional `serverName` parameter to `bootstrap` configuration
- When specified, CloudNativePG uses this value to search for backups in
S3
- Falls back to `oldName` when `serverName` is not provided (backwards
compatible)
## Changes
- Add `ServerName` field to PostgreSQL CRD type definition
- Add conditional `serverName` to Cluster `externalClusters` template
- Update `values.yaml` and README.md with `serverName` documentation
- Regenerate `values.schema.json` and `postgres.yaml` CRD
## Test plan
- [ ] Deploy postgres cluster with backups enabled
- [ ] Create backup
- [ ] Create new postgres instance with `bootstrap.enabled=true`,
`bootstrap.oldName=<original-name>`, and
`bootstrap.serverName=<value-from-backup.info>`
- [ ] Verify recovery completes successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an optional bootstrap.serverName setting to specify the Barman
server name from the original cluster’s backup.info (used when that
server name differs from the Kubernetes cluster name).
* **Documentation**
* Clarified that bootstrap.oldName must match the serverName value
recorded in backup.info; updated docs and examples to reflect the new
bootstrap.serverName option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an opt-in `publishing.exposure=loadBalancer` mode for the
ingress-nginx Service as a migration path away from
`Service.spec.externalIPs`, which is deprecated upstream in Kubernetes
v1.36
([KEP-5707](https://github.com/kubernetes/enhancements/issues/5707),
[kubernetes#137293](https://github.com/kubernetes/kubernetes/pull/137293)).
The `AllowServiceExternalIPs` feature gate is expected to default to off
around v1.40 and the implementation to be removed around v1.43.
Stacked on top of #2464 (cilium v1.19.3 bump) — depends on it for the
`CiliumLoadBalancerIPPool` at `cilium.io/v2`.
### Behavior
- New platform value `publishing.exposure` — enum `externalIPs |
loadBalancer`, default `externalIPs` (current behavior unchanged on
upgrade).
- Plumbed through `cozystack-values` into each tenant's ingress
HelmRelease via the new `expose-mode` key.
- When `exposure=loadBalancer` and the current namespace matches
`publishing.ingressName`, the Service becomes `type: LoadBalancer` with
`externalTrafficPolicy: Local` and a `CiliumLoadBalancerIPPool`
announces the addresses from `publishing.externalIPs` via Cilium LB
IPAM.
- The pool uses a namespace-only `serviceSelector`
(`io.kubernetes.service.namespace: <ns>`) — any LoadBalancer Service in
the tenant namespace draws from it. See "Pool ownership" below.
- IPv4 addresses get `/32` CIDRs, IPv6 addresses get `/128`. Mixed
families supported. Pre-CIDR entries (`192.0.2.10/32`) are accepted
without double-suffixing.
- Unknown values, `loadBalancer` with empty externalIPs, and stray empty
entries from `publishing.externalIPs` are rejected at render time with
explicit error messages.
### Pool ownership
The `CiliumLoadBalancerIPPool` is rendered from
`packages/apps/tenant/templates/cilium-lb-pool.yaml`, not from the
ingress chart. The tenant chart is the per-tenant owner of cross-cutting
resources (Namespace, `cozystack-values` Secret, HelmReleases for
ingress and gateway), so one pool per tenant lives there.
Cilium LB IPAM rejects overlapping CIDRs across pools regardless of
`serviceSelector` — the last-added pool gets `cilium.io/PoolConflict`
and stops allocating. Keeping the pool in the ingress chart would
collide with the Gateway-API PR (#2470), which materialises its own
LoadBalancer Service from the same `publishing.externalIPs` range when
`tenant.spec.gateway=true`. Moving the pool to the tenant chart with a
namespace-only selector lets a single pool back both services
(ingress-nginx today, a Cilium Gateway Service once #2470 lands).
Only the ingress-loadBalancer signal is wired here
(`_cluster.expose-mode=loadBalancer` + `.Values.ingress=true` +
publishing tenant). #2470 rebases on top of this PR, drops its own
`packages/extra/gateway/templates/cilium-lb-pool.yaml`, and adds an OR
branch for `.Values.gateway` in the tenant template.
### Scope
Only the ingress-nginx Service is migrated by this setting. Other
cozystack components that still write `Service.spec.externalIPs`
directly (notably the `vpn` app at
`packages/apps/vpn/templates/service.yaml`) need separate follow-up
before Kubernetes v1.40.
### Tests
- `packages/extra/ingress/tests/exposure_test.yaml` — 10 helm-unittest
cases on the ingress-nginx Service: type / externalTrafficPolicy /
externalIPs assertion for both modes, unknown-mode rejection
(case-sensitive enum), empty-IPs failure, empty-entry filtering,
non-publishing-tenant fallback.
- `packages/apps/tenant/tests/exposure_test.yaml` — 9 cases on the pool
itself: IPv4, IPv6, mixed, pre-CIDR input, empty-entry filtering,
`ingress=false` in publishing tenant (no pool), non-publishing tenant
(no pool), empty externalIPs (no pool).
- Both suites are auto-discovered by `hack/helm-unit-tests.sh` via the
`test:` target in each package's Makefile.
### Caveats (copied from the inline `values.yaml` comment)
- `loadBalancer` mode uses `externalTrafficPolicy: Local`. The external
IP must already be routed to a node that hosts an ingress pod (floating
IP / upstream router / podAntiAffinity).
- Cilium does not announce the IP on its own unless L2 announcements or
BGP are enabled in the Cilium values (disabled by default in cozystack).
- Switching the value on a running cluster causes the ingress-nginx
Service to be recreated (`upgrade.force: true` on the HelmRelease +
Service kind change); expect a brief interruption of ingress traffic.
### Release note
```release-note
feat(ingress): add opt-in publishing.exposure=loadBalancer mode that uses type: LoadBalancer + CiliumLoadBalancerIPPool instead of the deprecated Service.spec.externalIPs. Default (externalIPs) preserves existing behavior.
```
## What this PR does
Closes a bootstrap race where the `cnpg-webhook-service` Service gets
its EndpointSlice populated and the data plane (kube-proxy / Cilium)
programmed a second or two after `helm install --wait` on
`cozy-postgres-operator` declares the HelmRelease Ready. Any HelmRelease
that `dependsOn: postgres-operator` and creates a
`postgresql.cnpg.io/v1` resource in that window (cozy-keycloak, tenant
Postgres apps) has kube-apiserver's call to `mcluster.cnpg.io` fail with
```
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
```
which fails the downstream release's install.
The fix is a post-install / post-upgrade Helm hook (ServiceAccount +
ClusterRole + ClusterRoleBinding + Job) that probes `/readyz` on the
webhook through the apiserver service proxy. Apiserver's service proxy
uses the same endpoint-resolution and apiserver-initiated pod dial as
the admission webhook path, so once `/readyz` answers through the proxy,
the subsequent admission call will also succeed. Helm `--wait` blocks
the install from completing until the Job exits 0, so the HelmRelease
Ready condition does not lie anymore.
Hardened per review:
- RBAC scoped to a single `services/proxy` resourceName
(`https:cnpg-webhook-service:webhook-server`) — the exact string the
apiserver URL path parser expects for the Service proxy subresource.
- A drift-guard helm-unittest test renders the vendored cnpg Service
template and fails if its `metadata.name` / `ports[0].name` diverge from
the literals in the hook, so a future `make update` that renames the
service forces this template to be updated in the same change.
- Image digest-pinned (`clastix/kubectl:v1.32@sha256:…`) with a
`renovate:` annotation; `imagePullPolicy: IfNotPresent` when
digest-pinned, `Always` when tag-only.
- Job runs under PSA-restricted-compatible securityContext (non-root,
seccomp RuntimeDefault, readOnlyRootFilesystem, drop ALL caps).
- `activeDeadlineSeconds` is derived from `maxAttempts × sleepSeconds +
60s` so a values override raising retries does not get silently cut by a
fixed deadline.
- `backoffLimit: 2` (configurable) so a transient pod-level failure
(image pull rate limit, OOM, CNI hiccup) does not fail the whole
HelmRelease.
- On timeout the Job prints the last `kubectl get --raw` stderr so the
operator can distinguish DNS / refused / 401 / TLS from the Job logs.
21 helm-unittest assertions cover ordering, RBAC/URL parity, subchart
drift, image policy, retry-loop bounds, securityContext, and the
deadline-scaling invariant. Wired via `make test` in
`packages/system/postgres-operator/Makefile`.
First surfaced on #2470 E2E run 24862782568 where
`cozy-keycloak/keycloak` failed with the exact signature above. Not tied
to that PR's branch — this is independently applicable to `main`.
### Release note
```release-note
fix(postgres-operator): add a post-install readiness gate that blocks the HelmRelease from reporting Ready until the cnpg admission webhook actually serves through the cluster Service, preventing "dial tcp <svc-ip>:443: connect: connection refused" on the first HelmRelease that depends on postgres-operator (keycloak, tenant Postgres apps).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added webhook readiness validation during install/upgrade to block
completion until the admission webhook is reachable.
* New configuration options for the readiness probe image and
retry/timeouts.
* **Tests**
* Introduced a comprehensive test suite validating rendered manifests,
probe behavior, RBAC, image rendering, and retry/timeout logic.
* **Chores**
* Added a test entry point to the project Makefile to run chart/unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `linstor-gui` package (added in #2382) was never wired into the root
`Makefile`'s `build:` target, so the image is not built or published by
CI. `ghcr.io/cozystack/cozystack/linstor-gui` returns `NAME_UNKNOWN`
from the GHCR API, and `values.yaml` still pins `tag: 2.3.0` without a
digest because the per-package Makefile that rewrites the tag after a
successful push has never run in CI.
This PR adds the missing line so the next build publishes the image and
digest-pins `values.yaml` automatically.
### Release note
```release-note
build(linstor): include linstor-gui in root image build target so the image is built and published by CI (the chart previously referenced an image that did not exist in the registry)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Integrated the linstor-gui container image build into the main image
build workflow so the GUI image is produced as part of standard builds.
* Streamlined the GUI image build configuration to use consolidated
build arguments, improving consistency and maintainability of automated
image builds and metadata updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Bumps the vendored cozy-proxy chart and image tag from v0.2.0 to v0.3.0
by running `make update` in `packages/system/cozy-proxy`.
cozy-proxy v0.3.0 release:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
## Why
cozy-proxy v0.3.0 is the companion of #2501. The chart fix in #2501
finally honors the documented PortList semantics (only `externalPorts`
reachable, ICMP preserved by default), but only takes effect once
cozy-proxy v0.3.0 is in place — v0.2.0 has no port-aware logic, so the
chart fix silently no-ops on it. This PR ships that runtime side.
## Upgrade impact
For cozystack users this is a **bug fix, not a breaking change**.
cozy-proxy v0.3.0 introduces two contract changes upstream — (1)
label-only selector via `service.kubernetes.io/service-proxy-name:
cozy-proxy`, and (2) port-filter as the default ingress mode — both of
which are absorbed by the vm-instance chart (label landed in #2357,
`wholeIP` + `allowICMP` wired in #2501).
- **`externalMethod: WholeIP`** (default) — chart renders label +
`wholeIP: "true"` in both old and new versions. cozy-proxy keeps
passthrough. **No change.**
- **`externalMethod: PortList`** — previously the chart always rendered
`wholeIP: "true"` regardless of method, and cozy-proxy v0.2.0 had no
port-aware logic, so PortList was silently a no-op (all ports
reachable). After this bump + #2501, PortList behaves as documented:
only ports listed in `externalPorts` reach the VM. Users who configured
PortList but reached undeclared ports (relying on the silent no-op) will
see those ports become unreachable — that's the documented intent of
PortList finally taking effect. ICMP keeps working by default thanks to
`externalAllowICMP: true`.
- **DaemonSet rollover** — re-init of the nft table during cozy-proxy
pod restart is a brief blip on existing flows; same as any DaemonSet
upgrade.
Out-of-tree consumers using cozy-proxy directly (without the
`service-proxy-name` label, or relying on the absent-annotation
passthrough default) need to migrate per the upstream v0.3.0 release
notes — not a cozystack concern.
## Order of operations
This PR is functionally complete on its own (image bump), but the
user-facing PortList fix it enables only happens when #2501 also lands.
Recommended merge order:
1. #2501 first (chart adapts to the new contract)
2. This PR second (cozy-proxy DaemonSet picks up the new image)
The reverse order is also safe — cozy-proxy v0.3.0 with the unmodified
vm-instance chart still produces correct results for `externalMethod:
WholeIP` (most common case) and keeps PortList silently broken until
#2501 lands.
## Test plan
- [ ] `helm template packages/system/cozy-proxy` — image tag `v0.3.0`
rendered
- [ ] `make unit-tests` passes
- [ ] After #2501 lands, end-to-end: `externalMethod: PortList` with
`externalPorts: [22]` filters all other ports, ping works
- [ ] WholeIP-method VM Service unchanged after upgrade
## Companion PRs
- #2501 — vm-instance chart wires `wholeIP` / `allowICMP` per
`externalMethod`
- cozystack/cozy-proxy#13 (merged) — label-only selector + port-filter
default
- cozy-proxy v0.3.0 release notes —
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Released version 0.3.0 with updated Helm chart metadata and container
image configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `post-upgrade` Helm hook in `packages/extra/etcd` that was
`kubectl delete`ing the etcd TLS chain (`etcd-ca-tls`,
`etcd-peer-ca-tls`,
`etcd-client-tls`, `etcd-peer-tls`, `etcd-server-tls`) and then deleting
etcd pods on every chart upgrade.
The hook was gated by a semver compare of a stored
`etcd-deployed-version`
ConfigMap against `2.6.1`. That gate was written when chart versions
looked
like `2.6.0`, `2.6.1`, etc. Since commit `f871fbdb` ("Remove
versions_map
logic") all chart versions are stamped as `0.0.0+<git-hash>`, which per
semver is always `< 2.6.1`, so the gate always evaluates to "regenerate
certs" and the destructive hook fires on every upgrade.
On clusters running Kamaji-managed tenant control planes the consequence
is
severe: wiping the etcd CA triggers cert-manager to re-issue it with a
brand-new CA, but Kamaji's `datastore-certificate` Secrets still carry
the
old CA bundle mounted into tenant `kube-apiserver` pods. Those
apiservers
hit `x509: certificate signed by unknown authority` against
`etcd.<ns>.svc:2379` and go into CrashLoopBackOff until each tenant
`DataStore` is individually force-reconciled and the Deployments rolled.
Commit `47d81f70` ("Disabled private key rotation in CA certs") already
fixed the underlying `rotationPolicy: Always` issue the hook was written
to
paper over in March 2025, so the migration has no remaining use.
Changes:
- Delete `templates/hook/{job,role,rolebinding,serviceaccount}.yaml`
- Delete `templates/version.yaml` (only the hook read
`etcd-deployed-version`)
- Add `tests/no-post-upgrade-hook_test.yaml` as a regression guard — if
`templates/hook/job.yaml` or `templates/version.yaml` is ever brought
back,
`make unit-tests` fails
### Release note
```release-note
fix(etcd): remove destructive post-upgrade cert-regeneration hook. Previously the etcd chart ran a `post-upgrade` hook on every Helm upgrade that deleted etcd TLS Secrets and pods, causing cert-manager to re-issue the etcd CA. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 -> 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+<git-hash>` and the underlying `rotationPolicy: Always` issue was fixed in commit `47d81f70`. This is behavior removal, not cosmetic cleanup.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added Helm chart test suite verifying absence of post-upgrade hook Job
and version ConfigMap.
* **Chores**
* Removed post-upgrade hook Job and associated RBAC resources (Role,
RoleBinding, ServiceAccount).
* Removed version ConfigMap.
* Added `test` target to Makefile for running Helm chart unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `vm-instance` chart now drives the cozy-proxy `wholeIP` and
`allowICMP` annotations explicitly so that `externalMethod: PortList`
actually filters ingress traffic to declared ports while keeping
ping/PMTU functional.
- Render `networking.cozystack.io/wholeIP: "false"` when
`externalMethod: PortList` (was always `"true"`, which silently disabled
the PortList semantics in cozy-proxy).
- Add `externalAllowICMP` value (default `true`) propagated as
`networking.cozystack.io/allowICMP` when `externalMethod: PortList`.
Without this, cozy-proxy drops ICMP in port-filter mode (ping/PMTU
broken). Operators can set `externalAllowICMP: false` to opt out.
The changelog entry is intentionally **not** part of this PR — it will
be added in a dedicated `docs: add changelog for vX.Y.Z` commit at
release time, per project convention.
## Why
`externalMethod: PortList` is documented as filtering ingress traffic to
declared ports but has been non-functional on Cozystack v1.3.0 —
verified empirically on a 3-node Talos lab. Root cause was twofold:
chart always set `wholeIP: "true"`, and cozy-proxy v0.2.0 had no
port-aware logic. The cozy-proxy side was fixed in
cozystack/cozy-proxy#11 (merged) and cozystack/cozy-proxy#12 (allowICMP
follow-up); this PR completes the user-visible fix on the chart side.
## Companion PRs
- cozystack/cozy-proxy#11 (merged) — per-service ingress port filtering
- cozystack/cozy-proxy#12 (merged) — `allowICMP` annotation for
port-filter mode
## Test plan
- [x] Built cozy-proxy with the companion fix locally, deployed on a
3-node Talos lab (Cozystack v1.3.0)
- [x] `wholeIP: "false"` Service with `spec.ports: [22]`: only port 22
reachable from outside; ports 80/443/8080/9999 filtered
- [x] WholeIP-annotated Service unchanged: all listening ports reachable
- [x] Egress IP preservation works in both modes (TCP curl + UDP DNS)
- [x] `nft list table ip cozy_proxy` confirms expected ruleset
- [x] `helm template` renders the expected annotation matrix: `PortList`
default → `wholeIP=false, allowICMP=true`; `PortList` opt-out →
`allowICMP=false`; `WholeIP` → only `wholeIP=true`
- [x] `make unit-tests` passes locally
- [ ] CI unit tests
- [ ] CI E2E
## Backport
Suggesting `backport-v1.3` once merged.
## Release note
```release-note
[vm-instance] Make `externalMethod: PortList` actually filter ingress traffic to ports listed in `externalPorts`. New `externalAllowICMP` knob (default true) propagates the cozy-proxy `allowICMP` annotation to keep ping/PMTU functional in port-filter mode. Combined with cozy-proxy v0.3.0+, only listed ports plus ICMP are reachable from the VM's LoadBalancer IP.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `externalAllowICMP` configuration option to control ICMP traffic
acceptance for VM external access in PortList mode (enabled by default).
* **Documentation**
* Updated parameter documentation to include the new ICMP traffic
control setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pulls in the per-port filtering and allowICMP support that the companion
vm-instance chart fix in #2501 relies on. cozy-proxy v0.3.0 also tightens
the selector to the standard service.kubernetes.io/service-proxy-name=cozy-proxy
label and switches the default ingress mode to port-filter; both are
already covered by the vm-instance chart (label landed in #2357,
wholeIP/allowICMP wired explicitly in #2501), so VM workloads upgrade
transparently.
Out-of-tree consumers using cozy-proxy annotations directly (without the
label, or relying on the absent-annotation passthrough default) are
called out in the upstream v0.3.0 release notes:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
- Adds a pre-commit hook and a dedicated GitHub Actions workflow that
run `make generate` at the repo root to catch drift in generated API
code (CRDs, DeepCopy, clients, RBAC) before it lands on main.
- Pre-commit hook is scoped to paths that actually affect codegen
(`api/`, `pkg/apis/`, `hack/update-codegen.sh`,
`hack/boilerplate.go.txt`) so unrelated commits are not slowed down.
- CI workflow (`.github/workflows/codegen-drift.yml`) installs Go from
`go.mod`, runs `make generate`, and fails on drift with an error
pointing contributors to the local fix.
- Includes one drift fix the check surfaced: `RestoreJobSpec.Options`
(added in #2437) had no `DeepCopyInto` handling — regenerated.
### Release note
```release-note
ci(api): add pre-commit hook and GitHub Actions workflow that verify generated API code (CRDs, deepcopy, clients, RBAC) is in sync with `make generate`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a CI workflow that checks generated code during PRs and fails
the build if generated artifacts diverge.
* Added a pre-commit hook that runs generation checks locally to prevent
committing outdated generated files.
* Fixed deep-copy behavior for backup restore job specs so nested
options are correctly duplicated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Add `externalAllowICMP` value (default true) propagated as
`networking.cozystack.io/allowICMP` annotation on the rendered Service
when `externalMethod: PortList`. The cozy-proxy companion (released as
part of cozystack/cozy-proxy#11 + #12) drops ICMP by default in
port-filter mode, which breaks ping and PMTU discovery; defaulting the
chart to "true" preserves user expectations while still allowing
operators to opt out by setting `externalAllowICMP: false`.
- Remove the v1.3.1.md changelog entry. Project convention is to add
changelogs in a dedicated "docs: add changelog for vX.Y.Z" commit at
release time, not as part of feature/fix PRs.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
## What this PR does
Bumps `linstor-csi` from v1.10.5 to v1.10.6 and ships an out-of-tree
patch that fixes live migration of KubeVirt VMs whose volumes sit on a
DRBD Protocol-A/B resource (e.g. a `replicated-async` StorageClass).
### Problem
DRBD requires Protocol C whenever `allow-two-primaries=yes` is enabled.
Operators commonly opt into Protocol A on a per-resource-group basis for
async / WAN replication, which silently breaks every subsequent live
migration of consumers of those volumes: `drbdadm adjust` rejects the
second-attach with `Protocol C required` (errno 139), KubeVirt's
evacuation loop retries indefinitely, and the VM stays pinned to the
source node.
### Change
- `LINSTOR_CSI_VERSION` 1.10.5 → 1.10.6 (Makefile + Dockerfile default).
- New patch `002-protocol-c-override-for-dual-attach.diff`: when
`Attach` installs `allow-two-primaries=yes` on the resource-definition
during a second attach, it also installs `DrbdOptions/Net/protocol=C` as
an override on the resource-definition. The override applies to every
connection (including diskless TieBreaker peers, where a per-pair
override would still leave one connection broken). It is tagged with
`Aux/csi-protocol-override=yes` so `Detach` removes only the override
this driver installed, leaving any operator-set `Protocol` property on
the resource-definition untouched.
- Existing patch `001-relocate-after-clone-restore.diff` regenerated
against v1.10.6 (context shift only, no logic change — the old patch
hunks no longer aligned cleanly).
### Verification
- `make image-linstor-csi` builds successfully on linux/amd64 with both
patches applied.
- End-to-end test on dev5 cluster (KubeVirt v1.6.3, 3-node Talos):
created a Protocol-A resource-group + StorageClass, provisioned a VM on
top, and triggered live migration. Migration succeeds in a single Attach
with the override installed during dual-attach and removed by Detach.
Reproducer (without the patch) is the well-known evacuation loop with
`(node) Failed to adjust DRBD resource ... Protocol C required`.
### Upstream
Upstreamed as draft PR piraeusdatastore/linstor-csi#435.
### Compatibility
- No behaviour change for resources already using Protocol C (the common
case).
- No behaviour change for resources never attached with
allow-two-primaries.
- Idempotent: re-running `Attach` is a no-op once the override is
installed.
- Operator-set Protocol overrides on the resource-definition are
preserved (gated by the Aux marker).
### Release note
```release-note
fix(linstor): live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with "Protocol C required" — linstor-csi now installs a Protocol=C override on the resource-definition during dual-attach and reverts it on detach.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic replica relocation after volume clone and snapshot restore
to improve placement and load distribution.
* Conditional DRBD protocol override to enable/clean up dual-attach
(two-primaries) scenarios more reliably.
* **Chores**
* Updated LINSTOR CSI default to v1.10.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Render `networking.cozystack.io/wholeIP: "false"` on the Service when
`externalMethod: PortList` is configured (was always `"true"` before).
Combined with cozy-proxy v0.3.0+ which adds per-port filtering for
"false"-annotated services, this makes `externalMethod: PortList`
behave as documented: only ports listed in `externalPorts` are
reachable from the LoadBalancer IP.
Backward-compatible: existing services with `externalMethod: WholeIP`
continue to set `wholeIP: "true"` and behave identically. cozy-proxy
versions older than v0.3.0 ignore Services with `wholeIP: "false"`,
which means PortList Services on older cozy-proxy will lose their
egress IP preservation — but that path was already non-functional
ingress-wise, so this is not a regression for users actually relying
on PortList.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Bump linstor-csi to v1.10.6 and add an out-of-tree patch that overrides
DrbdOptions/Net/protocol to C on the resource-definition whenever a
volume is attached to a second node with allow-two-primaries (the
live-migration flow), and reverts it on detach.
Without the patch, any KubeVirt live migration of a VM whose volume
sits on a Protocol-A/B resource (for example a "replicated-async"
StorageClass) ends up in a permanent evacuation loop: drbdadm adjust
on the satellite rejects allow-two-primaries with "Protocol C
required" (errno 139), the migration fails, and KubeVirt retries
until manual intervention.
The override is set at the resource-definition level so it covers
every connection, including diskless TieBreaker peers, and is tagged
with an Aux marker so Detach reverts only the override we installed.
The 001 relocate-after-clone patch was regenerated against v1.10.6
(context shift only, no logic change).
Verified end-to-end on dev5: live migration of a KubeVirt VM whose
PVC sits on a Protocol-A resource-group now succeeds in a single
Attach.
Patch is upstreamed as draft PR piraeusdatastore/linstor-csi#435.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Closes#2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.
Implements a defense-in-depth fix:
- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.
Tests:
- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.
All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).
Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.
### Release note
```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
* Chart resources now render conditionally based on etcd presence.
* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.
* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The per-package Makefile added in #2382 hardcoded buildx flags
(--provenance, --builder, --platform=linux/amd64,linux/arm64,
--push, --load, --label) instead of using the shared $(BUILDX_ARGS)
macro from hack/common-envs.mk.
This broke CI: the runner's default docker driver does not support
multi-platform builds, and the hardcoded multi-arch platform list
crashed `make build` with "Multi-platform build is not supported for
the docker driver."
Replace the hardcoded flags with $(BUILDX_ARGS) to match every other
package (e.g. linstor, dashboard, cilium). $(BUILDX_ARGS) injects
--push, --load, --label, --provenance=false, and only sets --builder
or --platform when the operator explicitly exports BUILDER/PLATFORM.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The linstor-gui package was added in #2382 with its own per-package
Makefile and Dockerfile, but the root Makefile's `build:` target was
not updated to invoke it. As a result `ghcr.io/cozystack/cozystack/
linstor-gui` has never been published (registry returns NAME_UNKNOWN)
and the chart's `image.tag` was never digest-pinned. Any cluster
deploying the chart hits ImagePullBackOff.
Wire the package into the root build alongside the other system
images. The next CI build will publish the image and the per-package
Makefile will rewrite values.yaml `image.repository`/`image.tag` to a
digest-pinned reference automatically.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozystack.io/status-beacon: "true" annotation had no consumer in
the chart, no documented contract, and no convention defined for other
charts to follow. It would have become accidental precedent for
contributors copying the pattern without understanding it.
The ConfigMap itself is self-explanatory: the name <release>-awaiting-etcd,
data.status: "awaiting-etcd", and the human-readable message in
data.message all surface the same operator signal via kubectl get cm.
Drop the annotation; keep the ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Operators in air-gapped or rate-limited environments cannot reach
docker.io and the bundled busybox digest pin gives them no escape
hatch. Add an optional images.waitForKubeconfig chart value that, when
set, replaces the helper's image reference with any registry path
kubelet can pull. Empty value falls back to images/busybox.tag, so the
prior digest-pinned default is preserved.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adopt CNCF/Kubernetes label conventions for issues and PRs and add
automated labeling.
**Canonical label file**: `.github/labels.yml`. Synced into the
repository by `.github/workflows/labels.yaml` (EndBug/label-sync@v2) on
push to `main`, weekly cron, and manual dispatch. UI-only label edits
are overwritten — propose changes via PR to this file.
### Label namespaces
Following the [Kubernetes label
scheme](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md):
- `kind/*` — issue or PR type (bug, feature, documentation, support,
cleanup, regression, flake, failing-test, api-change, breaking-change)
- `priority/*` — urgency (critical-urgent, important-soon,
important-longterm, backlog)
- `triage/*` — review state (needs-triage, accepted, needs-information,
not-reproducible, duplicate, unresolved)
- `lifecycle/*` — issue/PR lifecycle (active, frozen, stale, rotten)
- `area/*` — subsystem; 15 seeded plus `area/uncategorized` fallback.
Extensible — propose a new area when no existing one fits.
- `do-not-merge/*` — PR merge blockers (work-in-progress, hold)
Cozystack-specific labels preserved: `epic`, `community`, `help wanted`,
`good first issue`, `quality-of-life`, `upstream-issue`, `backport`,
`backport-previous`, `release`, `automated`, `debug`, `sponsored`,
`lgtm`, `ok-to-test`, `security/*`, `size:*`.
### `area/*` set
15 areas seeded by activity in open issues and PRs: `area/ai`,
`area/api`, `area/build`, `area/ci`, `area/dashboard`, `area/database`,
`area/extra`, `area/kubernetes`, `area/monitoring`, `area/networking`,
`area/platform`, `area/release`, `area/storage`, `area/testing`,
`area/virtualization`. Plus `area/uncategorized` as the auto-labeler
fallback.
### Migration safety
Existing labels are renamed via `aliases:` in `labels.yml`. GitHub
preserves the label ID, so all currently tagged issues and PRs keep
their tags under the new name without losing references:
| Old | New |
|---|---|
| `bug` | `kind/bug` |
| `enhancement` | `kind/feature` |
| `documentation` | `kind/documentation` |
| `question` | `kind/support` |
| `frozen` | `lifecycle/frozen` |
| `stale` | `lifecycle/stale` |
| `duplicate` | `triage/duplicate` |
| `do-not-merge` | `do-not-merge/work-in-progress` |
| `do not merge` | `do-not-merge/work-in-progress` |
`delete-other-labels: false` on the initial rollout. Generic
GitHub-default labels (`wontfix`, `invalid`) are preserved untouched and
will be removed in a follow-up cleanup PR. EndBug processes aliases
sequentially, so the second of the two `do-not-merge*` aliases hits a
name collision and logs a warning — the legacy label survives that one
sync and is cleaned up in the same follow-up.
### PR auto-labeling
`.github/workflows/pr-labeler.yaml` parses each PR title on `opened`,
`edited`, `reopened`, and `synchronize` and applies labels additively
(never removes):
- **type → `kind/*`**: feat, fix, docs, chore, refactor (others get no
kind)
- **scope → `area/*`**: scope mapping covers all current cozystack
components (full table in `docs/agents/contributing.md`)
- **`!` after type or `BREAKING CHANGE:` footer**: applies
`kind/breaking-change`
- **`[Backport release-1.x]` prefix**: stripped before parsing;
`area/release` and `backport` labels added
- **Composite scope** (`feat(platform, system, apps): …`): each part
mapped independently
- **Bracket fallback** (`[scope] description`): maps `area/*` but cannot
infer `kind/*`
- **Unmapped scope or non-conventional title**: applies
`area/uncategorized` for human review
### Schema validation
`.github/workflows/labels.yaml` runs a `validate` job on every PR
touching `labels.yml` or its workflow. Asserts:
- description ≤ 100 chars (GitHub REST API limit)
- color is 6-char hex without leading `#`
- unique top-level names
- aliases do not collide with top-level names
Sync runs only on push to main, weekly cron, and manual dispatch; PR
runs validate-only.
### Hardcoded label/title references updated
- `.github/ISSUE_TEMPLATE/bug_report.md`: `labels: 'bug'` → `labels:
'kind/bug'`
- `.github/workflows/tags.yaml`:
- changelog PR labels `['documentation', 'automated']` →
`['kind/documentation', 'automated']`
- release PR title `Release v${version}` → `chore(release): cut
v${version}` (so the auto-labeler applies `kind/cleanup` +
`area/release`)
- changelog PR title `docs: add changelog for v${version}` →
`docs(release): add changelog for v${version}` (so the auto-labeler
applies `kind/documentation` + `area/release`)
### Documentation
- `AGENTS.md`: Activation entry pointing agents to `labels.yml` and the
PR title auto-labeling rules. States explicitly that `area/*` accuracy
outweighs reuse — propose a new area when none fits, do not shoehorn
into a wrong one.
- `docs/agents/contributing.md`: PR Title Auto-Labeling section with
type→kind and scope→area tables.
### Out of scope (follow-up PRs)
- Removal of redundant labels (`wontfix`, `invalid`, plus the surviving
legacy `do not merge` if alias-rename collision keeps it)
- Org-wide sync from `cozystack/.github/labels.yml`
- Dosu bot configuration update for `lifecycle/stale` (requires
dashboard access)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automated PR labeling from Conventional Commits (type → kind/*, scope
→ area/*), plus a comprehensive namespaced label taxonomy.
* **Chores**
* Workflows to validate, sync, and auto-apply labels (including
scheduled/manual runs and validation checks).
* Added repository-wide label configuration and normalized bug label
metadata to namespaced form.
* Updated release PR titling/labeling conventions.
* **Documentation**
* Contributor and agent guidance on PR title conventions, label
mappings, and triage procedures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:160,173:
emit core.warning when a Conventional Commits type has no kind/*
mapping or a scope has no area/* mapping. Without the warning, typos
(e.g., "hotfix" instead of "fix") and recurring new scopes silently
fall through to area/uncategorized, masking that the mapping has
drifted.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:155:
Conventional Commits 1.0 spec item 16 treats BREAKING CHANGE: and
BREAKING-CHANGE: as synonymous footers. The hyphen form was silently
ignored before, so PRs that use it would miss kind/breaking-change.
https://www.conventionalcommits.org/en/v1.0.0/#specification
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:129:
defensive (pr.labels || []) avoids TypeError if the webhook payload
arrives without the labels field on edge cases like stripped edited
events.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Add an opt-in `upgradeCRDs` field to `ComponentInstall` in
`PackageSource`. The field maps directly to
`HelmRelease.Spec.Upgrade.CRDs` so a component author can declare how
Flux should handle CRDs from the chart's `crds/` directory when the
release is upgraded.
The helm-controller default on upgrade is `Skip`, which means CRDs added
by a chart bump never reach clusters that already have the release
installed — they must be applied manually with `kubectl apply --filename
charts/.../crds/`. This surfaces on every upgrade of an operator whose
CRD set expands between versions (etcd-operator, cnpg, kubevirt, kamaji,
etc.).
Setting `upgradeCRDs: CreateReplace` lets Flux apply new CRDs
declaratively with the chart.
Values are restricted to `Skip`, `Create`, `CreateReplace` via a
kubebuilder enum marker. Empty / unset preserves the existing
helm-controller default, so every current `PackageSource` keeps working
unchanged.
Migration is out of scope here — follow-ups will opt individual packages
in case-by-case.
### Relation to existing CRD management approach
The project convention (per #377) is to extract CRDs into a dedicated
Helm chart that reconciles ahead of the operator chart. This PR does not
replace that pattern — it complements it for charts that keep CRDs
inline under `charts/<name>/crds/` where extraction isn't practical
(vendored upstream charts with tightly coupled CRDs). Packages that
already split CRDs out can leave `upgradeCRDs` unset and keep using
their existing separate chart.
### Release note
```release-note
feat(operator): add opt-in `upgradeCRDs` field to `PackageSource` component `install` block to control how CRDs from the chart's `crds/` directory are applied on HelmRelease upgrades (`Skip` by default; use `CreateReplace` for operators whose CRD set expands between versions).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable CRD upgrade policy for component installs: Skip, Create,
CreateReplace — controls CRD handling during package upgrades and
preserves controller default when unset.
* **Documentation**
* Guidance on CRD upgrade semantics, advice to use CreateReplace for
specific operators, and warning about potential data-loss risks;
clarified contributor scope examples and PR template guidance.
* **Tests**
* Added tests validating CRD policy parsing and presence of the CRD
policy enum in the published schema.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Match the wording adopted in docs/agents/contributing.md so that human
contributors and AI agents see the same guidance in both places.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Scopes section was read as an exhaustive enumeration, which led to
review feedback flagging any scope outside the list as invalid. The
intent has always been that contributors pick the most specific scope
for the change and extend the list when a genuinely new area appears.
Reword the section accordingly and add operator as an example scope.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe when to set upgradeCRDs: CreateReplace (operators that evolve
their CRD set additively between versions) and the data-loss risk of
enabling it on operators that drop fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add an opt-in UpgradeCRDs field to ComponentInstall that maps to
HelmRelease.Spec.Upgrade.CRDs, allowing a PackageSource component to
declare how Flux should handle CRDs from the chart's crds/ directory
on upgrade.
The helm-controller default on upgrade is Skip, which means new CRDs
added between chart versions never reach existing clusters and must be
applied manually. Setting upgradeCRDs: CreateReplace makes Flux apply
new CRDs declaratively with the chart.
Allowed values are restricted to Skip, Create, CreateReplace via a
kubebuilder enum marker. Empty / unset preserves the existing Flux
default, so all existing PackageSource resources keep working.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:73:
add space before closing pipe in the type to kind mapping table for
consistency with other rows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on AGENTS.md:33:
expand bare labels.yml and pr-labeler.yaml to .github/labels.yml and
.github/workflows/pr-labeler.yaml for consistency with surrounding refs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:88:
… replaced with ... for compatibility across editors and tools.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on .github/labels.yml:242:
all hex color values use lowercase characters for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to retrigger Build that lost the OCI registry tag race
on the previous attempt.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to retrigger Build job that failed on Docker Hub anonymous
pull rate limit (429 Too Many Requests) for golang:1.26.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
- Increase kamaji controller memory limit from 500Mi to 512Mi
- Increase kamaji controller memory request from 100Mi to 256Mi
- Add startup probe with 60-second timeout (12 attempts × 5s periods)
- Increase readiness/liveness probe initialDelaySeconds from 5s/15s to
30s
## Problem
The kamaji controller was experiencing frequent CrashLoopBackOff due to
OOMKilled errors. Analysis showed:
- Container was being killed with exit code 137 (OOMKilled) after ~20-25
seconds of runtime
- Memory limit of 500Mi was insufficient for controller initialization
- Readiness probe was failing because it started too early (5s
initialDelay), before the controller finished leader election (~17s)
## Solution
**Memory increase:**
- Limit: 500Mi → 512Mi (based on production testing)
- Request: 100Mi → 256Mi (ensures adequate reservation)
**Startup probe:**
- Added to give controller up to 60 seconds to initialize without being
killed by liveness probe
- 12 attempts × 5s period = 60s maximum startup time
**Probe delays:**
- ReadinessProbe: 5s → 30s initialDelay (controller needs ~17s to
acquire leader lease)
- LivenessProbe: 15s → 30s initialDelay (aligned with readiness)
## Testing
Verified in production cluster:
- Controller runs stable with 0 restarts
- No more OOMKilled events
- Successfully creates kubeconfig secrets for tenant clusters
## Related Issues
Fixes tenant cluster components stuck in ContainerCreating due to
missing kubeconfig secrets (caused by crashing kamaji controller).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Introduced automated health checks using HTTP-based probes to monitor
service status during startup, continuous operation, and readiness to
handle traffic.
* Adjusted container memory resource allocation for enhanced stability
and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The pool is now rendered from packages/apps/tenant/templates/cilium-lb-pool.yaml instead of packages/extra/ingress/templates/cilium-lb-pool.yaml. Cilium LB IPAM forbids overlapping CIDRs across pools regardless of serviceSelector, so both the ingress-loadBalancer path and the upcoming per-tenant Gateway in #2470 cannot each own their own pool on the same publishing.externalIPs range. The tenant chart is the natural per-tenant owner — it already creates the Namespace, the cozystack-values Secret, and the HelmReleases for both ingress and gateway.
The new pool uses a namespace-only serviceSelector (io.kubernetes.service.namespace: <ns>), which matches any LoadBalancer Service in the tenant namespace. The metadata.name changed from <trim>-ingress to <trim>-exposure to reflect that the pool is not ingress-specific.
Only the ingress-loadBalancer signal is wired in this commit (_cluster.expose-mode=loadBalancer plus .Values.ingress=true on the publishing tenant). The gateway branch is added in #2470 on top of this commit — it rebases, drops its own packages/extra/gateway/templates/cilium-lb-pool.yaml, and adds an OR branch for .Values.gateway in the tenant template.
Pool-rendering unit tests moved from packages/extra/ingress/tests/ to packages/apps/tenant/tests/. The ingress chart tests keep the Service-level asserts. packages/apps/tenant/Makefile gains a test target so hack/helm-unit-tests.sh picks up the new suite.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on
packages/system/postgres-operator/values.yaml:30.
The old comment's 5-minute ImagePullBackOff scenario conflicted with
the 180s activeDeadlineSeconds that the default maxAttempts/sleepSeconds
resolve to, so the numbers could not both be taken at face value.
Rewrite the comment to state the actual deadline math and frame the two
gates as an AND with activeDeadlineSeconds being the shorter one under
defaults, so readers understand why backoffLimit has little headroom
without an accompanying maxAttempts/sleepSeconds bump.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:115.
The wait container had no resource requests or limits, so schedulers
treated it as BestEffort and downstream quota enforcement had no signal.
Set small requests (10m CPU, 32Mi memory) and conservative limits
(100m CPU, 64Mi memory) matching the actual footprint of the kubectl
polling loop.
Add a matching unittest assertion so the values stay in sync if anyone
touches the template.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:83.
The Job targets the cnpg-webhook-service/services/proxy subresource, which
is namespaced and lives in the release namespace. A namespaced Role and
RoleBinding grant the exact permission needed without creating global
RBAC for a namespaced probe, which is the principle of least privilege.
Also update the kind assertions in tests/webhook-ready-hook_test.yaml so
the unittest suite tracks the new Role/RoleBinding objects.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Bumps the vendored `robotlb` chart to the latest upstream build.
Chart version remains `0.1.3`; the bundled `appVersion` moves from
`0.0.5` to `0.0.6`.
The new `robotlb` release adds RBAC permissions for
`discovery.k8s.io/endpointslices` (`get`, `list`, `watch`), which are
required to manage services backed by `EndpointSlice` — notably
KubeVirt-exposed workloads that do not publish classic `Endpoints`.
Notes:
- Upstream also replaced `replicas: {{ .Values.replicas }}` with a
hardcoded `replicas: 1` in `templates/deployment.yaml`. The
effective replica count is unchanged (we already set `1`), but the
value is no longer overridable via chart values. A minor cosmetic
reformat was applied to `templates/role.yaml`.
Closes#2256
### Release note
```release-note
chore(hetzner-robotlb): update robotlb to 0.0.6 — adds RBAC for EndpointSlices so services backed by EndpointSlice (e.g. KubeVirt) are supported.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Extended service account permissions to access Kubernetes endpoint
slices from the discovery API.
* **Bug Fixes**
* Deployment replica configuration now fixed to single instance.
* **Style**
* Improved YAML formatting in role template declarations.
* **Chores**
* Updated application version metadata to 0.0.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Refreshes the vendored Cilium chart in `packages/system/cilium` from
v1.19.1 to v1.19.3 via `make update`. Chart templates, values, CRDs and
the Cilium image reference are regenerated from upstream.
### Motivation
- **v1.19.2** ships a critical fix for cert-manager HTTP-01 Gateway API
challenges on hostnames that have both HTTP and HTTPS listeners
([cilium#44492](https://github.com/cilium/cilium/pull/44492), backport
[#44517](https://github.com/cilium/cilium/pull/44517)). Without this
fix, cert-manager cannot issue certificates via Gateway API when a
redirect HTTP listener and a TLS HTTPS listener share a hostname.
- **v1.19.3** is the latest stable patch release in the v1.19.x line (15
Apr 2026).
- This bump is a prerequisite for upcoming Gateway API work tracked
separately.
### Upstream changes pulled in
- Cilium Envoy bootstrap config, operator clusterrole, config template,
`values.schema.json` and the cilium-agent DaemonSet refreshed from
upstream.
- New `templates/ztunnel/` directory (DaemonSet, Secret, ServiceAccount)
added by upstream — not enabled by default in Cozystack values.
### Release note
```release-note
chore(cilium): bump to v1.19.3 (cert-manager HTTP-01 fix via cilium#44492)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added ztunnel encryption support with configurable deployment settings
* Added ConfigDriftDetection for monitoring ConfigMap changes
* Added endpoint policy update timeout configuration
* Added load balancer service topology support
* Extended Envoy circuit breaker configuration with connection and
request limits
* **Updates**
* Upgraded Cilium to v1.19.3
* Updated container images (Envoy, certgen, Hubble relay, clustermesh)
* Enhanced Cilium operator RBAC capabilities for managing ServiceImport
finalizers
* **Removals**
* Removed BIG TCP tunnel configuration option
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The cnpg admission webhook's controller pod passes readinessProbe as soon
as the local HTTPS server binds to :9443. The HelmRelease marks itself
Ready right after that via helm install --wait. But the Service
cnpg-webhook-service needs its EndpointSlice populated and the data plane
(kube-proxy / cilium) programmed before kube-apiserver can reach the
webhook through the Service ClusterIP. That gap is short but not zero,
and any HelmRelease that depends on postgres-operator (cozy-keycloak,
tenant Postgres apps) can fire its own install inside the window and hit
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
failed to call webhook: Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
which fails the install of the downstream release. Seen on cozystack/cozystack#2470
E2E run 24862782568.
Add a post-install,post-upgrade Helm hook Job that blocks the release
from reporting Ready until the webhook answers /readyz through the
apiserver service proxy. Apiserver proxy routes the call over the same
Service IP → EndpointSlice → pod path the admission webhook uses, so
once it responds, the webhook admission path is also working.
RBAC is minimal: a dedicated ServiceAccount with a ClusterRole that only
grants get on services/proxy scoped to https:cnpg-webhook-service:webhook-server.
The Job times out after 120s with 60 attempts at 2s intervals — longer
than any data-plane programming delay seen on E2E, but bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/extra/ingress/templates/cilium-lb-pool.yaml:19: if the operator
passes an externalIP already in CIDR form (192.0.2.10/32 or 2001:db8::1/128),
the template appended a second /32 or /128 suffix producing an invalid
CiliumLoadBalancerIPPool block. Guard the suffix append on the absence of
"/" in the input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add generated-output dirs (pkg/generated, internal/crdinstall/manifests,
packages/system/*/definitions) and Makefile to the workflow paths: filter
so PRs that modify only generated artifacts still trigger the drift check.
- Mirror the same paths in the root pre-commit hook's files: regex so
manual edits to generated files or changes to the root generate target
re-run make generate through pre-commit.
- Switch drift detection in the workflow from `git diff --exit-code` to
`git status --porcelain` so new untracked files produced by make generate
(e.g. generated YAML/Go for a new API type) also fail the job; dump
`git diff --color=always` on failure for easier debugging.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds explicit guidance to `docs/agents/contributing.md` about running
`make generate` in any touched package before committing. Pre-commit CI
runs `make generate` in every package and fails with exit code 123 on
any uncommitted generator output (regenerated `README.md`, reordered
`values.schema.json`, refreshed
`packages/system/<name>-rd/cozyrds/<name>.yaml`).
Recent PRs have tripped on this during review cycles. Documenting it in
the contributing checklist saves a round-trip.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Require regenerating and committing generated artifacts when
designated source files are edited.
* Added a section listing generated artifacts per package and concrete
regeneration/staging steps.
* Documented CI enforcement that detects unstaged generator output and
blocks PRs.
* Added guidance for rerunning generation after amended commits, plus
updated commit-scope guidance (now “not exhaustive”) and included
“agents” in allowed scopes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on docs/agents/contributing.md:11:
Scope linters kept flagging valid scopes like 'agents' as unknown because
the list read as exhaustive. Annotate it as examples (not exhaustive) and
add 'agents' to the Other group so both humans and review bots stop
tripping on scopes that are already in regular use across the repo
history.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:30:
Reword "likely to need regenerated" (regional construction) to
"likely needs to be regenerated" for standard technical prose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on docs/agents/contributing.md:26:
Replace the hard-coded packages/extra/<name> path in the example with
packages/<apps-or-extra>/<name> so the example matches the preceding
text that describes both apps and extra packages.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Service.spec.externalIPs is deprecated upstream in Kubernetes v1.36
(KEP-5707, kubernetes#137293). The AllowServiceExternalIPs feature gate
is expected to default to off around v1.40 and the implementation to
be removed around v1.43. For bare-metal installs that rely on
externalIPs today, cozystack needs a migration path.
This change adds an opt-in 'loadBalancer' exposure mode for the
ingress-nginx Service:
- New platform value 'publishing.exposure' (enum: externalIPs |
loadBalancer, default externalIPs). Plumbed through cozystack-values
into each tenant's ingress HelmRelease via the new 'expose-mode' key.
- Unknown values and loadBalancer with an empty externalIPs list fail
the chart render with explicit error messages, rather than silently
producing a broken Service.
- When exposure=loadBalancer and the current namespace matches
publishing.ingressName, the Service becomes type: LoadBalancer with
externalTrafficPolicy: Local.
- A new template renders a CiliumLoadBalancerIPPool whose blocks come
from publishing.externalIPs (IPv4 addresses get /32, IPv6 addresses
get /128) and whose serviceSelector uses Cilium's synthetic
io.kubernetes.service.namespace key combined with the standard
app.kubernetes.io/name: ingress-nginx label. No custom label is
written to the Service itself, avoiding cross-tenant collisions from
user-defined labels.
Default behaviour is unchanged: without opting in, the Service is
still ClusterIP + spec.externalIPs as today.
Scope: only ingress-nginx is migrated by this setting. Other cozystack
components that still write Service.spec.externalIPs directly (notably
the vpn app) must be migrated separately before the v1.40 feature gate
flip.
Tests: packages/extra/ingress/tests/exposure_test.yaml adds 13
helm-unittest cases covering both modes, IPv4/IPv6, empty-token
filtering, unknown-mode rejection, and the non-matching-namespace
fallback.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous image digest in values.yaml pointed at a single-arch
linux/arm64 manifest because 'make image' was run from an arm64 host
with the default buildx platform. Cozystack targets amd64 (Talos build
output, E2E runners, most real-world clusters) and also arm64 for
hybrid fleets, so Helm install would fail on amd64 nodes with 'no
matching manifest for linux/amd64 in the manifest list entries'
whenever somebody installed directly from this commit between merge
and the next release-tag CI rebuild.
Fix: rebuilt the image locally with
PLATFORM='linux/amd64,linux/arm64' make image from a buildx
docker-container driver, pushed the multi-arch manifest, and
refreshed values.yaml with:
- digest of the new multi-arch manifest list (verified via
'docker manifest inspect': amd64 sha256:e1977323..., arm64
sha256:8f5ab529...).
- tag bumped from 'latest' (emitted by the common-envs.mk settag
macro on a non-tagged checkout) to '1.19.3', matching the
established convention in every other packages/system/*/values.yaml
so reviewers and incident response have a human-readable version
anchor independent of digest chasing.
The Makefile is left untouched so the CI builder (which only uses the
default docker driver) keeps building single-arch for whatever
architecture it runs on; multi-arch is a responsibility of the
release-tag pipeline or an explicit local rebuild.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
