cozystack/packages/system
Aleksei Sviridkin bdf23e66d1
fix(kubernetes): close admin-kubeconfig race on tenant cluster bootstrap (#2413)
## What this PR does

Closes #2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.

Implements a defense-in-depth fix:

- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.

Tests:

- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.

All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).

Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.

### Release note

```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
  * Chart resources now render conditionally based on etcd presence.

* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.

* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-27 15:31:52 +03:00
..
application-definition-crd refactor(api): rename CozystackResourceDefinition to ApplicationDefinition 2026-01-15 22:35:56 +01:00
backup-controller Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
backupstrategy-controller fix(backups): move velero-configmap Role to velero chart (#2459) 2026-04-23 11:57:54 +05:00
bootbox refactor(labels): remove cozystack.io/ui label 2026-01-19 13:59:29 +01:00
bootbox-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
bucket Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
bucket-rd fix(bucket): regenerate schema and docs from values.yaml annotations 2026-03-03 19:32:32 +03:00
capi-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
capi-providers-bootstrap refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
capi-providers-core refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
capi-providers-cpprovider fix(capi-provider): update Kamaji CAPI provider to v0.16.0 2026-02-19 22:12:33 +03:00
capi-providers-infraprovider refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
cert-manager [cert-manager] Added cleanup before crds upgrade 2026-02-26 18:32:35 +05:00
cert-manager-crds [cert-manager] Updated cert-manager to 1.19.3 with new crd packaging 2026-02-26 15:28:19 +03:00
cert-manager-issuers fix(platform): migrate ACME HTTP-01 to ingressClassName API 2026-04-21 17:00:31 +05:00
cilium fix(cilium): rebuild image multi-arch and pin tag to 1.19.3 2026-04-23 17:33:19 +03:00
cilium-networkpolicy refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
clickhouse-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
clickhouse-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
cluster-autoscaler feat(cluster-autoscaler): enable enforce-node-group-min-size by default 2026-02-14 11:42:52 +01:00
clustersecret-operator Add clustersecret-operator system package 2026-02-10 22:47:20 +05:00
coredns fix coredns serviceaccount to match kubernetes bootstrap rbac 2026-02-02 15:32:50 +01:00
cozy-proxy refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
cozystack-api Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
cozystack-basics [system] Fix tenant RBAC for endpointslices read access (#2257) 2026-03-27 17:48:46 +05:00
cozystack-controller Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
cozystack-scheduler feat(scheduler): storage-aware scheduling 2026-04-13 17:15:09 +03:00
dashboard Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
etcd-operator fix(etcd-operator): replace deprecated kube-rbac-proxy image 2026-03-10 08:30:43 +01:00
etcd-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
external-dns feat(extra): add external-dns as standalone extra package 2026-03-10 12:48:01 +01:00
external-dns-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
external-secrets-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
flux-plunger Fix pre-commit check 2026-01-20 03:24:32 +01:00
fluxcd refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
fluxcd-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
foundationdb-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
foundationdb-rd refactor(labels): remove cozystack.io/ui label 2026-01-19 13:59:29 +01:00
gateway-api-crds refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
goldpinger refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
gpu-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
grafana-operator Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
harbor Fix system postgresql images to 17.7-standard-trixie 2026-04-10 12:47:46 +05:00
harbor-rd fix(harbor): remove incorrect tenant module flags 2026-04-21 20:26:13 +02:00
hetzner-robotlb chore(hetzner-robotlb): update robotlb chart to appVersion 0.0.6 2026-04-23 16:57:23 +05:00
http-cache-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
info-rd refactor(labels): remove cozystack.io/ui label 2026-01-19 13:59:29 +01:00
ingress-nginx Replaced direct chart edits with patch files 2026-03-02 12:35:14 +05:00
ingress-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
kafka-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
kafka-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
kamaji fix(kamaji): increase memory limits and add startup probe (#2421) 2026-04-24 23:11:53 +03:00
keycloak fix(platform): migrate ACME HTTP-01 to ingressClassName API 2026-04-21 17:00:31 +05:00
keycloak-configure [keycloak-configure] Make values.yaml self-documenting 2026-04-02 19:29:54 +03:00
keycloak-operator [keycloak-operator] Update to v1.32.0 2026-03-11 15:15:09 +03:00
kilo [kilo] Switch from fork to upstream squat/kilo 2026-03-23 15:18:23 +03:00
kubeovn fix(kube-ovn): bump kube-ovn to v1.15.10 with port-group regression fix 2026-04-21 20:09:19 +02:00
kubeovn-plunger Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
kubeovn-webhook Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
kubernetes-rd feat(kubernetes): make wait-for-kubeconfig image overridable for air-gapped registries 2026-04-27 13:50:08 +03:00
kubevirt fix(kubevirt): disable serial console log to fix VM pod initialization 2026-02-21 20:46:31 +01:00
kubevirt-cdi feat(cdi): switch clone strategy to csi-clone 2026-03-10 12:42:01 +01:00
kubevirt-cdi-operator [cdi-operator] Update to v1.64.0 2026-02-20 20:16:33 +01:00
kubevirt-csi-node Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
kubevirt-instancetypes refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
kubevirt-operator [kubevirt-operator] Update to v1.6.4 2026-02-20 20:16:34 +01:00
lineage-controller-webhook Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
linstor Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
linstor-gui fix(platform): migrate ACME HTTP-01 to ingressClassName API 2026-04-21 17:00:31 +05:00
linstor-scheduler feat(scheduler): storage-aware scheduling 2026-04-13 17:15:09 +03:00
local-ccm Update local-ccm v0.3.0 2026-02-10 20:19:03 +01:00
mariadb-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
mariadb-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
metallb Prepare release v1.0.0-alpha.1 2026-01-15 23:32:41 +00:00
metrics-server refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
mongodb-operator Fix pre-commit check 2026-01-20 03:24:32 +01:00
mongodb-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
monitoring fix(platform): migrate ACME HTTP-01 to ingressClassName API 2026-04-21 17:00:31 +05:00
monitoring-agents feat(monitoring): migrate VictoriaLogs from VLogs to VLCluster 2026-03-10 12:04:12 +05:00
monitoring-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
multus Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
nats Add monitoring for NATs 2026-02-17 22:54:12 +01:00
nats-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
nfs-driver refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
objectstorage-controller Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
openbao feat(openbao): add system chart with vendored upstream 2026-02-17 20:03:01 +03:00
openbao-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
opencost Fix pre-commit check 2026-01-20 03:24:32 +01:00
opensearch-operator [opensearch] Address PR review feedback 2026-03-17 11:20:44 +01:00
opensearch-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
piraeus-operator [piraeus-operator] Split LINSTOR controller availability and metrics alerts 2026-03-25 16:11:14 +03:00
piraeus-operator-crds Move CRDs installation logic to piraeus-operator-crds chart 2026-02-11 22:54:56 +01:00
postgres-operator [postgres-operator] Update to v1.27.3 2026-03-17 01:59:28 +03:00
postgres-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
prometheus-operator-crds refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
qdrant [qdrant] Add Qdrant vector database application 2026-02-06 13:30:58 +03:00
qdrant-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
rabbitmq-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
rabbitmq-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
redis-operator refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
redis-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
reloader refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
seaweedfs Prepare release v1.3.0 2026-04-22 07:28:09 +00:00
seaweedfs-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
snapshot-controller refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
tcp-balancer-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
telepresence refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
tenant-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
velero fix(backups): move velero-configmap Role to velero chart 2026-04-23 10:41:41 +05:00
vertical-pod-autoscaler Post upgrade fixes for v1.0 packagesources 2026-01-28 20:08:35 +03:00
vertical-pod-autoscaler-crds refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
victoria-metrics-operator feat(monitoring): upgrade victoria-metrics-operator to v0.68.4 2026-04-20 15:48:53 +03:00
virtualprivatecloud-rd [docs] Update cozyvalues-gen 2026-03-25 15:59:22 +05:00
vm-default-images [vm-default-images] Default to replicated storageClass and document storage requirements 2026-04-14 19:59:20 +05:00
vm-disk-rd [vm-disk] Fix vm-disk-rd 2026-04-14 19:59:19 +05:00
vm-instance-rd chore(vm-instance): regenerate CRD and schema via make generate 2026-03-30 10:55:26 +05:00
vpn-rd [docs] Updated app go types 2026-03-25 15:57:25 +05:00
vsnap-crd refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00
Makefile refactor: move scripts to hack directory 2026-01-15 16:06:56 +01:00