The previous fix (a21c18f9) sourced BucketInfo via Flux valuesFrom with
`targetPath: bucket.bucketInfo`. Flux runs values with `targetPath`
through Helm's `strvals.ParseInto`, which splits the value on commas as
list separators. The COSI BucketInfo JSON is comma-rich, so values
resolution bailed:
could not resolve Secret chart values reference
'tenant-X/harbor-X-registry-bucket' with key 'BucketInfo':
key "\"spec\":{\"bucketName\":\"bucket-1785...\""
has no value (cannot end with ,)
Drop `targetPath`. With only `valuesKey: BucketInfo`, helm-controller
unmarshals the value as YAML and merges at the chart's values root, so
JSON commas stay nested instead of being split. The system chart's
bucket-secret.yaml now reads `.Values.spec.bucketName` /
`.Values.spec.secretS3.{accessKeyID,accessSecretKey,endpoint}`,
guarded by nested `with` blocks so the registry-s3 Secret renders only
when secretS3 has been populated.
Gating semantics from the previous fix are preserved: with the default
`optional: false`, helm-controller still refuses to compose values
until the COSI BucketAccess controller writes `BucketInfo` into the
Secret, and the value's content still drives the HR config-digest.
Verified via `helm template` against the system chart with three
scenarios: missing `spec` and `spec` without `secretS3` render
nothing; full BucketInfo renders all five S3 env keys correctly.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Some workloads (OVN raft, LINSTOR controller) fail when replicas start
at different times due to image-pull stagger across nodes. Add a
DaemonSet-based pre-pull step that runs before helm install, ensuring
all nodes have the images cached so every replica starts within
milliseconds of each other.
The script accepts image refs on stdin and creates one container per
image (parallel pulls, total time = max of any single image rather than
sum). The bats test sources images directly from the rendered charts
via yq, walking only PodSpec-shaped objects so version bumps stay in
sync automatically without a separate hardcoded list.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Some workloads (OVN raft, LINSTOR controller) fail when replicas start
at different times due to image-pull stagger across nodes. Add a
DaemonSet-based pre-pull step that runs before helm install, ensuring
all nodes have the images cached so every replica starts within
milliseconds of each other.
Covers kube-ovn (confirmed OVN raft race), piraeus-server, and
linstor-csi. Comments in hack/e2e-prepull-images.sh point to the
source chart values so versions stay in sync.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Upstream COSI v0.2.2's BucketAccess reconciler does a Get->mutate->Update
on the parent Bucket and surfaces "Operation cannot be fulfilled ... the
object has been modified" as a FailedGrantAccess event when it races
against the Bucket reconciler in the same controller process. Wrap the
mutation in retry.RetryOnConflict so the reconcile loop refreshes and
retries instead of leaking the conflict to users.
Carried as 91-bucketaccess-conflict-retry.diff until upstreamed
(cf. 89-reconciliation.diff and 90-bucket-name.diff, both dropped
in c29d501b once merged upstream).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two bugs caused the harbor E2E to fail (CI #25049445125):
1. cozy-harbor's bucket-secret.yaml called `index $existingSecret.data
"BucketInfo"` unconditionally. During first-time install the COSI
BucketAccess controller may create the credentials Secret as a
placeholder before populating it, so `.data` is nil and the chart
render crashes with `index of untyped nil`.
2. The downstream `<release>-system` HelmRelease starts reconciling
immediately, in parallel with bucket provisioning, hitting bug #1
on every retry within the test's 5-minute window.
The previous shape relied on `lookup` to read the Secret at render
time. helm-controller's upgrade trigger is digest-based over composed
values + chart artifact, so a `lookup` returning new data on a later
reconcile is not enough on its own to force an upgrade — the rendered
output may diverge but the digest does not.
Switch to a values-driven shape:
- `bucket-secret.yaml` now reads `.Values.bucket.bucketInfo` (a JSON
string) and uses `dig` for safe access; if the value is empty or the
expected nested fields are missing, no `*-registry-s3` Secret is
rendered.
- The `<release>-system` HR sources `BucketInfo` via `valuesFrom` with
`targetPath: bucket.bucketInfo`. With the default `optional: false`,
helm-controller will refuse to compose values until the key exists,
which both gates initial reconciliation on the BucketAccess Secret
being populated and forces a config-digest change (and thus a helm
upgrade) when its contents change.
- `bucket.secretName` is removed from the system chart's values and
from the apps chart's `values:` block; the only consumer (the
`lookup` call) is gone.
Verified four scenarios via `helm template` against the system chart:
unset bucketInfo, empty string, empty JSON object `{}`, and a
fully-populated BucketInfo — only the last renders the registry-s3
Secret; the others render nothing without erroring.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`kubectl wait <kind> <name>` errors immediately with NotFound if the
resource doesn't exist yet — even with --for=condition or --for=jsonpath.
The redis test in CI run 25043350705 failed in 1 second for exactly this
reason: the redis-failover operator hadn't created the PVC by the time
the test waited for it. Previously the 3x retry on `Run E2E tests`
masked this race; with retry dropped, every such call is a flake risk.
Add a small `until kubectl get` existence backstop before each
kubectl wait, matching the pattern already established for HRs in
commit 66888c91.
33 backstops across 12 files:
bucket.bats — bucketclaims, bucketaccesses x2
clickhouse.bats — statefulset 0-0 (0-1 already covered)
harbor.bats — deploy x3, bucketclaims, bucketaccesses
kafka.bats — kafkas
mariadb.bats — statefulset, deployment
mongodb.bats — statefulset
openbao.bats — sts, pvc
postgres.bats — job.batch
qdrant.bats — sts, pvc
redis.bats — pvc, deploy, sts (the trigger)
vminstance.bats — dv, pvc, vm (with 120s for KubeVirt latency)
e2e-install-cozystack.bats — apiservices, sts/etcd, vmalert,
vmalertmanager, vlclusters, vmcluster,
clusters.postgresql.cnpg.io,
deploy/grafana-deployment, namespace
Same pattern, same 60s default timeout for the existence wait (120s for
nested-virt resources). Once the resource exists, the original wait
timeout takes over.
Run-kubernetes.sh has the same race shape on several waits (nfs, kamaji,
machinedeployment, etc.) — out of scope here; flagged for follow-up.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart fix (3a87485c, be0a6687) removes the chart-defined Namespace
without replacing it. On the Talos E2E cluster the apiserver enforces
PodSecurity baseline-by-default on bare namespaces, so the cozystack-
operator pod (hostNetwork=true) is rejected if cozy-system is created
without enforce=privileged.
Apply the namespace from the bats with the labels the operator depends
on, then run helm install without --create-namespace. Mirrors the
documented production install procedure for the new chart shape.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The pre-install Job approach (added in c0b76b16, b46151a4) hits a fatal
chicken-and-egg on Talos clusters that enforce PodSecurity baseline:latest
on bare namespaces:
pods "cozy-system-labeler-..." is forbidden:
violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true)
The labeler needs hostNetwork=true (no CNI yet) which requires the namespace
to be PSA-privileged, but the labeler's whole job is to add that label.
Cannot work from inside the namespace it's labeling.
Drop the labeler Job + ServiceAccount + ClusterRole + ClusterRoleBinding.
The chart now only removes the chart-defined Namespace and assumes the
caller pre-creates cozy-system with the right labels — same pattern as
kube-prometheus-stack, argo-cd, cert-manager, and other mainstream charts.
Install procedure becomes two-step:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: cozy-system
labels:
cozystack.io/system: "true"
cozystack.io/deletion-protected: "true"
pod-security.kubernetes.io/enforce: privileged
EOF
helm upgrade installer packages/core/installer \
--install --namespace cozy-system ...
The cozystack-operator pod (hostNetwork=true) is then admitted because the
namespace already has enforce=privileged.
Verified end-to-end on a kind cluster — clean install in 0.1s, no hooks
involved, no PSA conflict, no retry needed.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The pre-install Job runs before any CNI is installed, so all nodes are
tainted NotReady-NoSchedule and the pod has no pod network. Without
adjustment the Job's pod sits Pending until helm's pre-install timeout
fires, blocking the entire install.
Surfaced in PR #2500 CI run 25040584552 attempt 2:
FailedScheduling 0/3 nodes are available: 3 node(s) had untolerated
taint {node.kubernetes.io/not-ready: }.
Mirror the cozystack-operator deployment's scheduling pattern:
- hostNetwork=true so the pod doesn't depend on CNI
- Tolerations matching the operator
(not-ready / unreachable / cilium.agent-not-ready / cloudprovider.uninitialized)
- Variant-aware KUBERNETES_SERVICE_HOST/PORT env so kubectl reaches the
apiserver before kube-proxy + CNI are up:
talos: localhost:7445 (KubePrism)
generic: cozystack.apiServerHost / Port
hosted: default in-cluster
Verified on the sandbox kind cluster that the rendered Job manifest
schedules and the kubectl container starts (the kind cluster itself
doesn't run KubePrism so the env-var path can't be end-to-end-tested
there; the Talos E2E will exercise it).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The HelmRelease generation in package_reconciler hardcoded three values
that have caused recurring pain:
- Install.Timeout = 10m (line 214)
- Upgrade.Timeout = 10m (line 220)
- MaxHistory = unset (Helm default 5)
The 10m timeout in particular contradicts the per-Application timeout
work done in `pkg/config/config.go` and
`pkg/registry/apps/application/rest.go` (commit 7b146cbe), which lets
individual Applications override the install/upgrade timeout via
annotation. The PackageSource side had no equivalent — etcd works around
it by hand-rolling its HR YAML in `packages/apps/tenant/templates/etcd.yaml`
to set a 30m timeout, harbor has had multiple commits chasing the same
problem from the other side. This closes that gap.
New operator flags (each maps to a chart value with empty default;
operator uses its own default when value is empty):
- `--helmrelease-install-timeout` (default 10m) → Spec.Install.Timeout
- `--helmrelease-upgrade-timeout` (default 10m) → Spec.Upgrade.Timeout
- `--helmrelease-max-history` (default 5) → Spec.MaxHistory
Production behaviour unchanged. E2E and edge cases (cert rotation,
slow-installing charts) can override per-cluster without modifying
chart manifests.
Per-component overrides on `ComponentInstall` (mirroring the existing
`UpgradeCRDs` pattern) would be a strictly better fix for charts like
etcd that need an outlier timeout — left as a deliberate followup since
it touches the Package CR API.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The operator generated HelmReleases with `Install.Remediation{Retries: -1}`
and `Upgrade.Remediation{Retries: -1}`, which is the v2 way of saying
"retry forever, never remediate". Combined with the spec-level
`Interval: 5m`, this meant a failed install/upgrade waited a full 5
minutes before the next attempt — even when the underlying race that
caused the failure (e.g. a chart artifact not quite Ready, a HR
dependency reconciling out of order) cleared in seconds.
Audit of the last 30 successful PR runs found that every single run had
its `Install Cozystack` step's `kubectl wait hr/seaweedfs-system` time
out at the 2-minute mark and recover only after an inline
`flux reconcile --force` workaround. Same root cause: 5-minute retry
interval was longer than the test's wait window. That workaround can be
removed once this change lands.
Switch to `Strategy.Name=RetryOnFailure` with `RetryInterval` exposed via
a new `--helmrelease-retry-interval` operator flag (default 30s):
- Functionally equivalent to the previous configuration ("retry forever
on failure") since `Retries: -1` meant remediation never fired anyway.
- Decouples retry-on-failure timing from `spec.Interval` so failed
releases recover at 30s without polling healthy releases at the same
cadence (which would multiply controller load by ~10x for no benefit).
- Helm chart exposes `cozystackOperator.helmReleaseRetryInterval`,
defaulting to empty (operator uses its own 30s default).
- Same flag pattern as the existing `--helmrelease-interval` (5m default),
also added in this PR via cherry-pick.
Verified: go build green, go test ./internal/operator/... pass, helm
template renders both flags conditionally on values.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two reliability fixes in `hack/e2e-apps/vminstance.bats`:
1. Delete-recreate race. Both @test blocks deleted the prior VMInstance/VMDisk
with `kubectl ... --timeout=2m || true`. The `|| true` silently swallowed
timeout errors, letting the test continue with the resource still in
finalizer-drain. The next `kubectl apply` then no-ops with
"Detected changes to resource ... which is currently being deleted",
producing a NotFound on the downstream HR wait. Bumped the delete
timeout to 3m and removed `|| true` so true delete failures surface
loudly. End-of-test cleanup deletes also got the explicit timeout.
2. VM IP and VM ready timeouts. The 20s timeout for the VMI to acquire an
IP was unrealistic for nested KubeVirt under runner load — virt-launcher
+ libvirt + cloud-init DHCP routinely takes 30-60s. Bumped to 120s with
a 2s poll interval (60 polls instead of 4). VM ready timeout bumped from
20s to 60s for the same reason.
Both surfaced as recurring first-attempt failures in the past 30 successful
PR runs (7/17 analysable runs in each case) — masked by the 3x retry on
`Run E2E tests`.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Now that the chart fix (this same branch) bootstraps cozy-system via
--create-namespace + a pre-install label hook, the bats no longer needs
to pre-apply a hand-crafted helm-adoptable namespace. Restore plain
--create-namespace and drop the kubectl apply block.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Removes the chart's `Namespace cozy-system` resource and replaces it with
a pre-install/pre-upgrade Job hook (cozy-system-labeler) that patches the
required labels onto the namespace after `--create-namespace` creates it.
Why: helm v3 has a known chicken-and-egg with charts that ship their own
Namespace:
- WITH `--create-namespace` on the install command, helm pre-creates the
namespace via plain kubectl-create (no helm meta annotations); the
chart's own Namespace apply then fails with `already exists`.
- WITHOUT `--create-namespace`, helm fails immediately because it cannot
write its release-secret to a non-existent namespace.
Until now this was hidden by the 3x retry on `Install Cozystack` in
`.github/workflows/pull-requests.yaml`: first attempt always fails with
the conflict, second attempt sees the existing failed release and takes
the upgrade code path which patch-merges instead of strict-create.
Reproducible on every cold install. Surfaced cleanly when retries on the
install step were dropped.
After this change:
- install commands use `helm upgrade --install --namespace cozy-system
--create-namespace`. Standard pattern, matches kube-prometheus-stack /
argo-cd / cert-manager / others.
- the pre-install hook (SA + ClusterRole + ClusterRoleBinding + Job)
patches `cozystack.io/system=true` and
`pod-security.kubernetes.io/enforce=privileged` onto the namespace
before main resources apply.
- hook-delete-policy=before-hook-creation,hook-succeeded so the RBAC
surface only exists during install/upgrade.
Verified end-to-end on a kind cluster: cold install in 3.3s, upgrade
idempotent, cleanup clean. Image pinned to `alpine/k8s:1.32.0` for the
hook (small, public, includes kubectl).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozy-installer chart declares Namespace cozy-system itself, which
makes helm cold-install paradoxical:
- WITH --create-namespace: helm pre-creates the ns without helm
annotations; the chart's own Namespace apply then fails with
'already exists'.
- WITHOUT --create-namespace: helm fails before any apply because it
can't write its release-secret to a non-existent ns.
The 3x retry was masking both directions — second attempt saw a partial
release and took the upgrade path, which patch-merges instead of
strict-create. Reproducible on every cold install across PRs.
Pre-create cozy-system with meta.helm.sh/release-name and
app.kubernetes.io/managed-by=Helm so helm adopts it cleanly during
install. Labels mirror the chart's template (PSA enforce=privileged etc.)
so behaviour matches what the chart would have done.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozy-installer chart declares Namespace cozy-system itself (with
helm.sh/resource-policy: keep). Combining that with --create-namespace
makes Helm v3 pre-create the namespace via plain kubectl-create (without
helm annotations); the subsequent chart apply then fails with
'namespaces cozy-system already exists'.
The 3x retry on Install Cozystack was hiding this. First attempt failed,
second saw 'release exists' and treated it as upgrade. Reproducible
across PRs (PR #2507 E2E hit the same first-attempt failure today and
recovered on retry). Surfaced cleanly after dropping the retry on this
step.
Drop --create-namespace; let the chart manage its own namespace.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three poll/sleep sites in the install bats and the kubernetes runner are
NOT replaceable by `kubectl wait --for=condition=...` because the signal
they observe is not a Kubernetes API condition:
- e2e-install-cozystack.bats:55-56 - 5s pad lets late-arriving HRs join
the awk-snapshot the parallel `kubectl wait` runs against. There is no
k8s condition for "all expected platform HRs have been emitted" short
of hard-coding the list.
- e2e-install-cozystack.bats:75 - LINSTOR node membership is reported by
the linstor binary running inside the controller pod (kubectl exec),
not a CRD status, so kubectl wait cannot subscribe to it.
- e2e-apps/run-kubernetes.sh:231-238 - validates the external HTTP path
through MetalLB -> tenant ingress -> backend pod end-to-end. Not a
single API condition.
Annotate each with TODO(e2e-replace-fixed-timeouts) and the rationale so
future readers do not "fix" them with a kubectl wait that does not work.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`kubectl proxy --port=21234 & sleep 0.5` guesses how long the proxy
needs to start listening. On a slow runner the curl that follows can
race the proxy and fail with "connection refused".
Replace the fixed `sleep 0.5` with `nc -z localhost 21234` polled until
the listener accepts a connection (10s cap). Same idiom already used in
hack/e2e-apps/bucket.bats for the seaweedfs-s3 port-forward.
Also save the proxy PID to a named variable so the trap kills the right
process even if a later background command lands first in $!.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The app bats files all share the pattern:
kubectl apply -f - <<EOF ... EOF
sleep 5
kubectl wait hr <name> --for=condition=ready
The fixed sleep guesses how long the cozystack-operator needs to translate
the app CR into a HelmRelease. On a noisy CI runner or after a cold
install that wait is sometimes too short, so the immediately-following
`kubectl wait hr ...` fails because the HR object does not exist yet
(`kubectl wait` errors out instantly on a missing resource).
Replace each `sleep 5` (and the one outlier `sleep 15` in foundationdb
that was paying for the same risk) with an event-driven backstop that
polls for the HR's existence and exits the moment it appears, capped at
60s. The downstream `kubectl wait --for=condition=ready` is unchanged,
preserving the original total budget.
Same shape as the LINSTOR fix in commit eb87413 on feat/e2e-optimization:
wait for the actual prerequisite (the HelmRelease the operator creates),
not on a downstream artefact whose existence depends on it.
Files: postgres, mariadb, kafka, mongodb, clickhouse, redis, qdrant,
harbor, openbao, external-dns (both tests), vminstance (both tests, with
the existence backstop hoisted before the downstream vmi-ip poll), and
foundationdb.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Prepare environment is pure infrastructure (Talos image download, sandbox
VMs boot, network setup). Failures here are mostly noisy-runner /
transient infra hiccups (image-download stalls, NIC negotiation, etc.) —
not cozystack code under test. Retry-on-failure is the right policy for
infra setup.
Install Cozystack and Run E2E tests remain single-attempt: they exercise
cozystack code, where retries hide real bugs.
Refines the previous two retry-removal commits.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A failed cluster bootstrap or platform install is almost always a real
bug (Talos boot, chart, operator) — not a flake. The 3x retry on these
steps was the direct cause of the 250-min outlier run (24931612182):
each retry compounds 25-30 min of work, and the failure mode persists
across attempts. Single attempt + the existing 'collect-report' step
on always() gives faster signal and a debug archive.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Across 5 sampled failure runs (30 PR runs total), 25/25 retry attempts
failed. The retry loop never recovered a flake — it only stretched
deterministic failures (e.g. kubernetes-previous: 11:37 + 7:49 + 10:16
= 29:42 wasted on one broken test). Replace with a single attempt plus
inline diagnostics (HR list + recent events) so the actual failure
surfaces immediately. Re-runs remain available via the standard
'gh run rerun' / empty-commit retry path.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous 5-min timeout on `until kubectl get deploy/linstor-controller`
fired during cold-install runs where the LINSTOR HR's dependency chain
(cert-manager → piraeus-operator-crds → piraeus-operator → linstor)
takes longer than 5 min to resolve, killing the whole post-install-prep
script via `set -eu`. With CI's 3x retry the failure was hidden; with the
retry removed it would surface every cold install.
Wait on the LINSTOR HR being Ready first (the actual prerequisite),
then keep the existence-poll as a near-instant backstop, then wait
for Available. Same final semantics, no fragile fixed-window timeout.
Surfaced by PR #2500 cozyreport diagnostics (post-install-prep.log).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Previously these three configuration blocks ran as separate @test cases
after the 15m platform HR wait, adding ~1.5 min sequentially. None of
them gates platform HR reconcile, so they can run as a background prep
that completes during the main wait. The helper script polls for its
own prerequisites and is awaited at the end of the Install Cozystack
test so failures still surface deterministically.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).
E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.
Production defaults are unchanged. The override is opt-in per install.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A 50MB tarball with 200 directories is opaque on first contact.
summary.txt at the root surfaces 'what is broken right now' — non-Ready
HRs, ImagePullBackOff pods, OOMKilled events, cert-manager state,
Flux Source health, storage binding, node pressure — so triage starts
with one file instead of a directory tree dive.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous report omitted everything needed to diagnose failures that
originate deeper than a single system package install — Flux controller
logs, the actual rendered Helm manifest from the storage secret, recent
events, cert-manager state, the cozystack-operator's own logs, and
sandbox host context (talosctl logs, dmesg, disk).
This makes failures self-diagnosable from cozyreport.tgz alone, which is
the only artifact that survives a failed E2E run.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Closes#2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.
Implements a defense-in-depth fix:
- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.
Tests:
- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.
All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).
Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.
### Release note
```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
* Chart resources now render conditionally based on etcd presence.
* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.
* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The cozystack.io/status-beacon: "true" annotation had no consumer in
the chart, no documented contract, and no convention defined for other
charts to follow. It would have become accidental precedent for
contributors copying the pattern without understanding it.
The ConfigMap itself is self-explanatory: the name <release>-awaiting-etcd,
data.status: "awaiting-etcd", and the human-readable message in
data.message all surface the same operator signal via kubectl get cm.
Drop the annotation; keep the ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Operators in air-gapped or rate-limited environments cannot reach
docker.io and the bundled busybox digest pin gives them no escape
hatch. Add an optional images.waitForKubeconfig chart value that, when
set, replaces the helper's image reference with any registry path
kubelet can pull. Empty value falls back to images/busybox.tag, so the
prior digest-pinned default is preserved.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adopt CNCF/Kubernetes label conventions for issues and PRs and add
automated labeling.
**Canonical label file**: `.github/labels.yml`. Synced into the
repository by `.github/workflows/labels.yaml` (EndBug/label-sync@v2) on
push to `main`, weekly cron, and manual dispatch. UI-only label edits
are overwritten — propose changes via PR to this file.
### Label namespaces
Following the [Kubernetes label
scheme](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md):
- `kind/*` — issue or PR type (bug, feature, documentation, support,
cleanup, regression, flake, failing-test, api-change, breaking-change)
- `priority/*` — urgency (critical-urgent, important-soon,
important-longterm, backlog)
- `triage/*` — review state (needs-triage, accepted, needs-information,
not-reproducible, duplicate, unresolved)
- `lifecycle/*` — issue/PR lifecycle (active, frozen, stale, rotten)
- `area/*` — subsystem; 15 seeded plus `area/uncategorized` fallback.
Extensible — propose a new area when no existing one fits.
- `do-not-merge/*` — PR merge blockers (work-in-progress, hold)
Cozystack-specific labels preserved: `epic`, `community`, `help wanted`,
`good first issue`, `quality-of-life`, `upstream-issue`, `backport`,
`backport-previous`, `release`, `automated`, `debug`, `sponsored`,
`lgtm`, `ok-to-test`, `security/*`, `size:*`.
### `area/*` set
15 areas seeded by activity in open issues and PRs: `area/ai`,
`area/api`, `area/build`, `area/ci`, `area/dashboard`, `area/database`,
`area/extra`, `area/kubernetes`, `area/monitoring`, `area/networking`,
`area/platform`, `area/release`, `area/storage`, `area/testing`,
`area/virtualization`. Plus `area/uncategorized` as the auto-labeler
fallback.
### Migration safety
Existing labels are renamed via `aliases:` in `labels.yml`. GitHub
preserves the label ID, so all currently tagged issues and PRs keep
their tags under the new name without losing references:
| Old | New |
|---|---|
| `bug` | `kind/bug` |
| `enhancement` | `kind/feature` |
| `documentation` | `kind/documentation` |
| `question` | `kind/support` |
| `frozen` | `lifecycle/frozen` |
| `stale` | `lifecycle/stale` |
| `duplicate` | `triage/duplicate` |
| `do-not-merge` | `do-not-merge/work-in-progress` |
| `do not merge` | `do-not-merge/work-in-progress` |
`delete-other-labels: false` on the initial rollout. Generic
GitHub-default labels (`wontfix`, `invalid`) are preserved untouched and
will be removed in a follow-up cleanup PR. EndBug processes aliases
sequentially, so the second of the two `do-not-merge*` aliases hits a
name collision and logs a warning — the legacy label survives that one
sync and is cleaned up in the same follow-up.
### PR auto-labeling
`.github/workflows/pr-labeler.yaml` parses each PR title on `opened`,
`edited`, `reopened`, and `synchronize` and applies labels additively
(never removes):
- **type → `kind/*`**: feat, fix, docs, chore, refactor (others get no
kind)
- **scope → `area/*`**: scope mapping covers all current cozystack
components (full table in `docs/agents/contributing.md`)
- **`!` after type or `BREAKING CHANGE:` footer**: applies
`kind/breaking-change`
- **`[Backport release-1.x]` prefix**: stripped before parsing;
`area/release` and `backport` labels added
- **Composite scope** (`feat(platform, system, apps): …`): each part
mapped independently
- **Bracket fallback** (`[scope] description`): maps `area/*` but cannot
infer `kind/*`
- **Unmapped scope or non-conventional title**: applies
`area/uncategorized` for human review
### Schema validation
`.github/workflows/labels.yaml` runs a `validate` job on every PR
touching `labels.yml` or its workflow. Asserts:
- description ≤ 100 chars (GitHub REST API limit)
- color is 6-char hex without leading `#`
- unique top-level names
- aliases do not collide with top-level names
Sync runs only on push to main, weekly cron, and manual dispatch; PR
runs validate-only.
### Hardcoded label/title references updated
- `.github/ISSUE_TEMPLATE/bug_report.md`: `labels: 'bug'` → `labels:
'kind/bug'`
- `.github/workflows/tags.yaml`:
- changelog PR labels `['documentation', 'automated']` →
`['kind/documentation', 'automated']`
- release PR title `Release v${version}` → `chore(release): cut
v${version}` (so the auto-labeler applies `kind/cleanup` +
`area/release`)
- changelog PR title `docs: add changelog for v${version}` →
`docs(release): add changelog for v${version}` (so the auto-labeler
applies `kind/documentation` + `area/release`)
### Documentation
- `AGENTS.md`: Activation entry pointing agents to `labels.yml` and the
PR title auto-labeling rules. States explicitly that `area/*` accuracy
outweighs reuse — propose a new area when none fits, do not shoehorn
into a wrong one.
- `docs/agents/contributing.md`: PR Title Auto-Labeling section with
type→kind and scope→area tables.
### Out of scope (follow-up PRs)
- Removal of redundant labels (`wontfix`, `invalid`, plus the surviving
legacy `do not merge` if alias-rename collision keeps it)
- Org-wide sync from `cozystack/.github/labels.yml`
- Dosu bot configuration update for `lifecycle/stale` (requires
dashboard access)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automated PR labeling from Conventional Commits (type → kind/*, scope
→ area/*), plus a comprehensive namespaced label taxonomy.
* **Chores**
* Workflows to validate, sync, and auto-apply labels (including
scheduled/manual runs and validation checks).
* Added repository-wide label configuration and normalized bug label
metadata to namespaced form.
* Updated release PR titling/labeling conventions.
* **Documentation**
* Contributor and agent guidance on PR title conventions, label
mappings, and triage procedures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:160,173:
emit core.warning when a Conventional Commits type has no kind/*
mapping or a scope has no area/* mapping. Without the warning, typos
(e.g., "hotfix" instead of "fix") and recurring new scopes silently
fall through to area/uncategorized, masking that the mapping has
drifted.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:155:
Conventional Commits 1.0 spec item 16 treats BREAKING CHANGE: and
BREAKING-CHANGE: as synonymous footers. The hyphen form was silently
ignored before, so PRs that use it would miss kind/breaking-change.
https://www.conventionalcommits.org/en/v1.0.0/#specification
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:129:
defensive (pr.labels || []) avoids TypeError if the webhook payload
arrives without the labels field on edge cases like stripped edited
events.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Add an opt-in `upgradeCRDs` field to `ComponentInstall` in
`PackageSource`. The field maps directly to
`HelmRelease.Spec.Upgrade.CRDs` so a component author can declare how
Flux should handle CRDs from the chart's `crds/` directory when the
release is upgraded.
The helm-controller default on upgrade is `Skip`, which means CRDs added
by a chart bump never reach clusters that already have the release
installed — they must be applied manually with `kubectl apply --filename
charts/.../crds/`. This surfaces on every upgrade of an operator whose
CRD set expands between versions (etcd-operator, cnpg, kubevirt, kamaji,
etc.).
Setting `upgradeCRDs: CreateReplace` lets Flux apply new CRDs
declaratively with the chart.
Values are restricted to `Skip`, `Create`, `CreateReplace` via a
kubebuilder enum marker. Empty / unset preserves the existing
helm-controller default, so every current `PackageSource` keeps working
unchanged.
Migration is out of scope here — follow-ups will opt individual packages
in case-by-case.
### Relation to existing CRD management approach
The project convention (per #377) is to extract CRDs into a dedicated
Helm chart that reconciles ahead of the operator chart. This PR does not
replace that pattern — it complements it for charts that keep CRDs
inline under `charts/<name>/crds/` where extraction isn't practical
(vendored upstream charts with tightly coupled CRDs). Packages that
already split CRDs out can leave `upgradeCRDs` unset and keep using
their existing separate chart.
### Release note
```release-note
feat(operator): add opt-in `upgradeCRDs` field to `PackageSource` component `install` block to control how CRDs from the chart's `crds/` directory are applied on HelmRelease upgrades (`Skip` by default; use `CreateReplace` for operators whose CRD set expands between versions).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable CRD upgrade policy for component installs: Skip, Create,
CreateReplace — controls CRD handling during package upgrades and
preserves controller default when unset.
* **Documentation**
* Guidance on CRD upgrade semantics, advice to use CreateReplace for
specific operators, and warning about potential data-loss risks;
clarified contributor scope examples and PR template guidance.
* **Tests**
* Added tests validating CRD policy parsing and presence of the CRD
policy enum in the published schema.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Match the wording adopted in docs/agents/contributing.md so that human
contributors and AI agents see the same guidance in both places.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Scopes section was read as an exhaustive enumeration, which led to
review feedback flagging any scope outside the list as invalid. The
intent has always been that contributors pick the most specific scope
for the change and extend the list when a genuinely new area appears.
Reword the section accordingly and add operator as an example scope.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe when to set upgradeCRDs: CreateReplace (operators that evolve
their CRD set additively between versions) and the data-loss risk of
enabling it on operators that drop fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add an opt-in UpgradeCRDs field to ComponentInstall that maps to
HelmRelease.Spec.Upgrade.CRDs, allowing a PackageSource component to
declare how Flux should handle CRDs from the chart's crds/ directory
on upgrade.
The helm-controller default on upgrade is Skip, which means new CRDs
added between chart versions never reach existing clusters and must be
applied manually. Setting upgradeCRDs: CreateReplace makes Flux apply
new CRDs declaratively with the chart.
Allowed values are restricted to Skip, Create, CreateReplace via a
kubebuilder enum marker. Empty / unset preserves the existing Flux
default, so all existing PackageSource resources keep working.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:73:
add space before closing pipe in the type to kind mapping table for
consistency with other rows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on AGENTS.md:33:
expand bare labels.yml and pr-labeler.yaml to .github/labels.yml and
.github/workflows/pr-labeler.yaml for consistency with surrounding refs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:88:
… replaced with ... for compatibility across editors and tools.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on .github/labels.yml:242:
all hex color values use lowercase characters for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
- Increase kamaji controller memory limit from 500Mi to 512Mi
- Increase kamaji controller memory request from 100Mi to 256Mi
- Add startup probe with 60-second timeout (12 attempts × 5s periods)
- Increase readiness/liveness probe initialDelaySeconds from 5s/15s to
30s
## Problem
The kamaji controller was experiencing frequent CrashLoopBackOff due to
OOMKilled errors. Analysis showed:
- Container was being killed with exit code 137 (OOMKilled) after ~20-25
seconds of runtime
- Memory limit of 500Mi was insufficient for controller initialization
- Readiness probe was failing because it started too early (5s
initialDelay), before the controller finished leader election (~17s)
## Solution
**Memory increase:**
- Limit: 500Mi → 512Mi (based on production testing)
- Request: 100Mi → 256Mi (ensures adequate reservation)
**Startup probe:**
- Added to give controller up to 60 seconds to initialize without being
killed by liveness probe
- 12 attempts × 5s period = 60s maximum startup time
**Probe delays:**
- ReadinessProbe: 5s → 30s initialDelay (controller needs ~17s to
acquire leader lease)
- LivenessProbe: 15s → 30s initialDelay (aligned with readiness)
## Testing
Verified in production cluster:
- Controller runs stable with 0 restarts
- No more OOMKilled events
- Successfully creates kubeconfig secrets for tenant clusters
## Related Issues
Fixes tenant cluster components stuck in ContainerCreating due to
missing kubeconfig secrets (caused by crashing kamaji controller).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Introduced automated health checks using HTTP-based probes to monitor
service status during startup, continuous operation, and readiness to
handle traffic.
* Adjusted container memory resource allocation for enhanced stability
and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps the vendored `robotlb` chart to the latest upstream build.
Chart version remains `0.1.3`; the bundled `appVersion` moves from
`0.0.5` to `0.0.6`.
The new `robotlb` release adds RBAC permissions for
`discovery.k8s.io/endpointslices` (`get`, `list`, `watch`), which are
required to manage services backed by `EndpointSlice` — notably
KubeVirt-exposed workloads that do not publish classic `Endpoints`.
Notes:
- Upstream also replaced `replicas: {{ .Values.replicas }}` with a
hardcoded `replicas: 1` in `templates/deployment.yaml`. The
effective replica count is unchanged (we already set `1`), but the
value is no longer overridable via chart values. A minor cosmetic
reformat was applied to `templates/role.yaml`.
Closes#2256
### Release note
```release-note
chore(hetzner-robotlb): update robotlb to 0.0.6 — adds RBAC for EndpointSlices so services backed by EndpointSlice (e.g. KubeVirt) are supported.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Extended service account permissions to access Kubernetes endpoint
slices from the discovery API.
* **Bug Fixes**
* Deployment replica configuration now fixed to single instance.
* **Style**
* Improved YAML formatting in role template declarations.
* **Chores**
* Updated application version metadata to 0.0.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Refreshes the vendored Cilium chart in `packages/system/cilium` from
v1.19.1 to v1.19.3 via `make update`. Chart templates, values, CRDs and
the Cilium image reference are regenerated from upstream.
### Motivation
- **v1.19.2** ships a critical fix for cert-manager HTTP-01 Gateway API
challenges on hostnames that have both HTTP and HTTPS listeners
([cilium#44492](https://github.com/cilium/cilium/pull/44492), backport
[#44517](https://github.com/cilium/cilium/pull/44517)). Without this
fix, cert-manager cannot issue certificates via Gateway API when a
redirect HTTP listener and a TLS HTTPS listener share a hostname.
- **v1.19.3** is the latest stable patch release in the v1.19.x line (15
Apr 2026).
- This bump is a prerequisite for upcoming Gateway API work tracked
separately.
### Upstream changes pulled in
- Cilium Envoy bootstrap config, operator clusterrole, config template,
`values.schema.json` and the cilium-agent DaemonSet refreshed from
upstream.
- New `templates/ztunnel/` directory (DaemonSet, Secret, ServiceAccount)
added by upstream — not enabled by default in Cozystack values.
### Release note
```release-note
chore(cilium): bump to v1.19.3 (cert-manager HTTP-01 fix via cilium#44492)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added ztunnel encryption support with configurable deployment settings
* Added ConfigDriftDetection for monitoring ConfigMap changes
* Added endpoint policy update timeout configuration
* Added load balancer service topology support
* Extended Envoy circuit breaker configuration with connection and
request limits
* **Updates**
* Upgraded Cilium to v1.19.3
* Updated container images (Envoy, certgen, Hubble relay, clustermesh)
* Enhanced Cilium operator RBAC capabilities for managing ServiceImport
finalizers
* **Removals**
* Removed BIG TCP tunnel configuration option
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds explicit guidance to `docs/agents/contributing.md` about running
`make generate` in any touched package before committing. Pre-commit CI
runs `make generate` in every package and fails with exit code 123 on
any uncommitted generator output (regenerated `README.md`, reordered
`values.schema.json`, refreshed
`packages/system/<name>-rd/cozyrds/<name>.yaml`).
Recent PRs have tripped on this during review cycles. Documenting it in
the contributing checklist saves a round-trip.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Require regenerating and committing generated artifacts when
designated source files are edited.
* Added a section listing generated artifacts per package and concrete
regeneration/staging steps.
* Documented CI enforcement that detects unstaged generator output and
blocks PRs.
* Added guidance for rerunning generation after amended commits, plus
updated commit-scope guidance (now “not exhaustive”) and included
“agents” in allowed scopes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on docs/agents/contributing.md:11:
Scope linters kept flagging valid scopes like 'agents' as unknown because
the list read as exhaustive. Annotate it as examples (not exhaustive) and
add 'agents' to the Other group so both humans and review bots stop
tripping on scopes that are already in regular use across the repo
history.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:30:
Reword "likely to need regenerated" (regional construction) to
"likely needs to be regenerated" for standard technical prose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
