Some workloads (OVN raft, LINSTOR controller) fail when replicas start
at different times due to image-pull stagger across nodes. Add a
DaemonSet-based pre-pull step that runs before helm install, ensuring
all nodes have the images cached so every replica starts within
milliseconds of each other.
The script accepts image refs on stdin and creates one container per
image (parallel pulls, total time = max of any single image rather than
sum). The bats test sources images directly from the rendered charts
via yq, walking only PodSpec-shaped objects so version bumps stay in
sync automatically without a separate hardcoded list.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Some workloads (OVN raft, LINSTOR controller) fail when replicas start
at different times due to image-pull stagger across nodes. Add a
DaemonSet-based pre-pull step that runs before helm install, ensuring
all nodes have the images cached so every replica starts within
milliseconds of each other.
Covers kube-ovn (confirmed OVN raft race), piraeus-server, and
linstor-csi. Comments in hack/e2e-prepull-images.sh point to the
source chart values so versions stay in sync.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`kubectl wait <kind> <name>` errors immediately with NotFound if the
resource doesn't exist yet — even with --for=condition or --for=jsonpath.
The redis test in CI run 25043350705 failed in 1 second for exactly this
reason: the redis-failover operator hadn't created the PVC by the time
the test waited for it. Previously the 3x retry on `Run E2E tests`
masked this race; with retry dropped, every such call is a flake risk.
Add a small `until kubectl get` existence backstop before each
kubectl wait, matching the pattern already established for HRs in
commit 66888c91.
33 backstops across 12 files:
bucket.bats — bucketclaims, bucketaccesses x2
clickhouse.bats — statefulset 0-0 (0-1 already covered)
harbor.bats — deploy x3, bucketclaims, bucketaccesses
kafka.bats — kafkas
mariadb.bats — statefulset, deployment
mongodb.bats — statefulset
openbao.bats — sts, pvc
postgres.bats — job.batch
qdrant.bats — sts, pvc
redis.bats — pvc, deploy, sts (the trigger)
vminstance.bats — dv, pvc, vm (with 120s for KubeVirt latency)
e2e-install-cozystack.bats — apiservices, sts/etcd, vmalert,
vmalertmanager, vlclusters, vmcluster,
clusters.postgresql.cnpg.io,
deploy/grafana-deployment, namespace
Same pattern, same 60s default timeout for the existence wait (120s for
nested-virt resources). Once the resource exists, the original wait
timeout takes over.
Run-kubernetes.sh has the same race shape on several waits (nfs, kamaji,
machinedeployment, etc.) — out of scope here; flagged for follow-up.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart fix (3a87485c, be0a6687) removes the chart-defined Namespace
without replacing it. On the Talos E2E cluster the apiserver enforces
PodSecurity baseline-by-default on bare namespaces, so the cozystack-
operator pod (hostNetwork=true) is rejected if cozy-system is created
without enforce=privileged.
Apply the namespace from the bats with the labels the operator depends
on, then run helm install without --create-namespace. Mirrors the
documented production install procedure for the new chart shape.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two reliability fixes in `hack/e2e-apps/vminstance.bats`:
1. Delete-recreate race. Both @test blocks deleted the prior VMInstance/VMDisk
with `kubectl ... --timeout=2m || true`. The `|| true` silently swallowed
timeout errors, letting the test continue with the resource still in
finalizer-drain. The next `kubectl apply` then no-ops with
"Detected changes to resource ... which is currently being deleted",
producing a NotFound on the downstream HR wait. Bumped the delete
timeout to 3m and removed `|| true` so true delete failures surface
loudly. End-of-test cleanup deletes also got the explicit timeout.
2. VM IP and VM ready timeouts. The 20s timeout for the VMI to acquire an
IP was unrealistic for nested KubeVirt under runner load — virt-launcher
+ libvirt + cloud-init DHCP routinely takes 30-60s. Bumped to 120s with
a 2s poll interval (60 polls instead of 4). VM ready timeout bumped from
20s to 60s for the same reason.
Both surfaced as recurring first-attempt failures in the past 30 successful
PR runs (7/17 analysable runs in each case) — masked by the 3x retry on
`Run E2E tests`.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Now that the chart fix (this same branch) bootstraps cozy-system via
--create-namespace + a pre-install label hook, the bats no longer needs
to pre-apply a hand-crafted helm-adoptable namespace. Restore plain
--create-namespace and drop the kubectl apply block.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozy-installer chart declares Namespace cozy-system itself, which
makes helm cold-install paradoxical:
- WITH --create-namespace: helm pre-creates the ns without helm
annotations; the chart's own Namespace apply then fails with
'already exists'.
- WITHOUT --create-namespace: helm fails before any apply because it
can't write its release-secret to a non-existent ns.
The 3x retry was masking both directions — second attempt saw a partial
release and took the upgrade path, which patch-merges instead of
strict-create. Reproducible on every cold install across PRs.
Pre-create cozy-system with meta.helm.sh/release-name and
app.kubernetes.io/managed-by=Helm so helm adopts it cleanly during
install. Labels mirror the chart's template (PSA enforce=privileged etc.)
so behaviour matches what the chart would have done.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozy-installer chart declares Namespace cozy-system itself (with
helm.sh/resource-policy: keep). Combining that with --create-namespace
makes Helm v3 pre-create the namespace via plain kubectl-create (without
helm annotations); the subsequent chart apply then fails with
'namespaces cozy-system already exists'.
The 3x retry on Install Cozystack was hiding this. First attempt failed,
second saw 'release exists' and treated it as upgrade. Reproducible
across PRs (PR #2507 E2E hit the same first-attempt failure today and
recovered on retry). Surfaced cleanly after dropping the retry on this
step.
Drop --create-namespace; let the chart manage its own namespace.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three poll/sleep sites in the install bats and the kubernetes runner are
NOT replaceable by `kubectl wait --for=condition=...` because the signal
they observe is not a Kubernetes API condition:
- e2e-install-cozystack.bats:55-56 - 5s pad lets late-arriving HRs join
the awk-snapshot the parallel `kubectl wait` runs against. There is no
k8s condition for "all expected platform HRs have been emitted" short
of hard-coding the list.
- e2e-install-cozystack.bats:75 - LINSTOR node membership is reported by
the linstor binary running inside the controller pod (kubectl exec),
not a CRD status, so kubectl wait cannot subscribe to it.
- e2e-apps/run-kubernetes.sh:231-238 - validates the external HTTP path
through MetalLB -> tenant ingress -> backend pod end-to-end. Not a
single API condition.
Annotate each with TODO(e2e-replace-fixed-timeouts) and the rationale so
future readers do not "fix" them with a kubectl wait that does not work.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`kubectl proxy --port=21234 & sleep 0.5` guesses how long the proxy
needs to start listening. On a slow runner the curl that follows can
race the proxy and fail with "connection refused".
Replace the fixed `sleep 0.5` with `nc -z localhost 21234` polled until
the listener accepts a connection (10s cap). Same idiom already used in
hack/e2e-apps/bucket.bats for the seaweedfs-s3 port-forward.
Also save the proxy PID to a named variable so the trap kills the right
process even if a later background command lands first in $!.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The app bats files all share the pattern:
kubectl apply -f - <<EOF ... EOF
sleep 5
kubectl wait hr <name> --for=condition=ready
The fixed sleep guesses how long the cozystack-operator needs to translate
the app CR into a HelmRelease. On a noisy CI runner or after a cold
install that wait is sometimes too short, so the immediately-following
`kubectl wait hr ...` fails because the HR object does not exist yet
(`kubectl wait` errors out instantly on a missing resource).
Replace each `sleep 5` (and the one outlier `sleep 15` in foundationdb
that was paying for the same risk) with an event-driven backstop that
polls for the HR's existence and exits the moment it appears, capped at
60s. The downstream `kubectl wait --for=condition=ready` is unchanged,
preserving the original total budget.
Same shape as the LINSTOR fix in commit eb87413 on feat/e2e-optimization:
wait for the actual prerequisite (the HelmRelease the operator creates),
not on a downstream artefact whose existence depends on it.
Files: postgres, mariadb, kafka, mongodb, clickhouse, redis, qdrant,
harbor, openbao, external-dns (both tests), vminstance (both tests, with
the existence backstop hoisted before the downstream vmi-ip poll), and
foundationdb.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous 5-min timeout on `until kubectl get deploy/linstor-controller`
fired during cold-install runs where the LINSTOR HR's dependency chain
(cert-manager → piraeus-operator-crds → piraeus-operator → linstor)
takes longer than 5 min to resolve, killing the whole post-install-prep
script via `set -eu`. With CI's 3x retry the failure was hidden; with the
retry removed it would surface every cold install.
Wait on the LINSTOR HR being Ready first (the actual prerequisite),
then keep the existence-poll as a near-instant backstop, then wait
for Available. Same final semantics, no fragile fixed-window timeout.
Surfaced by PR #2500 cozyreport diagnostics (post-install-prep.log).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Previously these three configuration blocks ran as separate @test cases
after the 15m platform HR wait, adding ~1.5 min sequentially. None of
them gates platform HR reconcile, so they can run as a background prep
that completes during the main wait. The helper script polls for its
own prerequisites and is awaited at the end of the Install Cozystack
test so failures still surface deterministically.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).
E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.
Production defaults are unchanged. The override is opt-in per install.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A 50MB tarball with 200 directories is opaque on first contact.
summary.txt at the root surfaces 'what is broken right now' — non-Ready
HRs, ImagePullBackOff pods, OOMKilled events, cert-manager state,
Flux Source health, storage binding, node pressure — so triage starts
with one file instead of a directory tree dive.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous report omitted everything needed to diagnose failures that
originate deeper than a single system package install — Flux controller
logs, the actual rendered Helm manifest from the storage secret, recent
events, cert-manager state, the cozystack-operator's own logs, and
sandbox host context (talosctl logs, dmesg, disk).
This makes failures self-diagnosable from cozyreport.tgz alone, which is
the only artifact that survives a failed E2E run.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The test body asserts .status.history[].status extraction, but the
test name still referenced the old installFailures counter (leftover
from when the guard used that field before switching to status.history
to avoid ClearFailures zeroing the counters on successful reconcile).
Address review feedback from coderabbitai on
hack/remediation-guard.bats:84: rename so grep for what the test
actually pins matches the test name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Log .status.history regardless of content so a silently empty result
(Flux field rename) shows up in CI logs, and treat empty history on
a Ready HelmRelease as a distinct failure. A Ready HR by definition
has at least one snapshot; anything else is a shape-drift signal.
- Replace the unquoted heredoc in remediation-guard.sh with a printf |
grep pipeline. printf %s treats statuses as literal payload (no $
expansion surprises for future callers), grep --quiet --extended-regexp
returns exit status the caller can forward directly.
- Share the etcd-absent values file between both invariant tests
(packages/apps/kubernetes/tests/values-ci-no-etcd.yaml) instead of
duplicating the --set block.
- Fix typo "override applied" -> "override is applied" in the
Kubernetes ApplicationDefinition.
- Add a coupling comment in the ApplicationDefinition annotation that
points at the wait-for-kubeconfig init deadline in _helpers.tpl, so
a future operator raising the HR timeout updates the init deadline
too.
- Clarify the per-annotation timeout comment in rest.go so it stops
implying the feature is Kubernetes-only (it is not - only today's
one user is).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
17 child HelmReleases (cilium, coredns, csi, cert-manager,
metrics-server, ...) referenced *-admin-kubeconfig via
kubeConfig.secretRef and rendered even when _namespace.etcd was empty.
On an etcd-less tenant each one sat in NotReady forever because the
admin-kubeconfig Secret only exists after a KamajiControlPlane
reconciles, and KamajiControlPlane now only renders when etcd is set.
The outcome contradicted the "beacon only" contract claimed in the
soft-skip commit.
Extend the existing addon guards to also require _namespace.etcd, and
wrap the four unconditional HelmReleases (csi, metrics-server,
prometheus-operator-crds, volumesnapshot-crd) plus the always-on
cilium/coredns HR resources in the same gate. Add an invariant bats
test that renders the whole chart with etcd empty and asserts zero
HelmReleases reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Flux helm-controller's ClearFailures() zeroes installFailures and
upgradeFailures on every successful reconciliation (see the upstream
HelmReleaseStatus method). The previous guard ran after the HelmRelease
was Ready, at which point the counters were always 0 - the assertion
was vacuous and would have passed against a reverted fix.
Switch to .status.history, which retains per-revision release
Snapshots that survive a subsequent successful reconciliation. A
remediation cycle leaves behind a Snapshot with status=uninstalled
(the install-remediation code path) or status=failed (Helm release
failure that remediation then uninstalled). Either one signals the
race actually fired.
Rewrite the bats unit tests to cover: empty history, deployed-only,
deployed+superseded (happy path - not detected), single failed,
single uninstalled, uninstalled-then-deployed, and deployed-then-failed
(all detected). The pinned-shape test feeds a realistic HR status
snippet through yq the same way run-kubernetes.sh does via kubectl
-o jsonpath.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The soft-skip wrap in cluster.yaml only silenced Cluster and
KamajiControlPlane rendering when _namespace.etcd is empty. The three
CP-side Deployments (cluster-autoscaler, kccm, kcsi-controller) still
rendered, their wait-for-kubeconfig init containers CrashLoopBackOff'd
forever (no KamajiControlPlane = no admin-kubeconfig Secret),
HelmRelease hit its 15m wait timeout and triggered the very install
remediation cycle the rest of this PR prevents. Self-contradiction.
Wrap each of the three Deployment templates in
{{- if .Values._namespace.etcd }}...{{- end }} so they render only
when there is a DataStore to back them. Add an invariant bats test
that renders the whole chart with etcd empty and asserts zero
Deployments reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
run-kubernetes.sh extracts .status.installFailures and
.status.upgradeFailures via kubectl -o jsonpath. If a future flux
release renames the counters, kubectl returns empty, the guard reports
no cycle, and e2e silently misses real remediation loops.
Add a bats unit test that feeds a pinned HelmRelease v2 status snippet
through the same jsonpath and asserts the extraction still yields the
expected values. Also leave a pointer comment in run-kubernetes.sh so a
future flux bump surfaces the version coupling in review.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-template unittests in packages/apps/kubernetes/tests/ assert
that cluster-autoscaler, kccm, and the csi controller each mount the
admin-kubeconfig Secret optional and carry the wait-for-kubeconfig
init. That locks in today's three Deployments by name - a fourth
Deployment that mounts the same Secret but forgets the guard would
slip past them.
Add a bats-unit test that renders the entire chart, enumerates every
Deployment whose spec mounts a Secret ending in -admin-kubeconfig, and
asserts optional:true plus wait-for-kubeconfig init on all of them.
Verified by temporarily removing optional:true from csi/deploy.yaml:
the test correctly flagged invariant-kcsi-controller as an offender.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Shell && and || have equal precedence and left-to-right associativity,
so the previous guard parsed as (((A && B) || C) && D) and silently
passed on the canonical failure mode: install_failures=1 with an empty
upgrade_failures.
Extract the check into helmrelease_has_remediation_cycle() in a
dedicated helper sourced from run-kubernetes.sh, and add unit tests
under hack/remediation-guard.bats that pin the expected behavior for
every combination of empty, zero, and positive counter values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Before cleanup, inspect the parent HelmRelease installFailures and
upgradeFailures counters. A non-zero value means flux helm-controller
hit its wait timeout, ran install/upgrade remediation (uninstall),
and re-installed - the exact race condition this PR closes. Fail the
bats test in that case so the signal surfaces in CI instead of being
masked by a green retry.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
The aggregated API previously accepted tenant names containing hyphens
(e.g. `foo-bar`) because `ValidateApplicationName` delegated entirely to
`IsDNS1035Label`, which permits them. The tenant Helm chart's
`tenant.name` helper then rejected the release at template time — so
users saw a successful `kubectl apply` followed by a confusing
Flux/HelmRelease reconciliation error.
This PR extends `ValidateApplicationName` with a `kindName` parameter
and enforces an alphanumeric-only rule (`^[a-z][a-z0-9]*$`) for the
Tenant kind. A `TenantKind` constant centralizes the kind string, and
`REST.validateNameFormat` wraps the check symmetrically with the
existing `validateNameLength`. The tenant chart README is updated to
match the already-correct website documentation.
Test coverage includes:
- Unit tests for the validation package (format + error message
contract)
- REST wrapper and Update→Create fall-through path tests with a fake
client
- E2E BATS regression test with explicit exit-code checks and cleanup
Fixes#2375
### Release note
```release-note
[api] Tenant names containing dashes are now rejected at API level during creation, with a specific error message, instead of being silently accepted and failing later during Helm reconciliation.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant names now require lowercase letters and digits only, must start
with a letter; dashes are rejected with a tenant-specific error.
* **Documentation**
* Tenant naming guidelines updated to reflect the stricter
alphanumeric-only requirement.
* **Tests**
* New unit and end-to-end tests added to verify tenant naming
enforcement and tenant-specific error messaging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The api/apps/v1alpha1/* subtags share the same commit as the release
tags. git describe --exact-match picks the first match alphabetically,
returning api/apps/v1alpha1/vX.Y.Z instead of vX.Y.Z, which produces
invalid Docker image tags.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Four follow-ups from review round 4:
1. BLOCKER: the Update(forceAllowCreate=true) path delegates to Create() when the object does not yet exist (rest.go:452) — the typical kubectl apply upsert flow. Add TestUpdate_ForceAllowCreate_RejectsTenantDashName using a fake client so a future refactor of that delegation cannot silently bypass the tenant name check that r.validateNameFormat alone cannot catch.
2. BLOCKER: the e2e BATS test used || true inside the command substitution, which swallowed the kubectl exit code. Rework the test to capture exit code and stdout+stderr explicitly, then assert the exit code is non-zero before asserting on the error message. This distinguishes validation-success (kubectl exit 0 — regression) from environmental failures (exit non-zero but wrong message) from the happy path.
3. Extract TenantKind = "Tenant" as a named constant in the validation package with a comment pointing at the upstream ApplicationDefinition source of truth, and switch the kindName check to use it.
4. Add a clarifying comment on TestValidateApplicationName_TenantLengthFallthrough that it pins an architectural layering decision and is not a user-facing requirement, so a future promotion of tenant length into tenant-specific wording is a legitimate change rather than a test regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three follow-ups from review:
1. Add defensive cleanup of the foo-bar tenant before and after the e2e regression test. If a prior run left the object in the cluster (e.g. a transient validation regression), the test no longer inherits that state; and if the test itself trips a regression in the future, the cluster is left clean for subsequent tests.
2. Switch kubectl --validate=false to --validate=ignore. The bool form was deprecated in kubectl 1.25 and only kept as an alias; --validate=ignore is the modern, stable spelling.
3. Drop the brittle Contains("63") assertion in TestValidateApplicationName_TenantLengthFallthrough. It pinned the test to the exact DNS-1035 error text from k8s.io/apimachinery, which could break on unrelated upstream wording changes. The surviving assertions still cover the intent: an error exists and it is not the tenant-specific message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Follow-up from review round 2:
The BATS regression test now pins a namespace precondition, disables kubectl client-side validation with --validate=false so the request is guaranteed to reach the server-side name check, and documents the intent of each assertion. Previously a client-side schema rejection could have produced a false negative on the tenant-specific error grep, and a missing tenant-root namespace could have produced an unrelated failure that was hard to diagnose.
Also pin the edge case where a tenant name consists of valid characters but exceeds the DNS-1035 63-char label limit. The resulting error is intentionally the generic DNS-1035 message because length is not a tenant-specific constraint — the package-level function is not responsible for the stricter Helm-release-prefix length budget that REST.validateNameLength enforces.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two follow-ups from review:
1. The original tenantNameRegex ^[a-z0-9]+$ let leading-digit names like 123foo slip past the tenant check and receive a generic DNS-1035 error, which defeated the goal of surfacing a tenant-specific message. Require a leading lowercase letter so every tenant-invalid name returns the tenant-contract error.
2. Add a BATS regression test in hack/e2e-install-cozystack.bats that exercises the aggregated API end-to-end. Unit tests construct REST{kindName: "Tenant"} by hand, so a future change to ApplicationDefinition kind registration could break real behavior without breaking the unit tests.
Also pin the error-message contract with a new TestValidateApplicationName_TenantErrorMessage that asserts every tenant-invalid input returns a message containing 'tenant names must' (not the generic DNS-1035 text).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The disk_usage helper shells out to 'du -sh $path' for reporting in
the warning message. On the exact directories this script is meant
to warn about (a /var/lib/containerd accumulating millions of files
from months of unpruned builds), traversing the tree with du can
take minutes and stall the preflight indefinitely.
Wrap the call in 'timeout 5s' so the helper returns quickly even on
a pathologically slow filesystem. If the timeout binary is absent
(e.g. minimal busybox userland), the pipeline still exits 0 via the
existing '|| true' guard and usage stays empty — the warning still
prints, just without the size detail — so the change is backward
compatible.
Addresses an inline review comment from gemini-code-assist on the
open PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address three review findings on the bats suite:
- The 'standalone containerd service active' and 'standalone docker
service active' tests previously asserted 'systemctl disable --now
<service>' but did not require the 'sudo ' prefix, so the two-
service test was the only guard against a silent drop of the
prefix. Extend both single-service tests to require the prefix
exactly as the 'both services active' test does.
- The 'both services active' test now captures the exit code
explicitly with 'bash ... || status=$?' and asserts '$status -eq
0'. The script contract ('always exits 0') was previously enforced
implicitly via 'set -e', which produces a generic test failure on
a regression instead of a contract-specific error. The explicit
check locks in the contract and makes regressions readable.
- New test: 'docker socket paths with glob chars do not expand' —
sets COZYSTACK_DOCKER_SOCKET_PATHS to a literal glob against real
directories and asserts no docker warning fires. Locks in the
array-based parsing of the path list.
- Extend the file header with a 'title syntax constraints' section
documenting how cozytest.sh's awk parser treats double quotes and
non-alphanumeric characters in @test titles, so future
contributors do not stumble on the parser's quirks.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
'set -euo pipefail' does not include 'set -f', so the previous
'for sock in $DOCKER_SOCKET_PATHS' loop relied on both word splitting
AND unintended glob expansion. If the variable ever contained a
literal '*' or '?' — for example a test setting
COZYSTACK_DOCKER_SOCKET_PATHS='/run/docker-*.sock' against a tree
that happens to have real files matching the pattern — the loop
would iterate over directory entries instead of the intended socket
paths, producing false-positive docker warnings.
Replace the implicit split+glob loop with 'read -ra' into a bash
array and iterate 'for sock in "${_socks[@]}"'. This keeps the
space-separated input format, disables glob expansion, and removes
the fragile word splitting that shellcheck otherwise has to be
talked out of.
Covered by a new regression test ('docker socket paths with glob
chars do not expand') that sets the env var to a literal '*' pattern
pointing at real directories; without this fix the test fails with
an unexpected docker.service warning.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
DOCKER_SOCKET_PATHS is a space separated list of socket paths (its
default is "/run/docker.sock /var/run/docker.sock") that the loop
iterates by word splitting. Socket paths never contain whitespace on
Linux hosts, so this is the documented and correct idiom — not a
shellcheck oversight. Add an inline comment so a future maintainer (or
a linter that has not seen the comment) does not break the loop by
adding quotes around the expansion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two review findings:
1. The socket fallback tests previously branched on 'command -v
python3' and fell through to 'return 0' when it was missing.
cozytest.sh has no SKIP concept — 'return 0' is indistinguishable
from a real pass, so on a runner without python3 the socket
fallback paths were silently unverified. Since the script uses
'[ -e "$sock" ]' rather than '[ -S ... ]', a regular file is
sufficient to exercise the detection path. Replace the python3
unix-socket creation with 'touch "$SOCK"' so the tests run
unconditionally on every runner.
2. Extend the 'both services active' HINT assertion to require the
'sudo ' prefix on the 'systemctl disable --now' line. Without sudo
the operator would be instructed to run a privileged command as a
non-root user and hit a silent failure; the prefix is as important
as the verb itself and must be locked in by a test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expand the bats suite to address a second review pass:
- Assert that a containerd only warning produces a HINT that names
containerd.service and NOT docker.service, and mirror the check in
the docker only test. When both fire, assert the HINT lists both
services in a single systemctl disable invocation.
- New test: 'clean host without systemctl exits silently' — exercises
the COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 path when no standalone
sockets exist. Previously the tests only covered the systemd enabled
clean host, leaving the non systemd clean path unverified.
- New test: 'docker service plus socket still emits exactly one
warning' — mirrors the existing containerd service+socket test and
locks in the gated check in check_docker.
- Replace silent 'return 0' on missing python3 with a visible
'# SKIP: python3 unavailable' message on stderr so CI logs make it
obvious which tests were skipped on a given runner.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When only containerd.service was active, the HINT block still advised
the operator to disable both containerd.service and docker.service,
which is misleading and potentially dangerous on hosts where docker is
legitimately in use. Track per service warnings (CONTAINERD_WARN and
DOCKER_WARN) and build the HINT systemctl disable argument from the
services that actually fired.
Also guard the ANSI color escapes behind an 'is stderr a TTY' check so
log files (CI and reviewer captures) do not accumulate raw escape
sequences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback on the preflight test suite:
- Add a trap 'rm -rf $STUB_DIR' EXIT in every test immediately after
mktemp -d, so temp dirs are cleaned up even when an assertion fails
and terminates the test early under set -e.
- Assert the HINT block and 'systemctl disable --now' line in the
'both services active' test so a future silent removal or typo in
the HINT output is caught by CI.
- New test: 'docker socket fallback fires when systemctl is
unavailable' — mirrors the existing containerd socket fallback test
and exercises the same code path in check_docker.
- New test: 'containerd service plus socket still emits exactly one
warning' — documents and locks in the intent of the symmetric gated
check in check_containerd. Uses grep -c to assert the warning is not
double printed when both signals fire.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two small issues caught in review:
1. check_containerd probed the socket unconditionally even when
service_active had already set found=1, while check_docker short
circuited the same path behind 'if \[ $found -eq 0 \]'. The
asymmetry was not user visible but violated least surprise for
future maintainers. check_containerd now uses the same gated pattern.
2. The HINT block recommended 'sudo rm -rf /var/lib/docker
/var/lib/containerd' as a casual follow up, which could destroy data
the operator still needs. Replace that line with a warning telling
the operator to inspect and reclaim standalone runtime storage
manually rather than deleting it blindly.
Also drop a no op 'printf' from disk_usage that served no purpose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ubuntu hosts running the cozystack "generic" variant (k3s or kubeadm)
sometimes end up with a standalone containerd.service or docker.service
running alongside the embedded k3s runtime. The two runtimes do not fight
over sockets — k3s uses /run/k3s/containerd/containerd.sock while the
standalone package uses /run/containerd/containerd.sock — so both keep
running silently, and the standalone one accumulates unpruned images and
build cache in /var/lib/containerd. Over time this fills the root disk,
triggers DiskPressure, and sends cozystack-api into an eviction loop.
hack/check-host-runtime.sh warns an operator about this before install
without blocking it. The script probes systemctl and well-known socket
paths, reports disk usage of the standalone data directory when present,
prints a hint on how to disable the shadow runtime, and always exits 0.
Covered by hack/check-host-runtime.bats (six test cases).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add hack/check-host-runtime.bats with six self-contained test cases that
exercise the check-host-runtime.sh preflight script: clean host exits
silently, standalone containerd.service warns, standalone docker.service
warns, both services warn, du failures do not suppress warnings, and the
socket-only fallback fires when systemctl is unavailable.
Tests inject a stub systemctl and du binary via PATH and redirect the
script's probe paths through COZYSTACK_PREFLIGHT_* environment variables,
so they run without root and on any host (including non-systemd macOS).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Increase Kafka CR readiness timeout from 60s to 300s to account for
slow Strimzi startup on QEMU-based CI sandbox (4 JVM pods).
Add wait-for-delete before re-applying to prevent race condition where
kubectl apply hits a still-deleting resource on retry attempts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Reduces flakiness in E2E tests by addressing the most common failure
modes.
### Kubernetes tenant tests (`run-kubernetes.sh`)
- Increase node Ready timeout from 2m to 5m — CI runners are shared and
resource-constrained
- Fail fast when nodes are not Ready — saves ~7 minutes per failed
attempt by not running LB/NFS tests that will also fail
- Delete stale Kubernetes resources at test start — retries provision
from scratch instead of patching stuck state
- Wait for port-forward to be ready before using it (fixes race
condition)
- Reduce version check polling interval from 5s to 1s
### All app tests (postgres, redis, kafka, clickhouse, mariadb, mongodb,
qdrant, foundationdb, openbao, vminstance, bucket)
- Add pre-cleanup of stale resources from previous failed retries so
each attempt starts clean
### Test runner (`cozytest.sh`)
- Clean up temp directory on test failure (was only cleaned on success,
leaking `/tmp` dirs)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved e2e robustness by deleting stale resources before creation
(ignore-if-missing, bounded time) and killing leftover port-forwards.
* Added an exit cleanup to consistently remove temporary artifacts and
teardown port-forwards on any exit.
* Added readiness polling for forwarded endpoints (curl with timeout);
made API/version retries more responsive.
* Extended node readiness wait (2m → 5m), dump debug info and fail fast
if unready.
* Made load‑balancer reachability checking explicit and more reliable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
