The test body asserts .status.history[].status extraction, but the
test name still referenced the old installFailures counter (leftover
from when the guard used that field before switching to status.history
to avoid ClearFailures zeroing the counters on successful reconcile).
Address review feedback from coderabbitai on
hack/remediation-guard.bats:84: rename so grep for what the test
actually pins matches the test name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Log .status.history regardless of content so a silently empty result
(Flux field rename) shows up in CI logs, and treat empty history on
a Ready HelmRelease as a distinct failure. A Ready HR by definition
has at least one snapshot; anything else is a shape-drift signal.
- Replace the unquoted heredoc in remediation-guard.sh with a printf |
grep pipeline. printf %s treats statuses as literal payload (no $
expansion surprises for future callers), grep --quiet --extended-regexp
returns exit status the caller can forward directly.
- Share the etcd-absent values file between both invariant tests
(packages/apps/kubernetes/tests/values-ci-no-etcd.yaml) instead of
duplicating the --set block.
- Fix typo "override applied" -> "override is applied" in the
Kubernetes ApplicationDefinition.
- Add a coupling comment in the ApplicationDefinition annotation that
points at the wait-for-kubeconfig init deadline in _helpers.tpl, so
a future operator raising the HR timeout updates the init deadline
too.
- Clarify the per-annotation timeout comment in rest.go so it stops
implying the feature is Kubernetes-only (it is not - only today's
one user is).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
17 child HelmReleases (cilium, coredns, csi, cert-manager,
metrics-server, ...) referenced *-admin-kubeconfig via
kubeConfig.secretRef and rendered even when _namespace.etcd was empty.
On an etcd-less tenant each one sat in NotReady forever because the
admin-kubeconfig Secret only exists after a KamajiControlPlane
reconciles, and KamajiControlPlane now only renders when etcd is set.
The outcome contradicted the "beacon only" contract claimed in the
soft-skip commit.
Extend the existing addon guards to also require _namespace.etcd, and
wrap the four unconditional HelmReleases (csi, metrics-server,
prometheus-operator-crds, volumesnapshot-crd) plus the always-on
cilium/coredns HR resources in the same gate. Add an invariant bats
test that renders the whole chart with etcd empty and asserts zero
HelmReleases reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Flux helm-controller's ClearFailures() zeroes installFailures and
upgradeFailures on every successful reconciliation (see the upstream
HelmReleaseStatus method). The previous guard ran after the HelmRelease
was Ready, at which point the counters were always 0 - the assertion
was vacuous and would have passed against a reverted fix.
Switch to .status.history, which retains per-revision release
Snapshots that survive a subsequent successful reconciliation. A
remediation cycle leaves behind a Snapshot with status=uninstalled
(the install-remediation code path) or status=failed (Helm release
failure that remediation then uninstalled). Either one signals the
race actually fired.
Rewrite the bats unit tests to cover: empty history, deployed-only,
deployed+superseded (happy path - not detected), single failed,
single uninstalled, uninstalled-then-deployed, and deployed-then-failed
(all detected). The pinned-shape test feeds a realistic HR status
snippet through yq the same way run-kubernetes.sh does via kubectl
-o jsonpath.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The soft-skip wrap in cluster.yaml only silenced Cluster and
KamajiControlPlane rendering when _namespace.etcd is empty. The three
CP-side Deployments (cluster-autoscaler, kccm, kcsi-controller) still
rendered, their wait-for-kubeconfig init containers CrashLoopBackOff'd
forever (no KamajiControlPlane = no admin-kubeconfig Secret),
HelmRelease hit its 15m wait timeout and triggered the very install
remediation cycle the rest of this PR prevents. Self-contradiction.
Wrap each of the three Deployment templates in
{{- if .Values._namespace.etcd }}...{{- end }} so they render only
when there is a DataStore to back them. Add an invariant bats test
that renders the whole chart with etcd empty and asserts zero
Deployments reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
run-kubernetes.sh extracts .status.installFailures and
.status.upgradeFailures via kubectl -o jsonpath. If a future flux
release renames the counters, kubectl returns empty, the guard reports
no cycle, and e2e silently misses real remediation loops.
Add a bats unit test that feeds a pinned HelmRelease v2 status snippet
through the same jsonpath and asserts the extraction still yields the
expected values. Also leave a pointer comment in run-kubernetes.sh so a
future flux bump surfaces the version coupling in review.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-template unittests in packages/apps/kubernetes/tests/ assert
that cluster-autoscaler, kccm, and the csi controller each mount the
admin-kubeconfig Secret optional and carry the wait-for-kubeconfig
init. That locks in today's three Deployments by name - a fourth
Deployment that mounts the same Secret but forgets the guard would
slip past them.
Add a bats-unit test that renders the entire chart, enumerates every
Deployment whose spec mounts a Secret ending in -admin-kubeconfig, and
asserts optional:true plus wait-for-kubeconfig init on all of them.
Verified by temporarily removing optional:true from csi/deploy.yaml:
the test correctly flagged invariant-kcsi-controller as an offender.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Shell && and || have equal precedence and left-to-right associativity,
so the previous guard parsed as (((A && B) || C) && D) and silently
passed on the canonical failure mode: install_failures=1 with an empty
upgrade_failures.
Extract the check into helmrelease_has_remediation_cycle() in a
dedicated helper sourced from run-kubernetes.sh, and add unit tests
under hack/remediation-guard.bats that pin the expected behavior for
every combination of empty, zero, and positive counter values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Before cleanup, inspect the parent HelmRelease installFailures and
upgradeFailures counters. A non-zero value means flux helm-controller
hit its wait timeout, ran install/upgrade remediation (uninstall),
and re-installed - the exact race condition this PR closes. Fail the
bats test in that case so the signal surfaces in CI instead of being
masked by a green retry.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
The aggregated API previously accepted tenant names containing hyphens
(e.g. `foo-bar`) because `ValidateApplicationName` delegated entirely to
`IsDNS1035Label`, which permits them. The tenant Helm chart's
`tenant.name` helper then rejected the release at template time — so
users saw a successful `kubectl apply` followed by a confusing
Flux/HelmRelease reconciliation error.
This PR extends `ValidateApplicationName` with a `kindName` parameter
and enforces an alphanumeric-only rule (`^[a-z][a-z0-9]*$`) for the
Tenant kind. A `TenantKind` constant centralizes the kind string, and
`REST.validateNameFormat` wraps the check symmetrically with the
existing `validateNameLength`. The tenant chart README is updated to
match the already-correct website documentation.
Test coverage includes:
- Unit tests for the validation package (format + error message
contract)
- REST wrapper and Update→Create fall-through path tests with a fake
client
- E2E BATS regression test with explicit exit-code checks and cleanup
Fixes#2375
### Release note
```release-note
[api] Tenant names containing dashes are now rejected at API level during creation, with a specific error message, instead of being silently accepted and failing later during Helm reconciliation.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant names now require lowercase letters and digits only, must start
with a letter; dashes are rejected with a tenant-specific error.
* **Documentation**
* Tenant naming guidelines updated to reflect the stricter
alphanumeric-only requirement.
* **Tests**
* New unit and end-to-end tests added to verify tenant naming
enforcement and tenant-specific error messaging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The api/apps/v1alpha1/* subtags share the same commit as the release
tags. git describe --exact-match picks the first match alphabetically,
returning api/apps/v1alpha1/vX.Y.Z instead of vX.Y.Z, which produces
invalid Docker image tags.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Four follow-ups from review round 4:
1. BLOCKER: the Update(forceAllowCreate=true) path delegates to Create() when the object does not yet exist (rest.go:452) — the typical kubectl apply upsert flow. Add TestUpdate_ForceAllowCreate_RejectsTenantDashName using a fake client so a future refactor of that delegation cannot silently bypass the tenant name check that r.validateNameFormat alone cannot catch.
2. BLOCKER: the e2e BATS test used || true inside the command substitution, which swallowed the kubectl exit code. Rework the test to capture exit code and stdout+stderr explicitly, then assert the exit code is non-zero before asserting on the error message. This distinguishes validation-success (kubectl exit 0 — regression) from environmental failures (exit non-zero but wrong message) from the happy path.
3. Extract TenantKind = "Tenant" as a named constant in the validation package with a comment pointing at the upstream ApplicationDefinition source of truth, and switch the kindName check to use it.
4. Add a clarifying comment on TestValidateApplicationName_TenantLengthFallthrough that it pins an architectural layering decision and is not a user-facing requirement, so a future promotion of tenant length into tenant-specific wording is a legitimate change rather than a test regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three follow-ups from review:
1. Add defensive cleanup of the foo-bar tenant before and after the e2e regression test. If a prior run left the object in the cluster (e.g. a transient validation regression), the test no longer inherits that state; and if the test itself trips a regression in the future, the cluster is left clean for subsequent tests.
2. Switch kubectl --validate=false to --validate=ignore. The bool form was deprecated in kubectl 1.25 and only kept as an alias; --validate=ignore is the modern, stable spelling.
3. Drop the brittle Contains("63") assertion in TestValidateApplicationName_TenantLengthFallthrough. It pinned the test to the exact DNS-1035 error text from k8s.io/apimachinery, which could break on unrelated upstream wording changes. The surviving assertions still cover the intent: an error exists and it is not the tenant-specific message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Follow-up from review round 2:
The BATS regression test now pins a namespace precondition, disables kubectl client-side validation with --validate=false so the request is guaranteed to reach the server-side name check, and documents the intent of each assertion. Previously a client-side schema rejection could have produced a false negative on the tenant-specific error grep, and a missing tenant-root namespace could have produced an unrelated failure that was hard to diagnose.
Also pin the edge case where a tenant name consists of valid characters but exceeds the DNS-1035 63-char label limit. The resulting error is intentionally the generic DNS-1035 message because length is not a tenant-specific constraint — the package-level function is not responsible for the stricter Helm-release-prefix length budget that REST.validateNameLength enforces.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two follow-ups from review:
1. The original tenantNameRegex ^[a-z0-9]+$ let leading-digit names like 123foo slip past the tenant check and receive a generic DNS-1035 error, which defeated the goal of surfacing a tenant-specific message. Require a leading lowercase letter so every tenant-invalid name returns the tenant-contract error.
2. Add a BATS regression test in hack/e2e-install-cozystack.bats that exercises the aggregated API end-to-end. Unit tests construct REST{kindName: "Tenant"} by hand, so a future change to ApplicationDefinition kind registration could break real behavior without breaking the unit tests.
Also pin the error-message contract with a new TestValidateApplicationName_TenantErrorMessage that asserts every tenant-invalid input returns a message containing 'tenant names must' (not the generic DNS-1035 text).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The disk_usage helper shells out to 'du -sh $path' for reporting in
the warning message. On the exact directories this script is meant
to warn about (a /var/lib/containerd accumulating millions of files
from months of unpruned builds), traversing the tree with du can
take minutes and stall the preflight indefinitely.
Wrap the call in 'timeout 5s' so the helper returns quickly even on
a pathologically slow filesystem. If the timeout binary is absent
(e.g. minimal busybox userland), the pipeline still exits 0 via the
existing '|| true' guard and usage stays empty — the warning still
prints, just without the size detail — so the change is backward
compatible.
Addresses an inline review comment from gemini-code-assist on the
open PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address three review findings on the bats suite:
- The 'standalone containerd service active' and 'standalone docker
service active' tests previously asserted 'systemctl disable --now
<service>' but did not require the 'sudo ' prefix, so the two-
service test was the only guard against a silent drop of the
prefix. Extend both single-service tests to require the prefix
exactly as the 'both services active' test does.
- The 'both services active' test now captures the exit code
explicitly with 'bash ... || status=$?' and asserts '$status -eq
0'. The script contract ('always exits 0') was previously enforced
implicitly via 'set -e', which produces a generic test failure on
a regression instead of a contract-specific error. The explicit
check locks in the contract and makes regressions readable.
- New test: 'docker socket paths with glob chars do not expand' —
sets COZYSTACK_DOCKER_SOCKET_PATHS to a literal glob against real
directories and asserts no docker warning fires. Locks in the
array-based parsing of the path list.
- Extend the file header with a 'title syntax constraints' section
documenting how cozytest.sh's awk parser treats double quotes and
non-alphanumeric characters in @test titles, so future
contributors do not stumble on the parser's quirks.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
'set -euo pipefail' does not include 'set -f', so the previous
'for sock in $DOCKER_SOCKET_PATHS' loop relied on both word splitting
AND unintended glob expansion. If the variable ever contained a
literal '*' or '?' — for example a test setting
COZYSTACK_DOCKER_SOCKET_PATHS='/run/docker-*.sock' against a tree
that happens to have real files matching the pattern — the loop
would iterate over directory entries instead of the intended socket
paths, producing false-positive docker warnings.
Replace the implicit split+glob loop with 'read -ra' into a bash
array and iterate 'for sock in "${_socks[@]}"'. This keeps the
space-separated input format, disables glob expansion, and removes
the fragile word splitting that shellcheck otherwise has to be
talked out of.
Covered by a new regression test ('docker socket paths with glob
chars do not expand') that sets the env var to a literal '*' pattern
pointing at real directories; without this fix the test fails with
an unexpected docker.service warning.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
DOCKER_SOCKET_PATHS is a space separated list of socket paths (its
default is "/run/docker.sock /var/run/docker.sock") that the loop
iterates by word splitting. Socket paths never contain whitespace on
Linux hosts, so this is the documented and correct idiom — not a
shellcheck oversight. Add an inline comment so a future maintainer (or
a linter that has not seen the comment) does not break the loop by
adding quotes around the expansion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two review findings:
1. The socket fallback tests previously branched on 'command -v
python3' and fell through to 'return 0' when it was missing.
cozytest.sh has no SKIP concept — 'return 0' is indistinguishable
from a real pass, so on a runner without python3 the socket
fallback paths were silently unverified. Since the script uses
'[ -e "$sock" ]' rather than '[ -S ... ]', a regular file is
sufficient to exercise the detection path. Replace the python3
unix-socket creation with 'touch "$SOCK"' so the tests run
unconditionally on every runner.
2. Extend the 'both services active' HINT assertion to require the
'sudo ' prefix on the 'systemctl disable --now' line. Without sudo
the operator would be instructed to run a privileged command as a
non-root user and hit a silent failure; the prefix is as important
as the verb itself and must be locked in by a test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expand the bats suite to address a second review pass:
- Assert that a containerd only warning produces a HINT that names
containerd.service and NOT docker.service, and mirror the check in
the docker only test. When both fire, assert the HINT lists both
services in a single systemctl disable invocation.
- New test: 'clean host without systemctl exits silently' — exercises
the COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 path when no standalone
sockets exist. Previously the tests only covered the systemd enabled
clean host, leaving the non systemd clean path unverified.
- New test: 'docker service plus socket still emits exactly one
warning' — mirrors the existing containerd service+socket test and
locks in the gated check in check_docker.
- Replace silent 'return 0' on missing python3 with a visible
'# SKIP: python3 unavailable' message on stderr so CI logs make it
obvious which tests were skipped on a given runner.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When only containerd.service was active, the HINT block still advised
the operator to disable both containerd.service and docker.service,
which is misleading and potentially dangerous on hosts where docker is
legitimately in use. Track per service warnings (CONTAINERD_WARN and
DOCKER_WARN) and build the HINT systemctl disable argument from the
services that actually fired.
Also guard the ANSI color escapes behind an 'is stderr a TTY' check so
log files (CI and reviewer captures) do not accumulate raw escape
sequences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback on the preflight test suite:
- Add a trap 'rm -rf $STUB_DIR' EXIT in every test immediately after
mktemp -d, so temp dirs are cleaned up even when an assertion fails
and terminates the test early under set -e.
- Assert the HINT block and 'systemctl disable --now' line in the
'both services active' test so a future silent removal or typo in
the HINT output is caught by CI.
- New test: 'docker socket fallback fires when systemctl is
unavailable' — mirrors the existing containerd socket fallback test
and exercises the same code path in check_docker.
- New test: 'containerd service plus socket still emits exactly one
warning' — documents and locks in the intent of the symmetric gated
check in check_containerd. Uses grep -c to assert the warning is not
double printed when both signals fire.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two small issues caught in review:
1. check_containerd probed the socket unconditionally even when
service_active had already set found=1, while check_docker short
circuited the same path behind 'if \[ $found -eq 0 \]'. The
asymmetry was not user visible but violated least surprise for
future maintainers. check_containerd now uses the same gated pattern.
2. The HINT block recommended 'sudo rm -rf /var/lib/docker
/var/lib/containerd' as a casual follow up, which could destroy data
the operator still needs. Replace that line with a warning telling
the operator to inspect and reclaim standalone runtime storage
manually rather than deleting it blindly.
Also drop a no op 'printf' from disk_usage that served no purpose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ubuntu hosts running the cozystack "generic" variant (k3s or kubeadm)
sometimes end up with a standalone containerd.service or docker.service
running alongside the embedded k3s runtime. The two runtimes do not fight
over sockets — k3s uses /run/k3s/containerd/containerd.sock while the
standalone package uses /run/containerd/containerd.sock — so both keep
running silently, and the standalone one accumulates unpruned images and
build cache in /var/lib/containerd. Over time this fills the root disk,
triggers DiskPressure, and sends cozystack-api into an eviction loop.
hack/check-host-runtime.sh warns an operator about this before install
without blocking it. The script probes systemctl and well-known socket
paths, reports disk usage of the standalone data directory when present,
prints a hint on how to disable the shadow runtime, and always exits 0.
Covered by hack/check-host-runtime.bats (six test cases).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add hack/check-host-runtime.bats with six self-contained test cases that
exercise the check-host-runtime.sh preflight script: clean host exits
silently, standalone containerd.service warns, standalone docker.service
warns, both services warn, du failures do not suppress warnings, and the
socket-only fallback fires when systemctl is unavailable.
Tests inject a stub systemctl and du binary via PATH and redirect the
script's probe paths through COZYSTACK_PREFLIGHT_* environment variables,
so they run without root and on any host (including non-systemd macOS).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Increase Kafka CR readiness timeout from 60s to 300s to account for
slow Strimzi startup on QEMU-based CI sandbox (4 JVM pods).
Add wait-for-delete before re-applying to prevent race condition where
kubectl apply hits a still-deleting resource on retry attempts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Reduces flakiness in E2E tests by addressing the most common failure
modes.
### Kubernetes tenant tests (`run-kubernetes.sh`)
- Increase node Ready timeout from 2m to 5m — CI runners are shared and
resource-constrained
- Fail fast when nodes are not Ready — saves ~7 minutes per failed
attempt by not running LB/NFS tests that will also fail
- Delete stale Kubernetes resources at test start — retries provision
from scratch instead of patching stuck state
- Wait for port-forward to be ready before using it (fixes race
condition)
- Reduce version check polling interval from 5s to 1s
### All app tests (postgres, redis, kafka, clickhouse, mariadb, mongodb,
qdrant, foundationdb, openbao, vminstance, bucket)
- Add pre-cleanup of stale resources from previous failed retries so
each attempt starts clean
### Test runner (`cozytest.sh`)
- Clean up temp directory on test failure (was only cleaned on success,
leaking `/tmp` dirs)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved e2e robustness by deleting stale resources before creation
(ignore-if-missing, bounded time) and killing leftover port-forwards.
* Added an exit cleanup to consistently remove temporary artifacts and
teardown port-forwards on any exit.
* Added readiness polling for forwarded endpoints (curl with timeout);
made API/version retries more responsive.
* Extended node readiness wait (2m → 5m), dump debug info and fail fast
if unready.
* Made load‑balancer reachability checking explicit and more reliable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Root cause: EXIT trap called _k8s_test_cleanup after subshell exit,
but variables ${port} and ${test_name} were not available in that
context. set -u caught the unset variable and exited with error,
marking a successful test as failed. This caused every kubernetes
test to "fail" on attempt 1, then retry 2 more times with real
failures (NFS PVC timeout because the cluster was being deleted).
Fix: remove trap and cleanup function entirely. Pre-cleanup at
test start handles stale resources. Inline cleanup at test end
handles normal path. On failure, resources are cleaned up by
pre-cleanup on the next retry.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Root cause of NFS test failures: port-forward had `timeout 500s`
(8.3 minutes), but tests after it take up to 26 minutes (node
join 8m + node Ready 5m + LB test 2.5m + NFS 10m). Port-forward
was guaranteed to die mid-test.
Remove the timeout entirely. Cleanup is already handled by:
- EXIT trap (_k8s_test_cleanup does pkill port-forward)
- Explicit cleanup at every exit point
- Job-level timeout-minutes: 120 as backstop
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Increase NFS PVC Bound timeout from 2m to 5m — RWX via kubevirt
CSI provisions an NFS server pod which takes time
- Add pod describe and events dump when NFS test pod fails to
complete, making the root cause visible in CI logs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
On resource-constrained CI runners, HelmReleases may take close
to 15 minutes to reconcile, leaving very little time for CAPI
deployments to start and become available. Increase timeouts:
- CAPI deployment discovery: 60s -> 180s
- CAPI deployment availability: 1m -> 5m
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The explicit _k8s_test_cleanup calls cover known error paths, but
set -e can terminate the script at any kubectl/wait command. Add
an EXIT trap to catch these unexpected exits. The function runs
in a cozytest.sh subshell, so the trap is scoped and does not
affect parent traps. Removed 2>/dev/null from kubectl delete in
cleanup to preserve useful error output.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Use a _k8s_test_cleanup helper called at every exit point
instead of a process-scoped EXIT trap. This avoids any
potential trap scope issues and makes cleanup behavior
explicit and visible at each error path.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expand the EXIT trap in run_kubernetes_test to also delete the
Kubernetes tenant resource. Previously, early exit paths (exit 1
on node readiness failure, LB failure, etc.) would skip the final
kubectl delete, leaving the resource behind.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add pre-cleanup for VMInstance in second vminstance.bats test
- Fix indentation of LB_ADDR block in run-kubernetes.sh
- Add --ignore-not-found to final Kubernetes resource cleanup
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add EXIT trap in run-kubernetes.sh to clean up port-forward
process and tenantkubeconfig file on any exit path
- Fix LoadBalancer retry loop: success on attempt 20 was falsely
reported as failure due to $i -eq 20 check after break
- Kill stale port-forward in bucket.bats before retry
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add pre-cleanup of stale resources to all app E2E tests so
retries start fresh instead of patching stuck state
- Wait for port-forward to be ready before using it in
kubernetes tests (fixes race condition)
- Reduce version check sleep from 5s to 1s for faster retries
- Clean up temp directory on test failure in cozytest.sh
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three changes to reduce flakiness in kubernetes E2E tests:
- Increase node Ready timeout from 2m to 5m to accommodate
resource-constrained CI runners
- Fail fast when nodes are not Ready instead of continuing to
LB/NFS tests that will also fail, saving ~7 minutes per attempt
- Delete stale Kubernetes resources at test start to ensure
retries provision from scratch instead of patching stuck state
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
Add external-dns as a standalone self-managed application in
`packages/extra/external-dns/`, allowing tenants to deploy and configure
their own DNS management directly from the dashboard.
## Motivation
Tenants need the ability to manage their own DNS domains with their own
provider. Following the [developers
guide](https://github.com/cozystack/website/pull/413), this is
implemented as an extra package (like `ingress` and `seaweedfs`) using
the HelmRelease-based pattern, rather than embedding it in the tenant
chart.
This enables multi-tenant scenarios where:
- Tenant A uses Cloudflare for `domain-a.com`
- Tenant B uses AWS Route53 for `domain-b.com`
- Each tenant deploys and manages external-dns independently from the
dashboard
## Changes
- **New package**: `packages/extra/external-dns/` — standalone
HelmRelease-based application
- **New PackageSource**:
`packages/core/platform/sources/external-dns-application.yaml` —
references `system/external-dns` and `extra/external-dns` components
- **Cleaned tenant chart**: removed the previously embedded
`externalDns` block from `packages/apps/tenant/`
## Features
- Support for 9 DNS providers: cloudflare, aws, azure, google,
digitalocean, linode, ovh, exoscale, godaddy
- Per-provider credential configuration with full JSON schema validation
- Domain filtering via `domainFilters`
- Configurable sync policy (`sync` or `upsert-only`)
- Namespaced operation (`namespaced: true`) for tenant isolation
- Unique `txtOwnerId` per namespace to prevent DNS record conflicts
- Resource sizing via presets or explicit CPU/memory
## Usage Example
Deploy from the dashboard, or via values:
```yaml
# Cloudflare
provider: cloudflare
domainFilters:
- example.com
cloudflare:
apiToken: "your-cloudflare-api-token"
```
```yaml
# AWS Route53
provider: aws
domainFilters:
- example.org
aws:
accessKeyId: "AKIAXXXXXXXX"
secretAccessKey: "your-secret-key"
region: "us-east-1"
```
## Test plan
- [ ] `helm template external-dns packages/extra/external-dns/ --set
provider=cloudflare --set cloudflare.apiToken=test` renders correctly
- [ ] `helm template external-dns packages/extra/external-dns/` fails
(provider required)
- [ ] `helm template wrong-name packages/extra/external-dns/ --set
provider=cloudflare` fails (release name check)
- [ ] Deploy external-dns from tenant dashboard
- [ ] Verify HelmRelease is created in tenant namespace with namespaced
RBAC
- [ ] Create an Ingress and verify DNS record is created
- [ ] Verify no conflict with global external-dns instance
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an External DNS package for automatic DNS record management.
* Support for 9 DNS providers: Cloudflare, AWS, Azure, Google,
DigitalOcean, Linode, OVH, Exoscale, GoDaddy.
* Helm-based deployment with namespaced/system variants and release
configuration options.
* Configurable synchronization policies, domain filtering, provider
credentials, extra args, and resource presets.
* **Documentation**
* New README and schema-driven values documentation for installation and
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The victoria-metrics-operator v0.68.1 renamed VMCluster status field
from .status.clusterStatus to .status.updateStatus.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
After migrating VictoriaLogs from VLogs to VLCluster, the e2e test
still waited for the old vlogs/generic resource which no longer exists.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
