Commit graph

101 commits

Author SHA1 Message Date
Myasnikov Daniil
adc430e27c
test(e2e-apps): add existence backstops to all kubectl wait calls
`kubectl wait <kind> <name>` errors immediately with NotFound if the
resource doesn't exist yet — even with --for=condition or --for=jsonpath.
The redis test in CI run 25043350705 failed in 1 second for exactly this
reason: the redis-failover operator hadn't created the PVC by the time
the test waited for it. Previously the 3x retry on `Run E2E tests`
masked this race; with retry dropped, every such call is a flake risk.

Add a small `until kubectl get` existence backstop before each
kubectl wait, matching the pattern already established for HRs in
commit 66888c91.

33 backstops across 12 files:
  bucket.bats         — bucketclaims, bucketaccesses x2
  clickhouse.bats     — statefulset 0-0 (0-1 already covered)
  harbor.bats         — deploy x3, bucketclaims, bucketaccesses
  kafka.bats          — kafkas
  mariadb.bats        — statefulset, deployment
  mongodb.bats        — statefulset
  openbao.bats        — sts, pvc
  postgres.bats       — job.batch
  qdrant.bats         — sts, pvc
  redis.bats          — pvc, deploy, sts (the trigger)
  vminstance.bats     — dv, pvc, vm (with 120s for KubeVirt latency)
  e2e-install-cozystack.bats — apiservices, sts/etcd, vmalert,
                                vmalertmanager, vlclusters, vmcluster,
                                clusters.postgresql.cnpg.io,
                                deploy/grafana-deployment, namespace

Same pattern, same 60s default timeout for the existence wait (120s for
nested-virt resources). Once the resource exists, the original wait
timeout takes over.

Run-kubernetes.sh has the same race shape on several waits (nfs, kamaji,
machinedeployment, etc.) — out of scope here; flagged for follow-up.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 15:52:39 +05:00
Myasnikov Daniil
428868a3b2
test(e2e-apps): fix vminstance delete-recreate race + bump VM IP/ready timeouts
Two reliability fixes in `hack/e2e-apps/vminstance.bats`:

1. Delete-recreate race. Both @test blocks deleted the prior VMInstance/VMDisk
   with `kubectl ... --timeout=2m || true`. The `|| true` silently swallowed
   timeout errors, letting the test continue with the resource still in
   finalizer-drain. The next `kubectl apply` then no-ops with
   "Detected changes to resource ... which is currently being deleted",
   producing a NotFound on the downstream HR wait. Bumped the delete
   timeout to 3m and removed `|| true` so true delete failures surface
   loudly. End-of-test cleanup deletes also got the explicit timeout.

2. VM IP and VM ready timeouts. The 20s timeout for the VMI to acquire an
   IP was unrealistic for nested KubeVirt under runner load — virt-launcher
   + libvirt + cloud-init DHCP routinely takes 30-60s. Bumped to 120s with
   a 2s poll interval (60 polls instead of 4). VM ready timeout bumped from
   20s to 60s for the same reason.

Both surfaced as recurring first-attempt failures in the past 30 successful
PR runs (7/17 analysable runs in each case) — masked by the 3x retry on
`Run E2E tests`.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:13:55 +05:00
Myasnikov Daniil
461cdf961f
test(e2e): annotate genuine sleeps with TODO and rationale
Three poll/sleep sites in the install bats and the kubernetes runner are
NOT replaceable by `kubectl wait --for=condition=...` because the signal
they observe is not a Kubernetes API condition:

- e2e-install-cozystack.bats:55-56 - 5s pad lets late-arriving HRs join
  the awk-snapshot the parallel `kubectl wait` runs against. There is no
  k8s condition for "all expected platform HRs have been emitted" short
  of hard-coding the list.
- e2e-install-cozystack.bats:75 - LINSTOR node membership is reported by
  the linstor binary running inside the controller pod (kubectl exec),
  not a CRD status, so kubectl wait cannot subscribe to it.
- e2e-apps/run-kubernetes.sh:231-238 - validates the external HTTP path
  through MetalLB -> tenant ingress -> backend pod end-to-end. Not a
  single API condition.

Annotate each with TODO(e2e-replace-fixed-timeouts) and the rationale so
future readers do not "fix" them with a kubectl wait that does not work.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 00:07:19 +05:00
Myasnikov Daniil
66888c91ee
test(e2e-apps): replace fixed sleeps with HelmRelease existence backstop
The app bats files all share the pattern:

    kubectl apply -f - <<EOF ... EOF
    sleep 5
    kubectl wait hr <name> --for=condition=ready

The fixed sleep guesses how long the cozystack-operator needs to translate
the app CR into a HelmRelease. On a noisy CI runner or after a cold
install that wait is sometimes too short, so the immediately-following
`kubectl wait hr ...` fails because the HR object does not exist yet
(`kubectl wait` errors out instantly on a missing resource).

Replace each `sleep 5` (and the one outlier `sleep 15` in foundationdb
that was paying for the same risk) with an event-driven backstop that
polls for the HR's existence and exits the moment it appears, capped at
60s. The downstream `kubectl wait --for=condition=ready` is unchanged,
preserving the original total budget.

Same shape as the LINSTOR fix in commit eb87413 on feat/e2e-optimization:
wait for the actual prerequisite (the HelmRelease the operator creates),
not on a downstream artefact whose existence depends on it.

Files: postgres, mariadb, kafka, mongodb, clickhouse, redis, qdrant,
harbor, openbao, external-dns (both tests), vminstance (both tests, with
the existence backstop hoisted before the downstream vmi-ip poll), and
foundationdb.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 00:05:02 +05:00
Aleksei Sviridkin
b8aec9a973
fix(kubernetes): history guard non-empty check + nits from review
- Log .status.history regardless of content so a silently empty result
  (Flux field rename) shows up in CI logs, and treat empty history on
  a Ready HelmRelease as a distinct failure. A Ready HR by definition
  has at least one snapshot; anything else is a shape-drift signal.
- Replace the unquoted heredoc in remediation-guard.sh with a printf |
  grep pipeline. printf %s treats statuses as literal payload (no $
  expansion surprises for future callers), grep --quiet --extended-regexp
  returns exit status the caller can forward directly.
- Share the etcd-absent values file between both invariant tests
  (packages/apps/kubernetes/tests/values-ci-no-etcd.yaml) instead of
  duplicating the --set block.
- Fix typo "override applied" -> "override is applied" in the
  Kubernetes ApplicationDefinition.
- Add a coupling comment in the ApplicationDefinition annotation that
  points at the wait-for-kubeconfig init deadline in _helpers.tpl, so
  a future operator raising the HR timeout updates the init deadline
  too.
- Clarify the per-annotation timeout comment in rest.go so it stops
  implying the feature is Kubernetes-only (it is not - only today's
  one user is).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 21:50:10 +03:00
Aleksei Sviridkin
1ddd2aaea9
fix(hack): detect remediation via status.history, not transient counters
Flux helm-controller's ClearFailures() zeroes installFailures and
upgradeFailures on every successful reconciliation (see the upstream
HelmReleaseStatus method). The previous guard ran after the HelmRelease
was Ready, at which point the counters were always 0 - the assertion
was vacuous and would have passed against a reverted fix.

Switch to .status.history, which retains per-revision release
Snapshots that survive a subsequent successful reconciliation. A
remediation cycle leaves behind a Snapshot with status=uninstalled
(the install-remediation code path) or status=failed (Helm release
failure that remediation then uninstalled). Either one signals the
race actually fired.

Rewrite the bats unit tests to cover: empty history, deployed-only,
deployed+superseded (happy path - not detected), single failed,
single uninstalled, uninstalled-then-deployed, and deployed-then-failed
(all detected). The pinned-shape test feeds a realistic HR status
snippet through yq the same way run-kubernetes.sh does via kubectl
-o jsonpath.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 21:28:11 +03:00
Aleksei Sviridkin
03426fbd71
test(hack): pin HelmRelease v2 status shape used by remediation guard
run-kubernetes.sh extracts .status.installFailures and
.status.upgradeFailures via kubectl -o jsonpath. If a future flux
release renames the counters, kubectl returns empty, the guard reports
no cycle, and e2e silently misses real remediation loops.

Add a bats unit test that feeds a pinned HelmRelease v2 status snippet
through the same jsonpath and asserts the extraction still yields the
expected values. Also leave a pointer comment in run-kubernetes.sh so a
future flux bump surfaces the version coupling in review.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 21:13:06 +03:00
Aleksei Sviridkin
f87834a3ed
fix(hack): group e2e remediation guard conditions correctly
Shell && and || have equal precedence and left-to-right associativity,
so the previous guard parsed as (((A && B) || C) && D) and silently
passed on the canonical failure mode: install_failures=1 with an empty
upgrade_failures.

Extract the check into helmrelease_has_remediation_cycle() in a
dedicated helper sourced from run-kubernetes.sh, and add unit tests
under hack/remediation-guard.bats that pin the expected behavior for
every combination of empty, zero, and positive counter values.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 20:48:47 +03:00
Aleksei Sviridkin
a7d994365d
test(kubernetes): assert parent HelmRelease did not remediate in e2e
Before cleanup, inspect the parent HelmRelease installFailures and
upgradeFailures counters. A non-zero value means flux helm-controller
hit its wait timeout, ran install/upgrade remediation (uninstall),
and re-installed - the exact race condition this PR closes. Fail the
bats test in that case so the signal surfaces in CI instead of being
masked by a green retry.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-16 20:26:47 +03:00
Aleksei Sviridkin
e16908bb62
[tests] Fix Kafka E2E test timeout and retry race condition
Increase Kafka CR readiness timeout from 60s to 300s to account for
slow Strimzi startup on QEMU-based CI sandbox (4 JVM pods).

Add wait-for-delete before re-applying to prevent race condition where
kubectl apply hits a still-deleting resource on retry attempts.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-08 17:32:50 +03:00
Aleksei Sviridkin
67b4180152
[tests] Fix: remove EXIT trap that broke successful kubernetes tests
Root cause: EXIT trap called _k8s_test_cleanup after subshell exit,
but variables ${port} and ${test_name} were not available in that
context. set -u caught the unset variable and exited with error,
marking a successful test as failed. This caused every kubernetes
test to "fail" on attempt 1, then retry 2 more times with real
failures (NFS PVC timeout because the cluster was being deleted).

Fix: remove trap and cleanup function entirely. Pre-cleanup at
test start handles stale resources. Inline cleanup at test end
handles normal path. On failure, resources are cleaned up by
pre-cleanup on the next retry.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-24 21:30:06 +03:00
Aleksei Sviridkin
72cad3b7d6
[tests] Optimize cleanup and timeouts to reduce E2E duration
The previous changes increased worst-case E2E time by ~60 minutes
due to blocking cleanup (--timeout=2m × multiple calls) and
generous timeouts on retry.

Changes:
- Use --wait=false in _k8s_test_cleanup (non-blocking fire-and-forget)
- Pre-cleanup: delete --wait=false + wait --for=delete (only blocks once)
- Node Ready timeout: 5m -> 3m (was 2m originally)
- NFS PVC timeout: 5m -> 3m (was 2m originally)
- CAPI discovery: 180s -> 120s, availability: 5m -> 2m (was 1m originally)

Estimated savings: ~40 minutes on worst-case retry path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-24 18:25:50 +03:00
Aleksei Sviridkin
30e36a2225
[tests] Remove port-forward timeout that kills NFS tests
Root cause of NFS test failures: port-forward had `timeout 500s`
(8.3 minutes), but tests after it take up to 26 minutes (node
join 8m + node Ready 5m + LB test 2.5m + NFS 10m). Port-forward
was guaranteed to die mid-test.

Remove the timeout entirely. Cleanup is already handled by:
- EXIT trap (_k8s_test_cleanup does pkill port-forward)
- Explicit cleanup at every exit point
- Job-level timeout-minutes: 120 as backstop

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-24 13:44:38 +03:00
Aleksei Sviridkin
8c11ece4ce
[tests] Improve NFS test: increase PVC timeout, add debug on failure
- Increase NFS PVC Bound timeout from 2m to 5m — RWX via kubevirt
  CSI provisions an NFS server pod which takes time
- Add pod describe and events dump when NFS test pod fails to
  complete, making the root cause visible in CI logs

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-24 03:23:16 +03:00
Aleksei Sviridkin
a610d1331d
[tests] Add EXIT trap as safety net for unexpected set -e exits
The explicit _k8s_test_cleanup calls cover known error paths, but
set -e can terminate the script at any kubectl/wait command. Add
an EXIT trap to catch these unexpected exits. The function runs
in a cozytest.sh subshell, so the trap is scoped and does not
affect parent traps. Removed 2>/dev/null from kubectl delete in
cleanup to preserve useful error output.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 17:55:30 +03:00
Aleksei Sviridkin
8f2c452ff3
[tests] Replace EXIT trap with explicit cleanup function
Use a _k8s_test_cleanup helper called at every exit point
instead of a process-scoped EXIT trap. This avoids any
potential trap scope issues and makes cleanup behavior
explicit and visible at each error path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 17:51:57 +03:00
Aleksei Sviridkin
1c7c1f6c78
[tests] Include Kubernetes resource cleanup in EXIT trap
Expand the EXIT trap in run_kubernetes_test to also delete the
Kubernetes tenant resource. Previously, early exit paths (exit 1
on node readiness failure, LB failure, etc.) would skip the final
kubectl delete, leaving the resource behind.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 17:47:48 +03:00
Aleksei Sviridkin
c9b04c4ba3
[tests] Fix review findings: vminstance pre-cleanup, indentation, --ignore-not-found
- Add pre-cleanup for VMInstance in second vminstance.bats test
- Fix indentation of LB_ADDR block in run-kubernetes.sh
- Add --ignore-not-found to final Kubernetes resource cleanup

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 17:42:23 +03:00
Aleksei Sviridkin
e3acb3a67e
[tests] Fix port-forward leak and LB retry off-by-one
- Add EXIT trap in run-kubernetes.sh to clean up port-forward
  process and tenantkubeconfig file on any exit path
- Fix LoadBalancer retry loop: success on attempt 20 was falsely
  reported as failure due to $i -eq 20 check after break
- Kill stale port-forward in bucket.bats before retry

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 17:33:27 +03:00
Aleksei Sviridkin
18961de36b
[tests] Add pre-cleanup, fix port-forward race, fix temp leak
- Add pre-cleanup of stale resources to all app E2E tests so
  retries start fresh instead of patching stuck state
- Wait for port-forward to be ready before using it in
  kubernetes tests (fixes race condition)
- Reduce version check sleep from 5s to 1s for faster retries
- Clean up temp directory on test failure in cozytest.sh

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 17:25:01 +03:00
Aleksei Sviridkin
64b370bfef
[tests] Stabilize E2E kubernetes tests
Three changes to reduce flakiness in kubernetes E2E tests:

- Increase node Ready timeout from 2m to 5m to accommodate
  resource-constrained CI runners
- Fail fast when nodes are not Ready instead of continuing to
  LB/NFS tests that will also fail, saving ~7 minutes per attempt
- Delete stale Kubernetes resources at test start to ensure
  retries provision from scratch instead of patching stuck state

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-03-23 16:54:09 +03:00
mattia-eleuteri
f201fe4dac
feat(extra): add external-dns as standalone extra package
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-03-10 12:48:01 +01:00
IvanHunters
e1b169d6a7 fix(test): replace bats-specific run command with shell negation
cozytest.sh executes .bats files as plain shell functions where bats
builtins like `run` are not available. Use `!` negation to assert that
readonly user upload fails.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-03-05 08:45:57 +03:00
IvanHunters
4e8733091d fix(test): add --insecure flag to all mc commands in bucket E2E test
The mc client requires --insecure on each command when connecting to
SeaweedFS S3 with self-signed certificates via port-forward.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-03-04 22:37:37 +03:00
IvanHunters
15ed534c25 test(bucket): update E2E test for user model and readonly access
Update bucket E2E test to match the new per-user access model:
- Create bucket with admin (readwrite) and viewer (readonly) users
- Test that readwrite user can upload, list, and download objects
- Test that readonly user can list and download but cannot upload
- Use per-user BucketAccess and credential secret names

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-03-04 21:13:46 +03:00
Andrei Kvapil
daa3905b67
[ci] Debug improvements (#2111)
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does


### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
[ci] Added more debug information to ci tests
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Enhanced error handling and diagnostic output in development testing
infrastructure.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-27 18:24:07 +01:00
Andrei Kvapil
022ddf73a8
[apps] Add OpenBAO as a managed secrets management service (#2059)
## What this PR does

Adds OpenBAO (open-source Vault fork) as a new managed PaaS application
in Cozystack.

**Structure follows existing app patterns (qdrant, nats):**
- System chart with vendored upstream `openbao/openbao` (chart v0.25.3,
appVersion v2.5.0)
- App chart with standalone/HA mode switching based on replicas count
- TLS via cert-manager self-signed certificates per instance
- ApplicationDefinition, PackageSource, PaaS bundle entry
- E2E test with init/unseal workflow

**Key design decisions:**
- `replicas: 1` → standalone mode with file storage; `replicas > 1` → HA
with Raft integrated storage and retry_join with TLS peer verification
- TLS enabled by default — each instance gets a self-signed Certificate
with DNS SANs covering services and pod addresses
- `disable_mlock = true` in HCL config since default security context
drops IPC_LOCK capability
- Injector and CSI provider disabled (cluster-scoped components, not
safe per-tenant)
- No auto-init/unseal — OpenBAO requires manual initialization by design
- E2E test performs full lifecycle: deploy, wait for certificate + API,
init, unseal, verify readiness, cleanup

### Release note

```release-note
[apps] Add OpenBAO as a managed secrets management service with standalone and HA Raft modes, TLS enabled by default
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Added OpenBAO managed secrets management service with
high-availability and standalone deployment options
  * Integrated monitoring and dashboards for operational visibility
  * Enabled configurable external access and web UI
  * Added automated snapshot backup capability

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-27 11:11:59 +01:00
Myasnikov Daniil
fd6d0c3603
(ci) Added extra debug commands for k8s startup
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-02-27 12:41:40 +05:00
Aleksei Sviridkin
8f1e52690d
test(e2e): fix kubernetes-previous retry failures
- Kill stale port-forward processes before starting a new one;
  on retries, the previous attempt's port-forward still holds the
  port, causing all kubectl commands to get "connection refused"
- Use -ge 2 instead of -eq 2 for node count check; MachineHealthCheck
  may create a 3rd VM, leading to 3 nodes joining the tenant cluster
  which would never satisfy the exact equality check
- Increase node join timeout from 5m to 8m; QEMU VMs with v1.34 need
  more time to boot and join when running after kubernetes-latest

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-20 03:23:50 +03:00
Aleksei Sviridkin
00ab6e792c
test(e2e): increase worker node join timeout to 5 minutes
When running kubernetes-latest and kubernetes-previous E2E tests
simultaneously, worker VMs compete for resources in the sandbox
environment. 3 minutes was insufficient for nodes to boot and
join the tenant cluster under load. Increase to 5 minutes.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-20 01:30:10 +03:00
Aleksei Sviridkin
87d0390256
fix(harbor): include tenant domain in default hostname and add E2E cleanup
Use tenant base domain in default hostname construction (harbor.RELEASE.DOMAIN)
to match the pattern used by other apps (kubernetes, vpn). Remove unused $ingress
variable from harbor.yaml. Add cleanup of stale resources from previous failed
E2E runs.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 02:07:38 +03:00
Aleksei Sviridkin
e7ffc21743
feat(harbor): switch registry storage to S3 via COSI BucketClaim
Replace PVC-based registry storage with S3 via COSI BucketClaim/BucketAccess.
The system chart parses BucketInfo secret and creates a registry-s3 Secret
with REGISTRY_STORAGE_S3_* env vars that override Harbor's ConfigMap values.

- Add bucket-secret.yaml to system chart (BucketInfo parser)
- Remove storageType/size from registry config (S3 is now the only option)
- Use Harbor's existingSecret support for S3 credentials injection
- Add objectstorage-controller to PackageSource dependencies
- Update E2E test with COSI bucket provisioning waits and diagnostics

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 01:23:02 +03:00
Aleksei Sviridkin
0f2ba5aba2
fix(harbor): add diagnostic output on E2E system HelmRelease timeout
Dump HelmRelease status, pods, events, and ExternalArtifact info
when harbor-test-system fails to become ready, to diagnose the
root cause of the persistent timeout.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:12 +03:00
Aleksei Sviridkin
490faaf292
fix(harbor): add operator dependencies, fix persistence rendering, increase E2E timeout
Add postgres-operator and redis-operator to PackageSource dependsOn
to ensure CRDs are available before Harbor system chart deploys.

Make persistentVolumeClaim conditional to avoid empty YAML mapping
when using S3 storage without Trivy.

Increase E2E system HelmRelease timeout from 300s to 600s to account
for CPNG + Redis + Harbor bootstrap time on QEMU.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:12 +03:00
Aleksei Sviridkin
cea57f62c8
[harbor] Make registry storage configurable: S3 or PVC
Add registry.storageType parameter (pvc/s3) to let users choose
between PVC storage and S3 via COSI BucketClaim. Default is pvc,
which works without SeaweedFS in the tenant namespace.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:12 +03:00
Aleksei Sviridkin
c815725bcf
[harbor] Fix E2E test: use correct HelmRelease name with prefix
ApplicationDefinition has prefix "harbor-", so CR name "harbor" produces
HelmRelease "harbor-harbor". Use name="test" and release="harbor-test"
to correctly reference all resources.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:12 +03:00
Aleksei Sviridkin
0c85639fed
[harbor] Move to apps/, use S3 via BucketClaim for registry storage
Move Harbor from packages/extra/ to packages/apps/ as it is a
self-sufficient end-user application, not a singleton tenant module.
Update bundle entry from system to paas accordingly.

Replace registry PVC storage with S3 via COSI BucketClaim/BucketAccess,
provisioned from the namespace's SeaweedFS instance. S3 credentials are
injected into the HelmRelease via valuesFrom with targetPath.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:12 +03:00
Aleksei Sviridkin
2dd3c03279
[harbor] Use CPNG and redis-operator instead of internal databases
Replace Harbor's internal PostgreSQL with CloudNativePG operator and
internal Redis with redis-operator (RedisFailover), following established
Cozystack patterns from seaweedfs and redis apps.

Additional fixes from code review:
- Fix registry resources nesting level (registry.registry/controller)
- Persist token CA across upgrades to prevent JWT invalidation
- Update values schema and ApplicationDefinition

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:11 +03:00
Aleksei Sviridkin
543ce6e5fd
[harbor] Add managed Harbor container registry application
Add Harbor v2.14.2 as a tenant-level managed service with per-component
resource configuration, ingress with TLS termination, and internal
PostgreSQL/Redis.

Includes:
- extra/harbor wrapper chart with HelmRelease, WorkloadMonitors, Ingress
- system/harbor with vendored upstream chart (helm.goharbor.io v1.18.2)
- harbor-rd ApplicationDefinition for dynamic CRD registration
- PackageSource and system.yaml bundle entry
- E2E test with Secret and Service verification

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:50:11 +03:00
Aleksei Sviridkin
dd4723386f
test(openbao): add E2E test for standalone mode
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-17 21:23:27 +03:00
Aleksei Sviridkin
a52da8dd8d
style(e2e): consistently quote kubeconfig variable references
Quote all tenantkubeconfig-${test_name} references in run-kubernetes.sh
for consistent shell scripting style. The only exception is line 195
inside a sh -ec "..." double-quoted string where inner quotes would
break the outer quoting.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-17 02:00:44 +03:00
Aleksei Sviridkin
315e5dc0bd
fix(e2e): make kubernetes test retries effective by cleaning up stale resources
When the kubernetes E2E test fails at the deployment wait step, set -eu
causes immediate exit before cleanup. On retry, kubectl apply outputs
"unchanged" for the stuck deployment, making retries 2 and 3 guaranteed
to fail against the same stuck pod.

Add pre-creation cleanup of backend deployment/service and NFS test
resources using --ignore-not-found, so retries start fresh. Also
increase the deployment wait timeout from 90s to 300s to handle CI
resource pressure, aligning with other timeouts in the same function.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-17 01:58:13 +03:00
Andrei Kvapil
2bc5e01fda
fix kubernetes e2e test for rwx volume
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-14 11:02:12 +01:00
Andrei Kvapil
dbba5c325b
fix kubernetes e2e test
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-14 10:13:09 +01:00
Andrei Kvapil
13aa341a28
fix(platform): address review comments in vm migration script
Replace `|| echo ""` with `|| true` to avoid newline bugs in variable
assignments. Switch `for x in $(cmd)` loops to `while read` for safer
iteration over kubectl output.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-14 03:02:54 +01:00
Andrei Kvapil
168f6f2445
fix(csi): address review feedback for kubevirt-csi-driver RWX support
- Move nil check before req dereference in CreateVolume
- Scope CiliumNetworkPolicy endpointSelector to specific VMI
- Use vmNamespace from NodeId for VMI lookup instead of infraNamespace
- Log PVC lookup errors in ControllerExpandVolume
- Wrap CNP ownerReference updates in retry.RetryOnConflict
- Fix infraClusterLabels validation to check runControllerService flag
- Dereference nodeName pointer in error message
- Replace panic with klog.Fatal for consistent error handling
- Honor CSI readonly flag in NFS NodePublishVolume
- Log mount list errors in isNFSMount
- Reorder Dockerfile ENTRYPOINT after COPY for better layer caching
- Add cleanup on e2e test failure and --wait on pod deletion

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-13 09:04:33 +01:00
Andrei Kvapil
46103400f2
test(e2e): adapt kubernetes NFS test for native RWX CSI support
Remove separate NFS Application dependency from e2e test. The kubevirt
CSI driver wrapper now handles RWX Filesystem volumes natively - PVCs
with ReadWriteMany accessMode use the standard kubevirt StorageClass.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-13 08:57:14 +01:00
Andrei Kvapil
9a86551e40
fix(e2e): correct s3Bucket reference in mariadb test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-12 15:18:06 +01:00
Andrei Kvapil
bce5300116
refactor: rename mysql application to mariadb
The mysql chart actually deploys MariaDB via mariadb-operator, but was
incorrectly named "mysql". Rename all references to use the correct
"mariadb" name across the codebase.

Changes:
- Rename packages/apps/mysql -> packages/apps/mariadb
- Rename packages/system/mysql-rd -> packages/system/mariadb-rd
- Rename platform source and bundle references
- Update CRD kind from MySQL to MariaDB
- Update RBAC, e2e tests, backup controller tests
- Keep real MySQL CLI/config tool names unchanged (mysqldump, [mysqld], etc.)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-12 15:15:20 +01:00
Andrei Kvapil
0260b15aaf
refactor(apps): remove FerretDB application (#2028)
## What this PR does

Remove the FerretDB managed application from Cozystack. This includes
the application Helm chart, resource definition, platform source, PaaS
bundle entry, RBAC clusterrole entry, and e2e test. Historical migration
scripts are left intact for upgrade compatibility.

### Release note

```release-note
[ferretdb] Removed FerretDB managed application
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Removed FerretDB managed database service and associated Helm chart,
documentation, and test components from the platform.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-12 09:30:01 +01:00