## What this PR does
Adds Harbor v2.14.2 as a managed tenant-level container registry service
in the PaaS bundle.
**Architecture:**
- Wrapper chart (`apps/harbor`) — HelmRelease, Ingress,
WorkloadMonitors, BucketClaim, dashboard RBAC
- Vendored upstream chart (`system/harbor`) from helm.goharbor.io
v1.18.2
- System chart (`system/harbor`) provisions PostgreSQL via CloudNativePG
and Redis via redis-operator
- ApplicationDefinition (`system/harbor-rd`) for dynamic `Harbor` CRD
registration
- PackageSource and paas.yaml bundle entry for platform integration
**Key design decisions:**
- Database and Redis provisioned via CPNG and redis-operator (not
internal Helm-based instances) for reliable day-2 operations
- Registry image storage uses S3 via COSI BucketClaim/BucketAccess from
namespace SeaweedFS
- Trivy vulnerability scanner cache uses PVC (S3 not supported by
vendored chart)
- Token CA key/cert persisted across upgrades via Secret lookup
- Per-component resource configuration (core, registry, jobservice,
trivy)
- Ingress with TLS via cert-manager, cloudflare issuer type handling,
proxy timeouts for large image pushes
- Auto-generated admin credentials persisted across upgrades
**E2E test:** Creates Harbor instance, verifies HelmRelease readiness,
deployment availability, credentials secret, service port, then cleans
up.
### Release note
```release-note
[harbor] Add managed Harbor container registry as a tenant-level service
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Harbor container registry deployment with integrated Kubernetes
support, including database and cache layers.
* Enabled metrics monitoring via Prometheus integration.
* Configured dashboard management interface for Harbor administration.
* **Tests**
* Added end-to-end testing for Harbor deployment and verification.
* **Chores**
* Integrated Harbor into the platform's application package bundle.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Replace pre-rendered static YAML application (`kubectl apply`) with
direct `helm upgrade --install` of the `packages/core/installer` chart
in E2E tests
- Remove CRD/operator artifact upload/download from CI workflow — the
chart with correct values is already present in the sandbox via
workspace copy and `pr.patch`
- Remove `copy-installer-manifest` Makefile target and its dependencies
## Test plan
- [ ] CI build job completes without uploading CRD/operator artifacts
- [ ] E2E `install-cozystack` step succeeds with `helm upgrade
--install`
- [ ] All existing E2E app tests pass
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* PR workflows now only keep the primary disk asset; publishing/fetching
of auxiliary operator and CRD artifacts removed.
* CRD manifests are produced by concatenation and a verify-crds check
was added to unit tests; file-write permissions for embedded manifests
tightened.
* **New Features**
* Operator can install CRDs at startup to ensure resources exist before
reconcile.
* E2E install now uses the chart-based installer flow.
* **Tests**
* Added comprehensive tests for CRD-install handling and manifest
writing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace PVC-based registry storage with S3 via COSI BucketClaim/BucketAccess.
The system chart parses BucketInfo secret and creates a registry-s3 Secret
with REGISTRY_STORAGE_S3_* env vars that override Harbor's ConfigMap values.
- Add bucket-secret.yaml to system chart (BucketInfo parser)
- Remove storageType/size from registry config (S3 is now the only option)
- Use Harbor's existingSecret support for S3 credentials injection
- Add objectstorage-controller to PackageSource dependencies
- Update E2E test with COSI bucket provisioning waits and diagnostics
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
In v0.41.x the tenant-root Namespace and HelmRelease were applied via
kubectl apply with no Helm release tracking. In v1.0 these resources
are managed by the cozystack-basics Helm release. Without proper Helm
ownership annotations the install of cozystack-basics fails because
the resources already exist.
Add migration 31 that annotates and labels both the Namespace and
HelmRelease so Helm can adopt them, matching the pattern established
in migrations 22 and 27.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add postgres-operator and redis-operator to PackageSource dependsOn
to ensure CRDs are available before Harbor system chart deploys.
Make persistentVolumeClaim conditional to avoid empty YAML mapping
when using S3 storage without Trivy.
Increase E2E system HelmRelease timeout from 300s to 600s to account
for CPNG + Redis + Harbor bootstrap time on QEMU.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move Harbor from packages/extra/ to packages/apps/ as it is a
self-sufficient end-user application, not a singleton tenant module.
Update bundle entry from system to paas accordingly.
Replace registry PVC storage with S3 via COSI BucketClaim/BucketAccess,
provisioned from the namespace's SeaweedFS instance. S3 credentials are
injected into the HelmRelease via valuesFrom with targetPath.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add Harbor v2.14.2 as a tenant-level managed service with per-component
resource configuration, ingress with TLS termination, and internal
PostgreSQL/Redis.
Includes:
- extra/harbor wrapper chart with HelmRelease, WorkloadMonitors, Ingress
- system/harbor with vendored upstream chart (helm.goharbor.io v1.18.2)
- harbor-rd ApplicationDefinition for dynamic CRD registration
- PackageSource and system.yaml bundle entry
- E2E test with Secret and Service verification
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Remove duplicate "Starting controller manager" log before install
phases, keep only the one before mgr.Start()
- Rename misleading test "document without kind returns error" to
"decoder rejects document without kind" to match actual behavior
- Document Helm uninstall CRD behavior in deployment template comment
- Use --health-probe-bind-address=0 consistently with metrics-bind
- Exclude all dotfiles in verify-crds diff, not just .gitattributes
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Use explicit apiextensions.k8s.io/v1 CRD GVK in waitForCRDsEstablished
instead of fragile objects[0].GroupVersionKind()
- Add TestInstall_crdNotEstablished for context timeout path
- Add --recursive to diff in verify-crds Makefile target
- Document why both crds/ and --install-crds exist in deployment template
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add --install-crds=true to cozystack-operator container args so the
operator applies embedded CRD manifests on startup via server-side apply.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The test-cluster target references non-existent hack/e2e-cluster.bats
file. Remove it and its dependency from the test target.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm installs crds/ contents before processing templates, resolving the
chicken-and-egg problem where PackageSource CR validation fails because
its CRD hasn't been registered yet.
- Move definitions/ to crds/ in the installer chart
- Remove templates/crds.yaml (Helm auto-installs from crds/)
- Update codegen script to write CRDs to crds/
- Replace helm template with cat for static CRD manifest generation
- Remove pre-apply CRD workaround from e2e test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
Add a new `cilium-kilo` networking variant that combines Cilium as the
CNI with Kilo as the WireGuard mesh overlay. This replaces the
standalone kilo PackageSource with a unified variant under the
networking source.
## Changes
- Add `cilium-kilo` variant to `networking.yaml` PackageSource with
proper component ordering and dependencies
- Add `values-kilo.yaml` for Cilium to disable host firewall when used
with Kilo
- Remove standalone `kilo.yaml` PackageSource (now integrated into
networking source)
- Switch Kilo image to official
`ghcr.io/cozystack/cozystack/kilo:v0.8.2`
- Remove unused `podCIDR`/`serviceCIDR` options and `--service-cidr`
flag from Kilo chart
Add a new networking variant that integrates Kilo with Cilium
pre-configured. Cilium is deployed with host firewall disabled and
enable-ipip-termination enabled, which are required for correct IPIP
encapsulation through Cilium's overlay.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kilo is now integrated into the cilium-kilo networking variant instead
of being a separate package that users install manually.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Extend the installer chart to support generic and hosted Kubernetes
deployments via the existing `cozystackOperator.variant` parameter
(introduced in #2034)
- For `variant=generic`: render ConfigMaps (`cozystack`,
`cozystack-operator-config`) and an optional Platform Package CR —
resources previously required to be created manually before deploying on
non-Talos clusters
- Add variant validation in `packagesource.yaml` to fail fast on typos
- Publish the installer chart as an OCI Helm artifact
## Motivation
Deploying Cozystack on generic Kubernetes (k3s, kubeadm, RKE2) currently
requires manually creating ConfigMaps before applying the rendered
operator manifest. This change makes the installer chart variant-aware
so that:
1. `helm template -s` workflow continues to produce correct rendered
manifests
2. `helm install --set cozystackOperator.variant=generic` becomes a
viable single-command deployment path for generic clusters
3. Required ConfigMaps and optional Platform Package CR are generated
from values, eliminating manual steps
## OCI Helm chart publishing
The installer chart is now packaged and pushed to the OCI registry as
part of the `image` build target via `make chart`. A `.helmignore` file
ensures only chart-relevant files are included in the published
artifact.
## Test plan
- [ ] `helm template` with `variant=talos` (default) renders: operator +
PackageSource
- [ ] `helm template` with `variant=generic` renders: operator + 2
ConfigMaps + PackageSource
- [ ] `helm template` with `variant=generic` + `platform.enabled=true`
renders: + Package CR
- [ ] `helm template` with `variant=hosted` renders: operator +
PackageSource
- [ ] Invalid variant value produces a clear error message
- [ ] `make manifests` generates all asset files
- [ ] `helm package` produces a clean chart without build artifacts
Add a new "cilium" variant to the kilo PackageSource that deploys kilo
with --compatibility=cilium flag. This enables Cilium-aware IPIP
encapsulation, routing outer packets through Cilium's VxLAN overlay
instead of the host network.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add migration script that converts VPC HelmRelease values from map
format to array format. The script discovers all VirtualPrivateCloud
HelmReleases, reads their values Secrets, and converts subnets using
yq. Idempotent: skips if subnets are already an array or null.
Bumps migration targetVersion from 30 to 31.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace `|| echo ""` with `|| true` to avoid newline bugs in variable
assignments. Switch `for x in $(cmd)` loops to `while read` for safer
iteration over kubectl output.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The virtual-machine application is replaced by separate vm-disk and
vm-instance applications. Migration 28 handles the conversion of
existing VirtualMachine HelmReleases.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Disable both cdi-operator and cdi-apiserver during DataVolume creation
to prevent CDI webhook from rejecting PVC adoption. Also delete mutating
webhook configurations alongside validating ones. Fix cloud-init value
serialization to avoid spurious newline. Add volumeMode detection from PV.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Splits virtual-machine HelmReleases into separate vm-disk and vm-instance
components. Handles PV data preservation, kube-ovn IP migration, and
LoadBalancer IP pinning via metallb annotation for external VMs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Fixes Helm upgrade failure for `monitoring-agents` caused by unrendered
`{{ .Release.Namespace }}` template expressions in `values.yaml`.
Introduces `global.target` parameter to control the target namespace for
monitoring services (vminsert, vlogs). Default is `cozy-system`,
platform bundle passes `tenant-root`. Uses `tpl` in vmagent template to
render URLs containing template expressions.
### Release note
```release-note
[monitoring] Fix YAML parse error in monitoring-agents vmagent template
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Enhanced monitoring agents configuration to use centralized
target-based values, replacing namespace-based identifiers for improved
consistency across multiple environments.
* Updated monitoring component labeling and URL handling to leverage
parametrized configuration approach for greater flexibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Renames the `mysql` application to `mariadb` across the codebase. The
chart has always deployed MariaDB (via mariadb-operator), but was
incorrectly named "mysql", causing confusion.
- Rename `packages/apps/mysql` → `packages/apps/mariadb` with all
internal references updated
- Rename `packages/system/mysql-rd` → `packages/system/mariadb-rd` with
CRD kind `MySQL` → `MariaDB`
- Update platform source, bundle references, RBAC, e2e tests, and backup
controller tests
- Add platform migration 27 to auto-discover and rename all deployed
MySQL resources to mariadb
- Preserves PVC data via PV claimRef rebinding
- Handles protection-webhook and mariadb-operator webhook during
migration
- Idempotent: safe to re-run
Real MySQL CLI/config tool names (`mysqldump`, `[mysqld]`, `MYSQL_*` env
vars) are intentionally left unchanged.
### Release note
```release-note
[apps] Rename mysql application to mariadb. A platform migration automatically renames all deployed MySQL resources. No user action required.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added automated migration capability from MySQL to MariaDB with data
persistence.
* **Tests**
* Added end-to-end tests for MariaDB deployment and validation.
* **Chores**
* Updated platform components and configuration to use MariaDB as the
default database solution.
* Migrated all internal references, documentation, and build
configurations from MySQL to MariaDB.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Migration 23 already removes the `cozystack-resource-definition-crd`
HelmRelease, but existing Helm release secrets
for `*-rd` applications still reference the deleted
`CozystackResourceDefinition` CRD, causing upgrade failures.
This adds a step to delete those stale Helm secrets so that Flux can
recreate the releases cleanly.
### Release note
```release-note
[platform] Clean up stale Helm release secrets for *-rd applications during migration
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Enhanced system cleanup routine to remove deprecated Kubernetes
secrets during version updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add platform migration that auto-discovers all deployed MySQL
HelmReleases and renames their resources to use the mariadb prefix.
The migration handles PVC data preservation via PV claimRef rebinding,
temporarily disables protection-webhook for protected resource deletion,
and scales mariadb-operator once for all instances.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The mysql chart actually deploys MariaDB via mariadb-operator, but was
incorrectly named "mysql". Rename all references to use the correct
"mariadb" name across the codebase.
Changes:
- Rename packages/apps/mysql -> packages/apps/mariadb
- Rename packages/system/mysql-rd -> packages/system/mariadb-rd
- Rename platform source and bundle references
- Update CRD kind from MySQL to MariaDB
- Update RBAC, e2e tests, backup controller tests
- Keep real MySQL CLI/config tool names unchanged (mysqldump, [mysqld], etc.)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Separates Piraeus CRD management from the `piraeus-operator` chart into
the dedicated `piraeus-operator-crds` chart. This prevents
`piraeus-operator` from deleting CRDs during upgrades.
Changes:
- Add `installCRDs` toggle to `piraeus-operator-crds` chart
- Update `piraeus-operator` Makefile to move CRDs template on chart
update
- Add migration 27 to reassign Helm ownership on existing Piraeus CRDs
and clean up stale Helm secrets
### Release note
```release-note
[linstor] Move CRDs installation logic to piraeus-operator-crds chart
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* CRD installation can now be controlled via configuration settings
(enabled by default).
* System version upgraded to 28.
* **Chores**
* Added migration script to handle system upgrade from version 27 to 28,
including metadata updates and configuration management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Remove the FerretDB managed application from Cozystack. This includes
the application Helm chart, resource definition, platform source, PaaS
bundle entry, RBAC clusterrole entry, and e2e test. Historical migration
scripts are left intact for upgrade compatibility.
### Release note
```release-note
[ferretdb] Removed FerretDB managed application
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed FerretDB managed database service and associated Helm chart,
documentation, and test components from the platform.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace unrendered Helm template expressions in values.yaml with
global.target parameter. Pass tenant-root from platform bundle,
default to cozy-monitoring. Use tpl in vmagent template to render URLs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migrate Helm ownership labels and annotations on Piraeus CRDs from
piraeus-operator to piraeus-operator-crds, and delete stale Helm
secrets to prevent piraeus-operator from removing CRDs on upgrade.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add -talos suffix to the default variant output file for consistency
with -generic and -hosted variants. Update all references in CI
workflows, e2e tests, upload scripts, and testing Makefile.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Consolidate three separate operator deployment templates (talos, generic,
hosted) into a single template with variant selection via values.yaml
parameter. This resolves conflicts when running `make apply` from the
installer package, which previously rendered all three deployments.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Remove the FerretDB managed application, its resource definition,
platform source, RBAC entry, and e2e test. Historical migration
scripts are left intact for upgrade compatibility.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Add a new system package `cluster-autoscaler` that supports multiple
cloud providers (Hetzner and Azure) for automatic node scaling of
Cozystack management cluster.
Key features:
- Single base package with provider-specific variants via PackageSource
- Auto-update capability from upstream Helm chart
- RBAC rules for leader election (coordination.k8s.io/leases)
- Comprehensive documentation for Hetzner setup with Talos Linux
The Hetzner documentation covers:
- Talos image creation via rescue mode snapshot
- vSwitch (private network) configuration
- Correct Talos machine config structure (nodeLabels, nodeIP,
cloud-provider)
- Testing with pod anti-affinity
- Troubleshooting common issues
- Kilo mesh networking integration
### Release note
```release-note
[system] Add cluster-autoscaler package with support for Hetzner Cloud and Azure providers
```
Cluster-autoscaler does not require privileged installation.
Remove the unnecessary privileged: true setting from both
Hetzner and Azure PackageSource definitions.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Enables the installation of the backupstrategy controller by default.
### Release note
```release-note
[backups] Install the backupstrategy controller by default.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Backup strategy component is now included by default in all system
installations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Enable insecure TLS verification for Keycloak communication to support
environments with self-signed certificates:
- **keycloak-configure**: Switch ClusterKeycloak to use internal service
URL (`http://keycloak-http.cozy-keycloak.svc:8080`) and enable
`insecureSkipVerify: true`
- **dashboard**: Add `--ssl-insecure-skip-verify=true` flag to
oauth2-proxy
### Release note
```release-note
[keycloak-configure,dashboard] Enable insecure TLS verification for Keycloak by default to support self-signed certificates
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added an option to skip OIDC TLS certificate verification (disabled by
default).
* Authentication proxy can now be configured to omit SSL verification
when enabled.
* Keycloak connection switched to a non-TLS endpoint and TLS
verification handling updated.
* Removed the Keycloak authorization-services toggle from configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Enables the installation of the backupstrategy controller by default.
### Release note
```release-note
[backups] Install the backupstrategy controller by default.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
