## What this PR does
Adds Harbor v2.14.2 as a managed tenant-level container registry service
in the PaaS bundle.
**Architecture:**
- Wrapper chart (`apps/harbor`) — HelmRelease, Ingress,
WorkloadMonitors, BucketClaim, dashboard RBAC
- Vendored upstream chart (`system/harbor`) from helm.goharbor.io
v1.18.2
- System chart (`system/harbor`) provisions PostgreSQL via CloudNativePG
and Redis via redis-operator
- ApplicationDefinition (`system/harbor-rd`) for dynamic `Harbor` CRD
registration
- PackageSource and paas.yaml bundle entry for platform integration
**Key design decisions:**
- Database and Redis provisioned via CPNG and redis-operator (not
internal Helm-based instances) for reliable day-2 operations
- Registry image storage uses S3 via COSI BucketClaim/BucketAccess from
namespace SeaweedFS
- Trivy vulnerability scanner cache uses PVC (S3 not supported by
vendored chart)
- Token CA key/cert persisted across upgrades via Secret lookup
- Per-component resource configuration (core, registry, jobservice,
trivy)
- Ingress with TLS via cert-manager, cloudflare issuer type handling,
proxy timeouts for large image pushes
- Auto-generated admin credentials persisted across upgrades
**E2E test:** Creates Harbor instance, verifies HelmRelease readiness,
deployment availability, credentials secret, service port, then cleans
up.
### Release note
```release-note
[harbor] Add managed Harbor container registry as a tenant-level service
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Harbor container registry deployment with integrated Kubernetes
support, including database and cache layers.
* Enabled metrics monitoring via Prometheus integration.
* Configured dashboard management interface for Harbor administration.
* **Tests**
* Added end-to-end testing for Harbor deployment and verification.
* **Chores**
* Integrated Harbor into the platform's application package bundle.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add encoding, localeCollate, and localeCType to initdb bootstrap
configuration to ensure fulltext search (ilike) works correctly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This PR adds the changelog for release `v1.0.0-beta.6`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.0.0-beta.6.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Cilium-Kilo networking variant for enhanced multi-location setup
* NATS monitoring dashboards with Prometheus integration
* DNS validation for Application names
* Operator auto-installation of CRDs at startup
* **Bug Fixes**
* Fixed HelmRelease adoption during platform migration
* Improved test reliability and timeout handling
* **Documentation**
* Enhanced Azure autoscaling troubleshooting guidance
* Updated multi-location configuration docs
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR prepares the release `v1.0.0-beta.6`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated container image references from v1.0.0-beta.5 to v1.0.0-beta.6
across the platform, including updates to operators, controllers,
dashboards, storage components, and other services with corresponding
digest updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Specify initdb bootstrap with database and owner names explicitly
instead of relying on CNPG defaults which may change between versions.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Follow the same hostname pattern as kubernetes and vpn apps: use
Release.Name directly (which already includes the harbor- prefix)
instead of adding an extra harbor. subdomain.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Change sslmode from disable to require for CNPG PostgreSQL connection,
as CNPG supports TLS out of the box. Fix token key/cert preservation to
verify both values are present before passing to Harbor core.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Use tenant base domain in default hostname construction (harbor.RELEASE.DOMAIN)
to match the pattern used by other apps (kubernetes, vpn). Remove unused $ingress
variable from harbor.yaml. Add cleanup of stale resources from previous failed
E2E runs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
- Replace pre-rendered static YAML application (`kubectl apply`) with
direct `helm upgrade --install` of the `packages/core/installer` chart
in E2E tests
- Remove CRD/operator artifact upload/download from CI workflow — the
chart with correct values is already present in the sandbox via
workspace copy and `pr.patch`
- Remove `copy-installer-manifest` Makefile target and its dependencies
## Test plan
- [ ] CI build job completes without uploading CRD/operator artifacts
- [ ] E2E `install-cozystack` step succeeds with `helm upgrade
--install`
- [ ] All existing E2E app tests pass
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* PR workflows now only keep the primary disk asset; publishing/fetching
of auxiliary operator and CRD artifacts removed.
* CRD manifests are produced by concatenation and a verify-crds check
was added to unit tests; file-write permissions for embedded manifests
tightened.
* **New Features**
* Operator can install CRDs at startup to ensure resources exist before
reconcile.
* E2E install now uses the chart-based installer flow.
* **Tests**
* Added comprehensive tests for CRD-install handling and manifest
writing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace PVC-based registry storage with S3 via COSI BucketClaim/BucketAccess.
The system chart parses BucketInfo secret and creates a registry-s3 Secret
with REGISTRY_STORAGE_S3_* env vars that override Harbor's ConfigMap values.
- Add bucket-secret.yaml to system chart (BucketInfo parser)
- Remove storageType/size from registry config (S3 is now the only option)
- Use Harbor's existingSecret support for S3 credentials injection
- Add objectstorage-controller to PackageSource dependencies
- Update E2E test with COSI bucket provisioning waits and diagnostics
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[nats] add monitoring
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added Grafana dashboards for NATS JetStream and Server monitoring
* Added support for specifying container image digests and full image
names
* **Documentation**
* Enhanced NATS Helm chart documentation with container resource
configuration guidance
* **Chores**
* Updated NATS application version and component image versions
* Improved Kubernetes graceful shutdown and Prometheus exporter
configuration
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds migration 31 to adopt existing `tenant-root` Namespace and
HelmRelease
into the `cozystack-basics` Helm release during upgrade from v0.41.x to
v1.0.
In v0.41.x these resources were applied via `kubectl apply` (from the
platform
chart `apps.yaml`) with no Helm release tracking. In v1.0 they are
created by
the `cozystack-basics` chart. Without Helm ownership annotations, `helm
install`
of `cozystack-basics` fails because the resources already exist.
The migration adds:
- `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace`
annotations
for Helm adoption
- `app.kubernetes.io/managed-by: Helm` label
- `helm.sh/resource-policy: keep` annotation on the HelmRelease to
prevent
accidental deletion
- `sharding.fluxcd.io/key: tenants` label required by flux sharding in
v1.0
This follows the same pattern as migrations 22 and 27 (CRD adoption).
Related: #2063
### Release note
```release-note
[platform] adopt tenant-root resources into cozystack-basics Helm release during migration
```
In v0.41.x the tenant-root Namespace and HelmRelease were applied via
kubectl apply with no Helm release tracking. In v1.0 these resources
are managed by the cozystack-basics Helm release. Without proper Helm
ownership annotations the install of cozystack-basics fails because
the resources already exist.
Add migration 31 that annotates and labels both the Namespace and
HelmRelease so Helm can adopt them, matching the pattern established
in migrations 22 and 27.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Multiple Harbor instances in the same namespace would get the same
default hostname when derived from namespace host. Use Release.Name
instead for unique hostnames per instance.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The vendored Harbor chart does an unsafe `lookup` of the Redis auth
Secret at template rendering time to extract the password. On first
install, the Secret doesn't exist yet (created by the same chart),
causing a nil pointer error. Failed installs are rolled back, deleting
the Secret, so retries also fail — creating an infinite failure loop.
Fix by generating the Redis password in the wrapper chart (same
pattern as admin password), storing it in the credentials Secret,
and injecting it via HelmRelease valuesFrom with targetPath. This
bypasses the vendored chart's lookup entirely — it uses the password
value directly instead of looking up the Secret.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Dump HelmRelease status, pods, events, and ExternalArtifact info
when harbor-test-system fails to become ready, to diagnose the
root cause of the persistent timeout.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add postgres-operator and redis-operator to PackageSource dependsOn
to ensure CRDs are available before Harbor system chart deploys.
Make persistentVolumeClaim conditional to avoid empty YAML mapping
when using S3 storage without Trivy.
Increase E2E system HelmRelease timeout from 300s to 600s to account
for CPNG + Redis + Harbor bootstrap time on QEMU.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add registry.storageType parameter (pvc/s3) to let users choose
between PVC storage and S3 via COSI BucketClaim. Default is pvc,
which works without SeaweedFS in the tenant namespace.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ApplicationDefinition has prefix "harbor-", so CR name "harbor" produces
HelmRelease "harbor-harbor". Use name="test" and release="harbor-test"
to correctly reference all resources.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add | quote to host values in ingress for proper YAML escaping.
Add cloudflare issuer type handling following bucket/dashboard pattern.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move Harbor from packages/extra/ to packages/apps/ as it is a
self-sufficient end-user application, not a singleton tenant module.
Update bundle entry from system to paas accordingly.
Replace registry PVC storage with S3 via COSI BucketClaim/BucketAccess,
provisioned from the namespace's SeaweedFS instance. S3 credentials are
injected into the HelmRelease via valuesFrom with targetPath.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace Harbor's internal PostgreSQL with CloudNativePG operator and
internal Redis with redis-operator (RedisFailover), following established
Cozystack patterns from seaweedfs and redis apps.
Additional fixes from code review:
- Fix registry resources nesting level (registry.registry/controller)
- Persist token CA across upgrades to prevent JWT invalidation
- Update values schema and ApplicationDefinition
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Quote resourceNames values starting with {{ to prevent YAML parser
from interpreting them as flow mappings.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add Harbor v2.14.2 as a tenant-level managed service with per-component
resource configuration, ingress with TLS termination, and internal
PostgreSQL/Redis.
Includes:
- extra/harbor wrapper chart with HelmRelease, WorkloadMonitors, Ingress
- system/harbor with vendored upstream chart (helm.goharbor.io v1.18.2)
- harbor-rd ApplicationDefinition for dynamic CRD registration
- PackageSource and system.yaml bundle entry
- E2E test with Secret and Service verification
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
CollectCRDNames now requires both apiVersion "apiextensions.k8s.io/v1"
and kind "CustomResourceDefinition", consistent with the validation in
crdinstall.Install.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Remove duplicate "Starting controller manager" log before install
phases, keep only the one before mgr.Start()
- Rename misleading test "document without kind returns error" to
"decoder rejects document without kind" to match actual behavior
- Document Helm uninstall CRD behavior in deployment template comment
- Use --health-probe-bind-address=0 consistently with metrics-bind
- Exclude all dotfiles in verify-crds diff, not just .gitattributes
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The workload reconciler was refactored to use the label
workloads.cozystack.io/monitor but the test still used the old
workloadmonitor.cozystack.io/name label, causing the reconciler to
delete the workload instead of keeping it.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Check both apiVersion and kind when validating embedded CRD manifests
to prevent applying objects with wrong API group
- Move ctrl.SetupSignalHandler() before install phases so CRD and Flux
installs respect SIGTERM instead of blocking for up to 2 minutes
- Replace custom contains/searchString helpers with strings.Contains
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Validate that all parsed objects are CustomResourceDefinition before
applying with force server-side apply. This prevents accidental
application of arbitrary resources if a non-CRD file is placed in
the manifests directory.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move duplicated YAML parsing (ReadYAMLObjects, ParseManifestFile) and
CRD readiness check (WaitForCRDsEstablished, CollectCRDNames) into a
shared internal/manifestutil package. Both crdinstall and fluxinstall
now import from manifestutil instead of maintaining identical copies.
Replace fluxinstall's time.Sleep(2s) after CRD apply with proper
WaitForCRDsEstablished polling, matching the crdinstall behavior.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Use explicit apiextensions.k8s.io/v1 CRD GVK in waitForCRDsEstablished
instead of fragile objects[0].GroupVersionKind()
- Add TestInstall_crdNotEstablished for context timeout path
- Add --recursive to diff in verify-crds Makefile target
- Document why both crds/ and --install-crds exist in deployment template
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add verify-crds target that diffs packages/core/installer/crds/ and
internal/crdinstall/manifests/ to catch accidental divergence. Include
it in unit-tests target so it runs in CI.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Wait for CRDs to have Established condition after server-side apply,
instead of returning immediately
- Add TestInstall with fake client and interceptor to simulate CRD
establishment
- Add TestInstall_noManifests and TestInstall_writeManifestsFails for
error paths
- Fix fluxinstall/manifests.embed.go: use filepath.Join for OS paths
and restrict permissions from 0666 to 0600 (same fix as crdinstall)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Use filepath.Join instead of path.Join for OS file paths
- Restrict extracted manifest permissions from 0666 to 0600
- Add unit tests for readYAMLObjects, parseManifests, and
WriteEmbeddedManifests including permission verification
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
After generating CRDs to packages/core/installer/crds/, copy them to
internal/crdinstall/manifests/ so the operator binary embeds the latest
CRD definitions.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add --install-crds=true to cozystack-operator container args so the
operator applies embedded CRD manifests on startup via server-side apply.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Embed Package and PackageSource CRDs in the operator binary using Go
embed, following the same pattern as --install-flux. The operator applies
CRDs at startup using server-side apply, ensuring they are updated on
every operator restart/upgrade.
This addresses the CRD lifecycle concern: Helm crds/ directory handles
initial install, while the operator manages updates on subsequent
deployments.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The test-cluster target references non-existent hack/e2e-cluster.bats
file. Remove it and its dependency from the test target.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm installs crds/ contents before processing templates, resolving the
chicken-and-egg problem where PackageSource CR validation fails because
its CRD hasn't been registered yet.
- Move definitions/ to crds/ in the installer chart
- Remove templates/crds.yaml (Helm auto-installs from crds/)
- Update codegen script to write CRDs to crds/
- Replace helm template with cat for static CRD manifest generation
- Remove pre-apply CRD workaround from e2e test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm cannot validate PackageSource CR during install because the CRD
is part of the same chart. Pre-apply CRDs via helm template + kubectl
apply --server-side before running helm upgrade --install.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
Add a new `cilium-kilo` networking variant that combines Cilium as the
CNI with Kilo as the WireGuard mesh overlay. This replaces the
standalone kilo PackageSource with a unified variant under the
networking source.
## Changes
- Add `cilium-kilo` variant to `networking.yaml` PackageSource with
proper component ordering and dependencies
- Add `values-kilo.yaml` for Cilium to disable host firewall when used
with Kilo
- Remove standalone `kilo.yaml` PackageSource (now integrated into
networking source)
- Switch Kilo image to official
`ghcr.io/cozystack/cozystack/kilo:v0.8.2`
- Remove unused `podCIDR`/`serviceCIDR` options and `--service-cidr`
flag from Kilo chart
Switch kilo image to official ghcr.io/cozystack/cozystack/kilo:v0.8.2,
remove unnecessary enable-ipip-termination from cilium-kilo values,
and update platform source digest.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The podCIDR was incorrectly passed as --service-cidr to prevent
masquerading on pod traffic. This is unnecessary for multi-location
mesh and was a leftover from single-cluster assumptions.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The --service-cidr flag prevents masquerading for service IPs, but
service CIDRs are cluster-local and not useful for multi-location
mesh routing. Remove the serviceCIDR value and its template usage.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
