## What this PR does
Documentation: https://github.com/cozystack/website/pull/517
Integrates [HAMi](https://github.com/Project-HAMi/HAMi) v2.8.1 (CNCF
Sandbox) into Cozystack as a system-level package for fractional GPU
sharing in tenant Kubernetes clusters.
The integration covers three layers:
- **System chart** (`packages/system/hami/`): Vendors upstream HAMi Helm
chart with device plugin, scheduler extender, mutating webhook, and
RuntimeClass configuration. The broken DRA subchart was removed — it
renders resources even when disabled and references unpublished images.
- **Kubernetes app addon** (`packages/apps/kubernetes/`): HAMi exposed
as an optional toggle (`hami.enabled`). Automatically disables GPU
Operator's native device plugin when active to avoid conflicts. Enforces
hard dependency on GPU Operator.
- **Platform registration** (`packages/core/platform/`): HAMi declared
as PackageSource with gpu-operator dependency, included in the iaas
bundle.
**Known limitation**: HAMi-core relies on a private glibc symbol
(`_dl_sym`) removed in glibc 2.34, which breaks compute isolation on
modern container images (Ubuntu 22.04+, PyTorch/TensorFlow official
images). Alpine/musl is entirely incompatible. See the package README
for details and upstream issue references.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(hami): add HAMi GPU virtualization as an optional system package for fractional GPU sharing. Enables memory and compute isolation for NVIDIA GPUs across tenant workloads. Integrates with GPU Operator and can be enabled per-cluster via the hami.enabled toggle. Note: compute isolation requires glibc < 2.34 in workload containers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added HAMi GPU virtualization middleware as an optional cluster addon
to enable fractional GPU sharing.
* HAMi addon includes an enable toggle (default: disabled) and a
customizable Helm values override; when enabled it enforces GPU Operator
presence and adjusts GPU Operator values accordingly.
* New HelmRelease integration for deploying HAMi with dependency
ordering and conditional rendering.
* **Documentation**
* Added HAMi setup guide, compatibility notes, and updated cluster addon
configuration reference.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
In environments where control-plane nodes are not available (e.g., AWS
managed nodes, Cozy-in-Cozy tenant Kubernetes clusters), the DaemonSet
cannot schedule any pods because no nodes match this selector.
## What this PR does
This PR adds ability to replace DaemonSet by Deployment with custom
number of replicas for HA purpose. Default values saves current
behavior.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Optional Deployment mode for lineage-controller-webhook with
configurable replicas
* PodDisruptionBudget support when Deployment mode is enabled
* **Updates**
* Improved pod scheduling: node affinity replaces nodeSelector, optional
tolerations, and pod anti-affinity when using Deployment
* Clarified behavior for local API endpoint scheduling (only applies
when pod lands on nodes running an API server)
* **Documentation**
* Added README with usage examples and parameter details
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The `update:` recipe now reproduces the top-level vendoring overrides
(remove broken hami-dra subchart, clear Chart.yaml dependencies, strip
dra/hami-dra/podSecurityPolicy from upstream values.yaml) after
`helm pull`, so they no longer silently disappear on the next bump.
Template-level patches (DRA guards in scheduler/*, indent fix in
device-plugin/monitorservice.yaml) are documented with rationale and
commit references — they remain a manual step because automating them
would be fragile across upstream template restructures.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The HelmRelease at packages/apps/kubernetes/templates/helmreleases/hami.yaml
references chartRef.name 'cozystack-kubernetes-application-kubevirt-kubernetes-hami',
but that component was missing from the kubernetes-application PackageSource.
The HelmRelease would sit in Stalled: ArtifactNotFound at install time.
Add kubernetes-hami next to kubernetes-gpu-operator under variant kubevirt,
mirroring the existing pattern for every other tenant-cluster addon. The
standalone cozystack.hami PackageSource is retained — same shape as
gpu-operator, which is registered both standalone (for bundles/iaas) and
as a kubernetes-application component (for tenant-cluster HelmReleases).
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Match the pattern used by every other tenant-cluster addon HelmRelease
(gpu-operator, cilium, fluxcd, ...): only render when the upstream etcd
namespace is ready, not just when the addon is enabled. Add values-ci.yaml
to the HAMi test suite so the new gate has the fixture it needs.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The HAMi-driven default of disabling gpu-operator's device plugin is
applied via valuesOverride and can be re-enabled by users for advanced
topologies (mixed HAMi / vanilla NVIDIA device plugin pools). Document
this explicitly so the README aligns with the existing template merge
order.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
HAMi#173 was closed as "not planned" and only suggests a typo fix
(< 2.3.0 → < 2.30); it does not establish the actual 2.34 boundary.
HAMi-core#174 (symbol-level cause) and HAMi#1190 (empirical per-glibc
behavior) cover the same ground accurately.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The auto-generated changelog only listed #2459/#2467 (velero-configmap
Role move). Five additional PRs were backported and merged into
release-1.3 between then and the v1.3.1 tag (2026-04-28):
- #2471/#2524 - fix(api): IDOR in TenantNamespace Get/Watch
- #2496/#2505 - feat(linstor): linstor-csi v1.10.6 (Protocol-C dual-attach)
- #2462/#2511 - fix(etcd): remove destructive post-upgrade hook
- #2421/#2491 - fix(kamaji): memory limits + startup probe
- #2498/#2518 - build(linstor): wire linstor-gui into root build target
Update the release date to match the actual tag (2026-04-28), rewrite
the intro paragraph, and add @kvaps to contributors.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The AI-generated v1.3.1 changelog (#2480) was generated from
git log v1.3.0..main rather than git log v1.3.0..v1.3.1, because
the workflow checked out main while the v1.3.1 tag points to
release-1.3. As a result the changelog included:
- 8 PRs that were merged to main but never shipped in v1.3.1
- 6 backport PRs that were merged to release-1.3 *after* v1.3.1
was tagged
- Both originals and their backports as separate duplicate entries
- A 2024 PR (#435) that has nothing to do with this range
- Generic "Documentation updates" placeholders for website entries
- Title duplicated as both the bold label and the description
(`* **fix(...): X**: fix(...): X (...)`)
- The cozystack-ci bot listed as a human contributor
The actual v1.3.1 release range (v1.3.0..v1.3.1) contains exactly
one user-facing change:
41bcb0be [Backport release-1.3] fix(backups): move
velero-configmap Role to velero chart (#2467)
which is the backport of #2459 (myasnikovdaniil) shipped via
#2467 (IvanHunters). This commit replaces the contents of
docs/changelogs/v1.3.1.md with that one entry, the matching
two-person contributors list, and the standard footer.
The workflow + docs fixes that prevent this regression for future
patch releases will land in a separate PR against main.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds GPU observability to Cozystack: five Grafana dashboards, a set of
VictoriaMetrics recording rules with a throttle-regression alert, and
reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter
on Talos when running GPU workloads directly in pods.
### Components
1. **Dashboards** — `dashboards/gpu/*.json`, registered in
`packages/system/monitoring/dashboards-infra.list`. All follow the
project convention: the Prometheus data source is selected through the
`$ds_prometheus` template variable.
- `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics
engine, memory copy), VRAM, power, temperature, and health (XID errors,
power/thermal throttling). All metrics are shown per-GPU, filterable by
the `$Hostname` selector.
- `gpu-fleet` — cluster-wide inventory, capacity, utilization, and
per-node power draw as percentage of TDP.
- `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh
accounting; utilization and power are shown per-node (DCGM cannot
attribute hardware metrics to namespaces).
- `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a
per-namespace requests/limits table. All counts derive from the
`namespace:gpu_count:allocated` recording rule, which excludes
`Failed`/`Succeeded` pod carryover.
- `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation,
util-per-watt, and per-GPU power/thermal throttle fractions.
2. **Recording rules and alert** —
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`.
Four rule groups:
- `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview
panels.
- `gpu.recording.node.1m` — per-node hardware aggregates (grouped by
`Hostname`) plus per-namespace allocation counts from kube resource
requests; filters `cozy-*`/`kube-*`.
- `gpu.recording.efficiency.1m` — per-GPU tensor saturation and
util-per-watt, plus per-GPU power/thermal throttle fractions bounded via
`clamp_max(..., 1)`.
- `gpu.recording.throttle.validation.5m` — houses the
`GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp
throttle fraction exceeds 1.0. This guards against DCGM counter-unit
drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10
with DCGM 3.x.
Rules are safe on clusters without DCGM — they evaluate to empty series
when no matching metrics are scraped.
3. **Reference manifests** under
`packages/system/gpu-operator/examples/`. These are **not** templates;
they document one working configuration:
- `values-native-talos.yaml` — Cozystack Package values for the
native-pod scenario.
- `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with
profiling, ECC, throttling, and energy counters used by the dashboards
and recording rules.
- `nvidia-driver-compat.yaml` — DaemonSet that stages
`libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a
location the GPU Operator validator inspects. Workaround for
[NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687).
Requires `pod-security.kubernetes.io/enforce: privileged` on the target
namespace when the cluster enforces baseline/restricted
PodSecurityStandards.
- `README.md` — explains the two deployment paths (sandbox vs. native),
which DCGM metrics each dashboard depends on, and how the pieces fit
together.
4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats`
enforces that every recording-rule reference in tracked dashboards
resolves, and that every DCGM metric referenced by the rules is declared
in either the upstream default CSV snapshot
(`hack/dcgm-default-counters.csv`) or the custom CSV shipped under
`examples/`.
### Why reference manifests, not templates
The out-of-the-box `packages/system/gpu-operator/values-talos.yaml`
targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via
VFIO, driver disabled on the host, device plugin disabled. That is the
sensible default for Cozystack today.
Running GPU workloads directly in pods on Talos is also possible but
requires a different configuration that makes assumptions not every user
shares:
- the NVIDIA Talos system extension must be installed on GPU nodes,
- the operator's own driver and toolkit components must be disabled
because the extension provides them,
- the validator's hardcoded paths need a workaround until
NVIDIA/gpu-operator#1687 lands upstream.
Shipping those as active templates would silently impose those
assumptions. Shipping them as reference files next to the package lets
operators opt in with eyes open and understand the moving parts before
applying them.
### Scope note
The dashboards and recording rules are useful independently of the
reference manifests — any cluster that scrapes DCGM Exporter (however it
is installed) will populate them. The reference manifests are
self-contained and do not change any default behavior of the
`gpu-operator` package.
### Screenshots
<img width="2824" height="1055" alt="2026-04-27_12-36-52"
src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a"
/>
<img width="2913" height="1255" alt="2026-04-27_12-36-39"
src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229"
/>
<img width="2932" height="1251" alt="2026-04-27_12-36-30"
src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23"
/>
<img width="2864" height="1222" alt="2026-04-27_12-36-00"
src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9"
/>
<img width="2958" height="1226" alt="image"
src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76"
/>
### Release note
```release-note
feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added five GPU dashboards for performance, efficiency, fleet overview,
quotas/allocation, and tenant usage (utilization, power, throttling,
allocation, billing).
* Added cluster recording rules and a throttling validation alert; new
Prometheus-derived metrics and per-GPU rankings/timeseries.
* **Documentation**
* Added GPU Operator examples and README documenting native Talos setup
and required DCGM counters.
* **Tests**
* Added validation tests ensuring dashboard ↔ recording-rule and DCGM
metric consistency.
* **Chores**
* Registered new GPU dashboards in infra listings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Tests need _namespace.etcd from values-ci.yaml after the merge
introduced an etcd-namespace guard on the gpu-operator HelmRelease
condition.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## Summary
Fixed two IDOR (Insecure Direct Object Reference) vulnerabilities in the
TenantNamespace API handlers that allowed authenticated users to access
metadata of tenant namespaces without proper authorization checks.
## Changes
### New optimized function: hasAccessToNamespace()
- Lists RoleBindings **only in the target namespace** instead of all
cluster RoleBindings
- Used by Get() and Watch() for single-namespace access checks
- Order of magnitude faster than the previous approach
### Get() handler
- Uses `hasAccessToNamespace()` for efficient authorization
- Returns `NotFound` instead of `Forbidden` to prevent tenant
enumeration
- Now correctly enforces RoleBinding-based access control
### Watch() handler
- Uses `hasAccessToNamespace()` for per-event authorization
- Logs authorization errors with klog for security audit
- Events for unauthorized namespaces are silently filtered out
- Ensures users only receive watch events for namespaces they have
access to
### Additional fixes
- Fixed ServiceAccount subject handling when namespace is empty
(defaults to RoleBinding namespace)
- Added proper error logging in Watch handler
## Performance Impact
| Handler | Before | After |
|---------|--------|-------|
| List() | List all RoleBindings × 1 | No change ✅ |
| Get() | List all RoleBindings × 1 | List RoleBindings in 1 namespace 🚀
|
| Watch() | List all RoleBindings × N events | List RoleBindings in 1
namespace × N events 🚀 |
**For Watch with 100 events:**
- Before: 100 × (all cluster RoleBindings) = catastrophic
- After: 100 × (1-5 RoleBindings in namespace) = fast + cached by
controller-runtime
## Security Impact
**Before**: Any authenticated user could:
- Read metadata (labels, annotations, creation time) of any `tenant-*`
namespace via `Get()`
- Stream all tenant namespace events via `Watch()`, including
creation/modification/deletion
**After**: Users can only access tenant namespaces they have explicit
RoleBindings for, matching the behavior of the `List()` handler.
## Testing
Manually verified:
- Users can only `get` their own tenant namespaces
- Users can only `watch` events for their own tenant namespaces
- Unauthorized access returns `NotFound` (not `Forbidden`) to prevent
enumeration
- `List()` behavior remains unchanged and consistent with `Get()` and
`Watch()`
- Authorization errors are logged for security audit
## Checklist
- [x] Code follows project style and conventions
- [x] Security vulnerability is fully mitigated
- [x] Authorization logic is consistent across List/Get/Watch handlers
- [x] Performance optimized per feedback from @timofei.larkin
- [x] Error logging added for security audit
- [ ] Unit tests added (can be done in follow-up PR)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enforced per-namespace access control for Get and Watch; inaccessible
namespaces return Forbidden or NotFound as appropriate
* Forwarded field/label selectors to upstream watches and added
defensive filtering to skip inaccessible events (logged)
* Improved ServiceAccount subject namespace fallback and
privileged-group bypass
* **Tests**
* Added security tests covering RoleBinding subjects, groups, privileged
bypasses, ServiceAccounts, and access-denied behaviors
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Add `serverName` field to bootstrap configuration to support backup
recovery when the Barman server name in `backup.info` differs from the
Kubernetes cluster name.
This fixes "no target backup found" errors during CloudNativePG recovery
operations.
## Problem
CloudNativePG forms the backup path as `destinationPath + "postgres-" +
oldName`, but searches for backups using the `server_name` field from
`backup.info`. When these values don't match (e.g., cluster name is
`grafana` but `server_name` is `cloud`), recovery fails with "no target
backup found".
## Solution
- Add optional `serverName` parameter to `bootstrap` configuration
- When specified, CloudNativePG uses this value to search for backups in
S3
- Falls back to `oldName` when `serverName` is not provided (backwards
compatible)
## Changes
- Add `ServerName` field to PostgreSQL CRD type definition
- Add conditional `serverName` to Cluster `externalClusters` template
- Update `values.yaml` and README.md with `serverName` documentation
- Regenerate `values.schema.json` and `postgres.yaml` CRD
## Test plan
- [ ] Deploy postgres cluster with backups enabled
- [ ] Create backup
- [ ] Create new postgres instance with `bootstrap.enabled=true`,
`bootstrap.oldName=<original-name>`, and
`bootstrap.serverName=<value-from-backup.info>`
- [ ] Verify recovery completes successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an optional bootstrap.serverName setting to specify the Barman
server name from the original cluster’s backup.info (used when that
server name differs from the Kubernetes cluster name).
* **Documentation**
* Clarified that bootstrap.oldName must match the serverName value
recorded in backup.info; updated docs and examples to reflect the new
bootstrap.serverName option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an opt-in `publishing.exposure=loadBalancer` mode for the
ingress-nginx Service as a migration path away from
`Service.spec.externalIPs`, which is deprecated upstream in Kubernetes
v1.36
([KEP-5707](https://github.com/kubernetes/enhancements/issues/5707),
[kubernetes#137293](https://github.com/kubernetes/kubernetes/pull/137293)).
The `AllowServiceExternalIPs` feature gate is expected to default to off
around v1.40 and the implementation to be removed around v1.43.
Stacked on top of #2464 (cilium v1.19.3 bump) — depends on it for the
`CiliumLoadBalancerIPPool` at `cilium.io/v2`.
### Behavior
- New platform value `publishing.exposure` — enum `externalIPs |
loadBalancer`, default `externalIPs` (current behavior unchanged on
upgrade).
- Plumbed through `cozystack-values` into each tenant's ingress
HelmRelease via the new `expose-mode` key.
- When `exposure=loadBalancer` and the current namespace matches
`publishing.ingressName`, the Service becomes `type: LoadBalancer` with
`externalTrafficPolicy: Local` and a `CiliumLoadBalancerIPPool`
announces the addresses from `publishing.externalIPs` via Cilium LB
IPAM.
- The pool uses a namespace-only `serviceSelector`
(`io.kubernetes.service.namespace: <ns>`) — any LoadBalancer Service in
the tenant namespace draws from it. See "Pool ownership" below.
- IPv4 addresses get `/32` CIDRs, IPv6 addresses get `/128`. Mixed
families supported. Pre-CIDR entries (`192.0.2.10/32`) are accepted
without double-suffixing.
- Unknown values, `loadBalancer` with empty externalIPs, and stray empty
entries from `publishing.externalIPs` are rejected at render time with
explicit error messages.
### Pool ownership
The `CiliumLoadBalancerIPPool` is rendered from
`packages/apps/tenant/templates/cilium-lb-pool.yaml`, not from the
ingress chart. The tenant chart is the per-tenant owner of cross-cutting
resources (Namespace, `cozystack-values` Secret, HelmReleases for
ingress and gateway), so one pool per tenant lives there.
Cilium LB IPAM rejects overlapping CIDRs across pools regardless of
`serviceSelector` — the last-added pool gets `cilium.io/PoolConflict`
and stops allocating. Keeping the pool in the ingress chart would
collide with the Gateway-API PR (#2470), which materialises its own
LoadBalancer Service from the same `publishing.externalIPs` range when
`tenant.spec.gateway=true`. Moving the pool to the tenant chart with a
namespace-only selector lets a single pool back both services
(ingress-nginx today, a Cilium Gateway Service once #2470 lands).
Only the ingress-loadBalancer signal is wired here
(`_cluster.expose-mode=loadBalancer` + `.Values.ingress=true` +
publishing tenant). #2470 rebases on top of this PR, drops its own
`packages/extra/gateway/templates/cilium-lb-pool.yaml`, and adds an OR
branch for `.Values.gateway` in the tenant template.
### Scope
Only the ingress-nginx Service is migrated by this setting. Other
cozystack components that still write `Service.spec.externalIPs`
directly (notably the `vpn` app at
`packages/apps/vpn/templates/service.yaml`) need separate follow-up
before Kubernetes v1.40.
### Tests
- `packages/extra/ingress/tests/exposure_test.yaml` — 10 helm-unittest
cases on the ingress-nginx Service: type / externalTrafficPolicy /
externalIPs assertion for both modes, unknown-mode rejection
(case-sensitive enum), empty-IPs failure, empty-entry filtering,
non-publishing-tenant fallback.
- `packages/apps/tenant/tests/exposure_test.yaml` — 9 cases on the pool
itself: IPv4, IPv6, mixed, pre-CIDR input, empty-entry filtering,
`ingress=false` in publishing tenant (no pool), non-publishing tenant
(no pool), empty externalIPs (no pool).
- Both suites are auto-discovered by `hack/helm-unit-tests.sh` via the
`test:` target in each package's Makefile.
### Caveats (copied from the inline `values.yaml` comment)
- `loadBalancer` mode uses `externalTrafficPolicy: Local`. The external
IP must already be routed to a node that hosts an ingress pod (floating
IP / upstream router / podAntiAffinity).
- Cilium does not announce the IP on its own unless L2 announcements or
BGP are enabled in the Cilium values (disabled by default in cozystack).
- Switching the value on a running cluster causes the ingress-nginx
Service to be recreated (`upgrade.force: true` on the HelmRelease +
Service kind change); expect a brief interruption of ingress traffic.
### Release note
```release-note
feat(ingress): add opt-in publishing.exposure=loadBalancer mode that uses type: LoadBalancer + CiliumLoadBalancerIPPool instead of the deprecated Service.spec.externalIPs. Default (externalIPs) preserves existing behavior.
```
## What this PR does
Closes a bootstrap race where the `cnpg-webhook-service` Service gets
its EndpointSlice populated and the data plane (kube-proxy / Cilium)
programmed a second or two after `helm install --wait` on
`cozy-postgres-operator` declares the HelmRelease Ready. Any HelmRelease
that `dependsOn: postgres-operator` and creates a
`postgresql.cnpg.io/v1` resource in that window (cozy-keycloak, tenant
Postgres apps) has kube-apiserver's call to `mcluster.cnpg.io` fail with
```
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
```
which fails the downstream release's install.
The fix is a post-install / post-upgrade Helm hook (ServiceAccount +
ClusterRole + ClusterRoleBinding + Job) that probes `/readyz` on the
webhook through the apiserver service proxy. Apiserver's service proxy
uses the same endpoint-resolution and apiserver-initiated pod dial as
the admission webhook path, so once `/readyz` answers through the proxy,
the subsequent admission call will also succeed. Helm `--wait` blocks
the install from completing until the Job exits 0, so the HelmRelease
Ready condition does not lie anymore.
Hardened per review:
- RBAC scoped to a single `services/proxy` resourceName
(`https:cnpg-webhook-service:webhook-server`) — the exact string the
apiserver URL path parser expects for the Service proxy subresource.
- A drift-guard helm-unittest test renders the vendored cnpg Service
template and fails if its `metadata.name` / `ports[0].name` diverge from
the literals in the hook, so a future `make update` that renames the
service forces this template to be updated in the same change.
- Image digest-pinned (`clastix/kubectl:v1.32@sha256:…`) with a
`renovate:` annotation; `imagePullPolicy: IfNotPresent` when
digest-pinned, `Always` when tag-only.
- Job runs under PSA-restricted-compatible securityContext (non-root,
seccomp RuntimeDefault, readOnlyRootFilesystem, drop ALL caps).
- `activeDeadlineSeconds` is derived from `maxAttempts × sleepSeconds +
60s` so a values override raising retries does not get silently cut by a
fixed deadline.
- `backoffLimit: 2` (configurable) so a transient pod-level failure
(image pull rate limit, OOM, CNI hiccup) does not fail the whole
HelmRelease.
- On timeout the Job prints the last `kubectl get --raw` stderr so the
operator can distinguish DNS / refused / 401 / TLS from the Job logs.
21 helm-unittest assertions cover ordering, RBAC/URL parity, subchart
drift, image policy, retry-loop bounds, securityContext, and the
deadline-scaling invariant. Wired via `make test` in
`packages/system/postgres-operator/Makefile`.
First surfaced on #2470 E2E run 24862782568 where
`cozy-keycloak/keycloak` failed with the exact signature above. Not tied
to that PR's branch — this is independently applicable to `main`.
### Release note
```release-note
fix(postgres-operator): add a post-install readiness gate that blocks the HelmRelease from reporting Ready until the cnpg admission webhook actually serves through the cluster Service, preventing "dial tcp <svc-ip>:443: connect: connection refused" on the first HelmRelease that depends on postgres-operator (keycloak, tenant Postgres apps).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added webhook readiness validation during install/upgrade to block
completion until the admission webhook is reachable.
* New configuration options for the readiness probe image and
retry/timeouts.
* **Tests**
* Introduced a comprehensive test suite validating rendered manifests,
probe behavior, RBAC, image rendering, and retry/timeout logic.
* **Chores**
* Added a test entry point to the project Makefile to run chart/unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Collapse the chart to a single Deployment shape that mirrors
cozystack-api: 2 replicas, soft (preferred) nodeAffinity to
node-role.kubernetes.io/control-plane via Exists, permissive
tolerations, soft podAntiAffinity on hostname, an unconditional
PodDisruptionBudget with maxUnavailable: 1, and a Service with
spec.trafficDistribution: PreferClose. The same shape works on
Talos / kubeadm / k3s and on managed Kubernetes / Cozy-in-Cozy
tenant clusters without per-distro overrides — fixing #2417 by
making the soft control-plane affinity gracefully fall back to
worker scheduling when no control-plane nodes are visible.
Drop the DaemonSet path entirely. The previous PR's deployment.enabled
toggle, the workload-kind switch in templates/workload.yaml, the
fail-guard for localK8sAPIEndpoint+nodeAffinity=[], the conditional
PDB, and the values-shape knobs for nodeAffinity and tolerations all
go away as a consequence. Override the standard Deployment fields
through the usual component-values mechanism if a non-default topology
is ever needed.
Mark localK8sAPIEndpoint.enabled deprecated and flip the default to
false. The flag injects KUBERNETES_SERVICE_HOST=status.hostIP, which
is only valid when the pod is actually scheduled on an apiserver-
bearing node. With the new soft control-plane affinity, the pod can
land off-control-plane and crash-loop. The latency motivation for the
flag is real but pending separate webhook performance work; once
addressed, the flag can be removed.
Revert all changes to packages/core/platform/images/migrations/migrations/20.
The earlier ds/...-or-deploy/... fallback was over-engineered: per
run-migrations.sh, migration 20 fires only when CURRENT < 20 (i.e.
direct upgrades from pre-0.37 to 1.3+), and that path is unsupported
anyway. The original ds/... rollout-status line is dead code on every
supported install and upgrade path.
Add helm-unittest coverage for: the default Deployment shape (replicas,
soft nodeAffinity, soft podAntiAffinity, tolerations); no env vars at
the default localK8sAPIEndpoint.enabled=false; PDB rendered
unconditionally with maxUnavailable: 1, including at replicas=1 (the
no-op case); replicas value drives spec.replicas; and that
localK8sAPIEndpoint.enabled=true does inject the env vars. The Service
test continues to assert trafficDistribution: PreferClose with no
internalTrafficPolicy.
Add a slim README documenting the topology, parameters, and the
deprecation note for localK8sAPIEndpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
- Hoist user identity extraction out of the Watch goroutine; reuse a
cached username and groups map across events instead of re-fetching
them per event. Watch now returns Unauthorized up front when no user
is present in the context, rather than failing silently per event.
- Switch the per-event access-check error log to structured klog.ErrorS
to comply with the project Go style guide.
- Strengthen TestGet_WithAccess to assert the concrete *TenantNamespace
type plus Name, Kind, and APIVersion, so type or metadata regressions
fail fast.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `linstor-gui` package (added in #2382) was never wired into the root
`Makefile`'s `build:` target, so the image is not built or published by
CI. `ghcr.io/cozystack/cozystack/linstor-gui` returns `NAME_UNKNOWN`
from the GHCR API, and `values.yaml` still pins `tag: 2.3.0` without a
digest because the per-package Makefile that rewrites the tag after a
successful push has never run in CI.
This PR adds the missing line so the next build publishes the image and
digest-pins `values.yaml` automatically.
### Release note
```release-note
build(linstor): include linstor-gui in root image build target so the image is built and published by CI (the chart previously referenced an image that did not exist in the registry)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Integrated the linstor-gui container image build into the main image
build workflow so the GUI image is produced as part of standard builds.
* Streamlined the GUI image build configuration to use consolidated
build arguments, improving consistency and maintainability of automated
image builds and metadata updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Bumps the vendored cozy-proxy chart and image tag from v0.2.0 to v0.3.0
by running `make update` in `packages/system/cozy-proxy`.
cozy-proxy v0.3.0 release:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
## Why
cozy-proxy v0.3.0 is the companion of #2501. The chart fix in #2501
finally honors the documented PortList semantics (only `externalPorts`
reachable, ICMP preserved by default), but only takes effect once
cozy-proxy v0.3.0 is in place — v0.2.0 has no port-aware logic, so the
chart fix silently no-ops on it. This PR ships that runtime side.
## Upgrade impact
For cozystack users this is a **bug fix, not a breaking change**.
cozy-proxy v0.3.0 introduces two contract changes upstream — (1)
label-only selector via `service.kubernetes.io/service-proxy-name:
cozy-proxy`, and (2) port-filter as the default ingress mode — both of
which are absorbed by the vm-instance chart (label landed in #2357,
`wholeIP` + `allowICMP` wired in #2501).
- **`externalMethod: WholeIP`** (default) — chart renders label +
`wholeIP: "true"` in both old and new versions. cozy-proxy keeps
passthrough. **No change.**
- **`externalMethod: PortList`** — previously the chart always rendered
`wholeIP: "true"` regardless of method, and cozy-proxy v0.2.0 had no
port-aware logic, so PortList was silently a no-op (all ports
reachable). After this bump + #2501, PortList behaves as documented:
only ports listed in `externalPorts` reach the VM. Users who configured
PortList but reached undeclared ports (relying on the silent no-op) will
see those ports become unreachable — that's the documented intent of
PortList finally taking effect. ICMP keeps working by default thanks to
`externalAllowICMP: true`.
- **DaemonSet rollover** — re-init of the nft table during cozy-proxy
pod restart is a brief blip on existing flows; same as any DaemonSet
upgrade.
Out-of-tree consumers using cozy-proxy directly (without the
`service-proxy-name` label, or relying on the absent-annotation
passthrough default) need to migrate per the upstream v0.3.0 release
notes — not a cozystack concern.
## Order of operations
This PR is functionally complete on its own (image bump), but the
user-facing PortList fix it enables only happens when #2501 also lands.
Recommended merge order:
1. #2501 first (chart adapts to the new contract)
2. This PR second (cozy-proxy DaemonSet picks up the new image)
The reverse order is also safe — cozy-proxy v0.3.0 with the unmodified
vm-instance chart still produces correct results for `externalMethod:
WholeIP` (most common case) and keeps PortList silently broken until
#2501 lands.
## Test plan
- [ ] `helm template packages/system/cozy-proxy` — image tag `v0.3.0`
rendered
- [ ] `make unit-tests` passes
- [ ] After #2501 lands, end-to-end: `externalMethod: PortList` with
`externalPorts: [22]` filters all other ports, ping works
- [ ] WholeIP-method VM Service unchanged after upgrade
## Companion PRs
- #2501 — vm-instance chart wires `wholeIP` / `allowICMP` per
`externalMethod`
- cozystack/cozy-proxy#13 (merged) — label-only selector + port-filter
default
- cozy-proxy v0.3.0 release notes —
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Released version 0.3.0 with updated Helm chart metadata and container
image configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `post-upgrade` Helm hook in `packages/extra/etcd` that was
`kubectl delete`ing the etcd TLS chain (`etcd-ca-tls`,
`etcd-peer-ca-tls`,
`etcd-client-tls`, `etcd-peer-tls`, `etcd-server-tls`) and then deleting
etcd pods on every chart upgrade.
The hook was gated by a semver compare of a stored
`etcd-deployed-version`
ConfigMap against `2.6.1`. That gate was written when chart versions
looked
like `2.6.0`, `2.6.1`, etc. Since commit `f871fbdb` ("Remove
versions_map
logic") all chart versions are stamped as `0.0.0+<git-hash>`, which per
semver is always `< 2.6.1`, so the gate always evaluates to "regenerate
certs" and the destructive hook fires on every upgrade.
On clusters running Kamaji-managed tenant control planes the consequence
is
severe: wiping the etcd CA triggers cert-manager to re-issue it with a
brand-new CA, but Kamaji's `datastore-certificate` Secrets still carry
the
old CA bundle mounted into tenant `kube-apiserver` pods. Those
apiservers
hit `x509: certificate signed by unknown authority` against
`etcd.<ns>.svc:2379` and go into CrashLoopBackOff until each tenant
`DataStore` is individually force-reconciled and the Deployments rolled.
Commit `47d81f70` ("Disabled private key rotation in CA certs") already
fixed the underlying `rotationPolicy: Always` issue the hook was written
to
paper over in March 2025, so the migration has no remaining use.
Changes:
- Delete `templates/hook/{job,role,rolebinding,serviceaccount}.yaml`
- Delete `templates/version.yaml` (only the hook read
`etcd-deployed-version`)
- Add `tests/no-post-upgrade-hook_test.yaml` as a regression guard — if
`templates/hook/job.yaml` or `templates/version.yaml` is ever brought
back,
`make unit-tests` fails
### Release note
```release-note
fix(etcd): remove destructive post-upgrade cert-regeneration hook. Previously the etcd chart ran a `post-upgrade` hook on every Helm upgrade that deleted etcd TLS Secrets and pods, causing cert-manager to re-issue the etcd CA. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 -> 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+<git-hash>` and the underlying `rotationPolicy: Always` issue was fixed in commit `47d81f70`. This is behavior removal, not cosmetic cleanup.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added Helm chart test suite verifying absence of post-upgrade hook Job
and version ConfigMap.
* **Chores**
* Removed post-upgrade hook Job and associated RBAC resources (Role,
RoleBinding, ServiceAccount).
* Removed version ConfigMap.
* Added `test` target to Makefile for running Helm chart unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from sircthulhu and CodeRabbit:
1. Return Forbidden instead of NotFound for unauthorized access
- Get() now returns 403 Forbidden to follow standard K8s RBAC behavior
- Previously returned 404 NotFound for security-by-obscurity
- Updated test expectations to match new behavior
2. Propagate field and label selectors in Watch handler
- Pass opts.FieldSelector and opts.LabelSelector to upstream Watch
- Add defensive filtering before authorization to prevent RBAC bypass
- Fixes potential issue with resourceNames restrictions
3. Refactor subject-matching logic to eliminate duplication
- Extract matchesSubject() helper for Group/User/ServiceAccount checks
- Remove duplicated code from filterAccessible and hasAccessToNamespace
- Consolidates ServiceAccount namespace fallback logic
All tests pass successfully.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
The `vm-instance` chart now drives the cozy-proxy `wholeIP` and
`allowICMP` annotations explicitly so that `externalMethod: PortList`
actually filters ingress traffic to declared ports while keeping
ping/PMTU functional.
- Render `networking.cozystack.io/wholeIP: "false"` when
`externalMethod: PortList` (was always `"true"`, which silently disabled
the PortList semantics in cozy-proxy).
- Add `externalAllowICMP` value (default `true`) propagated as
`networking.cozystack.io/allowICMP` when `externalMethod: PortList`.
Without this, cozy-proxy drops ICMP in port-filter mode (ping/PMTU
broken). Operators can set `externalAllowICMP: false` to opt out.
The changelog entry is intentionally **not** part of this PR — it will
be added in a dedicated `docs: add changelog for vX.Y.Z` commit at
release time, per project convention.
## Why
`externalMethod: PortList` is documented as filtering ingress traffic to
declared ports but has been non-functional on Cozystack v1.3.0 —
verified empirically on a 3-node Talos lab. Root cause was twofold:
chart always set `wholeIP: "true"`, and cozy-proxy v0.2.0 had no
port-aware logic. The cozy-proxy side was fixed in
cozystack/cozy-proxy#11 (merged) and cozystack/cozy-proxy#12 (allowICMP
follow-up); this PR completes the user-visible fix on the chart side.
## Companion PRs
- cozystack/cozy-proxy#11 (merged) — per-service ingress port filtering
- cozystack/cozy-proxy#12 (merged) — `allowICMP` annotation for
port-filter mode
## Test plan
- [x] Built cozy-proxy with the companion fix locally, deployed on a
3-node Talos lab (Cozystack v1.3.0)
- [x] `wholeIP: "false"` Service with `spec.ports: [22]`: only port 22
reachable from outside; ports 80/443/8080/9999 filtered
- [x] WholeIP-annotated Service unchanged: all listening ports reachable
- [x] Egress IP preservation works in both modes (TCP curl + UDP DNS)
- [x] `nft list table ip cozy_proxy` confirms expected ruleset
- [x] `helm template` renders the expected annotation matrix: `PortList`
default → `wholeIP=false, allowICMP=true`; `PortList` opt-out →
`allowICMP=false`; `WholeIP` → only `wholeIP=true`
- [x] `make unit-tests` passes locally
- [ ] CI unit tests
- [ ] CI E2E
## Backport
Suggesting `backport-v1.3` once merged.
## Release note
```release-note
[vm-instance] Make `externalMethod: PortList` actually filter ingress traffic to ports listed in `externalPorts`. New `externalAllowICMP` knob (default true) propagates the cozy-proxy `allowICMP` annotation to keep ping/PMTU functional in port-filter mode. Combined with cozy-proxy v0.3.0+, only listed ports plus ICMP are reachable from the VM's LoadBalancer IP.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added `externalAllowICMP` configuration option to control ICMP traffic
acceptance for VM external access in PortList mode (enabled by default).
* **Documentation**
* Updated parameter documentation to include the new ICMP traffic
control setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Pulls in the per-port filtering and allowICMP support that the companion
vm-instance chart fix in #2501 relies on. cozy-proxy v0.3.0 also tightens
the selector to the standard service.kubernetes.io/service-proxy-name=cozy-proxy
label and switches the default ingress mode to port-filter; both are
already covered by the vm-instance chart (label landed in #2357,
wholeIP/allowICMP wired explicitly in #2501), so VM workloads upgrade
transparently.
Out-of-tree consumers using cozy-proxy annotations directly (without the
label, or relying on the absent-annotation passthrough default) are
called out in the upstream v0.3.0 release notes:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Address review feedback from coderabbitai on packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml:119
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback from coderabbitai on packages/system/gpu-operator/examples/nvidia-driver-compat.yaml:77
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
- Adds a pre-commit hook and a dedicated GitHub Actions workflow that
run `make generate` at the repo root to catch drift in generated API
code (CRDs, DeepCopy, clients, RBAC) before it lands on main.
- Pre-commit hook is scoped to paths that actually affect codegen
(`api/`, `pkg/apis/`, `hack/update-codegen.sh`,
`hack/boilerplate.go.txt`) so unrelated commits are not slowed down.
- CI workflow (`.github/workflows/codegen-drift.yml`) installs Go from
`go.mod`, runs `make generate`, and fails on drift with an error
pointing contributors to the local fix.
- Includes one drift fix the check surfaced: `RestoreJobSpec.Options`
(added in #2437) had no `DeepCopyInto` handling — regenerated.
### Release note
```release-note
ci(api): add pre-commit hook and GitHub Actions workflow that verify generated API code (CRDs, deepcopy, clients, RBAC) is in sync with `make generate`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a CI workflow that checks generated code during PRs and fails
the build if generated artifacts diverge.
* Added a pre-commit hook that runs generation checks locally to prevent
committing outdated generated files.
* Fixed deep-copy behavior for backup restore job specs so nested
options are correctly duplicated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Add `externalAllowICMP` value (default true) propagated as
`networking.cozystack.io/allowICMP` annotation on the rendered Service
when `externalMethod: PortList`. The cozy-proxy companion (released as
part of cozystack/cozy-proxy#11 + #12) drops ICMP by default in
port-filter mode, which breaks ping and PMTU discovery; defaulting the
chart to "true" preserves user expectations while still allowing
operators to opt out by setting `externalAllowICMP: false`.
- Remove the v1.3.1.md changelog entry. Project convention is to add
changelogs in a dedicated "docs: add changelog for vX.Y.Z" commit at
release time, not as part of feature/fix PRs.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
## What this PR does
Bumps `linstor-csi` from v1.10.5 to v1.10.6 and ships an out-of-tree
patch that fixes live migration of KubeVirt VMs whose volumes sit on a
DRBD Protocol-A/B resource (e.g. a `replicated-async` StorageClass).
### Problem
DRBD requires Protocol C whenever `allow-two-primaries=yes` is enabled.
Operators commonly opt into Protocol A on a per-resource-group basis for
async / WAN replication, which silently breaks every subsequent live
migration of consumers of those volumes: `drbdadm adjust` rejects the
second-attach with `Protocol C required` (errno 139), KubeVirt's
evacuation loop retries indefinitely, and the VM stays pinned to the
source node.
### Change
- `LINSTOR_CSI_VERSION` 1.10.5 → 1.10.6 (Makefile + Dockerfile default).
- New patch `002-protocol-c-override-for-dual-attach.diff`: when
`Attach` installs `allow-two-primaries=yes` on the resource-definition
during a second attach, it also installs `DrbdOptions/Net/protocol=C` as
an override on the resource-definition. The override applies to every
connection (including diskless TieBreaker peers, where a per-pair
override would still leave one connection broken). It is tagged with
`Aux/csi-protocol-override=yes` so `Detach` removes only the override
this driver installed, leaving any operator-set `Protocol` property on
the resource-definition untouched.
- Existing patch `001-relocate-after-clone-restore.diff` regenerated
against v1.10.6 (context shift only, no logic change — the old patch
hunks no longer aligned cleanly).
### Verification
- `make image-linstor-csi` builds successfully on linux/amd64 with both
patches applied.
- End-to-end test on dev5 cluster (KubeVirt v1.6.3, 3-node Talos):
created a Protocol-A resource-group + StorageClass, provisioned a VM on
top, and triggered live migration. Migration succeeds in a single Attach
with the override installed during dual-attach and removed by Detach.
Reproducer (without the patch) is the well-known evacuation loop with
`(node) Failed to adjust DRBD resource ... Protocol C required`.
### Upstream
Upstreamed as draft PR piraeusdatastore/linstor-csi#435.
### Compatibility
- No behaviour change for resources already using Protocol C (the common
case).
- No behaviour change for resources never attached with
allow-two-primaries.
- Idempotent: re-running `Attach` is a no-op once the override is
installed.
- Operator-set Protocol overrides on the resource-definition are
preserved (gated by the Aux marker).
### Release note
```release-note
fix(linstor): live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with "Protocol C required" — linstor-csi now installs a Protocol=C override on the resource-definition during dual-attach and reverts it on detach.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automatic replica relocation after volume clone and snapshot restore
to improve placement and load distribution.
* Conditional DRBD protocol override to enable/clean up dual-attach
(two-primaries) scenarios more reliably.
* **Chores**
* Updated LINSTOR CSI default to v1.10.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Render `networking.cozystack.io/wholeIP: "false"` on the Service when
`externalMethod: PortList` is configured (was always `"true"` before).
Combined with cozy-proxy v0.3.0+ which adds per-port filtering for
"false"-annotated services, this makes `externalMethod: PortList`
behave as documented: only ports listed in `externalPorts` are
reachable from the LoadBalancer IP.
Backward-compatible: existing services with `externalMethod: WholeIP`
continue to set `wholeIP: "true"` and behave identically. cozy-proxy
versions older than v0.3.0 ignore Services with `wholeIP: "false"`,
which means PortList Services on older cozy-proxy will lose their
egress IP preservation — but that path was already non-functional
ingress-wise, so this is not a regression for users actually relying
on PortList.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Bump linstor-csi to v1.10.6 and add an out-of-tree patch that overrides
DrbdOptions/Net/protocol to C on the resource-definition whenever a
volume is attached to a second node with allow-two-primaries (the
live-migration flow), and reverts it on detach.
Without the patch, any KubeVirt live migration of a VM whose volume
sits on a Protocol-A/B resource (for example a "replicated-async"
StorageClass) ends up in a permanent evacuation loop: drbdadm adjust
on the satellite rejects allow-two-primaries with "Protocol C
required" (errno 139), the migration fails, and KubeVirt retries
until manual intervention.
The override is set at the resource-definition level so it covers
every connection, including diskless TieBreaker peers, and is tagged
with an Aux marker so Detach reverts only the override we installed.
The 001 relocate-after-clone patch was regenerated against v1.10.6
(context shift only, no logic change).
Verified end-to-end on dev5: live migration of a KubeVirt VM whose
PVC sits on a Protocol-A resource-group now succeeds in a single
Attach.
Patch is upstreamed as draft PR piraeusdatastore/linstor-csi#435.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Closes#2412. On a cold tenant-Kubernetes bootstrap, the parent
HelmRelease raced the admin-kubeconfig Secret that Kamaji provisions
asynchronously. Three CP-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) mounted that Secret as a hard volume, flux
helm-controller's default wait budget was too short for Kamaji cold
start, and `install.remediation { retries: -1 }` then uninstalled the
Cluster CR and restarted the cycle forever.
Implements a defense-in-depth fix:
- `optional: true` on the admin-kubeconfig Secret volume in all three
Deployments so kubelet no longer FailedMounts while Kamaji is still
bootstrapping.
- A shared `wait-for-kubeconfig` init container (in
`templates/_helpers.tpl`) that polls for `super-admin.svc` with a 10m
deadline, strictly below the HelmRelease Install.Timeout so a broken
tenant falls into CrashLoopBackOff visibly instead of hanging forever.
- Per-Application HelmRelease Install/Upgrade timeout, driven by a new
`release.cozystack.io/helm-install-timeout` annotation on
ApplicationDefinition. Kubernetes-rd sets it to `15m`; other kinds leave
it unset and keep flux defaults, so their failed installs remediate on
the normal cadence. Parser rejects ns/us/µs (accepted by
`time.ParseDuration`, rejected by Flux's CRD pattern) at startup.
- Soft-skip when `_namespace.etcd` is empty: the CP-side Deployments,
the Cluster/KamajiControlPlane/KubevirtCluster/WorkloadMonitor CRs, and
every child HelmRelease that references admin-kubeconfig now render only
when an etcd DataStore exists for this tenant. An `awaiting-etcd`
ConfigMap is emitted as a user-visible status beacon so `helm install`
still succeeds and flux retries on its 5m interval until the Tenant
chart catches up.
- e2e remediation guard built on `.status.history[].status` (the
Snapshot shape), not on `.status.installFailures` - `ClearFailures()`
zeroes the latter on every successful reconciliation, which made the
previous guard vacuous.
Tests:
- Go unit tests for the annotation parser (accepted/rejected units) and
the HR builder (table-driven across kinds).
- helm unittest for the per-template structure (optional volume, init
container, dataStoreName, awaiting-etcd beacon).
- bats unit tests for the shell guard (every combination of
empty/zero/positive history entries, plus pinned HR v2 shape).
- Chart-wide bats invariants: every Deployment mounting admin-kubeconfig
has the guards; zero such Deployments and zero HelmReleases render when
etcd is empty.
All wired into the existing `make unit-tests` target (`go-unit-tests`
added alongside `helm-unit-tests` and `bats-unit-tests`).
Option 2 from the ticket (separate HelmRelease with `dependsOn`) was
intentionally not taken: the combination above closes the same race
without restructuring the chart's HelmRelease topology.
### Release note
```release-note
fix(kubernetes): close admin-kubeconfig race on tenant Kubernetes bootstrap. The parent HelmRelease no longer enters an uninstall/retry cycle when Kamaji control-plane cold start exceeds flux's default wait budget. A Kubernetes tenant created before the parent Tenant application has etcd enabled now renders only an awaiting-etcd beacon ConfigMap and waits quietly for the DataStore to appear, instead of producing half-installed Deployments that CrashLoopBackOff forever.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Per-application Helm install/upgrade timeout via metadata annotation.
* Init-container guards that wait for admin kubeconfig before workloads
start.
* Chart resources now render conditionally based on etcd presence.
* **Tests**
* Helm-template tests for admin-kubeconfig invariants and
remediation-cycle detection.
* New Go unit tests and CI Helm/unittest coverage plus test value files.
* **Chores**
* Added BusyBox image pin and new Makefile test targets (including Go
unit-tests).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The per-package Makefile added in #2382 hardcoded buildx flags
(--provenance, --builder, --platform=linux/amd64,linux/arm64,
--push, --load, --label) instead of using the shared $(BUILDX_ARGS)
macro from hack/common-envs.mk.
This broke CI: the runner's default docker driver does not support
multi-platform builds, and the hardcoded multi-arch platform list
crashed `make build` with "Multi-platform build is not supported for
the docker driver."
Replace the hardcoded flags with $(BUILDX_ARGS) to match every other
package (e.g. linstor, dashboard, cilium). $(BUILDX_ARGS) injects
--push, --load, --label, --provenance=false, and only sets --builder
or --platform when the operator explicitly exports BUILDER/PLATFORM.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The linstor-gui package was added in #2382 with its own per-package
Makefile and Dockerfile, but the root Makefile's `build:` target was
not updated to invoke it. As a result `ghcr.io/cozystack/cozystack/
linstor-gui` has never been published (registry returns NAME_UNKNOWN)
and the chart's `image.tag` was never digest-pinned. Any cluster
deploying the chart hits ImagePullBackOff.
Wire the package into the root build alongside the other system
images. The next CI build will publish the image and the per-package
Makefile will rewrite values.yaml `image.repository`/`image.tag` to a
digest-pinned reference automatically.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozystack.io/status-beacon: "true" annotation had no consumer in
the chart, no documented contract, and no convention defined for other
charts to follow. It would have become accidental precedent for
contributors copying the pattern without understanding it.
The ConfigMap itself is self-explanatory: the name <release>-awaiting-etcd,
data.status: "awaiting-etcd", and the human-readable message in
data.message all surface the same operator signal via kubectl get cm.
Drop the annotation; keep the ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Operators in air-gapped or rate-limited environments cannot reach
docker.io and the bundled busybox digest pin gives them no escape
hatch. Add an optional images.waitForKubeconfig chart value that, when
set, replaces the helper's image reference with any registry path
kubelet can pull. Empty value falls back to images/busybox.tag, so the
prior digest-pinned default is preserved.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Wrap the values: block in a conditional so the key is omitted entirely
when no defaults or overrides produce content. Previously the template
always emitted values: null, triggering unnecessary FluxCD reconciliation.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Use nindent instead of indent with leading whitespace to prevent
broken YAML rendering when devicePlugin.service.labels is set.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
OnDelete requires manual pod deletion to apply updates. RollingUpdate
with maxUnavailable constraint matches upstream default and is consistent
with all other system packages.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Adopt CNCF/Kubernetes label conventions for issues and PRs and add
automated labeling.
**Canonical label file**: `.github/labels.yml`. Synced into the
repository by `.github/workflows/labels.yaml` (EndBug/label-sync@v2) on
push to `main`, weekly cron, and manual dispatch. UI-only label edits
are overwritten — propose changes via PR to this file.
### Label namespaces
Following the [Kubernetes label
scheme](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md):
- `kind/*` — issue or PR type (bug, feature, documentation, support,
cleanup, regression, flake, failing-test, api-change, breaking-change)
- `priority/*` — urgency (critical-urgent, important-soon,
important-longterm, backlog)
- `triage/*` — review state (needs-triage, accepted, needs-information,
not-reproducible, duplicate, unresolved)
- `lifecycle/*` — issue/PR lifecycle (active, frozen, stale, rotten)
- `area/*` — subsystem; 15 seeded plus `area/uncategorized` fallback.
Extensible — propose a new area when no existing one fits.
- `do-not-merge/*` — PR merge blockers (work-in-progress, hold)
Cozystack-specific labels preserved: `epic`, `community`, `help wanted`,
`good first issue`, `quality-of-life`, `upstream-issue`, `backport`,
`backport-previous`, `release`, `automated`, `debug`, `sponsored`,
`lgtm`, `ok-to-test`, `security/*`, `size:*`.
### `area/*` set
15 areas seeded by activity in open issues and PRs: `area/ai`,
`area/api`, `area/build`, `area/ci`, `area/dashboard`, `area/database`,
`area/extra`, `area/kubernetes`, `area/monitoring`, `area/networking`,
`area/platform`, `area/release`, `area/storage`, `area/testing`,
`area/virtualization`. Plus `area/uncategorized` as the auto-labeler
fallback.
### Migration safety
Existing labels are renamed via `aliases:` in `labels.yml`. GitHub
preserves the label ID, so all currently tagged issues and PRs keep
their tags under the new name without losing references:
| Old | New |
|---|---|
| `bug` | `kind/bug` |
| `enhancement` | `kind/feature` |
| `documentation` | `kind/documentation` |
| `question` | `kind/support` |
| `frozen` | `lifecycle/frozen` |
| `stale` | `lifecycle/stale` |
| `duplicate` | `triage/duplicate` |
| `do-not-merge` | `do-not-merge/work-in-progress` |
| `do not merge` | `do-not-merge/work-in-progress` |
`delete-other-labels: false` on the initial rollout. Generic
GitHub-default labels (`wontfix`, `invalid`) are preserved untouched and
will be removed in a follow-up cleanup PR. EndBug processes aliases
sequentially, so the second of the two `do-not-merge*` aliases hits a
name collision and logs a warning — the legacy label survives that one
sync and is cleaned up in the same follow-up.
### PR auto-labeling
`.github/workflows/pr-labeler.yaml` parses each PR title on `opened`,
`edited`, `reopened`, and `synchronize` and applies labels additively
(never removes):
- **type → `kind/*`**: feat, fix, docs, chore, refactor (others get no
kind)
- **scope → `area/*`**: scope mapping covers all current cozystack
components (full table in `docs/agents/contributing.md`)
- **`!` after type or `BREAKING CHANGE:` footer**: applies
`kind/breaking-change`
- **`[Backport release-1.x]` prefix**: stripped before parsing;
`area/release` and `backport` labels added
- **Composite scope** (`feat(platform, system, apps): …`): each part
mapped independently
- **Bracket fallback** (`[scope] description`): maps `area/*` but cannot
infer `kind/*`
- **Unmapped scope or non-conventional title**: applies
`area/uncategorized` for human review
### Schema validation
`.github/workflows/labels.yaml` runs a `validate` job on every PR
touching `labels.yml` or its workflow. Asserts:
- description ≤ 100 chars (GitHub REST API limit)
- color is 6-char hex without leading `#`
- unique top-level names
- aliases do not collide with top-level names
Sync runs only on push to main, weekly cron, and manual dispatch; PR
runs validate-only.
### Hardcoded label/title references updated
- `.github/ISSUE_TEMPLATE/bug_report.md`: `labels: 'bug'` → `labels:
'kind/bug'`
- `.github/workflows/tags.yaml`:
- changelog PR labels `['documentation', 'automated']` →
`['kind/documentation', 'automated']`
- release PR title `Release v${version}` → `chore(release): cut
v${version}` (so the auto-labeler applies `kind/cleanup` +
`area/release`)
- changelog PR title `docs: add changelog for v${version}` →
`docs(release): add changelog for v${version}` (so the auto-labeler
applies `kind/documentation` + `area/release`)
### Documentation
- `AGENTS.md`: Activation entry pointing agents to `labels.yml` and the
PR title auto-labeling rules. States explicitly that `area/*` accuracy
outweighs reuse — propose a new area when none fits, do not shoehorn
into a wrong one.
- `docs/agents/contributing.md`: PR Title Auto-Labeling section with
type→kind and scope→area tables.
### Out of scope (follow-up PRs)
- Removal of redundant labels (`wontfix`, `invalid`, plus the surviving
legacy `do not merge` if alias-rename collision keeps it)
- Org-wide sync from `cozystack/.github/labels.yml`
- Dosu bot configuration update for `lifecycle/stale` (requires
dashboard access)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Automated PR labeling from Conventional Commits (type → kind/*, scope
→ area/*), plus a comprehensive namespaced label taxonomy.
* **Chores**
* Workflows to validate, sync, and auto-apply labels (including
scheduled/manual runs and validation checks).
* Added repository-wide label configuration and normalized bug label
metadata to namespaced form.
* Updated release PR titling/labeling conventions.
* **Documentation**
* Contributor and agent guidance on PR title conventions, label
mappings, and triage procedures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:160,173:
emit core.warning when a Conventional Commits type has no kind/*
mapping or a scope has no area/* mapping. Without the warning, typos
(e.g., "hotfix" instead of "fix") and recurring new scopes silently
fall through to area/uncategorized, masking that the mapping has
drifted.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:155:
Conventional Commits 1.0 spec item 16 treats BREAKING CHANGE: and
BREAKING-CHANGE: as synonymous footers. The hyphen form was silently
ignored before, so PRs that use it would miss kind/breaking-change.
https://www.conventionalcommits.org/en/v1.0.0/#specification
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:129:
defensive (pr.labels || []) avoids TypeError if the webhook payload
arrives without the labels field on edge cases like stripped edited
events.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
DCGM exporter metrics carry the exporter's own namespace
(cozy-gpu-operator), not the workload namespace. Recording rules that
filtered namespace!~"cozy-.*" silently dropped all DCGM series,
producing empty dashboard panels.
Replace namespace-level hardware aggregations with node-level
equivalents (grouped by Hostname), keep namespace-level allocation
rules that use kube_pod_container_resource_requests (which carries the
real workload namespace), and rename pod-level efficiency rules to
gpu-level since DCGM cannot attribute hardware metrics to individual
pods.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Add an opt-in `upgradeCRDs` field to `ComponentInstall` in
`PackageSource`. The field maps directly to
`HelmRelease.Spec.Upgrade.CRDs` so a component author can declare how
Flux should handle CRDs from the chart's `crds/` directory when the
release is upgraded.
The helm-controller default on upgrade is `Skip`, which means CRDs added
by a chart bump never reach clusters that already have the release
installed — they must be applied manually with `kubectl apply --filename
charts/.../crds/`. This surfaces on every upgrade of an operator whose
CRD set expands between versions (etcd-operator, cnpg, kubevirt, kamaji,
etc.).
Setting `upgradeCRDs: CreateReplace` lets Flux apply new CRDs
declaratively with the chart.
Values are restricted to `Skip`, `Create`, `CreateReplace` via a
kubebuilder enum marker. Empty / unset preserves the existing
helm-controller default, so every current `PackageSource` keeps working
unchanged.
Migration is out of scope here — follow-ups will opt individual packages
in case-by-case.
### Relation to existing CRD management approach
The project convention (per #377) is to extract CRDs into a dedicated
Helm chart that reconciles ahead of the operator chart. This PR does not
replace that pattern — it complements it for charts that keep CRDs
inline under `charts/<name>/crds/` where extraction isn't practical
(vendored upstream charts with tightly coupled CRDs). Packages that
already split CRDs out can leave `upgradeCRDs` unset and keep using
their existing separate chart.
### Release note
```release-note
feat(operator): add opt-in `upgradeCRDs` field to `PackageSource` component `install` block to control how CRDs from the chart's `crds/` directory are applied on HelmRelease upgrades (`Skip` by default; use `CreateReplace` for operators whose CRD set expands between versions).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable CRD upgrade policy for component installs: Skip, Create,
CreateReplace — controls CRD handling during package upgrades and
preserves controller default when unset.
* **Documentation**
* Guidance on CRD upgrade semantics, advice to use CreateReplace for
specific operators, and warning about potential data-loss risks;
clarified contributor scope examples and PR template guidance.
* **Tests**
* Added tests validating CRD policy parsing and presence of the CRD
policy enum in the published schema.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Match the wording adopted in docs/agents/contributing.md so that human
contributors and AI agents see the same guidance in both places.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Scopes section was read as an exhaustive enumeration, which led to
review feedback flagging any scope outside the list as invalid. The
intent has always been that contributors pick the most specific scope
for the change and extend the list when a genuinely new area appears.
Reword the section accordingly and add operator as an example scope.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Describe when to set upgradeCRDs: CreateReplace (operators that evolve
their CRD set additively between versions) and the data-loss risk of
enabling it on operators that drop fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add an opt-in UpgradeCRDs field to ComponentInstall that maps to
HelmRelease.Spec.Upgrade.CRDs, allowing a PackageSource component to
declare how Flux should handle CRDs from the chart's crds/ directory
on upgrade.
The helm-controller default on upgrade is Skip, which means new CRDs
added between chart versions never reach existing clusters and must be
applied manually. Setting upgradeCRDs: CreateReplace makes Flux apply
new CRDs declaratively with the chart.
Allowed values are restricted to Skip, Create, CreateReplace via a
kubebuilder enum marker. Empty / unset preserves the existing Flux
default, so all existing PackageSource resources keep working.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:73:
add space before closing pipe in the type to kind mapping table for
consistency with other rows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on AGENTS.md:33:
expand bare labels.yml and pr-labeler.yaml to .github/labels.yml and
.github/workflows/pr-labeler.yaml for consistency with surrounding refs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:88:
… replaced with ... for compatibility across editors and tools.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on .github/labels.yml:242:
all hex color values use lowercase characters for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to retrigger Build that lost the OCI registry tag race
on the previous attempt.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Empty commit to retrigger Build job that failed on Docker Hub anonymous
pull rate limit (429 Too Many Requests) for golang:1.26.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
- Increase kamaji controller memory limit from 500Mi to 512Mi
- Increase kamaji controller memory request from 100Mi to 256Mi
- Add startup probe with 60-second timeout (12 attempts × 5s periods)
- Increase readiness/liveness probe initialDelaySeconds from 5s/15s to
30s
## Problem
The kamaji controller was experiencing frequent CrashLoopBackOff due to
OOMKilled errors. Analysis showed:
- Container was being killed with exit code 137 (OOMKilled) after ~20-25
seconds of runtime
- Memory limit of 500Mi was insufficient for controller initialization
- Readiness probe was failing because it started too early (5s
initialDelay), before the controller finished leader election (~17s)
## Solution
**Memory increase:**
- Limit: 500Mi → 512Mi (based on production testing)
- Request: 100Mi → 256Mi (ensures adequate reservation)
**Startup probe:**
- Added to give controller up to 60 seconds to initialize without being
killed by liveness probe
- 12 attempts × 5s period = 60s maximum startup time
**Probe delays:**
- ReadinessProbe: 5s → 30s initialDelay (controller needs ~17s to
acquire leader lease)
- LivenessProbe: 15s → 30s initialDelay (aligned with readiness)
## Testing
Verified in production cluster:
- Controller runs stable with 0 restarts
- No more OOMKilled events
- Successfully creates kubeconfig secrets for tenant clusters
## Related Issues
Fixes tenant cluster components stuck in ContainerCreating due to
missing kubeconfig secrets (caused by crashing kamaji controller).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Introduced automated health checks using HTTP-based probes to monitor
service status during startup, continuous operation, and readiness to
handle traffic.
* Adjusted container memory resource allocation for enhanced stability
and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The pool is now rendered from packages/apps/tenant/templates/cilium-lb-pool.yaml instead of packages/extra/ingress/templates/cilium-lb-pool.yaml. Cilium LB IPAM forbids overlapping CIDRs across pools regardless of serviceSelector, so both the ingress-loadBalancer path and the upcoming per-tenant Gateway in #2470 cannot each own their own pool on the same publishing.externalIPs range. The tenant chart is the natural per-tenant owner — it already creates the Namespace, the cozystack-values Secret, and the HelmReleases for both ingress and gateway.
The new pool uses a namespace-only serviceSelector (io.kubernetes.service.namespace: <ns>), which matches any LoadBalancer Service in the tenant namespace. The metadata.name changed from <trim>-ingress to <trim>-exposure to reflect that the pool is not ingress-specific.
Only the ingress-loadBalancer signal is wired in this commit (_cluster.expose-mode=loadBalancer plus .Values.ingress=true on the publishing tenant). The gateway branch is added in #2470 on top of this commit — it rebases, drops its own packages/extra/gateway/templates/cilium-lb-pool.yaml, and adds an OR branch for .Values.gateway in the tenant template.
Pool-rendering unit tests moved from packages/extra/ingress/tests/ to packages/apps/tenant/tests/. The ingress chart tests keep the Service-level asserts. packages/apps/tenant/Makefile gains a test target so hack/helm-unit-tests.sh picks up the new suite.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Only emit values key in hami.yaml when valuesOverride has content,
matching gpu-operator pattern. Add test verifying empty valuesOverride
does not produce spec.values. Fix trailing whitespace and missing
newlines in vendored chart templates.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Unconditionally emit values in hami.yaml matching the project pattern.
Remove duplicate test case and add coverage for omitted valuesOverride key.
Delete dead PSP template and RBAC rules (policy/v1beta1 removed in K8s
1.25). Override kube-scheduler image registry to registry.k8s.io to avoid
Chinese registry for international users.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Remove broken hami-dra subchart dependency from vendored chart
(Chart.yaml, Chart.lock, values.yaml) and strip DRA condition guards
from all templates since the subchart was already deleted. Override
devicePlugin updateStrategy to OnDelete to prevent destructive rolling
updates of GPU workloads. Align gpu-operator template with project
pattern (unconditional values emission). Add nodeConfiguration format
documentation and test for conflicting valuesOverride scenario.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback from coderabbitai on
packages/system/postgres-operator/values.yaml:30.
The old comment's 5-minute ImagePullBackOff scenario conflicted with
the 180s activeDeadlineSeconds that the default maxAttempts/sleepSeconds
resolve to, so the numbers could not both be taken at face value.
Rewrite the comment to state the actual deadline math and frame the two
gates as an AND with activeDeadlineSeconds being the shorter one under
defaults, so readers understand why backoffLimit has little headroom
without an accompanying maxAttempts/sleepSeconds bump.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:115.
The wait container had no resource requests or limits, so schedulers
treated it as BestEffort and downstream quota enforcement had no signal.
Set small requests (10m CPU, 32Mi memory) and conservative limits
(100m CPU, 64Mi memory) matching the actual footprint of the kubectl
polling loop.
Add a matching unittest assertion so the values stay in sync if anyone
touches the template.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/system/postgres-operator/templates/webhook-ready-hook.yaml:83.
The Job targets the cnpg-webhook-service/services/proxy subresource, which
is namespaced and lives in the release namespace. A namespaced Role and
RoleBinding grant the exact permission needed without creating global
RBAC for a namespaced probe, which is the principle of least privilege.
Also update the kind assertions in tests/webhook-ready-hook_test.yaml so
the unittest suite tracks the new Role/RoleBinding objects.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Bumps the vendored `robotlb` chart to the latest upstream build.
Chart version remains `0.1.3`; the bundled `appVersion` moves from
`0.0.5` to `0.0.6`.
The new `robotlb` release adds RBAC permissions for
`discovery.k8s.io/endpointslices` (`get`, `list`, `watch`), which are
required to manage services backed by `EndpointSlice` — notably
KubeVirt-exposed workloads that do not publish classic `Endpoints`.
Notes:
- Upstream also replaced `replicas: {{ .Values.replicas }}` with a
hardcoded `replicas: 1` in `templates/deployment.yaml`. The
effective replica count is unchanged (we already set `1`), but the
value is no longer overridable via chart values. A minor cosmetic
reformat was applied to `templates/role.yaml`.
Closes#2256
### Release note
```release-note
chore(hetzner-robotlb): update robotlb to 0.0.6 — adds RBAC for EndpointSlices so services backed by EndpointSlice (e.g. KubeVirt) are supported.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Extended service account permissions to access Kubernetes endpoint
slices from the discovery API.
* **Bug Fixes**
* Deployment replica configuration now fixed to single instance.
* **Style**
* Improved YAML formatting in role template declarations.
* **Chores**
* Updated application version metadata to 0.0.6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Refreshes the vendored Cilium chart in `packages/system/cilium` from
v1.19.1 to v1.19.3 via `make update`. Chart templates, values, CRDs and
the Cilium image reference are regenerated from upstream.
### Motivation
- **v1.19.2** ships a critical fix for cert-manager HTTP-01 Gateway API
challenges on hostnames that have both HTTP and HTTPS listeners
([cilium#44492](https://github.com/cilium/cilium/pull/44492), backport
[#44517](https://github.com/cilium/cilium/pull/44517)). Without this
fix, cert-manager cannot issue certificates via Gateway API when a
redirect HTTP listener and a TLS HTTPS listener share a hostname.
- **v1.19.3** is the latest stable patch release in the v1.19.x line (15
Apr 2026).
- This bump is a prerequisite for upcoming Gateway API work tracked
separately.
### Upstream changes pulled in
- Cilium Envoy bootstrap config, operator clusterrole, config template,
`values.schema.json` and the cilium-agent DaemonSet refreshed from
upstream.
- New `templates/ztunnel/` directory (DaemonSet, Secret, ServiceAccount)
added by upstream — not enabled by default in Cozystack values.
### Release note
```release-note
chore(cilium): bump to v1.19.3 (cert-manager HTTP-01 fix via cilium#44492)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added ztunnel encryption support with configurable deployment settings
* Added ConfigDriftDetection for monitoring ConfigMap changes
* Added endpoint policy update timeout configuration
* Added load balancer service topology support
* Extended Envoy circuit breaker configuration with connection and
request limits
* **Updates**
* Upgraded Cilium to v1.19.3
* Updated container images (Envoy, certgen, Hubble relay, clustermesh)
* Enhanced Cilium operator RBAC capabilities for managing ServiceImport
finalizers
* **Removals**
* Removed BIG TCP tunnel configuration option
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The cnpg admission webhook's controller pod passes readinessProbe as soon
as the local HTTPS server binds to :9443. The HelmRelease marks itself
Ready right after that via helm install --wait. But the Service
cnpg-webhook-service needs its EndpointSlice populated and the data plane
(kube-proxy / cilium) programmed before kube-apiserver can reach the
webhook through the Service ClusterIP. That gap is short but not zero,
and any HelmRelease that depends on postgres-operator (cozy-keycloak,
tenant Postgres apps) can fire its own install inside the window and hit
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
failed to call webhook: Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
which fails the install of the downstream release. Seen on cozystack/cozystack#2470
E2E run 24862782568.
Add a post-install,post-upgrade Helm hook Job that blocks the release
from reporting Ready until the webhook answers /readyz through the
apiserver service proxy. Apiserver proxy routes the call over the same
Service IP → EndpointSlice → pod path the admission webhook uses, so
once it responds, the webhook admission path is also working.
RBAC is minimal: a dedicated ServiceAccount with a ClusterRole that only
grants get on services/proxy scoped to https:cnpg-webhook-service:webhook-server.
The Job times out after 120s with 60 attempts at 2s intervals — longer
than any data-plane programming delay seen on E2E, but bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on
packages/extra/ingress/templates/cilium-lb-pool.yaml:19: if the operator
passes an externalIP already in CIDR form (192.0.2.10/32 or 2001:db8::1/128),
the template appended a second /32 or /128 suffix producing an invalid
CiliumLoadBalancerIPPool block. Guard the suffix append on the absence of
"/" in the input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add PackageSource for HAMi with dependency on gpu-operator.
Include HAMi in the iaas bundle as an optional package, allowing
host-level deployment alongside the existing kubernetes addon path.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The hami-dra subchart renders resources even when dra.enabled=false
because Helm dependency conditions don't work for vendored subcharts.
The subchart also contains duplicate YAML label keys
(app.kubernetes.io/component, app.kubernetes.io/name) and references
unpublished container images (v0.0.1-dev). DRA support can be
re-added when the upstream project stabilizes it.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Add generated-output dirs (pkg/generated, internal/crdinstall/manifests,
packages/system/*/definitions) and Makefile to the workflow paths: filter
so PRs that modify only generated artifacts still trigger the drift check.
- Mirror the same paths in the root pre-commit hook's files: regex so
manual edits to generated files or changes to the root generate target
re-run make generate through pre-commit.
- Switch drift detection in the workflow from `git diff --exit-code` to
`git status --porcelain` so new untracked files produced by make generate
(e.g. generated YAML/Go for a new API type) also fail the job; dump
`git diff --color=always` on failure for easier debugging.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Added comprehensive unit tests for authorization logic:
- hasAccessToNamespace() tests:
- User subject access (positive and negative cases)
- Group subject access
- ServiceAccount subject access
- ServiceAccount with empty namespace (defaults to RoleBinding ns)
- Privileged groups (system:masters, cozystack-cluster-admin)
- Get() handler tests:
- Returns namespace when user has access
- Returns NotFound when user lacks access (not Forbidden)
- Returns NotFound for non-tenant namespaces
All tests verify that authorization correctly enforces RoleBinding-based
access control and prevents IDOR vulnerabilities.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Adds explicit guidance to `docs/agents/contributing.md` about running
`make generate` in any touched package before committing. Pre-commit CI
runs `make generate` in every package and fails with exit code 123 on
any uncommitted generator output (regenerated `README.md`, reordered
`values.schema.json`, refreshed
`packages/system/<name>-rd/cozyrds/<name>.yaml`).
Recent PRs have tripped on this during review cycles. Documenting it in
the contributing checklist saves a round-trip.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Require regenerating and committing generated artifacts when
designated source files are edited.
* Added a section listing generated artifacts per package and concrete
regeneration/staging steps.
* Documented CI enforcement that detects unstaged generator output and
blocks PRs.
* Added guidance for rerunning generation after amended commits, plus
updated commit-scope guidance (now “not exhaustive”) and included
“agents” in allowed scopes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from gemini-code-assist on docs/agents/contributing.md:11:
Scope linters kept flagging valid scopes like 'agents' as unknown because
the list read as exhaustive. Annotate it as examples (not exhaustive) and
add 'agents' to the Other group so both humans and review bots stop
tripping on scopes that are already in regular use across the repo
history.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from gemini-code-assist on docs/agents/contributing.md:30:
Reword "likely to need regenerated" (regional construction) to
"likely needs to be regenerated" for standard technical prose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai on docs/agents/contributing.md:26:
Replace the hard-coded packages/extra/<name> path in the example with
packages/<apps-or-extra>/<name> so the example matches the preceding
text that describes both apps and extra packages.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Service.spec.externalIPs is deprecated upstream in Kubernetes v1.36
(KEP-5707, kubernetes#137293). The AllowServiceExternalIPs feature gate
is expected to default to off around v1.40 and the implementation to
be removed around v1.43. For bare-metal installs that rely on
externalIPs today, cozystack needs a migration path.
This change adds an opt-in 'loadBalancer' exposure mode for the
ingress-nginx Service:
- New platform value 'publishing.exposure' (enum: externalIPs |
loadBalancer, default externalIPs). Plumbed through cozystack-values
into each tenant's ingress HelmRelease via the new 'expose-mode' key.
- Unknown values and loadBalancer with an empty externalIPs list fail
the chart render with explicit error messages, rather than silently
producing a broken Service.
- When exposure=loadBalancer and the current namespace matches
publishing.ingressName, the Service becomes type: LoadBalancer with
externalTrafficPolicy: Local.
- A new template renders a CiliumLoadBalancerIPPool whose blocks come
from publishing.externalIPs (IPv4 addresses get /32, IPv6 addresses
get /128) and whose serviceSelector uses Cilium's synthetic
io.kubernetes.service.namespace key combined with the standard
app.kubernetes.io/name: ingress-nginx label. No custom label is
written to the Service itself, avoiding cross-tenant collisions from
user-defined labels.
Default behaviour is unchanged: without opting in, the Service is
still ClusterIP + spec.externalIPs as today.
Scope: only ingress-nginx is migrated by this setting. Other cozystack
components that still write Service.spec.externalIPs directly (notably
the vpn app) must be migrated separately before the v1.40 feature gate
flip.
Tests: packages/extra/ingress/tests/exposure_test.yaml adds 13
helm-unittest cases covering both modes, IPv4/IPv6, empty-token
filtering, unknown-mode rejection, and the non-matching-namespace
fallback.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous image digest in values.yaml pointed at a single-arch
linux/arm64 manifest because 'make image' was run from an arm64 host
with the default buildx platform. Cozystack targets amd64 (Talos build
output, E2E runners, most real-world clusters) and also arm64 for
hybrid fleets, so Helm install would fail on amd64 nodes with 'no
matching manifest for linux/amd64 in the manifest list entries'
whenever somebody installed directly from this commit between merge
and the next release-tag CI rebuild.
Fix: rebuilt the image locally with
PLATFORM='linux/amd64,linux/arm64' make image from a buildx
docker-container driver, pushed the multi-arch manifest, and
refreshed values.yaml with:
- digest of the new multi-arch manifest list (verified via
'docker manifest inspect': amd64 sha256:e1977323..., arm64
sha256:8f5ab529...).
- tag bumped from 'latest' (emitted by the common-envs.mk settag
macro on a non-tagged checkout) to '1.19.3', matching the
established convention in every other packages/system/*/values.yaml
so reviewers and incident response have a human-readable version
anchor independent of digest chasing.
The Makefile is left untouched so the CI builder (which only uses the
default docker driver) keeps building single-arch for whatever
architecture it runs on; multi-arch is a responsibility of the
release-tag pipeline or an explicit local rebuild.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Built ghcr.io/cozystack/cozystack/cilium from the refreshed upstream
v1.19.3 base image and updated values.yaml with the new digest.
Previously values.yaml still pointed at the v1.19.1 cozystack rebuild
by digest while Chart.yaml and the Dockerfile were on v1.19.3 — with
chart default useDigest=true that would have silently pulled v1.19.1
until the next release-tag rebuild.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Vendored chart refreshed via make update in packages/system/cilium.
Motivation: v1.19.2 fixes a cert-manager HTTP-01 bug on hostnames with
both HTTP and HTTPS listeners (cilium#44492, backport PR #44517). This
is a prerequisite for upcoming Gateway API work.
v1.19.3 is the latest stable release in the v1.19.x line (15 Apr 2026).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pre-commit CI runs make generate in every package and fails with exit
123 on any uncommitted generator output. Add explicit guidance so
agents stage regenerated README.md, values.schema.json and
packages/system/<name>-rd artifacts alongside the hand edits.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pulls the latest robotlb chart (0.1.3) which ships robotlb 0.0.6.
The new appVersion adds RBAC permissions for discovery.k8s.io/endpointslices
needed to support EndpointSlice-based services such as KubeVirt.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
## What this PR does
Fixes the failures that blocked the v1.3.0 release pipeline (workflow
run [24765377017]) and closes the gap in `docs/agents/changelog.md` that
made them possible.
### Bug 1 — pathspec error in `Create changelog branch and commit`
After the Copilot step, the workflow runs `git checkout -b
"$CHANGELOG_BRANCH" origin/main`. Copilot CLI was invoked with
`--allow-all-tools` and had committed the generated changelog onto HEAD
of `main`, so the reset to `origin/main` deleted the tracked file, and
`git add "$CHANGELOG_FILE"` then failed with `fatal: pathspec ... did
not match any files`.
Fix (commit 2):
- Snapshot the file across the branch switch so the checkout cannot drop
it.
- `set -euo pipefail` so any failure surfaces loudly.
- Drop the dead "no changes to commit" soft-branch — the earlier
`check_changelog` step gates this step on the file being absent from
`origin/main`, so `git commit` must produce a diff; if it doesn't (e.g.
empty file), fail loud instead of pushing an empty branch.
- Fail-early file-existence check with `::error::` annotation, and
`VERSION` moved to step `env:`.
### Bug 2 — no timeout on `Generate changelog using AI`
The re-run hung in Copilot for 10+ minutes with zero log output. With no
`timeout-minutes`, a hung Copilot would hold a self-hosted runner for 6
hours (job default).
Fix (commit 2): `timeout-minutes: 30` (the prior successful run took ~26
minutes).
### Root-cause fix — scope the agent prompt
`docs/agents/changelog.md` is the actual prompt driving Copilot. It
ended with "Save the changelog" and gave no boundary, so an agent with
`--allow-all-tools` could reasonably interpret "done" as "commit, push,
open a PR".
Fix (commit 1):
- Add a "Scope and boundaries" section at the top stating the single
deliverable is `docs/changelogs/v<version>.md` and enumerating forbidden
operations (git commit / push / branch / tag / reset / merge / rebase,
PR creation, GitHub API writes, modifying any file other than the
changelog).
- Add an explicit "then exit" at the end of Step 9 with a back-reference
to the scope section.
- Scoped "unless the caller explicitly instructs otherwise" so
interactive IDE use stays flexible.
With the rules in the doc, CI and interactive callers share the same
boundary, and the workflow prompt becomes a one-line invocation
(`--prompt "Generate the release changelog for tag v${VERSION}. Follow
the instructions in @docs/agents/changelog.md exactly, including the
'Scope and boundaries' section at the top. ..."`).
### Files touched
- `docs/agents/changelog.md` — +16 / -0 (scope section + exit rule)
- `.github/workflows/tags.yaml` — +36 / -31 (timeout + rewritten commit
step + simplified prompt)
Other jobs in the workflow (`prepare-release`, `update-website-docs`)
are untouched.
### Release note
```release-note
NONE
```
[24765377017]:
https://github.com/cozystack/cozystack/actions/runs/24765377017
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved release automation reliability with stricter verification,
enforced error handling, and a timeout for changelog generation
* Ensured automated commits/pushes occur only after successful output
validation to prevent accidental repository mutations
* **Documentation**
* Clarified agent instructions to produce a single changelog file and
terminate, and to restrict the agent to read-only inspection during
generation
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
hack/update-codegen.sh sources kube_codegen.sh from the Go module
cache at ~/go/pkg/mod/k8s.io/code-generator@vX.Y.Z/, but the module
is not declared in go.mod so a fresh runner has nothing to source
from. Add a workflow step that parses the pinned version out of the
script and pulls the module into the cache before running
make generate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Post-release cleanup of `docs/changelogs/v1.3.0.md` so the notes match
what users actually experience in v1.3.0. No code changes.
- **Rewrite the postgres major-features entry** so author
(`@myasnikovdaniil`), PR (`#2369`), and description all line up with the
`17.7-standard-trixie` pin + migration-37 `imageName` rewrite that
actually shipped. The previous entry credited `#2304` with a description
matching a superseded `spec.version=v17` backfill approach.
- **Remove the duplicate `#2364` postgres bug-fix entry** — the same
work is now folded into the single major-features entry above, with
backport references to `#2309` (v1.2.1) and `#2364` (v1.2.2).
- **Remove the `[linstor-gui] Restrict to cozystack-cluster-admin group`
security entry.** The vulnerable state never shipped in a tagged
release, so there is nothing user-facing to announce. The
`cozystack-cluster-admin`-group restriction is already described in the
linstor-gui Feature Highlights section as part of the feature's day-one
shipping behavior.
### Release note
```release-note
[]
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated v1.3.0 changelog with clarified PostgreSQL system version
pinning details and removed redundant entries for improved documentation
clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The etcd chart shipped a `post-upgrade` Helm hook that `kubectl delete`d the
etcd TLS chain (`etcd-{ca,peer-ca,client,peer,server}-tls`) and then deleted
etcd pods on every chart upgrade, gated by a semver compare of an
`etcd-deployed-version` ConfigMap against `2.6.1`.
The hook was added as a one-shot migration for the chart `2.6.0 -> 2.6.1`
transition. Since commit f871fbdb ("Remove versions_map logic") all chart
versions are stamped as `0.0.0+<git-hash>`, which per semver is always
`< 2.6.1`. The gate therefore always resolves to "update certs", firing the
destructive hook on every etcd upgrade. On clusters running Kamaji-managed
tenant control planes this wipes the etcd CA, cert-manager re-issues it, and
tenant kube-apiservers hit `x509: certificate signed by unknown authority`
against `etcd.<ns>.svc:2379` until each tenant DataStore is manually
re-reconciled.
Commit 47d81f70 ("Disabled private key rotation in CA certs") already fixed
the underlying `rotationPolicy: Always` issue the migration was papering
over, so the hook has no remaining use. Remove the hook Job, its RBAC, the
version ConfigMap it read, and add a helm-unittest suite under
`packages/extra/etcd/tests/` that guards against re-introducing the hook
or the version ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the Generate changelog using AI step ran
Copilot with --allow-all-tools and GH_TOKEN set to the write-capable
installation token issued to the job (contents: write,
pull-requests: write on all cozystack/* repos). The scope rules in
docs/agents/changelog.md and the step prompt tell the agent not to
use those permissions, but nothing at the token layer prevented it.
Mint a second, read-only installation token from the same app
(same COZYSTACK_CI_APP_ID / COZYSTACK_CI_PRIVATE_KEY, scoped to
contents/pull-requests/metadata read) and pass that one to the AI
step instead. The write-capable token is still used by the checkout,
commit/push, and PR-creation steps that actually need it.
This is defense in depth: even if a future prompt change or agent
misbehavior ignored the scope rules, the token itself has no write
capability on any repository in the cozystack org. No new secret,
no new GitHub App install, no admin-side change — the RO token is
minted in the same workflow from the same app credentials.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the existing `[ -f ]` check catches
missing files but a zero-byte `docs/changelogs/v${VERSION}.md` would
still be staged and committed — `git add` + `git commit -s` on a new
empty file succeeds and produces a real commit, leaving the
downstream PR with no actual changelog content.
Add a `[ -s ]` guard after the existence check: if the Generate
changelog using AI step produces an empty file, emit a matching
`::error::` annotation and exit 1 before snapshotting.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the previous "do not write to refs, HEAD,
or remotes" wording contradicted the explicit allowance of `git fetch`
(which updates remote-tracking refs) and the mandatory cross-repo
checks in Step 6, which `cd` into `_repos/<repo>` and run
`git checkout`, `git pull`, etc.
Sharpen the scope paragraph:
- The git-write ban is now explicitly scoped to the cozystack working
tree — it bans writing to local branches, tags, or HEAD in that
repo, not "refs/HEAD/remotes" globally.
- `git fetch` is called out as expected.
- Local git operations inside disposable `_repos/` clones
(`git checkout`, `git pull`, etc.) are explicitly allowed, with
the remaining rules (no push, no PR creation, no API writes)
applying to any repository.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `backupstrategy-controller` chart declared a `Role` and
`RoleBinding` scoped to the `cozy-velero` namespace (for managing
`ResourceModifier` ConfigMaps consumed by Velero Restore). Because
`cozystack.velero` is an optional package, that namespace does not exist
in bundles that do not enable velero — and `backupstrategy-controller`
is a **default** package. Helm install aborted with:
```
namespaces "cozy-velero" not found
```
which blocked the entire
`cozy-backup-controller/backupstrategy-controller` HelmRelease on any
cluster where velero was not explicitly enabled (including the E2E
environment).
This PR moves that Role/RoleBinding into the velero chart
(`packages/system/velero/templates/backupstrategy-controller-rbac.yaml`),
so the permission grant only exists when velero is actually installed —
where it is useful. The RoleBinding subject points to the stable
`backupstrategy-controller` ServiceAccount in `cozy-backup-controller`.
### Release note
```release-note
fix(backups): moved the velero-namespaced ResourceModifier ConfigMap Role and RoleBinding from the backupstrategy-controller chart into the velero chart. This unblocks installs of backupstrategy-controller on bundles that do not enable velero (previously the HelmRelease failed with `namespaces "cozy-velero" not found`).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Reorganized RBAC configuration for the backup strategy controller by
consolidating namespace-scoped role definitions in the Velero chart
template
* Updated role bindings and permissions structure across system packages
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The backupstrategy-controller chart declared a Role/RoleBinding in the
cozy-velero namespace for ResourceModifier ConfigMap management. Because
velero is an optional package, that namespace does not exist in bundles
without velero, so Helm install aborted with "namespaces \"cozy-velero\"
not found" and blocked the default install of backupstrategy-controller.
Move the Role and RoleBinding into the velero chart so they are created
only when velero is actually installed. The RoleBinding subject points
to the backupstrategy-controller ServiceAccount in its fixed namespace
(cozy-backup-controller).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes to the generate-changelog job to fix the v1.3.0
release pipeline failure (run 24765377017) and make the job robust
to whatever state the Copilot step leaves behind.
1. Add `timeout-minutes: 30` to the Generate changelog using AI
step. On the v1.3.0 re-run the step hung silently for 10+
minutes; with no timeout a hung Copilot would hold a self-hosted
runner for up to 6 hours (job default). The previous successful
run took ~26 minutes, so 30 is a reasonable ceiling.
2. Replace the terse, ambiguous Copilot prompt with a one-liner
that invokes docs/agents/changelog.md directly. The "Scope and
boundaries" section added to that doc in the previous commit is
now the single source of truth for what the agent may and may
not do, so the workflow only needs to pass the version and
point at the relevant doc. VERSION is moved to step env: to
match GitHub's workflow-injection hardening guidance.
3. Rewrite the Create changelog branch and commit step:
- add `set -euo pipefail` so any failure is visible
- validate the file exists up front and fail loud with
`::error::` if not
- copy the file to a tempfile BEFORE `git checkout -b`, so the
checkout to `origin/main` cannot remove it (this is the fix
for the original pathspec error the v1.3.0 run hit)
- use `trap` to clean up the tempfile on any exit path
- move VERSION to env
- drop the dead "no changes to commit" branch: the check_changelog
step earlier in the job gates this step on the file being
absent from origin/main, so `git add` + `git commit` must
produce a diff. If they don't (e.g. Copilot emitted an empty
file), fail loud instead of pushing an empty branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The v1.3.0 release pipeline broke because Copilot, invoked by
.github/workflows/tags.yaml with --allow-all-tools, committed the
generated changelog onto HEAD of main on its own. The workflow's
next step — `git checkout -b ... origin/main` — then wiped the file,
and `git add` failed with a pathspec error.
The root cause is in this document. The checklist ends with "Save
the changelog", which an agent with broad tool access can reasonably
interpret as "also commit it, push it, and open a PR". There was no
explicit boundary.
Add a "Scope and boundaries" section at the top and an explicit
"then exit" at the end of Step 9:
- The single deliverable is docs/changelogs/v<version>.md.
- Forbidden by default: git commit / push / checkout (to switch
branches) / branch / tag / reset / merge / rebase; PR creation;
GitHub API writes (POST/PATCH/DELETE); modifying any file other
than the changelog.
- Read-only analysis (git log/show/fetch/diff, gh pr view, gh api
GET) remains expected.
- Auxiliary repo clones under _repos/ remain allowed for cross-repo
analysis per Step 6.
- Scoped "unless the caller explicitly instructs otherwise" so
interactive use with an IDE remains flexible.
With the rules in the doc, CI and interactive callers share the
same boundary; the workflow can invoke the doc with a one-line
prompt instead of re-stating the constraints every time.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Post-release cleanup of docs/changelogs/v1.3.0.md so the notes match
what users actually experience in the released v1.3.0:
- Rewrite the postgres major-features entry so author (myasnikovdaniil),
PR (#2369), and description all match the 17.7-standard-trixie pin +
migration-37 imageName rewrite that actually shipped. The previous
entry credited #2304 (superseded spec.version=v17 backfill approach).
- Remove the duplicate #2364 postgres bug-fix entry; the same work is
now folded into the single major-features entry above, with backport
references to #2309 (v1.2.1) and #2364 (v1.2.2).
- Remove the [linstor-gui] Restrict to cozystack-cluster-admin group
security entry. The vulnerable state never shipped in a tagged
release, so there is nothing user-facing to announce; the restriction
is already described in the linstor-gui Feature Highlights section
as part of the feature's day-one behavior.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This PR adds the changelog for release `v1.3.0`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.3.0.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Published Cozystack v1.3.0 release notes featuring LINSTOR scheduler
extender for storage-aware pod placement, managed LINSTOR GUI web
console, VM Default Images catalog, expanded observability with Events
dashboard and S3 metering, cross-namespace VMInstance backup/restore,
and various platform enhancements and bug fixes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
HAMi-core relies on _dl_sym (private glibc symbol removed in 2.34)
for CUDA interception. This breaks compute isolation on modern
container images using Ubuntu 22.04+ and makes Alpine/musl completely
incompatible.
Include upstream issue references and a compatibility matrix so users
can make informed decisions about base image selection.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add HAMi HelmRelease to the kubernetes app as a toggleable addon.
When enabled, GPU Operator's built-in device plugin is automatically
disabled via a values merge to avoid conflicts with HAMi's own
device plugin.
The HelmRelease fails fast if GPU Operator is not enabled, since HAMi
depends on it for driver management and container toolkit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add HAMi v2.8.1 (CNCF Sandbox) as a new system package for fractional
GPU sharing in tenant Kubernetes clusters. The chart enables workloads
to request specific amounts of GPU memory and compute cores instead of
claiming entire GPUs.
Vendored upstream chart includes device plugin, scheduler extender,
mutating webhook, and DRA subchart. Wrapper values configure nvidia
RuntimeClass and clear hardcoded node config from upstream defaults.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Harbor is a PaaS service (`category: PaaS`), not a tenant module. It is
not deployed automatically into tenant namespaces — there is no
corresponding manifest in `packages/apps/tenant/templates/`, unlike
actual tenant modules (monitoring, ingress, etcd, info, seaweedfs).
Two flags were incorrectly set on its `ApplicationDefinition`:
- `spec.dashboard.module: true` — caused Harbor to appear in the sidebar
"Modules" section and be excluded from its proper PaaS category.
- `spec.release.labels."internal.cozystack.io/tenantmodule": "true"` —
caused controllers (`pkg/registry/core/tenantmodule`, dashboard) to
treat the Harbor HelmRelease as a tenant module.
Both flags are removed so Harbor is listed correctly under PaaS and
handled as an ordinary managed application.
### Release note
```release-note
fix(harbor): remove incorrect tenant module flags from Harbor ApplicationDefinition so it appears under the PaaS category in the dashboard and is no longer treated as a tenant module
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Simplified internal application configuration settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Bumps `packages/system/kubeovn` to `cozystack/kubeovn-chart` tag
`v1.15.10-cozy.1`, which:
1. Updates upstream kube-ovn from v1.15.3 to v1.15.10 (latest patch in
the v1.15 series).
2. Carries a patch over `pkg/controller/pod.go` that preserves a VM
LSP's port-group memberships when kubernetes GCs a completed
virt-launcher pod while another virt-launcher pod of the same VM is
still running.
Without the patch, the destination pod of a successful live migration
loses its security groups, network policies and node-scoped routing
after kubernetes cleans up the source pod, and only recovers after a
`kube-ovn-controller` restart. The buggy code path is identical between
v1.15.3, v1.15.10, release-1.15 HEAD and master HEAD — confirmed by diff
and by reproduction on a cozystack cluster.
- Upstream issue: kubeovn/kube-ovn#6665
- Upstream fix PR: kubeovn/kube-ovn#6666
- Chart carry: cozystack/kubeovn-chart#4
### Release note
```release-note
fix(kube-ovn): bump kube-ovn to v1.15.10 and carry upstream fix that keeps VM LSP port-group memberships when kubernetes GCs a completed virt-launcher pod (post-migration regression)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VpcEgressGateway CRD extended with optional resources
(claims/limits/requests) and bandwidth (ingress/egress) fields.
* Added a pre-upgrade hook to run compatibility steps before upgrades.
* CNI DaemonSet now optionally respects MTU when configured.
* **Bug Fixes**
* Updated Kube-OVN and NAT gateway container images to the latest
maintenance release.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Fixes an RBAC failure in `kube-ovn-plunger` that prevented it from
reconciling the `ovn-central` Deployment:
```
deployments.apps is forbidden: User "system:serviceaccount:cozy-kubeovn:kube-ovn-plunger"
cannot list resource "deployments" in API group "apps" at the cluster scope
```
Two causes addressed:
- The controller-runtime cache issued cluster-wide list/watch for
Deployments and Pods by default. The manager cache is now scoped to the
kube-ovn namespace, so requests go to the namespaced endpoints covered
by the Role.
- The Role's deployments rule used `resourceNames`, which does not apply
to list/watch in Kubernetes RBAC. The restriction is removed; the Role
is still namespace-scoped via RoleBinding to the `cozy-kubeovn`
namespace.
`--kube-ovn-namespace` and `--ovn-central-name` are now passed to the
binary from chart values so the previously unused value is wired up.
### Release note
```release-note
fix(kube-ovn): fix kubeovn-plunger RBAC forbidden error on ovn-central Deployment reconcile
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable parameters for KubeOVN namespace and OVN central
name in deployment configuration.
* **Improvements**
* Enhanced cache configuration with namespace-scoped resource management
capabilities.
* Updated RBAC permissions to remove resource-specific constraints,
enabling broader deployment access.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Options field was added to RestoreJobSpec without re-running
'make generate', leaving zz_generated.deepcopy.go out of sync.
Regenerated to include the missing DeepCopyInto handling for the
runtime.RawExtension pointer.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Run root 'make generate' as a pre-commit hook and as a dedicated
CI workflow so missed codegen updates (CRDs, deepcopy, clients, RBAC)
are caught instead of merging stale generated files.
Pre-commit hook is scoped to files that actually affect codegen
(api/, pkg/apis/, hack/update-codegen.sh, hack/boilerplate.go.txt)
so unrelated commits are not slowed down. CI job sets up Go from
go.mod, runs make generate, and fails on drift with a pointer to
the local fix.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Harbor is a PaaS service, not a tenant module. It is not deployed
automatically into tenant namespaces (no manifest in
packages/apps/tenant/templates/). Remove the misplaced
`dashboard.module: true` flag and
`internal.cozystack.io/tenantmodule: "true"` release label so Harbor
appears under the PaaS category in the dashboard and is not treated as
a tenant module by the controllers.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Pulls cozystack/kubeovn-chart v1.15.10-cozy.1, which bumps upstream
kube-ovn from v1.15.3 to v1.15.10 and carries a patch over
pkg/controller/pod.go that preserves a VM LSP's port-group memberships
when kubernetes GCs a completed virt-launcher pod while another
virt-launcher pod of the same VM is still running.
Without the patch, the destination pod of a successful live migration
loses its security groups, network policies and node-scoped routing
after kubernetes cleans up the migration source pod, and only recovers
after a kube-ovn-controller restart.
Upstream issue: kubeovn/kube-ovn#6665
Upstream fix PR: kubeovn/kube-ovn#6666
Chart carry: cozystack/kubeovn-chart#4
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
- Introduce list, details, create, and sidebar views for `RestoreJob` in
the dashboard controller.
- Auto-generate stable component IDs for sidebar entries and detail
sub-blocks so upserts remain consistent across reconciles.
- Render a single "Same as backup" fallback for an omitted
`spec.targetApplicationRef` on the details view instead of concatenating
three missing-path fallbacks.
- Mark non-CRD-backed `stock-project-factory-*-details` sidebars
(`kube-*`, `plan`, `backupjob`, `backup`, `restorejob`) as static so
they pick up consistent managed-by labels.
### Release note
```release-note
[dashboard] Add RestoreJob views (list, details, create, sidebar link) to the dashboard controller.
```
## Test plan
- [x] Deploy controller to dev stand and open the restore pages in the
dashboard
- [x] Create a RestoreJob through the UI against an existing Backup;
reconcile succeeds and the details view renders
- [ ] CI green
## Previous PR
https://github.com/cozystack/cozystack/pull/2387
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Introduced comprehensive RestoreJobs dashboard interface enabling
users to view and manage backup restoration operations.
* New RestoreJobs section added to the Backups menu with dedicated
details page and navigation.
* Dashboard displays restore job status, phase, backup references,
target applications, and operation timestamps for easy monitoring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The kubeovn-plunger controller-runtime cache attempted cluster-wide
list/watch on Deployments and Pods, which the namespace-scoped Role
cannot satisfy. Additionally, the deployments rule relied on
resourceNames, which does not restrict list/watch verbs and left the
permission effectively unusable.
Scope the manager cache to the kube-ovn namespace so list/watch hit
the namespaced API, and drop resourceNames from the deployments rule.
Wire --kube-ovn-namespace and --ovn-central-name through the
Deployment so values are actually consumed.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
HTTP-01 ACME issuance in Cozystack has been relying on deprecated
cert-manager fields that happen to still work, but point at a
non-existent ingress class on default installs.
**The bug:**
- `ClusterIssuer`s (`letsencrypt-prod`, `letsencrypt-stage`) hardcode
`http01.ingress.class: nginx`, but Cozystack's nginx ingress classes are
named after tenants (`tenant-root`, `tenant-*`) — there is no class
called `nginx`.
- Every ingress that needs a cert overrides this with the deprecated
annotation `acme.cert-manager.io/http01-ingress-class: <class>`.
cert-manager still honors it, which is why issuance works at all.
- Mixing the modern `ingressClassName` field with the legacy
`http01-ingress-class` annotation produces `fields ingressClassName and
class cannot be set at the same time` (cert-manager#6651), so the
migration has to be atomic.
**The fix:**
- Parameterize the `ClusterIssuer` HTTP-01 solver from
`_cluster.expose-ingress` (default `tenant-root`) using the modern
`ingressClassName` field.
- Rename the override annotation on all ingresses to the modern
`acme.cert-manager.io/http01-ingress-ingressclassname`.
- No new features, no behavior change for operators who already set
`_cluster.expose-ingress` correctly — just removes the dependency on the
deprecated API and stops pointing at a phantom class.
Touched ingresses (9 templates): keycloak, dashboard, linstor-gui,
bucket, grafana, alerta, bootbox/matchbox, seaweedfs, harbor. Plus
`seaweedfs` values.yaml default annotation and the
`cert-manager-issuers` ClusterIssuer template.
Verified renders for `http01` + `tenant-root`, `http01` + custom class,
and `dns01` (correctly omits the http01 branch).
### Release note
```release-note
fix(platform): ACME HTTP-01 challenges now use the modern `ingressClassName` API and target the tenant ingress class from `_cluster.expose-ingress` (default `tenant-root`) instead of a hardcoded `nginx` class that did not exist on most installs. Ingress templates switched from the deprecated `acme.cert-manager.io/http01-ingress-class` annotation to `acme.cert-manager.io/http01-ingress-ingressclassname`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Chores**
* Updated cert-manager ACME HTTP-01 solver annotations across ingress
configurations to use the newer `ingressclassname` format for improved
compatibility with current cert-manager versions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Updates the `update-website-docs` job in `.github/workflows/tags.yaml`
to match the new docs-versioning contract introduced in
cozystack/website#495.
The website repo replaces the old "pre-create `vX.Y/` draft directory"
scheme with a permanent `content/en/docs/next/` trunk. Released version
directories are no longer implicitly pre-created — the upstream release
workflow owns the explicit decision of when to promote `next/` →
`vX.Y/`.
### Changes
- **New step `Determine if this release promotes next/`** — parses the
tag and only sets `promote=true` for final `vX.Y.Z` tags where
`content/en/docs/vX.Y/` does not already exist. Prereleases
(`vX.Y.Z-rc.N`, etc.) and patch releases skip promotion.
- **New step `Promote next/ to released version`** — gated on
`promote=true`; runs `make release-next RELEASE_TAG=<tag>`.
- **Step order**: `release-next` runs **before** `update-all`. This way
`update-all` routes into the freshly-created `vX.Y/` directory (per the
website Makefile's routing logic), and `next/` stays untouched as the
trunk for the following release.
- **`git add content hugo.yaml`** — `release-next` registers the new
version in `hugo.yaml` via `register_version.sh`, so the commit step
must stage it.
- Converted the existing `update-all` step to use `env:` vars for
consistency with the new step and workflow-injection hardening.
### Behaviour by tag
| Tag | `release-next`? | `update-all` targets |
|-----|-----------------|----------------------|
| `v1.3.0` (v1.3 does not exist) | yes — promotes next/ → v1.3/ | v1.3/
|
| `v1.3.1` (v1.3 exists) | no — skipped | v1.3/ |
| `v1.3.0-rc.1` | no — skipped (prerelease) | next/ |
| `v2.0.0` (v2.0 does not exist) | yes — promotes next/ → v2.0/ | v2.0/
|
### Release note
```release-note
Release tagging now promotes the website's `next/` docs trunk to `vX.Y/` when cutting a new minor or major release (requires cozystack/website#495 to be merged first).
```
### Dependency
Must not merge until cozystack/website#495 is merged on `main` of the
website repo, otherwise `make release-next` won't exist there and
new-minor release tags will fail.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated release automation to apply conditional logic for version
promotion based on tag patterns and documentation availability. Modified
documentation staging behavior in release workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Replace rename-only wording on spec.targetApplicationRef form labels so
users discover that the field also supports restoring into a different
application, not just renaming.
- Drop VMInstance-specific keys from the spec.options form label; the
restore options are driver-specific, so hardcoding one driver's keys is
misleading once other drivers land.
- Render a single "Same as backup" fallback for an omitted
spec.targetApplicationRef on the details page instead of the noisy
"-.-/-" produced by concatenating three missing path fallbacks.
- De-duplicate the time-block / time-icon / time-value component IDs
within the restorejob-details factory by prefixing them with
created-, started-at-, and completed-at-.
- Mark non-CRD-backed stock-project-factory-*-details sidebars (kube-*,
plan, backupjob, backup, restorejob) as static in
upsertMultipleSidebars so they pick up consistent managed-by labels
instead of being left unmanaged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
ClusterIssuer solver referenced IngressClass "nginx" which does not
exist on cozystack clusters — real classes are named after tenant
namespaces (e.g. tenant-root). Cert issuance only worked because every
requesting Ingress overrode the ClusterIssuer via the legacy
acme.cert-manager.io/http01-ingress-class annotation.
Switch both sides to the modern cert-manager API (available since
cert-manager 1.12; cozystack ships 1.19.3):
- ClusterIssuer: http01.ingress.ingressClassName, value parameterized
from _cluster.expose-ingress (default "tenant-root")
- Ingress annotation: http01-ingress-ingressclassname
These must migrate together — mixing ingressClassName (ClusterIssuer)
with the old http01-ingress-class annotation triggers cert-manager's
"fields ingressClassName and class cannot be set at the same time"
validation and breaks issuance.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Update oldName and serverName field descriptions based on code review
feedback to avoid confusion about their actual roles:
- oldName: Remove misleading "(matches serverName in backup.info)"
text. This field represents the Kubernetes cluster resource name,
not the Barman server name.
- serverName: Provide clearer explanation that it's the S3 path prefix
(barmanObjectStore.serverName) used by the original cluster, and should
only be set when it differs from the Kubernetes resource name.
Updated in:
- values.yaml (source of truth for field documentation)
- types.go (Go API type comments)
- values.schema.json (JSON schema for validation)
- postgres.yaml (CRD with embedded OpenAPI schema)
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add serverName field to bootstrap configuration to explicitly specify
Barman server name from backup.info. This fixes "no target backup found"
errors when server_name in backup.info differs from Kubernetes cluster name.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Upgrades the vendored `victoria-metrics-operator` Helm chart from
`0.59.1` to `0.61.0` (operator appVersion `v0.68.1` → `v0.68.4`) by
re-running `make update` in
`packages/system/victoria-metrics-operator/`.
Picks up upstream bugfixes in the `v0.68.x` operator line:
- Correct `VMPodScrape` port for `VMAgent` / `VLAgent` (previously
misrouted, causing scrape failures)
- `StatefulSet` pod *deletion* (not eviction) when `maxUnavailable=100%`
— matches the minimum-downtime upgrade strategy
- `VMDistributed` PVC no longer owned by both the `StatefulSet` and the
top-level object
- Finalizer cleanup on core Kubernetes resources and correct finalizer
targeting for `VLAgent`/`VLogs`/`VMAgent`/`VMSingle`
- Ingest-only `VMSingle`/`VMAgent` no longer mount scrape config secrets
(avoids RBAC errors)
- `StatefulSet` recreation when immutable fields change
- Operator waits for PVC resize completion before continuing reconcile
Chart `0.60.0` is skipped deliberately: it switched the operator
Deployment's selector to `app.kubernetes.io/component`, which required
Deployment recreation on upgrade. `0.61.0` reverts that change, so
jumping straight from `0.59.1` to `0.61.0` avoids the disruption.
Local patch `patches/disable-ca-key-rotation.patch` still applies
cleanly against the new `templates/webhook.yaml`.
### Release note
```release-note
[monitoring] Upgrade victoria-metrics-operator chart to 0.61.0 (operator v0.68.4) for upstream STS/PVC bugfixes, correct VMPodScrape port, and finalizer cleanup.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for configuring Deployment rolling update strategy
* Added Deployment unhealthyPodEvictionPolicy for Pod Disruption Budgets
* Added webhook annotations configuration
* Extended CRD with statefulRollingUpdateStrategyBehavior and
vmauth.enabled settings
* **Bug Fixes**
* Reverted Deployment matchLabels change from release 0.60.0
* **Documentation**
* Added changelog links to documentation
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds @myasnikovdaniil to the default owners in `.github/CODEOWNERS` so
PRs automatically request review from me alongside the existing
maintainers.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated internal repository code ownership configuration to extend
responsibility coverage.
---
**Note:** This release contains only internal administrative updates
with no end-user-facing changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Increases the LINSTOR satellite startup probe `failureThreshold` from
the Kubernetes default of 3 (30 seconds) to 30 (300 seconds / 5
minutes).
The default startup probe is too aggressive for nodes with slow storage
initialization — satellites get killed and restarted before they have a
chance to come up. This adds an explicit `startupProbe` override in the
`LinstorSatelliteConfiguration` pod template, keeping the same
`tcpSocket` check on the `linstor` port but giving satellites
significantly more time to start.
### Release note
```release-note
fix(linstor): increase satellite startup probe failure threshold from 3 to 30 (5 minutes to start)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a startup health check for the LINSTOR satellite container to
improve pod reliability and ensure the service fully initializes before
accepting traffic.
* This change only introduces container startup probing behavior and
does not alter other runtime behavior or public interfaces.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The website repo is switching to a permanent `content/en/docs/next/`
trunk (cozystack/website#495). Released version directories are no
longer pre-created — the upstream release workflow is now responsible
for explicitly promoting `next/` → `vX.Y/` on new minor/major releases
via `make release-next`.
Update the `update-website-docs` job to match that contract:
- New `Determine if this release promotes next/` step parses the tag
and only enables promotion for final `vX.Y.Z` tags where
`content/en/docs/vX.Y/` does not already exist. Prereleases and
patch releases keep the old behaviour (target routing is handled
by the website Makefile on its own).
- New `Promote next/ to released version` step runs
`make release-next RELEASE_TAG=...` before `make update-all`, so
the subsequent `update-all` routes into the freshly-created
`vX.Y/` directory and `next/` stays untouched as the trunk for
the following release.
- `git add content hugo.yaml` to pick up the version registration
that `release-next` writes via `register_version.sh`.
- Convert the existing `update-all` step to use `env:` vars,
matching the new step and addressing the workflow-injection
hardening guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Bumps the vendored victoria-metrics-operator chart from 0.59.1 to 0.61.0
(operator appVersion v0.68.1 to v0.68.4) via `make update`.
Picks up upstream bugfixes in the v0.68.x line: correct VMPodScrape port
for VMAgent/VLAgent, StatefulSet pod deletion when maxUnavailable=100%,
VMDistributed PVC ownership fix, finalizer cleanup, ingest-only mode not
mounting scrape config secrets, STS recreation on immutable field changes,
and PVC resize completion wait.
Chart 0.60.0 is skipped because it introduced a matchLabels change
requiring Deployment recreation that was reverted in 0.61.0.
Local patch disable-ca-key-rotation.patch reapplies cleanly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* WorkloadMonitor labels with the workloads.cozystack.io/ prefix are now
propagated onto created Workloads; created Workloads always include the
reserved workloads.cozystack.io/monitor label and source-object labels
take precedence on conflicts.
* Helm app charts now add workloads.cozystack.io/resource-preset
metadata to WorkloadMonitor manifests.
* **Tests**
* Added tests covering label extraction, propagation, conflict
resolution, and backward compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Make the 5-minute timeout self-documenting by setting periodSeconds: 10
explicitly rather than relying on the Kubernetes default.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The default Kubernetes startup probe allows only 30 seconds (3 retries
× 10s) for LINSTOR satellites to become ready. This is insufficient on
nodes with slow storage initialization, causing unnecessary pod restarts.
Raise failureThreshold to 30, giving satellites up to 300 seconds (5
minutes) to complete startup.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Increase memory limit from 500Mi to 512Mi
- Increase memory request from 100Mi to 256Mi
- Add startup probe with 60s timeout (12 attempts × 5s)
- Increase readiness/liveness initialDelaySeconds from 5/15 to 30s
This fixes OOMKilled crashes observed in production where kamaji
controller was being killed due to insufficient memory during startup.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The /1e9 divisor in gpu:{power,thermal}_throttle_fraction:rate5m was
derived empirically against DCGM 3.x on A10 — the counter documents
itself as microseconds but ticks in nanoseconds in practice. If a
future exporter release honors the documented units, pre-clamp values
would exceed 1.0 while clamp_max(..., 1) silently masks the drift,
plateauing every throttle fraction at 100% and making the panels
lie in unison.
Add a validation group that fires when the raw max/1e9 value exceeds
1.0 for 15m, so we notice and rescale to /1e6 before dashboards
silently mislead operators.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
pod:util_per_watt:avg5m divided two DCGM metrics without an explicit
on(...) clause, so the match used the intersection of their label
sets. If dcgm-exporter relabeling ever diverges between
DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_POWER_USAGE (e.g. a pod-mapping
label appears on one but not the other after a config change), the
entire result drops to empty silently. Pin the match to the labels we
group by so divergence becomes a missing side, not a missing rule.
Throttle fractions had a related shape problem: dcgm-exporter emits
one series per GPU for each pod-mapping combination. On a shared GPU
(restart races, MIG/MPS) the same physical counter appears under
multiple pod labels and downstream avg(...) panels get diluted by the
pod count. Fold duplicates with max by (Hostname, gpu, UUID) before
clamp_max so the fraction is tied to the physical GPU.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
kube-state-metrics keeps kube_pod_container_resource_requests series
for Failed/Succeeded pods until the apiserver garbage-collects them,
which could inflate :allocated beyond what tenants actually hold and
drive cluster:gpu_count:free negative.
Join the request metric against kube_pod_status_phase filtered to
Pending|Running — the canonical pattern from Kubernetes' own
container_resource recording rules — on both the cluster and namespace
aggregates. Add clamp_min(..., 0) on cluster:gpu_count:free as a
second line of defence against transient label drift between
kube-state-metrics and DCGM.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Drop the hardcoded metadata.namespace so the rule inherits the chart's
release namespace, and add an explicit empty params field on every
group for schema consistency. No behavior change.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The Pending GPU pods counter on gpu-quotas joined raw
kube_pod_container_resource_requests (per-container series) against
kube_pod_status_phase (per-pod series). Multi-container pods were
counted once per requesting container instead of once per pod, so the
widget over-reported whenever a Pending pod had more than one GPU
container. Collapse the requests to pod level before the join.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Pod-level panels on the efficiency dashboard and DCGM-level panels on
the performance dashboard ignored the $namespace template variable, so
changing it left the visualizations unchanged. Add the filter to each
query. Performance-side queries use the `$namespace|` empty-tolerant
form so host-level DCGM series without a namespace label remain
visible when a specific namespace is selected.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Explain why tolerations: [{operator: Exists}] is safe on the driver
compat DaemonSet: the nodeSelector already confines scheduling to GPU
nodes, so the blanket toleration only kicks in when those nodes carry
the dedicated=gpu / nvidia.com/gpu taints that the GPU Operator's
default policy and many deployments apply.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Clarify that the "Average utilization" panel on gpu-fleet reflects the
legacy NVML view (DCGM_FI_DEV_GPU_UTIL) rather than engine-active
profiling. For AI/LLM workloads the NVML number is optimistic; the
gpu-efficiency dashboard carries the profiling-based view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align pod:tensor_saturation:avg5m with namespace:tensor_active:avg and
DCGM's native 0..1 range by dropping the * 100 from the recording rule
and multiplying at display time in gpu-efficiency.json. Also scope
pod:util_per_watt:avg5m with avg by (Hostname, gpu, UUID, namespace,
pod) so the series mirrors pod:tensor_saturation's grouping and stays
usable in topk queries.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
gpu-fleet.json references DCGM_FI_DEV_POWER_MGMT_LIMIT for its
"TDP vs draw" panel, but the custom DCGM Exporter CSV did not declare
it, so the panel silently rendered "No data" on clusters using that
config. Declare the counter, fix the dashboards table in the
gpu-operator examples README, and add a bats test that cross-checks
every DCGM_FI_* reference in tracked dashboards and recording rules
against the union of the upstream default set (snapshotted under
hack/) and the project's custom CSV.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-efficiency: scope Tensor Saturation, Util-per-Watt and Power
Throttle stats to the $namespace selector. Cluster-wide means were
misleading when a user had narrowed the dashboard to specific
tenants — the headline numbers lied relative to the panels below.
- gpu-fleet: show per-node power draw as % of combined TDP cap
(DCGM_FI_DEV_POWER_MGMT_LIMIT) instead of raw watts. Thresholds
(60 / 80 %) generalize across GPU SKUs without per-model tuning.
- gpu-quotas: read cluster:gpu_count:allocated from the recording
rules instead of recomputing sum(kube_pod_container_resource_requests)
inline. Keeps the dashboard aligned with the canonical definition
in gpu-recording.rules.yaml so the two can't drift.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- namespace:gpu_count:sum — never consumed by any tracked dashboard;
the billable view is already covered by namespace:gpu_count:allocated,
and the admin view by cluster:gpu_count:allocated.
- namespace:energy_joules:sum — no panel integrates joules; kWh
readings on the tenant dashboard compute their own integrations
from namespace:power_watts:sum.
- pod:tensor_to_nvml_ratio:avg5m — interesting tenant signal in
theory, but not wired into any panel and carrying it just burns
cardinality on large fleets.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- gpu-fleet: cluster-wide admin view — inventory, capacity (total /
allocated / free), per-node utilization and power, throttling,
temperatures, XID errors.
- gpu-tenants: per-namespace view — live allocation, utilization,
tensor saturation, power, and 24h GPU-hours / kWh integrations for
billing inputs.
Register both under gpu/* in dashboards-infra.list so they ship as
GrafanaDashboard CRs and fall under the bats cross-check introduced
earlier on this branch.
Update examples/README to spell out which DCGM counters each of the
five gpu/* dashboards actually needs on top of the upstream default
CSV — gpu-performance needs profiling and throttling counters,
gpu-efficiency needs profiling, gpu-tenants needs only
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for its tensor panel, and gpu-fleet
and gpu-quotas work on the default counter set alone.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Align namespace:gpu_count:allocated with every other namespace:* rule
by filtering out cozy-*/kube-*. All other per-namespace rules
(gpu_util, tensor_active, fb_used_bytes, power_watts) already exclude
system namespaces, so the label set produced by :allocated diverged
from them — any dashboard variable or join that reads across these
rules could end up with a different namespace list depending on which
rule supplied the :allocated column.
Trade-off: per-namespace GPU accounting for system workloads is no
longer available through this rule. If it's ever needed, add a
dedicated system:gpu_count:allocated rather than widening this one —
the "billable tenant view" invariant is what the filter is protecting.
Cluster-level cluster:gpu_count:allocated intentionally keeps system
pods so it stays aligned with cluster:gpu_count:total and
cluster:gpu_count:free remains meaningful. As a consequence,
sum(namespace:gpu_count:allocated) no longer equals
cluster:gpu_count:allocated; the delta is system-pod GPU usage, which
is fine for the cluster-admin view.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Catch dangling references at PR time: every recording-rule name used
inside a tracked GPU dashboard must exist in
packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml. The
first iteration of gpu-efficiency.json shipped panels keyed on
pod:tensor_saturation:avg5m without the rule defined; the test fails
on exactly that class of bug.
Scoped to dashboards listed under gpu/* in dashboards-infra.list, so
untracked drafts stay out of scope until they are registered. Reverse
direction (rule defined but unused) is intentionally NOT enforced —
some rules exist for ad-hoc PromQL or upcoming dashboards.
Auto-discovered by make bats-unit-tests via hack/cozytest.sh.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The previous wording implied that the entire custom DCGM CSV was
required by the recording rules. In fact only the profiling counters
(DCGM_FI_PROF_*) need to be added on top of the upstream defaults —
everything else the rules consume is already in default-counters.csv.
Add a Verification status block flagging that the minimum-set claim is
derived from the DCGM Exporter version pinned in the currently shipped
gpu-operator package and must be re-checked when that package moves to
a newer release.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Restrict the nvidia-driver-compat DaemonSet to nodes labelled
nvidia.com/gpu.present=true (NFD/GPU Operator label). Without the
nodeSelector it was scheduling onto every node — control-plane and
CPU-only workers included — burning a privileged pod slot per host
for no benefit.
Add resource requests and limits to the init and pause containers so
the DaemonSet stays within control-plane budget on small clusters.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Drop gpu.recording.30s group: per-GPU 30s aggregates had no consumers
in tracked dashboards, only burned cardinality.
- Drop namespace:gpu_allocated_count:gauge: identical expression to
namespace:gpu_count:sum under a different name.
- Reground :allocated on kube_pod_container_resource_requests so it
reflects what tenants requested (Pending+Running) rather than what
DCGM currently sees. namespace:gpu_count:sum stays DCGM-based and
represents actually-running pods; the gap between the two is the
signal admins want.
- Add namespace:gpu_count:allocated as the per-namespace counterpart.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Before this change oauth2-proxy fronting linstor-gui only enforced that
the user could authenticate against the `cozy` Keycloak realm
(`--email-domain=*`, no group restriction). Any realm user could reach
the UI and, through it, the LINSTOR controller REST API — which the
gatekeeper proxies with a static mTLS client cert and no per-user RBAC.
Add `--allowed-group=cozystack-cluster-admin` and include `groups` in
the OIDC scope so the claim is present at validation time. The
`cozystack-cluster-admin` KeycloakRealmGroup and the `groups` client
scope are already provisioned by keycloak-configure, so no cluster-wide
changes are needed.
Assisted-By: Claude <noreply@anthropic.com>
(cherry picked from commit 9b54e46723)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Backport of #2415 to release-1.3.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated README to document that LINSTOR GUI access is restricted to
members of the `cozystack-cluster-admin` Keycloak group. Non-members
receive a 403 error.
* **Chores**
* Implemented group membership validation for LINSTOR GUI access
control.
* Added validation tests for group membership enforcement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Before this change oauth2-proxy fronting linstor-gui only enforced that
the user could authenticate against the `cozy` Keycloak realm
(`--email-domain=*`, no group restriction). Any realm user could reach
the UI and, through it, the LINSTOR controller REST API — which the
gatekeeper proxies with a static mTLS client cert and no per-user RBAC.
Add `--allowed-group=cozystack-cluster-admin` and include `groups` in
the OIDC scope so the claim is present at validation time. The
`cozystack-cluster-admin` KeycloakRealmGroup and the `groups` client
scope are already provisioned by keycloak-configure, so no cluster-wide
changes are needed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 9b54e46723)
Strip Grafana export boilerplate (__inputs, __elements, __requires,
default annotations, embedded datasource inputs) and tighten panel
layouts across the three GPU dashboards. All three continue to use
the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Revise gpu-performance and add two new dashboards, registered in
dashboards-infra.list:
- gpu-efficiency (GPU Efficiency Score) — utilization vs. capacity
and workload efficiency signals.
- gpu-quotas (GPU Quotas & Allocation) — per-namespace requested vs.
used GPUs for tenant capacity planning.
All three dashboards use the $ds_prometheus template variable, per
the project convention.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add reference manifests (not templates) under
packages/system/gpu-operator/examples/ documenting one working
configuration for running CUDA workloads directly in pods on a Talos
cluster, with DCGM metrics that drive the gpu/gpu-performance
dashboard.
- values-native-talos.yaml: Cozystack Package values that disable the
sandbox path, enable the device plugin, and wire DCGM to the custom
metrics ConfigMap.
- dcgm-custom-metrics.yaml: ConfigMap extending the default DCGM CSV
with profiling, ECC, throttling and energy counters used by the
dashboard and recording rules.
- nvidia-driver-compat.yaml: DaemonSet that stages libnvidia-ml.so.1
and nvidia-smi from the Talos glibc tree into a location the
gpu-operator validator inspects. Workaround for
NVIDIA/gpu-operator#1687.
- README.md: explains why these are shipped as references rather than
first-class templates (sandbox vs native is a deployment choice),
and how the pieces connect.
The out-of-the-box values-talos.yaml still targets the sandbox (VFIO
passthrough) scenario. Operators who want native pod GPU workloads can
start from these references.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add the gpu/gpu-performance dashboard and register it in the infra
dashboard list. The dashboard provides:
- Cluster overview: total/allocated GPUs, average utilization,
aggregate power draw.
- Utilization: GPU util (NVML), tensor pipe active (realistic load
for LLM/AI workloads), graphics engine active, memory copy util.
- Memory: VRAM used/free per GPU.
- Power and temperature per GPU.
- Health: XID errors, power and thermal throttling.
The dashboard relies on DCGM_FI_* metrics plus the cluster:gpu_* and
namespace:gpu_* recording rules added to monitoring-agents.
The JSON follows the cozystack convention — Prometheus data source is
selected via the $ds_prometheus template variable.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add VMRule with recording rules for DCGM metrics at three levels:
- gpu.recording.30s: per-GPU aggregates over 30s windows
- gpu.recording.cluster.1m: cluster-wide totals for overview panels
- gpu.recording.namespace.1m: per-namespace aggregates for tenant
reporting and GPU-hour calculations
The rules are safe to ship on clusters without DCGM — they evaluate to
empty series when no matching metrics are scraped.
Used by dashboards/gpu/gpu-performance.json.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
## What this PR does
Adds BucketClaim support to WorkloadMonitorReconciler, bringing S3
buckets in line with how Pods, PVCs, and Services are already tracked as
Workload resources.
- Add WorkloadMonitor CR to the bucket Helm chart with
`app.kubernetes.io/instance` label on BucketClaim for selector matching
- Add `reconcileBucketClaimForMonitor()` following the same pattern as
`reconcilePVCForMonitor()` — watches COSI `BucketClaim` objects and
creates corresponding `Workload` CRDs
- Query SeaweedFS bucket size metrics (logical + physical) from
VictoriaMetrics, with the monitoring endpoint resolved automatically
from the `namespace.cozystack.io/monitoring` namespace label
- COSI API types dependency (`container-object-storage-interface-api`)
for typed BucketClaim access
No configuration flags needed — the controller discovers the monitoring
stack for each tenant namespace automatically.
### Result
When a BucketClaim is matched by a WorkloadMonitor, the controller
creates a Workload with S3 storage metrics:
```yaml
apiVersion: cozystack.io/v1alpha1
kind: Workload
metadata:
name: bucket-bucket-test-billing
namespace: tenant-testing
labels:
workloads.cozystack.io/monitor: bucket-test-billing
ownerReferences:
- apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClaim
name: bucket-test-billing
status:
kind: bucket
type: s3
operational: true
resources:
s3-buckets: "1"
s3-storage-bytes: "10485864" # 10 MB logical
s3-physical-storage-bytes: "20971728" # 20 MB physical (replication factor 2)
```
- `s3-storage-bytes` — logical size (what the user stored), from
`SeaweedFS_s3_bucket_size_bytes`
- `s3-physical-storage-bytes` — physical size (with replicas), from
`SeaweedFS_s3_bucket_physical_size_bytes`
- When monitoring is not configured for the namespace, only `s3-buckets:
1` is tracked
- Sizes refresh every 60 seconds via `RequeueAfter`
Tested on a live dev cluster with SeaweedFS deployed in `tenant-root`
and a bucket created in `tenant-testing`.
### Release note
```release-note
[apps] Add WorkloadMonitor to bucket application. BucketClaims are now tracked as Workload resources with S3 storage size metrics resolved automatically from the tenant monitoring stack.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Monitor COSI bucket claims: creates/updates Workload records
reflecting bucket readiness and requeues periodically when buckets
exist.
* Prometheus-backed bucket storage metrics, with SeaweedFS-aware sizing
when available.
* Helm chart additions: WorkloadMonitor resource and instance label on
BucketClaim templates.
* **Tests**
* Unit tests covering bucket-monitor flows, metric querying, URL
resolution, and requeue behavior.
* **Chores**
* Updated module dependencies to enable COSI integration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace http.DefaultClient with a package-level *http.Client with an
explicit 10-second timeout. Avoids sharing the process-wide default
transport with other libraries.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Pass bucket names from BucketClaim.Status.BucketName into the PromQL
query as a bucket=~"name1|name2" filter. This prevents O(N²) load
where N WorkloadMonitors each fetch all buckets globally.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Wrap resolvePrometheusURL and queryAllBucketMetrics in a
len(bucketClaimList.Items) > 0 guard. Avoids unnecessary namespace
GET and HTTP request to Prometheus on every reconcile of non-bucket
WorkloadMonitors (postgres, redis, kubernetes, etc.).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Replace 2×N per-bucket HTTP requests with a single query that fetches
all SeaweedFS bucket size metrics at once:
{__name__=~"SeaweedFS_s3_bucket_(size|physical_size)_bytes"}
Results are keyed by bucket name in memory, then looked up per
BucketClaim. This reduces HTTP round-trips from 2N+1 to 2 (one
namespace lookup + one Prometheus query) regardless of bucket count.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Move resolvePrometheusURL call before the BucketClaim loop and pass
the URL as parameter. Avoids redundant namespace lookups and reduces
reconcile time from 2×N+1 to 2×N HTTP calls (where N = BucketClaims).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Change queryPrometheusMetric to return (int64, bool) so callers can
emit s3-storage-bytes=0 for empty buckets while omitting the field
entirely when monitoring is not configured.
Also:
- Add RBAC marker for core/namespaces GET (used by resolvePrometheusURL)
- Log namespace read errors instead of silently returning empty URL
- Add TestQueryPrometheusMetricZeroValue test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Bucket existence is already tracked via LifetimeHours. The s3-buckets
count resource produced a meaningless "s3-buckets-Hours" fallback type
in billing. Only storage size metrics (s3-storage-bytes,
s3-physical-storage-bytes) are now set on bucket Workloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Initialize workload.Labels map inside CreateOrUpdate mutate function
to prevent nil map panic when existing Workload has no labels
- Use strconv.ParseFloat instead of resource.ParseQuantity for
Prometheus metric values, which may use scientific notation
(e.g. "1.048576e+06") that ParseQuantity does not support
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Replace static --prometheus-url flag with dynamic resolution from
namespace.cozystack.io/monitoring label. Each tenant namespace knows
which tenant hosts its monitoring stack, so the controller constructs
the vmselect URL automatically. This correctly handles multi-tenant
setups where different tenants may use different monitoring instances.
- Remove PrometheusURL field from WorkloadMonitorReconciler struct
- Remove --prometheus-url flag and prometheusUrl chart value
- Add resolvePrometheusURL() that reads namespace label
- queryPrometheusMetric() now accepts prometheusBaseURL as parameter
- Add tests for resolvePrometheusURL with and without label
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Set replicas/minReplicas to 0 in bucket WorkloadMonitor (buckets
have no pods, minReplicas=1 would mark monitor as not operational)
- Remove dead IsNotFound check on List (List returns empty, not 404)
- Trim trailing slash from PrometheusURL before path append
- Add strings import for TrimRight
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Check resp.StatusCode before parsing Prometheus response
- Limit response body read to 1 MB via io.LimitReader
- Use strings.HasPrefix in test instead of fragile slice indexing
- Add TestQueryPrometheusMetricServerError for 500 responses
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Query SeaweedFS_s3_bucket_physical_size_bytes alongside the logical
size metric. Physical size includes all replicas and reflects actual
disk usage, while logical size reflects what the user stored.
Refactor queryBucketSizeBytes into generic queryPrometheusMetric
to reuse for both metrics.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Pass --prometheus-url to the controller container when configured in
values. This enables querying SeaweedFS bucket size metrics from a
Prometheus-compatible API for S3 bucket billing.
RBAC already covers BucketClaim access via existing wildcard rule
(apiGroups: ['*'], resources: ['*'], verbs: ["get", "list", "watch"]).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Query SeaweedFS_s3_bucket_size_bytes from a Prometheus-compatible API
to populate s3-storage-bytes resource on bucket Workloads. The
Prometheus URL is configurable via --prometheus-url flag. When set,
bucket WorkloadMonitors are requeued every 60s to keep sizes current.
When Prometheus is not configured, buckets still get tracked with
s3-buckets=1 for existence-based billing.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Add reconcileBucketClaimForMonitor() that watches COSI BucketClaim
objects and creates Workload CRDs with s3-buckets resource, following
the same pattern as PVC and Service reconcilers. This enables the
billing pipeline to discover and track S3 buckets per tenant.
Changes:
- Add COSI API types dependency (container-object-storage-interface-api)
- Register cosiv1alpha1 scheme in controller main
- Add BucketClaim watch in SetupWithManager
- Add BucketClaim list + reconcile in Reconcile loop
- Add RBAC annotation for objectstorage.k8s.io/bucketclaims
- Add unit tests for BucketClaim reconciliation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Add WorkloadMonitor CR to bucket Helm chart so that the billing
pipeline can discover and track S3 buckets. Add instance label
to BucketClaim metadata for WorkloadMonitor selector matching.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
## What this PR does
Tightens access control on the `linstor-gui` Ingress so it is reachable
only by members of the `cozystack-cluster-admin` Keycloak group (the
same group that grants cluster-admin RBAC on the host cluster).
Before this change, the oauth2-proxy gatekeeper in front of
`linstor-gui` enforced only that the user could authenticate against the
`cozy` realm (`--email-domain=*`, no group restriction). Any realm user
— including tenant-scoped accounts — could reach the UI and the
underlying LINSTOR controller REST API, which the gatekeeper proxies
with a static mTLS client cert and no per-user RBAC.
Changes:
- `packages/system/linstor-gui/templates/gatekeeper.yaml` — add
`--allowed-group=cozystack-cluster-admin` and include `groups` in
`--scope` so the claim is available for validation. Extend the
top-of-file rationale block to document the gate.
- `packages/system/linstor-gui/tests/ingress_auth_test.yaml` — assert
both new args on the rendered Deployment.
- `packages/system/linstor-gui/README.md` — document the group
restriction under "Option 1 — Keycloak-protected Ingress".
No Keycloak-side changes are required: the `cozystack-cluster-admin`
KeycloakRealmGroup and the `groups` client scope are already provisioned
by `keycloak-configure`.
Port-forward access (Option 2) is unchanged; it remains gated by kubectl
RBAC on `cozy-linstor`.
### Release note
```release-note
fix(linstor): restrict `linstor-gui` Ingress access to members of the `cozystack-cluster-admin` Keycloak group.
```
Before this change oauth2-proxy fronting linstor-gui only enforced that
the user could authenticate against the `cozy` Keycloak realm
(`--email-domain=*`, no group restriction). Any realm user could reach
the UI and, through it, the LINSTOR controller REST API — which the
gatekeeper proxies with a static mTLS client cert and no per-user RBAC.
Add `--allowed-group=cozystack-cluster-admin` and include `groups` in
the OIDC scope so the claim is present at validation time. The
`cozystack-cluster-admin` KeycloakRealmGroup and the `groups` client
scope are already provisioned by keycloak-configure, so no cluster-wide
changes are needed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The test body asserts .status.history[].status extraction, but the
test name still referenced the old installFailures counter (leftover
from when the guard used that field before switching to status.history
to avoid ClearFailures zeroing the counters on successful reconcile).
Address review feedback from coderabbitai on
hack/remediation-guard.bats:84: rename so grep for what the test
actually pins matches the test name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Log .status.history regardless of content so a silently empty result
(Flux field rename) shows up in CI logs, and treat empty history on
a Ready HelmRelease as a distinct failure. A Ready HR by definition
has at least one snapshot; anything else is a shape-drift signal.
- Replace the unquoted heredoc in remediation-guard.sh with a printf |
grep pipeline. printf %s treats statuses as literal payload (no $
expansion surprises for future callers), grep --quiet --extended-regexp
returns exit status the caller can forward directly.
- Share the etcd-absent values file between both invariant tests
(packages/apps/kubernetes/tests/values-ci-no-etcd.yaml) instead of
duplicating the --set block.
- Fix typo "override applied" -> "override is applied" in the
Kubernetes ApplicationDefinition.
- Add a coupling comment in the ApplicationDefinition annotation that
points at the wait-for-kubeconfig init deadline in _helpers.tpl, so
a future operator raising the HR timeout updates the init deadline
too.
- Clarify the per-annotation timeout comment in rest.go so it stops
implying the feature is Kubernetes-only (it is not - only today's
one user is).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pulls the release.cozystack.io/helm-install-timeout parsing out of
start.go into ParseHelmInstallTimeoutAnnotation in pkg/config. The
helper rejects units that time.ParseDuration accepts but Flux
helm-controller rejects (ns, us, µs): feeding one of those through
would cause the HelmRelease admission webhook to reject the object at
install time, giving a silent drop to flux defaults that is hard to
debug. Fail loudly at cozystack-api startup instead.
Adds a table-driven unit test covering: unset (empty), accepted units
ms/s/m/h, compound 2h30m, decimal 1.5m, and the rejected shapes
(bare digits, garbage, negative, ns/us/µs). The test lives in
pkg/config so it runs under the existing go-unit-tests make target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
17 child HelmReleases (cilium, coredns, csi, cert-manager,
metrics-server, ...) referenced *-admin-kubeconfig via
kubeConfig.secretRef and rendered even when _namespace.etcd was empty.
On an etcd-less tenant each one sat in NotReady forever because the
admin-kubeconfig Secret only exists after a KamajiControlPlane
reconciles, and KamajiControlPlane now only renders when etcd is set.
The outcome contradicted the "beacon only" contract claimed in the
soft-skip commit.
Extend the existing addon guards to also require _namespace.etcd, and
wrap the four unconditional HelmReleases (csi, metrics-server,
prometheus-operator-crds, volumesnapshot-crd) plus the always-on
cilium/coredns HR resources in the same gate. Add an invariant bats
test that renders the whole chart with etcd empty and asserts zero
HelmReleases reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The image-busybox Makefile target + images/busybox/Dockerfile wrapper
just rebuilt an upstream busybox digest as ghcr.io/cozystack/cozystack/
busybox. Payload is a one-shot sh loop run once per pod; the pinned
upstream digest is already immutable, so maintaining a private mirror
adds churn (rebuild on every release) for no real hardening benefit.
Delete the wrapper and reference docker.io/library/busybox:<digest>
directly from images/busybox.tag. Document the choice in _helpers.tpl.
Also drop the false coupling in the go table test: the "unrelated
kind without configured timeout" case used the real kind name Qdrant,
which tied the test to the Qdrant ApplicationDefinition for no reason.
Switch to a clearly fictional kind so a future Qdrant timeout override
does not break this assertion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
CI runs make unit-tests on every PR, which already covers helm
unittests and bats (hack/admin-kubeconfig-invariant.bats and
hack/remediation-guard.bats are both picked up by the existing
hack/*.bats glob). What was missing was any go test invocation.
Add a go-unit-tests target scoped to pkg/registry, pkg/config, and
pkg/cmd/server - the cozystack-api surface this repo actually owns
and tests in-tree. Running go test ./... pulls in generated-code
round-trip suites whose behavior is governed by generator tool
versions outside this repo's control; those are better exercised
from their own generator workflows.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Flux helm-controller's ClearFailures() zeroes installFailures and
upgradeFailures on every successful reconciliation (see the upstream
HelmReleaseStatus method). The previous guard ran after the HelmRelease
was Ready, at which point the counters were always 0 - the assertion
was vacuous and would have passed against a reverted fix.
Switch to .status.history, which retains per-revision release
Snapshots that survive a subsequent successful reconciliation. A
remediation cycle leaves behind a Snapshot with status=uninstalled
(the install-remediation code path) or status=failed (Helm release
failure that remediation then uninstalled). Either one signals the
race actually fired.
Rewrite the bats unit tests to cover: empty history, deployed-only,
deployed+superseded (happy path - not detected), single failed,
single uninstalled, uninstalled-then-deployed, and deployed-then-failed
(all detected). The pinned-shape test feeds a realistic HR status
snippet through yq the same way run-kubernetes.sh does via kubectl
-o jsonpath.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The soft-skip wrap in cluster.yaml only silenced Cluster and
KamajiControlPlane rendering when _namespace.etcd is empty. The three
CP-side Deployments (cluster-autoscaler, kccm, kcsi-controller) still
rendered, their wait-for-kubeconfig init containers CrashLoopBackOff'd
forever (no KamajiControlPlane = no admin-kubeconfig Secret),
HelmRelease hit its 15m wait timeout and triggered the very install
remediation cycle the rest of this PR prevents. Self-contradiction.
Wrap each of the three Deployment templates in
{{- if .Values._namespace.etcd }}...{{- end }} so they render only
when there is a DataStore to back them. Add an invariant bats test
that renders the whole chart with etcd empty and asserts zero
Deployments reference *-admin-kubeconfig.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
run-kubernetes.sh extracts .status.installFailures and
.status.upgradeFailures via kubectl -o jsonpath. If a future flux
release renames the counters, kubectl returns empty, the guard reports
no cycle, and e2e silently misses real remediation loops.
Add a bats unit test that feeds a pinned HelmRelease v2 status snippet
through the same jsonpath and asserts the extraction still yields the
expected values. Also leave a pointer comment in run-kubernetes.sh so a
future flux bump surfaces the version coupling in review.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-template unittests in packages/apps/kubernetes/tests/ assert
that cluster-autoscaler, kccm, and the csi controller each mount the
admin-kubeconfig Secret optional and carry the wait-for-kubeconfig
init. That locks in today's three Deployments by name - a fourth
Deployment that mounts the same Secret but forgets the guard would
slip past them.
Add a bats-unit test that renders the entire chart, enumerates every
Deployment whose spec mounts a Secret ending in -admin-kubeconfig, and
asserts optional:true plus wait-for-kubeconfig init on all of them.
Verified by temporarily removing optional:true from csi/deploy.yaml:
the test correctly flagged invariant-kcsi-controller as an offender.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the hardcoded r.kindName == "Kubernetes" switch in rest.go
with a config-driven path. The ApplicationDefinition CR now accepts a
release.cozystack.io/helm-install-timeout annotation that is parsed
at cozystack-api startup into config.ReleaseConfig.HelmInstallTimeout
and applied to both Install.Timeout and Upgrade.Timeout on the
rendered HelmRelease. Applications that leave the annotation unset
keep flux defaults so their failed installs remediate on the normal
cadence - only the Kubernetes kind carries the override and gets a
15m budget. New kinds with a similar race can opt in by setting the
same annotation; no rest.go patch needed.
Kubernetes-rd sets the annotation to 15m. Table-driven test in
rest_timeout_test.go covers three cases: Kubernetes with 15m, Qdrant
unset, and an arbitrary future kind with 20m - all of which pin the
Remediation.Retries == -1 contract as well.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The three control-plane-side Deployments (cluster-autoscaler, kccm,
kcsi-controller) carried three copies of the same 20-line init
container. That already drifted: the CSI copy used 4-space nesting
while the other two used 2-space. Any future update to the image,
the deadline, or the poll script had to land in three places or
silently diverge.
Extract the block into a new kubernetes.waitForAdminKubeconfig helper
in templates/_helpers.tpl and include it at each call site. Tighten
the deadline from 20m to 10m so it stays strictly below the 15m
HelmRelease Install.Timeout and the CrashLoopBackOff surfaces in
dashboards before flux remediation can fire. Also clarify the wait
message so operators debugging a stuck init container do not chase
Kamaji for what is actually kubelet's optional-Secret refresh cadence.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A hard helm fail in cluster.yaml made every cold bootstrap racy: if
the parent Tenant chart had not yet populated _namespace.etcd in
cozystack-values when the Kubernetes HelmRelease first reconciled, the
fail fired, install.remediation triggered, installFailures incremented
and the new e2e remediation-guard flagged it as a bug. That directly
contradicts the race the rest of this PR is trying to close.
Replace fail with a graceful skip: render only a status-beacon
ConfigMap (test-awaiting-etcd) when etcd is empty, wrap all
CAPI/Kamaji resources in {{ if $etcd }}. The HelmRelease installs
successfully and goes Ready; flux retries on its 5m interval and picks
up the DataStore as soon as the Tenant chart finishes reconciling.
Update the helm unittest: positive test still asserts dataStoreName on
KamajiControlPlane; the negative test now asserts exactly one
ConfigMap document with status=awaiting-etcd, no Cluster / KCP /
KubevirtCluster / WorkloadMonitor rendered.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cap the wait-for-kubeconfig init container at 20m. If Kamaji genuinely
fails to produce the admin-kubeconfig Secret (misconfigured tenant,
etcd outage after the guard already passed, Kamaji crash-loop), the
pod now exits non-zero and goes CrashLoopBackOff so the failure is
visible in dashboards, instead of silently sleeping in Init forever
and leaving only the flux helm-wait timeout to surface the problem.
Reword the etcd DataStore guard to reference the parent Tenant
application's etcd flag (not .Values.etcd of the Kubernetes chart,
which is a different chart). Update the helm unittest errorPattern
accordingly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds an assertion that cluster.yaml renders successfully when
_namespace.etcd is set and produces a KamajiControlPlane whose
dataStoreName equals the tenant's etcd DataStore name. Without this
positive case a future edit that inverts or removes the existing etcd
guard would pass the suite as long as the negative case still fails.
Also adds documentSelector: kind=Deployment to the kccm and csi
controller assertions so the jsonpath filter operates on a single
document, matching the cluster-autoscaler case and removing
reliance on helm-unittest filter-vs-single-value coercion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds images/busybox/Dockerfile and an image-busybox Makefile target
that mirror the same pattern as the rest of this chart's images (the
Dockerfile pins the upstream busybox by digest; the Makefile target
builds and tags for ghcr.io/cozystack/cozystack/busybox the same way
cluster-autoscaler et al. are handled). Also wires it into the
umbrella image target so 'make image' rebuilds everything.
Until the first release build runs image-busybox and rewrites the
.tag to point at ghcr.io, the .tag keeps a fully-qualified
docker.io/library/busybox:1.37.0@sha256:... reference so pods do not
silently resolve the short name via the default registry and pulls
remain immutable by digest. The release workflow overwrites this
file the same way it does for the other images.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous change set Install.Timeout and Upgrade.Timeout to 15m
on every Application's parent HelmRelease, but the admin-kubeconfig
race documented in #2412 is specific to the Kubernetes kind: only its
parent chart creates CAPI/Kamaji resources whose admin-kubeconfig
Secret is asynchronously provisioned and mounted by Deployments in
the same chart. Other kinds (Qdrant, MongoDB, Postgres, ...) have no
such race and should not have their failed installs linger three times
longer before flux triggers remediation.
Gate the timeout on r.kindName == "Kubernetes". Rewrite
rest_timeout_test.go to cover both sides: Kubernetes must get a
>= 15m timeout, other kinds must keep the flux defaults. Both tests
also pin Install/Upgrade Remediation.Retries == -1 so a future edit
that removes unbounded remediation would show up as a red test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Shell && and || have equal precedence and left-to-right associativity,
so the previous guard parsed as (((A && B) || C) && D) and silently
passed on the canonical failure mode: install_failures=1 with an empty
upgrade_failures.
Extract the check into helmrelease_has_remediation_cycle() in a
dedicated helper sourced from run-kubernetes.sh, and add unit tests
under hack/remediation-guard.bats that pin the expected behavior for
every combination of empty, zero, and positive counter values.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Before cleanup, inspect the parent HelmRelease installFailures and
upgradeFailures counters. A non-zero value means flux helm-controller
hit its wait timeout, ran install/upgrade remediation (uninstall),
and re-installed - the exact race condition this PR closes. Fail the
bats test in that case so the signal surfaces in CI instead of being
masked by a green retry.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When a Kubernetes tenant is created without a parent tenant that has
etcd enabled, .Values._namespace.etcd is empty and the rendered
KamajiControlPlane spec carries an empty dataStoreName. The Kamaji
admission webhook then rejects every TenantControlPlane create with
"tenant-root DataStore does not exist" and the control plane never
comes up.
Add a helm template-level guard that fails rendering with a
descriptive, actionable error message before the HelmRelease even
reaches the webhook. This also closes the narrow race where a
Kubernetes HelmRelease reconciles before the etcd HelmRelease has
created the DataStore CR - flux retries on its interval and picks up
the DataStore once it appears.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three Deployments in the Kubernetes app chart mount the tenant
admin-kubeconfig Secret directly as a volume: cluster-autoscaler,
kccm, and the kcsi controller. That Secret is provisioned
asynchronously by Kamaji after control-plane bootstrap, so on a fresh
install the pods used to hit FailedMount and the parent HelmRelease
ran out of its wait budget.
Mark the Secret volume optional and add a wait-for-kubeconfig
initContainer that polls the mounted path until the Secret appears.
Kubelet remounts the optional Secret within its sync period once
Kamaji publishes it, the init container exits, and the main container
starts cleanly. The Deployment becomes Available without the helm-wait
ever seeing a FailedMount.
Pins a busybox image for the init container via
images/busybox.tag (same format as the other pinned tags in this
chart).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds failing helm unittest suite for packages/apps/kubernetes covering:
- cluster-autoscaler, kccm, and csi controller Deployments mount the
admin-kubeconfig Secret with optional: true
- each of those Deployments has a wait-for-kubeconfig initContainer
that mounts the same kubeconfig path
- cluster.yaml renders a helm fail with a descriptive message when the
tenant has no etcd DataStore (empty _namespace.etcd)
Also wires up a test target in the chart Makefile so helm-unit-tests.sh
picks it up.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Parent HelmRelease created by cozystack-api for Kubernetes tenants
contains CAPI/Kamaji resources (Cluster, KamajiControlPlane,
MachineDeployment) that asynchronously provision the
*-admin-kubeconfig Secret. Three Deployments in the same chart
(cluster-autoscaler, kccm, kcsi-controller) mount that Secret
directly, so the helm-wait cannot complete until control-plane
bootstrap finishes.
Default flux helm-controller timeout is too short for a cold-node
first-tenant bootstrap (image pull + etcd bootstrap + apiserver Ready
+ admin-kubeconfig generation routinely exceed it). On timeout,
install.remediation triggers uninstall, which removes the Cluster CR
and restarts the cycle indefinitely.
Bumping Install.Timeout and Upgrade.Timeout to 15m gives realistic
bootstrap headroom while remaining bounded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Adds a failing unit test for convertApplicationToHelmRelease asserting
that Install.Timeout and Upgrade.Timeout are at least 15 minutes. The
default flux helm-controller timeout is too short to cover cold-start
Kamaji control-plane bootstrap (image pull + etcd bootstrap + apiserver
Ready + admin-kubeconfig Secret generation) and causes install
remediation loops.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds the v1.3.0-rc.1 release changelog covering all changes since
v1.2.0, and updates the changelog agent instructions to include two new
side repositories.
**Changelog** (`docs/changelogs/v1.3.0-rc.1.md`):
- Feature Highlights: storage-aware scheduling, LINSTOR GUI, VM Default
Images, WorkloadsReady conditions, cross-namespace VM backup restore
- Covers main repo, website, talm, ansible-cozystack, and
external-apps-example changes
- Backported fixes marked with *(backported to v1.2.x)* annotation
**Agent instructions** (`docs/agents/changelog.md`):
- Adds `external-apps-example` and `ansible-cozystack` to the optional
repositories list
### Screenshots
N/A — documentation only.
### Release note
```release-note
docs(changelog): add changelog for v1.3.0-rc.1
```
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated the README's introductory description to refine the platform
positioning and improve clarity on its core capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds two features to improve application observability in the dashboard,
plus a bug fix:
### 1. WorkloadsReady condition on Application status
- Queries WorkloadMonitor resources to determine if all application pods
are running
- Exposes `WorkloadsReady` as a separate condition alongside `Ready`
- `Ready` continues to reflect HelmRelease state only — no override — to
preserve backward compatibility with existing tooling (kubectl wait,
GitOps health checks) and avoid false-negative Ready=False during normal
startup windows
- Handles three states: operational, not operational, unknown (pending
reconciliation)
- Fails open on WorkloadMonitor query errors (prefers availability)
- Integrates with Application Watch to emit MODIFIED events on
WorkloadMonitor changes
- Registers cozystack.io/v1alpha1 types in API server scheme with
informer cache
### 2. Events tab in dashboard
- Shows Kubernetes Events scoped to the application's namespace
- Uses status.namespace for Tenant applications (consistent with
Resource Quotas tab)
- Includes both lastTimestamp and eventTime columns for Kubernetes
version compatibility
### 3. Bug fix: WorkloadMonitor Operational status persistence
- Operational field was written to stale `monitor` variable instead of
`fresh` inside RetryOnConflict, so it was never persisted to the cluster
Closes#2359Closes#2360
### Release note
```release-note
[dashboard] Added Events tab to application detail pages showing namespace-scoped Kubernetes Events
[application] Added WorkloadsReady condition exposing aggregated WorkloadMonitor status
[workloadmonitor] Fixed bug where Operational status was never persisted to the cluster
```
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[vm-default-images] Added package that brings set of images that can be used clusterwide
[vm-disk] Updated source "image" for prettier dropdown selection
[vm-disk] Added new source for vm-disk called disk - to use as source vm-disk from same namespace.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Clone VM disks by specifying an existing vm-disk as the source.
* Global default image collection and chart to publish pre-provisioned
images.
* **UI**
* Forms provide selectable lists for default images and existing VM
disks.
* **Migration**
* Migration to rename existing image DataVolumes to the new
default-images naming and bumped migration version.
* **Documentation**
* VM disk docs and README updated to reflect image sourcing and cloning.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
core/v1 Events populate .lastTimestamp and .firstTimestamp but leave
.eventTime null; events.k8s.io/v1 Events do the opposite. The previous
column bound to .eventTime alone and rendered 'Invalid Date' for every
Helm-generated event.
Extend createTimestampColumn with an optional second jsonPath that is
encoded as a nested reqsJsonPath fallback in the template, and use
.eventTime → .firstTimestamp for Event Time and .lastTimestamp →
.eventTime for Last Seen so both APIs render correctly.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Add a "Screenshots" section to the pull request template.
PRs that include UI changes are now required to attach screenshots
or screen recordings demonstrating the visual impact of the changes.
### Release note
```release-note
docs(ci): add mandatory screenshots requirement for UI-related pull requests
```
Add a Screenshots section to the pull request template that requires
contributors to attach screenshots or screen recordings when their
changes affect the UI.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
## Maintainer Nomination: Mattia Eleuteri
Per the process in
[CONTRIBUTOR_LADDER.md](./CONTRIBUTOR_LADDER.md#maintainer), this PR
nominates **Mattia Eleuteri** (@mattia-eleuteri, Hidora) as a
Maintainer.
### Summary
Mattia has demonstrated sustained, high-quality contributions across
CSI/storage, networking, monitoring, and security. Key areas include:
- CSI fixes for migration, RWX NFS mounts, and multi-node volumes
- CiliumNetworkPolicy and VPC peering for multi-tenant environments
- Monitoring improvements (vmagent, infrastructure dashboards)
- Security scanning and hardening
- New platform packages (external-dns)
Full discussion and vote: #2343
### Process checklist
- [x] @mattia-eleuteri please comment confirming you agree to all
[Maintainer responsibilities](./CONTRIBUTOR_LADDER.md#maintainer)
- [x] Majority of current Maintainers (5 of 8) approve this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added Mattia Eleuteri (Hidora) to the project maintainers list,
assigned to CSI, Storage, Networking & Security.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Maintainer Nomination: Matthieu Robin
Per the process in
[CONTRIBUTOR_LADDER.md](./CONTRIBUTOR_LADDER.md#maintainer), this PR
nominates **Matthieu Robin** (@matthieu-robin, Hidora) as a Maintainer.
### Summary
Matthieu has contributed consistently across managed applications,
platform quality, and community engagement. Key areas include:
- Full managed OpenSearch service (operator, packaging, validation)
- Workload monitoring with instance profile labels
- Operational hardening (etcd-defrag resource limits)
- Kubernetes benchmarking and platform validation
- CozySummit speaker and Program Committee member
Full discussion and vote: #2344
### Process checklist
- [x] @matthieu-robin please comment confirming you agree to all
[Maintainer responsibilities](./CONTRIBUTOR_LADDER.md#maintainer)
- [x] Majority of current Maintainers (5 of 8) approve this PR
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the project maintainers documentation to reflect the current
team composition and responsibility assignments. The update documents
team members responsible for overseeing managed applications, platform
quality initiatives, and benchmarking efforts to ensure comprehensive
coverage of critical project areas.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The package imports ~320Gi of golden-image PVCs (ubuntu-noble, fedora,
debian, centos, etc.) as soon as it's installed. On small test and dev
clusters that's enough to consume the entire replicated storage pool,
after which no tenant PVCs — including the ones E2E itself provisions —
can be bound. It's also unreasonable to force that cost on every iaas
user: many deployments don't need prebuilt images at all, and the ones
that do often want to curate their own subset.
Switch the bundle entry from 'package.default' to 'package.optional.default',
matching the treatment already applied to gpu-operator directly below it.
Users who want the golden images can opt in via:
bundles:
enabledPackages:
- cozystack.vm-default-images
The package, its Source, and migration 38 all stay in place — nothing
else changes for users who explicitly enable it. Users who previously
relied on the bundle auto-installing it will need to add the package
to enabledPackages on upgrade; this is called out in the release note.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Tenant applications live in a child namespace computed from the Tenant
name, not in the HelmRelease's namespace. Scope the WorkloadMonitor
watch cluster-wide when kindName is Tenant and reverse-map WM events
back to the owning HelmRelease via computeTenantNamespace. The
condition-enrichment path now looks up monitors in the computed child
namespace too, so WorkloadsReady is populated for Tenants.
Buffer WorkloadMonitor events that arrive before the initial-events-end
bookmark and replay them once the bookmark is emitted, to preserve the
watch-list contract while not dropping workload-state transitions that
happen during the snapshot window.
Pass the fresh WorkloadMonitor object from the watch event into the
conversion path so WorkloadsReady reflects the state that triggered
the event, even when the cache client is lagging behind the watch
client.
Derive WorkloadsReady.LastTransitionTime from a stable source
(max of HelmRelease creation and condition timestamps, plus monitor
timestamps) instead of metav1.Now(), so repeated conversions of the
same underlying state produce identical timestamps.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Inside the RetryOnConflict block, derive the operational status from
fresh.Spec.MinReplicas instead of the stale monitor.Spec.MinReplicas
so that concurrent spec updates observed by the retry are respected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add two features to improve application observability in the dashboard:
1. WorkloadsReady condition on Application status
- Query WorkloadMonitor resources to determine if all pods are running
- Expose WorkloadsReady as a separate condition alongside Ready
- Ready continues to reflect HelmRelease state only (no override) to
preserve backward compatibility with tooling and avoid false-negatives
during normal startup windows when pods are still coming up
- Handle three states: operational, not operational, unknown (pending)
- Fail-open on WorkloadMonitor query errors
- Integrate with Application Watch to emit MODIFIED events on
WorkloadMonitor changes (with initial-events-end safety)
- Register cozystack.io/v1alpha1 types in API server scheme
2. Events tab in dashboard
- Show Kubernetes Events scoped to application namespace
- Use status.namespace for Tenant applications
- Include both lastTimestamp and eventTime columns for K8s compat
3. Bug fix: WorkloadMonitor Operational status persistence
- Operational was written to stale 'monitor' instead of 'fresh'
inside RetryOnConflict, so it was never persisted to the cluster
Closes#2359Closes#2360
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Two independent cleanups in agent-facing documentation:
1. **Refocus `docs/agents/contributing.md` on project-side
conventions.** The file previously mixed project rules (commit format,
PR structure) with personal workflow preferences (e.g. "do not commit
automatically, show the diff first", subjective guidance on how to
evaluate AI-bot review comments). Personal preferences belong in
individual contributors' own agent config (`~/CLAUDE.md` or equivalent),
not in a shared project doc that every agent reads. This PR keeps only
the project-side artifacts: what a commit/PR must look like when it
reaches the repository.
2. **Adopt Conventional Commits.** Recent commit history mixes
`[component]` prefix and `type(scope):` styles. This PR picks
Conventional Commits and aligns all agent-facing docs and the PR
template on it, so contributors and bots get one consistent answer.
Also documents the `Assisted-By:` trailer convention for AI-authored
commits.
Changes:
- `.github/PULL_REQUEST_TEMPLATE.md`: update guidance and release-note
example
- `docs/agents/contributing.md`: drop personal workflow guidance, link
to the PR template instead of duplicating it, switch examples to
Conventional Commits, document `Assisted-By:` trailer
- `docs/agents/overview.md`, `AGENTS.md`, `.gemini/styleguide.md`:
update commit format references
### Release note
```release-note
docs: adopt Conventional Commits (`type(scope): description`) for commit and PR titles
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Switched contribution & PR guidance to Conventional Commits
(`type(scope): description`) and added explicit allowed types, scope
examples, and breaking-change notation (`!` or `BREAKING CHANGE:`).
* Updated PR template, release-note expectations, validation messaging,
and contributor checklist to match the new convention.
* Added AI Agent Attribution trailers for assisted commits and
streamlined branch/PR workflow guidance (rebase/cleanup and PR creation
examples).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Stacked on top of #2382. Publishes the linstor-gui UI on
`https://linstor-gui.<root-host>` behind the cluster Keycloak realm
using the same oauth2-proxy gatekeeper pattern that the dashboard uses,
so the LINSTOR REST API is never exposed unauthenticated through
Ingress.
The Ingress is rendered only when **both** gates are satisfied:
- `_cluster.oidc-enabled == "true"`
- `linstor-gui` is listed in `_cluster.expose-services` (i.e. in
`publishing.exposedServices` in the core cozystack values)
If OIDC is disabled cluster-wide we intentionally do **not** ship an
unauthenticated ``token-proxy`` fallback — unlike dashboard, there is no
reason to front a raw storage-management API with k8s bearer tokens.
Operators can still reach the UI via \`kubectl port-forward\` to the
ClusterIP service.
## What's new
- \`templates/gatekeeper.yaml\` — oauth2-proxy Deployment (OIDC mode),
upstream \`linstor-gui.<ns>.svc:80\`
- \`templates/gatekeeper-svc.yaml\` — ClusterIP :8000 in front of it
- \`templates/gatekeeper-sa.yaml\` — dedicated SA, no auto-token
- \`templates/keycloakclient.yaml\` — persistent \`linstor-gui-client\`
+ \`linstor-gui-auth-config\` Secrets, \`KeycloakClient\` CRD that
auto-provisions the OIDC client with \`redirectUris:
[/oauth2/callback/*]\`
- \`templates/ingress.yaml\` — Ingress to the gatekeeper Service with
cert-manager ClusterIssuer, gated on expose-services + oidc-enabled
- \`tests/ingress_auth_test.yaml\` — unit tests for each conditional
branch, KeycloakClient rendering, oauth2-proxy args
README updated with Option 1 (Keycloak-protected Ingress) and Option 2
(port-forward).
### Release note
\`\`\`release-note
Added an opt-in Keycloak-protected Ingress for the linstor-gui package.
Add \`linstor-gui\` to \`publishing.exposedServices\` to publish
\`https://linstor-gui.<root-host>\` behind the cluster OIDC realm.
\`\`\`
## Test plan
- [ ] Deploy to dev10 with \`linstor-gui\` added to
\`publishing.exposedServices\`
- [ ] Verify cert-manager issues the TLS certificate
- [ ] Open \`https://linstor-gui.<root-host>\` in a browser, confirm
Keycloak login challenge
- [ ] After login, confirm LINSTOR node list loads via the nginx → mTLS
proxy
- [ ] Verify helm-unittest cases pass in CI
- [ ] Sanity-check that when \`oidc-enabled=false\` or the service is
not in \`expose-services\`, the Ingress + gatekeeper resources are
skipped
## What this PR does
Adds a new Cozystack system package `linstor-gui` that ships [LINBIT's
LINSTOR web UI](https://github.com/LINBIT/linstor-gui) (GPL-3.0) so
operators can manage nodes, resources, volumes and snapshots from a
browser instead of the `linstor` CLI.
Changes:
- `packages/system/linstor-gui/` — umbrella chart with:
- `images/linstor-gui/Dockerfile` builds the image from the upstream
`pkg.linbit.com` tarball (v2.3.0) on top of
`nginxinc/nginx-unprivileged:1.29-alpine`, mirroring the build pattern
already used for `piraeus-server` and `linstor-csi`.
- `templates/configmap-nginx.yaml` — chart-supplied `nginx.conf` that
proxies `/v1` and `/metrics` to
`linstor-controller.cozy-linstor.svc:3371` over **mTLS**, using the
existing `linstor-client-tls` secret created by the `linstor` package.
- `templates/deployment.yaml` — read-only rootfs, non-root (UID 101),
`runAsNonRoot`, `RuntimeDefault` seccomp, no SA token mounted,
`reloader.stakater.com/auto` on the Deployment.
- `templates/service.yaml` — **ClusterIP only**. No Ingress is shipped
because the LINSTOR controller API is a privileged cluster-wide storage
surface and auth depends on the deployment's OIDC setup; operators wire
up ingress + auth explicitly.
- `tests/deployment_test.yaml` — `helm-unittest` covering Service shape,
TLS secret mount, securityContext, and nginx proxy+mTLS config.
- `packages/core/platform/sources/linstor-gui.yaml` — new
`PackageSource` depending on `cozystack.linstor`.
- `packages/core/platform/templates/bundles/system.yaml` — registers
`linstor-gui` as an **optional** system package (controlled via
`bundles.enabledPackages`), alongside `velero`, `telepresence`, etc.
### Why standalone (not a sidecar in the controller pod)
LINBIT's upstream Dockerfile is designed for this standalone
reverse-proxy topology, and keeping the UI as its own Deployment means
it scales, upgrades, and restarts independently of `linstor-controller`.
The price is the small amount of mTLS config in `nginx.conf`, all of
which lives in the chart's ConfigMap.
### Release note
```release-note
[linstor-gui] Add an opt-in system package that deploys LINBIT's linstor-gui web UI alongside the LINSTOR controller. Enable via `bundles.enabledPackages: [cozystack.linstor-gui]`, then `kubectl -n cozy-linstor port-forward svc/linstor-gui 3373:80` to access.
```
## Test plan
- [ ] `make unit-tests` (requires `helm-unittest` plugin) passes on the
new `tests/deployment_test.yaml`
- [ ] `make image` in `packages/system/linstor-gui/` builds and pushes
the image
- [ ] Install the package on a dev cluster with LINSTOR already
deployed; confirm the pod comes up
- [ ] `kubectl -n cozy-linstor port-forward svc/linstor-gui 3373:80` →
browser loads the UI at `localhost:3373`
- [ ] UI lists LINSTOR nodes, storage pools and resources correctly
(proves mTLS proxy to `/v1` works)
- [ ] Disable via `bundles.enabledPackages` removal; verify the
`Package` resource is cleaned up
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* LINSTOR GUI added as an optional system package deployable to the
cozy-linstor namespace.
* Web UI serves on port 3373, proxies LINSTOR controller over mTLS, and
exposes a /healthz endpoint.
* Image build and release targets added to produce multi-arch container
images.
* **Documentation**
* New README with deployment guidance, connection examples, and
configurable options (endpoint, client secret, image, replicas).
* **Tests**
* Helm chart tests validating service, deployment, config and mTLS
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
* Add changelog for v1.2.2 (`docs/changelogs/v1.2.2.md`)
* Add changelog for v1.1.6 (`docs/changelogs/v1.1.6.md`)
Once merged, the `update-releasenotes.yaml` workflow will sync these to
the corresponding GitHub releases.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed Docker image tagging during builds; pinned system PostgreSQL
images.
* Corrected Cilium AppArmor handling and BPF load‑balancing exclusions
for external VM LoadBalancer services.
* Restored monitoring dashboard rendering for default platform variant.
* **New Features**
* Upgraded LINSTOR (piraeus-server) to v1.33.2 with backported
reliability patches.
* **Documentation**
* Expanded website docs: controller naming, Talos/version pairing,
troubleshooting, bundle naming, new --take-ownership flag, networking.*
fields, OpenAPI refresh, badges, and release notes.
* **Chores**
* Switched CI/CD from long‑lived PATs to short‑lived GitHub App tokens
and updated bot commit identity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The package is wired into the iaas bundle where LINSTOR replicated storage is
available. Set storageClass to "replicated" by default and add a note that the
full image set needs ~320Gi (16 images × 20Gi).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Running make generate at repo root regenerates the zz_generated.deepcopy.go
files. Key fix: Source.DeepCopyInto in the vmdisk package now handles
the new Disk *SourceDisk pointer field, which was previously missing and
would cause pointer aliasing on deep-copy.
Other generated changes are incidental updates to vminstance (Networks
field added, Subnet renamed to Network) and backups CRD manifests picked
up by controller-gen from the current HEAD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Accessing imgName["properties"].(map[string]any) and
diskName["properties"].(map[string]any) without an ok-check would panic
if the schema does not contain a "properties" key. Wrap both assertions
in ok-guarded form consistent with the outer defensive style.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The x86_64 architecture token was placed inside the filename segment
(CentOS-Stream-GenericCloud-x86_64-N-latest.x86_64.qcow2) which does
not match the actual CentOS mirror layout. Correct both CentOS Stream 9
and 10 URLs to the canonical form without the extra architecture infix.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The \$dv variable assigned by the lookup call on the image PVC was never
read. Remove the dead line to keep the template tidy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The PackageSource for vm-default-images existed but was never included
in any bundle, so it would never be installed. Add it to iaas.yaml
immediately after kubevirt-cdi (which it depends on) so it is deployed
together with the rest of the KubeVirt IaaS stack.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vm-disk package previously referenced golden-image DataVolumes with
the prefix "vm-image-<name>" in the cozy-public namespace. The new
vm-default-images package creates them as "vm-default-images-<name>",
so any existing vm-disk that sources an image by name would fail to
find its PVC after upgrade.
Migration 38 renames every DataVolume in cozy-public whose name starts
with "vm-image-" to "vm-default-images-" (dropping the old object and
re-creating with the new name) so live vm-disk resources keep resolving
their source PVC transparently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous commit blocked /v1/security/* on the LINSTOR controller,
but that turned out to be wishful thinking: that path doesn't exist in
the LINSTOR REST API at all (the controller responds 404 for everything
under /v1/security), and the linstor-gui SPA never calls it.
The actual storage for the GUI's authentication panel is two LINSTOR
KeyValueStore instances managed by the SPA itself:
- __gui__settings — holds `authenticationEnabled` (the on/off toggle)
- __gui__users — holds the encrypted admin credential
The SPA reads them on every page load via
`GET /v1/key-value-store/__gui__settings`, so we cannot blanket-block
the path or the UI fails to render. Instead, restrict to read-only:
allow GET/HEAD (so the SPA can confirm auth=off and skip its login
screen), reject every mutating method (PUT/POST/PATCH/DELETE) with the
same explanatory 403 JSON. Other GUI key-value entries (e.g.
__gui__mode) are unaffected.
The dev10 cluster had this footgun tripped already
(authenticationEnabled=true, an admin entry in __gui__users); cleared
manually with `linstor key-value-store modify __gui__settings
authenticationEnabled false` so the existing release stops gating the
UI behind a login the user no longer has the password for.
Verified end-to-end on dev10 after upgrade:
GET /v1/key-value-store/__gui__settings -> 200
PUT /v1/key-value-store/__gui__settings -> 403 (with JSON body)
POST /v1/key-value-store/__gui__users -> 403
PUT /v1/key-value-store/__gui__mode -> 200 (other KV writes ok)
GET /v1/nodes -> 200
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The upstream linstor-gui SPA exposes a "Users" / sign-in panel that
POSTs to /v1/security/* on the LINSTOR controller. If a cozystack user
turns on HTTP auth via that panel, every subsequent REST call —
including from this very GUI, which talks to the controller via mTLS,
not bearer tokens — starts returning 401, locking the user out of
LINSTOR with no in-product recovery path.
Now that authentication for the cozystack-shipped GUI is enforced one
layer up at the Ingress (oauth2-proxy + Keycloak), the in-app auth is
both redundant and a footgun. Short-circuit /v1/security/* in the
gateway nginx with a 403 + explanatory JSON body so the setting cannot
be enabled regardless of what the SPA renders. Other LINSTOR REST paths
(/v1/nodes, /v1/resource-definitions, /metrics, …) continue to proxy
through unchanged.
Verified on dev10:
POST /v1/security/sign-in -> 403 (with explanation body)
GET /v1/security/users -> 403
GET /v1/nodes -> 200
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Keycloak access+refresh+id tokens push the oauth2-proxy session cookie
to ~8-10 KB (split across multiple Set-Cookie headers), which overflows
ingress-nginx's default proxy_buffer_size and produces "upstream sent too
big header" -> HTTP 502 on /oauth2/callback right after a successful
Keycloak login.
Mirror the dashboard ingress: proxy-buffer-size: 100m and
proxy-buffers-number: "4". Verified on dev10 — unauthenticated requests
now return 302 to Keycloak instead of 502.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Narrowing to (instance=linstor-gui, name=gatekeeper) also blocked the
transient cm-acme-http-solver pods that cert-manager spins up in the
same namespace during http-01 challenges — the solver pods carry
different labels, so the Certificate never became Ready and TLS
handshakes failed.
Broaden the rule to the whole cozy-linstor namespace to mirror
allow-to-dashboard / allow-to-keycloak. The ingress controller only
initiates upstream HTTP calls driven by Ingress resources, so allowing
namespace-wide egress is not a practical escalation.
Verified on dev10 after patching the live CNP: Certificate
linstor-gui-ingress-tls becomes Ready (Let's Encrypt prod), and the
Ingress returns HTTP 302 to the Keycloak authorize endpoint with the
correct client_id and redirect_uri.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The tenant chart declares per-namespace egress CNPs so the tenant-root
ingress-nginx can reach auth-fronted UIs in the system namespaces
(dashboard, keycloak, cdi-upload-proxy). linstor-gui's gatekeeper in
cozy-linstor needs the same allowance — without it, requests to
https://linstor-gui.<root-host> time out at the Ingress with 504
because tenant-root → cozy-linstor egress is denied by default-deny.
Scoped to the gatekeeper pods (instance=linstor-gui, name=gatekeeper)
rather than the whole cozy-linstor namespace — the ingress controller
does not need to reach the LINSTOR controller, satellites, or CSI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two issues surfaced while deploying to dev10:
1. The shared linstor-gui.labels / linstor-gui.selectorLabels helpers
set app.kubernetes.io/name=linstor-gui, which overrode the
app.kubernetes.io/name=gatekeeper that the gatekeeper templates set
first. The resulting Deployment had selector.matchLabels=gatekeeper
but template.metadata.labels=linstor-gui, failing admission with
"selector does not match template labels". Drop the helper include
from the three gatekeeper templates and inline the remaining common
labels (managed-by, part-of) so the gatekeeper keeps its own
name label.
2. Copying client.session.idle.timeout=86400 and
client.session.max.lifespan=604800 from the dashboard KeycloakClient
caused the keycloak-operator to reject the client ("Client session
idle timeout cannot exceed realm SSO session idle timeout"). Our
dev10 realm has smaller defaults, and dashboard's own client is
also stuck in this failed state. Drop the overrides and let the
realm defaults apply — the UI has no reason to demand longer
sessions than the realm is willing to grant.
Verified on dev10:
* gatekeeper Deployment rolls out cleanly
* KeycloakClient reconciles to status=OK
* https://linstor-gui.<root-host>/ returns HTTP 302 to
keycloak.<root-host>/realms/cozy/protocol/openid-connect/auth
with client_id=linstor-gui and the correct callback URI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Follow-up to the initial linstor-gui package (#2382). Publishes the UI on
https://linstor-gui.<root-host> behind the cluster Keycloak realm using
the same oauth2-proxy gatekeeper pattern that dashboard uses, so the
LINSTOR REST API is never exposed unauthenticated through Ingress.
Rendered only when both gates are satisfied:
* _cluster.oidc-enabled == "true"
* "linstor-gui" is listed in _cluster.expose-services
(i.e. in publishing.exposedServices in the core cozystack values).
If OIDC is disabled cluster-wide we intentionally do not ship an
unauthenticated token-proxy fallback — unlike dashboard, there is no
reason to front a raw storage-management API with k8s bearer tokens.
Operators can still reach the UI via `kubectl port-forward` to the
ClusterIP service.
New resources:
* templates/gatekeeper.yaml — oauth2-proxy Deployment (OIDC mode)
* templates/gatekeeper-svc.yaml — ClusterIP :8000 in front of it
* templates/gatekeeper-sa.yaml — dedicated SA, no autotoken
* templates/keycloakclient.yaml — persistent client + cookie
Secrets, KeycloakClient CRD with
redirectUris for /oauth2/callback
* templates/ingress.yaml — Ingress to the gatekeeper Service,
cert-manager ClusterIssuer, gated
on expose-services + oidc-enabled
* tests/ingress_auth_test.yaml — unit tests for each conditional
branch and the client CRD
README.md updated with both Option 1 (Keycloak-protected Ingress) and
the existing Option 2 (port-forward).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The matchRegex assertion used 5 literal spaces but the template
renders 9 (aligned with proxy_ssl_trusted_certificate). Use \s+
so the test is resilient to alignment changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback:
- Sync scope lists between PR template and contributing guide
- Add 'maintenance' to Other scopes in contributing guide
- Add scope to the docs example for format consistency
- Simplify rebase push example to drop unnecessary refspec mapping
- Add 'text' language tag to trailer code fence (MD040)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Refocus docs/agents/contributing.md on project-side conventions only
and drop personal agent-behavior guidance.
Switch commit format from [component] prefix to Conventional Commits
(type(scope): description) in contributing.md, overview.md, AGENTS.md,
.gemini/styleguide.md and the PR template.
Reference .github/PULL_REQUEST_TEMPLATE.md instead of duplicating the
PR body template in contributing.md.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Fix /healthz Content-Type header: nginx silently drops `add_header`
after `return` because the response is already finalized. Switch to
`default_type text/plain;` placed BEFORE the `return 200` so the
probe response is actually labeled as text/plain.
- Drop the redundant `checksum/nginx-config` pod-template annotation:
the deployment already carries `reloader.stakater.com/auto: "true"`,
so Stakater Reloader handles ConfigMap rollouts. Matches the sister
`linstor` package convention.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds configuration for the two AI reviewers active on this repository —
Gemini Code Assist and CodeRabbit.
Previously neither had a repo-level config, so both ran with defaults:
Gemini had no knowledge of the project's vendoring model and would flag
issues in vendored upstream charts and generated code; CodeRabbit only
ran on initial PR open, with no review on subsequent pushes.
Changes:
- `.gemini/config.yaml` — ignore patterns for vendored charts,
`vendor/`, generated code, build artifacts, dashboard JSONs, and
upstream patch files; severity threshold set to `LOW` for strict review
of hand-written code; comment cap to prevent floods on chart-update PRs.
- `.gemini/styleguide.md` — natural-language guide covering project
architecture, vendoring rules (where to send bug fixes instead of
editing `charts/`), commit and PR format requirements, sensitive
components, Go code standards, and explicit anti-patterns the reviewer
should not flag.
- `.coderabbit.yaml` — enables incremental reviews so CodeRabbit
re-reviews each push, and skips drafts.
### Release note
```release-note
[ci] Add Gemini Code Assist and CodeRabbit configuration
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added code review automation configuration settings
* Added repository style guide and development conventions documentation
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Makes `cozystack-scheduler` LINSTOR storage-aware by configuring it to
call the existing `linstor-scheduler-extender` as a scheduler extender.
This fixes#2328 (phase 1): pods with both a `SchedulingClass` and
LINSTOR PVCs now get storage-locality-aware placement instead of
bypassing LINSTOR's filter/prioritize logic.
Changes:
- **linstor-scheduler package**: expose the extender sidecar (port 8099)
via a new ClusterIP Service; patch the vendored deployment to add a
`app.kubernetes.io/component: scheduler` label for clean Service
targeting
- **cozystack-scheduler package**: bump to v0.3.0 which adds
configurable `extenders` support; set the LINSTOR extender URL in
wrapper values
- **tests**: helm-unittest for extender rendering in the ConfigMap
### Release note
```release-note
[scheduler] cozystack-scheduler now calls the LINSTOR scheduler extender for storage-aware pod placement. Pods assigned to a SchedulingClass that also use LINSTOR-backed PVCs will prefer nodes with local volume replicas.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added scheduler extender support: exposes an extender Service and adds
pod metadata to enable extender routing.
* **Chores**
* Bumped scheduler chart and image versions to 0.3.0.
* Added default extender configuration and a sample remote extender
entry.
* Updated build scripts to apply an additional patch and added a test
target.
* **Tests**
* Added Helm tests validating rendered ConfigMap with and without
extender configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Replace `3297617+cozystack-ci[bot]@users.noreply.github.com` with
`cozystack@users.noreply.github.com` across all CI workflows
- DCO probot rejects the `[bot]` brackets in email as invalid, causing
release PRs to fail DCO checks
## Test plan
- [ ] Verify DCO passes on release PRs after backport to release
branches
The cozystack-ci[bot] noreply email with brackets is rejected by DCO
probot as an invalid email address, causing release PRs to fail DCO
checks.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
The aggregated API previously accepted tenant names containing hyphens
(e.g. `foo-bar`) because `ValidateApplicationName` delegated entirely to
`IsDNS1035Label`, which permits them. The tenant Helm chart's
`tenant.name` helper then rejected the release at template time — so
users saw a successful `kubectl apply` followed by a confusing
Flux/HelmRelease reconciliation error.
This PR extends `ValidateApplicationName` with a `kindName` parameter
and enforces an alphanumeric-only rule (`^[a-z][a-z0-9]*$`) for the
Tenant kind. A `TenantKind` constant centralizes the kind string, and
`REST.validateNameFormat` wraps the check symmetrically with the
existing `validateNameLength`. The tenant chart README is updated to
match the already-correct website documentation.
Test coverage includes:
- Unit tests for the validation package (format + error message
contract)
- REST wrapper and Update→Create fall-through path tests with a fake
client
- E2E BATS regression test with explicit exit-code checks and cleanup
Fixes#2375
### Release note
```release-note
[api] Tenant names containing dashes are now rejected at API level during creation, with a specific error message, instead of being silently accepted and failing later during Helm reconciliation.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Tenant names now require lowercase letters and digits only, must start
with a letter; dashes are rejected with a tenant-specific error.
* **Documentation**
* Tenant naming guidelines updated to reflect the stricter
alphanumeric-only requirement.
* **Tests**
* New unit and end-to-end tests added to verify tenant naming
enforcement and tenant-specific error messaging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Three corrections from PR review:
- Package group list was incomplete. Added library/ (reusable helper
charts) and tests/ (tests for library charts, which are not directly
testable otherwise).
- CRD generation was described incorrectly. Static CRDs are generated
by hack/update-codegen.sh from types under api/v1alpha1/, api/backups/,
and api/dashboard/. Types under api/apps/v1alpha1/ are not static
CRDs — they are registered at runtime from ApplicationDefinition.
- Removed the tgz anti-pattern. No *.tgz files exist outside _out/,
which is gitignored. The guidance was fabricated.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expose the existing linstor-scheduler-extender sidecar as a ClusterIP
Service and configure cozystack-scheduler to call it during the
scheduling cycle. Pods with both a SchedulingClass and LINSTOR PVCs
now get storage-locality-aware placement.
- Add extender Service (port 8099) for linstor-scheduler
- Patch vendored deployment to label pods for Service selector
- Bump cozystack-scheduler to v0.3.0 (configurable extenders)
- Add "linstor" PackageSource variant with extender values
- Default variant ships without extender for non-LINSTOR clusters
- Select linstor variant in system bundle
- Add helm-unittest tests for both packages
Ref: #2328
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## Summary
- Add `--match 'v*'` to all `git describe` calls in
`hack/common-envs.mk`
- The `api/apps/v1alpha1/*` subtags share the same commit as release
tags, causing `git describe --exact-match` to pick
`api/apps/v1alpha1/vX.Y.Z` instead of `vX.Y.Z`, producing invalid Docker
image tags like `ghcr.io/.../image:v1.35-api/apps/v1alpha1/v1.1.6`
## Test plan
- [ ] Rerun Versioned Tag pipelines after merge and backport to release
branches
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Refined version and tag derivation in the build system to selectively
recognize Git tags, improving consistency in how release versions are
identified.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The api/apps/v1alpha1/* subtags share the same commit as the release
tags. git describe --exact-match picks the first match alphabetically,
returning api/apps/v1alpha1/vX.Y.Z instead of vX.Y.Z, which produces
invalid Docker image tags.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add a minimal .coderabbit.yaml that ensures CodeRabbit re-reviews each
push to a PR (incremental review on new commits) and skips drafts.
Without this file, the org-level configuration left incremental reviews
disabled, so only the initial PR open triggered a review.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Fix two issues caught in PR review:
- Use quadruple-backtick fence around the release-note example so the
inner triple-backtick block renders correctly and satisfies markdownlint.
- Correct the platform migration path. The previous text pointed at
scripts/migrations/, which does not exist. The actual migration flow
lives in packages/core/platform/templates/migration-hook.yaml and
packages/core/platform/images/migrations/.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
- Replace `GH_PAT` (cozystack-bot PAT) with `cozystack-ci` GitHub App
token in `pull-requests.yaml`
- This was missed in #2351 and broke the `resolve_assets` / `e2e` jobs
for release PRs (draft releases not visible with invalid token)
## Test plan
- [ ] Re-run the release PR pipeline for `release-1.1.6` and verify
`resolve_assets` job passes
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated continuous integration authentication mechanism to use GitHub
App tokens instead of personal access tokens.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Follow-up to 4b76a93d: the assertion still expected defaultMode 0400
and would fail in CI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The linstor-client-tls secret was mounted with defaultMode 0400 and
owned by root, so the nginx process (UID 101) got EACCES on tls.crt
and crash-looped with:
[emerg] cannot load certificate "/etc/linstor/client/tls.crt":
BIO_new_file() failed ... Permission denied
Set defaultMode to 0444. The secret volume is pod-local and
readOnlyRootFilesystem is on, so making it world-readable inside the
pod is not a broader disclosure.
Verified on dev10: pod Ready, /healthz 200, /v1/nodes proxied through
mTLS to linstor-controller:3371 returns real node data.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The GH_PAT secret tied to cozystack-bot is no longer valid after
migrating release workflows to the cozystack-ci GitHub App in #2351.
This broke the resolve_assets and e2e jobs for release PRs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Dockerfile: chown html root to 101:101 (match runAsGroup, not 101:0)
- Makefile: declare .PHONY targets (image, image-linstor-gui, test)
- deployment: drop priorityClassName system-cluster-critical (GUI is
not control-plane critical; chart does not need to reserve the slot)
- configmap-nginx:
- fail the render if linstor.endpoint is not https:// so a misconfig
cannot silently downgrade the mTLS-protected controller path
- hoist proxy_set_header and proxy_ssl_* directives to the server
block instead of duplicating across /v1 and /metrics locations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Ships LINBIT's LINSTOR web UI (GPL-3.0) as an opt-in system package under
packages/system/linstor-gui so operators can manage LINSTOR nodes, resources,
and volumes from a browser instead of the linstor CLI.
The UI is built from the upstream pkg.linbit.com tarball onto an
nginx-unprivileged base image. A chart-supplied nginx.conf proxies /v1 and
/metrics to the LINSTOR controller REST API over mTLS using the existing
linstor-client-tls secret created by the linstor package.
Only a ClusterIP Service is created; no Ingress is shipped, because LINSTOR's
controller API is a privileged cluster-wide storage surface and auth depends
on the deployment's OIDC setup. Operators opt in via bundles.enabledPackages
and wire up ingress + auth themselves (port-forward works out of the box).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Configure the Gemini Code Assist GitHub reviewer with project-specific
ignore patterns and a natural-language style guide.
The config silences noise from vendored upstream Helm charts, generated
Go code, build artifacts, and Grafana dashboard JSONs, while keeping the
severity threshold at LOW for strict review of hand-written code.
The style guide teaches the reviewer about the vendoring model, commit
format requirements, sensitive components like packages/core/platform/,
and lists anti-patterns not to flag (patch files, go.sum, upstream forks).
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace bare "Tenant" string literals with the validation.TenantKind constant in all test table entries, struct fields, and TypeMeta. Prevents silent drift if the canonical kind string is ever renamed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The convertHelmReleaseToApplication fall-through that populates Status.Namespace for tenant resources still compared r.kindName against the bare string "Tenant". The validation package already exposes TenantKind for exactly this reason: if the kind string ever diverges from the ApplicationDefinition source of truth, the constant and the call site drift together rather than silently desynchronizing.
Add TestConvertHelmReleaseToApplication_TenantNamespaceKindGate to pin both sides of the gate: tenant kind populates Status.Namespace, non-tenant kind leaves it empty. Also rewrite the TestUpdate comment to describe the upsert invariant directly instead of referencing an issue number.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Four follow-ups from review round 4:
1. BLOCKER: the Update(forceAllowCreate=true) path delegates to Create() when the object does not yet exist (rest.go:452) — the typical kubectl apply upsert flow. Add TestUpdate_ForceAllowCreate_RejectsTenantDashName using a fake client so a future refactor of that delegation cannot silently bypass the tenant name check that r.validateNameFormat alone cannot catch.
2. BLOCKER: the e2e BATS test used || true inside the command substitution, which swallowed the kubectl exit code. Rework the test to capture exit code and stdout+stderr explicitly, then assert the exit code is non-zero before asserting on the error message. This distinguishes validation-success (kubectl exit 0 — regression) from environmental failures (exit non-zero but wrong message) from the happy path.
3. Extract TenantKind = "Tenant" as a named constant in the validation package with a comment pointing at the upstream ApplicationDefinition source of truth, and switch the kindName check to use it.
4. Add a clarifying comment on TestValidateApplicationName_TenantLengthFallthrough that it pins an architectural layering decision and is not a user-facing requirement, so a future promotion of tenant length into tenant-specific wording is a legitimate change rather than a test regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three follow-ups from review:
1. Add defensive cleanup of the foo-bar tenant before and after the e2e regression test. If a prior run left the object in the cluster (e.g. a transient validation regression), the test no longer inherits that state; and if the test itself trips a regression in the future, the cluster is left clean for subsequent tests.
2. Switch kubectl --validate=false to --validate=ignore. The bool form was deprecated in kubectl 1.25 and only kept as an alias; --validate=ignore is the modern, stable spelling.
3. Drop the brittle Contains("63") assertion in TestValidateApplicationName_TenantLengthFallthrough. It pinned the test to the exact DNS-1035 error text from k8s.io/apimachinery, which could break on unrelated upstream wording changes. The surviving assertions still cover the intent: an error exists and it is not the tenant-specific message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Follow-up from review round 2:
The BATS regression test now pins a namespace precondition, disables kubectl client-side validation with --validate=false so the request is guaranteed to reach the server-side name check, and documents the intent of each assertion. Previously a client-side schema rejection could have produced a false negative on the tenant-specific error grep, and a missing tenant-root namespace could have produced an unrelated failure that was hard to diagnose.
Also pin the edge case where a tenant name consists of valid characters but exceeds the DNS-1035 63-char label limit. The resulting error is intentionally the generic DNS-1035 message because length is not a tenant-specific constraint — the package-level function is not responsible for the stricter Helm-release-prefix length budget that REST.validateNameLength enforces.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two follow-ups from review:
1. The original tenantNameRegex ^[a-z0-9]+$ let leading-digit names like 123foo slip past the tenant check and receive a generic DNS-1035 error, which defeated the goal of surfacing a tenant-specific message. Require a leading lowercase letter so every tenant-invalid name returns the tenant-contract error.
2. Add a BATS regression test in hack/e2e-install-cozystack.bats that exercises the aggregated API end-to-end. Unit tests construct REST{kindName: "Tenant"} by hand, so a future change to ApplicationDefinition kind registration could break real behavior without breaking the unit tests.
Also pin the error-message contract with a new TestValidateApplicationName_TenantErrorMessage that asserts every tenant-invalid input returns a message containing 'tenant names must' (not the generic DNS-1035 text).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The README stated that dashes are 'allowed but discouraged' in tenant names, but the tenant.name Helm helper explicitly fails on release names that contain more than one dash, and the platform guide on the website already says tenant names must be alphanumeric. Bring the chart README in line with both the website documentation and the enforcement now present on the aggregated API.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ValidateApplicationName previously delegated to IsDNS1035Label, which permits hyphens. The tenant Helm chart's tenant.name helper rejects them at template time, so users saw a successful kubectl apply followed by a Flux reconciliation error.
Extend ValidateApplicationName with a kindName parameter and enforce alphanumeric-only names for the Tenant kind, matching the documented naming rules. Wire the check into REST.Create via a small validateNameFormat wrapper that mirrors validateNameLength.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds a host preflight diagnostic script (`hack/check-host-runtime.sh`)
that warns operators when a standalone `containerd.service` or
`docker.service` is running on the host alongside the embedded k3s
runtime used by the cozystack `generic` variant (k3s / kubeadm on
Ubuntu).
**Why it matters.** K3s ships its own containerd at
`/run/k3s/containerd/containerd.sock` and
`/var/lib/rancher/k3s/agent/containerd`, while a system-package
containerd or docker uses `/run/containerd/containerd.sock` and
`/var/lib/containerd`. The two runtimes do not fight over sockets, so
both keep running silently. Over time the standalone one accumulates
unpruned images and build cache in `/var/lib/containerd` — enough to
fill the root disk, trigger `DiskPressure`, and put `cozystack-api` into
an eviction loop. This is silent on day zero and surfaces as a
mysterious production incident weeks later. The script exists to warn
the next operator before the failure mode surfaces.
The script is warning-only — it always exits 0 and never blocks the
install. It detects:
- `containerd.service` or `docker.service` active via `systemctl
is-active`
- Standalone runtime sockets at well-known paths, with a fallback that
works on hosts without systemd
- Standalone data directory sizes via `du -sh`
When a warning fires, the HINT names only the detected services and
instructs the operator to disable them with `sudo systemctl disable
--now <service>`. Reclaiming the data directory is called out separately
with an explicit note not to delete it blindly — the data may still be
in use.
**Entry points.**
- `make preflight` runs the script directly (for operators preparing a
generic-variant host)
- `make unit-tests` now runs `bats-unit-tests` alongside
`helm-unit-tests`, auto-discovering every `hack/*.bats` file that is not
an e2e test
**Test coverage.** `hack/check-host-runtime.bats` (11 cases, run via
`hack/cozytest.sh`) covers clean hosts with and without systemd,
single-service detection, both services simultaneously, socket-only
fallback for both runtimes, glob-expansion regression guard, explicit
exit-code-0 assertion, `sudo` prefix assertion, `du` failure robustness,
and the "service + socket = exactly one warning" de-duplication
invariant. Every test is self-contained with `trap 'rm -rf $STUB_DIR'
EXIT` for clean recovery on assertion failure, and has no runtime
dependencies beyond bash and core utilities — no python3, no real
systemd.
Irrelevant on Talos where the container runtime lifecycle is fully
managed by the distribution.
### Release note
```release-note
[hack] Add `check-host-runtime.sh` and `make preflight` target that warn when a standalone containerd or docker runtime is running alongside the embedded k3s runtime on the cozystack generic variant.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a non-blocking preflight check that detects standalone container
runtimes, reports disk-usage estimates, and displays an actionable hint
to disable detected services.
* **Tests**
* Added BATS-based unit tests with comprehensive coverage for the
preflight validations and various host scenarios.
* **Chores**
* Build updated to run the BATS unit tests alongside the existing test
suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The aggregated `cozystack-api` validates each individual Tenant name
length against the Helm release limit (46 chars for the `tenant-`
prefix), but it never validates the **computed workload namespace** that
is built by dash-joining the parent namespace with the tenant name. A
user can therefore `kubectl apply` a nested `Tenant` whose own name
passes the per-name check, only to have Kubernetes later reject the
generated `Namespace` for exceeding the 63-character DNS-1123 label
limit. The failure surfaces as an opaque HelmRelease reconcile error,
and the Tenant CR is left stranded in the cluster with a
`status.namespace` that can never exist.
This PR closes that gap at the aggregation layer by rejecting such
requests synchronously in `Create()` with a clear `field.Invalid` error
that names the computed namespace and its length, so the failure is
reported at `kubectl apply` time before any HelmRelease is created.
The change is scoped to the `Tenant` kind only. `Update()` is not
touched because Kubernetes names and namespaces are immutable and the
existing code already intentionally skips name validation there. No Helm
chart, CRD, or codegen changes are required.
The PR is split into two commits that document the bug and its fix via
TDD:
1. **Pin the gap** — adds a test that demonstrates the pre-fix behavior:
the pre-fix Create() path has no function that rejects a realistic
nested-tenant combination whose computed namespace exceeds 63 chars.
Reviewers can check out this commit and observe the gap.
2. **The fix** — adds `validateTenantNamespaceLength`, the `kindName ==
"Tenant"` gate in `Create()`, and `TestValidateTenantNamespaceLength`
covering boundary cases (pass at 63, fail at 64, fail at 68). The
pinning test from commit 1 is removed in the same commit now that the
gap is closed.
### Release note
```release-note
[platform] cozystack-api now rejects Tenant creation at admission time when the ancestor-chain namespace would exceed the 63-character Kubernetes namespace limit, instead of allowing the resource through and failing later at HelmRelease reconcile.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Tenant application creation now validates computed tenant namespace
length against the Kubernetes DNS-1123 63-character limit; creations
that would produce too-long namespace names now fail with a clear
validation error.
* **Tests**
* Added tests covering tenant namespace length validation, including
multiple parent/tenant-name scenarios, boundary conditions, and
verification that error messages include the offending namespace and its
length.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! -->
## What this PR does
Cilium 1.19 init containers `mount-cgroup`, `apply-sysctl-overwrites`
and `clean-cilium-state` call `nsenter` to join the host cgroup/mount
namespace. On any distribution that loads the
`cri-containerd.apparmor.d` AppArmor profile by default for containerd
workloads (Ubuntu 22.04+, Debian with AppArmor), the kernel denies the
operation and the audit log reports it as a blocked `ptrace` operation:
```text
apparmor="DENIED" operation="ptrace" class="ptrace"
profile="cri-containerd.apparmor.d" comm="nsenter"
requested_mask="read" denied_mask="read" peer="unconfined"
```
Symptom on non-Talos clusters: after a cilium rollout the new pod stays
in `Init:CrashLoopBackOff`, and the rest of the platform cascades into
`dependency not ready` until cilium is healthy on every node.
This PR opts the affected containers out of the
`cri-containerd.apparmor.d` profile via deprecated per-container
`container.apparmor.security.beta.kubernetes.io/<name>: unconfined`
annotations. Four containers are annotated: `cilium-agent`,
`clean-cilium-state`, `mount-cgroup`, `apply-sysctl-overwrites`.
`mount-bpf-fs` is intentionally excluded because it renders with its own
`securityContext.privileged: true`, and the kernel does not apply
AppArmor profiles to privileged containers.
### Why pod annotations instead of `podSecurityContext.appArmorProfile`
The upstream Cilium chart already sets
`podSecurityContext.appArmorProfile.type: Unconfined` (the modern k8s >=
1.30 API), but containerd's CRI implementation silently ignores this
field today and keeps applying `cri-containerd.apparmor.d` — see
kubernetes/kubernetes#125069 (still open). Only the deprecated
per-container annotations take effect end-to-end on current k3s /
kubeadm + containerd setups.
### Why a dedicated values file limited to non-Talos variants
On Talos (`isp-full` bundle) `values-talos.yaml` sets
`cgroup.autoMount.enabled: false`, so the upstream chart does not render
the `mount-cgroup` init container. The kube-apiserver rejects a
DaemonSet that carries a per-container AppArmor annotation for a
container that is not in the pod spec:
```
DaemonSet.apps "cilium" is invalid: spec.template.annotations[
container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: "mount-cgroup": container not found
```
To keep the Talos path untouched and only add the annotations where they
are both safe and useful, this PR puts the `cilium.podAnnotations` block
in a new `packages/system/cilium/values-apparmor.yaml` and wires it into
only the non-Talos PackageSource variants (`cilium-generic`,
`kubeovn-cilium-generic`). Those variants always run with
`cgroup.autoMount.enabled: true` (explicit bundle override in
`packages/core/platform/templates/bundles/system.yaml:83-86`), so every
annotated container really exists in the rendered DaemonSet.
On Talos the annotations are omitted entirely — that is safe because the
Talos kernel does not load the AppArmor LSM anyway.
### Vendored chart patch to avoid duplicate keys on unsupported k8s <
1.30
On k8s < 1.30 the upstream cilium chart template emits the same
annotation keys from its own `semverCompare "<1.30.0"` branch in
`cilium-agent/daemonset.yaml`. Letting both sources write to the
annotations mapping would produce a YAML map with duplicate keys.
Cozystack's supported k8s matrix starts at 1.30
(`packages/apps/kubernetes/files/versions.yaml`), so the duplicate
rendering is never observed in practice, but to keep the rendered
manifest clean on any k8s version this PR also adds a `perl -i -0pe`
multi-line delete to `packages/system/cilium/Makefile` that strips the
entire upstream `{{- if not .Values.securityContext.privileged }} / {{-
if semverCompare "<1.30.0" }}` block from the vendored template. A
fail-fast `grep` guard after the `perl` invocation turns a
silently-failed patch (e.g. if upstream reformats the block) into a loud
error. The same patch is applied to the currently vendored copy so the
repository state matches what `make update` produces. This follows the
existing sed-patch pattern already used for `values.yaml` (`Used in
iptables`, `SYS_MODULE`).
### Scope
- `packages/system/cilium/values-apparmor.yaml` — new file containing
the four `cilium.podAnnotations`
- `packages/system/cilium/values.yaml` — unchanged apart from reverting
the previously-added annotations block (the annotations now live in
`values-apparmor.yaml`)
- `packages/system/cilium/Makefile` — extends the `update:` target with
a `perl` multi-line delete (plus a `grep` fail-fast guard) that drops
the entire upstream AppArmor annotation block on every vendoring refresh
-
`packages/system/cilium/charts/cilium/templates/cilium-agent/daemonset.yaml`
— the vendored file with that block already removed to match the
Makefile patch
- `packages/core/platform/sources/networking.yaml` — adds
`values-apparmor.yaml` to the `cilium-generic` and
`kubeovn-cilium-generic` variants only
### Compatibility across supported platforms
- **Talos Linux** (`isp-full` bundle, `kubeovn-cilium` variant) —
AppArmor LSM is not loaded, `cgroup.autoMount.enabled: false`,
`values-apparmor.yaml` is not included. Zero annotations in the rendered
DaemonSet. No behavioural change. Verified with `helm template` +
`kubectl apply --dry-run=server` against a live k3s v1.35.2 cluster: no
errors, no AppArmor lines.
- **Ubuntu 22.04 / 24.04, Debian with AppArmor** (`isp-full-generic`
bundle, `kubeovn-cilium-generic` / `cilium-generic` variants) — the four
annotations are present in the rendered DaemonSet, kubelet starts the
listed containers with the unconfined profile, `nsenter` succeeds.
Verified with `helm template` + `kubectl apply --dry-run=server` against
a live k3s v1.35.2 cluster: no errors, exactly four
`container.apparmor.security.beta.kubernetes.io/*: unconfined` lines.
Verified on a fresh k3s + Ubuntu 24.04 deployment end-to-end.
- **RHEL / Rocky / AlmaLinux / Fedora** — use SELinux, not AppArmor.
When deployed via `*-generic` variants these platforms pick up the
annotations; kubelet silently ignores AppArmor annotations when the LSM
is not loaded, SELinux policy is unaffected.
- **k8s < 1.30 and k8s >= 1.30** — both now render the same four
annotations (on non-Talos), sourced exclusively from
`cilium.podAnnotations`. Verified via `helm template --kube-version
1.29.0` and `--kube-version 1.35.0` → exactly 4 annotations with no
duplicates.
### Release note
```release-note
[cilium] Opt four cilium-agent containers (cilium-agent, clean-cilium-state, mount-cgroup, apply-sysctl-overwrites) out of the cri-containerd.apparmor.d AppArmor profile via per-container annotations, applied only on non-Talos cilium variants (cilium-generic, kubeovn-cilium-generic). This unblocks cilium rollouts on Ubuntu 22.04+, Debian and other distributions that load this profile for containerd by default, where the kernel previously denied nsenter's namespace entry (reported as a blocked ptrace operation) and the cilium agent landed in Init:CrashLoopBackOff. Talos variants are untouched.
```
## Summary
Replaces the `cozystack-bot` user account + personal access token
(`GH_PAT`) with the `cozystack-ci` GitHub App across all CI/CD release
workflows.
### Why
- **Security**: GitHub App tokens are short-lived (1h) and scoped, vs. a
long-lived PAT tied to a user account
- **Auditability**: App actions are clearly attributed to
`cozystack-ci[bot]`
- **No 2FA issue**: The bot account lacked 2FA; GitHub Apps don't
require it
- **Best practice**: GitHub recommends Apps over PATs for automation
### Changes
| Workflow | What changed |
|---|---|
| `tags.yaml` | App token in all 3 jobs: prepare-release,
generate-changelog, update-website-docs |
| `auto-release.yaml` | App token for daily auto-patch tagging |
| `pull-requests-release.yaml` | App token for release PR finalization |
All `cozystack-bot` git identity replaced with `cozystack-ci[bot]`.
### New secrets (already configured at org level)
- `COZYSTACK_CI_APP_ID` — GitHub App ID
- `COZYSTACK_CI_PRIVATE_KEY` — GitHub App private key
### After merge
- [ ] Delete `GH_PAT` repo-level secret
- [ ] Remove `cozystack-bot` from the cozystack org
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated CI/CD automation authentication mechanisms across release and
tagging workflows for improved security practices.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Reject Tenant creation when the computed workload namespace (parent
namespace + "-" + tenant name) would exceed the 63-character DNS-1123
label limit. Previously only the per-name length was checked against
the Helm release limit, so a deeply-nested tenant could pass Create()
and then fail later as an opaque HelmRelease reconcile error.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The disk_usage helper shells out to 'du -sh $path' for reporting in
the warning message. On the exact directories this script is meant
to warn about (a /var/lib/containerd accumulating millions of files
from months of unpruned builds), traversing the tree with du can
take minutes and stall the preflight indefinitely.
Wrap the call in 'timeout 5s' so the helper returns quickly even on
a pathologically slow filesystem. If the timeout binary is absent
(e.g. minimal busybox userland), the pipeline still exits 0 via the
existing '|| true' guard and usage stays empty — the warning still
prints, just without the size detail — so the change is backward
compatible.
Addresses an inline review comment from gemini-code-assist on the
open PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
E2E revealed that the kube-apiserver rejects a DaemonSet carrying a
per-container AppArmor annotation for a container that is not present
in the pod spec:
DaemonSet.apps 'cilium' is invalid:
spec.template.annotations[container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: 'mount-cgroup': container not found
On Talos (isp-full bundle) values-talos.yaml sets
cgroup.autoMount.enabled=false, so the upstream chart does not render
the mount-cgroup init container, and our unconditional podAnnotations
entry for it becomes an invalid reference.
Move the podAnnotations block from the shared values.yaml into a new
values-apparmor.yaml and include it only in the cilium-generic and
kubeovn-cilium-generic PackageSource variants, where
cgroup.autoMount.enabled is explicitly true. Talos variants
(cilium, cilium-kilo, kubeovn-cilium) get no AppArmor annotations,
which is safe because the Talos kernel does not load the AppArmor
LSM anyway.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Broaden the grep guard in the Makefile update: target to match any
container.apparmor.security.beta.kubernetes.io key, not only the
quoted cilium-agent line. The previous pattern would silently pass
if upstream switched to an unquoted annotation value, masking a
failed perl patch; grepping for the prefix catches any residual
hardcoded AppArmor annotation regardless of format.
- Rewrite the values.yaml comment to distinguish the two Talos-safe
reasons: mount-cgroup is simply not rendered when
cgroup.autoMount.enabled is false (Talos default), while the other
annotations are harmless because kubelet ignores AppArmor metadata
on nodes without the LSM loaded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the misleading 'nsenter --ptrace' shorthand with an accurate
description: nsenter joins the host cgroup/mount namespace, and the
cri-containerd.apparmor.d profile reports the denial as a blocked
'ptrace' operation (operation name, not an nsenter CLI flag).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The perl multi-line delete in the update: target silently becomes a
no-op if the upstream template is ever reformatted. Verify the patch
applied by grepping for one of the stripped annotation keys after the
perl runs; fail the update: target loudly if it still exists, so a
future upstream reformat is caught instead of silently reintroducing
the duplicate-key rendering on k8s < 1.30.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous sed patch removed only the four annotation lines and
left an empty nested '{{- if ... }}{{- end }}{{- end }}{{- end }}'
skeleton behind. That skeleton was dead code, but it also meant that
the next make update would re-fetch the upstream chart with the
annotations back inside the same block, and the sed would strip them
again — leaving the skeleton behind forever.
Replace the per-line sed patch with a single perl -i -0pe multi-line
delete that removes the entire <1.30 AppArmor block in one step, and
apply the same patch to the currently vendored copy for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The upstream chart emits per-container AppArmor unconfined annotations
inside a k8s<1.30 branch of cilium-agent/daemonset.yaml. Cozystack now
owns these annotations via cilium.podAnnotations in values.yaml so they
also take effect on k8s>=1.30 (where upstream drops them in favour of
the containerd-broken podSecurityContext.appArmorProfile). Having both
sources emit the same keys produced a duplicate mapping on k8s<1.30.
Add a new SED_INPLACE invocation to the package Makefile that removes
the four hardcoded annotation lines from the vendored template, and
apply the patch to the currently vendored copy so make update will
reproduce the same state on future refreshes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
$(wildcard ...) returns a space-separated list, so a bats file with
a literal space in its name would silently split into multiple
tokens in the subsequent 'for' loop. All current bats files use
hyphen-separated names, so this is not an immediate risk, but the
caveat is worth calling out so a future contributor who adopts a
different naming convention rewrites the recipe instead of tripping
over it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address three review findings on the bats suite:
- The 'standalone containerd service active' and 'standalone docker
service active' tests previously asserted 'systemctl disable --now
<service>' but did not require the 'sudo ' prefix, so the two-
service test was the only guard against a silent drop of the
prefix. Extend both single-service tests to require the prefix
exactly as the 'both services active' test does.
- The 'both services active' test now captures the exit code
explicitly with 'bash ... || status=$?' and asserts '$status -eq
0'. The script contract ('always exits 0') was previously enforced
implicitly via 'set -e', which produces a generic test failure on
a regression instead of a contract-specific error. The explicit
check locks in the contract and makes regressions readable.
- New test: 'docker socket paths with glob chars do not expand' —
sets COZYSTACK_DOCKER_SOCKET_PATHS to a literal glob against real
directories and asserts no docker warning fires. Locks in the
array-based parsing of the path list.
- Extend the file header with a 'title syntax constraints' section
documenting how cozytest.sh's awk parser treats double quotes and
non-alphanumeric characters in @test titles, so future
contributors do not stumble on the parser's quirks.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
'set -euo pipefail' does not include 'set -f', so the previous
'for sock in $DOCKER_SOCKET_PATHS' loop relied on both word splitting
AND unintended glob expansion. If the variable ever contained a
literal '*' or '?' — for example a test setting
COZYSTACK_DOCKER_SOCKET_PATHS='/run/docker-*.sock' against a tree
that happens to have real files matching the pattern — the loop
would iterate over directory entries instead of the intended socket
paths, producing false-positive docker warnings.
Replace the implicit split+glob loop with 'read -ra' into a bash
array and iterate 'for sock in "${_socks[@]}"'. This keeps the
space-separated input format, disables glob expansion, and removes
the fragile word splitting that shellcheck otherwise has to be
talked out of.
Covered by a new regression test ('docker socket paths with glob
chars do not expand') that sets the env var to a literal '*' pattern
pointing at real directories; without this fix the test fails with
an unexpected docker.service warning.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
DOCKER_SOCKET_PATHS is a space separated list of socket paths (its
default is "/run/docker.sock /var/run/docker.sock") that the loop
iterates by word splitting. Socket paths never contain whitespace on
Linux hosts, so this is the documented and correct idiom — not a
shellcheck oversight. Add an inline comment so a future maintainer (or
a linter that has not seen the comment) does not break the loop by
adding quotes around the expansion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Without a guard, running make bats-unit-tests in a tree where the
hack/*.bats wildcard matches nothing (for example a rename, a shallow
clone, or a future reorganization that moves the tests elsewhere)
would silently succeed with zero tests run. Add an explicit guard that
fails the target with a clear error message when BATS_UNIT_FILES
expands to an empty list, so the developer cannot accidentally believe
tests passed when no tests executed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two review findings:
1. The socket fallback tests previously branched on 'command -v
python3' and fell through to 'return 0' when it was missing.
cozytest.sh has no SKIP concept — 'return 0' is indistinguishable
from a real pass, so on a runner without python3 the socket
fallback paths were silently unverified. Since the script uses
'[ -e "$sock" ]' rather than '[ -S ... ]', a regular file is
sufficient to exercise the detection path. Replace the python3
unix-socket creation with 'touch "$SOCK"' so the tests run
unconditionally on every runner.
2. Extend the 'both services active' HINT assertion to require the
'sudo ' prefix on the 'systemctl disable --now' line. Without sudo
the operator would be instructed to run a privileged command as a
non-root user and hit a silent failure; the prefix is as important
as the verb itself and must be locked in by a test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two small Makefile tweaks:
1. bats-unit-tests now iterates over every hack/*.bats file that is
not an e2e-*.bats file (wildcard + filter-out), so dropping a new
unit bats file into hack/ is enough to get it picked up by CI on
the next run. The previous hard coded single file path would have
required a Makefile edit for every new unit suite.
2. Add a new 'preflight' target that runs hack/check-host-runtime.sh.
This gives the script a discoverable entry point — an operator
preparing a generic (k3s/kubeadm) host for cozystack can now run
'make preflight' from the repository root to get the same warning
that is described in the script header, without having to know
the exact path under hack/.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expand the bats suite to address a second review pass:
- Assert that a containerd only warning produces a HINT that names
containerd.service and NOT docker.service, and mirror the check in
the docker only test. When both fire, assert the HINT lists both
services in a single systemctl disable invocation.
- New test: 'clean host without systemctl exits silently' — exercises
the COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 path when no standalone
sockets exist. Previously the tests only covered the systemd enabled
clean host, leaving the non systemd clean path unverified.
- New test: 'docker service plus socket still emits exactly one
warning' — mirrors the existing containerd service+socket test and
locks in the gated check in check_docker.
- Replace silent 'return 0' on missing python3 with a visible
'# SKIP: python3 unavailable' message on stderr so CI logs make it
obvious which tests were skipped on a given runner.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When only containerd.service was active, the HINT block still advised
the operator to disable both containerd.service and docker.service,
which is misleading and potentially dangerous on hosts where docker is
legitimately in use. Track per service warnings (CONTAINERD_WARN and
DOCKER_WARN) and build the HINT systemctl disable argument from the
services that actually fired.
Also guard the ANSI color escapes behind an 'is stderr a TTY' check so
log files (CI and reviewer captures) do not accumulate raw escape
sequences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback on the preflight test suite:
- Add a trap 'rm -rf $STUB_DIR' EXIT in every test immediately after
mktemp -d, so temp dirs are cleaned up even when an assertion fails
and terminates the test early under set -e.
- Assert the HINT block and 'systemctl disable --now' line in the
'both services active' test so a future silent removal or typo in
the HINT output is caught by CI.
- New test: 'docker socket fallback fires when systemctl is
unavailable' — mirrors the existing containerd socket fallback test
and exercises the same code path in check_docker.
- New test: 'containerd service plus socket still emits exactly one
warning' — documents and locks in the intent of the symmetric gated
check in check_containerd. Uses grep -c to assert the warning is not
double printed when both signals fire.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two small issues caught in review:
1. check_containerd probed the socket unconditionally even when
service_active had already set found=1, while check_docker short
circuited the same path behind 'if \[ $found -eq 0 \]'. The
asymmetry was not user visible but violated least surprise for
future maintainers. check_containerd now uses the same gated pattern.
2. The HINT block recommended 'sudo rm -rf /var/lib/docker
/var/lib/containerd' as a casual follow up, which could destroy data
the operator still needs. Replace that line with a warning telling
the operator to inspect and reclaim standalone runtime storage
manually rather than deleting it blindly.
Also drop a no op 'printf' from disk_usage that served no purpose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a bats-unit-tests Make target that runs hack/cozytest.sh against
hack/check-host-runtime.bats, and make unit-tests depend on it so the
existing CI step (make unit-tests in .github/workflows/pull-requests.yaml)
exercises the preflight script on every PR. Without this the bats file
exists in the tree but is never executed, leaving future regressions
undetected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ubuntu hosts running the cozystack "generic" variant (k3s or kubeadm)
sometimes end up with a standalone containerd.service or docker.service
running alongside the embedded k3s runtime. The two runtimes do not fight
over sockets — k3s uses /run/k3s/containerd/containerd.sock while the
standalone package uses /run/containerd/containerd.sock — so both keep
running silently, and the standalone one accumulates unpruned images and
build cache in /var/lib/containerd. Over time this fills the root disk,
triggers DiskPressure, and sends cozystack-api into an eviction loop.
hack/check-host-runtime.sh warns an operator about this before install
without blocking it. The script probes systemctl and well-known socket
paths, reports disk usage of the standalone data directory when present,
prints a hint on how to disable the shadow runtime, and always exits 0.
Covered by hack/check-host-runtime.bats (six test cases).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add hack/check-host-runtime.bats with six self-contained test cases that
exercise the check-host-runtime.sh preflight script: clean host exits
silently, standalone containerd.service warns, standalone docker.service
warns, both services warn, du failures do not suppress warnings, and the
socket-only fallback fires when systemctl is unavailable.
Tests inject a stub systemctl and du binary via PATH and redirect the
script's probe paths through COZYSTACK_PREFLIGHT_* environment variables,
so they run without root and on any host (including non-systemd macOS).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
On Kubernetes versions below 1.30 the upstream cilium chart still emits
the deprecated per-container AppArmor annotations from its own
semverCompare '<1.30.0' branch. Our podAnnotations override then emits
the same keys again, yielding a YAML mapping with duplicate entries
(identical values, last-wins by kubernetes API server JSON semantics).
Cozystack's supported Kubernetes matrix starts at 1.30
(packages/apps/kubernetes/files/versions.yaml), so this rendering
quirk never affects any shipped version. Document the behaviour in a
comment so future readers are not surprised.
No functional change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
mount-bpf-fs runs as privileged in the vendored chart, and the Linux
kernel does not apply AppArmor profiles to privileged containers. The
per-container unconfined annotation for this container is a no-op and
only widens the pod's admission-policy surface for no reason.
Keep the four meaningful annotations (cilium-agent, clean-cilium-state,
mount-cgroup, apply-sysctl-overwrites) which target unprivileged agent
and init containers actually affected by the cri-containerd.apparmor.d
profile.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cilium 1.19 init containers mount-cgroup, apply-sysctl-overwrites,
mount-bpf-fs and clean-cilium-state call nsenter --ptrace, which is
denied by the cri-containerd.apparmor.d profile on Ubuntu 22.04+ and
any other distro loading this AppArmor profile. As a result cilium
agent pods land in Init:CrashLoopBackOff and downstream HelmReleases
cascade into "dependency not ready".
The modern k8s >= 1.30 pod-level appArmorProfile: Unconfined field is
accepted into the DaemonSet spec but not honoured by containerd CRI
at runtime (kubernetes/kubernetes#125069), so the deprecated
per-container AppArmor annotations remain the only reliable opt-out.
Add the annotations via podAnnotations in the wrapper values.yaml.
The change is harmless on systems without AppArmor (Talos,
RHEL/Rocky/AlmaLinux with SELinux) because kubelet silently ignores
unknown AppArmor annotations when the LSM is not loaded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
Fix system postgresql images to 17.7-standard-trixie
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Database Helm charts now preserve and reuse existing PostgreSQL image
settings, normalizing tags to the standard-trixie variant when
appropriate.
* Migration configuration bumped to target version 38.
* **Bug Fixes**
* Migration logic updated to correctly locate and adjust CNPG Cluster
resources and to log per-cluster patch/skip decisions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
- Adds `cozy-monitoring` namespace to the infra dashboards rendering
condition in `dashboards.yaml`
- The default variant deploys the monitoring chart to `cozy-monitoring`,
but the condition from #2197 only checked for `tenant-root`, causing all
infrastructure dashboards to be missing
- This is consistent with the existing pattern in `vmagent.yaml` which
already handles both namespaces
## Release note
```release-note
[monitoring] Fix infrastructure dashboards not rendering in default variant where monitoring is deployed to cozy-monitoring namespace
```
Fixes#2349
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Infrastructure & Monitoring**
* Expanded Grafana dashboard availability across additional deployment
environments for infrastructure monitoring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- update `packages/system/linstor` to `linstor-server` `v1.33.1`
- keep only the selected `piraeus-server` backports needed for this
update
- document the upstream/source commits for each retained patch
- drop the older local patch set that is no longer carried in this
package revision
## Selected backports
- PR #475: backport maintainer implementation from
`3d97f71c95a493588d3d521c63eac4d846935fb3`
- PR #476: backport the TCP port preservation fix from
`79d6375c55d6181b35a7b7f0fe8dbdfb86e126cd` and
`bcc89902f4f61ac1589dd07ebb7f5aae1935370d`
- PR #472: backport the LUKS header sizing fix from
`ccc85fbd2c65f0b97c52403fa80f1efdb886ec4e` plus the required
`optimal_io_size` dependency from
`71b601554f41bcb50cd5bd06989c5b0d3a814acd`
## Verification
- applied all retained patches on top of upstream `v1.33.1` with `git
apply --check`
- verified the combined backport builds with `./gradlew --no-daemon
compileJava`
- ran `make image` in `packages/system/linstor` successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Upgraded LINSTOR server to v1.33.2 and refreshed related package
metadata.
* **Bug Fixes / Reliability**
* Safer toggle-disk flows with retry/abort handling and preservation of
DRBD TCP ports.
* Automatic retry/cleanup for stale bitmap attach errors to improve
recovery.
* **Storage / Compatibility**
* Improved LUKS2 header sizing, optimal I/O size detection, and related
patches for more reliable disk formatting and alignment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The default variant deploys monitoring to the cozy-monitoring namespace,
but the infra dashboards condition only checked for tenant-root. This
adds cozy-monitoring to the condition, consistent with the existing
pattern in vmagent.yaml.
Fixescozystack/cozystack#2349
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
add ability for restoring VMI to copy in another namespace
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* RestoreJob CR supports a free-form spec.options payload; restores now
support in-place preserves (IP/MAC, PVC handling), cross-namespace copy
restores, and safe rename semantics.
* **Documentation**
* Added admin and tenant walkthroughs for Velero-backed
VMInstance/VMDisk backup & restore.
* **Examples**
* New stepwise demo scripts, helpers, orchestration (run-all) and
cleanup scripts.
* **Tests**
* Extensive unit tests covering restore option parsing, target
resolution, and restore side effects.
* **Chores**
* RBAC updated to allow HelmRelease create/delete and namespace listing
for restore flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
All existing patches (allow-toggle-disk-retry, fix-duplicate-tcp-ports,
fix-luks-header-size, retry-adjust-after-stale-bitmap) apply cleanly on
the new version without modifications.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Adds the `service.kubernetes.io/service-proxy-name: "cozy-proxy"` label
to VM LoadBalancer services when `external: true`. This tells Cilium to
completely ignore the service — no BPF entries, no DNAT, no wildcard
drops.
Routing continues to work through kube-ovn
(`networking.cozystack.io/wholeIP` annotation) and MetalLB L2
advertisement is unaffected.
**This fixes two issues:**
1. **Inter-tenant connectivity via public LB IPs**: With
`kube-proxy-replacement: true`, Cilium DNATs traffic destined to LB IPs
(on ports declared in the Service) to the backend pod IP *before*
evaluating network policies. The `CiliumClusterwideNetworkPolicy` then
sees a cross-tenant pod-to-pod flow and blocks it. Ports not in the
Service pass as "world" traffic and work fine — so does ICMP. With this
label, Cilium doesn't DNAT anything, all traffic is classified as
"world", and the existing `allow-external-communication` policy allows
it. Pod IP isolation between tenants is preserved.
2. **WholeIP broken on Cilium 1.19+** (#2327): Cilium 1.19 introduced
wildcard service drop entries (`cilium/cilium#40684`) that drop all
traffic to a LB IP on ports not declared in the Service, before it
reaches netfilter/cozy-proxy. With this label, no BPF entries are
created at all, so there's nothing to drop.
**Tested on our lab cluster:**
- Cilium 1.18.6: inter-tenant via LB IP ✅, pod IP blocked ✅, external
SSH ✅, WholeIP all ports ✅
- Cilium 1.19.1: same results ✅, no wildcard drops ✅
- MetalLB L2 advertisement: unaffected ✅
- Other services without the label: unaffected ✅
**Note:** The `externalPorts` / PortList filtering is currently not
enforced at the network level (the `wholeIP` annotation causes kube-ovn
to route all ports regardless). This was the case before this change and
is a separate concern — per-VM CiliumNetworkPolicies could be
re-introduced for that (ref: `4cbf562a`, #1601).
```release-note
[virtual-machine] Fix inter-tenant connectivity via public LB IPs and WholeIP on Cilium 1.19+
```
Increase Kafka CR readiness timeout from 60s to 300s to account for
slow Strimzi startup on QEMU-based CI sandbox (4 JVM pods).
Add wait-for-delete before re-applying to prevent race condition where
kubectl apply hits a still-deleting resource on retry attempts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add service.kubernetes.io/service-proxy-name label to LoadBalancer
services when external: true. This prevents Cilium from adding the
service to its BPF service map, fixing two issues:
1. Inter-tenant connectivity via public LB IPs: Cilium's kube-proxy
replacement DNATs traffic to LB IPs before policy evaluation,
causing the CiliumClusterwideNetworkPolicy to block legitimate
cross-tenant traffic through public IPs.
2. WholeIP broken on Cilium 1.19+ (#2327): wildcard service drop
entries block all ports not declared in the Service spec before
traffic reaches cozy-proxy's nftables rules.
With this label, Cilium completely ignores the Service. Routing is
handled by kube-ovn (via the wholeIP annotation) and MetalLB
continues to advertise the IP via L2 unaffected.
Tested on Cilium 1.18.6 and 1.19.1: inter-tenant via LB IP works,
pod IP isolation preserved, external access unaffected.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Replace the cozystack-bot personal access token (GH_PAT) with a
GitHub App (cozystack-ci) installation token across all CI/CD
workflows. This improves security by using short-lived, scoped
tokens instead of a long-lived PAT tied to a user account.
Changes:
- tags.yaml: use app token in all 3 jobs (prepare-release,
generate-changelog, update-website-docs)
- auto-release.yaml: use app token for daily patch releases
- pull-requests-release.yaml: use app token for release finalization
The cozystack-ci GitHub App (ID: 3297617) is installed org-wide
with contents:write and pull-requests:write permissions. Secrets
COZYSTACK_CI_APP_ID and COZYSTACK_CI_PRIVATE_KEY are set at org
level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What this PR does
This PR adds configurable values for
`packages/system/keycloak-configure`:
- `login.userRegistration` to allow user self-registration
- `login.verifyEmail` to enforce email verification
- `smtp` which defines the SMTP configuration which will be used by
Keycloak
Fixes: aenix-org/cozyportal#365
### Release note
```release-note
["cchore(keycloak-configure): verify email and smtp config for Keycloak"]
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added configurable Keycloak login settings with user registration and
email verification controls disabled by default.
* Introduced SMTP configuration templates supporting server connection
details, TLS options, and secure credential management for email
delivery.
* Enabled conditional rendering of login and SMTP configuration sections
based on provided environment values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
CNPG operator in v1.2.0 updates default PostgreSQL image to v18
and cannot migrate from the previous major version automatically,
causing PostgreSQL clusters to fail after upgrade.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Add OpenSSF Best Practices passing badge to README alongside existing
project badges
- Badge links to https://www.bestpractices.dev/projects/10177
## Test plan
- [ ] Verify badge renders correctly on GitHub README
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added OpenSSF Best Practices badge to the project status section in
the README.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `SECURITY.md` to the repository.
The document defines:
- supported release lines
- how to report vulnerabilities without posting details publicly
- disclosure expectations
- how security fixes are communicated
It is written to match the current state of the project and avoids
claiming private channels or security tooling that are not clearly
published yet.
### Release note
```release-note
[docs] add SECURITY.md with vulnerability reporting and disclosure guidance
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added comprehensive security policy documentation outlining
vulnerability reporting procedures, responsible disclosure practices,
and response timelines for security concerns.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
- refactor backups status underlying resources structure
```
now it looks like:
```
underlyingResources:
kind: VMInstance
apiVersion: apps.cozystack.io/v1alpha1
dataVolumes:
- dataVolumeName: vm-disk-ubuntu-source
applicationName: ubuntu-source
ip: "10.244.2.98"
mac: "9e:92:c9:0c:10:9b"
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Backup status now stores underlying resources as a generic, opaque
JSON payload instead of a fixed typed structure.
* CRD and controller now preserve unknown fields in that payload,
allowing arbitrary application-specific metadata to be retained.
* Restore and controller flows were updated to consistently read and
forward this opaque payload through backup/restore operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
Introduce more accurate in-place backup/restores for VMDisk and VMInstance:
- keep IP/MAC of VM
- keep HR of VMDisk related to VMI volumes
- avoid Velero restore of DVs because it conflicts with DV from HR of VMDisk
- propagate to BackupJob/RestoreJob failure messages from Velero to be more clear for namespace users when some job fails
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Capture and persist underlying resource metadata (data volumes, OVN
IP/MAC) in Backup status and Velero annotations; use it to better scope
backups/restores.
* New Backup controller to ensure Velero cleanup for deleted backups.
* Pre-restore preparation: suspend releases, halt VMs, rename PVCs,
delete DataVolumes, and apply resource-modifier rules for claim adoption
and network annotations.
* **Bug Fixes**
* Improved Velero failure messages (includes hooks, async item ops, and
DataUpload failures).
* **Chores**
* Expanded RBAC and added RoleBinding to support Velero/restore
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR adds the changelog for release `v1.2.1`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.2.1.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes - v1.2.1
* **Bug Fixes**
* Fixed LINSTOR DRBD TCP port preservation during toggle operations
* Resolved Multus CNI initialization race conditions
* Improved Multus DEL operation reliability
* **New Features**
* Added custom Keycloak theme injection support
* **Documentation**
* Included guide for using generated Go types
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Always force-create and force-push the API submodule tag so it stays
in sync with the main version tag, even when re-tagging.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Remove broken `git push origin HEAD` in detached HEAD state (commit
artifacts are already pushed via the "Create release branch" step)
- Make subtag push idempotent: use explicit refs/tags/ prefix and
tolerate already-existing remote tags after verifying they point to
the correct commit
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Set explicit `domain.resources` with ephemeral-storage on
VirtualMachine spec in KubevirtMachineTemplate
- Limits and requests are calculated via `cozy-lib.resources.sanitize`
using the configured `ephemeralStorage` value and allocation ratio
## Problem
Without explicit ephemeral-storage resources on the VirtualMachine spec,
virt-launcher pods inherit default limits from the namespace LimitRange.
These defaults can be significantly lower than the actual emptyDisk
capacity (e.g. 2Gi limit vs 20Gi disk). When the VM's containerd unpacks
container images, it writes to emptyDir volumes on the host, quickly
exceeding the pod's total ephemeral-storage limit. This causes kubelet
to evict the pod, crashing the VM and creating a restart loop.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved virtual machine resource configuration with explicit
ephemeral storage management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Without explicit ephemeral-storage resources on the VirtualMachine spec,
virt-launcher pods inherit default limits from the namespace LimitRange,
which can be much lower than the emptyDisk capacity. This causes kubelet
to evict the pod when the VM's containerd unpacks images exceeding the
limit.
Set domain.resources with ephemeral-storage calculated via cozy-lib
allocation ratio based on the configured ephemeralStorage value.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Pin multusMasterCNI to `05-cilium.conflist` to prevent race condition
at boot
- Fixes issue where multus auto-detects wrong conflist, bypassing Cilium
CNI chain
## Problem
During node boot (e.g. Talos upgrade), multus auto-detects the master
CNI conflist by scanning
the CNI config directory. If kube-ovn writes `10-kube-ovn.conflist`
before Cilium writes
`05-cilium.conflist`, multus selects the wrong file. Pods on the
affected node then bypass
the Cilium chain entirely, have no Cilium endpoint, and their traffic
gets blocked by
cluster-wide network policies.
## Fix
Set `multusMasterCNI: "05-cilium.conflist"` in the multus daemon config
to explicitly
pin the master CNI file, eliminating the race condition.
## Upstream
- Issue: https://github.com/k8snetworkplumbingwg/multus-cni/issues/1499
- Tracking: https://github.com/cozystack/cozystack/issues/2310
## Test plan
- [x] Verified `multusMasterCNI` JSON key matches upstream struct tag
- [ ] Deploy and verify `00-multus.conf` points to `05-cilium.conflist`
after node reboot
Set multusMasterCNI to explicitly use 05-cilium.conflist instead of
auto-detecting the first available conflist at boot. This prevents a
race condition where multus picks 10-kube-ovn.conflist if it appears
before 05-cilium.conflist during node startup, bypassing the Cilium
CNI chain entirely.
Upstream issue: https://github.com/k8snetworkplumbingwg/multus-cni/issues/1499
Tracking issue: https://github.com/cozystack/cozystack/issues/2310
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Build custom multus-cni image with a patch that fixes sandbox cleanup
deadlock
- When CNI ADD never completes, DEL now returns success instead of
failing
- Prevents stale sandbox name reservations from permanently blocking pod
creation
## Problem
After a node disruption (e.g. DRBD/kube-ovn issues during upgrade),
containerd accumulates
stale sandbox name reservations. Cleanup fails because multus calls
delegate plugins for DEL
without cached state, and they reject the incomplete config. This blocks
all new pods on the
affected node indefinitely.
## Changes
- Add `images/multus-cni/` with Dockerfile and patch for custom image
build
- Add `image` target to multus Makefile
- Add multus to root Makefile build targets
- Update DaemonSet template to use the custom image
## Upstream
- PR: https://github.com/k8snetworkplumbingwg/multus-cni/pull/1498
- Tracking: https://github.com/cozystack/cozystack/issues/2310
## Test plan
- [x] Built and pushed image to ghcr.io/cozystack/cozystack/multus-cni
- [x] Deployed to cluster, verified fix in multus logs ("no cache
found... skip DEL")
- [x] Force-deleted stuck pods, all 24 pods started successfully within
70s
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Multus CNI container image build process to the system.
* **Bug Fixes**
* Fixed delete operation behavior when cache is unavailable, improving
reliability of network cleanup operations.
* **Chores**
* Updated Multus CNI container image reference in deployment
configuration to latest version with build improvements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
When CNI ADD never completes, multus DEL fails because delegate plugins
require runtime state that only exists after a successful ADD. This
creates a deadlock where containerd cannot release sandbox name
reservations, permanently blocking pod creation on the affected node.
Build a custom multus-cni image with a patch that returns success on
DEL when no cache file exists (ADD was never completed).
Upstream PR: https://github.com/k8snetworkplumbingwg/multus-cni/pull/1498
Tracking issue: https://github.com/cozystack/cozystack/issues/2310
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Set `DrbdOptions/Net/verify-alg` to `crc32c` at the controller level
to prevent DRBD connection failures on kernels where `crct10dif` is
unavailable
- Fixes an issue where all DRBD resources become Diskless after
upgrading to Talos v1.12.6 (kernel 6.18.18)
## Why
LINSTOR's auto-verify algorithm selection picks `crct10dif` by default,
but this kernel crypto module is no longer available in newer kernels.
This causes DRBD peer connections to fail with `VERIFYAlgNotAvail:
failed to allocate crct10dif for verify`, resulting in lost quorum on
all nodes.
## Test plan
- [ ] Deploy to a cluster running Talos v1.12.6 (kernel 6.18.18)
- [ ] Verify DRBD resources connect and replicate normally
- [ ] Confirm `drbdsetup` uses `crc32c` as verify-alg
Closes#2302
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Configuration**
* Added network verification algorithm configuration option for DRBD
cluster setup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR adds the changelog for release `v1.1.5`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.1.5.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Platform packages are now preserved when moved to disabled
configuration or removed from enabled packages.
* DRBD toggle-disk operations now maintain existing TCP ports across
updates, preventing connection failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
This PR ensures PostgreSQL version consistency across the platform by:
- Adding migration 37 to set version v17 for existing PostgreSQL
resources
- Hardcoding PostgreSQL 17.7 image for monitoring databases (Grafana and
Alerta)
## Motivation
CloudNativePG operator defaults to PostgreSQL 18.3 when no explicit
image is specified. However, monitoring queries are configured for
PostgreSQL 17 features (pg_stat_checkpointer, updated pg_stat_bgwriter).
This mismatch could cause issues with existing deployments.
## Changes
- **Migration 37**: Backfills spec.version="v17" for all
postgreses.apps.cozystack.io resources without a version set
- **Monitoring databases**: Explicitly set imageName:
ghcr.io/cloudnative-pg/postgresql:17.7 for:
- Grafana database
- Alerta database
## Testing
- [ ] Migration script tested on existing PostgreSQL resources
- [ ] Verified PostgreSQL 17.7 image is available
- [ ] Confirmed monitoring databases deploy with correct version
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added an automated migration to backfill PostgreSQL version fields
across existing resources.
* Explicitly set PostgreSQL image to 17.7 for monitoring and system
components: Alerta, Grafana, Harbor, Keycloak, and SeaweedFS.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add explicit PostgreSQL 17.7 image to Harbor, SeaweedFS, and Keycloak
databases to ensure consistent version across all system components.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add migration 37 to backfill spec.version=v17 for existing PostgreSQL
resources without a version set.
Hardcode PostgreSQL 17.7 image in monitoring databases (Grafana and Alerta)
to ensure compatibility with monitoring queries that expect PostgreSQL 17
features (pg_stat_checkpointer, updated pg_stat_bgwriter).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The crct10dif kernel crypto module is no longer available in Talos
v1.12.6 (kernel 6.18.18). DRBD auto-verify selects crct10dif by
default, causing peer connections to fail with VERIFYAlgNotAvail.
Setting verify-alg explicitly to crc32c at the controller level
ensures all resources use an available algorithm.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch lets Cozystack admins specify initContainers that will run
`cp -r /themes/ /opt/keycloak/themes/` on startup, effectively providing
an interface for operators to inject custom themes into the keycloak
deployment to customize the UI.
### Release note
```release-note
[keycloak] Enable injection of user-provided themes for Keycloak via
initContainers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for custom Keycloak themes through configuration,
allowing users to customize the appearance of the authentication
interface.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Adds script that tracks reconciliation of platform by checking readiness
of Packages, ArtifactGenerators, ExternalArtifacts, and HelmReleases
resources. It may be ran as singleshot or continuous check while running
upgrades or platform changes. Intended for cozystack devs and platform
admins.
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
Added check-readiness.sh script
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added new readiness check tool for Kubernetes custom resource types
including packages, artifact generators, external artifacts, and Helm
releases. Enables watch mode (`-w` flag) for continuous monitoring with
configurable refresh intervals, one-shot verification checks, colored
status indicators for clear resource health status, and support for
environment-specific configurations via KUBECONFIG.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Fixes a regression introduced in the bundle restructure (2d022e38) where
`cpuAllocationRatio`, `memoryAllocationRatio`, and
`ephemeralStorageAllocationRatio` from `platform/values.yaml` became
no-ops — never propagated to child packages.
The old monolithic bundles read these values from the cozystack
ConfigMap via `lookup` and passed them explicitly. After migration to
the Package CRD model, the bridge was lost: cozy-lib expects the ratios
in `_cluster` config (via `cozystack-values` Secret), but platform never
wrote them there, so the hardcoded defaults (10, 1, 40) were always used
regardless of what the user configured.
This commit restores the propagation by:
- Adding `cpu-allocation-ratio`, `memory-allocation-ratio`, and
`ephemeral-storage-allocation-ratio` to the `_cluster` section in the
`cozystack-values` Secret, so cozy-lib in all managed applications picks
them up
- Passing `cpuAllocationRatio` to the kubevirt Package component values,
so the KubeVirt CR gets the configured value
### Release note
```release-note
[platform] Fix propagation of resource allocation ratios (cpu, memory, ephemeral-storage) to managed applications and KubeVirt, which were silently ignored since the bundle restructure.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added new resource allocation configuration fields to platform
templates for flexible resource management.
* Enhanced conditional handling of resource settings within container
infrastructure components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Renames the misleading `subnets` field to `networks` in VMInstance. The
old field
name implied the VM was creating subnets, when in reality it attaches
the VM to
existing VPC network attachments (NetworkAttachmentDefinitions). This
change:
- Adds a new `networks` field as the primary way to attach VMs to VPC
networks
- Keeps `subnets` as deprecated with backward-compatible fallback via
Helm `default`
- Fails with a clear error if both `networks` and `subnets` are set
simultaneously
- Adds an API-backed dropdown selector for `networks[].name` listing
available
NetworkAttachmentDefinitions in the namespace
- Hides the deprecated `subnets` field from the dashboard UI
### Release note
```release-note
[vm-instance] Rename `subnets` field to `networks` in VMInstance for clarity.
The old `subnets` field is deprecated but still supported for backward compatibility.
A dropdown selector for available networks has been added to the dashboard.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added native networks field for VM attachments and a UI dropdown to
select network attachments; UI hides deprecated subnet fields for VM
instances.
* **Deprecations**
* subnets is deprecated; use networks moving forward (subnets retained
for backward compatibility).
* **Chores / Migration**
* Added a migration to copy existing subnets into networks and bumped
platform migration target version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Enables replication unconditionally in the MariaDB CR, regardless of
replica count.
Previously, single-replica MariaDB instances created a general service
(`mariadb-<name>`) without `-primary`/`-secondary` suffixes. This
caused:
- Dashboard not displaying the service (both dashboard-resourcemap RBAC
and the ApplicationDefinition expect `-primary`/`-secondary`)
- Backup CronJob referencing non-existent `<name>-secondary` service
Also removes a duplicate `template` key in the backup-cronjob YAML that
was silently ignored by the parser.
**BREAKING CHANGE:** Single-replica MariaDB instances will now expose
services as `<name>-primary` and `<name>-secondary` instead of the bare
`<name>`. Applications connecting via the old service DNS name must
update their connection strings.
### Release note
```release-note
[mariadb] BREAKING: MariaDB now always enables replication, creating -primary/-secondary services even for single-replica instances. This fixes dashboard visibility and backup functionality for single-replica setups. Existing single-replica users must update connection strings from `<name>` to `<name>-primary`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Removed a redundant backup job restart policy that could cause
conflicting behavior.
* Ensured replication is consistently enabled across all deployment
sizes.
* Made external service LoadBalancer settings apply more predictably for
external deployments.
* Adjusted database host selection so single-replica deployments target
the primary instance, improving connection correctness.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Updates the `fix-duplicate-tcp-ports` patch to preserve existing TCP
ports when DrbdRscData is recreated during toggle-disk operations.
Without this fix, `removeLayerData()` frees TCP ports from the number
pool, and `ensureStackDataExists()` may allocate different ports. If the
satellite misses the update (e.g. due to controller restart), it keeps
the old ports while peers receive the new ones, causing DRBD connections
to fail with StandAlone state.
The fix adds `copyDrbdTcpPortsIfExists()` which saves existing TCP ports
into the `LayerPayload` before `removeLayerData()` deletes them.
Also adds `dh_strip_nondeterminism` override in Dockerfile to fix build
failures on some JAR files.
Upstream:
https://github.com/LINBIT/linstor-server/pull/476#issuecomment-4147527442
### Release note
\`\`\`release-note
[linstor] Fix TCP port mismatches after toggle-disk operations that
could cause DRBD resources to enter StandAlone state
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed an issue where DRBD TCP ports were not correctly preserved
during disk toggle operations, which could result in TCP port mismatches
between the controller and satellite nodes.
* Improved robustness of the build and packaging process by addressing
non-determinism handling for Java library dependencies.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Adds annotation `helm.sh/resource-policy: keep` to all packages to
prevent its automatic deletion when:
1, Package added to `disabledPackages` parameter in
`cozystack.cozystack-platform` package
2. Package removed form `enabledPackages` parameter in
`cozystack.cozystack-platform` package
This change bring back [docs explained
behavior](https://cozystack.io/docs/v1/operations/configuration/components/)
- platform administrator should manually delete package if needed.
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[platform] Prevent installed packages deletion
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated Helm templates to configure resource-policy annotations on
package resources, ensuring proper resource lifecycle management and
retention policies during cluster deployments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add securityContext to theme init containers matching the main
container security posture. Add input validation for theme entries:
required fields, DNS-1123 name sanitization, duplicate detection,
and container name length limit. Add imagePullSecrets support for
private registries and sizeLimit on the emptyDir volume.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Update kamaji from edge-26.2.4 to 26.3.5-edge and remove two patches
that have been accepted upstream:
- `increase-startup-probe-threshold.diff` → upstream #1086 added
configurable startupProbeFailureThreshold
- `disable-datastore-check.diff` → upstream #1087 refactored
DataStore initialization
The remaining `fix-kubelet-config-compat.diff` (PR #1084) is still
needed — the maintainer has not accepted the upstream fix for kubelet
config compatibility with K8s < 1.35.
Also updates Makefile to match the new upstream tag format
(`26.x.x-edge` instead of `edge-26.x.x`) introduced in upstream
PR #1094.
Note: `make image` needs to be run to rebuild the container image
and update `values.yaml` with the new tag/digest.
Changes:
- Makefile: update tag grep pattern for new format
- Dockerfile: bump VERSION to 26.3.5-edge
- Remove 2 upstreamed patches
- Update vendored charts from 26.3.5-edge
### Release note
```release-note
[kamaji] Update to 26.3.5-edge, drop 2 upstreamed patches
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configurable probe tuning for Control Plane components (API Server,
Controller Manager, Scheduler)
* DataStore readiness visible in kubectl and status conditions tracking
* **Improvements**
* Stronger networkProfile validation (CIDR checks, DNS/IP consistency)
* DataStore driver is immutable after creation
* **Chores**
* Removed two validating webhooks
* Updated default packaged manager version used for builds
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR adds the changelog for release `v1.0.7`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.0.7.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Bug Fixes**
* Resolved RBAC permission issues enabling app creation
* Fixed Service details page access errors
* Corrected Pod serving table rendering
* Repaired navigation links in backup sidebar
* Improved alert accuracy and reduced false positives
* **Documentation**
* Added Backup and Recovery guide for virtualization
* Updated developer documentation with architecture details
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
After migration 36 copies subnets to networks, both fields are populated.
The fail guard would break Helm reconciliation. Instead, networks simply
takes priority via the existing default fallback.
Also handle missing VMInstance CRD gracefully in migration 36.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Existing VMInstance resources store network attachments in the deprecated
spec.subnets field. This migration copies subnets to the new spec.networks
field so the dashboard UI correctly displays attached networks. The old
subnets field is kept intact because migrations run before the new CRD is
applied.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Add fail validation when both 'networks' and 'subnets' are set to
prevent silent data loss during migration. Compute $networks variable
once at template top level to eliminate DRY violation. Regenerate CRD
via make generate instead of manual editing.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Rename the misleading 'subnets' field to 'networks' in VMInstance to
better reflect that it attaches the VM to VPC network attachments, not
creates subnets. The old 'subnets' field is kept as deprecated with
fallback logic for backward compatibility.
Add an API-backed dropdown (listInput) for networks[].name that lists
available NetworkAttachmentDefinitions in the namespace, and hide the
deprecated subnets field from the dashboard UI.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Since the bundle restructure in 2d022e38, the cpuAllocationRatio,
memoryAllocationRatio and ephemeralStorageAllocationRatio values from
platform values.yaml were no longer propagated to child packages.
The old monolithic bundles read these values from the cozystack
ConfigMap via lookup and passed them explicitly. After migration to
the Package CRD model, the bridge was lost: cozy-lib expects the
ratios in _cluster config (via cozystack-values Secret), but platform
never wrote them there, so the hardcoded defaults were always used.
This commit restores the propagation by:
- Adding cpu/memory/ephemeral-storage allocation ratios to the
_cluster section in the cozystack-values Secret, so cozy-lib in
all managed applications picks them up.
- Passing cpuAllocationRatio to the kubevirt Package component, so
the KubeVirt CR gets the configured value.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Update fix-duplicate-tcp-ports patch to preserve existing TCP ports when
DrbdRscData is recreated during toggle-disk operations. Without this,
removeLayerData() frees ports and ensureStackDataExists() may allocate
different ones, causing port mismatches between controller and satellites
if the satellite misses the update.
Also add dh_strip_nondeterminism override in Dockerfile to fix build
failures on some JAR files.
Upstream: https://github.com/LINBIT/linstor-server/pull/476#issuecomment-4147527442
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This PR adds the changelog for release `v1.2.0`.
✅ Changelog has been created in `docs/changelogs/v1.2.0.md` based on the
complete commit history between v1.1.0 and v1.2.0, including all bug
fixes merged up to the final v1.2.0 tag.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added v1.2.0 release notes documenting user-facing platform updates: a
fully managed OpenSearch PaaS with secure multi-role topology and
optional dashboards; tenant-level scheduling controls; VPC peering for
multi-tenant networking; clustered logging for improved durability and
scaling; storage improvements for automatic replica relocation after
clones/restores; monitoring and Grafana enhancements; dashboard/RBAC
fixes and various platform bug fixes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
When replicas=1, the secondary service has no endpoints because the
only pod is the primary. Route backups to the primary service in this
case to ensure they work correctly.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Enable replication unconditionally regardless of replica count.
Previously, single-replica instances created a general service
without -primary/-secondary suffixes, causing dashboard and
backup-cronjob to reference non-existent services.
Also fix duplicate YAML key in backup-cronjob template.
BREAKING CHANGE: Single-replica MariaDB instances will now have
service names with -primary/-secondary suffixes instead of the
bare instance name. Applications connecting via the old service
DNS name need to be updated.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Use -F/ with $NF (last field) instead of -F'[/^]' with $3.
The ^{} separator is unnecessary since the grep pattern already
filters out annotated tag dereferences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Update kamaji from edge-26.2.4 to 26.3.5-edge and remove patches
that have been accepted upstream:
- increase-startup-probe-threshold.diff: upstream PR #1086 added
configurable startupProbeFailureThreshold field
- disable-datastore-check.diff: upstream PR #1087 refactored
DataStore initialization, removing the blocking startup check
The remaining fix-kubelet-config-compat.diff patch (PR #1084) is
still needed as the maintainer has not accepted the upstream fix
for kubelet config compatibility with K8s < 1.35.
Also update Makefile to match the new upstream tag format
(26.x.x-edge instead of edge-26.x.x) introduced in PR #1094.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
This patch lets Cozystack admins specify initContainers that will run
`cp -r /themes/ /opt/keycloak/themes/` on startup, effectively providing
an interface for operators to inject custom themes into the keycloak
deployment to customize the UI.
```release-note
[keycloak] Enable injection of user-provided themes for Keycloak via
initContainers.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Cozystack is a Kubernetes-based PaaS built on Helm umbrella charts and FluxCD.
Packages live in `packages/{core,system,apps,extra,library,tests}/`. The `library/` group holds reusable helper charts; the `tests/` group exists because library charts are not directly testable.
Each package wraps one or more upstream Helm charts.
Go code in `cmd/`, `internal/`, `pkg/` implements Kubernetes controllers and API server.
Static CRDs are generated by `hack/update-codegen.sh` from types under `api/v1alpha1/`, `api/backups/`, and `api/dashboard/`.
App Kinds (like `Postgres`, `Kafka`) live in `api/apps/v1alpha1/` but are registered dynamically at runtime from `ApplicationDefinition` resources — they are not static CRDs.
## Vendored Code — Critical Rules
**Never suggest editing files inside any `charts/` directory under `packages/`.**
Those are upstream Helm charts vendored via `make update` (which runs `helm pull`).
Any direct edit is overwritten on the next update and provides zero value.
If you find an issue that appears to live in vendored chart code:
- For configuration-level changes: suggest overrides in the package root `values.yaml`.
- For structural changes: suggest a patch file in `packages/<name>/patches/` applied by the Makefile.
- For source-code changes in images: suggest a patch in `packages/<name>/images/<name>/patches/`.
- For true upstream bugs: point to the upstream repository and suggest an upstream issue/PR.
- Do NOT suggest creating `charts/patches/` — patches never live inside `charts/`.
Similarly, never propose edits to:
- `vendor/` — Go dependencies. Changes go through `go get` and `go mod tidy`.
- `zz_generated.*.go` — regenerated by `make generate`.
- Package-specific: any `<package-name>` matching a directory under `packages/`
Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer.
Each commit must have a `Signed-off-by:` trailer (produced by `git commit --signoff`).
PR body must contain a release note block:
````text
```release-note
type(scope): human-readable changelog entry
```
````
Flag any PR whose commits lack the Conventional Commits format or signoff, or whose body has no release-note block.
## Helm Chart Conventions
Packages follow an umbrella chart pattern:
- `charts/` — vendored upstream, read-only
- `templates/` — Cozystack-specific extra manifests
- `values.yaml` — override values for the upstream chart
- `values.schema.json` — JSON Schema for dashboard UI and input validation
When reviewing `values.schema.json`:
- `make generate` regenerates this file and strips `title`, `description`, and `x-*` custom annotations. Do not suggest adding fields that will be stripped on the next regeneration.
- Focus on type correctness, `required` fields, `enum` values, and default values.
- Flag breaking changes: removing a field, changing its type, or narrowing its enum.
## Sensitive Components
**`packages/core/platform/`**: the platform chart that deploys everything. Changes here can require updates to the migration flow — the Helm hook in `packages/core/platform/templates/migration-hook.yaml` and the runner image with scripts in `packages/core/platform/images/migrations/`. Flag any change to this package that lacks a corresponding migration update or an explicit note that backward compatibility is preserved.
**`api/apps/v1alpha1/`**: app CRD Kinds are registered at runtime from `ApplicationDefinition` resources and the matching `values.schema.json`. Suggest changes to the relevant package schema rather than hand-editing generated types.
**RBAC, ServiceAccounts, and SecurityContext**: flag overly broad RBAC (`*` on resources or verbs without justification), missing `securityContext`, containers running as root without an explicit reason, and `hostPath`/`hostNetwork` usage without clear rationale.
## Go Code Standards
- Use controller-runtime patterns for reconcilers.
- Use structured logging via `logr` — flag `fmt.Print*` and `log.Print*` in controller code.
- Handle errors explicitly. Discarding meaningful errors with `_` is a bug.
- Propagate `context.Context` through call chains. Flag `context.Background()` created inside a reconciler or request handler.
- Prefer `ctrl.Result{RequeueAfter: ...}` over empty requeue for predictable reconciliation loops.
- Tests live beside the code (`*_test.go`). New behavior without tests is worth flagging.
copilot --prompt "prepare changelog file for tagged release v${{ steps.tag.outputs.version }}, use @docs/agents/changelog.md for it. Create the changelog file at docs/changelogs/v${{ steps.tag.outputs.version }}.md" \
copilot \
--prompt "Generate the release changelog for tag v${VERSION}. Follow the instructions in @docs/agents/changelog.md exactly, including the 'Scope and boundaries' section at the top. Your deliverable is the single file docs/changelogs/v${VERSION}.md — write it and exit; this workflow handles branching, committing, pushing, and opening the PR." \
- Action: Use labels defined there. Conventions follow the Kubernetes scheme — `kind/*` (type), `area/*` (subsystem), `priority/*` (urgency), `triage/*` (review state), `lifecycle/*` (auto-close), `do-not-merge/*` (PR blockers), `security/*` (severity)
- For `area/*`: accuracy outweighs reuse. If no existing `area/*` truly fits the change, propose a new one via PR (extend `.github/labels.yml` and the scope mapping in `.github/workflows/pr-labeler.yaml`) — do not shoehorn the change into a wrong area. `area/uncategorized` is the auto-labeler fallback; treat it as a signal to pick a fit, create a new area, or correct the PR title
- PR titles: a Conventional Commits header (`type(scope): description`, types from [`contributing.md`](./docs/agents/contributing.md)) auto-applies `kind/*` and `area/*` via `.github/workflows/pr-labeler.yaml`. Append `!` (or add a `BREAKING CHANGE:` footer) to apply `kind/breaking-change`
**Important rules:**
- ✅ **ONLY read the file if the task matches the documented process scope** - do not read files for tasks that don't match their purpose
- ✅ **ALWAYS read the file FIRST** before starting the task (when applicable)
@ -53,7 +59,7 @@ working with the **Cozystack** project.
### Conventions
- **Helm Charts**: Umbrella pattern, vendored upstream charts in `charts/`
[](https://www.bestpractices.dev/projects/10177)
# Cozystack
**Cozystack** is a free PaaS platform and framework for building clouds.
**Cozystack** is a free platform and framework for building clouds.
Cozystack is a [CNCF Sandbox Level Project](https://www.cncf.io/sandbox-projects/) that was originally built and sponsored by [Ænix](https://aenix.io/).
This policy applies to the [`cozystack/cozystack`](https://github.com/cozystack/cozystack) repository and to release artifacts produced from it, including Cozystack core components, operators, packaged manifests, container images, and installation assets published by the project.
Cozystack integrates and ships many upstream cloud native components. If you believe a vulnerability originates in an upstream project rather than in Cozystack-specific code, packaging, defaults, or integration logic, please report it to the upstream project as well. If you are unsure, report it to Cozystack first and we will help route or coordinate the issue.
## Supported Versions
As of March 17, 2026, the Cozystack project maintains multiple release lines. Security fixes are prioritized for the latest stable release line and, when needed, backported to other supported lines.
| Version line | Status | Notes |
| --- | --- | --- |
| `v1.1.x` | Supported | Current stable release line. |
| `v1.0.x` | Supported | Previous stable release line; receives security and important maintenance fixes. |
| `v0.41.x` | Limited support | Legacy pre-v1 line during the v0 to v1 transition; critical security and upgrade-blocking fixes may be backported at maintainer discretion. |
| `< v0.41` | Not supported | Please upgrade to a supported release line before requesting a security fix. |
| `alpha`, `beta`, `rc` releases | Not supported | Pre-release builds are for testing and evaluation only. |
Supported versions may change over time as new release lines are cut. The authoritative source for current releases is the GitHub Releases page:
<https://github.com/cozystack/cozystack/releases>
## Reporting a Vulnerability
Please do **not** report security vulnerabilities through public GitHub issues, discussions, pull requests, Telegram, Slack, or other public community channels.
At the moment, this repository does not publish a dedicated private security mailbox in-tree. If you need to report a vulnerability:
1. Contact one of the project maintainers listed in `CODEOWNERS` using an existing private channel you already have.
2. If you do not already have a private maintainer contact, use a public community channel only to request a private contact path, without disclosing any vulnerability details.
Please do not include exploit details, credentials, tokens, private keys, customer data, or other sensitive material in any public message.
When reporting a vulnerability, please include as much of the following as possible:
- affected Cozystack version, tag, or commit
- affected component or package, for example operator, API server, dashboard, installer, or a packaged system component
- deployment environment and provider, for example bare metal, Hetzner, Oracle Cloud, or other infrastructure
- prerequisites and exact reproduction steps
- impact, attack scenario, and expected blast radius
- whether authentication, tenant access, cluster-admin access, or network adjacency is required
- known mitigations or workarounds
- whether you believe the issue also affects an upstream dependency
## What to Expect
The maintainers will aim to:
- acknowledge receipt within 3 business days
- perform an initial triage and severity assessment within 7 business days
- keep the reporter informed as the fix and disclosure plan are developed
Resolution timelines depend on severity, complexity, release branch applicability, and whether coordination with upstream projects is required.
## Disclosure Process
The Cozystack project follows a coordinated disclosure model.
- We ask reporters to keep details private until a fix or mitigation is available and users have had a reasonable opportunity to upgrade.
- When appropriate, maintainers may use GitHub Security Advisories or equivalent coordinated disclosure tooling to manage remediation and public disclosure.
- If appropriate, the project may request or publish a GHSA and/or CVE as part of the disclosure process.
- Fixes will normally be released in the supported version lines affected by the issue, subject to severity and feasibility.
Public disclosure will typically happen through one or more of the following:
- GitHub Releases and release notes
- project changelogs and documentation updates
- GitHub Security Advisories, when used for coordinated disclosure
## Project Security Practices
Security is part of the normal Cozystack development and release process. Current project practices include:
- maintainer-owned review through pull requests and `CODEOWNERS`
- automated pull request checks, including pre-commit validation, unit tests, builds, and end-to-end testing
- release automation with patch releases, release branches, and backport workflows
- ongoing maintenance of packaged dependencies and platform integrations across supported release lines
Because Cozystack is an integration-heavy platform, some vulnerabilities may require coordination across multiple repositories or with upstream maintainers before a public fix can be released.
## Security Fixes and Announcements
Security fixes are published in normal release artifacts whenever possible. Users should monitor:
- the Cozystack website and documentation: <https://cozystack.io>
## Out of Scope
The following are generally out of scope for private security reporting unless there is a clear Cozystack-specific impact:
- vulnerabilities in unsupported or end-of-life Cozystack versions
- issues that require access already equivalent to cluster-admin, node root, or direct infrastructure administrator privileges, unless they bypass an expected Cozystack security boundary
- vulnerabilities that exist only in an upstream dependency and are not introduced or materially worsened by Cozystack packaging, configuration, or defaults
- requests for security best-practice advice without a concrete vulnerability
## Credits
We appreciate responsible disclosure and will credit reporters in public advisories or release notes unless anonymous disclosure is requested.
// Timestamp (RFC3339) for point-in-time recovery; empty means latest.
// +kubebuilder:default:=""
RecoveryTimestring`json:"recoveryTime,omitempty"`
// Barman server name (S3 path prefix) used by the original cluster when writing backups. Set this only when the original cluster had an explicit barmanObjectStore.serverName that differed from its Kubernetes resource name.
// Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect.
// +kubebuilder:default:=true
ExternalAllowICMPbool`json:"externalAllowICMP"`
// Requested running state of the VirtualMachineInstance
// +kubebuilder:default:="Always"
RunStrategyRunStrategy`json:"runStrategy"`
@ -38,9 +41,12 @@ type ConfigSpec struct {
// List of disks to attach.
// +kubebuilder:default:={}
Disks[]Disk`json:"disks,omitempty"`
// Additional subnets
// Networks to attach the VM to.
// +kubebuilder:default:={}
Subnets[]Subnet`json:"subnets,omitempty"`
Networks[]Network`json:"networks,omitempty"`
// Deprecated: use networks instead.
// +kubebuilder:default:={}
Subnets[]Network`json:"subnets,omitempty"`
// List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
"description":"Tensor saturation, util/watt and throttling — reveals inefficient GPU workloads",
"tags":[
"gpu",
"efficiency",
"finops"
],
"timezone":"browser",
"editable":true,
"graphTooltip":1,
"time":{
"from":"now-1h",
"to":"now"
},
"fiscalYearStartMonth":0,
"schemaVersion":42,
"panels":[
{
"type":"row",
"collapsed":false,
"title":"Overall efficiency metrics",
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":0
},
"id":1,
"panels":[]
},
{
"type":"stat",
"id":2,
"targets":[
{
"expr":"avg(gpu:tensor_saturation:avg5m) * 100",
"refId":"A"
}
],
"title":"Avg Tensor Saturation",
"description":"Mean tensor core saturation across all GPUs. \u003c10% means GPUs are used inefficiently (workloads could move to CPU or optimize their code). Cluster-wide — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":5,
"w":8,
"x":0,
"y":1
},
"repeatDirection":"h",
"options":{
"graphMode":"area",
"colorMode":"background",
"justifyMode":"auto",
"textMode":"value",
"wideLayout":true,
"showPercentChange":false,
"reduceOptions":{
"calcs":[
"lastNotNull"
]
},
"percentChangeColorMode":"standard",
"orientation":""
},
"fieldConfig":{
"defaults":{
"unit":"percent",
"min":0,
"max":100,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"red"
},
{
"value":10,
"color":"orange"
},
{
"value":30,
"color":"yellow"
},
{
"value":60,
"color":"green"
}
]
}
},
"overrides":[]
}
},
{
"type":"stat",
"id":3,
"targets":[
{
"expr":"avg(gpu:util_per_watt:avg5m)",
"refId":"A"
}
],
"title":"Avg Utilization per Watt",
"description":"NVML utilization % per watt across all GPUs. Higher value = more efficient workload. Cluster-wide — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":5,
"w":8,
"x":8,
"y":1
},
"repeatDirection":"h",
"options":{
"graphMode":"area",
"colorMode":"background",
"justifyMode":"auto",
"textMode":"value",
"wideLayout":true,
"showPercentChange":false,
"reduceOptions":{
"calcs":[
"lastNotNull"
]
},
"percentChangeColorMode":"standard",
"orientation":""
},
"fieldConfig":{
"defaults":{
"unit":"none",
"decimals":3,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"red"
},
{
"value":0.5,
"color":"yellow"
},
{
"value":1,
"color":"green"
}
]
}
},
"overrides":[]
}
},
{
"type":"stat",
"id":4,
"targets":[
{
"expr":"avg(gpu:power_throttle_fraction:rate5m)",
"refId":"A"
}
],
"title":"Avg Power Throttling",
"description":"Fraction of time GPUs hit the TDP cap and lose performance. \u003e5% means tenants underutilize billed FLOPS. Cluster-wide — rule aggregates by (Hostname, gpu, UUID) and drops namespace label.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":5,
"w":8,
"x":16,
"y":1
},
"repeatDirection":"h",
"options":{
"graphMode":"area",
"colorMode":"background",
"justifyMode":"auto",
"textMode":"value",
"wideLayout":true,
"showPercentChange":false,
"reduceOptions":{
"calcs":[
"lastNotNull"
]
},
"percentChangeColorMode":"standard",
"orientation":""
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"decimals":2,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"green"
},
{
"value":0.05,
"color":"yellow"
},
{
"value":0.2,
"color":"red"
}
]
}
},
"overrides":[]
}
},
{
"type":"row",
"collapsed":false,
"title":"NVML vs Tensor (mismatch detector)",
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":6
},
"id":10,
"panels":[]
},
{
"type":"timeseries",
"id":11,
"targets":[
{
"expr":"DCGM_FI_DEV_GPU_UTIL",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"NVML GPU Utilization",
"description":"Classic utilization metric. Shows activity of any engine (SM, copy, encoder). Cluster-wide — DCGM namespace is the exporter's own namespace, not the workload namespace.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":7
},
"repeatDirection":"h",
"options":{
"legend":{
"displayMode":"table",
"placement":"bottom",
"showLegend":false,
"calcs":[
"mean",
"max"
]
},
"tooltip":{
"mode":"multi",
"sort":""
}
},
"fieldConfig":{
"defaults":{
"unit":"percent",
"min":0,
"max":100,
"custom":{
"drawStyle":"line",
"lineInterpolation":"linear",
"fillOpacity":10,
"showPoints":"never"
}
},
"overrides":[]
}
},
{
"type":"timeseries",
"id":12,
"targets":[
{
"expr":"DCGM_FI_PROF_PIPE_TENSOR_ACTIVE * 100",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"Tensor Pipe Active",
"description":"Real tensor core load (HMMA). For AI/LLM inference it should be ≥20%, otherwise the workload is unoptimized. Cluster-wide — DCGM namespace is the exporter's own namespace, not the workload namespace.",
"description":"Which GPUs are exercising tensor cores and which are not. Sorted descending. Grouped by Hostname/gpu/UUID — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":16
},
"repeatDirection":"h",
"transformations":[
{
"id":"organize",
"options":{
"excludeByName":{
"DCGM_FI_DRIVER_VERSION":true,
"Time":true,
"__name__":true,
"cluster":true,
"container":true,
"device":true,
"endpoint":true,
"gpu_driver_version":true,
"instance":true,
"job":true,
"modelName":true,
"namespace":true,
"pci_bus_id":true,
"pod":true,
"prometheus":true,
"service":true,
"tenant":true,
"tier":true,
"uid":true,
"unit":true
},
"indexByName":{
"Hostname":0,
"UUID":2,
"Value":3,
"gpu":1
},
"renameByName":{
"Hostname":"Node",
"UUID":"UUID",
"Value":"Saturation",
"gpu":"GPU"
}
}
}
],
"options":{
"frameIndex":0,
"showHeader":true,
"showTypeIcons":false,
"sortBy":[
{
"displayName":"Saturation",
"desc":true
}
],
"footer":{
"show":false,
"reducer":[]
},
"cellHeight":"sm"
},
"fieldConfig":{
"defaults":{},
"overrides":[
{
"matcher":{
"id":"byName",
"options":"Saturation"
},
"properties":[
{
"id":"unit",
"value":"percent"
},
{
"id":"min",
"value":0
},
{
"id":"max",
"value":100
},
{
"id":"custom.cellOptions",
"value":{
"mode":"gradient",
"type":"gauge"
}
},
{
"id":"thresholds",
"value":{
"mode":"absolute",
"steps":[
{
"color":"red"
},
{
"color":"orange",
"value":10
},
{
"color":"yellow",
"value":30
},
{
"color":"green",
"value":60
}
]
}
}
]
}
]
}
},
{
"type":"table",
"id":22,
"targets":[
{
"expr":"topk(20, gpu:util_per_watt:avg5m)",
"instant":true,
"range":false,
"format":"table",
"refId":"A"
}
],
"title":"Utilization / Watt per GPU (5m avg)",
"description":"How efficiently each GPU spends watts. Low value = poor optimization. Grouped by Hostname/gpu/UUID — DCGM metrics cannot be attributed to workload namespaces.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":12,
"y":16
},
"repeatDirection":"h",
"transformations":[
{
"id":"organize",
"options":{
"excludeByName":{
"DCGM_FI_DRIVER_VERSION":true,
"Time":true,
"__name__":true,
"cluster":true,
"container":true,
"device":true,
"endpoint":true,
"gpu_driver_version":true,
"instance":true,
"job":true,
"modelName":true,
"namespace":true,
"pci_bus_id":true,
"pod":true,
"prometheus":true,
"service":true,
"tenant":true,
"tier":true,
"uid":true,
"unit":true
},
"indexByName":{
"Hostname":0,
"UUID":2,
"Value":3,
"gpu":1
},
"renameByName":{
"Hostname":"Node",
"UUID":"UUID",
"Value":"Util/Watt",
"gpu":"GPU"
}
}
}
],
"options":{
"frameIndex":0,
"showHeader":true,
"showTypeIcons":false,
"sortBy":[
{
"displayName":"Util/Watt",
"desc":true
}
],
"footer":{
"show":false,
"reducer":[]
},
"cellHeight":"sm"
},
"fieldConfig":{
"defaults":{},
"overrides":[
{
"matcher":{
"id":"byName",
"options":"Util/Watt"
},
"properties":[
{
"id":"unit",
"value":"none"
},
{
"id":"decimals",
"value":3
},
{
"id":"min",
"value":0
},
{
"id":"max",
"value":1.5
},
{
"id":"custom.cellOptions",
"value":{
"mode":"gradient",
"type":"gauge"
}
},
{
"id":"thresholds",
"value":{
"mode":"absolute",
"steps":[
{
"color":"red"
},
{
"color":"yellow",
"value":0.5
},
{
"color":"green",
"value":1
}
]
}
}
]
}
]
}
},
{
"type":"row",
"collapsed":false,
"title":"Throttling",
"gridPos":{
"h":1,
"w":24,
"x":0,
"y":24
},
"id":30,
"panels":[]
},
{
"type":"timeseries",
"id":31,
"targets":[
{
"expr":"gpu:power_throttle_fraction:rate5m",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"Power throttle fraction per GPU",
"description":"Fraction of time the GPU was power-throttled. 1.0 = always throttled. Cluster-wide — rule aggregates by (Hostname, gpu, UUID) and drops namespace label.",
"transparent":false,
"datasource":{
"type":"prometheus",
"uid":"$ds_prometheus"
},
"gridPos":{
"h":8,
"w":12,
"x":0,
"y":25
},
"repeatDirection":"h",
"options":{
"legend":{
"displayMode":"table",
"placement":"bottom",
"showLegend":false,
"calcs":[
"mean",
"max"
]
},
"tooltip":{
"mode":"multi",
"sort":""
}
},
"fieldConfig":{
"defaults":{
"unit":"percentunit",
"min":0,
"max":1,
"thresholds":{
"mode":"absolute",
"steps":[
{
"value":null,
"color":"green"
},
{
"value":0.05,
"color":"yellow"
},
{
"value":0.2,
"color":"red"
}
]
},
"custom":{
"drawStyle":"line",
"lineInterpolation":"linear",
"fillOpacity":25,
"showPoints":"never"
}
},
"overrides":[]
}
},
{
"type":"timeseries",
"id":32,
"targets":[
{
"expr":"gpu:thermal_throttle_fraction:rate5m",
"legendFormat":"{{Hostname}}/{{gpu}}",
"refId":"A"
}
],
"title":"Thermal throttle fraction per GPU",
"description":"Fraction of time the GPU was thermal-throttled. Cluster-wide — rule aggregates by (Hostname, gpu, UUID) and drops namespace label.",
@ -6,6 +6,20 @@ This file contains detailed instructions for AI-powered IDE on how to generate c
Follow these instructions when the user explicitly asks to generate a changelog.
## Scope and boundaries
**Your single deliverable is the file `docs/changelogs/v<version>.md`.** Write the complete, verified changelog to that path. That is the entire task. Exit as soon as the file is written and verified against the checklist in Step 9.
Unless the caller explicitly instructs otherwise:
- **In the cozystack working tree**, do not run `git commit`, `git push`, `git checkout` (to switch branches), `git branch`, `git tag`, `git reset`, `git merge`, or `git rebase`. Do not write to local branches, tags, or HEAD. `git fetch` is expected and fine (see the read-only analysis list below).
- **Do not** push to any remote, open pull requests, or issue GitHub API write calls (POST / PATCH / DELETE) for any repository.
- In the cozystack working tree, the **only** file you create or modify is `docs/changelogs/v<version>.md`. Cloning auxiliary repositories under `_repos/` for cross-repo analysis (see Step 6) is fine; local git operations inside those disposable clones (`git checkout`, `git pull`, etc.) are allowed — just never push from them or open PRs against them.
The caller — a GitHub Actions workflow in CI, or a developer running you interactively — owns branching, committing, pushing, and PR creation. They will perform those actions after you exit. Do not pre-empt them even if the working tree looks ready.
Read-only analysis is expected and encouraged: `git log`, `git show`, `git fetch`, `git diff`, `gh pr view`, `gh api` GET requests, and reading any file in the repository.
## Required Tools
Before generating changelogs, ensure you have access to `gh` (GitHub CLI) tool, which is used to fetch commit and PR author information. The GitHub CLI is used to correctly identify PR authors from commits and pull requests.
@ -22,7 +36,7 @@ When the user asks to generate a changelog, follow these steps in the specified
- [ ] Step 5: Get the list of commits for the release period
- [ ] Step 6: Check additional repositories (website is REQUIRED, optional repos if tags exist)
- [ ] **MANDATORY**: Check website repository for documentation changes WITH authors and PR links via GitHub CLI
- [ ] **MANDATORY**: Check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy) for tags during release period
- [ ] **MANDATORY**: Check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy, external-apps-example, ansible-cozystack) for tags during release period
- [ ] **MANDATORY**: For ALL commits from additional repos, get GitHub username via CLI, prioritizing PR author over commit author.
**⚠️ IMPORTANT**: You MUST check ALL optional repositories for tags created during the release period. Do NOT skip this step even if you think there might not be any tags. Use the process below to verify.
@ -195,7 +211,7 @@ Cozystack release may include changes from related repositories. Check and inclu
3. **For optional repositories, check if tags exist during release period:**
**⚠️ MANDATORY: You MUST check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy). Do NOT skip any repository!**
**⚠️ MANDATORY: You MUST check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy, external-apps-example, ansible-cozystack). Do NOT skip any repository!**
**Use the helper script:**
```bash
@ -208,7 +224,7 @@ Cozystack release may include changes from related repositories. Check and inclu
```
The script will:
- Check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy)
- Get commits between tags (if tags exist) or by date range (if no tags)
- Extract PR numbers from commit messages
@ -569,7 +585,7 @@ Create a new changelog file in the format matching previous versions:
- [ ] Step 5 completed: **ALL commits included** (including merge commits and backports) - do not skip any commits
- [ ] Step 5 completed: **Backports identified and handled correctly** - original PR author used, both original and backport PR numbers included
- [ ] Step 6 completed: Website repository checked for documentation changes WITH authors and PR links via GitHub CLI
- [ ] Step 6 completed: **ALL** optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy) checked for tags during release period
- [ ] Step 6 completed: **ALL** optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy, external-apps-example, ansible-cozystack) checked for tags during release period
- [ ] Step 6 completed: For ALL commits from additional repos, GitHub username obtained via GitHub CLI (not skipped). For commits with PR numbers, PR author used via `gh pr view` (not commit author)
- [ ] Step 7 completed: For EVERY PR in main repo (including backports), PR author obtained via `gh pr view <PR_NUMBER> --json author --jq .author.login` (not skipped or assumed). Commit author NOT used - always use PR author
- [ ] Step 7 completed: **Backports verified** - for each backport PR, original PR found and original PR author used in changelog
@ -606,6 +622,8 @@ Create a new changelog file in the format matching previous versions:
**Save the changelog:**
Save the changelog to file `docs/changelogs/v<version>.md` according to the version for which the changelog is being generated.
**Then exit.** Do not commit, push, create a branch, or open a pull request — the caller handles all git and GitHub operations after you return. See the "Scope and boundaries" section at the top of this document.
### Important notes
- **After fetch with --force** local tags are up-to-date, use them for work
@ -628,7 +646,7 @@ Save the changelog to file `docs/changelogs/v<version>.md` according to the vers
- **⚠️ CRITICAL**: Always check the **website** repository for documentation changes during the release period. This is a required step and MUST NOT be skipped.
- **⚠️ CRITICAL**: You MUST check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy) for tags during the release period. Do NOT skip any repository even if you think there might not be tags.
- **⚠️ CRITICAL**: You MUST check ALL optional repositories (talm, boot-to-talos, cozyhr, cozy-proxy, external-apps-example, ansible-cozystack) for tags during the release period. Do NOT skip any repository even if you think there might not be tags.
- **CRITICAL**: For ALL entries from additional repositories (website and optional), you MUST:
- **MANDATORY**: Extract PR number from commit message first
- **MANDATORY**: For commits with PR numbers, ALWAYS use `gh pr view <PR_NUMBER> --repo cozystack/<repo> --json author --jq .author.login` to get PR author (not commit author)
@ -637,7 +655,7 @@ Save the changelog to file `docs/changelogs/v<version>.md` according to the vers
- **MANDATORY**: Do NOT use commit author for PRs - always use PR author
- Include PR link or commit hash reference
- Format: `* **[repo] Description**: details ([**@username**](https://github.com/username) in cozystack/repo#123)`
- For **optional repositories** (talm, boot-to-talos, cozyhr, cozy-proxy), you MUST check ALL of them for tags during the release period. Use the loop provided in Step 6 to check each repository systematically.
- For **optional repositories** (talm, boot-to-talos, cozyhr, cozy-proxy, external-apps-example, ansible-cozystack), you MUST check ALL of them for tags during the release period. Use the loop provided in Step 6 to check each repository systematically.
- When including changes from additional repositories, use the format: `[repo-name] Description` and link to the repository's PR/issue if available
- **Prefer PR numbers over commit hashes**: For commits from additional repositories, extract PR number from commit message using GitHub API. Use PR format (`cozystack/website#123`) instead of commit hash (`cozystack/website@abc1234`) when available
- **Never add entries without author and PR/commit reference**: Every entry from additional repositories must have both author and link
- [ ] Commit message follows Conventional Commits format
- [ ] Commit is signed off with `--signoff`
- [ ] Branch is rebased on `upstream/main` (no extra commits)
- [ ] PR body includes description and release note
- [ ] PR is pushed and created with `gh pr create`
- [ ] Ran `make generate` in every package whose `values.yaml`, `values.schema.json`, `Chart.yaml`, or `README.md` was touched, and committed the regenerated files
## How to Commit and Create Pull Requests
## Regenerate Artifacts Before Committing
### 1. Make Your Changes
Several files in each package are produced by `make generate` from `values.yaml` + `values.schema.json` and must stay in sync with the hand-edited sources:
Edit the necessary files in the codebase.
- `packages/(apps|extra)/<name>/README.md` — regenerated by `cozyvalues-gen` (parameter table, formatting).
- `packages/(apps|extra)/<name>/values.schema.json` — `cozyvalues-gen` rewrites ordering and derived fields.
- `packages/system/<name>-rd/cozyrds/<name>.yaml` — produced by `hack/update-crd.sh`, which `make generate` invokes.
### 2. Commit with Proper Format
Use the `[component]` prefix and `--signoff` flag:
**Before committing edits to any of those sources**, run `make generate` inside the package and stage the full diff:
```bash
git commit --signoff -m "[component] Brief description of changes"
The repo's pre-commit CI job runs `make generate` in every package and then `git diff --exit-code`. Any unstaged generator output fails the job with exit code 123 and blocks the PR. Also rerun `make generate` after a `git commit --amend` if the amended change touched any of the sources above.
To locate packages a WIP branch likely needs to be regenerated:
**Scopes** (examples — not an exhaustive list; pick the most specific scope that describes the change, and introduce a new one if a genuinely new area needs its own):
`.github/workflows/pr-labeler.yaml` parses the PR title on `opened`, `edited`, `reopened`, and `synchronize` events and applies labels additively (never removes). The title is expected to follow Conventional Commits — same format as commit messages above.
When the user asks to fix comments from AI bot reviewers (like Qodo, Copilot, etc.):
### 1. Get PR Comments
View all comments on the pull request:
```bash
gh pr view <PR-number> --comments
```
Or for the current branch:
```bash
gh pr view --comments
```
### 2. Review Each Comment Carefully
**Important**: Do NOT blindly apply all suggestions. Each comment should be evaluated:
- **Consider context** - Does the suggestion make sense for this specific case?
- **Check project conventions** - Does it align with Cozystack patterns?
- **Evaluate impact** - Will this improve code quality or introduce issues?
- **Question validity** - AI bots can be wrong or miss context
**When to apply:**
- ✅ Legitimate bugs or security issues
- ✅ Clear improvements to code quality
- ✅ Better error handling or edge cases
- ✅ Conformance to project conventions
**When to skip:**
- ❌ Stylistic preferences that don't match project style
- ❌ Over-engineering simple code
- ❌ Changes that break existing patterns
- ❌ Suggestions that show misunderstanding of the code
### 3. Apply Valid Fixes
Make changes addressing the valid comments. Use your judgment.
### 4. Leave Changes Uncommitted
**Critical**: Do NOT commit or push the changes automatically.
Leave the changes in the working directory so the user can:
- Review the fixes
- Decide whether to commit them
- Make additional adjustments if needed
```bash
# After making changes, show status but DON'T commit
git status
git diff
```
The user will commit and push when ready.
## Code Review Comments
When asked to fix code review comments, **always work only with unresolved (open) comments**. Resolved comments should be ignored as they have already been addressed.
### Getting Unresolved Review Comments
Use GitHub GraphQL API to fetch only unresolved review comments from a pull request:
The REST endpoint `/pulls/{pr}/reviews` returns review summaries, not individual review comments. Use the GraphQL API to access `reviewThreads` with `isResolved` status:
The key filter is `select(.isResolved == false)` which ensures only unresolved review threads are processed. Each thread can contain multiple comments, but if the thread is resolved, all its comments should be ignored.
### Working with Review Comments
1. **Fetch unresolved comments** using the GraphQL query above
2. **Parse the results** to identify:
- File path (`path`)
- Line number (`line` or `originalLine`)
- Comment text (`bodyText`)
- Author (`author.login`)
3. **Address each unresolved comment** by:
- Locating the relevant code section
- Making the requested changes
- Ensuring the fix addresses the concern raised
4. **Do NOT process resolved comments** - they have already been handled
- **REST API limitation**: The REST endpoint `/pulls/{pr}/reviews` returns review summaries, not individual review comments. Use GraphQL API for accessing `reviewThreads` with `isResolved` status.
- **Thread-based resolution**: Comments are organized in threads. If a thread is resolved (`isResolved: true`), ignore all comments in that thread.
- **Always filter**: Never process comments from resolved threads, even if they appear in the results.
### Example Workflow
```bash
# Get PR comments
gh pr view 1234 --comments
# Review comments and identify valid ones
# Make necessary changes to address valid comments
# ... edit files ...
# Show what was changed (but don't commit)
git status
git diff
# Tell the user what was fixed and what was skipped
```
## Git Permissions
Request these permissions when needed:
- `git_write` - For commit, rebase, cherry-pick, branch operations
- `network` - For push, fetch, pull operations
## Common Issues
**PR has extra commits?**
→ Rebase on `upstream/main` and cherry-pick only your commits
- Follow conventional commit format for changelogs
### PackageSource CRD upgrade policy
Each component in a `PackageSource` may set `install.upgradeCRDs` to control how CRDs from the chart's `crds/` directory are handled on `HelmRelease` upgrades. Allowed values: `Skip` (default — helm-controller does not touch CRDs on upgrade), `Create` (create new CRDs only), `CreateReplace` (create new and overwrite existing).
Set `upgradeCRDs: CreateReplace` for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them.
Do **not** set `CreateReplace` blindly: it overwrites every CRD in `crds/` and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually.
* **[platform] Fix tenant admins unable to create FoundationDB, Harbor, MongoDB, OpenBAO, OpenSearch, Qdrant, and VPN applications**: The `cozy:tenant:admin:base` ClusterRole was missing RBAC entries for `foundationdbs`, `harbors`, `mongodbs`, `openbaos`, `opensearches`, `qdrants`, and `vpns` resources from `apps.cozystack.io`. Without these permissions, tenant admins could not create these applications — the "Add" button was inactive in the dashboard. The fix adds all seven missing resource verbs ([**@sircthulhu**](https://github.com/sircthulhu) in #2268, #2271).
* **[system] Fix 403 error on Service details page for tenant users**: The `cozy:tenant:base` and `cozy:tenant:view:base` ClusterRoles were missing read permissions for `discovery.k8s.io/endpointslices`. The dashboard requests EndpointSlices to display the "Pod serving" section on the Service details page, and without this permission tenant users received a 403 error. The fix adds `get`, `list`, and `watch` verbs for endpointslices to both tenant roles ([**@sircthulhu**](https://github.com/sircthulhu) in #2257, #2284).
* **[dashboard] Fix "Pod serving" table showing "Raw:" prefixes and "Invalid Date" on Service details page**: The EndpointSlice table on the service details page displayed raw data and broken timestamps because the `EnrichedTable` component referenced the `factory-kube-service-details-endpointslice` customization ID which had no corresponding `CustomColumnsOverride`. The fix adds column definitions for Pod (`.targetRef.name`), Addresses (`.addresses`), Ready (`.conditions.ready`), and Node (`.nodeName`) ([**@sircthulhu**](https://github.com/sircthulhu) in #2266, #2282).
* **[dashboard] Fix broken backup menu links missing cluster context**: Backup resources (plans, backupjobs, backups) are not `ApplicationDefinitions`, so `ensureNavigation()` never created their `baseFactoriesMapping` entries. Without these mappings, the OpenUI frontend could not resolve the `{cluster}` context for backup pages, producing broken sidebar links with an empty cluster segment (e.g. `/openapi-ui//tenant-root/...` instead of `/openapi-ui/default/tenant-root/...`). The fix adds the three missing static entries to the Navigation resource ([**@sircthulhu**](https://github.com/sircthulhu) in #2232, #2270).
* **[linstor] Fix swapped VMPodScrape job labels causing incorrect alerts**: The `job` labels in the `cozy-linstor` VictoriaMetrics `VMPodScrape` templates were swapped: `linstor-satellite` metrics were relabeled as `job=linstor-controller` and vice versa. This caused `linstorControllerOffline` alerts to fire against satellite endpoints (`:9942`) while reporting the controller as unreachable. The fix ensures `linstor-satellite` metrics keep `job=linstor-satellite` and `linstor-controller` metrics keep `job=linstor-controller`, restoring consistent alerting and dashboard semantics ([**@sasha-sup**](https://github.com/sasha-sup) in #2264, #2288).
* **[piraeus-operator] Fix LINSTOR satellite alert annotations and reduce false-positive alerts**: Two issues in the LINSTOR alerts shipped by `cozy-piraeus-operator` were fixed. First, `linstorSatelliteErrorRate` used a non-existent `name` label in annotations, resulting in `Satellite ""` in alert notifications — corrected to use `{{ $labels.hostname }}`. Second, `linstorSatelliteErrorRate` produced false positives when the `linstor-controller` scrape flapped and historical `linstor_error_reports_count` counters reappeared inside the alert window — fixed by requiring stable `up{job="linstor-controller"}` for the full 15-minute window. Additionally, the controller availability alert was split to add a dedicated warning for metrics scrape failures with a 10-minute hold time to reduce transient noise ([**@sasha-sup**](https://github.com/sasha-sup) in #2265, #2287).
## Documentation
* **[website] Add Backup and Recovery guide for VMInstance and VMDisk**: Replaced the generic Kubernetes Backup and Recovery guide with a virtualization-focused Backup and Recovery doc covering VMInstance and VMDisk one-off and scheduled backups, restores, status checks, and troubleshooting (including Velero-related notes) ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#456).
* **[website] Update developer guide with operator-driven architecture and OCIRepository/migration flow**: Rewrote the development guide to describe the operator-driven in-cluster architecture, bootstrap flow, operator responsibilities, and platform install/update sequence. Added documentation for OCIRepositories and the migration flow with migration hook examples and sequencing rules for pre-upgrade/install migrations. Also updated the concepts guide with the two-repository update model, dependency ordering rules, namespace creation behavior, and cluster-wide values injection ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#458).
* **[boot-to-talos] Add support for ISO, RAW, and HTTP image sources**: The `boot-to-talos` tool can now use ISO files, raw disk images, and HTTP URLs as Talos image sources in addition to container registry images. This allows bootstrapping nodes in air-gapped environments or from locally stored images without requiring a container registry ([**@lexfrei**](https://github.com/lexfrei) in cozystack/boot-to-talos#13).
* **[boot-to-talos] Use permanent MAC address for predictable network interface names**: Interface name detection now reads the permanent MAC address directly from sysfs instead of relying on udev data, providing a stable hardware MAC that is unaffected by user modifications to the active MAC address. This makes network interface naming more reliable across reboots and hardware changes ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/boot-to-talos#14).
## Fixes
* **[dashboard] Fix broken backup menu links missing cluster context**: Backup resources (plans, backupjobs, backups) are not `ApplicationDefinition`s, so `ensureNavigation()` never created their `baseFactoriesMapping` entries. Without these entries the OpenUI frontend could not resolve the `{cluster}` context for backup pages, producing broken sidebar links with an empty cluster segment (e.g. `/openapi-ui//tenant-root/...`). The missing `baseFactoriesMapping` entries for all backup resource types are now added to the static `Navigation` resource ([**@sircthulhu**](https://github.com/sircthulhu) in #2232, #2269).
* **[platform] Fix tenant admins unable to create FoundationDB, Harbor, MongoDB, OpenBAO, OpenSearch, Qdrant, and VPN applications**: The `cozy:tenant:admin:base``ClusterRole` was missing seven application resources from `apps.cozystack.io` (`foundationdbs`, `harbors`, `mongodbs`, `openbaos`, `opensearches`, `qdrants`, `vpns`). Without these permissions, tenant admins could not create these applications — the "Add" button was inactive in the dashboard. The missing resources have been added to the ClusterRole ([**@sircthulhu**](https://github.com/sircthulhu) in #2268, #2272).
* **[dashboard] Fix StorageClass dropdown showing "Error" in application forms**: The dashboard UI fetches `StorageClass` resources to populate dropdowns (e.g. in the Postgres form), but the `cozystack-dashboard-readonly``ClusterRole` did not include `storage.k8s.io/storageclasses`. This caused authenticated users to see "Error" instead of the StorageClass name. `get`/`list`/`watch` permissions for `storageclasses` have been added to the dashboard readonly role ([**@sircthulhu**](https://github.com/sircthulhu) in #2267, #2274).
* **[system] Fix 403 error on Service details page by granting tenants read access to EndpointSlices**: The dashboard requested `EndpointSlices` from the `discovery.k8s.io` API group to display the "Pod serving" section on the Service details page, but `cozy:tenant:base` and `cozy:tenant:view:base``ClusterRole`s lacked permissions for this resource. Tenant users received a 403 error when opening the Service details page. `get`/`list`/`watch` permissions for `endpointslices` have been added to both tenant ClusterRoles ([**@sircthulhu**](https://github.com/sircthulhu) in #2257, #2285).
* **[dashboard] Fix "Pod serving" table displaying "Raw:" and "Invalid Date" on Service details page**: The Service details page `EndpointSlice` table showed "Raw:" prefixes and "Invalid Date" values because the `EnrichedTable` referenced `customizationId``factory-kube-service-details-endpointslice` which had no corresponding `CustomColumnsOverride`. Column definitions for Pod (`.targetRef.name`), Addresses (`.addresses`), Ready (`.conditions.ready`), and Node (`.nodeName`) have been added ([**@sircthulhu**](https://github.com/sircthulhu) in #2266, #2283).
* **[piraeus-operator] Fix LINSTOR satellite alert labels, reduce scrape-flap false positives, and improve controller alerting**: Three alerting issues in `cozy-piraeus-operator` have been addressed: (1) `linstorSatelliteErrorRate` used a non-existent `name` label in annotations, resulting in `Satellite ""` in alert notifications — corrected to `{{ $labels.hostname }}`; (2) `linstorSatelliteErrorRate` could produce false positives when the `linstor-controller` scrape flapped and historical `linstor_error_reports_count` counters reappeared inside the alert window — fixed by adding a minimum scrape-count guard; (3) The `LinstorControllerOffline` alert has been split into separate availability and metrics-availability alerts with configurable hold time to reduce noise during brief connectivity interruptions ([**@sasha-sup**](https://github.com/sasha-sup) in #2265, #2286).
* **[linstor] Fix swapped VMPodScrape job labels causing incorrect controller offline alerts**: The `cozy-linstor` VictoriaMetrics `VMPodScrape` templates had the `job` relabeling rules swapped: `linstor-satellite` metrics were labeled as `job=linstor-controller` and vice versa. This caused `linstorControllerOffline` alerts to fire for satellite endpoints (`:9942`) while reporting that the controller was unreachable. The `job` labels are now correctly assigned to their respective targets ([**@sasha-sup**](https://github.com/sasha-sup) in #2264, #2289).
* **[boot-to-talos] Fix triple-fault on hosts with 5-level paging (LA57) enabled**: On hosts with `CONFIG_X86_5LEVEL=y` in the kernel, kexec into Talos caused a triple-fault because the Talos kernel does not support 5-level page tables. `boot-to-talos` now detects LA57 before kexec and automatically patches GRUB with `no5lvl`, runs `update-grub`, and reboots. After reboot with 5-level paging disabled, `boot-to-talos` proceeds normally ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/boot-to-talos#15).
* **[boot-to-talos] Fix EFI boot entry creation when using loop device images**: Talos installer skips EFI variable creation when running on loop devices. `boot-to-talos` now creates a proper UEFI boot entry with an `HD()` device path pointing to the real target disk's ESP by reading the GPT partition table from the target disk after image copy, instead of relying on the Talos installer ([**@kvaps**](https://github.com/kvaps) in cozystack/boot-to-talos#16).
* **[talm] Fix silent empty output when no template files are specified**: Running `talm template` without `--file` or `--template` flags previously produced minimal or empty output without any error. Validation has been added to `engine.Render` to return a clear error message when no template files are specified, making misconfigured invocations immediately apparent ([**@kvaps**](https://github.com/kvaps) in cozystack/talm#112).
## Documentation
* **[website] Add documentation for VMInstance and VMDisk backups**: Added a new virtualization-focused Backup and Recovery guide covering one-off and scheduled backups for `VMInstance` and `VMDisk` resources, restore procedures, status verification commands, and troubleshooting notes including Velero-related issues ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#456).
* **[website] Update developer guide with operator-driven architecture and OCIRepository migration flow**: Rewrote the development guide to describe the operator-driven in-cluster architecture, bootstrap flow, operator responsibilities, and the platform install/update sequence. Added an "OCIRepositories and Migration Flow" section with migration hook examples and sequencing rules for pre-upgrade hooks ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#458).
* **[platform] Prevent installed packages deletion**: Added the `helm.sh/resource-policy: keep` annotation to all platform packages. Previously, moving a package to `disabledPackages` or removing it from `enabledPackages` caused Helm to automatically delete it, contradicting the documented behavior that requires the platform administrator to manually delete packages when needed ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2273, #2298).
* **[linstor] Fix TCP port mismatches after toggle-disk operations causing DRBD resources to enter StandAlone state**: During toggle-disk operations, `removeLayerData()` freed TCP ports from the number pool and `ensureStackDataExists()` could then allocate different ports. If a satellite missed the resulting update (e.g. due to a controller restart), it retained the old ports while peers received the new ones, causing DRBD connections to fail with StandAlone state. The fix introduces `copyDrbdTcpPortsIfExists()`, which preserves existing TCP ports in the `LayerPayload` before `removeLayerData()` releases them ([**@kvaps**](https://github.com/kvaps) in #2292, #2300).
* **[build] Filter git describe to match only v* tags**: Adds `--match 'v*'` to all `git describe` calls in `hack/common-envs.mk`. The `api/apps/v1alpha1/*` subtags share the same commit as release tags, causing `git describe --exact-match` to pick `api/apps/v1alpha1/vX.Y.Z` instead of `vX.Y.Z`, producing invalid Docker image tags ([**@kvaps**](https://github.com/kvaps) in #2386, #2388).
## Development, Testing, and CI/CD
* **[ci] Replace cozystack-bot PAT with cozystack-ci GitHub App**: Replaces the long-lived `cozystack-bot` personal access token with short-lived, scoped tokens from the `cozystack-ci` GitHub App across all release workflows. Improves security and auditability of CI operations ([**@tym83**](https://github.com/tym83) in #2351).
* **[ci] Replace GH_PAT with cozystack-ci GitHub App token in pull-requests workflow**: Switches the pull-requests release workflow to use the cozystack-ci GitHub App token instead of the personal access token ([**@kvaps**](https://github.com/kvaps) in #2383).
* **[ci] Use cozystack org noreply email for bot commits**: Updates CI workflows to use the cozystack organization noreply email for bot commits ([**@kvaps**](https://github.com/kvaps) in #2392).
> **⚠️ WARNING: Do not use this release.** This version includes CloudNativePG operator, which updates the default PostgreSQL image to version 18. CNPG is unable to perform the migration from the previous major version automatically, which will cause PostgreSQL clusters to fail to start after the upgrade. Please use [v1.2.1](https://github.com/cozystack/cozystack/releases/tag/v1.2.1) instead.
Cozystack v1.2.0 delivers significant platform enhancements: a fully managed **OpenSearch** service joining the application catalog, **VPC peering** for secure inter-tenant networking, tenant workload placement control via the new **SchedulingClass** system, a highly-available **VictoriaLogs cluster** replacing the single-node setup, and **Linstor volume relocation** for optimized clone and snapshot restore placement. Additional highlights include external-dns as a standalone extra package, multi-node RWX volume fixes, and a wave of dashboard and monitoring improvements.
## Feature Highlights
### OpenSearch: Managed Search and Analytics Service
Cozystack now ships **OpenSearch** as a fully managed PaaS application — supporting OpenSearch v1, v2, and v3 in a multi-role topology with dedicated master, data, ingest, coordinating, and ML nodes. TLS is enabled by default, HTTP Basic auth is provided out of the box, and custom user definitions allow per-application credentials. The optional **OpenSearch Dashboards** UI can be enabled alongside the engine. External access, topology spread policies, and a comprehensive JSON schema are all included.
A companion `opensearch-operator` system package wraps the upstream Opster OpenSearch Operator v2.8.0 and adds a sysctl DaemonSet to configure the required `vm.max_map_count` kernel parameter on every node automatically. An ApplicationDefinition package ties everything into the Cozystack platform dashboard with schema validation and resource management.
### SchedulingClass: Tenant Workload Placement
Cozystack now supports a **SchedulingClass** CRD that allows platform operators to define cluster-wide scheduling constraints — pinning tenant workloads to specific data centers, hardware generations, or node groups without requiring tenants to manage scheduler configuration themselves. Tenants declare a `schedulingClass` in their Tenant spec; the platform injects the appropriate `schedulerName` into all workloads in that namespace.
The `lineage-controller-webhook` has been extended to verify the referenced SchedulingClass CR before injection, and child tenants inherit their parent's scheduling constraints (children cannot override). A **SchedulingClass dropdown** in the Tenant creation form in the dashboard makes the feature fully self-service. The underlying `cozystack-scheduler` — a custom kube-scheduler extension with SchedulingClass-aware affinity plugins — is now installed and enabled by default as part of the platform.
### VPC Peering for Multi-Tenant Environments
The `vpc` application gains bilateral **VPC peering** using Kube-OVN's native `vpcPeerings` mechanism, allowing tenants to securely interconnect their private networks without routing traffic through public endpoints. Peering link-local IPs (`169.254.0.0/16`) are allocated deterministically from a hash of the sorted VPC pair names, ensuring stable addresses across reconciliations. Static route support (`staticRoutes`) enables fine-grained inter-VPC routing policies. A `cozy-lib` helper (`hexToInt`) performs the deterministic IP allocation, and a JSON Schema validation enforces the `^tenant-` namespace pattern for peered VPCs.
### VictoriaLogs: Clustered Mode for High Availability
The platform's log storage has been upgraded from the deprecated single-node `VLogs` CR to a **VLCluster** deployment with separate vlinsert, vlselect, and vlstorage components, each running with 2 replicas by default — consistent with the existing VMCluster setup. This brings horizontal scalability and resilience to the logging tier. VPA autoscaling is enabled for all VLCluster components, and the victoria-metrics-operator has been upgraded from v0.55.0 to v0.68.1 to add VLCluster CRD support.
### Linstor CSI: Volume Relocation After Clone and Restore
The Linstor CSI driver now carries upstream patches enabling **automatic replica relocation** after PVC clone and snapshot restore operations. Two new parameters control the behavior: `linstor.csi.linbit.com/relocateAfterClone` on StorageClasses moves replicas to optimal nodes after a clone, and `snap.linstor.csi.linbit.com/relocate-after-restore` on VolumeSnapshotClasses does the same after a restore. VolumeSnapshotClasses for Velero and Kasten use cases are pre-configured. This enables full PVC-level backup and restore workflows with automatic data rebalancing, a key prerequisite for production Velero/Kasten integrations.
## Major Features and Improvements
* **[apps] Add managed OpenSearch service**: Deployed as a PaaS application supporting OpenSearch v1/v2/v3 with multi-role node topology, TLS, HTTP Basic auth, custom users, optional OpenSearch Dashboards UI, external access, and topology spread policies; backed by the opster OpenSearch Operator v2.8.0 and a sysctl DaemonSet for `vm.max_map_count` ([**@matthieu-robin**](https://github.com/matthieu-robin) in #1953).
* **[vpc] Add VPC peering support for multi-tenant environments**: Bilateral VPC peering via Kube-OVN's `vpcPeerings`, deterministic link-local IP allocation from sorted VPC pair hash, static routes support, ConfigMap peer discovery enrichment, and JSON Schema validation enforcing `^tenant-` namespace pattern ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2152).
* **[monitoring] Migrate VictoriaLogs from VLogs to VLCluster**: Replaced deprecated single-node `VLogs` CR with clustered `VLCluster` (vlinsert/vlselect/vlstorage, 2 replicas each), added VPA for all components, upgraded victoria-metrics-operator to v0.68.1 ([**@sircthulhu**](https://github.com/sircthulhu) in #2153).
* **[scheduler] Integrate SchedulingClass support for tenant workloads**: Added `schedulingClass` Tenant parameter with inheritance enforcement, `scheduling.cozystack.io/class` namespace label, lineage-webhook extension to verify and inject `schedulerName`, SchedulingClass dropdown in Tenant dashboard form ([**@sircthulhu**](https://github.com/sircthulhu) in #2223).
* **[cozystack-scheduler] Add custom scheduler as an optional system package**: Vendored `cozystack-scheduler` from github.com/cozystack/cozystack-scheduler — a kube-scheduler extension with SchedulingClass-aware affinity plugins, including Helm chart with RBAC, ConfigMap, Deployment, and CRD ([**@lllamnyp**](https://github.com/lllamnyp) in #2205).
* **[platform] Enable cozystack-scheduler by default**: The cozystack-scheduler and SchedulingClass CRD are now installed as default system packages; the backup tool has been moved to optional packages ([**@lllamnyp**](https://github.com/lllamnyp) in #2253).
* **[extra] Add external-dns as a standalone extra package**: Packaged external-dns as an installable extra (tenant-level) component for automatic DNS record management from Kubernetes Service and Ingress resources ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #1988).
* **[linstor] Add linstor-csi patches for clone/snapshot relocation**: New patch enabling `relocateAfterClone` StorageClass parameter and `relocate-after-restore` VolumeSnapshotClass parameter; pre-configured VolumeSnapshotClasses for Velero and relocation workflows; CDI switched to csi-clone strategy ([**@kvaps**](https://github.com/kvaps) in #2133).
* **[monitoring] Add inlineScrapeConfig support to tenant vmagent**: Tenants can now define inline scrape configurations directly in their VMAgent spec, enabling custom metrics collection from services that are not discoverable via standard Kubernetes service discovery ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2200).
* **[monitoring] Add Slack dashboard URL, vmagent environment label, and dynamictext Grafana plugin**: Added `SLACK_DASHBOARD_URL` and `SLACK_SUMMARY_FMT` environment variables for richer alert notifications, per-vmagent `environment` label for metric source identification, and the `dynamictext-panel` plugin for Grafana dashboards ([**@vnyakas**](https://github.com/vnyakas) in #2210).
* **[monitoring] Scope infrastructure dashboards to tenant-root only**: Infrastructure-level Grafana dashboards are now scoped to the tenant-root namespace only, preventing them from appearing in tenant sub-namespaces and reducing dashboard noise ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2197).
* **[tenant] Allow egress to virt-handler for VM metrics scraping**: Extended tenant NetworkPolicy to permit egress to virt-handler pods, enabling Prometheus to scrape VM-level metrics from KubeVirt without additional policy exceptions ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2199).
* **[dashboard] Add keycloakInternalUrl for backend-to-backend OIDC requests**: Added a `keycloakInternalUrl` platform value for the dashboard backend to perform OIDC token introspection via an internal cluster URL, avoiding external round-trips and improving reliability in air-gapped environments ([**@sircthulhu**](https://github.com/sircthulhu) in #2224).
* **[dashboard] Add secret-hash annotation to KeycloakClient for secret sync**: Added a `secret-hash` annotation to the KeycloakClient resource so that changes to the client secret trigger automatic reconciliation and propagation to dependent components ([**@sircthulhu**](https://github.com/sircthulhu) in #2231).
* **[docs] Add OpenAPI and Go types code generation for apps**: Added tooling to generate OpenAPI schemas and Go types from Helm chart values, enabling type-safe programmatic access to managed application configurations and automatic API reference generation ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2214).
## Improvements (minor)
* **[cozystack-scheduler] Update to v0.2.0**: Updated the cozystack-scheduler to v0.2.0 with improved SchedulingClass affinity handling ([**@lllamnyp**](https://github.com/lllamnyp) in #2244).
* **[platform] Ensure cozystack-packages OCIRepository updates reliably**: Added configuration to ensure the `cozystack-packages` OCIRepository resource is consistently reconciled and reflects the latest package versions on upgrade ([**@sircthulhu**](https://github.com/sircthulhu) in #2246).
* **[etcd] Add protective limits to defrag CronJob**: Added CPU and memory resource limits to the etcd defragmentation CronJob to prevent it from starving other workloads during scheduled defragmentation runs ([**@sircthulhu**](https://github.com/sircthulhu) in #2233).
* **[platform] Add missing apps to tenant admin RBAC**: Extended the tenant admin ClusterRole to include RBAC permissions for recently added applications that were missing from the role binding ([**@sircthulhu**](https://github.com/sircthulhu) in #2268).
## Bug Fixes
* **[keycloak] Fix health probe configuration for Keycloak v26.x+**: Replaced deprecated `KC_PROXY=edge` with `KC_PROXY_HEADERS=xforwarded`/`KC_HTTP_ENABLED=true`; replaced liveness/readiness probes with management port endpoints (`/health/live`, `/health/ready`) and added a `startupProbe` to handle slow Keycloak startup without triggering premature restarts ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2162).
* **[migrations] Handle missing RabbitMQ CRD in migration 34**: Fixed a crash in migration script 34 that occurred when the RabbitMQ CRD was not yet installed, allowing upgrades from environments where RabbitMQ was never deployed ([**@IvanHunters**](https://github.com/IvanHunters) in #2168).
* **[platform] Fix VM MAC address not preserved during migration**: Fixed the `virtual-machine` to `vm-instance` migration script to correctly carry over the MAC address, preventing network identity changes after upgrading existing VM resources ([**@sircthulhu**](https://github.com/sircthulhu) in #2169).
* **[dashboard] Fix External IPs factory EnrichedTable rendering**: Corrected the External IPs factory component to use the EnrichedTable renderer, resolving blank/broken rendering of the external IP address list in the dashboard ([**@IvanHunters**](https://github.com/IvanHunters) in #2175).
* **[dashboard] Preserve disabled/hidden state on MarketplacePanel reconciliation**: Fixed a regression where MarketplacePanel reconciliation would reset the `disabled` and `hidden` flags set by operators, causing hidden applications to reappear in the catalog ([**@IvanHunters**](https://github.com/IvanHunters) in #2176).
* **[dashboard] Exclude hidden MarketplacePanel resources from sidebar menu**: Fixed the sidebar to omit applications that have been hidden via MarketplacePanel flags, preventing inaccessible menu entries from being displayed to users ([**@IvanHunters**](https://github.com/IvanHunters) in #2177).
* **[etcd-operator] Replace deprecated kube-rbac-proxy image**: Replaced the unmaintained `gcr.io/kubebuilder/kube-rbac-proxy` sidecar with the actively maintained `quay.io/brancz/kube-rbac-proxy` image to eliminate deprecation warnings and ensure continued security updates ([**@kvaps**](https://github.com/kvaps) in #2181).
* **[backups] Fix RBAC and backupstrategy-controller location**: Corrected role bindings and the deployment location for the backup strategy controller to restore full backup and restore functionality ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2149).
* **[api] Skip OpenAPI post-processor for non-apps group versions**: Fixed the API server to bypass OpenAPI schema post-processing for non-`apps` group versions, preventing schema corruption in unrelated API groups ([**@kvaps**](https://github.com/kvaps) in #2212).
* **[bucket] Fix s3manager endpoint mismatch with COSI credentials**: Corrected the S3 Manager UI to use the actual S3 endpoint from the BucketInfo COSI resource rather than a hardcoded value, resolving connection failures when the S3 endpoint differs from the default ([**@IvanHunters**](https://github.com/IvanHunters) in #2211).
* **[kube-ovn] Fix MASTER_NODES detection for multi-master Kubernetes clusters**: Updated kube-ovn configuration to discover control-plane nodes via the standard `node-role.kubernetes.io/control-plane` label rather than relying on static node lists, fixing OVN connectivity issues in multi-master generic Kubernetes deployments ([**@lexfrei**](https://github.com/lexfrei) in #2245).
* **[kubernetes] Fix CiliumNetworkPolicy endpointSelector for multi-node RWX volumes**: Corrected the CiliumNetworkPolicy endpoint selector for NFS-based ReadWriteMany volumes to properly allow NFS traffic when data is spread across multiple Linstor storage nodes ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2227).
* **[csi] Hide disk.img and lost+found from RWX NFS mounts**: Fixed the Linstor CSI NFS server to exclude internal files (`disk.img`, `lost+found`) from being visible inside NFS-mounted volumes, preventing application errors caused by unexpected files in volume root directories ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2243).
* **[dashboard] Fix broken backup menu links missing cluster context**: Restored cluster context in backup-related sidebar navigation links, fixing 404 errors when navigating to BackupJob and Plan pages from the cluster-level dashboard view ([**@kvaps**](https://github.com/kvaps) in #2232).
* **[dashboard] Fix StorageClass dropdown "Error" state by granting RBAC read access**: Added a ClusterRole/ClusterRoleBinding to grant authenticated users read access to StorageClass resources, resolving the "Error" state displayed in StorageClass dropdowns on application forms ([**@sircthulhu**](https://github.com/sircthulhu) in #2267).
* **[postgres] Fix database deletion lifecycle management**: Added cleanup stages to delete databases and orphaned roles when removed from `values.databases`, enabling declarative database lifecycle management and preventing stale data retention ([**@sircthulhu**](https://github.com/sircthulhu) in #2247).
* **[dashboard] Fix JSONPath crash on Tenant details with resourceQuotas**: Restored fallback protection for unresolved flatMap placeholders in the ResourceQuota "Used" column, preventing JSONPath parser crashes on the Tenant details page ([**@sircthulhu**](https://github.com/sircthulhu) in #2249).
* **[system] Fix tenant RBAC for endpointslices read access**: Added `discovery.k8s.io/endpointslices` read permissions to tenant ClusterRoles, resolving 403 errors on the Service details page when displaying the "Pod serving" section ([**@sircthulhu**](https://github.com/sircthulhu) in #2257).
* **[linstor] Fix swapped VMPodScrape job labels**: Corrected the `job` label relabeling in LINSTOR VictoriaMetrics PodScrape templates, fixing `linstorControllerOffline` alerts that incorrectly reported satellite endpoints as controller failures ([**@sasha-sup**](https://github.com/sasha-sup) in #2264).
* **[piraeus-operator] Fix LINSTOR satellite alert labels and scrape flapping false positives**: Fixed non-existent `name` label in `linstorSatelliteErrorRate` alert annotations (changed to `hostname`) and prevented false positives caused by scrape flapping and stale metric counters ([**@sasha-sup**](https://github.com/sasha-sup) in #2265).
* **[dashboard] Fix EndpointSlice column definitions for Pod serving table**: Added missing `CustomColumnsOverride` for the EndpointSlice table on service details page, replacing "Raw:" prefixes and "Invalid Date" values with proper Pod, Addresses, Ready, and Node columns ([**@sircthulhu**](https://github.com/sircthulhu) in #2266).
## Dependencies & Version Updates
* **[cilium] Update Cilium to v1.19.1**: Upgraded the Cilium CNI to v1.19.1 with latest bug fixes and performance improvements ([**@BROngineer**](https://github.com/BROngineer) in #2173).
* **[keycloak-operator] Update to v1.32.0**: Updated the Keycloak Operator to v1.32.0 (based on epam/edp-keycloak-operator with upstream patches), bumping Keycloak to 26.5.2 ([**@lllamnyp**](https://github.com/lllamnyp) in #2206).
* **[postgres-operator] Update to v1.27.3**: Upgraded the Postgres Operator (Patroni-based) to v1.27.3 with latest upstream fixes ([**@dmpopoff**](https://github.com/dmpopoff) in #2226).
* **[objectstorage-controller] Update to v0.2.2, drop upstreamed patches**: Updated the object storage controller to v0.2.2 and removed patches that were accepted upstream, reducing the maintenance delta ([**@lexfrei**](https://github.com/lexfrei) in #2261).
* **[kilo] Switch from fork to upstream squat/kilo**: Replaced the Cozystack-maintained Kilo fork with the upstream `squat/kilo` image now that required patches (`--internal-cidr`, allowed-location-ips fix, preferred source for WireGuard routes, Cilium IPIP overlay support) have been merged upstream ([**@lexfrei**](https://github.com/lexfrei) in #2259).
* **[talos] Bump Talos to v1.12.6**: Updated the pinned Talos version to v1.12.6 ([**@kvaps**](https://github.com/kvaps) in #2254).
* **[talm] Release v0.22.4** (github.com/cozystack/talm): Fixed `--file`/`--template` flag requirement to prevent ambiguous invocations ([**@kvaps**](https://github.com/kvaps) in cozystack/talm#112).
* **[boot-to-talos] Release v0.7.0** (github.com/cozystack/boot-to-talos): Added support for ISO, RAW, and HTTP image sources ([**@lexfrei**](https://github.com/lexfrei) in cozystack/boot-to-talos#13); permanent MAC addresses for predictable interface names ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/boot-to-talos#14); detection and workaround for 5-level paging (LA57) incompatibility with kexec ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/boot-to-talos#15).
* **[boot-to-talos] Release v0.7.1** (github.com/cozystack/boot-to-talos): Fixed EFI boot entry creation to use the target disk rather than relying on the installer disk, preventing boot failures on bare-metal systems ([**@kvaps**](https://github.com/kvaps) in cozystack/boot-to-talos#16).
## Development, Testing, and CI/CD
* **[tests] Stabilize E2E Kubernetes tests**: Comprehensive improvements to E2E test stability: pre-cleanup of leftover resources, fixes for port-forward race conditions and leaks, improved NFS PVC timeout and debug output, proper EXIT trap handling, and increased CAPI deployment timeouts ([**@lexfrei**](https://github.com/lexfrei) in #2262).
* **[ci] Fix E2E check blocking docs-only PRs**: Moved path filtering to the job level so that documentation-only pull requests are not blocked by pending E2E CI checks ([**@IvanHunters**](https://github.com/IvanHunters) in #2170).
* **[ci] Add timeout-minutes to Build and E2E jobs**: Added explicit `timeout-minutes` constraints to Build and E2E workflow jobs to prevent stuck runners from consuming CI resources indefinitely.
## Documentation
* **[website] Complete CA rotation documentation**: Added comprehensive CA certificate rotation procedures for all Cozystack system components ([**@kvaps**](https://github.com/kvaps) in cozystack/website#406).
* **[website] Add Ansible automated installation guide**: Added a step-by-step guide for automated Cozystack installation using Ansible playbooks ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#442).
* **[website] Add self-signed certificates configuration guide for OIDC**: Added documentation for configuring Cozystack to use self-signed TLS certificates with OIDC providers, covering certificate authority setup and kubeconfig integration ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/website#443).
* **[website] Add custom metrics collection guide**: Added a guide explaining how to configure custom Prometheus scrape targets using the new `inlineScrapeConfig` feature of tenant VMAgent ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/website#444).
* **[website] Add PackageSource/Package architecture to Key Concepts**: Documented the PackageSource and Package CRD architecture, explaining how operators extend the platform with custom application catalogs ([**@IvanHunters**](https://github.com/IvanHunters) in cozystack/website#445).
* **[website] Add SchedulingClass operations guide**: Added a guide covering SchedulingClass CRD creation, tenant assignment, and workload placement verification ([**@lllamnyp**](https://github.com/lllamnyp) in cozystack/website#455).
* **[website] Add VMInstance and VMDisk backups documentation**: Added user-facing documentation for backing up and restoring virtual machine instances and VM disk images using Velero ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#456).
* **[website] Update developer guide**: Updated the developer guide with current build, test, and contribution workflows including OCIRepository and migration tooling ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#458).
* **[website] Document keycloakInternalUrl platform value**: Added documentation explaining how to configure `keycloakInternalUrl` for backend-to-backend OIDC token introspection in cluster-internal environments ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#452).
* **[website] Add DependenciesNotReady troubleshooting guide**: Added a troubleshooting article explaining how to diagnose and resolve the `DependenciesNotReady` package status condition ([**@kvaps**](https://github.com/kvaps) in cozystack/website#450).
* **[website] Reorder installation steps for operator-before-platform**: Updated the installation guide to install the cozystack-operator before applying the platform package, reflecting the correct dependency order ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#449).
* **[website] Update managed apps reference**: Updated the automatically generated managed applications reference documentation to reflect new apps and schema changes in this release ([**@app/github-actions**](https://github.com/apps/github-actions) in cozystack/website#448).
* **[website] Update screenshots for Cozystack v1.1**: Refreshed dashboard screenshots to reflect the updated UI in Cozystack v1.1 ([**@kvaps**](https://github.com/kvaps) in cozystack/website#447).
* **[website] Enhance operator backups guide**: Improved the backup and recovery guide for operators with additional recovery scenarios and procedures ([**@androndo**](https://github.com/androndo) in cozystack/website#440).
## Contributors
We'd like to thank all contributors who made this release possible:
* **[postgres] Hardcode PostgreSQL 17 for monitoring databases and add migration**: CloudNativePG operator defaults to PostgreSQL 18.3 when no explicit image is specified, but monitoring queries in Grafana and Alerta rely on PostgreSQL 17 features such as `pg_stat_checkpointer` and the updated `pg_stat_bgwriter`. This mismatch could break monitoring after fresh installs or database recreation. PostgreSQL 17.7 images are now hardcoded for monitoring databases, and migration 37 is added to set version v17 for any existing PostgreSQL resources ([**@IvanHunters**](https://github.com/IvanHunters) in #2304, #2309).
## Fixes
* **[platform] Prevent installed packages deletion**: Added the `helm.sh/resource-policy: keep` annotation to all platform packages. Previously, moving a package to `disabledPackages` or removing it from `enabledPackages` caused Helm to automatically delete the corresponding resource, contradicting the documented behavior that requires the platform administrator to manually delete packages when needed ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2273, #2297).
* **[linstor] Preserve TCP ports during toggle-disk operations**: During toggle-disk operations, `removeLayerData()` freed TCP ports from the number pool and `ensureStackDataExists()` could then allocate different ports. If a satellite missed the resulting update (e.g. due to a controller restart), it retained the old ports while peers received the new ones, causing DRBD connections to fail with StandAlone state. The fix adds `copyDrbdTcpPortsIfExists()` which saves existing TCP ports into the `LayerPayload` before `removeLayerData()` deletes them ([**@kvaps**](https://github.com/kvaps) in #2292, #2299).
* **[platform] Fix resource allocation ratios not propagated to managed packages**: A regression introduced in the bundle restructure caused `cpuAllocationRatio`, `memoryAllocationRatio`, and `ephemeralStorageAllocationRatio` set in `platform/values.yaml` to become no-ops — they were never written to the `cozystack-values` Secret that cozy-lib reads in child packages. This meant all managed applications silently used the hardcoded defaults (10, 1, 40) regardless of operator-configured values. The fix restores propagation by writing the ratios into the `_cluster` section of the `cozystack-values` Secret and passing `cpuAllocationRatio` to the KubeVirt Package component ([**@sircthulhu**](https://github.com/sircthulhu) in #2296, #2301).
* **[linstor] Fix DRBD connectivity failures on kernels without `crct10dif` by setting verify-alg to `crc32c`**: LINSTOR's auto-verify algorithm selection defaults to `crct10dif`, but this kernel crypto module is no longer available in newer kernels (e.g. Talos v1.12.6, kernel 6.18.18). When `crct10dif` is unavailable, DRBD peer connections fail with `VERIFYAlgNotAvail: failed to allocate crct10dif for verify`, causing all DRBD resources to enter Diskless state and lose quorum. `DrbdOptions/Net/verify-alg` is now set to `crc32c` at the controller level ([**@kvaps**](https://github.com/kvaps) in #2303, #2312).
* **[multus] Fix stale sandbox reservations permanently blocking pod creation after CNI ADD failure**: After a node disruption (e.g. DRBD or kube-ovn issues during upgrade), containerd accumulated stale sandbox name reservations. Cleanup failed because multus called delegate plugins for DEL without cached state and they rejected the incomplete config, causing DEL to fail instead of succeeding. Stale entries were never released, permanently blocking new pod creation on the affected node. A custom multus-cni image is now built with a patch that returns success from DEL when CNI ADD never completed ([**@kvaps**](https://github.com/kvaps) in #2313, #2314).
* **[multus] Pin master CNI to `05-cilium.conflist` to prevent race condition at boot**: During node boot or Talos upgrade, multus auto-detects the master CNI conflist by scanning the CNI config directory. If kube-ovn writes `10-kube-ovn.conflist` before Cilium writes `05-cilium.conflist`, multus selects the wrong file and pods bypass the Cilium chain entirely, have no Cilium endpoint, and their traffic is blocked by cluster-wide network policies. `multusMasterCNI` is now pinned to `05-cilium.conflist` ([**@kvaps**](https://github.com/kvaps) in #2315, #2316).
## Documentation
* **[website] Add custom Keycloak themes documentation**: Added documentation for custom Keycloak theme injection to the White Labeling guide, covering the theme image contract (`/themes/` directory structure), configuration via the `cozystack.keycloak` Package resource, `imagePullSecrets` for private registries, and theme activation in the Keycloak admin console ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#463).
* **[website] Add documentation for Go types usage**: Added a guide for using the generated Go types for Cozystack managed applications as a Go module, including installation instructions, programmatic resource management examples, and deployment approaches ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#465).
* **[linstor] Update piraeus-server to v1.33.2 with selected backports**: Bumps LINSTOR server from v1.33.1 to v1.33.2 and adds backported patches for improved storage reliability: a stale bitmap adjust retry mechanism for automatic recovery after bitmap attach errors, LUKS2 header sizing and optimal I/O size detection improvements for more reliable disk formatting, and the maintainer implementation backport. All patches verified against upstream v1.33.2 with `git apply --check` and `gradlew compileJava` ([**@kvaps**](https://github.com/kvaps) in #2331, #2377).
## Fixes
* **[postgres] Fix system PostgreSQL images to 17.7-standard-trixie**: Hardcodes PostgreSQL 17.7-standard-trixie images for system PostgreSQL instances. This ensures system databases use the correct image variant consistent with the monitoring stack requirements introduced in v1.2.1 ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2364, #2369).
* **[cilium] Opt-out of cri-containerd.apparmor.d for nsenter init containers**: On Ubuntu 22.04+, Debian, and other distributions that load the `cri-containerd.apparmor.d` AppArmor profile by default for containerd workloads, the kernel denied `nsenter` namespace entry in cilium-agent init containers (`mount-cgroup`, `apply-sysctl-overwrites`, `clean-cilium-state`), causing the agent to land in `Init:CrashLoopBackOff` and cascading platform failures. Per-container `container.apparmor.security.beta.kubernetes.io` annotations now opt the affected containers out of this profile, applied only on non-Talos cilium variants (`cilium-generic`, `kubeovn-cilium-generic`). The vendored daemonset template is also patched to strip the upstream `semverCompare "<1.30.0"` AppArmor block, preventing duplicate annotation keys. Talos variants are untouched as Talos does not load the AppArmor LSM ([**@lexfrei**](https://github.com/lexfrei) in #2370, #2378).
* **[virtual-machine] Exclude external VM services from Cilium BPF LB**: Adds the `service.kubernetes.io/service-proxy-name: "cozy-proxy"` label to VM LoadBalancer services when `external: true`, telling Cilium to skip BPF processing entirely for these services. This fixes two issues: inter-tenant connectivity via public LB IPs (Cilium's DNAT caused cross-tenant pod-to-pod flow classification, triggering CiliumClusterwideNetworkPolicy blocks) and WholeIP broken on Cilium 1.19+ (wildcard service drop entries blocked traffic to LB IPs on undeclared ports before it reached netfilter/cozy-proxy). MetalLB L2 advertisement and kube-ovn routing remain unaffected ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2357, #2361).
* **[monitoring] Fix infra dashboards missing in default variant**: The default platform variant deploys the monitoring chart to the `cozy-monitoring` namespace, but the dashboard rendering condition introduced in #2197 only checked for `tenant-root`. Infrastructure dashboards were not rendered in the default variant. The `cozy-monitoring` namespace is now included in the rendering condition, consistent with the existing pattern in `vmagent.yaml` ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2365, #2367).
* **[build] Filter git describe to match only v* tags**: Adds `--match 'v*'` to all `git describe` calls in `hack/common-envs.mk`. The `api/apps/v1alpha1/*` subtags share the same commit as release tags, causing `git describe --exact-match` to pick `api/apps/v1alpha1/vX.Y.Z` instead of `vX.Y.Z`, producing invalid Docker image tags ([**@kvaps**](https://github.com/kvaps) in #2386, #2389).
## Development, Testing, and CI/CD
* **[ci] Replace cozystack-bot PAT with cozystack-ci GitHub App**: Replaces the long-lived `cozystack-bot` personal access token with short-lived, scoped tokens from the `cozystack-ci` GitHub App across all release workflows (`tags.yaml`, `auto-release.yaml`, `pull-requests-release.yaml`). Improves security and auditability of CI operations ([**@tym83**](https://github.com/tym83) in #2351).
* **[ci] Use cozystack org noreply email for bot commits**: Updates CI workflows to use the cozystack organization noreply email for bot commits ([**@kvaps**](https://github.com/kvaps) in #2392, #2393).
* **[ci] Replace GH_PAT with cozystack-ci GitHub App token in pull-requests workflow**: Switches the pull-requests release workflow to use the cozystack-ci GitHub App token instead of the personal access token ([**@kvaps**](https://github.com/kvaps) in #2383, #2384).
## Documentation
* **[website] Add ApplicationDefinition naming convention reference**: Added reference documentation on ApplicationDefinition naming conventions and how `cozystack-api` resolves kinds to their backing definitions ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#478).
* **[website] Document Talos / talosctl / Cozystack version pairing**: Added documentation covering Talos, talosctl, and Cozystack version compatibility matrix for installation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#484).
* **[website] Fix KubeOVN MASTER_NODES example path and key in troubleshooting**: Corrected the MASTER_NODES example path and key in the KubeOVN troubleshooting guide ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#483).
* **[website] Prefix bundle package names with cozystack. in v1 examples**: Updated documentation examples to use the correct `cozystack.` prefix for bundle package names in enabled/disabledPackages ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#482).
* **[website] Finish isolated-field removal and document opt-in policy labels**: Removed the obsolete `isolated` field from tenant documentation and documented the new opt-in policy labels approach ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#481).
* **[website] Add --take-ownership flag and describe networking.* fields**: Added documentation for the `--take-ownership` flag and described the `networking.*` fields in the installation guide ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#480).
* **[website] Add bonding (LACP) configuration how-to guide**: Added a guide for configuring network bonding with LACP on Cozystack installations ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#459).
* **[website] Improve registry mirrors for tenant Kubernetes in air-gapped guide**: Improved documentation for configuring registry mirrors in tenant Kubernetes clusters for air-gapped environments ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#461).
* **[website] Update backup/restore documentation for VMI/VMDisk**: Updated backup documentation with information related to VM instance and VM disk restore improvements ([**@androndo**](https://github.com/androndo) in cozystack/website#466).
* **[website] Add updated OpenAPI spec**: Updated the OpenAPI specification for managed applications reference ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#469).
* **[website] Add OSS Health pages and OpenSSF badge**: Added OSS Health section with OpenSSF Scorecard and Best Practices badge to the website footer ([**@tym83**](https://github.com/tym83) in cozystack/website#470).
* **[website] Add CozySummit Virtual 2026 program announcement**: Published the CozySummit Virtual 2026 program announcement blog post ([**@tym83**](https://github.com/tym83) in cozystack/website#472).
* **[website] Add missing release announcements for v0.1–v0.41**: Backfilled missing release announcement blog posts for Cozystack versions v0.1 through v0.41 ([**@tym83**](https://github.com/tym83) in cozystack/website#468).
* **[talm] Render templates online in apply to resolve lookups**: Fixed talm `apply` command to render templates online, resolving template lookup failures when using modeline templates ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/talm#119).
* **[talm] Update default Talos image to v1.12.6**: Updated the default Talos image version to v1.12.6 in talm ([**@kvaps**](https://github.com/kvaps) in cozystack/talm@03e9b6e).
A patch release with bug fixes and documentation updates.
## Features and Improvements
_No notable features in this patch release._
## Fixes
* **fix(kubernetes): set explicit ephemeral-storage on virt-launcher pods**: Prevents VM crashes caused by ephemeral-storage eviction by setting explicit `domain.resources` ephemeral-storage on the VirtualMachine spec. Uses sanitized limits and requests so virt-launcher pods do not inherit too-small namespace defaults. ([**@kvaps**](https://github.com/kvaps) in #2317, backport #2423).
## Documentation
* **[website] feat: add Telemetry page under OSS Health section**: Add Telemetry page and initial data seeding to OSS Health docs ([**@tym83**](https://github.com/tym83) in cozystack/website#471).
* **[website] Refactor docs versions to major.minor variants**: Move docs to major.minor versioning for v1.x series ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#477).
* **[website] docs(tenants): document namespace layout and parent/child derivation** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#479).
* **[website] docs(tenants): document the checkbox-then-edit-CR customization pattern** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#485).
* **[website] docs: fix 14 broken links and stale talm anchor across v1 docs** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#486).
* **[website] fix(og): update social badge image and title** ([**@tym83**](https://github.com/tym83) in cozystack/website#487).
* **[website] docs(external-apps): rewrite guide for ApplicationDefinition API** ([**@kitsunoff**](https://github.com/kitsunoff) in cozystack/website#488).
* **[website] docs: add CLAUDE.md for AI agent guidance** ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#489).
* **[website] fix: update /docs/v1/ redirect to latest v1.2** ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#492).
* **[website] fix(ci): add OpenAPI spec download to GitHub Pages build** ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#494).
* **[website] feat(blog): add managed PostgreSQL with synchronous replication post** ([**@tym83**](https://github.com/tym83) in cozystack/website#497).
* **[website] chore(blog): add images frontmatter for social preview on existing posts** ([**@tym83**](https://github.com/tym83) in cozystack/website#498).
* **[website] feat(blog): taxonomies and client-side filter UI** ([**@tym83**](https://github.com/tym83) in cozystack/website#499).
* **[website] style(oss-health): add breathing room between navbar and hero** ([**@tym83**](https://github.com/tym83) in cozystack/website#500).
## Other repositories
* **[talm] feat(config): migrate to Talos v1.12 multi-document config format**: Upgrade Talos config format and modernize configuration handling ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#116).
* **[talm] chore(deps): bump dependencies and modernize codebase** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#124).
* **[external-apps-example] feat: replace MongoDB example with Minecraft apps from cozylex** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/external-apps-example#2).
* **[ansible-cozystack] fix(examples): add v prefix to collection version in requirements.yml** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#23).
* **[ansible-cozystack] fix(plugins): replace ansible.utils.ipaddr with stdlib-based test plugin** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#24).
* **[ansible-cozystack] chore(deps): update dependency cozystack.installer to v1.2.3** ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#29).
* **[ansible-cozystack] feat(role): expose publishing.externalIPs and tenant-root ingress via role variables** ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#30).
## Contributors
Thanks to everyone who contributed to this patch release:
Cozystack v1.3.0-rc.1 is the first release candidate for v1.3.0, bringing **storage-aware scheduling** via the LINSTOR scheduler extender, a managed **LINSTOR GUI** web UI with Keycloak SSO, a **VM Default Images** catalog for out-of-the-box virtual machine provisioning, **WorkloadsReady conditions** with a real-time Events tab in the dashboard, and **cross-namespace VM backup restore** capabilities. Additional highlights include stricter tenant name validation, VM network selector improvements, Keycloak theme injection and SMTP configuration, and a comprehensive host runtime preflight check.
> **Note:** Fixes marked with *(backported to v1.2.x)* were also included in v1.2.1 or v1.2.2 patch releases.
## Feature Highlights
### Storage-Aware Scheduling via LINSTOR Extender
The `cozystack-scheduler` now calls the **LINSTOR scheduler extender** for storage-locality-aware pod placement. When a pod declares both a `SchedulingClass` and LINSTOR-backed PVCs, the scheduler consults LINSTOR to prefer nodes where volume replicas already exist — reducing cross-node replication traffic and improving I/O latency for storage-heavy workloads ([**@lllamnyp**](https://github.com/lllamnyp) in #2330).
### LINSTOR GUI: Managed Web UI for Storage Administration
A new opt-in `linstor-gui` system package deploys **LINBIT's linstor-gui web UI** alongside the LINSTOR controller with mTLS client authentication, non-root security context, and ClusterIP-only service. An optional **Keycloak-protected Ingress** (via oauth2-proxy) can be enabled for SSO-authenticated browser access when OIDC is configured on the platform ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2382, #2390).
### VM Default Images: Out-of-the-Box VM Provisioning
The new `vm-default-images` package provides a curated set of **cluster-wide virtual machine images** (Ubuntu, Debian, CentOS Stream, and others) as pre-populated DataVolumes. The package is opt-in via the `iaas` bundle and defaults to replicated storage for high availability. A companion migration (migration 38) renames legacy `vm-image-*` DataVolumes to the new `vm-default-images-*` naming scheme. The `vm-disk` chart also gains a new "disk" source type for cloning from existing vm-disks in the same namespace ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2258).
### WorkloadsReady Condition and Events Tab
Applications now expose a **WorkloadsReady** condition on their status by querying associated WorkloadMonitor resources, giving operators a single place to check whether all underlying workloads (Deployments, StatefulSets, DaemonSets) are healthy. The dashboard gains a new **Events tab** showing namespace-scoped Kubernetes events for each application, with fallback to `.firstTimestamp` when `.eventTime` is absent. A bug where WorkloadMonitor's `Operational` status was never persisted is also fixed ([**@lexfrei**](https://github.com/lexfrei) in #2356).
### Cross-Namespace VM Backup Restore
The backup system now supports **restoring VMInstance backups into a different namespace** (cross-namespace copy restores), with IP/MAC preservation and safe rename semantics. In-place backup/restores for VMDisk and VMInstance are improved: HelmReleases and DataVolumes are properly handled, and Velero failure messages are propagated to the Application status. The backup status structure has been refactored to store underlying resources as a generic opaque JSON object, enabling arbitrary application-specific metadata ([**@androndo**](https://github.com/androndo) in #2251, #2329, #2319).
## Major Features and Improvements
* **[api] Reject tenant names with dashes at Create time**: Enforces alphanumeric-only naming for Tenants at the API level, preventing names with hyphens that would silently fail during Helm reconciliation. A corresponding regex tightening and regression test suite hardens the validation ([**@lexfrei**](https://github.com/lexfrei) in #2380).
* **[platform] Validate computed tenant namespace length**: Rejects Tenant creation when the computed ancestor-chain namespace would exceed the 63-character Kubernetes namespace limit, preventing opaque HelmRelease reconcile errors downstream ([**@lexfrei**](https://github.com/lexfrei) in #2376).
* **[vm-instance] Rename subnets to networks and add dropdown selector**: Renames the misleading `subnets` field to `networks` in VMInstance for clarity, adds a dropdown selector for available networks in the dashboard form, and includes a migration to copy existing `subnets` values. The old field remains supported for backward compatibility ([**@sircthulhu**](https://github.com/sircthulhu) in #2263).
* **[keycloak] Enable injecting themes**: Cozystack administrators can now inject custom Keycloak themes via `initContainers` for UI white-labeling and customization ([**@lllamnyp**](https://github.com/lllamnyp) in #2142).
* **[keycloak-configure] Add email verification and SMTP configuration**: Adds configurable Keycloak settings for user self-registration, email verification, and SMTP server configuration, enabling automated user onboarding flows ([**@BROngineer**](https://github.com/BROngineer) in #2318).
* **[postgres] Hardcode PostgreSQL 17 for monitoring databases**: Pins PostgreSQL 17.7 images for system databases (Grafana, Alerta, Harbor, Keycloak, SeaweedFS) and adds migration 37 to backfill `spec.version=v17` for existing PostgreSQL resources, preventing CNPG from defaulting to PostgreSQL 18 *(backported to v1.2.1)* ([**@IvanHunters**](https://github.com/IvanHunters) in #2304).
* **[hack] Add host runtime preflight check**: New `check-host-runtime.sh` script and `make preflight` target that warns operators when a standalone containerd or docker runtime is running alongside the embedded k3s runtime, helping diagnose container runtime conflicts ([**@lexfrei**](https://github.com/lexfrei) in #2371).
* **[hack] Add check-readiness.sh diagnostic script**: A new diagnostic script for tracking platform reconciliation by checking readiness of Packages, ArtifactGenerators, ExternalArtifacts, and HelmReleases, with support for watch mode and continuous monitoring ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2294).
* **[mariadb] Always enable replication for consistent service naming**: MariaDB now always enables replication, creating `-primary`/`-secondary` services even for single-replica instances. This fixes dashboard visibility and backup functionality for single-replica setups ([**@sircthulhu**](https://github.com/sircthulhu) in #2279).
* **[platform] Prevent installed packages deletion**: Adds `helm.sh/resource-policy: keep` annotation to packages, preventing automatic deletion when packages are disabled and restoring documented behavior *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2273).
## Bug Fixes
* **[cilium] Opt-out of cri-containerd.apparmor.d for nsenter init containers**: Opts cilium-agent init containers out of the `cri-containerd.apparmor.d` AppArmor profile on non-Talos variants, fixing `Init:CrashLoopBackOff` on Ubuntu 22.04+ and Debian *(backported to v1.2.2)* ([**@lexfrei**](https://github.com/lexfrei) in #2370).
* **[virtual-machine] Exclude external VM services from Cilium BPF LB**: Adds `service-proxy-name: cozy-proxy` label to VM LoadBalancer services, telling Cilium to skip BPF processing. Fixes inter-tenant connectivity via public LB IPs and WholeIP functionality on Cilium 1.19+ *(backported to v1.2.2)* ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2357).
* **[monitoring] Fix infra dashboards missing in default variant**: Includes `cozy-monitoring` namespace in the dashboard rendering condition, fixing infrastructure Grafana dashboards not rendering in the default platform variant *(backported to v1.2.2)* ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2365).
* **[postgres] Fix system PostgreSQL images to 17.7-standard-trixie**: Normalizes system PostgreSQL image tags to use `17.7-standard-trixie` variant with migration logic for existing CNPG clusters *(backported to v1.2.2)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2364).
* **[build] Filter git describe to match only v\* tags**: Adds `--match 'v*'` to `git describe` calls, preventing API subtags from being picked up instead of release tags and producing invalid Docker image tags *(backported to v1.2.2)* ([**@kvaps**](https://github.com/kvaps) in #2386).
* **[platform] Fix resource allocation ratios not propagated to packages**: Restores propagation of CPU, memory, and ephemeral-storage allocation ratios to managed applications and KubeVirt, which were silently ignored since the bundle restructure *(backported to v1.2.1)* ([**@sircthulhu**](https://github.com/sircthulhu) in #2296).
* **[kubernetes] Set explicit ephemeral-storage on virt-launcher pods**: Sets explicit `domain.resources` with ephemeral-storage on VirtualMachine spec to prevent virt-launcher pods from being evicted due to LimitRange defaults being too low for actual emptyDisk capacity ([**@kvaps**](https://github.com/kvaps) in #2317).
* **[multus] Pin master CNI to 05-cilium.conflist**: Prevents a boot-time race condition where multus could auto-detect kube-ovn's conflist instead of Cilium's *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2315).
* **[multus] Build custom image with DEL cache fix**: Fixes sandbox cleanup deadlock when CNI ADD never completes, preventing stale sandbox name reservations from permanently blocking pod creation *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2313).
* **[linstor] Set verify-alg to crc32c**: Prevents DRBD connection failures on kernels where `crct10dif` is unavailable (e.g., Talos v1.12.6 with kernel 6.18.18) *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2303).
* **[linstor] Preserve TCP ports during toggle-disk operations**: Fixes TCP port mismatches after toggle-disk operations that could cause DRBD resources to enter StandAlone state *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2292).
## Dependencies & Version Updates
* **[linstor] Update piraeus-server to v1.33.2 with selected backports**: Bumps LINSTOR server from v1.33.1 to v1.33.2 with backported patches for stale bitmap adjust retry, LUKS2 header sizing, and optimal I/O size detection *(backported to v1.2.2)* ([**@kvaps**](https://github.com/kvaps) in #2331).
* **[kamaji] Update to 26.3.5-edge, drop upstreamed patches**: Updates Kamaji from edge-26.2.4 to 26.3.5-edge and removes two patches accepted upstream. Adds configurable probe tuning and DataStore readiness conditions ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2260).
* **[talm] Release v0.23.0, v0.23.1, v0.24.0** (github.com/cozystack/talm): Migrates to the Talos v1.12 multi-document machine config format ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#116); fixes template rendering in `apply` command to resolve lookups ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/talm#119); bumps dependencies and modernizes codebase ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#124).
* **[ansible-cozystack] Release v1.2.1, v1.2.2** (github.com/cozystack/ansible-cozystack): Exposes `publishing.externalIPs` and tenant-root ingress via role variables ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#30); adds comprehensive node prerequisites audit ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#27); replaces `ansible.utils.ipaddr` with a stdlib-based test plugin ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#24).
## Security
* **docs: add SECURITY.md**: Adds vulnerability reporting procedures, disclosure expectations, and supported release lines ([**@kvaps**](https://github.com/kvaps) in #2230).
* **docs: add OpenSSF Best Practices badge to README**: Adds the OpenSSF Best Practices passing badge to the project README ([**@lexfrei**](https://github.com/lexfrei) in #2320).
## Development, Testing, and CI/CD
* **[ci] Replace cozystack-bot PAT with cozystack-ci GitHub App**: Replaces the long-lived cozystack-bot personal access token with short-lived, scoped tokens from the cozystack-ci GitHub App across all CI release workflows ([**@tym83**](https://github.com/tym83) in #2351; [**@kvaps**](https://github.com/kvaps) in #2383, #2392).
* **[ci] Add Gemini Code Assist and CodeRabbit configuration**: Adds repository-level configuration for AI code reviewers with ignore patterns for vendored/generated code and incremental review settings ([**@lexfrei**](https://github.com/lexfrei) in #2385).
* **[ci] Make tags workflow idempotent on re-runs**: Fixes CI to force-update API subtags and handle re-runs gracefully ([**@kvaps**](https://github.com/kvaps)).
* **[tests] Fix Kafka E2E test timeout and retry race condition**: Increases Kafka E2E test timeout from 60s to 300s and fixes a retry race condition where `kubectl apply` could hit a still-deleting resource ([**@lexfrei**](https://github.com/lexfrei) in #2358).
* **docs: adopt Conventional Commits for commit and PR titles**: Standardizes commit and PR title format to `type(scope): description` across all contributing docs and the PR template ([**@lexfrei**](https://github.com/lexfrei) in #2395).
* **docs(ci): require screenshots for UI changes in PR template**: Adds a mandatory screenshots section to the PR template for UI-related changes ([**@kitsunoff**](https://github.com/kitsunoff) in #2407).
## Documentation
* **[website] Add ApplicationDefinition naming convention reference**: Documents how `cozystack-api` resolves kinds to their backing definitions ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#478).
* **[website] Document Talos / talosctl / Cozystack version pairing**: Adds version compatibility matrix for installation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#484).
* **[website] Document namespace layout and parent/child derivation**: Explains tenant namespace hierarchy and parent/child namespace derivation rules ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#479).
* **[website] Document the checkbox-then-edit-CR customization pattern for tenants**: Describes the workflow for customizing tenant settings via the CR after initial checkbox-based creation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#485).
* **[website] Add custom Keycloak themes documentation**: Covers the theme image contract, configuration, `imagePullSecrets`, and theme activation in the Keycloak admin console ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#463).
* **[website] Add bonding (LACP) configuration how-to guide**: Covers network bonding configuration for Cozystack installations ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#459).
* **[website] Improve registry mirrors for tenant Kubernetes in air-gapped guide**: Improved documentation for configuring registry mirrors in air-gapped environments ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#461).
* **[website] Rewrite guide for ApplicationDefinition API (external-apps)**: Comprehensive rewrite of the external apps guide using the ApplicationDefinition API ([**@kitsunoff**](https://github.com/kitsunoff) in cozystack/website#488).
* **[website] Add documentation for Go types usage**: Guide for using generated Go types for Cozystack managed applications as a Go module ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#465).
* **[website] Update backup/restore documentation for VMI/VMDisk**: Updated backup documentation with VM instance and VM disk restore improvements ([**@androndo**](https://github.com/androndo) in cozystack/website#466).
* **[website] Add OSS Health pages and OpenSSF badge**: Added OSS Health section with OpenSSF Scorecard and Best Practices badge to the website ([**@tym83**](https://github.com/tym83) in cozystack/website#470).
* **[website] Add CozySummit Virtual 2026 program announcement**: Published the CozySummit Virtual 2026 program announcement blog post ([**@tym83**](https://github.com/tym83) in cozystack/website#472).
* **[website] Add missing release announcements for v0.1–v0.41**: Backfilled missing release announcement blog posts for historical Cozystack versions ([**@tym83**](https://github.com/tym83) in cozystack/website#468).
* **[website] Fix broken links and stale anchors across v1 docs**: Fixes 14 broken links and stale talm anchors ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#486).
* **[website] Prefix bundle package names with cozystack. in v1 examples**: Corrects package naming in documentation examples ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#482).
* **[website] Finish isolated-field removal and document opt-in policy labels**: Removes obsolete `isolated` field from tenant documentation and documents the new approach ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#481).
* **[website] Add --take-ownership flag and describe networking.* fields**: Documents the `--take-ownership` flag and `networking.*` fields in the installation guide ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#480).
* **[website] Fix KubeOVN MASTER_NODES example path and key in troubleshooting**: Corrects the MASTER_NODES example path ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#483).
* **[external-apps-example] Replace MongoDB example with Minecraft apps**: Refactors the external apps example to use ApplicationDefinition API with Minecraft server applications ([**@lexfrei**](https://github.com/lexfrei) in cozystack/external-apps-example#2).
## Governance
* **Add Mattia Eleuteri ([@mattia-eleuteri](https://github.com/mattia-eleuteri)) as Maintainer**: CSI, Storage, Networking & Security ([**@tym83**](https://github.com/tym83) in #2345).
* **Add Matthieu Robin ([@matthieu-robin](https://github.com/matthieu-robin)) as Maintainer**: Managed applications, platform quality, and benchmarking ([**@tym83**](https://github.com/tym83) in #2346).
## Contributors
We'd like to thank all contributors who made this release possible:
Cozystack v1.3.0 brings **storage-aware pod scheduling** via a LINSTOR scheduler extender, a managed **LINSTOR GUI** web console with Keycloak SSO, a curated **VM Default Images** catalog for out-of-the-box virtual-machine provisioning, a new **WorkloadsReady / Events** observability surface with S3 bucket metering, and **cross-namespace VMInstance backup restore** with a full **RestoreJob dashboard** flow. The release also ships stricter tenant-name validation, VMInstance network-selector improvements, Keycloak theme injection and SMTP configuration, a host-runtime preflight check, and rolls up every fix from the v1.2.1 → v1.2.4 patch line.
> **Note:** Items marked *(backported to v1.2.x)* were also shipped in v1.2.1, v1.2.2, v1.2.3, or v1.2.4 patch releases.
## Feature Highlights
### Storage-Aware Scheduling via the LINSTOR Extender
The `cozystack-scheduler` now calls a **LINSTOR scheduler extender** for storage-locality-aware pod placement. When a pod declares both a `SchedulingClass` and LINSTOR-backed PVCs, the scheduler consults LINSTOR to prefer nodes where volume replicas already exist — reducing cross-node replication traffic and improving I/O latency for storage-heavy workloads such as databases, object stores, and VMs.
The integration builds on the existing `SchedulingClass` tenant workload placement system introduced in v1.2.0 and requires no tenant-side configuration — workloads simply benefit once a SchedulingClass is assigned. Administrators can mix storage locality with the existing data-center / hardware-generation constraints defined on SchedulingClass CRs ([**@lllamnyp**](https://github.com/lllamnyp) in #2330).
### LINSTOR GUI: Managed Web Console for Storage Administration
A new opt-in `linstor-gui` system package deploys **LINBIT's linstor-gui web UI** alongside the LINSTOR controller with mTLS client authentication, non-root security context, and a ClusterIP-only service by default. When OIDC is configured on the platform, an optional **Keycloak-protected Ingress** (via oauth2-proxy) exposes the UI for browser access. Access is restricted to members of the `cozystack-cluster-admin` Keycloak group, consistent with host-cluster admin RBAC, and the gatekeeper blocks in-app LINSTOR authentication setup at the nginx proxy layer so the managed configuration cannot be subverted through the UI.
Operators who prefer CLI access keep the existing `linstor` command; the GUI is strictly additive and stays disabled by default ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2382, #2390, #2415, #2419).
### VM Default Images: Out-of-the-Box VM Provisioning
The new `vm-default-images` package provides a curated set of **cluster-wide virtual-machine images** (Ubuntu, Debian, CentOS Stream, and others) as pre-populated DataVolumes, so tenants can provision VMs against well-known base images without first having to upload them. The package is opt-in via the `iaas` bundle and defaults to replicated storage for high availability. Migration 38 renames legacy `vm-image-*` DataVolumes to the new `vm-default-images-*` naming scheme, and the `vm-disk` chart gains a new "disk" source type for cloning from existing vm-disks in the same namespace ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2258).
### Application Observability: WorkloadsReady, Events, and S3 Bucket Metering
Applications now expose a **WorkloadsReady** condition on their status by querying associated WorkloadMonitor resources, giving operators a single place to check whether all underlying workloads (Deployments, StatefulSets, DaemonSets, PVCs) are healthy. The dashboard gains a new **Events tab** showing namespace-scoped Kubernetes events per application, with fallback to `.firstTimestamp` when `.eventTime` is absent. A long-standing bug where WorkloadMonitor's `Operational` status was never persisted is fixed in the same change ([**@lexfrei**](https://github.com/lexfrei) in #2356).
The WorkloadMonitor reconciler is extended to track **COSI BucketClaim** objects as first-class Workloads, and the bucket controller now queries SeaweedFS logical and physical bucket-size metrics from VictoriaMetrics via a namespace-scoped monitoring endpoint, enabling S3 billing integration on par with Pods and PVCs ([**@kitsunoff**](https://github.com/kitsunoff) in #2391). Workloads are also enriched with `workloads.cozystack.io/resource-preset` and source-object labels so downstream billing pipelines can correlate monitors with the tenant preset that produced them ([**@androndo**](https://github.com/androndo) in #2416).
### Cross-Namespace VM Backup Restore and RestoreJob Dashboard
The backup system now supports **restoring VMInstance backups into a different namespace** (cross-namespace copy restores) with IP/MAC preservation and safe rename semantics. In-place backup and restore flows for VMDisk and VMInstance are improved: HelmReleases and DataVolumes are properly handled, and Velero failure messages are propagated to the Application status. The backup status structure has been refactored to store underlying resources as a generic opaque JSON object, enabling arbitrary application-specific metadata without status-schema churn ([**@androndo**](https://github.com/androndo) in #2251, #2319, #2329).
The dashboard now ships a complete **RestoreJob experience**: list view, details page, create form, and sidebar entry, with a "Same as backup" fallback rendering when `spec.targetApplicationRef` is omitted. Non-CRD-backed sidebar factories (`kube-*`, `plan`, `backupjob`, `backup`, `restorejob`) are marked static so they pick up consistent managed-by labels across reconciles ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2437).
## Major Features and Improvements
* **[api] Reject tenant names with dashes at Create time**: Enforces alphanumeric-only naming for Tenants at the API level, preventing names with hyphens that would silently fail during Helm reconciliation. A corresponding regex tightening and regression test suite hardens the validation ([**@lexfrei**](https://github.com/lexfrei) in #2380).
* **[platform] Validate computed tenant namespace length**: Rejects Tenant creation when the computed ancestor-chain namespace would exceed the 63-character Kubernetes namespace limit, preventing opaque HelmRelease reconcile errors downstream ([**@lexfrei**](https://github.com/lexfrei) in #2376).
* **[vm-instance] Rename subnets to networks and add dropdown selector**: Renames the misleading `subnets` field to `networks` in VMInstance for clarity, adds a dropdown selector for available networks in the dashboard form, and includes migration 36 to copy existing `subnets` values. The old field remains supported for backward compatibility ([**@sircthulhu**](https://github.com/sircthulhu) in #2263).
* **[keycloak] Enable injecting themes**: Cozystack administrators can now inject custom Keycloak themes via `initContainers` for UI white-labeling and customization ([**@lllamnyp**](https://github.com/lllamnyp) in #2142).
* **[keycloak-configure] Add email verification and SMTP configuration**: Adds configurable Keycloak settings for user self-registration, email verification, and SMTP server configuration, enabling automated user onboarding flows ([**@BROngineer**](https://github.com/BROngineer) in #2318).
* **[postgres] Pin system PostgreSQL to 17.7-standard-trixie**: Pins the PostgreSQL image for system databases (Grafana, Alerta, Harbor, Keycloak, SeaweedFS) to `17.7-standard-trixie` across chart templates and `values.yaml`, and ships migration 37 to patch existing CNPG Cluster `imageName` fields to the same variant (handling unset, any PG 17 tag, and bare-version tags). This prevents CNPG from defaulting to PostgreSQL 18 and locks system databases to the trixie variant consistent with the monitoring stack requirements *(related backports shipped in v1.2.1 via #2309 and v1.2.2 via #2364)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2369).
* **[platform] Prevent installed packages deletion**: Adds the `helm.sh/resource-policy: keep` annotation to platform packages so disabling a package no longer triggers automatic Helm deletion, restoring the documented behavior where operators must explicitly delete a package *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2273).
* **[mariadb] Always enable replication for consistent service naming**: MariaDB now always enables replication, creating `-primary`/`-secondary` services even for single-replica instances. This fixes dashboard visibility and backup functionality for single-replica setups ([**@sircthulhu**](https://github.com/sircthulhu) in #2279).
* **[hack] Add host runtime preflight check**: New `check-host-runtime.sh` script and `make preflight` target that warns operators when a standalone containerd or docker runtime is running alongside the embedded k3s runtime, helping diagnose container-runtime conflicts early in an installation ([**@lexfrei**](https://github.com/lexfrei) in #2371).
* **[hack] Add check-readiness.sh diagnostic script**: A new diagnostic script for tracking platform reconciliation by checking readiness of Packages, ArtifactGenerators, ExternalArtifacts, and HelmReleases, with support for watch mode and continuous monitoring ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2294).
* **[platform] Add resourcePreset labels to WorkloadMonitor labels**: WorkloadMonitor labels with the `workloads.cozystack.io/` prefix are now propagated onto created Workloads; created Workloads always include the reserved `workloads.cozystack.io/monitor` label, and Helm app charts add `workloads.cozystack.io/resource-preset` metadata to WorkloadMonitor manifests, enabling downstream billing pipelines to correlate monitors with the tenant preset that produced them ([**@androndo**](https://github.com/androndo) in #2416).
## Bug Fixes
* **[platform] Migrate ACME HTTP-01 to ingressClassName API**: Switches ACME HTTP-01 issuance from the deprecated `acme.cert-manager.io/http01-ingress-class` annotation to the modern `ingressClassName` field on `ClusterIssuer` and solver pods. Previously, ClusterIssuers referenced a non-existent `nginx` class while each Ingress individually overrode it via annotation — producing `ingressClassName and class cannot be set at the same time` errors when tenants attempted to migrate to the modern field. The migration is atomic: both the ClusterIssuer and consuming Ingresses are updated together *(backported to v1.2.4)* ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2436).
* **[harbor] Remove incorrect tenant module flags**: Harbor is a PaaS service, not a tenant module. Incorrect `spec.dashboard.module: true` and `internal.cozystack.io/tenantmodule` flags caused Harbor to appear in the sidebar "Modules" section and be misclassified by controllers handling tenant modules. The flags are now removed so Harbor is displayed in its proper PaaS category and is no longer treated as a tenant-scoped HelmRelease ([**@kvaps**](https://github.com/kvaps) in #2444).
* **[kube-ovn] Resolve kubeovn-plunger RBAC forbidden on deployments**: Grants `kube-ovn-plunger` the RBAC needed to list Deployments so it can reconcile `ovn-central`, fixing `deployments.apps is forbidden` errors in `cozy-kubeovn` ([**@kvaps**](https://github.com/kvaps) in #2441).
* **[cilium] Opt-out of cri-containerd.apparmor.d for nsenter init containers**: Opts cilium-agent init containers out of the `cri-containerd.apparmor.d` AppArmor profile on non-Talos variants (`cilium-generic`, `kubeovn-cilium-generic`), fixing `Init:CrashLoopBackOff` on Ubuntu 22.04+ and Debian where the profile denies `nsenter` namespace entry. Talos variants are untouched as Talos does not load the AppArmor LSM *(backported to v1.2.2)* ([**@lexfrei**](https://github.com/lexfrei) in #2370).
* **[virtual-machine] Exclude external VM services from Cilium BPF LB**: Adds the `service.kubernetes.io/service-proxy-name: cozy-proxy` label to VM LoadBalancer services with `external: true`, telling Cilium to skip BPF processing entirely. Fixes inter-tenant connectivity via public LB IPs (Cilium's DNAT caused cross-tenant pod-to-pod flow classification, triggering CiliumClusterwideNetworkPolicy blocks) and restores WholeIP behavior on Cilium 1.19+ where wildcard service drop entries previously blocked traffic to LB IPs on undeclared ports *(backported to v1.2.2)* ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2357).
* **[monitoring] Fix infra dashboards missing in default variant**: Includes the `cozy-monitoring` namespace in the dashboard rendering condition, fixing infrastructure Grafana dashboards not rendering in the default platform variant (only the `tenant-root` namespace was previously checked) *(backported to v1.2.2)* ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #2365).
* **[build] Filter git describe to match only v* tags**: Adds `--match 'v*'` to all `git describe` calls in `hack/common-envs.mk`, preventing the `api/apps/v1alpha1/vX.Y.Z` subtag from being picked up instead of the release tag and producing invalid Docker image tags *(backported to v1.2.2)* ([**@kvaps**](https://github.com/kvaps) in #2386).
* **[platform] Fix resource allocation ratios not propagated to packages**: Restores propagation of `cpuAllocationRatio`, `memoryAllocationRatio`, and `ephemeralStorageAllocationRatio` from `platform/values.yaml` to the `cozystack-values` Secret that managed applications and KubeVirt read, fixing a regression introduced in the bundle restructure that silently ignored operator-configured ratios *(backported to v1.2.1)* ([**@sircthulhu**](https://github.com/sircthulhu) in #2296).
* **[kubernetes] Set explicit ephemeral-storage on virt-launcher pods**: Sets explicit `domain.resources` ephemeral-storage on the VirtualMachine spec to prevent virt-launcher pods from being evicted because LimitRange defaults were too small for the actual emptyDisk capacity *(backported to v1.2.3)* ([**@kvaps**](https://github.com/kvaps) in #2317).
* **[multus] Pin master CNI to 05-cilium.conflist**: Prevents a boot-time race where multus could auto-detect kube-ovn's conflist instead of Cilium's, which would cause pods to bypass the Cilium chain entirely and lose their endpoint *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2315).
* **[multus] Build custom image with DEL cache fix**: Fixes sandbox cleanup deadlock when CNI ADD never completes, preventing stale sandbox name reservations from permanently blocking pod creation after a node disruption *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2313).
* **[linstor] Set verify-alg to crc32c**: Prevents DRBD connection failures on kernels where `crct10dif` is unavailable (e.g., Talos v1.12.6 with kernel 6.18.18) by setting the LINSTOR verify-alg controller default to `crc32c`*(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2303).
* **[linstor] Preserve TCP ports during toggle-disk operations**: Saves existing TCP ports into the `LayerPayload` before `removeLayerData()` deletes them, preventing DRBD resources from entering StandAlone state when a satellite misses the resulting update *(backported to v1.2.1)* ([**@kvaps**](https://github.com/kvaps) in #2292).
* **[linstor] Increase satellite startup probe failure threshold**: Raises the LINSTOR satellite `startupProbe``failureThreshold` from 3 to 30 (30s → 300s) in the `LinstorSatelliteConfiguration` pod template, giving satellites with slow storage initialization enough time to come up without being killed and restarted ([**@Arsolitt**](https://github.com/Arsolitt) in #2425).
## Security
* **docs: add SECURITY.md**: Adds vulnerability reporting procedures, disclosure expectations, and supported release lines ([**@kvaps**](https://github.com/kvaps) in #2230).
* **docs: add OpenSSF Best Practices badge to README**: Adds the OpenSSF Best Practices passing badge to the project README ([**@lexfrei**](https://github.com/lexfrei) in #2320).
## Dependencies & Version Updates
* **[kube-ovn] Bump kube-ovn to v1.15.10 with port-group regression fix**: Updates `packages/system/kubeovn` to upstream v1.15.10 (from v1.15.3) and carries a patch for `pkg/controller/pod.go` that preserves a VM LSP's port-group memberships when Kubernetes GCs a completed virt-launcher pod while another virt-launcher pod of the same VM is still running. Without the patch, the destination pod of a successful live migration lost its security groups, network policies, and node-scoped routing until `kube-ovn-controller` was restarted ([**@kvaps**](https://github.com/kvaps) in #2443).
* **[monitoring] Upgrade victoria-metrics-operator to v0.68.4**: Bumps the vendored `victoria-metrics-operator` Helm chart from 0.59.1 to 0.61.0 (operator appVersion v0.68.1 → v0.68.4), picking up upstream fixes for `VMPodScrape` port routing on VMAgent/VLAgent and `StatefulSet` pod deletion (not eviction) when `maxUnavailable=100%` ([**@lexfrei**](https://github.com/lexfrei) in #2426).
* **[linstor] Update piraeus-server to v1.33.2 with selected backports**: Bumps LINSTOR server from v1.33.1 to v1.33.2 with backported patches for stale bitmap adjust retry, LUKS2 header sizing, optimal I/O size detection, and the maintainer implementation. All patches verified against upstream v1.33.2 with `git apply --check` and `gradlew compileJava`*(backported to v1.2.2)* ([**@kvaps**](https://github.com/kvaps) in #2331).
* **[kamaji] Update to 26.3.5-edge, drop upstreamed patches**: Updates Kamaji from edge-26.2.4 to 26.3.5-edge and removes two patches accepted upstream. Adds configurable probe tuning and DataStore readiness conditions ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2260).
* **[talm] Release v0.23.0, v0.23.1, v0.24.0** (github.com/cozystack/talm): Migrates to the Talos v1.12 multi-document machine config format ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#116); renders templates online in `apply` to resolve lookups ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/talm#119); bumps dependencies and modernizes the codebase ([**@lexfrei**](https://github.com/lexfrei) in cozystack/talm#124).
* **[ansible-cozystack] Release v1.2.1, v1.2.2, v1.2.4** (github.com/cozystack/ansible-cozystack): Exposes `publishing.externalIPs` and tenant-root ingress via role variables ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#30); adds a comprehensive node prerequisites audit ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#27); replaces `ansible.utils.ipaddr` with a stdlib-based test plugin ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#24); adds `v` prefix to collection version in requirements.yml examples ([**@lexfrei**](https://github.com/lexfrei) in cozystack/ansible-cozystack#23); tracks installer releases v1.2.1 through v1.2.4 ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#20, #22, #29, #31, #32).
## Development, Testing, and CI/CD
* **[ci] Replace cozystack-bot PAT with cozystack-ci GitHub App**: Replaces the long-lived `cozystack-bot` personal access token with short-lived, scoped tokens from the `cozystack-ci` GitHub App across all release workflows (`tags.yaml`, `auto-release.yaml`, `pull-requests-release.yaml`), improving security and auditability of CI operations ([**@tym83**](https://github.com/tym83) in #2351; [**@kvaps**](https://github.com/kvaps) in #2383, #2392).
* **[ci] Add Gemini Code Assist and CodeRabbit configuration**: Adds repository-level configuration for AI code reviewers with ignore patterns for vendored/generated code and incremental review settings ([**@lexfrei**](https://github.com/lexfrei) in #2385).
* **[ci] Promote next/ trunk on new minor/major releases**: Updates `update-website-docs` in `tags.yaml` to match the new docs-versioning contract — the website repo replaces the old "pre-create `vX.Y/` draft directory" scheme with a permanent `content/en/docs/next/` trunk, and released version directories are promoted explicitly by the release workflow ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2433).
* **[tests] Fix Kafka E2E test timeout and retry race condition**: Increases Kafka E2E test timeout from 60s to 300s and fixes a retry race where `kubectl apply` could hit a still-deleting resource ([**@lexfrei**](https://github.com/lexfrei) in #2358).
* **docs: adopt Conventional Commits for commit and PR titles**: Standardizes commit and PR title format to `type(scope): description` across all contributing docs and the PR template ([**@lexfrei**](https://github.com/lexfrei) in #2395).
* **docs(ci): require screenshots for UI changes in PR template**: Adds a mandatory screenshots section to the PR template for UI-related changes ([**@kitsunoff**](https://github.com/kitsunoff) in #2407).
* **chore(maintenance): add @myasnikovdaniil to CODEOWNERS**: Adds @myasnikovdaniil to the default owners in `.github/CODEOWNERS` for automatic review requests ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2434).
## Documentation
* **[website] Add ApplicationDefinition naming convention reference**: Documents how `cozystack-api` resolves kinds to their backing definitions ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#478).
* **[website] Document Talos / talosctl / Cozystack version pairing**: Adds a version compatibility matrix for installation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#484).
* **[website] Document namespace layout and parent/child derivation**: Explains tenant namespace hierarchy and parent/child namespace derivation rules ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#479).
* **[website] Document the checkbox-then-edit-CR customization pattern for tenants**: Describes the workflow for customizing tenant settings via the CR after initial checkbox-based creation ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#485).
* **[website] Add custom Keycloak themes documentation**: Covers the theme image contract, configuration, `imagePullSecrets`, and theme activation in the Keycloak admin console ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#463).
* **[website] Add bonding (LACP) configuration how-to guide**: Covers network bonding configuration for Cozystack installations ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#459).
* **[website] Improve registry mirrors for tenant Kubernetes in air-gapped guide**: Improves documentation for configuring registry mirrors in air-gapped environments ([**@sircthulhu**](https://github.com/sircthulhu) in cozystack/website#461).
* **[website] Rewrite guide for ApplicationDefinition API (external-apps)**: Comprehensive rewrite of the external apps guide using the ApplicationDefinition API with Minecraft server examples ([**@kitsunoff**](https://github.com/kitsunoff) in cozystack/website#488).
* **[website] Add documentation for Go types usage**: Guide for using generated Go types for Cozystack managed applications as a Go module ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#465).
* **[website] Update backup/restore documentation for VMI/VMDisk**: Updates backup documentation with VM instance and VM disk restore improvements ([**@androndo**](https://github.com/androndo) in cozystack/website#466).
* **[website] Refactor docs versions to major.minor variants**: Moves docs to major.minor versioning for the v1.x series ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#477).
* **[website] Trunk-based versioning with permanent next/ directory**: Replaces the old "pre-create `vX.Y/` draft directory" scheme with a permanent `content/en/docs/next/` trunk; released version directories are promoted explicitly by `hack/release_next.sh` on new minor/major releases, and routing between `next/` and `vX.Y/` is Makefile-driven ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#495).
* **[website] Add updated OpenAPI spec**: Updates the OpenAPI specification for managed applications reference ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#469).
* **[website] Add OpenAPI spec download to GitHub Pages build**: Fixes the GitHub Pages build to include the OpenAPI spec download ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#494).
* **[website] Add OSS Health pages and OpenSSF badge**: Adds OSS Health section with OpenSSF Scorecard and Best Practices badges to the website ([**@tym83**](https://github.com/tym83) in cozystack/website#470).
* **[website] Add Telemetry page under OSS Health section**: Adds the Telemetry page with initial data seeding to the OSS Health docs ([**@tym83**](https://github.com/tym83) in cozystack/website#471, cozystack/website#504).
* **[website] Blog: OSS Health section launch announcement**: Publishes the announcement blog post for the OSS Health section ([**@tym83**](https://github.com/tym83) in cozystack/website#474).
* **[website] Fix OpenSSF canonical status URL**: Changes the OpenSSF canonical status URL from pt-BR to en ([**@tym83**](https://github.com/tym83) in cozystack/website#475).
* **[website] Add CozySummit Virtual 2026 program announcement**: Publishes the CozySummit Virtual 2026 program announcement blog post ([**@tym83**](https://github.com/tym83) in cozystack/website#472).
* **[website] Add missing release announcements for v0.1–v0.41**: Backfills missing release announcement blog posts for historical Cozystack versions ([**@tym83**](https://github.com/tym83) in cozystack/website#468).
* **[website] Blog: managed PostgreSQL with synchronous replication**: Adds a post covering the managed PostgreSQL synchronous-replication feature ([**@tym83**](https://github.com/tym83) in cozystack/website#497).
* **[website] Blog taxonomies and client-side filter UI**: Registers article-type and topic taxonomies and adds a client-side filter on the blog list page ([**@tym83**](https://github.com/tym83) in cozystack/website#499).
* **[website] Add images frontmatter for social preview on existing posts**: Adds images frontmatter for social preview on existing blog posts ([**@tym83**](https://github.com/tym83) in cozystack/website#498).
* **[website] Fix broken links and stale anchors across v1 docs**: Fixes 14 broken links and stale talm anchors ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#486).
* **[website] Prefix bundle package names with cozystack. in v1 examples**: Corrects package naming in documentation examples ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#482).
* **[website] Finish isolated-field removal and document opt-in policy labels**: Removes the obsolete `isolated` field from tenant documentation and documents the new opt-in policy labels approach ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#481).
* **[website] Add --take-ownership flag and describe networking.* fields**: Documents the `--take-ownership` flag and `networking.*` fields in the installation guide ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#480).
* **[website] Fix KubeOVN MASTER_NODES example path and key in troubleshooting**: Corrects the MASTER_NODES example path and key ([**@lexfrei**](https://github.com/lexfrei) in cozystack/website#483).
* **[website] Add CLAUDE.md for AI agent guidance**: Adds a CLAUDE.md file describing the trunk-based docs architecture for AI agent guidance ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#489).
* **[website] Update /docs/v1/ redirect to latest v1.2**: Updates the `/docs/v1/` redirect target to point to the latest v1.2 docs on GitHub Pages ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#492).
* **[website] Remove nbykov from CODEOWNERS and CLAUDE.md**: Cleans up CODEOWNERS and CLAUDE.md entries ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#491).
* **[website] Add Ahrefs Analytics tracker**: Adds the Ahrefs Analytics tracker to the website ([**@tym83**](https://github.com/tym83) in cozystack/website#503).
* **[website] Add breathing room between navbar and hero on OSS Health**: Minor styling fix for the OSS Health section ([**@tym83**](https://github.com/tym83) in cozystack/website#500).
* **[website] Fix og social badge image and title**: Updates the social badge image and title ([**@tym83**](https://github.com/tym83) in cozystack/website#487).
* **[website] Update managed apps reference for v1.2.1**: Automated managed-apps reference update ([**@cozystack-bot**](https://github.com/cozystack-bot) in cozystack/website#464).
* **[external-apps-example] Replace MongoDB example with Minecraft apps**: Refactors the external apps example to use the ApplicationDefinition API with Minecraft server applications ([**@lexfrei**](https://github.com/lexfrei) in cozystack/external-apps-example#2).
* **docs: update README introductory description**: Refines the platform positioning and improves clarity on core capabilities in the main README ([**@tym83**](https://github.com/tym83) in #2409).
## Governance
* **Add Mattia Eleuteri ([@mattia-eleuteri](https://github.com/mattia-eleuteri)) as Maintainer**: CSI, Storage, Networking & Security ([**@tym83**](https://github.com/tym83) in #2345).
* **Add Matthieu Robin ([@matthieu-robin](https://github.com/matthieu-robin)) as Maintainer**: Managed applications, platform quality, and benchmarking ([**@tym83**](https://github.com/tym83) in #2346).
## Contributors
We'd like to thank all contributors who made this release possible:
Patch release covering a TenantNamespace IDOR fix in the API, a destructive `post-upgrade` hook removed from the etcd chart, kamaji controller stability, a `linstor-csi` bump that fixes live migration on Protocol-A/B DRBD resources, the missing `linstor-gui` build wiring, and a velero RBAC fix that unblocked installs on bundles without Velero.
## Security
* **fix(api): prevent IDOR in TenantNamespace Get and Watch handlers**: Two IDOR (Insecure Direct Object Reference) vulnerabilities allowed authenticated users to read TenantNamespace metadata they had no RoleBinding for. The `Get` and `Watch` handlers now go through a new `hasAccessToNamespace()` helper that lists RoleBindings scoped only to the target namespace (orders of magnitude faster than the previous all-cluster scan), returns `NotFound` instead of leaking existence on unauthorized access, and applies the same check on the `Watch` filter path. Includes regression tests for the unauthorized paths. ([**@IvanHunters**](https://github.com/IvanHunters) in #2471, backport #2524)
## Features
* **feat(linstor): bump linstor-csi to v1.10.6 with Protocol-C dual-attach fix**: Live migration of KubeVirt VMs on Protocol-A/B (async) DRBD volumes no longer fails with `Protocol C required`. `linstor-csi` v1.10.6 now installs a `Protocol=C` override on the resource-definition during dual-attach and reverts it on detach, so `replicated-async` StorageClasses and other Protocol-A/B resource groups support live migration without manual `drbdadm adjust` intervention. ([**@kvaps**](https://github.com/kvaps) in #2496, backport #2505)
## Fixes
* **fix(backups): move velero-configmap Role to velero chart**: The `backupstrategy-controller` (a default package) declared a Role/RoleBinding scoped to the `cozy-velero` namespace for managing `ResourceModifier` ConfigMaps. On bundles where Velero was not enabled, that namespace did not exist and the HelmRelease failed with `namespaces "cozy-velero" not found`, blocking installation. The Role/RoleBinding now lives in the velero chart, so it is created only when velero is actually deployed. ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2459, backport #2467)
* **fix(etcd): remove destructive post-upgrade cert-regeneration hook**: The etcd chart ran a `post-upgrade` Helm hook on every upgrade that deleted etcd TLS Secrets (`etcd-ca-tls`, `etcd-peer-ca-tls`, `etcd-client-tls`, `etcd-peer-tls`, `etcd-server-tls`) and then deleted etcd pods, forcing cert-manager to re-issue the entire etcd CA chain. On clusters with Kamaji-managed tenant control planes this put every tenant `kube-apiserver` into CrashLoopBackOff until each DataStore was manually re-reconciled. The hook was a one-shot `2.6.0 → 2.6.1` migration that became a permanent footgun once chart versioning moved to `0.0.0+<git-hash>` (always `< 2.6.1` per semver) and after the underlying `rotationPolicy: Always` issue was fixed in `47d81f70`. The hook is now removed entirely. ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2462, backport #2511)
* **fix(kamaji): increase memory limits and add startup probe**: The kamaji controller frequently entered CrashLoopBackOff due to OOMKills (exit 137) within ~20–25 seconds of startup, with the readiness probe failing while the controller was still finishing initialization. Memory limit raised from 500Mi to 512Mi, request from 100Mi to 256Mi, and a 60-second startup probe (12 attempts × 5s periods) is added so the controller has room to boot before liveness/readiness probes engage. ([**@IvanHunters**](https://github.com/IvanHunters) in #2421, backport #2491)
## Build
* **build(linstor): include linstor-gui in root image build target**: The `linstor-gui` package (added in #2382) was never wired into the root `Makefile`'s `build:` target, so CI never built or published the image. `ghcr.io/cozystack/cozystack/linstor-gui` returned `NAME_UNKNOWN` and `values.yaml` stayed pinned to `tag: 2.3.0` without a digest. The missing build line is added so the next CI run publishes the image and the per-package `Makefile` digest-pins `values.yaml` automatically. ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #2498, backport #2518)
## Contributors
Thanks to everyone who contributed to this patch release:
This scenario walks through the cluster-level setup required before users can
back up and restore their virtual machines. A cluster administrator creates
Velero strategies and a BackupClass that binds those strategies to Cozystack
application types.
## Prerequisites
- A running Cozystack cluster with the backup-controller installed.
- Velero deployed in the `cozy-velero` namespace with a configured
BackupStorageLocation (default: `default`).
- `kubectl` access with cluster-admin privileges.
## Steps
### 1. Create Velero Strategies
```bash
./01-create-strategies.sh
```
This creates two cluster-scoped `Velero` strategy objects:
| Strategy | Purpose |
|---|---|
| `vminstance-strategy` | Defines backup and restore templates for `VMInstance` applications. Includes VirtualMachine, PVCs, HelmReleases, pods (required by kubevirt-velero-plugin), and supporting resources. Uses `snapshotMoveData: true` for portable volume snapshots. |
| `vmdisk-strategy` | Defines backup and restore templates for standalone `VMDisk` applications. Covers HelmReleases, PVCs, and ConfigMaps. |
Both strategies use Go templates with `{{ .Application.metadata.name }}`,
`{{ .Application.metadata.namespace }}`, and `{{ .Parameters.backupStorageLocationName }}`
so they can be reused across any application instance.
**Key design details:**
- `orLabelSelectors` scope backups to only the resources belonging to a
specific application, preventing cross-contamination between VMs in the
same namespace.
- The restore spec excludes `datavolumes.cdi.kubevirt.io` to avoid conflicts
when the HelmRelease of VMDisk recreates DataVolumes after restore.
- `existingResourcePolicy: update` allows restoring over existing resources
during in-place restore.
### 2. Create BackupClass
```bash
./02-create-backupclass.sh
```
Creates a `BackupClass` named `velero` that maps application types to their
# Pinned to the DCGM Exporter image tag shipped by the gpu-operator
# chart under packages/system/gpu-operator/charts/gpu-operator/values.yaml
# (dcgmExporter.version = 4.1.1-4.0.4-ubuntu22.04). When that image is
# bumped, refresh this file from the matching tag in the NVIDIA/dcgm-exporter
# repo and re-run `./hack/cozytest.sh hack/check-gpu-recording-rules.bats`.
# Format
# If line starts with a '#' it is considered a comment
# DCGM FIELD, Prometheus metric type, help message
# Clocks
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
# Temperature
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
# Power
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).
# PCIE
# DCGM_FI_PROF_PCIE_TX_BYTES, counter, Total number of bytes transmitted through PCIe TX via NVML.
# DCGM_FI_PROF_PCIE_RX_BYTES, counter, Total number of bytes received through PCIe RX via NVML.
DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.
# Utilization (the sample period varies depending on the product)
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
DCGM_FI_DEV_ENC_UTIL, gauge, Encoder utilization (in %).
DCGM_FI_DEV_DEC_UTIL , gauge, Decoder utilization (in %).
# Errors and violations
DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
# DCGM_FI_DEV_POWER_VIOLATION, counter, Throttling duration due to power constraints (in us).
# DCGM_FI_DEV_THERMAL_VIOLATION, counter, Throttling duration due to thermal constraints (in us).
# DCGM_FI_DEV_SYNC_BOOST_VIOLATION, counter, Throttling duration due to sync-boost constraints (in us).
# DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in us).
# DCGM_FI_DEV_LOW_UTIL_VIOLATION, counter, Throttling duration due to low utilization (in us).
# DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in us).
# Memory usage
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
# ECC
# DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, Total number of single-bit volatile ECC errors.
# DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total number of double-bit volatile ECC errors.
# DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, counter, Total number of single-bit persistent ECC errors.
# DCGM_FI_DEV_ECC_DBE_AGG_TOTAL, counter, Total number of double-bit persistent ECC errors.
# Retired pages
# DCGM_FI_DEV_RETIRED_SBE, counter, Total number of retired pages due to single-bit errors.
# DCGM_FI_DEV_RETIRED_DBE, counter, Total number of retired pages due to double-bit errors.
# DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.
# NVLink
# DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total number of NVLink flow-control CRC errors.
# DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL, counter, Total number of NVLink data CRC errors.
# DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL, counter, Total number of NVLink retries.
# DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, Total number of NVLink recovery errors.
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, counter, Total number of NVLink bandwidth counters for all lanes.
# DCGM_FI_DEV_NVLINK_BANDWIDTH_L0, counter, The number of bytes of active NVLink rx or tx data including both header and payload.
# VGPU License status
DCGM_FI_DEV_VGPU_LICENSE_STATUS, gauge, vGPU License status
# Remapped rows
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for correctable errors
DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, Whether remapping of rows has failed
# Static configuration information. These appear as labels on the other metrics
DCGM_FI_DRIVER_VERSION, label, Driver Version
# DCGM_FI_NVML_VERSION, label, NVML Version
# DCGM_FI_DEV_BRAND, label, Device Brand
# DCGM_FI_DEV_SERIAL, label, Device Serial Number
# DCGM_FI_DEV_OEM_INFOROM_VER, label, OEM inforom version
# DCGM_FI_DEV_ECC_INFOROM_VER, label, ECC inforom version
# DCGM_FI_DEV_POWER_INFOROM_VER, label, Power management object inforom version
# DCGM_FI_DEV_INFOROM_IMAGE_VER, label, Inforom image version
# DCGM_FI_DEV_VBIOS_VERSION, label, VBIOS version of the device
# Datacenter Profiling (DCP) metrics
# NOTE: supported on Nvidia datacenter Volta GPUs and newer
DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, Ratio of time the graphics engine is active.
# DCGM_FI_PROF_SM_ACTIVE, gauge, The ratio of cycles an SM has at least 1 warp assigned.
# DCGM_FI_PROF_SM_OCCUPANCY, gauge, The ratio of number of warps resident on an SM.
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
DCGM_FI_PROF_DRAM_ACTIVE, gauge, Ratio of cycles the device memory interface is active sending or receiving data.
# DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles the fp64 pipes are active.
# DCGM_FI_PROF_PIPE_FP32_ACTIVE, gauge, Ratio of cycles the fp32 pipes are active.
# DCGM_FI_PROF_PIPE_FP16_ACTIVE, gauge, Ratio of cycles the fp16 pipes are active.
DCGM_FI_PROF_PCIE_TX_BYTES, gauge, The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
DCGM_FI_PROF_PCIE_RX_BYTES, gauge, The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
Can't render this file because it has a wrong number of fields in line 12.
