Add secret-hash annotation to the dashboard KeycloakClient CRD resource
so that when the Kubernetes Secret value changes, the operator detects
the CRD update and reconciles the client secret in Keycloak.
Without this annotation, if the dashboard-client Secret is recreated
with a new value (e.g. after upgrade), the KeycloakClient spec remains
unchanged, the operator skips reconciliation, and Keycloak retains the
stale secret — causing authentication failures for the dashboard.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit bb5ee3ea4a)
Without concurrencyPolicy and job limits, the defrag CronJob can
accumulate hundreds of running/failed pods during cluster upgrades
when etcd is temporarily unavailable. This was observed after
upgrading to v1.1.2 where defrag jobs piled up across tenants.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit ed8ba3beec)
When an NFS-backed RWX volume is published to multiple VMs, the
CiliumNetworkPolicy egress rule only allowed traffic from the first VM.
The endpointSelector.matchLabels was set once on creation and never
broadened, causing NFS mounts to hang on all nodes except the first.
Switch from matchLabels to matchExpressions (operator: In) so the
selector can list multiple VM names. Rebuild the selector whenever
ownerReferences are added or removed.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit cc5ec0b7a3)
Kube-OVN reads MAC address exclusively from the pod annotation
ovn.kubernetes.io/mac_address, not from the IP resource spec.macAddress.
Without pod-level annotations, migrated VMs receive a new random MAC,
breaking OS-level network config that matches by MAC (e.g. netplan).
Add a Helm lookup for the Kube-OVN IP resource in the vm-instance chart
template. When the IP resource exists, its macAddress and ipAddress are
automatically injected as pod annotations. This removes the need for
fragile Flux postRenderers on the HelmRelease — the chart itself handles
MAC/IP preservation based on actual cluster state.
Remove the postRenderers approach from migration 29 since the chart now
handles this natively.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 9a4f49238c)
The gcr.io/kubebuilder/kube-rbac-proxy image is no longer available
since GCR was deprecated. Replace it with quay.io/brancz/kube-rbac-proxy
from the original upstream author.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
(cherry picked from commit 4946383cf1)
Use a startupProbe to defer liveness/readiness checks until Keycloak
has fully started, instead of relying on initialDelaySeconds. This is
more robust for applications with variable startup times.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit d18ed79382)
Keycloak 26.x exposes dedicated health endpoints on the management
port (9000) via /health/live and /health/ready. The previous probes
used GET / on port 8080 which redirects to the configured KC_HOSTNAME
(HTTPS), causing kubelet to fail the probe with "Probe terminated
redirects" and eventually kill the pod in a crashloop.
Changes:
- Add KC_HEALTH_ENABLED=true to activate health endpoints
- Expose management port 9000 in container ports
- Switch liveness probe to /health/live on port 9000
- Switch readiness probe to /health/ready on port 9000
- Increase failure thresholds for more tolerance during startup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit 0873691913)
Add onCopy handler to SecretBase64Plain inputs to intercept native browser
copy events and explicitly write the full decoded text (including newlines)
to the clipboard. Without this, input[type=text] strips newlines on copy.
Upstream PR: https://github.com/PRO-Robotech/openapi-k8s-toolkit/pull/339
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 99ee0e34bf)
Update openapi-k8s-toolkit commit to d6b9e4ad (release/1.4.0) which
includes the FormListInput layout refactor, making formlistinput-value-binding.diff
obsolete.
Set allowEmpty: true on the VMInstance instanceType field so users can
explicitly clear the selection and override the default instance type.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 6e8ce65e49)
Update openapi-k8s-toolkit to release/1.4.0 (d6b9e4ad). The previous
value-binding layout refactor is already included upstream, so drop the
formlistinput-value-binding.diff patch.
Add formlistinput-allow-empty.diff patch which introduces two props to
the listInput component:
- allowEmpty: when set, auto-persists the field so BFF sends an empty
value instead of falling back to the schema default
- persistType: controls the type of empty value ('str' | 'number' | 'arr'
| 'obj'), allowing the feature to work correctly for any field type
Set allowEmpty: true on the VMInstance instanceType field so users can
explicitly clear the selection and override the default instance type.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit a3ccb4f87d)
KC_PROXY=edge was deprecated and removed in Keycloak 26.x, causing
"Non-secure context detected" warnings and broken cookie handling
behind reverse proxy. Replace with KC_PROXY_HEADERS=xforwarded and
KC_HTTP_ENABLED=true.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 14228aa0d7)
The dashboard BFF service account (incloud-web-web) falls under the
default "service-accounts" FlowSchema which maps to the "workload-low"
priority level. Under load, this causes API Priority and Fairness to
return 429 (Too Many Requests) responses to the BFF, resulting in 500
errors for dashboard users.
Add a FlowSchema that maps the BFF service account to the "exempt"
priority level to prevent APF throttling of dashboard API requests.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
(cherry picked from commit 161b5be8c2)
Add helm.sh/resource-policy=keep annotation to the cozy-system Namespace
in the installer helm chart. This prevents Helm from deleting the
namespace when the HelmRelease is removed, which would otherwise destroy
all other HelmReleases within it.
Update the migration script to annotate the cozy-system namespace and
cozystack-version ConfigMap with helm.sh/resource-policy=keep before
generating the Package resource.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
(cherry picked from commit c83e41ea14)
Remove stock-instance-* sidebars that were populated with namespace-scoped
menu items, causing the sidebar to incorrectly appear on cluster-level pages.
Clear the sidebar fallback ID so the frontend gracefully renders no sidebar
when no matching sidebar resource exists.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
## What this PR does
Add `ingress.host` field to cozy-keycloak values, allowing users to
override the default
`keycloak.<root-host>` Ingress hostname. The custom hostname is applied
to both the Ingress
resource and the `KC_HOSTNAME` environment variable in the StatefulSet.
When left empty,
behavior is unchanged (backward compatible).
### Release note
```release-note
[system] Add `ingress.host` option to cozy-keycloak for custom Ingress hostname override
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Keycloak ingress hostname is now configurable and automatically
defaults to "keycloak.<root-host>" when not explicitly specified.
* **Chores**
* Refactored hostname configuration across Keycloak templates to use
dynamic variable resolution instead of hard-coded values for improved
consistency and flexibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add a safety-net migration for users who already passed migrations 28/29
and still have orphaned -rd HelmReleases in cozy-system:
- ferretdb-rd (replaced by mongodb-rd, never had a dedicated migration)
- mysql-rd (migration 28 only handled user HRs)
- virtual-machine-rd (migration 29 only handled user HRs)
These HRs reference ExternalArtifacts that no longer exist after the
application renames (ferretdb→mongodb, mysql→mariadb,
virtual-machine→vm-disk+vm-instance), causing persistent errors.
Bump targetVersion from 33 to 34.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migrations 28 (mysql→mariadb) and 29 (virtual-machine→vm-disk+vm-instance)
only handled user-facing HelmReleases but left the system-level -rd
HelmReleases (mysql-rd, virtual-machine-rd) orphaned in cozy-system.
These HRs reference ExternalArtifacts that no longer exist, causing
persistent reconciliation failures.
Add cleanup steps to delete the orphaned -rd HRs and their Helm secrets.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add ingress.host field to values.yaml for cozy-keycloak. When set,
it overrides the default "keycloak.<root-host>" hostname in both the
Ingress resource and the KC_HOSTNAME environment variable. When left
empty, behavior is unchanged for backward compatibility.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The protection-webhook is not part of the cozystack platform and should
not be managed by the migration script. Old services are now deleted
directly instead of being batched through the webhook disable/enable cycle.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migration 26 was using apps.cozystack.io/application.kind=Monitoring label
which is added by migration 22 and may not be present on v0.41.1 clusters.
Switch to cozystack.io/ui=true (guaranteed on old HRs) with field-selector
for exact name match.
Also remove redundant bundle enabled flags from migrate-to-version-1.0.sh
since the variant already determines them via its values file.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The post-upgrade hook deletes TLS certificates and etcd pods to trigger
cert-manager regeneration. With 3 replicas and startup probes allowing
up to 25 minutes per pod, the previous 10m timeout was insufficient.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migration 27 failed with set -e when Piraeus CRDs did not exist on
clusters without linstor. Add existence check before annotating CRDs.
Also add name-pattern fallback for helm secret deletion, consistent
with migration 26.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
grep returns exit code 1 when no lines match. With set -euo pipefail,
this kills the script when all secrets are helm-release secrets or when
no matching resources exist. Wrap grep calls with { ... || true; }.
Also fix reconcile annotation in migration 29 to use RFC3339 timestamp
format instead of Unix epoch, which Flux v2 expects.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migration 26 silently skipped namespace processing when kubectl
queries failed, leaving helm release secrets intact. This caused
helm to diff old vs new chart manifests during upgrade, deleting
VLogs/CNPG resources and their PVCs.
- Remove silent error suppression (2>/dev/null || true) from
namespace discovery and HR suspend commands
- Add fallback secret deletion by name pattern when label selector
does not match
- Add verification that all helm release secrets are deleted
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
PR #2075 introduced `_cluster.cluster-domain` references in
monitoring-agents `values.yaml` for FQDN resolution in tenant clusters.
This broke the fluent-bit subchart because `_cluster` values are not
accessible from the Helm subchart context — only `global` values are
shared with subcharts.
This PR replaces `_cluster` references with a new `global.clusterDomain`
variable:
- Empty by default (management cluster uses short DNS names like
`service.namespace.svc`)
- Set to the management cluster domain (e.g. `cozy.local`) for tenant
clusters, enabling FQDN resolution for cross-cluster service discovery
Fixes#2084
### Release note
```release-note
[system] Fix monitoring-agents installation failure caused by inaccessible _cluster values in fluent-bit subchart context. Introduce global.clusterDomain for proper FQDN resolution in tenant workload clusters.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Monitoring agent configuration updated to support configurable cluster
domain names for greater flexibility.
* Remote write and log-forwarding endpoints adjusted to align with
cluster domain handling, improving compatibility when deploying across
different cluster DNS setups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
PR #2075 added _cluster.cluster-domain references to monitoring-agents
values.yaml for FQDN resolution. This broke fluent-bit because it is
a subchart where _cluster values are not accessible (only global
values are shared with subcharts), causing "index of untyped nil".
Revert values.yaml to short DNS names (.svc) for the management
cluster where they resolve locally, and add FQDN with cluster-domain
suffix in the tenant kubernetes HelmRelease where _cluster values are
available and cross-cluster DNS resolution is needed.
Closes#2084
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Updates kubevirt to v1.6.3 and CDI to v1.64.0.
Please note that VMs would be live-migrated as a part of the update.
### Release note
```release-note
Updated kubevirt to v1.6.3 and CDI to v1.64.0
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VM synchronization support with synchronization ports and address
reporting
* Cluster profiler and synchronization controller developer options
* **Updates**
* CDI operator bumped to v1.64.0; filesystem overhead default increased
* KubeVirt operator bumped to v1.6.3
* Added liveness/readiness probes and health/metrics ports
* Expanded operator tolerations for control-plane/master nodes
* Expanded operator permissions for synchronization and webhook-related
resources
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
KubeVirt v1.6.x has a known issue (#15989) where the guest-console-log
init container blocks virt-launcher pods from starting. The container
runs virt-tail as a long-running sidecar but fails to properly function
as a Kubernetes native sidecar, causing all VM pods to get stuck in
PodInitializing state indefinitely.
Disable serial console logging globally via the KubeVirt CR to prevent
the problematic init container from being created.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Upstream PR clastix/kamaji#1087 (refactor!: datastore conditions) removed
the startup datastore existence check that our disable-datastore-check.diff
patch was working around. Update to the merge commit and drop the now
redundant patch.
Remaining patches:
- fix-kubelet-config-compat.diff (pending upstream PR #1084)
- increase-startup-probe-threshold.diff (no upstream fix)
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Previously `_cluster.clusterissuer` controlled the ACME solver type
using
values `http01` / `cloudflare`, and every ingress template hardcoded
`cert-manager.io/cluster-issuer: letsencrypt-prod` with no way to
override it.
This PR adds new parameters in platform chart:
- `publishing.certificates.solver` (default `http01`)
- `publishing.certificates.issuerName` (default: `letsencrypt-prod`)
instead of single parameter before
- `publishing.certificates.issuerType`
Previous `certificates.issuerType` was renamed to `certificates.solver`;
Also its possible value
`cloudflare` was renamed to `dns01` to use standard ACME terminology.
New `certificates.issuerName` (default: `letsencrypt-prod`) — propagated
as
`_cluster.issuer-name` to all packages via `cozystack-values` then its
value appears in
`cert-manager.io/cluster-issuer` annotation across 8 templates of
ingresses in system applications.
`publishing.certificates.solver` can be set empty to clearly support
`selfsigned-cluster-issuer`,
or have any value, but it can be a bit confusing.
Operators can now point ingresses at any ClusterIssuer (custom ACME,
self-signed, internal CA) by setting `certificates.issuerName` without
touching individual package templates.
## Breaking changes
| What changed | Before | After |
|---|---|---|
| Solver key | `certificates.issuerType` | `certificates.solver` |
| Cloudflare solver value | `issuerType: cloudflare` | `solver: dns01` |
This changes handled by migration when upgrading cozystack from v1
or by `migration-to-v1.0.sh` script (also checked by migration later)
No actions from user needed.
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[platform] Added publishing.certificates.solver (http01/dns01) and
publishing.certificates.issuerName fields to allow configuring ACME challenge
type and ClusterIssuer per installation, replacing the old implicit issuerType field
[platform] Migration script and upgrade hook (migration 32) convert old
clusterissuer/issuerType fields to the new solver/issuerName fields
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Migrated certificate issuer configuration from legacy `issuerType`
field to new `solver` and `issuerName` fields system-wide.
* Automated migration script converts existing configurations, mapping
legacy values (cloudflare, http01) to new format.
* Updated all certificate-related templates to use new configurable
solver and issuer settings with sensible defaults.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Monitoring agents (vmagent, fluent-bit) in tenant workload clusters
failed to deliver metrics and logs because service addresses used short
DNS names (e.g. `vminsert-longterm.tenant-root.svc`) without the cluster
domain suffix. Tenant CoreDNS could not resolve these names across
cluster boundaries.
This PR appends the configured cluster domain from
`_cluster.cluster-domain` to all vmagent remoteWrite URLs and fluent-bit
output hosts, with a fallback to `cluster.local` when not set.
### Release note
```release-note
[system] Fix monitoring-agents endpoints to use FQDN with configured cluster domain, resolving metrics and logs delivery failures in tenant workload clusters.
```
## What this PR does
Updates Kubernetes version support to match current release landscape
and Talos 1.12 compatibility:
- Update Kamaji from `edge-25.4.1` to `edge-26.2.4` (adds K8s 1.35
support)
- Update Kubernetes version matrix: v1.30, v1.31, v1.32, v1.33, v1.34,
v1.35
- Drop EOL versions v1.28 and v1.29
- Remove merged-upstream patch (992.diff — label preservation fix)
- Regenerate disable-datastore-check.diff for new Kamaji version
Changes:
- Default Kubernetes version is now v1.35
- E2E tests will validate v1.35 (latest) and v1.34 (previous)
- Patch versions updated to latest available (v1.35.0, v1.34.4, v1.33.8,
v1.32.12, v1.31.14, v1.30.14)
### Release note
```release-note
[kubernetes] Update supported Kubernetes versions to v1.30-v1.35
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Kamaji CRDs Helm chart with DataStore and KubeconfigGenerator
resources, plus deployment templates and configurable
kubeconfigGenerator settings
* DataStore now supports multiple backends (etcd, MySQL, PostgreSQL,
NATS) with TLS/auth validations and status tracking (observedGeneration)
* **Chores**
* Bumped default Kubernetes version from v1.33 to v1.35 (added v1.34;
removed v1.28–v1.29)
* Updated charts, packaging metadata, README/docs and helm
ignore/Makefile entries; updated builder base image and chart
dependencies
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Kamaji edge-26.2.4 is compiled against Kubernetes 1.35 libraries.
SetDefaults_KubeletConfiguration() from 1.35 injects two fields
gated by feature gates that are not enabled in earlier versions:
- crashLoopBackOff.maxContainerRestartPeriod (KubeletCrashLoopBackOffMax)
- imagePullCredentialsVerificationPolicy (KubeletEnsureSecretPulledImages)
Kubelets < 1.35 reject these fields during configuration validation,
causing worker nodes to fail to join the tenant cluster.
Add a Go patch that clears these fields from the kubelet-config
ConfigMap when the target Kubernetes version is below 1.35.
See: https://github.com/clastix/kamaji/issues/1062
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
Fixed cozy:tenant:admin:base ClusterRole to deny deletion of tenant ResourceQuotas for the tenant admin and superadmin
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Removed resource quota management permissions from tenant admin role
to reduce unnecessary administrative access.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
