Without explicit ephemeral-storage resources on the VirtualMachine spec,
virt-launcher pods inherit default limits from the namespace LimitRange,
which can be much lower than the emptyDisk capacity. This causes kubelet
to evict the pod when the VM's containerd unpacks images exceeding the
limit.
Set domain.resources with ephemeral-storage calculated via cozy-lib
allocation ratio based on the configured ephemeralStorage value.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
ReadDir and Rename errors during migration are now hard failures
instead of warnings, preventing user data from becoming invisible
behind the /data subpath mount. Tolerate os.IsNotExist on Rename
for concurrent publish calls that already moved the file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
- Use os.MkdirTemp instead of predictable path (fixes race condition/TOCTOU)
- Use defer for cleanup (unmount + remove temp dir)
- Handle os.ReadDir and os.Rename errors with klog warnings
- Log unmount failures instead of ignoring them
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Mount the /data subdirectory of the NFS export instead of the root,
so pods no longer see internal LINSTOR/CDI artifacts (disk.img,
lost+found). On first mount after upgrade, any existing user files
at the NFS root are automatically migrated into /data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
When an NFS-backed RWX volume is published to multiple VMs, the
CiliumNetworkPolicy egress rule only allowed traffic from the first VM.
The endpointSelector.matchLabels was set once on creation and never
broadened, causing NFS mounts to hang on all nodes except the first.
Switch from matchLabels to matchExpressions (operator: In) so the
selector can list multiple VM names. Rebuild the selector whenever
ownerReferences are added or removed.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
This PR adds version specific ubuntu base images to fix errors when base
image has new deb packages of kubeadm and kubelet installed, but at
runtime it was downgraded by replacing just binaries. Now update by
replacing binaries works as intended - latest patch version of minor
version used.
Core issue was in kubeadm<1.32 expecting conntrack binary in its
preflight checks but it was not found. It happened because kubelet deb
package dropped conntrack dependency since 1.32 (actually it absent in
1.31.14 too).
So now status of supported tenant k8s versions is:
- 1.30 - works because kubelet package provided conntrack, also
conntrack preflight check ignored (see 1.31).
- 1.31 - works because conntrack preflight check ignored (for some
reason kubelet 1.31.14 did't provide conntrack dependency, unlike
1.31.13 did).
- \>=1.32 - works because conntrack preflight check removed from
`kubeadm init` entirely.
Conntrack preflight check ignoring is legit for tenant kubernetes
clusters because until 1.32 it was used in kube-proxy but cozystack k8s
approach doesn't use kube-proxy (replaced with cilium).
Issue with conntrack may be mitigated with only `ignorePreflightErrors`,
but I think proper base image build will help to avoid similar bugs in
future.
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[kubernetes] Fixed tenant k8s older than 1.32 creation by adding version specific ubuntu base images
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added multi-version Kubernetes support with version-specific container
images.
* Enhanced compatibility with newer Kubernetes releases, including
version 1.31.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.
Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Cilium's MTU auto-detection does not account for VXLAN overhead when
running inside KubeVirt VMs. The VM interface inherits MTU 1400 from
the parent OVN/Geneve overlay, and Cilium sets all interfaces
(cilium_vxlan, lxc*, cilium_host/net) to 1400 without subtracting
the 50-byte VXLAN encapsulation overhead.
This causes intermittent packet drops for large packets (TLS
handshakes, HTTP responses with data), resulting in timeouts and
499 errors for services running in tenant clusters.
Set MTU to 1350 (1400 - 50 VXLAN overhead) explicitly in the default
Cilium values for tenant Kubernetes clusters.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
PR #2075 introduced `_cluster.cluster-domain` references in
monitoring-agents `values.yaml` for FQDN resolution in tenant clusters.
This broke the fluent-bit subchart because `_cluster` values are not
accessible from the Helm subchart context — only `global` values are
shared with subcharts.
This PR replaces `_cluster` references with a new `global.clusterDomain`
variable:
- Empty by default (management cluster uses short DNS names like
`service.namespace.svc`)
- Set to the management cluster domain (e.g. `cozy.local`) for tenant
clusters, enabling FQDN resolution for cross-cluster service discovery
Fixes#2084
### Release note
```release-note
[system] Fix monitoring-agents installation failure caused by inaccessible _cluster values in fluent-bit subchart context. Introduce global.clusterDomain for proper FQDN resolution in tenant workload clusters.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Monitoring agent configuration updated to support configurable cluster
domain names for greater flexibility.
* Remote write and log-forwarding endpoints adjusted to align with
cluster domain handling, improving compatibility when deploying across
different cluster DNS setups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
PR #2075 added _cluster.cluster-domain references to monitoring-agents
values.yaml for FQDN resolution. This broke fluent-bit because it is
a subchart where _cluster values are not accessible (only global
values are shared with subcharts), causing "index of untyped nil".
Revert values.yaml to short DNS names (.svc) for the management
cluster where they resolve locally, and add FQDN with cluster-domain
suffix in the tenant kubernetes HelmRelease where _cluster values are
available and cross-cluster DNS resolution is needed.
Closes#2084
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kamaji auto-derives the konnectivity proxy image tag as v0.{minor}.0
from the Kubernetes version. For K8s v1.35, this produces v0.35.0,
but the kas-network-proxy/proxy-server:v0.35.0 image does not exist
in registry.k8s.io yet, causing ImagePullBackOff on new TCP pods.
Add konnectivity-versions.yaml mapping to explicitly override the
konnectivity version when the auto-derived tag is unavailable.
For v1.35, pin to v0.34.0 (latest available).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Update Kubernetes version matrix to match Talos 1.12 support range:
- Add v1.35.0 (latest) and v1.34.4
- Update existing patch versions (v1.33.8, v1.32.12)
- Drop EOL versions v1.28 and v1.29
- Set default version to v1.35
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Move nil check before req dereference in CreateVolume
- Scope CiliumNetworkPolicy endpointSelector to specific VMI
- Use vmNamespace from NodeId for VMI lookup instead of infraNamespace
- Log PVC lookup errors in ControllerExpandVolume
- Wrap CNP ownerReference updates in retry.RetryOnConflict
- Fix infraClusterLabels validation to check runControllerService flag
- Dereference nodeName pointer in error message
- Replace panic with klog.Fatal for consistent error handling
- Honor CSI readonly flag in NFS NodePublishVolume
- Log mount list errors in isNFSMount
- Reorder Dockerfile ENTRYPOINT after COPY for better layer caching
- Add cleanup on e2e test failure and --wait on pod deletion
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Remove separate NFS Application dependency from e2e test. The kubevirt
CSI driver wrapper now handles RWX Filesystem volumes natively - PVCs
with ReadWriteMany accessMode use the standard kubevirt StorageClass.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Implement a wrapper around upstream kubevirt-csi-driver that adds RWX
Filesystem volume support via LINSTOR NFS exports:
- CreateVolume: intercepts RWX+Filesystem requests, creates DataVolume
with explicit AccessMode=RWX and VolumeMode=Filesystem
- ControllerPublishVolume: waits for PVC bound, extracts NFS export URL
from infra PV, creates CiliumNetworkPolicy with VMI ownerReferences
- ControllerUnpublishVolume: manages CNP ownerReferences per-VMI
- ControllerExpandVolume: delegates to upstream, disables node expansion
for NFS volumes
- NodeStageVolume/NodePublishVolume: mounts NFS at target path
- NodeExpandVolume: no-op for NFS (LINSTOR handles resize)
Also includes:
- RBAC for CiliumNetworkPolicy management in infra cluster
- Explicit --run-node-service=false for controller deployment
- Explicit --run-controller-service=false for node DaemonSet
- nfs-utils in container image for NFS mount support
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
When the Gateway API addon is enabled, automatically configure
cert-manager with enableGatewayAPI: true. Uses the same default
values + mergeOverwrite pattern as Cilium for consistency.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
- Removes `hostNetwork` for Ingress NGINX Controller for Managed
kubernetes cluster
- Adds Service of type NodePort for load balancing
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[kubernetes] use NodePort service instead of hostNetwork for ingress-nginx
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Configuration**
* Updated ingress-nginx controller to use NodePort exposure method
(ports 30000 for HTTP, 30001 for HTTPS) when configured as "Proxied."
* Removed unnecessary hostNetwork configuration for the controller.
* Service port mappings updated to align with new NodePort settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Change the default resourcesPreset for kube-apiserver from "medium" to
"large" to prevent OOM kills during normal operation. The "medium"
preset provides insufficient memory for clusters with moderate workloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Integrate fix
- https://github.com/kubevirt/cloud-provider-kubevirt/pull/379
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[kubernetes] Fix endpoints for cilium-gateway
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved handling of incomplete endpoint data by introducing fallback
detection mechanisms.
* Enhanced service discovery to gather endpoints from all available
resources when standard detection fails.
* Updated logging to provide better visibility into fallback operations
and current resource status.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
More granular k8s app extensions dependencies.
As a part of the change, packages were updated:
- monitoring-agents: metrics-server moved to a separate package
- monitoring-agents: prometheus-operator-crds moved to a separate
package
Main Cozystack bundles were updated to reflect packages changes.
### Release note
```release-note
Granular k8s extensions dependencies.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a standalone metrics-server release and a
prometheus-operator-crds release.
* **Improvements**
* Expanded Prometheus-related CRDs with new configuration options and
status subresources.
* Adjusted monitoring orchestration so components (cilium, metrics,
CRDs) install in a clearer order.
* **Refactor**
* Split metrics-server out of the monitoring-agents bundle and moved
Prometheus CRDs into a dedicated package.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
