cozystack/docs/changelogs/v0.40.0.md

Cozystack v0.40 — "Enhanced Storage & Platform Architecture"

This release introduces LINSTOR scheduler for optimal pod placement, SeaweedFS traffic locality, a new valuesFrom-based configuration mechanism, auto-diskful for LINSTOR, automated version management systems, and numerous improvements across the platform.

Feature Highlights

LINSTOR Scheduler for Optimal Pod Placement

Cozystack now includes a custom Kubernetes scheduler extender that works alongside the default kube-scheduler to optimize pod placement on nodes with LINSTOR storage. When a pod requests LINSTOR-backed storage, the scheduler communicates with the LINSTOR controller to find nodes that have local replicas of the requested volumes, prioritizing placement on nodes with existing data to minimize network traffic and improve I/O performance.

The scheduler includes an admission webhook that automatically routes pods using LINSTOR CSI volumes to the custom scheduler, ensuring seamless integration without manual configuration. This feature significantly improves performance for workloads using LINSTOR storage by reducing network latency and improving data locality.

Learn more about LINSTOR in the documentation.

SeaweedFS Traffic Locality

SeaweedFS has been upgraded to version 4.05 with new traffic locality capabilities that optimize S3 service traffic distribution. The update includes a new admin component with a web-based UI and authentication support, as well as a worker component for distributed operations. These enhancements improve S3 service performance and provide better visibility through enhanced Grafana dashboard panels for buckets, API calls, costs, and performance metrics.

The traffic locality feature ensures that S3 requests are routed to the nearest available volume servers, reducing latency and improving overall performance for distributed storage operations. TLS certificate support for admin and worker components adds an extra layer of security for management operations.

ValuesFrom Configuration Mechanism

Cozystack now uses FluxCD's valuesFrom mechanism to replace Helm lookup functions for configuration propagation. This architectural improvement provides cleaner config propagation and eliminates the need for force reconcile controllers. Configuration from ConfigMaps (cozystack, cozystack-branding, cozystack-scheduling) and namespace service references (etcd, host, ingress, monitoring, seaweedfs) is now centrally managed through a cozystack-values Secret in each namespace.

This change simplifies Helm chart templates by replacing complex lookup functions with direct value references, improves configuration consistency, and reduces the reconciliation overhead. All HelmReleases now automatically receive cluster and namespace configuration through the valuesFrom mechanism, making configuration management more transparent and maintainable.

Auto-diskful for LINSTOR

The LINSTOR integration now includes automatic diskful functionality that converts diskless nodes to diskful when they hold DRBD resources in Primary state for an extended period (30 minutes). This feature addresses scenarios where workloads are scheduled on nodes without local storage replicas by automatically creating local disk replicas when needed, improving I/O performance for long-running workloads.

When enabled with cleanup options, the system can automatically remove disk replicas that are no longer needed, preventing storage waste from temporary replicas. This intelligent storage management reduces network traffic for frequently accessed data while maintaining efficient storage utilization.

Automated Version Management Systems

Cozystack now includes automated version management systems for PostgreSQL, Kubernetes, MariaDB, and Redis applications. These systems automatically track upstream versions and provide mechanisms for automated version updates, ensuring that platform users always have access to the latest stable versions while maintaining compatibility with existing deployments.

The version management systems integrate with the Cozystack API and dashboard, providing administrators with visibility into available versions and update paths. This infrastructure sets the foundation for future automated upgrade workflows and version compatibility management.

---

Major Features and Improvements

Storage

• [linstor] Add linstor-scheduler package: Added LINSTOR scheduler extender for optimal pod placement on nodes with LINSTOR storage. Includes admission webhook that automatically routes pods using LINSTOR CSI volumes to the custom scheduler, ensuring pods are placed on nodes with local replicas to minimize network traffic and improve I/O performance (@kvaps in #1824).
• [linstor] Enable auto-diskful for diskless nodes: Enabled DRBD auto-diskful functionality to automatically convert diskless nodes to diskful when they hold volumes in Primary state for more than 30 minutes. Improves I/O performance for long-running workloads by creating local replicas and includes automatic cleanup options to prevent storage waste (@kvaps in #1826).
• [linstor] Build linstor-server with custom patches: Added custom patches to linstor-server build process, enabling platform-specific optimizations and fixes (@kvaps in #1726).
• [seaweedfs] Traffic locality: Upgraded SeaweedFS to v4.05 with traffic locality capabilities, new admin component with web-based UI, worker component for distributed operations, and enhanced S3 monitoring with Grafana dashboards. Improves S3 service performance by routing requests to nearest available volume servers (@nbykov0 in #1748).
• [linstor] fix: prevent DRBD device race condition in updateDiscGran: Fixed race condition in DRBD device management during granularity updates, preventing potential data corruption or device conflicts (@kvaps in #1829).
• fix(linstor): prevent orphaned DRBD devices during toggle-disk retry: Fixed issue where retry logic during disk toggle operations could leave orphaned DRBD devices, now properly cleans up devices during retry attempts (@kvaps in #1823).

Platform Architecture

• [platform] Replace Helm lookup with valuesFrom mechanism: Replaced Helm lookup functions with FluxCD valuesFrom mechanism for configuration propagation. Configuration from ConfigMaps and namespace references is now managed through cozystack-values Secret, simplifying templates and eliminating force reconcile controllers (@kvaps in #1787).
• [platform] refactor: split cozystack-resource-definitions into separate packages: Refactored cozystack-resource-definitions into separate packages for better organization and maintainability, improving code structure and reducing coupling between components (@kvaps in #1778).
• [platform] Separate assets server into dedicated deployment: Separated assets server from main platform deployment, improving scalability and allowing independent scaling of asset delivery infrastructure (@kvaps in #1705).
• [core] Extract Talos package from installer: Extracted Talos package configuration from installer into a separate package, improving modularity and enabling independent updates (@kvaps in #1724).
• [registry] Add application labels and update filtering mechanism: Added application labels to registry resources and improved filtering mechanism for better resource discovery and organization (@kvaps in #1707).
• fix(registry): implement field selector filtering for label-based resources: Implemented field selector filtering for label-based resources in the registry, improving query performance and resource lookup efficiency (@kvaps in #1845).
• [platform] Add alphabetical sorting to registry resource lists: Added alphabetical sorting to registry resource lists in the API and dashboard, improving user experience when browsing available applications (@lexfrei in #1764).

Version Management

• [postgres] Add version management system with automated version updates: Introduced version management system for PostgreSQL with automated version tracking and update mechanisms (@kvaps in #1671).
• [kubernetes] Add version management system with automated version updates: Added version management system for Kubernetes tenant clusters with automated version tracking and update capabilities (@kvaps in #1672).
• [mariadb] Add version management system with automated version updates: Implemented version management system for MariaDB with automated version tracking and update mechanisms (@kvaps in #1680).
• [redis] Add version management system with automated version updates: Added version management system for Redis with automated version tracking and update capabilities (@kvaps in #1681).

Networking

• [kube-ovn] Update to v1.14.25: Updated Kube-OVN to version 1.14.25 with improved stability and new features (@kvaps in #1819).
• [kubeovn] Package from external repo: Extracted Kube-OVN packaging from main repository to external repository, improving modularity (@lllamnyp in #1535).
• [cilium] Update Cilium to v1.18.5: Updated Cilium to version 1.18.5 with latest features and bug fixes (@lexfrei in #1769).
• [system/cilium] Enable topology-aware routing for services: Enabled topology-aware routing for Cilium services, improving traffic distribution and reducing latency by routing traffic to endpoints in the same zone when possible (@nbykov0 in #1734).
• [cilium] Enable automatic pod rollout on configmap updates: Cilium and Cilium operator pods now automatically restart when the cilium-config ConfigMap is updated, ensuring configuration changes are applied immediately (@kvaps in #1728).
• [kubernetes] Fix endpoints for cilium-gateway: Fixed endpoint configuration for cilium-gateway, ensuring proper service discovery and connectivity (@kvaps in #1729).
• [multus] Increase memory limit: Increased memory limits for Multus components to handle larger network configurations and reduce out-of-memory issues (@nbykov0 in #1773).
• [main][paas-full] Add multus dependencies similar to other CNIs: Added Multus as a dependency in the paas-full package, consistent with how other CNIs are included (@nbykov0 in #1842).

Virtual Machines

• [vm] Always expose VMs with a service: Virtual machines are now always exposed with at least a ClusterIP service, ensuring they have in-cluster DNS names and can be accessed from other pods even without public IP addresses (@lllamnyp in #1738).
• [virtual-machine] Improve check for resizing job: Improved storage resize logic to only expand persistent volume claims when storage is being increased, preventing unintended storage reduction operations (@kvaps in #1688).
• [virtual-machine,vm-instance] Add nodeAffinity for Windows VMs based on scheduling config: Added nodeAffinity configuration to virtual-machine and vm-instance charts to support dedicated nodes for Windows VMs (@kvaps in #1693).

Monitoring

• [monitoring] Add SLACK_SEVERITY_FILTER field and VMAgent for tenant monitoring: Introduced SLACK_SEVERITY_FILTER environment variable in Alerta deployment to enable filtering of alert severities for Slack notifications. Added VMAgent resource template for scraping metrics within tenant namespaces, improving monitoring granularity (@IvanHunters in #1712).
• [monitoring] Improve tenant metrics collection: Improved tenant metrics collection mechanisms for better observability and monitoring coverage (@IvanHunters in #1684).

System Configuration

• [api, lineage] Tolerate all taints: Updated API and lineage webhook to tolerate all taints, ensuring controllers can run on any node regardless of taint configuration (@nbykov0 in #1781).
• [system] Add resource requests and limits to etcd-defrag: Added resource requests and limits to etcd-defrag job to ensure proper resource allocation and prevent resource contention (@matthieu-robin in #1785).
• [system:coredns] update coredns app labels to match Talos coredns labels: Updated coredns app labels to match Talos coredns labels, ensuring consistency across the platform (@nbykov0 in #1675).
• [system:monitoring-agents] rename coredns metrics service: Renamed coredns metrics service to avoid interference with coredns service used for name resolution in tenant k8s clusters (@nbykov0 in #1676).

Tenants and Namespaces

• [tenant] Allow egress to parent ingress pods: Updated tenant network policies to allow egress traffic to parent cluster ingress pods, enabling proper communication patterns (@lexfrei in #1765).
• [tenant] Run cleanup job from system namespace: Moved tenant cleanup job to run from system namespace, improving security and resource isolation (@lllamnyp in #1774).

FluxCD

• [fluxcd] Add flux-aio module and migration: Added FluxCD all-in-one module with migration support, simplifying FluxCD installation and management (@kvaps in #1698).
• [fluxcd] Enable source-watcher: Enabled source-watcher in FluxCD configuration for improved GitOps synchronization and faster update detection (@kvaps in #1706).

Applications

• [dashboard] Fix CustomFormsOverride schema to nest properties under spec.properties: Fixed CustomFormsOverride schema generation to properly nest properties under spec.properties instead of directly under properties, ensuring correct form schema generation (@kvaps in #1692).
• [apps] Refactor apiserver to use typed objects and fix UnstructuredList GVK: Refactored apiserver REST handlers to use typed objects instead of unstructured.Unstructured, eliminating runtime conversions. Fixed UnstructuredList GVK issue where objects were using the first registered kind instead of the correct kind (@kvaps in #1679).
• [keycloak] Make kubernetes client public: Made Kubernetes client public in Keycloak configuration, enabling broader access patterns for Kubernetes integrations (@lllamnyp in #1802).

Improvements

• [granular kubernetes application extensions dependencies]: Improved dependency management for Kubernetes application extensions with more granular control over dependencies (@nbykov0 in #1683).
• [core:installer] Address buildx warnings: Fixed Dockerfile syntax warnings from buildx, ensuring clean builds without warnings (@nbykov0 in #1682).
• [linstor] Update piraeus-operator v2.10.2: Updated LINSTOR CSI to version 2.10.2 with improved stability and bug fixes (@kvaps in #1689).
• Update SeaweedFS v4.02: Updated SeaweedFS to version 4.02 with improved S3 daemon performance and fixes (@kvaps in #1725).
• [installer,dx] Rename cozypkg to cozyhr: Renamed cozypkg tool to cozyhr for better branding and consistency (@kvaps in #1763).

Fixes

• fix(platform): fix migrations for v0.40 release: Fixed platform migrations for v0.40 release, ensuring smooth upgrades from previous versions (@kvaps in #1846).
• [platform] fix migration for removing fluxcd-operator: Fixed migration logic for removing fluxcd-operator, ensuring clean removal without leaving orphaned resources (@kvaps in commit 4a83d2c7).
• [kubevirt-operator] Fix typo in VMNotRunningFor10Minutes alert: Fixed typo in VM alert name, ensuring proper alert triggering and monitoring (@kvaps in #1770).
• [kubevirt-operator] Revert incorrect case change in VM alerts: Reverted incorrect case change in VM alert names to maintain consistency (@lexfrei in #1804).
• [cozystack-controller] Fix: move crds to definitions: Fixed CRD placement by moving them to definitions directory, ensuring proper resource organization (@kvaps in #1759).

Dependencies

• Update SeaweedFS v4.02: Updated SeaweedFS to version 4.02 (@kvaps in #1725).
• [seaweedfs] Traffic locality: Upgraded SeaweedFS to v4.05 with traffic locality capabilities (@nbykov0 in #1748).
• [linstor] Update piraeus-operator v2.10.2: Updated piraeus-operator to version 2.10.2 (@kvaps in #1689).
• [kube-ovn] Update to v1.14.25: Updated Kube-OVN to version 1.14.25 (@kvaps in #1819).
• [cilium] Update Cilium to v1.18.5: Updated Cilium to version 1.18.5 (@lexfrei in #1769).
• Update go modules: Updated Go modules to latest versions (@kvaps in #1736).

Development, Testing, and CI/CD

• [ci] Fix auto-release workflow: Fixed auto-release workflow to ensure correct release publishing and tagging (@kvaps in commit 526af294).
• fix(ci): ensure correct latest release after backport publishing: Fixed CI workflow to correctly identify and tag the latest release after backport publishing (@kvaps in #1800).
• [workflows] Add auto patch release workflow: Added automated patch release workflow for streamlined release management (@kvaps in #1754).
• [workflow] Add GitHub Action to update release notes from changelogs: Added GitHub Action to automatically update release notes from changelog files (@kvaps in #1752).
• [ci] Improve backport workflow with merge_commits skip and conflict resolution: Improved backport workflow with better merge commit handling and conflict resolution (@kvaps in #1694).
• [testing] Add aliases and autocomplete: Added shell aliases and autocomplete support for testing commands, improving developer experience (@lllamnyp in #1803).
• [kubernetes] Add lb tests for tenant k8s: Added load balancer tests for tenant Kubernetes clusters, improving test coverage (@IvanHunters in #1783).
• [agents] Add instructions for working with unresolved code review comments: Added documentation and instructions for working with unresolved code review comments in agent workflows (@kvaps in #1710).
• feat(ci): add /retest command to rerun tests from Prepare environment: Added /retest command to rerun tests from Prepare environment workflow (@kvaps in commit 30c1041e).
• fix(ci): remove GITHUB_TOKEN extraheader to trigger workflows: Removed GITHUB_TOKEN extraheader to properly trigger workflows (@kvaps in commit 68a639b3).
• Fix: Add missing components to distro-full bundle: Fixed missing components in distro-full bundle, ensuring all required components are included (@LoneExile in #1620).
• Update Flux Operator (v0.33.0): Updated Flux Operator to version 0.33.0 (@kingdonb in #1649).
• Add changelogs for v0.38.3 and v.0.38.4: Added missing changelogs for v0.38.3 and v0.38.4 releases (@androndo in #1743).
• Add changelogs to v.0.39.1: Added changelog for v0.39.1 release (@androndo in #1750).
• Add Cloupard to ADOPTERS.md: Added Cloupard to the adopters list (@SerjioTT in #1733).

Documentation

• [website] docs: expand monitoring and alerting documentation: Expanded monitoring and alerting documentation with comprehensive guides, examples, and troubleshooting information (@IvanHunters in cozystack/website#388).
• [website] fix auto-generation of documentation: Fixed automatic documentation generation process, ensuring all documentation is properly generated and formatted (@IvanHunters in cozystack/website#391).
• [website] secure boot: Added documentation for Secure Boot support in Talos Linux (@kvaps in cozystack/website#387).

Tools

• [talm] feat(helpers): add bond interface discovery helpers: Added bond interface discovery helpers to talm for easier network configuration (@kvaps in cozystack/talm#94).
• [talm] feat(talosconfig): add certificate regeneration from secrets.yaml: Added certificate regeneration functionality to talm talosconfig command, allowing certificates to be regenerated from secrets.yaml (@kvaps in cozystack/talm@1319dde).
• [talm] fix(init): make name optional for -u flag: Made name parameter optional for init command with -u flag, improving flexibility (@kvaps in cozystack/talm@da29320).
• [talm] fix(wrapper): copy NoOptDefVal when remapping -f to -F flag: Fixed wrapper to properly copy NoOptDefVal when remapping flags, ensuring correct default value handling (@kvaps in cozystack/talm@f6a6f1d).
• [talm] fix(root): detect project root with secrets.encrypted.yaml: Fixed root detection to properly identify project root when secrets.encrypted.yaml is present (@kvaps in cozystack/talm@cf56780).
• [talm] Fix interfaces helper for Talos v1.12: Fixed interfaces helper to work correctly with Talos v1.12 (@kvaps in cozystack/talm@34984ae).
• [talm] Fix typo on README.md: Fixed typo in README documentation (@diegolakatos in cozystack/talm#92).
• [talm] fix(template): return error for invalid YAML in template output: Fixed template command to return proper error for invalid YAML output (@kvaps in cozystack/talm#93).
• [talm] feat(cozystack): enable allocateNodeCIDRs by default: Enabled allocateNodeCIDRs by default in talm cozystack preset (@lexfrei in cozystack/talm#91).
• [boot-to-talos] feat(network): add VLAN interface support via netlink: Added VLAN interface support via netlink in boot-to-talos for advanced network configuration (@kvaps in cozystack/boot-to-talos@02874d7).
• [boot-to-talos] feat(network): add bond interface support via netlink: Added bond interface support via netlink in boot-to-talos for network bonding configurations (@kvaps in cozystack/boot-to-talos@067822d).
• [boot-to-talos] Draft EFI Support: Added draft EFI support in boot-to-talos for UEFI boot scenarios (@kvaps in cozystack/boot-to-talos@e194bc8).
• [boot-to-talos] Change default install image size from 2GB to 3GB: Changed default install image size from 2GB to 3GB to accommodate larger installations (@kvaps in cozystack/boot-to-talos@3bfb035).
• [cozyhr] feat(values): add valuesFrom support for HelmRelease: Added valuesFrom support for HelmRelease in cozyhr tool, enabling better configuration management (@kvaps in cozystack/cozyhr@7dff0c8).
• [cozyhr] Rename cozypkg to cozyhr: Renamed cozypkg tool to cozyhr for better branding (@kvaps in cozystack/cozyhr@1029461).

---

Contributors

We'd like to thank all contributors who made this release possible:

• @IvanHunters
• @kvaps
• @lexfrei
• @lllamnyp
• @nbykov0
• @LoneExile
• @kingdonb
• @androndo
• @SerjioTT
• @matthieu-robin
• @diegolakatos

---

Full Changelog: v0.39.0...v0.40.0