vrr/cozystack
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(cozystack-engine): add managed Kubernetes without visible control-plane nodes support to lineage-controller-webhook (#2481)

Browse source 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

In environments where control-plane nodes are not available (e.g., AWS
managed nodes, Cozy-in-Cozy tenant Kubernetes clusters), the DaemonSet
cannot schedule any pods because no nodes match this selector.

## What this PR does

This PR adds ability to replace DaemonSet by Deployment with custom
number of replicas for HA purpose. Default values saves current
behavior.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional Deployment mode for lineage-controller-webhook with
configurable replicas
  * PodDisruptionBudget support when Deployment mode is enabled

* **Updates**
* Improved pod scheduling: node affinity replaces nodeSelector, optional
tolerations, and pod anti-affinity when using Deployment
* Clarified behavior for local API endpoint scheduling (only applies
when pod lands on nodes running an API server)

* **Documentation**
  * Added README with usage examples and parameter details
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Timofei Larkin 2026-04-29 15:24:27 +04:00 • committed by GitHub
commit 69ad74e457
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 239 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

3
packages/core/platform/templates/bundles/system.yaml
View file

@ -105,9 +105,6 @@
{{- /* cozystack-api DaemonSet */ -}}
{{- $apiValues := dict "cozystackAPI" (dict "nodeSelector" $genericNodeSelector) -}}
{{- $_ := set $cozystackEngineComponents "cozystack-api" (dict "values" $apiValues) -}}
{{- /* lineage-controller-webhook DaemonSet */ -}}
{{- $lineageValues := dict "lineageControllerWebhook" (dict "nodeSelector" $genericNodeSelector) -}}
{{- $_ := set $cozystackEngineComponents "lineage-controller-webhook" (dict "values" $lineageValues) -}}
{{- end -}}
{{- if .Values.authentication.oidc.enabled }}
{{include "cozystack.platform.package" (list "cozystack.cozystack-engine" "oidc" $ $cozystackEngineComponents) }}

5
packages/system/lineage-controller-webhook/Makefile
View file

@ -4,8 +4,13 @@ NAMESPACE=cozy-system
include ../../../hack/common-envs.mk
include ../../../hack/package.mk
.PHONY: test
image: image-lineage-controller-webhook
test:
helm unittest .
image-lineage-controller-webhook:
docker buildx build -f images/lineage-controller-webhook/Dockerfile ../../.. \
--tag $(REGISTRY)/lineage-controller-webhook:$(call settag,$(TAG)) \

67
packages/system/lineage-controller-webhook/README.md Normal file
View file

@ -0,0 +1,67 @@
# lineage-controller-webhook
Cozystack system package for the **lineage controller webhook** — a mutating
admission webhook that stamps "lineage" labels onto tenant workloads, linking
each resource back to the Cozystack `Application` that ultimately owns it.
The webhook intercepts CREATE and UPDATE on `pods`, `secrets`, `services`,
`persistentvolumeclaims`, `ingresses.networking.k8s.io`, and
`workloadmonitors.cozystack.io` outside system namespaces. For each request it
walks the ownership graph upward (Kubernetes `ownerReferences`, then the
`HelmRelease` Flux installed the resource with, then the Cozystack
`Application`-derived CRD that produced the HelmRelease) and writes the
discovered application's group, kind and name as labels
(`apps.cozystack.io/application.{group,kind,name}`) on the incoming object.
Those labels let the aggregated API server, the Cozystack dashboard, and other
lineage-aware consumers reason about which application a resource belongs to.
The webhook serves TLS on port 9443 with a cert-manager issued certificate, and
runs with `failurePolicy: Fail` — every CREATE/UPDATE on the resources above
is gated on the webhook being reachable.
## Topology
The chart ships a single shape, modelled on `cozystack-api`:
- **Deployment** with two replicas (override via `replicas`).
- **Soft `nodeAffinity`** preferring `node-role.kubernetes.io/control-plane`
(`Exists`, so it matches both Talos's empty value and k3s/kubeadm's `"true"`).
Soft means: the pod lands on a control-plane node when one is reachable, and
on any worker otherwise — no override needed for managed Kubernetes,
Cozy-in-Cozy tenants, or any other cluster where control-plane nodes aren't
visible.
- **Permissive `tolerations`** (`[{operator: Exists}]`) so the pod can land on
tainted control-plane nodes when the soft affinity is satisfiable.
- **Soft `podAntiAffinity`** on `kubernetes.io/hostname` so replicas spread
across nodes when possible (best-effort).
- **`PodDisruptionBudget`** with `maxUnavailable: 1`, unconditional. At
`replicas: 1` it's a useful no-op (allows full drain); at `replicas >= 2`
it caps disruption to one pod.
- **`Service` with `spec.trafficDistribution: PreferClose`** so the apiserver
prefers a webhook endpoint on its own node when one exists, and transparently
falls over to a remote endpoint otherwise. Requires Kubernetes ≥ 1.31; on
older clusters the field is silently ignored and traffic uses default
cluster-wide distribution.
## Parameters
All keys live under the top-level `lineageControllerWebhook:` map.
| Name | Description | Value |
| ----------------------------- | --------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `image` | Container image (digest-pinned) | `ghcr.io/cozystack/cozystack/lineage-controller-webhook:v1.3.0@sha256:e898…6fb0` |
| `debug` | Enable `--zap-log-level=debug` instead of `info` | `false` |
| `replicas` | Deployment replica count | `2` |
| `localK8sAPIEndpoint.enabled` | **Deprecated.** See note below. | `false` |
### `localK8sAPIEndpoint.enabled` (deprecated)
When enabled, this injects `KUBERNETES_SERVICE_HOST=status.hostIP` and
`KUBERNETES_SERVICE_PORT=6443` so the webhook talks to the kube-apiserver on
its own node. It was originally added to avoid latency on the
webhook-to-apiserver path, but it is only valid when the pod is actually
scheduled on an apiserver-bearing node — which the chart's soft control-plane
affinity no longer guarantees. With this flag enabled and the pod scheduled
off a control-plane node, the controller will crash-loop dialing a non-
apiserver hostIP. Slated for removal once the latency motivation is addressed
in the webhook itself; **leave disabled**.

9
packages/system/lineage-controller-webhook/templates/pdb.yaml Normal file
View file

@ -0,0 +1,9 @@
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: lineage-controller-webhook
spec:
maxUnavailable: 1
selector:
matchLabels:
app: lineage-controller-webhook

2
packages/system/lineage-controller-webhook/templates/service.yaml
View file

@ -5,7 +5,7 @@ metadata:
labels:
app: lineage-controller-webhook
spec:
internalTrafficPolicy: Local
trafficDistribution: PreferClose
type: ClusterIP
ports:
- port: 443

25
packages/system/lineage-controller-webhook/templates/daemonset.yaml → packages/system/lineage-controller-webhook/templates/workload.yaml
View file

@ -1,10 +1,11 @@
apiVersion: apps/v1
kind: DaemonSet
kind: Deployment
metadata:
name: lineage-controller-webhook
labels:
app: lineage-controller-webhook
spec:
replicas: {{ .Values.lineageControllerWebhook.replicas }}
selector:
matchLabels:
app: lineage-controller-webhook
@ -13,13 +14,25 @@ spec:
labels:
app: lineage-controller-webhook
spec:
{{- with .Values.lineageControllerWebhook.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: lineage-controller-webhook
tolerations:
- operator: Exists
serviceAccountName: lineage-controller-webhook
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: lineage-controller-webhook
containers:
- name: lineage-controller-webhook
image: "{{ .Values.lineageControllerWebhook.image }}"

30
packages/system/lineage-controller-webhook/tests/service_test.yaml Normal file
View file

@ -0,0 +1,30 @@
suite: lineage-controller-webhook service
templates:
- templates/service.yaml
release:
name: lineage-controller-webhook
namespace: cozy-system
tests:
- it: renders a ClusterIP service on 443 -> 9443 with PreferClose traffic distribution
asserts:
- isKind:
of: Service
- equal:
path: spec.type
value: ClusterIP
- equal:
path: spec.trafficDistribution
value: PreferClose
- notExists:
path: spec.internalTrafficPolicy
- equal:
path: spec.ports[0].port
value: 443
- equal:
path: spec.ports[0].targetPort
value: 9443
- equal:
path: spec.selector.app
value: lineage-controller-webhook

99
packages/system/lineage-controller-webhook/tests/workload_test.yaml Normal file
View file

@ -0,0 +1,99 @@
suite: lineage-controller-webhook workload + PDB
templates:
- templates/workload.yaml
- templates/pdb.yaml
release:
name: lineage-controller-webhook
namespace: cozy-system
tests:
# ---- Default Deployment shape ----------------------------------------------
- it: defaults render a 2-replica Deployment with soft control-plane nodeAffinity, soft podAntiAffinity, and Exists tolerations
template: templates/workload.yaml
asserts:
- isKind:
of: Deployment
- equal:
path: spec.replicas
value: 2
- equal:
path: spec.template.spec.tolerations
value:
- operator: Exists
- equal:
path: spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution
value:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists
- notExists:
path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution
- equal:
path: spec.template.spec.affinity.podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].podAffinityTerm.topologyKey
value: kubernetes.io/hostname
- it: defaults inject no localK8sAPIEndpoint env vars (knob is opt-in)
template: templates/workload.yaml
asserts:
- notExists:
path: spec.template.spec.containers[0].env
# ---- PDB --------------------------------------------------------------------
- it: PDB renders unconditionally with maxUnavailable=1 at the default replica count
template: templates/pdb.yaml
asserts:
- isKind:
of: PodDisruptionBudget
- equal:
path: spec.maxUnavailable
value: 1
- notExists:
path: spec.minAvailable
- it: PDB still renders at replicas=1 (no-op, but consistent shape)
set:
lineageControllerWebhook.replicas: 1
template: templates/pdb.yaml
asserts:
- isKind:
of: PodDisruptionBudget
- equal:
path: spec.maxUnavailable
value: 1
# ---- Replica override -------------------------------------------------------
- it: replicas value drives spec.replicas
set:
lineageControllerWebhook.replicas: 5
template: templates/workload.yaml
asserts:
- equal:
path: spec.replicas
value: 5
# ---- Deprecated localK8sAPIEndpoint knob -----------------------------------
- it: localK8sAPIEndpoint.enabled=true injects KUBERNETES_SERVICE_HOST and PORT env vars
set:
lineageControllerWebhook.localK8sAPIEndpoint.enabled: true
template: templates/workload.yaml
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: KUBERNETES_SERVICE_HOST
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- contains:
path: spec.template.spec.containers[0].env
content:
name: KUBERNETES_SERVICE_PORT
value: "6443"

15
packages/system/lineage-controller-webhook/values.yaml
View file

@ -1,10 +1,13 @@
lineageControllerWebhook:
image: ghcr.io/cozystack/cozystack/lineage-controller-webhook:v1.3.0@sha256:e8984709686a5eaf19b89da378d7b8c688ea5607e0783a88d9c9e4ccfca96fb0
debug: false
replicas: 2
# DEPRECATED. Injects KUBERNETES_SERVICE_HOST=status.hostIP and
# KUBERNETES_SERVICE_PORT=6443 so the webhook talks to the apiserver on its
# own node — only valid when the pod is actually scheduled on an apiserver-
# bearing node. Now that the chart's nodeAffinity to control-plane is soft
# (preferred, not required), the pod can land off-control-plane and crash-
# loop dialing a non-apiserver hostIP. Slated for removal once webhook
# performance work obviates the locality motivation. Leave disabled.
localK8sAPIEndpoint:
enabled: true
# nodeSelector for the DaemonSet
# Talos uses empty value: "node-role.kubernetes.io/control-plane": ""
# Generic k8s (k3s, kubeadm) uses: "node-role.kubernetes.io/control-plane": "true"
nodeSelector:
node-role.kubernetes.io/control-plane: ""
enabled: false