diff --git a/docs/proposals/affinity-class.md b/docs/proposals/affinity-class.md new file mode 100644 index 00000000..090f627b --- /dev/null +++ b/docs/proposals/affinity-class.md @@ -0,0 +1,356 @@ +# AffinityClass: Named Placement Classes for CozyStack Applications (Draft) + +## Concept + +Similar to StorageClass in Kubernetes, a new resource **AffinityClass** is introduced — a named abstraction over scheduling constraints. When creating an Application, the user selects an AffinityClass by name without knowing the details of the cluster topology. + +``` +StorageClass → "which disk" → PV provisioning +AffinityClass → "where to place" → Pod scheduling +``` + +## Design + +### 1. AffinityClass CRD + +A cluster-scoped resource created by the platform administrator: + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: AffinityClass +metadata: + name: dc1 +spec: + # nodeSelector that MUST be present on every pod of the application. + # Used for validation by the lineage webhook. + nodeSelector: + topology.kubernetes.io/zone: dc1 +``` + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: AffinityClass +metadata: + name: dc2 +spec: + nodeSelector: + topology.kubernetes.io/zone: dc2 +``` + +```yaml +apiVersion: cozystack.io/v1alpha1 +kind: AffinityClass +metadata: + name: gpu +spec: + nodeSelector: + node.kubernetes.io/gpu: "true" +``` + +An AffinityClass contains a `nodeSelector` — a set of key=value pairs that must be present in `pod.spec.nodeSelector` on every pod of the application. This is a contract: the chart is responsible for setting these selectors, the webhook is responsible for verifying them. + +### 2. Tenant: Restricting Available Classes + +Tenant gets `allowedAffinityClasses` and `defaultAffinityClass` fields: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Tenant +metadata: + name: acme + namespace: tenant-root +spec: + defaultAffinityClass: dc1 # default class for applications + allowedAffinityClasses: # which classes are allowed + - dc1 + - dc2 + etcd: false + ingress: true + monitoring: false +``` + +These values are propagated to the `cozystack-values` Secret in the child namespace: + +```yaml +# Secret cozystack-values in namespace tenant-acme +stringData: + values.yaml: | + _cluster: + # ... existing cluster config + _namespace: + # ... existing namespace config + defaultAffinityClass: dc1 + allowedAffinityClasses: + - dc1 + - dc2 +``` + +### 3. Application: Selecting a Class + +Each application can specify an `affinityClass`. If not specified, the `defaultAffinityClass` from the tenant is used: + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Postgres +metadata: + name: main-db + namespace: tenant-acme +spec: + affinityClass: dc1 # explicit selection + replicas: 3 +``` + +```yaml +apiVersion: apps.cozystack.io/v1alpha1 +kind: Redis +metadata: + name: cache + namespace: tenant-acme +spec: + # affinityClass not specified → uses tenant's defaultAffinityClass (dc1) + replicas: 2 +``` + +### 4. How affinityClass Reaches the HelmRelease + +When creating an Application, the API server (`pkg/registry/apps/application/rest.go`): + +1. Extracts `affinityClass` from `spec` (or uses the default from `cozystack-values`) +2. Records `affinityClass` as a **label on the HelmRelease**: + ``` + apps.cozystack.io/affinity-class: dc1 + ``` +3. Resolves AffinityClass to `nodeSelector` and passes it into HelmRelease values as `_scheduling`: + ```yaml + _scheduling: + affinityClass: dc1 + nodeSelector: + topology.kubernetes.io/zone: dc1 + ``` + +### 5. How Charts Apply Scheduling + +A helper is added to `cozy-lib`: + +```yaml +{{- define "cozy-lib.scheduling.nodeSelector" -}} +{{- if .Values._scheduling }} +{{- if .Values._scheduling.nodeSelector }} +nodeSelector: + {{- .Values._scheduling.nodeSelector | toYaml | nindent 2 }} +{{- end }} +{{- end }} +{{- end -}} +``` + +Each app chart uses the helper when rendering Pod/StatefulSet/Deployment specs: + +```yaml +# packages/apps/postgres/templates/db.yaml +spec: + instances: {{ .Values.replicas }} + {{- include "cozy-lib.scheduling.nodeSelector" . | nindent 2 }} +``` + +```yaml +# packages/apps/redis/templates/redis.yaml +spec: + replicas: {{ .Values.replicas }} + template: + spec: + {{- include "cozy-lib.scheduling.nodeSelector" . | nindent 6 }} +``` + +Charts **must** apply `_scheduling.nodeSelector`. If they don't, pods will be rejected by the webhook. + +--- + +## Validation via Lineage Webhook + +### Why Validation, Not Mutation + +Mutation (injecting nodeSelector into a pod) creates problems: +- Requires merging with existing pod nodeSelector/affinity — complex logic with edge cases +- Operators (CNPG, Strimzi) may overwrite nodeSelector on pod restart +- Hidden behavior: pod is created with one spec but actually runs with another + +Validation is simpler and more reliable: +- Webhook checks: "does this pod **have** the required nodeSelector?" +- If not, the pod is **rejected** with a clear error message +- The chart and operator are responsible for setting the correct spec + +### What Already Exists in the Lineage Webhook + +The lineage webhook (`internal/lineagecontrollerwebhook/webhook.go`) on every Pod creation: + +1. Decodes the Pod +2. Walks the ownership graph (`lineage.WalkOwnershipGraph`) — finds the **owning HelmRelease** +3. Extracts labels from the HelmRelease: `apps.cozystack.io/application.kind`, `.group`, `.name` +4. Applies these labels to the Pod + +**Key point:** the webhook already knows which HelmRelease owns each Pod. + +### What Is Added + +After computing lineage labels, a validation step is added: + +``` +Handle(pod): + 1. [existing] computeLabels(pod) → finds owning HelmRelease + 2. [existing] applyLabels(pod, labels) → mutates labels + 3. [NEW] validateAffinity(pod, hr) → checks nodeSelector + 4. Return patch or Denied +``` + +The `validateAffinity` logic: + +```go +func (h *LineageControllerWebhook) validateAffinity( + ctx context.Context, + pod *unstructured.Unstructured, + hr *helmv2.HelmRelease, +) *admission.Response { + // 1. Extract affinityClass from HelmRelease label + affinityClassName, ok := hr.Labels["apps.cozystack.io/affinity-class"] + if !ok { + return nil // no affinityClass — no validation needed + } + + // 2. Look up AffinityClass from cache + affinityClass, ok := h.affinityClassMap[affinityClassName] + if !ok { + resp := admission.Denied(fmt.Sprintf( + "AffinityClass %q not found", affinityClassName)) + return &resp + } + + // 3. Check pod's nodeSelector + podNodeSelector := extractNodeSelector(pod) // from pod.spec.nodeSelector + for key, expected := range affinityClass.Spec.NodeSelector { + actual, exists := podNodeSelector[key] + if !exists || actual != expected { + resp := admission.Denied(fmt.Sprintf( + "pod %s/%s belongs to application with AffinityClass %q "+ + "but missing required nodeSelector %s=%s", + pod.GetNamespace(), pod.GetName(), + affinityClassName, key, expected)) + return &resp + } + } + + return nil // validation passed +} +``` + +### AffinityClass Caching + +The lineage webhook controller already caches ApplicationDefinitions (`runtimeConfig.appCRDMap`). An AffinityClass cache is added in the same way: + +```go +type runtimeConfig struct { + appCRDMap map[appRef]*cozyv1alpha1.ApplicationDefinition + affinityClassMap map[string]*cozyv1alpha1.AffinityClass // NEW +} +``` + +The controller adds a watch on AffinityClass: + +```go +func (c *LineageControllerWebhook) SetupWithManagerAsController(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&cozyv1alpha1.ApplicationDefinition{}). + Watches(&cozyv1alpha1.AffinityClass{}, &handler.EnqueueRequestForObject{}). + Complete(c) +} +``` + +When an AffinityClass changes, the cache is rebuilt. + +--- + +## End-to-End Flow + +``` +1. Admin creates AffinityClass "dc1" (nodeSelector: zone=dc1) + +2. Admin creates Tenant "acme" (defaultAffinityClass: dc1, allowed: [dc1, dc2]) + → namespace tenant-acme + → cozystack-values Secret with defaultAffinityClass + +3. User creates Postgres "main-db" (affinityClass: dc1) + → API server checks: dc1 ∈ allowedAffinityClasses? ✓ + → API server resolves AffinityClass → nodeSelector + → HelmRelease is created with: + - label: apps.cozystack.io/affinity-class=dc1 + - values: _scheduling.nodeSelector.topology.kubernetes.io/zone=dc1 + +4. FluxCD deploys HelmRelease → Helm renders the chart + → Chart uses cozy-lib helper + → CNPG Cluster is created with nodeSelector: {zone: dc1} + +5. CNPG operator creates Pod + → Pod has nodeSelector: {zone: dc1} + +6. Lineage webhook intercepts the Pod: + a. WalkOwnershipGraph → finds HelmRelease "main-db" + b. HelmRelease label → affinityClass=dc1 + c. AffinityClass "dc1" → nodeSelector: {zone: dc1} + d. Checks: pod.spec.nodeSelector contains zone=dc1? ✓ + e. Admits Pod (+ standard lineage labels) + +7. Scheduler places the Pod on a node in dc1 +``` + +### Error Scenario (chart forgot to apply nodeSelector): + +``` +5. CNPG operator creates Pod WITHOUT nodeSelector + +6. Lineage webhook: + d. Checks: pod.spec.nodeSelector contains zone=dc1? ✗ + e. REJECTS Pod: + "pod main-db-1 belongs to application with AffinityClass dc1 + but missing required nodeSelector topology.kubernetes.io/zone=dc1" + +7. Pod is not created. CNPG operator sees the error and retries. + → Chart developer gets a signal that the chart does not support scheduling. +``` + +--- + +## Code Changes + +### New Files + +| File | Description | +|------------------------------------------------------|-------------------------| +| `api/v1alpha1/affinityclass_types.go` | AffinityClass CRD types | +| `config/crd/bases/cozystack.io_affinityclasses.yaml` | CRD manifest | + +### Modified Files + +| File | Change | +|-------------------------------------------------------|-------------------------------------------------------------------| +| `internal/lineagecontrollerwebhook/webhook.go` | Add `validateAffinity()` to `Handle()` | +| `internal/lineagecontrollerwebhook/config.go` | Add `affinityClassMap` to `runtimeConfig` | +| `internal/lineagecontrollerwebhook/controller.go` | Add watch on AffinityClass | +| `pkg/registry/apps/application/rest.go` | On Create/Update: resolve affinityClass, pass to values and label | +| `packages/apps/tenant/values.yaml` | Add `defaultAffinityClass`, `allowedAffinityClasses` | +| `packages/apps/tenant/templates/namespace.yaml` | Propagate to cozystack-values | +| `packages/system/tenant-rd/cozyrds/tenant.yaml` | Extend OpenAPI schema | +| `packages/library/cozy-lib/templates/_cozyconfig.tpl` | Add `cozy-lib.scheduling.nodeSelector` helper | +| `packages/apps/*/templates/*.yaml` | Each app chart: add helper usage | + +--- + +## Open Questions + +1. **AffinityClass outside Tenants**: Should AffinityClass work for applications outside tenant namespaces (system namespace)? Or only for tenant workloads? + +2. **affinityClass validation on Application creation**: The API server should verify that the specified affinityClass exists and is included in the tenant's `allowedAffinityClasses`. Where should this be done — in the REST handler (`rest.go`) or in a separate validating webhook? + +3. **Soft mode (warn vs deny)**: Is a mode needed where the webhook issues a warning instead of rejecting? This would simplify gradual adoption while not all charts support `_scheduling`. + +4. **affinityClass inheritance**: If a child Tenant does not specify `defaultAffinityClass`, should it be inherited from the parent? The current `cozystack-values` architecture supports this inheritance natively. + +5. **Multiple nodeSelectors**: Is OR-logic support needed (pod can be in dc1 OR dc2)? With `nodeSelector` this is impossible — AffinityClass would need to be extended to `nodeAffinity`. However, validation becomes significantly more complex.