fix(kamaji): increase memory limits and add startup probe (#2421)
## Summary - Increase kamaji controller memory limit from 500Mi to 512Mi - Increase kamaji controller memory request from 100Mi to 256Mi - Add startup probe with 60-second timeout (12 attempts × 5s periods) - Increase readiness/liveness probe initialDelaySeconds from 5s/15s to 30s ## Problem The kamaji controller was experiencing frequent CrashLoopBackOff due to OOMKilled errors. Analysis showed: - Container was being killed with exit code 137 (OOMKilled) after ~20-25 seconds of runtime - Memory limit of 500Mi was insufficient for controller initialization - Readiness probe was failing because it started too early (5s initialDelay), before the controller finished leader election (~17s) ## Solution **Memory increase:** - Limit: 500Mi → 512Mi (based on production testing) - Request: 100Mi → 256Mi (ensures adequate reservation) **Startup probe:** - Added to give controller up to 60 seconds to initialize without being killed by liveness probe - 12 attempts × 5s period = 60s maximum startup time **Probe delays:** - ReadinessProbe: 5s → 30s initialDelay (controller needs ~17s to acquire leader lease) - LivenessProbe: 15s → 30s initialDelay (aligned with readiness) ## Testing Verified in production cluster: - Controller runs stable with 0 restarts - No more OOMKilled events - Successfully creates kubeconfig secrets for tenant clusters ## Related Issues Fixes tenant cluster components stuck in ContainerCreating due to missing kubeconfig secrets (caused by crashing kamaji controller). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Introduced automated health checks using HTTP-based probes to monitor service status during startup, continuous operation, and readiness to handle traffic. * Adjusted container memory resource allocation for enhanced stability and performance. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
commit
52edffbb9a
1 changed files with 25 additions and 2 deletions
|
|
@ -8,9 +8,32 @@ kamaji:
|
|||
resources:
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 500Mi
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 100Mi
|
||||
memory: 256Mi
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: healthcheck
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
successThreshold: 1
|
||||
failureThreshold: 12
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: healthcheck
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 1
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /readyz
|
||||
port: healthcheck
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 1
|
||||
extraArgs:
|
||||
- --migrate-image=ghcr.io/cozystack/cozystack/kamaji:v1.3.0@sha256:8c9af083b60600c0800eb56a2cda75f26007b7272a1cf019140de003bfce1a4d
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue