[hack] Add host runtime preflight check for standalone containerd/docker (#2371)
## What this PR does Adds a host preflight diagnostic script (`hack/check-host-runtime.sh`) that warns operators when a standalone `containerd.service` or `docker.service` is running on the host alongside the embedded k3s runtime used by the cozystack `generic` variant (k3s / kubeadm on Ubuntu). **Why it matters.** K3s ships its own containerd at `/run/k3s/containerd/containerd.sock` and `/var/lib/rancher/k3s/agent/containerd`, while a system-package containerd or docker uses `/run/containerd/containerd.sock` and `/var/lib/containerd`. The two runtimes do not fight over sockets, so both keep running silently. Over time the standalone one accumulates unpruned images and build cache in `/var/lib/containerd` — enough to fill the root disk, trigger `DiskPressure`, and put `cozystack-api` into an eviction loop. This is silent on day zero and surfaces as a mysterious production incident weeks later. The script exists to warn the next operator before the failure mode surfaces. The script is warning-only — it always exits 0 and never blocks the install. It detects: - `containerd.service` or `docker.service` active via `systemctl is-active` - Standalone runtime sockets at well-known paths, with a fallback that works on hosts without systemd - Standalone data directory sizes via `du -sh` When a warning fires, the HINT names only the detected services and instructs the operator to disable them with `sudo systemctl disable --now <service>`. Reclaiming the data directory is called out separately with an explicit note not to delete it blindly — the data may still be in use. **Entry points.** - `make preflight` runs the script directly (for operators preparing a generic-variant host) - `make unit-tests` now runs `bats-unit-tests` alongside `helm-unit-tests`, auto-discovering every `hack/*.bats` file that is not an e2e test **Test coverage.** `hack/check-host-runtime.bats` (11 cases, run via `hack/cozytest.sh`) covers clean hosts with and without systemd, single-service detection, both services simultaneously, socket-only fallback for both runtimes, glob-expansion regression guard, explicit exit-code-0 assertion, `sudo` prefix assertion, `du` failure robustness, and the "service + socket = exactly one warning" de-duplication invariant. Every test is self-contained with `trap 'rm -rf $STUB_DIR' EXIT` for clean recovery on assertion failure, and has no runtime dependencies beyond bash and core utilities — no python3, no real systemd. Irrelevant on Talos where the container runtime lifecycle is fully managed by the distribution. ### Release note ```release-note [hack] Add `check-host-runtime.sh` and `make preflight` target that warn when a standalone containerd or docker runtime is running alongside the embedded k3s runtime on the cozystack generic variant. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a non-blocking preflight check that detects standalone container runtimes, reports disk-usage estimates, and displays an actionable hint to disable detected services. * **Tests** * Added BATS-based unit tests with comprehensive coverage for the preflight validations and various host scenarios. * **Chores** * Build updated to run the BATS unit tests alongside the existing test suite. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
Andrei Kvapil
GitHub
commit
1436fee2dd
3 changed files with 605 additions and 2 deletions
31
Makefile
31
Makefile
|
|
@ -1,4 +1,4 @@
|
|||
.PHONY: manifests assets unit-tests helm-unit-tests
|
||||
.PHONY: manifests assets unit-tests helm-unit-tests bats-unit-tests preflight
|
||||
|
||||
include hack/common-envs.mk
|
||||
|
||||
|
|
@ -82,11 +82,38 @@ test:
|
|||
make -C packages/core/testing apply
|
||||
make -C packages/core/testing test
|
||||
|
||||
unit-tests: helm-unit-tests
|
||||
unit-tests: helm-unit-tests bats-unit-tests
|
||||
|
||||
helm-unit-tests:
|
||||
hack/helm-unit-tests.sh
|
||||
|
||||
# Discover every hack/*.bats file that is NOT an e2e test and run it
|
||||
# through cozytest.sh. Drop a new *.bats file in hack/ and it is picked
|
||||
# up automatically on the next `make unit-tests` run.
|
||||
#
|
||||
# Caveat: $(wildcard ...) returns space-separated names, so a filename
|
||||
# containing a literal space would split into multiple tokens here. All
|
||||
# current bats files use hyphen-separated names; if the project ever
|
||||
# introduces whitespace-bearing filenames this recipe must be rewritten
|
||||
# (e.g. to use `find ... -print0 | xargs -0`).
|
||||
BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))
|
||||
|
||||
bats-unit-tests:
|
||||
@if [ -z "$(BATS_UNIT_FILES)" ]; then \
|
||||
echo "ERROR: no hack/*.bats unit test files found"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@for f in $(BATS_UNIT_FILES); do \
|
||||
echo "--- running $$f ---"; \
|
||||
hack/cozytest.sh "$$f" || exit 1; \
|
||||
done
|
||||
|
||||
# Operator-facing host preflight check. Warns about a standalone
|
||||
# containerd.service or docker.service running alongside the embedded
|
||||
# k3s runtime. Safe to run at any time; always exits 0.
|
||||
preflight:
|
||||
@hack/check-host-runtime.sh
|
||||
|
||||
prepare-env:
|
||||
make -C packages/core/testing apply
|
||||
make -C packages/core/testing prepare-cluster
|
||||
|
|
|
|||
415
hack/check-host-runtime.bats
Normal file
415
hack/check-host-runtime.bats
Normal file
|
|
@ -0,0 +1,415 @@
|
|||
#!/usr/bin/env bats
|
||||
# -----------------------------------------------------------------------------
|
||||
# Unit tests for hack/check-host-runtime.sh
|
||||
#
|
||||
# The script warns when a standalone containerd.service or docker.service is
|
||||
# active alongside the embedded k3s runtime on Ubuntu hosts running the
|
||||
# cozystack "generic" variant. Warnings go to stderr; exit code is always 0.
|
||||
#
|
||||
# Test strategy: each test builds its own temporary stub directory and prepends
|
||||
# it to PATH to inject a fake `systemctl` (and optionally `du`) binary. The
|
||||
# script itself honors a small set of COZYSTACK_PREFLIGHT_* environment
|
||||
# variables to redirect socket/dir probes into the stub tree, so tests do not
|
||||
# need root privileges or a real systemd host.
|
||||
#
|
||||
# Each test installs a `trap 'rm -rf "$STUB_DIR"' EXIT` immediately after
|
||||
# creating the stub dir so cleanup runs even when an assertion fails mid-test
|
||||
# under `set -e`. cozytest.sh runs each @test in its own subshell, so traps
|
||||
# scope per test and do not leak across tests.
|
||||
#
|
||||
# Tests are otherwise self-contained — no shared setup/teardown helpers,
|
||||
# because cozytest.sh's awk parser only recognizes @test blocks and treats a
|
||||
# bare `}` on its own line as the end of a test function.
|
||||
#
|
||||
# Title syntax constraints (inherited from cozytest.sh's awk parser):
|
||||
# - Titles must be delimited by ASCII double quotes; embedded literal
|
||||
# double quotes are NOT escaped and will silently truncate the title.
|
||||
# - Only alphanumeric characters from the title survive into the shell
|
||||
# function name (everything else becomes '_'), so titles that differ
|
||||
# only in punctuation collapse to the same function name. Keep titles
|
||||
# distinctive in their alphanumeric run.
|
||||
#
|
||||
# Run with: hack/cozytest.sh hack/check-host-runtime.bats
|
||||
# (or `bats hack/check-host-runtime.bats` if the bats binary is
|
||||
# installed; cozytest.sh is the CI path.)
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
@test "clean host with no runtime services exits silently" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker1.sock $STUB_DIR/missing-docker2.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh >"$STUB_DIR/stdout" 2>"$STDERR_FILE"
|
||||
|
||||
[ ! -s "$STDERR_FILE" ]
|
||||
[ ! -s "$STUB_DIR/stdout" ]
|
||||
}
|
||||
|
||||
@test "standalone containerd service active prints warning" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$1" = "is-active" ] && [ "$2" = "containerd.service" ]; then
|
||||
echo active
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
mkdir -p "$STUB_DIR/var-lib-containerd"
|
||||
echo dummy >"$STUB_DIR/var-lib-containerd/dummy"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/var-lib-containerd" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
grep -q 'standalone containerd.service' "$STDERR_FILE"
|
||||
if grep -q 'standalone docker.service' "$STDERR_FILE"; then
|
||||
echo "unexpected docker warning found:" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
# HINT line must name only the detected service, not advise disabling
|
||||
# docker.service when only containerd.service is running. The sudo
|
||||
# prefix is also required — without it the command silently no-ops
|
||||
# for a non-root operator, so the prefix is part of the contract.
|
||||
grep -q 'sudo systemctl disable --now containerd.service' "$STDERR_FILE"
|
||||
if grep -q 'systemctl disable --now.*docker' "$STDERR_FILE"; then
|
||||
echo "HINT unexpectedly mentions docker:" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "standalone docker service active prints warning" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$1" = "is-active" ] && [ "$2" = "docker.service" ]; then
|
||||
echo active
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
mkdir -p "$STUB_DIR/var-lib-docker"
|
||||
echo dummy >"$STUB_DIR/var-lib-docker/dummy"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/var-lib-docker" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
grep -q 'standalone docker.service' "$STDERR_FILE"
|
||||
if grep -q 'standalone containerd.service' "$STDERR_FILE"; then
|
||||
echo "unexpected containerd warning found:" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
# HINT line must name only the detected service. As in the
|
||||
# containerd test, the sudo prefix is part of the contract.
|
||||
grep -q 'sudo systemctl disable --now docker.service' "$STDERR_FILE"
|
||||
if grep -q 'systemctl disable --now.*containerd' "$STDERR_FILE"; then
|
||||
echo "HINT unexpectedly mentions containerd:" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "both services active prints two warnings and the HINT block" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$1" = "is-active" ]; then
|
||||
case "$2" in
|
||||
containerd.service|docker.service) echo active; exit 0 ;;
|
||||
esac
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
mkdir -p "$STUB_DIR/var-lib-containerd" "$STUB_DIR/var-lib-docker"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
# Capture exit code explicitly: the script contract says exit 0
|
||||
# unconditionally (warning, not blocker). `set -e` in the test
|
||||
# function body would already fail on a nonzero exit, but an
|
||||
# explicit status check locks in the contract and makes a
|
||||
# regression show up as "expected 0, got N" rather than as a
|
||||
# generic test failure.
|
||||
status=0
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/var-lib-containerd" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/var-lib-docker" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE" || status=$?
|
||||
|
||||
[ "$status" -eq 0 ]
|
||||
grep -q 'standalone containerd.service' "$STDERR_FILE"
|
||||
grep -q 'standalone docker.service' "$STDERR_FILE"
|
||||
# HINT block must fire whenever warnings exist; otherwise a future silent
|
||||
# removal of the HINT would go unnoticed. When both services fire the HINT
|
||||
# must list both in a single sudo systemctl disable invocation — the sudo
|
||||
# prefix is as important as the systemctl verb, otherwise the operator
|
||||
# would be told to run it as a non-root user and quietly fail.
|
||||
grep -q 'HINT:' "$STDERR_FILE"
|
||||
grep -q 'sudo systemctl disable --now containerd.service docker.service' "$STDERR_FILE"
|
||||
}
|
||||
|
||||
@test "failing du does not suppress the containerd warning" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$1" = "is-active" ] && [ "$2" = "containerd.service" ]; then
|
||||
echo active
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
cat >"$STUB_DIR/du" <<'DUEOF'
|
||||
#!/bin/sh
|
||||
exit 1
|
||||
DUEOF
|
||||
chmod +x "$STUB_DIR/du"
|
||||
|
||||
mkdir -p "$STUB_DIR/var-lib-containerd"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/var-lib-containerd" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
grep -q 'standalone containerd.service' "$STDERR_FILE"
|
||||
}
|
||||
|
||||
@test "containerd socket fallback fires when systemctl is unavailable" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
# The script uses `[ -e "$sock" ]`, not `[ -S ... ]`, so a regular
|
||||
# file is a valid stand-in for a unix socket in tests. This also
|
||||
# removes any optional runtime dependency on python3 and makes the
|
||||
# test unconditional on every CI runner.
|
||||
SOCK="$STUB_DIR/containerd.sock"
|
||||
touch "$SOCK"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 \
|
||||
COZYSTACK_CONTAINERD_SOCKET="$SOCK" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
grep -q 'standalone containerd.service' "$STDERR_FILE"
|
||||
}
|
||||
|
||||
@test "docker socket fallback fires when systemctl is unavailable" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
SOCK="$STUB_DIR/docker.sock"
|
||||
touch "$SOCK"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 \
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$SOCK" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
grep -q 'standalone docker.service' "$STDERR_FILE"
|
||||
if grep -q 'standalone containerd.service' "$STDERR_FILE"; then
|
||||
echo "unexpected containerd warning found:" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "clean host without systemctl exits silently" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 \
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker1.sock $STUB_DIR/missing-docker2.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
bash hack/check-host-runtime.sh >"$STUB_DIR/stdout" 2>"$STDERR_FILE"
|
||||
|
||||
[ ! -s "$STDERR_FILE" ]
|
||||
[ ! -s "$STUB_DIR/stdout" ]
|
||||
}
|
||||
|
||||
@test "docker service plus socket still emits exactly one warning" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$1" = "is-active" ] && [ "$2" = "docker.service" ]; then
|
||||
echo active
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
SOCK="$STUB_DIR/docker.sock"
|
||||
touch "$SOCK"
|
||||
|
||||
mkdir -p "$STUB_DIR/var-lib-docker"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$SOCK" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/var-lib-docker" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
count=$(grep -c 'standalone docker.service' "$STDERR_FILE")
|
||||
if [ "$count" != "1" ]; then
|
||||
echo "expected exactly one docker warning, got $count" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "docker socket paths with glob chars do not expand" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
# Create two directories that a naive `for sock in $PATHS` loop
|
||||
# would glob-expand and treat as existing "sockets". With the
|
||||
# array-based parsing the literal path "$STUB_DIR/var-lib-*" does
|
||||
# not exist and no warning must fire.
|
||||
mkdir -p "$STUB_DIR/var-lib-docker" "$STUB_DIR/var-lib-containerd"
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$STUB_DIR/missing-containerd.sock" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/var-lib-*" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/missing-containerd-dir" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
if grep -q 'standalone docker.service' "$STDERR_FILE"; then
|
||||
echo "glob pattern expanded — docker warning should not fire:" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
@test "containerd service plus socket still emits exactly one warning" {
|
||||
STUB_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$STUB_DIR"' EXIT
|
||||
|
||||
cat >"$STUB_DIR/systemctl" <<'STUBEOF'
|
||||
#!/bin/sh
|
||||
if [ "$1" = "--version" ]; then
|
||||
echo "systemd stub"
|
||||
exit 0
|
||||
fi
|
||||
if [ "$1" = "is-active" ] && [ "$2" = "containerd.service" ]; then
|
||||
echo active
|
||||
exit 0
|
||||
fi
|
||||
exit 1
|
||||
STUBEOF
|
||||
chmod +x "$STUB_DIR/systemctl"
|
||||
|
||||
# The script uses `[ -e "$sock" ]`, not `[ -S ... ]`, so a regular
|
||||
# file is a valid stand-in for a unix socket in tests. This also
|
||||
# removes any optional runtime dependency on python3 and makes the
|
||||
# test unconditional on every CI runner.
|
||||
SOCK="$STUB_DIR/containerd.sock"
|
||||
touch "$SOCK"
|
||||
|
||||
mkdir -p "$STUB_DIR/var-lib-containerd"
|
||||
|
||||
STDERR_FILE="$STUB_DIR/stderr"
|
||||
COZYSTACK_CONTAINERD_SOCKET="$SOCK" \
|
||||
COZYSTACK_DOCKER_SOCKET_PATHS="$STUB_DIR/missing-docker.sock" \
|
||||
COZYSTACK_CONTAINERD_DIR="$STUB_DIR/var-lib-containerd" \
|
||||
COZYSTACK_DOCKER_DIR="$STUB_DIR/missing-docker-dir" \
|
||||
PATH="$STUB_DIR:$PATH" \
|
||||
bash hack/check-host-runtime.sh 2>"$STDERR_FILE"
|
||||
|
||||
count=$(grep -c 'standalone containerd.service' "$STDERR_FILE")
|
||||
if [ "$count" != "1" ]; then
|
||||
echo "expected exactly one containerd warning, got $count" >&2
|
||||
cat "$STDERR_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
161
hack/check-host-runtime.sh
Executable file
161
hack/check-host-runtime.sh
Executable file
|
|
@ -0,0 +1,161 @@
|
|||
#!/usr/bin/env bash
|
||||
# -----------------------------------------------------------------------------
|
||||
# check-host-runtime.sh — operator preflight warning
|
||||
#
|
||||
# Purpose:
|
||||
# Warn when a standalone containerd.service or docker.service is running on
|
||||
# the host alongside the embedded k3s runtime. This mismatch is silent on
|
||||
# day 0 (k3s uses its own containerd at /run/k3s/containerd/containerd.sock
|
||||
# and /var/lib/rancher/k3s/agent/containerd) but over time the standalone
|
||||
# runtime accumulates unpruned images and build cache in /var/lib/containerd
|
||||
# — enough to trigger DiskPressure and crash cozystack-api with eviction
|
||||
# loops. The script does NOT block install; it only prints a warning.
|
||||
#
|
||||
# When to run:
|
||||
# Before `helm install cozy-installer` on an Ubuntu host prepared with k3s
|
||||
# or kubeadm (cozystack "generic" variant). Irrelevant on Talos where the
|
||||
# container runtime lifecycle is fully managed. Discoverable via
|
||||
# `make preflight` from the repository root.
|
||||
#
|
||||
# Exit code:
|
||||
# Always 0 (warning, not a blocker). Warnings go to stderr.
|
||||
#
|
||||
# Environment variables (test hooks — override default probe paths):
|
||||
# COZYSTACK_CONTAINERD_SOCKET standalone containerd socket path
|
||||
# COZYSTACK_DOCKER_SOCKET_PATHS space-separated list of docker socket paths
|
||||
# COZYSTACK_CONTAINERD_DIR standalone containerd data directory
|
||||
# COZYSTACK_DOCKER_DIR standalone docker data directory
|
||||
# COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 pretend systemctl is absent
|
||||
# -----------------------------------------------------------------------------
|
||||
set -euo pipefail
|
||||
|
||||
if [ -t 2 ]; then
|
||||
YELLOW=$'\033[1;33m'
|
||||
RESET=$'\033[0m'
|
||||
else
|
||||
YELLOW=''
|
||||
RESET=''
|
||||
fi
|
||||
|
||||
CONTAINERD_SOCKET=${COZYSTACK_CONTAINERD_SOCKET:-/run/containerd/containerd.sock}
|
||||
DOCKER_SOCKET_PATHS=${COZYSTACK_DOCKER_SOCKET_PATHS:-/run/docker.sock /var/run/docker.sock}
|
||||
CONTAINERD_DIR=${COZYSTACK_CONTAINERD_DIR:-/var/lib/containerd}
|
||||
DOCKER_DIR=${COZYSTACK_DOCKER_DIR:-/var/lib/docker}
|
||||
|
||||
CONTAINERD_WARN=0
|
||||
DOCKER_WARN=0
|
||||
|
||||
warn() {
|
||||
printf '%sWARNING:%s %s\n' "$YELLOW" "$RESET" "$1" >&2
|
||||
}
|
||||
|
||||
detect_systemctl() {
|
||||
if [ "${COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL:-0}" = "1" ]; then
|
||||
return 1
|
||||
fi
|
||||
if command -v systemctl >/dev/null 2>&1 && systemctl --version >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
disk_usage() {
|
||||
local path=$1
|
||||
local usage
|
||||
if [ -d "$path" ]; then
|
||||
# Wrap `du` in `timeout 5s` so a container data directory with
|
||||
# millions of files (the exact scenario this script exists to
|
||||
# warn about) cannot stall the preflight indefinitely. If the
|
||||
# `timeout` binary is absent the pipeline still exits 0 via
|
||||
# `|| true` and `usage` stays empty; the warning itself is
|
||||
# still printed, just without the size detail.
|
||||
usage=$(timeout 5s du -sh "$path" 2>/dev/null | awk '{print $1}' || true)
|
||||
if [ -n "${usage:-}" ]; then
|
||||
printf ' (%s uses %s)' "$path" "$usage"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
service_active() {
|
||||
local service=$1
|
||||
if [ "$HAS_SYSTEMCTL" = "1" ]; then
|
||||
if systemctl is-active "$service" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
check_containerd() {
|
||||
local detail=""
|
||||
local found=0
|
||||
if service_active containerd.service; then
|
||||
found=1
|
||||
fi
|
||||
if [ "$found" -eq 0 ] && [ -e "$CONTAINERD_SOCKET" ]; then
|
||||
found=1
|
||||
fi
|
||||
if [ "$found" -eq 1 ]; then
|
||||
detail=$(disk_usage "$CONTAINERD_DIR")
|
||||
warn "standalone containerd.service detected alongside k3s embedded runtime${detail}"
|
||||
CONTAINERD_WARN=1
|
||||
fi
|
||||
}
|
||||
|
||||
check_docker() {
|
||||
local detail=""
|
||||
local found=0
|
||||
if service_active docker.service; then
|
||||
found=1
|
||||
fi
|
||||
if [ "$found" -eq 0 ]; then
|
||||
# DOCKER_SOCKET_PATHS is a space separated list of paths. Parse
|
||||
# it into an array via `read -ra` so that word splitting is
|
||||
# explicit AND glob expansion is suppressed — `for sock in
|
||||
# $DOCKER_SOCKET_PATHS` would both word split and glob, so a
|
||||
# path containing a literal `*` or `?` could expand into
|
||||
# directory entries and produce false positives.
|
||||
local -a _socks
|
||||
read -ra _socks <<<"$DOCKER_SOCKET_PATHS"
|
||||
for sock in "${_socks[@]}"; do
|
||||
if [ -e "$sock" ]; then
|
||||
found=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
if [ "$found" -eq 1 ]; then
|
||||
detail=$(disk_usage "$DOCKER_DIR")
|
||||
warn "standalone docker.service detected alongside k3s embedded runtime${detail}"
|
||||
DOCKER_WARN=1
|
||||
fi
|
||||
}
|
||||
|
||||
if detect_systemctl; then
|
||||
HAS_SYSTEMCTL=1
|
||||
else
|
||||
HAS_SYSTEMCTL=0
|
||||
fi
|
||||
|
||||
check_containerd
|
||||
check_docker
|
||||
|
||||
if [ "$CONTAINERD_WARN" -eq 1 ] || [ "$DOCKER_WARN" -eq 1 ]; then
|
||||
services=""
|
||||
if [ "$CONTAINERD_WARN" -eq 1 ]; then
|
||||
services="containerd.service"
|
||||
fi
|
||||
if [ "$DOCKER_WARN" -eq 1 ]; then
|
||||
if [ -n "$services" ]; then
|
||||
services="$services docker.service"
|
||||
else
|
||||
services="docker.service"
|
||||
fi
|
||||
fi
|
||||
printf '%sHINT:%s cozystack runs its own containerd under k3s. To stop the shadow runtime:\n' "$YELLOW" "$RESET" >&2
|
||||
printf ' sudo systemctl disable --now %s\n' "$services" >&2
|
||||
printf 'Inspect and reclaim standalone runtime storage separately — it may contain container data\n' >&2
|
||||
printf 'that the operator still needs; do not delete it blindly.\n' >&2
|
||||
fi
|
||||
|
||||
exit 0
|
||||
Loading…
Add table
Add a link
Reference in a new issue