Fix auto-update leaving service down after installer failure
Some checks are pending
Build and Test / Secret Scan (push) Waiting to run
Build and Test / Frontend & Backend (push) Waiting to run
Canonical Governance / governance (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 1/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 2/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 3/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 4/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 5/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 6/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 7/8) (push) Waiting to run
Core E2E Tests / Playwright Core E2E (shard 8/8) (push) Waiting to run
Core E2E Tests / Agent registration lifecycle (push) Waiting to run
Core E2E Tests / E2E verdict (push) Blocked by required conditions
Unified Agent Native Verification / Linux ARM64 (push) Waiting to run
Unified Agent Native Verification / Linux x64 (push) Waiting to run
Unified Agent Native Verification / Windows x64 (push) Waiting to run
Unified Agent Native Verification / macOS ARM64 (push) Waiting to run
Unified Agent Native Verification / macOS Intel (push) Waiting to run
Unified Agent Native Verification / FreeBSD cross-build contract (push) Waiting to run

Two bugs from #1630 that combined to take installs down silently:

1. perform_update()'s install-failed rollback branch restored the backup
   but never restarted pulse.service. Since the generated
   pulse-update.service gates on ExecCondition=systemctl is-active,
   every later timer run was then skipped and the install stayed down
   until manual intervention. Restart is now guaranteed by a
   service_was_active-guarded restart in that branch plus an
   ensure_service_restarted RETURN trap so no exit path can miss it
   (re-fix of #1323, originally c0b3a0e66, lost in 778a2577b and only
   partially restored in 672e81985).

2. install.sh aborted under errexit when writing the /bin/update helper
   on a read-only filesystem - after the new binary was installed and
   the service stopped, landing in bug 1's no-restart branch. The stock
   pulse-update.service uses ProtectSystem=strict, so /bin and
   /usr/local/bin are read-only on stock unattended updates; transient
   read-only remounts hit the same path. The helper write, PATH
   appends, and the /usr/local/bin/pulse symlink are now idempotent and
   non-fatal with a warning (install_binary_symlink).

Contract: deployment-installability now pins fail-closed service
availability for unattended updates and non-fatal writes outside the
hardened unit's writable set, with proofs in pulse_auto_update_test.go
and root_install_sh_test.go plus shell regression coverage in
scripts/tests/test-pulse-auto-update.sh (installer-exits-nonzero path)
and scripts/tests/test-install-update-resilience.sh (read-only helper
and symlink paths, verified under set -e, root-safe via ENOTDIR).

Fixes #1630

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
courtmanr@gmail.com 2026-07-27 09:28:45 +01:00
parent bf24a9a9cf
commit 806cbe83dc
7 changed files with 569 additions and 15 deletions

View file

@ -346,6 +346,35 @@ wait_for_service_active() {
return 1
}
# Guarantee the Pulse service is left running after an update attempt if (and
# only if) it was running beforehand. Invoked from perform_update's RETURN trap
# so that no exit path — present or future — can leave Pulse stopped (#1630:
# the install-failed rollback branch restored the backup but never restarted
# the service). This matters doubly because the generated pulse-update.service
# uses ExecCondition=systemctl is-active pulse.service: once Pulse is down,
# every subsequent timer run is skipped and the install stays down until
# someone intervenes. Always returns 0 (it runs under set -e in a trap).
ensure_service_restarted() {
local service_name=$1
local service_was_active=$2
if [[ "$service_was_active" != "true" ]]; then
return 0
fi
if systemctl is-active --quiet "$service_name" 2>/dev/null; then
return 0
fi
log warn "Pulse service is not active after update attempt; starting it"
systemctl start "$service_name" 2>/dev/null || true
if wait_for_service_active "$service_name" 20; then
log info "Pulse service is running again"
else
log error "Pulse service could not be started after update attempt; manual intervention required (systemctl start $service_name)"
fi
return 0
}
perform_update() {
local new_version=$1
local service_name=$(detect_service_name)
@ -360,6 +389,11 @@ perform_update() {
service_was_active="true"
fi
# Whatever way this function exits, never leave Pulse stopped when it was
# running before the update (#1630). Extended below once the installer
# tempfiles exist.
trap 'ensure_service_restarted "$service_name" "$service_was_active"' RETURN
# Refuse to install a prerelease via the unattended updater. The stable
# channel must never cross onto a tag like v6.0.0-rc.2, even if every
# caller above this point thought it was safe.
@ -406,7 +440,7 @@ perform_update() {
installer_tmp=$(mktemp /tmp/pulse-update-installer.XXXXXX)
signature_tmp=$(mktemp /tmp/pulse-update-installer.sig.XXXXXX)
trap 'rm -f "$installer_tmp" "$signature_tmp"' RETURN
trap 'rm -f "$installer_tmp" "$signature_tmp"; ensure_service_restarted "$service_name" "$service_was_active"' RETURN
if ! curl -fsSL "$install_script_url" -o "$installer_tmp"; then
log error "Failed to download installer from $install_script_url"
@ -510,10 +544,18 @@ perform_update() {
if [[ -f "$backup_dir/VERSION" ]]; then
cp -f "$backup_dir/VERSION" "$INSTALL_DIR/VERSION"
fi
# Restart the restored binary if Pulse was running before the update.
# The installer stops the service before it can fail, so skipping this
# left Pulse down indefinitely (#1630); the RETURN trap above is the
# backstop if this path ever changes.
if [[ "$service_was_active" == "true" ]]; then
systemctl restart "$service_name" || true
fi
# Clean up backup
rm -rf "$backup_dir"
return 1
fi
}