The installer was trying to write binaries to /opt/pulse/sensor-proxy/bin/
before creating the directory structure, causing 'No such file or directory'
errors on fresh installs.
Moved directory creation for INSTALL_ROOT and bin/ to before binary installation
section (before line 657), ensuring directories exist before use.
Related to cleanup implementation testing.
Extends cleanup script to completely remove Pulse footprint from hosts
when nodes are removed, not just SSH keys. Now removes: SSH keys, proxy
service, binaries, API tokens, pulse-monitor user, and LXC bind mounts.
Key improvements:
1. **flock Serialization**: Prevents concurrent cleanup runs
- Acquires exclusive lock on cleanup.lock file
- Prevents race conditions and cleanup loops
2. **Immediate Request File Deletion**: Delete cleanup-request.json
before any long-running operations to prevent re-triggering
3. **API Token Cleanup**: Removes all pulse-monitor@pam API tokens
- Tries JSON output first (Proxmox 7.0+)
- Falls back to table parsing with proper filtering (no decoration chars)
- Deletes pulse-monitor@pam user after removing all tokens
4. **LXC Bind Mount Removal**: Scans all container configs and removes
pulse-sensor-proxy bind mount entries
5. **Process Isolation for Uninstaller**: Uses systemd-run to spawn
isolated transient unit that won't be killed when proxy service stops
- Unit name: pulse-uninstall-{timestamp}
- Properties: Type=oneshot, Conflicts=pulse-sensor-proxy.service
- Runs non-blocking so cleanup service can exit cleanly
- Falls back to direct call if systemd-run unavailable
6. **Complete Service/Binary Removal**: Calls installer's --uninstall
- Stops and disables pulse-sensor-proxy.service
- Removes all systemd units
- Deletes all binaries from /opt/pulse/sensor-proxy/
- Removes configuration files
- Cleans up directories
Changes to cleanup script logic:
- Added LOCKFILE and INSTALLER_PATH configuration
- Acquire flock before processing (prevents concurrent runs)
- Delete request file immediately after reading
- Full localhost cleanup: SSH keys → API tokens → bind mounts → uninstall
- Remote cleanup still SSH-key-only (can't orchestrate uninstall remotely)
- Better error handling with appropriate log levels
Updated cleanup service unit:
- ExecStart now uses ${CLEANUP_SCRIPT_PATH} variable (new /opt location)
- Changed heredoc from 'SERVICE_EOF' to SERVICE_EOF for variable expansion
Addresses all issues documented in CLEANUP_TODO.md:
- ✅ Read-only filesystem (binaries now in /opt, removable)
- ✅ Process isolation (systemd-run transient unit)
- ✅ Cleanup loops (flock + immediate file deletion)
- ✅ API token parsing (JSON first, filtered table fallback)
The UI message is now accurate: "Removing this proxmox ve node also
scrubs the Pulse footprint on the host — the proxy service, SSH key,
API token, and bind mount are all cleaned up automatically."
Part of: CLEANUP_TODO.md Phase 2-4
Supersedes: ed65fda74 (original cleanup attempt with process issues)
Depends on: b192c60e9 (binary relocation to /opt)
Moves all sensor-proxy binaries and scripts from /usr/local/bin to
/opt/pulse/sensor-proxy/bin to ensure they can be removed during cleanup
even on systems with read-only /usr (hardened Proxmox setups).
Changes:
- INSTALL_ROOT=/opt/pulse/sensor-proxy (new writable location)
- Binary path: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy
- Wrapper script: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh
- Cleanup script: /opt/pulse/sensor-proxy/bin/pulse-sensor-cleanup.sh
- Selfheal script: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy-selfheal.sh
- Installer storage: /opt/pulse/sensor-proxy/install-sensor-proxy.sh
Updated:
- Directory creation to include ${INSTALL_ROOT}/bin
- Systemd service ExecStart paths to use ${BINARY_PATH}
- Self-heal service ExecStart to use ${SELFHEAL_SCRIPT}
- Changed heredoc delimiters from 'EOF' to EOF for variable expansion
Rationale:
Proxmox VE can mount /usr as read-only in hardened configurations.
The previous /usr/local/bin location prevented complete uninstallation
on these systems, violating Pulse's correctness principle. The /opt
location is guaranteed writable and appropriate for third-party software.
This is Phase 1 of implementing full cleanup functionality per
CLEANUP_TODO.md. Subsequent commits will add process isolation,
API token deletion, and bind mount removal.
Part of: #CLEANUP_TODO.md Phase 1
Related: ed65fda74 (original cleanup attempt)
The update_allowed_nodes function was changing ownership of the temp file
before all writes were complete, causing 'Permission denied' errors when
appending the allowed_nodes section.
Root cause:
- mktemp creates file owned by script runner (root)
- chown changed ownership to pulse-sensor-proxy:pulse-sensor-proxy
- Subsequent append (>>) failed because root can't write to the file
Fix: Defer chown until after all writes complete and file is moved to
final location. Ownership is still correctly set on the final config file.
When a Proxmox node is removed from Pulse, the cleanup now performs full uninstallation:
- SSH keys removal (existing functionality)
- Uninstalls pulse-sensor-proxy service
- Removes LXC bind mounts from container configs
- Deletes Proxmox API tokens
- Removes pulse-monitor@pam user
This aligns with security best practices and user expectations - "remove node"
should completely sever trust with that machine, not leave credentials and
privileged services behind.
The cleanup script now calls the uninstaller (--uninstall) and uses pveum
to remove API tokens. This prevents leftover artifacts if the host is
repurposed or compromised.
Related: config_handlers.go triggerPVEHostCleanup() at node deletion
- Change mktemp to use /tmp/pulse-config.XXXXXXXXXX template
- Prevents conflicts with stale temp files from previous runs
- Fixes 'Permission denied' errors when script re-runs
- Add trap to remove temp file on function return (success or failure)
- Add error check for mv command with descriptive message
- Ensure config file has proper permissions after update
This prevents orphaned temp files when errors occur and provides
better diagnostics when file operations fail.
The all_nodes arrays were declared with 'local' keyword outside of
functions, causing bash syntax error:
'local: can only be used in a function'
Fixed by removing 'local' keyword - arrays in main script scope don't
need it and it's actually invalid syntax.
The awk logic was removing allowed_nodes sections but leaving their
comment headers behind. When multiple sections existed, comments would
accumulate.
New approach:
- Buffer all comment lines encountered outside sections
- When a non-comment line is found, flush buffered comments
- When allowed_nodes is found, discard buffered comments (they belonged
to the section we're removing)
- This cleanly removes section headers like:
'# Cluster nodes (auto-discovered during installation)'
'# These nodes are allowed to request...'
Tested with config containing duplicate allowed_nodes sections - now
correctly produces clean output with all duplicates and headers removed.
The installer was only creating base config.yaml in standalone mode,
but update_allowed_nodes() is also called in LXC mode. When the config
didn't exist, update_allowed_nodes() would create an empty file and only
add the allowed_nodes section, missing required fields like
allowed_peer_uids, metrics_address, rate_limit, etc.
This caused the proxy to fail when it tried to parse the incomplete config.
Now creates a proper base config with all required fields if the file
doesn't exist, before any mode-specific configuration is added.
The install-sensor-proxy.sh script was blindly appending allowed_nodes
sections to the config file without checking if they already existed.
When the script was re-run or if the initial config already had an
allowed_nodes section, this created duplicate YAML keys that caused
the proxy service to fail with parse errors.
Changes:
- Add update_allowed_nodes() helper function that safely updates the
allowed_nodes section by removing any existing ones first
- Replace all three cat >> config.yaml heredocs with calls to the
helper function (cluster nodes, standalone mode, pvecm fallback)
- Uses awk to properly parse and remove multi-line YAML sections
This makes the installer idempotent and prevents config corruption on
re-runs.
Fixes issue where proxy service crashed with:
'mapping key "allowed_nodes" already defined at line X'
The HTTP mode installer now includes 127.0.0.1/32 in allowed_source_subnets
to permit self-monitoring queries from localhost. This fixes 403 Forbidden
errors when nodes query their own sensor-proxy instance.
Related to HTTP mode implementation for external PVE hosts.
## HTTP Server Fixes
- Add source IP middleware to enforce allowed_source_subnets
- Fix missing source subnet validation for external HTTP requests
- HTTP health endpoint now respects subnet restrictions
## Installer Improvements
- Auto-configure allowed_source_subnets with Pulse server IP
- Add cluster node hostnames to allowed_nodes (not just IPs)
- Fix node validation to accept both hostnames and IPs
- Add Pulse server reachability check before installation
- Add port availability check for HTTP mode
- Add automatic rollback on service startup failure
- Add HTTP endpoint health check after installation
- Fix config backup and deduplication (prevent duplicate keys)
- Fix IPv4 validation with loopback rejection
- Improve registration retry logic with detailed errors
- Add automatic LXC bind mount cleanup on uninstall
## Temperature Collection Fixes
- Add local temperature collection for self-monitoring nodes
- Fix node identifier matching (use hostname not SSH host)
- Fix JSON double-encoding in HTTP client response
Related to #XXX (temperature monitoring fixes)
When cluster IPC validation fails (due to systemd hardening), the proxy
falls back to allowlist-based validation. The installer now automatically
populates allowed_nodes with:
- Cluster mode: All discovered cluster member IPs
- Standalone mode: localhost IP addresses (including 127.0.0.1/localhost)
- Fallback mode: localhost IPs when pvecm unavailable
This ensures out-of-the-box temperature monitoring works on fresh installs
without manual configuration.
Root cause: pulse-sensor-proxy runs with strict systemd hardening that prevents
access to Proxmox corosync IPC (abstract UNIX sockets). When pvecm fails with
IPC errors, the code incorrectly treated it as "standalone mode" and only
discovered localhost addresses, rejecting legitimate cluster members and external
nodes.
Changes:
1. **Distinguish IPC failures from true standalone mode**
- Detect ipcc_send_rec and access control list errors specifically
- These indicate a cluster exists but isn't accessible (LXC, systemd restrictions)
- Return error to disable cluster validation instead of misusing standalone logic
2. **Graceful degradation when cluster validation fails**
- When cluster IPC is unavailable, fall through to permissive mode
- Log debug message suggesting allowed_nodes configuration
- Allows requests to proceed rather than blocking all temperature monitoring
3. **Improve local address discovery for true standalone nodes**
- Use Go's native net.Interfaces() instead of shelling out to 'ip addr'
- More reliable and works with AF_NETLINK restrictions
- Add helpful logging when only hostnames are discovered
4. **Systemd hardening adjustments**
- Add AF_NETLINK to RestrictAddressFamilies (for net.Interfaces())
- Remove RemoveIPC=true (attempted fix for corosync, insufficient)
- Add ReadWritePaths=-/run/corosync (optional path, corosync uses abstract sockets anyway)
Result: Temperature monitoring now works in:
- Clustered Proxmox hosts (falls back to permissive when IPC blocked)
- LXC containers (correctly detects IPC failure, allows requests)
- Standalone nodes (proper local address discovery with IPs)
Workaround for maximum security: Configure allowed_nodes in /etc/pulse-sensor-proxy/config.yaml
when cluster validation cannot be used.
Root cause: The systemd service hardening blocked AF_NETLINK sockets,
preventing IP address discovery on standalone nodes. The proxy could
only discover hostnames, causing node_not_cluster_member rejections
when users configured Pulse with IP addresses.
Changes:
1. Add AF_NETLINK to RestrictAddressFamilies in all systemd services
- pulse-sensor-proxy.service
- install-sensor-proxy.sh (both modes)
- pulse-sensor-cleanup.service
2. Replace shell-based 'ip addr' with Go native net.Interfaces() API
- More reliable and doesn't require external commands
- Works even with strict systemd restrictions
- Properly filters loopback, link-local, and down interfaces
3. Improve error logging and user guidance
- Warn when no IP addresses can be discovered
- Provide clear instructions about allowed_nodes workaround
- Include address counts in logs for debugging
This fix ensures standalone Proxmox nodes can properly validate
temperature requests by IP address without requiring manual
allowed_nodes configuration.
- Replace unreliable git fetch --dry-run check
- Use git rev-parse to compare local and remote commits
- Prevents false warnings about diverged branches
- Check VERSION file matches before triggering workflow
- Validate working directory is clean
- Confirm on main branch and up to date
- Load release notes from /tmp/release_notes_X.Y.Z.md
- Prevents wasting CI time on misconfigured releases
Users were abandoning Pulse due to catastrophic temperature monitoring setup failures. This commit addresses the root causes:
**Problem 1: Silent Failures**
- Installations reported "SUCCESS" even when proxy never started
- UI showed green checkmarks with no temperature data
- Zero feedback when things went wrong
**Problem 2: Missing Diagnostics**
- Service failures logged only in journald
- Users saw "Something going on with the proxy" with no actionable guidance
- No way to troubleshoot from error messages
**Problem 3: Standalone Node Issues**
- Proxy daemon logged continuous pvecm errors as warnings
- "ipcc_send_rec" and "Unknown error -1" messages confused users
- These are expected for non-clustered/LXC setups
**Solutions Implemented:**
1. **Health Gate in install.sh (lines 1588-1629)**
- Verify service is running after installation
- Check socket exists on host
- Confirm socket visible inside container via bind mount
- Fail loudly with specific diagnostics if any check fails
2. **Actionable Error Messages in install-sensor-proxy.sh (lines 822-877)**
- When service fails to start: dump full systemctl status + 40 lines of logs
- When socket missing: show permissions, service status, and remediation command
- Include common issues checklist (missing user, permission errors, lm-sensors, etc.)
- Direct link to troubleshooting docs
3. **Better Standalone Node Detection in ssh.go (lines 585-595)**
- Recognize "Unknown error -1" and "Unable to load access control list" as LXC indicators
- Log at INFO level (not WARN) since this is expected behavior
- Clarify message: "using localhost for temperature collection"
**Impact:**
- Eliminates "green checkmark but no temps" scenario
- Users get immediate actionable feedback on failures
- Standalone/LXC installations work silently without error spam
- Reduces support burden from #571 (15+ comments of user frustration)
Related to #571
Snap-installed Docker does not automatically create a docker group,
causing permission denied errors when the pulse-docker service user
tries to access /var/run/docker.sock.
Changes:
- Auto-detect Snap Docker installations
- Create docker group if missing when Snap Docker is detected
- Restart Snap Docker after group creation to refresh socket ACLs
- Add socket access validation before starting the service
- Handle symlinked Docker sockets in systemd unit ReadWritePaths
- Document troubleshooting steps in DOCKER_MONITORING.md
Bare metal installations couldn't serve Windows host agent downloads because
the Windows and macOS binaries weren't included in the universal tarball. The
download endpoint would return 404 when Windows users tried to install the
host agent from a bare metal Pulse deployment (Proxmox LXC, Debian VM, etc.).
Changes:
- build-release.sh: Copy Windows/macOS host agent binaries into universal tarball
- build-release.sh: Create symlinks for Windows binaries without .exe extension
- validate-release.sh: Add Windows 386 binary and symlink to Docker validation
- validate-release.sh: Add explicit validation that universal tarball contains all Windows/macOS binaries
The universal tarball now matches the Docker image, ensuring both deployment
methods can serve the complete set of downloadable binaries for the /download/
endpoint.
Replace mv with install command to ensure correct SELinux context.
The mv command preserves the user_tmp_t label from /tmp, which
prevents systemd from executing the binary on SELinux systems.
The install command creates a new file with the correct label for
/usr/local/bin. Added automatic restorecon call for SELinux systems
to ensure policy compliance.
Related to #688
Following best practices for release format transitions:
- build-release.sh now generates both formats from same sha256sum run
- Workflow uploads both checksums.txt and individual .sha256 files
- Validation ensures both formats exist and match
This provides a safe transition period for users with older install scripts
while maintaining the cleaner checksums.txt format going forward. After 2-3
releases when most users have updated scripts, we can remove .sha256 generation.
Related: Install script already supports both formats (falls back gracefully).
Linux host-agent binaries don't have separate archives - they're included in
the main pulse-v*.tar.gz files. Only macOS and Windows have separate archives.
Removed validation checks for standalone binaries that are no longer
uploaded to GitHub releases. These binaries are only needed in Docker
images for the /download/ endpoint.
Updated required assets list to include all versioned tarballs/zips
instead of standalone binaries.
Removed:
- Individual .sha256 files (checksums.txt already contains all checksums)
- Standalone binaries without version numbers (users should download versioned tarballs/zips)
Standalone binaries are only needed in Docker images for the /download/ endpoint.
GitHub releases should only contain versioned archives for user downloads.
This reduces release assets from ~54 files to ~19 files per release.
Users don't care about CI/CD improvements, release workflows, build
processes, or testing infrastructure. Only include user-visible changes.
Related to #671
Commit hashes clutter the release notes and aren't useful for end users.
Only include issue references when explicitly mentioned in commits.
Related to #671
Remove # symbol from commit hash references so GitHub auto-links them.
Format: (abc123) instead of (#abc123)
Issue references still use #: (#123)
Related to #671
- Use exact template format from v4.28.0 and prior releases
- Include all standard sections: New Features, Bug Fixes, Improvements, Breaking Changes
- Add complete installation instructions (systemd, Docker, Manual Binary, Helm)
- Include Downloads section with all artifact types
- Add Notes section for important highlights and upgrade considerations
- Ensure LLM outputs format exactly matching previous releases
Related to #671 (automated release workflow)
- Create scripts/generate-release-notes.sh to auto-generate release notes from git commits
- Supports both Anthropic Claude and OpenAI APIs
- Uses Claude Haiku 4.5 (claude-haiku-4-5-20251001) for cost efficiency ($1/$5 per million tokens)
- Falls back to OpenAI gpt-4o-mini if Anthropic key not available
- Integrates into release workflow between validation and release creation
- Compares current version with previous git tag to generate changelog
- Outputs categorized, user-friendly release notes with installation instructions
Workflow now automatically:
1. Finds previous release tag
2. Analyzes all commits since last release
3. Generates structured release notes via LLM
4. Uses generated notes for draft release body
Requires ANTHROPIC_API_KEY or OPENAI_API_KEY in GitHub secrets.
Related to #671 (automated release workflow)
The script does pushd into RELEASE_DIR, so tarball paths should not include
the RELEASE_DIR prefix. Also fixed checksum validation glob patterns to
exclude .sha256 files from matching.
Tarballs are created with ./bin/pulse paths (relative from inside staging dir)
but validation was looking for bin/pulse paths. Updated all tar -tzf checks
to use correct ./ prefix.
The validation script was looking for tarballs in the current directory
instead of the release/ directory, causing all validations to fail.
Now properly prepends $RELEASE_DIR to all file paths.
