Commit graph

117 commits

Author SHA1 Message Date
rcourtman
a236d308d3 Gracefully handle missing PVE instance during proxy registration 2025-11-15 22:25:50 +00:00
rcourtman
0d70063642 Fix proxy installer dedupe 2025-11-15 22:04:36 +00:00
rcourtman
47d5c14aef Improve temperature proxy control-plane flow 2025-11-15 21:49:51 +00:00
rcourtman
ad35a60cfe Ensure sensor proxy installer configures Pulse env 2025-11-15 18:28:42 +00:00
rcourtman
48799d74a4 Ensure sensor proxy installer configures Pulse env 2025-11-15 18:23:40 +00:00
rcourtman
1f55a44547 Deduplicate allowed_nodes when installing sensor proxy 2025-11-15 18:14:38 +00:00
rcourtman
b90ee83ef3 Fix installer adding invalid hostname entries to allowed_nodes
The installer was adding node hostnames (and accidentally the header "Name")
to allowed_nodes in addition to IPs. This caused:
1. Invalid entries like "Name", "minipc", "delly" in config
2. These are not valid for SSH temperature collection

Only IPs should be in allowed_nodes since that's what the proxy uses for SSH.
Removed the loop that added CLUSTER_NODE_NAMES to the array.

Also fixed: Removed extraction of CLUSTER_NODE_NAMES since it's no longer used.
2025-11-15 10:07:22 +00:00
rcourtman
2d3d5fab8c Fix cleanup systemd-run deadlock
Problem:
Cleanup script uses systemd-run with both --wait and
--property="After=pulse-sensor-cleanup.service", creating a circular
dependency:
- cleanup.service runs and waits for uninstaller to complete
- uninstaller has After=cleanup.service, so it waits for cleanup to finish
- Result: Both services stuck waiting for each other

Fix:
Remove the --property="After=pulse-sensor-cleanup.service" line. The
Conflicts=pulse-sensor-proxy.service is sufficient to ensure the proxy
stops before uninstallation. The cleanup script doesn't need to finish
before the uninstaller starts.

Testing:
Cleanup now completes successfully, removing all artifacts:
- Systemd units removed
- Binaries deleted from /opt/pulse/sensor-proxy/
- Data directory /var/lib/pulse-sensor-proxy/ removed
- SSH keys cleaned from authorized_keys
- pulse-monitor user and API tokens deleted
- LXC bind mounts removed from container configs

Related to #605 (temperature monitoring cleanup)
2025-11-15 09:03:17 +00:00
rcourtman
ce88da77de Fix missed /usr/local path migration and add backward compatibility
**Missed Migration**:
- Line 2204 still used /usr/local/bin/pulse-sensor-wrapper.sh in fallback path
- Updated to use /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh

**Backward Compatibility**:
- When pushing SSH keys to cluster nodes, installer now checks if remote node
  has old installation (/usr/local/bin wrapper exists but /opt path doesn't)
- Automatically creates symlink on remote nodes to maintain compatibility
- Prevents temperature collection failures when cluster has mixed old/new installs

**Root Cause**:
When installer runs on upgraded node (delly), it pushes SSH keys with new forced
command path to all cluster nodes. If remote node (minipc) has old installation,
the forced command fails because wrapper doesn't exist at new path.

This fix ensures "it works straight out the box" by bridging old and new paths
automatically during SSH key deployment.
2025-11-15 08:37:44 +00:00
rcourtman
d9b830c3c3 Fix update_allowed_nodes to be properly idempotent
Rewrote AWK state machine to correctly handle:
- Multiple allowed_nodes sections (removes all of them)
- Comment lines immediately preceding allowed_nodes (discards them)
- Empty lines within allowed_nodes section
- Indented list items and comments

The function now:
1. Buffers comment lines that might precede allowed_nodes
2. When allowed_nodes: is detected, discards buffered comments
3. Skips all content until hitting a non-indented, non-comment line
4. Flushes buffered comments when hitting non-comment content

This ensures running the installer multiple times won't create duplicate
allowed_nodes sections in config.yaml.

Tested with script that verifies duplicate sections are removed correctly.
2025-11-15 08:03:28 +00:00
rcourtman
384c017f04 Address remaining Codex review findings
**LXC Bind Mount Removal**:
- Changed from sed to `pct set -delete` for safer mount removal
- Validates syntax and prevents breaking container configs
- Finds mount points by grepping for pulse-sensor-proxy, extracts mp number

**API Token Parsing** (three-tier fallback):
1. Try `pveum --output-format json` with python3 JSON parsing
2. Fall back to `pvesh get /access/users/pulse-monitor@pam/token` JSON API
3. Last resort: parse table output with improved filtering (handles more Unicode chars)

**Retry Logic**:
- Rename cleanup-request.json to .processing instead of deleting immediately
- Allows retry on failure (processing file persists if script crashes)
- Remove .processing file only on successful completion
- Prevents loops while enabling failure recovery

These complete all 8 issues identified by Codex review.
2025-11-15 00:35:22 +00:00
rcourtman
c1f636edb9 Fix critical cleanup implementation issues found by Codex review
**Host Detection**:
- Now detects localhost by hostname and FQDN, not just IP
- Fixes issue where nodes configured as https://hostname:8006 would skip
  localhost cleanup (API tokens, bind mounts, service removal)

**Systemd Sandbox**:
- Added /etc/pve and /etc/systemd/system to ReadWritePaths
- Allows cleanup script to modify Proxmox configs and systemd units

**Uninstaller Improvements**:
- Use UUID for transient unit names (prevents same-second collisions)
- Added --purge flag for complete removal
- Added --wait and --collect flags to capture exit code
- Now fails cleanup if uninstaller exits non-zero

**Path Migration**:
- Fixed all /usr/local references to use /opt/pulse/sensor-proxy
- Updated forced command in SSH authorized_keys
- Updated self-heal script installer path
- Updated Go backend removal helpers (supports both new and legacy paths)

These fixes address Codex findings: hostname detection, sandbox permissions,
transient unit collisions, incomplete purging, and incomplete path migration.

Related to cleanup implementation testing.
2025-11-15 00:33:41 +00:00
rcourtman
a63f5acd4d Fix unbound SHARE_DIR variable in installer
After relocating binaries to INSTALL_ROOT, the SHARE_DIR variable was removed
but one reference remained in cache_installer_for_self_heal() causing
'unbound variable' error.

Changed to use INSTALL_ROOT directly since that's where the cached installer
is stored (STORED_INSTALLER=${INSTALL_ROOT}/install-sensor-proxy.sh).
2025-11-15 00:09:34 +00:00
rcourtman
8bb20a7ae1 Fix directory creation order in sensor-proxy installer
The installer was trying to write binaries to /opt/pulse/sensor-proxy/bin/
before creating the directory structure, causing 'No such file or directory'
errors on fresh installs.

Moved directory creation for INSTALL_ROOT and bin/ to before binary installation
section (before line 657), ensuring directories exist before use.

Related to cleanup implementation testing.
2025-11-15 00:08:25 +00:00
rcourtman
7141c8b1b5 Implement full cleanup when nodes are removed from Pulse
Extends cleanup script to completely remove Pulse footprint from hosts
when nodes are removed, not just SSH keys. Now removes: SSH keys, proxy
service, binaries, API tokens, pulse-monitor user, and LXC bind mounts.

Key improvements:

1. **flock Serialization**: Prevents concurrent cleanup runs
   - Acquires exclusive lock on cleanup.lock file
   - Prevents race conditions and cleanup loops

2. **Immediate Request File Deletion**: Delete cleanup-request.json
   before any long-running operations to prevent re-triggering

3. **API Token Cleanup**: Removes all pulse-monitor@pam API tokens
   - Tries JSON output first (Proxmox 7.0+)
   - Falls back to table parsing with proper filtering (no decoration chars)
   - Deletes pulse-monitor@pam user after removing all tokens

4. **LXC Bind Mount Removal**: Scans all container configs and removes
   pulse-sensor-proxy bind mount entries

5. **Process Isolation for Uninstaller**: Uses systemd-run to spawn
   isolated transient unit that won't be killed when proxy service stops
   - Unit name: pulse-uninstall-{timestamp}
   - Properties: Type=oneshot, Conflicts=pulse-sensor-proxy.service
   - Runs non-blocking so cleanup service can exit cleanly
   - Falls back to direct call if systemd-run unavailable

6. **Complete Service/Binary Removal**: Calls installer's --uninstall
   - Stops and disables pulse-sensor-proxy.service
   - Removes all systemd units
   - Deletes all binaries from /opt/pulse/sensor-proxy/
   - Removes configuration files
   - Cleans up directories

Changes to cleanup script logic:
- Added LOCKFILE and INSTALLER_PATH configuration
- Acquire flock before processing (prevents concurrent runs)
- Delete request file immediately after reading
- Full localhost cleanup: SSH keys → API tokens → bind mounts → uninstall
- Remote cleanup still SSH-key-only (can't orchestrate uninstall remotely)
- Better error handling with appropriate log levels

Updated cleanup service unit:
- ExecStart now uses ${CLEANUP_SCRIPT_PATH} variable (new /opt location)
- Changed heredoc from 'SERVICE_EOF' to SERVICE_EOF for variable expansion

Addresses all issues documented in CLEANUP_TODO.md:
-  Read-only filesystem (binaries now in /opt, removable)
-  Process isolation (systemd-run transient unit)
-  Cleanup loops (flock + immediate file deletion)
-  API token parsing (JSON first, filtered table fallback)

The UI message is now accurate: "Removing this proxmox ve node also
scrubs the Pulse footprint on the host — the proxy service, SSH key,
API token, and bind mount are all cleaned up automatically."

Part of: CLEANUP_TODO.md Phase 2-4
Supersedes: ed65fda74 (original cleanup attempt with process issues)
Depends on: b192c60e9 (binary relocation to /opt)
2025-11-15 00:03:09 +00:00
rcourtman
d62c0d8f52 Relocate sensor-proxy binaries to /opt for guaranteed cleanup
Moves all sensor-proxy binaries and scripts from /usr/local/bin to
/opt/pulse/sensor-proxy/bin to ensure they can be removed during cleanup
even on systems with read-only /usr (hardened Proxmox setups).

Changes:
- INSTALL_ROOT=/opt/pulse/sensor-proxy (new writable location)
- Binary path: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy
- Wrapper script: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh
- Cleanup script: /opt/pulse/sensor-proxy/bin/pulse-sensor-cleanup.sh
- Selfheal script: /opt/pulse/sensor-proxy/bin/pulse-sensor-proxy-selfheal.sh
- Installer storage: /opt/pulse/sensor-proxy/install-sensor-proxy.sh

Updated:
- Directory creation to include ${INSTALL_ROOT}/bin
- Systemd service ExecStart paths to use ${BINARY_PATH}
- Self-heal service ExecStart to use ${SELFHEAL_SCRIPT}
- Changed heredoc delimiters from 'EOF' to EOF for variable expansion

Rationale:
Proxmox VE can mount /usr as read-only in hardened configurations.
The previous /usr/local/bin location prevented complete uninstallation
on these systems, violating Pulse's correctness principle. The /opt
location is guaranteed writable and appropriate for third-party software.

This is Phase 1 of implementing full cleanup functionality per
CLEANUP_TODO.md. Subsequent commits will add process isolation,
API token deletion, and bind mount removal.

Part of: #CLEANUP_TODO.md Phase 1
Related: ed65fda74 (original cleanup attempt)
2025-11-14 23:59:25 +00:00
rcourtman
c56155ff6d Fix permission denied error when updating allowed_nodes
The update_allowed_nodes function was changing ownership of the temp file
before all writes were complete, causing 'Permission denied' errors when
appending the allowed_nodes section.

Root cause:
- mktemp creates file owned by script runner (root)
- chown changed ownership to pulse-sensor-proxy:pulse-sensor-proxy
- Subsequent append (>>) failed because root can't write to the file

Fix: Defer chown until after all writes complete and file is moved to
final location. Ownership is still correctly set on the final config file.
2025-11-14 23:21:46 +00:00
rcourtman
324bb8fa05 Use unique temp file prefix to avoid permission conflicts
- Change mktemp to use /tmp/pulse-config.XXXXXXXXXX template
- Prevents conflicts with stale temp files from previous runs
- Fixes 'Permission denied' errors when script re-runs
2025-11-14 22:44:33 +00:00
rcourtman
1dd2775f65 Add cleanup trap and better error handling to update_allowed_nodes
- Add trap to remove temp file on function return (success or failure)
- Add error check for mv command with descriptive message
- Ensure config file has proper permissions after update

This prevents orphaned temp files when errors occur and provides
better diagnostics when file operations fail.
2025-11-14 22:13:35 +00:00
rcourtman
8ebff03fa3 Remove invalid 'local' keyword from main script scope
The all_nodes arrays were declared with 'local' keyword outside of
functions, causing bash syntax error:
  'local: can only be used in a function'

Fixed by removing 'local' keyword - arrays in main script scope don't
need it and it's actually invalid syntax.
2025-11-14 22:06:44 +00:00
rcourtman
3e38159882 Fix update_allowed_nodes to remove comment headers
The awk logic was removing allowed_nodes sections but leaving their
comment headers behind. When multiple sections existed, comments would
accumulate.

New approach:
- Buffer all comment lines encountered outside sections
- When a non-comment line is found, flush buffered comments
- When allowed_nodes is found, discard buffered comments (they belonged
  to the section we're removing)
- This cleanly removes section headers like:
  '# Cluster nodes (auto-discovered during installation)'
  '# These nodes are allowed to request...'

Tested with config containing duplicate allowed_nodes sections - now
correctly produces clean output with all duplicates and headers removed.
2025-11-14 22:02:10 +00:00
rcourtman
49cc2927f3 Ensure base config exists before updating allowed_nodes
The installer was only creating base config.yaml in standalone mode,
but update_allowed_nodes() is also called in LXC mode. When the config
didn't exist, update_allowed_nodes() would create an empty file and only
add the allowed_nodes section, missing required fields like
allowed_peer_uids, metrics_address, rate_limit, etc.

This caused the proxy to fail when it tried to parse the incomplete config.

Now creates a proper base config with all required fields if the file
doesn't exist, before any mode-specific configuration is added.
2025-11-14 21:57:16 +00:00
rcourtman
dda29b2e22 Fix duplicate allowed_nodes in sensor proxy installer
The install-sensor-proxy.sh script was blindly appending allowed_nodes
sections to the config file without checking if they already existed.
When the script was re-run or if the initial config already had an
allowed_nodes section, this created duplicate YAML keys that caused
the proxy service to fail with parse errors.

Changes:
- Add update_allowed_nodes() helper function that safely updates the
  allowed_nodes section by removing any existing ones first
- Replace all three cat >> config.yaml heredocs with calls to the
  helper function (cluster nodes, standalone mode, pvecm fallback)
- Uses awk to properly parse and remove multi-line YAML sections

This makes the installer idempotent and prevents config corruption on
re-runs.

Fixes issue where proxy service crashed with:
  'mapping key "allowed_nodes" already defined at line X'
2025-11-14 21:43:31 +00:00
rcourtman
105215b5c3 Improve sensor proxy socket verification 2025-11-14 20:14:25 +00:00
rcourtman
36429f6540 Related to #701: improve sensor proxy installer caching 2025-11-14 00:51:54 +00:00
rcourtman
59c630633f Add localhost to allowed_source_subnets for self-monitoring
The HTTP mode installer now includes 127.0.0.1/32 in allowed_source_subnets
to permit self-monitoring queries from localhost. This fixes 403 Forbidden
errors when nodes query their own sensor-proxy instance.

Related to HTTP mode implementation for external PVE hosts.
2025-11-13 19:34:11 +00:00
rcourtman
aa357e5013 Fix HTTP mode for pulse-sensor-proxy and improve installer safety
## HTTP Server Fixes
- Add source IP middleware to enforce allowed_source_subnets
- Fix missing source subnet validation for external HTTP requests
- HTTP health endpoint now respects subnet restrictions

## Installer Improvements
- Auto-configure allowed_source_subnets with Pulse server IP
- Add cluster node hostnames to allowed_nodes (not just IPs)
- Fix node validation to accept both hostnames and IPs
- Add Pulse server reachability check before installation
- Add port availability check for HTTP mode
- Add automatic rollback on service startup failure
- Add HTTP endpoint health check after installation
- Fix config backup and deduplication (prevent duplicate keys)
- Fix IPv4 validation with loopback rejection
- Improve registration retry logic with detailed errors
- Add automatic LXC bind mount cleanup on uninstall

## Temperature Collection Fixes
- Add local temperature collection for self-monitoring nodes
- Fix node identifier matching (use hostname not SSH host)
- Fix JSON double-encoding in HTTP client response

Related to #XXX (temperature monitoring fixes)
2025-11-13 18:22:36 +00:00
rcourtman
559dfd0235 Add HTTP mode support to sensor-proxy installer
Implements complete HTTP mode installation workflow for external PVE hosts.

New installer features:
- `--http-mode` flag: Enable HTTP server mode for remote temperature monitoring
- `--http-addr <addr>` flag: Configure listen address (default :8443)
- Auto-generates self-signed TLS certificates (4096-bit RSA, 10-year validity)
- Registers with Pulse API and receives authentication token
- Configures systemd service with proper security hardening

Installation workflow (HTTP mode):
1. Validate --pulse-server parameter is provided
2. Generate TLS certificate with SAN (hostname + IPs)
3. Call Pulse API POST /api/temperature-proxy/register
4. Receive and store auth token securely (mode 600)
5. Append HTTP config to config.yaml
6. Update systemd service with TLS paths
7. Start service

TLS certificate generation:
- Uses openssl req with RSA 4096-bit keys
- 10-year validity period
- SubjectAltName includes hostname + all IPs
- Files stored in /etc/pulse-sensor-proxy/tls/
- Permissions: 640 root:pulse-sensor-proxy
- Logs SHA256 fingerprint for audit

API registration:
- Calls POST /api/temperature-proxy/register
- Payload: {"hostname": "...", "proxy_url": "https://..."}
- Response: {"token": "...", "pve_instance": "..."}
- Aborts installation on registration failure (fail-fast)
- Token stored in config.yaml

Systemd service updates:
- Adds ReadOnlyPaths=/etc/pulse-sensor-proxy/tls for HTTP mode
- RestrictAddressFamilies already includes AF_INET/AF_INET6
- Maintains all existing security hardening

Error handling:
- Validates required parameters before starting
- Aborts on TLS generation failure
- Aborts on API registration failure
- Provides actionable troubleshooting guidance
- Logs clear error messages

Security:
- Tokens stored with mode 600, owned by service user
- TLS keys protected with mode 640
- Service runs as unprivileged pulse-sensor-proxy user
- Full systemd hardening maintained

Usage example:
  curl -fsSL https://pulse-server/download/install-sensor-proxy.sh | \
    bash -s -- --http-mode --pulse-server https://pulse.example.com:7655

Related to #571
2025-11-13 16:33:12 +00:00
rcourtman
aba0eabd21 Fix #571: Installer now auto-configures allowed_nodes for temperature monitoring
When cluster IPC validation fails (due to systemd hardening), the proxy
falls back to allowlist-based validation. The installer now automatically
populates allowed_nodes with:

- Cluster mode: All discovered cluster member IPs
- Standalone mode: localhost IP addresses (including 127.0.0.1/localhost)
- Fallback mode: localhost IPs when pvecm unavailable

This ensures out-of-the-box temperature monitoring works on fresh installs
without manual configuration.
2025-11-13 15:37:30 +00:00
rcourtman
4bb8ab15a7 Fix temperature monitoring for clustered and LXC Proxmox environments (addresses #571)
Root cause: pulse-sensor-proxy runs with strict systemd hardening that prevents
access to Proxmox corosync IPC (abstract UNIX sockets). When pvecm fails with
IPC errors, the code incorrectly treated it as "standalone mode" and only
discovered localhost addresses, rejecting legitimate cluster members and external
nodes.

Changes:

1. **Distinguish IPC failures from true standalone mode**
   - Detect ipcc_send_rec and access control list errors specifically
   - These indicate a cluster exists but isn't accessible (LXC, systemd restrictions)
   - Return error to disable cluster validation instead of misusing standalone logic

2. **Graceful degradation when cluster validation fails**
   - When cluster IPC is unavailable, fall through to permissive mode
   - Log debug message suggesting allowed_nodes configuration
   - Allows requests to proceed rather than blocking all temperature monitoring

3. **Improve local address discovery for true standalone nodes**
   - Use Go's native net.Interfaces() instead of shelling out to 'ip addr'
   - More reliable and works with AF_NETLINK restrictions
   - Add helpful logging when only hostnames are discovered

4. **Systemd hardening adjustments**
   - Add AF_NETLINK to RestrictAddressFamilies (for net.Interfaces())
   - Remove RemoveIPC=true (attempted fix for corosync, insufficient)
   - Add ReadWritePaths=-/run/corosync (optional path, corosync uses abstract sockets anyway)

Result: Temperature monitoring now works in:
- Clustered Proxmox hosts (falls back to permissive when IPC blocked)
- LXC containers (correctly detects IPC failure, allows requests)
- Standalone nodes (proper local address discovery with IPs)

Workaround for maximum security: Configure allowed_nodes in /etc/pulse-sensor-proxy/config.yaml
when cluster validation cannot be used.
2025-11-13 13:25:27 +00:00
rcourtman
573851a388 Fix temperature monitoring on standalone Proxmox nodes (addresses #571)
Root cause: The systemd service hardening blocked AF_NETLINK sockets,
preventing IP address discovery on standalone nodes. The proxy could
only discover hostnames, causing node_not_cluster_member rejections
when users configured Pulse with IP addresses.

Changes:
1. Add AF_NETLINK to RestrictAddressFamilies in all systemd services
   - pulse-sensor-proxy.service
   - install-sensor-proxy.sh (both modes)
   - pulse-sensor-cleanup.service

2. Replace shell-based 'ip addr' with Go native net.Interfaces() API
   - More reliable and doesn't require external commands
   - Works even with strict systemd restrictions
   - Properly filters loopback, link-local, and down interfaces

3. Improve error logging and user guidance
   - Warn when no IP addresses can be discovered
   - Provide clear instructions about allowed_nodes workaround
   - Include address counts in logs for debugging

This fix ensures standalone Proxmox nodes can properly validate
temperature requests by IP address without requiring manual
allowed_nodes configuration.
2025-11-13 13:02:15 +00:00
rcourtman
d3875eaae5 Dramatically improve temperature proxy installation robustness
Users were abandoning Pulse due to catastrophic temperature monitoring setup failures. This commit addresses the root causes:

**Problem 1: Silent Failures**
- Installations reported "SUCCESS" even when proxy never started
- UI showed green checkmarks with no temperature data
- Zero feedback when things went wrong

**Problem 2: Missing Diagnostics**
- Service failures logged only in journald
- Users saw "Something going on with the proxy" with no actionable guidance
- No way to troubleshoot from error messages

**Problem 3: Standalone Node Issues**
- Proxy daemon logged continuous pvecm errors as warnings
- "ipcc_send_rec" and "Unknown error -1" messages confused users
- These are expected for non-clustered/LXC setups

**Solutions Implemented:**

1. **Health Gate in install.sh (lines 1588-1629)**
   - Verify service is running after installation
   - Check socket exists on host
   - Confirm socket visible inside container via bind mount
   - Fail loudly with specific diagnostics if any check fails

2. **Actionable Error Messages in install-sensor-proxy.sh (lines 822-877)**
   - When service fails to start: dump full systemctl status + 40 lines of logs
   - When socket missing: show permissions, service status, and remediation command
   - Include common issues checklist (missing user, permission errors, lm-sensors, etc.)
   - Direct link to troubleshooting docs

3. **Better Standalone Node Detection in ssh.go (lines 585-595)**
   - Recognize "Unknown error -1" and "Unable to load access control list" as LXC indicators
   - Log at INFO level (not WARN) since this is expected behavior
   - Clarify message: "using localhost for temperature collection"

**Impact:**
- Eliminates "green checkmark but no temps" scenario
- Users get immediate actionable feedback on failures
- Standalone/LXC installations work silently without error spam
- Reduces support burden from #571 (15+ comments of user frustration)

Related to #571
2025-11-13 10:14:19 +00:00
rcourtman
798f91792b Improve sensor-proxy release detection (related to #701) 2025-11-12 17:49:20 +00:00
rcourtman
5bac91a664 Fix pulse-sensor-proxy configuration not applied in LXC containers (related to #600)
This fixes two bugs that prevented temperature monitoring from working
after running install-sensor-proxy.sh on LXC deployments:

1. CRITICAL: Pulse service not restarted after systemd override
   - The installer wrote PULSE_SENSOR_PROXY_SOCKET env var to systemd
     drop-in and ran daemon-reload, but never restarted Pulse service
   - Running Pulse instances continued using old environment variables
   - Temperatures wouldn't work until manual Pulse restart
   - Now: Automatically restart Pulse if running after writing override

2. Added guard to check if Pulse service exists before configuring
   - Installer would write systemd override even if Pulse not installed
   - Left orphaned drop-in files that confused users
   - Now: Check if pulse.service exists, warn and skip if not found

3. MINOR: Fix inconsistent Docker mount instructions
   - docker-compose.yml showed :ro (read-only) mount
   - Installer output showed :rw (read-write) mount
   - Changed installer to match compose file (:ro is correct and secure)

Impact: Users in #600 reported "socketFound=false" even after running
installer successfully. This was because Pulse never picked up the new
socket path without a restart.
2025-11-09 16:44:08 +00:00
rcourtman
4834dea05b Add support for linux-386 and linux-armv6 architectures (related to #674)
Adds build support for 32-bit x86 (i386/i686) and ARMv6 (older Raspberry Pi models) architectures across all agents and install scripts.

Changes:
- Add linux-386 and linux-armv6 to build-release.sh builds array
- Update Dockerfile to build docker-agent, host-agent, and sensor-proxy for new architectures
- Update all install scripts to detect and handle i386/i686 and armv6l architectures
- Add architecture normalization in router download endpoints
- Update update manager architecture mapping
- Update validate-release.sh to expect 24 binaries (was 18)

This enables Pulse agents to run on older/legacy hardware including 32-bit x86 systems and Raspberry Pi Zero/Zero W devices.
2025-11-09 08:35:24 +00:00
rcourtman
334b8c727f Fix SMART temperature collection on smartctl 7.4+ (related to #672)
Fixes two critical bugs in refresh_smart_cache() that prevented SMART
temperature collection from working:

1. Invalid smartctl parameter: Changed -n standby,after to -n standby
   The 'after' parameter is not valid in smartctl 7.4 and causes:
   "INVALID ARGUMENT TO -n: standby,after"
   Valid syntax is standby[,STATUS[,STATUS2]] where STATUS must be numeric.

2. Broken process detection: Replaced exec -a with lock file approach
   The original exec -a pulse-sensor-wrapper-refresh bash line replaced
   the subshell with a new bash process that had no script to run, causing
   the function to exit immediately without collecting any SMART data.

   New approach uses a lock file ($CACHE_DIR/smart-refresh.lock) with
   trap-based cleanup to prevent concurrent refresh operations.

Credits to @ZaDarkSide for identifying these issues in PR #672.
2025-11-08 23:40:43 +00:00
rcourtman
679225510e Silence broken pipe error in sensor proxy self-heal script (related to #628)
The self-heal timer runs 'systemctl list-unit-files | grep -q' every hour.
When grep matches and exits early, systemctl logs "Failed to print table:
Broken pipe" to syslog. This is cosmetic but floods Proxmox logs and
confuses operators.

Changes:
- Redirect stderr from systemctl to /dev/null
- Prevents the broken pipe message from reaching syslog
- Self-heal functionality unchanged

This addresses the concern raised in discussion #628.
2025-11-07 17:46:23 +00:00
rcourtman
2a79d57f73 Add SMART temperature collection for physical disks (related to #652)
Extends temperature monitoring to collect SMART temps for SATA/SAS disks,
addressing issue #652 where physical disk temperatures showed as empty.

Architecture:
- Deploys pulse-sensor-wrapper.sh as SSH forced command on Proxmox nodes
- Wrapper collects both CPU/GPU temps (sensors -j) and disk temps (smartctl)
- Implements 30-min cache with background refresh to avoid performance impact
- Uses smartctl -n standby,after to skip sleeping drives without waking them
- Returns unified JSON: {sensors: {...}, smart: [...]}

Backend changes:
- Add DiskTemp model with device, serial, WWN, temperature, lastUpdated
- Extend Temperature model with SMART []DiskTemp field and HasSMART flag
- Add WWN field to PhysicalDisk for reliable disk matching
- Update parseSensorsJSON to handle both legacy and new wrapper formats
- Rewrite mergeNVMeTempsIntoDisks to match SMART temps by WWN → serial → devpath
- Preserve legacy NVMe temperature support for backward compatibility

Performance considerations:
- SMART data cached for 30 minutes per node to avoid excessive smartctl calls
- Background refresh prevents blocking temperature requests
- Respects drive standby state to avoid spinning up idle arrays
- Staggered disk scanning with 0.1s delay to avoid saturating SATA controllers

Install script:
- Deploys wrapper to /usr/local/bin/pulse-sensor-wrapper.sh
- Updates SSH forced command from "sensors -j" to wrapper script
- Backward compatible - falls back to direct sensors output if wrapper missing

Testing note:
- Requires real hardware with smartmontools installed for full functionality
- Empty smart array returned gracefully when smartctl unavailable
- Legacy sensor-only nodes continue working without changes
2025-11-07 11:46:57 +00:00
rcourtman
8a052baa2a Fix temperature monitoring for standalone Proxmox nodes and add multi-arch sensor proxy builds
Related to #571

This addresses multiple temperature monitoring issues:

1. Fix single-node Proxmox installation failure: Add '|| true' to pvecm status
   calls to prevent script exit on standalone (non-clustered) nodes with
   'set -euo pipefail'. The script now properly falls through to standalone
   node configuration when cluster detection fails.

2. Build pulse-sensor-proxy for all Linux architectures (amd64, arm64, armv7)
   in Dockerfile to ensure binaries are available for download on all supported
   platforms. This resolves the missing binary issue from v4.23.0.

Note: AMD Tctl sensor support was already implemented in a previous commit.
2025-11-05 19:41:09 +00:00
rcourtman
26144ae558 Improve temperature proxy setup guidance for Docker deployments
This addresses GitHub Discussion #605 where users were unclear about
configuring the temperature proxy when running Pulse in Docker.

Changes:

**install-sensor-proxy.sh:**
- Add Docker-specific post-install instructions when --standalone flag is used
- Show required docker-compose.yml bind mount configuration
- Provide verification commands for Docker deployments
- Link to full documentation for troubleshooting

**TEMPERATURE_MONITORING.md:**
- Add prominent "Quick Start for Docker Deployments" section at the top
- Move Docker instructions earlier in the document for better visibility
- Provide complete 4-step setup process with verification commands

These changes ensure Docker users immediately see:
1. How to install the proxy on the Proxmox host
2. What bind mount to add to docker-compose.yml
3. How to restart and verify the setup
4. Where to find detailed troubleshooting

The installer now provides actionable next steps instead of just
confirming installation, reducing confusion for containerized deployments.
2025-11-05 18:18:04 +00:00
rcourtman
5c4be1921c chore: snapshot current changes 2025-11-02 22:47:55 +00:00
rcourtman
ff4dc49ae4 Update Pulse install flow and related components 2025-10-21 19:58:53 +00:00
rcourtman
f9cb96ceb8 feat: add --uninstall support to Docker agent and sensor proxy scripts
Users can now cleanly uninstall components with optional data removal.

Docker Agent (install-docker-agent.sh):
- --uninstall: Remove service, binary, systemd unit, Unraid startup hook
- --purge: Also remove log files (optional, must be used with --uninstall)
- Stops/disables service even if unit file is missing (resilient cleanup)
- Validates --purge requires --uninstall

Sensor Proxy (install-sensor-proxy.sh):
- --uninstall: Remove service, binary, cleanup scripts, socket directory
- Calls existing cleanup helper to remove SSH keys from cluster nodes
- Manual fallback if cleanup helper is missing
- --purge: Also remove state/logs and service account
- Validates --purge requires --uninstall

Usage:
  # Uninstall Docker agent (keep logs)
  curl ... | bash -s -- --uninstall

  # Uninstall Docker agent (remove everything)
  curl ... | bash -s -- --uninstall --purge

  # Uninstall sensor proxy (keep state/logs)
  curl ... | bash -s -- --uninstall

  # Uninstall sensor proxy (remove everything)
  curl ... | bash -s -- --uninstall --purge

Changes:
- scripts/install-docker-agent.sh: Add --purge flag, improve uninstall flow
- scripts/install-sensor-proxy.sh: Add perform_uninstall() function
- Both: Non-interactive, idempotent, resilient cleanup

Next: Update UI to show uninstall commands when removing hosts/nodes

Co-authored-by: Codex AI
2025-10-21 10:21:48 +00:00
rcourtman
7e871780f6 feat: improve LXC installer robustness and temperature monitoring UX
Major improvements to the install script based on comprehensive review:

## 1. Temperature Monitoring - No Restart Required 
- Ask about temperature monitoring BEFORE container creation (not after)
- Add bind mount during `pct create` instead of requiring restart later
- Quick mode defaults to "yes", Advanced mode asks user
- Host path: /run/pulse-sensor-proxy → /mnt/pulse-proxy in container
- Support --skip-restart flag in install-sensor-proxy.sh
- Eliminates disruptive container restart on fresh installs

## 2. Shell Injection Prevention 🔒
- Replace `eval pct create` with array-based command building
- Prevents quoting bugs with special characters in hostnames/nameservers
- Safer handling of user input in container creation

## 3. Non-Interactive Install Support 🤖
- Replace bare `read` with `safe_read_with_default` in prompts
- Prevents hangs when running `curl | bash` non-interactively
- Proper fallback to sensible defaults

## 4. Cleanup on Interrupt 🧹
- Track container ID globally during creation
- Properly cleanup orphaned containers on Ctrl+C/SIGTERM
- New handle_install_interrupt() function
- Prevents leftover containers after cancelled installs

## 5. Air-Gapped Network Support 🌐
- Replace 8.8.8.8 ping check with `hostname -I` IP detection
- Supports restricted/firewalled networks where external ping fails
- More reliable for DHCP-only environments

Changes:
- install.sh: Refactor temperature prompt timing and mount setup
- install.sh: Convert pct create to array-based args (lines 1018-1055)
- install.sh: Add handle_install_interrupt trap (lines 38-48)
- install.sh: Replace ping check with IP detection (line 1082)
- scripts/install-sensor-proxy.sh: Add --skip-restart flag support
- scripts/install-sensor-proxy.sh: Improve mount detection and updates

Impact:
- Fresh installs now complete without any container restarts
- Temperature monitoring works immediately after first boot
- Safer and more robust for automation/CI scenarios
- Better experience on restricted networks

Co-authored-by: Codex AI
2025-10-21 09:22:43 +00:00
rcourtman
95c85f6e01 fix: use correct service name (pulse.service) for proxy environment override
The installer was configuring pulse-backend.service.d but the actual
service is pulse.service, so the PULSE_SENSOR_PROXY_SOCKET environment
variable wasn't being set.

Changed: pulse-backend.service → pulse.service

This ensures Pulse actually uses the proxy socket for temperature
monitoring instead of attempting SSH connections.
2025-10-20 22:28:33 +00:00
rcourtman
8faa9040fb fix: show curl errors in installer download failures
Changed curl flags from -fsSL to -fSL to enable error output.
The -s flag was silencing all curl errors including SSL/TLS issues,
making it impossible to diagnose download failures.

With -S (show errors), stderr now captures meaningful error messages
like certificate problems, connection failures, etc.
2025-10-20 21:31:54 +00:00
rcourtman
90d51a2b1b feat: add rollback mechanism for container config changes
- Back up container config before making mount modifications
- Restore original config if socket verification fails
- Clean up backup file on success or when verification is skipped
- Leave host-level resources (user, binary, service) in place for idempotency

This ensures failed installations don't leave containers in an
inconsistent state while keeping successfully installed host services
for faster re-runs.
2025-10-20 21:16:06 +00:00
rcourtman
d421f101ba feat: harden temperature proxy installation with better validation and error handling
Setup script improvements (config_handlers.go):
- Remove redundant mount configuration and container restart logic
- Let installer handle all mount/restart operations (single source of truth)
- Eliminate hard-coded mp0 assumption

Installer improvements (install-sensor-proxy.sh):
- Add mount configuration persistence validation via pct config check
- Surface pct set errors instead of silencing with 2>/dev/null
- Capture and display curl download errors with temp files
- Check systemd daemon-reload/enable/restart exit codes
- Show journalctl output when service fails to start
- Make socket verification fatal (was warning)
- Provide clear manual steps when hot-plug fails on running container

This makes the installation fail fast with actionable error messages
instead of silently proceeding with broken configuration.
2025-10-20 21:14:00 +00:00
rcourtman
001d7f5f1c fix: comprehensive temperature proxy setup improvements
Addresses multiple issues that prevented successful temperature monitoring setup:

1. **Missing log directory (install-sensor-proxy.sh)**
   - Added LogsDirectory=pulse/sensor-proxy to both systemd service templates
   - Fixes crash: "open /var/log/pulse/sensor-proxy/audit.log: read-only file system"
   - Uses systemd's LogsDirectory directive for proper permissions

2. **Invalid pct restart command (install-sensor-proxy.sh:822)**
   - Changed from `pct restart` (doesn't exist) to `pct stop && sleep 2 && pct start`
   - Fixes container restart failures during proxy setup

3. **Version compatibility check (config_handlers.go)**
   - Added const minProxyReadyVersion = "4.24.0"
   - Setup script now queries /api/version endpoint
   - Blocks proxy setup on Pulse < v4.24.0 with clear upgrade message
   - Prevents users from attempting proxy setup on incompatible versions

4. **Proxy service health validation (config_handlers.go)**
   - Verifies pulse-sensor-proxy service is actually running
   - Checks socket exists at /run/pulse-sensor-proxy/pulse-sensor-proxy.sock
   - Shows journalctl command for troubleshooting on failure
   - Sets TEMP_MONITORING_AVAILABLE=false to skip remaining steps

5. **Interactive LXC restart prompt (config_handlers.go)**
   - Replaced passive "please restart" message with interactive prompt
   - Default action is "yes" for easy acceptance
   - Actually executes pct stop/start on confirmation
   - Handles non-interactive environments gracefully

6. **Post-restart socket verification (config_handlers.go)**
   - Validates socket is accessible inside container after restart
   - Provides clear error if mount didn't work
   - Prevents claiming success when setup is incomplete

All changes tested with fresh LXC installation. Temperature monitoring now
works end-to-end with proper error handling and user guidance.

Fixes temperature proxy setup flow for v4.24.0+
2025-10-20 18:00:21 +00:00
rcourtman
c91b7874ac docs: comprehensive v4.24.0 documentation audit and updates
Complete documentation overhaul for Pulse v4.24.0 release covering all new
features and operational procedures.

Documentation Updates (19 files):

P0 Release-Critical:
- Operations: Rewrote ADAPTIVE_POLLING_ROLLOUT.md as GA operations runbook
- Operations: Updated ADAPTIVE_POLLING_MANAGEMENT_ENDPOINTS.md with DEFERRED status
- Operations: Enhanced audit-log-rotation.md with scheduler health checks
- Security: Updated proxy hardening docs with rate limit defaults
- Docker: Added runtime logging and rollback procedures

P1 Deployment & Integration:
- KUBERNETES.md: Runtime logging config, adaptive polling, post-upgrade verification
- PORT_CONFIGURATION.md: Service naming, change tracking via update history
- REVERSE_PROXY.md: Rate limit headers, error pass-through, v4.24.0 verification
- PROXY_AUTH.md, OIDC.md, WEBHOOKS.md: Runtime logging integration
- TROUBLESHOOTING.md, VM_DISK_MONITORING.md, zfs-monitoring.md: Updated workflows

Features Documented:
- X-RateLimit-* headers for all API responses
- Updates rollback workflow (UI & CLI)
- Scheduler health API with rich metadata
- Runtime logging configuration (no restart required)
- Adaptive polling (GA, enabled by default)
- Enhanced audit logging
- Circuit breakers and dead-letter queue

Supporting Changes:
- Discovery service enhancements
- Config handlers updates
- Sensor proxy installer improvements

Total Changes: 1,626 insertions(+), 622 deletions(-)
Files Modified: 24 (19 docs, 5 code)

All documentation is production-ready for v4.24.0 release.
2025-10-20 17:20:13 +00:00