Commit graph

123 commits

Author SHA1 Message Date
rcourtman
4f824ab148 style: Apply gofmt to 37 files
Standardize code formatting across test files and monitor.go.
No functional changes.
2025-12-02 17:21:48 +00:00
rcourtman
0bc58f678e perf: Cache err.Error() in storage timeout error handling
Cache err.Error() result in two locations:
- monitor.go: storage query retry logic (2x calls to 1)
- monitor_polling.go: storage timeout handling (2x calls to 1)
2025-12-02 15:39:37 +00:00
rcourtman
dc707b2225 perf: Cache err.Error() in disk monitoring error handling
Compute err.Error() once and reuse errStr instead of calling Error()
four times when checking disk monitoring error types.
2025-12-02 15:37:48 +00:00
rcourtman
c81bbba8a3 perf: Use strconv.Itoa instead of fmt.Sprintf for int conversion
strconv.Itoa is faster than fmt.Sprintf("%d", ...) because it doesn't
need to parse a format string. Changed 4 occurrences in monitoring
package where integers are converted to strings.
2025-12-02 15:21:41 +00:00
rcourtman
158669296e refactor: Remove unreachable dead code branches
- firstForwardedValue: strings.Split always returns at least one element
- shouldRunBackupPoll: remaining is always >= 1 by math
- convertContainerDiskInfo: lowerLabel is never empty for non-rootfs

All three functions now at 100% coverage.
2025-12-02 14:41:53 +00:00
rcourtman
139deb73aa Filter virtual/system filesystems from host disk display
Host disk bars were showing virtual filesystems like tmpfs, /dev, /run,
/sys, and Docker overlay mounts. These clutter the UI and don't represent
meaningful disk usage.

Changed from `shouldIgnoreReadOnlyFilesystem` (read-only only) to the
full `fsfilters.ShouldSkipFilesystem` which also excludes:
- Virtual FS types: tmpfs, devtmpfs, sysfs, proc, cgroup, etc.
- Special mountpoints: /dev, /proc, /sys, /run, /var/lib/docker, /snap
- Network filesystems: fuse, nfs, cifs, etc.

Related to #790
2025-12-02 00:16:39 +00:00
rcourtman
dfc0059bd9 test: Add tests for convertDockerSwarmInfo, namespacePathsForDatastore, preserveFailedStorageBackups
- convertDockerSwarmInfo: 66.7%→100% (4 cases for nil, empty, populated structs)
- namespacePathsForDatastore: 92.3%→100% (removed unreachable dead code)
- preserveFailedStorageBackups: 91.3%→100% (6 cases for filtering, deduplication)
2025-12-01 19:04:23 +00:00
rcourtman
7129582639 Fix outdated error message path for removed Docker hosts
The error message referenced "Settings -> Docker -> Removed hosts" but
that UI path no longer exists. The correct path is now
"Settings -> Agents -> Removed Docker Hosts".

Related to #778
2025-12-01 12:03:05 +00:00
rcourtman
4122fc6c9b refactor: Extract guest metadata functions to separate file
Move 17 guest-metadata-related functions from monitor.go to new
guest_metadata.go file. This includes:
- Guest agent API call retry logic
- Guest metadata caching (fetch, reserve, schedule, clear)
- Network interface processing and cloning
- OS info extraction
- Type conversion utilities (stringValue, anyToInt64)

Reduces monitor.go by 541 lines (8932 → 8391).
2025-12-01 10:53:41 +00:00
rcourtman
62407f0a00 refactor: Extract container parsing functions to separate file
Move 13 related container parsing functions from monitor.go (9359 lines)
to container_parsing.go (451 lines). Reduces monitor.go by 427 lines.

Functions extracted:
- ensureContainerRootDiskEntry
- convertContainerDiskInfo
- sanitizeRootFSDevice
- parseContainerRawIPs
- collectIPsFromInterface
- sanitizeGuestAddressStrings
- dedupeStringsPreserveOrder
- containerNetworkDetails struct
- containerMountMetadata struct
- parseContainerConfigNetworks
- parseContainerMountMetadata
- mergeContainerNetworkInterface
- extractContainerRootDeviceFromConfig
2025-12-01 10:40:48 +00:00
rcourtman
21e3c74d9d refactor: Extract Docker host identifier functions to separate file
Move 9 related pure functions from monitor.go (9616 lines) to
docker_host_identity.go (281 lines). Reduces monitor.go by 257 lines.

Functions extracted:
- tokenHintFromRecord
- resolveDockerHostIdentifier
- findMatchingDockerHost
- dockerHostIDExists
- generateDockerHostIdentifier
- dockerHostSuffixCandidates
- sanitizeDockerHostSuffix
- fallbackDockerHostID
- uniqueNonEmptyStrings
2025-12-01 10:03:47 +00:00
rcourtman
8361042ada Fix backup status indicator not showing for guests
The backup status indicator feature was incomplete - it added the UI
component but never populated VM/Container LastBackup from actual
backup data. Now SyncGuestBackupTimes() is called after storage
backups and PBS backups are polled, matching each guest's VMID to
its most recent backup timestamp.

Fixes #786
2025-11-30 22:13:46 +00:00
rcourtman
2eea0335a2 Extract filesystem filtering logic into pkg/fsfilters
Move the inline filesystem skip logic from pollVMsAndContainersEfficient
into a reusable ShouldSkipFilesystem function. This consolidates filtering
for virtual filesystems (tmpfs, cgroup, etc.), network mounts (nfs, cifs,
fuse), and special mountpoints (/dev, /proc, /snap, etc.) into one tested
location.

Reduces cyclomatic complexity of pollVMsAndContainersEfficient and adds
28 test cases covering virtual fs types, network mounts, special mounts,
Windows paths, and edge cases.
2025-11-29 16:38:08 +00:00
rcourtman
dbf32baaa0 Rebuild agent token bindings on API token config reload
When api_tokens.json is modified on disk, the ConfigWatcher reloads
the tokens into memory. However, the Monitor's dockerTokenBindings and
hostTokenBindings maps were not synchronized with the new token set,
causing orphaned bindings when agents reconnect after reinstall.

Add SetAPITokenReloadCallback to ConfigWatcher that triggers Monitor's
new RebuildTokenBindings method after token reload. This method
reconstructs the binding maps from current Docker host and host agent
state, keeping only bindings for tokens that still exist in config.

Related to #773
2025-11-29 14:09:30 +00:00
rcourtman
2ff0a0988f refactor: remove unnecessary type conversions
Remove redundant type conversions identified by unconvert linter:
- Remove int() conversions for already-int VMID fields
- Remove int64() conversions for already-int64 arithmetic results
- Remove uint64() conversions for already-uint64 Disk/MaxDisk fields
- Remove int() on syscall.Stdin (already int constant)
2025-11-27 10:33:35 +00:00
rcourtman
8152197207 fix: mark unused parameters to satisfy unparam linter
Mark intentionally unused parameters with underscore to:
- Silence unparam warnings for legitimate unused parameters
- Keep function signatures intact for API compatibility
- Remove unused req from serveChecksum helper
2025-11-27 10:12:48 +00:00
rcourtman
8f6a481cd2 refactor: use builtin max() and fix unused parameter
- Replace custom maxInt64 helper with Go 1.21+ builtin max()
- Mark unused cfg parameter in newAdaptiveIntervalSelector
- Remove test for deleted helper function
2025-11-27 10:08:37 +00:00
rcourtman
6ff345fb6b chore: fix staticcheck SA warnings
- Fix SA4006 unused value issues in ssh.go, validation.go, generator.go
- Replace deprecated ioutil with io/os in config.go
- Replace deprecated tar.TypeRegA with tar.TypeReg
- Remove deprecated rand.Seed calls (auto-seeded in Go 1.20+)
- Fix always-true nil check in main.go
- Fix impossible nil comparison in tempproxy/client.go
- Add nil check for config in monitor.New()
2025-11-27 09:16:53 +00:00
rcourtman
ea335546fc feat: improve legacy agent detection and migration UX
Add seamless migration path from legacy agents to unified agent:

- Add AgentType field to report payloads (unified vs legacy detection)
- Update server to detect legacy agents by type instead of version
- Add UI banner showing upgrade command when legacy agents are detected
- Add deprecation notice to install-host-agent.ps1
- Create install-docker-agent.sh stub that redirects to unified installer

Legacy agents (pulse-host-agent, pulse-docker-agent) now show a "Legacy"
badge in the UI with a one-click copy command to upgrade to the unified
agent.
2025-11-25 23:26:22 +00:00
courtmanr@gmail.com
1716774e71 feat: adaptive node table layout, guest row fixes, and legacy agent detection
- Implemented adaptive layout for NodeSummaryTable with responsive columns and sticky name column.
- Fixed GuestRow background display issues.
- Added IsLegacy field to Host and DockerHost models to flag legacy agents (version < 1.0.0).
- Updated monitor to populate IsLegacy based on agent version.
2025-11-25 17:19:36 +00:00
courtmanr@gmail.com
584ad94ee5 Refactor: Parallelize PVE node polling 2025-11-25 08:38:03 +00:00
courtmanr@gmail.com
d6addee8b4 Fix: Correct context cancellation in loop (Fixes #727)
- Replaced defer in loop with explicit cancellation to avoid resource leak
- Properly tagged issue #727
2025-11-23 22:28:28 +00:00
courtmanr@gmail.com
78308cbc10 Fix: Prevent single node auth failure from disabling global SSH temperature collection
- Removed global legacySSHDisabled flag that was triggered by any single node auth failure
- Changed disableLegacySSHOnAuthFailure to only log warnings
- Fixed potential context leak in monitor.go
- Updated tests to reflect removal of global disable logic
2025-11-23 22:24:15 +00:00
courtmanr@gmail.com
5f3fd17025 Fix temperature collection regression for cluster nodes. Related to #727 2025-11-23 12:13:57 +00:00
courtmanr@gmail.com
83e07969f0 fix: ensure proxmox nodes are displayed even if cluster endpoints are missing
Fixes #727. Previously, if temperature monitoring was enabled and a node wasn't found in ClusterEndpoints, the entire node processing was skipped. This change ensures we only skip temperature collection.
2025-11-22 23:31:30 +00:00
rcourtman
a0c27e84ae Persist PVE fallback host after portless retry 2025-11-22 17:06:15 +00:00
rcourtman
9fc019b90c Handle PVE portless fallback when default port fails 2025-11-22 17:01:16 +00:00
rcourtman
255357d2fe Add recovery notifications and grouping controls 2025-11-21 22:07:00 +00:00
rcourtman
61a7c2829c Honor configured PVE polling interval in scheduler 2025-11-20 22:00:56 +00:00
rcourtman
bd0c47ed1b Improve token collision handling and installer subnet support 2025-11-20 09:45:36 +00:00
rcourtman
ce0fb90182 feat: avoid redundant PBS snapshot polling (Related to #717) 2025-11-18 23:10:43 +00:00
rcourtman
15d32adb10 feat: surface LXC mountpoints in UI (related to #715) 2025-11-18 22:57:20 +00:00
rcourtman
51b368ddc1 feat: make PVE polling interval configurable (related to #467) 2025-11-18 21:30:04 +00:00
rcourtman
b474a77b65 Add direct node fallback for storage polling 2025-11-18 19:58:38 +00:00
rcourtman
a03f8115b6 Improve installer temperature proxy and backup polling 2025-11-18 18:42:33 +00:00
rcourtman
a9e751d165 Allow PBS backup poll to finish after poller returns 2025-11-18 16:55:46 +00:00
rcourtman
1abff55feb Improve temperature proxy detection 2025-11-18 14:25:09 +00:00
rcourtman
23d194128d Skip inactive storages during content scans 2025-11-18 09:46:48 +00:00
rcourtman
13daa61d1d Harden turnkey install and proxy auto-registration 2025-11-18 00:24:50 +00:00
rcourtman
f9341ae1fc Improve temperature proxy workflow 2025-11-17 14:25:46 +00:00
rcourtman
47d5c14aef Improve temperature proxy control-plane flow 2025-11-15 21:49:51 +00:00
rcourtman
b1aad303b7 Fix incorrect temperature data during cluster initialization
During cluster startup, nodes were temporarily using the primary cluster
endpoint for temperature collection before cluster metadata validation
completed. This caused all nodes to show the same (incorrect) temperature
values for ~4 minutes until validation finished and per-node endpoints
were established.

Example: minipc would show delly's temperature (90°C) instead of its own
(50°C) from startup until cluster validation completed.

Root cause:
- Temperature collection started immediately at startup
- Cluster endpoint validation happened asynchronously
- Code fell back to primary endpoint when ClusterEndpoints was empty
- All nodes used same endpoint, got same temperature data

Fix: Skip temperature collection for cluster nodes until:
1. ClusterEndpoints array is populated (validation complete)
2. Node's specific endpoint is found in the cluster metadata

This ensures correct temperature data from the very first collection,
maintaining data integrity during startup. When persisted config exists,
endpoints are available immediately so no delay occurs. For new clusters,
temperature collection begins once validation completes (~30s).

Preserves Pulse's correctness guarantee: users can trust metrics
immediately after restart without waiting for "warm-up" period.
2025-11-14 23:38:44 +00:00
rcourtman
d49c333283 monitoring: add poll watchdog to prevent worker leaks (refs #696) 2025-11-14 11:24:59 +00:00
rcourtman
2ee693cc63 Add HTTP mode to pulse-sensor-proxy for multi-instance temperature monitoring
This implements HTTP/HTTPS support for pulse-sensor-proxy to enable
temperature monitoring across multiple separate Proxmox instances.

Architecture changes:
- Dual-mode operation: Unix socket (local) + HTTPS (remote)
- Unix socket remains default for security/performance (no breaking change)
- HTTP mode enables temps from external PVE hosts

Backend implementation:
- Add HTTPS server with TLS + Bearer token authentication to sensor-proxy
- Add TemperatureProxyURL and TemperatureProxyToken fields to PVEInstance
- Add HTTP client (internal/tempproxy/http_client.go) for remote proxy calls
- Update temperature collector to prefer HTTP proxy when configured
- Fallback logic: HTTP proxy → Unix socket → direct SSH (if not containerized)

Configuration:
- pulse-sensor-proxy config: http_enabled, http_listen_addr, http_tls_cert/key, http_auth_token
- PVEInstance config: temperature_proxy_url, temperature_proxy_token
- Environment variables: PULSE_SENSOR_PROXY_HTTP_* for all HTTP settings

Security:
- TLS 1.2+ with modern cipher suites
- Constant-time token comparison (timing attack prevention)
- Rate limiting applied to HTTP requests (shared with socket mode)
- Audit logging for all HTTP requests

Next steps:
- Update installer script to support HTTP mode + auto-registration
- Add Pulse API endpoint for proxy registration
- Generate TLS certificates during installation
- Test multi-instance temperature collection

Related to #571 (multi-instance architecture)
2025-11-13 16:13:53 +00:00
rcourtman
ddac48e640 Ensure agent ID collisions respect token boundaries (Related to #658) 2025-11-12 22:46:56 +00:00
rcourtman
e27d9f7deb Preserve storage backups after partial failures (Related to #704) 2025-11-12 21:10:18 +00:00
rcourtman
d2247d3ad7 Related to #692: Skip unsupported guest OS info calls 2025-11-12 19:17:09 +00:00
rcourtman
2e1ef44ecd Filter read-only filesystems from host agent disk metrics (related to #690)
Squashfs snap mounts on Ubuntu (and similar read-only filesystems like
erofs on Home Assistant OS) always report near-full usage and trigger
false disk alerts. The filter logic existed in Proxmox monitoring but
wasn't applied to host agents.

Changes:
- Extract read-only filesystem filter to shared pkg/fsfilters package
- Apply filter in hostmetrics.collectDisks() for host/docker agents
- Apply filter in monitor.ApplyHostReport() for backward compatibility
- Convert internal/monitoring/fs_filters.go to wrapper functions

This prevents squashfs, erofs, iso9660, cdfs, udf, cramfs, romfs, and
saturated overlay filesystems from generating alerts. Filtering happens
at both collection time (agents) and ingestion time (server) to ensure
older agents don't cause false alerts until they're updated.
2025-11-12 09:47:02 +00:00
rcourtman
cc595da28b Fix guest agent OS info calls causing OpenBSD VM crashes (related to #692)
Add defensive mitigation to prevent repeated guest-get-osinfo calls that
trigger buggy behavior in QEMU guest agent 9.0.2 on OpenBSD 7.6.

The issue: OpenBSD doesn't have /etc/os-release (Linux convention), and
qemu-ga 9.0.2 appears to spawn excessive helper processes trying to read
this file whenever guest-get-osinfo is called. These helpers don't clean
up properly, eventually exhausting the process table and crashing the VM.

The fix: Track consecutive OS info failures per VM. After 3 failures,
automatically skip future guest-get-osinfo calls for that VM while
continuing to fetch other guest agent data (network interfaces, version).
This prevents triggering the buggy code path while maintaining most guest
agent functionality.

The counter resets on success, so if the guest agent is upgraded or the
issue is resolved, Pulse will automatically resume OS info collection.

Related to #692
2025-11-11 22:27:22 +00:00
rcourtman
bb7ca93c18 feat: Add mdadm RAID monitoring support for host agents
Implements comprehensive mdadm RAID array monitoring for Linux hosts
via pulse-host-agent. Arrays are automatically detected and monitored
with real-time status updates, rebuild progress tracking, and automatic
alerting for degraded or failed arrays.

Key changes:

**Backend:**
- Add mdadm package for parsing mdadm --detail output
- Extend host agent report structure with RAID array data
- Integrate mdadm collection into host agent (Linux-only, best-effort)
- Add RAID array processing in monitoring system
- Implement automatic alerting:
  - Critical alerts for degraded arrays or arrays with failed devices
  - Warning alerts for rebuilding/resyncing arrays with progress tracking
  - Auto-clear alerts when arrays return to healthy state

**Frontend:**
- Add TypeScript types for RAID arrays and devices
- Display RAID arrays in host details drawer with:
  - Array status (clean/degraded/recovering) with color-coded indicators
  - Device counts (active/total/failed/spare)
  - Rebuild progress percentage and speed when applicable
  - Green for healthy, amber for rebuilding, red for degraded

**Documentation:**
- Document mdadm monitoring feature in HOST_AGENT.md
- Explain requirements (Linux, mdadm installed, root access)
- Clarify scope (software RAID only, hardware RAID not supported)

**Testing:**
- Add comprehensive tests for mdadm output parsing
- Test parsing of healthy, degraded, and rebuilding arrays
- Verify proper extraction of device states and rebuild progress

All builds pass successfully. RAID monitoring is automatic and best-effort
- if mdadm is not installed or no arrays exist, host agent continues
reporting other metrics normally.

Related to #676
2025-11-09 16:36:33 +00:00