Pulse/docs/TROUBLESHOOTING.md

# 🔧 Troubleshooting Guide

## ⚡ Quick Fixes

### I forgot my password

Pulse does not provide an email password-reset flow. Choose the path that
matches how this self-hosted instance authenticates:

- **Local Pulse username and password**: recovery requires shell access to the
  Pulse host or container. Follow the deployment-specific steps below.
- **OIDC, SAML, or proxy authentication**: contact the identity-provider or
  Pulse administrator. If the administrator deliberately kept local login as a
  fallback, they can open the Pulse URL with `?show_local=true`; this only
  reveals the existing local form and does not reset credentials.
- **Temporary lockout**: wait for the lockout to expire. Another signed-in
  administrator can use the lockout reset in Pulse; password recovery is not
  required.

The local recovery steps remove only the Pulse-generated authentication file.
After restart, Pulse still requires the host-only bootstrap token before it
will accept replacement credentials. If `PULSE_AUTH_USER` or `PULSE_AUTH_PASS`
is supplied by Docker Compose, Kubernetes, systemd, or another deployment
manager, update that deployment configuration instead; its environment values
override the generated file.

**Docker**:

```bash
docker exec pulse rm /data/.env
docker restart pulse
# Access UI again. Pulse will require a bootstrap token for setup.
# Get it with:
docker exec pulse /app/pulse bootstrap-token
```

**Systemd**:

```bash
sudo rm /etc/pulse/.env
sudo systemctl restart pulse
sudo pulse bootstrap-token
```

**Proxmox LXC** (installed from the Proxmox shell):
Pulse runs inside the container, so run the same steps through `pct exec` on the Proxmox host. The binary needs its absolute path here, because `pct exec` runs with `PATH=/sbin:/bin:/usr/sbin:/usr/bin` and that does not include `/usr/local/bin`:

```bash
pct exec <ctid> -- rm /etc/pulse/.env
pct exec <ctid> -- systemctl restart pulse
pct exec <ctid> -- /usr/local/bin/pulse bootstrap-token
```

If you only missed the token during a fresh install (no password set yet), skip the first two commands and just read it back with the last one.

Treat the bootstrap token like a password: enter it only in the Pulse setup
screen for this instance and do not paste it into support requests or issue
reports.

### Port change didn't take effect
1. Check which service is running: `systemctl status pulse` (legacy installs may use `pulse-backend`).
2. Verify environment override: `systemctl show pulse --property=Environment`.
3. Docker: Ensure you updated the `-p` flag (e.g., `-p 8080:7655`).

### "Connection Refused"
- Check if Pulse is running.
- Verify the port is open on your firewall.
- **PBS**: Remember PBS uses port **8007** and requires **HTTPS**.

---

## 🔍 Common Issues

### Authentication

#### "Invalid username or password" after setup
- **Docker Compose**: Did you escape the `$` signs in your hash? Use `$$2a$$...`.
- **Truncated Hash**: Ensure your bcrypt hash is exactly 60 characters.

#### Cannot login / 401 Unauthorized
- Clear browser cookies.
- Check if your IP is locked out (wait 15 mins).
- If another admin can log in, use `POST /api/security/reset-lockout` to clear the lockout for your username or IP.

#### Audit Log verification shows unsigned events
- **Symptom**: Audit Log entries show “Unsigned” or verification fails in the UI.
- **Root cause**: Audit signing is disabled (crypto manager unavailable), so events are stored without signatures.
- **Fix**: Ensure `.encryption.key` is present and Pro/legacy Pro+/Cloud audit logging is enabled, then restart Pulse to regenerate `.audit-signing.key`. Newly created events will be signed; existing unsigned events remain unsigned.

#### Audit Log is empty
- **Symptom**: Audit Log shows zero events or "Console Logging Only."
- **Root cause**: Community plan uses console logging only, or Pro/legacy Pro+/Cloud audit logging is not enabled.
- **Fix**: Use Pro, legacy Pro+, or Cloud with audit logging enabled, then generate new audit events (logins, token creation, password changes).

#### Audit Log verification fails for older events
- **Symptom**: Older events fail verification while newer events pass.
- **Root cause**: The audit signing key changed (for example, `.audit-signing.key` was regenerated), so signatures no longer match.
- **Fix**: Restore the previous `.audit-signing.key` from backup to verify older events. If rotated intentionally, expect older events to fail verification.

### Monitoring Data

#### Agent fleet update or identity issue

- Open an outdated-agent notice or
  `/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** and
  copy the platform-specific command for each reported host. This is a manual
  handoff; Pulse does not remotely execute the command.
- Administrators can call the read-only Agent Fleet Doctor endpoint,
  `GET /api/agents/diagnostics`, to inspect liveness, version drift, profile
  deployment drift, expected telemetry gaps, and identity-split evidence. It
  does not change agent configuration or enqueue a repair.
- A current Pulse server does not prove fleet convergence. Eligible v6 agents
  update asynchronously; v5, PVE, disabled, and failed updates require manual
  handling.

#### Removed Pulse server but `pulse-agent` still logs connection failures

Removing the Pulse server does not remove agent services installed on monitored
hosts. On a systemd host, stop and disable the orphaned service to halt retries:

```bash
sudo systemctl disable --now pulse-agent.service
```

If the Pulse server is still reachable, use its generated uninstall command so
the agent can deregister cleanly. Otherwise, stopping the service is the safe
first step before platform-local cleanup.

#### Filter Pulse's systemd journal by severity

Current systemd installs preserve Pulse's structured log level as the journal
priority. For example, show warnings and more important records with:

```bash
sudo journalctl -u pulse -p warning
```

The stored message remains JSON (`"level":"warn"`, for example), while
`PRIORITY` is available to `journalctl` and syslog forwarding. If a current
Pulse warning appears only with `-p info`, inspect `systemctl cat pulse`; the
installed service must contain `SyslogLevelPrefix=true` and
`PULSE_LOG_JOURNAL_LEVEL_PREFIX=true`. Re-run the current signed installer to
repair an older generated unit rather than adding a JSON-parsing wrapper.

#### VMs show "-" for disk usage
- Install **QEMU Guest Agent** in the VM.
- Enable "QEMU Guest Agent" in Proxmox VM Options.
- Restart the VM.
- See [VM Disk Monitoring](VM_DISK_MONITORING.md).

#### Temperature data missing
- Install `lm-sensors` on the host.
- Run `sensors-detect`.
- Install the unified agent on the Proxmox host with `--enable-proxmox`.
- See [Temperature Monitoring](TEMPERATURE_MONITORING.md).

#### Docker hosts appearing/disappearing
- **Duplicate IDs**: Cloned VMs often share `/etc/machine-id`.
- **Fix**: Run `rm /etc/machine-id && systemd-machine-id-setup` on the clone.
- **Identity note**: The displayed IP is not the durable identity. Pulse uses
  the machine ID or an explicit agent ID, so two clones with the same value can
  collapse into one record even when their hostnames or IP addresses differ.

### Notifications

#### No alert when Pulse, power or internet goes down

Pulse cannot send a notification while its host is stopped or its outbound
network is unavailable. A local delivery-health warning is not an external
outage detector, and a successful **Send test** does not prove outage coverage.

- In **Alerts → Notifications → External watchdog**, configure a
  Healthchecks-compatible **success ping URL** and save the configuration.
  Keep the URL secret; do not include it in screenshots or support reports.
- Run the watchdog outside the failure you want to detect. Another machine on
  the same power supply or internet connection does not cover a whole-site
  outage. Its notification destination must also remain reachable independently
  of that site.
- At the watchdog, configure a one-minute period and a three-minute grace
  period, and enable its notification integration. Pulse sends a heartbeat
  every minute and a `/fail` signal if its monitoring loop stalls. For
  Healthchecks simple schedules, a missing heartbeat becomes down after
  **period plus grace**: approximately four minutes after the last success,
  not three. Recipient delivery can take longer. See
  [Healthchecks timing and notification concepts](https://healthchecks.io/docs/).
- Verify in an authorised test environment: confirm incoming heartbeats at
  the watchdog, stop the test Pulse instance, and check that the intended
  recipient actually receives the external alert. Restore Pulse and verify
  heartbeat recovery. Test loss of internet separately if you need that
  coverage; a stopped-process test does not prove it.
- To retire the check, pause or remove it at the watchdog too. Clearing the
  URL in Pulse stops its signals but does not pause the remote check.

A healthy heartbeat indicates Pulse monitoring-loop progress, not successful
delivery of every resource alert or external reachability of your services.
Continue checking delivery activity for destination failures.

#### Emails not sending
- Open **Alerts → Notifications** first. Pulse shows a delivery warning when
  failed or dead-lettered notifications remain in the persistent queue; a
  missing queue-health read is shown as unavailable rather than healthy.
- **Recent delivery activity** appears directly below that warning. It names
  the destination and affected alert, shows an absolute timestamp for timeline
  correlation, and includes safely redacted provider errors. Completed attempts
  remain for 7 days and dead-letter attempts remain for 30 days.
- After correcting the destination, use **Retry retained deliveries**. Use
  **Dismiss retained failures** only when those deliveries should not be sent.
  Both actions preserve delivery history; do not delete `notification_queue.db`
  to clear the warning.
- Check SMTP settings in **Alerts → Notifications**.
- Check logs: `docker logs pulse | grep email`.
- Ensure your SMTP provider allows the connection (e.g., Gmail App Passwords).

#### Webhooks failing
- Check the delivery warning in **Alerts → Notifications** and use **Send test**
  after correcting the destination. Recoverable retries do not trigger the
  warning; retained terminal failures do.
- If the test succeeds, use **Retry retained deliveries** to give the retained
  items a fresh retry budget. Dismiss them only when delivery is no longer
  wanted; neither action deletes the audit trail.
- Verify the URL is reachable from the Pulse server.
- If targeting private IPs, allow them in **Settings → System → Network → Webhook Security**.
- Check Pulse logs for HTTP status codes and response bodies.

### TrueNAS

#### "TrueNAS service unavailable"
- Ensure TrueNAS was added in **Settings → TrueNAS** with a valid HTTPS URL,
  API key, and the username that owns the key.
- Check that the TrueNAS system is reachable from the Pulse server (default
  HTTPS port).
- Verify the API-key owner has read access, then use **Test Connection** in
  Pulse. TrueNAS 25.04 and later should report the `jsonrpc-websocket`
  transport; TrueNAS 26 removed the former `/api/v2.0` REST endpoints.

#### TrueNAS pools/datasets not appearing
- TrueNAS data appears in the unified resource model and may take one polling cycle (30s) to appear.
- Check **Infrastructure** (TrueNAS host), **Storage** (pools/datasets), and **Recovery** (snapshots/replication).

### Navigation (v6)

#### Old bookmarks don't work
- Legacy URLs (`/proxmox`, `/docker`, `/kubernetes`, `/hosts`, `/services`) are not supported in v6.
- Update bookmarks to canonical routes. See [Migration Guide](MIGRATION_UNIFIED_NAV.md).

### Relay / Mobile

#### Relay showing "Disconnected"
- Confirm a valid Relay, Pro, grandfathered Pro+, or Cloud license is active (**Settings → Plans & Billing**).
- Check Pulse server can reach the relay server (outbound WebSocket to `relay.pulserelay.pro`).
- Review logs: `journalctl -u pulse | grep relay` or `docker logs pulse | grep relay`.

---

## 🛠️ Advanced Diagnostics

### Correlate Logs with Requests
Every API response has an `X-Request-ID` header. Use it to find the exact log entry:
```bash
# systemd / Proxmox LXC
journalctl -u pulse --no-pager | grep "request_id=abc123"

# Docker
docker logs pulse 2>&1 | grep "request_id=abc123"
```

### Check Permissions (Proxmox)
If Pulse can't see VMs or storage, check the user permissions on Proxmox:
```bash
pveum user permissions <user>@pam
```
At minimum, ensure the user/token has read access for inventory and metrics:

- `Sys.Audit`
- `Datastore.Audit`

For VM guest agent features on PVE 9+, prefer:

- `VM.GuestAgent.Audit` — required for disk usage and guest info
- `VM.GuestAgent.FileRead` — required for accurate memory monitoring (excludes buff/cache)

For PVE 8 only, use `VM.Monitor` instead of the `VM.GuestAgent.*` privileges.

Note: The built-in `PVEAuditor` role cannot be modified. Create a custom role (e.g. `PulseMonitor`) with the above privileges added, and assign it to your Pulse API token. After upgrading to PVE 9, add the `VM.GuestAgent.*` privileges and remove legacy `VM.Monitor` from the custom role.

**Rocky Linux / RHEL VMs**: The default qemu-guest-agent configuration may block file-read RPCs (`guest-file-open`, `guest-file-read`, `guest-file-close`). If memory or disk data is missing for these VMs, check `/etc/sysconfig/qemu-ga` and ensure those operations are not blocked, then restart the agent. Refer to your distro's qemu-guest-agent documentation for the exact config syntax.

### Proxmox pending-update access

A successful inventory check does not establish access to the package-update list.
Pulse requests `GET /nodes/{node}/apt/update` with its configured Proxmox credential.
[Proxmox's endpoint implementation](https://github.com/proxmox/pve-manager/blob/614bede5d65599c67e068cbf18d49717ea8ab33b/PVE/API2/APT.pm)
requires `Sys.Modify` on `/nodes/{node}` for that GET; `Sys.Audit` alone is not sufficient
in that implementation. Check the API requirements for your installed Proxmox version.

`Sys.Modify` is broader than read-only monitoring: the same privilege also authorises
the separate POST that refreshes package indexes. Do not add it automatically to a
monitoring role just to obtain an update badge. Keeping the narrower role and an
unavailable update check is a valid choice; unavailable is not a confirmed zero.
Older Pulse wording that says “Sys.Audit permission required” does not identify the
actual missing privilege.

For diagnosis, compare the affected node and endpoint using the exact credential
configured in Pulse, not an administrator's browser session. Keep TLS verification
enabled and credentials private; report only the HTTP status and a redacted denial
or whether `data` is empty/nonempty. Do not use POST, Refresh or Upgrade as a test.
If the matching GET succeeds but Pulse remains unavailable after its next update
check, report the Pulse version and displayed check time/status separately; API
success alone does not confirm the Pulse display has recovered.

### Recovery Mode
If you are completely locked out, you can trigger a recovery token from localhost:
```bash
curl -X POST http://localhost:7655/api/security/recovery \
  -d '{"action":"generate_token","duration":30}'
```
Use the returned token in `X-Recovery-Token` when calling `/api/security/recovery` to enable or disable local-only auth bypass (`disable_auth` / `enable_auth`). Token generation is localhost-only.

Example (enable recovery mode):
```bash
curl -X POST http://localhost:7655/api/security/recovery \
  -H "X-Recovery-Token: <token>" \
  -d '{"action":"disable_auth"}'
```

---

## 🆘 Getting Help

If you're still stuck:
1. **Check Logs**: `journalctl -u pulse -n 100` or `docker logs --tail 100 pulse`.
2. **Check Version**: `curl http://localhost:7655/api/version`.
3. **Open Issue**: Report on [GitHub Issues](https://github.com/rcourtman/Pulse/issues) with your logs and version info.