mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-07-09 16:00:59 +00:00
feat: capture Proxmox memory snapshots in diagnostics
This commit is contained in:
parent
f46ff1792b
commit
a74baed121
9 changed files with 882 additions and 58 deletions
303
docs/DOCKER_HUB_README.md
Normal file
303
docs/DOCKER_HUB_README.md
Normal file
|
|
@ -0,0 +1,303 @@
|
|||
# Pulse
|
||||
|
||||
[](https://github.com/rcourtman/Pulse/releases/latest)
|
||||
[](https://hub.docker.com/r/rcourtman/pulse)
|
||||
[](LICENSE)
|
||||
|
||||
**Real-time monitoring for Proxmox VE, Proxmox Mail Gateway, PBS, and Docker infrastructure with alerts and webhooks.**
|
||||
|
||||
Monitor your hybrid Proxmox and Docker estate from a single dashboard. Get instant alerts when nodes go down, containers misbehave, backups fail, or storage fills up. Supports email, Discord, Slack, Telegram, and more.
|
||||
|
||||
**[Try the live demo →](https://demo.pulserelay.pro)** (read-only with mock data)
|
||||
|
||||
## Support Pulse Development
|
||||
|
||||
Pulse is built by a solo developer in evenings and weekends. Your support helps:
|
||||
- Keep me motivated to add new features
|
||||
- Prioritize bug fixes and user requests
|
||||
- Ensure Pulse stays 100% free and open-source forever
|
||||
|
||||
[](https://github.com/sponsors/rcourtman)
|
||||
[](https://ko-fi.com/rcourtman)
|
||||
|
||||
**Not ready to sponsor?** Star the project or share it with your homelab community!
|
||||
|
||||
## Features
|
||||
|
||||
- **Auto-Discovery**: Finds Proxmox nodes on your network, one-liner setup via generated scripts
|
||||
- **Cluster Support**: Configure one node, monitor entire cluster
|
||||
- **Enterprise Security**:
|
||||
- Credentials encrypted at rest, masked in logs, never sent to frontend
|
||||
- CSRF protection for all state-changing operations
|
||||
- Rate limiting (500 req/min general, 10 attempts/min for auth)
|
||||
- Account lockout after failed login attempts
|
||||
- Secure session management with HttpOnly cookies
|
||||
- bcrypt password hashing (cost 12) - passwords NEVER stored in plain text
|
||||
- API tokens stored securely with restricted file permissions
|
||||
- Security headers (CSP, X-Frame-Options, etc.)
|
||||
- Comprehensive audit logging
|
||||
- Live monitoring of VMs, containers, nodes, storage
|
||||
- **Smart Alerts**: Email and webhooks (Discord, Slack, Telegram, Teams, ntfy.sh, Gotify)
|
||||
- Example: "VM 'webserver' is down on node 'pve1'"
|
||||
- Example: "Storage 'local-lvm' at 85% capacity"
|
||||
- Example: "VM 'database' is back online"
|
||||
- **Adaptive Thresholds**: Hysteresis-based trigger/clear levels, fractional network thresholds, per-metric search, reset-to-defaults, and Custom overrides with inline audit trail
|
||||
- **Alert Timeline Analytics**: Rich history explorer with acknowledgement/clear markers, escalation breadcrumbs, and quick filters for noisy resources
|
||||
- **Ceph Awareness**: Surface Ceph health, pool utilisation, and daemon status automatically when Proxmox exposes Ceph-backed storage
|
||||
- Unified view of PBS backups, PVE backups, and snapshots
|
||||
- **Interactive Backup Explorer**: Cross-highlighted bar chart + grid with quick time-range pivots (24h/7d/30d/custom) and contextual tooltips for the busiest jobs
|
||||
- Proxmox Mail Gateway analytics: mail volume, spam/virus trends, quarantine health, and cluster node status
|
||||
- Optional Docker container monitoring via lightweight agent
|
||||
- Config export/import with encryption and authentication
|
||||
- Automatic stable updates with safe rollback (opt-in)
|
||||
- Dark/light themes, responsive design
|
||||
- Built with Go for minimal resource usage
|
||||
|
||||
[View screenshots and full documentation on GitHub →](https://github.com/rcourtman/Pulse)
|
||||
|
||||
## Privacy
|
||||
|
||||
**Pulse respects your privacy:**
|
||||
- No telemetry or analytics collection
|
||||
- No phone-home functionality
|
||||
- No external API calls (except for configured webhooks)
|
||||
- All data stays on your server
|
||||
- Open source - verify it yourself
|
||||
|
||||
Your infrastructure data is yours alone.
|
||||
|
||||
## Quick Start with Docker
|
||||
|
||||
### Basic Setup
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name pulse \
|
||||
-p 7655:7655 \
|
||||
-v pulse_data:/data \
|
||||
--restart unless-stopped \
|
||||
rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
Then open `http://localhost:7655` and complete the security setup wizard.
|
||||
|
||||
### Network Discovery
|
||||
|
||||
Pulse automatically discovers Proxmox nodes on your network! By default, it scans:
|
||||
- 192.168.0.0/16 (home networks)
|
||||
- 10.0.0.0/8 (private networks)
|
||||
- 172.16.0.0/12 (Docker/internal networks)
|
||||
|
||||
To scan a custom subnet instead:
|
||||
```bash
|
||||
docker run -d \
|
||||
--name pulse \
|
||||
-p 7655:7655 \
|
||||
-v pulse_data:/data \
|
||||
-e DISCOVERY_SUBNET="192.168.50.0/24" \
|
||||
--restart unless-stopped \
|
||||
rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
### Automated Deployment with Pre-configured Auth
|
||||
|
||||
```bash
|
||||
# Deploy with authentication pre-configured
|
||||
docker run -d \
|
||||
--name pulse \
|
||||
-p 7655:7655 \
|
||||
-v pulse_data:/data \
|
||||
-e API_TOKEN="your-secure-token" \
|
||||
-e PULSE_AUTH_USER="admin" \
|
||||
-e PULSE_AUTH_PASS="your-password" \
|
||||
--restart unless-stopped \
|
||||
rcourtman/pulse:latest
|
||||
|
||||
# Plain text credentials are automatically hashed for security
|
||||
# No setup required - API works immediately
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
pulse:
|
||||
image: rcourtman/pulse:latest
|
||||
container_name: pulse
|
||||
ports:
|
||||
- "7655:7655"
|
||||
volumes:
|
||||
- pulse_data:/data
|
||||
environment:
|
||||
# NOTE: Env vars override UI settings. Remove env var to allow UI configuration.
|
||||
|
||||
# Network discovery (usually not needed - auto-scans common networks)
|
||||
# - DISCOVERY_SUBNET=192.168.50.0/24 # Only for non-standard networks
|
||||
|
||||
# Ports
|
||||
# - PORT=7655 # Backend port (default: 7655)
|
||||
# - FRONTEND_PORT=7655 # Frontend port (default: 7655)
|
||||
|
||||
# Security (all optional - runs open by default)
|
||||
# - PULSE_AUTH_USER=admin # Username for web UI login
|
||||
# - PULSE_AUTH_PASS=your-password # Plain text or bcrypt hash (auto-hashed if plain)
|
||||
# - API_TOKEN=your-token # Plain text or SHA3-256 hash (auto-hashed if plain)
|
||||
# - ALLOW_UNPROTECTED_EXPORT=false # Allow export without auth (default: false)
|
||||
|
||||
# Security: Plain text credentials are automatically hashed
|
||||
# You can provide either:
|
||||
# 1. Plain text (auto-hashed): PULSE_AUTH_PASS=mypassword
|
||||
# 2. Pre-hashed (advanced): PULSE_AUTH_PASS='$$2a$$12$$...'
|
||||
# Note: Escape $ as $$ in docker-compose.yml for pre-hashed values
|
||||
|
||||
# Performance
|
||||
# - CONNECTION_TIMEOUT=10 # Connection timeout in seconds (default: 10)
|
||||
|
||||
# CORS & logging
|
||||
# - ALLOWED_ORIGINS=https://app.example.com # CORS origins (default: none, same-origin only)
|
||||
# - LOG_LEVEL=info # Log level: debug/info/warn/error (default: info)
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
pulse_data:
|
||||
```
|
||||
|
||||
## Initial Setup
|
||||
|
||||
1. Open `http://<your-server>:7655`
|
||||
2. **Complete the mandatory security setup** (first-time only)
|
||||
3. Create your admin username and password
|
||||
4. Save the generated API token for automation
|
||||
|
||||
## Configure Proxmox/PBS Nodes
|
||||
|
||||
After logging in:
|
||||
|
||||
1. Go to Settings → Nodes
|
||||
2. Discovered nodes appear automatically
|
||||
3. Click "Setup Script" next to any node
|
||||
4. Click "Generate Setup Code" button (creates a 6-character code valid for 5 minutes)
|
||||
5. Copy and run the provided one-liner on your Proxmox/PBS host
|
||||
6. Node is configured and monitoring starts automatically
|
||||
|
||||
**Example setup command:**
|
||||
```bash
|
||||
curl -sSL "http://pulse:7655/api/setup-script?type=pve&host=https://pve:8006&auth_token=ABC123" | bash
|
||||
```
|
||||
|
||||
## Docker Updates
|
||||
|
||||
```bash
|
||||
# Latest stable
|
||||
docker pull rcourtman/pulse:latest
|
||||
|
||||
# Latest RC/pre-release
|
||||
docker pull rcourtman/pulse:rc
|
||||
|
||||
# Specific version
|
||||
docker pull rcourtman/pulse:v4.22.0
|
||||
|
||||
# Then recreate your container
|
||||
docker stop pulse && docker rm pulse
|
||||
# Run your docker run or docker-compose command again
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
- **Authentication required** - Protects your Proxmox infrastructure credentials
|
||||
- **Quick setup wizard** - Secure your installation in under a minute
|
||||
- **Multiple auth methods**: Password authentication, API tokens, proxy auth (SSO), or combinations
|
||||
- **Proxy/SSO support** - Integrate with Authentik, Authelia, and other authentication proxies
|
||||
- **Enterprise-grade protection**:
|
||||
- Credentials encrypted at rest (AES-256-GCM)
|
||||
- CSRF tokens for state-changing operations
|
||||
- Rate limiting and account lockout protection
|
||||
- Secure session management with HttpOnly cookies
|
||||
- bcrypt password hashing (cost 12) - passwords NEVER stored in plain text
|
||||
- API tokens stored securely with restricted file permissions
|
||||
- Security headers (CSP, X-Frame-Options, etc.)
|
||||
- Comprehensive audit logging
|
||||
- **Security by design**:
|
||||
- Frontend never receives node credentials
|
||||
- API tokens visible only to authenticated users
|
||||
- Export/import requires authentication when configured
|
||||
|
||||
See [Security Documentation](https://github.com/rcourtman/Pulse/blob/main/docs/SECURITY.md) for details.
|
||||
|
||||
## HTTPS/TLS Configuration
|
||||
|
||||
Enable HTTPS by setting these environment variables:
|
||||
|
||||
```bash
|
||||
docker run -d -p 7655:7655 \
|
||||
-e HTTPS_ENABLED=true \
|
||||
-e TLS_CERT_FILE=/data/cert.pem \
|
||||
-e TLS_KEY_FILE=/data/key.pem \
|
||||
-v pulse_data:/data \
|
||||
-v /path/to/certs:/data/certs:ro \
|
||||
rcourtman/pulse:latest
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
#### Cannot login after setting up security
|
||||
- **Docker**: Ensure bcrypt hash is exactly 60 characters and wrapped in single quotes
|
||||
- **Docker Compose**: MUST escape $ characters as $$ (e.g., `$$2a$$12$$...`)
|
||||
- **Example (docker run)**: `PULSE_AUTH_PASS='$2a$12$YTZXOCEylj4TaevZ0DCeI.notayQZ..b0OZ97lUZ.Q24fljLiMQHK'`
|
||||
- **Example (docker-compose.yml)**: `PULSE_AUTH_PASS='$$2a$$12$$YTZXOCEylj4TaevZ0DCeI.notayQZ..b0OZ97lUZ.Q24fljLiMQHK'`
|
||||
- If hash is truncated or mangled, authentication will fail
|
||||
- Use Quick Security Setup in the UI to avoid manual configuration errors
|
||||
|
||||
#### .env file not created (Docker)
|
||||
- **Expected behavior**: When using environment variables, no .env file is created in /data
|
||||
- The .env file is only created when using Quick Security Setup or password changes
|
||||
- If you provide credentials via environment variables, they take precedence
|
||||
- To use Quick Security Setup: Start container WITHOUT auth environment variables
|
||||
|
||||
### VM Disk Stats Show "-"
|
||||
- VMs require QEMU Guest Agent to report disk usage (Proxmox API returns 0 for VMs)
|
||||
- Install guest agent in VM: `apt install qemu-guest-agent` (Linux) or virtio-win tools (Windows)
|
||||
- Enable in VM Options → QEMU Guest Agent, then restart VM
|
||||
- Container (LXC) disk stats always work (no guest agent needed)
|
||||
|
||||
### Connection Issues
|
||||
- Check Proxmox API is accessible (port 8006/8007)
|
||||
- Verify credentials have PVEAuditor role minimum
|
||||
- For PBS: ensure API token has Datastore.Audit permission
|
||||
|
||||
### Logs
|
||||
```bash
|
||||
# View logs
|
||||
docker logs pulse
|
||||
|
||||
# Follow logs
|
||||
docker logs -f pulse
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
Full documentation available on GitHub:
|
||||
|
||||
- [Complete Installation Guide](https://github.com/rcourtman/Pulse/blob/main/docs/INSTALL.md)
|
||||
- [Configuration Guide](https://github.com/rcourtman/Pulse/blob/main/docs/CONFIGURATION.md)
|
||||
- [VM Disk Monitoring](https://github.com/rcourtman/Pulse/blob/main/docs/VM_DISK_MONITORING.md) - Set up QEMU Guest Agent for accurate VM disk usage
|
||||
- [Troubleshooting](https://github.com/rcourtman/Pulse/blob/main/docs/TROUBLESHOOTING.md)
|
||||
- [API Reference](https://github.com/rcourtman/Pulse/blob/main/docs/API.md)
|
||||
- [Webhook Guide](https://github.com/rcourtman/Pulse/blob/main/docs/WEBHOOKS.md)
|
||||
- [Proxy Authentication](https://github.com/rcourtman/Pulse/blob/main/docs/PROXY_AUTH.md) - SSO integration with Authentik, Authelia, etc.
|
||||
- [Reverse Proxy Setup](https://github.com/rcourtman/Pulse/blob/main/docs/REVERSE_PROXY.md) - nginx, Caddy, Apache, Traefik configs
|
||||
- [Security](https://github.com/rcourtman/Pulse/blob/main/docs/SECURITY.md)
|
||||
- [FAQ](https://github.com/rcourtman/Pulse/blob/main/docs/FAQ.md)
|
||||
|
||||
## Links
|
||||
|
||||
- [GitHub Repository](https://github.com/rcourtman/Pulse)
|
||||
- [Releases & Changelog](https://github.com/rcourtman/Pulse/releases)
|
||||
- [Issues & Feature Requests](https://github.com/rcourtman/Pulse/issues)
|
||||
- [Live Demo](https://demo.pulserelay.pro)
|
||||
|
||||
## License
|
||||
|
||||
MIT - See [LICENSE](https://github.com/rcourtman/Pulse/blob/main/LICENSE)
|
||||
|
|
@ -703,7 +703,7 @@ export function Dashboard(props: DashboardProps) {
|
|||
<thead>
|
||||
<tr class="bg-gray-50 dark:bg-gray-700/50 text-gray-600 dark:text-gray-300 border-b border-gray-200 dark:border-gray-600">
|
||||
<th
|
||||
class="pl-4 pr-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[220px] xl:w-[260px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600 focus:outline-none focus:ring-2 focus:ring-inset focus:ring-blue-500"
|
||||
class="pl-4 pr-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[170px] lg:w-[190px] xl:w-[210px] 2xl:w-[260px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600 focus:outline-none focus:ring-2 focus:ring-inset focus:ring-blue-500"
|
||||
onClick={() => handleSort('name')}
|
||||
onKeyDown={(e) => e.key === 'Enter' && handleSort('name')}
|
||||
tabindex="0"
|
||||
|
|
@ -713,63 +713,63 @@ export function Dashboard(props: DashboardProps) {
|
|||
Name {sortKey() === 'name' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[60px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[56px] lg:w-[60px] xl:w-[64px] 2xl:w-[72px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('type')}
|
||||
>
|
||||
Type {sortKey() === 'type' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[70px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[64px] lg:w-[72px] xl:w-[80px] 2xl:w-[90px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('vmid')}
|
||||
>
|
||||
VMID {sortKey() === 'vmid' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[100px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[90px] lg:w-[100px] 2xl:w-[120px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('uptime')}
|
||||
>
|
||||
Uptime {sortKey() === 'uptime' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[160px] xl:w-[200px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[120px] lg:w-[140px] xl:w-[150px] 2xl:w-[190px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('cpu')}
|
||||
>
|
||||
CPU {sortKey() === 'cpu' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[160px] xl:w-[200px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[120px] lg:w-[140px] xl:w-[150px] 2xl:w-[190px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('memory')}
|
||||
>
|
||||
Memory {sortKey() === 'memory' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[160px] xl:w-[200px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[120px] lg:w-[140px] xl:w-[150px] 2xl:w-[190px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('disk')}
|
||||
>
|
||||
Disk {sortKey() === 'disk' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[90px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('diskRead')}
|
||||
>
|
||||
Disk Read{' '}
|
||||
{sortKey() === 'diskRead' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[90px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('diskWrite')}
|
||||
>
|
||||
Disk Write{' '}
|
||||
{sortKey() === 'diskWrite' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[90px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('networkIn')}
|
||||
>
|
||||
Net In {sortKey() === 'networkIn' && (sortDirection() === 'asc' ? '▲' : '▼')}
|
||||
</th>
|
||||
<th
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[90px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
class="px-2 py-1 text-left text-[11px] sm:text-xs font-medium uppercase tracking-wider w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px] cursor-pointer hover:bg-gray-200 dark:hover:bg-gray-600"
|
||||
onClick={() => handleSort('networkOut')}
|
||||
>
|
||||
Net Out{' '}
|
||||
|
|
|
|||
|
|
@ -218,7 +218,8 @@ export function GuestRow(props: GuestRowProps) {
|
|||
|
||||
// Get first cell styling
|
||||
const firstCellClass = createMemo(() => {
|
||||
const base = 'py-0.5 pr-2 whitespace-nowrap relative';
|
||||
const base =
|
||||
'py-0.5 pr-2 whitespace-nowrap relative w-[170px] lg:w-[190px] xl:w-[210px] 2xl:w-[260px]';
|
||||
const indent = 'pl-6';
|
||||
return `${base} ${indent}`;
|
||||
});
|
||||
|
|
@ -285,7 +286,7 @@ export function GuestRow(props: GuestRowProps) {
|
|||
</td>
|
||||
|
||||
{/* Type */}
|
||||
<td class="py-0.5 px-2 whitespace-nowrap">
|
||||
<td class="py-0.5 px-2 whitespace-nowrap w-[56px] lg:w-[60px] xl:w-[64px] 2xl:w-[72px]">
|
||||
<div class="flex h-[24px] items-center">
|
||||
<span
|
||||
class={`inline-block px-1.5 py-0.5 text-xs font-medium rounded ${
|
||||
|
|
@ -300,13 +301,13 @@ export function GuestRow(props: GuestRowProps) {
|
|||
</td>
|
||||
|
||||
{/* VMID */}
|
||||
<td class="py-0.5 px-2 whitespace-nowrap text-sm text-gray-600 dark:text-gray-400 align-middle">
|
||||
<td class="py-0.5 px-2 whitespace-nowrap w-[64px] lg:w-[72px] xl:w-[80px] 2xl:w-[90px] text-sm text-gray-600 dark:text-gray-400 align-middle">
|
||||
{props.guest.vmid}
|
||||
</td>
|
||||
|
||||
{/* Uptime */}
|
||||
<td
|
||||
class={`py-0.5 px-2 text-sm whitespace-nowrap align-middle ${
|
||||
class={`py-0.5 px-2 w-[90px] lg:w-[100px] 2xl:w-[120px] text-sm whitespace-nowrap align-middle ${
|
||||
props.guest.uptime < 3600 ? 'text-orange-500' : 'text-gray-600 dark:text-gray-400'
|
||||
}`}
|
||||
>
|
||||
|
|
@ -316,7 +317,7 @@ export function GuestRow(props: GuestRowProps) {
|
|||
</td>
|
||||
|
||||
{/* CPU */}
|
||||
<td class="py-0.5 px-2 w-[160px] xl:w-[200px]">
|
||||
<td class="py-0.5 px-2 w-[120px] lg:w-[140px] xl:w-[150px] 2xl:w-[190px]">
|
||||
<Show when={showGuestMetrics()} fallback={<span class="text-sm text-gray-400">-</span>}>
|
||||
<MetricBar
|
||||
value={cpuPercent()}
|
||||
|
|
@ -332,7 +333,7 @@ export function GuestRow(props: GuestRowProps) {
|
|||
</td>
|
||||
|
||||
{/* Memory */}
|
||||
<td class="py-0.5 px-2 w-[160px] xl:w-[200px]">
|
||||
<td class="py-0.5 px-2 w-[120px] lg:w-[140px] xl:w-[150px] 2xl:w-[190px]">
|
||||
<div title={memoryTooltip() ?? undefined}>
|
||||
<Show when={showGuestMetrics()} fallback={<span class="text-sm text-gray-400">-</span>}>
|
||||
<MetricBar
|
||||
|
|
@ -346,7 +347,7 @@ export function GuestRow(props: GuestRowProps) {
|
|||
</td>
|
||||
|
||||
{/* Disk – surface usage even if guest is currently stopped so users can see last reported values */}
|
||||
<td class="py-0.5 px-2 w-[160px] xl:w-[200px]">
|
||||
<td class="py-0.5 px-2 w-[120px] lg:w-[140px] xl:w-[150px] 2xl:w-[190px]">
|
||||
<Show
|
||||
when={hasDiskUsage()}
|
||||
fallback={
|
||||
|
|
@ -369,24 +370,24 @@ export function GuestRow(props: GuestRowProps) {
|
|||
</td>
|
||||
|
||||
{/* Disk I/O */}
|
||||
<td class="py-0.5 px-2">
|
||||
<td class="py-0.5 px-2 w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px]">
|
||||
<div class="flex h-[24px] items-center">
|
||||
<IOMetric value={props.guest.diskRead} disabled={!isRunning()} />
|
||||
</div>
|
||||
</td>
|
||||
<td class="py-0.5 px-2">
|
||||
<td class="py-0.5 px-2 w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px]">
|
||||
<div class="flex h-[24px] items-center">
|
||||
<IOMetric value={props.guest.diskWrite} disabled={!isRunning()} />
|
||||
</div>
|
||||
</td>
|
||||
|
||||
{/* Network I/O */}
|
||||
<td class="py-0.5 px-2">
|
||||
<td class="py-0.5 px-2 w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px]">
|
||||
<div class="flex h-[24px] items-center">
|
||||
<IOMetric value={props.guest.networkIn} disabled={!isRunning()} />
|
||||
</div>
|
||||
</td>
|
||||
<td class="py-0.5 px-2">
|
||||
<td class="py-0.5 px-2 w-[68px] lg:w-[76px] xl:w-[82px] 2xl:w-[100px]">
|
||||
<div class="flex h-[24px] items-center">
|
||||
<IOMetric value={props.guest.networkOut} disabled={!isRunning()} />
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -3805,12 +3805,25 @@ const Settings: Component<SettingsProps> = (props) => {
|
|||
{(() => {
|
||||
const sanitizeForGitHub = (data: Record<string, unknown>) => {
|
||||
// Deep clone the data
|
||||
const sanitized = JSON.parse(JSON.stringify(data)) as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
const sanitized = JSON.parse(JSON.stringify(data)) as Record<
|
||||
string,
|
||||
unknown
|
||||
>;
|
||||
|
||||
const connectionKeyMap = new Map<string, string>();
|
||||
const connectionKeyMap = new Map<string, string>();
|
||||
const instanceKeyMap = new Map<string, string>();
|
||||
|
||||
const getSanitizedInstance = (instance: string) => {
|
||||
if (!instance) return instance;
|
||||
if (instanceKeyMap.has(instance)) {
|
||||
return instanceKeyMap.get(instance)!;
|
||||
}
|
||||
const suffixMatch = instance.match(/\.(lan|local|home|internal)$/);
|
||||
const suffix = suffixMatch ? suffixMatch[0] : '';
|
||||
const label = `instance-REDACTED${suffix}`;
|
||||
instanceKeyMap.set(instance, label);
|
||||
return label;
|
||||
};
|
||||
|
||||
// Sanitize IP addresses (keep first octet for network type identification)
|
||||
const sanitizeIP = (ip: string) => {
|
||||
|
|
@ -3838,6 +3851,90 @@ const Settings: Component<SettingsProps> = (props) => {
|
|||
.replace(/\b\d{1,3}(?:\.\d{1,3}){3}\b/g, 'xxx.xxx.xxx.xxx');
|
||||
};
|
||||
|
||||
const sanitizeNotesArray = (notes: unknown) => {
|
||||
if (!Array.isArray(notes)) return notes;
|
||||
return notes.map((note) => {
|
||||
if (typeof note !== 'string') return note;
|
||||
const sanitizedNote = sanitizeText(note);
|
||||
return sanitizedNote ?? note;
|
||||
});
|
||||
};
|
||||
|
||||
const sanitizeNodeSnapshots = (
|
||||
snapshots: Array<Record<string, unknown>>,
|
||||
) =>
|
||||
snapshots.map((snapshot, index: number) => {
|
||||
const sanitizedSnapshot = { ...snapshot };
|
||||
const originalNode =
|
||||
typeof sanitizedSnapshot.node === 'string'
|
||||
? (sanitizedSnapshot.node as string)
|
||||
: '';
|
||||
if (originalNode) {
|
||||
sanitizedSnapshot.node =
|
||||
connectionKeyMap.get(originalNode) ||
|
||||
sanitizeHostname(originalNode);
|
||||
}
|
||||
|
||||
const originalInstance =
|
||||
typeof sanitizedSnapshot.instance === 'string'
|
||||
? (sanitizedSnapshot.instance as string)
|
||||
: '';
|
||||
if (originalInstance) {
|
||||
sanitizedSnapshot.instance = getSanitizedInstance(originalInstance);
|
||||
}
|
||||
|
||||
if (
|
||||
typeof sanitizedSnapshot.id === 'string' &&
|
||||
sanitizedSnapshot.id
|
||||
) {
|
||||
sanitizedSnapshot.id = `node-snapshot-${index}`;
|
||||
}
|
||||
|
||||
return sanitizedSnapshot;
|
||||
});
|
||||
|
||||
const sanitizeGuestSnapshots = (
|
||||
snapshots: Array<Record<string, unknown>>,
|
||||
) =>
|
||||
snapshots.map((snapshot, index: number) => {
|
||||
const sanitizedSnapshot = { ...snapshot };
|
||||
const originalNode =
|
||||
typeof sanitizedSnapshot.node === 'string'
|
||||
? (sanitizedSnapshot.node as string)
|
||||
: '';
|
||||
if (originalNode) {
|
||||
sanitizedSnapshot.node =
|
||||
connectionKeyMap.get(originalNode) ||
|
||||
sanitizeHostname(originalNode);
|
||||
}
|
||||
|
||||
const originalInstance =
|
||||
typeof sanitizedSnapshot.instance === 'string'
|
||||
? (sanitizedSnapshot.instance as string)
|
||||
: '';
|
||||
if (originalInstance) {
|
||||
sanitizedSnapshot.instance = getSanitizedInstance(originalInstance);
|
||||
}
|
||||
|
||||
if (typeof sanitizedSnapshot.name === 'string') {
|
||||
sanitizedSnapshot.name = 'vm-REDACTED';
|
||||
}
|
||||
|
||||
if (typeof sanitizedSnapshot.vmid === 'number') {
|
||||
sanitizedSnapshot.vmid = index + 1;
|
||||
} else if (typeof sanitizedSnapshot.vmid === 'string') {
|
||||
sanitizedSnapshot.vmid = `vm-${index + 1}`;
|
||||
}
|
||||
|
||||
if (Array.isArray(sanitizedSnapshot.notes)) {
|
||||
sanitizedSnapshot.notes = sanitizeNotesArray(
|
||||
sanitizedSnapshot.notes,
|
||||
);
|
||||
}
|
||||
|
||||
return sanitizedSnapshot;
|
||||
});
|
||||
|
||||
// Sanitize nodes
|
||||
if (sanitized.nodes) {
|
||||
sanitized.nodes = (
|
||||
|
|
@ -3944,6 +4041,18 @@ const Settings: Component<SettingsProps> = (props) => {
|
|||
});
|
||||
}
|
||||
|
||||
if (Array.isArray(sanitized.nodeSnapshots)) {
|
||||
sanitized.nodeSnapshots = sanitizeNodeSnapshots(
|
||||
sanitized.nodeSnapshots as Array<Record<string, unknown>>,
|
||||
);
|
||||
}
|
||||
|
||||
if (Array.isArray(sanitized.guestSnapshots)) {
|
||||
sanitized.guestSnapshots = sanitizeGuestSnapshots(
|
||||
sanitized.guestSnapshots as Array<Record<string, unknown>>,
|
||||
);
|
||||
}
|
||||
|
||||
// Sanitize backend diagnostics (if present)
|
||||
if (
|
||||
sanitized.backendDiagnostics &&
|
||||
|
|
@ -3951,6 +4060,18 @@ const Settings: Component<SettingsProps> = (props) => {
|
|||
) {
|
||||
const backend = sanitized.backendDiagnostics as Record<string, unknown>;
|
||||
|
||||
if (Array.isArray(backend.nodeSnapshots)) {
|
||||
backend.nodeSnapshots = sanitizeNodeSnapshots(
|
||||
backend.nodeSnapshots as Array<Record<string, unknown>>,
|
||||
);
|
||||
}
|
||||
|
||||
if (Array.isArray(backend.guestSnapshots)) {
|
||||
backend.guestSnapshots = sanitizeGuestSnapshots(
|
||||
backend.guestSnapshots as Array<Record<string, unknown>>,
|
||||
);
|
||||
}
|
||||
|
||||
if (Array.isArray(backend.nodes)) {
|
||||
backend.nodes = backend.nodes.map((rawNode, index: number) => {
|
||||
const node = rawNode as Record<string, unknown>;
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/updates"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/pbs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
|
||||
|
|
@ -25,6 +26,10 @@ type DiagnosticsInfo struct {
|
|||
PBS []PBSDiagnostic `json:"pbs"`
|
||||
System SystemDiagnostic `json:"system"`
|
||||
Errors []string `json:"errors"`
|
||||
// NodeSnapshots captures the raw memory payload and derived usage Pulse last observed per node.
|
||||
NodeSnapshots []monitoring.NodeMemorySnapshot `json:"nodeSnapshots,omitempty"`
|
||||
// GuestSnapshots captures recent per-guest memory breakdowns (VM/LXC) with the raw Proxmox fields.
|
||||
GuestSnapshots []monitoring.GuestMemorySnapshot `json:"guestSnapshots,omitempty"`
|
||||
}
|
||||
|
||||
// NodeDiagnostic contains diagnostic info for a Proxmox node
|
||||
|
|
@ -280,6 +285,17 @@ func (r *Router) handleDiagnostics(w http.ResponseWriter, req *http.Request) {
|
|||
diag.PBS = append(diag.PBS, pbsDiag)
|
||||
}
|
||||
|
||||
// Include cached monitor snapshots for memory diagnostics if available
|
||||
if r.monitor != nil {
|
||||
snapshots := r.monitor.GetDiagnosticSnapshots()
|
||||
if len(snapshots.Nodes) > 0 {
|
||||
diag.NodeSnapshots = snapshots.Nodes
|
||||
}
|
||||
if len(snapshots.Guests) > 0 {
|
||||
diag.GuestSnapshots = snapshots.Guests
|
||||
}
|
||||
}
|
||||
|
||||
// Add any recent errors from logs (this would need a log collector)
|
||||
// For now, just check basic connectivity
|
||||
|
||||
|
|
|
|||
172
internal/monitoring/diagnostic_snapshots.go
Normal file
172
internal/monitoring/diagnostic_snapshots.go
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
package monitoring
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
)
|
||||
|
||||
// NodeMemoryRaw captures the raw memory fields returned by Proxmox for a node.
|
||||
type NodeMemoryRaw struct {
|
||||
Total uint64 `json:"total"`
|
||||
Used uint64 `json:"used"`
|
||||
Free uint64 `json:"free"`
|
||||
Available uint64 `json:"available"`
|
||||
Avail uint64 `json:"avail"`
|
||||
Buffers uint64 `json:"buffers"`
|
||||
Cached uint64 `json:"cached"`
|
||||
Shared uint64 `json:"shared"`
|
||||
EffectiveAvailable uint64 `json:"effectiveAvailable"`
|
||||
FallbackTotal uint64 `json:"fallbackTotal,omitempty"`
|
||||
FallbackUsed uint64 `json:"fallbackUsed,omitempty"`
|
||||
FallbackFree uint64 `json:"fallbackFree,omitempty"`
|
||||
FallbackCalculated bool `json:"fallbackCalculated,omitempty"`
|
||||
ProxmoxMemorySource string `json:"proxmoxMemorySource,omitempty"`
|
||||
}
|
||||
|
||||
// NodeMemorySnapshot records the final node memory calculation alongside the raw data.
|
||||
type NodeMemorySnapshot struct {
|
||||
Instance string `json:"instance"`
|
||||
Node string `json:"node"`
|
||||
RetrievedAt time.Time `json:"retrievedAt"`
|
||||
MemorySource string `json:"memorySource"`
|
||||
FallbackReason string `json:"fallbackReason,omitempty"`
|
||||
Memory models.Memory `json:"memory"`
|
||||
Raw NodeMemoryRaw `json:"raw"`
|
||||
}
|
||||
|
||||
// VMMemoryRaw captures both the listing and detailed status memory fields for a VM/CT.
|
||||
type VMMemoryRaw struct {
|
||||
ListingMem uint64 `json:"listingMem"`
|
||||
ListingMaxMem uint64 `json:"listingMaxmem"`
|
||||
StatusMem uint64 `json:"statusMem,omitempty"`
|
||||
StatusFreeMem uint64 `json:"statusFreemem,omitempty"`
|
||||
StatusMaxMem uint64 `json:"statusMaxmem,omitempty"`
|
||||
Balloon uint64 `json:"balloon,omitempty"`
|
||||
BalloonMin uint64 `json:"balloonMin,omitempty"`
|
||||
MemInfoUsed uint64 `json:"meminfoUsed,omitempty"`
|
||||
MemInfoFree uint64 `json:"meminfoFree,omitempty"`
|
||||
MemInfoTotal uint64 `json:"meminfoTotal,omitempty"`
|
||||
Agent int `json:"agent,omitempty"`
|
||||
DerivedFromBall bool `json:"derivedFromBalloon,omitempty"`
|
||||
}
|
||||
|
||||
// GuestMemorySnapshot records the memory calculation for a guest (VM/LXC).
|
||||
type GuestMemorySnapshot struct {
|
||||
GuestType string `json:"guestType"`
|
||||
Instance string `json:"instance"`
|
||||
Node string `json:"node"`
|
||||
Name string `json:"name"`
|
||||
VMID int `json:"vmid"`
|
||||
Status string `json:"status"`
|
||||
RetrievedAt time.Time `json:"retrievedAt"`
|
||||
MemorySource string `json:"memorySource"`
|
||||
Memory models.Memory `json:"memory"`
|
||||
Raw VMMemoryRaw `json:"raw"`
|
||||
Notes []string `json:"notes,omitempty"`
|
||||
}
|
||||
|
||||
// DiagnosticSnapshotSet aggregates the latest node and guest snapshots.
|
||||
type DiagnosticSnapshotSet struct {
|
||||
Nodes []NodeMemorySnapshot `json:"nodes"`
|
||||
Guests []GuestMemorySnapshot `json:"guests"`
|
||||
}
|
||||
|
||||
func makeNodeSnapshotKey(instance, node string) string {
|
||||
return fmt.Sprintf("%s|%s", instance, node)
|
||||
}
|
||||
|
||||
func makeGuestSnapshotKey(instance, guestType, node string, vmid int) string {
|
||||
return fmt.Sprintf("%s|%s|%s|%d", instance, guestType, node, vmid)
|
||||
}
|
||||
|
||||
func (m *Monitor) recordNodeSnapshot(instance, node string, snapshot NodeMemorySnapshot) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
|
||||
snapshot.Instance = instance
|
||||
snapshot.Node = node
|
||||
if snapshot.RetrievedAt.IsZero() {
|
||||
snapshot.RetrievedAt = time.Now()
|
||||
}
|
||||
|
||||
m.diagMu.Lock()
|
||||
defer m.diagMu.Unlock()
|
||||
if m.nodeSnapshots == nil {
|
||||
m.nodeSnapshots = make(map[string]NodeMemorySnapshot)
|
||||
}
|
||||
m.nodeSnapshots[makeNodeSnapshotKey(instance, node)] = snapshot
|
||||
}
|
||||
|
||||
func (m *Monitor) recordGuestSnapshot(instance, guestType, node string, vmid int, snapshot GuestMemorySnapshot) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
|
||||
snapshot.Instance = instance
|
||||
snapshot.GuestType = guestType
|
||||
snapshot.Node = node
|
||||
snapshot.VMID = vmid
|
||||
if snapshot.RetrievedAt.IsZero() {
|
||||
snapshot.RetrievedAt = time.Now()
|
||||
}
|
||||
|
||||
m.diagMu.Lock()
|
||||
defer m.diagMu.Unlock()
|
||||
if m.guestSnapshots == nil {
|
||||
m.guestSnapshots = make(map[string]GuestMemorySnapshot)
|
||||
}
|
||||
m.guestSnapshots[makeGuestSnapshotKey(instance, guestType, node, vmid)] = snapshot
|
||||
}
|
||||
|
||||
// GetDiagnosticSnapshots returns copies of the latest node and guest memory snapshots.
|
||||
func (m *Monitor) GetDiagnosticSnapshots() DiagnosticSnapshotSet {
|
||||
result := DiagnosticSnapshotSet{
|
||||
Nodes: []NodeMemorySnapshot{},
|
||||
Guests: []GuestMemorySnapshot{},
|
||||
}
|
||||
|
||||
if m == nil {
|
||||
return result
|
||||
}
|
||||
|
||||
m.diagMu.RLock()
|
||||
defer m.diagMu.RUnlock()
|
||||
|
||||
if len(m.nodeSnapshots) > 0 {
|
||||
result.Nodes = make([]NodeMemorySnapshot, 0, len(m.nodeSnapshots))
|
||||
for _, snapshot := range m.nodeSnapshots {
|
||||
result.Nodes = append(result.Nodes, snapshot)
|
||||
}
|
||||
sort.Slice(result.Nodes, func(i, j int) bool {
|
||||
if result.Nodes[i].Instance == result.Nodes[j].Instance {
|
||||
return result.Nodes[i].Node < result.Nodes[j].Node
|
||||
}
|
||||
return result.Nodes[i].Instance < result.Nodes[j].Instance
|
||||
})
|
||||
}
|
||||
|
||||
if len(m.guestSnapshots) > 0 {
|
||||
result.Guests = make([]GuestMemorySnapshot, 0, len(m.guestSnapshots))
|
||||
for _, snapshot := range m.guestSnapshots {
|
||||
result.Guests = append(result.Guests, snapshot)
|
||||
}
|
||||
sort.Slice(result.Guests, func(i, j int) bool {
|
||||
if result.Guests[i].Instance == result.Guests[j].Instance {
|
||||
if result.Guests[i].Node == result.Guests[j].Node {
|
||||
if result.Guests[i].GuestType == result.Guests[j].GuestType {
|
||||
return result.Guests[i].VMID < result.Guests[j].VMID
|
||||
}
|
||||
return result.Guests[i].GuestType < result.Guests[j].GuestType
|
||||
}
|
||||
return result.Guests[i].Node < result.Guests[j].Node
|
||||
}
|
||||
return result.Guests[i].Instance < result.Guests[j].Instance
|
||||
})
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
|
@ -203,6 +203,9 @@ type Monitor struct {
|
|||
pbsBackupPollers map[string]bool // Track PBS backup polling goroutines per instance
|
||||
runtimeCtx context.Context // Context used while monitor is running
|
||||
wsHub *websocket.Hub // Hub used for broadcasting state
|
||||
diagMu sync.RWMutex // Protects diagnostic snapshot maps
|
||||
nodeSnapshots map[string]NodeMemorySnapshot
|
||||
guestSnapshots map[string]GuestMemorySnapshot
|
||||
}
|
||||
|
||||
// safePercentage calculates percentage safely, returning 0 if divisor is 0
|
||||
|
|
@ -763,6 +766,8 @@ func New(cfg *config.Config) (*Monitor, error) {
|
|||
lastPhysicalDiskPoll: make(map[string]time.Time),
|
||||
persistence: config.NewConfigPersistence(cfg.DataPath),
|
||||
pbsBackupPollers: make(map[string]bool),
|
||||
nodeSnapshots: make(map[string]NodeMemorySnapshot),
|
||||
guestSnapshots: make(map[string]GuestMemorySnapshot),
|
||||
}
|
||||
|
||||
// Load saved configurations
|
||||
|
|
@ -1698,6 +1703,19 @@ func (m *Monitor) pollPVEInstance(ctx context.Context, instanceName string, clie
|
|||
ClusterName: instanceCfg.ClusterName,
|
||||
}
|
||||
|
||||
nodeSnapshotRaw := NodeMemoryRaw{
|
||||
Total: node.MaxMem,
|
||||
Used: node.Mem,
|
||||
Free: node.MaxMem - node.Mem,
|
||||
FallbackTotal: node.MaxMem,
|
||||
FallbackUsed: node.Mem,
|
||||
FallbackFree: node.MaxMem - node.Mem,
|
||||
FallbackCalculated: true,
|
||||
ProxmoxMemorySource: "nodes-endpoint",
|
||||
}
|
||||
nodeMemorySource := "nodes-endpoint"
|
||||
var nodeFallbackReason string
|
||||
|
||||
// Debug logging for disk metrics - note that these values can fluctuate
|
||||
// due to thin provisioning and dynamic allocation
|
||||
if node.Disk > 0 && node.MaxDisk > 0 {
|
||||
|
|
@ -1716,6 +1734,7 @@ func (m *Monitor) pollPVEInstance(ctx context.Context, instanceName string, clie
|
|||
if node.Status == "online" {
|
||||
nodeInfo, nodeErr := client.GetNodeStatus(ctx, node.Node)
|
||||
if nodeErr != nil {
|
||||
nodeFallbackReason = "node-status-unavailable"
|
||||
// If we can't get node status, log but continue with data from /nodes endpoint
|
||||
if node.Disk > 0 && node.MaxDisk > 0 {
|
||||
log.Warn().
|
||||
|
|
@ -1735,6 +1754,20 @@ func (m *Monitor) pollPVEInstance(ctx context.Context, instanceName string, clie
|
|||
Msg("Could not get node status - no fallback metrics available (memory will include cache/buffers)")
|
||||
}
|
||||
} else if nodeInfo != nil {
|
||||
if nodeInfo.Memory != nil {
|
||||
nodeSnapshotRaw.Total = nodeInfo.Memory.Total
|
||||
nodeSnapshotRaw.Used = nodeInfo.Memory.Used
|
||||
nodeSnapshotRaw.Free = nodeInfo.Memory.Free
|
||||
nodeSnapshotRaw.Available = nodeInfo.Memory.Available
|
||||
nodeSnapshotRaw.Avail = nodeInfo.Memory.Avail
|
||||
nodeSnapshotRaw.Buffers = nodeInfo.Memory.Buffers
|
||||
nodeSnapshotRaw.Cached = nodeInfo.Memory.Cached
|
||||
nodeSnapshotRaw.Shared = nodeInfo.Memory.Shared
|
||||
nodeSnapshotRaw.EffectiveAvailable = nodeInfo.Memory.EffectiveAvailable()
|
||||
nodeSnapshotRaw.ProxmoxMemorySource = "node-status"
|
||||
nodeSnapshotRaw.FallbackCalculated = false
|
||||
}
|
||||
|
||||
// Convert LoadAvg from interface{} to float64
|
||||
loadAvg := make([]float64, 0, len(nodeInfo.LoadAvg))
|
||||
for _, val := range nodeInfo.LoadAvg {
|
||||
|
|
@ -1807,14 +1840,17 @@ func (m *Monitor) pollPVEInstance(ctx context.Context, instanceName string, clie
|
|||
switch {
|
||||
case nodeInfo.Memory.Available > 0:
|
||||
logCtx.Msg("Node memory: using available field (excludes reclaimable cache)")
|
||||
nodeMemorySource = "available-field"
|
||||
case nodeInfo.Memory.Avail > 0:
|
||||
logCtx.Msg("Node memory: using avail field (excludes reclaimable cache)")
|
||||
nodeMemorySource = "avail-field"
|
||||
default:
|
||||
logCtx.
|
||||
Uint64("free", nodeInfo.Memory.Free).
|
||||
Uint64("buffers", nodeInfo.Memory.Buffers).
|
||||
Uint64("cached", nodeInfo.Memory.Cached).
|
||||
Msg("Node memory: derived available from free+buffers+cached (excludes reclaimable cache)")
|
||||
nodeMemorySource = "derived-free-buffers-cached"
|
||||
}
|
||||
default:
|
||||
// Fallback to traditional used memory if no cache-aware data is exposed
|
||||
|
|
@ -1824,6 +1860,7 @@ func (m *Monitor) pollPVEInstance(ctx context.Context, instanceName string, clie
|
|||
Uint64("total", nodeInfo.Memory.Total).
|
||||
Uint64("used", nodeInfo.Memory.Used).
|
||||
Msg("Node memory: no cache-aware metrics - using traditional calculation (includes cache)")
|
||||
nodeMemorySource = "node-status-used"
|
||||
}
|
||||
|
||||
free := int64(nodeInfo.Memory.Total - actualUsed)
|
||||
|
|
@ -1894,9 +1931,25 @@ func (m *Monitor) pollPVEInstance(ctx context.Context, instanceName string, clie
|
|||
Str("instance", instanceName).
|
||||
Str("node", node.Node).
|
||||
Msg("Preserving previous memory metrics - node status unavailable this cycle")
|
||||
|
||||
if nodeFallbackReason == "" {
|
||||
nodeFallbackReason = "preserved-previous-snapshot"
|
||||
}
|
||||
nodeMemorySource = "previous-snapshot"
|
||||
if nodeSnapshotRaw.ProxmoxMemorySource == "node-status" && nodeSnapshotRaw.Total == 0 {
|
||||
nodeSnapshotRaw.ProxmoxMemorySource = "previous-snapshot"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m.recordNodeSnapshot(instanceName, node.Node, NodeMemorySnapshot{
|
||||
RetrievedAt: time.Now(),
|
||||
MemorySource: nodeMemorySource,
|
||||
FallbackReason: nodeFallbackReason,
|
||||
Memory: modelNode.Memory,
|
||||
Raw: nodeSnapshotRaw,
|
||||
})
|
||||
|
||||
// Collect temperature data via SSH (non-blocking, best effort)
|
||||
// Only attempt for online nodes
|
||||
if node.Status == "online" && m.tempCollector != nil {
|
||||
|
|
@ -2432,6 +2485,15 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
continue
|
||||
}
|
||||
|
||||
memTotal := res.MaxMem
|
||||
memUsed := res.Mem
|
||||
memorySource := "cluster-resources"
|
||||
guestRaw := VMMemoryRaw{
|
||||
ListingMem: res.Mem,
|
||||
ListingMaxMem: res.MaxMem,
|
||||
}
|
||||
var detailedStatus *proxmox.VMStatus
|
||||
|
||||
// Try to get actual disk usage from guest agent if VM is running
|
||||
diskUsed := res.Disk
|
||||
diskTotal := res.MaxDisk
|
||||
|
|
@ -2449,7 +2511,7 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
// We should prefer guest agent data when available for accurate metrics
|
||||
if res.Status == "running" && res.Type == "qemu" {
|
||||
// First check if agent is enabled by getting VM status
|
||||
vmStatus, err := client.GetVMStatus(ctx, res.Node, res.VMID)
|
||||
status, err := client.GetVMStatus(ctx, res.Node, res.VMID)
|
||||
if err != nil {
|
||||
log.Debug().
|
||||
Err(err).
|
||||
|
|
@ -2457,15 +2519,47 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
Str("vm", res.Name).
|
||||
Int("vmid", res.VMID).
|
||||
Msg("Could not get VM status to check guest agent availability")
|
||||
} else if vmStatus != nil {
|
||||
} else if status != nil {
|
||||
detailedStatus = status
|
||||
guestRaw.StatusMaxMem = detailedStatus.MaxMem
|
||||
guestRaw.StatusMem = detailedStatus.Mem
|
||||
guestRaw.StatusFreeMem = detailedStatus.FreeMem
|
||||
guestRaw.Balloon = detailedStatus.Balloon
|
||||
guestRaw.BalloonMin = detailedStatus.BalloonMin
|
||||
guestRaw.Agent = detailedStatus.Agent
|
||||
if detailedStatus.MemInfo != nil {
|
||||
guestRaw.MemInfoUsed = detailedStatus.MemInfo.Used
|
||||
guestRaw.MemInfoFree = detailedStatus.MemInfo.Free
|
||||
guestRaw.MemInfoTotal = detailedStatus.MemInfo.Total
|
||||
}
|
||||
|
||||
// Use actual disk I/O values from detailed status
|
||||
diskReadBytes = int64(vmStatus.DiskRead)
|
||||
diskWriteBytes = int64(vmStatus.DiskWrite)
|
||||
networkInBytes = int64(vmStatus.NetIn)
|
||||
networkOutBytes = int64(vmStatus.NetOut)
|
||||
diskReadBytes = int64(detailedStatus.DiskRead)
|
||||
diskWriteBytes = int64(detailedStatus.DiskWrite)
|
||||
networkInBytes = int64(detailedStatus.NetIn)
|
||||
networkOutBytes = int64(detailedStatus.NetOut)
|
||||
|
||||
if detailedStatus.Balloon > 0 && detailedStatus.Balloon < detailedStatus.MaxMem {
|
||||
memTotal = detailedStatus.Balloon
|
||||
guestRaw.DerivedFromBall = true
|
||||
} else if detailedStatus.MaxMem > 0 {
|
||||
memTotal = detailedStatus.MaxMem
|
||||
guestRaw.DerivedFromBall = false
|
||||
}
|
||||
|
||||
if detailedStatus.FreeMem > 0 && memTotal >= detailedStatus.FreeMem {
|
||||
memUsed = memTotal - detailedStatus.FreeMem
|
||||
memorySource = "status-freemem"
|
||||
} else if detailedStatus.Mem > 0 {
|
||||
memUsed = detailedStatus.Mem
|
||||
memorySource = "status-mem"
|
||||
}
|
||||
if memUsed > memTotal {
|
||||
memUsed = memTotal
|
||||
}
|
||||
|
||||
// Gather guest metadata from the agent when available
|
||||
guestIPs, guestIfaces, guestOSName, guestOSVersion := fetchGuestAgentMetadata(ctx, client, instanceName, res.Node, res.Name, res.VMID, vmStatus)
|
||||
guestIPs, guestIfaces, guestOSName, guestOSVersion := fetchGuestAgentMetadata(ctx, client, instanceName, res.Node, res.Name, res.VMID, detailedStatus)
|
||||
if len(guestIPs) > 0 {
|
||||
ipAddresses = guestIPs
|
||||
}
|
||||
|
|
@ -2481,12 +2575,12 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
|
||||
// Always try to get filesystem info if agent is enabled
|
||||
// Prefer guest agent data over cluster/resources data for accuracy
|
||||
if vmStatus.Agent > 0 {
|
||||
if detailedStatus.Agent > 0 {
|
||||
log.Debug().
|
||||
Str("instance", instanceName).
|
||||
Str("vm", res.Name).
|
||||
Int("vmid", res.VMID).
|
||||
Int("agent", vmStatus.Agent).
|
||||
Int("agent", detailedStatus.Agent).
|
||||
Uint64("current_disk", diskUsed).
|
||||
Uint64("current_maxdisk", diskTotal).
|
||||
Msg("Guest agent enabled, querying filesystem info for accurate disk usage")
|
||||
|
|
@ -2748,7 +2842,7 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
Str("instance", instanceName).
|
||||
Str("vm", res.Name).
|
||||
Int("vmid", res.VMID).
|
||||
Int("agent", vmStatus.Agent).
|
||||
Int("agent", detailedStatus.Agent).
|
||||
Msg("VM does not have guest agent enabled in config")
|
||||
}
|
||||
} else {
|
||||
|
|
@ -2761,16 +2855,43 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
}
|
||||
}
|
||||
|
||||
// Calculate I/O rates after we have the actual values
|
||||
if res.Status != "running" {
|
||||
memorySource = "powered-off"
|
||||
memUsed = 0
|
||||
}
|
||||
|
||||
memFree := uint64(0)
|
||||
if memTotal >= memUsed {
|
||||
memFree = memTotal - memUsed
|
||||
}
|
||||
|
||||
sampleTime := time.Now()
|
||||
currentMetrics := IOMetrics{
|
||||
DiskRead: diskReadBytes,
|
||||
DiskWrite: diskWriteBytes,
|
||||
NetworkIn: networkInBytes,
|
||||
NetworkOut: networkOutBytes,
|
||||
Timestamp: time.Now(),
|
||||
Timestamp: sampleTime,
|
||||
}
|
||||
diskReadRate, diskWriteRate, netInRate, netOutRate := m.rateTracker.CalculateRates(guestID, currentMetrics)
|
||||
|
||||
memoryUsage := safePercentage(float64(memUsed), float64(memTotal))
|
||||
memory := models.Memory{
|
||||
Total: int64(memTotal),
|
||||
Used: int64(memUsed),
|
||||
Free: int64(memFree),
|
||||
Usage: memoryUsage,
|
||||
}
|
||||
if memory.Free < 0 {
|
||||
memory.Free = 0
|
||||
}
|
||||
if memory.Used > memory.Total {
|
||||
memory.Used = memory.Total
|
||||
}
|
||||
if detailedStatus != nil && detailedStatus.Balloon > 0 {
|
||||
memory.Balloon = int64(detailedStatus.Balloon)
|
||||
}
|
||||
|
||||
vm := models.VM{
|
||||
ID: guestID,
|
||||
VMID: res.VMID,
|
||||
|
|
@ -2781,12 +2902,7 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
Type: "qemu",
|
||||
CPU: safeFloat(res.CPU),
|
||||
CPUs: res.MaxCPU,
|
||||
Memory: models.Memory{
|
||||
Total: int64(res.MaxMem),
|
||||
Used: int64(res.Mem),
|
||||
Free: int64(res.MaxMem - res.Mem),
|
||||
Usage: safePercentage(float64(res.Mem), float64(res.MaxMem)),
|
||||
},
|
||||
Memory: memory,
|
||||
Disk: models.Disk{
|
||||
Total: int64(diskTotal),
|
||||
Used: int64(diskUsed),
|
||||
|
|
@ -2804,7 +2920,7 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
DiskWrite: maxInt64(0, int64(diskWriteRate)),
|
||||
Uptime: int64(res.Uptime),
|
||||
Template: res.Template == 1,
|
||||
LastSeen: time.Now(),
|
||||
LastSeen: sampleTime,
|
||||
}
|
||||
|
||||
// Parse tags
|
||||
|
|
@ -2826,6 +2942,15 @@ func (m *Monitor) pollVMsAndContainersEfficient(ctx context.Context, instanceNam
|
|||
|
||||
allVMs = append(allVMs, vm)
|
||||
|
||||
m.recordGuestSnapshot(instanceName, vm.Type, res.Node, res.VMID, GuestMemorySnapshot{
|
||||
Name: vm.Name,
|
||||
Status: vm.Status,
|
||||
RetrievedAt: sampleTime,
|
||||
MemorySource: memorySource,
|
||||
Memory: vm.Memory,
|
||||
Raw: guestRaw,
|
||||
})
|
||||
|
||||
// For non-running VMs, zero out resource usage metrics to prevent false alerts
|
||||
// Proxmox may report stale or residual metrics for stopped VMs
|
||||
if vm.Status != "running" {
|
||||
|
|
@ -3026,6 +3151,13 @@ func (m *Monitor) pollVMsWithNodes(ctx context.Context, instanceName string, cli
|
|||
guestID = fmt.Sprintf("%s-%s-%d", instanceName, node.Node, vm.VMID)
|
||||
}
|
||||
|
||||
guestRaw := VMMemoryRaw{
|
||||
ListingMem: vm.Mem,
|
||||
ListingMaxMem: vm.MaxMem,
|
||||
Agent: vm.Agent,
|
||||
}
|
||||
memorySource := "listing-mem"
|
||||
|
||||
// Initialize I/O metrics from VM listing (may be 0 for disk I/O)
|
||||
diskReadBytes := int64(vm.DiskRead)
|
||||
diskWriteBytes := int64(vm.DiskWrite)
|
||||
|
|
@ -3042,6 +3174,17 @@ func (m *Monitor) pollVMsWithNodes(ctx context.Context, instanceName string, cli
|
|||
if vm.Status == "running" {
|
||||
// Try to get detailed VM status for more accurate memory reporting and disk I/O
|
||||
if status, err := client.GetVMStatus(ctx, node.Node, vm.VMID); err == nil && status != nil {
|
||||
guestRaw.StatusMaxMem = status.MaxMem
|
||||
guestRaw.StatusMem = status.Mem
|
||||
guestRaw.StatusFreeMem = status.FreeMem
|
||||
guestRaw.Balloon = status.Balloon
|
||||
guestRaw.BalloonMin = status.BalloonMin
|
||||
guestRaw.Agent = status.Agent
|
||||
if status.MemInfo != nil {
|
||||
guestRaw.MemInfoUsed = status.MemInfo.Used
|
||||
guestRaw.MemInfoFree = status.MemInfo.Free
|
||||
guestRaw.MemInfoTotal = status.MemInfo.Total
|
||||
}
|
||||
|
||||
// Use actual disk I/O values from detailed status
|
||||
diskReadBytes = int64(status.DiskRead)
|
||||
|
|
@ -3052,16 +3195,21 @@ func (m *Monitor) pollVMsWithNodes(ctx context.Context, instanceName string, cli
|
|||
// If balloon is enabled, use balloon as the total available memory
|
||||
if status.Balloon > 0 && status.Balloon < status.MaxMem {
|
||||
memTotal = status.Balloon
|
||||
guestRaw.DerivedFromBall = true
|
||||
} else {
|
||||
guestRaw.DerivedFromBall = false
|
||||
}
|
||||
|
||||
// If we have free memory from guest agent, calculate actual usage
|
||||
if status.FreeMem > 0 {
|
||||
// Guest agent reports free memory, so calculate used
|
||||
memUsed = memTotal - status.FreeMem
|
||||
memorySource = "status-freemem"
|
||||
} else if status.Mem > 0 {
|
||||
// No guest agent free memory data, but we have actual memory usage
|
||||
// Use the reported memory usage from Proxmox
|
||||
memUsed = status.Mem
|
||||
memorySource = "status-mem"
|
||||
} else {
|
||||
// No memory data available at all - show 0% usage
|
||||
memUsed = 0
|
||||
|
|
@ -3083,10 +3231,12 @@ func (m *Monitor) pollVMsWithNodes(ctx context.Context, instanceName string, cli
|
|||
} else {
|
||||
// Failed to get detailed status - show 0% usage
|
||||
memUsed = 0
|
||||
memorySource = "status-unavailable"
|
||||
}
|
||||
} else {
|
||||
// VM is not running, show 0 usage
|
||||
memUsed = 0
|
||||
memorySource = "powered-off"
|
||||
}
|
||||
|
||||
// Set CPU to 0 for non-running VMs to avoid false alerts
|
||||
|
|
@ -3296,12 +3446,13 @@ func (m *Monitor) pollVMsWithNodes(ctx context.Context, instanceName string, cli
|
|||
}
|
||||
|
||||
// Calculate I/O rates after we have the actual values
|
||||
sampleTime := time.Now()
|
||||
currentMetrics := IOMetrics{
|
||||
DiskRead: diskReadBytes,
|
||||
DiskWrite: diskWriteBytes,
|
||||
NetworkIn: networkInBytes,
|
||||
NetworkOut: networkOutBytes,
|
||||
Timestamp: time.Now(),
|
||||
Timestamp: sampleTime,
|
||||
}
|
||||
diskReadRate, diskWriteRate, netInRate, netOutRate := m.rateTracker.CalculateRates(guestID, currentMetrics)
|
||||
|
||||
|
|
@ -3341,19 +3492,27 @@ func (m *Monitor) pollVMsWithNodes(ctx context.Context, instanceName string, cli
|
|||
Template: vm.Template == 1,
|
||||
Tags: tags,
|
||||
Lock: vm.Lock,
|
||||
LastSeen: time.Now(),
|
||||
LastSeen: sampleTime,
|
||||
}
|
||||
allVMs = append(allVMs, modelVM)
|
||||
|
||||
m.recordGuestSnapshot(instanceName, modelVM.Type, node.Node, vm.VMID, GuestMemorySnapshot{
|
||||
Name: vm.Name,
|
||||
Status: vm.Status,
|
||||
RetrievedAt: sampleTime,
|
||||
MemorySource: memorySource,
|
||||
Memory: modelVM.Memory,
|
||||
Raw: guestRaw,
|
||||
})
|
||||
|
||||
// Record metrics history
|
||||
now := time.Now()
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "cpu", modelVM.CPU*100, now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "memory", modelVM.Memory.Usage, now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "disk", modelVM.Disk.Usage, now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "diskread", float64(modelVM.DiskRead), now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "diskwrite", float64(modelVM.DiskWrite), now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "netin", float64(modelVM.NetworkIn), now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "netout", float64(modelVM.NetworkOut), now)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "cpu", modelVM.CPU*100, sampleTime)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "memory", modelVM.Memory.Usage, sampleTime)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "disk", modelVM.Disk.Usage, sampleTime)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "diskread", float64(modelVM.DiskRead), sampleTime)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "diskwrite", float64(modelVM.DiskWrite), sampleTime)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "netin", float64(modelVM.NetworkIn), sampleTime)
|
||||
m.metricsHistory.AddGuestMetric(modelVM.ID, "netout", float64(modelVM.NetworkOut), sampleTime)
|
||||
|
||||
// Check thresholds for alerts
|
||||
m.alertManager.CheckGuest(modelVM, instanceName)
|
||||
|
|
|
|||
|
|
@ -131,6 +131,13 @@ func (m *Monitor) pollVMsWithNodesOptimized(ctx context.Context, instanceName st
|
|||
guestID = fmt.Sprintf("%s-%s-%d", instanceName, n.Node, vm.VMID)
|
||||
}
|
||||
|
||||
guestRaw := VMMemoryRaw{
|
||||
ListingMem: vm.Mem,
|
||||
ListingMaxMem: vm.MaxMem,
|
||||
Agent: vm.Agent,
|
||||
}
|
||||
memorySource := "listing-mem"
|
||||
|
||||
// Initialize metrics from VM listing (may be 0 for disk I/O)
|
||||
diskReadBytes := int64(vm.DiskRead)
|
||||
diskWriteBytes := int64(vm.DiskWrite)
|
||||
|
|
@ -150,13 +157,27 @@ func (m *Monitor) pollVMsWithNodesOptimized(ctx context.Context, instanceName st
|
|||
statusCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
if status, err := client.GetVMStatus(statusCtx, n.Node, vm.VMID); err == nil {
|
||||
vmStatus = status
|
||||
guestRaw.StatusMaxMem = status.MaxMem
|
||||
guestRaw.StatusMem = status.Mem
|
||||
guestRaw.StatusFreeMem = status.FreeMem
|
||||
guestRaw.Balloon = status.Balloon
|
||||
guestRaw.BalloonMin = status.BalloonMin
|
||||
guestRaw.Agent = status.Agent
|
||||
if status.MemInfo != nil {
|
||||
guestRaw.MemInfoUsed = status.MemInfo.Used
|
||||
guestRaw.MemInfoFree = status.MemInfo.Free
|
||||
guestRaw.MemInfoTotal = status.MemInfo.Total
|
||||
}
|
||||
if vmStatus.Balloon > 0 && vmStatus.Balloon < vmStatus.MaxMem {
|
||||
memTotal = vmStatus.Balloon
|
||||
guestRaw.DerivedFromBall = true
|
||||
}
|
||||
if vmStatus.FreeMem > 0 {
|
||||
memUsed = memTotal - vmStatus.FreeMem
|
||||
memorySource = "status-freemem"
|
||||
} else if vmStatus.Mem > 0 {
|
||||
memUsed = vmStatus.Mem
|
||||
memorySource = "status-mem"
|
||||
}
|
||||
// Use actual disk I/O values from detailed status
|
||||
diskReadBytes = int64(vmStatus.DiskRead)
|
||||
|
|
@ -167,6 +188,12 @@ func (m *Monitor) pollVMsWithNodesOptimized(ctx context.Context, instanceName st
|
|||
cancel()
|
||||
}
|
||||
|
||||
if vm.Status != "running" {
|
||||
memorySource = "powered-off"
|
||||
} else if vmStatus == nil {
|
||||
memorySource = "status-unavailable"
|
||||
}
|
||||
|
||||
if vm.Status == "running" && vmStatus != nil {
|
||||
guestIPs, guestIfaces, guestOSName, guestOSVersion := fetchGuestAgentMetadata(ctx, client, instanceName, n.Node, vm.Name, vm.VMID, vmStatus)
|
||||
if len(guestIPs) > 0 {
|
||||
|
|
@ -184,12 +211,13 @@ func (m *Monitor) pollVMsWithNodesOptimized(ctx context.Context, instanceName st
|
|||
}
|
||||
|
||||
// Calculate I/O rates after we have the actual values
|
||||
sampleTime := time.Now()
|
||||
currentMetrics := IOMetrics{
|
||||
DiskRead: diskReadBytes,
|
||||
DiskWrite: diskWriteBytes,
|
||||
NetworkIn: networkInBytes,
|
||||
NetworkOut: networkOutBytes,
|
||||
Timestamp: time.Now(),
|
||||
Timestamp: sampleTime,
|
||||
}
|
||||
diskReadRate, diskWriteRate, netInRate, netOutRate := m.rateTracker.CalculateRates(guestID, currentMetrics)
|
||||
|
||||
|
|
@ -469,7 +497,7 @@ func (m *Monitor) pollVMsWithNodesOptimized(ctx context.Context, instanceName st
|
|||
DiskWrite: maxInt64(0, int64(diskWriteRate)),
|
||||
Uptime: int64(vm.Uptime),
|
||||
Template: vm.Template == 1,
|
||||
LastSeen: time.Now(),
|
||||
LastSeen: sampleTime,
|
||||
Tags: tags,
|
||||
IPAddresses: ipAddresses,
|
||||
OSName: osName,
|
||||
|
|
@ -490,6 +518,15 @@ func (m *Monitor) pollVMsWithNodesOptimized(ctx context.Context, instanceName st
|
|||
|
||||
nodeVMs = append(nodeVMs, modelVM)
|
||||
|
||||
m.recordGuestSnapshot(instanceName, modelVM.Type, n.Node, vm.VMID, GuestMemorySnapshot{
|
||||
Name: vm.Name,
|
||||
Status: vm.Status,
|
||||
RetrievedAt: sampleTime,
|
||||
MemorySource: memorySource,
|
||||
Memory: modelVM.Memory,
|
||||
Raw: guestRaw,
|
||||
})
|
||||
|
||||
// Check alerts
|
||||
m.alertManager.CheckGuest(modelVM, instanceName)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -31,6 +31,21 @@ fi
|
|||
restart_backend() {
|
||||
echo -e "${YELLOW}Restarting backend...${NC}"
|
||||
|
||||
# Check if running under systemd
|
||||
if systemctl is-active --quiet pulse-hot-dev; then
|
||||
echo "Detected systemd hot-dev service, restarting it..."
|
||||
sudo systemctl restart pulse-hot-dev
|
||||
sleep 3
|
||||
if systemctl is-active --quiet pulse-hot-dev; then
|
||||
echo -e "${GREEN}✓ Hot-dev service restarted successfully${NC}"
|
||||
return 0
|
||||
else
|
||||
echo -e "${RED}✗ Hot-dev service failed to start. Check: sudo journalctl -u pulse-hot-dev -n 50${NC}"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Fallback: manual restart (for standalone hot-dev.sh)
|
||||
# Kill existing pulse backend
|
||||
pkill -x pulse 2>/dev/null || true
|
||||
sleep 1
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue