Pulse/docs/api/SCHEDULER_HEALTH.md

Scheduler Health API

New in v4.24.0

Endpoint: GET /api/monitoring/scheduler/health

Returns a snapshot of the adaptive polling scheduler, queue state, circuit breakers, and per-instance status. Requires authentication (session cookie or bearer token).

Key Features:

• Real-time scheduler health monitoring
• Circuit breaker status per instance
• Dead-letter queue tracking (tasks that repeatedly fail)
• Per-instance staleness metrics
• No query parameters required
• Read-only endpoint (rate-limited under general 500 req/min bucket)

---

Request

GET /api/monitoring/scheduler/health
Authorization: Bearer <token>

No query parameters are needed.

---

Response Overview

{
  "updatedAt": "2025-10-20T13:05:42Z",  // RFC 3339 timestamp
  "enabled": true,                       // Mirrors AdaptivePollingEnabled setting
  "queue": {...},
  "deadLetter": {...},
  "breakers": [...],          // legacy summary (for backward compatibility)
  "staleness": [...],         // legacy summary (for backward compatibility)
  "instances": [ ... ]        // authoritative per-instance view (v4.24.0+)
}

Field Notes:

• updatedAt: RFC 3339 timestamp of when this snapshot was generated
• enabled: Reflects the current AdaptivePollingEnabled system setting
• breakers and staleness: Legacy arrays maintained for backward compatibility; use instances for complete data
• instances: Authoritative source for per-instance health (v4.24.0+)

Queue Snapshot (queue)

Field	Type	Description
depth	integer	Current queue size
dueWithinSeconds	integer	Items scheduled within the next 12 seconds
perType	object	Counts per instance type, e.g. {"pve":4}

Dead-letter Snapshot (deadLetter)

Field	Type	Description
count	integer	Total items in the dead-letter queue
tasks	array	Limited to 25 entries for performance. Each task includes instance, type, nextRun, lastError, and failures count. For complete per-instance DLQ data, use instances[].deadLetter

Note: The top-level deadLetter.tasks array is capped at 25 items to prevent large responses. Use the instances array for exhaustive coverage.

Instances (instances)

Each element gives a complete view of one instance.

Field	Type	Description
key	string	Unique key type::name
type	string	Instance type (pve, pbs, pmg, etc.)
displayName	string	Friendly name (falls back to host/name)
instance	string	Raw instance identifier
connection	string	Connection URL or host
pollStatus	object	Recent poll outcomes
breaker	object	Circuit breaker state
deadLetter	object	Dead-letter insight for this instance

Poll Status (pollStatus)

Field	Type	Description
lastSuccess	timestamp nullable	RFC 3339 timestamp of most recent successful poll
lastError	object nullable	{ at, message, category } where at is RFC 3339, message describes the error, and category is transient (network issues, timeouts) or permanent (auth failures, invalid config)
consecutiveFailures	integer	Current failure streak length (resets on successful poll)
firstFailureAt	timestamp nullable	New in v4.24.0: RFC 3339 timestamp when the current failure streak began. Useful for calculating failure duration

Timing Metadata (v4.24.0+):

• firstFailureAt: Tracks when a failure streak started, enabling "failing for X minutes" calculations
• Resets to null when a successful poll occurs
• Combine with consecutiveFailures to assess severity

Breaker (breaker)

Field	Type	Description
state	string	closed (healthy), open (failing), half_open (testing recovery), or unknown (not initialized)
since	timestamp nullable	New in v4.24.0: RFC 3339 timestamp when the current state began. Use to calculate how long a breaker has been open
lastTransition	timestamp nullable	New in v4.24.0: RFC 3339 timestamp of the most recent state change (e.g., closed → open)
retryAt	timestamp nullable	New in v4.24.0: RFC 3339 timestamp of next scheduled retry attempt when breaker is open or half-open
failureCount	integer	New in v4.24.0: Number of failures in the current breaker cycle. Resets when breaker closes

Circuit Breaker Timing (v4.24.0+):

• since: When did the current state start? (e.g., "breaker has been open for 5 minutes")
• lastTransition: When was the last state change? (useful for detecting flapping)
• retryAt: When will the next retry attempt occur? (for open/half-open states)
• failureCount: How many failures have accumulated? (triggers state transitions)

State Transitions:

• closed → open: Triggered after N failures (default: 5)
• open → half_open: After timeout period, allows one test request
• half_open → closed: If test request succeeds
• half_open → open: If test request fails

Dead-letter (deadLetter)

Field	Type	Description
present	boolean	true if instance is in the DLQ
reason	string	max_retry_attempts or permanent_failure
firstAttempt	timestamp nullable	First time the instance hit DLQ
lastAttempt	timestamp nullable	Most recent DLQ enqueue
retryCount	integer	Number of DLQ attempts
nextRetry	timestamp nullable	Next scheduled retry time

---

Example Response

{
  "updatedAt": "2025-10-20T13:05:42Z",
  "enabled": true,
  "queue": {
    "depth": 7,
    "dueWithinSeconds": 2,
    "perType": { "pve": 4, "pbs": 2, "pmg": 1 }
  },
  "deadLetter": {
    "count": 1,
    "tasks": [
      {
        "instance": "pbs-b",
        "type": "pbs",
        "nextRun": "2025-10-20T13:30:00Z",
        "lastError": "401 unauthorized",
        "failures": 5
      }
    ]
  },
  "breakers": [
    {
      "instance": "pve-a",
      "type": "pve",
      "state": "half_open",
      "failures": 3,
      "retryAt": "2025-10-20T13:06:15Z"
    }
  ],
  "staleness": [
    {
      "instance": "pve-a",
      "type": "pve",
      "score": 0.42,
      "lastSuccess": "2025-10-20T13:05:10Z",
      "lastError": "2025-10-20T13:05:40Z"
    }
  ],
  "instances": [
    {
      "key": "pve::pve-a",
      "type": "pve",
      "displayName": "Pulse PVE Cluster",
      "instance": "pve-a",
      "connection": "https://pve-a:8006",
      "pollStatus": {
        "lastSuccess": "2025-10-20T13:05:10Z",
        "lastError": {
          "at": "2025-10-20T13:05:40Z",
          "message": "connection timeout",
          "category": "transient"
        },
        "consecutiveFailures": 2,
        "firstFailureAt": "2025-10-20T13:05:20Z"
      },
      "breaker": {
        "state": "half_open",
        "since": "2025-10-20T13:05:40Z",
        "lastTransition": "2025-10-20T13:05:40Z",
        "retryAt": "2025-10-20T13:06:15Z",
        "failureCount": 3
      },
      "deadLetter": {
        "present": false
      }
    },
    {
      "key": "pbs::pbs-b",
      "type": "pbs",
      "displayName": "Backup PBS",
      "instance": "pbs-b",
      "connection": "https://pbs-b:8007",
      "pollStatus": {
        "lastSuccess": "2025-10-20T12:55:00Z",
        "lastError": {
          "at": "2025-10-20T13:00:01Z",
          "message": "401 unauthorized",
          "category": "permanent"
        },
        "consecutiveFailures": 5,
        "firstFailureAt": "2025-10-20T12:58:30Z"
      },
      "breaker": {
        "state": "open",
        "since": "2025-10-20T13:00:01Z",
        "lastTransition": "2025-10-20T13:00:01Z",
        "retryAt": "2025-10-20T13:02:01Z",
        "failureCount": 5
      },
      "deadLetter": {
        "present": true,
        "reason": "max_retry_attempts",
        "firstAttempt": "2025-10-20T12:58:30Z",
        "lastAttempt": "2025-10-20T13:00:01Z",
        "retryCount": 5,
        "nextRetry": "2025-10-20T13:30:00Z"
      }
    }
  ]
}

---

Useful jq Queries

Instances with recent errors

curl -s http://HOST:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.pollStatus.lastError != null) | {key, lastError: .pollStatus.lastError}'

Current dead-letter queue entries

curl -s http://HOST:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.deadLetter.present) | {key, reason: .deadLetter.reason, retryCount: .deadLetter.retryCount}'

Breakers not closed

curl -s http://HOST:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.breaker.state != "closed") | {key, breaker: .breaker}'

Stale instances (score > 0.5)

curl -s http://HOST:7655/api/monitoring/scheduler/health \
  | jq '.staleness[] | select(.score > 0.5)'

Instances sorted by failure streak

curl -s http://HOST:7655/api/monitoring/scheduler/health \
  | jq '.instances[] | select(.pollStatus.consecutiveFailures > 0) | {key, failures: .pollStatus.consecutiveFailures}'

---

Migration Notes

Legacy Field	Status	Replacement
breakers array	retains summary	use instances[].breaker for detailed view
deadLetter.tasks	retains summary	use instances[].deadLetter for per-instance enrichment
staleness array	unchanged	combined with pollStatus.lastSuccess gives precise timestamps

The instances array centralizes per-instance telemetry; existing integrations can migrate at their own pace.

---

Operational Notes

v4.24.0 Behavior:

• Read-only endpoint: This endpoint is informational only and does not modify scheduler state
• Rate limiting: Falls under the general API limit (500 requests/minute per IP)
• Authentication required: Must provide valid session cookie or API token
• Adaptive polling disabled: When adaptive polling is disabled (enabled: false), the response includes empty breakers, staleness, and instances arrays
• Real-time data: Reflects current scheduler state; not historical (for trends, use metrics/logs)
• No query parameters: Returns complete snapshot on every request
• Automatic adjustments: The enabled field automatically reflects the AdaptivePollingEnabled system setting

Use Cases:

• Monitoring dashboards: Embed in Grafana/Prometheus for real-time scheduler health
• Alerting: Trigger alerts on open circuit breakers or high DLQ counts
• Debugging: Investigate why specific instances aren't polling successfully
• Capacity planning: Monitor queue depth trends to assess if polling intervals need adjustment

Breaking Changes:

• None: v4.24.0 only adds fields; all existing consumers continue to work
• Consumers just gain access to richer metadata (firstFailureAt, breaker timestamps, DLQ retry windows)

---

Troubleshooting Examples

1. Transient outages: look for pollStatus.lastError.category == "transient" to confirm network hiccups; check breaker.retryAt to see when retries resume.
2. Permanent failures: deadLetter.present == true with reason == "permanent_failure" indicates credential or configuration issues.
3. Breaker stuck: breaker.state != "closed" with since > 5 minutes suggests manual intervention or rollback.
4. Staleness spike: compare pollStatus.lastSuccess with updatedAt to estimate data age; cross-reference staleness.score for alert thresholds.

Use Grafana dashboards for historical trends; the API complements dashboards by revealing instant state and precise failure context.