Contract-Neutral: E2E test tier metadata and validation only; no deployment runtime or public contract change.
Co-authored-by: Pulse Autonomous Maintainer <rcourtman@users.noreply.github.com>
Update source-shape guards for the shared per-generation resource list and cached mock unified view. Demote the onboarding E2E spec after its rate-limit/shared-state retry flake on main so it continues running without reddening the advisory verdict.
Contract-Neutral: advisory CI guardrails and probation tier bookkeeping only; no runtime or public contract delta
Contract-Neutral: Demotes one flaky E2E spec under the documented stable-to-probation tier policy; product behavior and public contracts are unchanged.
tests/90-operational-trust-protection-posture.spec.ts failed the
stable tier on main in run 29731882505 — the first tiered run. One
incident on main demotes: it rejoins the gate after 10 consecutive
green runs, per the rule above PROBATION_SPECS.
Contract-Neutral: probation demotion per documented tier rule: spec 90 failed on main run 29731882505; CI gating list only, no dev-runtime delta
Mined per-spec failure data from the 32 completed Core E2E main runs
since the 2026-07-18 quarantine delist (failed-shard logs; a green
shard means every spec in it passed). 26 specs failed or retry-flaked
within the 10 most recent completed runs; they seed PROBATION_SPECS in
tests/integration/playwright.config.ts. The remaining 62 files form the
stable tier and are the only specs that can fail the e2e-verdict job.
Mechanism extends the existing quarantine list rather than adding a
parallel one: PULSE_E2E_TIER=stable ignores probation specs,
PULSE_E2E_TIER=probation runs only them, unset runs the full suite
(local behavior unchanged). CI runs both tiers per shard against the
same containers; the probation pass sits behind continue-on-error with
its own report/results dirs and artifacts, so a probation flake is
reported in the shard summary without painting main red.
Promotion rule, documented next to the list: a probation spec promotes
to stable after 10 consecutive green main runs with no failure and no
retry-flake; one incident on main demotes a stable spec back. In the
newest completed run (29729544151) every failure was in a probation
spec — under this split that run's verdict is green.
Contract-Neutral: CI-only E2E tier split: gating semantics of push-CI verdict; local npm test behavior and dev-runtime orchestration unchanged, no public contract delta
Delist nineteen quarantined specs (69 green, 13 env-gated skips in one
full-batch run); 05/19/20/30/52/60/63/77/79 passed unchanged. 03 stays
quarantined with its reason recorded inline: multi-tenant scenarios 6/7
carry an unresolved org-resolution bug that CI's multi-tenant
environment would exercise, and 47/48 still await their concept rework.
- 15: organization panels only exist in multi-tenant installs, so those
routes now skip when the feature is off, and the Remote Access shell
description follows the refreshed Relay copy.
- 17: the backups page defaults to the guest-centric Coverage view when
anything needs attention; the day-activity assertions enter the By
date view first.
- 18/51: the Patrol page crashed to its error boundary under stale
autonomy stubs; the stubs now carry the requested/effective levels and
autopilot acknowledgement the page reads, and the paused copy is
sentence-case.
- 43: rewritten as a platform-pages tour (TrueNAS, vSphere, Proxmox,
Docker, PBS, PMG) since the settings platform views it pinned were
retired to a 404 with the legacy aliases.
- 49: the demo-estate proof follows the per-platform IA and drops a
self-contradictory storage exclusion that could never pass alongside
the west-a-service-pool assertion.
- 57: the RC Preview badge only renders in the authenticated shell, so
the spec signs in instead of stubbing the session.
- 62: the retired platform-management aliases deliberately 404 now
(cfb16e6da); the spec pins the graceful not-found page instead of the
removed redirect.
- 75: switches to cookie-session auth (the token path leaves
page.request unauthenticated), anchors on the Connection posture
region, and drops the retired operations sub-route.
- 84: a missing lab artifact is now a skip rather than a hard failure.
Contract-Neutral: E2E quarantine recovery, spec re-pins verified green on a fresh managed backend
Re-pin five quarantined commercial-funnel specs and delist all eight
from the quarantine (54/55/56 already passed unchanged; the two managed
demo runtime tests stay env-gated).
- The old stubbed auth shell renders the login page now that the app
shell makes auth-gated calls beyond the stub set; 59/61/70 sign in for
real and stub only the license or diagnostics surface under test.
- 53: hidden demo billing routes redirect to the settings infrastructure
root (the /install sub-route was retired), and the browser probe list
gains the trial/activate endpoint its hidden-keys contract already
named but never probed.
- 58: Remote Access stays in the free-tier nav on purpose (the panel
renders its own upgrade gate; hiding it made Relay undiscoverable), so
the spec pins its presence instead of its absence.
- 59/70: billing moved to the Pulse Intelligence settings group with a
Plans & Billing heading; plan-card assertions re-anchor at page scope
on the current copy (capability sentences, Apply key, Manual key
recovery disclosure, Patrol-flavoured activation summary).
- 61: the Commercial Funnel diagnostics card was deliberately removed
(c6bcad255); the spec now pins the inverse boundary, asserting the
internal funnel analytics never render even when the API payload
carries them.
Contract-Neutral: E2E quarantine recovery, spec re-pins verified green on a fresh managed backend
Re-pin seven quarantined specs to the surfaces that replaced the retired
/workloads and /storage pages and delist them from the quarantine.
- 44/45: the summary chart strip and its range buttons are gone; the
sparkline path maps x by timestamp, so the payload proofs assert
window coverage and tail stability directly against
/api/charts/workloads.
- 46: multi-range storage history proofs move onto /proxmox/storage
with point-count floors a fresh seeded backend guarantees.
- 50: physical disks live under the Physical Disks tab on
/proxmox/storage; rows are pinned by identity because live refreshes
re-sort the table, and late in-flight poll responses no longer fail
teardown.
- 59: the wide-set assertions run at 1600px (wide engages at 1536 after
the layout fix) plus a no-horizontal-scroll proof for the 1440px
compact default.
- 62: the Growth column is fixed to a 24h window; the rendered label is
proven against the same used-capacity history the endpoint serves.
- 64: the workloads surface lives on /proxmox, drawer tabs expose tab
roles, and row selection no longer writes a resource deep link.
Contract-Neutral: E2E quarantine recovery, spec re-pins verified green on a fresh managed backend
Re-pin the six /infrastructure-era specs to the vSphere platform page:
37 keeps its full facets/audit re-pin but is fixme'd because table-row
drawers disable remote history, leaving VMware hosts no reachable
change/action history surface (tracked); 38 and 42 move mention and
chat-turn coverage onto live mock state and the session transport (the
retired /api/ai/chat SSE stub is gone, so 42 proves the rendered
exchange instead of the wire payload); 39 and 41 pin the guest drawer's
read-only vCenter context and admin-action exclusions; 40 pins the
Datastores tab without backup semantics. 22 and 36 pass unchanged.
Verified via the managed local backend with PULSE_MOCK_MODE=true (8
passed, 1 tracked fixme).
Contract-Neutral: E2E quarantine-list shrink is test selection only; no deployment contract delta, mirrors bypass on 725837dc3
Re-pin 21 to the consolidated Connected-systems table (the address
column is gone; rows carry source, surfaces, state, and freshness) and
rewrite 26 against live mock websocket state with the stubable
alerts-config activation bypass, since REST resource stubs get
overwritten by state frames. 27, 28, and 31 were environmental-only
failures and pass unchanged against a mock-mode backend. Verified via
the managed local backend with PULSE_MOCK_MODE=true.
Contract-Neutral: E2E quarantine-list shrink is test selection only; no deployment contract delta, mirrors bypass on 725837dc3
Core E2E has failed on every push since 2026-06-29. Fifty of the 81 spec
files still assert the pre-platform-first IA (locators, URLs, subheader
copy the June rebuild replaced), and with maxFailures aborting shards at
20 the healthy specs produced no verdict at all. The quarantine list in
playwright.config.ts excludes the rotted files from all three projects so
the remaining 38 files (323 tests, including the smoke and journey
coverage) return to being a live per-push gate.
This is a stopgap, named as such in the config comment. Each quarantined
spec leaves the list by being re-pinned to the current UI, never by
deletion; the recovery effort is tracked separately.
Contract-Neutral: E2E spec quarantine is test selection only; no deployment contract delta, and the contract file is another agent's in-progress work
Proving run 28925348032 showed a shard can still hit the 45-minute job
timeout while the suite carries many real failures: shard 4 held the
visual crawl (5.3 minutes per attempt, currently failing) plus dozens of
14-second failing attempts, and with two retries the failure storm
outran the budget even under the 20-failure cap.
Retry once instead of twice on CI, and run the visual crawl only on the
desktop project. The mobile emulation projects keep their dedicated
mobile specs; a per-project 5-minute crawl was the single most expensive
line in the suite.
Every main push since the v6 branch flip was cancelled at the 45-minute
job timeout with no verdict. The flip brought the full 94-spec suite onto
main (the last green run, 2026-06-29, ran only 2 specs on the v5 main),
and it runs sequentially against a release-tagged image whose mock-fixture
gate returns 403 without a demo entitlement. Dozens of specs fail, retry
twice each, and burn the budget: of the 31 minutes of suite time in run
28907574469, 18.8 minutes were failing attempts.
- Add GO_BUILD_TAGS build arg (default release) and build the pulse:test
e2e image with it empty, matching the dev harness the suite is green
under. Shipped images keep the release tag; release-gate behavior keeps
its dedicated -tags release Go tests.
- Shard Playwright 4 ways across a CI matrix (214/202/205/203 tests per
shard) with per-shard report artifacts and an aggregate verdict job.
- Cap CI at 20 failures so an env-broken run reports red in minutes
instead of grinding into a no-verdict cancellation.
Backend:
- Add smart provider fallback when selected model's provider isn't configured
- Automatically switch to a model from a configured provider instead of failing
- Log warning when fallback occurs for visibility
Frontend (AISettings.tsx):
- Add helper functions to check if model's provider is configured
- Group model dropdown: configured providers first, unconfigured marked with ⚠️
- Add inline warning when selecting model from unconfigured provider
- Validate on save that model's provider is configured (or being added)
- Warn before clearing last configured provider (would disable AI)
- Warn before clearing provider that current model uses
- Add patrol interval validation (must be 0 or >= 10 minutes)
- Show red border + inline error for invalid patrol intervals 1-9
- Update patrol interval hint: '(0=off, 10+ to enable)'
These changes prevent confusing '500 Internal Server Error' and
'AI is not enabled or configured' errors when model/provider mismatch.
