Fail actions stranded without agent completion evidence and add force-fail (#1649)

An action that reached executing had exactly one way out: a terminal
operation receipt from the agent. Expiry skips executing rows on purpose,
and reconciliation bailed for every non-terminal query answer, so a Docker
update whose agent restarted mid-pull sat in executing forever with no
operator route out of it.

The agent-side receipt store rewrites accepted and started receipts to
interrupted on every Open(), and nothing can move an interrupted or
tombstoned receipt back to terminal. An identity-correlated answer of
either kind is therefore proof the operation will never report, and the
three executors now settle the action immediately as inconclusive with a
message telling the operator Pulse cannot confirm the effect and the
resource needs checking by hand.

A not_found answer, or a receipt still merely accepted or started, may
still be completed, so those keep waiting and only settle once the
dispatch attempt is older than one hour, the same threshold the
pulse-intelligence telemetry already uses to call an executing action
stuck. Every typed operation timeout is far shorter than that, so an
in-flight mutation is never cut short. A transport error answering the
query is still not evidence and preserves receipt_pending unchanged.

RecoverExecutingActions drives all of this on the existing two-minute
recovery loop, so rows already wedged before the upgrade heal themselves
without anyone touching them.

For the residue that reconciliation cannot reach, an agent that was
reinstalled or a legacy executing row with no dispatch attempt, POST
/api/actions/{id}/force-fail writes the same inconclusive terminal truth
under an operator attribution. It never touches the transport, refuses
anything already terminal, and is gated on admin plus settings:write on
top of the execute capability check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/release-control/v6/internal/subsystems/agent-lifecycle.md

@@ -5175,7 +5175,19 @@ The server derives operation version and request digest after canonical request
 fields are fixed, persists that binding on the action dispatch attempt, and
 queries the authenticated agent by the exact binding after callback loss or
 restart. `not_found`, interrupted, tombstoned, malformed, late, duplicate,
-wrong-agent, and mismatched responses never authorize resend or terminal truth.
+wrong-agent, and mismatched responses never authorize resend, and none of them
+can ever produce success or agent-attested verification. They are not, however,
+an open-ended wait. Because a reopen converts accepted or started work to
+`interrupted_unknown` and nothing can move an interrupted or tombstoned receipt
+back to terminal, an identity-correlated interrupted or tombstoned answer is
+proof that this agent will never complete the operation, and the API-owned
+lifecycle settles the action as inconclusive on the spot. A `not_found` answer,
+or a receipt still merely accepted or started, keeps the attempt receipt-pending
+until the API-owned bounded reconciliation window elapses and is only then
+settled the same inconclusive way. The agent still authors no execution,
+verification, evidence, or compensation truth in any of these paths: it reports
+receipt state, and the server alone decides that the absence of evidence is now
+durable.
 Terminal payload kind/version and strict adapter codecs are revalidated on
 write, read, query, and reopen. APT update and cleanup are adapters over this
 generic owner, persist only bounded sanitized typed results, and still share

docs/release-control/v6/internal/subsystems/api-contracts.md

@@ -8086,10 +8086,26 @@ Legacy executing rows without an attempt remain readable but inert.
 Task 07 Phase B1 owns server transport continuity. Phase B2 now adds the
 generic agent-side durable operation-receipt owner, exact server/agent binding,
 query-only reconciliation, payload-to-tombstone retention, and explicit agent
-receipt-protocol capability gate consumed by the APT adapters. A query
-`not_found`, interrupted record, or replay-denial tombstone leaves the action
-`executing` with its dispatch `receipt_pending`; it never authorizes executor
-or transport resend. Query-only executor routing is derived from the immutable
+receipt-protocol capability gate consumed by the APT adapters. No non-terminal
+query answer ever authorizes executor or transport resend. It does, however,
+bound how long the action may wait: an authenticated, identity-correlated query
+that returns an interrupted record or a replay-denial tombstone terminalizes
+the action immediately, because the agent-side store marks a receipt
+interrupted only after the agent restarted post-admission and neither an
+interrupted nor a tombstoned receipt can ever become terminal again. A
+`not_found` answer, or a receipt still merely accepted or started, leaves the
+action `executing`/`receipt_pending` while completion evidence may still
+arrive, and terminalizes only once the dispatch attempt is older than the
+bounded reconciliation window — the same one-hour threshold the
+pulse-intelligence telemetry uses to call an executing action stuck, so an
+action can never be reported stuck while reconciliation still waits on it.
+Every terminalization of that kind is durable inconclusive truth with no
+evidence and a stable reason code (`dispatch_evidence_interrupted`,
+`dispatch_evidence_discarded`, `dispatch_evidence_missing`,
+`dispatch_evidence_incomplete`); it is never success, and its operator-facing
+summary states that Pulse cannot confirm the effect and the resource must be
+checked directly. A transport error answering the query is still not evidence
+and preserves `receipt_pending` unchanged. Query-only executor routing is derived from the immutable
 operation kind and action identity committed on the dispatch attempt. Each
 durable executor registers the operation kinds it owns; recovery must not
 rediscover that route through current resource inventory. A successful mutation
@@ -8112,6 +8128,21 @@ authority, and a disconnected agent simply leaves the action
 `executing`/`receipt_pending` until a later periodic or registration pass can
 query it.
 
+`POST /api/actions/{id}/force-fail` is the operator escape hatch for whatever
+that automatic path cannot settle: an agent that was reinstalled and will never
+answer, or a legacy executing row with no dispatch attempt at all. It is a
+local repair route, not an agent capability and not a relay-mobile route, and
+it is gated harder than execute — admin plus `settings:write` on top of the
+same action-execute capability check the execute route applies. It only writes
+terminal audit truth: it never sends, resends, cancels, or queries the
+transport. It accepts only an `executing` action and refuses every terminal
+state with the existing already-final conflict token, so a stale client cannot
+overwrite a settled outcome, and it refuses planned, pending, or approved rows
+because plan expiry still owns those. The recorded result is the same durable
+inconclusive shape as automatic terminalization under reason code
+`operator_force_failed`, attributed to the authenticated operator, with the
+operator's justification preserved verbatim in the terminal audit event.
+
 Typed Docker / Podman container start, stop, and restart operations extend that
 same durable boundary rather than creating a provider-local action protocol.
 The agent wire request is closed and binds the exact action, dispatch attempt,

docs/release-control/v6/internal/subsystems/storage-recovery.md

@@ -2134,6 +2134,15 @@ Storage and recovery consumers may observe action detail, pending/settled
 projections, and correlated receipt state, but may not infer restore,
 verification, rollback, or compensation truth from those transport-only fields
 or create a local retry path when the core attempt is `receipt_pending`.
+The shared lifecycle, not this subsystem, also owns the exits from
+`receipt_pending`. A correlated interrupted or tombstoned receipt settles the
+action immediately, and a missing or still-unfinished receipt settles it only
+after the shared bounded reconciliation window, always as durable inconclusive
+truth with no evidence. Storage and recovery consumers must read that outcome
+as "effect unknown, verify the resource" and must never treat it as a confirmed
+failure, a restore/rollback signal, or a licence to redispatch. The operator
+force-fail override on the shared action route carries exactly the same
+inconclusive meaning and the same prohibition.
 The adjacent command transport now namespaces live sessions and pending
 response channels by organization plus agent identity, while durable receipt
 identity remains the immutable attempt/action/operation/digest/agent tuple.

internal/actionlifecycle/service.go

@@ -1262,6 +1262,79 @@ func (s *Service) RecordDispatchReceipt(orgID string, receipt unified.ActionDisp
 	return attempt, nil
 }
 
+// StrandedDispatchResult is the durable inconclusive truth for a committed
+// dispatch whose agent can no longer produce completion evidence: the receipt
+// was interrupted by an agent restart, was compacted away, or never existed
+// and never arrived inside the bounded wait. The mutation may or may not have
+// taken effect, so execution and verification are both inconclusive with no
+// evidence; the audit row becomes failed rather than staying executing
+// forever. Reconcilers use it to terminalize; it is never success truth.
+func StrandedDispatchResult(reasonCode, summary string, rollbackAvailable bool) *unified.ExecutionResult {
+	return unified.ExecutorContractViolationResult(reasonCode, summary, rollbackAvailable)
+}
+
+// ForceFailReasonCode is the canonical execution reason code recorded when an
+// operator terminalizes a wedged action through the explicit override.
+const ForceFailReasonCode = "operator_force_failed"
+
+// ErrActionForceFailNotExecuting reports that the requested override target is
+// not a dispatched, non-terminal action. Terminal rows unwrap to
+// unified.ErrActionExecutionFinal instead so callers reuse the existing
+// already-final transport mapping.
+var ErrActionForceFailNotExecuting = fmt.Errorf("%w: only a dispatched action can be force-failed", unified.ErrActionNotExecuting)
+
+// ForceFail is the operator escape hatch for an action wedged in executing:
+// the agent never returned completion evidence and reconciliation cannot
+// manufacture it. It records terminal inconclusive truth (the mutation may
+// have taken effect) attributed to the operator, so the row leaves the
+// executing state with a full audit trail instead of being edited out of band.
+// It refuses terminal actions and never touches the transport.
+func (s *Service) ForceFail(orgID, actionID, actor, reason string) (unified.ActionAuditRecord, error) {
+	actionID = strings.TrimSpace(actionID)
+	store, err := s.store(orgID)
+	if err != nil {
+		return unified.ActionAuditRecord{}, err
+	}
+	record, found, err := store.GetActionAudit(actionID)
+	if err != nil {
+		return unified.ActionAuditRecord{}, &QueryError{Op: "action audit", Err: err}
+	}
+	if !found {
+		return unified.ActionAuditRecord{}, &ActionNotFoundError{ActionID: actionID}
+	}
+	switch record.State {
+	case unified.ActionStateExecuting:
+	case unified.ActionStateCompleted, unified.ActionStateFailed, unified.ActionStateRejected, unified.ActionStateExpired:
+		return record, unified.ErrActionExecutionFinal
+	default:
+		return record, ErrActionForceFailNotExecuting
+	}
+	actor = strings.TrimSpace(actor)
+	if actor == "" {
+		actor = "api:authenticated"
+	}
+	summary := "Operator force-failed this action after it was left executing without agent completion evidence. Pulse cannot confirm whether the operation took effect; verify the resource directly before retrying."
+	if reason = strings.TrimSpace(reason); reason != "" {
+		summary += " Operator reason: " + reason
+	}
+	result := StrandedDispatchResult(ForceFailReasonCode, summary, record.Plan.RollbackAvailable)
+	failed, doneEvent, err := unified.CompleteActionExecution(record, result, actor, s.now())
+	if err != nil {
+		return unified.ActionAuditRecord{}, err
+	}
+	if err := store.RecordActionExecutionResult(failed, doneEvent); err != nil {
+		if errors.Is(err, unified.ErrActionNotExecuting) || errors.Is(err, unified.ErrActionExecutionFinal) {
+			if current, ok, queryErr := store.GetActionAudit(actionID); queryErr == nil && ok {
+				return current, unified.ErrActionExecutionFinal
+			}
+		}
+		return unified.ActionAuditRecord{}, &PersistError{Op: "action force fail", Err: err}
+	}
+	s.publishTransition(orgID, failed)
+	s.publishCompleted(failed)
+	return failed, nil
+}
+
 // RecoverExecutingActions drives restart recovery without blind re-execution.
 // Queued or pre-send expired claims may dispatch once; post-start attempts are
 // reconciled by durable attempt identity only.

internal/actionlifecycle/service_test.go

@@ -1555,3 +1555,103 @@ func TestActionPolicyProvenanceParticipatesInFreshnessWithoutAuthorizingDispatch
 		t.Fatalf("descriptive provenance authorized dispatch: %v", err)
 	}
 }
+
+// Issue1649: an executing action whose agent never returns completion evidence
+// used to have no exit at all. Expiry skips executing rows and only a terminal
+// agent receipt could complete one, so a Docker update interrupted by an agent
+// restart stayed "executing" forever with no operator route out. Reconciliation
+// now self-heals the known stranded shapes; ForceFail is the manual override
+// for anything left, and it must refuse to rewrite settled truth.
+func TestIssue1649ForceFailTerminalizesWedgedExecutingActionWithAudit(t *testing.T) {
+	store := unified.NewMemoryStore()
+	executor := &reconcilingExecutor{executeErr: context.DeadlineExceeded, found: false}
+	service := serviceForStore(t, store, testResource(time.Now().UTC(), unified.ApprovalNone), executor)
+	var completed []unified.ActionAuditRecord
+	var transitions []unified.ActionAuditRecord
+	service.OnActionCompleted = func(record unified.ActionAuditRecord) { completed = append(completed, record) }
+	service.OnActionTransition = func(_ string, record unified.ActionAuditRecord) { transitions = append(transitions, record) }
+	plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := service.Execute(context.Background(), "default", plan.ActionID, testActionActor("operator", "default"), "wedge it"); !errors.Is(err, context.DeadlineExceeded) {
+		t.Fatalf("Execute error=%v", err)
+	}
+	if current, _, err := service.Get("default", plan.ActionID); err != nil || current.State != unified.ActionStateExecuting {
+		t.Fatalf("precondition current=%#v err=%v", current, err)
+	}
+
+	failed, err := service.ForceFail("default", plan.ActionID, "operator", "verified by hand, container is running the new image")
+	if err != nil || failed.State != unified.ActionStateFailed || failed.Result == nil || failed.Result.ActionResultV2 == nil {
+		t.Fatalf("ForceFail failed=%#v err=%v", failed, err)
+	}
+	truth := failed.Result.ActionResultV2
+	if truth.Execution.Status != unified.ActionExecutionInconclusive || truth.Execution.ReasonCode != ForceFailReasonCode {
+		t.Fatalf("execution truth=%#v", truth.Execution)
+	}
+	if truth.Verification.Status != unified.ActionVerificationInconclusive || truth.Verification.EvidenceClass != unified.ActionEvidenceNone {
+		t.Fatalf("verification truth=%#v", truth.Verification)
+	}
+	if !strings.Contains(failed.Result.ErrorMessage, "verified by hand") {
+		t.Fatalf("operator reason missing from audit result: %q", failed.Result.ErrorMessage)
+	}
+	if executor.executeCalls != 1 {
+		t.Fatalf("ForceFail touched the transport: execute=%d", executor.executeCalls)
+	}
+	persisted, found, err := store.GetActionAudit(plan.ActionID)
+	if err != nil || !found || persisted.State != unified.ActionStateFailed {
+		t.Fatalf("persisted=%#v found=%v err=%v", persisted, found, err)
+	}
+	events, err := store.GetActionLifecycleEvents(plan.ActionID, time.Time{}, 100)
+	if err != nil || len(events) == 0 {
+		t.Fatalf("events=%#v err=%v", events, err)
+	}
+	terminal := events[0]
+	if terminal.State != unified.ActionStateFailed || terminal.Actor != "operator" || !strings.Contains(terminal.Message, "Operator force-failed") {
+		t.Fatalf("terminal audit event=%#v", terminal)
+	}
+	if len(completed) != 1 || completed[0].State != unified.ActionStateFailed {
+		t.Fatalf("completed=%#v", completed)
+	}
+	if len(transitions) == 0 || transitions[len(transitions)-1].State != unified.ActionStateFailed {
+		t.Fatalf("transitions=%#v", transitions)
+	}
+	// The row is settled, so the recovery loop stops reconciling it and the
+	// override cannot be replayed over the persisted outcome.
+	recovered, err := service.RecoverExecutingActions(context.Background(), "default", "system:restart-recovery", 100)
+	if err != nil || len(recovered) != 0 {
+		t.Fatalf("recovered=%#v err=%v", recovered, err)
+	}
+	if _, err := service.ForceFail("default", plan.ActionID, "operator", "again"); !errors.Is(err, unified.ErrActionExecutionFinal) {
+		t.Fatalf("replayed ForceFail error=%v", err)
+	}
+}
+
+func TestIssue1649ForceFailRefusesTerminalAndUndispatchedActions(t *testing.T) {
+	store := unified.NewMemoryStore()
+	service := serviceForStore(t, store, testResource(time.Now().UTC(), unified.ApprovalNone), &stubExecutor{result: &unified.ExecutionResult{Success: true, Output: "restarted"}})
+	plan, err := service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	// Planned but never dispatched: expiry still owns this row, so the
+	// override must not manufacture an unknown-effect failure for it.
+	if _, err := service.ForceFail("default", plan.ActionID, "operator", ""); !errors.Is(err, unified.ErrActionNotExecuting) {
+		t.Fatalf("undispatched ForceFail error=%v", err)
+	}
+	current, err := service.Execute(context.Background(), "default", plan.ActionID, testActionActor("operator", "default"), "run it")
+	if err != nil || current.State != unified.ActionStateCompleted {
+		t.Fatalf("current=%#v err=%v", current, err)
+	}
+	if _, err := service.ForceFail("default", plan.ActionID, "operator", ""); !errors.Is(err, unified.ErrActionExecutionFinal) {
+		t.Fatalf("completed ForceFail error=%v", err)
+	}
+	settled, found, err := store.GetActionAudit(plan.ActionID)
+	if err != nil || !found || settled.State != unified.ActionStateCompleted || !settled.Result.Success {
+		t.Fatalf("refused override rewrote settled truth: %#v found=%v err=%v", settled, found, err)
+	}
+	var notFound *ActionNotFoundError
+	if _, err := service.ForceFail("default", "act_missing", "operator", ""); !errors.As(err, &notFound) {
+		t.Fatalf("missing action ForceFail error=%v", err)
+	}
+}

internal/api/action_executor.go

@@ -4,12 +4,98 @@ import (
 	"context"
 	"fmt"
 	"strings"
+	"time"
 
 	"github.com/rcourtman/pulse-go-rewrite/internal/actionlifecycle"
 	"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
+	"github.com/rcourtman/pulse-go-rewrite/internal/operationreceipt"
 	unified "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
 )
 
+// strandedActionDispatchWindow bounds how long a committed dispatch may sit
+// without any durable completion evidence before Pulse stops waiting and
+// terminalizes the audit row. It is deliberately the same threshold the
+// pulse-intelligence telemetry uses to call an executing action stuck, so an
+// action can never be reported stuck while reconciliation is still waiting on
+// it. Every typed operation timeout is far shorter than this window, so an
+// in-flight mutation is never cut short by it.
+const strandedActionDispatchWindow = pulseIntelligenceStuckExecutingThreshold
+
+// strandedActionDispatchClassifiable reports whether a non-terminal receipt
+// query carries enough protocol meaning to be reasoned about at all. An
+// unrecognized status is not evidence of anything, so the attempt stays
+// receipt_pending rather than being validated or terminalized.
+func strandedActionDispatchClassifiable(status operationreceipt.QueryStatus) bool {
+	return status == operationreceipt.QueryNotFound || status == operationreceipt.QueryFoundInterrupted
+}
+
+// strandedActionDispatch converts an authenticated, correlated non-terminal
+// receipt query into terminal audit truth when the agent can no longer produce
+// completion evidence for the committed attempt. It returns found=false while
+// evidence may still arrive, so the deliberate receipt_pending semantics stay:
+// a transport error never reaches here, and a fresh missing receipt keeps
+// waiting. Callers must have validated the query against the attempt identity
+// and must not call it for a terminal receipt, which carries real result truth.
+func strandedActionDispatch(query operationreceipt.QueryResult, record unified.ActionAuditRecord, attempt unified.ActionDispatchAttempt, now time.Time) (*unified.ExecutionResult, unified.ActionDispatchReceipt, bool) {
+	reasonCode, summary, stranded := strandedActionDispatchReason(query, attempt, now)
+	if !stranded {
+		return nil, unified.ActionDispatchReceipt{}, false
+	}
+	result := actionlifecycle.StrandedDispatchResult(reasonCode, summary, record.Plan.RollbackAvailable)
+	receipt := unified.ActionDispatchReceipt{
+		AttemptID: attempt.ID, ActionID: record.ID, TransportRequestID: attempt.ID, ReceivedAt: now.UTC(),
+	}
+	return result, receipt, true
+}
+
+// strandedActionDispatchReason classifies a non-terminal receipt query. An
+// interrupted or tombstoned receipt is unrecoverable the moment it is read:
+// the agent-side store only marks a receipt interrupted when the agent
+// restarted after admitting the operation, and no interrupted or tombstoned
+// receipt can ever become terminal again. A receipt that is merely still
+// accepted or started, or one the agent has no record of at all, may still be
+// completed, so those only strand once the bounded window has elapsed.
+func strandedActionDispatchReason(query operationreceipt.QueryResult, attempt unified.ActionDispatchAttempt, now time.Time) (string, string, bool) {
+	switch query.Status {
+	case operationreceipt.QueryNotFound:
+		if !strandedActionDispatchWindowElapsed(attempt, now) {
+			return "", "", false
+		}
+		return "dispatch_evidence_missing", "The Pulse agent has no record of this operation and no completion evidence arrived within the reconciliation window, so Pulse cannot confirm whether it took effect. Check the resource directly before retrying.", true
+	case operationreceipt.QueryFoundInterrupted:
+		if query.Record == nil {
+			return "", "", false
+		}
+		switch query.Record.State {
+		case operationreceipt.StateInterrupted:
+			return "dispatch_evidence_interrupted", "The Pulse agent restarted while this operation was running and kept no durable completion evidence, so Pulse cannot confirm whether it took effect. Check the resource directly before retrying.", true
+		case operationreceipt.StateTombstone:
+			return "dispatch_evidence_discarded", "The Pulse agent discarded this operation's durable completion evidence before Pulse could read it, so Pulse cannot confirm whether it took effect. Check the resource directly before retrying.", true
+		default:
+			if !strandedActionDispatchWindowElapsed(attempt, now) {
+				return "", "", false
+			}
+			return "dispatch_evidence_incomplete", "The Pulse agent still reports this operation as unfinished long after it was dispatched, so Pulse cannot confirm whether it took effect. Check the resource directly before retrying.", true
+		}
+	default:
+		return "", "", false
+	}
+}
+
+// strandedActionDispatchWindowElapsed measures from the last durable transport
+// transition on the attempt. An attempt with no usable timestamp never strands
+// on its own; the operator override remains the escape hatch for it.
+func strandedActionDispatchWindowElapsed(attempt unified.ActionDispatchAttempt, now time.Time) bool {
+	since := attempt.UpdatedAt
+	if since.Before(attempt.CreatedAt) {
+		since = attempt.CreatedAt
+	}
+	if since.IsZero() || now.IsZero() {
+		return false
+	}
+	return !now.UTC().Before(since.UTC().Add(strandedActionDispatchWindow))
+}
+
 type actionAgentCommander interface {
 	ExecuteCommand(ctx context.Context, agentID string, cmd agentexec.ExecuteCommandPayload) (*agentexec.CommandResultPayload, error)
 	GetAgentForHost(hostname string) (string, bool)

internal/api/actions.go

@@ -20,6 +20,7 @@ import (
 const maxActionPlanRequestBytes = 1 << 20
 const maxActionDecisionRequestBytes = 64 << 10
 const maxActionExecutionRequestBytes = 64 << 10
+const maxActionForceFailRequestBytes = 64 << 10
 const maxPendingActionAudits = 100
 
 // ActionExecutor is the API-facing name for the canonical action lifecycle
@@ -63,6 +64,12 @@ type actionExecutionRequest struct {
 	PlanHash string `json:"planHash,omitempty"`
 }
 
+// actionForceFailRequest carries the operator's justification for the
+// override. It is optional and is recorded verbatim in the terminal result.
+type actionForceFailRequest struct {
+	Reason string `json:"reason,omitempty"`
+}
+
 type actionExecutionResponse struct {
 	ActionID string                    `json:"actionId"`
 	State    unified.ActionState       `json:"state"`
@@ -546,6 +553,80 @@ func (h *ResourceHandlers) HandleExecuteAction(w http.ResponseWriter, r *http.Re
 	}
 }
 
+// HandleForceFailAction is the operator escape hatch for an action wedged in
+// executing because its agent never returned durable completion evidence.
+// Reconciliation self-heals the known stranded shapes on its own; this exists
+// for the residue an operator has verified by hand. It only terminalizes the
+// audit row — it never sends, resends, or cancels anything on the transport —
+// and it refuses actions that are already final.
+func (h *ResourceHandlers) HandleForceFailAction(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost {
+		http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
+		return
+	}
+	if mock.IsMockEnabled() {
+		writeJSONError(w, http.StatusForbidden, agentcapabilities.AgentErrCodeMockModeEnabled, "Cannot force-fail actions in mock mode")
+		return
+	}
+
+	actionID := strings.TrimSpace(r.PathValue("id"))
+	if actionID == "" {
+		writeJSONError(w, http.StatusBadRequest, agentcapabilities.AgentErrCodeMissingID, "Missing action ID")
+		return
+	}
+	if !validAuditEventID.MatchString(actionID) || len(actionID) > 128 {
+		writeJSONError(w, http.StatusBadRequest, agentcapabilities.AgentErrCodeInvalidID, "Invalid action ID format")
+		return
+	}
+
+	var override actionForceFailRequest
+	decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxActionForceFailRequestBytes))
+	decoder.DisallowUnknownFields()
+	if err := decoder.Decode(&override); err != nil {
+		if !errors.Is(err, io.EOF) {
+			writeJSONErrorWithDetails(w, http.StatusBadRequest, agentcapabilities.AgentErrCodeInvalidActionExecution, "Invalid action force-fail request", map[string]string{
+				"body": "request body must be a valid action force-fail JSON object",
+			})
+			return
+		}
+	} else if err := decoder.Decode(&struct{}{}); err != io.EOF {
+		writeJSONErrorWithDetails(w, http.StatusBadRequest, agentcapabilities.AgentErrCodeInvalidActionExecution, "Invalid action force-fail request", map[string]string{
+			"body": "request body must contain one JSON object",
+		})
+		return
+	}
+	override.Reason = strings.TrimSpace(override.Reason)
+
+	orgID := GetOrgID(r.Context())
+	actor, err := actionActorForRequest(h.cfg, r, orgID)
+	if err != nil {
+		writeJSONError(w, http.StatusForbidden, agentcapabilities.AgentErrCodeActionActorUnavailable, "Authenticated action actor is unavailable")
+		return
+	}
+	failed, err := h.ActionLifecycle().ForceFail(orgID, actionID, actor.SubjectID, override.Reason)
+	if err != nil {
+		writeActionLifecycleReadError(w, err, func() {
+			var persist *actionlifecycle.PersistError
+			if errors.As(err, &persist) {
+				writeJSONError(w, http.StatusInternalServerError, "action_force_fail_persist_failed", sanitizeErrorForClient(err, "Failed to persist action force-fail"))
+				return
+			}
+			writeActionExecutionApplyError(w, err)
+		})
+		return
+	}
+
+	w.Header().Set("Content-Type", "application/json")
+	if err := json.NewEncoder(w).Encode(actionExecutionResponse{
+		ActionID: failed.ID,
+		State:    failed.State,
+		Result:   failed.Result,
+		Audit:    failed,
+	}); err != nil {
+		writeJSONError(w, http.StatusInternalServerError, "action_force_fail_encode_failed", "Failed to encode action force-fail")
+	}
+}
+
 func mockActionAuditsByState(limit int, states ...unified.ActionState) []unified.ActionAuditRecord {
 	allowed := make(map[unified.ActionState]struct{}, len(states))
 	for _, state := range states {

internal/api/actions_test.go

@@ -1287,3 +1287,103 @@ func TestHandlePlanActionRejectsMissingCapability(t *testing.T) {
 		t.Fatalf("unexpected response body: %s", rec.Body.String())
 	}
 }
+
+// Issue1649: the transport surface for the operator override. A dispatch that
+// never came back leaves the audit row executing with nothing able to settle
+// it, so POST /api/actions/{id}/force-fail terminalizes it — and refuses once
+// the row is already final so a stale tab cannot overwrite real truth.
+func TestIssue1649HandleForceFailActionSettlesWedgedExecutingAction(t *testing.T) {
+	now := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	h := newActionTestResourceHandlers(t, &config.Config{DataPath: t.TempDir()})
+	h.SetStateProvider(resourceUnifiedSeedProvider{
+		snapshot: models.StateSnapshot{LastUpdate: now},
+		resources: []unified.Resource{
+			{
+				ID: "vm:42", Type: unified.ResourceTypeVM, Name: "web-42", Status: unified.StatusWarning,
+				LastSeen: now, UpdatedAt: now, Sources: []unified.DataSource{unified.SourceProxmox},
+				Capabilities: []unified.ResourceCapability{
+					{
+						Name: "restart", Type: unified.CapabilityTypeCommon, Description: "Restart the VM",
+						MinimumApprovalLevel: unified.ApprovalNone, InternalHandler: "proxmox.vm.restart",
+					},
+				},
+			},
+		},
+	})
+	// The executor answers with a transport failure, which deliberately keeps
+	// the dispatch receipt-pending and the audit row executing.
+	h.SetActionExecutor(&stubActionExecutor{err: errors.New("agent connection dropped mid-dispatch")})
+
+	planRec := httptest.NewRecorder()
+	planReq := httptest.NewRequest(http.MethodPost, "/api/actions/plan", bytes.NewBufferString(`{
+		"requestId":"issue1649-force-fail",
+		"resourceId":"vm:42",
+		"capabilityName":"restart",
+		"reason":"Recover after confirmed outage"
+	}`))
+	h.HandlePlanAction(planRec, actionHandlerTestRequest(planReq, ""))
+	if planRec.Code != http.StatusOK {
+		t.Fatalf("plan status = %d, body=%s", planRec.Code, planRec.Body.String())
+	}
+	var plan unified.ActionPlan
+	if err := json.Unmarshal(planRec.Body.Bytes(), &plan); err != nil {
+		t.Fatal(err)
+	}
+
+	executeRec := httptest.NewRecorder()
+	executeReq := httptest.NewRequest(http.MethodPost, "/api/actions/"+plan.ActionID+"/execute", bytes.NewBufferString(`{"reason":"run it"}`))
+	executeReq.SetPathValue("id", plan.ActionID)
+	h.HandleExecuteAction(executeRec, actionHandlerTestRequest(executeReq, ""))
+	if executeRec.Code != http.StatusInternalServerError {
+		t.Fatalf("execute status = %d, body=%s", executeRec.Code, executeRec.Body.String())
+	}
+	store, err := h.getStore("default")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if wedged, ok, err := store.GetActionAudit(plan.ActionID); err != nil || !ok || wedged.State != unified.ActionStateExecuting {
+		t.Fatalf("precondition wedged=%#v ok=%v err=%v", wedged, ok, err)
+	}
+
+	forceRec := httptest.NewRecorder()
+	forceReq := httptest.NewRequest(http.MethodPost, "/api/actions/"+plan.ActionID+"/force-fail", bytes.NewBufferString(`{"reason":"agent host was rebuilt"}`))
+	forceReq.SetPathValue("id", plan.ActionID)
+	h.HandleForceFailAction(forceRec, actionHandlerTestRequest(forceReq, ""))
+	if forceRec.Code != http.StatusOK {
+		t.Fatalf("force-fail status = %d, body=%s", forceRec.Code, forceRec.Body.String())
+	}
+	var settled actionExecutionResponse
+	if err := json.Unmarshal(forceRec.Body.Bytes(), &settled); err != nil {
+		t.Fatal(err)
+	}
+	if settled.ActionID != plan.ActionID || settled.State != unified.ActionStateFailed {
+		t.Fatalf("force-fail response = %#v", settled)
+	}
+	if settled.Result == nil || settled.Result.Success || settled.Result.ActionResultV2 == nil ||
+		settled.Result.ActionResultV2.Execution.ReasonCode != actionlifecycle.ForceFailReasonCode {
+		t.Fatalf("force-fail result = %#v", settled.Result)
+	}
+	if !strings.Contains(settled.Result.ErrorMessage, "agent host was rebuilt") {
+		t.Fatalf("operator reason missing: %q", settled.Result.ErrorMessage)
+	}
+	persisted, ok, err := store.GetActionAudit(plan.ActionID)
+	if err != nil || !ok || persisted.State != unified.ActionStateFailed {
+		t.Fatalf("persisted=%#v ok=%v err=%v", persisted, ok, err)
+	}
+
+	replayRec := httptest.NewRecorder()
+	replayReq := httptest.NewRequest(http.MethodPost, "/api/actions/"+plan.ActionID+"/force-fail", bytes.NewBufferString(`{}`))
+	replayReq.SetPathValue("id", plan.ActionID)
+	h.HandleForceFailAction(replayRec, actionHandlerTestRequest(replayReq, ""))
+	if replayRec.Code != http.StatusConflict || !strings.Contains(replayRec.Body.String(), "action_execution_final") {
+		t.Fatalf("replayed force-fail status = %d, body=%s", replayRec.Code, replayRec.Body.String())
+	}
+
+	missingRec := httptest.NewRecorder()
+	missingReq := httptest.NewRequest(http.MethodPost, "/api/actions/act_missing/force-fail", bytes.NewBufferString(`{}`))
+	missingReq.SetPathValue("id", "act_missing")
+	h.HandleForceFailAction(missingRec, actionHandlerTestRequest(missingReq, ""))
+	if missingRec.Code != http.StatusNotFound {
+		t.Fatalf("missing force-fail status = %d, body=%s", missingRec.Code, missingRec.Body.String())
+	}
+}

internal/api/contract_test.go

@@ -7,6 +7,7 @@ import (
 	"crypto/tls"
 	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"math"
 	"net/http"
@@ -41,6 +42,7 @@ import (
 	"github.com/rcourtman/pulse-go-rewrite/internal/models"
 	"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
 	"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
+	"github.com/rcourtman/pulse-go-rewrite/internal/operationreceipt"
 	"github.com/rcourtman/pulse-go-rewrite/internal/recovery"
 	"github.com/rcourtman/pulse-go-rewrite/internal/relay"
 	"github.com/rcourtman/pulse-go-rewrite/internal/telemetry"
@@ -20490,6 +20492,8 @@ func TestContract_AgentSurfaceErrorCodesMatchManifestDeclarations(t *testing.T)
 		"action_execution_persist_failed":                      true,
 		"action_execution_encode_failed":                       true,
 		"action_execution_failed":                              true,
+		"action_force_fail_persist_failed":                     true,
+		"action_force_fail_encode_failed":                      true,
 		"action_queue_unavailable":                             true,
 		"action_queue_query_failed":                            true,
 		"action_queue_encode_failed":                           true,
@@ -22117,3 +22121,191 @@ func TestContract_SSOProviderResponseBaseURLNeverGuessesLocalhost(t *testing.T)
 		}
 	})
 }
+
+// Issue1649 regression coverage. Docker update (and every other typed) action
+// used to sit in "executing" forever whenever the agent could not produce a
+// terminal receipt: the agent-side receipt store marks an in-flight receipt
+// interrupted on every restart, and no executor ever turned that, a missing
+// receipt, or a compacted one into terminal audit truth. Expiry deliberately
+// skips executing rows, so nothing else could clear them either. These pin the
+// three shapes that now terminalize, and the ones that must keep waiting.
+func issue1649HostUpdateReceiptQuery(status operationreceipt.QueryStatus, state operationreceipt.State, attempt unifiedresources.ActionDispatchAttempt, at time.Time) operationreceipt.QueryResult {
+	query := operationreceipt.QueryResult{Version: operationreceipt.ProtocolVersion, Status: status}
+	if status == operationreceipt.QueryNotFound {
+		return query
+	}
+	record := &operationreceipt.Record{
+		Identity: operationreceipt.Identity{
+			AttemptID: attempt.ID, ActionID: attempt.ActionID, OperationKind: attempt.OperationKind,
+			OperationVersion: attempt.OperationVersion, RequestDigest: attempt.RequestDigest, AgentID: attempt.AgentID,
+		},
+		State:      state,
+		AcceptedAt: at.Add(-2 * time.Second),
+	}
+	if state != operationreceipt.StateAccepted {
+		record.StartedAt = at.Add(-time.Second)
+	}
+	if state == operationreceipt.StateTombstone {
+		record.TerminalAt = at
+	}
+	query.Record = record
+	return query
+}
+
+func TestIssue1649InterruptedReceiptTerminalizesStrandedActionImmediately(t *testing.T) {
+	dispatchedAt := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	attempt := hostUpdateDispatchAttempt(t, "action-host-update", dispatchedAt)
+	// The agent restarted one minute after dispatch: well inside the bounded
+	// wait, but the receipt can never become terminal again, so waiting is
+	// pointless and the row must terminalize now.
+	receivedAt := dispatchedAt.Add(time.Minute)
+	agents := &fakeHostUpdateAgent{queryResult: issue1649HostUpdateReceiptQuery(operationreceipt.QueryFoundInterrupted, operationreceipt.StateInterrupted, attempt, dispatchedAt)}
+	executor := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return receivedAt }}
+
+	result, receipt, found, err := executor.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt)
+	if err != nil || !found || result == nil || result.ActionResultV2 == nil {
+		t.Fatalf("reconcile result=%#v found=%v err=%v", result, found, err)
+	}
+	if result.Success {
+		t.Fatalf("stranded dispatch reported success: %#v", result)
+	}
+	truth := result.ActionResultV2
+	if truth.Execution.Status != unifiedresources.ActionExecutionInconclusive || truth.Execution.ReasonCode != "dispatch_evidence_interrupted" {
+		t.Fatalf("execution truth=%#v", truth.Execution)
+	}
+	if truth.Verification.Status != unifiedresources.ActionVerificationInconclusive || truth.Verification.EvidenceClass != unifiedresources.ActionEvidenceNone {
+		t.Fatalf("verification truth=%#v", truth.Verification)
+	}
+	if !strings.Contains(result.ErrorMessage, "restarted") || !strings.Contains(result.ErrorMessage, "Check the resource directly") {
+		t.Fatalf("operator-facing message=%q", result.ErrorMessage)
+	}
+	if receipt.AttemptID != attempt.ID || receipt.ActionID != attempt.ActionID || !receipt.ReceivedAt.Equal(receivedAt) {
+		t.Fatalf("receipt=%#v", receipt)
+	}
+	if len(agents.requests) != 0 {
+		t.Fatalf("terminalizing a stranded dispatch resent the mutation: %#v", agents.requests)
+	}
+
+	// The completion is real terminal audit truth, so the wedged row leaves
+	// executing instead of being reconciled forever.
+	completed, event, err := unifiedresources.CompleteActionExecution(hostUpdateActionRecord(attempt.ActionID), result, "system:restart-recovery", receivedAt)
+	if err != nil || completed.State != unifiedresources.ActionStateFailed || event.State != unifiedresources.ActionStateFailed {
+		t.Fatalf("completed=%#v event=%#v err=%v", completed, event, err)
+	}
+}
+
+func TestIssue1649DiscardedReceiptTerminalizesStrandedActionImmediately(t *testing.T) {
+	dispatchedAt := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	attempt := hostUpdateDispatchAttempt(t, "action-host-update", dispatchedAt)
+	receivedAt := dispatchedAt.Add(time.Minute)
+	agents := &fakeHostUpdateAgent{queryResult: issue1649HostUpdateReceiptQuery(operationreceipt.QueryFoundInterrupted, operationreceipt.StateTombstone, attempt, dispatchedAt)}
+	executor := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return receivedAt }}
+
+	result, _, found, err := executor.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt)
+	if err != nil || !found || result == nil || result.ActionResultV2 == nil {
+		t.Fatalf("reconcile result=%#v found=%v err=%v", result, found, err)
+	}
+	if result.Success || result.ActionResultV2.Execution.ReasonCode != "dispatch_evidence_discarded" {
+		t.Fatalf("execution truth=%#v", result.ActionResultV2.Execution)
+	}
+}
+
+func TestIssue1649MissingReceiptWaitsInsideTheBoundedWindow(t *testing.T) {
+	dispatchedAt := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	attempt := hostUpdateDispatchAttempt(t, "action-host-update", dispatchedAt)
+	agents := &fakeHostUpdateAgent{queryResult: issue1649HostUpdateReceiptQuery(operationreceipt.QueryNotFound, "", attempt, dispatchedAt)}
+	// A receipt that has not been written yet is not evidence of anything, so
+	// a fresh not-found keeps the attempt receipt_pending.
+	executor := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return dispatchedAt.Add(strandedActionDispatchWindow - time.Minute) }}
+
+	result, _, found, err := executor.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt)
+	if err != nil || found || result != nil {
+		t.Fatalf("fresh not-found terminalized early: result=%#v found=%v err=%v", result, found, err)
+	}
+}
+
+func TestIssue1649MissingReceiptTerminalizesAfterTheBoundedWindow(t *testing.T) {
+	dispatchedAt := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	attempt := hostUpdateDispatchAttempt(t, "action-host-update", dispatchedAt)
+	agents := &fakeHostUpdateAgent{queryResult: issue1649HostUpdateReceiptQuery(operationreceipt.QueryNotFound, "", attempt, dispatchedAt)}
+	executor := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return dispatchedAt.Add(strandedActionDispatchWindow) }}
+
+	result, receipt, found, err := executor.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt)
+	if err != nil || !found || result == nil || result.ActionResultV2 == nil {
+		t.Fatalf("aged not-found result=%#v found=%v err=%v", result, found, err)
+	}
+	if result.Success || result.ActionResultV2.Execution.ReasonCode != "dispatch_evidence_missing" {
+		t.Fatalf("execution truth=%#v", result.ActionResultV2.Execution)
+	}
+	if receipt.AttemptID != attempt.ID || len(agents.requests) != 0 {
+		t.Fatalf("receipt=%#v resent=%#v", receipt, agents.requests)
+	}
+}
+
+func TestIssue1649UnfinishedReceiptWaitsThenTerminalizes(t *testing.T) {
+	dispatchedAt := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	attempt := hostUpdateDispatchAttempt(t, "action-host-update", dispatchedAt)
+	query := issue1649HostUpdateReceiptQuery(operationreceipt.QueryFoundInterrupted, operationreceipt.StateStarted, attempt, dispatchedAt)
+	agents := &fakeHostUpdateAgent{queryResult: query}
+
+	// A started receipt on a live agent means the operation may still be
+	// running (an image pull can be slow), so it must not be cut short.
+	inFlight := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return dispatchedAt.Add(time.Minute) }}
+	if result, _, found, err := inFlight.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt); err != nil || found || result != nil {
+		t.Fatalf("in-flight operation terminalized: result=%#v found=%v err=%v", result, found, err)
+	}
+
+	aged := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return dispatchedAt.Add(strandedActionDispatchWindow) }}
+	result, _, found, err := aged.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt)
+	if err != nil || !found || result == nil || result.ActionResultV2 == nil {
+		t.Fatalf("aged unfinished result=%#v found=%v err=%v", result, found, err)
+	}
+	if result.Success || result.ActionResultV2.Execution.ReasonCode != "dispatch_evidence_incomplete" {
+		t.Fatalf("execution truth=%#v", result.ActionResultV2.Execution)
+	}
+}
+
+func TestIssue1649TransportErrorNeverTerminalizesTheAction(t *testing.T) {
+	dispatchedAt := time.Date(2026, 7, 29, 9, 0, 0, 0, time.UTC)
+	attempt := hostUpdateDispatchAttempt(t, "action-host-update", dispatchedAt)
+	transportErr := errors.New("agent command channel is not connected")
+	agents := &fakeHostUpdateAgent{queryErr: transportErr}
+	// Long past the bounded window: an unanswered query is still not evidence,
+	// so receipt_pending semantics survive and nothing is terminalized.
+	executor := hostUpdateActionExecutor{agents: agents, now: func() time.Time { return dispatchedAt.Add(48 * time.Hour) }}
+
+	result, _, found, err := executor.ReconcileActionDispatch(context.Background(), hostUpdateActionRecord(attempt.ActionID), attempt)
+	if !errors.Is(err, transportErr) || found || result != nil {
+		t.Fatalf("transport error result=%#v found=%v err=%v", result, found, err)
+	}
+}
+
+func TestIssue1649StrandedDockerUpdateReconcilesWithoutRedispatch(t *testing.T) {
+	now := time.Now().UTC()
+	resource := dockerContainerUpdateActionResource("app-container:api", "docker", now)
+	h := newActionTestResourceHandlers(t, &config.Config{DataPath: t.TempDir()})
+	h.SetStateProvider(resourceUnifiedSeedProvider{snapshot: models.StateSnapshot{LastUpdate: now}, resources: []unifiedresources.Resource{resource}})
+	agents := &fakeDockerActionAgentCommander{}
+	executor := newDockerContainerActionExecutor(h, agents).(dockerContainerActionExecutor)
+	record := dockerContainerActionRecord("act_container", resource.ID, "update")
+	attempt, err := unifiedresources.NewActionDispatchAttempt(record.ID, now.Add(-30*time.Minute))
+	if err != nil {
+		t.Fatal(err)
+	}
+	attempt, err = executor.BindActionDispatch(context.Background(), record, attempt)
+	if err != nil {
+		t.Fatal(err)
+	}
+	agents.queryResult = issue1649HostUpdateReceiptQuery(operationreceipt.QueryFoundInterrupted, operationreceipt.StateInterrupted, attempt, now.Add(-30*time.Minute))
+
+	result, receipt, found, err := executor.ReconcileActionDispatch(context.Background(), record, attempt)
+	if err != nil || !found || result == nil || result.ActionResultV2 == nil {
+		t.Fatalf("docker reconcile result=%#v found=%v err=%v", result, found, err)
+	}
+	if result.Success || result.ActionResultV2.Execution.ReasonCode != "dispatch_evidence_interrupted" {
+		t.Fatalf("execution truth=%#v", result.ActionResultV2.Execution)
+	}
+	if receipt.ActionID != record.ID || len(agents.typedUpdateCalls) != 0 || len(agents.typedCalls) != 0 || len(agents.calls) != 0 {
+		t.Fatalf("receipt=%#v update=%d lifecycle=%d raw=%d", receipt, len(agents.typedUpdateCalls), len(agents.typedCalls), len(agents.calls))
+	}
+}

internal/api/docker_container_action_executor.go

@@ -265,10 +265,17 @@ func (e dockerContainerActionExecutor) ReconcileActionDispatch(ctx context.Conte
 	if err != nil {
 		return nil, unified.ActionDispatchReceipt{}, false, err
 	}
-	if query.Status != operationreceipt.QueryFoundTerminal {
-		return nil, unified.ActionDispatchReceipt{}, false, nil
-	}
 	receivedAt := time.Now().UTC()
+	if query.Status != operationreceipt.QueryFoundTerminal {
+		if !strandedActionDispatchClassifiable(query.Status) {
+			return nil, unified.ActionDispatchReceipt{}, false, nil
+		}
+		if err := agentexec.ValidateOperationQueryResultForIdentity(query, identity, receivedAt); err != nil {
+			return nil, unified.ActionDispatchReceipt{}, false, err
+		}
+		result, receipt, stranded := strandedActionDispatch(query, record, attempt, receivedAt)
+		return result, receipt, stranded, nil
+	}
 	if err := agentexec.ValidateOperationQueryResultForIdentity(query, identity, receivedAt); err != nil {
 		return nil, unified.ActionDispatchReceipt{}, false, err
 	}

internal/api/host_storage_cleanup_action_executor.go

@@ -134,10 +134,17 @@ func (e hostStorageCleanupActionExecutor) ReconcileActionDispatch(ctx context.Co
 	if err != nil {
 		return nil, unified.ActionDispatchReceipt{}, false, err
 	}
-	if query.Status != operationreceipt.QueryFoundTerminal {
-		return nil, unified.ActionDispatchReceipt{}, false, nil
-	}
 	receivedAt := e.currentTime()
+	if query.Status != operationreceipt.QueryFoundTerminal {
+		if !strandedActionDispatchClassifiable(query.Status) {
+			return nil, unified.ActionDispatchReceipt{}, false, nil
+		}
+		if err := agentexec.ValidateOperationQueryResultForIdentity(query, identity, receivedAt); err != nil {
+			return nil, unified.ActionDispatchReceipt{}, false, err
+		}
+		execution, receipt, stranded := strandedActionDispatch(query, record, attempt, receivedAt)
+		return execution, receipt, stranded, nil
+	}
 	if err := agentexec.ValidateOperationQueryResultForIdentity(query, identity, receivedAt); err != nil {
 		return nil, unified.ActionDispatchReceipt{}, false, err
 	}

internal/api/host_update_action_executor.go

@@ -139,10 +139,17 @@ func (e hostUpdateActionExecutor) ReconcileActionDispatch(ctx context.Context, r
 	if err != nil {
 		return nil, unified.ActionDispatchReceipt{}, false, err
 	}
-	if query.Status != operationreceipt.QueryFoundTerminal {
-		return nil, unified.ActionDispatchReceipt{}, false, nil
-	}
 	receivedAt := e.currentTime()
+	if query.Status != operationreceipt.QueryFoundTerminal {
+		if !strandedActionDispatchClassifiable(query.Status) {
+			return nil, unified.ActionDispatchReceipt{}, false, nil
+		}
+		if err := agentexec.ValidateOperationQueryResultForIdentity(query, identity, receivedAt); err != nil {
+			return nil, unified.ActionDispatchReceipt{}, false, err
+		}
+		execution, receipt, stranded := strandedActionDispatch(query, record, attempt, receivedAt)
+		return execution, receipt, stranded, nil
+	}
 	if err := agentexec.ValidateOperationQueryResultForIdentity(query, identity, receivedAt); err != nil {
 		return nil, unified.ActionDispatchReceipt{}, false, err
 	}

internal/api/route_inventory_test.go

@@ -435,6 +435,7 @@ var allRouteAllowlist = []string{
 	"GET /api/actions/{id}",
 	"POST /api/actions/{id}/decision",
 	"POST /api/actions/{id}/execute",
+	"POST /api/actions/{id}/force-fail",
 	"/api/guests/metadata",
 	"/api/guests/metadata/",
 	"/api/docker/metadata",

internal/api/router_routes_monitoring.go

@@ -159,6 +159,14 @@ func (r *Router) registerMonitoringResourceRoutes(
 		agentcapabilities.ExecuteActionCapabilityName,
 		requireActionCapability(r.authorizer, auth.ActionExecute, r.resourceHandlers.HandleExecuteAction),
 	))))
+	// Operator override for an action wedged in executing without agent
+	// completion evidence. It is deliberately not an agent capability and not
+	// a relay-mobile route: it is a local admin repair, gated harder than
+	// execute (admin plus settings:write on top of the execute capability)
+	// because it writes terminal audit truth Pulse could not observe.
+	r.mux.HandleFunc("POST /api/actions/{id}/force-fail", RequireAdmin(r.config, RequireScope(config.ScopeSettingsWrite,
+		requireActionCapability(r.authorizer, auth.ActionExecute, r.resourceHandlers.HandleForceFailAction),
+	)))
 	// Guest metadata routes
 	r.mux.HandleFunc("/api/guests/metadata", RequireAuth(r.config, RequireScope(config.ScopeMonitoringRead, guestMetadataHandler.HandleGetMetadata)))
 	r.mux.HandleFunc("/api/guests/metadata/", RequireAuth(r.config, func(w http.ResponseWriter, req *http.Request) {