diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index ef21a6af0..d8a611308 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -888,6 +888,20 @@ the fixed `/var/cache/apt/archives` scan, bounded entry/byte limits, and sole inspection failure, command failure, and unconfirmed reclaimed bytes all fail closed. The envelope has no command, path, package selector, arbitrary argument, installed-package removal, or reboot authority. +Before a concrete Docker lifecycle/update, host update, or package-cache +cleanup action is persisted for approval, the current Unified Agent may be +asked to evaluate the exact already-bound operation through the versioned +`action_preflight` protocol. That request is read-only, carries exactly one +closed typed envelope and its action-bound request digest, and never enters the +durable operation-receipt store. The result is limited to feasibility, the +same bounded refusal-code vocabulary, the exact operation binding, and a fresh +agent timestamp. It cannot approve, admit, reserve, execute, or verify a +mutation. Current agents advertise protocol version 1 at registration; a +server with the transport but an older connected agent fails concrete +feasibility closed, while internal executors that predate the optional +transport retain their compatibility path. Dispatch still repeats all local +preconditions after durable admission because preflight evidence is not a +lease and target state may race after approval. Proxmox VM and LXC lifecycle affordances follow the same adjacent boundary: lifecycle and fleet surfaces may consume backend-advertised `start`, `shutdown`, `reboot`, and `stop` capabilities and typed `actionReadiness`, but diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 38eeee64a..0b81c1dd8 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -2305,6 +2305,17 @@ a new API state machine, queue contract, or verification-accounting field. uses that hook for command-agent connectivity and runtime posture; browser controls may consume advertised capabilities, but must not replace this check with direct shell, SSH, provider, or agent-command calls. + Concrete action feasibility is a separate lifecycle gate from cheap + capability availability. `AvailabilityChecker` remains safe for bulk + `/api/resources` projection; the optional `FeasibilityChecker` runs only + for a specific action during planning and again before approval/dispatch, + where it may ask the current Unified Agent to perform the versioned, + read-only `action_preflight` for the exact action-bound operation digest. + An infeasible, stale, mismatched, timed-out, or unsupported agent result + returns the same `409 action_execution_unavailable` contract and cannot + persist an approval or enter `executing`. The preflight result is readiness + evidence only, never approval, dispatch authority, durable receipt, + execution success, or verification. Docker / Podman lifecycle execution resolves the command WebSocket by the Docker reporting agent ID first, then by canonical Docker host name when the runtime source ID differs from the command-agent registration ID. Once @@ -4063,16 +4074,19 @@ count only non-integration-backed hosts as registered agents. `POST /api/actions/{actionId}/execute` routes through the transport-independent Actions lifecycle, which revalidates the approved plan against the current canonical resource and then asks the optional executor-owned -`AvailabilityChecker` for live readiness before entering `executing`, creating -a dispatch attempt, or calling the executor. The same gate applies to +`AvailabilityChecker` for cheap live readiness and the optional +`FeasibilityChecker` for exact agent-local preflight before entering +`executing`, creating a dispatch attempt, or calling the executor. The same gates apply to `ExecuteUnderPolicy`, so an automatic broker cannot bypass it. A resource that disappears remains `action_plan_drift`; an explicitly unavailable capability returns HTTP `409` with shared code `action_execution_unavailable` and bounded `resourceId`, `capabilityName`, `reasonCode`, and `reason` details. Pulse persists a terminal failed/no-effect audit and lifecycle event and publishes -the normal completion notification. Executors without the optional checker, +the normal completion notification. Executors without either optional checker, and checkers returning an empty readiness result, preserve the existing -compatibility path. Registry or readiness-check infrastructure failures remain +compatibility path. A current executor whose agent supports the feasibility +transport fails closed when the connected agent cannot answer it. Registry or +readiness-check infrastructure failures remain nonterminal internal errors rather than false permanent refusals. The public Patrol investigation boundary now carries independent diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index ca4bb21bf..a2f369217 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -678,6 +678,12 @@ fingerprint-bound `install_os_updates` operation after canonical lifecycle approval. Storage/recovery consumers may observe its redacted audit outcome and reboot-required fact as context, but must not treat package installation as storage maintenance, recovery evidence, or a storage-owned mutation path. +The adjacent `action_preflight` request for a package update or package-cache +cleanup is API/action-lifecycle and agent-transport readiness evidence only. +Storage/recovery may display its bounded refusal reason but must not treat that +read-only exact-digest check as storage repair, cleanup execution, recovery +proof, a reservation on package-manager state, or authority to bypass the +second precondition check at durable dispatch. The API-owned `internal/api/host_storage_cleanup_action_executor.go` is a narrow exception only in product purpose, not ownership: it may reclaim the fixed APT package cache through the canonical action lifecycle when the cache's diff --git a/internal/actionlifecycle/service.go b/internal/actionlifecycle/service.go index 6a7bf27bc..28e2a2bbd 100644 --- a/internal/actionlifecycle/service.go +++ b/internal/actionlifecycle/service.go @@ -46,6 +46,14 @@ type AvailabilityChecker interface { CheckActionAvailable(ctx context.Context, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness } +// FeasibilityChecker is the trust-critical, operation-specific counterpart to +// AvailabilityChecker. Availability is cheap enough for resource projection; +// feasibility may perform bounded remote reads and is called only while +// planning, approving, or dispatching a concrete action. +type FeasibilityChecker interface { + CheckActionFeasible(ctx context.Context, actionID string, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness +} + // RefreshPlanner reconstructs broker-owned planning inputs for a replacement // plan. Public/operator actions use the lifecycle default; first-party // brokers use this hook to re-evaluate current policy factors without letting @@ -474,6 +482,13 @@ func (s *Service) PlanWithOptions(ctx context.Context, orgID string, req unified } } } + if checker, ok := s.Executor.(FeasibilityChecker); ok { + if readiness := checker.CheckActionFeasible(ctx, plan.ActionID, req, *resource); readiness.Name != "" && !readiness.Available { + return unified.ActionPlan{}, &AvailabilityRefusedError{ + ResourceID: req.ResourceID, CapabilityName: req.CapabilityName, Readiness: readiness, + } + } + } store, err := s.store(orgID) if err != nil { @@ -1600,8 +1615,9 @@ const ( // the optional executor-owned readiness checker immediately before admission. // An absent checker or an empty readiness result preserves compatibility. func (s *Service) ValidateExecutionAvailable(ctx context.Context, orgID string, record unified.ActionAuditRecord) error { - checker, ok := s.Executor.(AvailabilityChecker) - if !ok { + availability, hasAvailability := s.Executor.(AvailabilityChecker) + feasibility, hasFeasibility := s.Executor.(FeasibilityChecker) + if !hasAvailability && !hasFeasibility { return nil } normalized, err := unified.NormalizeActionAuditRecord(record) @@ -1616,18 +1632,30 @@ func (s *Service) ValidateExecutionAvailable(ctx context.Context, orgID string, if !ok || resource == nil { return fmt.Errorf("%w: resource %q is no longer present", unified.ErrActionPlanDrift, normalized.Request.ResourceID) } - readiness := checker.CheckActionAvailable(ctx, normalized.Request, *resource) + readiness := unified.ResourceActionReadiness{} + if hasAvailability { + readiness = availability.CheckActionAvailable(ctx, normalized.Request, *resource) + } readiness.Name = strings.TrimSpace(readiness.Name) readiness.ReasonCode = boundedActionAvailabilityText(readiness.ReasonCode, actionAvailabilityReasonCodeMaxRunes) readiness.Reason = boundedActionAvailabilityText(readiness.Reason, actionAvailabilityReasonMaxRunes) - if readiness.Name == "" || readiness.Available { - return nil + if readiness.Name != "" && !readiness.Available { + return &AvailabilityRefusedError{ + ResourceID: normalized.Request.ResourceID, CapabilityName: normalized.Request.CapabilityName, Readiness: readiness, + } } - return &AvailabilityRefusedError{ - ResourceID: normalized.Request.ResourceID, - CapabilityName: normalized.Request.CapabilityName, - Readiness: readiness, + if hasFeasibility { + readiness = feasibility.CheckActionFeasible(ctx, normalized.ID, normalized.Request, *resource) + readiness.Name = strings.TrimSpace(readiness.Name) + readiness.ReasonCode = boundedActionAvailabilityText(readiness.ReasonCode, actionAvailabilityReasonCodeMaxRunes) + readiness.Reason = boundedActionAvailabilityText(readiness.Reason, actionAvailabilityReasonMaxRunes) + if readiness.Name != "" && !readiness.Available { + return &AvailabilityRefusedError{ + ResourceID: normalized.Request.ResourceID, CapabilityName: normalized.Request.CapabilityName, Readiness: readiness, + } + } } + return nil } func boundedActionAvailabilityText(value string, maxRunes int) string { diff --git a/internal/actionlifecycle/service_test.go b/internal/actionlifecycle/service_test.go index ecfec008d..7c9e122b0 100644 --- a/internal/actionlifecycle/service_test.go +++ b/internal/actionlifecycle/service_test.go @@ -883,11 +883,12 @@ func TestQueuedPolicyActionRevalidatesAfterSQLiteRestart(t *testing.T) { } type stubExecutor struct { - result *unified.ExecutionResult - err error - calls int - received unified.ActionAuditRecord - readiness *unified.ResourceActionReadiness + result *unified.ExecutionResult + err error + calls int + received unified.ActionAuditRecord + readiness *unified.ResourceActionReadiness + feasibility *unified.ResourceActionReadiness } type reconcilingExecutor struct { @@ -971,6 +972,13 @@ func (s *stubExecutor) CheckActionAvailable(_ context.Context, _ unified.ActionR return *s.readiness } +func (s *stubExecutor) CheckActionFeasible(_ context.Context, _ string, _ unified.ActionRequest, _ unified.Resource) unified.ResourceActionReadiness { + if s.feasibility == nil { + return unified.ResourceActionReadiness{} + } + return *s.feasibility +} + type serviceEnv struct { store unified.ResourceStore registry *unified.ResourceRegistry @@ -1179,6 +1187,24 @@ func TestPlanAvailabilityRefusalPersistsNothing(t *testing.T) { } } +func TestPlanAgentFeasibilityRefusalPersistsNothing(t *testing.T) { + now := time.Now().UTC() + env := newServiceEnv(t, testResource(now, unified.ApprovalAdmin)) + env.executor.feasibility = &unified.ResourceActionReadiness{ + Name: "restart", Available: false, ReasonCode: "target_state_changed", + Reason: "The target changed before approval.", + } + _, err := env.service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default")) + var refused *AvailabilityRefusedError + if !errors.As(err, &refused) || refused.Readiness.ReasonCode != "target_state_changed" { + t.Fatalf("error=%v refused=%#v", err, refused) + } + audits, queryErr := env.store.GetActionAudits("vm:42", time.Time{}, 10) + if queryErr != nil || len(audits) != 0 { + t.Fatalf("audits=%#v err=%v", audits, queryErr) + } +} + func TestDecideApprovesPendingAction(t *testing.T) { now := time.Now().UTC() env := newServiceEnv(t, testResource(now, unified.ApprovalAdmin)) @@ -1247,6 +1273,27 @@ func TestApprovalRequiresCurrentReadinessButRejectionRemainsAvailable(t *testing } } +func TestApprovalRechecksAgentFeasibilityBeforePersistingDecision(t *testing.T) { + now := time.Now().UTC() + env := newServiceEnv(t, testResource(now, unified.ApprovalAdmin)) + plan, err := env.service.Plan(context.Background(), "default", restartRequest(), testActionActor("requester", "default")) + if err != nil { + t.Fatal(err) + } + env.executor.feasibility = &unified.ResourceActionReadiness{ + Name: "restart", Available: false, ReasonCode: "target_state_changed", + Reason: "The target changed after planning.", + } + decision := testActionDecision(t, env.service, "default", plan.ActionID, unified.ActionApprovalRecord{Actor: "operator", Outcome: unified.OutcomeApproved}) + if _, err := env.service.Decide(context.Background(), "default", plan.ActionID, decision); !errors.Is(err, unified.ErrActionExecutionUnavailable) { + t.Fatalf("approval error=%v", err) + } + current := mustActionRecord(t, env.service, plan.ActionID) + if current.State != unified.ActionStatePending || len(current.Approvals) != 0 { + t.Fatalf("refused approval mutated record: %#v", current) + } +} + func TestRefreshReplacesExpiredPlanAndPreservesTrustedOrigin(t *testing.T) { now := time.Now().UTC() env := newServiceEnv(t, testResource(now, unified.ApprovalAdmin)) diff --git a/internal/agentexec/action_preflight.go b/internal/agentexec/action_preflight.go new file mode 100644 index 000000000..c14a2ac5e --- /dev/null +++ b/internal/agentexec/action_preflight.go @@ -0,0 +1,204 @@ +package agentexec + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "strings" + "time" +) + +const ActionPreflightProtocolVersion = 1 + +// ActionPreflightError lets an in-process typed module preserve a safe refusal +// category without making callers parse provider error text. Err remains local +// and is never serialized by the action preflight protocol. +type ActionPreflightError struct { + ReasonCode string + Err error +} + +func (e *ActionPreflightError) Error() string { + if e == nil || e.Err == nil { + return "action preflight refused" + } + return e.Err.Error() +} + +func (e *ActionPreflightError) Unwrap() error { + if e == nil { + return nil + } + return e.Err +} + +func NewActionPreflightError(reasonCode string, err error) error { + reasonCode = strings.TrimSpace(reasonCode) + if !IsActionRefusalReasonCode(reasonCode) { + reasonCode = ActionRefusalTargetPreconditionFailed + } + return &ActionPreflightError{ReasonCode: reasonCode, Err: err} +} + +func ActionPreflightReasonCode(err error, fallback string) string { + var refusal *ActionPreflightError + if errors.As(err, &refusal) && IsActionRefusalReasonCode(refusal.ReasonCode) { + return refusal.ReasonCode + } + if IsActionRefusalReasonCode(fallback) { + return fallback + } + return ActionRefusalTargetPreconditionFailed +} + +func DecodeActionPreflightPayload(data []byte) (ActionPreflightPayload, error) { + var payload ActionPreflightPayload + if err := decodeStrictActionPreflight(data, &payload); err != nil { + return ActionPreflightPayload{}, err + } + if err := ValidateActionPreflightPayload(&payload); err != nil { + return ActionPreflightPayload{}, err + } + return payload, nil +} + +func DecodeActionPreflightResultPayload(data []byte) (ActionPreflightResultPayload, error) { + var result ActionPreflightResultPayload + if err := decodeStrictActionPreflight(data, &result); err != nil { + return ActionPreflightResultPayload{}, err + } + if err := ValidateActionPreflightResultPayload(&result); err != nil { + return ActionPreflightResultPayload{}, err + } + return result, nil +} + +func decodeStrictActionPreflight(data []byte, target any) error { + if len(bytes.TrimSpace(data)) == 0 { + return fmt.Errorf("action preflight payload is empty") + } + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(target); err != nil { + return err + } + if err := decoder.Decode(&struct{}{}); err != io.EOF { + if err == nil { + return fmt.Errorf("action preflight payload contains trailing JSON") + } + return fmt.Errorf("action preflight payload contains trailing data: %w", err) + } + return nil +} + +func ValidateActionPreflightPayload(payload *ActionPreflightPayload) error { + if payload == nil { + return fmt.Errorf("action preflight payload is required") + } + payload.RequestID = strings.TrimSpace(payload.RequestID) + if payload.RequestID == "" || len(payload.RequestID) > maxRequestIDLength { + return fmt.Errorf("invalid action preflight request id") + } + if payload.ProtocolVersion != ActionPreflightProtocolVersion { + return fmt.Errorf("unsupported action preflight protocol version %d", payload.ProtocolVersion) + } + count := 0 + typedRequestID := "" + if payload.HostUpdate != nil { + count++ + if err := ValidateHostUpdatePayload(payload.HostUpdate); err != nil { + return err + } + typedRequestID = payload.HostUpdate.RequestID + } + if payload.StorageCleanup != nil { + count++ + if err := ValidateHostStorageCleanupPayload(payload.StorageCleanup); err != nil { + return err + } + typedRequestID = payload.StorageCleanup.RequestID + } + if payload.DockerLifecycle != nil { + count++ + if err := ValidateDockerContainerLifecyclePayload(payload.DockerLifecycle); err != nil { + return err + } + typedRequestID = payload.DockerLifecycle.RequestID + } + if payload.DockerUpdate != nil { + count++ + if err := ValidateDockerContainerUpdatePayload(payload.DockerUpdate); err != nil { + return err + } + typedRequestID = payload.DockerUpdate.RequestID + } + if count != 1 { + return fmt.Errorf("action preflight requires exactly one typed operation") + } + if payload.RequestID != typedRequestID { + return fmt.Errorf("action preflight request id does not match typed operation") + } + return nil +} + +func ActionPreflightBinding(payload ActionPreflightPayload) (operation string, version int, digest string) { + switch { + case payload.HostUpdate != nil: + return payload.HostUpdate.Operation, payload.HostUpdate.OperationVersion, payload.HostUpdate.RequestDigest + case payload.StorageCleanup != nil: + return payload.StorageCleanup.Operation, payload.StorageCleanup.OperationVersion, payload.StorageCleanup.RequestDigest + case payload.DockerLifecycle != nil: + return payload.DockerLifecycle.Operation, payload.DockerLifecycle.OperationVersion, payload.DockerLifecycle.RequestDigest + case payload.DockerUpdate != nil: + return payload.DockerUpdate.Operation, payload.DockerUpdate.OperationVersion, payload.DockerUpdate.RequestDigest + default: + return "", 0, "" + } +} + +func ValidateActionPreflightResultPayload(result *ActionPreflightResultPayload) error { + if result == nil { + return fmt.Errorf("action preflight result is required") + } + result.RequestID = strings.TrimSpace(result.RequestID) + result.Operation = strings.TrimSpace(result.Operation) + result.RequestDigest = strings.TrimSpace(result.RequestDigest) + result.ReasonCode = strings.TrimSpace(result.ReasonCode) + if result.RequestID == "" || len(result.RequestID) > maxRequestIDLength { + return fmt.Errorf("invalid action preflight result request id") + } + if result.ProtocolVersion != ActionPreflightProtocolVersion || result.Operation == "" || result.OperationVersion <= 0 || !hostUpdateInventoryHashPattern.MatchString(result.RequestDigest) { + return fmt.Errorf("invalid action preflight result binding") + } + if result.CheckedAt.IsZero() { + return fmt.Errorf("action preflight checked_at is required") + } + result.CheckedAt = result.CheckedAt.UTC() + if result.Feasible && result.ReasonCode != "" { + return fmt.Errorf("feasible action preflight cannot carry a refusal reason") + } + if !result.Feasible && !IsActionRefusalReasonCode(result.ReasonCode) { + return fmt.Errorf("infeasible action preflight requires a valid refusal reason") + } + return nil +} + +func ValidateActionPreflightResultForRequest(req ActionPreflightPayload, result ActionPreflightResultPayload, receivedAt time.Time) error { + if err := ValidateActionPreflightPayload(&req); err != nil { + return err + } + if err := ValidateActionPreflightResultPayload(&result); err != nil { + return err + } + operation, version, digest := ActionPreflightBinding(req) + if result.RequestID != req.RequestID || result.ProtocolVersion != req.ProtocolVersion || result.Operation != operation || result.OperationVersion != version || result.RequestDigest != digest { + return fmt.Errorf("action preflight result does not match request binding") + } + receivedAt = receivedAt.UTC() + if receivedAt.IsZero() || result.CheckedAt.After(receivedAt.Add(5*time.Minute)) || result.CheckedAt.Before(receivedAt.Add(-5*time.Minute)) { + return fmt.Errorf("action preflight result is stale or has invalid chronology") + } + return nil +} diff --git a/internal/agentexec/action_preflight_test.go b/internal/agentexec/action_preflight_test.go new file mode 100644 index 000000000..303f1eddb --- /dev/null +++ b/internal/agentexec/action_preflight_test.go @@ -0,0 +1,83 @@ +package agentexec + +import ( + "encoding/json" + "strings" + "testing" + "time" +) + +func boundHostUpdatePreflight(t *testing.T) ActionPreflightPayload { + t.Helper() + typed := HostUpdatePayload{ + RequestID: "preflight-1", ActionID: "action-1", Operation: HostUpdateOperationInstall, + ExpectedInventoryHash: "sha256:" + strings.Repeat("a", 64), + } + if err := BindHostUpdatePayload(&typed); err != nil { + t.Fatal(err) + } + return ActionPreflightPayload{RequestID: typed.RequestID, ProtocolVersion: ActionPreflightProtocolVersion, HostUpdate: &typed} +} + +func TestActionPreflightPayloadRequiresExactlyOneBoundOperation(t *testing.T) { + payload := boundHostUpdatePreflight(t) + if err := ValidateActionPreflightPayload(&payload); err != nil { + t.Fatalf("valid payload: %v", err) + } + payload.StorageCleanup = &HostStorageCleanupPayload{} + if err := ValidateActionPreflightPayload(&payload); err == nil { + t.Fatal("payload with two operations was accepted") + } + payload = boundHostUpdatePreflight(t) + payload.RequestID = "different-request" + if err := ValidateActionPreflightPayload(&payload); err == nil { + t.Fatal("mismatched outer and typed request ids were accepted") + } +} + +func TestActionPreflightStrictDecodeRejectsUnknownAuthority(t *testing.T) { + payload := boundHostUpdatePreflight(t) + encoded, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + encoded = append(encoded[:len(encoded)-1], []byte(`,"command":"apt-get upgrade"}`)...) + if _, err := DecodeActionPreflightPayload(encoded); err == nil { + t.Fatal("unknown command field was accepted") + } +} + +func TestActionPreflightResultMustMatchExactDigestAndChronology(t *testing.T) { + req := boundHostUpdatePreflight(t) + operation, version, digest := ActionPreflightBinding(req) + now := time.Now().UTC() + result := ActionPreflightResultPayload{ + RequestID: req.RequestID, ProtocolVersion: req.ProtocolVersion, + Operation: operation, OperationVersion: version, RequestDigest: digest, + Feasible: true, CheckedAt: now, + } + if err := ValidateActionPreflightResultForRequest(req, result, now); err != nil { + t.Fatalf("valid result: %v", err) + } + result.RequestDigest = "sha256:" + strings.Repeat("b", 64) + if err := ValidateActionPreflightResultForRequest(req, result, now); err == nil { + t.Fatal("mismatched digest was accepted") + } +} + +func TestActionPreflightRefusalRequiresTypedReason(t *testing.T) { + req := boundHostUpdatePreflight(t) + operation, version, digest := ActionPreflightBinding(req) + result := ActionPreflightResultPayload{ + RequestID: req.RequestID, ProtocolVersion: req.ProtocolVersion, + Operation: operation, OperationVersion: version, RequestDigest: digest, + CheckedAt: time.Now().UTC(), + } + if err := ValidateActionPreflightResultPayload(&result); err == nil { + t.Fatal("unclassified refusal was accepted") + } + result.ReasonCode = ActionRefusalPackageManagerUnhealthy + if err := ValidateActionPreflightResultPayload(&result); err != nil { + t.Fatalf("typed refusal rejected: %v", err) + } +} diff --git a/internal/agentexec/server.go b/internal/agentexec/server.go index 0336f2674..51a57b40d 100644 --- a/internal/agentexec/server.go +++ b/internal/agentexec/server.go @@ -68,6 +68,7 @@ type Server struct { pendingHostUpdates map[string]chan HostUpdateResultPayload // scoped request key -> typed host-update response pendingDockerContainerLifecycles map[string]chan DockerContainerLifecycleResultPayload pendingDockerContainerUpdates map[string]chan DockerContainerUpdateResultPayload + pendingActionPreflights map[string]chan ActionPreflightResultPayload pendingHostOperations map[string]pendingHostOperation // scoped request key -> exact typed APT operation/query identity pendingOperationQueries map[string]pendingOperationQuery deploySubs map[string]chan DeployProgressPayload // deploySubKey(agentID, jobID) -> progress subscriber @@ -195,6 +196,7 @@ func NewServerWithAdmissionValidator(admit AgentRegistrationValidator, validateS pendingHostUpdates: make(map[string]chan HostUpdateResultPayload), pendingDockerContainerLifecycles: make(map[string]chan DockerContainerLifecycleResultPayload), pendingDockerContainerUpdates: make(map[string]chan DockerContainerUpdateResultPayload), + pendingActionPreflights: make(map[string]chan ActionPreflightResultPayload), pendingHostOperations: make(map[string]pendingHostOperation), pendingOperationQueries: make(map[string]pendingOperationQuery), deploySubs: make(map[string]chan DeployProgressPayload), @@ -974,6 +976,7 @@ func (s *Server) HandleWebSocket(w http.ResponseWriter, r *http.Request) { Tags: reg.Tags, ConnectedAt: time.Now(), OperationReceiptVersion: reg.OperationReceiptVersion, + ActionPreflightVersion: reg.ActionPreflightVersion, }, admission: admission, sessionKey: agentSessionKey(admission.OrganizationID, admission.AgentID), @@ -1244,6 +1247,23 @@ func (s *Server) readLoop(ac *agentConn) { } } + case MsgTypeActionPreflightResult: + result, decodeErr := DecodeActionPreflightResultPayload(msg.Payload) + if decodeErr != nil { + log.Warn().Err(decodeErr).Str("agent_id", ac.agent.AgentID).Msg("Dropping invalid action preflight result") + continue + } + s.mu.RLock() + ch, ok := s.pendingActionPreflights[pendingRequestKey(connectionSessionKey(ac), result.RequestID)] + s.mu.RUnlock() + if ok { + select { + case ch <- result: + default: + log.Warn().Str("agent_id", ac.agent.AgentID).Str("request_id", result.RequestID).Msg("Action preflight result channel full, dropping") + } + } + case MsgTypeHostStorageCleanupResult: result, decodeErr := DecodeHostStorageCleanupResultPayload(msg.Payload) if decodeErr != nil { @@ -1775,6 +1795,72 @@ func (s *Server) ExecuteHostUpdate(ctx context.Context, agentID string, req Host }) } +// PreflightAction asks the current Unified Agent to evaluate the exact typed +// operation without admitting a durable operation or starting a mutation. +func (s *Server) PreflightAction(ctx context.Context, agentID string, req ActionPreflightPayload) (*ActionPreflightResultPayload, error) { + if s == nil { + return nil, fmt.Errorf("agent execution server is unavailable") + } + agentID = strings.TrimSpace(agentID) + if agentID == "" { + return nil, fmt.Errorf("agent id is required") + } + if strings.TrimSpace(req.RequestID) == "" { + req.RequestID = uuid.NewString() + } + if err := ValidateActionPreflightPayload(&req); err != nil { + return nil, err + } + ac, ok := s.connectionForContext(ctx, agentID) + if !ok { + return nil, fmt.Errorf("agent %s not connected", agentID) + } + if ac.agent.ActionPreflightVersion != ActionPreflightProtocolVersion { + return nil, fmt.Errorf("agent does not support action preflight protocol") + } + ch := make(chan ActionPreflightResultPayload, 1) + key := pendingRequestKey(connectionSessionKey(ac), req.RequestID) + s.mu.Lock() + if _, exists := s.pendingActionPreflights[key]; exists { + s.mu.Unlock() + return nil, fmt.Errorf("action preflight request %q is already pending", req.RequestID) + } + s.pendingActionPreflights[key] = ch + s.mu.Unlock() + defer func() { + s.mu.Lock() + delete(s.pendingActionPreflights, key) + s.mu.Unlock() + }() + msg, err := NewMessage(MsgTypeActionPreflight, req.RequestID, req) + if err != nil { + return nil, err + } + ac.writeMu.Lock() + err = s.sendMessage(ac.conn, msg) + ac.writeMu.Unlock() + if err != nil { + return nil, fmt.Errorf("failed to send action preflight request: %w", err) + } + timer := time.NewTimer(20 * time.Second) + defer timer.Stop() + select { + case result := <-ch: + if err := ValidateActionPreflightResultForRequest(req, result, s.currentTime()); err != nil { + return nil, fmt.Errorf("action preflight result validation failed: %w", err) + } + return &result, nil + case <-timer.C: + return nil, fmt.Errorf("action preflight timed out") + case <-ctx.Done(): + return nil, ctx.Err() + case <-ac.done: + return nil, fmt.Errorf("agent %s disconnected before action preflight result", agentID) + case <-s.shutdown: + return nil, errServerShuttingDown + } +} + // ExecuteHostStorageCleanup dispatches the closed package-cache cleanup // operation. No command text, path, package selector, or removal policy crosses // the server/agent boundary. diff --git a/internal/agentexec/server_websocket_test.go b/internal/agentexec/server_websocket_test.go index 5e10583c7..f11701f89 100644 --- a/internal/agentexec/server_websocket_test.go +++ b/internal/agentexec/server_websocket_test.go @@ -696,6 +696,59 @@ func TestExecuteHostUpdateRoundTripUsesTypedCommandFreeEnvelope(t *testing.T) { } } +func TestActionPreflightRoundTripIsReadOnlyAndDigestBound(t *testing.T) { + s := NewServer(allowAllTestTokens) + ts := newWSServer(t, s) + defer ts.Close() + conn, _, err := dialAgentExecWebSocket(t, ts.URL) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + wsWriteMessage(t, conn, mustNewMessage(t, MsgTypeAgentRegister, "", AgentRegisterPayload{ + AgentID: "preflight-agent", Hostname: "host1", Version: "6", Platform: "linux", Token: "any", + OperationReceiptVersion: operationreceipt.ProtocolVersion, ActionPreflightVersion: ActionPreflightProtocolVersion, + })) + _ = wsReadRegisteredPayload(t, conn) + req := boundHostUpdatePreflight(t) + agentErr := make(chan error, 1) + go func() { + msg, readErr := wsReadRawMessageWithTimeout(conn, 2*time.Second) + if readErr != nil { + agentErr <- readErr + return + } + if msg.Type != MsgTypeActionPreflight || msg.Payload == nil || bytes.Contains(*msg.Payload, []byte(`"command"`)) { + agentErr <- fmt.Errorf("unexpected preflight envelope: %#v", msg) + return + } + payload, decodeErr := DecodeActionPreflightPayload(*msg.Payload) + if decodeErr != nil { + agentErr <- decodeErr + return + } + operation, version, digest := ActionPreflightBinding(payload) + result := ActionPreflightResultPayload{ + RequestID: payload.RequestID, ProtocolVersion: payload.ProtocolVersion, + Operation: operation, OperationVersion: version, RequestDigest: digest, + ReasonCode: ActionRefusalPackageManagerUnhealthy, CheckedAt: time.Now().UTC(), + } + agentErr <- conn.WriteJSON(mustNewMessage(t, MsgTypeActionPreflightResult, payload.RequestID, result)) + }() + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + result, err := s.PreflightAction(ctx, "preflight-agent", req) + if err != nil { + t.Fatal(err) + } + if result.Feasible || result.ReasonCode != ActionRefusalPackageManagerUnhealthy { + t.Fatalf("result=%#v", result) + } + if err := <-agentErr; err != nil { + t.Fatal(err) + } +} + func TestValidateHostUpdatePayloadRejectsOpenEndedAuthority(t *testing.T) { for _, req := range []HostUpdatePayload{ {RequestID: "r1", ActionID: "a1", Operation: "run_command", ExpectedInventoryHash: "sha256:" + strings.Repeat("a", 64)}, diff --git a/internal/agentexec/types.go b/internal/agentexec/types.go index 761035283..898290c60 100644 --- a/internal/agentexec/types.go +++ b/internal/agentexec/types.go @@ -19,6 +19,7 @@ const ( MsgTypeHostUpdateResult MessageType = "host_update_result" MsgTypeDockerContainerLifecycleResult MessageType = "docker_container_lifecycle_result" MsgTypeDockerContainerUpdateResult MessageType = "docker_container_update_result" + MsgTypeActionPreflightResult MessageType = "action_preflight_result" MsgTypeOperationQueryResult MessageType = "agent_operation_query_result" // Server -> Agent messages @@ -30,6 +31,7 @@ const ( MsgTypeHostUpdate MessageType = "host_update" MsgTypeDockerContainerLifecycle MessageType = "docker_container_lifecycle" MsgTypeDockerContainerUpdate MessageType = "docker_container_update" + MsgTypeActionPreflight MessageType = "action_preflight" MsgTypeOperationQuery MessageType = "agent_operation_query" MsgTypeDeployPreflight MessageType = "deploy_preflight" MsgTypeDeployInstall MessageType = "deploy_install" @@ -92,6 +94,7 @@ type AgentRegisterPayload struct { Tags []string `json:"tags,omitempty"` Token string `json:"token"` // API token for authentication OperationReceiptVersion int `json:"operation_receipt_version,omitempty"` + ActionPreflightVersion int `json:"action_preflight_version,omitempty"` } // RegisteredPayload is sent by server after successful registration @@ -395,6 +398,32 @@ type HostStorageCleanupResultPayload struct { Duration int64 `json:"duration_ms"` } +// ActionPreflightPayload carries exactly one already-bound typed operation to +// the Unified Agent for a read-only feasibility decision. It grants no durable +// dispatch authority and must never enter the operation receipt store. +type ActionPreflightPayload struct { + RequestID string `json:"request_id"` + ProtocolVersion int `json:"protocol_version"` + HostUpdate *HostUpdatePayload `json:"host_update,omitempty"` + StorageCleanup *HostStorageCleanupPayload `json:"storage_cleanup,omitempty"` + DockerLifecycle *DockerContainerLifecyclePayload `json:"docker_lifecycle,omitempty"` + DockerUpdate *DockerContainerUpdatePayload `json:"docker_update,omitempty"` +} + +// ActionPreflightResultPayload is bounded agent evidence about the exact +// operation digest. No provider output, paths, package names, or command text +// cross this boundary. +type ActionPreflightResultPayload struct { + RequestID string `json:"request_id"` + ProtocolVersion int `json:"protocol_version"` + Operation string `json:"operation"` + OperationVersion int `json:"operation_version"` + RequestDigest string `json:"request_digest"` + Feasible bool `json:"feasible"` + ReasonCode string `json:"reason_code,omitempty"` + CheckedAt time.Time `json:"checked_at"` +} + const ( HostStorageCleanupPhasePreflight = "preflight" HostStorageCleanupPhaseClean = "clean" @@ -450,6 +479,7 @@ type ConnectedAgent struct { Tags []string ConnectedAt time.Time OperationReceiptVersion int + ActionPreflightVersion int } // --- Deploy protocol payloads --- diff --git a/internal/api/action_executor.go b/internal/api/action_executor.go index ed3cf8f21..8bc102048 100644 --- a/internal/api/action_executor.go +++ b/internal/api/action_executor.go @@ -102,6 +102,10 @@ type actionAgentCommander interface { IsAgentConnected(agentID string) bool } +type actionPreflightAgentCommander interface { + PreflightAction(context.Context, string, agentexec.ActionPreflightPayload) (*agentexec.ActionPreflightResultPayload, error) +} + type scopedActionAgentCommander interface { IsAgentConnectedForOrganization(organizationID, agentID string) bool GetAgentForHostForOrganization(organizationID, hostname string) (string, bool) @@ -298,6 +302,65 @@ func (e routedActionExecutor) CheckActionAvailable(ctx context.Context, req unif return checker.CheckActionAvailable(ctx, req, resource) } +func (e routedActionExecutor) CheckActionFeasible(ctx context.Context, actionID string, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness { + capability, ok := resourceCapabilityByName(resource.Capabilities, req.CapabilityName) + if !ok || strings.TrimSpace(capability.InternalHandler) == "" { + return unified.ResourceActionReadiness{} + } + executor := e.byHandler[strings.TrimSpace(capability.InternalHandler)] + if executor == nil { + return unified.ResourceActionReadiness{} + } + checker, ok := executor.(interface { + CheckActionFeasible(context.Context, string, unified.ActionRequest, unified.Resource) unified.ResourceActionReadiness + }) + if !ok { + return unified.ResourceActionReadiness{} + } + return checker.CheckActionFeasible(ctx, actionID, req, resource) +} + +func actionPreflightReadiness(capability string, result *agentexec.ActionPreflightResultPayload, err error) unified.ResourceActionReadiness { + readiness := unified.ResourceActionReadiness{Name: strings.TrimSpace(capability), Available: false} + if err != nil || result == nil { + readiness.ReasonCode = "agent_preflight_unavailable" + readiness.Reason = "Pulse could not confirm that the exact operation is currently feasible on the target agent." + return readiness + } + if result.Feasible { + readiness.Available = true + return readiness + } + readiness.ReasonCode = strings.TrimSpace(result.ReasonCode) + switch readiness.ReasonCode { + case agentexec.ActionRefusalCapabilityUnavailable: + readiness.Reason = "The target agent no longer exposes the required operation." + case agentexec.ActionRefusalTargetInspectionUnavailable: + readiness.Reason = "The target agent could not inspect the resource safely." + case agentexec.ActionRefusalTargetStateChanged: + readiness.Reason = "The target state changed after this action was planned. Refresh the plan before approving it." + case agentexec.ActionRefusalTargetPreconditionFailed: + readiness.Reason = "The target no longer satisfies the operation's local preconditions. Refresh the plan before approving it." + case agentexec.ActionRefusalPackageManagerBusy: + readiness.Reason = "The host package manager is busy. Retry readiness after the current package operation finishes." + case agentexec.ActionRefusalPackagePreflightFailed: + readiness.Reason = "The target agent could not inspect package update feasibility safely." + case agentexec.ActionRefusalPackageInventoryChanged: + readiness.Reason = "The package inventory changed after this action was planned. Refresh the plan before approving it." + case agentexec.ActionRefusalPackageManagerUnhealthy: + readiness.Reason = "The host package manager needs recovery before updates can run safely." + case agentexec.ActionRefusalCleanupPreflightFailed: + readiness.Reason = "The target agent could not inspect package-cache cleanup feasibility safely." + case agentexec.ActionRefusalCleanupInventoryChanged: + readiness.Reason = "The package-cache inventory changed after this action was planned. Refresh the plan before approving it." + case agentexec.ActionRefusalContractInvalid: + readiness.Reason = "The exact operation contract is no longer valid. Refresh the plan before approving it." + default: + readiness.Reason = "The target agent refused the operation during its read-only feasibility check." + } + return readiness +} + func (e routedActionExecutor) executorForAction(ctx context.Context, req unified.ActionRequest) (ActionExecutor, error) { if e.resources == nil { return nil, fmt.Errorf("resource handler unavailable") diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go index 6011206a5..158c0e686 100644 --- a/internal/api/contract_test.go +++ b/internal/api/contract_test.go @@ -21211,6 +21211,23 @@ func TestContract_ResourceActionReadinessPayloadShape(t *testing.T) { } } +func TestContract_ActionPreflightRefusalProjectsStableReadiness(t *testing.T) { + result := &agentexec.ActionPreflightResultPayload{ + Feasible: false, + ReasonCode: agentexec.ActionRefusalPackageManagerUnhealthy, + } + readiness := actionPreflightReadiness(hostPackageUpdateCapability, result, nil) + if readiness.Name != hostPackageUpdateCapability || readiness.Available || readiness.ReasonCode != agentexec.ActionRefusalPackageManagerUnhealthy { + t.Fatalf("readiness=%#v", readiness) + } + if readiness.Reason != "The host package manager needs recovery before updates can run safely." { + t.Fatalf("operator reason=%q", readiness.Reason) + } + if strings.Contains(strings.ToLower(readiness.Reason), "apt-get") || strings.Contains(strings.ToLower(readiness.Reason), "dpkg") { + t.Fatalf("readiness leaked agent command detail: %q", readiness.Reason) + } +} + func TestContract_DockerLifecycleActionsResolveCommandAgentAndDispatchOneTypedOperation(t *testing.T) { source, err := os.ReadFile("docker_container_action_executor.go") if err != nil { diff --git a/internal/api/docker_container_action_executor.go b/internal/api/docker_container_action_executor.go index df61c4100..d3eac14fb 100644 --- a/internal/api/docker_container_action_executor.go +++ b/internal/api/docker_container_action_executor.go @@ -6,6 +6,7 @@ import ( "strings" "time" + "github.com/google/uuid" "github.com/rcourtman/pulse-go-rewrite/internal/actionlifecycle" "github.com/rcourtman/pulse-go-rewrite/internal/agentexec" "github.com/rcourtman/pulse-go-rewrite/internal/ai/safety" @@ -346,6 +347,45 @@ func (e dockerContainerActionExecutor) CheckActionAvailable(ctx context.Context, return readiness } +func (e dockerContainerActionExecutor) CheckActionFeasible(ctx context.Context, actionID string, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness { + if readiness := e.CheckActionAvailable(ctx, req, resource); readiness.Name == "" || !readiness.Available { + return readiness + } + commander, ok := e.agents.(actionPreflightAgentCommander) + if !ok { + return unified.ResourceActionReadiness{} + } + runtime, err := dockerContainerRuntime(resource) + if err != nil { + return actionPreflightReadiness(req.CapabilityName, nil, err) + } + agentID, err := e.connectedDockerCommandAgentID(ctx, resource) + if err != nil { + return actionPreflightReadiness(req.CapabilityName, nil, err) + } + requestID := uuid.NewString() + preflight := agentexec.ActionPreflightPayload{RequestID: requestID, ProtocolVersion: agentexec.ActionPreflightProtocolVersion} + if isDockerContainerUpdateOperation(req.CapabilityName) { + typed, bindErr := dockerContainerUpdateRequest(requestID, actionID, runtime, resource) + if bindErr != nil { + return actionPreflightReadiness(req.CapabilityName, nil, bindErr) + } + preflight.DockerUpdate = &typed + } else { + operation, operationErr := dockerAgentLifecycleOperation(req.CapabilityName) + if operationErr != nil { + return actionPreflightReadiness(req.CapabilityName, nil, operationErr) + } + typed := dockerContainerLifecycleRequest(requestID, actionID, operation, runtime, resource) + if bindErr := agentexec.BindDockerContainerLifecyclePayload(&typed); bindErr != nil { + return actionPreflightReadiness(req.CapabilityName, nil, bindErr) + } + preflight.DockerLifecycle = &typed + } + result, err := commander.PreflightAction(agentCommandContext(ctx), agentID, preflight) + return actionPreflightReadiness(req.CapabilityName, result, err) +} + func (e dockerContainerActionExecutor) currentDockerContainerResource(ctx context.Context, resourceID, operation string) (unified.Resource, error) { if e.resources == nil { return unified.Resource{}, fmt.Errorf("resource handler unavailable") diff --git a/internal/api/host_storage_cleanup_action_executor.go b/internal/api/host_storage_cleanup_action_executor.go index 882f0dc30..99dac1547 100644 --- a/internal/api/host_storage_cleanup_action_executor.go +++ b/internal/api/host_storage_cleanup_action_executor.go @@ -7,6 +7,7 @@ import ( "strings" "time" + "github.com/google/uuid" "github.com/rcourtman/pulse-go-rewrite/internal/actionlifecycle" "github.com/rcourtman/pulse-go-rewrite/internal/agentexec" "github.com/rcourtman/pulse-go-rewrite/internal/operationreceipt" @@ -77,6 +78,26 @@ func (e hostStorageCleanupActionExecutor) CheckActionAvailable(ctx context.Conte return unified.ResourceActionReadiness{Name: hostStorageCleanupCapability, Available: true} } +func (e hostStorageCleanupActionExecutor) CheckActionFeasible(ctx context.Context, actionID string, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness { + if readiness := e.CheckActionAvailable(ctx, req, resource); readiness.Name == "" || !readiness.Available { + return readiness + } + commander, ok := e.agents.(actionPreflightAgentCommander) + if !ok { + return unified.ResourceActionReadiness{} + } + typed := agentexec.HostStorageCleanupPayload{ + RequestID: uuid.NewString(), ActionID: strings.TrimSpace(actionID), Operation: agentexec.HostStorageCleanupOperationPackageCache, + ExpectedFingerprint: resource.Agent.StorageCleanup.Fingerprint, + } + if err := agentexec.BindHostStorageCleanupPayload(&typed); err != nil { + return actionPreflightReadiness(req.CapabilityName, nil, err) + } + preflight := agentexec.ActionPreflightPayload{RequestID: typed.RequestID, ProtocolVersion: agentexec.ActionPreflightProtocolVersion, StorageCleanup: &typed} + result, err := commander.PreflightAction(agentCommandContext(ctx), resource.Agent.AgentID, preflight) + return actionPreflightReadiness(req.CapabilityName, result, err) +} + func (e hostStorageCleanupActionExecutor) ExecuteAction(ctx context.Context, record unified.ActionAuditRecord) (*unified.ExecutionResult, error) { attempt, ok := actionlifecycle.DispatchAttemptFromContext(ctx) if !ok || attempt.ActionID != record.ID { diff --git a/internal/api/host_update_action_executor.go b/internal/api/host_update_action_executor.go index 2868e03fc..424c5ac59 100644 --- a/internal/api/host_update_action_executor.go +++ b/internal/api/host_update_action_executor.go @@ -7,6 +7,7 @@ import ( "strings" "time" + "github.com/google/uuid" "github.com/rcourtman/pulse-go-rewrite/internal/actionlifecycle" "github.com/rcourtman/pulse-go-rewrite/internal/agentexec" "github.com/rcourtman/pulse-go-rewrite/internal/operationreceipt" @@ -82,6 +83,26 @@ func (e hostUpdateActionExecutor) CheckActionAvailable(ctx context.Context, req return unified.ResourceActionReadiness{Name: hostPackageUpdateCapability, Available: true} } +func (e hostUpdateActionExecutor) CheckActionFeasible(ctx context.Context, actionID string, req unified.ActionRequest, resource unified.Resource) unified.ResourceActionReadiness { + if readiness := e.CheckActionAvailable(ctx, req, resource); readiness.Name == "" || !readiness.Available { + return readiness + } + commander, ok := e.agents.(actionPreflightAgentCommander) + if !ok { + return unified.ResourceActionReadiness{} + } + typed := agentexec.HostUpdatePayload{ + RequestID: uuid.NewString(), ActionID: strings.TrimSpace(actionID), Operation: agentexec.HostUpdateOperationInstall, + ExpectedInventoryHash: resource.Agent.PackageUpdates.InventoryHash, + } + if err := agentexec.BindHostUpdatePayload(&typed); err != nil { + return actionPreflightReadiness(req.CapabilityName, nil, err) + } + preflight := agentexec.ActionPreflightPayload{RequestID: typed.RequestID, ProtocolVersion: agentexec.ActionPreflightProtocolVersion, HostUpdate: &typed} + result, err := commander.PreflightAction(agentCommandContext(ctx), resource.Agent.AgentID, preflight) + return actionPreflightReadiness(req.CapabilityName, result, err) +} + func (e hostUpdateActionExecutor) ExecuteAction(ctx context.Context, record unified.ActionAuditRecord) (*unified.ExecutionResult, error) { attempt, ok := actionlifecycle.DispatchAttemptFromContext(ctx) if !ok || attempt.ActionID != record.ID { diff --git a/internal/api/host_update_action_executor_test.go b/internal/api/host_update_action_executor_test.go index 1a5d285d2..723d1dceb 100644 --- a/internal/api/host_update_action_executor_test.go +++ b/internal/api/host_update_action_executor_test.go @@ -29,6 +29,23 @@ type fakeHostUpdateAgent struct { queries []operationreceipt.Identity } +type preflightHostUpdateAgent struct { + *fakeHostUpdateAgent + feasible bool + reasonCode string + preflights []agentexec.ActionPreflightPayload +} + +func (f *preflightHostUpdateAgent) PreflightAction(_ context.Context, _ string, req agentexec.ActionPreflightPayload) (*agentexec.ActionPreflightResultPayload, error) { + f.preflights = append(f.preflights, req) + operation, version, digest := agentexec.ActionPreflightBinding(req) + return &agentexec.ActionPreflightResultPayload{ + RequestID: req.RequestID, ProtocolVersion: req.ProtocolVersion, + Operation: operation, OperationVersion: version, RequestDigest: digest, + Feasible: f.feasible, ReasonCode: f.reasonCode, CheckedAt: time.Now().UTC(), + }, nil +} + func (f *fakeHostUpdateAgent) QueryAgentOperation(_ context.Context, _ string, identity operationreceipt.Identity) (operationreceipt.QueryResult, error) { f.queries = append(f.queries, identity) return f.queryResult, f.queryErr @@ -114,6 +131,26 @@ func TestHostUpdateActionExecutorDispatchesTypedOperationAndProjectsVerification } } +func TestHostUpdateActionFeasibilityReturnsAgentRefusalBeforeApproval(t *testing.T) { + now := time.Now().UTC() + h := NewResourceHandlers(&config.Config{DataPath: t.TempDir()}) + resource := hostUpdateActionResource(now) + h.SetStateProvider(resourceUnifiedSeedProvider{snapshot: models.StateSnapshot{LastUpdate: now}, resources: []unified.Resource{resource}}) + agents := &preflightHostUpdateAgent{ + fakeHostUpdateAgent: &fakeHostUpdateAgent{connected: true}, + reasonCode: agentexec.ActionRefusalPackageManagerUnhealthy, + } + executor := newHostUpdateActionExecutor(h, agents).(hostUpdateActionExecutor) + readiness := executor.CheckActionFeasible(context.Background(), "action-preflight", unified.ActionRequest{CapabilityName: hostPackageUpdateCapability}, resource) + if readiness.Available || readiness.ReasonCode != agentexec.ActionRefusalPackageManagerUnhealthy || len(agents.preflights) != 1 { + t.Fatalf("readiness=%#v preflights=%d", readiness, len(agents.preflights)) + } + typed := agents.preflights[0].HostUpdate + if typed == nil || typed.ActionID != "action-preflight" || typed.ExpectedInventoryHash != testHostPackageInventoryHash { + t.Fatalf("typed preflight=%#v", typed) + } +} + func TestHostUpdateActionExecutorReportsInconclusiveVerificationHonestly(t *testing.T) { now := time.Now().UTC() h := NewResourceHandlers(&config.Config{DataPath: t.TempDir()}) diff --git a/internal/dockeragent/container_update_typed.go b/internal/dockeragent/container_update_typed.go index 26dfc8d12..b67f331f5 100644 --- a/internal/dockeragent/container_update_typed.go +++ b/internal/dockeragent/container_update_typed.go @@ -15,19 +15,42 @@ import ( // same pull/backup/recreate/rollback implementation the module has always // used and reports the neutral outcome the unified agent bridge forwards. func (a *Agent) TypedContainerUpdate(ctx context.Context, runtime, containerID, expectedImageDigest string, progress func(string)) (agentexec.DockerContainerUpdateOutcome, error) { + if err := a.TypedContainerUpdatePreflight(ctx, runtime, containerID, expectedImageDigest); err != nil { + return agentexec.DockerContainerUpdateOutcome{}, err + } + + result := a.updateContainerWithProgress(ctx, containerID, progress) + return agentexec.DockerContainerUpdateOutcome{ + Success: result.Success, + ContainerName: result.ContainerName, + OldContainerID: result.OldContainerID, + NewContainerID: result.NewContainerID, + OldImageDigest: result.OldImageDigest, + NewImageDigest: result.NewImageDigest, + BackupCreated: result.BackupCreated, + BackupContainer: result.BackupContainer, + RollbackAttempted: result.RollbackAttempted, + RolledBack: result.RolledBack, + Error: result.Error, + }, nil +} + +// TypedContainerUpdatePreflight performs only the daemon reads needed to +// prove that the exact planned container and image digest remain current. +func (a *Agent) TypedContainerUpdatePreflight(ctx context.Context, runtime, containerID, expectedImageDigest string) error { if a == nil || a.docker == nil { - return agentexec.DockerContainerUpdateOutcome{}, fmt.Errorf("docker module is not connected to a container runtime") + return agentexec.NewActionPreflightError(agentexec.ActionRefusalCapabilityUnavailable, fmt.Errorf("docker module is not connected to a container runtime")) } requestedRuntime := strings.ToLower(strings.TrimSpace(runtime)) if requestedRuntime != "" && requestedRuntime != strings.ToLower(string(a.runtime)) { - return agentexec.DockerContainerUpdateOutcome{}, fmt.Errorf("container runtime mismatch: module runs %s", a.runtime) + return agentexec.NewActionPreflightError(agentexec.ActionRefusalCapabilityUnavailable, fmt.Errorf("container runtime mismatch: module runs %s", a.runtime)) } inspect, err := dockerCallWithRetry(ctx, dockerUpdateCallTimeout, func(callCtx context.Context) (containertypes.InspectResponse, error) { return a.docker.ContainerInspect(callCtx, containerID) }) if err != nil { - return agentexec.DockerContainerUpdateOutcome{}, fmt.Errorf("container preflight inspect unavailable: %v", annotateDockerConnectionError(err)) + return agentexec.NewActionPreflightError(agentexec.ActionRefusalTargetInspectionUnavailable, fmt.Errorf("container preflight inspect unavailable: %v", annotateDockerConnectionError(err))) } if expectedImageDigest != "" { expectedImageDigest = strings.TrimSpace(expectedImageDigest) @@ -48,22 +71,8 @@ func (a *Agent) TypedContainerUpdate(ctx context.Context, runtime, containerID, if repoDigest == "" { repoDigest = "unavailable" } - return agentexec.DockerContainerUpdateOutcome{}, fmt.Errorf("container image digest no longer matches the planned update (expected %s, local image id %s, local repo digest %s)", expectedImageDigest, localImageID, repoDigest) + return agentexec.NewActionPreflightError(agentexec.ActionRefusalTargetPreconditionFailed, fmt.Errorf("container image digest no longer matches the planned update (expected %s, local image id %s, local repo digest %s)", expectedImageDigest, localImageID, repoDigest)) } } - - result := a.updateContainerWithProgress(ctx, containerID, progress) - return agentexec.DockerContainerUpdateOutcome{ - Success: result.Success, - ContainerName: result.ContainerName, - OldContainerID: result.OldContainerID, - NewContainerID: result.NewContainerID, - OldImageDigest: result.OldImageDigest, - NewImageDigest: result.NewImageDigest, - BackupCreated: result.BackupCreated, - BackupContainer: result.BackupContainer, - RollbackAttempted: result.RollbackAttempted, - RolledBack: result.RolledBack, - Error: result.Error, - }, nil + return nil } diff --git a/internal/hostagent/commands.go b/internal/hostagent/commands.go index 96129a520..ca80b738f 100644 --- a/internal/hostagent/commands.go +++ b/internal/hostagent/commands.go @@ -194,6 +194,8 @@ const ( msgTypeDockerContainerLifecycleResult messageType = "docker_container_lifecycle_result" msgTypeDockerContainerUpdate messageType = "docker_container_update" msgTypeDockerContainerUpdateResult messageType = "docker_container_update_result" + msgTypeActionPreflight messageType = "action_preflight" + msgTypeActionPreflightResult messageType = "action_preflight_result" msgTypeOperationQuery messageType = "agent_operation_query" msgTypeOperationQueryResult messageType = "agent_operation_query_result" msgTypeDeployPreflight messageType = "deploy_preflight" @@ -217,6 +219,7 @@ type registerPayload struct { Tags []string `json:"tags,omitempty"` Token string `json:"token"` OperationReceiptVersion int `json:"operation_receipt_version,omitempty"` + ActionPreflightVersion int `json:"action_preflight_version,omitempty"` } type registeredPayload struct { @@ -442,6 +445,7 @@ func (c *CommandClient) sendRegistration(conn *websocket.Conn) error { Platform: c.platform, Token: c.apiToken, OperationReceiptVersion: c.operationReceiptVersion(), + ActionPreflightVersion: agentexec.ActionPreflightProtocolVersion, }) if err != nil { return fmt.Errorf("marshal registration payload: %w", err) @@ -571,6 +575,14 @@ func (c *CommandClient) handleMessages(ctx context.Context, conn *websocket.Conn } go c.handleHostUpdate(ctx, conn, payload) + case msgTypeActionPreflight: + payload, err := agentexec.DecodeActionPreflightPayload(msg.Payload) + if err != nil { + c.logger.Warn().Err(err).Msg("Dropping invalid action preflight request") + continue + } + go c.handleActionPreflight(ctx, conn, payload) + case msgTypeHostStorageCleanup: payload, err := agentexec.DecodeHostStorageCleanupPayload(msg.Payload) if err != nil { @@ -653,6 +665,76 @@ func (c *CommandClient) handleHostUpdate(ctx context.Context, conn *websocket.Co completeHostAPTOperation(c, conn, identity, sanitizeHostUpdateReceipt(result), agentexec.HostUpdateReceiptKind, payload.RequestID, "Failed to persist host update terminal receipt", c.sendHostUpdateResult) } +func (c *CommandClient) handleActionPreflight(ctx context.Context, conn *websocket.Conn, payload agentexec.ActionPreflightPayload) { + operation, version, digest := agentexec.ActionPreflightBinding(payload) + result := agentexec.ActionPreflightResultPayload{ + RequestID: payload.RequestID, ProtocolVersion: payload.ProtocolVersion, + Operation: operation, OperationVersion: version, RequestDigest: digest, + CheckedAt: time.Now().UTC(), + } + preflightCtx, cancel := context.WithTimeout(ctx, 15*time.Second) + defer cancel() + switch { + case payload.HostUpdate != nil: + result.Feasible, result.ReasonCode = c.preflightHostUpdate(preflightCtx, *payload.HostUpdate) + case payload.StorageCleanup != nil: + result.Feasible, result.ReasonCode = c.preflightStorageCleanup(preflightCtx, *payload.StorageCleanup) + case payload.DockerLifecycle != nil: + result.Feasible, result.ReasonCode = c.preflightDockerLifecycle(preflightCtx, *payload.DockerLifecycle) + case payload.DockerUpdate != nil: + result.Feasible, result.ReasonCode = c.preflightDockerUpdate(preflightCtx, *payload.DockerUpdate) + default: + result.ReasonCode = agentexec.ActionRefusalContractInvalid + } + result.CheckedAt = time.Now().UTC() + encoded, err := json.Marshal(result) + if err != nil { + return + } + msg := wsMessage{Type: msgTypeActionPreflightResult, ID: result.RequestID, Timestamp: time.Now(), Payload: encoded} + c.connMu.Lock() + err = conn.WriteJSON(msg) + c.connMu.Unlock() + if err != nil { + c.logger.Debug().Err(err).Str("request_id", result.RequestID).Msg("Failed to send action preflight result") + } +} + +func (c *CommandClient) preflightHostUpdate(ctx context.Context, payload agentexec.HostUpdatePayload) (bool, string) { + if c.packageUpdates == nil { + return false, agentexec.ActionRefusalCapabilityUnavailable + } + return c.packageUpdates.Preflight(ctx, payload) +} + +func (c *CommandClient) preflightStorageCleanup(ctx context.Context, payload agentexec.HostStorageCleanupPayload) (bool, string) { + if c.storageCleanup == nil { + return false, agentexec.ActionRefusalCapabilityUnavailable + } + return c.storageCleanup.Preflight(ctx, payload) +} + +func (c *CommandClient) preflightDockerLifecycle(ctx context.Context, payload agentexec.DockerContainerLifecyclePayload) (bool, string) { + preflighter, ok := c.dockerLifecycle.(interface { + Preflight(context.Context, agentexec.DockerContainerLifecyclePayload) (bool, string) + }) + if !ok || preflighter == nil { + return false, agentexec.ActionRefusalCapabilityUnavailable + } + return preflighter.Preflight(ctx, payload) +} + +func (c *CommandClient) preflightDockerUpdate(ctx context.Context, payload agentexec.DockerContainerUpdatePayload) (bool, string) { + preflighter, ok := c.dockerUpdater.(DockerContainerUpdatePreflighter) + if !ok || preflighter == nil { + return false, agentexec.ActionRefusalCapabilityUnavailable + } + if err := preflighter.TypedContainerUpdatePreflight(ctx, payload.Runtime, payload.ContainerID, payload.ExpectedImageDigest); err != nil { + return false, agentexec.ActionPreflightReasonCode(err, agentexec.ActionRefusalTargetPreconditionFailed) + } + return true, "" +} + // beginHostAPTOperation runs the shared durable admission prefix of the typed // host APT operation handlers: admit → replay-if-terminal → mark started → // derive the bounded execution context. ok is false when the handler must diff --git a/internal/hostagent/docker_lifecycle.go b/internal/hostagent/docker_lifecycle.go index ba6c075e0..2d43ee1fd 100644 --- a/internal/hostagent/docker_lifecycle.go +++ b/internal/hostagent/docker_lifecycle.go @@ -86,6 +86,20 @@ func (m *localDockerLifecycleManager) Apply(ctx context.Context, req agentexec.D return result } +func (m *localDockerLifecycleManager) Preflight(ctx context.Context, req agentexec.DockerContainerLifecyclePayload) (bool, string) { + if err := agentexec.ValidateDockerContainerLifecyclePayload(&req); err != nil { + return false, agentexec.ActionRefusalContractInvalid + } + before, err := m.inspect(ctx, req.Runtime, req.ContainerID) + if err != nil { + return false, agentexec.ActionRefusalTargetInspectionUnavailable + } + if !dockerLifecycleBeforeMatches(req, before) { + return false, agentexec.ActionRefusalTargetStateChanged + } + return true, "" +} + func (m *localDockerLifecycleManager) command(ctx context.Context, runtime string, args ...string) ([]byte, error) { if m == nil || m.run == nil { return nil, fmt.Errorf("container runtime command runner unavailable") diff --git a/internal/hostagent/docker_lifecycle_test.go b/internal/hostagent/docker_lifecycle_test.go index bc23bb7e8..851434f88 100644 --- a/internal/hostagent/docker_lifecycle_test.go +++ b/internal/hostagent/docker_lifecycle_test.go @@ -76,6 +76,25 @@ func TestDockerLifecycleManagerStaleBeforeStateIsTripleZero(t *testing.T) { } } +func TestDockerLifecyclePreflightRefusesStateDriftWithoutMutation(t *testing.T) { + t.Setenv("DOCKER_CONTEXT", "") + before := time.Now().UTC().Add(-time.Minute) + mutations := 0 + manager := &localDockerLifecycleManager{now: time.Now, run: func(_ context.Context, _ string, args ...string) ([]byte, error) { + if len(args) > 0 && args[0] == "restart" { + mutations++ + } + if len(args) > 2 && strings.Contains(args[2], ".State") { + return []byte(fmt.Sprintf(`{"Status":"exited","Running":false,"StartedAt":%q}`, before.Format(time.RFC3339Nano))), nil + } + return []byte("0"), nil + }} + feasible, reason := manager.Preflight(context.Background(), dockerLifecycleTestRequest(t, before)) + if feasible || reason != agentexec.ActionRefusalTargetStateChanged || mutations != 0 { + t.Fatalf("feasible=%t reason=%q mutations=%d", feasible, reason, mutations) + } +} + func TestDockerLifecycleManagerFailedInspectIsTripleZero(t *testing.T) { t.Setenv("DOCKER_CONTEXT", "") manager := &localDockerLifecycleManager{now: time.Now, run: func(context.Context, string, ...string) ([]byte, error) { diff --git a/internal/hostagent/docker_update.go b/internal/hostagent/docker_update.go index 89c7f8904..01b85e8f3 100644 --- a/internal/hostagent/docker_update.go +++ b/internal/hostagent/docker_update.go @@ -21,6 +21,10 @@ type DockerContainerUpdater interface { TypedContainerUpdate(ctx context.Context, runtime, containerID, expectedImageDigest string, progress func(string)) (agentexec.DockerContainerUpdateOutcome, error) } +type DockerContainerUpdatePreflighter interface { + TypedContainerUpdatePreflight(ctx context.Context, runtime, containerID, expectedImageDigest string) error +} + func (c *CommandClient) handleDockerContainerUpdate(ctx context.Context, conn *websocket.Conn, payload agentexec.DockerContainerUpdatePayload) { identity := agentexec.DockerContainerUpdateOperationIdentity(c.agentID, payload) record, admitted, err := c.admitOperation(identity) @@ -84,7 +88,7 @@ func (c *CommandClient) runDockerContainerUpdate(ctx context.Context, payload ag outcome, err := c.dockerUpdater.TypedContainerUpdate(ctx, payload.Runtime, payload.ContainerID, payload.ExpectedImageDigest, progress) if err != nil { c.logger.Warn().Err(err).Str("request_id", payload.RequestID).Str("container_id", payload.ContainerID).Msg("Docker update refused before mutation") - result.ReasonCode = agentexec.ActionRefusalTargetPreconditionFailed + result.ReasonCode = agentexec.ActionPreflightReasonCode(err, agentexec.ActionRefusalTargetPreconditionFailed) result.Error = boundDockerUpdateError(err.Error()) return result } diff --git a/internal/hostagent/package_updates.go b/internal/hostagent/package_updates.go index d505adda1..cc1a91a25 100644 --- a/internal/hostagent/package_updates.go +++ b/internal/hostagent/package_updates.go @@ -69,6 +69,34 @@ func (m *packageUpdateManager) Snapshot(ctx context.Context, force bool) agentex return m.snapshotLocked(ctx, force) } +func (m *packageUpdateManager) Preflight(ctx context.Context, req agentexec.HostUpdatePayload) (bool, string) { + if err := agentexec.ValidateHostUpdatePayload(&req); err != nil { + return false, agentexec.ActionRefusalContractInvalid + } + release, err := m.lease.acquire(ctx) + if err != nil { + return false, agentexec.ActionRefusalPackageManagerBusy + } + defer release() + m.mu.Lock() + defer m.mu.Unlock() + snapshot := m.snapshotLocked(ctx, true) + if !snapshot.Supported { + return false, agentexec.ActionRefusalCapabilityUnavailable + } + if snapshot.Error != "" { + return false, agentexec.ActionRefusalPackagePreflightFailed + } + if snapshot.InventoryHash != strings.TrimSpace(req.ExpectedInventoryHash) { + return false, agentexec.ActionRefusalPackageInventoryChanged + } + checked, healthy := m.checkPackageManagerHealth(ctx) + if !checked || !healthy { + return false, agentexec.ActionRefusalPackageManagerUnhealthy + } + return true, "" +} + func (m *packageUpdateManager) snapshotLocked(ctx context.Context, force bool) agentexec.HostPackageUpdateSnapshot { now := m.currentTime() if !force && m.cached != nil && now.Sub(m.cached.CheckedAt) < m.cacheTTL { diff --git a/internal/hostagent/package_updates_test.go b/internal/hostagent/package_updates_test.go index c8af905b7..2ec81bdbe 100644 --- a/internal/hostagent/package_updates_test.go +++ b/internal/hostagent/package_updates_test.go @@ -108,6 +108,38 @@ func TestPackageUpdateManagerApplyUsesClosedAPTCommandCatalogAndVerifies(t *test } } +func TestPackageUpdatePreflightRefusesUnhealthyManagerWithoutMutation(t *testing.T) { + pending := "Inst openssl [1.0] (1.1 repo [amd64])\n" + m := newPackageUpdateManager("linux", newPackageManagerLease()) + m.lookPath = func(name string) (string, error) { return "/usr/bin/" + name, nil } + m.stat = func(string) (os.FileInfo, error) { return nil, os.ErrNotExist } + var calls [][]string + m.run = func(_ context.Context, _ []string, name string, args ...string) packageUpdateCommandResult { + calls = append(calls, append([]string{name}, args...)) + if name == "apt-get" && strings.Contains(strings.Join(args, " "), "-s") { + return packageUpdateCommandResult{stdout: pending} + } + if name == "dpkg" { + return packageUpdateCommandResult{stdout: "packages require configuration"} + } + t.Fatalf("unexpected mutating command: %s %v", name, args) + return packageUpdateCommandResult{} + } + req := agentexec.HostUpdatePayload{RequestID: "preflight-1", ActionID: "action-1", Operation: agentexec.HostUpdateOperationInstall, ExpectedInventoryHash: aptUpgradeInventoryHash(pending)} + if err := agentexec.BindHostUpdatePayload(&req); err != nil { + t.Fatal(err) + } + feasible, reason := m.Preflight(context.Background(), req) + if feasible || reason != agentexec.ActionRefusalPackageManagerUnhealthy { + t.Fatalf("feasible=%t reason=%q calls=%#v", feasible, reason, calls) + } + for _, call := range calls { + if len(call) > 1 && (call[1] == "update" || call[1] == "upgrade" || call[1] == "clean") { + t.Fatalf("preflight mutated package state: %#v", calls) + } + } +} + func TestPackageUpdateManagerFailsClosedWhenRefreshFails(t *testing.T) { m := newPackageUpdateManager("linux", newPackageManagerLease()) m.lookPath = func(string) (string, error) { return "/usr/bin/apt-get", nil } diff --git a/internal/hostagent/storage_cleanup.go b/internal/hostagent/storage_cleanup.go index 63ae13aa7..5697c9d09 100644 --- a/internal/hostagent/storage_cleanup.go +++ b/internal/hostagent/storage_cleanup.go @@ -60,6 +60,30 @@ func (m *storageCleanupManager) Snapshot(ctx context.Context, force bool) agente return m.snapshotLocked(force) } +func (m *storageCleanupManager) Preflight(ctx context.Context, req agentexec.HostStorageCleanupPayload) (bool, string) { + if err := agentexec.ValidateHostStorageCleanupPayload(&req); err != nil { + return false, agentexec.ActionRefusalContractInvalid + } + release, err := m.lease.acquire(ctx) + if err != nil { + return false, agentexec.ActionRefusalPackageManagerBusy + } + defer release() + m.mu.Lock() + defer m.mu.Unlock() + snapshot := m.snapshotLocked(true) + if !snapshot.Supported { + return false, agentexec.ActionRefusalCapabilityUnavailable + } + if snapshot.Error != "" { + return false, agentexec.ActionRefusalCleanupPreflightFailed + } + if snapshot.Fingerprint != strings.TrimSpace(req.ExpectedFingerprint) { + return false, agentexec.ActionRefusalCleanupInventoryChanged + } + return true, "" +} + func (m *storageCleanupManager) snapshotLocked(force bool) agentexec.HostStorageCleanupSnapshot { now := m.currentTime() if !force && m.cached != nil && now.Sub(m.cached.CheckedAt) < m.cacheTTL { diff --git a/internal/hostagent/storage_cleanup_test.go b/internal/hostagent/storage_cleanup_test.go index dce64cdcb..0e19f08a4 100644 --- a/internal/hostagent/storage_cleanup_test.go +++ b/internal/hostagent/storage_cleanup_test.go @@ -117,6 +117,26 @@ func TestStorageCleanupManagerRefusesFingerprintDriftBeforeMutation(t *testing.T } } +func TestStorageCleanupPreflightRefusesFingerprintDriftWithoutClean(t *testing.T) { + manager := newStorageCleanupManager("linux", newPackageManagerLease()) + manager.lookPath = func(string) (string, error) { return "/usr/bin/apt-get", nil } + manager.scan = func() (agentexec.HostStorageCleanupSnapshot, error) { + return agentexec.HostStorageCleanupSnapshot{Fingerprint: "sha256:" + strings.Repeat("b", 64), ReclaimableBytes: 400}, nil + } + manager.run = func(context.Context, []string, string, ...string) packageUpdateCommandResult { + t.Fatal("read-only preflight must not invoke apt-get clean") + return packageUpdateCommandResult{} + } + req := agentexec.HostStorageCleanupPayload{RequestID: "preflight-1", ActionID: "action-1", Operation: agentexec.HostStorageCleanupOperationPackageCache, ExpectedFingerprint: "sha256:" + strings.Repeat("a", 64)} + if err := agentexec.BindHostStorageCleanupPayload(&req); err != nil { + t.Fatal(err) + } + feasible, reason := manager.Preflight(context.Background(), req) + if feasible || reason != agentexec.ActionRefusalCleanupInventoryChanged { + t.Fatalf("feasible=%t reason=%q", feasible, reason) + } +} + func TestStorageCleanupManagerFailurePhasesPreserveMeasuredEffect(t *testing.T) { before := agentexec.HostStorageCleanupSnapshot{Supported: true, Provider: "apt-package-cache", Fingerprint: "sha256:" + strings.Repeat("a", 64), ReclaimableBytes: 400, CheckedAt: time.Now().UTC().Add(-time.Second)} for _, tc := range []struct {