diff --git a/docs/gateway/cloud-workers.md b/docs/gateway/cloud-workers.md index 54a4c1856ddd..6081f8c523e5 100644 --- a/docs/gateway/cloud-workers.md +++ b/docs/gateway/cloud-workers.md @@ -48,6 +48,12 @@ Cloud workers are opt-in. Until you configure a profile, clients hide the Cloud | Transcript and live session state | Gateway, fed by the worker's replayable event stream | Gateway through the normal local harness path | | Workspace file state | Changed on the box; reconciled by the Gateway | Changed remotely; reconciled by the Gateway | +Applications that make their own model API calls need a separate credential +route. For an exclusively owned coordinator-backed Linux lease, use +[`openclaw crabbox run`](/gateway/secrets/secret-store-and-egress#model-credentials-for-crabbox-commands) +to keep the configured API key on the host while the application uses a protected +egress bridge. + The bundled Crabbox cloud provider advertises both `worker-turn` and `remote-exec` through its enrolled node transport, so the same cloud profile is available to both harnesses. Codex can also use an explicitly authorized paired device or a provider that retains an SSH-backed remote-execution carrier. A profile that advertises only one mode remains unavailable to the other runtime. After Crabbox setup, the cloud node dials the Gateway's public TLS endpoint over outbound WebSocket. Worker control, Codex remote execution, and workspace transfer use authenticated node or worker channels, not a Gateway-created reverse tunnel or rsync. Crabbox itself may still require SSH reachability while its CLI runs the provider-owned setup command. Outbound internet access and setup reachability follow the selected backend's network policy; configure them in Crabbox. diff --git a/docs/gateway/secrets/secret-store-and-egress.md b/docs/gateway/secrets/secret-store-and-egress.md index 2940b2aaab4f..5ca73269bfc1 100644 --- a/docs/gateway/secrets/secret-store-and-egress.md +++ b/docs/gateway/secrets/secret-store-and-egress.md @@ -3,6 +3,7 @@ summary: "The shared secret store, the default-off secret egress proxy, and file read_when: - Storing team-wide secrets and environment values in the shared secret store - Enabling the destination-bound secret egress proxy or its traffic allowlist + - Running a Crabbox application with a configured model credential kept on the host title: "Shared secret store and egress proxy" --- @@ -145,9 +146,110 @@ Current limits: - Identity-scoped secrets are not supported; only the team store participates. - Allowed-host policy is exact-hostname authorization only. It does not validate the resolved IP or prevent an allowed origin from reflecting credentials. - Plain HTTP is refused; it is not upgraded or substituted. -- Secret egress applies only to Gateway-hosted exec. Sandbox and remote `node` exec receive neither proxy variables nor sentinels, so shared-store `secret` entries are unavailable there. Provider-native harness subprocesses also do not use this proxy. +- Automatic shared-store secret egress applies only to Gateway-hosted exec. Sandbox and remote `node` exec receive neither proxy variables nor sentinels, so shared-store `secret` entries are unavailable there. Provider-native harness subprocesses also do not use this proxy. The explicit Crabbox command below grants a configured model credential separately. - Background subprocesses retain their original secret snapshot until they exit or are stopped. Changes to stored credentials or destination bindings require a new run and a new command; stop existing commands to revoke their older grants immediately. +## Model credentials for Crabbox commands + +The Crabbox plugin lets a foreground application in a Linux lease call an +OpenAI-compatible API using a credential kept on the host. Cloud agents already +keep their own inference and provider authentication on the Gateway; this command +is for API calls made by the application itself. + +### Requirements + +Run the command on the host that owns the configured credential, from the local +project directory that owns the lease. Use an exclusively owned, +coordinator-backed Linux lease with a configured Crabbox login and no active +egress session. The Crabbox binary must support `egress run` with +`--upstream-proxy-env`; the command checks support before reading the credential. +Use `--binary ` to select another binary. + +The selected provider must use an API-key SecretRef in +`models.providers..apiKey`. File, environment, exec, and shared-store +SecretRefs use the existing resolver without copying the credential into another +store. The model must resolve to an `openai-responses` or `openai-completions` +route with an HTTPS endpoint on port 443. Endpoint URLs cannot contain credentials, +a query, or a fragment. Auth-profile and OAuth credentials, custom headers, +request proxy/TLS overrides, disabled auth headers, and local-service +configuration are unsupported. + +### Prepare and run + +Sync files, hydrate the workspace, and install dependencies through the normal +Crabbox workflow first. The model command passes `--no-sync --no-hydrate`, so it +uses the prepared workspace and cannot fetch dependencies through its +model-host-only bridge. Keep the same local project directory for preparation +and execution: `--id` selects a lease but does not override Crabbox's repository +claim or workspace selection. + +For an existing lease and configured OpenAI SecretRef, this example checks that +`curl` is available, then makes a Responses API request without reading the key: + +```bash +cd ~/path/to/project +crabbox run --id -- curl --version +openclaw crabbox run --id --model openai/gpt-5.6-sol -- sh -c ' + curl --fail-with-body --silent --show-error "${OPENAI_BASE_URL%/}/responses" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -H "Content-Type: application/json" \ + --data "{\"model\":\"$OPENAI_MODEL\",\"input\":\"Reply with OK.\"}" +' +``` + +Success returns the provider's response JSON. Replace `sh -c ...` with your +application command, such as `node test-app.js`. Use the endpoint appropriate to +the configured route; a Completions-only provider needs its matching API call. +`--provider ` selects a Crabbox backend, while `--model` selects the +model provider. `--timeout ` bounds setup and execution together +(1–86400 seconds, default 600). Cancellation and timeout revoke credential access +immediately and give Crabbox 75 seconds for graceful cleanup. + +OpenClaw resolves the selected model's configured or provider-owned API endpoint +and starts an isolated secret proxy. Crabbox's native `egress run` owns the +foreground bridge, remote command, and session cleanup. OpenClaw supplies +`OPENAI_API_KEY` as an opaque sentinel, +`OPENAI_BASE_URL`, and `OPENAI_MODEL`, plus HTTP proxy settings and a temporary +public CA bundle. The API key, upstream proxy authentication, and CA private key +stay on the host. The bridge permits only the selected hostname; this does not +block a program from opening direct sockets. Model selection sets the app's +default environment, not a limit on the provider credential's API operations or +models. + +The application's HTTP client must honor both proxy and CA settings. `curl` and +Python's default `urllib.request` opener use the injected environment. For Node.js, +use a runtime supporting `NODE_USE_ENV_PROXY` (for example Node.js 24+) with +`NODE_EXTRA_CA_CERTS`. A custom Node dispatcher, Python opener, or SDK client may +override these defaults; configure its proxy and trust explicitly if needed. +Disabling certificate verification or ignoring the proxy does not establish +protected model access. + +### Lifetime and recovery + +Cancellation and timeout revoke credential use immediately, before command +cleanup settles. Completion closes the grant and bridge and stops the matching +lease-side egress client. The lease and prepared workspace remain available. +Keep the foreground command running for the entire app lifetime; detached apps +lose model access when it exits. Ordinary remote `exec`, `background`, and +sandbox commands do not acquire this grant. + +An old binary is refused with an update message. An active-egress error requires +an idle lease or stopping an existing session you own. A repository-claim error +means you must return to the lease's owning local project directory. Do not +reclaim another job's lease to bypass either check. + +If cleanup cannot confirm settlement, the command fails. Inspect +`crabbox egress status --id ` and, when the failed command reported a +session ID, retry `crabbox egress stop --id --session ` +for that session. Also confirm the remote workload has stopped before reusing +the lease, or release the disposable lease through its normal owner. Revocation +is not proof that an unreachable remote process has exited. + +This standalone command does not require `secrets.egressProxy.enabled`, change +Gateway configuration, or restart the Gateway. After a completed or canceled +command, start a new command to obtain a fresh grant; its old sentinel and CA +files are not reusable credentials. + ## File-backed API keys Do not put `file:...` strings in the config `env` block. That block is literal and non-overriding, so `file:...` is never resolved there. diff --git a/docs/plugins/sdk-runtime/models.md b/docs/plugins/sdk-runtime/models.md index 001dad6799f2..850a04443919 100644 --- a/docs/plugins/sdk-runtime/models.md +++ b/docs/plugins/sdk-runtime/models.md @@ -10,6 +10,41 @@ sidebarTitle: "Model helpers" Call a model, resolve model-selection policy, and resolve provider auth without importing host internals. Part of the [Plugin runtime helpers](/plugins/sdk-runtime) reference. +## Protected model egress for standalone commands + +`withConfiguredModelEgress` from `openclaw/plugin-sdk/secret-egress-runtime` +runs an explicit standalone command for an official plugin with a destination-bound credential +sentinel. It resolves the selected provider's configured API-key SecretRef +through the normal secret resolver, including file-backed keys, and uses the +provider's model route policy for its HTTPS endpoint on port 443. It supports +OpenAI-compatible Responses and Completions routes. OAuth, auth-profile +references, custom request headers, and custom request transport are unsupported. +This is a private JavaScript-only host binding for bundled and separately +published official plugins, not a third-party plugin API. + +```typescript +const { withConfiguredModelEgress } = await import("openclaw/plugin-sdk/secret-egress-runtime"); +await withConfiguredModelEgress({ config, provider, model, signal }, async (egress) => { + // Keep hostEnv on the credential-owning host for the authenticated bridge. + // Send only sentinel, baseUrl, model, and public caBundle to the remote app. + await runRemoteAppThroughBridge(egress, signal); +}); +``` + +The callback receives `sentinel`, `baseUrl`, `model`, `allowedHosts`, `hostEnv`, +and the public `caBundle` contents. Supply `onOutput(text, stream)` to receive +live output with resolved credentials redacted, including values split between +chunks; pass the callback's `onOutputChunk` to the command runner. The caller owns its bridge and remote +process, must honor cancellation, and must join both before its callback +settles. Cancellation revokes proxy access immediately; callback completion or +failure revokes access, stops the isolated proxy, and removes its private CA +directory. Credentials and proxy grants never enter the shared secret store. + +This is an explicit command-scoped proxy using the existing secret egress +implementation. It does not require enabling or restarting the Gateway's +persistent egress proxy. Import the SDK module only in the command execution +path; importing it alone does not load model or secret runtime code. + ## Prepared simple completions The `openclaw/plugin-sdk/simple-completion-runtime` helpers support preparing a diff --git a/docs/plugins/sdk-subpaths.md b/docs/plugins/sdk-subpaths.md index 6517f929d3f5..6cf51ec3c8ad 100644 --- a/docs/plugins/sdk-subpaths.md +++ b/docs/plugins/sdk-subpaths.md @@ -268,6 +268,7 @@ usage endpoint failed or returned no usable usage data. | `plugin-sdk/channel-secret-basic-runtime` | Narrow secret-contract exports and target-registry builders for non-TTS channel/plugin secret surfaces | | `plugin-sdk/channel-secret-tts-runtime` | Private-local after July 2026; Narrow nested channel TTS secret assignment helpers | | `plugin-sdk/secret-ref-runtime` | Narrow SecretRef typing, resolution, setup-plan construction, and setup CLI scaffolding for plugin-owned secret providers | + | `plugin-sdk/secret-egress-runtime` | Private official-plugin runtime; command-scoped protected model credentials, public CA transfer, and revocable HTTPS egress through `withConfiguredModelEgress`; not a third-party plugin API | | `plugin-sdk/security-runtime` | Deprecated broad barrel for trust, DM gating, root-bounded file/path helpers including create-only writes, sync/async atomic file replacement, sibling temp writes, cross-device move fallback, private file-store helpers, symlink-parent guards, external-content, `redactSensitiveText`, constant-time secret comparison, and secret-collection helpers; prefer focused security/SSRF/secret subpaths | | `plugin-sdk/ssrf-policy` | Host allowlist and private-network SSRF policy helpers | | `plugin-sdk/ssrf-dispatcher` | Private-local after July 2026; Narrow pinned-dispatcher helpers without the broad infra runtime surface | diff --git a/extensions/crabbox/index.ts b/extensions/crabbox/index.ts index 064c2d9ad105..25ae5c68fbd1 100644 --- a/extensions/crabbox/index.ts +++ b/extensions/crabbox/index.ts @@ -33,16 +33,18 @@ export default definePluginEntry({ tags: ["cloud", "desktop"], }); api.registerCli( - async ({ program }) => { + async ({ program, config }) => { const { registerCrabboxWarmImageCommands } = await import("./src/crabbox-worker-warm-image-cli.js"); registerCrabboxWarmImageCommands(program, api.runtime.state); + const { registerCrabboxModelRunCommand } = await import("./src/crabbox-model-run-cli.js"); + registerCrabboxModelRunCommand({ program, config }); }, { descriptors: [ { name: "crabbox", - description: "Inspect and recover Crabbox warm images", + description: "Run model-backed commands and manage Crabbox warm images", hasSubcommands: true, }, ], diff --git a/extensions/crabbox/openclaw.plugin.json b/extensions/crabbox/openclaw.plugin.json index 306ee574743b..e4a6e56a6b29 100644 --- a/extensions/crabbox/openclaw.plugin.json +++ b/extensions/crabbox/openclaw.plugin.json @@ -9,7 +9,7 @@ "cliCommands": [ { "name": "crabbox", - "description": "Inspect and recover Crabbox warm images", + "description": "Run model-backed commands and manage Crabbox warm images", "hasSubcommands": true } ], diff --git a/extensions/crabbox/skills/crabbox-apps/SKILL.md b/extensions/crabbox/skills/crabbox-apps/SKILL.md index 69f2b613470c..f78f875c93a9 100644 --- a/extensions/crabbox/skills/crabbox-apps/SKILL.md +++ b/extensions/crabbox/skills/crabbox-apps/SKILL.md @@ -52,6 +52,39 @@ capability and the actual completed state. A portal that the user's browser cannot reach is not a completed preview. Preserve the separate-origin portal transport; never expose arbitrary application scripts on the Gateway origin. +## Apps that call a model API + +Cloud-agent inference already uses Gateway-held authentication. An application +inside the lease making its own API calls needs a separate protected route. +For an exclusively owned coordinator-backed Linux lease, use the host CLI: + +```sh +openclaw crabbox run --id --model -- +``` + +Run from the credential-owning host and the local project directory that owns +the lease. Prepare source and dependencies first: the command skips sync and +hydration, and its bridge permits only the model host. A lease ID does not +override Crabbox's repository claim. Require no active egress session and a +Crabbox binary supporting `egress run --upstream-proxy-env`. Crabbox owns the +foreground bridge, remote command, and session cleanup. + +The provider must have a configured API-key SecretRef and an OpenAI-compatible +HTTPS endpoint on port 443. Auth-profile/OAuth credentials, custom headers, and +request transport overrides are unsupported. The CLI injects a sentinel as +`OPENAI_API_KEY`, plus `OPENAI_BASE_URL`, `OPENAI_MODEL`, proxy settings, and +public CA trust. The actual key and upstream proxy credentials stay on the host. +Use a client that honors the proxy and CA environment; Node.js needs +`NODE_USE_ENV_PROXY` support, while Python's default `urllib.request` opener +honors the environment. Custom SDK clients may need explicit proxy/trust setup. + +Keep this command alive for the full app lifetime. Do not detach the app or +copy a key into the box. Ordinary tool `exec` and `background` do not acquire +this model grant. Cancellation revokes access before cleanup; if settlement is +uncertain, inspect the named session and remote process before reusing the lease. +This path needs no Gateway restart or persistent egress setting. See the +[setup, runnable example, and recovery guide](https://docs.openclaw.ai/gateway/secrets/secret-store-and-egress#model-credentials-for-crabbox-commands). + ## Follow-ups and cleanup Reopen the current environment or portal for "show me again". A viewer reconnect diff --git a/extensions/crabbox/src/crabbox-model-run-cli.ts b/extensions/crabbox/src/crabbox-model-run-cli.ts new file mode 100644 index 000000000000..d0eef933ddd2 --- /dev/null +++ b/extensions/crabbox/src/crabbox-model-run-cli.ts @@ -0,0 +1,65 @@ +import type { OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry"; +import { resolveCrabboxBinary } from "./crabbox-binary.js"; +import { runCrabboxModelCommand } from "./crabbox-model-run.js"; + +type CliContext = Parameters[0]>[0]; + +export function registerCrabboxModelRunCommand({ + program, + config, +}: Pick): void { + const crabbox = + program.commands.find((command) => command.name() === "crabbox") ?? program.command("crabbox"); + crabbox.description("Run model-backed commands and manage Crabbox warm images"); + crabbox + .command("run") + .description( + "Run a foreground command on an exclusive Linux lease using a protected model credential", + ) + .requiredOption("--id ", "Existing, exclusively owned Crabbox lease") + .requiredOption("--model ", "Configured OpenAI-compatible API-key model") + .option("--provider ", "Crabbox backend override; otherwise use the lease's provider") + .option("--binary ", "Crabbox binary with upstream proxy support") + .option("--timeout ", "Total command and setup deadline in seconds", "600") + .argument("", "Remote executable and arguments, following --") + .action( + async ( + argv: string[], + options: { + id: string; + model: string; + provider?: string; + binary?: string; + timeout: string; + }, + ) => { + const seconds = Number(options.timeout); + if (!Number.isSafeInteger(seconds) || seconds < 1 || seconds > 86_400) { + throw new Error("--timeout must be an integer between 1 and 86400 seconds"); + } + const controller = new AbortController(); + const cancel = () => controller.abort(new Error("Crabbox model command interrupted")); + process.once("SIGINT", cancel); + process.once("SIGTERM", cancel); + try { + const result = await runCrabboxModelCommand({ + config, + binary: resolveCrabboxBinary({ explicit: options.binary, pathEnv: process.env.PATH }), + id: options.id, + model: options.model, + provider: options.provider, + argv, + timeoutMs: seconds * 1000, + signal: controller.signal, + onOutput: (text, stream) => { + process[stream].write(text); + }, + }); + process.exitCode = result.termination === "exit" ? (result.code ?? 1) : 1; + } finally { + process.off("SIGINT", cancel); + process.off("SIGTERM", cancel); + } + }, + ); +} diff --git a/extensions/crabbox/src/crabbox-model-run.test.ts b/extensions/crabbox/src/crabbox-model-run.test.ts new file mode 100644 index 000000000000..f367b28e9f6a --- /dev/null +++ b/extensions/crabbox/src/crabbox-model-run.test.ts @@ -0,0 +1,122 @@ +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import type { CommandOptions, SpawnResult } from "openclaw/plugin-sdk/process-runtime"; +import type { ConfiguredModelEgress } from "openclaw/plugin-sdk/secret-egress-runtime"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ run: vi.fn(), withEgress: vi.fn() })); +vi.mock("openclaw/plugin-sdk/process-runtime", () => ({ runCommandWithTimeout: mocks.run })); +vi.mock("openclaw/plugin-sdk/secret-egress-runtime", () => ({ + withConfiguredModelEgress: mocks.withEgress, +})); +const { runCrabboxModelCommand } = await import("./crabbox-model-run.js"); +const execFileAsync = promisify(execFile); +const capabilityHelp = " -upstream-proxy-env string\n"; +const egress: ConfiguredModelEgress = { + sentinel: "oc-sent-v2.synthetic-placeholder.end", + baseUrl: "https://api.example.com/v1/", + model: "example-model", + allowedHosts: ["api.example.com"], + hostEnv: { HTTPS_PROXY: "http://openclaw:proxy-password-fixture@127.0.0.1:43210" }, + caBundle: "public-ca-fixture\n", +}; +const options = { + config: {}, + binary: "crabbox-fixture", + id: "cbx_owned", + model: "example/example-model", + argv: ["node", "test-app.js"], + timeoutMs: 60_000, +}; +function result(stdout = "", code = 0): SpawnResult { + return { stdout, stderr: "", code, signal: null, killed: false, termination: "exit" }; +} +beforeEach(() => { + mocks.run.mockReset(); + mocks.withEgress.mockReset(); + mocks.withEgress.mockImplementation(async (_options, run) => await run(egress)); +}); + +describe("Crabbox protected model command", () => { + it("delegates one native job with a host-only proxy grant and sentinel-only app environment", async () => { + let remoteInput = ""; + mocks.run.mockImplementation(async (argv: string[], params: CommandOptions) => { + if (argv.includes("--help")) { + return result(capabilityHelp); + } + expect(argv).toEqual([ + "crabbox-fixture", + "egress", + "run", + "--id", + "cbx_owned", + "--allow", + "api.example.com", + "--upstream-proxy-env", + "CRABBOX_MODEL_PROXY", + "--no-sync", + "--no-hydrate", + "--script-stdin", + "--", + "node", + "test-app.js", + ]); + expect(params.env?.CRABBOX_MODEL_PROXY).toBe(egress.hostEnv.HTTPS_PROXY); + expect(params.env?.CRABBOX_ENV_ALLOW).toBe(","); + expect(params.killGraceMs).toBeGreaterThan(60_000); + expect(argv.join(" ")).not.toContain("proxy-password-fixture"); + remoteInput = String(params.input); + expect(remoteInput).not.toContain("proxy-password-fixture"); + return result("app finished\n", 7); + }); + expect(await runCrabboxModelCommand(options)).toMatchObject({ + code: 7, + stdout: "app finished\n", + }); + expect(mocks.run).toHaveBeenCalledTimes(2); + + const fixture = `const fs = require('fs'); console.log(JSON.stringify({ + key: process.env.OPENAI_API_KEY, baseUrl: process.env.OPENAI_BASE_URL, + model: process.env.OPENAI_MODEL, proxy: process.env.HTTPS_PROXY, + ca: fs.readFileSync(process.env.SSL_CERT_FILE, 'utf8'), path: process.env.SSL_CERT_FILE + }));`; + const child = execFile("bash", ["-s", "--", process.execPath, "-e", fixture], { + env: { ...process.env, NODE_OPTIONS: undefined }, + }); + const completion = new Promise((resolve, reject) => { + let output = ""; + child.stdout?.on("data", (chunk) => { + output += String(chunk); + }); + child.once("error", reject); + child.once("exit", (code) => + code === 0 ? resolve(output) : reject(new Error(`fixture exited ${code}`)), + ); + }); + child.stdin!.end(remoteInput); + const observed = JSON.parse(await completion); + expect(observed).toMatchObject({ + key: egress.sentinel, + baseUrl: egress.baseUrl, + model: egress.model, + proxy: "http://127.0.0.1:3128", + ca: `${egress.caBundle}\n`, + }); + await expect(execFileAsync("test", ["-e", observed.path])).rejects.toThrow(); + }); + + it.each(["", " --upstream-proxy-env-name string\n per-upstream-proxy-env help"])( + "refuses incompatible native commands before credential access", + async (help) => { + mocks.run.mockResolvedValue(result(help)); + await expect(runCrabboxModelCommand(options)).rejects.toThrow("lacks native egress run"); + expect(mocks.withEgress).not.toHaveBeenCalled(); + }, + ); + + it("reports unconfirmed native process cleanup as failure", async () => { + mocks.run.mockResolvedValueOnce(result(capabilityHelp)); + mocks.run.mockResolvedValueOnce({ ...result(), cleanup: "uncertain" }); + await expect(runCrabboxModelCommand(options)).rejects.toThrow("could not confirm"); + }); +}); diff --git a/extensions/crabbox/src/crabbox-model-run.ts b/extensions/crabbox/src/crabbox-model-run.ts new file mode 100644 index 000000000000..fa0c8c4a8ad1 --- /dev/null +++ b/extensions/crabbox/src/crabbox-model-run.ts @@ -0,0 +1,128 @@ +import { randomBytes } from "node:crypto"; +import { redactSensitiveText } from "openclaw/plugin-sdk/logging-core"; +import { parseModelRef } from "openclaw/plugin-sdk/model-ref-parse"; +import type { OpenClawConfig } from "openclaw/plugin-sdk/plugin-entry"; +import { + runCommandWithTimeout, + type CommandOptions, + type SpawnResult, +} from "openclaw/plugin-sdk/process-runtime"; +import { + withConfiguredModelEgress, + type ConfiguredModelEgress, +} from "openclaw/plugin-sdk/secret-egress-runtime"; + +const UPSTREAM_PROXY_ENV = "CRABBOX_MODEL_PROXY"; +const MAX_OUTPUT_BYTES = 1024 * 1024; + +export type CrabboxModelRunOptions = { + config: OpenClawConfig; + binary: string; + id: string; + model: string; + provider?: string; + argv: string[]; + timeoutMs: number; + signal?: AbortSignal; + onOutput?: (text: string, stream: "stdout" | "stderr") => void; +}; + +function shellQuote(value: string): string { + if (value.includes("\0")) { + throw new Error("Crabbox model command cannot contain NUL bytes"); + } + return `'${value.replaceAll("'", `'"'"'`)}'`; +} + +function remoteCommandScript(egress: ConfiguredModelEgress): string { + const delimiter = `MODEL_EGRESS_CA_${randomBytes(12).toString("hex")}`; + return `#!/bin/bash +set -euo pipefail +umask 077 +egress_dir=$(mktemp -d) +trap 'rm -rf -- "$egress_dir"' EXIT +cat > "$egress_dir/ca.pem" <<'${delimiter}' +${egress.caBundle} +${delimiter} +export OPENAI_API_KEY=${shellQuote(egress.sentinel)} +export OPENAI_BASE_URL=${shellQuote(egress.baseUrl)} +export OPENAI_MODEL=${shellQuote(egress.model)} +export HTTPS_PROXY=http://127.0.0.1:3128 HTTP_PROXY=http://127.0.0.1:3128 +export https_proxy="$HTTPS_PROXY" http_proxy="$HTTP_PROXY" +export NO_PROXY=localhost,127.0.0.1,::1 no_proxy=localhost,127.0.0.1,::1 +unset ALL_PROXY all_proxy +export NODE_USE_ENV_PROXY=1 +export NODE_EXTRA_CA_CERTS="$egress_dir/ca.pem" SSL_CERT_FILE="$egress_dir/ca.pem" +export CURL_CA_BUNDLE="$egress_dir/ca.pem" REQUESTS_CA_BUNDLE="$egress_dir/ca.pem" +export GIT_SSL_CAINFO="$egress_dir/ca.pem" +"$@" +`; +} + +export async function runCrabboxModelCommand(params: CrabboxModelRunOptions): Promise { + const model = parseModelRef(params.model, ""); + if (!params.model.includes("/") || !model?.provider || !model.model) { + throw new Error("--model must name an explicit provider/model"); + } + const signal = AbortSignal.any([ + AbortSignal.timeout(params.timeoutMs), + ...(params.signal ? [params.signal] : []), + ]); + const command = ( + args: string[], + options: Pick = {}, + ) => + runCommandWithTimeout([params.binary, "egress", "run", ...args], { + timeoutMs: params.timeoutMs, + maxOutputBytes: MAX_OUTPUT_BYTES, + killProcessTree: true, + requireProcessTreeExtinction: true, + // Native egress owns remote cleanup after SIGTERM; credential use ends immediately. + killGraceMs: 75_000, + signal, + ...options, + env: { CRABBOX_ENV_ALLOW: ",", [UPSTREAM_PROXY_ENV]: undefined, ...options.env }, + }); + const help = await command(["--help"]); + if ( + help.termination !== "exit" || + help.code !== 0 || + !/(?:^|\n)\s+-{1,2}upstream-proxy-env(?:\s|$)/u.test(`${help.stdout}\n${help.stderr}`) + ) { + throw new Error("This Crabbox binary lacks native egress run support; update Crabbox first"); + } + return await withConfiguredModelEgress( + { config: params.config, ...model, signal, onOutput: params.onOutput }, + async (egress) => { + const result = await command( + [ + "--id", + params.id, + ...(params.provider ? ["--provider", params.provider] : []), + "--allow", + egress.allowedHosts.join(","), + "--upstream-proxy-env", + UPSTREAM_PROXY_ENV, + "--no-sync", + "--no-hydrate", + "--script-stdin", + "--", + ...params.argv, + ], + { + env: { [UPSTREAM_PROXY_ENV]: egress.hostEnv.HTTPS_PROXY }, + input: remoteCommandScript(egress), + onOutputChunk: egress.onOutputChunk, + }, + ); + if (result.cleanup === "uncertain") { + throw new Error("Crabbox could not confirm that the command stopped"); + } + return { + ...result, + stdout: redactSensitiveText(result.stdout), + stderr: redactSensitiveText(result.stderr), + }; + }, + ); +} diff --git a/extensions/tsconfig.package-boundary.paths.json b/extensions/tsconfig.package-boundary.paths.json index 03a0b3fc5f93..910800096f5a 100644 --- a/extensions/tsconfig.package-boundary.paths.json +++ b/extensions/tsconfig.package-boundary.paths.json @@ -954,6 +954,9 @@ ], "openclaw/plugin-sdk/agent-harness-attempt-runtime": [ "../packages/plugin-sdk/dist/src/plugin-sdk/agent-harness-attempt-runtime.d.ts" + ], + "openclaw/plugin-sdk/secret-egress-runtime": [ + "../packages/plugin-sdk/dist/src/plugin-sdk/secret-egress-runtime.d.ts" ] } } diff --git a/extensions/xai/tsconfig.json b/extensions/xai/tsconfig.json index 20e85678121c..2d9086dfe1e6 100644 --- a/extensions/xai/tsconfig.json +++ b/extensions/xai/tsconfig.json @@ -938,6 +938,9 @@ ], "openclaw/plugin-sdk/agent-harness-attempt-runtime": [ "../../packages/plugin-sdk/dist/src/plugin-sdk/agent-harness-attempt-runtime.d.ts" + ], + "openclaw/plugin-sdk/secret-egress-runtime": [ + "../../packages/plugin-sdk/dist/src/plugin-sdk/secret-egress-runtime.d.ts" ] } } diff --git a/package.json b/package.json index 0299ae4effaa..f90c78733ac3 100644 --- a/package.json +++ b/package.json @@ -439,7 +439,8 @@ "!dist/plugin-sdk/realtime-voice-playback.d.ts", "!dist/plugin-sdk/text-grapheme.d.ts", "!dist/plugin-sdk/agent-harness-session-runtime.d.ts", - "!dist/plugin-sdk/agent-harness-attempt-runtime.d.ts" + "!dist/plugin-sdk/agent-harness-attempt-runtime.d.ts", + "!dist/plugin-sdk/secret-egress-runtime.d.ts" ], "type": "module", "main": "dist/index.js", @@ -847,6 +848,9 @@ "types": "./dist/plugin-sdk/secret-ref-runtime.d.ts", "default": "./dist/plugin-sdk/secret-ref-runtime.js" }, + "./plugin-sdk/secret-egress-runtime": { + "default": "./dist/plugin-sdk/secret-egress-runtime.js" + }, "./plugin-sdk/secret-file-runtime": { "default": "./dist/plugin-sdk/secret-file-runtime.js" }, diff --git a/scripts/lib/plugin-sdk-entrypoints.json b/scripts/lib/plugin-sdk-entrypoints.json index 470b34ff2dbd..1905c6e1b8bd 100644 --- a/scripts/lib/plugin-sdk-entrypoints.json +++ b/scripts/lib/plugin-sdk-entrypoints.json @@ -113,6 +113,7 @@ "channel-secret-owner-runtime", "channel-secret-tts-runtime", "secret-ref-runtime", + "secret-egress-runtime", "secret-file-runtime", "security-runtime", "gateway-config-runtime", diff --git a/scripts/lib/plugin-sdk-private-local-only-subpaths.json b/scripts/lib/plugin-sdk-private-local-only-subpaths.json index f8cdd0d98e59..875910bfea2c 100644 --- a/scripts/lib/plugin-sdk-private-local-only-subpaths.json +++ b/scripts/lib/plugin-sdk-private-local-only-subpaths.json @@ -168,6 +168,7 @@ "runtime-doctor-migrations", "runtime-fetch", "sandbox", + "secret-egress-runtime", "secret-file-runtime", "secret-provider-alias", "secure-random-runtime", diff --git a/src/fleet/containers.redaction.ts b/src/fleet/containers.redaction.ts deleted file mode 100644 index 432777e7c5d8..000000000000 --- a/src/fleet/containers.redaction.ts +++ /dev/null @@ -1,64 +0,0 @@ -// Fleet container stream redaction preserves chunked output without leaking split secrets. -import { StringDecoder } from "node:string_decoder"; - -// Longest suffix of `text` that is a proper prefix of any secret. Retaining it -// across emissions guarantees no complete secret is ever split between two -// emitted chunks, and emitted text can never grow into a match later. -function secretPrefixSuffixLength(text: string, redactValues: readonly string[]): number { - let longest = 0; - for (const value of redactValues) { - const max = Math.min(value.length - 1, text.length); - for (let length = max; length > longest; length -= 1) { - if (value.startsWith(text.slice(text.length - length))) { - longest = length; - break; - } - } - } - return longest; -} - -export function createRedactingStreamWriter( - target: NodeJS.WriteStream, - redactValues: readonly string[], -): { write: (chunk: Buffer) => boolean; flush: () => void } { - const decoder = new StringDecoder("utf8"); - let pending = ""; - const redact = (text: string): string => { - let redacted = text; - for (const value of redactValues) { - if (value) { - redacted = redacted.replaceAll(value, ""); - } - } - return redacted; - }; - // Returns the raw target.write() backpressure signal so callers can pause - // the child stream instead of buffering a noisy follow stream without bound. - const emit = (text: string): boolean => { - if (!text) { - return true; - } - return target.write(redact(text)); - }; - return { - // Emit everything except a possible secret prefix at the tail on every - // chunk, so unterminated output (progress lines, prompts) streams live - // instead of stalling until a newline arrives. - write: (chunk) => { - pending += decoder.write(chunk); - const keep = secretPrefixSuffixLength(pending, redactValues); - const cut = pending.length - keep; - if (cut <= 0) { - return true; - } - const writable = emit(pending.slice(0, cut)); - pending = pending.slice(cut); - return writable; - }, - flush: () => { - emit(pending + decoder.end()); - pending = ""; - }, - }; -} diff --git a/src/fleet/containers.runtime.test.ts b/src/fleet/containers.runtime.test.ts index 61139ae16bf2..1420c04722eb 100644 --- a/src/fleet/containers.runtime.test.ts +++ b/src/fleet/containers.runtime.test.ts @@ -29,7 +29,6 @@ const profileMocks = vi.hoisted(() => ({ vi.mock("./cell-profile.js", () => profileMocks); import type { CellContainerProfile } from "./cell-profile.js"; -import { createRedactingStreamWriter } from "./containers.redaction.js"; import { createFleetContainerRuntime } from "./containers.runtime.js"; type FleetContainerCommandExecutor = NonNullable[0]>; @@ -599,32 +598,6 @@ describe("fleet container runtime", () => { expect(executor.mock.calls[0]?.[1].includes("--internal")).toBe(expected); }); - it("redacts secrets from streamed log output, including across chunk boundaries", () => { - const written: string[] = []; - const target = { write: (text: string) => written.push(text) } as unknown as NodeJS.WriteStream; - const writer = createRedactingStreamWriter(target, ["gw-secret-token"]); - writer.write(Buffer.from("boot ok\ntoken=gw-sec")); - writer.write(Buffer.from("ret-token done\ntail without newline")); - writer.flush(); - const output = written.join(""); - expect(output).toContain("token= done"); - expect(output).toContain("tail without newline"); - expect(output).not.toContain("gw-secret-token"); - }); - - it("never splits a secret across a forced long-line flush", () => { - const written: string[] = []; - const target = { write: (text: string) => written.push(text) } as unknown as NodeJS.WriteStream; - const writer = createRedactingStreamWriter(target, ["gw-secret-token"]); - // An unterminated line ending exactly in a secret prefix at the flush point. - writer.write(Buffer.from(`${"x".repeat(64 * 1024)}gw-sec`)); - writer.write(Buffer.from("ret-token trailing")); - writer.flush(); - const output = written.join(""); - expect(output).toContain(" trailing"); - expect(output).not.toContain("gw-secret-token"); - }); - it("parses hardened inspect fields and Docker network internal state", async () => { const executor = vi.fn(async (_runtime, args) => args[0] === "network" diff --git a/src/fleet/containers.runtime.ts b/src/fleet/containers.runtime.ts index 49eb23e0fa17..0bacb1cf387e 100644 --- a/src/fleet/containers.runtime.ts +++ b/src/fleet/containers.runtime.ts @@ -1,6 +1,7 @@ import { spawn } from "node:child_process"; import { isRecord, isStringRecord } from "@openclaw/normalization-core/record-coerce"; import { withContainerEnvFile } from "../infra/container-env-file.js"; +import { createRedactingStreamWriter } from "../logging/redacting-stream.js"; import { attachChildProcessBridge } from "../process/child-process-bridge.js"; import { runCommandWithTimeout } from "../process/exec.js"; import { @@ -11,7 +12,6 @@ import { type CellContainerProfile, type FleetContainerRuntimeName, } from "./cell-profile.js"; -import { createRedactingStreamWriter } from "./containers.redaction.js"; type FleetContainerCommandOptions = { allowFailure?: boolean; diff --git a/src/logging/redacting-stream.test.ts b/src/logging/redacting-stream.test.ts new file mode 100644 index 000000000000..75cde09f3d1c --- /dev/null +++ b/src/logging/redacting-stream.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it, vi } from "vitest"; +import { createRedactingStreamWriter } from "./redacting-stream.js"; + +describe("createRedactingStreamWriter", () => { + it.each([ + { values: ["abcabc"], input: "abcabc", expected: "" }, + { values: ["abcabc"], input: "abcabcabcabc", expected: "" }, + { values: ["abc", "abcdef"], input: "abcdef abc!", expected: " !" }, + { values: ["bc", "abcdef"], input: "abcdef abc", expected: " a" }, + { values: ["abcabc", "redact"], input: "abcabc redact", expected: " " }, + { values: ["", "🦞密🦞"], input: "🦞密🦞 🦞!", expected: " 🦞!" }, + { values: ["gw-secret-token"], input: "partial gw-sec", expected: "partial gw-sec" }, + { values: [], input: "progress 🦞", expected: "progress 🦞" }, + ])("redacts $input independently of byte chunk boundaries", ({ values, input, expected }) => { + const bytes = Buffer.from(input); + const partitions = Array.from({ length: bytes.length + 1 }, (_, index) => [ + bytes.subarray(0, index), + bytes.subarray(index), + ]); + partitions.push(Array.from(bytes, (byte) => Buffer.from([byte]))); + for (const chunks of partitions) { + let output = ""; + const writer = createRedactingStreamWriter( + { + write: (text) => { + output += text; + return true; + }, + }, + values, + ); + for (const chunk of chunks) { + writer.write(chunk); + } + writer.flush(); + expect(output).toBe(expected); + } + }); + + it.each(["boot ok\ntoken=", "x".repeat(64 * 1024)])( + "streams unterminated progress with bounded carry and forwards backpressure (%#)", + (prefix) => { + const write = vi + .fn<(text: string) => boolean>() + .mockReturnValueOnce(false) + .mockReturnValue(true); + const writer = createRedactingStreamWriter({ write }, ["gw-secret-token"]); + expect(writer.write(Buffer.from(`${prefix}gw-sec`))).toBe(false); + expect(write.mock.calls).toEqual([[prefix]]); + expect(writer.write(Buffer.from("ret-token done\ntail without newline"))).toBe(true); + expect(write.mock.calls).toEqual([[prefix], [" done\ntail without newline"]]); + writer.flush(); + expect(write).toHaveBeenCalledTimes(2); + }, + ); + + it("holds an ambiguous secret prefix until the match is complete", () => { + const write = vi.fn<(text: string) => boolean>().mockReturnValue(false); + const writer = createRedactingStreamWriter({ write }, ["abc", "abcabc"]); + expect(writer.write(Buffer.from("abc"))).toBe(true); + expect(write).not.toHaveBeenCalled(); + expect(writer.write(Buffer.from("abc"))).toBe(false); + expect(write.mock.calls).toEqual([[""]]); + writer.flush(); + expect(write).toHaveBeenCalledTimes(1); + }); +}); diff --git a/src/logging/redacting-stream.ts b/src/logging/redacting-stream.ts new file mode 100644 index 000000000000..6d6fc4e91041 --- /dev/null +++ b/src/logging/redacting-stream.ts @@ -0,0 +1,48 @@ +import { StringDecoder } from "node:string_decoder"; + +export function createRedactingStreamWriter( + target: { write(text: string): boolean }, + redactValues: readonly string[], +): { write: (chunk: Buffer) => boolean; flush: () => void } { + const decoder = new StringDecoder("utf8"); + const values = redactValues.filter(Boolean).toSorted((left, right) => right.length - left.length); + const firstCharacters = new Set(values.map((value) => value.charAt(0))); + let pending = ""; + const emit = (ending: boolean): boolean => { + let output = ""; + let cursor = 0; + while (cursor < pending.length) { + const value = firstCharacters.has(pending.charAt(cursor)) + ? values.find( + (candidate) => + pending.startsWith(candidate, cursor) || + (!ending && + pending.length - cursor < candidate.length && + candidate.startsWith(pending.slice(cursor))), + ) + : undefined; + if (!value) { + output += pending[cursor]; + cursor += 1; + } else if (cursor + value.length > pending.length) { + // A longer possible match owns the tail until it completes or the stream ends. + break; + } else { + output += ""; + cursor += value.length; + } + } + pending = pending.slice(cursor); + return !output || target.write(output); + }; + return { + write: (chunk) => { + pending += decoder.write(chunk); + return emit(false); + }, + flush: () => { + pending += decoder.end(); + emit(true); + }, + }; +} diff --git a/src/plugin-sdk/secret-egress-runtime.ts b/src/plugin-sdk/secret-egress-runtime.ts new file mode 100644 index 000000000000..0a9e9916dc66 --- /dev/null +++ b/src/plugin-sdk/secret-egress-runtime.ts @@ -0,0 +1,17 @@ +import type { + ConfiguredModelEgress, + ConfiguredModelEgressOptions, +} from "../secrets/model-egress.js"; +import { createLazyRuntimeModule } from "../shared/lazy-runtime.js"; + +export type { ConfiguredModelEgress, ConfiguredModelEgressOptions }; + +const loadModelEgressRuntime = createLazyRuntimeModule(() => import("../secrets/model-egress.js")); + +/** Private official-plugin runtime for a standalone job's protected model credential. */ +export async function withConfiguredModelEgress( + options: ConfiguredModelEgressOptions, + run: (egress: ConfiguredModelEgress) => Promise, +): Promise { + return (await loadModelEgressRuntime()).withConfiguredModelEgress(options, run); +} diff --git a/src/secrets/model-egress.test.ts b/src/secrets/model-egress.test.ts new file mode 100644 index 000000000000..3233ce231e00 --- /dev/null +++ b/src/secrets/model-egress.test.ts @@ -0,0 +1,203 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { useAutoCleanupTempDirTracker } from "../../test/helpers/temp-dir.js"; +import type { OpenClawConfig } from "../config/types.openclaw.js"; +import { createDeferredCore } from "../shared/deferred.js"; +import type { SecretEgressProxyHandle } from "./egress-proxy/proxy-server.js"; +import { withConfiguredModelEgress } from "./model-egress.js"; +import { looksLikeSecretSentinel, resolveSecretSentinel } from "./sentinel.js"; + +const startProxy = vi.hoisted(() => + vi.fn(), +); +vi.mock("./egress-proxy/proxy-server.js", () => ({ + startSecretEgressProxyServer: startProxy, +})); + +const tempDirs = useAutoCleanupTempDirTracker(afterEach); +const credential = "synthetic-model-egress-api-key"; +const publicCa = "synthetic-public-ca-bundle"; +let config: OpenClawConfig; +let proxy: SecretEgressProxyHandle; +const revoke = vi.fn(); + +beforeEach(async () => { + vi.clearAllMocks(); + vi.stubEnv("OPENAI_BASE_URL", undefined); + vi.stubEnv("OPENCLAW_SECRET_SENTINELS", "off"); + const dir = tempDirs.make("model-egress-test-"); + const secretPath = path.join(dir, "key.txt"); + const caPath = path.join(dir, "ca.pem"); + await fs.writeFile(secretPath, credential, { mode: 0o600 }); + await fs.writeFile(caPath, publicCa); + config = { + secrets: { + providers: { model_key: { source: "file", path: secretPath, mode: "singleValue" } }, + }, + models: { + providers: { + openai: { + apiKey: { source: "file", provider: "model_key", id: "value" }, + baseUrl: "", + models: [], + }, + }, + }, + }; + proxy = { + caCertPath: caPath, + proxyOrigin: "http://127.0.0.1:12345", + getCertificateStatus: () => ({ + state: "ready", + caExpiresAt: "2030-01-01", + failedCertificates: 0, + }), + registerProcess: vi.fn(() => ({ + env: { + HTTPS_PROXY: "http://openclaw:synthetic-proxy-token@127.0.0.1:12345", + NODE_EXTRA_CA_CERTS: caPath, + }, + revoke, + })), + stop: vi.fn(async () => {}), + }; + startProxy.mockResolvedValue(proxy); +}); + +afterEach(() => vi.unstubAllEnvs()); + +function runOptions() { + return { config, provider: "openai", model: "gpt-5.5" }; +} + +describe("configured model egress", () => { + it("uses the provider-owned default route and seals a file-backed credential only for its host", async () => { + // Authored provider config can omit its default transport, even though materialized config has one. + Reflect.deleteProperty(config.models!.providers!.openai!, "baseUrl"); + const result = await withConfiguredModelEgress(runOptions(), async (egress) => { + expect(egress).toMatchObject({ + baseUrl: "https://api.openai.com/v1", + model: "gpt-5.5", + allowedHosts: ["api.openai.com"], + caBundle: publicCa, + }); + expect(looksLikeSecretSentinel(egress.sentinel)).toBe(true); + expect(resolveSecretSentinel(egress.sentinel)).toBe(credential); + expect(JSON.stringify(egress)).not.toContain(credential); + expect(proxy.registerProcess).toHaveBeenCalledWith([ + { + name: "openai model API key", + sentinel: egress.sentinel, + allowedHosts: ["api.openai.com"], + }, + ]); + expect(revoke).not.toHaveBeenCalled(); + return 17; + }); + expect(result).toBe(17); + expect(revoke).toHaveBeenCalled(); + expect(proxy.stop).toHaveBeenCalledOnce(); + const options = startProxy.mock.calls[0]![0]; + expect(options.allowedHosts).toEqual(["api.openai.com"]); + await expect(fs.stat(options.caDir)).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it("binds a configured compatible endpoint instead of the provider default", async () => { + config.models!.providers!.openai!.baseUrl = "https://inference.example.test/tenant/v1"; + await withConfiguredModelEgress(runOptions(), async (egress) => { + expect(egress.baseUrl).toBe("https://inference.example.test/tenant/v1"); + expect(egress.allowedHosts).toEqual(["inference.example.test"]); + }); + }); + + it("streams progress before completion and redacts credentials split between chunks", async () => { + const stdout: string[] = []; + const stderr: string[] = []; + await withConfiguredModelEgress( + { + ...runOptions(), + onOutput: (text, stream) => (stream === "stdout" ? stdout : stderr).push(text), + }, + async (egress) => { + egress.onOutputChunk!(Buffer.from("ready "), "stdout"); + expect(stdout.join("")).toBe("ready "); + egress.onOutputChunk!(Buffer.from(credential.slice(0, 8)), "stdout"); + egress.onOutputChunk!(Buffer.from(credential.slice(8)), "stdout"); + egress.onOutputChunk!(Buffer.from("diagnostic"), "stderr"); + expect(stdout.join("")).toBe("ready "); + expect(stderr.join("")).toBe("diagnostic"); + }, + ); + expect(stdout.join("")).not.toContain(credential); + }); + + it.each([ + ["literal or profile credential", { apiKey: "profile-id" }], + ["OAuth authentication", { auth: "oauth" }], + ["custom headers", { headers: { Authorization: "synthetic-header" } }], + ["custom request transport", { request: { proxy: { url: "http://proxy.example.test" } } }], + ["unencrypted endpoint", { baseUrl: "http://api.openai.com/v1" }], + ["unsupported endpoint port", { baseUrl: "https://inference.example.test:8443/v1" }], + ["credential-bearing endpoint", { baseUrl: "https://user:password@inference.example.test/v1" }], + ["subscription route", { api: "openai-chatgpt-responses" }], + ])("rejects %s before starting a proxy", async (_label, override) => { + Object.assign(config.models!.providers!.openai!, override); + const run = vi.fn(); + await expect(withConfiguredModelEgress(runOptions(), run)).rejects.toThrow(); + expect(startProxy).not.toHaveBeenCalled(); + expect(run).not.toHaveBeenCalled(); + }); + + it("revokes synchronously on cancellation while the remote job is still settling", async () => { + const controller = new AbortController(); + const entered = createDeferredCore(); + const settle = createDeferredCore(); + const running = withConfiguredModelEgress( + { ...runOptions(), signal: controller.signal }, + async () => { + entered.resolve(); + await settle.promise; + }, + ); + await entered.promise; + controller.abort(new Error("job cancelled")); + expect(revoke).toHaveBeenCalled(); + expect(proxy.stop).not.toHaveBeenCalled(); + settle.resolve(); + await expect(running).rejects.toThrow("job cancelled"); + expect(proxy.stop).toHaveBeenCalledOnce(); + }); + + it("refuses late admission after cancellation during proxy preparation", async () => { + const controller = new AbortController(); + const entered = createDeferredCore(); + const prepared = createDeferredCore(); + startProxy.mockImplementationOnce(async () => { + entered.resolve(); + await prepared.promise; + return proxy; + }); + const run = vi.fn(); + const running = withConfiguredModelEgress({ ...runOptions(), signal: controller.signal }, run); + await entered.promise; + controller.abort(new Error("preparation cancelled")); + prepared.resolve(); + await expect(running).rejects.toThrow("preparation cancelled"); + expect(proxy.registerProcess).not.toHaveBeenCalled(); + expect(run).not.toHaveBeenCalled(); + expect(proxy.stop).toHaveBeenCalledOnce(); + }); + + it("revokes and removes the private CA directory when the command fails", async () => { + await expect( + withConfiguredModelEgress(runOptions(), async () => { + throw new Error("remote command failed"); + }), + ).rejects.toThrow("remote command failed"); + expect(revoke).toHaveBeenCalled(); + expect(proxy.stop).toHaveBeenCalledOnce(); + const options = startProxy.mock.calls[0]![0]; + await expect(fs.stat(options.caDir)).rejects.toMatchObject({ code: "ENOENT" }); + }); +}); diff --git a/src/secrets/model-egress.ts b/src/secrets/model-egress.ts new file mode 100644 index 000000000000..e5b4bb1a45ab --- /dev/null +++ b/src/secrets/model-egress.ts @@ -0,0 +1,191 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { normalizeProviderId } from "@openclaw/model-catalog-core/provider-id"; +import { resolveProviderConfigSecretInput } from "../agents/model-auth-provider-config.js"; +import { findConfiguredProviderModel } from "../config/model-provider-config.js"; +import type { OpenClawConfig } from "../config/types.openclaw.js"; +import { createRedactingStreamWriter } from "../logging/redacting-stream.js"; +import { captureSecretRedactionRegistrySnapshot } from "../logging/secret-redaction-registry.js"; +import { + createProviderModelCatalogIdNormalizer, + resolveProviderModelRoutes, +} from "../plugins/provider-model-routes.js"; +import { startSecretEgressProxyServer } from "./egress-proxy/proxy-server.js"; +import { normalizeExactAllowedHost } from "./exact-hostname.js"; +import { resolveSecretRefString } from "./resolve.js"; +import { sealSecretSentinel } from "./sentinel.js"; + +export type ConfiguredModelEgress = { + /** Opaque API-key substitute for the remote application. */ + sentinel: string; + baseUrl: string; + model: string; + allowedHosts: readonly string[]; + /** Authenticated loopback proxy environment; keep it on the credential-owning host. */ + hostEnv: Readonly>; + /** Public trust bundle contents, suitable for the remote application's CA file. */ + caBundle: string; + onOutputChunk?: (chunk: Buffer, stream: "stdout" | "stderr") => void; +}; + +export type ConfiguredModelEgressOptions = { + config: OpenClawConfig; + provider: string; + model: string; + signal?: AbortSignal; + onOutput?: (text: string, stream: "stdout" | "stderr") => void; +}; + +function hasEntries(value: object | undefined): boolean { + return value !== undefined && Object.keys(value).length > 0; +} + +function resolveModelEgressSelection(params: ConfiguredModelEgressOptions) { + const provider = normalizeProviderId(params.provider); + const model = params.model.trim(); + if (!provider || !model) { + throw new Error("Model egress requires an explicit provider and model"); + } + const { providerConfig, ref } = resolveProviderConfigSecretInput(params.config, provider); + if (!providerConfig) { + throw new Error("Model egress requires a configured provider"); + } + if (!ref || (providerConfig.auth !== undefined && providerConfig.auth !== "api-key")) { + throw new Error( + "Model egress requires a configured API-key SecretRef; auth profiles and OAuth are unsupported", + ); + } + const configuredModel = findConfiguredProviderModel( + providerConfig, + provider, + model, + createProviderModelCatalogIdNormalizer(provider), + ); + if ( + hasEntries(providerConfig.headers) || + hasEntries(configuredModel?.headers) || + hasEntries(providerConfig.request) || + providerConfig.authHeader === false || + providerConfig.localService !== undefined + ) { + throw new Error( + "Model egress does not support custom request headers, authentication, proxy, TLS, or local-service configuration", + ); + } + const resolution = resolveProviderModelRoutes({ + provider, + modelId: model, + config: params.config, + }); + if (resolution?.kind === "incompatible") { + throw new Error(resolution.message); + } + const route = resolution + ? resolution.kind === "routes" + ? resolution.routes.find((candidate) => candidate.authRequirement === "api-key") + : undefined + : { + api: configuredModel?.api ?? providerConfig.api, + baseUrl: configuredModel?.baseUrl ?? providerConfig.baseUrl, + }; + if (!route?.baseUrl || (route.api !== "openai-responses" && route.api !== "openai-completions")) { + throw new Error("Model egress requires an OpenAI-compatible API-key model route"); + } + let url: URL; + try { + url = new URL(route.baseUrl); + } catch { + throw new Error("Model egress requires a valid HTTPS model base URL"); + } + if ( + url.protocol !== "https:" || + url.port || + url.username || + url.password || + url.search || + url.hash + ) { + throw new Error( + "Model egress requires an HTTPS model base URL on port 443 without credentials, query, or fragment", + ); + } + return { + provider, + model, + ref, + baseUrl: url.href, + allowedHosts: [normalizeExactAllowedHost(url.hostname)], + }; +} + +/** Owns one explicit CLI job's protected model credential and its revocable egress path. */ +export async function withConfiguredModelEgress( + params: ConfiguredModelEgressOptions, + run: (egress: ConfiguredModelEgress) => Promise, +): Promise { + params.signal?.throwIfAborted(); + const selection = resolveModelEgressSelection(params); + const apiKey = await resolveSecretRefString(selection.ref, { config: params.config }); + params.signal?.throwIfAborted(); + if (Buffer.byteLength(apiKey) > 64 * 1024) { + throw new Error("Model egress API key exceeds the protected credential size limit"); + } + const sentinel = sealSecretSentinel(apiKey, { label: `model-egress:${selection.provider}` }); + await using resources = new AsyncDisposableStack(); + const caDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-model-egress-")); + resources.defer(() => fs.rm(caDir, { recursive: true, force: true })); + params.signal?.throwIfAborted(); + const proxy = await startSecretEgressProxyServer({ + caDir, + allowedHosts: selection.allowedHosts, + onAudit: () => {}, + }); + resources.defer(() => proxy.stop()); + params.signal?.throwIfAborted(); + const grant = proxy.registerProcess([ + { + name: `${selection.provider} model API key`, + sentinel, + allowedHosts: selection.allowedHosts, + }, + ]); + resources.defer(() => { + grant.revoke(); + params.signal?.removeEventListener("abort", grant.revoke); + }); + params.signal?.addEventListener("abort", grant.revoke, { once: true }); + params.signal?.throwIfAborted(); + const caBundle = await fs.readFile(grant.env.NODE_EXTRA_CA_CERTS!, "utf8"); + params.signal?.throwIfAborted(); + const values = [...captureSecretRedactionRegistrySnapshot().values, grant.env.HTTPS_PROXY!]; + const output = (stream: "stdout" | "stderr") => + createRedactingStreamWriter( + { + write(text) { + params.onOutput?.(text, stream); + return true; + }, + }, + values, + ); + const stdout = output("stdout"); + const stderr = output("stderr"); + resources.defer(stdout.flush); + resources.defer(stderr.flush); + const result = await run({ + sentinel, + baseUrl: selection.baseUrl, + model: selection.model, + allowedHosts: selection.allowedHosts, + hostEnv: grant.env, + caBundle, + onOutputChunk: params.onOutput + ? (chunk, stream) => { + (stream === "stdout" ? stdout : stderr).write(chunk); + } + : undefined, + }); + params.signal?.throwIfAborted(); + return result; +}