mirror of
https://github.com/wirenboard/agent-vm.git
synced 2026-07-09 16:00:54 +00:00
Two coordinated changes that together raise the per-sandbox virtio
device cap on x86_64 by ~10x:
1. **Enable msb_krun's userspace split irqchip** in the runtime
(vendor/microsandbox submodule bump). KVM's in-kernel IOAPIC is
hardcoded to 24 pins, of which libkrun's allocator hands out only
IRQs 5..15 — saturated by a typical agent-vm config (rootfs, upper,
runtime fs, network, vsock, console, balloon, rng) plus a couple of
`--mount`s, so an extra mount has historically tripped
`RegisterNetDevice(IrqsExhausted)` at boot. The userspace IOAPIC
exposes 256 pins, lifting the libkrun allocator ceiling to
IRQ_MAX_SPLIT = 223. Requires a libkrun fork (pinned in
Cargo.toml) with three correctness fixes on top of upstream 0.1.13;
without them, enabling split_irqchip crashes the VMM during boot or
silently truncates the kernel cmdline.
2. **Patch libkrunfw to bump x86 COMMAND_LINE_SIZE 2048 → 16384**.
Each virtio_mmio device adds ~36 bytes of cmdline; past ~10 user
mounts the assembled cmdline crosses the stock 2048 cap, the kernel
silently truncates the tail (which includes virtio-console), and
the guest hangs in early boot with `kernel.log` stuck at 0 bytes.
New `libkrunfw-overrides/cmdline-size_x86_64.patch` lifts the cap;
CI workflow picks it up via libkrunfw's `patches/0*.patch` glob
(the `0999-overrides-` prefix sorts strictly after libkrunfw's own
numbered patches).
CI workflow hardening that landed alongside:
- cache key now keyed per-arch via `*_${ARCH}.patch` glob (an
aarch64 patch edit no longer invalidates the x86_64 cache);
- `config-libkrunfw*` skip pattern uses a bare suffix so future
`config-libkrunfw-tdx_${ARCH}.patch` / `-sev_` variants are also
excluded from the kernel source-patch pipeline.
User-facing knobs (`--mount` doc in run.rs, README troubleshooting
guide, ARCHITECTURE/PLAN narrative) updated to reflect the new
practical ceiling.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
692 lines
32 KiB
Markdown
692 lines
32 KiB
Markdown
# agent-vm — ARCHITECTURE
|
|
|
|
How the rewrite is put together and *why*. Reading this top-to-bottom should
|
|
tell you what every nontrivial design choice in the codebase exists for.
|
|
Updated after each phase lands. Section per phase; subsection per major
|
|
decision.
|
|
|
|
## Phase 0 — Scaffolding
|
|
|
|
### Repository layout
|
|
|
|
```
|
|
microsandbox-rewrite/
|
|
├── PLAN.md # phased roadmap
|
|
├── ARCHITECTURE.md # this file
|
|
├── Cargo.toml # workspace
|
|
├── crates/
|
|
│ └── agent-vm/
|
|
│ ├── Cargo.toml
|
|
│ └── src/main.rs # hello-world sandbox boot
|
|
├── vendor/
|
|
│ └── microsandbox/ # git submodule, wirenboard/microsandbox
|
|
└── .gitmodules
|
|
```
|
|
|
|
### Why a Cargo workspace from day one
|
|
|
|
The binary is small today but we already know we'll need at least one
|
|
internal crate per concern (creds, image, session). A workspace lets us add
|
|
those without restructuring later, and keeps `vendor/microsandbox` out of
|
|
our crate's manifest noise.
|
|
|
|
### Why a git submodule for microsandbox (vs. crates.io, vs. path dep)
|
|
|
|
- **Phase 3 requires extending microsandbox.** The new `SecretValue::File`
|
|
variant lives in `microsandbox-network`. A path dep against a sibling
|
|
checkout works for one developer but not for CI or contributors. A submodule
|
|
pinned to a branch on our fork (`wirenboard/microsandbox`) makes the
|
|
checkout self-contained and the upstream diff reviewable.
|
|
- **`[patch]` against crates.io** also works, but it duplicates the source-of-
|
|
truth pointer (Cargo.lock + patch table) and hides the fact that we are
|
|
shipping a fork. Submodule is more explicit.
|
|
|
|
### Why depend on the path under `vendor/microsandbox` even before we fork
|
|
|
|
Phase 0 doesn't change microsandbox, but we point at the submodule path so
|
|
the build wiring we set up here is the same wiring Phase 3 uses. Avoids a
|
|
mid-rewrite refactor of `Cargo.toml`.
|
|
|
|
### Why `Sandbox::builder("hello").image("alpine")` for the smoke test
|
|
|
|
Smallest possible exercise of the SDK that proves we can talk to the runtime.
|
|
Alpine is in the microsandbox examples, downloads quickly, and exits cleanly.
|
|
No need to involve our own image (that's Phase 1).
|
|
|
|
### Phase 0 runtime validation
|
|
|
|
`cargo run -p agent-vm` was exercised end-to-end on a Linux KVM host:
|
|
|
|
- One-time setup required outside the source tree: `apt install libcap-ng-dev`
|
|
(link-time dep pulled in transitively by `msb_krun`'s `capng` crate), and
|
|
user membership in the `kvm` group so `/dev/kvm` is openable. Both are host
|
|
prerequisites and don't belong in the repo.
|
|
- microsandbox's build script downloads its prebuilt runtime artifacts the
|
|
first time `cargo check` runs against the workspace
|
|
(`microsandbox@0.4.6: downloading microsandbox runtime dependencies`).
|
|
Nothing in our crate has to opt into this; the `prebuilt` feature is on by
|
|
default in `microsandbox-runtime`.
|
|
- Wall-clock for the full boot + `echo` + teardown with the alpine image
|
|
already in cache: **2.7s** on a release build. Cold first run includes the
|
|
OCI pull on top.
|
|
|
|
This is the latest point we can confirm we're talking to a real runtime before
|
|
adding our own scaffolding; pinning the validation here means a Phase 1 image
|
|
regression won't masquerade as an SDK-integration regression.
|
|
|
|
## Phase 1 — Base OCI image
|
|
|
|
### Layout
|
|
|
|
```
|
|
images/
|
|
├── Dockerfile # Debian 13 slim + agents
|
|
└── build.sh # ensures registry, docker build, docker push
|
|
|
|
crates/agent-vm/src/
|
|
├── main.rs # clap entry; dispatches to subcommands
|
|
└── setup.rs # `agent-vm setup`: invoke build.sh, then verify in microsandbox
|
|
```
|
|
|
|
### Image distribution: local Docker registry vs. alternatives
|
|
|
|
`RootfsSource` (microsandbox-side) supports three image origins:
|
|
|
|
1. `Oci(reference)` — pulled from a registry (Docker Hub, GHCR, local, etc.).
|
|
2. `Bind(path)` — host directory used as the rootfs directly.
|
|
3. `DiskImage(path)` — qcow2/raw/vmdk file.
|
|
|
|
We pick **(1) with a local `registry:2` container on 127.0.0.1:5000**, exposed
|
|
to the sandbox builder as `.image("localhost:5000/agent-vm:latest")
|
|
.registry(|r| r.insecure())`. Rationale:
|
|
|
|
- **Standard OCI semantics.** microsandbox's layer cache, GC, snapshotting,
|
|
and metadata DB all key off OCI references. Going through the registry path
|
|
means we get all of that for free instead of working around it.
|
|
- **Same wiring as a future remote registry.** When/if we publish images to
|
|
GHCR, the launcher's `.image(...)` call doesn't change; only the tag does.
|
|
- **Bind would require write-through or COW management.** `RootfsSource::Bind`
|
|
hands the host directory to the VM as the rootfs. The microsandbox example
|
|
uses it for a single one-shot sandbox; we'd need an overlay on top to share
|
|
a template across multiple concurrent invocations. The OCI path already
|
|
handles this via the layer cache.
|
|
- **Disk-image (qcow2) would mean building rootfs images ourselves.** Doable
|
|
with `debootstrap` + `mkfs.ext4`, but the build steps are less familiar
|
|
than `docker build` and the rebuild loop is slower.
|
|
|
|
The price is that we now run a Docker daemon and a `registry:2` container on
|
|
the host. Acceptable: every dev who needs to *build* the image already needs
|
|
Docker, and the registry container is ~30 MB and starts in <1 s. End users who
|
|
only pull a prebuilt image won't run the local registry at all (Phase 9
|
|
distribution territory).
|
|
|
|
### Image content: deliberately minimal
|
|
|
|
The current Dockerfile installs only what each of the three agents needs to
|
|
run plus the dev tools that are universally useful:
|
|
|
|
- Base: `ca-certificates`, `curl`, `wget`, `git`, `jq`, `bash`,
|
|
`python3`/`pip`, `ripgrep`, `fd-find`.
|
|
- Chromium + `fonts-liberation` + `sudo` + `libnss3-tools` for the
|
|
Chrome DevTools MCP (Phase 7). Symlinks `/usr/bin/google-chrome` and
|
|
`/opt/google/chrome/chrome` to `/usr/bin/chromium` so puppeteer's
|
|
default discovery paths resolve. Dedicated `chrome` user (UID 9999)
|
|
with home `/home/chrome`, empty NSS DB at `~/.pki/nssdb`, sudoers
|
|
rule `root ALL=(chrome) NOPASSWD: ALL`, and a wrapper at
|
|
`/usr/local/bin/agent-vm-chrome-mcp` that re-execs the MCP under
|
|
that user. Pre-warmed npm cache for `chrome-devtools-mcp@1.0.1`
|
|
under `/home/chrome/.npm/_npx/` so first launch is a cache hit.
|
|
- `gh` from cli.github.com/packages (Phase 6 — gh/git credential
|
|
injection).
|
|
- Node.js 22 from NodeSource (needed by Claude Code, OpenCode, MCP servers).
|
|
- Agents installed via their canonical installer scripts so we track upstream
|
|
release channels: `claude.ai/install.sh`, `opencode.ai/install`, and the
|
|
Codex `install.sh` from GitHub releases.
|
|
|
|
Explicitly skipped in v1 (per PLAN.md scope cuts): Docker-in-VM, LSP
|
|
plugins, `mitmproxy` (microsandbox does the interception in Phase 3,
|
|
no in-VM proxy needed), GitHub Copilot CLI. Each line we keep is a
|
|
line that has to keep working through `apt-get update` churn, so the
|
|
bar to add anything is "needed by an in-scope agent flow."
|
|
|
|
Resulting image: a few GB uncompressed (chromium is the largest
|
|
contributor at ~400 MB, followed by Node.js and the agent CLIs;
|
|
re-measure with `docker images` when you care about exact bytes).
|
|
Registry layer count is bounded by the `RUN` granularity in the
|
|
Dockerfile.
|
|
|
|
### Image build is shelled to Bash, not done in Rust
|
|
|
|
`crates/agent-vm/src/setup.rs::run_build_script` spawns
|
|
`bash images/build.sh`. We don't talk to the Docker daemon directly because:
|
|
|
|
- Docker has a CLI that every developer already knows how to read, run, and
|
|
debug. A Rust caller wrapping the API would only add a layer.
|
|
- The build script is the right place for host-shell idioms (volumes,
|
|
port-forwarding the registry, `docker inspect` checks) and stays out of
|
|
the way of the Rust binary's logic.
|
|
- Rebuilding the image doesn't require recompiling the binary, and vice
|
|
versa.
|
|
|
|
The Rust side does own the **verify** step (boot from the freshly pushed
|
|
image, run the three `--version` commands), because that step is exactly the
|
|
microsandbox SDK call the launcher will make in Phase 2 — exercising it from
|
|
`setup` ensures we catch image/SDK-integration regressions before any user
|
|
session depends on them.
|
|
|
|
### `setup --no-verify` and `--image`
|
|
|
|
Two escape hatches surfaced from the start:
|
|
|
|
- `--no-verify` lets a developer iterate on the Dockerfile without paying for
|
|
a sandbox boot each loop.
|
|
- `--image` / `AGENT_VM_IMAGE_TAG` lets us point at an alternative tag (a
|
|
prebuilt image on GHCR, a developer's experimental tag, etc.) without
|
|
touching `build.sh`. The default stays `localhost:5000/agent-vm:latest` so
|
|
the happy path matches what `build.sh` produces.
|
|
|
|
## Phase 2 — Launcher MVP
|
|
|
|
### Layout
|
|
|
|
```
|
|
crates/agent-vm/src/
|
|
├── main.rs # clap entry: setup | claude | codex | opencode | shell
|
|
├── setup.rs # unchanged from Phase 1
|
|
├── run.rs # `agent-vm <agent>`: build sandbox, attach or exec
|
|
└── session.rs # project-dir hash, state dirs, sandbox name
|
|
```
|
|
|
|
### Project-scoped sandbox name + state dir
|
|
|
|
Each project gets:
|
|
|
|
- A short hash: first 6 bytes of `SHA256(canonical(cwd))` rendered as 12 hex
|
|
chars (~48 bits — plenty for "no two project dirs on one host collide").
|
|
- A state directory at
|
|
`${AGENT_VM_STATE_DIR-${XDG_STATE_HOME-$HOME/.local/state}/agent-vm}/<hash>/`.
|
|
- A sandbox name of `agent-vm-<hash>`.
|
|
|
|
The hash being short enough to fit in `<hostname>` is convenient when
|
|
debugging from inside the sandbox (`hostname` shows it). The sandbox name
|
|
being deterministic means a second `agent-vm claude` in the same project
|
|
*replaces* the first one (`.replace()` on the builder gives it 10 s to exit
|
|
gracefully, then SIGKILLs) instead of spawning a parallel VM.
|
|
|
|
The launcher prints a one-line banner on startup
|
|
(`==> agent-vm-<hash> in <cwd> (state: <dir>)`) so users always know which
|
|
project a given sandbox is bound to.
|
|
|
|
### Mounts: one for workspace, one for state, no third
|
|
|
|
When this layout was chosen, the microsandbox runtime was running with
|
|
libkrun's default in-kernel IOAPIC, which hands ~11 IRQs to virtio-mmio
|
|
devices total on x86_64. The OCI rootfs already consumes two slots (EROFS
|
|
lower + ext4 upper), plus virtio-net + vsock + console + agentd's
|
|
serial — adding a bind mount per agent state directory (claude, codex,
|
|
opencode) pushed us over and `RegisterBlockDevice(IrqsExhausted)` at boot
|
|
followed. We later lifted the underlying cap by enabling `msb_krun`'s
|
|
userspace split irqchip (see "Split irqchip and the virtio-IRQ ceiling"
|
|
below; cap is now ~219), but the one-workspace + one-state layout still
|
|
makes sense regardless: one virtio-fs server, one rootfs `patch` entry
|
|
per agent, and a stable on-host layout.
|
|
|
|
Resolution:
|
|
|
|
- One bind mount for `cwd → /workspace` (the project).
|
|
- One bind mount for `<state-dir> → /agent-vm-state` (everything else).
|
|
- The agents' expected paths are wired up *inside* the rootfs by Phase 1's
|
|
`patch` API (rootfs symlinks baked into the upper overlay before VM start):
|
|
- `/root/.claude → /agent-vm-state/claude`
|
|
- `/root/.local/share/opencode → /agent-vm-state/opencode`
|
|
- Codex uses the `CODEX_HOME=/agent-vm-state/codex` env var instead of a
|
|
symlink, because `/root/.codex/packages/...` in the base image contains
|
|
the codex binary itself and a symlink would shadow it.
|
|
|
|
This keeps us at two virtio bind mounts no matter how many agents we add
|
|
later, and leaves plenty of IRQ headroom for user-supplied `--mount`
|
|
arguments now that the split irqchip is on.
|
|
|
|
### Split irqchip and the virtio-IRQ ceiling
|
|
|
|
`msb_krun` exposes a `MachineBuilder::split_irqchip(bool)` knob. With it
|
|
off (default), libkrun uses KVM's in-kernel IOAPIC, which is hard-capped at
|
|
24 pins and only hands IRQs 5..=15 to virtio-mmio — about 11 usable IRQs
|
|
for the whole VM. That fills up fast: rootfs lower + rootfs upper +
|
|
virtio-net + virtio-vsock + virtio-console + virtio-fs (project) +
|
|
virtio-fs (state) already saturates it on this build, so an extra
|
|
`--mount` would trip `RegisterNetDevice(IrqsExhausted)` at boot.
|
|
|
|
With split_irqchip enabled, `msb_krun` runs a userspace IOAPIC backed by
|
|
an event-loop thread it spawns automatically. The pin count rises to 219,
|
|
which puts the practical ceiling on `--mount` well into the hundreds. The
|
|
trade-off is one extra worker thread per VM and a slightly hotter IRQ
|
|
delivery path; we accept it because the IRQ headroom is the difference
|
|
between "one or two extra mounts work" and "you can stop worrying about
|
|
the cap." aarch64/riscv64 ignore the knob — their GIC/AIA models already
|
|
expose >200 IRQs.
|
|
|
|
The runtime sets `split_irqchip(true)` unconditionally in
|
|
`vendor/microsandbox/crates/runtime/lib/vm.rs`. The user-facing
|
|
`--mount` doc and Phase 7's wiring in `crates/agent-vm/src/run.rs` were
|
|
updated to drop the pre-cap warning that previously fronted the limit.
|
|
|
|
The change also bumped `msb_krun` from 0.1.12 → 0.1.13 across the
|
|
vendor crates (`runtime`, `filesystem`, `network`). 0.1.12's userspace
|
|
IOAPIC was unusable in practice: its IRR was a single `u32` so any IRQ
|
|
delivered on pin ≥ 32 was dropped without notice, and the redirection-
|
|
table register-index calculation in `read`/`write` did an unchecked
|
|
`ioregsel - IOAPIC_REG_REDTBL_BASE` that wrapped on any access below
|
|
the redirection-table base — which the guest performs during normal
|
|
IOAPIC programming. Both fixes landed in 0.1.13, published 2026-05-26.
|
|
The PLAN's Discovered Upstream Issue #3 was originally attributed to
|
|
"the libkrun IRQ cap"; with hindsight, the cap is a real KVM-level
|
|
ceiling but the multi-mount boot failure that finally drove this work
|
|
was a separate, fixable bug inside `msb_krun_devices`'s userspace
|
|
IOAPIC, only ever reached once the split irqchip was turned on.
|
|
|
|
### Interactive attach vs. non-TTY exec
|
|
|
|
`Sandbox::attach()` requires a real controlling TTY: it puts stdin in raw
|
|
mode and opens `/dev/tty` for its non-blocking input fd. When stdin isn't a
|
|
TTY (pipe, redirect, smoke test under `sg`/`sudo -c`, CI), `attach` returns
|
|
ENXIO.
|
|
|
|
The launcher checks `std::io::stdin().is_terminal()` and branches:
|
|
|
|
- **TTY** → `attach(cmd, args)` — the agent's TUI gets a full PTY and is
|
|
fully interactive.
|
|
- **No TTY** → `exec_with(cmd, |e| e.args(args).cwd("/workspace"))` — runs to
|
|
completion, then forwards collected stdout/stderr and the exit code.
|
|
|
|
Non-TTY mode loses the live streaming TUI experience but gives the caller a
|
|
clean `stdout | other-tool` story. Streaming stdout/stderr during run landed
|
|
in the Phase 4 verification session (2026-05-24) — see PLAN.md.
|
|
|
|
### Credentials: env-var only, deliberately
|
|
|
|
Phase 2 reads `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` from the host
|
|
environment and forwards them via `.env()`. This is the simplest possible
|
|
path that exercises everything else end-to-end. Phase 3 replaces this with
|
|
microsandbox's secret-substitution API backed by host-rooted token files,
|
|
and Phase 4 adds refresh semantics on top.
|
|
|
|
Concretely, the env-var path is intentionally insufficient for our real use
|
|
case (Claude Code's host OAuth, Codex's host ChatGPT auth, OpenCode's
|
|
OAuth flows). That gap stays open until Phase 3.
|
|
|
|
### `PATH` is set explicitly, not inherited
|
|
|
|
The Phase 1 Dockerfile puts the agent binaries on `PATH` via an `ENV`
|
|
directive, but that PATH only takes effect when an interactive shell sources
|
|
the image's profile. `attach()` and `exec()` both spawn the command via
|
|
`execve` directly, so we re-publish the same PATH on the sandbox builder
|
|
(`/root/.local/bin:/root/.claude/local/bin:/root/.opencode/bin:/usr/local/
|
|
bin:/usr/bin:/bin`). Otherwise `agent-vm claude` would `ENOENT` immediately.
|
|
|
|
### Tunables: env-var-driven for now
|
|
|
|
`AGENT_VM_IMAGE_TAG`, `AGENT_VM_MEMORY_MIB`, `AGENT_VM_CPUS` cover the three
|
|
knobs you actually want to change session-to-session. `--memory` and
|
|
`--cpus` were promoted to clap flags (`1817391`); `--image` and friends are
|
|
on the Phase 9 polish list. Env-var-only kept the
|
|
Phase 2 surface small and means we don't have to design the `--memory 4G`
|
|
vs `--memory 4096` ergonomics yet.
|
|
|
|
### What Phase 2 deliberately doesn't do
|
|
|
|
- **No live DoD smoke against the Anthropic API.** The Phase 2 DoD in
|
|
PLAN.md calls for `agent-vm claude -p 'say hi'` returning a real Claude
|
|
response, but on this host we only have a Claude OAuth credential (not an
|
|
`ANTHROPIC_API_KEY`), and Phase 2 explicitly does *not* implement OAuth
|
|
plumbing. We verified all of Phase 2's wiring end-to-end via the `shell`
|
|
subcommand (workspace mount, state persistence, env propagation, all three
|
|
agent CLIs resolvable on PATH) and explicitly chose to close the API-call
|
|
gap during Phase 3's host-OAuth work rather than ferry an ephemeral API
|
|
key through this session.
|
|
|
|
## Phase 3 — Host-rooted secrets
|
|
|
|
### Layout
|
|
|
|
```
|
|
vendor/microsandbox/ (branch: agent-vm-secret-file)
|
|
└── crates/network/lib/secrets/
|
|
├── config.rs # new: SecretValue { Static, File } enum
|
|
└── handler.rs # resolves SecretValue at connection-setup
|
|
|
|
crates/agent-vm/src/
|
|
├── secrets.rs # new: read host creds, write placeholders
|
|
├── run.rs # wire TLS intercept + secret_env per provider
|
|
└── main.rs # register secrets module
|
|
```
|
|
|
|
### Two-layer placeholder dance
|
|
|
|
Real tokens never enter the VM. The dance per provider:
|
|
|
|
1. **Host side.** agent-vm reads the host's credential file
|
|
(`~/.claude/.credentials.json` for Claude,
|
|
`~/.codex/auth.json` for Codex) at every launch. It extracts the
|
|
access token, keeps it in a short-lived Rust `String`, and registers
|
|
it as a microsandbox secret with a stable placeholder string
|
|
(`msb-anthropic-placeholder-a-v2` and
|
|
`msb-openai-placeholder-a-v2`). The placeholder *constants* live in
|
|
`crates/agent-vm/src/secrets.rs` (`ANTHROPIC_ACCESS_PLACEHOLDER` etc.) —
|
|
prefer the constant over the literal so a future rename doesn't drift.
|
|
2. **Guest side.** agent-vm writes a "placeholder credentials" JSON
|
|
into the per-project state dir
|
|
(`<state>/claude/.credentials.json`,
|
|
`<state>/codex/auth.json`) using the placeholder string instead of
|
|
the real token. Other fields (`expiresAt`, `scopes`, `account_id`,
|
|
`last_refresh`, etc.) are copied from the host file so the in-VM
|
|
agent sees a plausible JSON shape. `refreshToken` is set to a
|
|
sentinel string — Phase 3 doesn't handle refresh.
|
|
3. **TLS interception.** microsandbox's network proxy intercepts the
|
|
sandbox's HTTPS traffic. When the agent makes a request to any
|
|
allowed host (`api.anthropic.com`, `platform.claude.com`,
|
|
`api.openai.com`, `chatgpt.com`, `auth.openai.com`), the proxy
|
|
sees `Authorization: Bearer msb-…-placeholder-…` in the outgoing
|
|
request, splices in the real token from the secret config, then
|
|
forwards.
|
|
|
|
The agent inside the VM never sees the real token in any form
|
|
(`/proc/$$/environ`, `cat ~/.claude/.credentials.json`, network
|
|
introspection inside the guest all show only the placeholder). It
|
|
gets the real token only as a header-mangled middlebox effect on the
|
|
way out — which is structurally what microsandbox was designed for.
|
|
|
|
### Upstream extension: `SecretValue { Static, File }`
|
|
|
|
Pre-Phase 3, `SecretEntry.value` was a `String` captured at builder
|
|
time. That worked for static API keys but precluded host-side OAuth
|
|
rotation — there was no way to surface a new token to a running
|
|
sandbox short of rebuilding it.
|
|
|
|
The `agent-vm-secret-file` branch of vendor/microsandbox adds
|
|
`SecretValue { Static(String), File(PathBuf) }` and changes
|
|
`SecretEntry.value` to that enum. The handler resolves `File` at
|
|
connection-setup time, so each new request to an allowed host sees the
|
|
current file contents. Wire format stays a single JSON string for
|
|
backward compatibility with the prebuilt `msb` daemon already on
|
|
users' hosts:
|
|
|
|
| Variant | Wire format |
|
|
|---|---|
|
|
| `Static(v)` | `"v"` — a bare JSON string, identical to the old `value: String` form |
|
|
| `File(p)` | `"\0msbfile:<path>"` — a NUL-prefixed sentinel string |
|
|
|
|
The NUL prefix is unforgeable in API tokens (always printable ASCII).
|
|
Old `msb` daemons that don't recognise the sentinel treat the whole
|
|
thing as an opaque string and substitute it verbatim — broken for
|
|
`File`, but never crashes.
|
|
|
|
### Phase 3 uses `Static` only
|
|
|
|
`SecretValue::File` is the right primitive for refresh-aware
|
|
substitution, but turning it on end-to-end requires a `msb` daemon
|
|
built from our forked source replacing the prebuilt one at
|
|
`~/.microsandbox/bin/msb`. Phase 3 doesn't ship that distribution
|
|
plumbing — it captures the host token as a `String` at launch time
|
|
and passes `SecretValue::Static(token)` to microsandbox. The sandbox
|
|
lives until the token's TTL (usually hours); rotation is a Phase 4
|
|
problem.
|
|
|
|
### Allowed-host lists
|
|
|
|
Per-provider, we allow the API host *and* the OAuth-token host. The
|
|
OAuth-token host doesn't actually need substitution in Phase 3 (we
|
|
don't intercept the refresh flow yet), but we have to allow it,
|
|
otherwise the in-VM agent's refresh attempt would trigger
|
|
microsandbox's secret-violation detector (placeholder going to a
|
|
disallowed host = `BlockAndLog` blocks the request). Letting the
|
|
placeholder reach the OAuth host means the upstream server just
|
|
rejects it normally, which is at least a comprehensible failure.
|
|
|
|
| Provider | Allowed hosts |
|
|
|---|---|
|
|
| Anthropic | `api.anthropic.com`, `platform.claude.com` |
|
|
| OpenAI | `api.openai.com`, `chatgpt.com`, `auth.openai.com` |
|
|
|
|
### `IS_SANDBOX=1`
|
|
|
|
Claude Code refuses to run as root with
|
|
`--dangerously-skip-permissions` unless `IS_SANDBOX=1` is set. The
|
|
in-guest user is root, and the whole point of the microVM is that
|
|
the sandbox itself is the boundary — so we set `IS_SANDBOX=1` on
|
|
the builder. Same env var the original Bash agent-vm used.
|
|
|
|
### Smoke verification
|
|
|
|
End-to-end verified on a nested-VM test host (cwd
|
|
`/home/boger.linux/agent-vm-phase3-test`):
|
|
|
|
- `cat /root/.claude/.credentials.json` inside the guest shows the
|
|
placeholder, not the real token. ✓
|
|
- `cat /proc/1/environ | tr '\0' '\n' | grep -i token` finds only
|
|
`MSB_AGENT_VM_ANTHROPIC_UNUSED=msb-…-placeholder-…`. ✓
|
|
- TLS-intercepted curl to `https://api.anthropic.com` sees the
|
|
microsandbox CA on the server cert (`CN=microsandbox CA`),
|
|
confirming requests go through the substitution proxy. ✓
|
|
- `AGENT_VM_DEBUG_CONFIG=1` dumps the SandboxConfig JSON and the
|
|
secret value is the host's real `accessToken` (which on this nested
|
|
test host is itself a placeholder relayed to the outer host's real
|
|
bridge — see below). ✓
|
|
|
|
The *final* leg ("api.anthropic.com returns a real response") can't
|
|
be verified on this host because we're running inside an outer
|
|
agent-vm whose own credential bridge intercepts requests on the outer
|
|
host's localhost — which our nested microsandbox can't reach. On a
|
|
non-nested host with a real Claude OAuth credential, the substituted
|
|
bearer reaches Anthropic verbatim and the response is real. The same
|
|
flow is structurally identical to how the original Bash agent-vm's
|
|
credential-proxy works.
|
|
|
|
### What Phase 3 deliberately doesn't do
|
|
|
|
- **No refresh.** Long sessions will hit a 401 when the captured
|
|
access token expires (typically hours). Phase 4 closes this.
|
|
- **No `~/.microsandbox/bin/msb` replacement.** The `SecretValue::File`
|
|
variant requires a `msb` rebuilt from our fork to actually re-read
|
|
the file. Without that the Static path is what gets exercised, and
|
|
it does work against unpatched `msb` (wire-format compatibility was
|
|
the explicit design goal of the bare-string sentinel encoding).
|
|
- **No host-side OAuth token endpoint short-circuiting.** When the
|
|
in-VM agent tries to refresh, the request goes upstream and is
|
|
rejected by Anthropic/OpenAI because the placeholder refresh token
|
|
isn't real. The original Bash agent-vm has logic to MITM the
|
|
`platform.claude.com/v1/oauth/token` and `auth.openai.com/oauth/token`
|
|
endpoints and forge responses from re-reads of the host file. That's
|
|
Phase 4.
|
|
|
|
## Phase 4 — OAuth refresh: file-backed secrets + interceptor hook
|
|
|
|
Phase 3 left a 401-then-die failure mode: when the captured access
|
|
token expires mid-session the in-VM agent gets 401 from the API, tries
|
|
to refresh against the OAuth endpoint with the placeholder refresh
|
|
token, and gets 401 again. The user has to exit and re-launch.
|
|
|
|
Phase 4 closes the loop end-to-end. Two upstream microsandbox
|
|
extensions plus an agent-vm subprocess handle it.
|
|
|
|
### Pieces
|
|
|
|
```
|
|
vendor/microsandbox/ (branch: agent-vm-secret-file)
|
|
└── crates/network/lib/
|
|
├── secrets/config.rs # SecretValue::File (from Phase 3) is now actually used
|
|
└── intercept/ # new: per-route request-interceptor hook
|
|
├── config.rs # InterceptConfig (rules + hook command)
|
|
└── handler.rs # per-connection state machine
|
|
|
|
crates/agent-vm/src/
|
|
├── msb_install.rs # new: build patched msb from vendor; point MSB_PATH at it
|
|
├── intercept_hook.rs # new: `agent-vm _intercept-hook` subprocess
|
|
├── secrets.rs # switched from Static(token) to File(<state>.secrets/{anthropic,openai})
|
|
└── run.rs # registers the interceptor with two rules
|
|
```
|
|
|
|
### Patched `msb` shipped via `MSB_PATH`
|
|
|
|
`agent-vm setup` now runs
|
|
`cargo build --release -p microsandbox-cli --bin msb` in
|
|
`vendor/microsandbox` and leaves the artifact at
|
|
`vendor/microsandbox/target/release/msb`. At startup, every agent-vm
|
|
invocation sets `MSB_PATH` to that path (top of microsandbox's
|
|
resolution ladder), so the patched binary is what actually runs the
|
|
VM. The user's `~/.microsandbox/bin/msb` is never touched and
|
|
upstream-installed tooling on the same host keeps using its own
|
|
prebuilt.
|
|
|
|
The real msb binary lives in the `microsandbox-cli` crate; the
|
|
`microsandbox` crate has a separate `microsandbox` binary that's
|
|
just a 5-line shim forwarding to `~/.microsandbox/bin/msb`. Building
|
|
the wrong target produces a 389 KB shim that boots silently then
|
|
hangs at VM init — about 30 minutes of debugging into a
|
|
no-VMM-symbols-in-the-binary surprise. Recorded here so the next
|
|
person doesn't redo it.
|
|
|
|
### `SecretValue::File`
|
|
|
|
Phase 3's per-launch snapshot becomes a per-launch *file write*. We
|
|
write the host's `accessToken` to a host-only secret file (see next
|
|
subsection for *where*) with 0600 perms via atomic-write-then-rename.
|
|
The launcher passes the file path to microsandbox as a
|
|
`SecretValue::File`.
|
|
|
|
The patched msb's TLS-intercept proxy calls `SecretValue::resolve()`
|
|
at *connection-setup* time — every new TCP connection re-reads the
|
|
file. So any host-side rotation (whether triggered by the user's
|
|
external `claude` use or by our interceptor hook below) is visible
|
|
to the very next request, without rebuilding the sandbox.
|
|
|
|
### Token files live *outside* the guest bind mount
|
|
|
|
The launcher bind-mounts the per-project `state_dir` into the guest at
|
|
`/agent-vm-state` as a *single* mount. The single-bind shape originally
|
|
fell out of libkrun's tight virtio-IRQ cap (one bind for all per-agent
|
|
state instead of one per agent — see "Mounts: one for workspace, one
|
|
for state, no third"), and we've kept it after lifting the cap because
|
|
it gives a stable on-host layout and a single virtio-fs server. That
|
|
makes mount placement security-critical: **anything under `state_dir`
|
|
is readable from inside the VM.**
|
|
|
|
The real access-token files therefore must *not* live under
|
|
`state_dir`. They sit in a sibling host-only directory
|
|
`${state_root}/<hash>.secrets/` (mode 0700), derived from `state_dir`
|
|
by `secrets::{anthropic,openai}_token_path` so the launcher and the
|
|
refresh hook agree on the path without passing it explicitly. The
|
|
microsandbox proxy reads these files on the *host* side via
|
|
`SecretValue::File`, so they never need to be mounted into the guest at
|
|
all.
|
|
|
|
This was a real leak found during Phase 4 end-to-end verification: the
|
|
first cut wrote the tokens to `<state>/tokens/{anthropic,openai}`, i.e.
|
|
*inside* the mount, so `cat /agent-vm-state/tokens/anthropic` in the
|
|
guest returned the host's real bearer — silently defeating the entire
|
|
"real tokens never enter the VM" guarantee. The nested test host masked
|
|
it (there the "real" token is itself the outer bridge's placeholder), so
|
|
it only surfaced once we grepped the guest filesystem for the token
|
|
during verification. A `token_files_live_outside_the_guest_mount` unit
|
|
test now guards the invariant.
|
|
|
|
### Request-interceptor hook (the OAuth refresh MITM)
|
|
|
|
`microsandbox-network` gained an `InterceptConfig` that the launcher
|
|
fills with:
|
|
|
|
```rust
|
|
.intercept(|i| i
|
|
.hook(["…/agent-vm", "_intercept-hook", "--state-dir", "…"])
|
|
.rule("platform.claude.com", "POST", "/v1/oauth/token")
|
|
.rule("auth.openai.com", "POST", "/oauth/token"))
|
|
```
|
|
|
|
When the in-VM agent posts an OAuth refresh request, the proxy:
|
|
|
|
1. Buffers the full request (it's tiny — <1 KB — so this is cheap
|
|
and capped at 64 KiB).
|
|
2. Spawns the hook command with the request bytes on stdin and four
|
|
env vars (`MSB_INTERCEPT_SNI/_HOST_RULE/_METHOD/_PATH_PREFIX`).
|
|
3. Reads the hook's stdout as the response and writes it back to the
|
|
guest, encrypted under the forged TLS cert.
|
|
4. Closes the connection without ever touching the upstream server.
|
|
|
|
The hook (`agent-vm _intercept-hook`) is the same binary in a hidden
|
|
clap subcommand mode:
|
|
|
|
1. Reads the request from stdin (sanity-checks it's `POST …`).
|
|
2. Spawns `claude -p hi --model sonnet` (or `codex exec --skip-git-
|
|
repo-check 'Reply OK'`) on the **host** so the host CLI rotates
|
|
`~/.claude/.credentials.json` / `~/.codex/auth.json` the normal way.
|
|
3. Re-reads the rotated host file, rewrites the host-only token file
|
|
(`<state>.secrets/{anthropic,openai}`) so the next non-refresh
|
|
request from the guest gets the new bearer via `SecretValue::File`.
|
|
4. Synthesizes an OAuth refresh-response JSON shaped like what the
|
|
upstream server would return, but with **placeholder** strings in
|
|
the `access_token` / `refresh_token` fields. The in-VM agent
|
|
updates its credentials.json to those placeholders and continues.
|
|
5. Writes the response to stdout, exits 0.
|
|
|
|
The guest never holds a real token at any layer:
|
|
- `~/.claude/.credentials.json` always contains placeholders (Phase 3).
|
|
- The real-token file is on the host *outside* the guest bind mount
|
|
(see "Token files live outside the guest bind mount").
|
|
- The proxy substitutes real-for-placeholder on the way *out* (Phase 3).
|
|
- The OAuth refresh response also returns placeholders (Phase 4).
|
|
- The host CLI on the host is the only thing that ever touches real
|
|
OAuth machinery, and it writes to a file we re-read.
|
|
|
|
### Hook-process boundary, not callback
|
|
|
|
The interceptor uses a subprocess (fork+exec per request) rather than
|
|
a callback into the SDK. Reasons:
|
|
|
|
- `Vec<Box<dyn RequestInterceptor>>` isn't serializable. The
|
|
network config is JSON-piped from the SDK to a separate `msb`
|
|
process, so anything we configure on the SDK side has to round-trip
|
|
through JSON.
|
|
- Refresh requests are rare (once per hour at worst). Fork-per-request
|
|
overhead is irrelevant against the latency of the host `claude`
|
|
invocation that the hook does anyway.
|
|
- A subprocess can dispatch on any logic without us having to
|
|
re-extend microsandbox each time we add a provider.
|
|
|
|
### Smoke verification
|
|
|
|
```
|
|
Inside the guest:
|
|
POST https://platform.claude.com/v1/oauth/token
|
|
body: {"grant_type":"refresh_token","refresh_token":"…PLACEHOLDER_REFRESH…"}
|
|
|
|
Response:
|
|
HTTP 200 application/json
|
|
{"access_token":"msb-anthropic-placeholder-a-v2",
|
|
"refresh_token":"msb-anthropic-placeholder-r-v2",
|
|
"expires_in":3499, "token_type":"Bearer",
|
|
"scope":["user:file_upload","user:inference",…]}
|
|
```
|
|
|
|
Confirmed on the same nested-VM test host as Phase 3. The hook ran,
|
|
host `claude -p` rotated the host file, the new bearer landed in
|
|
`<state>.secrets/anthropic`, and the synthesized response reached the
|
|
guest. `expires_in: 3499` is the freshly-derived seconds-until-expiry
|
|
of the just-rotated token.
|
|
|
|
### What Phase 4 deliberately doesn't do
|
|
|
|
- **No proactive expiry timer.** Discussed and rejected: the
|
|
guest's own refresh attempt at 401-time triggers our hook, which
|
|
triggers the host-side refresh. If the user runs `claude` on the
|
|
host between sessions, the host file is already fresh and the
|
|
`SecretValue::File` re-read picks it up with no hook involved.
|
|
A timer would be belt-and-suspenders.
|
|
- **No msb shipped via `~/.microsandbox/bin/msb`.** The MSB_PATH
|
|
override is per-agent-vm-invocation only; other microsandbox SDK
|
|
consumers on the same host keep using the upstream prebuilt.
|
|
- **No single-flight for concurrent in-guest refreshes.** Two
|
|
concurrent refresh attempts could each spawn a host `claude -p`.
|
|
The host CLI's own file lock prevents corruption, so the worst
|
|
outcome is one extra `claude -p` invocation. If this becomes a
|
|
pain point, a `<state>/tokens/.refresh.lock` flock around the host
|
|
CLI invocation is two lines.
|