fix(secrets): keep real tokens out of the guest bind mount

The launcher bind-mounts the per-project state_dir into the guest at
/agent-vm-state (a single mount, to stay under libkrun's virtio-IRQ
cap). secrets.rs wrote the host's real access token to
state_dir/tokens/{anthropic,openai} — i.e. INSIDE that mount — so the
in-VM agent could `cat /agent-vm-state/tokens/anthropic` and read the
host bearer, silently defeating the whole Phase 3/4 "real tokens never
enter the VM" guarantee.

The nested test host masked this: there the "real" token is itself the
outer agent-vm bridge's placeholder, so a leak isn't a real leak. It
only surfaced when grepping the guest filesystem during end-to-end
verification on a host with a genuine credential.

Move the token files to a host-only sibling dir <hash>.secrets/ (0700),
derived from state_dir by secrets::{anthropic,openai}_token_path so the
launcher and the refresh hook agree on the path. The microsandbox proxy
reads them host-side via SecretValue::File, so they never need to be
mounted. Best-effort remove of the legacy state_dir/tokens on launch so
an upgrade doesn't leave a real token lingering in the mount.

Adds token_files_live_outside_the_guest_mount as a regression guard.
Verified in-guest: no host token anywhere under /agent-vm-state;
credentials.json and PID1 environ show only placeholders;
api.anthropic.com server cert is issued by CN=microsandbox CA.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Evgeny Boger 2026-05-21 21:33:44 +03:00
parent 2e5b5c546e
commit db50cc26ab
3 changed files with 141 additions and 13 deletions

View file

@ -459,7 +459,7 @@ vendor/microsandbox/ (branch: agent-vm-secret-file)
crates/agent-vm/src/
├── msb_install.rs # new: build patched msb from vendor; point MSB_PATH at it
├── intercept_hook.rs # new: `agent-vm _intercept-hook` subprocess
├── secrets.rs # switched from Static(token) to File(<state>/tokens/{anthropic,openai})
├── secrets.rs # switched from Static(token) to File(<state>.secrets/{anthropic,openai})
└── run.rs # registers the interceptor with two rules
```
@ -486,9 +486,10 @@ person doesn't redo it.
### `SecretValue::File`
Phase 3's per-launch snapshot becomes a per-launch *file write*. We
write the host's `accessToken` to `<state>/tokens/anthropic` (and
`.../openai`) with 0600 perms via atomic-write-then-rename. The
launcher passes the file path to microsandbox as a `SecretValue::File`.
write the host's `accessToken` to a host-only secret file (see next
subsection for *where*) with 0600 perms via atomic-write-then-rename.
The launcher passes the file path to microsandbox as a
`SecretValue::File`.
The patched msb's TLS-intercept proxy calls `SecretValue::resolve()`
at *connection-setup* time — every new TCP connection re-reads the
@ -496,6 +497,33 @@ file. So any host-side rotation (whether triggered by the user's
external `claude` use or by our interceptor hook below) is visible
to the very next request, without rebuilding the sandbox.
### Token files live *outside* the guest bind mount
The launcher bind-mounts the per-project `state_dir` into the guest at
`/agent-vm-state` as a *single* mount (libkrun caps virtio IRQs, so we
deliberately use one bind for all per-agent state instead of one per
agent). That makes mount placement security-critical: **anything under
`state_dir` is readable from inside the VM.**
The real access-token files therefore must *not* live under
`state_dir`. They sit in a sibling host-only directory
`${state_root}/<hash>.secrets/` (mode 0700), derived from `state_dir`
by `secrets::{anthropic,openai}_token_path` so the launcher and the
refresh hook agree on the path without passing it explicitly. The
microsandbox proxy reads these files on the *host* side via
`SecretValue::File`, so they never need to be mounted into the guest at
all.
This was a real leak found during Phase 4 end-to-end verification: the
first cut wrote the tokens to `<state>/tokens/{anthropic,openai}`, i.e.
*inside* the mount, so `cat /agent-vm-state/tokens/anthropic` in the
guest returned the host's real bearer — silently defeating the entire
"real tokens never enter the VM" guarantee. The nested test host masked
it (there the "real" token is itself the outer bridge's placeholder), so
it only surfaced once we grepped the guest filesystem for the token
during verification. A `token_files_live_outside_the_guest_mount` unit
test now guards the invariant.
### Request-interceptor hook (the OAuth refresh MITM)
`microsandbox-network` gained an `InterceptConfig` that the launcher
@ -525,8 +553,8 @@ clap subcommand mode:
2. Spawns `claude -p hi --model sonnet` (or `codex exec --skip-git-
repo-check 'Reply OK'`) on the **host** so the host CLI rotates
`~/.claude/.credentials.json` / `~/.codex/auth.json` the normal way.
3. Re-reads the rotated host file, rewrites
`<state>/tokens/{anthropic,openai}` so the next non-refresh
3. Re-reads the rotated host file, rewrites the host-only token file
(`<state>.secrets/{anthropic,openai}`) so the next non-refresh
request from the guest gets the new bearer via `SecretValue::File`.
4. Synthesizes an OAuth refresh-response JSON shaped like what the
upstream server would return, but with **placeholder** strings in
@ -536,6 +564,8 @@ clap subcommand mode:
The guest never holds a real token at any layer:
- `~/.claude/.credentials.json` always contains placeholders (Phase 3).
- The real-token file is on the host *outside* the guest bind mount
(see "Token files live outside the guest bind mount").
- The proxy substitutes real-for-placeholder on the way *out* (Phase 3).
- The OAuth refresh response also returns placeholders (Phase 4).
- The host CLI on the host is the only thing that ever touches real
@ -573,7 +603,7 @@ Response:
Confirmed on the same nested-VM test host as Phase 3. The hook ran,
host `claude -p` rotated the host file, the new bearer landed in
`<state>/tokens/anthropic`, and the synthesized response reached the
`<state>.secrets/anthropic`, and the synthesized response reached the
guest. `expires_in: 3499` is the freshly-derived seconds-until-expiry
of the just-rotated token.

36
PLAN.md
View file

@ -146,7 +146,7 @@ they're in already and doesn't redo the work.
only after a successful pull, so an interrupted pull never leaves the
microsandbox cache in an empty or stale state.
### Phase 3 — Static host-rooted secrets [done — submodule branch `agent-vm-secret-file`, agent-vm commit pending]
### Phase 3 — Static host-rooted secrets [done — submodule branch `agent-vm-secret-file`, committed `8cc036b`]
The big architectural payoff of moving to microsandbox: real tokens
never enter the VM.
@ -191,7 +191,7 @@ verified on the nested test host because of an outer credential bridge
that itself substitutes placeholders. Structurally equivalent to the
original Bash agent-vm's credential-proxy flow.
### Phase 4 — Refresh semantics [done — submodule branch `agent-vm-secret-file`, agent-vm commit pending]
### Phase 4 — Refresh semantics [done — committed `8e262c0`..`c7f7386`; end-to-end verified + leak fixed this session]
Tokens rotate; long-running sandbox sessions must survive that without
re-attaching. Phase 3 makes the access token swappable in principle;
@ -236,6 +236,38 @@ externally-rotated case.
end-to-end without the agent seeing an auth error and without manual
intervention.
**Verification session (2026-05-21):** stood the runtime back up on a
fresh host (installed `libkrunfw` bundle + the patched
`0.4.6+agent-vm.phase4` msb), rebuilt the image, and ran the launcher
against a real host Claude credential. Findings:
- **`images/build.sh` registry bug fixed.** Docker 29.x emits a stray
blank line to *stdout* on `docker inspect` of a missing container, so
`state=$(... || echo missing)` became `"\nmissing"` and never matched
the `missing` case — the script tried to `docker start` a nonexistent
container. Now whitespace-stripped with an empty→missing fallback.
- **Real-token leak into the guest found + fixed.** The token files
were written under `state_dir`, which is bind-mounted into the guest
at `/agent-vm-state`, so `cat /agent-vm-state/tokens/anthropic`
returned the host bearer. Moved to a host-only sibling
`<hash>.secrets/` (0700), never mounted; added a guard test. See
ARCHITECTURE.md "Token files live outside the guest bind mount".
- **Network layer verified.** Inside the guest, `credentials.json` and
PID1 environ show only placeholders; `api.anthropic.com`'s server
cert is issued by `CN=microsandbox CA` (traffic goes through the
intercept proxy). The final real-response leg still can't be checked
here — this is a doubly-nested host whose *outer* agent-vm bridge
already replaced the host token with its own placeholder
(`sk-ant-oat01-placeholder-proxy-managed`) and doesn't re-intercept
the nested VM's egress, so Anthropic returns 401. Same documented
limitation as Phase 3.
- **Codex path not exercisable here:** no `~/.codex/auth.json` on this
host, so the OpenAI/Codex websocket flow (the original stop point)
can't be authenticated. The `chatgpt.com` WebSocket support
(`inject_basic_auth(false)` + zero-copy fast path) is in place and
the codex CLI (now 0.133.0) is present in the verified image, but a
host with real Codex credentials is needed to confirm it end-to-end.
### Phase 5 — Fast-launch (deferred — wrong instrument for the job)
Originally framed around `Sandbox::from_snapshot(...)` on the assumption

View file

@ -52,13 +52,35 @@ pub struct CredsState {
pub openai_token_file: Option<PathBuf>,
}
/// Per-project location of the token file the proxy re-reads.
/// Host-only directory holding the real access-token files the proxy
/// re-reads via `SecretValue::File`.
///
/// **Must live outside `state_dir`.** The launcher bind-mounts
/// `state_dir` into the guest at `/agent-vm-state` (a single mount, to
/// stay under libkrun's virtio-IRQ cap), so anything written *under*
/// `state_dir` is readable from inside the VM — a `cat
/// /agent-vm-state/tokens/anthropic` would hand the in-VM agent the
/// host's real token and defeat the entire point of Phase 3/4. The
/// microsandbox proxy reads these files on the *host* side, so they
/// never need to be mounted; we keep them in a sibling `<hash>.secrets/`
/// directory that is never bind-mounted anywhere.
fn host_secret_dir(state_dir: &Path) -> PathBuf {
let name = state_dir
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("default");
let parent = state_dir.parent().unwrap_or(state_dir);
parent.join(format!("{name}.secrets"))
}
/// Per-project location of the token file the proxy re-reads. Lives in
/// the host-only [`host_secret_dir`], never inside the guest mount.
pub fn anthropic_token_path(state_dir: &Path) -> PathBuf {
state_dir.join("tokens/anthropic")
host_secret_dir(state_dir).join("anthropic")
}
pub fn openai_token_path(state_dir: &Path) -> PathBuf {
state_dir.join("tokens/openai")
host_secret_dir(state_dir).join("openai")
}
/// Read host credentials, write the token file (atomically, 0600) and
@ -66,7 +88,22 @@ pub fn openai_token_path(state_dir: &Path) -> PathBuf {
/// the written token files so the launcher can plumb them into
/// microsandbox's SecretValue::File config.
pub fn refresh(state_dir: &Path, project_guest_path: &str) -> Result<CredsState> {
std::fs::create_dir_all(state_dir.join("tokens"))?;
// The token files hold the host's *real* access tokens, so their
// directory must never be bind-mounted into the guest. Create it
// 0700 in the host-only sibling location (see `host_secret_dir`).
let secret_dir = host_secret_dir(state_dir);
std::fs::create_dir_all(&secret_dir)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(&secret_dir, std::fs::Permissions::from_mode(0o700))
.with_context(|| format!("chmod 700 {}", secret_dir.display()))?;
}
// Best-effort: an older agent-vm wrote token files to
// `state_dir/tokens/`, which is inside the guest bind mount. If a
// user upgrades over such a state dir, remove the stale dir so a real
// token doesn't linger where the guest can read it.
let _ = std::fs::remove_dir_all(state_dir.join("tokens"));
// First-run bypasses, run regardless of whether the user has host
// credentials for the provider. Without these the in-VM agent
@ -289,3 +326,32 @@ fn write_default_codex_config(path: &Path) -> Result<()> {
f.write_all(body.as_bytes())?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// Security invariant: the real-token files must never live under
/// `state_dir`, because the launcher bind-mounts `state_dir` into the
/// guest at `/agent-vm-state`. A token file under that path would be
/// readable by the in-VM agent (`cat /agent-vm-state/tokens/...`),
/// defeating the whole "real tokens never enter the VM" design.
#[test]
fn token_files_live_outside_the_guest_mount() {
let state_dir = Path::new("/home/u/.local/state/agent-vm/abc123");
for token in [
anthropic_token_path(state_dir),
openai_token_path(state_dir),
] {
assert!(
!token.starts_with(state_dir),
"{} must not be under the bind-mounted state dir {}",
token.display(),
state_dir.display(),
);
// ...but still derivable from it (same parent) so the launcher
// and the refresh hook agree on the path.
assert_eq!(token.parent().unwrap().parent(), state_dir.parent());
}
}
}