docs: refresh PLAN.md after Phase 2.x and re-plan remaining phases

- Add a "Phase 2.x — Post-MVP polish" section listing the work that
  landed between Phase 2 and Phase 3 (tracing, registry auto-recover,
  host-path cwd, profile flag, pull progress bar, explicit pull +
  update banner). Each entry references its commit so the next reader
  doesn't redo any of it.
- Expand Phase 3 with the things we learned: SecretEntry.value is
  captured at builder time so we need a File variant upstream; TLS
  intercept must be enabled for substitution to fire; only access
  tokens cross into the marker file, never refresh tokens.
- Add Phase 5 (snapshot-based fast launch) as a real phase rather than
  a vague polish bullet. Surfaced naturally from the "why is pull so
  slow" profiling — the EROFS materialize cost is the dominant pull
  cost, and snapshot/restore sidesteps both that and the ~1s libkrun
  boot. DoD: `agent-vm shell -- -c true` well under 500ms.
- Rename Phase 5→6 ("Distribution + polish + docs"). Add concrete
  items: microsandbox runtime auto-install, CLI flag promotion,
  .agent-vm.runtime.sh hook, CI smoke, macOS/aarch64 builds.
- Add "Discovered upstream issues" section recording the four
  microsandbox bugs/quirks we worked around (PullPolicy::Always not
  refreshing manifest digest; LayerDownloadProgress elided on fast
  registries; libkrun IRQ cap; no SDK-level "resolve manifest" helper).
  These should eventually be filed upstream.
- README status list extended with Phase 2.x done and 5/6 split.
This commit is contained in:
Evgeny Boger 2026-05-17 20:36:11 +03:00
parent d3914b990b
commit d979c31028
2 changed files with 144 additions and 24 deletions

160
PLAN.md
View file

@ -111,44 +111,158 @@ CODEX_HOME redirect, env propagation). The live API smoke was deferred — see
ARCHITECTURE.md "What Phase 2 deliberately doesn't do". Phase 3's host-OAuth
work closes the gap naturally.
### Phase 3 — Static host-rooted secrets [pending]
### Phase 2.x — Post-MVP polish [done — commits `7608f27`..`d3914b9`]
A series of small fixes landed between Phase 2 and Phase 3, all triggered by
real testing on the user's laptop. Listed here so the next reader knows
they're in already and doesn't redo the work.
- **`RUST_LOG` wiring** (`7608f27`). `tracing-subscriber` initialized in
`main`, defaults to `warn`. The microsandbox stack is silent otherwise.
- **Auto-recover the local registry container** (`a57ed6d`). `build.sh`'s
`ensure_registry` was rewritten as a state machine that handles every
`docker ps` state, polls `/v2/` after start, and recreates from scratch
if a stale container is running with no port mapping. Plus per-phase
banners so long waits don't look like a hang.
- **Mirror the host project path inside the guest** (`92ff582`). `cwd` is
bind-mounted at the same absolute path, so anything the agent emits
(compiler errors, stack traces, file:line references) is interpretable on
the host. Paths under tmpfs mount points (`/tmp`, `/run`, `/dev/shm`,
`/var/run`) fall back to `/workspace` with a warning, because the guest
tmpfs-mounts them at boot and wipes any patch-created mount point.
- **`AGENT_VM_PROFILE=1`** (`127f6b3`). Prints per-phase wall-time
(create / run / stop / remove) for the launcher. Confirmed total is
~1.5 s, dominated by VM boot (~1.0 s of libkrun kernel boot).
- **Pull progress bar** (`2489168`..`66ed8f3`..`3ffd6b6`..`984680a`).
Two-phase indicatif renderer: spinner with text during download, real
byte-weighted bar during materialize. Single line, single spinner, ETA
based on materialize-only rate (no more "29 minute" → "17 second" jumps).
- **`agent-vm pull` + per-launch update-available banner**
(`bfab9d3`..`d3914b9`). Pulls are explicit; `agent-vm shell` only does a
cheap manifest-digest HEAD against the registry and prints a banner when
the per-platform digest differs from what we last pulled. The "what we
last pulled" digest is tracked in our own marker file
(`~/.local/state/agent-vm/pulled-digests/<hash>`), atomically written
only after a successful pull, so an interrupted pull never leaves the
microsandbox cache in an empty or stale state.
### Phase 3 — Static host-rooted secrets [next]
The big architectural payoff of moving to microsandbox: real tokens never
enter the VM.
- Branch `vendor/microsandbox` to add a `SecretValue::File(PathBuf)` variant
alongside the existing `String` value. The TLS-intercept proxy re-reads the
file on every substitution.
alongside the existing `String` value. The TLS-intercept proxy re-reads
the file on every substitution. Required because microsandbox's
`SecretEntry.value: String` is captured at sandbox-builder time, with no
runtime update path.
- Enable TLS interception on the sandbox (`network(|n| n.tls(...))`) —
microsandbox's secrets are only substituted on TLS-intercepted
connections to allowed hosts, and the guest needs the sandbox CA in its
trust store.
- `agent-vm` on startup snapshots host `~/.claude/.credentials.json` and
`~/.codex/auth.json` into `<state>/tokens/{anthropic,openai}.token` files.
`~/.codex/auth.json` into `<state>/tokens/{anthropic,openai}.token`
files (just the `access_token`, never the refresh token).
- Register file-backed secret entries for `api.anthropic.com`,
`api.openai.com`, `chatgpt.com`, `platform.claude.com`, etc.
- Cross-instance lock so two `agent-vm` processes don't fight over the token
files.
`api.openai.com`, `chatgpt.com`, `platform.claude.com`,
`auth.openai.com`, etc.
- Cross-instance lockfile under `<state>/tokens/` so two `agent-vm`
processes don't race when both want to refresh.
**Done when:** inside the guest, `cat /proc/$$/environ | tr '\0' '\n' | grep
ANTHROPIC` shows only placeholders, while a real Claude request succeeds.
**Done when:** inside the guest, `cat /proc/$$/environ | tr '\0' '\n' |
grep -i token` shows only placeholders, while a real Claude (host OAuth)
request succeeds end-to-end and writes a turn to disk.
### Phase 4 — Refresh semantics [pending]
Tokens rotate; long-running sandbox sessions must survive that without
re-attaching. Phase 3 makes the access token swappable; this phase teaches
agent-vm to actually do the swap.
- inotify (Linux) / kqueue (macOS) watcher on host creds files: when host
Claude/Codex rotates tokens, we re-snapshot to the file microsandbox watches.
- Proactive expiry watch: when the access token is < 5 minutes from expiry
and no host activity has refreshed it, spawn `claude -p "ping" --model
sonnet` / `codex exec --skip-git-repo-check "Reply OK"` on the host.
Claude/Codex rotates tokens externally (the user runs `claude` on the
host, etc.), we re-snapshot to the file microsandbox watches.
- Proactive expiry watch: when the access token is < 5 minutes from
expiry and no host activity has refreshed it, spawn `claude -p "ping"
--model sonnet` / `codex exec --skip-git-repo-check "Reply OK"` on the
host so the host-side CLI rotates the file.
- Single-flight per credential; cross-instance lockfile so concurrent
`agent-vm` instances don't all kick off a refresh at once.
**Done when:** a multi-hour session crosses a token rotation without the agent
seeing an auth error.
**Done when:** a multi-hour session crosses a token rotation without the
agent seeing an auth error and without manual intervention.
### Phase 5 — Polish & docs [pending]
### Phase 5 — Snapshot-based fast launch [pending — perf win]
- `.agent-vm.runtime.sh` project hook.
- `--memory N` flag (passed through to microsandbox builder).
- Full arg passthrough to the agent command.
- README rewrite: install, setup, usage, troubleshooting.
- Smoke tests: at least one end-to-end test that boots the image and runs a
trivial agent command.
Surfaced during Phase 2.x profiling and discussed with the user when they
asked why `agent-vm pull` was so slow. Every `agent-vm shell` currently
spends ~1.0 s booting the libkrun kernel + initing the guest agent;
microsandbox has a snapshot/restore API that should drop that to ~100 ms
*and* skip the EROFS materialize step on a re-pull.
**Done when:** README is publishable; one CI smoke test green.
- One-shot snapshot generation: `agent-vm setup` (or a new `agent-vm
snapshot`) boots the image once, lets it reach a quiet steady state,
takes a snapshot, and stashes it under `<microsandbox-home>/snapshots/`.
- Launcher: if a snapshot exists, use `Sandbox::builder(...).from_snapshot
(snapshot_path)` instead of `.image(...)`. Falls back to cold boot if
the snapshot is missing or wrong-digest.
- Re-snapshot whenever `agent-vm pull` lands a new manifest (snapshots are
pinned to a specific image digest by microsandbox).
- Bake the per-agent symlinks + state mount point into the snapshot so
the launcher doesn't need to re-apply patches each invocation.
**Done when:** `time agent-vm shell -- -c true` is well under 500 ms in
the steady state.
### Phase 6 — Distribution + polish + docs [pending]
The "ready to share with a teammate" phase.
- **Auto-install of the microsandbox runtime.** The agent-vm binary ships
on its own; `~/.microsandbox/{bin/msb, lib/libkrunfw.so.5.2.1}` are
needed but not bundled. Wrap `microsandbox::setup::install()` so first
run downloads them automatically if missing. Verify version matches the
prebuilt the binary was built against.
- **CLI flag promotion.** `--memory N`, `--cpus N`, `--image REF`,
`--no-update-check`. Today these are env-var only (`AGENT_VM_*`).
- **`.agent-vm.runtime.sh` project hook.** Script executed in the guest
immediately before the agent starts, for project-local setup
(`npm install`, `docker compose up`, etc.).
- **README rewrite.** Install, prereqs, setup, usage, troubleshooting,
the registry/marker/snapshot internals at a high level.
- **CI smoke test.** GitHub Actions workflow that builds the image, runs
`agent-vm setup --no-verify`, and `agent-vm shell -- -c 'echo ok'`.
- **macOS/aarch64 binary.** Cross-compile or native-build on each
platform microsandbox supports.
**Done when:** README is publishable, CI smoke green on at least linux-amd64,
binary works from a fresh checkout on a host where microsandbox runtime is
not pre-installed.
## Discovered upstream issues
Things we worked around during Phase 2.x that should eventually be filed
or fixed in `wirenboard/microsandbox`:
1. **`PullPolicy::Always` doesn't refresh the cached manifest digest.**
It re-fetches layer blobs correctly, but `Image::persist`'s fast-path
detection skips the DB update under the same reference even when the
per-platform manifest digest changed. We work around it with our own
marker file rather than `Image::remove` (because remove + re-pull
opens an empty-cache window).
2. **`LayerDownloadProgress` events are often elided** for fast registries
(we never see them with localhost). Only `LayerDownloadComplete` fires.
Not exactly a bug, but undocumented and bit us when we tried to drive a
download-bytes bar.
3. **libkrun virtio IRQ cap is low** (~6 devices). We're constrained to
2 bind mounts on top of the OCI overlay's 2-device cost. Bigger fan-out
needs upstream tuning of the libkrun build.
4. **Manifest media-type assumptions.** Microsandbox stores the
per-platform manifest digest; a registry HEAD on a tag returns the
multi-arch index digest by default. Either would be fine to use, but
the SDK doesn't expose either as "ask the registry what's there now"
so we end up doing raw HTTP. A `Image::resolve(reference) -> RemoteRef`
helper would clean this up.
## Working agreements

View file

@ -21,7 +21,13 @@ until v1 of the rewrite ships.
- Phase 1 (base OCI image + `agent-vm setup`): done.
- Phase 2 (launcher MVP — `claude`/`codex`/`opencode`/`shell`): done; live
API smoke deferred to Phase 3.
- Phase 3 (host-rooted secrets via microsandbox TLS intercept): pending.
- Phase 2.x (RUST_LOG, host-path mounting, pull progress bar,
`agent-vm pull` + update-available banner, registry auto-recovery):
done.
- Phase 3 (host-rooted secrets via microsandbox TLS intercept): **next**.
- Phase 4 (token-refresh semantics): pending.
- Phase 5 (snapshot-based fast launch): pending.
- Phase 6 (distribution + polish + docs): pending.
## Building