mirror of
https://github.com/wirenboard/agent-vm.git
synced 2026-07-09 16:00:54 +00:00
6513050a31
Profiling (LAUNCH-PROFILE.md) found the launch was dominated by PRE-BOOT host work, not the VM boot. Three fixes, ~2.5s -> ~1.8s on a warm launch: - secrets.rs: cache the host git identity (24h TTL, validated strings only). `gh api user` was an uncached ~1.29s HTTPS round-trip on every launch's critical path (added in v0.1.14) — the main 1.3s->2.5s regression. - run.rs: spawn the ghcr.io update-banner check instead of awaiting it (~0.9s off the hot path; banner still fires, just doesn't block boot). - run.rs + images/Dockerfile: move the chrome-MCP MITM-CA `certutil` import out of the per-launch in-guest prelude into the agent-vm-chrome-mcp wrapper. It now runs once at MCP startup (as the chrome user that owns the NSS DB), off the launch path and skipped when chrome is unused (~270ms). Verified end-to-end: the proxy serves a `microsandbox CA`-signed cert that validates against the CA the wrapper imports. Also: kernel A/B showed the nested-virt libkrunfw rebuild adds only ~190ms (KVM ~100, conntrack ~62); DEFERRED_STRUCT_PAGE_INIT and split_irqchip have no wall-clock effect. Build/measure scripts under profiling/. NOTE: the Dockerfile (wrapper) and binary (prelude removal) must ship together — the image must be rebuilt to :latest with the new wrapper before this binary is published, or chrome MCP loses its CA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
135 lines
7.4 KiB
Markdown
135 lines
7.4 KiB
Markdown
# agent-vm launch profiling — findings
|
||
|
||
Host: AMD EPYC, 16 vCPU, nested virt (`/dev/kvm`, `kvm_amd.nested=1`). Image cached.
|
||
All headline numbers are **wall-clock**, from interleaved A/Bs (host drift cancels),
|
||
`drop_caches` between rounds where noted. Measured with `AGENT_VM_PROFILE=1` plus
|
||
sub-timers added to `run.rs` (pre-boot phases + build/spawn+boot+relay) and the
|
||
runtime's own `runtime.log` wall timestamps.
|
||
|
||
## TL;DR
|
||
|
||
A launch in a GitHub-remote repo broke down (default 2 vCPU / 2 GiB) as:
|
||
|
||
```
|
||
pre: session+reap ............... 0.3 ms
|
||
pre: update-check (ghcr HEAD) ... 886 ms ← notify_if_update_available (banner only)
|
||
pre: repo-detect ................ 2 ms
|
||
pre: secrets (gh auth token) .... 40 ms
|
||
pre: gh api user ................ 1290 ms ← v0.1.14 author identity ← THE regression
|
||
pre: TOTAL pre-boot ............. 2220 ms ← LARGER than the actual VM boot
|
||
create (guest kernel boot) ..... 1500 ms
|
||
run (incl. chrome certutil) .... 270 ms
|
||
stop / remove .................. 50 ms
|
||
TOTAL ~4.1 s
|
||
```
|
||
|
||
**~2.2s of the launch happens before the kernel even starts, and it's two uncached
|
||
blocking GitHub/registry network round-trips.** The original `[profile] create` timer
|
||
started *after* all of this, which is why earlier profiling missed it entirely.
|
||
|
||
## The regression: `gh api user` (pre-boot), not the kernel
|
||
|
||
`discover_host_git_identity()` (added in commit `0c3bb51`, **v0.1.14** — "bake host
|
||
gh/git identity into the guest gitconfig", the exact regression window) calls
|
||
**`gh api user` first**, an HTTPS round-trip to api.github.com, falling back to the
|
||
instant local gitconfig only if it fails. No cache. Measured ~1.26–1.31s here, every
|
||
launch. Adding ~1.29s to a ~1.3s baseline ≈ the reported 2.5s.
|
||
|
||
The ghcr.io **update-check** (`notify_if_update_available`, commit `bfab9d3`) is a second
|
||
per-launch blocking HEAD (~0.89s) purely to print a "newer image available" banner.
|
||
|
||
### Fixes (implemented on this branch, measured)
|
||
|
||
1. **Cache the resolved git identity** (`secrets.rs`, 24h TTL, validated strings only,
|
||
never tokens; preserves the canonical `gh` identity incl. `gh_login`). Pays the
|
||
`gh api user` cost once per day instead of every launch.
|
||
2. **Make the update-check non-blocking** (`run.rs`): spawn it concurrently with boot
|
||
instead of awaiting it. Banner still prints (during boot); never delays launch.
|
||
|
||
| | pre-boot | total wall |
|
||
|---|---|---|
|
||
| before | ~2220 ms | ~4100 ms |
|
||
| after, run 1 (cold identity cache) | ~1300 ms | ~3280 ms |
|
||
| **after, warm cache** | **~32 ms** | **~1900 ms** |
|
||
|
||
`gh api user` 1290ms → 22µs; update-check 886ms → ~0. **~2.19s off every launch after
|
||
the first.** No downside, no kernel rebuild. This alone undoes the regression.
|
||
|
||
## Secondary: the ~1.5s guest-kernel boot floor
|
||
|
||
`create` ≈ guest kernel boot (`build()` is ~3µs; `entering VM → agentd core.ready`).
|
||
The console (`hvc0`) attaches ~1.3s in, so early boot isn't visible in `kernel.log`;
|
||
the A/B below is the real attribution. Five+1 kernels built from one tree
|
||
(configs grepped, not assumed), 10 interleaved rounds @1 GiB, `drop_caches` per round:
|
||
|
||
| kernel | config | create mean ± sd | Δ |
|
||
|---|---|---|---|
|
||
| stock | upstream libkrunfw (no KVM, no netfilter) | 1.488 ± 0.10 s | — |
|
||
| stock+KVM | + CONFIG_KVM/KVM_INTEL/KVM_AMD | 1.587 ± 0.15 s | **+99 ms (KVM)** |
|
||
| heavy_nonf | KVM, netfilter off (+mqueue) | 1.615 ± 0.12 s | ≈ stock+KVM ✓ |
|
||
| heavy_legacy | + conntrack/NAT/iptables-legacy/bridge | 1.677 ± 0.03 s | **+62 ms (conntrack)** |
|
||
| heavy (current) | + full nf_tables/XT/IPv6/VLAN | 1.680 ± 0.03 s | +3 ms (nf_tables ≈ 0) |
|
||
| heavy+deferred | heavy + DEFERRED_STRUCT_PAGE_INIT | 1.632 ± 0.03 s | no help |
|
||
|
||
**The nested-virt kernel rebuild adds only ~190ms total to boot** (KVM ~100ms,
|
||
conntrack/iptables ~62ms, nf_tables ~3ms). So it's a real but *minor* secondary cost:
|
||
|
||
- **KVM (~100ms)** is *required* for nested virt — irremovable.
|
||
- **conntrack/iptables-legacy (~62ms)** is the unavoidable cost of docker bridge+SNAT
|
||
(the conntrack hashtable auto-sizes from RAM). With `CONFIG_MODULES is not set` +
|
||
`nomodule`, it can't be made a module — it's built-in or absent. Drop it only if you
|
||
don't need docker networking by default.
|
||
- **nf_tables/XT/IPv6/VLAN (~3ms)** — droppable, but saves essentially nothing. Not
|
||
worth the docker-iptables-nft→legacy fallback risk.
|
||
|
||
### Two levers that do NOT work (verified, don't pursue)
|
||
|
||
- **`CONFIG_DEFERRED_STRUCT_PAGE_INIT=y`**: no help at 1 GiB, slightly *worse* at 4 GiB.
|
||
Its `defer_init()` heuristic only defers past a 128 MB section threshold after low
|
||
zones init; a 1–4 GiB single-node guest has nothing to defer (it targets TB-scale RAM).
|
||
- **`split_irqchip`**: ~515ms swing in the runtime's `boot_time_ms` metric but **zero
|
||
wall-clock effect** (1.91s vs 1.88s create). `boot_time_ms` excludes ~0.9s of early
|
||
boot and is a misleading proxy — rank kernels on wall-clock `create` only. vCPU count:
|
||
also negligible (1/2/4 ≈ 1.84/1.84/1.89s).
|
||
|
||
## Other real levers
|
||
|
||
- **Guest memory** (real wall-clock, but EPT/page-materialization under nested virt, not
|
||
struct-page init): create ≈ 1.49s @1G / 1.68–1.92s @2G / 2.9s @4G. Lower the default
|
||
(`AGENT_VM_MEMORY_GIB`, currently 2) for sessions that don't need 2 GiB (~0.2s+).
|
||
- **Chrome-MCP CA `certutil` (~270ms)** in the `run` phase — **fixed on this branch.**
|
||
`run.rs` used to run `sudo -u chrome certutil -A …` synchronously in the in-guest
|
||
prelude before exec'ing the agent, on *every* launch. But chromium ignores the system
|
||
CA bundle (which `update-ca-certificates` already populates at boot) and only honors
|
||
its per-user NSS DB, so the CA must be imported there or chrome-devtools-mcp fails
|
||
every HTTPS page with `ERR_CERT_AUTHORITY_INVALID`. The cert can't be baked into the
|
||
shared image (the CA is generated per-install on the host); the import repeated every
|
||
launch only because the NSS DB lives in the ephemeral rootfs. **Fix:** moved the import
|
||
into the in-image `agent-vm-chrome-mcp` wrapper (`images/Dockerfile`) — it runs once at
|
||
MCP startup, as the `chrome` user that owns the DB, off the launch critical path and
|
||
skipped entirely when chrome is unused. Measured: `run` phase 310ms → ~38ms.
|
||
**Coupling:** the `run.rs` and Dockerfile changes must ship together — until the
|
||
template image is rebuilt with the new wrapper, dropping the prelude import would leave
|
||
chrome MCP without the CA.
|
||
|
||
## Recommended order (by impact × safety)
|
||
|
||
1. **Cache the git identity** + **non-blocking update-check** — ~2.19s, implemented here,
|
||
zero downside, no rebuild. This *is* the regression fix.
|
||
2. **Chrome-MCP certutil moved into the image wrapper** — ~270ms off the `run` phase,
|
||
implemented on this branch (run.rs + images/Dockerfile; ship together).
|
||
3. **Lower default guest memory** if 2 GiB is more than agents need — ~0.2s+.
|
||
4. Kernel: leave it. The ~190ms it adds is mostly required (KVM, conntrack). Do **not**
|
||
enable deferred-page-init or chase split_irqchip. Optionally drop nf_tables/IPv6 NF to
|
||
shave ~3ms only if you also accept iptables-legacy.
|
||
|
||
## Reproduce
|
||
|
||
```bash
|
||
cd <github-remote repo>
|
||
AGENT_VM_PROFILE=1 agent-vm shell true # prints pre-boot phases + create/run/stop
|
||
# pre-boot network cost, in isolation:
|
||
time gh api user >/dev/null ; time curl -sI https://ghcr.io/v2/wirenboard/agent-vm-template/manifests/latest
|
||
# kernel A/B: build variants from libkrunfw-src (one tree, grep each .config), swap the
|
||
# .so next to msb, measure interleaved with drop_caches — see measure_variants.sh.
|
||
```
|