mirror of
https://github.com/wirenboard/agent-vm.git
synced 2026-07-09 16:00:54 +00:00
Merge irq-multi-mount: lift --mount cap from ~11 to ~100+ on x86_64
Two coordinated changes that together raise the per-sandbox virtio
device cap on x86_64 by ~10x:
1. Enable msb_krun's userspace split irqchip in the runtime
(vendor/microsandbox bump). Lifts the libkrun virtio-IRQ allocator
ceiling from IRQ 15 (KVM in-kernel IOAPIC, 24 pins) to IRQ 223
(userspace IOAPIC, 256 pins). Requires the wirenboard/libkrun fork
(pinned in Cargo.toml [patch.crates-io] at 38e46f2) with three
correctness fixes on top of upstream msb_krun 0.1.13.
2. Patch libkrunfw to bump x86 COMMAND_LINE_SIZE 2048 → 16384.
New `libkrunfw-overrides/cmdline-size_x86_64.patch` lifts the cap
so the assembled cmdline (~36 bytes per virtio_mmio device) doesn't
silently truncate past ~10 user mounts.
CI workflow hardening that landed alongside:
- Cache key now keyed per-arch (an aarch64 patch edit no longer
invalidates the x86_64 cache).
- `config-libkrunfw*` skip pattern uses a bare suffix so future
`config-libkrunfw-tdx_${ARCH}.patch` / `-sev_` variants are also
excluded from the kernel source-patch pipeline.
User-facing knobs (`--mount` doc in run.rs, README troubleshooting,
ARCHITECTURE/PLAN narrative) updated to reflect the new ceiling.
This commit is contained in:
Evgeny Boger
commit
a850ec8edb
10 changed files with 253 additions and 85 deletions
29
.github/workflows/release-npm.yml
vendored
29
.github/workflows/release-npm.yml
vendored
|
|
@ -223,9 +223,10 @@ jobs:
|
|||
id: lk-cache
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
# Key on version + arch + sha256 of the patch + workflow
|
||||
# file so a patch edit or pin bump invalidates the cache.
|
||||
key: libkrunfw-${{ steps.lkv.outputs.libkrunfw_version }}-${{ matrix.arch }}-${{ hashFiles('libkrunfw-overrides/config-libkrunfw_x86_64.patch', '.github/workflows/release-npm.yml') }}
|
||||
# Key on version + arch + sha256 of the patches that affect
|
||||
# this arch + workflow file. Hash only `*_${arch}.patch` so an
|
||||
# aarch64 patch edit doesn't invalidate the x86_64 cache.
|
||||
key: libkrunfw-${{ steps.lkv.outputs.libkrunfw_version }}-${{ matrix.arch }}-${{ hashFiles(format('libkrunfw-overrides/*_{0}.patch', matrix.arch), '.github/workflows/release-npm.yml') }}
|
||||
path: libkrunfw-out/
|
||||
|
||||
- name: Install kernel build deps
|
||||
|
|
@ -248,7 +249,29 @@ jobs:
|
|||
git clone --branch "v${LK_V}" --depth 1 \
|
||||
https://github.com/containers/libkrunfw.git libkrunfw-src
|
||||
cd libkrunfw-src
|
||||
# 1) kbuild .config seed (CONFIG_KVM=y + Intel/AMD backends).
|
||||
patch -p1 < ../libkrunfw-overrides/config-libkrunfw_${ARCH}.patch
|
||||
# 2) Kernel source patches: libkrunfw's Makefile runs
|
||||
# `patch -p1 -d $(KERNEL_SOURCES) < $patch` for every file
|
||||
# matching `patches/0*.patch` after extracting the tarball.
|
||||
# Drop arch-matching source patches from libkrunfw-overrides/
|
||||
# there so they ride the same pipeline. File-naming convention:
|
||||
# `<topic>_${ARCH}.patch` (arch-specific). The `0999-overrides-`
|
||||
# prefix sorts them strictly after libkrunfw's own numbered
|
||||
# patches (which currently top out in the 0010-range).
|
||||
#
|
||||
# `config-libkrunfw*` files are kbuild .config seeds, not
|
||||
# kernel source patches — exclude them here. The skip pattern
|
||||
# uses a bare `config-libkrunfw*` (no trailing underscore) so
|
||||
# future variants like `config-libkrunfw-tdx_${ARCH}.patch`
|
||||
# (hyphen) are also caught.
|
||||
shopt -s nullglob
|
||||
for p in ../libkrunfw-overrides/*_${ARCH}.patch; do
|
||||
case "$(basename "$p")" in
|
||||
config-libkrunfw*) continue;;
|
||||
esac
|
||||
cp "$p" "patches/0999-overrides-$(basename "$p")"
|
||||
done
|
||||
# olddefconfig (run by make rule on first invocation) resolves
|
||||
# support symbols (KVM_X86, KVM_SMM, KVM_HYPERV, KVM_VFIO, …).
|
||||
make -j"$(nproc)"
|
||||
|
|
|
|||
|
|
@ -219,10 +219,17 @@ project a given sandbox is bound to.
|
|||
|
||||
### Mounts: one for workspace, one for state, no third
|
||||
|
||||
The microsandbox runtime caps virtio devices via libkrun's IRQ pool, and an
|
||||
OCI rootfs already consumes two slots (EROFS lower + ext4 upper). Adding a
|
||||
bind mount per agent state directory (claude, codex, opencode) puts us over
|
||||
the cap — `RegisterBlockDevice(IrqsExhausted)` at boot.
|
||||
When this layout was chosen, the microsandbox runtime was running with
|
||||
libkrun's default in-kernel IOAPIC, which hands ~11 IRQs to virtio-mmio
|
||||
devices total on x86_64. The OCI rootfs already consumes two slots (EROFS
|
||||
lower + ext4 upper), plus virtio-net + vsock + console + agentd's
|
||||
serial — adding a bind mount per agent state directory (claude, codex,
|
||||
opencode) pushed us over and `RegisterBlockDevice(IrqsExhausted)` at boot
|
||||
followed. We later lifted the underlying cap by enabling `msb_krun`'s
|
||||
userspace split irqchip (see "Split irqchip and the virtio-IRQ ceiling"
|
||||
below; cap is now ~219), but the one-workspace + one-state layout still
|
||||
makes sense regardless: one virtio-fs server, one rootfs `patch` entry
|
||||
per agent, and a stable on-host layout.
|
||||
|
||||
Resolution:
|
||||
|
||||
|
|
@ -237,7 +244,46 @@ Resolution:
|
|||
the codex binary itself and a symlink would shadow it.
|
||||
|
||||
This keeps us at two virtio bind mounts no matter how many agents we add
|
||||
later.
|
||||
later, and leaves plenty of IRQ headroom for user-supplied `--mount`
|
||||
arguments now that the split irqchip is on.
|
||||
|
||||
### Split irqchip and the virtio-IRQ ceiling
|
||||
|
||||
`msb_krun` exposes a `MachineBuilder::split_irqchip(bool)` knob. With it
|
||||
off (default), libkrun uses KVM's in-kernel IOAPIC, which is hard-capped at
|
||||
24 pins and only hands IRQs 5..=15 to virtio-mmio — about 11 usable IRQs
|
||||
for the whole VM. That fills up fast: rootfs lower + rootfs upper +
|
||||
virtio-net + virtio-vsock + virtio-console + virtio-fs (project) +
|
||||
virtio-fs (state) already saturates it on this build, so an extra
|
||||
`--mount` would trip `RegisterNetDevice(IrqsExhausted)` at boot.
|
||||
|
||||
With split_irqchip enabled, `msb_krun` runs a userspace IOAPIC backed by
|
||||
an event-loop thread it spawns automatically. The pin count rises to 219,
|
||||
which puts the practical ceiling on `--mount` well into the hundreds. The
|
||||
trade-off is one extra worker thread per VM and a slightly hotter IRQ
|
||||
delivery path; we accept it because the IRQ headroom is the difference
|
||||
between "one or two extra mounts work" and "you can stop worrying about
|
||||
the cap." aarch64/riscv64 ignore the knob — their GIC/AIA models already
|
||||
expose >200 IRQs.
|
||||
|
||||
The runtime sets `split_irqchip(true)` unconditionally in
|
||||
`vendor/microsandbox/crates/runtime/lib/vm.rs`. The user-facing
|
||||
`--mount` doc and Phase 7's wiring in `crates/agent-vm/src/run.rs` were
|
||||
updated to drop the pre-cap warning that previously fronted the limit.
|
||||
|
||||
The change also bumped `msb_krun` from 0.1.12 → 0.1.13 across the
|
||||
vendor crates (`runtime`, `filesystem`, `network`). 0.1.12's userspace
|
||||
IOAPIC was unusable in practice: its IRR was a single `u32` so any IRQ
|
||||
delivered on pin ≥ 32 was dropped without notice, and the redirection-
|
||||
table register-index calculation in `read`/`write` did an unchecked
|
||||
`ioregsel - IOAPIC_REG_REDTBL_BASE` that wrapped on any access below
|
||||
the redirection-table base — which the guest performs during normal
|
||||
IOAPIC programming. Both fixes landed in 0.1.13, published 2026-05-26.
|
||||
The PLAN's Discovered Upstream Issue #3 was originally attributed to
|
||||
"the libkrun IRQ cap"; with hindsight, the cap is a real KVM-level
|
||||
ceiling but the multi-mount boot failure that finally drove this work
|
||||
was a separate, fixable bug inside `msb_krun_devices`'s userspace
|
||||
IOAPIC, only ever reached once the split irqchip was turned on.
|
||||
|
||||
### Interactive attach vs. non-TTY exec
|
||||
|
||||
|
|
@ -517,10 +563,13 @@ to the very next request, without rebuilding the sandbox.
|
|||
### Token files live *outside* the guest bind mount
|
||||
|
||||
The launcher bind-mounts the per-project `state_dir` into the guest at
|
||||
`/agent-vm-state` as a *single* mount (libkrun caps virtio IRQs, so we
|
||||
deliberately use one bind for all per-agent state instead of one per
|
||||
agent). That makes mount placement security-critical: **anything under
|
||||
`state_dir` is readable from inside the VM.**
|
||||
`/agent-vm-state` as a *single* mount. The single-bind shape originally
|
||||
fell out of libkrun's tight virtio-IRQ cap (one bind for all per-agent
|
||||
state instead of one per agent — see "Mounts: one for workspace, one
|
||||
for state, no third"), and we've kept it after lifting the cap because
|
||||
it gives a stable on-host layout and a single virtio-fs server. That
|
||||
makes mount placement security-critical: **anything under `state_dir`
|
||||
is readable from inside the VM.**
|
||||
|
||||
The real access-token files therefore must *not* live under
|
||||
`state_dir`. They sit in a sibling host-only directory
|
||||
|
|
|
|||
73
Cargo.lock
generated
73
Cargo.lock
generated
|
|
@ -1641,11 +1641,6 @@ name = "hashbrown"
|
|||
version = "0.17.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
|
||||
dependencies = [
|
||||
"allocator-api2",
|
||||
"equivalent",
|
||||
"foldhash 0.2.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hashlink"
|
||||
|
|
@ -2409,15 +2404,6 @@ dependencies = [
|
|||
"hashbrown 0.16.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lru"
|
||||
version = "0.18.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8a860605968fce16869fd239cf4237a82f3ac470723415db603b0e8b6c8d4fb9"
|
||||
dependencies = [
|
||||
"hashbrown 0.17.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "lru-slab"
|
||||
version = "0.1.2"
|
||||
|
|
@ -2574,7 +2560,7 @@ dependencies = [
|
|||
"hickory-proto",
|
||||
"ipnetwork",
|
||||
"libc",
|
||||
"lru 0.16.4",
|
||||
"lru",
|
||||
"microsandbox-protocol",
|
||||
"microsandbox-utils",
|
||||
"msb_krun",
|
||||
|
|
@ -2682,9 +2668,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7c29ba9c9c8a38cd001c0c9af16bd3c75be328ab8c95a3aa933b125c1d93f41c"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"crossbeam-channel",
|
||||
"kvm-bindings",
|
||||
|
|
@ -2702,9 +2687,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_arch"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5eb4327e3efe423f3bcdca0390589193834b9f73eb54bb13b90f48f1755b499e"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"kvm-bindings",
|
||||
"kvm-ioctls",
|
||||
|
|
@ -2718,15 +2702,13 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_arch_gen"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4bc7dbc08509138172ccd3b40df5c1501ad498ba12f2643eb013320205f9c987"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
|
||||
[[package]]
|
||||
name = "msb_krun_cpuid"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "03d88b81a52ae5b3fe1afb1692b0785a4fe946c77bbc9e43a030e84e90627819"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"kvm-bindings",
|
||||
"kvm-ioctls",
|
||||
|
|
@ -2735,9 +2717,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_devices"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d0f6d808b3c4d5a7c24fba02a64f2f704da973cf6820deedd2811c2ec795b9fc"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"bitflags 1.3.2",
|
||||
"capng",
|
||||
|
|
@ -2749,7 +2730,7 @@ dependencies = [
|
|||
"libc",
|
||||
"libloading",
|
||||
"log",
|
||||
"lru 0.18.0",
|
||||
"lru",
|
||||
"msb_krun_arch",
|
||||
"msb_krun_hvf",
|
||||
"msb_krun_polly",
|
||||
|
|
@ -2763,9 +2744,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_hvf"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bce2de785e2cceda28ce03f20a9eeb895ac947ab66b09db31077a070bd6dda86"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"crossbeam-channel",
|
||||
"libloading",
|
||||
|
|
@ -2775,9 +2755,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_kernel"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2047a3f886330138dd6a681ef96b0b2257b494f295535bdd641841068f5847bc"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"msb_krun_utils",
|
||||
"vm-memory 0.16.2",
|
||||
|
|
@ -2785,9 +2764,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_polly"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "db57f147c6e1aca9a14d137ada76b6df78b538a0673d86b33652be78e9784499"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"msb_krun_utils",
|
||||
|
|
@ -2795,18 +2773,16 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_smbios"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6c0c3a0923f194852028fd0e809a8d26c5465bc550f59c08b25bae5dd9885457"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"vm-memory 0.16.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "msb_krun_utils"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9fbae5590c539ee8b9ed0e2d4c50c6ee5b546f03f9eb2408b885628514ddb9a6"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"bitflags 1.3.2",
|
||||
"crossbeam-channel",
|
||||
|
|
@ -2819,9 +2795,8 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "msb_krun_vmm"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0bd62153e5ffe89a76b775c89b9c01b65d38c67a32ca8d516a910d4cb497d367"
|
||||
version = "0.1.13"
|
||||
source = "git+https://github.com/wirenboard/libkrun?rev=38e46f2#38e46f2ea49fba33814191b7e16756d6801a8d11"
|
||||
dependencies = [
|
||||
"bzip2",
|
||||
"crossbeam-channel",
|
||||
|
|
|
|||
31
Cargo.toml
31
Cargo.toml
|
|
@ -11,6 +11,37 @@ edition = "2024"
|
|||
license = "MIT"
|
||||
repository = "https://github.com/wirenboard/agent-vm"
|
||||
|
||||
# wirenboard's libkrun fork: msb_krun_* 0.1.13 with three fixes on top
|
||||
# that are required for the userspace split irqchip to work and for the
|
||||
# per-VM virtio-mmio cap to scale past ~11 devices:
|
||||
# - virtio-console pre-activation panic guard (a3d4e3b) — without
|
||||
# this, enabling split_irqchip(true) crashes the VMM during boot
|
||||
# when a spurious epoll event hits the console device before the
|
||||
# guest driver activates it.
|
||||
# - mptable: emit one MP_INTSRC entry per IOAPIC pin (01d8661) — the
|
||||
# pre-fix 16-entry table broke virtio-mmio probes on pin > 15.
|
||||
# - cmdline overflow warning (faf8a58) — surfaces the otherwise
|
||||
# silent x86 COMMAND_LINE_SIZE=2048 truncation in early boot.
|
||||
# All sibling msb_krun_* crates are patched from the same git source
|
||||
# because they share types across crate boundaries (e.g. msb_krun_vmm
|
||||
# holds `dyn Subscriber` impls of msb_krun_devices' Console); mixing
|
||||
# registry + git crates from the same cohort gives crate-identity
|
||||
# mismatches even when the version numbers agree.
|
||||
# Pinned to a specific commit so this Cargo.lock is reproducible;
|
||||
# bump when filing the upstream PR is the right time to revisit.
|
||||
[patch.crates-io]
|
||||
msb_krun = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_arch = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_arch_gen = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_cpuid = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_devices = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_hvf = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_kernel = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_polly = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_smbios = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_utils = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
msb_krun_vmm = { git = "https://github.com/wirenboard/libkrun", rev = "38e46f2" }
|
||||
|
||||
[profile.release]
|
||||
codegen-units = 16
|
||||
lto = "thin"
|
||||
|
|
|
|||
27
PLAN.md
27
PLAN.md
|
|
@ -454,9 +454,12 @@ Each is independent and lands as its own PR per the working agreement.
|
|||
**`--mount HOST:GUEST`** (repeatable). Pass extra host directories
|
||||
through to the guest as bind mounts. The launcher already mounts the
|
||||
project at its host path + the per-project state dir at
|
||||
`/agent-vm-state`; add user-supplied extras. **Mind the libkrun
|
||||
virtio-IRQ cap** (~6 devices; see Discovered Upstream Issue #3) — print
|
||||
a clear error if the user crosses it, *don't* silently fail at boot.
|
||||
`/agent-vm-state`; add user-supplied extras. Originally we worried
|
||||
about libkrun's tight virtio-IRQ cap (~11 IRQs with the in-kernel
|
||||
IOAPIC) and capped/warned on extras; that ceiling went away when we
|
||||
flipped on `msb_krun`'s userspace split irqchip in the runtime
|
||||
(`vendor/microsandbox/crates/runtime/lib/vm.rs`), raising the cap to
|
||||
~219 — see Discovered Upstream Issue #3 for the history.
|
||||
|
||||
**Clipboard bridge.** Original `clipboard-pty.py` does a live PTY
|
||||
bridge; that's more than we need. v1 design: a per-project
|
||||
|
|
@ -629,9 +632,21 @@ or fixed in `wirenboard/microsandbox`:
|
|||
(we never see them with localhost). Only `LayerDownloadComplete` fires.
|
||||
Not exactly a bug, but undocumented and bit us when we tried to drive a
|
||||
download-bytes bar.
|
||||
3. **libkrun virtio IRQ cap is low** (~6 devices). We're constrained to
|
||||
2 bind mounts on top of the OCI overlay's 2-device cost. Bigger fan-out
|
||||
needs upstream tuning of the libkrun build.
|
||||
3. **libkrun virtio IRQ cap is low** with the default in-kernel IOAPIC
|
||||
(~11 IRQs handed to virtio-mmio total on x86_64), so a config with
|
||||
the OCI overlay's 2-device cost + virtio-net + vsock + console + a
|
||||
couple of bind mounts saturates it and any extra `--mount` trips
|
||||
`RegisterNetDevice(IrqsExhausted)` at boot. *Resolved* by enabling
|
||||
`msb_krun`'s userspace split irqchip via `MachineBuilder::split_irqchip(true)`
|
||||
in `vendor/microsandbox/crates/runtime/lib/vm.rs` — that swaps in a
|
||||
userspace IOAPIC with ~219 usable IRQs at the cost of one extra
|
||||
msb_krun worker thread. **Also required bumping `msb_krun` 0.1.12 →
|
||||
0.1.13**: 0.1.12's userspace IOAPIC used a `u32` IRR (pins ≥32
|
||||
silently dropped) and had an integer-underflow bug in the
|
||||
redirection-table register-index reads/writes that crashed the VMM
|
||||
mid-boot once the guest started programming RTEs. Both are fixed in
|
||||
0.1.13. Verified e2e: 8 user --mount entries → guest brings up 19
|
||||
virtio devices on IO-APIC pins 5..23. No effect on aarch64 / riscv64.
|
||||
4. **Manifest media-type assumptions.** Microsandbox stores the
|
||||
per-platform manifest digest; a registry HEAD on a tag returns the
|
||||
multi-arch index digest by default. Either would be fine to use, but
|
||||
|
|
|
|||
10
README.md
10
README.md
|
|
@ -95,7 +95,7 @@ Each launcher accepts:
|
|||
| `--no-update-check` | skip the registry HEAD on launch |
|
||||
| `--no-git` | skip gh/git auth injection (still respects `--repo`) |
|
||||
| `--repo OWNER/NAME` | add to the GitHub allow-list (repeatable) |
|
||||
| `--mount HOST[:GUEST]` | extra bind mount (subject to libkrun IRQ cap) |
|
||||
| `--mount HOST[:GUEST]` | extra bind mount (one virtio-fs each, ~210 mount headroom) |
|
||||
|
||||
Trailing args go to the agent: `agent-vm claude -p "say hi"`,
|
||||
`agent-vm shell -- -c 'cargo test'`.
|
||||
|
|
@ -155,9 +155,11 @@ exit aborts the launch.
|
|||
|
||||
## Troubleshooting
|
||||
|
||||
- **`RegisterNetDevice(IrqsExhausted)` at boot** — libkrun's virtio
|
||||
IRQ pool is saturated by project + state + net + secrets. Drop a
|
||||
`--mount` or pass `--no-git`.
|
||||
- **`RegisterNetDevice(IrqsExhausted)` at boot** — the userspace split
|
||||
irqchip raises the cap to ~219 IRQs, so this should only happen with
|
||||
hundreds of `--mount`s or on a host whose KVM lacks
|
||||
`KVM_CAP_SPLIT_IRQCHIP` (pre-Linux 4.7 or some nested-virt /
|
||||
seccomp-restricted setups). Drop a `--mount` to recover.
|
||||
- **`handshake read id_offset: timed out`** — `free -h`; the VM needs
|
||||
more memory than is available. Try `--memory 1`.
|
||||
- **GitHub 403 from the proxy** — repo isn't in the allow-list.
|
||||
|
|
|
|||
|
|
@ -121,11 +121,14 @@ pub struct Args {
|
|||
/// `HOST[:GUEST]`. `GUEST` defaults to `HOST` (mirror at the
|
||||
/// same absolute path). Repeatable.
|
||||
///
|
||||
/// Each `--mount` consumes one virtio device — libkrun caps the
|
||||
/// IRQ pool around 6, so we only have room for a couple of
|
||||
/// extras on top of the project + state binds. The launcher will
|
||||
/// error clearly if you cross the cap rather than failing at
|
||||
/// boot.
|
||||
/// Each `--mount` consumes one virtio-fs device. The microsandbox
|
||||
/// runtime enables msb_krun's userspace split irqchip, which on
|
||||
/// x86_64 lifts the per-VM IRQ ceiling from 11 to ~219; aarch64 /
|
||||
/// riscv64 always had >200 GIC/AIA IRQs and are unchanged. Either
|
||||
/// way the practical cap on `--mount` is well into the hundreds
|
||||
/// (shared with rootfs, network, vsock, console, and any
|
||||
/// `--volume` disks — call it ~210 user mounts for the common
|
||||
/// config). You can stop worrying about it for typical workloads.
|
||||
#[arg(long = "mount")]
|
||||
mount: Vec<String>,
|
||||
|
||||
|
|
@ -183,12 +186,14 @@ pub async fn launch(agent: Agent, args: Args) -> Result<i32> {
|
|||
// boot — that wipes any mount point our `patch` builder baked into the
|
||||
// rootfs. Fall back to /workspace and tell the user once.
|
||||
//
|
||||
// microsandbox VMs also cap the number of virtio devices via libkrun's
|
||||
// IRQ pool; each bind mount is one device on top of the OCI rootfs's two
|
||||
// (EROFS lower + ext4 upper). We therefore bind a *single* host
|
||||
// directory for all per-agent state and either symlink the agent's
|
||||
// expected home (claude, opencode) or redirect via an env var (codex).
|
||||
// Codex needs the env-var path because its CLI binary lives under
|
||||
// Per-agent state lives behind a *single* bind mount, with the
|
||||
// agent's expected home wired up via symlink (claude, opencode) or
|
||||
// an env var (codex). The single-bind layout predates the
|
||||
// split-irqchip switch in the runtime — it kept IRQ pressure down
|
||||
// back when libkrun only handed out 11 virtio IRQs total. Today
|
||||
// it's still the better shape: one virtio-fs server, one rootfs
|
||||
// patch entry per agent, and a stable on-host layout. Codex needs
|
||||
// the env-var path because its CLI binary lives under
|
||||
// /root/.codex/packages, which a symlink would shadow.
|
||||
let host_path = session
|
||||
.project_dir
|
||||
|
|
@ -340,6 +345,20 @@ pub async fn launch(agent: Agent, args: Args) -> Result<i32> {
|
|||
)
|
||||
.context("writing guest gh/git config")?;
|
||||
|
||||
// Phase 7: parse `--mount HOST[:GUEST]` extras. The microsandbox
|
||||
// runtime now enables msb_krun's userspace split irqchip (requires
|
||||
// msb_krun >= 0.1.13 — earlier versions' userspace IOAPIC silently
|
||||
// dropped IRQs on pin ≥ 32 and underflowed on RTE register
|
||||
// accesses), raising the virtio-mmio IRQ cap from 11 to ~219, so
|
||||
// we no longer need to warn or pre-cap on extra mounts. See the
|
||||
// vendored vm.rs `build_vm` for details.
|
||||
let extra_mounts = parse_extra_mounts(&args.mount).context("parsing --mount")?;
|
||||
for em in &extra_mounts {
|
||||
if !em.host.exists() {
|
||||
anyhow::bail!("--mount host path {:?} does not exist", em.host);
|
||||
}
|
||||
}
|
||||
|
||||
let is_local_registry = crate::pull::is_plain_http_registry(&image);
|
||||
let mut builder = Sandbox::builder(&session.sandbox_name)
|
||||
.image(image.as_str())
|
||||
|
|
|
|||
|
|
@ -7,12 +7,14 @@ building the `.so` that ships in the agent-vm npm package.
|
|||
The CI `package` job in `.github/workflows/release-npm.yml`:
|
||||
|
||||
1. `git clone --branch v$LIBKRUNFW_VERSION --depth 1 https://github.com/containers/libkrunfw`
|
||||
2. `patch -p1 < libkrunfw-overrides/config-libkrunfw_$ARCH.patch`
|
||||
2. Apply each `*.patch` in this directory.
|
||||
3. `make`
|
||||
4. Copies the resulting `libkrunfw.so.$LIBKRUNFW_VERSION` into
|
||||
`npm-dist/agent-vm-$PLATFORM/lib/`.
|
||||
|
||||
## Why these patches
|
||||
## Patches
|
||||
|
||||
### `config-libkrunfw_x86_64.patch` — nested KVM
|
||||
|
||||
Stock libkrunfw ships only the paravirt-guest helpers
|
||||
(`CONFIG_KVM_GUEST=y`) — the guest kernel can talk to a host KVM but
|
||||
|
|
@ -33,6 +35,31 @@ Docker-in-Docker / nested KVM use cases, so we flip:
|
|||
`devtmpfs` and `DEVTMPFS_MOUNT` are already enabled upstream so
|
||||
`/dev/kvm` materializes at boot without further config changes.
|
||||
|
||||
## Kernel-source overrides
|
||||
|
||||
In addition to the kbuild `.config` seed above, libkrunfw-overrides/
|
||||
can ship arch-suffixed source patches (named `<topic>_<arch>.patch`)
|
||||
that the CI workflow drops into libkrunfw's `patches/0999-overrides-*`
|
||||
slot for the build to apply.
|
||||
|
||||
### `cmdline-size_x86_64.patch` — higher `--mount` cap
|
||||
|
||||
x86 Linux statically sizes `boot_command_line[COMMAND_LINE_SIZE]`
|
||||
(default 2048) and silently truncates anything past that at
|
||||
`setup_arch()` time. libkrun appends one
|
||||
`virtio_mmio.device=4K@0xd0XXX000:NN` entry (~36 bytes) per virtio
|
||||
device, so a baseline boot (rootfs/upper/runtime/network/vsock/
|
||||
console/etc.) plus ~10 `--mount` entries already pushes the assembled
|
||||
cmdline past 2048 bytes; the trailing virtio devices then fail to
|
||||
register and the guest hangs in early boot with `kernel.log` stuck at
|
||||
0 bytes (the lost devices include virtio-console, so even printk has
|
||||
nowhere to land).
|
||||
|
||||
Bumping `COMMAND_LINE_SIZE` to 16384 leaves room for ~100 user mounts
|
||||
before re-hitting the cap. libkrun also warns when the assembled
|
||||
cmdline crosses 2048 bytes, so configurations relying on this patch
|
||||
flag themselves if run against stock libkrunfw.
|
||||
|
||||
## Cost of each override
|
||||
|
||||
Measured by rebuilding the `.so` from a pristine v5.2.1 checkout with each
|
||||
|
|
|
|||
27
libkrunfw-overrides/cmdline-size_x86_64.patch
Normal file
27
libkrunfw-overrides/cmdline-size_x86_64.patch
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
Bump x86_64 COMMAND_LINE_SIZE so libkrun's per-virtio-mmio cmdline entries
|
||||
don't truncate at higher --mount counts.
|
||||
|
||||
x86 Linux statically allocates `boot_command_line[COMMAND_LINE_SIZE]` and
|
||||
silently truncates anything longer at setup_arch() time. libkrun appends
|
||||
one `virtio_mmio.device=4K@0xd0XXX000:NN` entry (~36 bytes) per virtio
|
||||
device, so the cmdline grows ~36 bytes per --mount on top of a ~1100-byte
|
||||
base (DEFAULT_KERNEL_CMDLINE + KRUN_ENV + MSB_BLOCK_ROOT / MSB_DIR_MOUNTS
|
||||
/ MSB_NET_* / PATH / ...). At ~26 virtio devices (N=11 user mounts) we
|
||||
hit 2048; one more mount pushes the trailing entries off the buffer,
|
||||
the kernel never registers those devices, and the guest hangs in early
|
||||
boot (kernel.log stays 0 bytes because virtio-console didn't probe).
|
||||
|
||||
Bumping to 16384 leaves headroom for ~400 user mounts before re-hitting
|
||||
the cap.
|
||||
|
||||
--- a/arch/x86/include/asm/setup.h
|
||||
+++ b/arch/x86/include/asm/setup.h
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
#include <uapi/asm/setup.h>
|
||||
|
||||
-#define COMMAND_LINE_SIZE 2048
|
||||
+#define COMMAND_LINE_SIZE 16384
|
||||
|
||||
#include <linux/linkage.h>
|
||||
#include <asm/page_types.h>
|
||||
2
vendor/microsandbox
vendored
2
vendor/microsandbox
vendored
|
|
@ -1 +1 @@
|
|||
Subproject commit 1e9864e0203fc03d2b91794990445649c69943d4
|
||||
Subproject commit e7590e28d53f8816a94ec38cdafd0ab5a4a4b543
|
||||
Loading…
Add table
Add a link
Reference in a new issue