mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-09 16:08:31 +00:00
feat(sandbox): add BoxLite micro-VM sandbox provider (scaffold) (#3940)
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run
* feat(sandbox): add BoxLite micro-VM sandbox provider (scaffold) Community SandboxProvider backed by BoxLite, a daemonless OCI-native micro-VM runtime. execute_command is wired end to end through a private asyncio loop that bridges BoxLite's async SDK to DeerFlow's sync Sandbox contract; the remaining Sandbox methods are stubbed pending approach review. Opt-in via sandbox.use (pip install boxlite); a packaged [boxlite] extra will follow. Existing providers are unchanged. Refs #3936, #3439, #3213 * feat(sandbox): implement full Sandbox contract for BoxLite backend Rename the community BoxLite integration to read as a compute backend rather than "a sandbox": module boxlite_sandbox -> boxlite, BoxliteSandboxProvider -> BoxliteProvider, BoxliteSandbox -> BoxliteBox. Implement the file surface DeerFlow's default tool path assumes -- read_file, write_file, update_file, download_file, list_dir, glob, grep -- as shell commands inside the box (cat/find/grep/chunked base64), reusing deerflow.sandbox.search and busybox-portable flags. Removes the reachable NotImplementedError regression once the provider is selected. download_file keeps the /mnt/user-data prefix + traversal guards; the provider materialises those virtual dirs on box start. Refs #3936, #3940 * fix(sandbox): resolve CI + review findings on BoxLite backend - provider: import DEFAULT_SKILLS_CONTAINER_PATH instead of the "/mnt/skills" literal (backend-unit-tests guard), and drop the redundant env-var re-resolution -- AppConfig.resolve_env_variables already resolves $VARS and raises on missing. - box: grep now passes the raw pattern to grep (-F/-E); it previously handed the re.escape'd pattern to grep -F, so a literal search of e.g. foo.bar looked for foo\.bar and never matched. - box: execute_command now calls _validate_extra_env(env), matching the Sandbox POSIX env-key contract that the local/e2b/aio sandboxes enforce. - add tests/test_boxlite_provider.py (CI-safe, no BoxLite): actionable ImportError on the lazy import, clean acquire-failure + shutdown, the traversal / download-prefix guards, and env-key rejection. Refs #3936, #3940
This commit is contained in:
parent
6a4e5a3bb2
commit
358bacad89
6 changed files with 672 additions and 1 deletions
|
|
@ -381,7 +381,7 @@ Scheduled-task runtime note:
|
|||
- `image_search/` - Image search via DuckDuckGo
|
||||
- `aio_sandbox/` - Docker-based isolation (`AioSandboxProvider`)
|
||||
|
||||
Additional providers also live here (`brave`, `browserless`, `crawl4ai`, `ddg_search`, `e2b_sandbox`, `exa`, `fastcrw`, `groundroute`, `infoquest`, `searxng`, `serper`); see each subpackage for specifics.
|
||||
Additional providers also live here (`boxlite`, `brave`, `browserless`, `crawl4ai`, `ddg_search`, `e2b_sandbox`, `exa`, `fastcrw`, `groundroute`, `infoquest`, `searxng`, `serper`); see each subpackage for specifics.
|
||||
|
||||
**ACP agent tools**:
|
||||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||||
|
|
|
|||
|
|
@ -0,0 +1,68 @@
|
|||
# BoxLite backend
|
||||
|
||||
Runs each DeerFlow sandbox as a [BoxLite](https://github.com/boxlite-ai/boxlite)
|
||||
micro-VM — a daemonless, OCI-native VM with its own kernel (libkrun/KVM on Linux,
|
||||
Hypervisor.framework on macOS). Motivated by the resource/cold-start pain with
|
||||
the default AIO Docker sandbox in
|
||||
[#3439](https://github.com/bytedance/deer-flow/issues/3439) and
|
||||
[#3213](https://github.com/bytedance/deer-flow/issues/3213); discussion in
|
||||
[#3936](https://github.com/bytedance/deer-flow/issues/3936).
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
sandbox:
|
||||
use: deerflow.community.boxlite:BoxliteProvider
|
||||
image: python:3.12-slim # any OCI image, run unchanged (default: python:3.12-slim)
|
||||
memory_mib: 1024 # per-box memory cap (optional)
|
||||
cpus: 2 # per-box vCPUs (optional)
|
||||
environment: # injected into every command
|
||||
PYTHONUNBUFFERED: "1"
|
||||
```
|
||||
|
||||
```bash
|
||||
pip install boxlite # an optional `[boxlite]` extra + uv.lock update will follow once the approach lands
|
||||
```
|
||||
|
||||
**Host requirement:** BoxLite boots micro-VMs, so a Linux host needs KVM — i.e.
|
||||
nested virtualization when DeerFlow runs inside a cloud VM. macOS uses
|
||||
Hypervisor.framework. This is the main deployment constraint to weigh vs. the
|
||||
container-based providers.
|
||||
|
||||
## Design
|
||||
|
||||
DeerFlow's `Sandbox` contract is synchronous; BoxLite's SDK is async-native and
|
||||
its box handles are event-loop-affine. The provider owns **one** private asyncio
|
||||
loop on a daemon thread and marshals every coroutine onto it via
|
||||
`run_coroutine_threadsafe`. This keeps all operations on the loop the box was
|
||||
started on and is safe under DeerFlow's `asyncio.to_thread` worker pool — without
|
||||
using BoxLite's greenlet sync facade, which refuses to run inside an async
|
||||
context and is thread-affine.
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `provider.py` | `SandboxProvider` lifecycle + the private-loop bridge |
|
||||
| `box.py` | `Sandbox` adapter; `execute_command` + file ops |
|
||||
|
||||
## Contract coverage
|
||||
|
||||
The full `Sandbox` surface is implemented. File operations run as shell commands
|
||||
inside the box and reuse `deerflow.sandbox.search`, mirroring `e2b_sandbox`:
|
||||
|
||||
- `execute_command` — `sh -lc`, with per-call env and timeout.
|
||||
- `read_file` / `write_file` / `update_file` — `cat` and chunked `base64` (binary-safe, no arg-size limit).
|
||||
- `download_file` — 100 MB cap, restricted to the `/mnt/user-data` prefix.
|
||||
- `list_dir` / `glob` / `grep` — `find` / `grep` with busybox-portable flags; results filtered/capped in Python.
|
||||
|
||||
The provider creates `/mnt/user-data/{workspace,uploads,outputs}` and
|
||||
`/mnt/skills` on box start so those virtual paths resolve natively.
|
||||
|
||||
**Out of scope for this pass** (follow-ups): warm pooling, idle reaping, mount
|
||||
syncing, and remote/provisioner modes.
|
||||
|
||||
## Status
|
||||
|
||||
Verified end-to-end against a live box (provider resolution → `execute_command`
|
||||
→ file ops) on macOS/HVF. Linux/KVM validation and benchmarks vs. the AIO
|
||||
sandbox are tracked in
|
||||
[#3936](https://github.com/bytedance/deer-flow/issues/3936).
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
"""BoxLite micro-VM backend for DeerFlow sandboxes.
|
||||
|
||||
Integrates `BoxLite <https://github.com/boxlite-ai/boxlite>`_ — a daemonless,
|
||||
OCI-native micro-VM runtime (libkrun/KVM on Linux, Hypervisor.framework on
|
||||
macOS) — behind DeerFlow's :class:`Sandbox` / :class:`SandboxProvider` contract.
|
||||
Each sandbox is a hardware-isolated VM with its own kernel that runs any OCI
|
||||
image unchanged. See https://github.com/bytedance/deer-flow/issues/3936.
|
||||
|
||||
The full contract is implemented: ``execute_command`` plus ``read_file`` /
|
||||
``write_file`` / ``update_file`` / ``download_file`` / ``list_dir`` / ``glob`` /
|
||||
``grep`` (file ops run as shell commands inside the box).
|
||||
|
||||
Configuration example (``config.yaml``)::
|
||||
|
||||
sandbox:
|
||||
use: deerflow.community.boxlite:BoxliteProvider
|
||||
image: python:3.12-slim # any OCI image; runs unchanged
|
||||
memory_mib: 1024 # per-box memory cap (optional)
|
||||
cpus: 2 # per-box vCPUs (optional)
|
||||
environment: # injected into every command
|
||||
PYTHONUNBUFFERED: "1"
|
||||
|
||||
Install the runtime (an optional ``[boxlite]`` extra + lockfile update will
|
||||
follow once the approach lands)::
|
||||
|
||||
pip install boxlite
|
||||
|
||||
Host requirement: BoxLite boots micro-VMs, so a Linux host needs KVM (nested
|
||||
virtualization when DeerFlow itself runs inside a cloud VM); macOS uses
|
||||
Hypervisor.framework.
|
||||
"""
|
||||
|
||||
from .box import BoxliteBox
|
||||
from .provider import BoxliteProvider
|
||||
|
||||
__all__ = [
|
||||
"BoxliteBox",
|
||||
"BoxliteProvider",
|
||||
]
|
||||
304
backend/packages/harness/deerflow/community/boxlite/box.py
Normal file
304
backend/packages/harness/deerflow/community/boxlite/box.py
Normal file
|
|
@ -0,0 +1,304 @@
|
|||
"""``BoxliteBox`` — DeerFlow :class:`Sandbox` backed by a BoxLite micro-VM.
|
||||
|
||||
DeerFlow's ``Sandbox`` contract is synchronous; BoxLite's SDK is async-native and
|
||||
its box handles are event-loop-affine. The provider (:mod:`.provider`) owns one
|
||||
private asyncio loop on a daemon thread and injects a ``run`` callable that
|
||||
marshals each coroutine onto it via ``run_coroutine_threadsafe`` — so every op
|
||||
runs on the loop the box was started on, and stays safe no matter which
|
||||
``asyncio.to_thread`` worker DeerFlow invokes us from.
|
||||
|
||||
Every operation is a shell command run inside the box (``cat`` / ``find`` /
|
||||
``grep`` / chunked ``base64``), parsed with the shared ``deerflow.sandbox.search``
|
||||
helpers — the same exec-driven approach as ``community/e2b_sandbox``. Commands
|
||||
use only busybox-portable flags so any OCI image works.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import errno
|
||||
import logging
|
||||
import posixpath
|
||||
import re
|
||||
import shlex
|
||||
import threading
|
||||
from typing import TYPE_CHECKING, TypeVar
|
||||
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX
|
||||
from deerflow.sandbox.sandbox import Sandbox, _validate_extra_env
|
||||
from deerflow.sandbox.search import GrepMatch, path_matches, should_ignore_path, truncate_line
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from boxlite import SimpleBox
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
_MAX_DOWNLOAD_SIZE = 100 * 1024 * 1024 # 100 MB
|
||||
# One base64 chunk stays well under Linux MAX_ARG_STRLEN (128 KiB per argv entry),
|
||||
# and 60000 is a multiple of 4 so each chunk is a self-contained base64 unit whose
|
||||
# decoded bytes concatenate losslessly.
|
||||
_B64_CHUNK = 60000
|
||||
|
||||
|
||||
class BoxliteBox(Sandbox):
|
||||
"""Adapter that delegates to a running BoxLite ``SimpleBox``.
|
||||
|
||||
Args:
|
||||
id: DeerFlow-side sandbox id (the BoxLite box id).
|
||||
box: A started async ``SimpleBox``. The provider owns its lifecycle; this
|
||||
adapter stops it on :meth:`close`.
|
||||
run: Runs a coroutine on the provider's private loop, returning its result
|
||||
(blocking the caller thread).
|
||||
default_env: Static environment merged into every command, overridden by
|
||||
per-call ``env`` (request-scoped secrets).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
id: str,
|
||||
box: SimpleBox,
|
||||
run: Callable[[Awaitable[T]], T],
|
||||
*,
|
||||
default_env: dict[str, str] | None = None,
|
||||
) -> None:
|
||||
super().__init__(id)
|
||||
self._box = box
|
||||
self._run = run
|
||||
self._default_env = dict(default_env or {})
|
||||
self._lock = threading.Lock()
|
||||
self._closed = False
|
||||
|
||||
# ── bridge helpers ──────────────────────────────────────────────────
|
||||
|
||||
def _exec(self, *argv: str, env: dict[str, str] | None = None):
|
||||
with self._lock:
|
||||
if self._closed:
|
||||
raise RuntimeError("sandbox has been closed")
|
||||
box = self._box
|
||||
return self._run(box.exec(*argv, env=env))
|
||||
|
||||
def _sh(self, script: str, env: dict[str, str] | None = None):
|
||||
return self._exec("sh", "-lc", script, env=env)
|
||||
|
||||
def close(self) -> None:
|
||||
with self._lock:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
try:
|
||||
self._run(self._box.stop())
|
||||
except Exception as e:
|
||||
logger.warning("Error stopping BoxLite box %s: %s", self.id, e)
|
||||
|
||||
# ── path safety (mirrors community/e2b_sandbox) ─────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _guard_traversal(path: str) -> str:
|
||||
if not path:
|
||||
raise ValueError("path must be a non-empty string")
|
||||
normalized = path.replace("\\", "/")
|
||||
for segment in normalized.split("/"):
|
||||
if segment == "..":
|
||||
raise PermissionError(f"Access denied: path traversal detected in '{path}'")
|
||||
return normalized
|
||||
|
||||
def _resolve_path(self, path: str) -> str:
|
||||
# The provider materialises the /mnt/user-data prefix on the box rootfs,
|
||||
# so DeerFlow's virtual paths are used as-is; we only reject traversal.
|
||||
return self._guard_traversal(path)
|
||||
|
||||
# ── command execution ───────────────────────────────────────────────
|
||||
|
||||
def execute_command(
|
||||
self,
|
||||
command: str,
|
||||
env: dict[str, str] | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> str:
|
||||
"""Run ``command`` through a shell in the box and return its output.
|
||||
|
||||
DeerFlow passes a bash command *string*; BoxLite's ``exec`` takes argv, so
|
||||
it runs through ``sh -lc``. Per-call ``env`` is layered over the static
|
||||
config environment and scoped to this command only.
|
||||
"""
|
||||
_validate_extra_env(env) # POSIX env-var key rule; raises ValueError on a bad key
|
||||
merged_env = {**self._default_env, **(env or {})} or None
|
||||
with self._lock:
|
||||
if self._closed:
|
||||
return "Error: sandbox has been closed"
|
||||
box = self._box
|
||||
try:
|
||||
result = self._run(box.exec("sh", "-lc", command, env=merged_env, timeout=timeout))
|
||||
except Exception as e:
|
||||
logger.error("Failed to execute command in BoxLite box %s: %s", self.id, e)
|
||||
return f"Error: {e}"
|
||||
|
||||
stdout = result.stdout or ""
|
||||
stderr = result.stderr or ""
|
||||
if stdout and stderr:
|
||||
output = f"{stdout}\n{stderr}"
|
||||
else:
|
||||
output = stdout or stderr
|
||||
if result.exit_code not in (0, None) and not output:
|
||||
output = f"Command exited with code {result.exit_code}"
|
||||
return output if output else "(no output)"
|
||||
|
||||
# ── file operations ─────────────────────────────────────────────────
|
||||
|
||||
def read_file(self, path: str) -> str:
|
||||
resolved = self._resolve_path(path)
|
||||
try:
|
||||
r = self._exec("cat", "--", resolved)
|
||||
except Exception as e:
|
||||
logger.error("read_file %s failed: %s", resolved, e)
|
||||
return f"Error: {e}"
|
||||
if r.exit_code not in (0, None):
|
||||
return f"Error: {(r.stderr or '').strip() or 'cannot read file'}"
|
||||
return r.stdout or ""
|
||||
|
||||
def write_file(self, path: str, content: str, append: bool = False) -> None:
|
||||
self._write_bytes(self._resolve_path(path), content.encode("utf-8"), append=append)
|
||||
|
||||
def update_file(self, path: str, content: bytes) -> None:
|
||||
self._write_bytes(self._resolve_path(path), content, append=False)
|
||||
|
||||
def _write_bytes(self, resolved: str, data: bytes, *, append: bool) -> None:
|
||||
parent = posixpath.dirname(resolved)
|
||||
if parent:
|
||||
mk = self._sh(f"mkdir -p {shlex.quote(parent)}")
|
||||
if mk.exit_code not in (0, None):
|
||||
raise OSError(f"cannot create parent of '{resolved}': {(mk.stderr or '').strip()}")
|
||||
|
||||
b64 = base64.b64encode(data).decode("ascii")
|
||||
if not b64: # empty file — create/truncate without piping
|
||||
r = self._sh(f": {'>>' if append else '>'} {shlex.quote(resolved)}")
|
||||
if r.exit_code not in (0, None):
|
||||
raise OSError(f"write '{resolved}' failed: {(r.stderr or '').strip()}")
|
||||
return
|
||||
|
||||
first = True
|
||||
for i in range(0, len(b64), _B64_CHUNK):
|
||||
chunk = b64[i : i + _B64_CHUNK]
|
||||
redir = ">>" if (append or not first) else ">"
|
||||
r = self._sh(f"printf %s {shlex.quote(chunk)} | base64 -d {redir} {shlex.quote(resolved)}")
|
||||
if r.exit_code not in (0, None):
|
||||
raise OSError(f"write '{resolved}' failed: {(r.stderr or '').strip()}")
|
||||
first = False
|
||||
|
||||
def download_file(self, path: str) -> bytes:
|
||||
normalized = self._guard_traversal(path)
|
||||
stripped = normalized.lstrip("/")
|
||||
allowed = VIRTUAL_PATH_PREFIX.lstrip("/")
|
||||
if stripped != allowed and not stripped.startswith(f"{allowed}/"):
|
||||
raise PermissionError(f"Access denied: path must be under '{VIRTUAL_PATH_PREFIX}': '{path}'")
|
||||
|
||||
# Enforce the size cap before buffering the whole payload.
|
||||
size_r = self._sh(f"wc -c < {shlex.quote(normalized)}")
|
||||
if size_r.exit_code not in (0, None):
|
||||
raise OSError(f"cannot read '{path}' from box: {(size_r.stderr or '').strip() or 'not found'}")
|
||||
try:
|
||||
size = int((size_r.stdout or "0").strip() or "0")
|
||||
except ValueError:
|
||||
size = 0
|
||||
if size > _MAX_DOWNLOAD_SIZE:
|
||||
raise OSError(errno.EFBIG, f"File exceeds maximum download size of {_MAX_DOWNLOAD_SIZE} bytes", path)
|
||||
|
||||
r = self._sh(f"base64 {shlex.quote(normalized)}")
|
||||
if r.exit_code not in (0, None):
|
||||
raise OSError(f"cannot read '{path}' from box: {(r.stderr or '').strip()}")
|
||||
try:
|
||||
return base64.b64decode("".join((r.stdout or "").split()))
|
||||
except Exception as e:
|
||||
raise OSError(f"failed to decode '{path}' from box: {e}") from e
|
||||
|
||||
def list_dir(self, path: str, max_depth: int = 2) -> list[str]:
|
||||
resolved = self._resolve_path(path)
|
||||
r = self._sh(f"find {shlex.quote(resolved)} -maxdepth {int(max_depth)} \\( -type f -o -type d \\) 2>/dev/null | head -500")
|
||||
return [line.strip() for line in (r.stdout or "").splitlines() if line.strip()]
|
||||
|
||||
def glob(
|
||||
self,
|
||||
path: str,
|
||||
pattern: str,
|
||||
*,
|
||||
include_dirs: bool = False,
|
||||
max_results: int = 200,
|
||||
) -> tuple[list[str], bool]:
|
||||
resolved = self._resolve_path(path)
|
||||
types = ("f", "d") if include_dirs else ("f",)
|
||||
type_expr = " -o ".join(f"-type {t}" for t in types)
|
||||
hard_limit = max(max_results * 4, max_results + 50)
|
||||
r = self._sh(f"find {shlex.quote(resolved)} \\( {type_expr} \\) -print 2>/dev/null | head -{hard_limit}")
|
||||
|
||||
matches: list[str] = []
|
||||
root = resolved.rstrip("/") or "/"
|
||||
root_prefix = root if root == "/" else f"{root}/"
|
||||
for entry in (r.stdout or "").splitlines():
|
||||
entry = entry.strip()
|
||||
if not entry or (entry != root and not entry.startswith(root_prefix)):
|
||||
continue
|
||||
if should_ignore_path(entry):
|
||||
continue
|
||||
rel_path = entry[len(root) :].lstrip("/")
|
||||
if not rel_path:
|
||||
continue
|
||||
if path_matches(pattern, rel_path):
|
||||
matches.append(entry)
|
||||
if len(matches) >= max_results:
|
||||
return matches, True
|
||||
return matches, False
|
||||
|
||||
def grep(
|
||||
self,
|
||||
path: str,
|
||||
pattern: str,
|
||||
*,
|
||||
glob: str | None = None,
|
||||
literal: bool = False,
|
||||
case_sensitive: bool = False,
|
||||
max_results: int = 100,
|
||||
) -> tuple[list[GrepMatch], bool]:
|
||||
# Sanity-check a regex pattern as a Python regex at the boundary (grep uses
|
||||
# POSIX ERE, but this catches gross errors); a literal needs no validation.
|
||||
# grep receives the RAW pattern: -F matches it literally, -E as a regex.
|
||||
if not literal:
|
||||
re.compile(pattern, 0 if case_sensitive else re.IGNORECASE)
|
||||
|
||||
resolved = self._resolve_path(path)
|
||||
# busybox+GNU-portable flags: -r recursive (also prints the filename),
|
||||
# -n line numbers, -I skip binary, -E/-F regex vs fixed. --include and -m
|
||||
# are omitted for busybox portability; glob-scoping and the result cap are
|
||||
# applied in Python below.
|
||||
flags = ["-r", "-n", "-I"]
|
||||
if not case_sensitive:
|
||||
flags.append("-i")
|
||||
flags.append("-F" if literal else "-E")
|
||||
total_cap = max(max_results * 4, max_results + 50)
|
||||
cmd = "grep " + " ".join(flags) + f" -e {shlex.quote(pattern)} {shlex.quote(resolved)} 2>/dev/null | head -{total_cap}"
|
||||
r = self._sh(cmd)
|
||||
|
||||
include = glob.split("/")[-1] if glob else None
|
||||
matches: list[GrepMatch] = []
|
||||
truncated = False
|
||||
for raw in (r.stdout or "").splitlines():
|
||||
try:
|
||||
file_path, line_no_str, line_text = raw.split(":", 2)
|
||||
except ValueError:
|
||||
continue
|
||||
try:
|
||||
line_number = int(line_no_str)
|
||||
except ValueError:
|
||||
continue
|
||||
if should_ignore_path(file_path):
|
||||
continue
|
||||
if include and not path_matches(include, posixpath.basename(file_path)):
|
||||
continue
|
||||
matches.append(GrepMatch(path=file_path, line_number=line_number, line=truncate_line(line_text)))
|
||||
if len(matches) >= max_results:
|
||||
truncated = True
|
||||
break
|
||||
return matches, truncated
|
||||
188
backend/packages/harness/deerflow/community/boxlite/provider.py
Normal file
188
backend/packages/harness/deerflow/community/boxlite/provider.py
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
"""``BoxliteProvider`` — DeerFlow :class:`SandboxProvider` backed by BoxLite.
|
||||
|
||||
Integrates `BoxLite <https://github.com/boxlite-ai/boxlite>`_ — a daemonless,
|
||||
OCI-native micro-VM runtime — as a DeerFlow sandbox backend. See
|
||||
https://github.com/bytedance/deer-flow/issues/3936.
|
||||
|
||||
Config is read off :class:`SandboxConfig` (``extra="allow"``), so BoxLite keys
|
||||
may appear under ``sandbox:`` in ``config.yaml`` even though they are not declared
|
||||
on the model — see this package's ``__init__`` docstring for the full set. The
|
||||
provider creates one micro-VM per ``(user, thread)`` and reuses it within the
|
||||
process; warm pooling, idle reaping and remote modes are out of scope for now.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import atexit
|
||||
import logging
|
||||
import threading
|
||||
from collections.abc import Awaitable
|
||||
from typing import TYPE_CHECKING, Any, TypeVar
|
||||
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX
|
||||
from deerflow.constants import DEFAULT_SKILLS_CONTAINER_PATH
|
||||
from deerflow.sandbox.sandbox import Sandbox
|
||||
from deerflow.sandbox.sandbox_provider import SandboxProvider
|
||||
|
||||
from .box import BoxliteBox
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from boxlite import SimpleBox
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
DEFAULT_IMAGE = "python:3.12-slim"
|
||||
# DeerFlow's virtual prefixes, materialised on the box rootfs at start so the
|
||||
# Sandbox file APIs (which address /mnt/user-data/...) resolve natively.
|
||||
_VIRTUAL_DIRS = (
|
||||
f"{VIRTUAL_PATH_PREFIX}/workspace",
|
||||
f"{VIRTUAL_PATH_PREFIX}/uploads",
|
||||
f"{VIRTUAL_PATH_PREFIX}/outputs",
|
||||
DEFAULT_SKILLS_CONTAINER_PATH,
|
||||
)
|
||||
|
||||
|
||||
def _import_simplebox() -> type[SimpleBox]:
|
||||
"""Import BoxLite's async ``SimpleBox`` lazily.
|
||||
|
||||
Kept out of module import so the harness (and every other provider) installs
|
||||
without BoxLite; the dependency is only needed once this provider is selected.
|
||||
"""
|
||||
try:
|
||||
from boxlite import SimpleBox
|
||||
except ImportError as e: # pragma: no cover - depends on the optional dependency
|
||||
raise ImportError("BoxliteProvider requires the 'boxlite' package. Install it with: pip install boxlite.") from e
|
||||
return SimpleBox
|
||||
|
||||
|
||||
class _EventLoopThread:
|
||||
"""A private asyncio event loop running on a dedicated daemon thread.
|
||||
|
||||
BoxLite is async-native and its box handles are loop-affine, while DeerFlow's
|
||||
``Sandbox`` contract is synchronous and may be invoked from arbitrary
|
||||
``asyncio.to_thread`` workers. Owning one loop here and marshalling every
|
||||
coroutine onto it via ``run_coroutine_threadsafe`` gives a stable, thread-safe
|
||||
bridge without BoxLite's greenlet sync facade (which refuses to run inside an
|
||||
async context and is thread-affine).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._loop = asyncio.new_event_loop()
|
||||
self._thread = threading.Thread(target=self._loop.run_forever, name="boxlite-loop", daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def run(self, coro: Awaitable[T], *, timeout: float | None = None) -> T:
|
||||
return asyncio.run_coroutine_threadsafe(coro, self._loop).result(timeout)
|
||||
|
||||
def close(self) -> None:
|
||||
self._loop.call_soon_threadsafe(self._loop.stop)
|
||||
self._thread.join(timeout=5)
|
||||
if not self._loop.is_running():
|
||||
self._loop.close()
|
||||
|
||||
|
||||
class BoxliteProvider(SandboxProvider):
|
||||
"""Run each DeerFlow sandbox as a BoxLite micro-VM."""
|
||||
|
||||
uses_thread_data_mounts = False
|
||||
needs_upload_permission_adjustment = True
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.Lock()
|
||||
self._boxes: dict[str, BoxliteBox] = {}
|
||||
self._thread_boxes: dict[tuple[str, str], str] = {}
|
||||
self._shutdown_called = False
|
||||
self._config = self._load_config()
|
||||
self._loop = _EventLoopThread()
|
||||
atexit.register(self.shutdown)
|
||||
|
||||
def _load_config(self) -> dict[str, Any]:
|
||||
sandbox_config = get_app_config().sandbox
|
||||
|
||||
def _opt(name: str, default: Any = None) -> Any:
|
||||
return getattr(sandbox_config, name, default)
|
||||
|
||||
# $VARS in config.yaml are already resolved by AppConfig.resolve_env_variables
|
||||
# (which raises on a missing var), so the environment dict is used as-is.
|
||||
return {
|
||||
"image": _opt("image") or DEFAULT_IMAGE,
|
||||
"memory_mib": _opt("memory_mib"),
|
||||
"cpus": _opt("cpus"),
|
||||
"environment": dict(_opt("environment") or {}),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _thread_key(thread_id: str, user_id: str | None) -> tuple[str, str]:
|
||||
return (user_id or "", thread_id)
|
||||
|
||||
def acquire(self, thread_id: str | None = None, *, user_id: str | None = None) -> str:
|
||||
if thread_id is not None:
|
||||
key = self._thread_key(thread_id, user_id)
|
||||
with self._lock:
|
||||
existing = self._thread_boxes.get(key)
|
||||
if existing is not None and existing in self._boxes:
|
||||
return existing
|
||||
|
||||
box = self._create_box()
|
||||
|
||||
with self._lock:
|
||||
self._boxes[box.id] = box
|
||||
if thread_id is not None:
|
||||
self._thread_boxes[self._thread_key(thread_id, user_id)] = box.id
|
||||
return box.id
|
||||
|
||||
def _create_box(self) -> BoxliteBox:
|
||||
simplebox_cls = _import_simplebox()
|
||||
mkdir_cmd = "mkdir -p " + " ".join(_VIRTUAL_DIRS)
|
||||
|
||||
async def _make() -> SimpleBox:
|
||||
box = simplebox_cls(
|
||||
image=self._config["image"],
|
||||
memory_mib=self._config["memory_mib"],
|
||||
cpus=self._config["cpus"],
|
||||
)
|
||||
await box.start()
|
||||
# Materialise DeerFlow's virtual prefixes so file ops resolve natively.
|
||||
await box.exec("sh", "-lc", mkdir_cmd)
|
||||
return box
|
||||
|
||||
box = self._loop.run(_make())
|
||||
logger.info("Created BoxLite box %s (image=%s)", box.id, self._config["image"])
|
||||
return BoxliteBox(box.id, box, self._loop.run, default_env=self._config["environment"])
|
||||
|
||||
def get(self, sandbox_id: str) -> Sandbox | None:
|
||||
with self._lock:
|
||||
return self._boxes.get(sandbox_id)
|
||||
|
||||
def release(self, sandbox_id: str) -> None:
|
||||
with self._lock:
|
||||
box = self._boxes.pop(sandbox_id, None)
|
||||
for key in [k for k, sid in self._thread_boxes.items() if sid == sandbox_id]:
|
||||
self._thread_boxes.pop(key, None)
|
||||
if box is not None:
|
||||
box.close()
|
||||
|
||||
def reset(self) -> None:
|
||||
with self._lock:
|
||||
self._boxes.clear()
|
||||
self._thread_boxes.clear()
|
||||
|
||||
def shutdown(self) -> None:
|
||||
with self._lock:
|
||||
if self._shutdown_called:
|
||||
return
|
||||
self._shutdown_called = True
|
||||
active = list(self._boxes.values())
|
||||
self._boxes.clear()
|
||||
self._thread_boxes.clear()
|
||||
|
||||
for box in active:
|
||||
try:
|
||||
box.close()
|
||||
except Exception as e: # pragma: no cover - defensive
|
||||
logger.warning("Error closing BoxLite box %s during shutdown: %s", box.id, e)
|
||||
self._loop.close()
|
||||
72
backend/tests/test_boxlite_provider.py
Normal file
72
backend/tests/test_boxlite_provider.py
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
"""Unit tests for the BoxLite community provider.
|
||||
|
||||
These run in CI without BoxLite installed: they cover the lazy-import error path,
|
||||
provider lifecycle, and the path-safety guards — none of which need a live box.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import types
|
||||
|
||||
import pytest
|
||||
|
||||
from deerflow.community.boxlite.box import BoxliteBox
|
||||
from deerflow.community.boxlite.provider import BoxliteProvider, _import_simplebox
|
||||
|
||||
|
||||
def _no_boxlite(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Make ``import boxlite`` raise, regardless of whether it is installed."""
|
||||
monkeypatch.setitem(sys.modules, "boxlite", None)
|
||||
|
||||
|
||||
def test_import_simplebox_missing_raises_actionable(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_no_boxlite(monkeypatch)
|
||||
with pytest.raises(ImportError, match=r"pip install boxlite"):
|
||||
_import_simplebox()
|
||||
|
||||
|
||||
def test_acquire_without_boxlite_raises_and_shuts_down_cleanly(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
# Stub config so the provider constructs without a config.yaml on disk.
|
||||
stub = types.SimpleNamespace(sandbox=types.SimpleNamespace())
|
||||
monkeypatch.setattr("deerflow.community.boxlite.provider.get_app_config", lambda: stub)
|
||||
_no_boxlite(monkeypatch)
|
||||
|
||||
provider = BoxliteProvider()
|
||||
try:
|
||||
with pytest.raises(ImportError, match=r"pip install boxlite"):
|
||||
provider.acquire("thread-1", user_id="u")
|
||||
finally:
|
||||
provider.shutdown() # must not raise even though no box was ever created
|
||||
# Idempotent shutdown.
|
||||
provider.shutdown()
|
||||
|
||||
|
||||
def test_guard_traversal() -> None:
|
||||
assert BoxliteBox._guard_traversal("/mnt/user-data/workspace/a.txt") == "/mnt/user-data/workspace/a.txt"
|
||||
assert BoxliteBox._guard_traversal("relative/ok.txt") == "relative/ok.txt"
|
||||
with pytest.raises(PermissionError):
|
||||
BoxliteBox._guard_traversal("/mnt/user-data/../etc/passwd")
|
||||
with pytest.raises(ValueError):
|
||||
BoxliteBox._guard_traversal("")
|
||||
|
||||
|
||||
def test_download_file_guards_reject_before_touching_box() -> None:
|
||||
# ``run`` must never be called: both guards raise before any exec.
|
||||
def _fail_run(_coro: object) -> None:
|
||||
raise AssertionError("download_file must reject the path before running a command")
|
||||
|
||||
box = BoxliteBox("box-id", box=object(), run=_fail_run)
|
||||
with pytest.raises(PermissionError):
|
||||
box.download_file("/etc/passwd") # outside the /mnt/user-data prefix
|
||||
with pytest.raises(PermissionError):
|
||||
box.download_file("/mnt/user-data/../etc/passwd") # traversal
|
||||
|
||||
|
||||
def test_execute_command_rejects_invalid_env_key() -> None:
|
||||
def _fail_run(_coro: object) -> None:
|
||||
raise AssertionError("execute_command must reject a bad env key before running")
|
||||
|
||||
box = BoxliteBox("box-id", box=object(), run=_fail_run)
|
||||
with pytest.raises(ValueError, match=r"POSIX"):
|
||||
box.execute_command("echo hi", env={"BAD KEY": "x"})
|
||||
Loading…
Add table
Add a link
Reference in a new issue