mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-09 16:08:31 +00:00
72f033fbbe
* feat: add redis stream bridge * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(gateway): address redis stream bridge review Redis was imported eagerly through deerflow.runtime and declared as a hard dependency, which made memory-only installs load redis.asyncio at startup and left the lazy factory import ineffective. Move redis behind an optional extra, remove the public eager re-export, and keep make_stream_bridge as the only runtime import path with an actionable install hint when the extra is missing. Because Docker deployments now default the stream bridge to Redis via DEER_FLOW_STREAM_BRIDGE_REDIS_URL, install the redis extra explicitly in Docker/dev container flows and teach the local uv-extra detector to infer redis from both stream_bridge.type and the Redis URL env var. This keeps Docker working while preserving slim non-Docker installs. Harden the Redis bridge by batching XREAD replay, replacing brittle ResponseError string matching with a single fallback to 0-0 for malformed Last-Event-ID values, documenting connection/retention/fail-hard behavior, and adding fake plus opt-in real Redis coverage for XADD/XREAD, replay, invalid IDs, and MAXLEN trimming. * fix(config): bump config version for stream bridge * fix redis stream bridge terminal handling * fix: repair uv.lock, format redis.py, and align Dockerfile extras test The uv.lock file was missing a closing bracket for the redis extras section, redis.py had a formatting issue caught by ruff, and the Dockerfile extras test did not account for the hardcoded --extra redis flag. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
103 lines
4.6 KiB
Bash
Executable file
103 lines
4.6 KiB
Bash
Executable file
#!/usr/bin/env sh
|
|
#
|
|
# DeerFlow gateway dev entrypoint — runs inside the docker-compose-dev gateway
|
|
# container. Extracted from docker/docker-compose-dev.yaml's inline `command:`
|
|
# (PR #2767, addressing review on Issue #2754).
|
|
#
|
|
# Responsibilities:
|
|
# 1. Resolve `--extra X` flags from UV_EXTRAS (comma- or whitespace-separated,
|
|
# mirroring scripts/detect_uv_extras.py for parity with local `make dev`).
|
|
# 2. Validate each extra against [A-Za-z][A-Za-z0-9_-]* so a stray shell
|
|
# metacharacter in `.env` cannot reach `uv sync`.
|
|
# 3. `uv sync --all-packages` so workspace member extras (deerflow-harness's
|
|
# postgres extra in particular) are installed — see PR #2584.
|
|
# 4. Self-heal: if the first sync fails, recreate .venv and retry once.
|
|
# 5. Hand off to uvicorn with reload, replacing this shell so uvicorn becomes
|
|
# PID 1 inside the container.
|
|
#
|
|
# Anchored at /bin/sh (not bash) since alpine-based base images may not ship
|
|
# bash. Uses POSIX-only constructs throughout.
|
|
|
|
set -e
|
|
|
|
# `--print-extras` is a dry-run hook: parse + validate UV_EXTRAS, print the
|
|
# resulting `--extra X` flags to stdout, and exit. Used by the unit test in
|
|
# backend/tests/test_dev_entrypoint.py and useful for ad-hoc debugging.
|
|
PRINT_EXTRAS_ONLY=0
|
|
if [ "${1:-}" = "--print-extras" ]; then
|
|
PRINT_EXTRAS_ONLY=1
|
|
fi
|
|
|
|
# Mirror the legacy command's behavior: redirect both stdout and stderr to the
|
|
# host-mounted log file (../logs/gateway.log → /app/logs/gateway.log). Skip
|
|
# the redirect under --print-extras so the test runner can capture stdout.
|
|
if [ "$PRINT_EXTRAS_ONLY" = "0" ]; then
|
|
exec >/app/logs/gateway.log 2>&1
|
|
fi
|
|
|
|
# ── Resolve extras ──────────────────────────────────────────────────────────
|
|
|
|
EXTRAS_FLAGS=""
|
|
if [ -n "${UV_EXTRAS:-}" ]; then
|
|
# Normalize comma → space, then split on whitespace via the unquoted `for`.
|
|
for raw in $(printf '%s' "$UV_EXTRAS" | tr ',' ' '); do
|
|
[ -z "$raw" ] && continue
|
|
# Reject anything that does not look like an identifier.
|
|
# Two patterns: leading non-letter, or any non-[A-Za-z0-9_-] character.
|
|
case "$raw" in
|
|
[!A-Za-z]* | *[!A-Za-z0-9_-]*)
|
|
echo "[startup] UV_EXTRAS entry '$raw' is invalid (must match [A-Za-z][A-Za-z0-9_-]*) — aborting" >&2
|
|
exit 1
|
|
;;
|
|
esac
|
|
EXTRAS_FLAGS="$EXTRAS_FLAGS --extra $raw"
|
|
done
|
|
fi
|
|
|
|
if [ "$PRINT_EXTRAS_ONLY" = "1" ]; then
|
|
# Trim leading space for tidier output, then exit.
|
|
printf '%s\n' "${EXTRAS_FLAGS# }"
|
|
exit 0
|
|
fi
|
|
|
|
if [ -n "$EXTRAS_FLAGS" ]; then
|
|
echo "[startup] uv extras:$EXTRAS_FLAGS"
|
|
fi
|
|
|
|
# Keep runtime-owned files out of uvicorn's reload watcher. Each excluded path
|
|
# must exist before uvicorn starts so watchfiles treats it as an excluded
|
|
# directory, not as a plain glob pattern — on Python 3.12, globbing an absolute
|
|
# pattern raises NotImplementedError and crashes startup (#3459 / #3454). That
|
|
# means `sandbox` must be created here too, not just `.deer-flow`.
|
|
: "${DEER_FLOW_HOME:=/app/backend/.deer-flow}"
|
|
export DEER_FLOW_HOME
|
|
mkdir -p "$DEER_FLOW_HOME" /app/backend/.deer-flow /app/backend/sandbox
|
|
|
|
# ── Sync dependencies (with self-heal) ──────────────────────────────────────
|
|
|
|
cd /app/backend
|
|
|
|
# `--all-packages` propagates extras into workspace members (PR #2584).
|
|
# `--extra redis` is always installed because docker-compose-dev defaults the
|
|
# stream bridge to Redis (DEER_FLOW_STREAM_BRIDGE_REDIS_URL); redis is an
|
|
# optional extra elsewhere. It is kept out of EXTRAS_FLAGS so the --print-extras
|
|
# contract (UV_EXTRAS-derived flags only) stays unchanged.
|
|
# `$EXTRAS_FLAGS` intentionally unquoted so each `--extra X` becomes its own arg.
|
|
# shellcheck disable=SC2086 # word-splitting is intentional here
|
|
if ! uv sync --all-packages --extra redis $EXTRAS_FLAGS; then
|
|
echo "[startup] uv sync failed; recreating .venv and retrying once"
|
|
uv venv --allow-existing .venv
|
|
# shellcheck disable=SC2086
|
|
uv sync --all-packages --extra redis $EXTRAS_FLAGS
|
|
fi
|
|
|
|
# ── Hand off to uvicorn ─────────────────────────────────────────────────────
|
|
|
|
PYTHONPATH=. exec uv run uvicorn app.gateway.app:app \
|
|
--host 0.0.0.0 --port 8001 \
|
|
--reload \
|
|
--reload-include='*.yaml' \
|
|
--reload-include='.env' \
|
|
--reload-exclude=/app/backend/sandbox \
|
|
--reload-exclude="$DEER_FLOW_HOME" \
|
|
--reload-exclude=/app/backend/.deer-flow
|