Adds ``deerflow.community.e2b_sandbox.E2BSandboxProvider`` with parity
to AioSandboxProvider: metadata-keyed per-thread persistence, server-
side idle timeout, warm-pool reclaim with liveness checks, /mnt/user-data
bootstrap symlinks, dead-sandbox auto-rebuild, and release-time mirror
of agent outputs back to the host artifact directory.
Signed-off-by: joey <zchengjoey@gmail.com>
* fix(frontend): keep orphan tool messages visible
LangGraph `messages-tuple` stream mode can emit tool-result events
out of order or replay them from subagent state (e.g. the bash subagent
under LocalSandboxProvider with allow_host_bash: true). When that
happens, the tool message arrives after a terminal assistant/human
group, so getMessageGroups' lastOpenGroup() returns null.
The previous behaviour was console.error + drop, which silently hid
the tool result from the UI - the user could not see the tool output
or tool-call records even though the agent executed the action
correctly (backend + Langfuse trace were both fine).
Fallback now attaches the orphan tool message to the most recent group
so the UI shows it. Adds a unit test that covers the orphan-tool path
and a duplicate-stream regression test.
* test(frontend): make orphan-tool replay test actually reach the fallback
The previous fixture was `human → ai(tool_calls) → tool → tool(replay)`
with no terminal group in between, so both tool messages hit the
unchanged happy path (`open.messages.push(...)`). Neither message was
an orphan, the new `else if (groups.length > 0)` fallback was never
exercised, and the `>= 1` length assertion was satisfied trivially by
t-1a alone.
Interleave a terminal assistant message between the original tool
result and the replayed one, so t-1b arrives when lastOpenGroup()
returns null. Now the strict assertion that t-1b is reachable can only
pass via the new fallback branch.
Review feedback from @willem-bd on PR #3880.
* fix(frontend): satisfy noUncheckedIndexedAccess in orphan-tool fallback
The fallback branch pushed into `groups[groups.length - 1].messages`
directly, which trips `noUncheckedIndexedAccess` under the project's
strict TS config and breaks lint-frontend, e2e-tests, and the
full-stack render CI layer.
Take `groups[groups.length - 1]` into a local `lastGroup` and check
for `undefined` explicitly. The `else if (groups.length > 0)` guard
becomes redundant once `lastGroup` is checked, so the branches are
folded into the outer `else` to keep the structure flat. Behavior is
unchanged: orphan tools still attach to the most recent group, and
the empty-groups diagnostic still logs at ERROR level.
Review feedback from @willem-bd on PR #3880.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): add delegations ledger field + reducer to ThreadState
* feat(subagents): pure helpers to derive + format the delegation ledger
* feat(subagents): DelegationLedgerMiddleware records + injects the ledger
* feat(subagents): register DelegationLedgerMiddleware for lead when subagents enabled + docs
* add runtime log
* chore(subagents): make delegation-ledger injection log production-ready
* test(subagents): make delegation-ledger registration tests config-free; refresh ultra replay golden for delegations channel
* refactor(subagents): derive TERMINAL_STATUSES from SUBAGENT_STATUS_VALUES + pin it
Make thread_state's TERMINAL_STATUSES a frozenset over the status contract's
SUBAGENT_STATUS_VALUES instead of a hardcoded literal, so the terminal-status
set can never drift from the contract. Add a pinning test asserting the
derivation and that the non-terminal "in_progress" stays excluded.
Addresses PR #3877 review.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(sandbox): stop blocking bash commands from hanging the turn
Starting a server through the host bash tool (e.g. `python -m http.server`)
could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command`
used `subprocess.run(capture_output=True)`, whose captured pipes are inherited
by any process the command spawns — so a backgrounded long-lived process
(`server &`) keeps the read end open and blocks `communicate()` until the
timeout fires, even though the foreground command already returned. Commands
that read stdin blocked the same way, and on timeout only the direct child was
killed, leaving orphaned process groups.
Rework the POSIX path to capture stdout/stderr via temp files instead of pipes,
take stdin from /dev/null, and run the command in its own session/process group:
- Backgrounded long-lived processes (servers) now return immediately while the
process keeps running.
- A command reading stdin gets immediate EOF instead of blocking.
- A genuinely blocking foreground command is bounded by a configurable
wall-clock timeout; on timeout the whole process group is killed and the agent
gets an explanatory notice telling it to background long-lived processes.
The timeout is configurable via `sandbox.bash_command_timeout` (default 600).
The Windows path is unchanged. Adds focused regression tests and updates docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): instruct the agent to background long-lived processes
The code fix bounds a foreground server with a timeout, but the turn still
waits the full timeout before the run continues. Add the prompt-side half:
the bash tool description now tells the model to ALWAYS start long-lived
processes (e.g. web servers) in the background with output redirected, so the
tool returns immediately. The timeout notice points at the same readable
workspace log path. Pins the guidance with a test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): make fallback-kill exception explicit and observable
Address automated review: the inner `except OSError: pass` in
_terminate_process_group silently swallowed the case where the direct-child
fallback kill found the process already gone. Make the intent explicit with a
comment and a debug log instead of a bare pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): address bash timeout review feedback
* fix(sandbox): document fd cleanup races
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Split `_build_runtime_middlewares`'s flat list into three named declarative
sublists (outer_wrappers / thread_hooks / tail) and drop the
`middlewares.insert(2, UploadsMiddleware())` magic-index pattern. The
declarative structure makes the layering self-documenting and immune to
position drift when the head of the list changes.
Move UploadsMiddleware to run after ThreadDataMiddleware in the chain.
Under the previous order (a magic-index artifact introduced when #3662
prepended InputSanitizationMiddleware), UploadsMiddleware scanned the
uploads directory before ThreadDataMiddleware created it under
lazy_init=False, so historical files could be missed on the first run of
a thread. Narrow path — the upload endpoint normally pre-creates the
directory — but the order is the correct semantic and is now locked.
Documentation:
- backend/AGENTS.md middleware chain renumbered: ThreadDataMiddleware is
now #3, UploadsMiddleware #4 (was reversed).
Tests (backend/tests/test_tool_error_handling_middleware.py):
- test_build_lead_runtime_middlewares_orders_thread_data_before_uploads
— focused td_idx < um_idx assertion.
- test_build_lead_runtime_middlewares_chain_order_matches_agents_md
— full-chain order pin using real classes, so a swap between any pair
is caught (the existing FakeMiddleware-stubbed tests cannot detect
this).
- test_lead_runtime_chain_finds_historical_uploads_under_lazy_init_false
— integration anchor: under lazy_init=False, ThreadDataMiddleware
creates the dirs, then UploadsMiddleware surfaces a pre-existing
historical file in the injected <uploaded_files> context.
After SummarizationMiddleware runs, the merged conversation view could drop
already-displayed messages (previous assistant output, current user input),
leaving a nearly-empty thread.
Root cause: the display merge combines `visibleHistory` (archived history, a
React `useState` in useThreadHistory) with `persistedMessages` (live thread,
the LangGraph SDK external store via useSyncExternalStore). On summarization
the backend removes every live message and onUpdateEvent re-archives them via
an async `appendMessages` setState. Those two state systems are scheduled
independently, so a render can observe the post-summary (shrunk) thread before
the archive setState commits — the rescued messages are then absent from BOTH
merge inputs and get dropped.
Fix: bridge the async gap with a synchronous `pendingArchivedMessagesRef`
buffer written the moment onUpdateEvent computes the moved messages and read by
the merge on every render, so correctness no longer depends on how the two
channels interleave. The buffer drains once history confirms absorption and
only injects messages missing from history (live copies stay authoritative,
order preserved). It is tagged with the thread it was captured from and the
merge overlays it only when that matches the viewed `threadId` (the same prop
visibleHistory is gated on), so it can never leak into another thread or the
new-chat screen — a read-only check, no render-phase ref mutation.
Extracts the moved-message derivation and the merge overlay into pure,
unit-tested helpers (computeSummarizationMovedMessages, resolvePreservedHistory,
pruneConfirmedArchivedMessages) with regression coverage for the full rescue
pipeline.
* fix(feishu): stop creating thread topics and throttle card updates
- Remove reply_in_thread(True) so replies appear as normal messages (#1332)
- Use chat_type=p2p for shared LangGraph thread in P2P chats
- Add two-level stream throttle (1.0s interval / 60 char buffer) with OR logic
- Add cursor indicator for streaming
- Filter values from overwriting delta text during streaming
Closes#3801
* fix(feishu): stop creating thread topics, throttle card updates, preserve clarification
- Remove reply_in_thread(True) so replies appear as normal messages (#1332)
- Use chat_type=p2p for shared LangGraph thread in P2P chats, with stored
mapping fallback for backward compatibility with pre-upgrade threads
- Add two-level stream throttle (1.0s interval / 60 char buffer) with OR
logic and cursor indicator for non-final outbounds
- Re-publish clarification text from values snapshots mid-stream
- Update streaming tests to the new contract (cursor glyph, clarification)
Closes#3801
Closes#3192
Root cause
----------
The artifact preview header is driven by a Radix Select whose <SelectValue>
renders the label of the <SelectItem> matching the current value. The option
list was built solely from `artifacts` in ArtifactsContext, which is only
synced from `thread.values.artifacts` (chat-box.tsx). Artifacts surfaced via
the message-layer `present_files` tool call are never written back to
`thread.values.artifacts`, so when such a file is selected its filepath has
no matching <SelectItem>. Radix then renders an empty trigger and the header
filename appears blank, even though the preview body loads correctly.
Fix
---
Compute `artifactOptions` as a defensive union: if the currently selected
filepath is missing from `artifacts`, prepend it so a matching <SelectItem>
always exists. This keeps the header label in sync with the active file
without changing context semantics or coupling the UI to a future
auto-discovery mechanism (see existing TODO in chat-box.tsx).
Tests
-----
Add an e2e case that mocks a thread whose only artifact is delivered via
`present_files` (thread.values.artifacts = []) and asserts both the header
title and preview content render. All 21 e2e tests pass.
When DeerFlow's nginx runs behind another TLS-terminating reverse proxy
(Pangolin/Traefik, Cloudflare, Caddy), every location block overwrote the
already-correct X-Forwarded-Proto with $scheme (= http on the private hop).
The Gateway then treated HTTPS browser traffic as HTTP: the auth-origin check
rejected the login POST with 403 "Cross-site auth request denied", and session
cookies lost the Secure flag and max-age.
Preserve an upstream X-Forwarded-Proto via a map that falls back to $scheme when
nginx is itself the TLS edge, so standalone `make dev` / Docker is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_or_new_skill_storage() and reset_skill_storage() touch the
process-global skill storage singleton without a lock - the same
unsynchronized check-then-create that #3730 just fixed in
sandbox_provider.py, the module this file documents itself as mirroring.
Two callers racing a cold start can both see _default_skill_storage is
None and each build a SkillStorage, so the second overwrites the first;
a reset_skill_storage() racing a get can also null the global between
the None-check and the return.
Guard the build/return and the reset with a module-level threading.Lock
and a double check, mirroring get_memory_storage(). Construction stays
inside the lock (rather than sandbox_provider's build-outside-then-
discard-the-loser) because SkillStorage has no teardown hook, so a
losing racer's orphan could not be cleaned up.
Add backend/tests/test_skill_storage_lifecycle.py with concurrency
regression tests (8-thread cold-start race asserting a single instance;
reset racing gets asserting no None is returned).
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* feat(tui): add Hermes-like terminal workbench backed by DeerFlowClient
Implements the `deerflow` TUI from RFC #3540: a terminal-native, embedded
workbench over the existing harness (no Gateway/frontend/nginx/Docker), built
Python-native with Textual and learning UX patterns from tao-pi.
Architecture — every layer except the Textual app is pure and unit-tested:
- view_state.py: ViewState + reduce(state, action), the testable heart
- runtime.py: StreamEvent -> reducer actions (pure translate + threaded driver)
- message_format / command_registry / input_history / render / theme: pure
- app.py: Textual App; runs the sync DeerFlowClient.stream() on a worker thread
and marshals actions back to the UI thread. Slash command palette, model and
thread modal pickers, ↑/↓ history, Ctrl+C interrupt, TTY-aware fallback.
- cli.py: pure launch-mode planning + headless --print/--json + `deerflow`
console script (textual is an optional [tui] extra; degrades to headless help)
Web UI visibility (the RFC's key decision): persistence.py writes a threads_meta
row under the local default user into the same DB the Gateway reads, so terminal
sessions appear in the Web UI sidebar without running the Gateway. Best-effort,
no-op on the memory backend; all DB work on one long-lived background loop.
Tests: 95 TUI tests — pure layers via pytest, app/palette/overlays via Textual's
pilot harness with a fake session, and a threads_meta read/write round-trip.
ruff clean; respects the harness->app import boundary. Docs: backend/docs/TUI.md
plus CLAUDE.md/README updates and preview screenshots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): de-duplicate streamed assistant text and tool cards; keep Tab in composer
Self-test surfaced three issues, all root-caused to consuming non-strict
streaming from DeerFlowClient (proven by the client's own
test_dedup_requires_messages_before_values_invariant, which shows the client can
re-emit a message id's full content twice):
- Assistant text was doubled (e.g. "answer answer") because the reducer blindly
concatenated same-id deltas. Now merges by content: a re-send or cumulative
snapshot replaces; only genuine increments append.
- Tool activity showed duplicate and empty "gear" cards from partial/re-emitted
tool-call chunks. ToolStarted now de-dupes by tool_call_id, drops id-less
noise chunks, and fills the name on a later chunk; a tool result with no prior
card still surfaces as a completed card.
- Tab moved focus off the composer to the scroll region (felt like broken cursor
logic). Tab is now consumed by the composer (completes a command when the
palette is open, no-op otherwise).
Adds reducer tests for each case plus a Tab-focus test; 102 TUI tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): make Esc interrupt an active run (matches the status hint)
The status line advertised "esc interrupt" but Esc was only wired to close the
slash palette, so it did nothing during a run. Esc now: closes the palette when
open, interrupts the active run when streaming, and is a no-op when idle. The
interrupt logic is shared with Ctrl+C via _interrupt_run(). Adds a regression
test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): stop prior answers duplicating on threads with history
On a thread with history, DeerFlowClient re-emits every prior message on each
new turn (its streamed_ids dedup is per-stream-call), and a re-emitted older
message can arrive after a newer message has already started. The reducer only
matched the *most recent* assistant row by id and otherwise appended, so each
re-emitted older answer was duplicated verbatim at the end of the transcript.
Match an assistant row by id anywhere in the transcript and merge in place.
Tool cards already de-dupe by call id globally, so they were unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): correct CJK cursor drift in the composer
Confirmed a Textual Input bug (latest 8.2.7): Input._cursor_offset adds an
unconditional +1 at the end of the value, overshooting by one cell after
double-width (CJK) characters. That misplaces the hardware/IME cursor — the
drift seen when typing Chinese in iTerm2 (the on-screen block cursor, drawn
separately in render_line, is fine; English doesn't use an IME so it looks
correct). Reproduced with a bare Input, so it's upstream, not our layout.
Add ComposerInput(Input) overriding _cursor_offset to the true cell position and
use it for the composer. Numeric tests pin the CJK end/mid and ASCII cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(tui): render finalized assistant messages as Markdown
The transcript showed raw Markdown (literal **bold**, ## headings, - lists,
links). Finalized assistant messages now render as Rich Markdown — headings,
bold/italic, lists, inline code + code blocks, blockquotes, horizontal rules and
links — with the ● speaker marker aligned to the top of the body.
The actively-streaming message stays plain text so partial Markdown doesn't
reflow/jump, then snaps to its rendered form when the run ends. Transcript
re-renders are coalesced on a ~60ms timer (dirty flag) so per-token Markdown
re-parsing stays smooth on long threads. Tests cover both the rendered and the
streaming-plain paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style(tui): apply ruff format
CI lint runs `ruff format --check` via uvx (latest ruff); apply the formatter so
the lint-backend job passes. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(tui): address code-quality review comments
From github-code-quality[bot] on #3760:
- runtime.py: give the `_ClientLike` Protocol method a docstring body instead of
a bare `...` (flagged as a no-effect statement), matching the harness
convention for Protocol stubs (e.g. SafetyTerminationDetector).
- test_tui_cli_main.py: drop the unnecessary `lambda: _FakeSession()` wrappers in
monkeypatch.setattr; pass `_FakeSession` directly (same behavior).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): keep history Markdown-rendered when a follow-up run starts
Previously the transcript rendered "the last assistant row" as plain text while
streaming. But when a follow-up turn starts, the last assistant row is the
*previous, finalized* answer until the new message begins — and the client
re-emits prior messages early in the turn — so sending a follow-up reverted the
previous answer from rendered Markdown back to raw text.
Track the actively-streaming message id in ViewState instead: it's reset on
RunStarted, set only when an AssistantDelta actually adds new content (history
re-emits are no-ops and don't mark it), and cleared on RunEnded. The renderer
keeps only that one message plain; all history stays Markdown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(readme): add Terminal Workbench (TUI) section to root README
Mention the new `deerflow` TUI alongside the Embedded Python Client in the root
README.md and README_zh.md (install, launch/headless commands, feature summary,
Web UI visibility), with a ToC entry and a preview screenshot. Links to
backend/docs/TUI.md for the full guide.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(tui): address review feedback (willem-bd)
Ten findings from the TUI code review:
1. /resume was dead-ended — registered + in /help + tested as a builtin, but no
dispatch branch. Wired it to thread resolution / the switcher.
2. --resume <title> was forwarded raw into the checkpointer (blank thread).
Added Session.resolve_ref() to resolve id-or-title via list_threads; used by
--resume and /resume.
3. str(get("id","")) returned "None" for an explicit id:None (truthy), defeating
the empty-id guard so unrelated null-id tool calls collapsed into one card.
Coerce via a None-safe helper.
4. Headless --print/--json no longer spin up the persistence loop/engine/pool
(open_session(persistence=False)).
5. _LoopThread + engine are now closed: Session.close() (dispose engine + stop
loop) called from a try/finally around app.run().
6. --cli --continue (and piped --cli) now run headless instead of erroring.
7. Cancelled runs no longer persist a truncated title (guard on _cancelled).
8. Palette highlight resets to the top when the filter set changes.
9. Dropped the never-populated tools count from the header.
10. Documented the `not row.error` merge guard.
Adds regression tests for each; 126 TUI tests pass, ruff check + format clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* docs: add root-level CLAUDE.md to orient the monorepo
Adds a thin top-level CLAUDE.md that maps the monorepo and delegates depth
to backend/CLAUDE.md and frontend/CLAUDE.md, per issue #3761.
Includes the project overview + service topology (Nginx 2026, Gateway 8001,
Frontend 3000, optional Provisioner 8002), a top-level repository map, root
`make` vs. per-module command sections, "where to go next" links to the module
guides and primary root docs, and the repo-wide cross-cutting conventions
(documentation-update policy, TDD expectation, format before pushing).
No code or behavior changes; root points down, modules own depth.
Closes#3761
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: make AGENTS.md the source of truth, CLAUDE.md a thin @AGENTS.md importer
Adopt the AGENTS.md convention so the same agent guidance serves Claude Code,
Codex, and other tools. At each level (root, backend, frontend) the content
lives in AGENTS.md and CLAUDE.md just imports it via `@AGENTS.md`.
- root: move the monorepo orientation layer to AGENTS.md; CLAUDE.md -> @AGENTS.md.
Fix an incorrect "TUI" reference (not present on main) and repoint the module
links to the AGENTS.md files.
- backend: move the guide to AGENTS.md (was an AGENTS.md -> @CLAUDE.md pointer;
direction is now flipped). Refresh stale content: rebuild the full middleware
chain (~26 ordered steps incl. InputSanitization, ToolOutputBudget,
DynamicContext, TokenBudget, SafetyFinishReason) from the actual build
functions; drop the brittle "11 middleware components" count; expand the
community-tools list to the real set.
- frontend: merge the practical Next.js guide with the existing AGENTS.md's
unique sections (LangGraph diagram, tech-stack versions, interaction
ownership, resources) into one AGENTS.md (CLAUDE.md -> @AGENTS.md). Fix the
stale src/ layout (remove the no-longer-present server/ better-auth entry;
add the now-active auth/agents/blog/... modules and routes) and drop a bogus
interaction-ownership bullet referencing files that don't exist.
Docs only; no code or behavior changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): synchronize sandbox provider singleton lifecycle
get_sandbox_provider() used an unsynchronized check-then-create, so two OS
threads (e.g. the main event loop and the Feishu channel thread, which runs
its own loop) could double-initialize the provider. With AioSandboxProvider
the overwritten instance leaks its idle-checker thread, since the only code
that joins it (shutdown()) is reachable only through the reference that was
overwritten.
reset_sandbox_provider(), shutdown_sandbox_provider() and set_sandbox_provider()
also touched the global without a lock, so a reset/shutdown racing an in-flight
create could clear it mid-construction or tear down an instance another thread
was about to return.
Guard all four lifecycle sites with a single module-level threading.Lock and
use double-checked locking in the getter, mirroring get_memory_storage().
* test(sandbox): add concurrent regression tests for provider singleton
- 8 threads racing on cold start, synchronized with a threading.Barrier so
the check-then-create race fires deterministically; asserts exactly one
provider instance is created.
- reset racing concurrent gets: asserts every returned value is a fully
constructed provider (never None / half-built).
* fix(sandbox): lock get read path, run provider callbacks outside the lock
Addresses the three review findings on #3730:
1. get_sandbox_provider()'s read+return ran outside _provider_lock, so a
concurrent reset/shutdown/set could null or tear the global between the
check and the return, handing callers None / a torn instance. The hot read
and the install reconciliation now both happen under the lock.
2. The non-reentrant _provider_lock was held across plugin-supplied callbacks
(resolve_class import + provider __init__ in get; provider.reset()/shutdown()
in reset/shutdown). A custom provider that re-entered these lifecycle
functions would self-deadlock, and a slow teardown blocked every concurrent
get(). resolve_class + construction now run outside the lock; reset/shutdown
detach the reference under the lock and invoke the callback outside it.
Tradeoff: racing cold-start callers may each construct a candidate. Exactly
one is installed and returned to everyone; the losers (e.g. an AioSandbox
instance that already started an idle-checker thread) are shut down so they
do not leak the orphan thread #3721 is about. set_sandbox_provider() documents
that it replaces but does not shut down the prior instance.
3. test_reset_racing_get reset the singleton to None *before* the barrier, so
the racing reset was a no-op and never exercised reset-of-a-live-provider.
It now populates the singleton up front so the reset tears down a live
instance while getters read it.
Tests: rename the cold-start test to assert "one installed singleton, observed
by all" (construction is no longer single under the new design); add
shutdown-vs-get, set-vs-get, and a losing-racer-shuts-down-its-orphan case.
All five pass; the existing sandbox/middleware/mounts/uploads suites are green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add endswith('__user') guard in _is_user_injection_target to prevent
unbounded suffix growth (id__user__user__user...) when a prior ID-swap
peer is mistakenly treated as a new injection target
- Add peer rescue in _preserve_dynamic_context_reminders to keep
ID-swap __user and __memory messages out of summary compression,
preventing orphaned SystemMessage reminders and lost user context
- Add 6 regression tests covering both failure modes
* perf(runtime): index MemoryRunEventStore events by run_id to avoid O(n) scans
list_messages was already served from a thread-wide messages projection
(#3531), but list_events and list_messages_by_run still scanned the whole
thread's event log (every run, every category) to return one run's events --
O(N_thread) on every run-scoped /messages page-load and /events request.
Add _events_by_run / _messages_by_run projections (same dict objects, kept
in lockstep in _put_one / delete_by_run / delete_by_thread), so both reads
are O(M_run), with bisect cursor pagination for messages. Semantics are
unchanged, pinned by a brute-force parity test over interleaved traces and
both cursors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style: ruff format test_run_event_store_by_run_index.py
Clears the lint-backend (ruff format --check) failure on the PR; the original
commit ran `ruff check` but not `ruff format`. No behavior change (test fixture
formatting only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): improve chat math rendering
* fix(frontend): refine math rendering stability
* fix(frontend): address review feedback on math preprocessing
- Fix escaped-backslash blind spot: consume \\\\ as a unit so \\( and
\\[ are not mis-interpreted as math delimiters when the backslash
itself is escaped.
- Thread inInlineCode state across lines so multi-line backtick code
spans are protected from delimiter conversion.
- Remove double math normalization: preprocessStreamdownMarkdown now
only handles Mermaid; math normalization lives solely in
ClipboardSafeStreamdown, preventing non-idempotent double passes.
- Wire parseIncompleteMarkdown to isLoading in MarkdownContent so
Streamdown's streaming-incomplete-math handling is actually active
during streaming.
* fix(frontend): address streamdown math review issues
* refactor(frontend): move streamdown preprocessing out of ai elements
* feat(persistence): wire alembic migrations + bootstrap schema on startup
Closes#3682. Pre-#3658 DBs lack the `runs.token_usage_by_model` column
because alembic was never wired up — startup only ran `create_all`,
which never ALTERs existing tables.
Adds a hybrid bootstrap in FastAPI lifespan (replaces bare `create_all`):
- empty DB → create_all + stamp head
- legacy DB → stamp 0001_baseline + upgrade head
- versioned DB → upgrade head
Concurrency: Postgres `pg_advisory_lock` (cross-process); SQLite
per-engine `asyncio.Lock` + 30s `PRAGMA busy_timeout` on both prod and
alembic engines. Column revisions use `safe_add_column` /
`safe_drop_column` idempotent helpers as fallback.
Other bits:
- 0001 baseline (chain root) + 0002 add `runs.token_usage_by_model`
- `include_object` filter so alembic ignores LangGraph checkpointer tables
- `make migrate-rev MSG="..."` for authoring new revisions
(no migrate/stamp targets — startup is the only execution path)
- Tests: three-branch decision, concurrency, #3682 regression, env
filter, blocking-IO gate anchor
- CLAUDE.md: new "Schema migrations" section
* fix(style): fix lint error
* perf(persistence): address review feedback on alembic bootstrap
Behavioural fixes
- _SQLITE_LOCKS now keyed via WeakKeyDictionary so id-reuse after GC
cannot return a stale, loop-bound lock and the cache cannot leak one
entry per disposed engine.
- safe_add_column compares nullable / server_default against the desired
column when the name already exists and emits a warning on drift,
surfacing manual-ALTER workarounds instead of silently no-op'ing.
- _postgres_lock issues SET LOCAL idle_in_transaction_session_timeout=0
before pg_advisory_lock, so managed Postgres cannot kill the idle
lock-holding session mid-upgrade and silently release the advisory
lock.
- legacy branch now backfills missing baseline tables via a restricted
create_all (Base.metadata.create_all scoped to _BASELINE_TABLE_NAMES).
Restores pre-#1930 upgraders whose channel_* tables were never
provisioned, without pre-empting future create_table revisions for
newly-added models.
Schema parity
- runs.token_usage_by_model gains server_default=text("'{}'") in both
the ORM model and the 0001_baseline create_table, matching what 0002
adds via ALTER. create_all and alembic-upgrade paths now produce
identical column definitions.
- New parity test compares Base.metadata.create_all output against a
pure alembic upgrade base->head, asserting column-set, nullable, and
server_default agree across all tables (normalized through the same
helper safe_add_column's drift check uses).
Guards
- test_baseline_table_names_constant_matches_0001 pins
_BASELINE_TABLE_NAMES to 0001_baseline.upgrade()'s actual output --
the constant cannot drift silently when someone edits 0001.
- test_legacy_backfill_skips_non_baseline_tables verifies the restricted
backfill does not create a phantom table on Base.metadata, modelling
a future revision that would otherwise collide on op.create_table.
Doc residuals
- Three-branch decision table is now consistent across bootstrap.py
top docstring, engine.py comment, test module docstring, and
CLAUDE.md.
- Stale test anchor in blocking_io/test_persistence_engine_sqlite.py
docstring now points at the real file.
* fix(style): fix lint error
* fix(persistence): close drift detection holes
- _check_column_drift compares column type via a family equivalence
allowlist ({JSON, JSONB}). Catches the wrong-type workaround
`TEXT NOT NULL DEFAULT '{}'` that previously slipped through silently,
while keeping Postgres JSON/JSONB dialect reflection quiet. Reflected
and desired type are also echoed in every drift warning's payload for
operator triage.
- Extract _escape_url_for_alembic so bootstrap._alembic_safe_url and
scripts/_autogen_revision share the ConfigParser % escape rule
instead of duplicating it.
- backend/README.md: add `make migrate-rev MSG=...` to Commands and a
Schema Migrations section per the repo's README/CLAUDE.md sync policy.
- test_base_to_dict.py: scope the test ORM class to an isolated MetaData
so the create_all-vs-alembic parity test (added in the previous
commit) is not polluted by the phantom table on the full pytest
session.
* perf(runtime): index MemoryRunStore by thread_id to avoid O(n) scans
MemoryRunStore is the default run backend (database.backend=memory) and backs
RunManager.list_by_thread, which calls it on every thread-runs query to hydrate
persisted runs. list_by_thread scanned every run in the store (O(total runs))
to filter by thread_id, so listing one thread's runs got linearly slower as
unrelated runs accumulated across all threads.
Add a thread_id -> insertion-ordered run_id set secondary index, maintained in
lockstep with _runs in put()/delete(), and use it in list_by_thread for an
O(runs-in-thread) lookup. This mirrors the index RunManager already keeps over
its own in-memory records (#3499); the store extraction — whose docstring notes
it is "Equivalent to the original RunManager._runs dict behavior" — did not
carry the index across.
Behavior is unchanged: same user_id filtering, newest-first ordering, and limit.
The store runs each method without awaits on the event loop, so the index and
_runs stay consistent without a lock.
Extends tests/test_persistence_scaffold.py::TestMemoryRunStore with coverage for
unknown-thread, newest-first ordering, limit, and index cleanup on delete
(including empty-bucket removal).
* perf(runtime): route aggregate_tokens_by_thread through the thread index too
list_by_thread already uses the _runs_by_thread index this PR adds, but
aggregate_tokens_by_thread (the /token-usage endpoint) still scanned every run
in the process to pick out one thread's runs. Route it through the same index
for an O(runs-in-thread) lookup, completing the thread-scoped read coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_artifact ran its filesystem work directly on the event loop: virtual-path
resolution (os.path.abspath via .resolve()), exists/is_file probes, MIME sniffing
(mimetypes lazily stats the system MIME DB on first use), full-file
read_text/read_bytes, is_text_file_by_content (open+read), and .skill ZIP
open+extract. So serving any artifact blocked the loop for the whole read;
`make detect-blocking-io` flagged it. Same class as #3457 / #3529.
Offload each branch's IO via asyncio.to_thread: one sync helper per branch
(_load_skill_archive_member, _read_artifact_payload) folds stat + MIME + read /
extract into a single worker hop and returns a small (kind, mime, payload) plan
the handler turns into the response on the loop. FileResponse (download / active
content) keeps streaming the file itself. Behavior, branching, error codes, and
security boundaries are unchanged.
Add tests/blocking_io/test_artifacts_router.py anchor (text / binary / .skill
member), verified red->green under the strict Blockbuster gate. The gate also
caught a blocking call the static scan missed: resolve_thread_virtual_path's
.resolve() (os.path.abspath), now offloaded too.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(channels): let UI runtime channel config win over config.yaml
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* test(channels): update assertion to expect runtime/UI value to win over yaml
The test was written when yaml took precedence over runtime config.
This PR inverts that precedence so UI-entered credentials win; the
assertion now correctly reflects that runtime value (xapp-ui) beats
the yaml value (xapp) on a shared key.
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Part of #3742. RunJournal._message_text and the gateway thread-messages
helper (thread_runs._message_text) reimplemented the same 'extract display
text from a message' logic — str / list of string|{text}|nested{content}
blocks joined without a separator / mapping with text|content key. They
differed only in two ways: journal reads a BaseMessage attribute while
thread_runs reads dict-shaped run_events rows, and journal falls back to
message.text.
Add deerflow.utils.messages.message_to_text(message, *,
text_attribute_fallback=False) that handles both message shapes (attribute
or mapping content access) and gates the .text fallback behind a flag, and
have both call sites delegate. journal passes text_attribute_fallback=True;
thread_runs uses the default. Behavior is unchanged at both sites.
Verified behavior-preserving with an equivalence harness running both
original implementations vs the shared helper over 98 inputs (BaseMessage
and dict messages; str/list/mapping/None/numeric content; mixed blocks;
.text attribute present/absent/non-str) -> 0 mismatches. Added
tests/test_utils_messages.py; the journal last_ai_message extraction tests
still pass.
These three module-private helpers have no callers anywhere in the repo
(verified by a repo-wide identifier scan + direct greps; each appears only
at its own definition):
- agents/memory/updater.py: _create_empty_memory (a no-caller wrapper around
storage.create_empty_memory, which is used directly elsewhere)
- runtime/runs/worker.py: _extract_human_message (also drops the now-unused
HumanMessage TYPE_CHECKING import and TYPE_CHECKING from typing)
- sandbox/tools.py: _looks_like_unsafe_cwd_target (cwd safety is enforced via
_is_allowed_local_bash_absolute_path)
Underscore-prefixed = module-internal, so zero references is conclusive.
Route handlers and @compiles-registered functions were excluded as false
positives. Public-surface candidates are noted in the issue, not touched here.
Fixes#3748.
DynamicContextMiddleware (PR #3630, p0) uses the ID-swap technique to
inject a SystemMessage(reminder) into the middle of the conversation.
create_agent then prepends the static system_prompt as another
SystemMessage at request time, so strict OpenAI-compatible backends
(vLLM, SGLang, Qwen) and Anthropic reject the request with
'System message must be at the beginning'.
New SystemMessageCoalescingMiddleware runs in wrap_model_call — after
create_agent prepends system_prompt — and merges every SystemMessage
into a single leading one before the request reaches the provider.
Non-system messages keep their original order; the merged SystemMessage
preserves the id of the first system message. Only the request payload
is touched; checkpoint state is unchanged, so every consumer that scans
history (memory builder, journal, summarization, dynamic-context
detection) keeps working.
Mirrors the per-request coalescing already done for Claude in
claude_provider._coalesce_system_messages (PR #3702) but at a
provider-agnostic layer so every backend benefits from a single fix.
Closes#3707
* fix(middleware): fix positional fallback consuming unrelated todo when same-content list is exhausted
When next_todos contains the same content string twice (e.g. two "A"
entries), the first iteration pops the matching previous todo from
previous_by_content["A"], leaving an empty list. The second iteration
finds that empty list, treats it as falsy, and sets previous_match=None.
The positional fallback then fires unconditionally — consuming
previous_todos[index] even if it holds a completely different entry
(e.g. "B"). That marks "B" as matched, so the final loop never emits
a todo_remove action for it.
Fix: guard the positional fallback with `content not in previous_by_content`
so it only fires for genuinely new content (the key was never in previous),
not for same-content entries whose list has simply been exhausted.
Add a regression test to _build_todo_actions covering this exact case.
* style: ruff-format the token usage middleware test
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
get_model_config / get_tool_config / get_tool_group_config did a next(...)
linear scan of self.models / self.tools / self.tool_groups on every call.
These sit on hot paths: get_tool_config runs 2-3x per community-tool
invocation (web_search etc.) and get_model_config several times per agent
build, across 40+ call sites.
Build name -> config dicts once in a mode="after" validator (stored as
PrivateAttr), so each getter is an O(1) dict lookup. A config reload
constructs a fresh AppConfig, which rebuilds the indexes; setdefault keeps
first-match-wins on duplicate names, matching the prior next(...) semantics.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MemoryStreamBridge._resolve_start_offset scanned the retained event buffer
(up to queue_maxsize=256 entries) on every subscribe/reconnect carrying a
Last-Event-ID. Event ids are "{ts}-{seq}" where seq is a per-run monotonic
counter that equals the event's absolute offset, so the offset is computable
arithmetically. Parse seq, index into the buffer, and verify the id matches
exactly -- a stale/evicted/foreign/malformed id falls back to
replay-from-earliest, identical to the previous scan.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Middleware-injected hidden messages (TodoMiddleware.todo_reminder,
ViewImageMiddleware, p0 DynamicContextMiddleware.__memory) were being fed
to the memory-updating LLM as if they were real user input, polluting
long-term memory with framework-internal text. The p0 __memory payload
could also trigger a self-amplification loop.
Fix: skip HumanMessages with additional_kwargs['hide_from_ui'] in
filter_messages_for_memory, consistent with the frontend logic.
Closes#3695
mask_local_paths_in_output runs once per glob/grep match (tools.py:1540,
:1625) and once per bash/ls output, and each call rebuilt every skills /
ACP / user-data masking regex from scratch — Path.resolve() syscalls +
re.escape + re.compile. A grep returning up to 100 matches recompiled the
same patterns 100x.
Compile the host->virtual patterns once per ordered source set via
functools.lru_cache (keyed on the config-stable + per-thread (host,
virtual) pairs, so it self-invalidates when they change) and collapse the
three byte-identical replacer blocks into one applier. Behavior is
unchanged: same patterns, same application order, same per-thread mappings.
Verified behavior-preserving with an equivalence harness (original vs
refactored logic over 54 inputs incl. slash-style variants, base-only
matches, subpaths, overlapping mappings, missing skills/ACP, thread_data
None) -> 0 mismatches. Added cache/consistency tests alongside the
existing masking tests.
Fixes#3712.
* feat(memory): add guaranteed injection for correction facts with graceful fallback
When the token budget is tight, high-value facts (e.g. user corrections)
can be silently evicted by lower-priority regular facts. This change:
- Introduces configurable 'guaranteed_categories' (default: [correction])
whose facts draw from a separate 'guaranteed_token_budget', ensuring
they are never dropped due to budget pressure.
- Adds a graceful fallback to confidence-only ranking when the
guaranteed-category path raises an unexpected exception.
- Refactors fact selection into a header-agnostic helper
(_select_fact_lines) with explicit token accounting in the caller,
eliminating double-counting of separators.
- Emits a single 'Facts:' header regardless of whether both guaranteed
and regular facts are present.
- Extends the final safety truncation limit to account for the
additional guaranteed budget so guaranteed facts survive end-to-end.
* refactor(memory): address review feedback on guaranteed injection
- Restore strict break-on-overflow in `_select_fact_lines` to preserve
the caller's confidence-ordered ranking; add a regression test locking
in the invariant that a shorter lower-confidence fact never slips
ahead of a skipped higher-confidence one.
- Account for the inter-group `\n` separator between guaranteed and
regular fact blocks in the regular budget (1-token precision fix).
- Clarify docstrings on `format_memory_for_injection` and
`MemoryConfig.guaranteed_token_budget` to distinguish the common
*displacement* case (total stays within `max_tokens`) from the rarer
*additive* case (safety-truncation ceiling raised when guaranteed
lines alone would overflow).
* fix(memory): address P1 safety truncation + P2s from review
- Structure-aware safety truncation: Facts block is now a protected
suffix so guaranteed-category facts can never be silently discarded
by a prefix-cut on overflow. Only the preceding (user/history)
sections are eligible for truncation.
- Extend the same protected-suffix treatment to the except/fallback
path by returning fact lines alongside the formatted section from
_fallback_format_facts, avoiding string parsing.
- Single inter-section separator: facts section no longer embeds its
own leading \n\n; the final "\n\n".join(sections) is the single
source of truth for section-to-section spacing.
- Bare string for guaranteed_categories now raises TypeError instead
of silently iterating single characters.
- Category-less / malformed facts no longer default-promote into the
guaranteed "context" pool — only facts with an explicit category
field qualify.
- Lift valid_facts pre-filter outside the try so the fallback path
reuses it instead of re-doing validation work.
- MemoryConfigResponse + DeerFlowClient.get_memory_config now expose
guaranteed_categories / guaranteed_token_budget.
- config.example.yaml: document the two new fields and bump
config_version from 12 to 13.
- Add regression tests for every finding.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
