Atlas Cloud (https://atlascloud.ai) exposes a single OpenAI-compatible
endpoint in front of many open models (DeepSeek, Qwen, Kimi, GLM,
MiniMax, Llama, ...). It needs no new provider code — it uses the same
ChatOpenAI + base_url pattern already documented for OpenRouter, Novita
and other gateways.
Add a commented example to config.example.yaml: a plain ChatOpenAI entry
plus a PatchedChatOpenAI variant for *-thinking model ids so
reasoning_content is replayed across multi-turn tool calls. The key is
read from the ATLASCLOUD_API_KEY environment variable via the $VAR form,
consistent with the other examples.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
_lock was defined in __init__ and used only in reset(), leaving _apply(),
before_agent(), _clear_run_state(), and _drain_pending_warnings() unprotected.
Concurrent after_model callbacks for the same run_id could race on
usage_accum.input += diff_input, causing lost-update on token counters.
Also fix stray bare 'a' character in frontend/src/content/zh/introduction/index.mdx
that rendered as visible text on the documentation page.
Co-authored-by: kapilvus <kapilvus@gmail.com>
The Official Website hero image and the Coding Plan banner referenced
deleted GitHub user-attachment assets that now 404. No working
replacement exists, so remove them; section headings and prose/bullet
links already cover the same destinations.
Closes#3715
* test(agents): multi-turn message-stream invariants (graph integration)
Add a graph-integration net for the class of bug behind #3684: build a real
create_agent graph with DynamicContextMiddleware plus a checkpointer, drive two
user turns on one thread with a deterministic fake model and memory injection
stubbed on, then assert the message stream stays well-formed -- the newest user
message is the latest human turn, no duplicate ids, no __user__user suffix
explosion.
Runs at unit speed in `make test` (backend-unit-tests), with no gateway, SSE,
fixtures, or API key. Verified red on the pre-fix middleware and green after.
Catches this class earlier than e2e replay, which disables memory, uses a
single-turn golden, and asserts SSE shape only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(agents): address review — semantic-first asserts, _STREAM_MIDDLEWARES, DRY
- Check the semantic invariant (newest user message is the latest human turn)
before the structural id checks, so a regression surfaces as a meaning-level
failure; verified it now fails first on the pre-#3685 middleware.
- Add module-level _STREAM_MIDDLEWARES (the docstring referenced it; it did not
exist) so the net is trivially widened with more state-touching middlewares.
- _last_human_text reuses _msg_text instead of re-implementing content flattening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): skip dateless reminders in dynamic-context date scan
_last_injected_date returned at the first message flagged
dynamic_context_reminder regardless of whether it carried a
<current_date>. Since #3630 split injected context into a date
SystemMessage plus a separate dateless <memory> HumanMessage, the
reverse scan hit the memory message first and returned None, making a
later turn look like the first turn. The middleware then re-injected,
re-targeted the previous turn's __user message via the ID swap, and left
the stale message as the latest human turn -- so the model re-answered
the previous message and the persisted message stream got scrambled.
Skip flagged reminders without a <current_date> so the scan reaches the
real date SystemMessage. Add a regression test for the 2nd-turn +
memory-enabled scenario.
Fixes#3684
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): make injected reminder date authoritative via metadata
Promote the dynamic-context date out of message content into
additional_kwargs["reminder_date"] on the date SystemMessage.
_last_injected_date reads that key instead of regex-parsing content, with a
SystemMessage-scoped content fallback for checkpoints written before the key
existed.
This removes two structural fragilities raised in review of #3685:
- the boolean dynamic_context_reminder flag no longer has to also mean
"carries a date" -- dateless reminders simply lack reminder_date and are
skipped, so a future reminder type cannot reproduce the shadowing bug.
- the date regex no longer runs on the user-influenceable memory HumanMessage,
so a memory fact containing a literal <current_date> cannot spoof the
injected date (false same-day skip / false midnight crossing).
Add regression tests for memory date-spoofing, structured-date stamping, and
legacy (no-key) detection; update existing date-reminder fixtures to the
production shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
_aexecute collects AI messages from agent.astream(stream_mode="values"),
which re-yields the full state every super-step. The duplicate check rescanned
the append-only ai_messages list on every chunk -- any(m["id"] == message_id) --
so a run with M messages did O(M^2) work, and M reaches max_turns=150 for the
general-purpose / deep-research subagent.
Track an id-keyed set alongside ai_messages: id-bearing messages become O(1)
set lookups, and the id-less full-dict-compare fallback is preserved. Behavior
is unchanged.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(config): coerce null object config sections to their defaults
#3434 made commented-out list sections (models/tools/tool_groups) parse as []
instead of crashing, but the same scenario still crashes for object sections:
commenting out every key under e.g. memory: / summarization: / guardrails:
makes PyYAML parse the value as None, and AppConfig then raises "Input should
be a valid dictionary" for that section — breaking the documented
`cp config.example.yaml config.yaml` first-run flow.
Generalize the handling: a model_validator(mode="before") drops None-valued
sections so each field falls back to its default (list sections -> [], object
sections -> their default config). This subsumes the previous list-only
field_validator and the database special-case. Required sections without a
default (sandbox) still error when null.
Adds test_app_config_coerces_commented_out_object_sections; the existing
list-section regression test still passes.
* docs+test: address review on null-section coercion
- Correct the _drop_null_config_sections docstring: it does not subsume the
database special-case; _apply_database_defaults still owns `database` and
applies concrete defaults beyond null-coercion.
- Strengthen test_app_config_coerces_commented_out_object_sections to assert
each null section falls back to its expected default config type, not just
that it is non-None.
- Add test_app_config_null_required_section_still_errors covering the
required-section (sandbox) "still errors when null" claim.
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
list_thread_messages matched event_type == "ai_message" to find each
run's last AI message, but RunJournal stores AI messages as
"llm.ai.response" and the event store returns that verbatim. No code
writes "ai_message", so the match never hit: feedback was never attached
(every message returned feedback=null) and the grouped-feedback query ran
on every request for nothing.
Match "llm.ai.response", and only run the grouped-feedback query when the
thread actually has an AI message to attach it to. Adds a regression test
for the per-run attachment and the no-AI-message lazy-query path.
Fixes#3650.
P0: DynamicContextMiddleware role isolation — framework data as SystemMessage,
memory as HumanMessage (OWASP LLM01), user input unchanged.
Includes E2E hash stabilization: frontend hide_from_ui filter fix in
suggest_agent, backend _canonical_messages() hide_from_ui skip, fixture
hash updates.
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Add a uv lock --check guard in two places so a stale uv.lock cannot be
committed unnoticed (as happened with the groundroute extra):
- pre-commit: a local uv-lock-check hook, scoped to the backend
pyproject.toml files and uv.lock.
- CI: a "Check uv.lock is in sync" step in the lint-backend job, run
before uv sync so the install step cannot mask a stale lock by
regenerating it.
The deerflow-harness pyproject.toml declares a groundroute optional
extra (added in #3675), but uv.lock provides-extras was left out of
sync. Regenerate the lock so the declared extra is recorded.
The login page used hardcoded English strings. Add a `login` section to
the i18n Translations interface and both locale files, then wire the
login page to `useI18n()` so all titles, labels, placeholders, buttons,
SSO hints, and error messages resolve from the active locale.
init_engine runs on the FastAPI lifespan event loop, but created the SQLite
data directory with a synchronous os.makedirs (a stat + mkdir syscall),
blocking startup. Dispatch it via asyncio.to_thread, mirroring the #1912 fix
for the checkpointer's ensure_sqlite_parent_dir.
Adds a Blockbuster-gated regression test in tests/blocking_io/ that drives the
real init_engine path with a not-yet-existing sqlite_dir; it trips
BlockingError if the makedirs regresses onto the event loop.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
* feat(groundroute): add GroundRoute community web_search + web_fetch tools
GroundRoute is a meta search layer over six engines (Serper, Brave, Exa,
Tavily, Firecrawl, Perplexity) with price-based routing and failover. This
adds a self-contained community engine module (httpx only, no new required
deps) mirroring community/brave + community/tavily:
- web_search: POST /v1/search, normalize to {title,url,snippet,source_engine}.
- web_fetch: fetch a URL via mode=page.
- unit tests covering normalization, auth, clamping, and graceful errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(groundroute): register GroundRoute search + fetch in wizard and config
Add GroundRoute to the setup wizard provider lists (SEARCH_PROVIDERS +
WEB_FETCH_PROVIDERS) and as commented web_search + web_fetch examples in
config.example.yaml, mirroring tavily/serper/brave so SEARCH_API can select it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(groundroute): apply repo ruff format (line-length 240)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(groundroute): add GroundRoute to tools docs and config reference
Adds GroundRoute as a web_search and web_fetch option in the en + zh
tools.mdx pages (new tab alongside Tavily/Brave/Exa/etc.) and documents
GROUNDROUTE_API_KEY in CONFIGURATION.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(groundroute): define empty groundroute extra for clean install
The docs install line 'uv add deerflow-harness[groundroute]' (mirroring the
tavily/exa/firecrawl pattern) referenced an undefined extra, which uv accepts
but warns about. GroundRoute needs no extra packages (httpx is a core dep), so
declare an empty 'groundroute' extra in deerflow-harness optional-dependencies
so the documented command resolves without a warning.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(groundroute): per-tool api key + honor caller max_results (review)
Address maintainer review on PR #3675:
- _get_api_key(tool_name): web_fetch now reads the web_fetch config block's key
instead of always web_search, so a flow that pairs GroundRoute fetch with a
different search engine authenticates correctly. Mirrors serper/exa/firecrawl.
- web_search honors a caller-supplied max_results (sentinel default None),
falling back to the configured value only when omitted, so the documented
parameter is no longer silently discarded.
- warn-once is now keyed per tool. Tests cover both fixes (web_fetch key,
agent max_results honored).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): reset new chat on client-side navigation
Drive the chat reset effect with Next.js's reactive pathname instead of the render-time window.location-derived flag.
During App Router transitions, window.location may still point to the previous thread until commit, leaving chat state stale until another UI interaction triggers a render. Preserve the stale 'new' param guard so created thread UUIDs are not overwritten.
* test(frontend): add e2e to cover history-only new chat reset
* docs(frontend): clarify new chat pathname synchronization
* feat: persist AI turn duration in backend and UI
* chore: restore uv.lock to match main
* refactor(frontend): use Math.floor instead of Math.ceil for reasoning timer
Follow-up to #3301 / #3344. The Gateway-embedded runtime is the standard
topology — there is no standalone LangGraph service — but two stale
references slipped past the cleanup guard:
- .github/copilot-instructions.md still told contributors that `make dev`
"Starts LangGraph (2024)" and wrote logs/langgraph.log.
- SandboxAuditMiddleware's docstring pointed audit logs at langgraph.log.
Update both to the Gateway-embedded reality (gateway.log) and extend
test_gateway_runtime_cleanup.py to pin the agent-instruction docs so the
standalone references can't reappear.
Stays within the safe scope of #3304: does not touch langgraph_auth.py,
langgraph.json, or the langgraph-api / langgraph-cli / langgraph-runtime-inmem
deps, which the issue gates on maintainer confirmation of Studio / direct
LangGraph Server support.
Dependabot security update #3620 bumped cryptography to 48.0.1 in uv.lock
but left the harness manifest at >=43.0.0, since the dependency lives in
the workspace member packages/harness, not backend/pyproject.toml. The
drift caused 'uv lock' to keep rewriting the recorded specifier. Align the
source constraint with the locked version.
* fix(scripts): avoid lsof hang during make dev cleanup on NFS
`_is_deerflow_pid` and `_report_reclaimed_ports` call `lsof -p <pid>` to
enumerate a process's open files. On hosts whose working tree or home is
on a network filesystem (NFS/autofs), `lsof -p` blocks indefinitely on the
kernel stat calls, so `make dev` / `make stop` hang forever at
"Stopping all services...".
Add `-b` (avoid kernel blocking functions) and `-w` (suppress the
resulting warnings) to both calls. The network-only `lsof -nP -iTCP`
probes are unaffected and already returned quickly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(nginx): set global error_log so local-dev nginx starts as non-root
nginx.local.conf only declared `error_log` inside the `http {}` block.
nginx opens its compiled-in default error log (on Debian/Ubuntu builds,
the absolute /var/log/nginx/error.log) at startup, before it reaches the
http-block directive. When `make dev` launches nginx as a non-root user
that path is not writable, so startup fails with:
[emerg] open() "/var/log/nginx/error.log" failed (13: Permission denied)
Declare a global (main-context) `error_log logs/nginx-error.log warn;`.
Combined with the existing `-p $REPO_ROOT`, logging resolves to the
repo-local logs/ directory and nginx starts without elevated privileges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
should_ignore_name runs once per directory entry during glob/grep tree
walks and looped over ~57 IGNORE_PATTERNS calling fnmatch per pattern.
Precompute the literal names into a frozenset (O(1)) and the few glob
patterns into one combined compiled regex via fnmatch.translate; per name
it's now one normcase + a set lookup + at most one regex match.
os.path.normcase preserves fnmatch's platform case behavior, so results
are identical (covered by an equivalence test against the old loop).
Fixes#3655.
* fix(mcp): migrate local MCP-produced files into sandbox outputs (#3597)
Stdio MCP servers (e.g. Playwright) write files to host paths that the
sandbox/artifact API cannot resolve, since it only serves paths under
/mnt/user-data. Copy local files referenced by ResourceLink results into
the thread's sandbox outputs dir and rewrite their URIs to
/mnt/user-data/outputs/... so they become readable.
Also scope pooled MCP sessions by user_id:thread_id instead of thread_id
alone, matching the per-(user_id, thread_id) filesystem isolation.
* fix(mcp): restrict file migration to trusted source roots (#3597)
Add a source-root allowlist to the MCP file-migration path so a
malicious or buggy MCP server cannot have us copy arbitrary host files
(e.g. /etc/passwd) into a thread's outputs directory, from where the
artifact API would serve them. Files are migrated only when located
under the OS temp dir (Playwright's default), the thread's own
user-data tree, or an operator-configured root via
DEERFLOW_MCP_MIGRATION_SOURCE_ROOTS.
Expand test coverage with allowlist/security cases (path escape refusal,
trusted-root acceptance), URL-encoded file:// paths, converter content
branches (image/embedded/error/structured), and copy/resolve failure
fallbacks.
* fix(mcp): harden local file migration into sandbox outputs
Address robustness and security gaps in the MCP ResourceLink file
migration:
- Set migrated files to 0o644 so a differently-UID sandbox container can
read them, instead of inheriting the source's (possibly 0o600) mode.
- Enforce the 100MB size cap during the copy (chunked, byte-counted)
rather than from a prior stat(), closing the grow-after-stat TOCTOU.
- Create the destination atomically with O_CREAT|O_EXCL to remove the
check-then-create name-collision race.
- Document the shared-$TMPDIR multi-tenant read surface and mitigation.
Add regression tests: symlink escape refusal, explicit $TMPDIR source
migration, 0o644 mode, and the outputs/user-data resolve() OSError
fallback branches.
* fix(mcp): migrate playwright text file outputs
* fix(mcp): translate MCP file outputs to virtual paths instead of copying (#3597)
Pin stdio MCP subprocess cwd and TMPDIR/TMP/TEMP under the thread workspace
so produced files always land in the mounted user-data tree, then rewrite
returned references via deterministic host->virtual path translation. Free
text is best-effort only: a reference is rewritten only when it resolves to
an existing file inside the thread's tree, and bare filenames are matched
against files created/modified by the same tool call. Replaces the previous
copy-into-outputs + regex approach (which missed cases like temp/page-*.yml).
* style(mcp): apply ruff format to mcp path translation tests
* perf(mcp): offload stdio FS work off event loop and gate on transport
Address review on #3600:
- Wrap the workspace dir prep, snapshot diff, and per-token path
resolution in asyncio.to_thread so they no longer block the event
loop (matches the repo's blocking-IO gate convention).
- Gate the cwd/temp pinning and snapshots on stdio transport only;
SSE/HTTP servers skip the filesystem work entirely.
- Skip the post-call snapshot diff when the result has no text content.
* test(mcp): cover stdio transport gating and text-content after-walk skip
Add unit/integration coverage for the new review-driven behavior:
- _prepare_stdio_workspace dir/temp/snapshot bundle
- _result_has_text_content detection (text, embedded text, image, empty)
- non-stdio transport skips cwd/temp pinning and touches no workspace dirs
- post-call snapshot diff is skipped without text content and runs with it
* fix(mcp): address stdio path rewrite review feedback
- Restrict the stdio MCP temp directory to 0700 instead of 0777.
- Preserve operator-provided stdio cwd values while keeping injected cwd values as strings.
- Add debug logging for deterministic path rewrites and bare-filename rewrite decisions.
- Document the stdio cwd/temp pinning, virtual-path translation, and user/thread session scope.
- Cover explicit cwd preservation and temp-dir permissions in session-pool tests.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* perf(sandbox): cache LocalSandbox path-rewrite regexes per instance
LocalSandbox re-sorted path_mappings and re-compiled its path-rewrite
regex on every tool call: _resolve_paths_in_command (every bash),
_resolve_paths_in_content (every write_file), and
_reverse_resolve_paths_in_output (every bash output / read_file). The
inputs are derived solely from self.path_mappings, which is assigned once
in __init__ and never mutated, so the work is identical every call.
Compile the patterns once per sandbox via functools.cached_property and
reuse them; hoist 'import re' to module scope. Behavior is unchanged —
only the per-call sort+escape+compile on the agent's hot path is removed.
Fixes#3647. Adds tests covering caching identity, unchanged rewriting,
path-segment boundary matching, and the empty-mappings pass-through.
* perf(sandbox): also cache resolved local paths and sorted mapping views
Beyond the regex compilation, _find_path_mapping, _is_read_only_path,
_resolve_path_with_mapping and _reverse_resolve_path re-sorted
path_mappings and re-ran Path(local_path).resolve() (a filesystem
syscall) on every call. Since path_mappings is immutable, cache the
resolved local root per mapping and the two sorted views via
cached_property and reuse them. Behavior is unchanged; the reverse-output
pattern builder now reuses the same resolved-path cache.
* fix(frontend): render full content for multi-part AI messages
Gemini streams the first content block as a {type:text} object carrying
the thinking signature, then emits continuation deltas as bare strings.
The content extractors only kept {type:text} objects and dropped the
string parts, truncating each AI message to its first block.
Handle string elements in extractContentFromMessage, extractTextFromMessage,
and textOfMessage so the full message renders.
Fixes#1000
* test(frontend): cover multi-part AI message content extraction
Add a regression test for the #1000 truncation: an AI message whose
content is [{type:text, signature}, "bare string"] (Gemini's shape after
LangChain merge_content). Fails on main, passes with the extractor fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(frontend): cover textOfMessage multi-part content extraction
Add tests for the bare-string continuation case and the null-when-empty
contract, and document why textOfMessage joins flat ("") while the body
extractors join with "\n". Addresses review feedback on #3649.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(gateway): attribute token usage to actual models
Capture per-call model names from LLM response metadata for lead, middleware, and subagent calls.
Persist a per-run token_usage_by_model breakdown and aggregate by that map in both SQL and memory stores, with legacy fallback to the run-level model_name for older rows.
Add regression coverage for by_model totals, caller consistency, active progress snapshots, store parity, and SubagentTokenCollector model propagation.
* fix(gateway): harden by-model token aggregation
Use usage.get("total_tokens", 0) when reducing per-model token usage maps so aggregation tolerates partially written or manually edited JSON blobs without changing behavior for journal-written rows.
* docs(gateway): clarify by-model run count semantics
Document that by_model[*].runs counts the number of runs in which a model appeared, so multi-model runs can increment multiple model buckets.
Base.to_dict() and __repr__() ran sqlalchemy.inspect(type(self)).mapper
reflection on every call, but to_dict() is invoked once per row when
serializing ORM results (e.g. every event in a messages/events page).
The mapped columns are fixed at class-definition time, so cache the
column keys per class with functools.cache and iterate the cached tuple.
Behavior is unchanged.
Fixes#3653.
* feat(suggest_agent): stop frontend from fetching when suggestions disabled
* wait for suggestions config to load before fetching
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(frontend): resolve stale subagent running state after stop
Subtask cards were recreated from task tool calls as in_progress whenever
the thread was loading. If a user stopped a run before the task tool returned
a ToolMessage, the card could remain running after reload, and could flip back
to running when a later user turn started streaming.
Derive pending subtask state from the current assistant turn instead of the
global thread loading flag. A task now stays in progress only while its own
turn is loading or while a matching ToolMessage exists for result parsing;
otherwise it is marked failed.
Add unit coverage for task ToolMessage matching and for the later-turn loading
regression.
* fix(frontend): refresh subagent card on terminal status transition
- notify React via a ref + post-render effect when a subtask flips to
completed/failed, so SubtaskCard updates without relying on an
unrelated MessageList re-render
- tighten the current-turn heuristic to `groupIndex === lastGroupIndex`
(precomputed once) — fixes the rare case where an earlier subagent
group in the same turn was kept in_progress, and drops the per-group
slice+some scan to O(1)
- rename `getPendingSubtaskStatus` → `derivePendingSubtaskStatus`
- narrow `toolCall.id` at MessageList call sites; skip task tool calls
without an id instead of `id!`
- add Playwright e2e covering reload-after-stop showing `Subtask failed`
The SSE stream's `values` snapshots serialize the full state, including the
hide_from_ui human messages that ViewImageMiddleware fills with base64 image
payloads. #3535 stripped those from the REST wait/history/state endpoints via
serialize_channel_values_for_api, but the streaming path (worker publishes
serialize(chunk, mode="values")) still went through serialize_channel_values,
so the same base64 leaked to the frontend over SSE. Route values-mode
serialization through serialize_channel_values_for_api as well. Non-hidden
messages and https image URLs are left untouched.
