mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-10 00:18:36 +00:00
* feat(channels): add GitHub event-driven agents (#3754) Add a webhook-driven GitHub channel with fail-closed webhook routing, deterministic per-agent PR/issue threads, mention-gated trigger fan-out, GitHub App token injection for sandboxed gh/git commands, and backend/AGENTS.md documentation. * fix(llm-middleware): classify bare IndexError as transient Upstream chat providers occasionally return 200 OK with an empty generations list (observed against Volces "coding" on ark.cn-beijing.volces.com). When that happens, langchain_core.language_models.chat_models.ainvoke raises ``IndexError: list index out of range`` at ``llm_result.generations[0][0].message`` and kills the run. Treat a bare IndexError reaching the middleware as a transient upstream-payload glitch and route it through the existing retry/backoff path instead of failing the whole agent run. The retry budget and backoff schedule are unchanged. Adds three regression tests covering the classifier and both the recover-on-retry and exhausted-retries paths. * fix(runtime): ignore stale LLM fallback markers from prior runs When a run on a thread ends with the LLM-error-handling middleware emitting a `deerflow_error_fallback`-marked AIMessage (e.g. after the IndexError empty-generations classification fix lands), that message is persisted to the thread's checkpoint as part of the messages channel. LangGraph replays the full message history in `stream_mode="values"` chunks, so every subsequent run on the same thread re-streams the stale fallback marker — and the worker's chunk scanner faithfully picks it up, flipping `RunStatus.success` to `RunStatus.error` for runs that themselves had no LLM failure at all. Snapshot the set of pre-existing message ids from the pre-run checkpoint and thread it through `_extract_llm_error_fallback_message` / `_try_extract_from_message` as a filter. Markers on history messages are ignored; markers on fresh messages produced during this run still trip the error path. Falls back to an empty set when the checkpointer is absent or the snapshot can't be captured, preserving the prior behavior on first-run / no-state paths. Adds unit tests for the new filter (helper-level and `_collect_pre_existing_message_ids`) plus an integration test exercising the full `run_agent` path with a stale history checkpointer. * fix(channels): make github channel fire-and-forget to avoid httpx.ReadTimeout on long runs GitHub agent runs (clone -> edit -> test -> push -> PR) routinely exceed the langgraph_sdk default 300s read deadline. The manager's runs.wait call kept an HTTP stream open for the entire run lifetime, so the long run blew up with httpx.ReadTimeout and the outer except branch then released the dedupe key and emitted a false 'internal error' outbound. The GitHub channel's outbound send is log-only by design: agents post to the issue/PR via the gh CLI in the sandbox when they choose to comment or create a PR. There is nothing for the manager to ferry back, so the long-poll was pure overhead. This change adds ChannelRunPolicy.fire_and_forget (default False) and sets it True for the github channel. When fire_and_forget is True, _handle_chat dispatches via client.runs.create (short POST, returns once the run is pending) instead of client.runs.wait, and skips the response-extraction + outbound-publish block. ConflictError on a busy thread still trips the standard THREAD_BUSY_MESSAGE path so behavior on the busy case is preserved for any future non-github fire-and-forget channel. Other (non-github) channels are unchanged: their policy defaults fire_and_forget=False and they continue to dispatch via runs.wait. Adds 6 regression tests in tests/test_channels.py::TestGithubFireAndForget: - Default ChannelRunPolicy.fire_and_forget is False. - The github policy registers fire_and_forget=True. - github inbound calls runs.create, not runs.wait, with the right kwargs. - github inbound publishes no outbound on success. - ConflictError from runs.create still emits THREAD_BUSY_MESSAGE. - Non-github channels (slack) still dispatch via runs.wait. * test(lead-agent): accept user_id kwarg in skill-policy test stubs The two GitHub-channel tests added in #3754 stubbed _load_enabled_skills_for_tool_policy with a lambda that only accepted `available_skills` and `app_config`, but the real function (and its call site in agent.py) also passes `user_id`. This raised TypeError on every run, failing backend-unit-tests. Add `user_id=None` to match the three sibling stubs in the same file. * refactor(gateway): disambiguate context-key set names The two frozensets _INTERNAL_ONLY_CONTEXT_KEYS and _CONTEXT_ONLY_KEYS shared a confusable "CONTEXT_ONLY" token in different orders, and the first broke the _CONTEXT_<X>_KEYS pattern of its sibling _CONTEXT_CONFIGURABLE_KEYS. Rename to make the distinct axes explicit: _CONTEXT_INTERNAL_CALLER_KEYS - WHO: internal callers (scheduler) only _CONTEXT_RUNTIME_ONLY_KEYS - WHERE: runtime context only, never configurable Pure rename, no behavior change.
410 lines
15 KiB
Python
410 lines
15 KiB
Python
"""Tests for the GitHub binding registry."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
import yaml
|
|
|
|
from app.gateway.github.registry import (
|
|
_invalidate_cache,
|
|
build_github_agent_registry,
|
|
lookup_agents,
|
|
)
|
|
|
|
|
|
def _write_agent(base: Path, user_id: str, name: str, body: dict) -> Path:
|
|
agent_dir = base / "users" / user_id / "agents" / name
|
|
agent_dir.mkdir(parents=True, exist_ok=True)
|
|
(agent_dir / "config.yaml").write_text(yaml.safe_dump(body), encoding="utf-8")
|
|
return agent_dir
|
|
|
|
|
|
def _write_legacy_agent(base: Path, name: str, body: dict) -> Path:
|
|
"""Write an agent under the pre-user-isolation shared layout at ``{base}/agents/{name}/``."""
|
|
agent_dir = base / "agents" / name
|
|
agent_dir.mkdir(parents=True, exist_ok=True)
|
|
(agent_dir / "config.yaml").write_text(yaml.safe_dump(body), encoding="utf-8")
|
|
return agent_dir
|
|
|
|
|
|
@pytest.fixture()
|
|
def base_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
|
"""Isolate the agents store for this test.
|
|
|
|
Also drops the registry's module-level mtime cache: tmp_path is
|
|
freshly minted per test and the previous test's cache signature
|
|
would otherwise survive into this one and short-circuit the scan.
|
|
"""
|
|
monkeypatch.setenv("DEER_FLOW_HOME", str(tmp_path))
|
|
from deerflow.config import paths as paths_module
|
|
|
|
monkeypatch.setattr(paths_module, "_paths", None)
|
|
_invalidate_cache()
|
|
return tmp_path
|
|
|
|
|
|
def test_registry_empty_when_no_agents(base_dir: Path) -> None:
|
|
registry = build_github_agent_registry()
|
|
assert registry == {}
|
|
|
|
|
|
def test_registry_skips_agents_without_github_block(base_dir: Path) -> None:
|
|
_write_agent(base_dir, "default", "plain", {"name": "plain"})
|
|
registry = build_github_agent_registry()
|
|
assert registry == {}
|
|
|
|
|
|
def test_registry_indexes_by_repo_and_event(base_dir: Path) -> None:
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"coding-llm-gateway",
|
|
{
|
|
"name": "coding-llm-gateway",
|
|
"github": {
|
|
"bindings": [
|
|
{
|
|
"repo": "zhfeng/llm-gateway",
|
|
"triggers": {"pull_request": {"actions": ["opened"]}},
|
|
}
|
|
],
|
|
},
|
|
},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
matched = lookup_agents(registry, "zhfeng/llm-gateway", "pull_request")
|
|
assert len(matched) == 1
|
|
assert matched[0].agent.name == "coding-llm-gateway"
|
|
assert matched[0].user_id == "default"
|
|
assert matched[0].agent.github.installation_id is None # default
|
|
|
|
|
|
def test_registry_does_not_register_events_the_binding_omits(base_dir: Path) -> None:
|
|
# Events are opt-in per binding. A binding with an empty trigger map
|
|
# registers for nothing — the dispatcher will never fan a webhook out
|
|
# to this agent. (Tightened from the old behavior, which auto-included
|
|
# all default-enabled events.)
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"silent",
|
|
{
|
|
"name": "silent",
|
|
"github": {"bindings": [{"repo": "a/b"}]},
|
|
},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
for event in (
|
|
"pull_request",
|
|
"issue_comment",
|
|
"pull_request_review_comment",
|
|
"pull_request_review",
|
|
"issues",
|
|
"ping",
|
|
):
|
|
assert lookup_agents(registry, "a/b", event) == [], event
|
|
|
|
|
|
def test_registry_indexes_only_explicitly_listed_events(base_dir: Path) -> None:
|
|
# An explicit ``pull_request: {}`` opts the agent in for PR events ONLY —
|
|
# not the other default-enabled events.
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"pr-only",
|
|
{
|
|
"name": "pr-only",
|
|
"github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]},
|
|
},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
assert len(lookup_agents(registry, "a/b", "pull_request")) == 1
|
|
assert lookup_agents(registry, "a/b", "issue_comment") == []
|
|
assert lookup_agents(registry, "a/b", "pull_request_review_comment") == []
|
|
|
|
|
|
def test_registry_picks_up_event_only_via_explicit_trigger_override(base_dir: Path) -> None:
|
|
# ``issues`` is disabled by default, but an explicit trigger entry opts in.
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"issue-bot",
|
|
{
|
|
"name": "issue-bot",
|
|
"github": {"bindings": [{"repo": "a/b", "triggers": {"issues": {}}}]},
|
|
},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
matched = lookup_agents(registry, "a/b", "issues")
|
|
assert len(matched) == 1
|
|
assert matched[0].agent.name == "issue-bot"
|
|
|
|
|
|
def test_registry_supports_multiple_agents_on_same_repo_event(base_dir: Path) -> None:
|
|
for n in ("alpha", "beta"):
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
n,
|
|
{
|
|
"name": n,
|
|
"github": {
|
|
"bindings": [
|
|
{"repo": "a/b", "triggers": {"pull_request": {}}},
|
|
],
|
|
},
|
|
},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
matched = lookup_agents(registry, "a/b", "pull_request")
|
|
names = sorted(m.agent.name for m in matched)
|
|
assert names == ["alpha", "beta"]
|
|
|
|
|
|
def test_registry_supports_one_agent_on_multiple_repos(base_dir: Path) -> None:
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"multi",
|
|
{
|
|
"name": "multi",
|
|
"github": {
|
|
"bindings": [
|
|
{"repo": "a/one", "triggers": {"pull_request": {}}},
|
|
{"repo": "a/two", "triggers": {"pull_request": {}}},
|
|
],
|
|
},
|
|
},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
assert len(lookup_agents(registry, "a/one", "pull_request")) == 1
|
|
assert len(lookup_agents(registry, "a/two", "pull_request")) == 1
|
|
assert lookup_agents(registry, "a/three", "pull_request") == []
|
|
|
|
|
|
def test_registry_scans_multiple_users(base_dir: Path) -> None:
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"agent-a",
|
|
{"name": "agent-a", "github": {"bindings": [{"repo": "x/y", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
_write_agent(
|
|
base_dir,
|
|
"alice",
|
|
"agent-b",
|
|
{"name": "agent-b", "github": {"bindings": [{"repo": "x/y", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
matched = lookup_agents(registry, "x/y", "pull_request")
|
|
user_ids = sorted(m.user_id for m in matched)
|
|
assert user_ids == ["alice", "default"]
|
|
|
|
|
|
def test_registry_skips_broken_agent_config(base_dir: Path, caplog: pytest.LogCaptureFixture) -> None:
|
|
# One good, one with malformed YAML.
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"good",
|
|
{"name": "good", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
bad_dir = base_dir / "users" / "default" / "agents" / "broken"
|
|
bad_dir.mkdir(parents=True)
|
|
(bad_dir / "config.yaml").write_text("this: : not : valid\n", encoding="utf-8")
|
|
with caplog.at_level("WARNING", logger="app.gateway.github.registry"):
|
|
registry = build_github_agent_registry()
|
|
assert any("broken" in rec.message for rec in caplog.records)
|
|
matched = lookup_agents(registry, "a/b", "pull_request")
|
|
assert [m.agent.name for m in matched] == ["good"]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# mtime-keyed cache
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_registry_cache_returns_same_object_on_warm_call(base_dir: Path) -> None:
|
|
"""Identical (user_id, agent, mtime) signature → no YAML reparse.
|
|
|
|
The warm path only does iterdir + stat per agent — the dominant
|
|
YAML-parse cost is skipped. We verify by reference identity: the
|
|
cache returns the same dict object across calls so any caller that
|
|
holds a reference sees a coherent snapshot.
|
|
"""
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"alpha",
|
|
{"name": "alpha", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
first = build_github_agent_registry()
|
|
second = build_github_agent_registry()
|
|
assert first is second # same object, no rebuild
|
|
assert lookup_agents(first, "a/b", "pull_request")[0].agent.name == "alpha"
|
|
|
|
|
|
def test_registry_cache_invalidates_on_new_agent(base_dir: Path) -> None:
|
|
"""Adding a new agent on disk invalidates the cache.
|
|
|
|
Operator edits between webhook deliveries must be visible without a
|
|
process restart — the mtime signature changes when a new entry is
|
|
added, so the next call rebuilds.
|
|
"""
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"alpha",
|
|
{"name": "alpha", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
first = build_github_agent_registry()
|
|
assert {m.agent.name for m in lookup_agents(first, "a/b", "pull_request")} == {"alpha"}
|
|
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"beta",
|
|
{"name": "beta", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
second = build_github_agent_registry()
|
|
assert second is not first
|
|
assert {m.agent.name for m in lookup_agents(second, "a/b", "pull_request")} == {"alpha", "beta"}
|
|
|
|
|
|
def test_registry_cache_invalidates_on_config_edit(base_dir: Path) -> None:
|
|
"""Editing config.yaml advances mtime and triggers a rebuild.
|
|
|
|
Pure mtime-bump suffices: an operator who edits the file with the
|
|
same byte content (touch) still gets a fresh parse — which is
|
|
desirable; no harm done.
|
|
"""
|
|
agent_dir = _write_agent(
|
|
base_dir,
|
|
"default",
|
|
"edited",
|
|
{"name": "edited", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
first = build_github_agent_registry()
|
|
assert {m.agent.name for m in lookup_agents(first, "a/b", "pull_request")} == {"edited"}
|
|
|
|
# Rewrite with a different repo binding and bump the mtime well past
|
|
# filesystem granularity so the change is observable.
|
|
import os
|
|
import time
|
|
|
|
(agent_dir / "config.yaml").write_text(
|
|
yaml.safe_dump({"name": "edited", "github": {"bindings": [{"repo": "c/d", "triggers": {"pull_request": {}}}]}}),
|
|
encoding="utf-8",
|
|
)
|
|
future = time.time() + 5
|
|
os.utime(agent_dir / "config.yaml", (future, future))
|
|
|
|
second = build_github_agent_registry()
|
|
assert second is not first
|
|
assert lookup_agents(second, "a/b", "pull_request") == []
|
|
assert {m.agent.name for m in lookup_agents(second, "c/d", "pull_request")} == {"edited"}
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Legacy shared layout ({base_dir}/agents/) for unmigrated installations
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_registry_indexes_legacy_shared_agent(base_dir: Path) -> None:
|
|
"""Legacy ``{base_dir}/agents/{name}/`` still indexes for unmigrated installs.
|
|
|
|
CLAUDE.md commits to the legacy layout as a read-only fallback, and
|
|
``load_agent_config(name)`` resolves it under DEFAULT_USER_ID. The
|
|
GitHub registry must agree, or an unmigrated install with a
|
|
``github:`` block silently fans out to nothing.
|
|
"""
|
|
_write_legacy_agent(
|
|
base_dir,
|
|
"shared-bot",
|
|
{"name": "shared-bot", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
matched = lookup_agents(registry, "a/b", "pull_request")
|
|
assert len(matched) == 1
|
|
assert matched[0].agent.name == "shared-bot"
|
|
# Legacy agents are bucketed under DEFAULT_USER_ID — same as load_agent_config().
|
|
assert matched[0].user_id == "default"
|
|
|
|
|
|
def test_registry_per_user_entry_shadows_legacy_with_same_name(base_dir: Path) -> None:
|
|
"""A ``users/default/agents/{name}/`` entry hides the legacy ``agents/{name}/`` entry.
|
|
|
|
Mirrors ``list_custom_agents``' precedence so migration is a no-op for
|
|
the registry — the legacy row stops appearing the moment the per-user
|
|
copy lands, and no duplicate trigger sets bleed through.
|
|
"""
|
|
# Different trigger set on each so we can tell which one won.
|
|
_write_legacy_agent(
|
|
base_dir,
|
|
"shadowed",
|
|
{"name": "shadowed", "github": {"bindings": [{"repo": "legacy/repo", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"shadowed",
|
|
{"name": "shadowed", "github": {"bindings": [{"repo": "user/repo", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
registry = build_github_agent_registry()
|
|
# Per-user binding wins.
|
|
assert len(lookup_agents(registry, "user/repo", "pull_request")) == 1
|
|
# Legacy binding does not appear.
|
|
assert lookup_agents(registry, "legacy/repo", "pull_request") == []
|
|
|
|
|
|
def test_registry_legacy_cache_invalidates_on_legacy_config_edit(base_dir: Path) -> None:
|
|
"""Editing a legacy ``{base_dir}/agents/{name}/config.yaml`` invalidates the cache."""
|
|
agent_dir = _write_legacy_agent(
|
|
base_dir,
|
|
"legacy-edited",
|
|
{"name": "legacy-edited", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
first = build_github_agent_registry()
|
|
assert {m.agent.name for m in lookup_agents(first, "a/b", "pull_request")} == {"legacy-edited"}
|
|
|
|
import os
|
|
import time
|
|
|
|
(agent_dir / "config.yaml").write_text(
|
|
yaml.safe_dump({"name": "legacy-edited", "github": {"bindings": [{"repo": "c/d", "triggers": {"pull_request": {}}}]}}),
|
|
encoding="utf-8",
|
|
)
|
|
future = time.time() + 5
|
|
os.utime(agent_dir / "config.yaml", (future, future))
|
|
|
|
second = build_github_agent_registry()
|
|
assert second is not first
|
|
assert lookup_agents(second, "a/b", "pull_request") == []
|
|
assert {m.agent.name for m in lookup_agents(second, "c/d", "pull_request")} == {"legacy-edited"}
|
|
|
|
|
|
def test_registry_cache_invalidates_on_agent_deletion(base_dir: Path) -> None:
|
|
"""Removing an agent dir drops it from the next registry."""
|
|
_write_agent(
|
|
base_dir,
|
|
"default",
|
|
"alpha",
|
|
{"name": "alpha", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
agent_dir = _write_agent(
|
|
base_dir,
|
|
"default",
|
|
"doomed",
|
|
{"name": "doomed", "github": {"bindings": [{"repo": "a/b", "triggers": {"pull_request": {}}}]}},
|
|
)
|
|
first = build_github_agent_registry()
|
|
assert {m.agent.name for m in lookup_agents(first, "a/b", "pull_request")} == {"alpha", "doomed"}
|
|
|
|
import shutil
|
|
|
|
shutil.rmtree(agent_dir)
|
|
second = build_github_agent_registry()
|
|
assert second is not first
|
|
assert {m.agent.name for m in lookup_agents(second, "a/b", "pull_request")} == {"alpha"}
|