mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-09 16:08:31 +00:00
fix(title): avoid default LLM call before stream end (#3885)
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run
* fix(title): avoid default LLM call before stream end * fix(title): keep default fallback local * fix(title): harden fallback and replay e2e * docs(title): align fallback title behavior
This commit is contained in:
parent
fe82552023
commit
2453718acd
16 changed files with 202 additions and 102 deletions
|
|
@ -581,7 +581,7 @@ Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only de
|
|||
- `tool_groups[]` - Logical groupings for tools
|
||||
- `sandbox.use` - Sandbox provider class path
|
||||
- `skills.path` / `skills.container_path` - Host and container paths to skills directory
|
||||
- `title` - Auto-title generation (enabled, max_words, max_chars, prompt_template)
|
||||
- `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path)
|
||||
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
|
||||
- `subagents.enabled` - Master switch for subagent delegation
|
||||
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@
|
|||
使用 `TitleMiddleware` 在 `after_model` 钩子中:
|
||||
1. 检测是否是首次对话(1个用户消息 + 1个助手回复)
|
||||
2. 检查 state 是否已有 title
|
||||
3. 调用 LLM 生成简洁的标题(默认最多6个词)
|
||||
3. 默认从首条用户消息生成本地 fallback 标题,避免在流式回复结束前额外等待一次 LLM 调用;显式配置 `model_name` 时才调用 LLM 生成标题(默认最多6个词)
|
||||
4. 将 title 存储到 `ThreadState` 中(会被 checkpointer 持久化)
|
||||
|
||||
TitleMiddleware 会先把 LangChain message content 里的结构化 block/list 内容归一化为纯文本,再拼到 title prompt 里,避免把 Python/JSON 的原始 repr 泄漏到标题生成模型。
|
||||
|
|
@ -67,7 +67,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 6
|
||||
max_chars: 60
|
||||
model_name: null # 使用默认模型
|
||||
model_name: null # null = 快速本地 fallback;填模型名才启用 LLM 标题
|
||||
```
|
||||
|
||||
或在代码中配置:
|
||||
|
|
@ -149,17 +149,21 @@ sequenceDiagram
|
|||
participant Client
|
||||
participant LangGraph
|
||||
participant TitleMiddleware
|
||||
participant LLM
|
||||
participant TitleModel as Title model (optional)
|
||||
participant Checkpointer
|
||||
|
||||
User->>Client: 发送首条消息
|
||||
Client->>LangGraph: POST /threads/{id}/runs
|
||||
LangGraph->>Agent: 处理消息
|
||||
Agent-->>LangGraph: 返回回复
|
||||
LangGraph->>TitleMiddleware: after_agent()
|
||||
LangGraph->>TitleMiddleware: after_model()/aafter_model()
|
||||
TitleMiddleware->>TitleMiddleware: 检查是否需要生成 title
|
||||
TitleMiddleware->>LLM: 生成 title
|
||||
LLM-->>TitleMiddleware: 返回 title
|
||||
alt title.model_name 为空(默认)
|
||||
TitleMiddleware->>TitleMiddleware: 从首条用户消息生成本地 fallback title
|
||||
else 显式配置 title.model_name
|
||||
TitleMiddleware->>TitleModel: 生成 LLM title
|
||||
TitleModel-->>TitleMiddleware: 返回 title
|
||||
end
|
||||
TitleMiddleware->>LangGraph: return {"title": "..."}
|
||||
LangGraph->>Checkpointer: 保存 state (含 title)
|
||||
LangGraph-->>Client: 返回响应
|
||||
|
|
@ -178,20 +182,15 @@ sequenceDiagram
|
|||
## 注意事项
|
||||
|
||||
1. **读取方式不同**:Title 在 `state.values.title` 而非 `thread.metadata.title`
|
||||
2. **性能考虑**:title 生成会增加约 0.5-1 秒延迟,可通过使用更快的模型优化
|
||||
3. **并发安全**:middleware 在 agent 执行后运行,不会阻塞主流程
|
||||
4. **Fallback 策略**:如果 LLM 调用失败,会使用用户消息的前几个词作为 title
|
||||
2. **性能考虑**:默认配置不调用标题模型;只有显式配置 `title.model_name` 时,才会在首轮回复后额外等待一次 LLM title 生成
|
||||
3. **并发安全**:middleware 在 agent 首次完整回复后更新 state,不需要客户端额外请求
|
||||
4. **Fallback 策略**:默认使用用户消息前几个字符作为 title;如果显式启用的 LLM 调用失败,也会回退到该策略
|
||||
|
||||
## 测试
|
||||
|
||||
```python
|
||||
# 测试 title 生成
|
||||
import pytest
|
||||
from deerflow.agents.title_middleware import TitleMiddleware
|
||||
|
||||
def test_title_generation():
|
||||
# TODO: 添加单元测试
|
||||
pass
|
||||
```bash
|
||||
cd backend
|
||||
uv run pytest tests/test_title_middleware_core_logic.py tests/test_title_generation.py
|
||||
```
|
||||
|
||||
## 故障排查
|
||||
|
|
@ -199,8 +198,8 @@ def test_title_generation():
|
|||
### Title 没有生成
|
||||
|
||||
1. 检查配置是否启用:`get_title_config().enabled == True`
|
||||
2. 检查日志:查找 "Generated thread title" 或错误信息
|
||||
3. 确认是首次对话:只有 1 个用户消息和 1 个助手回复时才会触发
|
||||
2. 确认是首次对话:只有 1 个用户消息和 1 个助手回复时才会触发
|
||||
3. 如果显式配置了 `title.model_name`,检查标题模型是否可用;未配置时会走本地 fallback
|
||||
|
||||
### Title 生成但客户端看不到
|
||||
|
||||
|
|
@ -231,16 +230,19 @@ def test_title_generation():
|
|||
```python
|
||||
# TitleMiddleware 核心逻辑
|
||||
@override
|
||||
def after_agent(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | None:
|
||||
"""Generate and set thread title after the first agent response."""
|
||||
if self._should_generate_title(state, runtime):
|
||||
title = self._generate_title(runtime)
|
||||
print(f"Generated thread title: {title}")
|
||||
|
||||
# ✅ 返回 state 更新,会被 checkpointer 自动持久化
|
||||
return {"title": title}
|
||||
|
||||
return None
|
||||
async def aafter_model(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | None:
|
||||
return await self._agenerate_title_result(state)
|
||||
|
||||
async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None:
|
||||
if not self._should_generate_title(state):
|
||||
return None
|
||||
|
||||
config = self._get_title_config()
|
||||
if not config.model_name:
|
||||
return self._generate_title_result(state)
|
||||
|
||||
# 显式配置 title.model_name 时才调用标题模型;失败会回退到本地 title。
|
||||
...
|
||||
```
|
||||
|
||||
## 相关文件
|
||||
|
|
|
|||
|
|
@ -387,7 +387,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 6
|
||||
max_chars: 60
|
||||
model_name: null # Use first model in list
|
||||
model_name: null # null = fast local fallback; set a model name to use LLM title generation
|
||||
```
|
||||
|
||||
### GitHub API Token (Optional for GitHub Deep Research Skill)
|
||||
|
|
|
|||
|
|
@ -16,9 +16,9 @@
|
|||
#### [`packages/harness/deerflow/agents/middlewares/title_middleware.py`](../packages/harness/deerflow/agents/middlewares/title_middleware.py) (新建)
|
||||
- ✅ 创建 `TitleMiddleware` 类
|
||||
- ✅ 实现 `_should_generate_title()` 检查是否需要生成
|
||||
- ✅ 实现 `_generate_title()` 调用 LLM 生成标题
|
||||
- ✅ 实现 `after_agent()` 钩子,在首次对话后自动触发
|
||||
- ✅ 包含 fallback 策略(LLM 失败时使用用户消息前几个词)
|
||||
- ✅ 默认使用本地 fallback 生成标题,避免流式回复结束前等待额外 LLM 调用;显式配置 `model_name` 时可使用 LLM 标题
|
||||
- ✅ 实现 `after_model()` / `aafter_model()` 钩子,在首次对话后自动触发
|
||||
- ✅ 包含 fallback 策略(LLM 未配置或失败时使用用户消息前几个字符)
|
||||
|
||||
#### [`packages/harness/deerflow/config/app_config.py`](../packages/harness/deerflow/config/app_config.py)
|
||||
- ✅ 导入 `load_title_config_from_dict`
|
||||
|
|
@ -37,7 +37,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 6
|
||||
max_chars: 60
|
||||
model_name: null
|
||||
model_name: null # null = 快速本地 fallback;填模型名才启用 LLM 标题
|
||||
```
|
||||
|
||||
### 3. 文档
|
||||
|
|
@ -81,11 +81,13 @@ title:
|
|||
↓
|
||||
Agent 处理并返回回复
|
||||
↓
|
||||
TitleMiddleware.after_agent() 触发
|
||||
TitleMiddleware.after_model()/aafter_model() 触发
|
||||
↓
|
||||
检查:是否首次对话?是否已有 title?
|
||||
↓
|
||||
调用 LLM 生成 title
|
||||
默认从首条用户消息生成本地 fallback title
|
||||
↓
|
||||
如果显式配置 title.model_name,才调用 LLM 生成更精炼的 title
|
||||
↓
|
||||
返回 {"title": "..."} 更新 state
|
||||
↓
|
||||
|
|
@ -113,7 +115,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 8 # 标题最多 8 个词
|
||||
max_chars: 80 # 标题最多 80 个字符
|
||||
model_name: null # 使用默认模型
|
||||
model_name: null # null = 快速本地 fallback;填模型名才启用 LLM 标题
|
||||
```
|
||||
|
||||
3. **配置持久化(可选)**
|
||||
|
|
@ -169,8 +171,8 @@ pytest
|
|||
### Title 没有生成?
|
||||
|
||||
1. 检查配置:`title.enabled = true`
|
||||
2. 查看日志:搜索 "Generated thread title"
|
||||
3. 确认是首次对话(1 个用户消息 + 1 个助手回复)
|
||||
2. 确认是首次对话(1 个用户消息 + 1 个助手回复)
|
||||
3. 如果显式配置了 `title.model_name`,检查标题模型是否可用;未配置时会走本地 fallback
|
||||
|
||||
### Title 生成但看不到?
|
||||
|
||||
|
|
@ -188,22 +190,22 @@ pytest
|
|||
|
||||
## 📊 性能影响
|
||||
|
||||
- **延迟增加**:约 0.5-1 秒(LLM 调用)
|
||||
- **并发安全**:在 `after_agent` 中运行,不阻塞主流程
|
||||
- **默认延迟**:默认 `title.model_name: null` 不会发起额外 LLM 调用,仅从首条用户消息生成本地 fallback title
|
||||
- **显式 LLM 标题延迟**:只有配置 `title.model_name` 时,首轮回复后才会等待一次标题模型调用
|
||||
- **并发安全**:在 `after_model()` / `aafter_model()` 中更新 state,不需要客户端额外请求
|
||||
- **资源消耗**:每个 thread 只生成一次
|
||||
|
||||
### 优化建议
|
||||
|
||||
1. 使用更快的模型(如 `gpt-3.5-turbo`)
|
||||
2. 减少 `max_words` 和 `max_chars`
|
||||
3. 调整 prompt 使其更简洁
|
||||
1. 默认保持 `model_name: null`,避免流式回复结束前的额外 LLM 等待
|
||||
2. 如需更精炼标题,再显式配置较快的标题模型
|
||||
3. 减少 `max_words` 和 `max_chars`,并让 prompt 保持简洁
|
||||
|
||||
---
|
||||
|
||||
## 🚀 下一步
|
||||
|
||||
- [ ] 添加集成测试(需要 mock LangGraph Runtime)
|
||||
- [ ] 支持自定义 prompt template
|
||||
- [ ] 补充 prompt template 的集成测试
|
||||
- [ ] 支持多语言 title 生成
|
||||
- [ ] 添加 title 重新生成功能
|
||||
- [ ] 监控 title 生成成功率和延迟
|
||||
|
|
|
|||
|
|
@ -67,6 +67,11 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
|
|||
def _is_user_message_for_title(message: object) -> bool:
|
||||
return getattr(message, "type", None) == "human" and not is_dynamic_context_reminder(message)
|
||||
|
||||
def _get_title_user_message(self, state: TitleMiddlewareState) -> str:
|
||||
messages = state.get("messages", [])
|
||||
user_msg_content = next((m.content for m in messages if self._is_user_message_for_title(m)), "")
|
||||
return self._normalize_content(user_msg_content)
|
||||
|
||||
def _should_generate_title(self, state: TitleMiddlewareState) -> bool:
|
||||
"""Check if we should generate a title for this thread."""
|
||||
config = self._get_title_config()
|
||||
|
|
@ -97,10 +102,9 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
|
|||
config = self._get_title_config()
|
||||
messages = state.get("messages", [])
|
||||
|
||||
user_msg_content = next((m.content for m in messages if self._is_user_message_for_title(m)), "")
|
||||
assistant_msg_content = next((m.content for m in messages if m.type == "ai"), "")
|
||||
|
||||
user_msg = self._normalize_content(user_msg_content)
|
||||
user_msg = self._get_title_user_message(state)
|
||||
assistant_msg = self._strip_think_tags(self._normalize_content(assistant_msg_content))
|
||||
|
||||
prompt = config.prompt_template.format(
|
||||
|
|
@ -153,18 +157,23 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
|
|||
if not self._should_generate_title(state):
|
||||
return None
|
||||
|
||||
_, user_msg = self._build_title_prompt(state)
|
||||
user_msg = self._get_title_user_message(state)
|
||||
return {"title": self._fallback_title(user_msg)}
|
||||
|
||||
async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None:
|
||||
"""Generate a title asynchronously and fall back locally on failure."""
|
||||
"""Generate a configured LLM title asynchronously and fall back locally."""
|
||||
if not self._should_generate_title(state):
|
||||
return None
|
||||
|
||||
config = self._get_title_config()
|
||||
prompt, user_msg = self._build_title_prompt(state)
|
||||
if not config.model_name:
|
||||
user_msg = self._get_title_user_message(state)
|
||||
return {"title": self._fallback_title(user_msg)}
|
||||
|
||||
user_msg = self._get_title_user_message(state)
|
||||
|
||||
try:
|
||||
prompt, user_msg = self._build_title_prompt(state)
|
||||
# attach_tracing=False because ``_get_runnable_config()`` inherits
|
||||
# the graph-level RunnableConfig (set in ``_make_lead_agent``) whose
|
||||
# callbacks already carry tracing handlers; binding them again at
|
||||
|
|
@ -172,10 +181,7 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
|
|||
model_kwargs = {"thinking_enabled": False, "attach_tracing": False}
|
||||
if self._app_config is not None:
|
||||
model_kwargs["app_config"] = self._app_config
|
||||
if config.model_name:
|
||||
model = create_chat_model(name=config.model_name, **model_kwargs)
|
||||
else:
|
||||
model = create_chat_model(**model_kwargs)
|
||||
model = create_chat_model(name=config.model_name, **model_kwargs)
|
||||
response = await model.ainvoke(prompt, config=self._get_runnable_config())
|
||||
title = self._parse_title(response.content)
|
||||
if title:
|
||||
|
|
|
|||
|
|
@ -24,11 +24,11 @@ class TitleConfig(BaseModel):
|
|||
)
|
||||
model_name: str | None = Field(
|
||||
default=None,
|
||||
description="Model name to use for title generation (None = use default model)",
|
||||
description="Model name to use for LLM title generation (None = use local fallback title)",
|
||||
)
|
||||
prompt_template: str = Field(
|
||||
default=("Generate a concise title (max {max_words} words) for this conversation.\nUser: {user_msg}\nAssistant: {assistant_msg}\n\nReturn ONLY the title, no quotes, no explanation."),
|
||||
description="Prompt template for title generation",
|
||||
description="Prompt template for LLM title generation when model_name is set",
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -82,7 +82,8 @@ tools:
|
|||
use: deerflow.sandbox.tools:write_file_tool
|
||||
# Memory + summarization make background / debounced model calls whose timing is
|
||||
# non-deterministic; disable them so record and replay see the same model-call
|
||||
# set. (Title stays — it is an in-graph, deterministic call we record.)
|
||||
# set. Title stays enabled, but the default title.model_name: null path is a
|
||||
# local state update rather than a recorded model call.
|
||||
memory:
|
||||
enabled: false
|
||||
injection_enabled: false
|
||||
|
|
|
|||
|
|
@ -94,7 +94,7 @@ class TestTitleMiddlewareCoreLogic:
|
|||
assert middleware._should_generate_title(state) is False
|
||||
|
||||
def test_generate_title_uses_async_model_and_respects_max_chars(self, monkeypatch):
|
||||
_set_test_title_config(max_chars=12, model_name=None)
|
||||
_set_test_title_config(max_chars=12, model_name="title-model")
|
||||
middleware = TitleMiddleware()
|
||||
model = MagicMock()
|
||||
model.ainvoke = AsyncMock(return_value=AIMessage(content="短标题"))
|
||||
|
|
@ -110,7 +110,7 @@ class TestTitleMiddlewareCoreLogic:
|
|||
title = result["title"]
|
||||
|
||||
assert title == "短标题"
|
||||
title_middleware_module.create_chat_model.assert_called_once_with(thinking_enabled=False, attach_tracing=False)
|
||||
title_middleware_module.create_chat_model.assert_called_once_with(name="title-model", thinking_enabled=False, attach_tracing=False)
|
||||
model.ainvoke.assert_awaited_once()
|
||||
assert model.ainvoke.await_args.kwargs["config"] == {
|
||||
"run_name": "title_agent",
|
||||
|
|
@ -160,7 +160,7 @@ class TestTitleMiddlewareCoreLogic:
|
|||
)
|
||||
|
||||
def test_generate_title_normalizes_structured_message_content(self, monkeypatch):
|
||||
_set_test_title_config(max_chars=20)
|
||||
_set_test_title_config(max_chars=20, model_name="title-model")
|
||||
middleware = TitleMiddleware()
|
||||
model = MagicMock()
|
||||
model.ainvoke = AsyncMock(return_value=AIMessage(content="请帮我总结这段代码"))
|
||||
|
|
@ -179,7 +179,7 @@ class TestTitleMiddlewareCoreLogic:
|
|||
assert title == "请帮我总结这段代码"
|
||||
|
||||
def test_generate_title_fallback_for_long_message(self, monkeypatch):
|
||||
_set_test_title_config(max_chars=20)
|
||||
_set_test_title_config(max_chars=20, model_name="title-model")
|
||||
middleware = TitleMiddleware()
|
||||
model = MagicMock()
|
||||
model.ainvoke = AsyncMock(side_effect=RuntimeError("model unavailable"))
|
||||
|
|
@ -199,6 +199,7 @@ class TestTitleMiddlewareCoreLogic:
|
|||
assert title.startswith("这是一个非常长的问题描述")
|
||||
|
||||
def test_aafter_model_delegates_to_async_helper(self, monkeypatch):
|
||||
_set_test_title_config(model_name="title-model")
|
||||
middleware = TitleMiddleware()
|
||||
|
||||
monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value={"title": "异步标题"}))
|
||||
|
|
@ -208,6 +209,78 @@ class TestTitleMiddlewareCoreLogic:
|
|||
monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value=None))
|
||||
assert asyncio.run(middleware.aafter_model({"messages": []}, runtime=MagicMock())) is None
|
||||
|
||||
def test_aafter_model_uses_local_fallback_when_no_title_model_is_configured(self, monkeypatch):
|
||||
"""Default async path must not block stream completion on a second LLM call."""
|
||||
_set_test_title_config(max_chars=20, model_name=None)
|
||||
middleware = TitleMiddleware()
|
||||
create_chat_model = MagicMock()
|
||||
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
|
||||
|
||||
state = {
|
||||
"messages": [
|
||||
HumanMessage(content="请帮我写测试"),
|
||||
AIMessage(content="好的"),
|
||||
]
|
||||
}
|
||||
result = asyncio.run(middleware.aafter_model(state, runtime=MagicMock()))
|
||||
|
||||
assert result == {"title": "请帮我写测试"}
|
||||
create_chat_model.assert_not_called()
|
||||
|
||||
def test_async_generate_title_result_uses_local_fallback_without_model_name(self, monkeypatch):
|
||||
"""The default async helper path avoids the hidden title-model LLM call."""
|
||||
_set_test_title_config(max_chars=20, model_name=None)
|
||||
middleware = TitleMiddleware()
|
||||
create_chat_model = MagicMock()
|
||||
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
|
||||
|
||||
state = {
|
||||
"messages": [
|
||||
HumanMessage(content="流式回答结束后不要再等待标题模型"),
|
||||
AIMessage(content="好的"),
|
||||
]
|
||||
}
|
||||
result = asyncio.run(middleware._agenerate_title_result(state))
|
||||
|
||||
assert result == {"title": "流式回答结束后不要再等待标题模型"}
|
||||
create_chat_model.assert_not_called()
|
||||
|
||||
def test_async_local_fallback_does_not_format_unused_prompt_template(self, monkeypatch):
|
||||
"""Local fallback should not depend on the LLM prompt template."""
|
||||
_set_test_title_config(max_chars=20, model_name=None, prompt_template="{missing_placeholder}")
|
||||
middleware = TitleMiddleware()
|
||||
create_chat_model = MagicMock()
|
||||
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
|
||||
|
||||
state = {
|
||||
"messages": [
|
||||
HumanMessage(content="默认标题路径不应读取模型 prompt"),
|
||||
AIMessage(content="好的"),
|
||||
]
|
||||
}
|
||||
result = asyncio.run(middleware._agenerate_title_result(state))
|
||||
|
||||
assert result == {"title": "默认标题路径不应读取模型 prompt"}
|
||||
create_chat_model.assert_not_called()
|
||||
|
||||
def test_async_title_model_falls_back_when_prompt_template_is_invalid(self, monkeypatch):
|
||||
"""Opt-in LLM title generation still degrades locally on template errors."""
|
||||
_set_test_title_config(max_chars=20, model_name="title-model", prompt_template="{usr_msg}")
|
||||
middleware = TitleMiddleware()
|
||||
create_chat_model = MagicMock()
|
||||
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
|
||||
|
||||
state = {
|
||||
"messages": [
|
||||
HumanMessage(content="请帮我写测试"),
|
||||
AIMessage(content="好的"),
|
||||
]
|
||||
}
|
||||
result = asyncio.run(middleware._agenerate_title_result(state))
|
||||
|
||||
assert result == {"title": "请帮我写测试"}
|
||||
create_chat_model.assert_not_called()
|
||||
|
||||
def test_after_model_sync_delegates_to_sync_helper(self, monkeypatch):
|
||||
middleware = TitleMiddleware()
|
||||
|
||||
|
|
@ -296,7 +369,7 @@ class TestTitleMiddlewareCoreLogic:
|
|||
|
||||
def test_generate_title_async_strips_think_tags_in_response(self, monkeypatch):
|
||||
"""Async title generation strips <think> blocks from the model response."""
|
||||
_set_test_title_config(max_chars=50)
|
||||
_set_test_title_config(max_chars=50, model_name="title-model")
|
||||
middleware = TitleMiddleware()
|
||||
model = MagicMock()
|
||||
model.ainvoke = AsyncMock(return_value=AIMessage(content="<think>用户想研究贵阳。</think>贵阳发展研究"))
|
||||
|
|
|
|||
|
|
@ -1114,7 +1114,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 6
|
||||
max_chars: 60
|
||||
model_name: null # Use default model (first model in models list)
|
||||
model_name: null # null = fast local fallback; set a model name to use LLM title generation
|
||||
|
||||
# ============================================================================
|
||||
# Summarization Configuration
|
||||
|
|
|
|||
|
|
@ -1,15 +1,21 @@
|
|||
import { defineConfig, devices } from "@playwright/test";
|
||||
|
||||
const frontendPort = process.env.E2E_FRONTEND_PORT ?? "3000";
|
||||
const gatewayPort = process.env.E2E_GATEWAY_PORT ?? "8011";
|
||||
const frontendUrl = `http://localhost:${frontendPort}`;
|
||||
const gatewayUrl = `http://localhost:${gatewayPort}`;
|
||||
const gatewayInternalUrl = `http://127.0.0.1:${gatewayPort}`;
|
||||
|
||||
/**
|
||||
* Layer 2 of the record/replay e2e: the REAL Next.js frontend rendering data
|
||||
* from a REAL gateway whose LLM is the deterministic `ReplayChatModel` (no API
|
||||
* key). This is separate from `playwright.config.ts` (which mocks the backend)
|
||||
* so the mock-based suite is untouched.
|
||||
*
|
||||
* Two webServers are started: the replay gateway (:8011) and the frontend
|
||||
* (:3000, pointed at the gateway). Auth-disabled mode is enabled on both
|
||||
* servers so the no-cookie e2e contract is covered; specs that need session
|
||||
* cookies still register a throwaway test account at runtime.
|
||||
* Two webServers are started: the replay gateway and the frontend pointed at
|
||||
* it. Auth-disabled mode is enabled on both servers so the no-cookie e2e
|
||||
* contract is covered; specs that need session cookies still register a
|
||||
* throwaway test account at runtime.
|
||||
*/
|
||||
export default defineConfig({
|
||||
testDir: "./tests/e2e-real-backend",
|
||||
|
|
@ -21,7 +27,7 @@ export default defineConfig({
|
|||
timeout: 90_000,
|
||||
|
||||
use: {
|
||||
baseURL: "http://localhost:3000",
|
||||
baseURL: frontendUrl,
|
||||
trace: "on-first-retry",
|
||||
},
|
||||
|
||||
|
|
@ -29,9 +35,9 @@ export default defineConfig({
|
|||
|
||||
webServer: [
|
||||
{
|
||||
command: "uv run python scripts/run_replay_gateway.py --port 8011",
|
||||
command: `uv run python scripts/run_replay_gateway.py --port ${gatewayPort} --cors ${frontendUrl}`,
|
||||
cwd: "../backend",
|
||||
url: "http://localhost:8011/health",
|
||||
url: `${gatewayUrl}/health`,
|
||||
reuseExistingServer: !process.env.CI,
|
||||
timeout: 180_000,
|
||||
stdout: "pipe",
|
||||
|
|
@ -46,10 +52,11 @@ export default defineConfig({
|
|||
},
|
||||
{
|
||||
command: "pnpm build && pnpm start",
|
||||
url: "http://localhost:3000",
|
||||
url: frontendUrl,
|
||||
reuseExistingServer: !process.env.CI,
|
||||
timeout: 240_000,
|
||||
env: {
|
||||
PORT: frontendPort,
|
||||
SKIP_ENV_VALIDATION: "1",
|
||||
DEER_FLOW_AUTH_DISABLED: "1",
|
||||
BETTER_AUTH_SECRET: "local-dev-secret",
|
||||
|
|
@ -57,7 +64,7 @@ export default defineConfig({
|
|||
// next.config rewrites (same-origin proxy) instead of talking to the
|
||||
// gateway cross-origin — cross-origin fetches drop the auth cookies.
|
||||
// Just point that proxy at the replay gateway.
|
||||
DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: "http://127.0.0.1:8011",
|
||||
DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: gatewayInternalUrl,
|
||||
},
|
||||
},
|
||||
],
|
||||
|
|
|
|||
|
|
@ -82,7 +82,7 @@ Limits the number of parallel subagent task calls the agent can make in a single
|
|||
|
||||
### TitleMiddleware
|
||||
|
||||
Automatically generates a title for the thread after the first exchange. The title is derived from the user's first message and the agent's response.
|
||||
Automatically generates a title for the thread after the first exchange. By default, the title is a fast local fallback derived from the user's first message. Set `title.model_name` only when you want the optional LLM title path.
|
||||
|
||||
**Configuration**: `title:` section in `config.yaml`.
|
||||
|
||||
|
|
@ -91,7 +91,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 6
|
||||
max_chars: 60
|
||||
model_name: null # use default model
|
||||
model_name: null # local fallback; set a model name to use LLM title generation
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -82,7 +82,7 @@ memory:
|
|||
|
||||
### TitleMiddleware
|
||||
|
||||
在第一次交互后自动为线程生成标题。标题从用户的第一条消息和 Agent 的响应中派生。
|
||||
在第一次交互后自动为线程生成标题。默认从用户的第一条消息生成快速本地 fallback 标题;只有设置 `title.model_name` 时,才启用可选的 LLM 标题路径。
|
||||
|
||||
**配置**:`config.yaml` 中的 `title:` 部分。
|
||||
|
||||
|
|
@ -91,7 +91,7 @@ title:
|
|||
enabled: true
|
||||
max_words: 6
|
||||
max_chars: 60
|
||||
model_name: null # 使用默认模型
|
||||
model_name: null # 本地 fallback;填模型名才启用 LLM 标题
|
||||
```
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@ import { expect, test } from "@playwright/test";
|
|||
|
||||
import { AUTH_DISABLED_USER } from "../../src/core/auth/auth-disabled-user";
|
||||
|
||||
const APP = "http://localhost:3000";
|
||||
const APP =
|
||||
process.env.E2E_APP_URL ??
|
||||
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
|
||||
|
||||
test.describe("auth-disabled contract (real backend)", () => {
|
||||
test("gateway /auth/me returns the frontend synthetic user without a cookie", async ({
|
||||
|
|
|
|||
|
|
@ -20,7 +20,9 @@ import { expect, test } from "@playwright/test";
|
|||
* No model, no recording, no API key — the runs are seeded via a test-only
|
||||
* endpoint mounted only on the replay gateway.
|
||||
*/
|
||||
const APP = "http://localhost:3000";
|
||||
const APP =
|
||||
process.env.E2E_APP_URL ??
|
||||
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
|
||||
|
||||
// Distinctive markers so getByText can't collide with UI chrome.
|
||||
const ALPHA = "ALPHA-FIRST-QUESTION-7f3a2c";
|
||||
|
|
|
|||
|
|
@ -11,13 +11,15 @@ const here = dirname(fileURLToPath(import.meta.url));
|
|||
* API key) and assert the browser renders the backend's data correctly.
|
||||
*
|
||||
* The prompt is read from the same fixture the gateway replays, so the input
|
||||
* hash matches and the recorded turns (write_file -> auto-title -> read_file ->
|
||||
* final answer) reproduce deterministically.
|
||||
* hash matches and the recorded model turns reproduce deterministically. The
|
||||
* default auto-title is local fallback state, not a replayed model turn.
|
||||
*/
|
||||
// Register through the frontend origin (same-origin proxy) so the auth cookies
|
||||
// are stored for and sent to localhost:3000 — the gateway is reached via the
|
||||
// are stored for and sent to the browser origin — the gateway is reached via the
|
||||
// next.config rewrite, never cross-origin from the browser.
|
||||
const APP = "http://localhost:3000";
|
||||
const APP =
|
||||
process.env.E2E_APP_URL ??
|
||||
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
|
||||
const fixture = JSON.parse(
|
||||
readFileSync(
|
||||
join(
|
||||
|
|
@ -32,10 +34,17 @@ const fixture = JSON.parse(
|
|||
};
|
||||
|
||||
const PROMPT = fixture.prompt;
|
||||
// Derive the assertions from the fixture so a re-record auto-updates them. Both
|
||||
// are model-generated strings absent from the user prompt, so a pass proves the
|
||||
// replay drove the render (not a prompt echo): the first plain-text turn is the
|
||||
// in-graph auto-title; the JSON-array turn is the follow-up suggestions.
|
||||
const FALLBACK_TITLE_MAX_CHARS = 50;
|
||||
|
||||
function fallbackTitle(userMsg: string): string {
|
||||
if (!userMsg) return "New Conversation";
|
||||
if (userMsg.length <= FALLBACK_TITLE_MAX_CHARS) return userMsg;
|
||||
return `${userMsg.slice(0, FALLBACK_TITLE_MAX_CHARS).trimEnd()}...`;
|
||||
}
|
||||
|
||||
// Suggestions still come from the recorded model fixture. The default title no
|
||||
// longer does: TitleMiddleware uses a local fallback when title.model_name is
|
||||
// unset, so derive that expected title from the prompt.
|
||||
const textTurns = fixture.turns
|
||||
.map((t) => t.output?.data?.content)
|
||||
.filter((c): c is string => typeof c === "string" && c.trim().length > 0);
|
||||
|
|
@ -52,7 +61,7 @@ const EXPECTED_SUGGESTION = ((): string => {
|
|||
return "";
|
||||
}
|
||||
})();
|
||||
const EXPECTED_TITLE = textTurns.find((c) => !c.trim().startsWith("[")) ?? "";
|
||||
const EXPECTED_TITLE = fallbackTitle(PROMPT);
|
||||
|
||||
test.describe("real backend render (replay, no API key)", () => {
|
||||
test.beforeEach(async ({ context }) => {
|
||||
|
|
@ -66,7 +75,7 @@ test.describe("real backend render (replay, no API key)", () => {
|
|||
expect(resp.status(), await resp.text()).toBe(201);
|
||||
});
|
||||
|
||||
test("renders the replayed auto-title + suggestions from a real backend", async ({
|
||||
test("renders the local auto-title + replayed suggestions from a real backend", async ({
|
||||
page,
|
||||
}) => {
|
||||
// ultra mode so the context the frontend sends (is_plan_mode + subagent_enabled)
|
||||
|
|
@ -85,17 +94,13 @@ test.describe("real backend render (replay, no API key)", () => {
|
|||
await textarea.fill(PROMPT);
|
||||
await textarea.press("Enter");
|
||||
|
||||
// Replay-only DOM assertions (derived from the fixture): both are
|
||||
// model-generated strings absent from the user prompt, so they render only if
|
||||
// the recorded turns replayed AND the real frontend rendered them — the
|
||||
// in-graph auto-title and the post-answer follow-up suggestion. Together they
|
||||
// prove the whole pipeline (replay backend -> real frontend render). The
|
||||
// record spec waits for the /suggestions response, so a re-recorded fixture
|
||||
// always captures the suggestion turn — a missing one is a broken recording
|
||||
// and must fail loud here, not pass silently.
|
||||
// The title is the default local fallback, while the suggestion is a
|
||||
// replayed model output absent from the prompt. Together they prove the
|
||||
// backend state update and the replayed post-answer model call both render
|
||||
// through the real frontend.
|
||||
expect(
|
||||
EXPECTED_TITLE,
|
||||
"fixture should contain an auto-title turn",
|
||||
"default local fallback title should be derived from the prompt",
|
||||
).not.toBe("");
|
||||
expect(
|
||||
EXPECTED_SUGGESTION,
|
||||
|
|
|
|||
|
|
@ -6,9 +6,9 @@ import { expect, test } from "@playwright/test";
|
|||
* RECORD driver (Plan A): drive the real frontend through the write/read-file
|
||||
* scenario against the real-model gateway. The gateway captures every model
|
||||
* call to DEERFLOW_RECORD_OUT; this just needs to drive the flow and wait until
|
||||
* the captures stop arriving (main turns + in-graph title + follow-up
|
||||
* suggestions all fired). It asserts nothing about content — it produces the
|
||||
* fixture, it doesn't verify it.
|
||||
* the captures stop arriving (main turns + follow-up suggestions all fired;
|
||||
* the default auto-title is local state). It asserts nothing about content —
|
||||
* it produces the fixture, it doesn't verify it.
|
||||
*/
|
||||
const APP = "http://localhost:3000";
|
||||
const SCENARIO = "write_read_file";
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue