fix(title): avoid default LLM call before stream end (#3885)
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run

* fix(title): avoid default LLM call before stream end

* fix(title): keep default fallback local

* fix(title): harden fallback and replay e2e

* docs(title): align fallback title behavior
This commit is contained in:
xiawiie 2026-06-30 18:58:02 +08:00 committed by GitHub
parent fe82552023
commit 2453718acd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 202 additions and 102 deletions

View file

@ -581,7 +581,7 @@ Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only de
- `tool_groups[]` - Logical groupings for tools - `tool_groups[]` - Logical groupings for tools
- `sandbox.use` - Sandbox provider class path - `sandbox.use` - Sandbox provider class path
- `skills.path` / `skills.container_path` - Host and container paths to skills directory - `skills.path` / `skills.container_path` - Host and container paths to skills directory
- `title` - Auto-title generation (enabled, max_words, max_chars, prompt_template) - `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path)
- `summarization` - Context summarization (enabled, trigger conditions, keep policy) - `summarization` - Context summarization (enabled, trigger conditions, keep policy)
- `subagents.enabled` - Master switch for subagent delegation - `subagents.enabled` - Master switch for subagent delegation
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens) - `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)

View file

@ -9,7 +9,7 @@
使用 `TitleMiddleware``after_model` 钩子中: 使用 `TitleMiddleware``after_model` 钩子中:
1. 检测是否是首次对话1个用户消息 + 1个助手回复 1. 检测是否是首次对话1个用户消息 + 1个助手回复
2. 检查 state 是否已有 title 2. 检查 state 是否已有 title
3. 调用 LLM 生成简洁的标题默认最多6个词 3. 默认从首条用户消息生成本地 fallback 标题,避免在流式回复结束前额外等待一次 LLM 调用;显式配置 `model_name` 时才调用 LLM 生成标题默认最多6个词
4. 将 title 存储到 `ThreadState` 中(会被 checkpointer 持久化) 4. 将 title 存储到 `ThreadState` 中(会被 checkpointer 持久化)
TitleMiddleware 会先把 LangChain message content 里的结构化 block/list 内容归一化为纯文本,再拼到 title prompt 里,避免把 Python/JSON 的原始 repr 泄漏到标题生成模型。 TitleMiddleware 会先把 LangChain message content 里的结构化 block/list 内容归一化为纯文本,再拼到 title prompt 里,避免把 Python/JSON 的原始 repr 泄漏到标题生成模型。
@ -67,7 +67,7 @@ title:
enabled: true enabled: true
max_words: 6 max_words: 6
max_chars: 60 max_chars: 60
model_name: null # 使用默认模型 model_name: null # null = 快速本地 fallback填模型名才启用 LLM 标题
``` ```
或在代码中配置: 或在代码中配置:
@ -149,17 +149,21 @@ sequenceDiagram
participant Client participant Client
participant LangGraph participant LangGraph
participant TitleMiddleware participant TitleMiddleware
participant LLM participant TitleModel as Title model (optional)
participant Checkpointer participant Checkpointer
User->>Client: 发送首条消息 User->>Client: 发送首条消息
Client->>LangGraph: POST /threads/{id}/runs Client->>LangGraph: POST /threads/{id}/runs
LangGraph->>Agent: 处理消息 LangGraph->>Agent: 处理消息
Agent-->>LangGraph: 返回回复 Agent-->>LangGraph: 返回回复
LangGraph->>TitleMiddleware: after_agent() LangGraph->>TitleMiddleware: after_model()/aafter_model()
TitleMiddleware->>TitleMiddleware: 检查是否需要生成 title TitleMiddleware->>TitleMiddleware: 检查是否需要生成 title
TitleMiddleware->>LLM: 生成 title alt title.model_name 为空(默认)
LLM-->>TitleMiddleware: 返回 title TitleMiddleware->>TitleMiddleware: 从首条用户消息生成本地 fallback title
else 显式配置 title.model_name
TitleMiddleware->>TitleModel: 生成 LLM title
TitleModel-->>TitleMiddleware: 返回 title
end
TitleMiddleware->>LangGraph: return {"title": "..."} TitleMiddleware->>LangGraph: return {"title": "..."}
LangGraph->>Checkpointer: 保存 state (含 title) LangGraph->>Checkpointer: 保存 state (含 title)
LangGraph-->>Client: 返回响应 LangGraph-->>Client: 返回响应
@ -178,20 +182,15 @@ sequenceDiagram
## 注意事项 ## 注意事项
1. **读取方式不同**Title 在 `state.values.title` 而非 `thread.metadata.title` 1. **读取方式不同**Title 在 `state.values.title` 而非 `thread.metadata.title`
2. **性能考虑**title 生成会增加约 0.5-1 秒延迟,可通过使用更快的模型优化 2. **性能考虑**默认配置不调用标题模型;只有显式配置 `title.model_name` 时,才会在首轮回复后额外等待一次 LLM title 生成
3. **并发安全**middleware 在 agent 执行后运行,不会阻塞主流程 3. **并发安全**middleware 在 agent 首次完整回复后更新 state不需要客户端额外请求
4. **Fallback 策略**如果 LLM 调用失败,会使用用户消息的前几个词作为 title 4. **Fallback 策略**默认使用用户消息前几个字符作为 title如果显式启用的 LLM 调用失败,也会回退到该策略
## 测试 ## 测试
```python ```bash
# 测试 title 生成 cd backend
import pytest uv run pytest tests/test_title_middleware_core_logic.py tests/test_title_generation.py
from deerflow.agents.title_middleware import TitleMiddleware
def test_title_generation():
# TODO: 添加单元测试
pass
``` ```
## 故障排查 ## 故障排查
@ -199,8 +198,8 @@ def test_title_generation():
### Title 没有生成 ### Title 没有生成
1. 检查配置是否启用:`get_title_config().enabled == True` 1. 检查配置是否启用:`get_title_config().enabled == True`
2. 检查日志:查找 "Generated thread title" 或错误信息 2. 确认是首次对话:只有 1 个用户消息和 1 个助手回复时才会触发
3. 确认是首次对话:只有 1 个用户消息和 1 个助手回复时才会触发 3. 如果显式配置了 `title.model_name`,检查标题模型是否可用;未配置时会走本地 fallback
### Title 生成但客户端看不到 ### Title 生成但客户端看不到
@ -231,16 +230,19 @@ def test_title_generation():
```python ```python
# TitleMiddleware 核心逻辑 # TitleMiddleware 核心逻辑
@override @override
def after_agent(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | None: async def aafter_model(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | None:
"""Generate and set thread title after the first agent response.""" return await self._agenerate_title_result(state)
if self._should_generate_title(state, runtime):
title = self._generate_title(runtime) async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None:
print(f"Generated thread title: {title}") if not self._should_generate_title(state):
return None
# ✅ 返回 state 更新,会被 checkpointer 自动持久化
return {"title": title} config = self._get_title_config()
if not config.model_name:
return None return self._generate_title_result(state)
# 显式配置 title.model_name 时才调用标题模型;失败会回退到本地 title。
...
``` ```
## 相关文件 ## 相关文件

View file

@ -387,7 +387,7 @@ title:
enabled: true enabled: true
max_words: 6 max_words: 6
max_chars: 60 max_chars: 60
model_name: null # Use first model in list model_name: null # null = fast local fallback; set a model name to use LLM title generation
``` ```
### GitHub API Token (Optional for GitHub Deep Research Skill) ### GitHub API Token (Optional for GitHub Deep Research Skill)

View file

@ -16,9 +16,9 @@
#### [`packages/harness/deerflow/agents/middlewares/title_middleware.py`](../packages/harness/deerflow/agents/middlewares/title_middleware.py) (新建) #### [`packages/harness/deerflow/agents/middlewares/title_middleware.py`](../packages/harness/deerflow/agents/middlewares/title_middleware.py) (新建)
- ✅ 创建 `TitleMiddleware` - ✅ 创建 `TitleMiddleware`
- ✅ 实现 `_should_generate_title()` 检查是否需要生成 - ✅ 实现 `_should_generate_title()` 检查是否需要生成
- ✅ 实现 `_generate_title()` 调用 LLM 生成标题 - ✅ 默认使用本地 fallback 生成标题,避免流式回复结束前等待额外 LLM 调用;显式配置 `model_name` 时可使用 LLM 标题
- ✅ 实现 `after_agent()` 钩子,在首次对话后自动触发 - ✅ 实现 `after_model()` / `aafter_model()` 钩子,在首次对话后自动触发
- ✅ 包含 fallback 策略LLM 失败时使用用户消息前几个词 - ✅ 包含 fallback 策略LLM 未配置或失败时使用用户消息前几个字符
#### [`packages/harness/deerflow/config/app_config.py`](../packages/harness/deerflow/config/app_config.py) #### [`packages/harness/deerflow/config/app_config.py`](../packages/harness/deerflow/config/app_config.py)
- ✅ 导入 `load_title_config_from_dict` - ✅ 导入 `load_title_config_from_dict`
@ -37,7 +37,7 @@ title:
enabled: true enabled: true
max_words: 6 max_words: 6
max_chars: 60 max_chars: 60
model_name: null model_name: null # null = 快速本地 fallback填模型名才启用 LLM 标题
``` ```
### 3. 文档 ### 3. 文档
@ -81,11 +81,13 @@ title:
Agent 处理并返回回复 Agent 处理并返回回复
TitleMiddleware.after_agent() 触发 TitleMiddleware.after_model()/aafter_model() 触发
检查:是否首次对话?是否已有 title 检查:是否首次对话?是否已有 title
调用 LLM 生成 title 默认从首条用户消息生成本地 fallback title
如果显式配置 title.model_name才调用 LLM 生成更精炼的 title
返回 {"title": "..."} 更新 state 返回 {"title": "..."} 更新 state
@ -113,7 +115,7 @@ title:
enabled: true enabled: true
max_words: 8 # 标题最多 8 个词 max_words: 8 # 标题最多 8 个词
max_chars: 80 # 标题最多 80 个字符 max_chars: 80 # 标题最多 80 个字符
model_name: null # 使用默认模型 model_name: null # null = 快速本地 fallback填模型名才启用 LLM 标题
``` ```
3. **配置持久化(可选)** 3. **配置持久化(可选)**
@ -169,8 +171,8 @@ pytest
### Title 没有生成? ### Title 没有生成?
1. 检查配置:`title.enabled = true` 1. 检查配置:`title.enabled = true`
2. 查看日志:搜索 "Generated thread title" 2. 确认是首次对话1 个用户消息 + 1 个助手回复)
3. 确认是首次对话1 个用户消息 + 1 个助手回复) 3. 如果显式配置了 `title.model_name`,检查标题模型是否可用;未配置时会走本地 fallback
### Title 生成但看不到? ### Title 生成但看不到?
@ -188,22 +190,22 @@ pytest
## 📊 性能影响 ## 📊 性能影响
- **延迟增加**:约 0.5-1 秒LLM 调用) - **默认延迟**:默认 `title.model_name: null` 不会发起额外 LLM 调用,仅从首条用户消息生成本地 fallback title
- **并发安全**:在 `after_agent` 中运行,不阻塞主流程 - **显式 LLM 标题延迟**:只有配置 `title.model_name` 时,首轮回复后才会等待一次标题模型调用
- **并发安全**:在 `after_model()` / `aafter_model()` 中更新 state不需要客户端额外请求
- **资源消耗**:每个 thread 只生成一次 - **资源消耗**:每个 thread 只生成一次
### 优化建议 ### 优化建议
1. 使用更快的模型(如 `gpt-3.5-turbo` 1. 默认保持 `model_name: null`,避免流式回复结束前的额外 LLM 等待
2. 减少 `max_words``max_chars` 2. 如需更精炼标题,再显式配置较快的标题模型
3. 调整 prompt 使其更简洁 3. 减少 `max_words``max_chars`,并让 prompt 保持简洁
--- ---
## 🚀 下一步 ## 🚀 下一步
- [ ] 添加集成测试(需要 mock LangGraph Runtime - [ ] 补充 prompt template 的集成测试
- [ ] 支持自定义 prompt template
- [ ] 支持多语言 title 生成 - [ ] 支持多语言 title 生成
- [ ] 添加 title 重新生成功能 - [ ] 添加 title 重新生成功能
- [ ] 监控 title 生成成功率和延迟 - [ ] 监控 title 生成成功率和延迟

View file

@ -67,6 +67,11 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
def _is_user_message_for_title(message: object) -> bool: def _is_user_message_for_title(message: object) -> bool:
return getattr(message, "type", None) == "human" and not is_dynamic_context_reminder(message) return getattr(message, "type", None) == "human" and not is_dynamic_context_reminder(message)
def _get_title_user_message(self, state: TitleMiddlewareState) -> str:
messages = state.get("messages", [])
user_msg_content = next((m.content for m in messages if self._is_user_message_for_title(m)), "")
return self._normalize_content(user_msg_content)
def _should_generate_title(self, state: TitleMiddlewareState) -> bool: def _should_generate_title(self, state: TitleMiddlewareState) -> bool:
"""Check if we should generate a title for this thread.""" """Check if we should generate a title for this thread."""
config = self._get_title_config() config = self._get_title_config()
@ -97,10 +102,9 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
config = self._get_title_config() config = self._get_title_config()
messages = state.get("messages", []) messages = state.get("messages", [])
user_msg_content = next((m.content for m in messages if self._is_user_message_for_title(m)), "")
assistant_msg_content = next((m.content for m in messages if m.type == "ai"), "") assistant_msg_content = next((m.content for m in messages if m.type == "ai"), "")
user_msg = self._normalize_content(user_msg_content) user_msg = self._get_title_user_message(state)
assistant_msg = self._strip_think_tags(self._normalize_content(assistant_msg_content)) assistant_msg = self._strip_think_tags(self._normalize_content(assistant_msg_content))
prompt = config.prompt_template.format( prompt = config.prompt_template.format(
@ -153,18 +157,23 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
if not self._should_generate_title(state): if not self._should_generate_title(state):
return None return None
_, user_msg = self._build_title_prompt(state) user_msg = self._get_title_user_message(state)
return {"title": self._fallback_title(user_msg)} return {"title": self._fallback_title(user_msg)}
async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None: async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None:
"""Generate a title asynchronously and fall back locally on failure.""" """Generate a configured LLM title asynchronously and fall back locally."""
if not self._should_generate_title(state): if not self._should_generate_title(state):
return None return None
config = self._get_title_config() config = self._get_title_config()
prompt, user_msg = self._build_title_prompt(state) if not config.model_name:
user_msg = self._get_title_user_message(state)
return {"title": self._fallback_title(user_msg)}
user_msg = self._get_title_user_message(state)
try: try:
prompt, user_msg = self._build_title_prompt(state)
# attach_tracing=False because ``_get_runnable_config()`` inherits # attach_tracing=False because ``_get_runnable_config()`` inherits
# the graph-level RunnableConfig (set in ``_make_lead_agent``) whose # the graph-level RunnableConfig (set in ``_make_lead_agent``) whose
# callbacks already carry tracing handlers; binding them again at # callbacks already carry tracing handlers; binding them again at
@ -172,10 +181,7 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
model_kwargs = {"thinking_enabled": False, "attach_tracing": False} model_kwargs = {"thinking_enabled": False, "attach_tracing": False}
if self._app_config is not None: if self._app_config is not None:
model_kwargs["app_config"] = self._app_config model_kwargs["app_config"] = self._app_config
if config.model_name: model = create_chat_model(name=config.model_name, **model_kwargs)
model = create_chat_model(name=config.model_name, **model_kwargs)
else:
model = create_chat_model(**model_kwargs)
response = await model.ainvoke(prompt, config=self._get_runnable_config()) response = await model.ainvoke(prompt, config=self._get_runnable_config())
title = self._parse_title(response.content) title = self._parse_title(response.content)
if title: if title:

View file

@ -24,11 +24,11 @@ class TitleConfig(BaseModel):
) )
model_name: str | None = Field( model_name: str | None = Field(
default=None, default=None,
description="Model name to use for title generation (None = use default model)", description="Model name to use for LLM title generation (None = use local fallback title)",
) )
prompt_template: str = Field( prompt_template: str = Field(
default=("Generate a concise title (max {max_words} words) for this conversation.\nUser: {user_msg}\nAssistant: {assistant_msg}\n\nReturn ONLY the title, no quotes, no explanation."), default=("Generate a concise title (max {max_words} words) for this conversation.\nUser: {user_msg}\nAssistant: {assistant_msg}\n\nReturn ONLY the title, no quotes, no explanation."),
description="Prompt template for title generation", description="Prompt template for LLM title generation when model_name is set",
) )

View file

@ -82,7 +82,8 @@ tools:
use: deerflow.sandbox.tools:write_file_tool use: deerflow.sandbox.tools:write_file_tool
# Memory + summarization make background / debounced model calls whose timing is # Memory + summarization make background / debounced model calls whose timing is
# non-deterministic; disable them so record and replay see the same model-call # non-deterministic; disable them so record and replay see the same model-call
# set. (Title stays — it is an in-graph, deterministic call we record.) # set. Title stays enabled, but the default title.model_name: null path is a
# local state update rather than a recorded model call.
memory: memory:
enabled: false enabled: false
injection_enabled: false injection_enabled: false

View file

@ -94,7 +94,7 @@ class TestTitleMiddlewareCoreLogic:
assert middleware._should_generate_title(state) is False assert middleware._should_generate_title(state) is False
def test_generate_title_uses_async_model_and_respects_max_chars(self, monkeypatch): def test_generate_title_uses_async_model_and_respects_max_chars(self, monkeypatch):
_set_test_title_config(max_chars=12, model_name=None) _set_test_title_config(max_chars=12, model_name="title-model")
middleware = TitleMiddleware() middleware = TitleMiddleware()
model = MagicMock() model = MagicMock()
model.ainvoke = AsyncMock(return_value=AIMessage(content="短标题")) model.ainvoke = AsyncMock(return_value=AIMessage(content="短标题"))
@ -110,7 +110,7 @@ class TestTitleMiddlewareCoreLogic:
title = result["title"] title = result["title"]
assert title == "短标题" assert title == "短标题"
title_middleware_module.create_chat_model.assert_called_once_with(thinking_enabled=False, attach_tracing=False) title_middleware_module.create_chat_model.assert_called_once_with(name="title-model", thinking_enabled=False, attach_tracing=False)
model.ainvoke.assert_awaited_once() model.ainvoke.assert_awaited_once()
assert model.ainvoke.await_args.kwargs["config"] == { assert model.ainvoke.await_args.kwargs["config"] == {
"run_name": "title_agent", "run_name": "title_agent",
@ -160,7 +160,7 @@ class TestTitleMiddlewareCoreLogic:
) )
def test_generate_title_normalizes_structured_message_content(self, monkeypatch): def test_generate_title_normalizes_structured_message_content(self, monkeypatch):
_set_test_title_config(max_chars=20) _set_test_title_config(max_chars=20, model_name="title-model")
middleware = TitleMiddleware() middleware = TitleMiddleware()
model = MagicMock() model = MagicMock()
model.ainvoke = AsyncMock(return_value=AIMessage(content="请帮我总结这段代码")) model.ainvoke = AsyncMock(return_value=AIMessage(content="请帮我总结这段代码"))
@ -179,7 +179,7 @@ class TestTitleMiddlewareCoreLogic:
assert title == "请帮我总结这段代码" assert title == "请帮我总结这段代码"
def test_generate_title_fallback_for_long_message(self, monkeypatch): def test_generate_title_fallback_for_long_message(self, monkeypatch):
_set_test_title_config(max_chars=20) _set_test_title_config(max_chars=20, model_name="title-model")
middleware = TitleMiddleware() middleware = TitleMiddleware()
model = MagicMock() model = MagicMock()
model.ainvoke = AsyncMock(side_effect=RuntimeError("model unavailable")) model.ainvoke = AsyncMock(side_effect=RuntimeError("model unavailable"))
@ -199,6 +199,7 @@ class TestTitleMiddlewareCoreLogic:
assert title.startswith("这是一个非常长的问题描述") assert title.startswith("这是一个非常长的问题描述")
def test_aafter_model_delegates_to_async_helper(self, monkeypatch): def test_aafter_model_delegates_to_async_helper(self, monkeypatch):
_set_test_title_config(model_name="title-model")
middleware = TitleMiddleware() middleware = TitleMiddleware()
monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value={"title": "异步标题"})) monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value={"title": "异步标题"}))
@ -208,6 +209,78 @@ class TestTitleMiddlewareCoreLogic:
monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value=None)) monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value=None))
assert asyncio.run(middleware.aafter_model({"messages": []}, runtime=MagicMock())) is None assert asyncio.run(middleware.aafter_model({"messages": []}, runtime=MagicMock())) is None
def test_aafter_model_uses_local_fallback_when_no_title_model_is_configured(self, monkeypatch):
"""Default async path must not block stream completion on a second LLM call."""
_set_test_title_config(max_chars=20, model_name=None)
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="请帮我写测试"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware.aafter_model(state, runtime=MagicMock()))
assert result == {"title": "请帮我写测试"}
create_chat_model.assert_not_called()
def test_async_generate_title_result_uses_local_fallback_without_model_name(self, monkeypatch):
"""The default async helper path avoids the hidden title-model LLM call."""
_set_test_title_config(max_chars=20, model_name=None)
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="流式回答结束后不要再等待标题模型"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware._agenerate_title_result(state))
assert result == {"title": "流式回答结束后不要再等待标题模型"}
create_chat_model.assert_not_called()
def test_async_local_fallback_does_not_format_unused_prompt_template(self, monkeypatch):
"""Local fallback should not depend on the LLM prompt template."""
_set_test_title_config(max_chars=20, model_name=None, prompt_template="{missing_placeholder}")
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="默认标题路径不应读取模型 prompt"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware._agenerate_title_result(state))
assert result == {"title": "默认标题路径不应读取模型 prompt"}
create_chat_model.assert_not_called()
def test_async_title_model_falls_back_when_prompt_template_is_invalid(self, monkeypatch):
"""Opt-in LLM title generation still degrades locally on template errors."""
_set_test_title_config(max_chars=20, model_name="title-model", prompt_template="{usr_msg}")
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="请帮我写测试"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware._agenerate_title_result(state))
assert result == {"title": "请帮我写测试"}
create_chat_model.assert_not_called()
def test_after_model_sync_delegates_to_sync_helper(self, monkeypatch): def test_after_model_sync_delegates_to_sync_helper(self, monkeypatch):
middleware = TitleMiddleware() middleware = TitleMiddleware()
@ -296,7 +369,7 @@ class TestTitleMiddlewareCoreLogic:
def test_generate_title_async_strips_think_tags_in_response(self, monkeypatch): def test_generate_title_async_strips_think_tags_in_response(self, monkeypatch):
"""Async title generation strips <think> blocks from the model response.""" """Async title generation strips <think> blocks from the model response."""
_set_test_title_config(max_chars=50) _set_test_title_config(max_chars=50, model_name="title-model")
middleware = TitleMiddleware() middleware = TitleMiddleware()
model = MagicMock() model = MagicMock()
model.ainvoke = AsyncMock(return_value=AIMessage(content="<think>用户想研究贵阳。</think>贵阳发展研究")) model.ainvoke = AsyncMock(return_value=AIMessage(content="<think>用户想研究贵阳。</think>贵阳发展研究"))

View file

@ -1114,7 +1114,7 @@ title:
enabled: true enabled: true
max_words: 6 max_words: 6
max_chars: 60 max_chars: 60
model_name: null # Use default model (first model in models list) model_name: null # null = fast local fallback; set a model name to use LLM title generation
# ============================================================================ # ============================================================================
# Summarization Configuration # Summarization Configuration

View file

@ -1,15 +1,21 @@
import { defineConfig, devices } from "@playwright/test"; import { defineConfig, devices } from "@playwright/test";
const frontendPort = process.env.E2E_FRONTEND_PORT ?? "3000";
const gatewayPort = process.env.E2E_GATEWAY_PORT ?? "8011";
const frontendUrl = `http://localhost:${frontendPort}`;
const gatewayUrl = `http://localhost:${gatewayPort}`;
const gatewayInternalUrl = `http://127.0.0.1:${gatewayPort}`;
/** /**
* Layer 2 of the record/replay e2e: the REAL Next.js frontend rendering data * Layer 2 of the record/replay e2e: the REAL Next.js frontend rendering data
* from a REAL gateway whose LLM is the deterministic `ReplayChatModel` (no API * from a REAL gateway whose LLM is the deterministic `ReplayChatModel` (no API
* key). This is separate from `playwright.config.ts` (which mocks the backend) * key). This is separate from `playwright.config.ts` (which mocks the backend)
* so the mock-based suite is untouched. * so the mock-based suite is untouched.
* *
* Two webServers are started: the replay gateway (:8011) and the frontend * Two webServers are started: the replay gateway and the frontend pointed at
* (:3000, pointed at the gateway). Auth-disabled mode is enabled on both * it. Auth-disabled mode is enabled on both servers so the no-cookie e2e
* servers so the no-cookie e2e contract is covered; specs that need session * contract is covered; specs that need session cookies still register a
* cookies still register a throwaway test account at runtime. * throwaway test account at runtime.
*/ */
export default defineConfig({ export default defineConfig({
testDir: "./tests/e2e-real-backend", testDir: "./tests/e2e-real-backend",
@ -21,7 +27,7 @@ export default defineConfig({
timeout: 90_000, timeout: 90_000,
use: { use: {
baseURL: "http://localhost:3000", baseURL: frontendUrl,
trace: "on-first-retry", trace: "on-first-retry",
}, },
@ -29,9 +35,9 @@ export default defineConfig({
webServer: [ webServer: [
{ {
command: "uv run python scripts/run_replay_gateway.py --port 8011", command: `uv run python scripts/run_replay_gateway.py --port ${gatewayPort} --cors ${frontendUrl}`,
cwd: "../backend", cwd: "../backend",
url: "http://localhost:8011/health", url: `${gatewayUrl}/health`,
reuseExistingServer: !process.env.CI, reuseExistingServer: !process.env.CI,
timeout: 180_000, timeout: 180_000,
stdout: "pipe", stdout: "pipe",
@ -46,10 +52,11 @@ export default defineConfig({
}, },
{ {
command: "pnpm build && pnpm start", command: "pnpm build && pnpm start",
url: "http://localhost:3000", url: frontendUrl,
reuseExistingServer: !process.env.CI, reuseExistingServer: !process.env.CI,
timeout: 240_000, timeout: 240_000,
env: { env: {
PORT: frontendPort,
SKIP_ENV_VALIDATION: "1", SKIP_ENV_VALIDATION: "1",
DEER_FLOW_AUTH_DISABLED: "1", DEER_FLOW_AUTH_DISABLED: "1",
BETTER_AUTH_SECRET: "local-dev-secret", BETTER_AUTH_SECRET: "local-dev-secret",
@ -57,7 +64,7 @@ export default defineConfig({
// next.config rewrites (same-origin proxy) instead of talking to the // next.config rewrites (same-origin proxy) instead of talking to the
// gateway cross-origin — cross-origin fetches drop the auth cookies. // gateway cross-origin — cross-origin fetches drop the auth cookies.
// Just point that proxy at the replay gateway. // Just point that proxy at the replay gateway.
DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: "http://127.0.0.1:8011", DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: gatewayInternalUrl,
}, },
}, },
], ],

View file

@ -82,7 +82,7 @@ Limits the number of parallel subagent task calls the agent can make in a single
### TitleMiddleware ### TitleMiddleware
Automatically generates a title for the thread after the first exchange. The title is derived from the user's first message and the agent's response. Automatically generates a title for the thread after the first exchange. By default, the title is a fast local fallback derived from the user's first message. Set `title.model_name` only when you want the optional LLM title path.
**Configuration**: `title:` section in `config.yaml`. **Configuration**: `title:` section in `config.yaml`.
@ -91,7 +91,7 @@ title:
enabled: true enabled: true
max_words: 6 max_words: 6
max_chars: 60 max_chars: 60
model_name: null # use default model model_name: null # local fallback; set a model name to use LLM title generation
``` ```
--- ---

View file

@ -82,7 +82,7 @@ memory:
### TitleMiddleware ### TitleMiddleware
在第一次交互后自动为线程生成标题。标题从用户的第一条消息和 Agent 的响应中派生 在第一次交互后自动为线程生成标题。默认从用户的第一条消息生成快速本地 fallback 标题;只有设置 `title.model_name` 时,才启用可选的 LLM 标题路径
**配置**`config.yaml` 中的 `title:` 部分。 **配置**`config.yaml` 中的 `title:` 部分。
@ -91,7 +91,7 @@ title:
enabled: true enabled: true
max_words: 6 max_words: 6
max_chars: 60 max_chars: 60
model_name: null # 使用默认模型 model_name: null # 本地 fallback填模型名才启用 LLM 标题
``` ```
--- ---

View file

@ -2,7 +2,9 @@ import { expect, test } from "@playwright/test";
import { AUTH_DISABLED_USER } from "../../src/core/auth/auth-disabled-user"; import { AUTH_DISABLED_USER } from "../../src/core/auth/auth-disabled-user";
const APP = "http://localhost:3000"; const APP =
process.env.E2E_APP_URL ??
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
test.describe("auth-disabled contract (real backend)", () => { test.describe("auth-disabled contract (real backend)", () => {
test("gateway /auth/me returns the frontend synthetic user without a cookie", async ({ test("gateway /auth/me returns the frontend synthetic user without a cookie", async ({

View file

@ -20,7 +20,9 @@ import { expect, test } from "@playwright/test";
* No model, no recording, no API key the runs are seeded via a test-only * No model, no recording, no API key the runs are seeded via a test-only
* endpoint mounted only on the replay gateway. * endpoint mounted only on the replay gateway.
*/ */
const APP = "http://localhost:3000"; const APP =
process.env.E2E_APP_URL ??
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
// Distinctive markers so getByText can't collide with UI chrome. // Distinctive markers so getByText can't collide with UI chrome.
const ALPHA = "ALPHA-FIRST-QUESTION-7f3a2c"; const ALPHA = "ALPHA-FIRST-QUESTION-7f3a2c";

View file

@ -11,13 +11,15 @@ const here = dirname(fileURLToPath(import.meta.url));
* API key) and assert the browser renders the backend's data correctly. * API key) and assert the browser renders the backend's data correctly.
* *
* The prompt is read from the same fixture the gateway replays, so the input * The prompt is read from the same fixture the gateway replays, so the input
* hash matches and the recorded turns (write_file -> auto-title -> read_file -> * hash matches and the recorded model turns reproduce deterministically. The
* final answer) reproduce deterministically. * default auto-title is local fallback state, not a replayed model turn.
*/ */
// Register through the frontend origin (same-origin proxy) so the auth cookies // Register through the frontend origin (same-origin proxy) so the auth cookies
// are stored for and sent to localhost:3000 — the gateway is reached via the // are stored for and sent to the browser origin — the gateway is reached via the
// next.config rewrite, never cross-origin from the browser. // next.config rewrite, never cross-origin from the browser.
const APP = "http://localhost:3000"; const APP =
process.env.E2E_APP_URL ??
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
const fixture = JSON.parse( const fixture = JSON.parse(
readFileSync( readFileSync(
join( join(
@ -32,10 +34,17 @@ const fixture = JSON.parse(
}; };
const PROMPT = fixture.prompt; const PROMPT = fixture.prompt;
// Derive the assertions from the fixture so a re-record auto-updates them. Both const FALLBACK_TITLE_MAX_CHARS = 50;
// are model-generated strings absent from the user prompt, so a pass proves the
// replay drove the render (not a prompt echo): the first plain-text turn is the function fallbackTitle(userMsg: string): string {
// in-graph auto-title; the JSON-array turn is the follow-up suggestions. if (!userMsg) return "New Conversation";
if (userMsg.length <= FALLBACK_TITLE_MAX_CHARS) return userMsg;
return `${userMsg.slice(0, FALLBACK_TITLE_MAX_CHARS).trimEnd()}...`;
}
// Suggestions still come from the recorded model fixture. The default title no
// longer does: TitleMiddleware uses a local fallback when title.model_name is
// unset, so derive that expected title from the prompt.
const textTurns = fixture.turns const textTurns = fixture.turns
.map((t) => t.output?.data?.content) .map((t) => t.output?.data?.content)
.filter((c): c is string => typeof c === "string" && c.trim().length > 0); .filter((c): c is string => typeof c === "string" && c.trim().length > 0);
@ -52,7 +61,7 @@ const EXPECTED_SUGGESTION = ((): string => {
return ""; return "";
} }
})(); })();
const EXPECTED_TITLE = textTurns.find((c) => !c.trim().startsWith("[")) ?? ""; const EXPECTED_TITLE = fallbackTitle(PROMPT);
test.describe("real backend render (replay, no API key)", () => { test.describe("real backend render (replay, no API key)", () => {
test.beforeEach(async ({ context }) => { test.beforeEach(async ({ context }) => {
@ -66,7 +75,7 @@ test.describe("real backend render (replay, no API key)", () => {
expect(resp.status(), await resp.text()).toBe(201); expect(resp.status(), await resp.text()).toBe(201);
}); });
test("renders the replayed auto-title + suggestions from a real backend", async ({ test("renders the local auto-title + replayed suggestions from a real backend", async ({
page, page,
}) => { }) => {
// ultra mode so the context the frontend sends (is_plan_mode + subagent_enabled) // ultra mode so the context the frontend sends (is_plan_mode + subagent_enabled)
@ -85,17 +94,13 @@ test.describe("real backend render (replay, no API key)", () => {
await textarea.fill(PROMPT); await textarea.fill(PROMPT);
await textarea.press("Enter"); await textarea.press("Enter");
// Replay-only DOM assertions (derived from the fixture): both are // The title is the default local fallback, while the suggestion is a
// model-generated strings absent from the user prompt, so they render only if // replayed model output absent from the prompt. Together they prove the
// the recorded turns replayed AND the real frontend rendered them — the // backend state update and the replayed post-answer model call both render
// in-graph auto-title and the post-answer follow-up suggestion. Together they // through the real frontend.
// prove the whole pipeline (replay backend -> real frontend render). The
// record spec waits for the /suggestions response, so a re-recorded fixture
// always captures the suggestion turn — a missing one is a broken recording
// and must fail loud here, not pass silently.
expect( expect(
EXPECTED_TITLE, EXPECTED_TITLE,
"fixture should contain an auto-title turn", "default local fallback title should be derived from the prompt",
).not.toBe(""); ).not.toBe("");
expect( expect(
EXPECTED_SUGGESTION, EXPECTED_SUGGESTION,

View file

@ -6,9 +6,9 @@ import { expect, test } from "@playwright/test";
* RECORD driver (Plan A): drive the real frontend through the write/read-file * RECORD driver (Plan A): drive the real frontend through the write/read-file
* scenario against the real-model gateway. The gateway captures every model * scenario against the real-model gateway. The gateway captures every model
* call to DEERFLOW_RECORD_OUT; this just needs to drive the flow and wait until * call to DEERFLOW_RECORD_OUT; this just needs to drive the flow and wait until
* the captures stop arriving (main turns + in-graph title + follow-up * the captures stop arriving (main turns + follow-up suggestions all fired;
* suggestions all fired). It asserts nothing about content it produces the * the default auto-title is local state). It asserts nothing about content
* fixture, it doesn't verify it. * it produces the fixture, it doesn't verify it.
*/ */
const APP = "http://localhost:3000"; const APP = "http://localhost:3000";
const SCENARIO = "write_read_file"; const SCENARIO = "write_read_file";