fix(title): avoid default LLM call before stream end (#3885)
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run

* fix(title): avoid default LLM call before stream end

* fix(title): keep default fallback local

* fix(title): harden fallback and replay e2e

* docs(title): align fallback title behavior
This commit is contained in:
xiawiie 2026-06-30 18:58:02 +08:00 committed by GitHub
parent fe82552023
commit 2453718acd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
16 changed files with 202 additions and 102 deletions

View file

@ -581,7 +581,7 @@ Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only de
- `tool_groups[]` - Logical groupings for tools
- `sandbox.use` - Sandbox provider class path
- `skills.path` / `skills.container_path` - Host and container paths to skills directory
- `title` - Auto-title generation (enabled, max_words, max_chars, prompt_template)
- `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path)
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
- `subagents.enabled` - Master switch for subagent delegation
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)

View file

@ -9,7 +9,7 @@
使用 `TitleMiddleware``after_model` 钩子中:
1. 检测是否是首次对话1个用户消息 + 1个助手回复
2. 检查 state 是否已有 title
3. 调用 LLM 生成简洁的标题默认最多6个词
3. 默认从首条用户消息生成本地 fallback 标题,避免在流式回复结束前额外等待一次 LLM 调用;显式配置 `model_name` 时才调用 LLM 生成标题默认最多6个词
4. 将 title 存储到 `ThreadState` 中(会被 checkpointer 持久化)
TitleMiddleware 会先把 LangChain message content 里的结构化 block/list 内容归一化为纯文本,再拼到 title prompt 里,避免把 Python/JSON 的原始 repr 泄漏到标题生成模型。
@ -67,7 +67,7 @@ title:
enabled: true
max_words: 6
max_chars: 60
model_name: null # 使用默认模型
model_name: null # null = 快速本地 fallback填模型名才启用 LLM 标题
```
或在代码中配置:
@ -149,17 +149,21 @@ sequenceDiagram
participant Client
participant LangGraph
participant TitleMiddleware
participant LLM
participant TitleModel as Title model (optional)
participant Checkpointer
User->>Client: 发送首条消息
Client->>LangGraph: POST /threads/{id}/runs
LangGraph->>Agent: 处理消息
Agent-->>LangGraph: 返回回复
LangGraph->>TitleMiddleware: after_agent()
LangGraph->>TitleMiddleware: after_model()/aafter_model()
TitleMiddleware->>TitleMiddleware: 检查是否需要生成 title
TitleMiddleware->>LLM: 生成 title
LLM-->>TitleMiddleware: 返回 title
alt title.model_name 为空(默认)
TitleMiddleware->>TitleMiddleware: 从首条用户消息生成本地 fallback title
else 显式配置 title.model_name
TitleMiddleware->>TitleModel: 生成 LLM title
TitleModel-->>TitleMiddleware: 返回 title
end
TitleMiddleware->>LangGraph: return {"title": "..."}
LangGraph->>Checkpointer: 保存 state (含 title)
LangGraph-->>Client: 返回响应
@ -178,20 +182,15 @@ sequenceDiagram
## 注意事项
1. **读取方式不同**Title 在 `state.values.title` 而非 `thread.metadata.title`
2. **性能考虑**title 生成会增加约 0.5-1 秒延迟,可通过使用更快的模型优化
3. **并发安全**middleware 在 agent 执行后运行,不会阻塞主流程
4. **Fallback 策略**如果 LLM 调用失败,会使用用户消息的前几个词作为 title
2. **性能考虑**默认配置不调用标题模型;只有显式配置 `title.model_name` 时,才会在首轮回复后额外等待一次 LLM title 生成
3. **并发安全**middleware 在 agent 首次完整回复后更新 state不需要客户端额外请求
4. **Fallback 策略**默认使用用户消息前几个字符作为 title如果显式启用的 LLM 调用失败,也会回退到该策略
## 测试
```python
# 测试 title 生成
import pytest
from deerflow.agents.title_middleware import TitleMiddleware
def test_title_generation():
# TODO: 添加单元测试
pass
```bash
cd backend
uv run pytest tests/test_title_middleware_core_logic.py tests/test_title_generation.py
```
## 故障排查
@ -199,8 +198,8 @@ def test_title_generation():
### Title 没有生成
1. 检查配置是否启用:`get_title_config().enabled == True`
2. 检查日志:查找 "Generated thread title" 或错误信息
3. 确认是首次对话:只有 1 个用户消息和 1 个助手回复时才会触发
2. 确认是首次对话:只有 1 个用户消息和 1 个助手回复时才会触发
3. 如果显式配置了 `title.model_name`,检查标题模型是否可用;未配置时会走本地 fallback
### Title 生成但客户端看不到
@ -231,16 +230,19 @@ def test_title_generation():
```python
# TitleMiddleware 核心逻辑
@override
def after_agent(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | None:
"""Generate and set thread title after the first agent response."""
if self._should_generate_title(state, runtime):
title = self._generate_title(runtime)
print(f"Generated thread title: {title}")
# ✅ 返回 state 更新,会被 checkpointer 自动持久化
return {"title": title}
async def aafter_model(self, state: TitleMiddlewareState, runtime: Runtime) -> dict | None:
return await self._agenerate_title_result(state)
async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None:
if not self._should_generate_title(state):
return None
config = self._get_title_config()
if not config.model_name:
return self._generate_title_result(state)
# 显式配置 title.model_name 时才调用标题模型;失败会回退到本地 title。
...
```
## 相关文件

View file

@ -387,7 +387,7 @@ title:
enabled: true
max_words: 6
max_chars: 60
model_name: null # Use first model in list
model_name: null # null = fast local fallback; set a model name to use LLM title generation
```
### GitHub API Token (Optional for GitHub Deep Research Skill)

View file

@ -16,9 +16,9 @@
#### [`packages/harness/deerflow/agents/middlewares/title_middleware.py`](../packages/harness/deerflow/agents/middlewares/title_middleware.py) (新建)
- ✅ 创建 `TitleMiddleware`
- ✅ 实现 `_should_generate_title()` 检查是否需要生成
- ✅ 实现 `_generate_title()` 调用 LLM 生成标题
- ✅ 实现 `after_agent()` 钩子,在首次对话后自动触发
- ✅ 包含 fallback 策略LLM 失败时使用用户消息前几个词
- ✅ 默认使用本地 fallback 生成标题,避免流式回复结束前等待额外 LLM 调用;显式配置 `model_name` 时可使用 LLM 标题
- ✅ 实现 `after_model()` / `aafter_model()` 钩子,在首次对话后自动触发
- ✅ 包含 fallback 策略LLM 未配置或失败时使用用户消息前几个字符
#### [`packages/harness/deerflow/config/app_config.py`](../packages/harness/deerflow/config/app_config.py)
- ✅ 导入 `load_title_config_from_dict`
@ -37,7 +37,7 @@ title:
enabled: true
max_words: 6
max_chars: 60
model_name: null
model_name: null # null = 快速本地 fallback填模型名才启用 LLM 标题
```
### 3. 文档
@ -81,11 +81,13 @@ title:
Agent 处理并返回回复
TitleMiddleware.after_agent() 触发
TitleMiddleware.after_model()/aafter_model() 触发
检查:是否首次对话?是否已有 title
调用 LLM 生成 title
默认从首条用户消息生成本地 fallback title
如果显式配置 title.model_name才调用 LLM 生成更精炼的 title
返回 {"title": "..."} 更新 state
@ -113,7 +115,7 @@ title:
enabled: true
max_words: 8 # 标题最多 8 个词
max_chars: 80 # 标题最多 80 个字符
model_name: null # 使用默认模型
model_name: null # null = 快速本地 fallback填模型名才启用 LLM 标题
```
3. **配置持久化(可选)**
@ -169,8 +171,8 @@ pytest
### Title 没有生成?
1. 检查配置:`title.enabled = true`
2. 查看日志:搜索 "Generated thread title"
3. 确认是首次对话1 个用户消息 + 1 个助手回复)
2. 确认是首次对话1 个用户消息 + 1 个助手回复)
3. 如果显式配置了 `title.model_name`,检查标题模型是否可用;未配置时会走本地 fallback
### Title 生成但看不到?
@ -188,22 +190,22 @@ pytest
## 📊 性能影响
- **延迟增加**:约 0.5-1 秒LLM 调用)
- **并发安全**:在 `after_agent` 中运行,不阻塞主流程
- **默认延迟**:默认 `title.model_name: null` 不会发起额外 LLM 调用,仅从首条用户消息生成本地 fallback title
- **显式 LLM 标题延迟**:只有配置 `title.model_name` 时,首轮回复后才会等待一次标题模型调用
- **并发安全**:在 `after_model()` / `aafter_model()` 中更新 state不需要客户端额外请求
- **资源消耗**:每个 thread 只生成一次
### 优化建议
1. 使用更快的模型(如 `gpt-3.5-turbo`
2. 减少 `max_words``max_chars`
3. 调整 prompt 使其更简洁
1. 默认保持 `model_name: null`,避免流式回复结束前的额外 LLM 等待
2. 如需更精炼标题,再显式配置较快的标题模型
3. 减少 `max_words``max_chars`,并让 prompt 保持简洁
---
## 🚀 下一步
- [ ] 添加集成测试(需要 mock LangGraph Runtime
- [ ] 支持自定义 prompt template
- [ ] 补充 prompt template 的集成测试
- [ ] 支持多语言 title 生成
- [ ] 添加 title 重新生成功能
- [ ] 监控 title 生成成功率和延迟

View file

@ -67,6 +67,11 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
def _is_user_message_for_title(message: object) -> bool:
return getattr(message, "type", None) == "human" and not is_dynamic_context_reminder(message)
def _get_title_user_message(self, state: TitleMiddlewareState) -> str:
messages = state.get("messages", [])
user_msg_content = next((m.content for m in messages if self._is_user_message_for_title(m)), "")
return self._normalize_content(user_msg_content)
def _should_generate_title(self, state: TitleMiddlewareState) -> bool:
"""Check if we should generate a title for this thread."""
config = self._get_title_config()
@ -97,10 +102,9 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
config = self._get_title_config()
messages = state.get("messages", [])
user_msg_content = next((m.content for m in messages if self._is_user_message_for_title(m)), "")
assistant_msg_content = next((m.content for m in messages if m.type == "ai"), "")
user_msg = self._normalize_content(user_msg_content)
user_msg = self._get_title_user_message(state)
assistant_msg = self._strip_think_tags(self._normalize_content(assistant_msg_content))
prompt = config.prompt_template.format(
@ -153,18 +157,23 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
if not self._should_generate_title(state):
return None
_, user_msg = self._build_title_prompt(state)
user_msg = self._get_title_user_message(state)
return {"title": self._fallback_title(user_msg)}
async def _agenerate_title_result(self, state: TitleMiddlewareState) -> dict | None:
"""Generate a title asynchronously and fall back locally on failure."""
"""Generate a configured LLM title asynchronously and fall back locally."""
if not self._should_generate_title(state):
return None
config = self._get_title_config()
prompt, user_msg = self._build_title_prompt(state)
if not config.model_name:
user_msg = self._get_title_user_message(state)
return {"title": self._fallback_title(user_msg)}
user_msg = self._get_title_user_message(state)
try:
prompt, user_msg = self._build_title_prompt(state)
# attach_tracing=False because ``_get_runnable_config()`` inherits
# the graph-level RunnableConfig (set in ``_make_lead_agent``) whose
# callbacks already carry tracing handlers; binding them again at
@ -172,10 +181,7 @@ class TitleMiddleware(AgentMiddleware[TitleMiddlewareState]):
model_kwargs = {"thinking_enabled": False, "attach_tracing": False}
if self._app_config is not None:
model_kwargs["app_config"] = self._app_config
if config.model_name:
model = create_chat_model(name=config.model_name, **model_kwargs)
else:
model = create_chat_model(**model_kwargs)
response = await model.ainvoke(prompt, config=self._get_runnable_config())
title = self._parse_title(response.content)
if title:

View file

@ -24,11 +24,11 @@ class TitleConfig(BaseModel):
)
model_name: str | None = Field(
default=None,
description="Model name to use for title generation (None = use default model)",
description="Model name to use for LLM title generation (None = use local fallback title)",
)
prompt_template: str = Field(
default=("Generate a concise title (max {max_words} words) for this conversation.\nUser: {user_msg}\nAssistant: {assistant_msg}\n\nReturn ONLY the title, no quotes, no explanation."),
description="Prompt template for title generation",
description="Prompt template for LLM title generation when model_name is set",
)

View file

@ -82,7 +82,8 @@ tools:
use: deerflow.sandbox.tools:write_file_tool
# Memory + summarization make background / debounced model calls whose timing is
# non-deterministic; disable them so record and replay see the same model-call
# set. (Title stays — it is an in-graph, deterministic call we record.)
# set. Title stays enabled, but the default title.model_name: null path is a
# local state update rather than a recorded model call.
memory:
enabled: false
injection_enabled: false

View file

@ -94,7 +94,7 @@ class TestTitleMiddlewareCoreLogic:
assert middleware._should_generate_title(state) is False
def test_generate_title_uses_async_model_and_respects_max_chars(self, monkeypatch):
_set_test_title_config(max_chars=12, model_name=None)
_set_test_title_config(max_chars=12, model_name="title-model")
middleware = TitleMiddleware()
model = MagicMock()
model.ainvoke = AsyncMock(return_value=AIMessage(content="短标题"))
@ -110,7 +110,7 @@ class TestTitleMiddlewareCoreLogic:
title = result["title"]
assert title == "短标题"
title_middleware_module.create_chat_model.assert_called_once_with(thinking_enabled=False, attach_tracing=False)
title_middleware_module.create_chat_model.assert_called_once_with(name="title-model", thinking_enabled=False, attach_tracing=False)
model.ainvoke.assert_awaited_once()
assert model.ainvoke.await_args.kwargs["config"] == {
"run_name": "title_agent",
@ -160,7 +160,7 @@ class TestTitleMiddlewareCoreLogic:
)
def test_generate_title_normalizes_structured_message_content(self, monkeypatch):
_set_test_title_config(max_chars=20)
_set_test_title_config(max_chars=20, model_name="title-model")
middleware = TitleMiddleware()
model = MagicMock()
model.ainvoke = AsyncMock(return_value=AIMessage(content="请帮我总结这段代码"))
@ -179,7 +179,7 @@ class TestTitleMiddlewareCoreLogic:
assert title == "请帮我总结这段代码"
def test_generate_title_fallback_for_long_message(self, monkeypatch):
_set_test_title_config(max_chars=20)
_set_test_title_config(max_chars=20, model_name="title-model")
middleware = TitleMiddleware()
model = MagicMock()
model.ainvoke = AsyncMock(side_effect=RuntimeError("model unavailable"))
@ -199,6 +199,7 @@ class TestTitleMiddlewareCoreLogic:
assert title.startswith("这是一个非常长的问题描述")
def test_aafter_model_delegates_to_async_helper(self, monkeypatch):
_set_test_title_config(model_name="title-model")
middleware = TitleMiddleware()
monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value={"title": "异步标题"}))
@ -208,6 +209,78 @@ class TestTitleMiddlewareCoreLogic:
monkeypatch.setattr(middleware, "_agenerate_title_result", AsyncMock(return_value=None))
assert asyncio.run(middleware.aafter_model({"messages": []}, runtime=MagicMock())) is None
def test_aafter_model_uses_local_fallback_when_no_title_model_is_configured(self, monkeypatch):
"""Default async path must not block stream completion on a second LLM call."""
_set_test_title_config(max_chars=20, model_name=None)
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="请帮我写测试"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware.aafter_model(state, runtime=MagicMock()))
assert result == {"title": "请帮我写测试"}
create_chat_model.assert_not_called()
def test_async_generate_title_result_uses_local_fallback_without_model_name(self, monkeypatch):
"""The default async helper path avoids the hidden title-model LLM call."""
_set_test_title_config(max_chars=20, model_name=None)
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="流式回答结束后不要再等待标题模型"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware._agenerate_title_result(state))
assert result == {"title": "流式回答结束后不要再等待标题模型"}
create_chat_model.assert_not_called()
def test_async_local_fallback_does_not_format_unused_prompt_template(self, monkeypatch):
"""Local fallback should not depend on the LLM prompt template."""
_set_test_title_config(max_chars=20, model_name=None, prompt_template="{missing_placeholder}")
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="默认标题路径不应读取模型 prompt"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware._agenerate_title_result(state))
assert result == {"title": "默认标题路径不应读取模型 prompt"}
create_chat_model.assert_not_called()
def test_async_title_model_falls_back_when_prompt_template_is_invalid(self, monkeypatch):
"""Opt-in LLM title generation still degrades locally on template errors."""
_set_test_title_config(max_chars=20, model_name="title-model", prompt_template="{usr_msg}")
middleware = TitleMiddleware()
create_chat_model = MagicMock()
monkeypatch.setattr(title_middleware_module, "create_chat_model", create_chat_model)
state = {
"messages": [
HumanMessage(content="请帮我写测试"),
AIMessage(content="好的"),
]
}
result = asyncio.run(middleware._agenerate_title_result(state))
assert result == {"title": "请帮我写测试"}
create_chat_model.assert_not_called()
def test_after_model_sync_delegates_to_sync_helper(self, monkeypatch):
middleware = TitleMiddleware()
@ -296,7 +369,7 @@ class TestTitleMiddlewareCoreLogic:
def test_generate_title_async_strips_think_tags_in_response(self, monkeypatch):
"""Async title generation strips <think> blocks from the model response."""
_set_test_title_config(max_chars=50)
_set_test_title_config(max_chars=50, model_name="title-model")
middleware = TitleMiddleware()
model = MagicMock()
model.ainvoke = AsyncMock(return_value=AIMessage(content="<think>用户想研究贵阳。</think>贵阳发展研究"))

View file

@ -1114,7 +1114,7 @@ title:
enabled: true
max_words: 6
max_chars: 60
model_name: null # Use default model (first model in models list)
model_name: null # null = fast local fallback; set a model name to use LLM title generation
# ============================================================================
# Summarization Configuration

View file

@ -1,15 +1,21 @@
import { defineConfig, devices } from "@playwright/test";
const frontendPort = process.env.E2E_FRONTEND_PORT ?? "3000";
const gatewayPort = process.env.E2E_GATEWAY_PORT ?? "8011";
const frontendUrl = `http://localhost:${frontendPort}`;
const gatewayUrl = `http://localhost:${gatewayPort}`;
const gatewayInternalUrl = `http://127.0.0.1:${gatewayPort}`;
/**
* Layer 2 of the record/replay e2e: the REAL Next.js frontend rendering data
* from a REAL gateway whose LLM is the deterministic `ReplayChatModel` (no API
* key). This is separate from `playwright.config.ts` (which mocks the backend)
* so the mock-based suite is untouched.
*
* Two webServers are started: the replay gateway (:8011) and the frontend
* (:3000, pointed at the gateway). Auth-disabled mode is enabled on both
* servers so the no-cookie e2e contract is covered; specs that need session
* cookies still register a throwaway test account at runtime.
* Two webServers are started: the replay gateway and the frontend pointed at
* it. Auth-disabled mode is enabled on both servers so the no-cookie e2e
* contract is covered; specs that need session cookies still register a
* throwaway test account at runtime.
*/
export default defineConfig({
testDir: "./tests/e2e-real-backend",
@ -21,7 +27,7 @@ export default defineConfig({
timeout: 90_000,
use: {
baseURL: "http://localhost:3000",
baseURL: frontendUrl,
trace: "on-first-retry",
},
@ -29,9 +35,9 @@ export default defineConfig({
webServer: [
{
command: "uv run python scripts/run_replay_gateway.py --port 8011",
command: `uv run python scripts/run_replay_gateway.py --port ${gatewayPort} --cors ${frontendUrl}`,
cwd: "../backend",
url: "http://localhost:8011/health",
url: `${gatewayUrl}/health`,
reuseExistingServer: !process.env.CI,
timeout: 180_000,
stdout: "pipe",
@ -46,10 +52,11 @@ export default defineConfig({
},
{
command: "pnpm build && pnpm start",
url: "http://localhost:3000",
url: frontendUrl,
reuseExistingServer: !process.env.CI,
timeout: 240_000,
env: {
PORT: frontendPort,
SKIP_ENV_VALIDATION: "1",
DEER_FLOW_AUTH_DISABLED: "1",
BETTER_AUTH_SECRET: "local-dev-secret",
@ -57,7 +64,7 @@ export default defineConfig({
// next.config rewrites (same-origin proxy) instead of talking to the
// gateway cross-origin — cross-origin fetches drop the auth cookies.
// Just point that proxy at the replay gateway.
DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: "http://127.0.0.1:8011",
DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: gatewayInternalUrl,
},
},
],

View file

@ -82,7 +82,7 @@ Limits the number of parallel subagent task calls the agent can make in a single
### TitleMiddleware
Automatically generates a title for the thread after the first exchange. The title is derived from the user's first message and the agent's response.
Automatically generates a title for the thread after the first exchange. By default, the title is a fast local fallback derived from the user's first message. Set `title.model_name` only when you want the optional LLM title path.
**Configuration**: `title:` section in `config.yaml`.
@ -91,7 +91,7 @@ title:
enabled: true
max_words: 6
max_chars: 60
model_name: null # use default model
model_name: null # local fallback; set a model name to use LLM title generation
```
---

View file

@ -82,7 +82,7 @@ memory:
### TitleMiddleware
在第一次交互后自动为线程生成标题。标题从用户的第一条消息和 Agent 的响应中派生
在第一次交互后自动为线程生成标题。默认从用户的第一条消息生成快速本地 fallback 标题;只有设置 `title.model_name` 时,才启用可选的 LLM 标题路径
**配置**`config.yaml` 中的 `title:` 部分。
@ -91,7 +91,7 @@ title:
enabled: true
max_words: 6
max_chars: 60
model_name: null # 使用默认模型
model_name: null # 本地 fallback填模型名才启用 LLM 标题
```
---

View file

@ -2,7 +2,9 @@ import { expect, test } from "@playwright/test";
import { AUTH_DISABLED_USER } from "../../src/core/auth/auth-disabled-user";
const APP = "http://localhost:3000";
const APP =
process.env.E2E_APP_URL ??
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
test.describe("auth-disabled contract (real backend)", () => {
test("gateway /auth/me returns the frontend synthetic user without a cookie", async ({

View file

@ -20,7 +20,9 @@ import { expect, test } from "@playwright/test";
* No model, no recording, no API key the runs are seeded via a test-only
* endpoint mounted only on the replay gateway.
*/
const APP = "http://localhost:3000";
const APP =
process.env.E2E_APP_URL ??
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
// Distinctive markers so getByText can't collide with UI chrome.
const ALPHA = "ALPHA-FIRST-QUESTION-7f3a2c";

View file

@ -11,13 +11,15 @@ const here = dirname(fileURLToPath(import.meta.url));
* API key) and assert the browser renders the backend's data correctly.
*
* The prompt is read from the same fixture the gateway replays, so the input
* hash matches and the recorded turns (write_file -> auto-title -> read_file ->
* final answer) reproduce deterministically.
* hash matches and the recorded model turns reproduce deterministically. The
* default auto-title is local fallback state, not a replayed model turn.
*/
// Register through the frontend origin (same-origin proxy) so the auth cookies
// are stored for and sent to localhost:3000 — the gateway is reached via the
// are stored for and sent to the browser origin — the gateway is reached via the
// next.config rewrite, never cross-origin from the browser.
const APP = "http://localhost:3000";
const APP =
process.env.E2E_APP_URL ??
`http://localhost:${process.env.E2E_FRONTEND_PORT ?? "3000"}`;
const fixture = JSON.parse(
readFileSync(
join(
@ -32,10 +34,17 @@ const fixture = JSON.parse(
};
const PROMPT = fixture.prompt;
// Derive the assertions from the fixture so a re-record auto-updates them. Both
// are model-generated strings absent from the user prompt, so a pass proves the
// replay drove the render (not a prompt echo): the first plain-text turn is the
// in-graph auto-title; the JSON-array turn is the follow-up suggestions.
const FALLBACK_TITLE_MAX_CHARS = 50;
function fallbackTitle(userMsg: string): string {
if (!userMsg) return "New Conversation";
if (userMsg.length <= FALLBACK_TITLE_MAX_CHARS) return userMsg;
return `${userMsg.slice(0, FALLBACK_TITLE_MAX_CHARS).trimEnd()}...`;
}
// Suggestions still come from the recorded model fixture. The default title no
// longer does: TitleMiddleware uses a local fallback when title.model_name is
// unset, so derive that expected title from the prompt.
const textTurns = fixture.turns
.map((t) => t.output?.data?.content)
.filter((c): c is string => typeof c === "string" && c.trim().length > 0);
@ -52,7 +61,7 @@ const EXPECTED_SUGGESTION = ((): string => {
return "";
}
})();
const EXPECTED_TITLE = textTurns.find((c) => !c.trim().startsWith("[")) ?? "";
const EXPECTED_TITLE = fallbackTitle(PROMPT);
test.describe("real backend render (replay, no API key)", () => {
test.beforeEach(async ({ context }) => {
@ -66,7 +75,7 @@ test.describe("real backend render (replay, no API key)", () => {
expect(resp.status(), await resp.text()).toBe(201);
});
test("renders the replayed auto-title + suggestions from a real backend", async ({
test("renders the local auto-title + replayed suggestions from a real backend", async ({
page,
}) => {
// ultra mode so the context the frontend sends (is_plan_mode + subagent_enabled)
@ -85,17 +94,13 @@ test.describe("real backend render (replay, no API key)", () => {
await textarea.fill(PROMPT);
await textarea.press("Enter");
// Replay-only DOM assertions (derived from the fixture): both are
// model-generated strings absent from the user prompt, so they render only if
// the recorded turns replayed AND the real frontend rendered them — the
// in-graph auto-title and the post-answer follow-up suggestion. Together they
// prove the whole pipeline (replay backend -> real frontend render). The
// record spec waits for the /suggestions response, so a re-recorded fixture
// always captures the suggestion turn — a missing one is a broken recording
// and must fail loud here, not pass silently.
// The title is the default local fallback, while the suggestion is a
// replayed model output absent from the prompt. Together they prove the
// backend state update and the replayed post-answer model call both render
// through the real frontend.
expect(
EXPECTED_TITLE,
"fixture should contain an auto-title turn",
"default local fallback title should be derived from the prompt",
).not.toBe("");
expect(
EXPECTED_SUGGESTION,

View file

@ -6,9 +6,9 @@ import { expect, test } from "@playwright/test";
* RECORD driver (Plan A): drive the real frontend through the write/read-file
* scenario against the real-model gateway. The gateway captures every model
* call to DEERFLOW_RECORD_OUT; this just needs to drive the flow and wait until
* the captures stop arriving (main turns + in-graph title + follow-up
* suggestions all fired). It asserts nothing about content it produces the
* fixture, it doesn't verify it.
* the captures stop arriving (main turns + follow-up suggestions all fired;
* the default auto-title is local state). It asserts nothing about content
* it produces the fixture, it doesn't verify it.
*/
const APP = "http://localhost:3000";
const SCENARIO = "write_read_file";