deer-flow/backend/docs/TITLE_GENERATION_IMPLEMENTATION.md
xiawiie 2e15e3fe0d
Some checks are pending
Backend Blocking IO / backend-blocking-io (push) Waiting to run
Unit Tests / backend-unit-tests (push) Waiting to run
E2E Tests / e2e-tests (push) Waiting to run
Frontend Unit Tests / frontend-unit-tests (push) Waiting to run
Lint Check / lint-backend (push) Waiting to run
Lint Check / lint-frontend (push) Waiting to run
Replay E2E (front-back contract) / Layer 1 — backend golden (no API key) (push) Waiting to run
Replay E2E (front-back contract) / Layer 2 — full-stack render (no API key) (push) Waiting to run
fix: generate fallback title for interrupted first-turn runs (#3874)
* fix: generate title for interrupted first turn

* test(title): cover partial-exchange + dict-form messages

Harden the interrupted-run fallback path added in 19fc34fd:

- TitleMiddleware._should_generate_title now accepts a lone first-turn
  user message when allow_partial_exchange=True, so the worker can still
  derive a title if cancellation lands before any AI chunk is checkpointed.
- runtime/runs/worker._ensure_interrupted_title computes the next
  checkpoint step defensively (treat missing/non-int step as 0) and
  renames a shadowed ckpt_config local for readability.
- Add four unit tests in tests/test_title_middleware_core_logic.py:
  partial-exchange allows user-only, partial-exchange still respects an
  existing title, dict-form messages are recognized, and the sync
  fallback path derives a title from dict-form messages — matching what
  channel_values stores in the checkpoint.

Refs #3859.

* fix: persist interrupted-title via channel_versions bump

Address PR #3874 review feedback: ``_ensure_interrupted_title`` previously
called ``aput(..., new_versions={})``. LangGraph's DB-backed savers
(``PostgresSaver`` and the v4 ``SqliteSaver`` blob layout) strip inline
``channel_values`` from ``put`` and only persist blobs for channels named
in ``new_versions`` — so the fallback ``title`` channel was dropped on
read-back and ``threads_meta.display_name`` stayed ``"Untitled"`` after
refresh on those backends. The original in-memory e2e passed because
``InMemorySaver`` keeps the inline snapshot verbatim.

Fix mirrors ``_rollback_to_pre_run_checkpoint`` in the same file: bump
``channel_versions["title"]`` (via ``checkpointer.get_next_version`` when
available, else int/string fallbacks), persist the new version on the
checkpoint, and declare it in ``new_versions`` so the DB savers actually
write the blob.

Regression coverage in ``tests/test_run_worker_rollback.py``:

- ``test_ensure_interrupted_title_bumps_channel_version_and_declares_it_in_new_versions``
  — exact ``aput`` invariants: ``new_versions == {"title": 1}``, the
  written checkpoint's ``channel_versions["title"]`` is bumped, and the
  pre-existing ``messages`` version is preserved.
- ``test_ensure_interrupted_title_bumps_existing_string_version`` —
  string-shaped prior version (some savers use UUID-style versions);
  bumped value must differ from the prior, no overwrite-in-place.
- ``test_ensure_interrupted_title_skips_when_title_already_set`` — title
  short-circuit; no extra ``aput``.
- ``test_ensure_interrupted_title_returns_none_when_no_checkpoint`` —
  no checkpoint yet; returns ``None`` without writing.
- ``test_ensure_interrupted_title_round_trip_with_real_sqlite_checkpointer``
  — full round-trip against a real ``AsyncSqliteSaver`` on a disk-backed
  DB, then closes and re-opens the saver to simulate a fresh
  connection. The fallback title must still be present on the second
  ``aget_tuple``. This is the exact scenario the review flagged.

Validated locally with the full backend suite: 5195 passed, 18 skipped.

Refs #3859. Addresses review on #3874.

* test(worker, title): harden interrupted-title fallback for every saver

Defensive coverage on top of the channel_versions fix (commit 05253957),
addressing edge cases surfaced during a second-pass review of #3874.

Worker:
- Extract version bump into ``_bump_channel_version(checkpointer, current)``
  with explicit fallbacks for int / float / numeric-string / UUID-shaped
  string / None / bool, AND a wrap-around defense when the saver's
  ``get_next_version`` raises or returns an unchanged value. The
  invariant is: returned version MUST differ from the prior. Without
  this, a saver bug (or a custom backend) could leave
  ``new_versions={"title": v}`` no-op on DB savers — the very class of
  bug the original review pointed out.

Title middleware:
- Coerce ``state.get("messages")`` from ``None`` to ``[]`` on both
  ``_should_generate_title`` and ``_build_title_prompt``. A
  partially-initialized checkpoint can carry ``messages=None`` on the
  channel_values channel (the worker reads raw channel_values, not
  BaseMessages), and the default kwarg only protects against a missing
  key. Repro: ``TypeError: 'NoneType' object is not iterable`` from
  the next() generator — confirmed by reverting the fix and watching
  ``test_*_handles_none_messages_channel`` go red.

Tests (TDD-verified red→green for the new asserts):
- ``test_run_worker_rollback.py``:
  * ``_bump_channel_version`` — 8 tests covering every version type
    (int, float, numeric string, UUID-style string, None, bool) and
    every saver-side fault mode (no ``get_next_version`` / raising /
    stuck on identity).
  * ``test_ensure_interrupted_title_*`` — 5 additional helper
    boundary tests: title.enabled=false short-circuit; empty
    messages list; messages=None; aput-error propagation (helper
    contract: caller swallows, not the helper); idempotency on a
    real InMemorySaver across two invocations.
  * ``test_ensure_interrupted_title_preserves_non_title_channel_versions``
    — pins that ``new_versions`` only contains ``"title"`` and that
    other channels' versions are untouched (regression anchor for a
    sloppier draft that bumped every channel).
  * ``test_worker_finally_block_swallows_helper_exceptions`` — pins
    the integration contract: even if the helper raises, the worker's
    threads_meta status sync still runs and ``publish_end`` is still
    awaited so the SSE stream closes cleanly.

- ``test_title_middleware_core_logic.py``:
  * 4 additional tests: ``messages=None`` on both
    ``_should_generate_title`` and ``_build_title_prompt``; the
    ``role: user`` / ``role: assistant`` (OpenAI-style) dict
    normalization; partial-exchange path with a dict-form message.

Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
  → 5215 passed, 18 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Red/green TDD verification: temporarily reverted the
  ``new_versions={}`` fix → 4 new tests went red as expected; restored
  and the suite is green again. Same red/green dance for the
  ``messages=None`` coercion.

Refs #3859. Addresses second-pass review on #3874.

* fix(title): ignore dict context reminders in fallback

* fix(worker): link interrupted-title checkpoint to its parent

The title-bump checkpoint written by ``_ensure_interrupted_title`` was
landing without a ``parent_checkpoint_id`` — a real orphan in the
LangGraph history graph. Reproduction (disk-backed AsyncSqliteSaver):

  [seed] checkpoint_id = 1f173dbc...
  [helper] wrote title = "Why is the sky blue?"
  [issue 1] new checkpoint = 1f173dbc..., parent = None
  [issue 1] is new checkpoint orphaned? True

Root cause: ``_ensure_interrupted_title`` built ``write_config`` as
``{"thread_id": ..., "checkpoint_ns": ...}`` only. ``BaseCheckpointSaver``
implementations read ``configurable.checkpoint_id`` from that config as
the *parent* id when inserting (see ``langgraph/checkpoint/sqlite/aio.py``
``aput``: ``config["configurable"].get("checkpoint_id")`` becomes the
``parent_checkpoint_id`` column). With no value, the saver writes NULL —
the new checkpoint is a tree root.

Consequences:
- Any future LangGraph ``runs.resume_from`` / time-travel feature has no
  backward edge to walk past the title-bump.
- History-visualization UIs built on ``alist()`` render the title-bump as
  a sibling of the prior checkpoint, not its descendant.

Fix: read ``checkpoint_id`` off the tuple's own config and thread it into
``write_config["configurable"]["checkpoint_id"]`` before calling ``aput``,
the same pattern every middleware-driven write uses.

Three new regression tests against real ``AsyncSqliteSaver`` (disk-backed,
fresh connections so we exercise the on-disk read path):

- ``test_ensure_interrupted_title_links_new_checkpoint_to_its_parent`` —
  asserts ``latest.parent_config["configurable"]["checkpoint_id"]`` equals
  the seeded checkpoint id. TDD red-green verified: reverting the fix
  flips this test red with ``AssertionError: title-bump checkpoint must
  have a parent_config``.
- ``test_ensure_interrupted_title_appears_in_history_with_audit_marker``
  — pins the audit contract: the title-bump entry in ``alist()`` carries
  ``metadata.source == "update"`` and ``metadata.writes`` contains
  ``runtime_interrupt_title``. This is a deliberate design choice — we
  do NOT hide the entry from history (audit trail belongs in the saver),
  but its source and writes marker MUST be unambiguous so UIs/tools can
  identify it.
- ``test_ensure_interrupted_title_survives_immediate_next_turn`` —
  cancel → immediate user follow-up scenario. Simulates the agent's next
  turn appending a (user, ai) pair without touching the title channel,
  then opens a fresh saver and verifies the title is still present after
  the next-turn checkpoint write. Pins the channel-version-blob
  invariant established by commit 05253957 — without the
  ``new_versions={"title": v}`` declaration there, the title blob would
  vanish from the DB and this test would read back ``None``.

Verification:
- ``PYTHONPATH=. uv run pytest tests/ -x --ignore=tests/blocking_io -q``
  → 5222 passed, 15 skipped.
- ``ruff check`` + ``ruff format --check`` clean on every touched file.
- Reproduction script confirms ``parent_checkpoint_id`` is now non-null
  and the next-turn read-back preserves the fallback title.

Refs #3859.

* Revert "fix(worker): link interrupted-title checkpoint to its parent"

This reverts commit c763ed9334781db1acdce0f5f33d663d8d5f80ff.

* test: trim over-engineered test coverage

Reduce review surface area on PR #3874 by dropping defensive tests that
don't pin a real invariant. After self-review:

- ``_bump_channel_version``: 8 tests → 2 (happy path + saver-error
  fallback). Dropped float / bool / numeric-string / UUID-string /
  missing-get-next-version / stuck-get-next-version branches — those
  are speculative scaffolding for savers we don't ship.
- ``_ensure_interrupted_title``: dropped ``returns_none_when_title_disabled``,
  ``returns_none_with_no_user_message``, ``returns_none_when_no_checkpoint``
  — boundary guards already exercised by the e2e test and the
  ``handles_none_messages_channel`` regression anchor.

Net: -107 lines of test code. Remaining coverage still pins every
red-green-verified invariant (channel_versions bump, string-version
bump, idempotency, sqlite round-trip, non-title channel preservation,
aput-error contract, worker finally swallowing, partial-exchange).

Verification: 5209 passed, 15 skipped.

* fix: harden interrupted title finalization

* fix: serialize interrupted title finalization

* fix: preserve interrupt semantics during title finalization

* fix: preserve delayed interrupted title recovery

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-07-02 00:13:01 +08:00

6.9 KiB
Raw Blame History

自动 Title 生成功能实现总结

已完成的工作

1. 核心实现文件

packages/harness/deerflow/agents/thread_state.py

  • 添加 title: str | None = None 字段到 ThreadState

packages/harness/deerflow/config/title_config.py (新建)

  • 创建 TitleConfig 配置类
  • 支持配置enabled, max_words, max_chars, model_name, prompt_template
  • 提供 get_title_config()set_title_config() 函数
  • 提供 load_title_config_from_dict() 从配置文件加载

packages/harness/deerflow/agents/middlewares/title_middleware.py (新建)

  • 创建 TitleMiddleware
  • 实现 _should_generate_title() 检查是否需要生成
  • 默认使用本地 fallback 生成标题,避免流式回复结束前等待额外 LLM 调用;显式配置 model_name 时可使用 LLM 标题
  • 实现 after_model() / aafter_model() 钩子,在首次对话后自动触发
  • 包含 fallback 策略LLM 未配置或失败时使用用户消息前几个字符)

packages/harness/deerflow/config/app_config.py

  • 导入 load_title_config_from_dict
  • from_file() 中加载 title 配置

packages/harness/deerflow/agents/lead_agent/agent.py

  • 导入 TitleMiddleware
  • 注册到 middleware 列表:[SandboxMiddleware(), TitleMiddleware()]

2. 配置文件

config.yaml

  • 添加 title 配置段:
title:
  enabled: true
  max_words: 6
  max_chars: 60
  model_name: null  # null = 快速本地 fallback填模型名才启用 LLM 标题

3. 文档

docs/AUTO_TITLE_GENERATION.md (新建)

  • 完整的功能说明文档
  • 实现方式和架构设计
  • 配置说明
  • 客户端使用示例TypeScript
  • 工作流程图Mermaid
  • 故障排查指南
  • State vs Metadata 对比

TODO.md

  • 添加功能完成记录

4. 测试

tests/test_title_generation.py (新建)

  • 配置类测试
  • Middleware 初始化测试
  • TODO: 集成测试(需要 mock Runtime

🎯 核心设计决策

为什么使用 State 而非 Metadata

方面 State ( 采用) Metadata ( 未采用)
持久化 自动(通过 checkpointer 取决于实现,不可靠
版本控制 支持时间旅行 不支持
类型安全 TypedDict 定义 任意字典
标准化 LangGraph 核心机制 扩展功能

工作流程

用户发送首条消息
  ↓
Agent 处理并返回回复
  ↓
TitleMiddleware.after_model()/aafter_model() 触发
  ↓
检查:是否首次对话?是否已有 title
  ↓
默认从首条用户消息生成本地 fallback title
  ↓
如果显式配置 title.model_name才调用 LLM 生成更精炼的 title
  ↓
返回 {"title": "..."} 更新 state
  ↓
Checkpointer 自动持久化(如果配置了)
  ↓
客户端从 state.values.title 读取

📋 使用指南

后端配置

  1. 启用/禁用功能
# config.yaml
title:
  enabled: true  # 设为 false 禁用
  1. 自定义配置
title:
  enabled: true
  max_words: 8      # 标题最多 8 个词
  max_chars: 80     # 标题最多 80 个字符
  model_name: null  # null = 快速本地 fallback填模型名才启用 LLM 标题
  1. 配置持久化(可选)

如果需要在本地开发时持久化 title

# checkpointer.py
from langgraph.checkpoint.sqlite import SqliteSaver

checkpointer = SqliteSaver.from_conn_string("deerflow.db")
// langgraph.json
{
  "graphs": {
    "lead_agent": "deerflow.agents:lead_agent"
  },
  "checkpointer": "checkpointer:checkpointer"
}

客户端使用

// 获取 thread title
const state = await client.threads.getState(threadId);
const title = state.values.title || "New Conversation";

// 显示在对话列表
<li>{title}</li>

⚠️ 注意Title 在 state.values.title,而非 thread.metadata.title


🧪 测试

# 运行测试
pytest tests/test_title_generation.py -v

# 运行所有测试
pytest

🔍 故障排查

Title 没有生成?

  1. 检查配置:title.enabled = true
  2. 确认是首次对话1 个用户消息 + 1 个助手回复)
  3. 如果显式配置了 title.model_name,检查标题模型是否可用;未配置时会走本地 fallback

Title 生成但看不到?

  1. 确认读取位置:state.values.title(不是 thread.metadata.title
  2. 检查 API 响应是否包含 title
  3. 重新获取 state

Title 重启后丢失?

  1. 本地开发需要配置 checkpointer
  2. LangGraph Platform 会自动持久化
  3. 检查数据库确认 checkpointer 工作正常

中断首轮后仍显示默认标题?

  1. runtime/runs/worker.py 会在 interrupted-run cleanup 中保持 run 处于 finalizing 状态,避免同线程新 run 在 fallback title 写入期间覆盖 checkpoint
  2. 如果取消发生在可用 checkpoint 写入前worker 会使用本次 graph_input 中的首条用户消息生成本地 fallback title
  3. fallback title 写入前会重新读取 latest checkpoint如果同线程状态已经前进只对最新 snapshot 做 title-only 更新,避免旧消息重新成为 latest

📊 性能影响

  • 默认延迟:默认 title.model_name: null 不会发起额外 LLM 调用,仅从首条用户消息生成本地 fallback title
  • 显式 LLM 标题延迟:只有配置 title.model_name 时,首轮回复后才会等待一次标题模型调用
  • 并发安全:在 after_model() / aafter_model() 中更新 state不需要客户端额外请求
  • 资源消耗:每个 thread 只生成一次

优化建议

  1. 默认保持 model_name: null,避免流式回复结束前的额外 LLM 等待
  2. 如需更精炼标题,再显式配置较快的标题模型
  3. 减少 max_wordsmax_chars,并让 prompt 保持简洁

🚀 下一步

  • 补充 prompt template 的集成测试
  • 支持多语言 title 生成
  • 添加 title 重新生成功能
  • 监控 title 生成成功率和延迟

📚 相关资源


实现完成时间: 2026-01-14