diff --git a/frontend/src/content/en/application/agents-and-threads.mdx b/frontend/src/content/en/application/agents-and-threads.mdx index 0b2662581..3bf3a2baf 100644 --- a/frontend/src/content/en/application/agents-and-threads.mdx +++ b/frontend/src/content/en/application/agents-and-threads.mdx @@ -82,38 +82,37 @@ Each user message triggers an agent run. The run streams tokens and tool calls b #### Checkpoint -If a checkpointer is configured, the thread state is persisted after each turn. This means the conversation survives process restarts. +The configured persistence backend stores thread state after each turn. This means the conversation survives process restarts when a persistent backend is used. #### Resume -Opening a thread from the sidebar loads its full history from the checkpointer. The agent picks up from where it left off. +Opening a thread from the sidebar loads its full history from persisted state. The agent picks up from where it left off. -### Checkpointer configuration +### Thread persistence configuration -The checkpointer controls how thread state is persisted: +The `database` section controls the default persistence backend for thread state and application data: ```yaml -# In-memory (default if omitted — state lost on restart) -# checkpointer: -# type: memory +# SQLite (local development and single-user deployments) +database: + backend: sqlite + sqlite_dir: .deer-flow/data -# SQLite (survives restarts, single-process) -checkpointer: - type: sqlite - connection_string: checkpoints.db - -# PostgreSQL (multi-process, production) -# checkpointer: -# type: postgres -# connection_string: postgresql://user:password@localhost:5432/deerflow +# Postgres (production and multi-user deployments) +# database: +# backend: postgres +# postgres_url: $DATABASE_URL +# +# run_events: +# backend: db ``` - The Gateway embedded runtime uses the checkpointer setting in - config.yaml. The same setting is also used by - DeerFlowClient in direct Python integrations. + The legacy checkpointer section is still accepted for LangGraph + state compatibility and takes precedence when configured. Prefer{" "} + database for new deployments. ### Thread data storage diff --git a/frontend/src/content/en/application/configuration.mdx b/frontend/src/content/en/application/configuration.mdx index b94b252bf..603f37806 100644 --- a/frontend/src/content/en/application/configuration.mdx +++ b/frontend/src/content/en/application/configuration.mdx @@ -182,33 +182,41 @@ tools: - use: deerflow.sandbox.tools:bash_tool ``` -### Thread state persistence (checkpointer) +### Database backend -By default, DeerFlow uses an SQLite checkpointer for thread state persistence: +DeerFlow uses the `database` section for both LangGraph checkpoint data and application data such as runs, feedback, and thread metadata. + +By default, DeerFlow uses SQLite for local, single-node persistence: ```yaml -checkpointer: - type: sqlite - connection_string: checkpoints.db # stored in backend/.deer-flow/ +database: + backend: sqlite + sqlite_dir: .deer-flow/data ``` -For production deployments with multiple processes: +SQLite mode stores everything in one `deerflow.db` file. This is fine for development or single-user deployments, but concurrent production traffic can hit SQLite's single-writer limit and raise `sqlite3.OperationalError: database is locked`. + +For production or multi-user deployments, use Postgres: ```yaml -checkpointer: - type: postgres - connection_string: postgresql://user:password@localhost:5432/deerflow +database: + backend: postgres + postgres_url: $DATABASE_URL + +run_events: + backend: db ``` -Install PostgreSQL support: `cd backend && uv add langgraph-checkpoint-postgres psycopg[binary] psycopg-pool` +Set `DATABASE_URL` in your environment, for example `postgresql://user:password@localhost:5432/deerflow`. -For in-memory only (state lost on restart): +Install PostgreSQL support for local runs: -```yaml -checkpointer: - type: memory +```bash +cd backend && uv sync --all-packages --extra postgres ``` +For Docker or scripted starts, set `UV_EXTRAS=postgres` before installing or building. The legacy standalone `checkpointer` section is still accepted for compatibility, but prefer `database` for new deployments. + ### Memory ```yaml diff --git a/frontend/src/content/en/application/deployment-guide.mdx b/frontend/src/content/en/application/deployment-guide.mdx index 969884a15..5f66499da 100644 --- a/frontend/src/content/en/application/deployment-guide.mdx +++ b/frontend/src/content/en/application/deployment-guide.mdx @@ -119,6 +119,32 @@ For production, use a named volume or a Persistent Volume Claim (PVC) instead of For any deployment with more than one concurrent user, use a container-based sandbox to prevent users from interfering with each other's execution environments. +### Database backend + +SQLite is convenient for local development and single-user deployments, but it is not a production backend for concurrent users. + +In SQLite mode, DeerFlow stores LangGraph checkpoint data and application data in one `deerflow.db` file. WAL mode allows concurrent reads, but SQLite still permits only one writer at a time. When several users run agents at once, checkpoint writes and application writes can contend for the same file and raise `sqlite3.OperationalError: database is locked`. + +For production or any multi-user deployment, use Postgres: + +```bash +# .env +DATABASE_URL=postgresql://user:password@postgres:5432/deerflow +UV_EXTRAS=postgres +``` + +```yaml +# config.yaml +database: + backend: postgres + postgres_url: $DATABASE_URL + +run_events: + backend: db +``` + +Keep SQLite deployments to local, single-node, or short-lived evaluation environments. + ### K8s Provisioner setup The provisioner manages sandbox Pods in a Kubernetes cluster. It is included in `docker/docker-compose-dev.yaml`. diff --git a/frontend/src/content/en/application/operations-and-troubleshooting.mdx b/frontend/src/content/en/application/operations-and-troubleshooting.mdx index 52230bd97..26fae89f7 100644 --- a/frontend/src/content/en/application/operations-and-troubleshooting.mdx +++ b/frontend/src/content/en/application/operations-and-troubleshooting.mdx @@ -105,6 +105,25 @@ If file tools (`read_file`, `ls`, `bash`) fail with permission or path errors: --- +### SQLite reports `database is locked` + +This usually means SQLite is handling concurrent writes from multiple users or long-running agent runs. SQLite supports concurrent reads, but only one writer can hold the database lock at a time. In DeerFlow, checkpoint writes and application writes share the same SQLite database file. + +For production or multi-user deployments, switch to Postgres: + +```yaml +database: + backend: postgres + postgres_url: $DATABASE_URL + +run_events: + backend: db +``` + +SQLite should be kept for local development, single-user deployments, or short-lived evaluation environments. + +--- + ### K8s Provisioner not connecting ``` @@ -136,23 +155,15 @@ If MCP tools appear in `extensions_config.json` but are not available in the age ## Data backup -Thread data and memory are stored under `backend/.deer-flow/`: +Back up the storage backends you use: -``` -backend/.deer-flow/ - memory.json # global agent memory - agents/ # per-agent memory - threads/ # thread working directories - {thread_id}/ - user-data/ - uploads/ - outputs/ - checkpoints.db # SQLite checkpoints (if configured) -``` +- `backend/.deer-flow/threads/` for thread working directories, uploads, and artifacts. +- `backend/.deer-flow/memory.json` and `backend/.deer-flow/agents/` for learned memory. +- `${database.sqlite_dir}/deerflow.db` when using `database.backend: sqlite` (default: `.deer-flow/data/deerflow.db`). +- Regular database dumps when using `database.backend: postgres`. +- The legacy `checkpointer.connection_string` file if your deployment still configures a standalone `checkpointer`. -Back up this entire directory to preserve conversation history, artifacts, and learned memory. - -In Docker deployments, the bind-mounted host path (`$DEER_FLOW_ROOT/backend/.deer-flow/`) is the source of truth — back up the host path. +In Docker deployments, back up the bind-mounted host paths rather than the container filesystem. ## Restarting services diff --git a/frontend/src/content/zh/application/agents-and-threads.mdx b/frontend/src/content/zh/application/agents-and-threads.mdx index 9074a9fe4..df3134657 100644 --- a/frontend/src/content/zh/application/agents-and-threads.mdx +++ b/frontend/src/content/zh/application/agents-and-threads.mdx @@ -72,29 +72,32 @@ curl -X POST http://localhost:8001/api/agents \ 每次你发送消息,LangGraph 从最新检查点恢复线程状态,运行 Lead Agent,并更新状态。流式事件在 Agent 工作时发送到浏览器。 -### 检查点和持久化 +### 线程持久化配置 -DeerFlow 在每次 Agent 轮次后自动保存线程状态(如果配置了检查点器)。这允许: +DeerFlow 在每次 Agent 轮次后自动保存线程状态。使用持久化后端时,这允许: - 在服务器重启后恢复线程。 - 在长时间间隔后继续对话。 - 在出现问题时重放或从特定时间点恢复。 -配置检查点器: +`database` 配置控制线程状态和应用数据的默认持久化后端: ```yaml -checkpointer: - type: sqlite - connection_string: .deer-flow/checkpoints.db +# SQLite(本地开发和单用户部署) +database: + backend: sqlite + sqlite_dir: .deer-flow/data + +# Postgres(生产环境和多用户部署) +# database: +# backend: postgres +# postgres_url: $DATABASE_URL +# +# run_events: +# backend: db ``` -对于生产高负载环境使用 Redis: - -```yaml -checkpointer: - type: redis - connection_string: redis://localhost:6379/0 -``` +旧的 `checkpointer` 配置仍然兼容,用于 LangGraph 状态持久化;如果配置了它,会优先于 `database` 的检查点设置。新的部署应优先使用 `database`。 ### 线程数据目录 diff --git a/frontend/src/content/zh/application/configuration.mdx b/frontend/src/content/zh/application/configuration.mdx index 2114a8d49..b9bb3cd25 100644 --- a/frontend/src/content/zh/application/configuration.mdx +++ b/frontend/src/content/zh/application/configuration.mdx @@ -147,19 +147,41 @@ skills: 技能可用性在 `extensions_config.json` 中管理(参见下方)。 -### 检查点(线程持久化) +### 数据库后端 + +DeerFlow 使用 `database` 配置同时管理 LangGraph 检查点数据和应用数据,例如运行记录、反馈和线程元数据。 + +默认使用 SQLite,适合本地开发和单节点单用户持久化: ```yaml -checkpointer: - type: sqlite - connection_string: .deer-flow/checkpoints.db - -# 或使用 Redis(高负载生产环境): -# checkpointer: -# type: redis -# connection_string: redis://localhost:6379/0 +database: + backend: sqlite + sqlite_dir: .deer-flow/data ``` +SQLite 模式会把数据写入同一个 `deerflow.db` 文件。它适合开发或单用户部署,但生产并发流量可能触发 SQLite 的单写者限制,出现 `sqlite3.OperationalError: database is locked`。 + +生产环境或多用户部署请使用 Postgres: + +```yaml +database: + backend: postgres + postgres_url: $DATABASE_URL + +run_events: + backend: db +``` + +在环境变量中设置 `DATABASE_URL`,例如 `postgresql://user:password@localhost:5432/deerflow`。 + +本地运行时安装 PostgreSQL 支持: + +```bash +cd backend && uv sync --all-packages --extra postgres +``` + +Docker 或脚本启动场景,在安装或构建前设置 `UV_EXTRAS=postgres`。旧的独立 `checkpointer` 配置仍然兼容,但新的部署应优先使用 `database`。 + ## 前端环境变量 前端通过 `.env.local`(本地开发)或 Docker Compose 环境中的环境变量配置。 diff --git a/frontend/src/content/zh/application/deployment-guide.mdx b/frontend/src/content/zh/application/deployment-guide.mdx index d5220168a..33fdebe8d 100644 --- a/frontend/src/content/zh/application/deployment-guide.mdx +++ b/frontend/src/content/zh/application/deployment-guide.mdx @@ -116,6 +116,32 @@ BETTER_AUTH_SECRET=your-secret-here-min-32-chars 对于有多个并发用户的任何部署,使用基于容器的沙箱,防止用户之间的执行环境相互干扰。 +### 数据库后端 + +SQLite 适合本地开发和单用户部署,但不适合作为多用户生产环境的数据库后端。 + +在 SQLite 模式下,DeerFlow 会把 LangGraph 检查点数据和应用数据写入同一个 `deerflow.db` 文件。WAL 模式可以支持并发读取,但 SQLite 同一时间仍然只有一个写入者。多个用户同时运行 Agent 时,检查点写入和应用写入会竞争同一个文件锁,可能触发 `sqlite3.OperationalError: database is locked`。 + +生产环境或任何多用户部署请使用 Postgres: + +```bash +# .env +DATABASE_URL=postgresql://user:password@postgres:5432/deerflow +UV_EXTRAS=postgres +``` + +```yaml +# config.yaml +database: + backend: postgres + postgres_url: $DATABASE_URL + +run_events: + backend: db +``` + +SQLite 部署应限制在本地、单节点或短期评估场景。 + ### K8s Provisioner 配置 diff --git a/frontend/src/content/zh/application/operations-and-troubleshooting.mdx b/frontend/src/content/zh/application/operations-and-troubleshooting.mdx index 43b8b463f..c7bfe783d 100644 --- a/frontend/src/content/zh/application/operations-and-troubleshooting.mdx +++ b/frontend/src/content/zh/application/operations-and-troubleshooting.mdx @@ -94,6 +94,27 @@ ls -la backend/.deer-flow/threads/ --- +### SQLite 报 `database is locked` + +**症状**:Docker 或团队部署中,多个用户同时使用时出现 `sqlite3.OperationalError: database is locked`。 + +**原因**:SQLite 支持并发读取,但同一时间只有一个写入者。DeerFlow 的检查点写入和应用写入会共享同一个 SQLite 数据库文件,多用户并发时可能竞争文件锁。 + +**解决**:生产环境或多用户部署切换到 Postgres: + +```yaml +database: + backend: postgres + postgres_url: $DATABASE_URL + +run_events: + backend: db +``` + +SQLite 应只用于本地开发、单用户部署或短期评估环境。 + +--- + ### 前端构建失败 **症状**:`make install` 或前端构建步骤失败,提示 `BETTER_AUTH_SECRET` 错误。 @@ -142,14 +163,16 @@ grep -i "mcp\|timeout" logs/gateway.log | tail -20 ## 数据备份 -DeerFlow 将持久化数据存储在: +根据你启用的存储后端备份: -- **线程数据**:`backend/.deer-flow/threads/` — 每个线程的上传文件、输出和工作区文件 -- **检查点**:取决于检查点器配置(SQLite:`backend/.deer-flow/checkpoints.db`,Redis:外部存储) -- **记忆**:`backend/.deer-flow/memory.json`(以及 `agents/*/memory.json`) -- **自定义 Agent 配置**:`backend/agents/*/config.yaml` +- `backend/.deer-flow/threads/`:线程工作目录、上传文件和产出物。 +- `backend/.deer-flow/memory.json` 和 `backend/.deer-flow/agents/`:全局和 Agent 记忆。 +- `${database.sqlite_dir}/deerflow.db`:当使用 `database.backend: sqlite` 时备份,默认路径为 `.deer-flow/data/deerflow.db`。 +- Postgres 数据库备份:当使用 `database.backend: postgres` 时定期导出。 +- 旧的 `checkpointer.connection_string` 文件:如果你的部署仍配置独立 `checkpointer`。 +- `backend/agents/*/config.yaml`:自定义 Agent 配置。 -对于生产部署,定期备份这些目录。Docker 部署中,确保这些目录绑定挂载到持久卷,而不是容器内部。 +Docker 部署中,备份绑定挂载的宿主机路径,而不是容器内部文件系统。 ## 停止和重启服务