mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-09 16:08:31 +00:00
docs: add production database guidance (#3830)
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
This commit is contained in:
ly-wang19
GitHub
parent
c9a5f23e7b
commit
ddca86411b
8 changed files with 194 additions and 76 deletions
|
|
@ -82,38 +82,37 @@ Each user message triggers an agent run. The run streams tokens and tool calls b
|
|||
|
||||
#### Checkpoint
|
||||
|
||||
If a checkpointer is configured, the thread state is persisted after each turn. This means the conversation survives process restarts.
|
||||
The configured persistence backend stores thread state after each turn. This means the conversation survives process restarts when a persistent backend is used.
|
||||
|
||||
#### Resume
|
||||
|
||||
Opening a thread from the sidebar loads its full history from the checkpointer. The agent picks up from where it left off.
|
||||
Opening a thread from the sidebar loads its full history from persisted state. The agent picks up from where it left off.
|
||||
|
||||
</Steps>
|
||||
|
||||
### Checkpointer configuration
|
||||
### Thread persistence configuration
|
||||
|
||||
The checkpointer controls how thread state is persisted:
|
||||
The `database` section controls the default persistence backend for thread state and application data:
|
||||
|
||||
```yaml
|
||||
# In-memory (default if omitted — state lost on restart)
|
||||
# checkpointer:
|
||||
# type: memory
|
||||
# SQLite (local development and single-user deployments)
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
|
||||
# SQLite (survives restarts, single-process)
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: checkpoints.db
|
||||
|
||||
# PostgreSQL (multi-process, production)
|
||||
# checkpointer:
|
||||
# type: postgres
|
||||
# connection_string: postgresql://user:password@localhost:5432/deerflow
|
||||
# Postgres (production and multi-user deployments)
|
||||
# database:
|
||||
# backend: postgres
|
||||
# postgres_url: $DATABASE_URL
|
||||
#
|
||||
# run_events:
|
||||
# backend: db
|
||||
```
|
||||
|
||||
<Callout type="info">
|
||||
The Gateway embedded runtime uses the <code>checkpointer</code> setting in
|
||||
<code>config.yaml</code>. The same setting is also used by
|
||||
<code>DeerFlowClient</code> in direct Python integrations.
|
||||
The legacy <code>checkpointer</code> section is still accepted for LangGraph
|
||||
state compatibility and takes precedence when configured. Prefer{" "}
|
||||
<code>database</code> for new deployments.
|
||||
</Callout>
|
||||
|
||||
### Thread data storage
|
||||
|
|
|
|||
|
|
@ -182,33 +182,41 @@ tools:
|
|||
- use: deerflow.sandbox.tools:bash_tool
|
||||
```
|
||||
|
||||
### Thread state persistence (checkpointer)
|
||||
### Database backend
|
||||
|
||||
By default, DeerFlow uses an SQLite checkpointer for thread state persistence:
|
||||
DeerFlow uses the `database` section for both LangGraph checkpoint data and application data such as runs, feedback, and thread metadata.
|
||||
|
||||
By default, DeerFlow uses SQLite for local, single-node persistence:
|
||||
|
||||
```yaml
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: checkpoints.db # stored in backend/.deer-flow/
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
```
|
||||
|
||||
For production deployments with multiple processes:
|
||||
SQLite mode stores everything in one `deerflow.db` file. This is fine for development or single-user deployments, but concurrent production traffic can hit SQLite's single-writer limit and raise `sqlite3.OperationalError: database is locked`.
|
||||
|
||||
For production or multi-user deployments, use Postgres:
|
||||
|
||||
```yaml
|
||||
checkpointer:
|
||||
type: postgres
|
||||
connection_string: postgresql://user:password@localhost:5432/deerflow
|
||||
database:
|
||||
backend: postgres
|
||||
postgres_url: $DATABASE_URL
|
||||
|
||||
run_events:
|
||||
backend: db
|
||||
```
|
||||
|
||||
Install PostgreSQL support: `cd backend && uv add langgraph-checkpoint-postgres psycopg[binary] psycopg-pool`
|
||||
Set `DATABASE_URL` in your environment, for example `postgresql://user:password@localhost:5432/deerflow`.
|
||||
|
||||
For in-memory only (state lost on restart):
|
||||
Install PostgreSQL support for local runs:
|
||||
|
||||
```yaml
|
||||
checkpointer:
|
||||
type: memory
|
||||
```bash
|
||||
cd backend && uv sync --all-packages --extra postgres
|
||||
```
|
||||
|
||||
For Docker or scripted starts, set `UV_EXTRAS=postgres` before installing or building. The legacy standalone `checkpointer` section is still accepted for compatibility, but prefer `database` for new deployments.
|
||||
|
||||
### Memory
|
||||
|
||||
```yaml
|
||||
|
|
|
|||
|
|
@ -119,6 +119,32 @@ For production, use a named volume or a Persistent Volume Claim (PVC) instead of
|
|||
|
||||
For any deployment with more than one concurrent user, use a container-based sandbox to prevent users from interfering with each other's execution environments.
|
||||
|
||||
### Database backend
|
||||
|
||||
SQLite is convenient for local development and single-user deployments, but it is not a production backend for concurrent users.
|
||||
|
||||
In SQLite mode, DeerFlow stores LangGraph checkpoint data and application data in one `deerflow.db` file. WAL mode allows concurrent reads, but SQLite still permits only one writer at a time. When several users run agents at once, checkpoint writes and application writes can contend for the same file and raise `sqlite3.OperationalError: database is locked`.
|
||||
|
||||
For production or any multi-user deployment, use Postgres:
|
||||
|
||||
```bash
|
||||
# .env
|
||||
DATABASE_URL=postgresql://user:password@postgres:5432/deerflow
|
||||
UV_EXTRAS=postgres
|
||||
```
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
database:
|
||||
backend: postgres
|
||||
postgres_url: $DATABASE_URL
|
||||
|
||||
run_events:
|
||||
backend: db
|
||||
```
|
||||
|
||||
Keep SQLite deployments to local, single-node, or short-lived evaluation environments.
|
||||
|
||||
### K8s Provisioner setup
|
||||
|
||||
The provisioner manages sandbox Pods in a Kubernetes cluster. It is included in `docker/docker-compose-dev.yaml`.
|
||||
|
|
|
|||
|
|
@ -105,6 +105,25 @@ If file tools (`read_file`, `ls`, `bash`) fail with permission or path errors:
|
|||
|
||||
---
|
||||
|
||||
### SQLite reports `database is locked`
|
||||
|
||||
This usually means SQLite is handling concurrent writes from multiple users or long-running agent runs. SQLite supports concurrent reads, but only one writer can hold the database lock at a time. In DeerFlow, checkpoint writes and application writes share the same SQLite database file.
|
||||
|
||||
For production or multi-user deployments, switch to Postgres:
|
||||
|
||||
```yaml
|
||||
database:
|
||||
backend: postgres
|
||||
postgres_url: $DATABASE_URL
|
||||
|
||||
run_events:
|
||||
backend: db
|
||||
```
|
||||
|
||||
SQLite should be kept for local development, single-user deployments, or short-lived evaluation environments.
|
||||
|
||||
---
|
||||
|
||||
### K8s Provisioner not connecting
|
||||
|
||||
```
|
||||
|
|
@ -136,23 +155,15 @@ If MCP tools appear in `extensions_config.json` but are not available in the age
|
|||
|
||||
## Data backup
|
||||
|
||||
Thread data and memory are stored under `backend/.deer-flow/`:
|
||||
Back up the storage backends you use:
|
||||
|
||||
```
|
||||
backend/.deer-flow/
|
||||
memory.json # global agent memory
|
||||
agents/ # per-agent memory
|
||||
threads/ # thread working directories
|
||||
{thread_id}/
|
||||
user-data/
|
||||
uploads/
|
||||
outputs/
|
||||
checkpoints.db # SQLite checkpoints (if configured)
|
||||
```
|
||||
- `backend/.deer-flow/threads/` for thread working directories, uploads, and artifacts.
|
||||
- `backend/.deer-flow/memory.json` and `backend/.deer-flow/agents/` for learned memory.
|
||||
- `${database.sqlite_dir}/deerflow.db` when using `database.backend: sqlite` (default: `.deer-flow/data/deerflow.db`).
|
||||
- Regular database dumps when using `database.backend: postgres`.
|
||||
- The legacy `checkpointer.connection_string` file if your deployment still configures a standalone `checkpointer`.
|
||||
|
||||
Back up this entire directory to preserve conversation history, artifacts, and learned memory.
|
||||
|
||||
In Docker deployments, the bind-mounted host path (`$DEER_FLOW_ROOT/backend/.deer-flow/`) is the source of truth — back up the host path.
|
||||
In Docker deployments, back up the bind-mounted host paths rather than the container filesystem.
|
||||
|
||||
## Restarting services
|
||||
|
||||
|
|
|
|||
|
|
@ -72,29 +72,32 @@ curl -X POST http://localhost:8001/api/agents \
|
|||
|
||||
每次你发送消息，LangGraph 从最新检查点恢复线程状态，运行 Lead Agent，并更新状态。流式事件在 Agent 工作时发送到浏览器。
|
||||
|
||||
### 检查点和持久化
|
||||
### 线程持久化配置
|
||||
|
||||
DeerFlow 在每次 Agent 轮次后自动保存线程状态（如果配置了检查点器）。这允许：
|
||||
DeerFlow 在每次 Agent 轮次后自动保存线程状态。使用持久化后端时，这允许：
|
||||
|
||||
- 在服务器重启后恢复线程。
|
||||
- 在长时间间隔后继续对话。
|
||||
- 在出现问题时重放或从特定时间点恢复。
|
||||
|
||||
配置检查点器：
|
||||
`database` 配置控制线程状态和应用数据的默认持久化后端：
|
||||
|
||||
```yaml
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: .deer-flow/checkpoints.db
|
||||
# SQLite（本地开发和单用户部署）
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
|
||||
# Postgres（生产环境和多用户部署）
|
||||
# database:
|
||||
# backend: postgres
|
||||
# postgres_url: $DATABASE_URL
|
||||
#
|
||||
# run_events:
|
||||
# backend: db
|
||||
```
|
||||
|
||||
对于生产高负载环境使用 Redis：
|
||||
|
||||
```yaml
|
||||
checkpointer:
|
||||
type: redis
|
||||
connection_string: redis://localhost:6379/0
|
||||
```
|
||||
旧的 `checkpointer` 配置仍然兼容，用于 LangGraph 状态持久化；如果配置了它，会优先于 `database` 的检查点设置。新的部署应优先使用 `database`。
|
||||
|
||||
### 线程数据目录
|
||||
|
||||
|
|
|
|||
|
|
@ -147,19 +147,41 @@ skills:
|
|||
|
||||
技能可用性在 `extensions_config.json` 中管理（参见下方）。
|
||||
|
||||
### 检查点（线程持久化）
|
||||
### 数据库后端
|
||||
|
||||
DeerFlow 使用 `database` 配置同时管理 LangGraph 检查点数据和应用数据，例如运行记录、反馈和线程元数据。
|
||||
|
||||
默认使用 SQLite，适合本地开发和单节点单用户持久化：
|
||||
|
||||
```yaml
|
||||
checkpointer:
|
||||
type: sqlite
|
||||
connection_string: .deer-flow/checkpoints.db
|
||||
|
||||
# 或使用 Redis（高负载生产环境）：
|
||||
# checkpointer:
|
||||
# type: redis
|
||||
# connection_string: redis://localhost:6379/0
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: .deer-flow/data
|
||||
```
|
||||
|
||||
SQLite 模式会把数据写入同一个 `deerflow.db` 文件。它适合开发或单用户部署，但生产并发流量可能触发 SQLite 的单写者限制，出现 `sqlite3.OperationalError: database is locked`。
|
||||
|
||||
生产环境或多用户部署请使用 Postgres：
|
||||
|
||||
```yaml
|
||||
database:
|
||||
backend: postgres
|
||||
postgres_url: $DATABASE_URL
|
||||
|
||||
run_events:
|
||||
backend: db
|
||||
```
|
||||
|
||||
在环境变量中设置 `DATABASE_URL`，例如 `postgresql://user:password@localhost:5432/deerflow`。
|
||||
|
||||
本地运行时安装 PostgreSQL 支持：
|
||||
|
||||
```bash
|
||||
cd backend && uv sync --all-packages --extra postgres
|
||||
```
|
||||
|
||||
Docker 或脚本启动场景，在安装或构建前设置 `UV_EXTRAS=postgres`。旧的独立 `checkpointer` 配置仍然兼容，但新的部署应优先使用 `database`。
|
||||
|
||||
## 前端环境变量
|
||||
|
||||
前端通过 `.env.local`（本地开发）或 Docker Compose 环境中的环境变量配置。
|
||||
|
|
|
|||
|
|
@ -116,6 +116,32 @@ BETTER_AUTH_SECRET=your-secret-here-min-32-chars
|
|||
|
||||
对于有多个并发用户的任何部署，使用基于容器的沙箱，防止用户之间的执行环境相互干扰。
|
||||
|
||||
### 数据库后端
|
||||
|
||||
SQLite 适合本地开发和单用户部署，但不适合作为多用户生产环境的数据库后端。
|
||||
|
||||
在 SQLite 模式下，DeerFlow 会把 LangGraph 检查点数据和应用数据写入同一个 `deerflow.db` 文件。WAL 模式可以支持并发读取，但 SQLite 同一时间仍然只有一个写入者。多个用户同时运行 Agent 时，检查点写入和应用写入会竞争同一个文件锁，可能触发 `sqlite3.OperationalError: database is locked`。
|
||||
|
||||
生产环境或任何多用户部署请使用 Postgres：
|
||||
|
||||
```bash
|
||||
# .env
|
||||
DATABASE_URL=postgresql://user:password@postgres:5432/deerflow
|
||||
UV_EXTRAS=postgres
|
||||
```
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
database:
|
||||
backend: postgres
|
||||
postgres_url: $DATABASE_URL
|
||||
|
||||
run_events:
|
||||
backend: db
|
||||
```
|
||||
|
||||
SQLite 部署应限制在本地、单节点或短期评估场景。
|
||||
|
||||
### K8s Provisioner 配置
|
||||
|
||||
<Steps>
|
||||
|
|
|
|||
|
|
@ -94,6 +94,27 @@ ls -la backend/.deer-flow/threads/
|
|||
|
||||
---
|
||||
|
||||
### SQLite 报 `database is locked`
|
||||
|
||||
**症状**：Docker 或团队部署中，多个用户同时使用时出现 `sqlite3.OperationalError: database is locked`。
|
||||
|
||||
**原因**：SQLite 支持并发读取，但同一时间只有一个写入者。DeerFlow 的检查点写入和应用写入会共享同一个 SQLite 数据库文件，多用户并发时可能竞争文件锁。
|
||||
|
||||
**解决**：生产环境或多用户部署切换到 Postgres：
|
||||
|
||||
```yaml
|
||||
database:
|
||||
backend: postgres
|
||||
postgres_url: $DATABASE_URL
|
||||
|
||||
run_events:
|
||||
backend: db
|
||||
```
|
||||
|
||||
SQLite 应只用于本地开发、单用户部署或短期评估环境。
|
||||
|
||||
---
|
||||
|
||||
### 前端构建失败
|
||||
|
||||
**症状**：`make install` 或前端构建步骤失败，提示 `BETTER_AUTH_SECRET` 错误。
|
||||
|
|
@ -142,14 +163,16 @@ grep -i "mcp\|timeout" logs/gateway.log | tail -20
|
|||
|
||||
## 数据备份
|
||||
|
||||
DeerFlow 将持久化数据存储在：
|
||||
根据你启用的存储后端备份：
|
||||
|
||||
- **线程数据**：`backend/.deer-flow/threads/` — 每个线程的上传文件、输出和工作区文件
|
||||
- **检查点**：取决于检查点器配置（SQLite：`backend/.deer-flow/checkpoints.db`，Redis：外部存储）
|
||||
- **记忆**：`backend/.deer-flow/memory.json`（以及 `agents/*/memory.json`）
|
||||
- **自定义 Agent 配置**：`backend/agents/*/config.yaml`
|
||||
- `backend/.deer-flow/threads/`：线程工作目录、上传文件和产出物。
|
||||
- `backend/.deer-flow/memory.json` 和 `backend/.deer-flow/agents/`：全局和 Agent 记忆。
|
||||
- `${database.sqlite_dir}/deerflow.db`：当使用 `database.backend: sqlite` 时备份，默认路径为 `.deer-flow/data/deerflow.db`。
|
||||
- Postgres 数据库备份：当使用 `database.backend: postgres` 时定期导出。
|
||||
- 旧的 `checkpointer.connection_string` 文件：如果你的部署仍配置独立 `checkpointer`。
|
||||
- `backend/agents/*/config.yaml`：自定义 Agent 配置。
|
||||
|
||||
对于生产部署，定期备份这些目录。Docker 部署中，确保这些目录绑定挂载到持久卷，而不是容器内部。
|
||||
Docker 部署中，备份绑定挂载的宿主机路径，而不是容器内部文件系统。
|
||||
|
||||
## 停止和重启服务
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue