3965f7133f
Cherry-pick of #57844 to preview ---- Summary: - Stop loading persisted default Zed Rules into native agent project context. - Remove legacy user-rules rendering from agent system prompt templates. - Keep Skills and project rules file support unchanged. Tests: - cargo test -p agent test_system_prompt_does_not_render_legacy_zed_rules_section --lib - cargo test -p prompt_store test_empty_skills_sets_has_skills_false - cargo test -p prompt_store test_project_context_does_not_filter_by_budget Closes AI-325 Release Notes: - Fixed deprecated Rules being automatically included in new agent requests. Co-authored-by: MartinYe1234 <52641447+MartinYe1234@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| script | ||
| src | ||
| zed_eval | ||
| .gitignore | ||
| build.rs | ||
| Cargo.toml | ||
| Dockerfile | ||
| Dockerfile.dockerignore | ||
| LICENSE-GPL | ||
| README.md | ||
eval-cli
Headless CLI binary for running Zed's agent in evaluation/benchmark environments. Designed to work inside containerized environments like Harbor where the repository is already checked out and API keys are provided via environment variables.
Uses the same NativeAgent + AcpThread pipeline as the production Zed
editor — full agentic loop with tool calls, subagents, and retries, just
without a GUI.
Building
Native (for local testing on the same OS)
cargo build --release -p eval_cli
Cross-compile for Linux x86_64 (from macOS or other hosts)
Harbor containers run Linux x86_64. Use the Docker-based build script:
crates/eval_cli/script/build-linux
This produces target/eval-cli (an x86_64 Linux ELF binary). You can
also specify a custom output path:
crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux
Standalone usage
eval-cli \
--workdir /testbed \
--model anthropic/claude-sonnet-4-6-latest \
--instruction "Fix the bug described in..." \
--timeout 600 \
--output-dir /logs/agent
Reads API keys from environment variables (ANTHROPIC_API_KEY,
OPENAI_API_KEY, etc.). Writes result.json, thread.md, and
thread.json to the output directory.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Agent finished |
| 1 | Error (model/auth/runtime failure) |
| 2 | Timeout |
| 3 | Interrupted (SIGTERM/SIGINT) |
Harbor integration
The zed_eval/ directory contains a Python package that
implements Harbor's BaseInstalledAgent interface, allowing eval-cli to
be used with --agent-import-path without modifying Harbor's source code.
Setup
pip install -e crates/eval_cli/harbor/
Running with a local binary
Build for Linux first, then pass the binary path:
crates/eval_cli/script/build-linux
harbor run -d "swebench_verified@latest" \
--agent-import-path zed_eval.agent:ZedAgent \
--ae binary_path=target/eval-cli \
-m anthropic/claude-sonnet-4-6-latest
The agent uploads the binary into the container during setup — no download URL needed during local iteration.
Running with a download URL
For CI or when the binary is hosted somewhere:
harbor run -d "swebench_verified@latest" \
--agent-import-path zed_eval.agent:ZedAgent \
--ak download_url=https://example.com/eval-cli \
-m anthropic/claude-sonnet-4-6-latest
Setting a timeout
Pass EVAL_CLI_TIMEOUT via --ae:
harbor run -d "swebench_verified@latest" \
--agent-import-path zed_eval.agent:ZedAgent \
--ak binary_path=target/eval-cli \
--ae EVAL_CLI_TIMEOUT=600 \
-m anthropic/claude-sonnet-4-6-latest