Commit graph

10 commits

Author SHA1 Message Date
Anant Goel
10f501d700
eval_cli: Add remote benchmark orchestration (#59802)
Summary:

- Add the `zed-eval` Python CLI for Modal/Harbor/Pier benchmark
orchestration, including content-addressed remote builds, run/suite
management, reporting, rejudge, baseline, and cleanup workflows.
- Extend `eval-cli` for remote evals with provider/model overrides and
step/tool-call metrics in `result.json`.
- Add install/source-run helper scripts so `zed-eval` can be installed
or run from the checkout without manually setting `PYTHONPATH`.
- Harden the remote harness wrappers around exit-code preservation,
archive extraction, custom secret wiring, and Harbor/Pier option parity,
with regression coverage.

Testing:

- Using the CLI for two weeks
- `PYTHONPATH=crates/eval_cli python3 -m compileall -q
crates/eval_cli/zed_eval`
- `uv run --project crates/eval_cli/zed_eval python -m unittest discover
-s crates/eval_cli/zed_eval/tests`
- `bash -n crates/eval_cli/script/install-zed-eval
crates/eval_cli/script/zed-eval`
- `cargo check -p eval_cli`
- `cargo fmt --package eval_cli -- --check`
- `cargo test -p eval_cli --no-run`
- `./script/clippy -p eval_cli`

Release Notes:

- N/A
2026-06-24 15:32:41 +00:00
Ben Brandt
5efd6f68c7
git_ui: Make git_ui call support optional (#58413)
Broke the headless builds for evals. Open to cleaner ways of doing
this...

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-06-03 13:18:45 +00:00
Ben Brandt
4557ad7ad1
eval_cli: Initialize themes in eval headless mode (#57139)
Also fix patch generation

Release Notes:

- N/A
2026-05-19 11:29:00 +00:00
Ben Brandt
5aeb8a7e0f
eval_cli: Wait for model discovery (#57038)
Given the model list is dynamic now, we need a wait

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-05-18 08:41:02 +00:00
Ben Brandt
7a34bc059e
eval_cli: Update eval_cli toolchain and pin harbor (#55768)
Move `git_ui` to `agent_ui` test dependencies
and bump the eval CLI Docker image to Rust 1.95.0 while pinning
the Python `harbor` dependency to 0.6.4

Release Notes:

- N/A
2026-05-05 13:40:58 +00:00
Ben Brandt
b4a96f9c14
eval_cli: Fixes for termbench (#55762)
Release Notes:

- N/A
2026-05-05 12:32:58 +00:00
Ben Brandt
0d5504e3d4
eval_cli: Make things a bit more resilient to different Docker envs (#52731)
Release Notes:

- N/A
2026-03-30 13:18:22 +00:00
Ben Brandt
24ea5e98da
eval_cli: Simplify build setup for more datasets (#52686)
Cleans up build setup and simplifies it a bunch so that we can use the
binary in more eval docker containers

Release Notes:

- N/A
2026-03-29 20:58:50 +00:00
Ben Brandt
1f5d280870
eval_cli: Improve setup for the eval_cli args (#52209)
## Context

Fixes some issues I ran into when running this on a remote machine.

## Self-Review Checklist

<!-- Check before requesting review: -->
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-03-24 11:53:08 +00:00
Ben Brandt
2457e27437
eval: Add eval_cli crate (#50922)
Very much wip

Release Notes:

- N/A
2026-03-06 11:12:38 +00:00