Commit graph

23 commits

Author SHA1 Message Date
Anant Goel
a923597341
Fix agent terminal in headless eval sandbox (#59969)
# Objective

Fix the agent `terminal` tool in headless eval environments. In the eval
sandbox, terminal commands failed before the shell ran with `IOError:
Not a tty (os error 25)` because PTY setup attempted to acquire a
controlling terminal.

## Solution

- Add a `terminal::HeadlessTerminal` global that is set by `eval_cli`
only.
- When headless mode is enabled, run terminal task commands as plain
subprocesses with piped stdout/stderr instead of opening a PTY.
- Pump subprocess output through the existing terminal emulator/event
channel so output capture, completion, and task killing keep working.
- Build ACP terminal commands non-interactively in headless mode.
- Keep the normal editor terminal path unchanged when the global is
unset.
- Handle non-PTY output edge cases by preserving split CRLF sequences
and avoiding an indefinite wait if subprocess exit-status polling
errors.

## Testing

- `cargo fmt --all`
- `cargo test -p util non_interactive_omits_interactive_flag`
- `cargo test -p terminal test_no_pty_task_terminal_captures_output`
- `cargo test -p terminal test_convert_lf_to_crlf_preserves_split_crlf`
- `cargo test -p terminal test_write_output`
- `cargo check -p eval_cli`
- `git --no-pager diff --check`

## Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content adheres to Zed's UI standards
([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
and
[icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md)
guidelines)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- Fixed the agent terminal tool failing with "Not a tty" in
headless/eval environments
2026-06-29 15:58:53 +00:00
Ben Brandt
70fd3c5774
acp: Handle acp message ids when we get them (#59927)
This separates the two user-message identities we currently need in
agent threads:

- `protocol_id`: the ACP `messageId` used to group or split streamed
message chunks from ACP agents.
- `client_id`: the Zed-generated user message ID used by the native
agent for truncate, rewind, edit/regenerate, checkpoints, token usage,
and persistence.

The main behavior change is that ACP protocol message IDs now affect
message merging/rendering without leaking Zed-native client IDs into ACP
agents by default. Agents that can accept Zed-generated user message IDs
opt into that path through `AgentSessionClientUserMessageIds`, which
owns both ID generation and prompt submission. The native Zed agent
implements that capability; normal ACP agents continue through the
protocol-only `prompt` path.

In ACP v2 we'll have a mechanism for getting user message replays that
will allow the agent to adequately generate it's own opaque id that we
can then use for these other capabilities. But for now, this split at
least makes sure we keep the two concepts distinct.

Release Notes:

- N/A

---------

Co-authored-by: Bennet Bo Fenner <bennetbo@gmx.de>
2026-06-29 07:21:09 +00:00
Anant Goel
10f501d700
eval_cli: Add remote benchmark orchestration (#59802)
Summary:

- Add the `zed-eval` Python CLI for Modal/Harbor/Pier benchmark
orchestration, including content-addressed remote builds, run/suite
management, reporting, rejudge, baseline, and cleanup workflows.
- Extend `eval-cli` for remote evals with provider/model overrides and
step/tool-call metrics in `result.json`.
- Add install/source-run helper scripts so `zed-eval` can be installed
or run from the checkout without manually setting `PYTHONPATH`.
- Harden the remote harness wrappers around exit-code preservation,
archive extraction, custom secret wiring, and Harbor/Pier option parity,
with regression coverage.

Testing:

- Using the CLI for two weeks
- `PYTHONPATH=crates/eval_cli python3 -m compileall -q
crates/eval_cli/zed_eval`
- `uv run --project crates/eval_cli/zed_eval python -m unittest discover
-s crates/eval_cli/zed_eval/tests`
- `bash -n crates/eval_cli/script/install-zed-eval
crates/eval_cli/script/zed-eval`
- `cargo check -p eval_cli`
- `cargo fmt --package eval_cli -- --check`
- `cargo test -p eval_cli --no-run`
- `./script/clippy -p eval_cli`

Release Notes:

- N/A
2026-06-24 15:32:41 +00:00
Ben Brandt
8ba35e5eac
acp: Update agent client protocol crate to 0.15.0 (#59593)
Some checks are pending
Congratsbot / check-author (push) Waiting to run
Congratsbot / congrats (push) Blocked by required conditions
deploy_nightly_docs / deploy_docs (push) Waiting to run
run_tests / orchestrate (push) Waiting to run
run_tests / check_style (push) Waiting to run
run_tests / clippy_windows (push) Blocked by required conditions
run_tests / clippy_linux (push) Blocked by required conditions
run_tests / clippy_mac (push) Blocked by required conditions
run_tests / clippy_mac_x86_64 (push) Blocked by required conditions
run_tests / run_tests_windows (push) Blocked by required conditions
run_tests / run_tests_linux (push) Blocked by required conditions
run_tests / run_tests_mac (push) Blocked by required conditions
run_tests / miri_scheduler (push) Blocked by required conditions
run_tests / doctests (push) Blocked by required conditions
run_tests / check_workspace_binaries (push) Blocked by required conditions
run_tests / build_visual_tests_binary (push) Blocked by required conditions
run_tests / check_wasm (push) Blocked by required conditions
run_tests / check_dependencies (push) Blocked by required conditions
run_tests / check_docs (push) Blocked by required conditions
run_tests / check_licenses (push) Blocked by required conditions
run_tests / check_scripts (push) Blocked by required conditions
run_tests / check_postgres_and_protobuf_migrations (push) Blocked by required conditions
run_tests / extension_tests (push) Blocked by required conditions
run_tests / tests_pass (push) Blocked by required conditions
We finally have a cancellation mechanism to use! Made the non
side-effectful handlers stop their work if we get a cancel request
notification.

Release Notes:

- N/A
2026-06-22 09:04:29 +00:00
Ben Brandt
5efd6f68c7
git_ui: Make git_ui call support optional (#58413)
Broke the headless builds for evals. Open to cleaner ways of doing
this...

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-06-03 13:18:45 +00:00
MartinYe1234
aa6062edaf
Stop loading deprecated agent rules (#57844)
Summary:

- Stop loading persisted default Zed Rules into native agent project
context.
- Remove legacy user-rules rendering from agent system prompt templates.
- Keep Skills and project rules file support unchanged.

Tests:

- cargo test -p agent
test_system_prompt_does_not_render_legacy_zed_rules_section --lib
- cargo test -p prompt_store test_empty_skills_sets_has_skills_false
- cargo test -p prompt_store
test_project_context_does_not_filter_by_budget

Closes AI-325
Release Notes:

- Fixed deprecated Rules being automatically included in new agent
requests.
2026-05-27 16:21:30 +00:00
Ben Brandt
4557ad7ad1
eval_cli: Initialize themes in eval headless mode (#57139)
Also fix patch generation

Release Notes:

- N/A
2026-05-19 11:29:00 +00:00
Ben Brandt
5aeb8a7e0f
eval_cli: Wait for model discovery (#57038)
Given the model list is dynamic now, we need a wait

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-05-18 08:41:02 +00:00
Ben Brandt
9a679b3226
eval_cli: Set global filesystem in eval CLI init (#55862)
Some dependency started requiring this, so fixing some runtime errors.

Release Notes:

- N/A
2026-05-06 09:35:43 +00:00
Ben Brandt
7a34bc059e
eval_cli: Update eval_cli toolchain and pin harbor (#55768)
Move `git_ui` to `agent_ui` test dependencies
and bump the eval CLI Docker image to Rust 1.95.0 while pinning
the Python `harbor` dependency to 0.6.4

Release Notes:

- N/A
2026-05-05 13:40:58 +00:00
Ben Brandt
b4a96f9c14
eval_cli: Fixes for termbench (#55762)
Release Notes:

- N/A
2026-05-05 12:32:58 +00:00
Ben Brandt
58e2b7ecdd
acp: Use new Rust SDK (#52997)
Testing out Niko's new SDK design

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable


Release Notes:

- N/A
2026-04-22 15:02:57 +00:00
Jakub Konka
29609d3599
language_model: Decouple from Zed-specific implementation details (#52913)
This PR decouples `language_model`'s dependence on Zed-specific
implementation details. In particular
* `credentials_provider` is split into a generic `credentials_provider`
crate that provides a trait, and `zed_credentials_provider` that
implements the said trait for Zed-specific providers and has functions
that can populate a global state with them
* `zed_env_vars` is split into a generic `env_var` crate that provides
generic tooling for managing env vars, and `zed_env_vars` that contains
Zed-specific statics
* `client` is now dependent on `language_model` and not vice versa

Release Notes:

- N/A
2026-04-02 17:06:57 -03:00
Ben Brandt
76c6004b27
Remove text thread and slash command crates (#52757)
🫡

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- Removed legacy Text Threads feature to help streamline the new agentic
workflows in Zed. Thanks to all of you who were enthusiastic Text Thread
users over the years ❤️!

---------

Co-authored-by: Bennet Bo Fenner <bennetbo@gmx.de>
2026-03-31 17:55:05 +02:00
Mikayla Maki
8b0d49f474
Add existing user agent onboarding flow (#52787)
Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Closes #ISSUE

Release Notes:

- N/A
2026-03-31 01:39:47 +00:00
Ben Brandt
0d5504e3d4
eval_cli: Make things a bit more resilient to different Docker envs (#52731)
Release Notes:

- N/A
2026-03-30 13:18:22 +00:00
Piotr Osiewicz
b59ae5cf81
Bump Rust version to 1.94 (#51086)
Closes #ISSUE

Before you mark this PR as ready for review, make sure that you have:
- [ ] Added a solid test coverage and/or screenshots from doing manual
testing
- [ ] Done a self-review taking into account security and performance
aspects
- [ ] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)

Release Notes:

- N/A
2026-03-30 09:06:59 +00:00
Ben Brandt
24ea5e98da
eval_cli: Simplify build setup for more datasets (#52686)
Cleans up build setup and simplifies it a bunch so that we can use the
binary in more eval docker containers

Release Notes:

- N/A
2026-03-29 20:58:50 +00:00
Ben Brandt
1f5d280870
eval_cli: Improve setup for the eval_cli args (#52209)
## Context

Fixes some issues I ran into when running this on a remote machine.

## Self-Review Checklist

<!-- Check before requesting review: -->
- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-03-24 11:53:08 +00:00
Bennet Bo Fenner
015225196d
Store ACP thread metadata (#51657)
Before you mark this PR as ready for review, make sure that you have:
- [x] Added a solid test coverage and/or screenshots from doing manual
testing
- [x] Done a self-review taking into account security and performance
aspects
- [x] Aligned any UI changes with the [UI
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)

Release Notes:

- N/A

---------

Co-authored-by: cameron <cameron.studdstreet@gmail.com>
Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com>
2026-03-16 17:23:02 +01:00
Bennet Bo Fenner
f4b04af3dc
agent: Allow NativeAgent to work with multiple projects (#51202)
This removes the assumption that one project <-> one native agent. The
native agent now maintains a project per session. We don't make use of
this right now, but it will come in handy once we start sharing ACP
connections globally.

Release Notes:

- N/A
2026-03-10 17:04:48 +01:00
Neel
dc0e41f834
Refresh LLM API token on organization change (#50931)
Emit client-side organization changed events through
`RefreshLlmTokenListener` so it produces the same `RefreshLlmTokenEvent`
used for server-pushed `UserUpdated` messages.

This keeps token refresh fan-out in one place.

Closes CLO-383.

Release Notes:

- N/A

---------

Co-authored-by: Tom Houlé <tom@tomhoule.com>
2026-03-06 19:15:21 +00:00
Ben Brandt
2457e27437
eval: Add eval_cli crate (#50922)
Very much wip

Release Notes:

- N/A
2026-03-06 11:12:38 +00:00