Commit graph

12 commits

Author SHA1 Message Date
Richard Feldman
8dff89c8d1
Fix Gemini schema conversion dropping anyOf and nullability (#57160)
Follow-up to #49292:

- **Bug fix in `push_any_of_constraint`**: when both an `anyOf` and a
non-empty `allOf` were already present at the same level, the existing
`anyOf` was silently dropped. Now it's always preserved.
- **Canonical OpenAPI nullability**: collapse `{nullable: true}`-only
entries out of `anyOf` onto the parent so `anyOf: [{type: "string"},
{type: "null"}]` becomes `{type: "string", nullable: true}` (the form
Gemini actually expects) instead of `anyOf: [{type: "string"},
{nullable: true}]`.
- **Compile-time path consistency**: route `ToJsonSchemaSubsetTransform`
through the same `convert_null_in_types_to_nullable` helper so
Rust-defined tools using `Option<T>` also keep nullability on Gemini,
instead of silently truncating to the non-null type.
- Drop an unnecessary clone in `push_any_of_constraint` and simplify
`convert_types_to_any_of_defs`.
- Add a small `obj()` test helper and a regression test for the `anyOf +
allOf + multi-type` case.

Release Notes:

- Fixed `Option<T>` tool parameters being sent to Gemini without their
nullability, and fixed tool schemas with `anyOf` + `allOf` losing
constraints during the OpenAPI 3.0 conversion

Co-authored-by: Daniel Strobusch <1438302+dastrobu@users.noreply.github.com>
2026-05-19 19:04:48 +00:00
Daniel Strobusch
ae47ec9ac0
language_models: Fix Gemini tool parameter nullability and multi-type schema (#49292)
Some checks are pending
Congratsbot / check-author (push) Waiting to run
Congratsbot / congrats (push) Blocked by required conditions
deploy_nightly_docs / deploy_docs (push) Waiting to run
run_tests / check_postgres_and_protobuf_migrations (push) Blocked by required conditions
run_tests / extension_tests (push) Blocked by required conditions
run_tests / check_scripts (push) Blocked by required conditions
run_tests / clippy_windows (push) Blocked by required conditions
run_tests / clippy_linux (push) Blocked by required conditions
run_tests / clippy_mac (push) Blocked by required conditions
run_tests / clippy_mac_x86_64 (push) Blocked by required conditions
run_tests / run_tests_mac (push) Blocked by required conditions
run_tests / miri_scheduler (push) Blocked by required conditions
run_tests / doctests (push) Blocked by required conditions
run_tests / check_workspace_binaries (push) Blocked by required conditions
run_tests / build_visual_tests_binary (push) Blocked by required conditions
run_tests / check_wasm (push) Blocked by required conditions
run_tests / check_dependencies (push) Blocked by required conditions
run_tests / check_docs (push) Blocked by required conditions
run_tests / check_licenses (push) Blocked by required conditions
run_tests / tests_pass (push) Blocked by required conditions
run_tests / orchestrate (push) Waiting to run
run_tests / check_style (push) Waiting to run
run_tests / run_tests_windows (push) Blocked by required conditions
run_tests / run_tests_linux (push) Blocked by required conditions
Transform JSON schemas for Google AI tools to use `nullable: true`
instead of `type: ["type", "null"]`, which is not supported by the
Gemini API.

Additionally, convert multi-type arrays (e.g., `type: ["string",
"number"]`)
to `anyOf` constraints, as Gemini expects a single string for the `type`
field.

This handles recursive transformation of properties, items, definitions,
and logical operators, safely merging conflicting `anyOf` and `allOf`
constraints.

Closes https://github.com/zed-industries/zed/issues/44875
Closes https://github.com/zed-industries/zed/issues/32429

Release Notes:

- Fixed a bug where using Gemini with certain tools (especially via MCP)
resulted in "Invalid JSON payload received" errors due to incompatible
JSON schema formats.

## Testing

Added unit tests in `crates/language_model/src/tool_schema.rs` covering
nullability, multi-types, and `oneOf` conversions.

### Manual Testing with MCP Test Server
The [MCP Test Server](https://github.com/dastrobu/mcp-test-server) was
used to verify these edge cases with Gemini 3 Flash.

#### Setup
1. Install the test server: `cargo install --git
https://github.com/dastrobu/mcp-test-server`
2. Add to Zed `settings.json`:
   ```json
   "context_servers": [
     {
       "command": "mcp-test-server"
     }
   ]
   ```


Use the following pattern in a chat window:

```
call the add_tool function to create a new tool: weather with input schema: 

  {
    "type": "object",
    "properties": {
      "city": { "type": ["string", "null"] }
    }
  }
```
Afterwards:
...

```
call it
```

Without fix: 

<img width="671" height="448" alt="image"
src="https://github.com/user-attachments/assets/e8b6e7bd-d431-4a7e-8c9a-fff73d82127d"
/>

With fix:

<img width="729" height="334" alt="image"
src="https://github.com/user-attachments/assets/eb758765-4c34-4915-a8cf-dd4bfb993619"
/>

#### Cases verified manually:



**1. Nullability in properties**
- **Input:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": { "type": ["string", "null"] }
    }
  }
  ```
- **Converted:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": { "type": "string", "nullable": true }
    }
  }
  ```

**2. Multi-type properties**
- **Input:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": { "type": ["string", "number"] }
    }
  }
  ```
- **Converted:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": {
        "anyOf": [
          { "type": "string" },
          { "type": "number" }
        ]
      }
    }
  }
  ```

**3. Explicit `anyOf` with nullability**
- **Input:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": {
        "anyOf": [
          { "type": "string" },
          { "type": "null" }
        ]
      }
    }
  }
  ```
- **Converted:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": {
        "anyOf": [
          { "type": "string" },
          { "nullable": true }
        ]
      }
    }
  }
  ```

**4. Conflicting `anyOf` sources (Multi-type + existing `anyOf`)**
- **Input:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": {
        "type": ["string", "number"],
        "anyOf": [
          { "minLength": 5 }
        ]
      }
    }
  }
  ```
- **Converted:**
  ```json
  {
    "type": "object",
    "properties": {
      "city": {
        "allOf": [
          { "anyOf": [{ "minLength": 5 }] },
          { "anyOf": [{ "type": "string" }, { "type": "number" }] }
        ]
      }
    }
  }
  ```

Co-authored-by: Richard Feldman <richard@zed.dev>
2026-05-19 15:56:34 +00:00
Bennet Bo Fenner
da43bdb648
agent: Support image output from MCP tools (#57134)
Release Notes:

- agent: Support image output from MCP tools
2026-05-19 12:51:42 +00:00
Bennet Bo Fenner
68340172a1
agent: Remove unused LanguageModelImage APIs (#57050)
Pulled out from #56866. Will help with MCP image support

Release Notes:

- N/A
2026-05-18 12:22:22 +00:00
Bennet Bo Fenner
3a742b5e0d
language_models: Remove unused cache_configuration API (#56884)
Release Notes:

- N/A
2026-05-15 16:27:11 +00:00
Ben Brandt
78c889c21d
open_ai: Responses API improvements (#56476)
Release Notes:

- Removed deprecated OpenAI models
- Added support for gpt-5.4-nano/mini models for OpenAI provider
- Improved output quality when using OpenAI models

---------

Co-authored-by: Bennet Bo Fenner <bennetbo@gmx.de>
Co-authored-by: Smit Barmase <heysmitbarmase@gmail.com>
Co-authored-by: Gaauwe Rombouts <mail@grombouts.nl>
2026-05-12 14:47:16 +00:00
Ben Brandt
a8ae1677a3
Implement tool result conversion for anyhow errors (#55001)
Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- N/A
2026-04-27 13:03:33 +00:00
Bennet Bo Fenner
bf3fc2336d
agent: Allow tools to output multiple content parts (#54518)
Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Closes #ISSUE

Release Notes:

- N/A
2026-04-27 12:36:11 +00:00
Lukas Wirth
c5a2807492
Remove smol as a dependency from a bunch of crates (#53603)
We aren't making use of it in these crates and it unblocks some
web-related work

Release Notes:

- N/A or Added/Fixed/Improved ...
2026-04-24 10:29:51 +00:00
Bennet Bo Fenner
dec1a2a3b8
gemini: Fix tool schema issues when type field is array (#53834)
Closes #53919

Self-Review Checklist:

- [x] I've reviewed my own diff for quality, security, and reliability
- [x] Unsafe blocks (if any) have justifying comments
- [x] The content is consistent with the [UI/UX
checklist](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist)
- [x] Tests cover the new/changed behavior
- [x] Performance impact has been considered and is acceptable

Release Notes:

- gemini: Fixed an issue with MCP servers specifying tools with specific
schemas
2026-04-15 10:11:44 +02:00
Nathan Sobo
4a02367db2
Persist fast mode across new threads (#53356)
When toggling fast mode, the setting is now written to `settings.json`
under `agent.default_model.speed`, so new threads start with the same
speed. This follows the same pattern as `enable_thinking` and `effort`.

The `speed` field uses the existing `Speed` enum (`"fast"` /
`"standard"`) rather than a boolean, to leave room for future speed
tiers.

Example settings:
```json
{
  "agent": {
    "default_model": {
      "provider": "zed.dev",
      "model": "claude-sonnet-4",
      "speed": "fast"
    }
  }
}
```

cc @benbrandt

Release Notes:

- N/A

---------

Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com>
2026-04-08 18:59:33 +00:00
Agus Zubiaga
98c17ca160
language_models: Refactor deps and extract cloud (#53270)
- `language_model` no longer depends on provider-specific crates such as
`anthropic` and `open_ai` (inverted dependency)
- `language_model_core` was extracted from `language_model` which
contains the types for the provider-specific crates to convert to/from.
- `gpui::SharedString` has been extracted into its own crate (still
exposed by `gpui`), so `language_model_core` and provider API crates
don't have to depend on `gpui`.
- Removes some unnecessary `&'static str` | `SharedString` -> `String`
-> `SharedString` conversions across the codebase.
- Extracts the core logic of the cloud `LanguageModelProvider` into its
own crate with simpler dependencies.


Release Notes:

- N/A

---------

Co-authored-by: John Tur <john-tur@outlook.com>
2026-04-07 12:28:19 -03:00