mirror of
https://github.com/zed-industries/zed.git
synced 2026-07-09 16:00:35 +00:00
4b119fc547
Hi there, I'm Celina from Hugging Face! Opening this PR to add [llama.cpp](https://llama.app) as a model provider # Objective Today Zed users running llama.cpp have to fall back to the generic OpenAI-compatible provider, which means no auto-discovery (the router mode (`llama serve`) discovers models from the cache and loads them on demand) and manual configuration of every model and its capabilities. This PR makes `llama.cpp` a first-class provider with the same auto-discovery experience. ## Solution - Add a `llama_cpp` client crate with the OpenAI-compatible chat types (`/v1/chat/completions`, including `reasoning_content`) and the discovery types (`/v1/models`, `/props`), mirroring the existing `ollama` crate. - Add the provider in `crates/language_models/src/provider/llama_cpp.rs`, modeled on the Ollama provider (settings, configuration view, event mapping). - Auto discover served models and their context length and tool/vision support from `/props`. Set `auto_discover: false` to list models manually instead. - An unloaded model can't be inspected without loading it, so it is listed with optimistic defaults (large context, tools enabled) and is usable from the first message; its real context length and tool support are filled in once it loads. These live behind a shared map, so a model already selected in an open conversation picks them up without being re-selected. - Show load progress. The provider subscribes to `/models/sse` and surfaces each model's load progress (e.g. "Loading weights 42%") in its display name, reconciling stale labels against `/v1/models`. Builds without `/models/sse` degrade gracefully - no progress, and no capability refresh after the initial discovery. - Add settings (`api_url`, `auto_discover`, `available_models` with per-model `max_tokens` / `supports_tools` / `supports_images`, `context_window`, `custom_headers`), the provider icon, a `default.json` entry, and documentation under "Use a Local Model". No new dependencies: the crate reuses existing workspace dependencies, and shared state uses `std::sync::RwLock`. ## Testing - Unit tests in both new crates cover wire/response parsing, model discovery for single-model and router shapes, the cold-start optimistic defaults, the in-place capability refresh once a model loads, and the `/models/sse` event handling (state changes, load failure, load progress). - Built Zed locally and ran it against a local `llama serve` router: confirmed models are discovered without manual configuration, that the first message works before a model has finished loading, that load progress is shown in the model's display name while it loads, and that the reported context length and tool support refine to the model's real values once it finishes loading. - Platforms: tested on macOS (Apple Silicon). ## Self-Review Checklist: - [x] I've reviewed my own diff for quality, security, and reliability - [x] Unsafe blocks (if any) have justifying comments - [x] The content adheres to Zed's UI standards ([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist) and [icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md) guidelines) - [x] Tests cover the new/changed behavior - [x] Performance impact has been considered and is acceptable ## Showcase The generation speed (tokens/sec) depends on the machine you're running the model, here it's a Apple M3 Max 64GB running a 4-bit quant of https://huggingface.co/Qwen/Qwen3.5-35B-A3B. For the load progress status, make sure to upgrade your llama.cpp version to the latest build. https://github.com/user-attachments/assets/0254f6ef-abe9-42ed-810b-ef1a5b8fa3bd --- Release Notes: - Added llama.cpp as a language model provider --------- Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com> |
||
|---|---|---|
| .. | ||
| badge | ||
| fonts | ||
| icons | ||
| images | ||
| keymaps | ||
| prompts | ||
| settings | ||
| sounds | ||
| themes | ||