Add llama.cpp and vLLM local providers

Add llama.cpp / llama-server support:
- Register llama.cpp chat and embedding providers through hosted_vllm defaults on host.docker.internal:8080.
- Add onboarding metadata, a bundled icon, no-key provider metadata, model discovery coverage, setup docs, and tests.

Add vLLM support:
- Register vLLM chat and embedding providers through hosted_vllm defaults on host.docker.internal:8000.
- Add onboarding metadata, a bundled icon, no-key provider metadata, model discovery coverage, setup docs, and tests.
- Strip empty tools arrays from the Responses-to-chat fallback path so strict OpenAI-compatible servers accept local vLLM calls.
This commit is contained in:
Alessandro 2026-06-15 05:37:28 +02:00
parent 0450098117
commit 9bcef39028
18 changed files with 450 additions and 6 deletions

View file

@ -439,6 +439,8 @@ Use the naming format required by your selected provider:
| OpenRouter | Provider prefix mostly required | `anthropic/claude-sonnet-4-5` |
| Ollama | Model name only | `gpt-oss:20b` |
| oMLX | API-visible model name from `/v1/models` | `Qwen3-0.6B-4bit` |
| llama.cpp | API-visible model name from `/v1/models` or `--alias` | `local-gguf` |
| vLLM | Hugging Face model ID or served model alias | `Qwen/Qwen2.5-1.5B-Instruct` |
> [!TIP]
> If you see "Invalid model ID," verify the provider and naming format on the provider website, or search the web for "<name-of-ai-model> model naming".
@ -504,6 +506,71 @@ omlx serve --model-dir ~/.omlx/models --paged-ssd-cache-dir ~/.omlx/cache
---
## Installing and Using llama.cpp (GGUF Local Models)
llama.cpp provides `llama-server`, a lightweight OpenAI-compatible HTTP server for GGUF models. Agent Zero talks to it through the same `/v1` API used by OpenAI-compatible clients.
### macOS llama.cpp Installation
**Using Homebrew:**
```bash
brew install llama.cpp
```
Start a server with a downloaded GGUF model:
```bash
llama-server -m ~/models/model.gguf --port 8080 --alias local-gguf
```
By default, Agent Zero expects llama.cpp at `http://host.docker.internal:8080/v1`. The model name can be the model path returned by `/v1/models`, but using `--alias` gives you a short stable name such as `local-gguf`.
### Configuring llama.cpp in Agent Zero
1. Start `llama-server` and confirm `http://localhost:8080/v1/models` returns your model.
2. In Agent Zero Settings, choose **llama.cpp** as the Chat model, Utility model, or Embedding model provider.
3. Use the model ID shown by `/v1/models`, or the alias you passed with `--alias`.
4. Override the API base URL only if you started `llama-server` on another host or port.
5. Click `Save` to confirm your settings.
> [!NOTE]
> If Agent Zero runs in Docker and cannot reach a host-side `llama-server`, start the server on an address Docker can reach, for example `--host 0.0.0.0`, and keep the port firewalled to trusted clients.
---
## Installing and Using vLLM (Local OpenAI-Compatible Serving)
vLLM is a high-throughput local inference server with an OpenAI-compatible API. It is most common on Linux GPU hosts, and can also run on Apple Silicon through the vLLM Apple Silicon path or vLLM-Metal.
For Apple Silicon Macs, install and activate vLLM-Metal:
```bash
curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
source ~/.venv-vllm-metal/bin/activate
```
Start a basic OpenAI-compatible server:
```bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000
```
By default, Agent Zero expects vLLM at `http://host.docker.internal:8000/v1`, matching vLLM's default HTTP port. If another local provider already uses port 8000, start vLLM on another port and update Agent Zero's API base, for example `http://host.docker.internal:8001/v1`.
### Configuring vLLM in Agent Zero
1. Start vLLM and confirm `http://localhost:8000/v1/models` returns the served model.
2. In Agent Zero Settings, choose **vLLM** as the Chat model, Utility model, or Embedding model provider.
3. Use the model ID returned by vLLM's model list endpoint.
4. If you started vLLM with `--api-key`, enter the same key in the advanced provider settings or environment.
5. Click `Save` to confirm your settings.
> [!NOTE]
> vLLM serves one model at a time by default. Use a generation model for Chat and Utility slots, and a separate embedding-capable vLLM server if you want vLLM embeddings.
---
## Installing and Using Ollama (Local Models)
Ollama is a powerful tool that allows you to run various large language models locally.