mirror of
https://github.com/agent0ai/agent-zero.git
synced 2026-07-09 17:08:29 +00:00
Add llama.cpp and vLLM local providers
Add llama.cpp / llama-server support: - Register llama.cpp chat and embedding providers through hosted_vllm defaults on host.docker.internal:8080. - Add onboarding metadata, a bundled icon, no-key provider metadata, model discovery coverage, setup docs, and tests. Add vLLM support: - Register vLLM chat and embedding providers through hosted_vllm defaults on host.docker.internal:8000. - Add onboarding metadata, a bundled icon, no-key provider metadata, model discovery coverage, setup docs, and tests. - Strip empty tools arrays from the Responses-to-chat fallback path so strict OpenAI-compatible servers accept local vLLM calls.
This commit is contained in:
parent
0450098117
commit
9bcef39028
18 changed files with 450 additions and 6 deletions
|
|
@ -439,6 +439,8 @@ Use the naming format required by your selected provider:
|
|||
| OpenRouter | Provider prefix mostly required | `anthropic/claude-sonnet-4-5` |
|
||||
| Ollama | Model name only | `gpt-oss:20b` |
|
||||
| oMLX | API-visible model name from `/v1/models` | `Qwen3-0.6B-4bit` |
|
||||
| llama.cpp | API-visible model name from `/v1/models` or `--alias` | `local-gguf` |
|
||||
| vLLM | Hugging Face model ID or served model alias | `Qwen/Qwen2.5-1.5B-Instruct` |
|
||||
|
||||
> [!TIP]
|
||||
> If you see "Invalid model ID," verify the provider and naming format on the provider website, or search the web for "<name-of-ai-model> model naming".
|
||||
|
|
@ -504,6 +506,71 @@ omlx serve --model-dir ~/.omlx/models --paged-ssd-cache-dir ~/.omlx/cache
|
|||
|
||||
---
|
||||
|
||||
## Installing and Using llama.cpp (GGUF Local Models)
|
||||
|
||||
llama.cpp provides `llama-server`, a lightweight OpenAI-compatible HTTP server for GGUF models. Agent Zero talks to it through the same `/v1` API used by OpenAI-compatible clients.
|
||||
|
||||
### macOS llama.cpp Installation
|
||||
|
||||
**Using Homebrew:**
|
||||
|
||||
```bash
|
||||
brew install llama.cpp
|
||||
```
|
||||
|
||||
Start a server with a downloaded GGUF model:
|
||||
|
||||
```bash
|
||||
llama-server -m ~/models/model.gguf --port 8080 --alias local-gguf
|
||||
```
|
||||
|
||||
By default, Agent Zero expects llama.cpp at `http://host.docker.internal:8080/v1`. The model name can be the model path returned by `/v1/models`, but using `--alias` gives you a short stable name such as `local-gguf`.
|
||||
|
||||
### Configuring llama.cpp in Agent Zero
|
||||
|
||||
1. Start `llama-server` and confirm `http://localhost:8080/v1/models` returns your model.
|
||||
2. In Agent Zero Settings, choose **llama.cpp** as the Chat model, Utility model, or Embedding model provider.
|
||||
3. Use the model ID shown by `/v1/models`, or the alias you passed with `--alias`.
|
||||
4. Override the API base URL only if you started `llama-server` on another host or port.
|
||||
5. Click `Save` to confirm your settings.
|
||||
|
||||
> [!NOTE]
|
||||
> If Agent Zero runs in Docker and cannot reach a host-side `llama-server`, start the server on an address Docker can reach, for example `--host 0.0.0.0`, and keep the port firewalled to trusted clients.
|
||||
|
||||
---
|
||||
|
||||
## Installing and Using vLLM (Local OpenAI-Compatible Serving)
|
||||
|
||||
vLLM is a high-throughput local inference server with an OpenAI-compatible API. It is most common on Linux GPU hosts, and can also run on Apple Silicon through the vLLM Apple Silicon path or vLLM-Metal.
|
||||
|
||||
For Apple Silicon Macs, install and activate vLLM-Metal:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh | bash
|
||||
source ~/.venv-vllm-metal/bin/activate
|
||||
```
|
||||
|
||||
Start a basic OpenAI-compatible server:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
By default, Agent Zero expects vLLM at `http://host.docker.internal:8000/v1`, matching vLLM's default HTTP port. If another local provider already uses port 8000, start vLLM on another port and update Agent Zero's API base, for example `http://host.docker.internal:8001/v1`.
|
||||
|
||||
### Configuring vLLM in Agent Zero
|
||||
|
||||
1. Start vLLM and confirm `http://localhost:8000/v1/models` returns the served model.
|
||||
2. In Agent Zero Settings, choose **vLLM** as the Chat model, Utility model, or Embedding model provider.
|
||||
3. Use the model ID returned by vLLM's model list endpoint.
|
||||
4. If you started vLLM with `--api-key`, enter the same key in the advanced provider settings or environment.
|
||||
5. Click `Save` to confirm your settings.
|
||||
|
||||
> [!NOTE]
|
||||
> vLLM serves one model at a time by default. Use a generation model for Chat and Utility slots, and a separate embedding-capable vLLM server if you want vLLM embeddings.
|
||||
|
||||
---
|
||||
|
||||
## Installing and Using Ollama (Local Models)
|
||||
|
||||
Ollama is a powerful tool that allows you to run various large language models locally.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue