mirror of
https://github.com/eigent-ai/eigent.git
synced 2026-05-24 22:04:09 +00:00
d606fae458
Co-authored-by: bytecii <bytecii@users.noreply.github.com> Co-authored-by: bytecii <994513625@qq.com>
70 lines
2.1 KiB
Markdown
70 lines
2.1 KiB
Markdown
---
|
|
title: Models (Local Model)
|
|
description: Configure and deploy your preferred LLM models with Eigent.
|
|
---
|
|
|
|
## **Self-Host Model**
|
|
|
|
1. Configure your self-host model
|
|
|
|
First, you need to set up your local models and expose them as an **OpenAI-Compatible Server.**
|
|
|
|
```bash
|
|
#Vllm https://docs.vllm.ai/en/latest/getting_started/quickstart.html#openai-compatible-server
|
|
vllm serve Qwen/Qwen2.5-1.5B-Instruct
|
|
```
|
|
|
|
```python
|
|
#SGLang https://docs.sglang.ai/backend/openai_api_completions.html
|
|
from sglang.test.test_utils import is_in_ci
|
|
|
|
if is_in_ci():
|
|
from patch import launch_server_cmd
|
|
else:
|
|
from sglang.utils import launch_server_cmd
|
|
|
|
from sglang.utils import wait_for_server, print_highlight, terminate_process
|
|
|
|
server_process, port = launch_server_cmd(
|
|
"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --mem-fraction-static 0.8"
|
|
)
|
|
|
|
wait_for_server(f"http://localhost:{port}")
|
|
print(f"Server started on http://localhost:{port}")
|
|
```
|
|
|
|
```bash
|
|
#Ollama https://github.com/ollama/ollama
|
|
ollama pull qwen2.5:7b
|
|
```
|
|
|
|
```bash
|
|
# LLaMA.cpp server https://github.com/ggml-org/llama.cpp/tree/master/tools/server
|
|
./llama-server -m /path/to/model.gguf --host 0.0.0.0 --port 8080
|
|
```
|
|
|
|
2. Setting your model
|
|
|
|
|
|
|
|
3. Configure the Google Search toolkit
|
|
|
|
|
|
|
|
<img src="/docs/images/models_configure_tools_key.png" alt="configure_searchtoolsapi" /> You can refer to the following document for detailed information on how to configure **GOOGLE_API_KEY** and **SEARCH_ENGINE_ID :** https://developers.google.com/custom-search/v1/overview
|
|
|
|
## **API KEY Reference**
|
|
|
|
Gemini: https://ai.google.dev/gemini-api/docs/api-key
|
|
|
|
OpenAI: https://platform.openai.com/api-keys
|
|
|
|
Anthropic: https://console.anthropic.com/
|
|
|
|
Qwen:https://www.alibabacloud.com/help/en/model-studio/get-api-key
|
|
|
|
Deepseek: https://platform.deepseek.com/api_keys
|
|
|
|
AWS Bedrock:https://github.com/aws-samples/bedrock-access-gateway/blob/main/README.md
|
|
|
|
Azure:https://azure.microsoft.com/products/cognitive-services/openai-service/
|