open-notebook/docs/5-CONFIGURATION/openai-compatible.md

OpenAI-Compatible Providers

Use any server that implements the OpenAI API format with Open Notebook. This includes LM Studio, Text Generation WebUI, vLLM, and many others.

---

What is OpenAI-Compatible?

Many AI tools implement the same API format as OpenAI:

POST /v1/chat/completions
POST /v1/embeddings
POST /v1/audio/speech

Open Notebook can connect to any server using this format.

---

Common Compatible Servers

Server	Use Case	URL
LM Studio	Desktop GUI for local models	https://lmstudio.ai
Text Generation WebUI	Full-featured local inference	https://github.com/oobabooga/text-generation-webui
vLLM	High-performance serving	https://github.com/vllm-project/vllm
Ollama	Simple local models	(Use native Ollama provider instead)
LocalAI	Local AI inference	https://github.com/mudler/LocalAI
llama.cpp server	Lightweight inference	https://github.com/ggerganov/llama.cpp

---

Quick Setup: LM Studio

Step 1: Install and Start LM Studio

1. Download from https://lmstudio.ai
2. Install and launch
3. Download a model (e.g., Llama 3)
4. Start the local server (default: port 1234)

Step 2: Configure Environment

# For language models
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
export OPENAI_COMPATIBLE_API_KEY=not-needed  # LM Studio doesn't require key

Step 3: Add Model in Open Notebook

1. Go to Settings → Models
2. Click Add Model
3. Configure:
   • Provider: openai_compatible
   • Model Name: Your model name from LM Studio
   • Display Name: LM Studio - Llama 3
4. Click Save

---

Environment Variables

Language Models (Chat)

OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=optional-api-key

Embeddings

OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:1234/v1
OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=optional-api-key

Text-to-Speech

OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
OPENAI_COMPATIBLE_API_KEY_TTS=optional-api-key

Speech-to-Text

OPENAI_COMPATIBLE_BASE_URL_STT=http://localhost:9000/v1
OPENAI_COMPATIBLE_API_KEY_STT=optional-api-key

---

Docker Networking

When Open Notebook runs in Docker and your compatible server runs on the host:

macOS / Windows

OPENAI_COMPATIBLE_BASE_URL=http://host.docker.internal:1234/v1

Linux

# Option 1: Docker bridge IP
OPENAI_COMPATIBLE_BASE_URL=http://172.17.0.1:1234/v1

# Option 2: Host networking mode
docker run --network host ...

Same Docker Network

# docker-compose.yml
services:
  open-notebook:
    # ...
    environment:
      - OPENAI_COMPATIBLE_BASE_URL=http://lm-studio:1234/v1

  lm-studio:
    # your LM Studio container
    ports:
      - "1234:1234"

---

Text Generation WebUI Setup

Start with API Enabled

python server.py --api --listen

Configure Open Notebook

OPENAI_COMPATIBLE_BASE_URL=http://localhost:5000/v1

Docker Compose Example

services:
  text-gen:
    image: atinoda/text-generation-webui:default
    ports:
      - "5000:5000"
      - "7860:7860"
    volumes:
      - ./models:/app/models
    command: --api --listen

  open-notebook:
    image: lfnovo/open_notebook:v1-latest-single
    pull_policy: always
    environment:
      - OPENAI_COMPATIBLE_BASE_URL=http://text-gen:5000/v1
    depends_on:
      - text-gen

---

vLLM Setup

Start vLLM Server

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000

Configure Open Notebook

OPENAI_COMPATIBLE_BASE_URL=http://localhost:8000/v1

Docker Compose with GPU

services:
  vllm:
    image: vllm/vllm-openai:latest
    command: --model meta-llama/Llama-3.1-8B-Instruct
    ports:
      - "8000:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-notebook:
    image: lfnovo/open_notebook:v1-latest-single
    pull_policy: always
    environment:
      - OPENAI_COMPATIBLE_BASE_URL=http://vllm:8000/v1
    depends_on:
      - vllm

---

Adding Models in Open Notebook

Via Settings UI

1. Go to Settings → Models
2. Click Add Model in appropriate section
3. Select Provider: openai_compatible
4. Enter Model Name: exactly as the server expects
5. Enter Display Name: your preferred name
6. Click Save

Model Name Format

The model name must match what your server expects:

Server	Model Name Format
LM Studio	As shown in LM Studio UI
vLLM	HuggingFace model path
Text Gen WebUI	As loaded in UI
llama.cpp	Model file name

---

Testing Connection

Test API Endpoint

# Test chat completions
curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-name",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Test from Inside Docker

docker exec -it open-notebook curl http://host.docker.internal:1234/v1/models

---

Troubleshooting

Connection Refused

Problem: Cannot connect to server

Solutions:
1. Verify server is running
2. Check port is correct
3. Test with curl directly
4. Check Docker networking (use host.docker.internal)
5. Verify firewall allows connection

Model Not Found

Problem: Server returns "model not found"

Solutions:
1. Check model is loaded in server
2. Verify exact model name spelling
3. List available models: curl http://localhost:1234/v1/models
4. Update model name in Open Notebook

Slow Responses

Problem: Requests take very long

Solutions:
1. Check server resources (RAM, GPU)
2. Use smaller/quantized model
3. Reduce context length
4. Enable GPU acceleration if available

Authentication Errors

Problem: 401 or authentication failed

Solutions:
1. Check if server requires API key
2. Set OPENAI_COMPATIBLE_API_KEY
3. Some servers need any non-empty key

Timeout Errors

Problem: Request times out

Solutions:
1. Model may be loading (first request slow)
2. Increase timeout settings
3. Check server logs for errors
4. Reduce request size

---

Multiple Compatible Endpoints

You can use different compatible servers for different purposes:

# Chat model from LM Studio
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1

# Embeddings from different server
OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8080/v1

# TTS from Speaches
OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1

Add each as a separate model in Open Notebook settings.

---

Performance Tips

Model Selection

Model Size	RAM Needed	Speed
7B	8GB	Fast
13B	16GB	Medium
70B	64GB+	Slow

Quantization

Use quantized models (Q4, Q5) for faster inference with less RAM:

llama-3-8b-q4_k_m.gguf  → ~4GB RAM, fast
llama-3-8b-f16.gguf     → ~16GB RAM, slower

GPU Acceleration

Enable GPU in your server for much faster inference:

• LM Studio: Settings → GPU layers
• vLLM: Automatic with CUDA
• llama.cpp: --n-gpu-layers 35

---

Comparison: Native vs Compatible

Aspect	Native Provider	OpenAI Compatible
Setup	API key only	Server + configuration
Models	Provider's models	Any compatible model
Cost	Pay per token	Free (local)
Speed	Usually fast	Depends on hardware
Features	Full support	Basic features

Use OpenAI-compatible when:

• Running local models
• Using custom/fine-tuned models
• Privacy requirements
• Cost control

---

Related

• Local TTS Setup - Text-to-speech with Speaches
• AI Providers - All provider options
• Ollama Setup - Native Ollama integration