kvcache-ai-ktransformers/doc/en/kt-kernel/deepseek-v3.2-sglang-tutorial.md

Running DeepSeek V3.2 with SGLang and KT-Kernel

This tutorial demonstrates how to run DeepSeek V3.2 model inference using SGLang integrated with KT-Kernel for CPU-GPU heterogeneous inference. This setup enables efficient deployment of large MoE models by offloading experts to CPU.

Table of Contents

• Hardware Requirements
• Prerequisites
• Step 1: Download Model Weights
• Step 2: Quantize CPU Weights
• Step 3: Launch SGLang Server
• Step 4: Send Inference Requests

Hardware Requirements

Minimum Configuration:

• GPU: NVIDIA L20 48GB (or equivalent with at least 27GB VRAM available)
• CPU: Intel Xeon with AMX support (e.g., Sapphire Rapids)
• RAM: At least 350GB system memory for INT4 quantization
• Storage: ~1TB for model weights (FP8 + INT4 quantized)

Tested Configuration:

• GPU: NVIDIA L20 48GB
• CPU: Intel(R) Xeon(R) Platinum 8488C
• RAM: 2TB DDR5
• OS: Linux (Ubuntu 20.04+ recommended)

Prerequisites

Before starting, ensure you have:

1. KT-Kernel installed - Follow the installation guide
2. SGLang installed - Install the kvcache-ai fork: pip install kt-kernel sglang-kt or run ./install.sh from the ktransformers root
3. CUDA toolkit - Compatible with your GPU (CUDA 11.8+ recommended)
4. Hugging Face CLI - For downloading models:

   pip install huggingface-hub
   

Step 1: Download Model Weights

DeepSeek V3.2 requires downloading model repositories:

1. DeepSeek-V3.2
2. DeepSeek-V3.2-Speciale

# Create a directory for models
mkdir -p /path/to/models
cd /path/to/models

# Download DeepSeek-V3.2 (FP8 weights for GPU)
huggingface-cli download deepseek-ai/DeepSeek-V3.2 \
  --local-dir /path/to/deepseek-v3.2

# Download DeepSeek-V3.2-Speciale (if needed)
huggingface-cli download deepseek-ai/DeepSeek-V3.2-Speciale \
  --local-dir /path/to/deepseek-v3.2-speciale

Note: Replace /path/to/models with your actual storage path throughout this tutorial.

Step 2: Quantize CPU Weights

Convert the FP8 GPU weights to INT4 quantized CPU weights using the provided conversion script.

Conversion Command

For a 2-NUMA system with 60 physical cores:

cd /path/to/ktransformers/kt-kernel

python scripts/convert_cpu_weights.py \
  --input-path /path/to/deepseek-v3.2 \
  --input-type fp8 \
  --output /path/to/deepseek-v3.2-INT4 \
  --quant-method int4 \
  --cpuinfer-threads 60 \
  --threadpool-count 2 \
  --no-merge-safetensor

Step 3: Launch SGLang Server

Start the SGLang server with KT-Kernel integration for CPU-GPU heterogeneous inference.

Launch Command

For single NVIDIA L20 48GB + 2-NUMA CPU system:

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 30000 \
  --model /path/to/deepseek-v3.2 \
  --kt-weight-path /path/to/deepseek-v3.2-INT4 \
  --kt-cpuinfer 60 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 1 \
  --attention-backend triton \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 4096 \
  --max-running-requests 32 \
  --max-total-tokens 40000 \
  --served-model-name DeepSeek-V3.2 \
  --enable-mixed-chunk \
  --tensor-parallel-size 1 \
  --enable-p2p-check \
  --disable-shared-experts-fusion \
  --kt-method AMXINT4

Resource Usage

• GPU VRAM: ~27GB (for 1 GPU expert per layer + attention)
• System RAM: ~350GB (for INT4 quantized CPU experts)

Step 4: Send Inference Requests

Once the server is running, you can send inference requests using the OpenAI-compatible API.

Basic Chat Completion Request

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V3.2",
    "stream": false,
    "messages": [
      {"role": "user", "content": "hi"}
    ]
  }'

Example Response

{
  "id": "adbb44f6aafb4b58b167e42fbbb1eed3",
  "object": "chat.completion",
  "created": 1764675126,
  "model": "DeepSeek-V3.2",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hi there! 👋 \n\nThanks for stopping by! How can I help you today? Feel free to ask me anything - I'm here to assist with questions, explanations, conversations, or whatever you need! 😊\n\nIs there something specific on your mind, or would you like to know more about what I can do?",
        "reasoning_content": null,
        "tool_calls": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "matched_stop": 1
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "total_tokens": 72,
    "completion_tokens": 67,
    "prompt_tokens_details": null,
    "reasoning_tokens": 0
  },
  "metadata": {
    "weight_version": "default"
  }
}

Additional Resources

• KT-Kernel Documentation
• DeepSeek V3.2 Model Card
• SGLang GitHub