Inference engines

Docker Model Runner supports two inference engines: llama.cpp and vLLM. Each engine has different strengths, supported platforms, and model format requirements. This guide helps you choose the right engine and configure it for your use case.

Engine comparison

llama.cpp

llama.cpp is the default inference engine in Docker Model Runner. It's designed for efficient local inference and supports a wide range of hardware configurations.

Platform support

Model format: GGUF

llama.cpp uses the GGUF format, which supports efficient quantization for reduced memory usage without significant quality loss.

Quantization levels

Recommended: Q4_K_M offers the best balance of quality and memory usage for most use cases.

Pulling quantized models

Models on Docker Hub often include quantization in the tag:

$ docker model pull ai/llama3.2:3B-Q4_K_M

Using llama.cpp

llama.cpp is the default engine. No special configuration is required:

$ docker model run ai/smollm2

To explicitly specify llama.cpp when running models:

$ docker model run ai/smollm2 --backend llama.cpp

llama.cpp API endpoints

When using llama.cpp, API calls use the llama.cpp engine path:

POST /engines/llama.cpp/v1/chat/completions

Or without the engine prefix:

POST /engines/v1/chat/completions

vLLM

vLLM is a high-performance inference engine optimized for production workloads with high throughput requirements.

Platform support

Important

vLLM requires an NVIDIA GPU with CUDA support. It does not support CPU-only inference.

Model format: Safetensors

vLLM works with models in Safetensors format, which is the standard format for HuggingFace models. These models typically use more memory than quantized GGUF models but may offer better quality and faster inference on powerful hardware.

Setting up vLLM

Docker Engine (Linux)

Install the Model Runner with vLLM backend:

$ docker model install-runner --backend vllm --gpu cuda

Verify the installation:

$ docker model status
Docker Model Runner is running

Status:
llama.cpp: running llama.cpp version: c22473b
vllm: running vllm version: 0.11.0

Docker Desktop (Windows with WSL2)

1. Ensure you have:

   • Docker Desktop 4.54 or later
   • NVIDIA GPU with updated drivers
   • WSL2 enabled

2. Install vLLM backend:

   $ docker model install-runner --backend vllm --gpu cuda
   

Running models with vLLM

vLLM models are typically tagged with -vllm suffix:

$ docker model run ai/smollm2-vllm

To specify the vLLM backend explicitly:

$ docker model run ai/model --backend vllm

vLLM API endpoints

When using vLLM, specify the engine in the API path:

POST /engines/vllm/v1/chat/completions

vLLM configuration

HuggingFace overrides

Use --hf_overrides to pass model configuration overrides:

$ docker model configure --hf_overrides '{"max_model_len": 8192}' ai/model-vllm

Common vLLM settings

vLLM and llama.cpp performance comparison

Running both engines

You can run both llama.cpp and vLLM simultaneously. Docker Model Runner routes requests to the appropriate engine based on the model or explicit engine selection.

Check which engines are running:

$ docker model status
Docker Model Runner is running

Status:
llama.cpp: running llama.cpp version: c22473b
vllm: running vllm version: 0.11.0

Engine-specific API paths

Managing inference engines

Install an engine

$ docker model install-runner --backend <engine> [--gpu <type>]

Options:

• --backend: llama.cpp or vllm
• --gpu: cuda, rocm, vulkan, or metal (depends on platform)

Reinstall an engine

$ docker model reinstall-runner --backend <engine>

Check engine status

$ docker model status

View engine logs

$ docker model logs

Packaging models for each engine

Package a GGUF model (llama.cpp)

$ docker model package --gguf ./model.gguf --push myorg/mymodel:Q4_K_M

Package a Safetensors model (vLLM)

$ docker model package --safetensors ./model/ --push myorg/mymodel-vllm

Troubleshooting

vLLM won't start

1. Verify NVIDIA GPU is available:

   $ nvidia-smi
   

2. Check Docker has GPU access:

   $ docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi
   

3. Verify you're on a supported platform (Linux x86_64 or Windows WSL2).

llama.cpp is slow

1. Ensure GPU acceleration is working (check logs for Metal/CUDA messages).

2. Try a more aggressive quantization:

   $ docker model pull ai/model:Q4_K_M
   

3. Reduce context size:

   $ docker model configure --context-size 2048 ai/model
   

Out of memory errors

1. Use a smaller quantization (Q4 instead of Q8).
2. Reduce context size.
3. For vLLM, adjust gpu_memory_utilization:

   $ docker model configure --hf_overrides '{"gpu_memory_utilization": 0.8}' ai/model
   

What's next

• Configuration options - Detailed parameter reference
• API reference - API documentation
• GPU support - GPU configuration for Docker Desktop