NVIDIA Dynamo

Why can't you just throw a model on one GPU?

Modern LLMs like DeepSeek-R1 (671B parameters) or Llama-3-70B don't fit on a single GPU. Even an 80GB H100 can only hold ~40B parameters in FP16. So you must split the model across multiple GPUs (tensor parallelism), sometimes across multiple machines (pipeline parallelism).

But splitting creates problems:

How a request flows through Dynamo

Every request goes through these stages. Click the simulation button below to watch a request travel through the pipeline.

1. Frontend — the Rust HTTP server

Receives your /v1/chat/completions request. Handles tokenization, applies chat templates, validates it, and forwards to the Router. It's OpenAI-compatible — any OpenAI SDK works.

2. Smart Router — the brain

Decides which GPU worker handles your request. This is not random load balancing — it's intelligent.

It maintains a global radix tree — a data structure tracking which token prefixes are cached on which GPUs across the whole cluster.

3. Prefill Worker — the heavy lifter

Processes your entire prompt in a single forward pass. Compute-intensive — matrix multiplications across every attention layer for every input token simultaneously. Output: the KV cache.

4. NIXL — the data highway

Moves the KV cache from prefill GPU to decode GPU using the fastest available transport.

5. Decode Worker — the token factory

Receives KV cache and generates tokens one at a time. Each new token's attention is computed against all previous KV pairs, then its new K/V are appended. Continues until EOS or max_tokens.

Disaggregated Serving in Action

Watch how requests flow differently in aggregated vs disaggregated mode. Click to see 8 requests being processed.

KV-Aware Routing: Why it's a game changer

In multi-turn conversations, each follow-up shares a prefix with the previous message. Without smart routing, a new request lands on a GPU with no cached context, forcing expensive recomputation.

KVBM: 4-tier memory hierarchy for KV cache

GPU memory (HBM) is fast but tiny. For many concurrent users or long conversations, KV cache overflows. KVBM creates a 4-tier hierarchy, automatically spilling to cheaper storage.

The Planner: SLA-driven GPU allocation

Demand fluctuates. At 2 AM you need 2 GPUs; at noon, 32. The Planner monitors metrics and dynamically adjusts how many GPUs run prefill vs decode.

Three engines, one framework

Dynamo is engine-agnostic. Your choice depends on priority:

Feature Matrix

Quick Start: 0 to inference in 5 minutes

Fastest way — Docker containers with everything pre-installed:

# Pick your backend
docker run --gpus all --network host --rm -it \
  nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.8.1       # SGLang

docker run --gpus all --network host --rm -it \
  nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.8.1   # TensorRT-LLM

docker run --gpus all --network host --rm -it \
  nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.8.1          # vLLM

Start the frontend and a worker:

python3 -m dynamo.frontend --http-port 8000 --store-kv file

# SGLang
python3 -m dynamo.sglang --model-path Qwen/Qwen3-0.6B --store-kv file

# TensorRT-LLM
python3 -m dynamo.trtllm --model-path Qwen/Qwen3-0.6B --store-kv file

# vLLM (note: --model, not --model-path)
python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --store-kv file \
  --kv-events-config '{"enable_kv_cache_events": false}'

curl localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true,
    "max_tokens": 200
  }'

PyPI Install (no Docker)

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv venv && source venv/bin/activate && uv pip install pip

# Pick one:
uv pip install "ai-dynamo[sglang]"
uv pip install "ai-dynamo[vllm]"
pip install --pre --extra-index-url https://pypi.nvidia.com "ai-dynamo[trtllm]"

Build from Source

# 1. System deps
sudo apt install -y build-essential libhwloc-dev libudev-dev \
  pkg-config libclang-dev protobuf-compiler python3-dev cmake

# 2. Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# 3. Python + build tools
uv venv dynamo && source dynamo/bin/activate
uv pip install pip maturin

# 4. Build Rust bindings
cd lib/bindings/python && maturin develop --uv

# 5. Install
cd $PROJECT_ROOT
uv pip install -e lib/gpu_memory_service
uv pip install -e .

Kubernetes: single GPU to data center

Dynamo uses Grove — a topology-optimized Kubernetes operator that understands GPU relationships and manages the inference graph as one resource.

export NAMESPACE=dynamo-system VERSION=0.9.0

helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${VERSION}.tgz
helm install dynamo-crds dynamo-crds-${VERSION}.tgz -n default

helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${VERSION}.tgz
helm install dynamo-platform dynamo-platform-${VERSION}.tgz \
  -n ${NAMESPACE} --create-namespace

apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: llama-70b-disagg
spec:
  services:
    Frontend:
      replicas: 2
    PrefillWorker:
      replicas: 4
      resources:
        limits:
          nvidia.com/gpu: "2"
    DecodeWorker:
      replicas: 4
      resources:
        limits:
          nvidia.com/gpu: "2"

Production Recipes

Service Discovery

OpenAI-compatible API (drop-in replacement)

Tool Calling

Every knob you can turn

Frontend

Planner

Logging

export DYN_LOG=debug                                    # Everything
export DYN_LOG=dynamo_llm=debug,dynamo_runtime=info     # Per-crate

Hardware and software compatibility

GPU Support

OS and CPU

Version Compatibility

NVIDIA Dynamo — Apache 2.0 License — Rust + Python

GitHub · Docs · NVIDIA