Instructions to use deepseek-ai/DeepSeek-V4-Flash-0731 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V4-Flash-0731")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731", device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4-Flash-0731"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731

SGLang

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731
```

Here's how to run on SM120 / RTX 6000 Pro's (two of them) - not a troll/rickroll, I promise

#44

by pooterton - opened Aug 5

Discussion

pooterton

Aug 5

•

edited Aug 9

I have it running on 2 RTX 6000 Pro MaxQ's

Without using any offloading, you'll be capped at 343296 context length (that's all we got, timmy)

I started by setting up a venv

curl -LsSf https://astral.sh/uv/install.sh | sh
exec $SHELL -l

mkdir -p ~/serving && cd ~/serving
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto  # vllm==0.26.0, torch==2.11.0+cu130, flashinfer-python==0.6.14

You need to install the cuda toolkit as well as vllm will JIT compile a few things

FlashInfer
DeepGEMM (yes, at least under 0.26.0 a third-party vendored kernel is available and compiles DeepGemmFp8BlockScaledMMKernel and DEEPGEMM_MXFP4)
Triton will compile on fresh batch sizes once running

vllm.third_party.deep_gemm -> DeepGEMM E8M0 enabled on current platform

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2604/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit-13-3

Startup

OMP_NUM_THREADS=8 \
VLLM_ENGINE_READY_TIMEOUT_S=3600 \
CUDA_VISIBLE_DEVICES=0,2 \
vllm serve /tank/models/dsv4 \
  --served-model-name deepseek-v4-flash \
  --port 8010 \
  --trust-remote-code \
  --data-parallel-size 2 \
  --enable-expert-parallel \
  --disable-custom-all-reduce \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --gpu-memory-utilization 0.95 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 32
  --max-model-len 343296 \ # 1048576
  --kv-offloading-size 32 \
  --kv-offloading-backend native

Benchmarks

$ vllm bench serve   \
--model deepseek-v4-flash   \
--tokenizer /tank/models/dsv4   \
--base-url http://localhost:8010   \
--dataset-name random   \
--random-input-len 16000   \
--random-output-len 2048   \
--num-prompts 32   \
--max-concurrency 8
...
============ Serving Benchmark Result ============
Successful requests:                     32
Failed requests:                         0
Maximum request concurrency:             8
Benchmark duration (s):                  235.46
Total input tokens:                      512000
Total generated tokens:                  65536
Request throughput (req/s):              0.14
Output token throughput (tok/s):         278.33
Peak output token throughput (tok/s):    416.00
Peak concurrent requests:                12.00
Total token throughput (tok/s):          2452.81
---------------Time to First Token----------------
Mean TTFT (ms):                          8434.05
Median TTFT (ms):                        6081.17
P99 TTFT (ms):                           23351.07
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.63
Median TPOT (ms):                        25.03
P99 TPOT (ms):                           26.69
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.63
Median ITL (ms):                         21.34
P99 ITL (ms):                            23.42
==================================================




$ vllm bench serve \
  --model deepseek-v4-flash \
  --tokenizer /tank/models/dsv4 \
  --base-url http://localhost:8010 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 512 \
  --num-prompts 32 \
  --max-concurrency 4
...
============ Serving Benchmark Result ============
Successful requests:                     32
Failed requests:                         0
Maximum request concurrency:             4
Benchmark duration (s):                  93.57
Total input tokens:                      131072
Total generated tokens:                  16384
Request throughput (req/s):              0.34
Output token throughput (tok/s):         175.10
Peak output token throughput (tok/s):    256.00
Peak concurrent requests:                8.00
Total token throughput (tok/s):          1575.86
---------------Time to First Token----------------
Mean TTFT (ms):                          2861.64
Median TTFT (ms):                        1394.32
P99 TTFT (ms):                           14287.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.28
Median TPOT (ms):                        17.07
P99 TPOT (ms):                           19.34
---------------Inter-token Latency----------------
Mean ITL (ms):                           17.28
Median ITL (ms):                         16.22
P99 ITL (ms):                            17.33
==================================================

pooterton changed discussion title from How to run on SM120 / RTX 6000 Pro's (two of them) to Here's how to run on SM120 / RTX 6000 Pro's (two of them) - not a troll/rickroll, I promise Aug 5

CyySky

Aug 6

https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4dspark-v20.md

try this, the inference is much faster. hopefully vLLM/sglang will integrate the pro 6000 custom sm120 kernel soon.

pooterton

Aug 9

I'm noticing that this vllm setup shits itself on compaction when you have kv offloading enabled

(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332] EngineCore encountered a fatal error.
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332] Traceback (most recent call last):
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1323, in run_engine_core
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     engine_core.run_busy_loop()
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 2042, in run_busy_loop
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     executed = self._process_engine_step()
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1403, in _process_engine_step
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     outputs, model_executed = self.step_fn()
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]                               ^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 645, in step_with_batch_queue
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     scheduler_output = self.scheduler.schedule(self._should_throttle_prefills())
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/core/sched/scheduler.py", line 1174, in schedule
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     meta = self._build_kv_connector_meta(self.connector, scheduler_output)
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/core/sched/scheduler.py", line 1196, in _build_kv_connector_meta
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     return connector.build_connector_meta(scheduler_output)
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py", line 157, in bui
ld_connector_meta
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     return self.connector_scheduler.build_connector_meta(scheduler_output)
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py", line 1157, in bu
ild_connector_meta
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     store_jobs=self._build_store_jobs(scheduler_output),
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py", line 975, in _bu
ild_store_jobs
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]     assert len(offload_keys) == len(offload_block_ids)
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=120297) ERROR 08-09 16:48:07 [core.py:1332] AssertionError
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704] AsyncLLM output_handler failed.
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704] AsyncLLM output_handler failed.
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704] Traceback (most recent call last):
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704] Traceback (most recent call last):
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704]     outputs = await engine_core.get_output_async()
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704]     outputs = await engine_core.get_output_async()
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1061, in get_output_async
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704]   File "/home/poot/serving/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1061, in get_output_async
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704]     raise self._format_exception(outputs) from None
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704]     raise self._format_exception(outputs) from None
(ApiServer_1 pid=120300) ERROR 08-09 16:48:07 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(ApiServer_0 pid=120299) ERROR 08-09 16:48:07 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

$ cat deepseek.sh


#!/usr/bin/env bash
# DeepSeek-V4-Flash-0731 on blackwell GPUs 0 and 2. DP=2 over attention, EP over the
# MoE, MLA KV at FP8, CPU offload as a second-tier prefix cache.
set -euo pipefail

source /home/poot/serving/.venv/bin/activate

export CUDA_HOME=${CUDA_HOME:-/usr/local/cuda}
export PATH="$CUDA_HOME/bin:$PATH"
command -v nvcc >/dev/null || { echo "nvcc not on PATH; FlashInfer JIT will be disabled" >&2; exit 1; }

VENV=${VENV:-/home/poot/serving/.venv}
MODEL_PATH=${MODEL_PATH:-/tank/models/dsv4}
SERVED_NAME=${SERVED_NAME:-deepseek-v4-flash}
PORT=${PORT:-8010}
MAX_MODEL_LEN=${MAX_MODEL_LEN:-524288}
KV_OFFLOAD_GIB=${KV_OFFLOAD_GIB:-32}  # THIS IS PER RANK :hodor:
# CUDA_VISIBLE_DEVICES is meaningless without a stable enumeration order, and
# systemd won't inherit this from a shell profile
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,2
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
export OMP_NUM_THREADS=8
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export FLASHINFER_DISABLE_VERSION_CHECK=1

[[ -x $VENV/bin/vllm ]] || { echo "vllm not found at $VENV/bin/vllm" >&2; exit 1; }
[[ -d $MODEL_PATH ]] || { echo "model path not found: $MODEL_PATH" >&2; exit 1; }

# the offload pool is pinned host memory, allocated up front — if it doesn't
# fit we take the box down mid-startup rather than failing cleanly
available_gib=$(( $(awk '/^MemAvailable:/ {print $2}' /proc/meminfo) / 1024 / 1024 ))
required_gib=$(( KV_OFFLOAD_GIB + 16 ))
if (( available_gib < required_gib )); then
  echo "host RAM: ${available_gib} GiB available, need ~${required_gib} GiB for a ${KV_OFFLOAD_GIB} GiB offload pool" >&2
  echo "lower KV_OFFLOAD_GIB or free memory before starting" >&2
  exit 1
fi

# array form rather than backslash continuations: a comment inside a continued
# command silently drops every flag after it
args=(
  serve "$MODEL_PATH"
  --served-model-name "$SERVED_NAME"
  --port "$PORT"
  --trust-remote-code

  --data-parallel-size 2
  --enable-expert-parallel

  --kv-cache-dtype fp8
  --gpu-memory-utilization 0.965
  --max-model-len "$MAX_MODEL_LEN"

  # max-num-seqs 1 is load-bearing: the KV pool holds ~1.4 sessions at full
  # length, and any preemption desyncs the offloading connector's block
  # bookkeeping into a fatal assert
  --max-num-seqs 1
  --enable-chunked-prefill
  --max-num-batched-tokens 8192

  --kv-offloading-size "$KV_OFFLOAD_GIB"
  --kv-offloading-backend native

  --enable-auto-tool-choice
  --tool-call-parser deepseek_v4
  --reasoning-parser deepseek_v4

  #--attention-backend FLASHMLA_SPARSE_DSV4
)

echo "serving ${SERVED_NAME} on :${PORT} (ctx ${MAX_MODEL_LEN}, offload ${KV_OFFLOAD_GIB} GiB)"
exec "$VENV/bin/vllm" "${args[@]}"

dw1932

about 1 month ago

https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4dspark-v20.md

try this, the inference is much faster. hopefully vLLM/sglang will integrate the pro 6000 custom sm120 kernel soon.

I’d like to ask: what is the decoding speed after you’ve deployed it?

pooterton

about 1 month ago

https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4dspark-v20.md

try this, the inference is much faster. hopefully vLLM/sglang will integrate the pro 6000 custom sm120 kernel soon.

I’d like to ask: what is the decoding speed after you’ve deployed it?

I haven’t finished looking into this, but max context becomes something like 127k because of the extra hbm the decoder head etc consume. On two cards, for my use case, losing that much context, isn’t worth it

Also, kv cache offloading is busted on 0.26.0 (but that’s unrelated)

pooterton

about 1 month ago

As an update -rRunning PP=2 with max-num-batched-tokens=2048 nets me a KV cache pool of ~4.4M

No TP, no EP.

I tired PP=3 across three cards - buys a lot of kv at 2048 (blocksize 256), but nothing in terms of performance.

I don't have a fourth card despite the space for it, so I can't reliably run TP=4 (TP=2 isn't really worth it afaict).

I'm going to give local-inference-lab's container a whirl with a token batch of 2048 to see what KV turns into

dw1932

about 1 month ago

https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4dspark-v20.md

try this, the inference is much faster. hopefully vLLM/sglang will integrate the pro 6000 custom sm120 kernel soon.

I’d like to ask: what is the decoding speed after you’ve deployed it?

I haven’t finished looking into this, but max context becomes something like 127k because of the extra hbm the decoder head etc consume. On two cards, for my use case, losing that much context, isn’t worth it

Also, kv cache offloading is busted on 0.26.0 (but that’s unrelated)

Okay, thank you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment