Dual RTX3090 and 64GB DDR4 + 78GB Swap. Speed: ~1.78 TPS
My config is dual RTX3090 and 64GB DDR4 + 78GB Swap. (model is DeepSeek-V4-Flash-UD-Q8_K_XL.)
Speed: ~1.78 TPS
And for a single question:
What model are you?
Response:
I'm Claude, an AI assistant made by Anthropic. I don't have a specific version number visible in my current configuration.
Speed:
Prompt Processing: ~29.8 tokens/sec (Fast because GPUs handle the prefill phase).
Generation Speed: ~1.78 tokens/sec (No swap used only GPU/RAM).
The llama.cpp server script -with swap create- :
#!/usr/bin/env bash
# Serve unsloth/DeepSeek-V4-Flash-0731-GGUF (UD-Q8_K_XL) with llama.cpp and create self-contained swap create/teardown.
set -euo pipefail
LLAMA=/adat/ai/llama.cpp/llama-server
MODEL=/adat/ai/models/DeepSeek-V4-Flash-0731-UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
HOST=0.0.0.0
PORT=1234
CTX=65536 # 64k ctx: ~1.4 GB KV @ q8_0
ALIAS=DeepSeek-V4-Flash-UD-Q8_K_XL
SWAPFILE=/swapfile2
SWAPSIZE="${SWAPSIZE:-80G}"
SWAP_CREATED=0
cleanup() {
echo "[cleanup] Terminating server and cleaning up..."
# 1. Stop server and wait for process memory to be freed by kernel
if [[ -n "${LLAMA_PID:-}" ]]; then
kill "$LLAMA_PID" 2>/dev/null || true
wait "$LLAMA_PID" 2>/dev/null || true
fi
# 2. Tear down swap we created only after memory is freed
if [[ "$SWAP_CREATED" == "1" ]]; then
echo "[cleanup] Removing swap $SWAPFILE"
sudo swapoff "$SWAPFILE" 2>/dev/null || true
sudo rm -f "$SWAPFILE" 2>/dev/null || true
fi
}
trap cleanup EXIT INT TERM
[ -x "$LLAMA" ] || { echo "missing llama-server at $LLAMA" >&2; exit 1; }
[ -f "$MODEL" ] || { echo "missing model at $MODEL" >&2; exit 1; }
# ---- Swap Setup (FIXED LOGIC) -----------------------------------------------
if swapon --show=NAME | grep -qx "$SWAPFILE"; then
echo "[swap] $SWAPFILE already active, leaving as-is"
elif [[ -f "$SWAPFILE" ]]; then
echo "[swap] Activating existing $SWAPFILE"
sudo swapon "$SWAPFILE"
SWAP_CREATED=1
else
echo "[swap] Creating $SWAPFILE ($SWAPSIZE) and enabling..."
# fallocate is fast; fallback to dd if fallocate isn't supported by filesystem
if ! sudo fallocate -l "$SWAPSIZE" "$SWAPFILE" 2>/dev/null; then
echo "[swap] fallocate failed, falling back to dd..."
# Convert '80G' -> 80000 (roughly for dd)
SIZE_NUM=$(echo "$SWAPSIZE" | tr -d 'Gg')
sudo dd if=/dev/zero of="$SWAPFILE" bs=1M count=$((SIZE_NUM * 1024)) status=progress
fi
sudo chmod 600 "$SWAPFILE"
sudo mkswap "$SWAPFILE" >/dev/null
sudo swapon "$SWAPFILE"
SWAP_CREATED=1
fi
free -h | head -2
# ---- Launch llama-server ----------------------------------------------------
"$LLAMA" \
--model "$MODEL" \
--alias "$ALIAS" \
--host "$HOST" --port "$PORT" \
--ctx-size "$CTX" \
--threads 20 \
--threads-batch 40 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-ram 16384 \
--n-gpu-layers 11 \
--tensor-split 1,1 \
-fa on \
--jinja \
--cont-batching \
--top-p 0.95 \
--temp 0.7 \
--repeat-penalty 1.2 \
-b 2048 \
-ub 2048 \
--metrics &
LLAMA_PID=$!
wait "$LLAMA_PID" ```
I dont know why it say model is Claude 😄
My config is dual RTX3090 and 64GB DDR4 + 78GB Swap. (model is DeepSeek-V4-Flash-UD-Q8_K_XL.)
Speed: ~1.78 TPS
Man, use lower quants. I tried DeepSeek-V4-Flash-UD-Q4_K_XL (155Gb) on my rig with dual 3090 and 128Gb of DDR4 RAM. Around 30Gb where going to swap, but after that model was working quite nice giving me 4tps. The main problem was very uneven load of 3090s, one was fully loaded, the other was sitting half empty.
I dont know why it say model is Claude 😄
It is normal for OS models to confuse their names.
Q4 vs Q8: Night and day is the quality difference and only 7GB bigger (@162GB). Doesn't matter.
I dont know why it say model is Claude 😄
Could it be because it was trained/fine-tuned on Claude? 😉
Q4 vs Q8: Night and day is the quality difference and only 7GB bigger (@162GB). Doesn't matter.
I dont know why it say model is Claude 😄
Could it be because it was trained/fine-tuned on Claude? 😉
And Claude in Chinese says it's Deepseek/Qwen. What's your point?
Prove it. 😃
It is normal for OS models to confuse their names.
@perelmanych No it is not. I have never seen that happen on any model before. They always give their own companies name. The fact that you freely just say things like that is wild or you are using low quality quaints that are lobotomized.
It is normal for OS models to confuse their names.
@perelmanych No it is not. I have never seen that happen on any model before. They always give their own companies name. The fact that you freely just say things like that is wild or you are using low quality quaints that are lobotomized.
I've seen it fairly often, and many reports of it on r/locallama.
Not sure if that qualifies as 'normal' or just 'not unheard-of'.
I'm wondering if those who claim this normal for a model to claim it is a different model are asking Chinese models? Because open source llm space is dominated by Chinese? I have zero experience in this, just a question.
Prompt processing seems really slow for a 3090, let alone two - I suspect you really need to crank up b and ub a lot. I get pcie bandwidth limited up to 6k batch size with an rtx 5000 mobile (Turing generation); being disk-read-limited and on a pair of much faster GPUs, you probably need it as high as it will go without crashing !
FWIW, I get around 150 pp on that rtx 5000 + 128gb ddr4 laptop, with the ud-iq3_xxs quant; but TG is also a bit under 2t/s.
Also, you probably don't need the swap - do you see any performance improvements over llama.cpp's mmap mode (enabled by default nowadays) ?
Running inference with weights on disk is the cause of the slow tg.
I’m running DeepSeek-V4-Flash-0731-UD-Q3_K_XL on a single RTX 4090 with 128 GB of system RAM, and I’m getting around 11 tokens per second.
Would adding a second RTX 4090 provide a meaningful performance improvement, or would PCIe communication and model-splitting overhead limit the gains? I’d appreciate hearing from anyone who has tested a similar dual-4090 setup.
Using 5090m and 192gb 4000MT Notebook,
PP seems on the low side
@echo off
"E:\llama_ai\llama-b10243-bin-win-cuda-13.3-x64\llama-server.exe" ^
-hf "unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL" ^
--alias "DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL" ^
--gpu-layers 999 ^
-ot "\.[0-9]+\.ffn_(gate|up|down)_exps\.=CPU" ^
--no-mmproj ^
--flash-attn on ^
--no-mmap ^
--cache-type-k q8_0 ^
--cache-type-v q8_0 ^
--ctx-size 200536 ^
--ubatch-size 4048 ^
--batch-size 4048 ^
--reasoning on ^
--threads 16 ^
--parallel 1 ^
--host 0.0.0.0 ^
--port 11434 ^
--presence_penalty 0.0 ^
--temp 1.00 ^
--top-p 1.00 ^
--min-p 0.00 ^
--top-k 20 ^
--jinja
pause
How to improve? I would like to have around 200-400PP especially since I have not even 20k context reached here..
Using 5090m and 192gb 4000MT Notebook,
PP seems on the low side
2x3090 here with 3600MT yielding 8.4
I would try:
--gpu-layers 999 ^ -ot "\.[0-9]+\.ffn_(gate|up|down)_exps\.=CPU" ^replace with -fit on
--cache-type-k q8_0 ^ --cache-type-v q8_0 ^Documentation suggests to not specify any cache type
--threads 16 ^Documentation suggests to run threads = physical cores / 2
--presence_penalty 0.0 ^ --temp 1.00 ^ --top-p 1.00 ^ --min-p 0.00 ^ --top-k 20 ^Unrelated to speed, but I do not override the model's --jinja
--jinja
Each batch basically needs the entire model streamed from memory to gpu, so you're probably memory or pcie bandwidth-limited at this batch size.
Try larger b and ub values, at least 8192 ! (That won't fix small-batch processing speed though, there's no solution for that other than unified memory or huge VRAM.)
And for TG performance, I'd go with -fit off and replace the -ot regex with --n-cpu-moe tuned as low as it will go without crashing (41 here on 16gb gpu, you can probably go a bit lower)
q8_0 kv in modern llama.cpp is just fine (horror stories about 8-bit kv are either from before Hadamard rotation was implemented a few months back, or a confusion between q8_0 and fp8)