it works and here is how you compile llama.cpp
#1
by aydintb - opened
cd ~/Projects
git clone --branch my_build_deepseek41 --depth 1 \
https://github.com/smalinin/llama.cpp.git llama.cpp-dsv41
cd llama.cpp-dsv41
cmake -B build \
-DGGML_CUDA=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build -j$(nproc) --target llama-server
and this is my llmscript
#!/bin/bash
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MODEL="$SCRIPT_DIR/DeepSeek-V4.1-Flash-Q2_K-00001-of-00007.gguf"
TFILE="$SCRIPT_DIR/deepseek-ai-DeepSeek-V4.1.jinja"
~/Projects/llama.cpp-dsv41/build/bin/llama-server \
-m "$MODEL" \
--port 8080 \
--host 0.0.0.0 \
--ctx-size $((8*8*1024)) \
--batch-size 2048 \
--ubatch-size 256 \
--parallel 1 \
--n-gpu-layers auto \
--fit on \
--fit-ctx 64000 \
--fit-target 2048 \
--load-mode mmap \
--lazy-mode auto \
--flash-attn on \
--presence-penalty 0.0 \
--repeat-penalty 1.02 \
--no-warmup \
--no-context-shift \
--jinja \
--chat-template-file "$TFILE" \
--chat-template-kwargs '{"reasoning_effort":80,"enable_thinking":true}' \
--reasoning-format deepseek \
--no-reasoning-preserve \
--no-prefill-assistant
aydintb changed discussion status to closed
I built with (it is for RTX3090 + RTX4090 arch) :
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DGGML_CUDA_NCCL=ON \
-DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_FA=ON '-DCMAKE_CUDA_ARCHITECTURES=86-real;89-real'
cmake --build build -j10
Note:
- It is NOT final version of code.
- Engramm data is supported already.
- Also I will prepare new better version of GGUF files and DSpark head, when it will be ready.
- And this code has already improved support for GLM-5.3-Flash and Qwen3.8-Flash-Next better perf, mix MTP/image in input data and etc.
smalinin changed discussion status to open