--- license: mit base_model: - deepseek-ai/DeepSeek-V4-Flash - huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF tags: - deepseek - abliterated - uncensored - safetensors - vllm - blackwell - rtx-pro-6000 - sm_120 --- # DeepSeek V4 Flash Abliterated ***Ported to vLLM for massive throughput on RTX Pro 6000** — by [Fraser Price](https://x.com/fraserpricee)* This is probs the best uncensored model you can run on 2 x RTX Pro 6000 at high throughput (source: `✨ my opinion ✨`). Served on the `voipmonitor` vLLM fork it runs **~13× prefill and ~2.8× single-stream decode faster** than the widely available abliterated GGUF versions. --- ## Quick start (one command) > Requires Docker with the NVIDIA Container Toolkit, and ≥2 GPUs (see [Hardware](#hardware)). ```bash bash run.sh ``` `run.sh` defaults to this repo, so no arguments are needed. Everything is configured via env vars (see the table below) — e.g. `GPUS=0,1,2,3 TP=4 bash run.sh` to run on 4 GPUs. `run.sh` (included) will: 1. **Download the weights** (\~156 GB) into your Hugging Face cache (`\~/.cache/huggingface`, or `$HF_HOME`/`$HF_HUB_CACHE`) if not already present. 2. **Pull the inference image** if not already present. 3. **Serve** the model on `http://localhost:8000/v1` (OpenAI API spec). > **Optional — MTP speculative decoding** (~1.8× single-stream decode speed). The checkpoint ships its MTP layer; enable it with `MTP=2 bash run.sh` (see the table below). Override anything via env vars (all optional): | Variable | Default | Meaning | |---|---|---| | `GPUS` | `0,1` | GPU indices to use (comma-separated) | | `TP` | `2` | Tensor-parallel size — **set to the number of GPUs in `GPUS`** | | `MTP` | *(off)* | MTP speculative-decoding draft tokens (e.g. `2`); empty disables it | | `PORT` | `8000` | API port | | `MAX_MODEL_LEN` | `262144` | Context length | | `GPU_MEM_UTIL` | `0.92` | Fraction of VRAM to use | | `HF_REPO` | `fraserprice/DeepSeek-V4-Flash-Abliterated` | Hugging Face repo to download | | `MODEL_DIR` | *(HF cache)* | Serve from a specific local dir instead of downloading | | `SERVED_NAME` | `DeepSeek-V4-Flash-Abliterated` | Model name exposed by the API | | `IMAGE` | *(pinned `voipmonitor/vllm` build)* | Inference container image | Example on 4 GPUs: ```bash GPUS=0,1,2,3 TP=4 bash run.sh ``` --- ## Hardware/Image Tuned for **NVIDIA RTX Pro 6000 Blackwell (96 GB, sm_120).** It will run on other Blackwell-class GPUs, but the numbers below and the default flags assume RTX Pro 6000. It needs roughly **170 GB of total VRAM** for weights + KV cache at long context, split across GPUs by tensor parallelism. | Setup | Works? | |---|---| | **2 × RTX Pro 6000 Blackwell (96 GB) — `TP=2`** | ✅ **reference config** | | 4 × RTX Pro 6000 Blackwell — `TP=4` | ✅ (more KV headroom / throughput) | | 4 × 80 GB (e.g. H100/H200) — `TP=4` | ✅ (untested here, I believe can just use base vLLM) | | 1 × DXG Spark | ❓ idk why you didn't get an RTX Pro lmao (but lmk if you manage to run this and I can add to docs) | | 1 × anything | ❌ (blessed are the poor in VRAM 🙏) | The serving image targets **Blackwell (sm_120)** and CUDA 13.2. The DeepSeek-V4 architecture (sparse attention / lightning indexer, MLA, MTP, etc.) requires this image to run on RTX Pro. See [local-inference-lab/rtx6kpro](https://github.com/local-inference-lab/rtx6kpro) for image, bench, and serving details. --- ## Performance Measured under load on **RTX Pro 6000 Blackwell (connected via PCIe 5.0)**, 128 generated tokens/request, `TP` = number of GPUs. `MTP` is speculative decoding (`—` = off, `2` = 2 draft tokens). `PP` / `TG` are prefill / per-request decode throughput (tok/s); `Total` is aggregate end-to-end throughput across all concurrent requests (tok/s). Rows are paired off/on so the MTP decode gain is directly comparable. | GPUs | MTP | Prompt | Conc | TTFT | PP | TG | Total | |-----:|:---:|--------:|-----:|----------:|-------:|------:|-------:| | 2 | — | 1,000 | 1 | 132 ms | 7,613 | 108.6 | 870 | | 2 | 2 | 1,000 | 1 | 143 ms | 7,005 | 176.2 | 1,310 | | 2 | — | 1,000 | 3 | 275 ms | 4,190 | 76.7 | 1,746 | | 2 | 2 | 1,000 | 3 | 332 ms | 3,746 | 125.7 | 2,435 | | 2 | — | 10,000 | 1 | 1,036 ms | 9,658 | 108.0 | 4,581 | | 2 | 2 | 10,000 | 1 | 1,069 ms | 9,357 | 181.6 | 5,729 | | 2 | — | 10,000 | 3 | 2,138 ms | 5,302 | 57.1 | 6,600 | | 2 | 2 | 10,000 | 3 | 2,205 ms | 5,146 | 85.3 | 7,496 | | 2 | — | 100,000 | 1 | 11,575 ms | 8,640 | 107.6 | 7,850 | | 2 | 2 | 100,000 | 1 | 11,933 ms | 8,380 | 196.2 | 7,959 | | 2 | — | 100,000 | 3 | 23,579 ms | 5,153 | 33.0 | 8,161 | | 2 | 2 | 100,000 | 3 | 24,349 ms | 4,991 | 59.6 | 8,065 | | 4 | — | 1,000 | 1 | 119 ms | 8,437 | 117.9 | 946 | | 4 | 2 | 1,000 | 1 | 128 ms | 7,858 | 216.5 | 1,584 | | 4 | — | 1,000 | 3 | 255 ms | 4,539 | 98.0 | 2,175 | | 4 | 2 | 1,000 | 3 | 507 ms | 3,276 | 129.7 | 1,996 | | 4 | — | 10,000 | 1 | 842 ms | 11,886 | 118.4 | 5,293 | | 4 | 2 | 10,000 | 1 | 873 ms | 11,459 | 223.9 | 7,034 | | 4 | — | 10,000 | 3 | 1,741 ms | 6,521 | 69.0 | 8,041 | | 4 | 2 | 10,000 | 3 | 1,813 ms | 6,257 | 108.7 | 9,216 | | 4 | — | 100,000 | 1 | 9,530 ms | 10,494 | 118.5 | 9,445 | | 4 | 2 | 100,000 | 1 | 9,856 ms | 10,146 | 227.4 | 9,614 | | 4 | — | 100,000 | 3 | 19,226 ms | 6,299 | 40.0 | 10,028 | | 4 | 2 | 100,000 | 3 | 20,037 ms | 6,049 | 73.1 | 9,824 | MTP=2 lifts single-stream decode by **~1.6–1.9×** (e.g. 108 → 196 tok/s at 100k on `TP=2`) for a negligible TTFT cost. --- ## How it was made DeepSeek-V4-Flash's published abliterations existed only as a GGUF for the bespoke `ds4`/`llama.cpp` engines, which leave Blackwell hardware massively underutilised. Rather than re-run abliteration from scratch, the **refusal subspace was simply recovered and re-applied** from an existing abliteration: 1. **Recover the refusal directions.** The rank-3 refusal subspace in the 4096-dim residual stream was extracted from the delta between [huihui-ai's abliterated GGUF](https://huggingface.co/huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF) and the clean base. 2. **Re-apply to the official weights.** The same rank-3 projection was applied to every **residual-writing matrix** of the official checkpoint — attention output (`attn.wo_b`) and shared-expert down-projection (`shared_experts.w2`) across all 43 layers **and** the MTP layer (88 tensors total). Each tensor is dequantised → projected → requantised, with a fixed-point solver that compensates for FP8 requantisation so the intended projection survives quantisation. 3. **Leave everything else untouched.** The FP4-quantised routed experts and all gate/up projections are **byte-identical** to the official release; only the 88 FP8 residual-writing matrices change. **Validation:** on every edited tensor the achieved projection matched target within ~0.002, and all 33,880 non-target tensors were verified byte-identical to the base. The served model drops refusals while keeping correct tool-calling and coherent step-by-step reasoning. No formal evals yet — tested manually; it should match the base [huihui-ai abliteration](https://huggingface.co/huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF) at far higher speed. --- ## Credits - **[deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)** — the base model (MIT). - **[huihui-ai](https://huggingface.co/huihui-ai)** — the original abliteration ([`Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF`](https://huggingface.co/huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF)), from which the refusal directions were derived (MIT). - **[`voipmonitor/vllm`](https://hub.docker.com/r/voipmonitor/vllm)** — the Blackwell/CUDA-13.2 vLLM build that runs the DeepSeek-V4 architecture on RTX Pro. - **[local-inference-lab/rtx6kpro](https://github.com/local-inference-lab/rtx6kpro)** — the serving recipe (backends, flags, and benchmarks) this config is based on. This checkpoint contains **no huihui-ai weights** — only the official DeepSeek-V4-Flash weights with a refusal-direction projection applied. ## Disclaimer This is an **uncensored** model: the refusal subspace has been removed, so it will not decline requests the way the base model does. It is released **as-is, for research/lawful use only**. - **Not a safety-aligned model.** Because guardrails have been removed, outputs may be inaccurate, offensive, or harmful. Apply your own filtering, review, and safeguards before relying on or deploying it. - **No warranty.** The weights and scripts are provided "as is", without warranty of any kind, express or implied. You use them entirely at your own risk. - **You are responsible for what you generate.** The author does **not** endorse, encourage, or accept any responsibility or liability for how this model is used, including any unlawful, harmful, or unethical use. Outputs do not reflect the views of the author. ## License MIT, inheriting from the base model and the abliteration source. --- Built by Fraser Price — [@fraserpricee](https://x.com/fraserpricee)