taurusduan/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF

TL;DR: DeepSeek's 552B-parameter backbone plus 196B-parameter Engram memory, reduced from the original 510.3 GB FP8/FP4 checkpoint to a 426.1 GB package including the DSpark draft model and vision projector.

DeepSeek-V4.1-Flash · GSQ-RCO GGUF

Non-uniform GGUF quantization produced with GSQ and RCO, with the DSpark draft model and the vision projector.

Independent community reproduction. These files were produced by a third party using the published GSQ and RCO methods. They are not an IST-DASLab release and carry no endorsement from the authors of either paper.

---

Overview

This repository provides a GGUF quantization of deepseek-ai/DeepSeek-V4.1-Flash at 3.0 bits per weight for the backbone, together with the DSpark speculative-decoding draft and the BF16 vision projector (mmproj). RCO assigns quantization types to eligible backbone projections according to their sensitivity, within an exact total size budget. The Engram conditional-memory tables use Q8_0 and are excluded from the search.

Method	Description
GSQ (Gumbel-Softmax Quantization, paper, code)	Post-training scalar quantization that jointly learns per-coordinate grid assignments and per-group scales through a Gumbel-Softmax relaxation.
RCO (Riemannian Constrained Optimization, paper, code)	Assigns one of K quantization types to each of N tensors under an exact total size budget, reformulated as a smooth Riemannian manifold in logit space.

Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

---

Read this before using these files

The file is 415.76 GB. Engram (208.90 GB at Q8_0) is intended for host memory (-ot 'engram_embd=CPU'); the remaining 206.86 GB contains the backbone and file metadata. Allow additional memory for the KV cache, runtime buffers, draft model and vision projector.

---

Available files

File	bpw	Size	Notes
DeepSeek-V4.1-Flash-GSQ-RCO-3.0bit.gguf	2.9986 (backbone)	415.76 GB	206.86 GB backbone + 208.90 GB Engram at Q8_0
DeepSeek-V4.1-Flash-DSpark-draft.gguf		9.32 GB	DSpark speculative-decoding draft
DeepSeek-V4.1-Flash-mmproj-BF16.gguf	16	0.97 GB	Vision encoder and projector

---

Results

MMLU-Pro: no reasoning; 2,048-token context limit. Scores use Spark's fixed 2,000-question subset with raw zero-shot prompts and the log probabilities of single-token answer letters. These evaluations used F32 matrix multiplication and F32 KV caches in the supplied runtime, with Flash Attention disabled.

Test	GSQ-RCO 3.0-bit result
MMLU-Pro	61.00% (1,220/2,000)
Held-out PPL, 4,088 scored positions	3.4845570141
IFEval, 16 prompts	16 passed the strict checker; 15 completed, 1 truncated
GSM8K, 8 problems	8 completed correctly

All 23 completed responses passed official parsing. The truncated IFEval response was not parsed. The evaluation guide includes the full protocol and saved responses.

---

Download

hf download pfeifferj/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF --local-dir .

For text inference, follow the supplied runtime build guide and reproduction instructions. Vision and speculative decoding have not been validated.

---

Quantization procedure

1. GSQ generated a candidate bank for backbone projections across 40 layers and 384 experts. The bank is 1.1 TB; Engram tables and the vision tower were excluded from the search.
2. RCO selected the allocation by validation teacher KL. No step improved on step 0, so it kept the initial allocation.
3. The size budget included packed weights, metadata and alignment. A roundtrip check verified the payload.

Source revision: dba1be0a.

---

Citation

If you use these files, please cite this release together with the base model and both methods.

This release

@misc{dsv41flashgsqrco2026,
  title        = {DeepSeek-V4.1-Flash GSQ-RCO GGUF quantization},
  author       = {Josephine Pfeiffer},
  year         = {2026},
  publisher    = {Hugging Face},
  doi          = {10.57967/hf/10402},
  howpublished = {\url{https://huggingface.co/pfeifferj/DeepSeek-V4.1-Flash-GSQ-RCO-GGUF}}
}

Base model

@misc{deepseekai2026deepseekv41flash,
  title  = {DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression},
  author = {DeepSeek-AI},
  year   = {2026},
  url    = {https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash}
}

Methods

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

---

Acknowledgements

Huge kudos to the Deep Algorithms and Systems Lab (DASLab) at the Institute of Science and Technology Austria for developing GSQ and RCO and for releasing the papers and reference implementations publicly.

Thanks also to ella for giving me the idea and supporting me at every step of the process, and Chris van Hoof (Red Hat) for contributing the compute on which this quantization was produced and evaluated.

---

License

These quantized weights inherit the license of the base model, deepseek-ai/DeepSeek-V4.1-Flash (MIT); the original license text ships as LICENSE. The GSQ and RCO tooling is released by DASLab under its own repository licenses.