heath0xFF/DeepSeek-V4-Flash-0731-REAP-GGUF

DeepSeek-V4-Flash-0731-REAP-K160-GGUF

GGUF quantizations of 0xSero/DeepSeek-V4-Flash-0731-REAP — a REAP expert-pruned DeepSeek-V4-Flash checkpoint retaining 160 of 256 routed experts per MoE layer.

For llama.cpp on a single DGX Spark, use the newer REAP K216 IQ3_XXS build. It applies 0xSero's K216 map directly to Unsloth's stock quantized GGUF and copies retained expert rows byte-for-byte, avoiding the REAP-checkpoint → GGUF quantization path used here. The K216 build is 82.94 GiB and has generated successfully with configured 32K, 64K, and 128K contexts on GB10; 128K is the current recommendation. A 256K slot initializes but crashes on its first prompt in the current llama.cpp CUDA path.

Experimental runtime compatibility (updated 2026-08-01): Q2_K and Q4_K_M are not currently recommended. In controlled llama.cpp testing both repeatedly exhausted the output budget restating their plans without reaching an answer, including with the source model's sampling configuration. MXFP4 produced a normal answer in the matched comparison, but full GPU offload of both Q2_K and MXFP4 exposed backend kernel failures on NVIDIA GB10. A Strix Halo user has separately reported looping with Q2_K and MXFP4. See Runtime status before downloading.

• Base model: 0xSero/DeepSeek-V4-Flash-0731-REAP
• Original model: deepseek-ai/DeepSeek-V4-Flash-0731
• Format: GGUF
• Total params: ~193B
• Routed experts: 160 per layer (REAP-pruned from 256)
• Architecture: deepseek4

Variants

File	Quant	Size	Status
DeepSeek-V4-Flash-0731-REAP-K160-MXFP4.gguf	MXFP4 (native)	95 GB	Best result in limited testing; GPU-backend compatibility remains experimental
DeepSeek-V4-Flash-0731-REAP-K160-Q4_K_M.gguf	Q4_K_M	102 GB	Not recommended: repetitive/non-answering behavior in controlled testing
DeepSeek-V4-Flash-0731-REAP-K160-Q2_K.gguf	Q2_K	~65 GB	Not recommended: repetitive/non-answering behavior in controlled testing

Runtime status

These GGUFs use the deepseek4 architecture and an unusual REAP-compacted K160 expert shape. Runtime support should currently be treated as experimental and backend-dependent.

Testing on an NVIDIA DGX Spark / GB10 used llama.cpp b10223 (11924d4c1), built for CUDA architecture 121a with GGML_CUDA_FA_ALL_QUANTS=ON:

• The reported 262,144-token configuration loaded successfully with Q2_K, but full GPU execution failed at the first prompt in a CUDA quantized-matrix (MMQ) kernel.
• Full GPU execution of MXFP4 could likewise fail in its MMQ kernel depending on prompt shape.
• Keeping MoE weights on CPU (--cpu-moe) allowed diagnostic generation.
• With identical 512-token prompts, Q2_K repeatedly restated its plan and never answered. This remained true after removing extra candidate filters with --top-k 0 --min-p 0.
• Q4_K_M behaved the same in two 512-token tests: one with the reported --top-k 20 configuration and one with --top-k 0 --min-p 0. Both remained entirely in repetitive planning and produced no final answer.
• MXFP4 transitioned from reasoning into a concrete answer under the originally reported --top-k 20 settings.

The 102 GB Q4_K_M file does not safely fit on a 121 GiB GB10 system with --no-mmap: that configuration exhausted available memory before model initialization completed. A diagnostic run was possible with --load-mode mmap, 2K context, and --cpu-moe.

This does not establish that the GGUFs are corrupt. CPU/CPU-MoE execution can generate coherent text, while GPU behavior varies by quant, prompt shape, and backend. It does mean these files should not yet be presented as broadly compatible with llama.cpp GPU backends.

AMD Strix Halo is a different platform from NVIDIA GB10: it uses Radeon 8060S RDNA 3.5 graphics and will generally exercise Vulkan or HIP/ROCm rather than CUDA. If you encounter looping there, please report:

• exact GGUF filename;
• llama.cpp build/commit;
• backend (Vulkan, HIP/ROCm, or CPU);
• a minimal prompt and the beginning/end of the looping output;
• whether -ngl 0 (CPU-only) or --cpu-moe changes the result.

Sampling

The source checkpoint's generation_config.json specifies temperature: 1.0 and top_p: 1.0; it does not specify top-k, min-p, or a repetition penalty. For a clean diagnostic run, avoid adding candidate filters:

--temp 1.0 --top-p 1.0 --top-k 0 --min-p 0 --repeat-penalty 1.0

Increasing repeat penalty may hide a symptom, but it did not address the Q2_K quality failure observed above.

Serving

Known runtime paths:

• DwarfStar (ds4) — reference engine with native MXFP4 support.
• llama.cpp — deepseek4 support is under active development; validate the exact build and backend before relying on GPU inference.

License

MIT — inherited from the base model.

Acknowledgements

• DeepSeek for the base model
• 0xSero for the REAP pruning
• Cerebras Research for REAP
• llama.cpp contributors
• antirez for DwarfStar