woongkk/deepseek-v2-lite-bcq4-gguf

DeepSeek-V2-Lite, BCQ4 GGUF for flash-offloaded expert inference

This is deepseek-ai/DeepSeek-V2-Lite (the base model, not Chat), prepared for on-device inference where the routed experts stay in flash storage and only the parts that are needed are read into DRAM.

• Routed experts (78 tensors: 26 MoE layers x gate/up/down, 64 experts each) are 4-bit BCQ with group size 128. Each expert is stored as 4 separate bit planes. The first 2 planes on their own are a valid 2-bit weight. So a runtime can run an expert at 2 planes for a token that gives it a small router weight and at 4 planes for a token that gives it a large one, from the same file.
• Layer 0 is a dense FFN and has no routed experts, as in the original model. The MoE layers are 1 to 26.
• The other 326 tensors:
  • 218 are Q8_0: the attention weights (MLA), the dense FFN of layer 0, the shared experts, the token embeddings and the LM head.
  • 108 stay F32: the norms and the router.

This file does not load in upstream llama.cpp. The BCQ tensor type, the flash expert arena and the Hexagon NPU kernels come from a patch in the companion code repository https://github.com/Woong-DoubleK/s26-moe-offload (private; request access from the owner). This model also needs the repository's support for models whose first layers are dense.

File

file	bytes	md5	sha256
deepseek-v2-lite-bcq4-q8.gguf	9,505,094,656	6c79b8d07cacdaea52adba10b3a84d7d	e529a454e73e9b8861be76ac923fd4bec4c390e80845435312947cd324eceb76

MD5SUMS and SHA256SUMS hold the same values.

On the phone, the runtime packs the experts from this GGUF into a separate arena file that holds the 26 MoE layers. It is 8,097,107,968 bytes, and a correct pack has md5 f2200b84e17d7e989d9dfd36867c3bb4. The arena is not in this repository, because it is derived from the GGUF.

How it was made

1. The routed experts of deepseek-ai/DeepSeek-V2-Lite were quantized with AnyBCQ (plain 4-bit joint BCQ, group size 128, 128 calibration samples, no MRE steps). The run took 26.1 minutes on a GPU server.
2. The result was exported plane-major: 4 planes and 4 scale columns per projection.
3. The non-expert tensors were converted to Q8_0 with llama.cpp at commit ce8caa6 (convert_hf_to_gguf.py --outtype f16, then llama-quantize --pure).
4. Both parts were merged into one GGUF.

The quantizer state and driver are not public, so this GGUF cannot be regenerated from public inputs. It is published here instead.

Use

See the README of https://github.com/Woong-DoubleK/s26-moe-offload. In short:

• Target device: a Galaxy S26 Ultra (Snapdragon SM8850, Hexagon v81 NPU) on a userdebug build with adb root. Root is required.
• Build the patched llama.cpp.
• Push this GGUF to the phone, pack the arena on the phone, and run with per-token precision (an expert runs at 4 planes for a token whose raw router weight for it is above a threshold, else at 2).

License

This is a Derivative of the Model DeepSeek-V2-Lite and is distributed under the DEEPSEEK LICENSE AGREEMENT, Version 1.0 (see LICENSE, and NOTICE for what was changed).

• The use-based restrictions of paragraph 5 and Attachment A of that license apply to this file and to anything derived from it. Anyone you give it to must be told so.
• The code in the companion repository is MIT licensed separately.