caiovicentino1/DeepSeek-V4.1-Flash-HLWQ-Engram-Q4

DeepSeek-V4.1-Flash · HLWQ Q4 Engram

Sibling repo with 5-bit tables (half the reconstruction error, +25 GB): DeepSeek-V4.1-Flash-HLWQ-Engram-Q5.

What this is: a complete, drop-in copy of DeepSeek-V4.1-Flash in which the two Engram hash tables (layers.1.engram.embed and layers.14.engram.embed, 384 M rows × 256 each, ~196 B parameters) are re-encoded with HLWQ (Hadamard-Lloyd Weight Quantization) at 4 bits per weight. All 46 other shards are byte-identical to the official release; shards 47 and 48 keep their small q/k/wkv tensors and drop the FP8 tables. Download this repo, run serve.sh, done.

Why: the FP8 tables take 203 GB of host RAM (or NVMe) when served with the row-store adapters used on consumer/workstation GPUs. HLWQ Q4 halves that to **101 GB**, and the download drops from 510 GB to ~408 GB, with a per-row reconstruction error of 9.7 % relative RMSE (cosine 0.9954), and on the same server and prompts, next-token quality matches FP8 on prose, costs about 0.015 nats/token on code and about 0.04 nats/token on long-context retrieval prompts, with all needles still retrieved (table below).

	FP8 (official)	HLWQ Q4 (this repo)
Bytes per row (256 dims)	264 (256 fp8 + 8 e8m0)	132 (128 packed nibbles + 2 × fp16 norms)
Both layers on disk / RAM	~203 GB	~101 GB
Rel. RMSE vs FP8 rows	—	0.0967
Mean cosine vs FP8 rows	—	0.99536
Lookup cost (1024 rows, RAM mode)	~50 µs	~50 µs + 3 tiny torch ops

Quick start (4× RTX PRO 6000 or any 4× 96 GB SM120 box, ≥ 128 GB host RAM)

hf download caiovicentino1/DeepSeek-V4.1-Flash-HLWQ-Engram-Q4 --local-dir /models/v41-hlwq
docker run --gpus all --ipc=host --shm-size 32g -v /models/v41-hlwq:/model -p 8000:8000 \
  lmsysorg/sglang:dev-dsv41 bash -c "python /model/patches/patch_mask.py && API_KEY=change-me bash /model/serve.sh"

serve.sh sets PYTHONPATH=/model/adapter (the Engram row store + HLWQ dequant), points DSV41_SOURCE at the model, serves the tables from host RAM (OFFLOAD_MODE=nvme if you have < 128 GB), and launches SGLang with the flags validated here (TP4/EP4, dsv4 attention, MXFP4 MoE, DSpark block 5, 16 concurrent requests, 409 600 context). Every knob is an env var; see the header of the script. The OpenAI-compatible API answers on port 8000 as deepseek-v4.1-flash; pass chat_template_kwargs: {"thinking": true} to enable reasoning (off by default in this build).

Blackwell datacenter GPUs (B200/B300) do not need patches/; only SM120 does.

Files

model-000{01..46}-of-00048.safetensors   byte-identical to deepseek-ai/DeepSeek-V4.1-Flash
model-00047-of-00048.safetensors         layers.1.engram.{q_weight,k_weight,wkv.*} only (157 MB)
model-00048-of-00048.safetensors         layers.14.engram.{q_weight,k_weight,wkv.*} only (157 MB)
model.safetensors.index.json             official index minus the two FP8 Engram tables
engram_hlwq_q4_layer1-0000{1,2}-of-00002.safetensors    layers.1.engram.embed.hlwq_codes  U8  [rows, 128]  (two row-range parts, ~23.6 GiB each)
                                                        layers.1.engram.embed.hlwq_norms  F16 [rows, 2]
engram_hlwq_q4_layer14-0000{1,2}-of-00002.safetensors   same for layer 14
hlwq_config_layer{1,14}.json             nbits, centroids, packing, part list with row ranges, reconstruction stats, provenance
config.json, tokenizer*, encoding/, inference/, LICENSE   from the official release
adapter/                                 engram_backend_hlwq.py, row_store_hlwq.cpp, sitecustomize.py
patches/                                 SM120 prefill fix (see patches/README.md)
tools/                                   hlwq_engram_quant.py (the quantizer), split_hlwq.py, ab_tf2.py / ab_engram.py (the validation)
serve.sh                                 one-command launcher

How HLWQ encodes a row

Each 256-dim row is split into two blocks of 128:

1. Normalize the block by its L2 norm (stored as fp16).
2. Rotate with the normalized Walsh-Hadamard matrix H₁₂₈. After rotation the coordinates are close to N(0, 1/128).
3. Scale by √128 and map each coordinate to the nearest of 16 Lloyd-Max centroids for N(0,1) (100 fixed-point iterations of the conditional-expectation update).
4. Pack the 4-bit codes two per byte (low nibble = even index).

Dequantization is the mirror image and costs three tensor ops per lookup: centroids[codes] → (·) @ H₁₂₈ → × norm. Because H is orthogonal and its own inverse, nothing else is stored. The centroids used are in hlwq_config_layer*.json.

No calibration data, no Hessian, no gradient: the quantizer is deterministic and runs on CPU (≈ 0.2 M rows/s per 96 threads).

Serving (SGLang, 4× RTX PRO 6000, no NVLink)

This repo was produced and validated on the recipe from 0xSero/deepseek-v4.1-flash-4x-rtx-pro-6000 (SGLang dev-dsv41, Engram tables served from host RAM or NVMe through a cudaLaunchHostFunc row store). adapter/sitecustomize.py autodetects hlwq_config_layer1.json in the model directory and installs engram_backend_hlwq.py; with the official FP8 checkpoint it falls back to 0xSero's FP8 row store, so the same adapter serves both. Everything else (hashing, gating, wkv, TP all-reduce) is the model's own code. The SM120 prefill fix is in patches/ (context in flashinfer#5095).

Validation (same server, same prompts, FP8 vs HLWQ)

Measured on 4× RTX PRO 6000 (SM120), SGLang dev-dsv41, DSpark block 5, TP4/EP4, 2026-09-10.

Teacher-forced next-token probability on 25 fixed FP8 greedy continuations (10 pt-BR prose, 10 code, 5 needle-in-a-haystack from 3 k to 52 k prompt tokens), 8 positions each (16 for needles). Lower NLL is better. Both columns were measured with the server idle, where two consecutive passes reproduce to the fourth decimal; under concurrent traffic the same metric moves by up to ±0.015 because batching changes the numerics, so compare only idle-to-idle.

Category	FP8 NLL / token	HLWQ Q4 NLL / token	top-1 agreement FP8	top-1 agreement HLWQ
pt-BR prose (80 positions)	0.0668	0.0601	98.8 %	100.0 %
code (80 positions)	0.1086	0.1233	97.5 %	93.8 %
needle, 3–52 k ctx (37 positions)	0.0840	0.1228	94.6 %	97.3 %

Needle-in-a-haystack (free generation): 5/5 facts retrieved correctly at 3.4 k, 9.6 k, 22 k, 39 k and 52 k prompt tokens (identical answers to FP8; wording differs in one case). Token-level identity to the FP8 greedy output was 79 %, against 88 % for FP8 re-run against itself under load, so the difference is within the server's own batching non-determinism.

Functional gates: 7/9 on the same functional gate suite as FP8 (the two failures are test-harness false negatives shared with FP8: usage.cached_tokens is not reported by this build, and a strict string match on a vision answer that is actually correct). Long generation (6.4 k tokens, pt-BR) coherent, tool calling and vision unchanged.

Speed: unchanged. Batch-1 prose 100 tok/s, batch-1 code 170 tok/s, 8 concurrent streams 592 tok/s aggregate (idle server), prefill 4.9 k tok/s at 29 k prompt tokens; the dequant adds three small tensor ops per Engram lookup, invisible next to the 40 ms forward.

Limitations and honesty notes

• 4-bit Lloyd-Max on a Gaussian has an inherent ~9.7 % relative RMSE per row; the tables are already FP8 in the official checkpoint, so this is a second quantization. Use the numbers above, not intuition, to decide whether it fits your use.
• Validated only through SGLang with the row-store adapter on SM120. MLX/GGUF runtimes have their own Engram quantizers (e.g. mixed 4/8-bit with 6-bit Engram); this repo does not target them.
• Video and multi-image inputs were not tested (the base recipe does not support them either).

Attribution

HLWQ (Hadamard-Lloyd Weight Quantization) is the weight-quantization method by Caio Vicentino (arXiv:2603.29078). The rotation-plus-scalar-codebook lineage goes back to PolarQuant by Han, Kacham, Karbasi, Mirrokni and Zandieh, 2025 (arXiv:2502.02617), a KV-cache method; the name HLWQ replaces the author's earlier "PolarQuant" branding to disambiguate from that work. Base model © DeepSeek, MIT license. Serving recipe by 0xSero.