DeepSeek-V4.1-Flash-EXL3-3bpw

Calibrated EXL3 3bpw routed-expert quantization of DeepSeek-V4.1-Flash. The original BF16 vision transformer and aligner are preserved unchanged.

Upload status: this repository is being populated. Treat the download as incomplete until upload-status.json reports complete and the downloaded files match release-manifest.json.

Runtime compatibility: this is an experimental weights release for a specialized vLLM/EXL3 integration, not a drop-in stock-vLLM or standalone ExLlamaV3 checkpoint. The reusable public serving bundle is still being finalized and is not included here yet. Do not assume that a stock vllm serve command can load these weights.

What is quantized?

• All 40 backbone layers and 15,360 routed experts use calibrated 3-bit EXL3 MUL1 weights.
• The 3bpw label applies to routed experts, not every parameter or the average storage cost of the entire checkpoint. EXL3 metadata and padding add storage beyond the packed 3-bit payload.
• Vision transformer and aligner retain their original BF16 bytes. Routers, shared experts and other non-routed tensors are copied unchanged from the published source, which is itself a mixed-precision/native-FP4 checkpoint.
• Original FP8 engram tables/scales are retained under engrams/, for SSD-backed lookup rather than loading those tables onto the GPUs.
• Original draft/MTP weights are retained under draft/. They are not loaded in the tested recipe; speculative decoding is disabled.

Measured quality

The full 40-layer candidate was compared with the original native-FP4 model on 64 held-out records. These are teacher-forced measurements, not a BF16 comparison or a claim about broad downstream task accuracy.

Text metric	Original source	EXL3 3bpw
Prediction targets	81,880	81,880
Mean negative log likelihood	1.33438994	1.34028314
Perplexity	3.79767842	3.82012498
Correct next-token prediction	70.71690%	70.60821%

Perplexity increases 0.59106%. Top-choice agreement with the original is 93.24255%, which is agreement, not task accuracy. The image-answer slice contains only 84 prediction targets: it is too small for a broad vision claim. See evaluation-summary.json for aggregate measurements and their scope.

Small candidate-only generated-answer checks produced 123/128 on ARC-Challenge and 63/64 on GSM8K, with thinking disabled. These are sampled checks, not full benchmarks or original-model generation comparisons; calibration overlap and pretraining contamination were not assessed. The original 24-image question suite scored 18/24. Its six misses remain failures; separately rephrased diagnostics do not replace them.

Serving measurements and limitations

The previous working custom runtime served text and images across two DGX Sparks with TP=2, DCP=2, eager execution and one active request. It used native FP8 KV, original BF16 vision and SSD engrams. FP4 KV is not implemented.

• Short-context decode: approximately 7.18 tokens/s, with 0.379 s server time to first token in the measured 24-token-prompt/128-token-output test.
• A synthetic 1,047,990-token retrieval request recovered all three planted facts in 4,277.419 s (71.29 minutes).
• An identical repeat reused 1,047,808 cached tokens and completed in 10.070 s; server time to first token was 2.955 s.
• The two short replies at roughly 1M context decoded at approximately 2.406 tokens/s. These are not sustained-throughput benchmarks.

The tested profile used GPU memory utilization 0.90 and a downward-only KV cap of 1,009,612,800 bytes per GPU. Memory margins are tight and host-specific. Six concurrent requests or six independent 1M histories are not qualified. Cold 1M prefill remains slow; cached-prefix reuse is substantially faster.

Repeated uncached requests for one image question still varied in the previous working runtime. Tracing found the first recorded divergence at DCP attention selected-key ordering; the saved last rows had identical key membership but different order. A deterministic-order candidate has passed bounded native-kernel tests, but its full-model repair is not yet verified. This is a serving-repeatability caveat, not a measurement of quantization loss. Broader comparative generation, coding and vision evaluation remains open.

Runtime source revisions used in the experiments:

• vLLM: e47aa780bccf59f59dfa2cbb18e17a10b4fe69ba
• ExLlamaV3: 6ff3a17ea7f3d0026b273d43239398d57f71b788
• Base checkpoint: df42c109f1defefcbfcedbe7d905718a12266e40

Those revisions alone are insufficient: custom DCP/cache-ownership hooks, SSD engram integration, streaming loading and bounded GB10 kernels are also required. No clean external rebuild or stock-runtime compatibility is claimed.

Files and integrity

Component	Shards	Tensor payload bytes
Active model	51	215,344,840,128
SSD engram tables/scales	2	202,758,032,400
Retained draft/MTP	2	7,932,874,632

Total shard storage including safetensors headers is 426,055,648,464 bytes. All 55 shards are byte-identical to the independently verified canonical candidate. Public metadata is a separate revision; no re-quantization occurred. Tokenizer, model metadata, source license and historical packaging provenance are included. Calibration examples, activation captures, operational logs, credentials and runtime container images are not included.

release-manifest.json lists the expected size and SHA256 of every payload. verify-release.py verifies a materialized download using standard Python; run it before loading a GPU model. Obtain the manifest SHA256 from the completed upload receipt or another trusted channel, then run:

hf download coolbho3k/DeepSeek-V4.1-Flash-EXL3-3bpw --local-dir ./DeepSeek-V4.1-Flash-EXL3-3bpw
python3 ./DeepSeek-V4.1-Flash-EXL3-3bpw/verify-release.py --directory ./DeepSeek-V4.1-Flash-EXL3-3bpw --manifest-sha256 MANIFEST_SHA256

Integrity verification is not a quality benchmark or proof of publisher authenticity. Historical records under provenance/ describe packaging and the evaluation candidate, not a claim that every runtime issue is resolved.

Selected quantization manifest logical SHA256: 4639e1cf64c5845bb89ae4f9b3648293014bb8c0c32daee2d19f7df0a3f5e86e. Original evaluation-candidate package manifest SHA256: 7f7cfee4a7dc618196b0699a57034dce3316b83257a7320687d079d72d13ee73.

The upstream MIT license is retained in LICENSE.