vellum-ai/Kimi-K3-W3A16-g64

Kimi-K3 W3A16 (3-bit GPTQ, group 64, compressed-tensors)

A 3-bit weight-only quantization of moonshotai/Kimi-K3 in the standard compressed-tensors pack-quantized format. 1.29 TB total (vs 1.56 TB for the original MXFP4 release; a BF16 render of this model would be ~5.6 TB), sized so the full model fits resident on a single 8x B200 / 8x H200-class node.

Status: format-complete, kernel support in progress. This artifact is a valid compressed-tensors checkpoint and every expert tensor round-trips bit-exactly through the reference dequantization. However, no currently released inference stack (vLLM, SGLang) ships a fused 3-bit MoE kernel: their WNA16 MoE paths execute 4-bit and 8-bit only. Until 3-bit kernels land, options are (a) a lossless W3-to-W4 recode (every 3-bit code is representable in the 4-bit kernel's range; +340 GB, then it serves on stock vLLM on 16x H200-class hardware), or (b) custom dequant-per-forward execution as used for the quality evaluation below. An evaluation table (perplexity + a held-out probe battery comparing this quant against the original checkpoint's activations) is being finalized and will be added here.

Quantization recipe

• Targets: the routed-expert Linears only (language_model.model.layers.{1..92}.block_sparse_moe.experts.{0..895}.{w1,w2,w3}, 247,296 Linears). Everything else (attention including KDA/MLA, shared experts, router, latent projections, norms, embeddings, lm_head, vision tower) ships unchanged in BF16.
• Scheme: GPTQ, weight-only int3, symmetric, group size 64, static activation ordering (llmcompressor 0.12.0.1 / compressed-tensors 0.17.1).
• Calibration: 512 sequences x 2048 tokens, public data only (ultrachat_200k chat + fineweb-edu prose), tokenized with the model's own tokenizer, batch-equal maskless packing.
• Pipeline: layer-sequential GPTQ with error propagation, run directly from the packed MXFP4 source (no BF16 intermediate ever materialized), quantized layer-by-layer on 8x B200.

Files

Same layout as the source repository: safetensors shards + index, the model's remote-code files, tokenizer, and the original Moonshot LICENSE (this is a derivative work of Kimi-K3, redistributed under its terms with attribution; see LICENSE for the scale-triggered obligations).

Acknowledgements

Weights by Moonshot AI. Quantization by Vellum using llmcompressor and compressed-tensors.