schoggie/K2-Horizon-MoVA-36B-A4B-W4A16

K2-Horizon-MoVA-36B-A4B — W4A16 (compressed-tensors)

INT4 group-128 symmetric weights on the routed MoE experts only, produced with compressed-tensors (pack-quantized). Everything else — most importantly the entire MoVA mechanism — stays BF16.

HumanEval pass@1: 92.1% (151/164, greedy) on a single NVIDIA CMP 170HX.

Why this exists: do not quantize MoVA

K2-Horizon's attention replaces v_proj with MoVA — a router (v_router) over 64 value experts per layer. A widely-used community GPTQ Int4 of this model quantizes those value experts and the router to 4 bits. On our hardware that produced systematically corrupted output: fluent English and correct reasoning, but broken code —

n = len(numbers))
for i in range(n)):

Token-level inspection showed the model emitting '))))\n\n' as a single token where '))' was correct, i.e. picking the wrong member of the ) / )) / ))) / )))) family. That is the signature of slightly-off logits, not a text bug. Measured HumanEval 15.2%, with 106/164 samples containing the doubled-paren pattern. It reproduced identically with CUDA graphs off (15.9%) and with an unquantized KV cache (16.5%), so it was neither.

The cause is that an INT4 router selects slightly wrong value experts, so the attention value vectors are subtly wrong at every layer.

IFM's own FP8 release says the same thing in its config. Its ignored_layers (3408 entries) leave in full precision: all 64 self_attn.v_experts.* per layer, v_router, q/k/o/gate_proj, mlp.gate, mlp.shared_experts.*, the dense mlp_only_layers, every norm, lm_head, embed_tokens, model.norm. IFM quantizes only mlp.experts.*.

This checkpoint follows that contract exactly: 13,500 tensors quantized (45 MoE layers × 100 experts × {gate,up,down}_proj), 3,498 copied in BF16. It is therefore larger than the GPTQ (35.5 GB vs 21 GB) — that is the point.

	GPTQ Int4 (MoVA quantized)	this W4A16 (MoVA BF16)
HumanEval pass@1	15.2%	92.1%
doubled-paren samples	106/164	none observed
size	21 GB	35.5 GB

Serving

compressed-tensors W4A16. Note the ecosystem situation at time of writing:

• SGLang (xllm.py, the native K2 path) refuses any quantized K2 — _XllmMoVAAttentionBase.__init__ raises "K2 Horizon MoVA supports unquantized bf16/fp16 weights only" whenever quant_config is not None. It serves K2 in BF16 only, which needs ~70 GB (TP≥2).
• vLLM needs the out-of-tree plugin k2-horizon-vllm for the architecture. Upstream, that plugin requires MoVA to be 4-bit — it reads self.v_experts_fused.qweight — so this checkpoint fails to load with AttributeError: 'MergedColumnParallelLinear' object has no attribute 'qweight'.

A small patch adds a BF16 MoVA path (K2_MOVA_BF16=1): build v_experts_fused with quant_config=None, then compute the value projection densely and gather the top-k experts. It is mathematically identical to the sparse path (silu is elementwise per expert). With E·kv_dim = 65,536 and hidden = 2560 that is ~0.13 GB of activations per 1000 tokens.

podman run -d --device nvidia.com/gpu=0 \
  -v /path/to/models:/models -p 8000:8080 \
  -e K2_MOVA_BF16=1 \
  <vllm-image-with-patched-plugin> \
  --model /models/k2-horizon-w4a16 --trust-remote-code \
  --max-model-len 131072 --max-num-seqs 32 \
  --gpu-memory-utilization 0.93 \
  --attention-backend FLASHINFER --kv-cache-dtype fp8 --enable-prefix-caching

Measured on one CMP 170HX (GA100, sm_80, 64 GB)

Weights 33.6 GiB resident. With --kv-cache-dtype fp8: 266,016-token KV pool, 131,072 per request (max_position_embeddings is 524,288).

np	1	2	4	8	16	32
aggregate tok/s	42.5	73.1	137.0	255.0	498.0	874.7

Single-stream is ~42% below the (corrupt) sparse-Marlin build because the dense fallback evaluates all 64 value experts rather than the top-4. A sparse BF16 gather path would recover most of that, but is data-dependent and therefore not CUDA-graph safe.

KV is expensive on this model: full attention on all 48 layers, 8 KV heads × 128 head_dim → ~192 KB/token BF16, ~96 KB/token FP8. Roughly 6× a hybrid-attention model of similar size.

Credits

Base model and the MoVA architecture: IFM/K2-Horizon-MoVA-36B-A4B by MBZUAI's Institute of Foundation Models. This repository is only a quantization.