pavanperi/sarvam-30b-awq-w4a16-broad-5120

Sarvam 30B compression by Atos

Usage :

In your fresh python environment, use :

pip install vllm==0.15.0
python3 hotpatch_vllm.py
python3 patch_compressed.py --moe-kernel marlin

Then serve the model with :

vllm serve --config vllm_config.yaml

We are not using A100 GPU as we heard that Marlin could have problems with the A100 architecture. Contact us if that is the case.

AWQ

The key idea is to use real activations from a calibration dataset to quantize the targeted weight tensors in a smart way.

We tried many other compression techniques :

• GPTQ, SparseGPT: Results were less capable
• Unstructured pruning: Low energy savings
• Structured pruning: Too much loss of capability
• AWQ but with fewer layers compressed: Ended up creating mismatches in and within layers that caused total hallucinations in output

In practice, AWQ does the following.

• Runs representative inputs through the model
• Observes the activations entering important linear layers
• Identifies channels that are more sensitive to quantization error
• Rescales or normalizes the computation so these sensitive channels are better preserved
• Quantizes the targeted weights

W4A16 means that weights are quantized to 4 bits, while activations remain at 16-bit precision. That matters because it gives a good tradeoff for LLM inference:

• the model weights become much smaller
• memory bandwidth pressure drops significantly
• activations stay in a safer precision regime
• quality is often better preserved than with more aggressive full low-bit activation quantization

On the model card on HuggingFace, it shows 6B parameters, but it is not the case, no parameter was cut-off, they were just quantized to int4 at most

Why AWQ Makes Sense For An MoE Model

MoE models are not just bigger dense transformers. They have token-dependent specialist paths.

That creates two important constraints.

• some expert channels are used in highly specialized ways
• routing mistakes or distorted expert outputs can hurt quality disproportionately

AWQ is useful here because it uses observed activations to preserve the most sensitive directions better than naive low-bit rounding.

After different trial-and-error processes, we concluded that we should limit ourselves to quantizing Linear layers, which means AWQ is allowed to quantize linear modules across the model rather than only the routed experts.

It includes

• attention projections such as attention.query_key_value and attention.dense
• dense MLP projections where they exist
• routed expert MLP projections
• shared expert MLP projections

But we did not compress

• lm_head because it directly controls the final token logits
• the MoE router gate weights because the router is a very important part in a MoE model
• non-linear modules such as layer norms
• any weights that are not exposed through the targeted linear-module path

Calibration

The calibration dataset is not training data, and it is not fine-tuning. Its purpose is to show AWQ realistic activation patterns so weight quantization can be adapted to the way the model is actually used.

For Sarvam, calibration quality matters a lot because:

• the model is multilingual
• the model is MoE, so different experts may specialize differently
• broad quantization touches several important subsystems

If the calibration set is too narrow, the quantizer protects the wrong activation ranges and the model can become less coherent, less stable across languages, or more brittle on certain tasks.

The 5120-Sample Strategy

The original master design used a 512-sample benchmark-aware multilingual calibration set, but we expanded it to 5,120 samples due to significant performance degradation. While 512 samples is generally considered the optimal size for a standard dense model, Mixture-of-Experts (MoE) architectures require a much larger calibration set to properly align all experts. Although an even larger dataset would likely be preferable, memory constraints on our hardware limited us to 5,120 samples

Task Category	Language / Source Split	Count	Direct Repository Link
Math	English (en): 430 samples French (fr): 430 samples Indic (10 scripts): 420 samples	1280	juletxara/mgsm
Medical	English (en): 430 samples French (fr): 430 samples Indic scripts: 420 samples	1280	Medical-Corpus
Writing	English (en): 430 samples French (fr): 430 samples Indic scripts: 420 samples	1280	wikimedia/wikipedia
Questions	English (en): 420 samples French (fr): 420 samples Indic scripts: 440 samples	1280	aya dataset
TOTAL	25% English / 25% French / 50% Indic	5120

Results and conclusion

On our hardware, the model achieved over 50% energy savings and reduced memory usage (under 20GB vs. 120GB VRAM), while retaining more than 81% of the base model’s capabilities based on our internal benchmarks. Energy savings are highly dependent on the underlying kernel, which is more or less optimized for W4A16 compression and not compatible with all hardware types.