| --- |
| license: other |
| base_model: sarvamai/sarvam-30b |
| tags: |
| - sarvam |
| - moe |
| - quantization |
| - w4a16 |
| - gptq |
| - compressed-tensors |
| - energy-efficient |
| - vllm |
| language: |
| - en |
| - hi |
| - ta |
| - bn |
| - te |
| - kn |
| - ml |
| - mr |
| - gu |
| - pa |
| pipeline_tag: text-generation |
| --- |
| |
| # Sarvam-30B · Godspeed (W4A16, energy-optimized) |
|
|
| A compressed, **energy-efficient** build of [`sarvamai/sarvam-30b`](https://huggingface.co/sarvamai/sarvam-30b), |
| prepared for the **Resilient AI Challenge — Track 1 (text-to-text)**. The objective of that track is the |
| **lowest inference energy** (joules over the full benchmark suite) on a single A100-80GB served via stock |
| `vllm serve`, while retaining **≥80 % quality recovery in every evaluation category**. |
|
|
| This model preserves the base model's reasoning quality while cutting per-token memory traffic — the dominant |
| driver of decode-time energy in a batch-1, memory-bandwidth-bound regime — through careful, serving-safe 4-bit |
| weight quantization. |
|
|
| --- |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | **Developed from** | `sarvamai/sarvam-30b` (SarvamMoEForCausalLM) | |
| | **Architecture** | Mixture-of-Experts decoder — 19 layers (1 dense + 18 MoE), 128 routed + 1 shared expert, top-6 routing, 4 KV heads, hybrid "thinking" chat template | |
| | **Precision** | **W4A16** (4-bit weights, 16-bit activations), compressed-tensors / Marlin-ready | |
| | **Serving** | `vllm serve . --config vllm_config.yaml` (stock vLLM, `trust_remote_code`) | |
| | **Languages** | English + major Indic languages (Hindi, Tamil, Bengali, Telugu, Kannada, Malayalam, Marathi, Gujarati, Punjabi, …) | |
|
|
| --- |
|
|
| ## Quantization recipe |
|
|
| The recipe is deliberately **quality-first and serving-safe**: |
|
|
| - **GPTQ (calibrated, 4-bit)** on the **attention** projections (`query_key_value`, `dense`) and the |
| **shared-expert** MLP — the quality-sensitive, always-active paths — calibrated on a multilingual + math/STEM |
| mix derived from `sarvamai/indivibe`. |
| - **RTN (round-to-nearest, 4-bit)** on the **128 routed experts** — data-free and fast, suitable for the |
| sparsely-activated expert bank. |
| - **Kept in bf16** for stability and serving safety: the **LM head** (vLLM rejects a quantized `lm_head`), |
| the **router gates** (routing must stay precise), and **layer 0** (input-sensitive). |
|
|
| The original **thinking** chat template is retained unchanged, so the model behaves exactly like the base model |
| under the evaluator's chat interface. |
|
|
| --- |
|
|
| ## Why this is energy-efficient |
|
|
| At batch-1 decode the GPU is **memory-bandwidth-bound**: energy per generated token is governed by the number of |
| **bytes read from HBM per forward pass**. Moving the attention, shared-expert and routed-expert weights from |
| 16-bit to **4-bit** reduces those bytes substantially, lowering joules-per-token **without inflating the token |
| count** — quality is preserved, so the model does not "think longer" to compensate (a failure mode that erases |
| byte savings). The model passes **every category quality floor with margin**, which is the prerequisite for being |
| ranked on energy at all. |
|
|
| --- |
|
|
| ## Intended use & limitations |
|
|
| - **Intended use:** efficient text generation / reasoning for the challenge benchmark and general multilingual |
| assistant tasks. |
| - **Limitations:** inherits the base model's knowledge cutoff, biases and capabilities. W4A16 is the practical |
| floor for stock-vLLM serving — sub-4-bit or `lm_head` quantization were evaluated and **rejected** because they |
| either fail to serve on stock vLLM or degrade quality below the floors. |
|
|
| --- |
|
|
| ## How to serve |
|
|
| ```bash |
| vllm serve . --config vllm_config.yaml |
| ``` |
|
|
| The included `vllm_config.yaml` is stripped of infrastructure-specific parameters (no `tensor-parallel-size`, |
| `swap_space`, etc.), per the challenge configuration-hygiene requirement, and sets `max_model_len: 32768`, |
| `trust_remote_code: true`. |
|
|
| --- |
|
|
| *Prepared for the Resilient AI Challenge. Base model © Sarvam AI; please refer to the base model card for its |
| license and usage terms.* |
|
|
