---
license: apache-2.0
library_name: transformers
tags:
  - quantization
  - fp8
  - autoround
  - moe
  - sarvam
  - resilient-ai-challenge
base_model: sarvamai/sarvam-30b
---

# Sarvam-30B W8A8-Dynamic (AutoRound FP8)

## Model Description

This is an FP8 (W8A8) quantized version of Sarvam-30B, a Mixture-of-Experts (MoE) model with 128 experts (6 active per token) plus 1 shared expert. The model was quantized using **AutoRound** with dynamic activation quantization.

| Property | Value |
|----------|-------|
| Base Model | Sarvam-30B |
| Architecture | SarvamMoEForCausalLM |
| Parameters (total) | ~30B |
| Layers | 19 |
| Hidden Size | 4096 |
| Attention Heads | 64 (4 KV heads, GQA) |
| Experts | 128 routed + 1 shared |
| Active Experts/Token | 6 |
| Max Context Length | 131,072 tokens |
| Quantized Size | ~37 GB |

## Compression Technique

### Method: AutoRound (FP8 Dynamic Quantization)

**AutoRound** is a weight-rounding optimization technique that minimizes quantization error via learned rounding decisions. It iteratively optimizes rounding choices to preserve model accuracy.

#### Quantization Configuration

| Component | Precision | Strategy | Details |
|-----------|-----------|----------|---------|
| Weights | FP8 (8-bit float) | Per-channel, symmetric | Static (memoryless minmax observer) |
| Input Activations | FP8 (8-bit float) | Per-token, symmetric | Dynamic quantization |
| Output Activations | Not quantized | — | — |

#### AutoRound Hyperparameters

| Parameter | Value |
|-----------|-------|
| Iterations | 200 |
| Batch Size | 4 |
| Scheme | FP8_DYNAMIC |
| Torch Compile | Enabled |

#### Layers/Modules Kept at Full Precision

The following modules are **not quantized** (kept in original precision) to preserve model quality:

- `lm_head` (output projection)
- All self-attention layers (`query_key_value`, `dense`)
- All shared expert layers (`shared_experts.gate_proj`, `up_proj`, `down_proj`)

This selective quantization strategy preserves the most sensitive components (attention and shared experts) while compressing the routed expert MLP weights and activations to FP8.

## Inference

### vLLM (Recommended)

```bash
vllm serve --config vllm_config.yaml
```

A `vllm_config.yaml` is included in the model root with the following settings:

```yaml
model: .
trust_remote_code: true
tensor_parallel_size: 1
gpu_memory_utilization: 0.85
max_model_len: 65536
dtype: auto
```

## License

Apache 2.0 — same as the original [Sarvam-30B](https://huggingface.co/sarvamai/sarvam-30b) model.