bquast/SmolLM2-135B-int8-absmax

SmolLM2-135M — INT8 absmax (per-tensor, symmetric)

A research artifact, not a production model. This is the simplest possible weight-only post-training quantization, applied to SmolLM2-135M, to establish a baseline.

What this is

• Symmetric per-tensor absmax scaling
• 8-bit signed integer weights (torch.int8)
• FP32 scale factors, one per quantized tensor
• No calibration data, no activation quantization
• 1D parameters (biases, layernorm scales, etc.) left in bf16
• Embedding and LM head matrices are quantized along with all other 2D weight tensors

The math, in full:

scale = max(|W|) / 127
W_q   = round(W / scale).clip(-127, 127).to(int8)
W_hat = W_q.to(bf16) * scale

What this is not

• Not loadable with AutoModelForCausalLM.from_pretrained directly. Use the included loader.py.
• Not int8 inference. Weights are dequantized to bf16 at load time. For real int8 compute, reach for bitsandbytes, AutoGPTQ, or AutoAWQ.
• Not better than other quantization methods. Per-tensor absmax is the textbook baseline that GPTQ / AWQ / SmoothQuant exist to improve on. Expect the worst error on down_proj and lm_head — fat-tailed weight distributions, one outlier blows up the scale and crushes everyone else's resolution.

Usage

from loader import load_dequantized

model, tokenizer = load_dequantized("your-username/SmolLM2-135M-int8-absmax")

inputs = tokenizer("The capital of France is", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=10, do_sample=False)
print(tokenizer.decode(out[0]))

Why it exists

To see what naive absmax quantization actually does to a small language model, before reaching for more sophisticated methods. Companion to the experiment notebook at https://github.com/your-username/... (link your repo if you publish one).