Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-W4A16

Model description

This is a W4A16 quantized version of gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1, which is a fine-tuned Mistral-Small-3.2-24B-Instruct-2506 designed to improve the evaluation of SBVR (Semantics of Business Vocabulary and Business Rules) rules.

The quantization was performed using AutoRound 0.15.0 with SignRoundV2 optimization, reducing the model size from 45 GB (BF16) to 15 GB while maintaining high fidelity to the original model.

Quantization configuration

Parameter	Value
Quantization method	AutoRound (SignRoundV2)
AutoRound version	0.15.0
Weight bits	4 (W4A16)
Activation bits	16 (BF16)
Group size	128
Symmetric	True
Iterations	1000
Calibration samples	512
Sequence length	4096
Packing format	auto_round:auto_gptq
lm_head	Kept in 16-bit (not quantized)
enable_alg_ext	True (SignRoundV2)
enable_torch_compile	True
low_gpu_mem_usage	True
GPU	NVIDIA RTX 5090 (32 GB)

Calibration dataset

The calibration dataset was constructed from multiple domain-specific sources to ensure the quantized model retains accuracy across all relevant domains:

Domain	Samples	Description
SBVR rules	~50%	SBVR compliance rules in JSON format, rule status assignment (PASSED/FAILED/UNRESOLVABLE), insurance terminology
Slovak text	~25%	Slovak Wikipedia, FineWeb2, EN↔SK parallel corpus (opus100) — ensures diacritics preservation
English general	~15%	UltraChat conversations, general English text
Reasoning	~10%	MMLU-Pro (14 categories), BoolQ, TruthfulQA

All 512 calibration samples were concatenated to a minimum of 4,096 tokens each (20,480 characters) to maximize the calibration signal per sample. The dataset is published alongside this model as calibration_dataset.jsonl.

Quantization script

The quantization script is published as quantize.py in this repository.

Evaluation results

WikiText-2 Perplexity

Standard WikiText-2-raw-v1 benchmark with sliding window (stride=512, max_length=2048, 590 windows, 301,616 tokens):

Model	WikiText-2 PPL	Degradation
BF16 (original)	4.7596	—
W4A16 (this model)	4.9160	+3.28%

Comprehensive evaluation

30 domain-specific prompts (SBVR, Slovak, English, code, reasoning) comparing token-level agreement, KL divergence, and generation fidelity against the BF16 baseline:

Metric	Value
Average PPL degradation	+1.39%
KL divergence	0.0231
Cosine similarity	0.9945
Spearman correlation	0.9984
Top-1 token agreement	70.00%
Top-5 token agreement	90.69%
Top-10 token agreement	91.05%
Greedy match	90.46%

Per-category PPL

Category	BF16 PPL	W4A16 PPL	Degradation
SBVR	30.43	30.89	+1.51%
Slovak	11.09	11.16	+0.63%
English	5.89	6.16	+4.58%
Code	3.41	3.38	-0.88%
Reasoning	4.03	3.99	-0.99%

Usage

from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-W4A16",
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-W4A16",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Evaluate this SBVR rule: ..."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
output = model.generate(inputs.to(model.device), max_new_tokens=512)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Model details

Property	Value
Architecture	Mistral3ForConditionalGeneration
Parameters	1.78B (quantized) / 24B (original)
Model size	15 GB
Precision	W4A16 (4-bit weights, 16-bit activations)
Context length	128K (inherited from base model)
License	Apache 2.0

HELM comparison: fine-tuned BF16 vs W4A16

The BF16 fine-tuned source model and this W4A16 quantized version were evaluated side by side on Stanford HELM benchmarks (crfm-helm >= 0.5.0, 1,000 instances per scenario, served via an OpenAI-compatible endpoint).

General benchmarks

Suite	FT BF16	FT W4A16	Δ
bbq	90.60%	91.90%	+1.30 pp
boolq	85.10%	89.40%	+4.30 pp
commonsense	82.80%	82.90%	+0.10 pp
damaged_items_mc	100.00%	100.00%	0.00 pp
insurance_claim_extraction	64.14%	61.63%	−2.51 pp
polycontext_rules	71.50%	62.10%	−9.40 pp
quac	59.97%	58.85%	−1.12 pp
synthetic_reasoning	93.40%	91.75%	−1.65 pp
truthful_qa	68.35%	68.50%	+0.15 pp

MMLU Pro

Subject	FT BF16	FT W4A16	Δ
biology	76.43%	76.01%	−0.42 pp
business	34.98%	37.77%	+2.79 pp
chemistry	31.50%	32.20%	+0.70 pp
computer_science	53.90%	52.68%	−1.22 pp
economics	64.57%	63.74%	−0.83 pp
engineering	33.85%	37.36%	+3.51 pp
health	64.55%	62.47%	−2.08 pp
history	60.63%	57.48%	−3.15 pp
law	42.20%	42.10%	−0.10 pp
math	34.50%	34.60%	+0.10 pp
other	54.98%	53.14%	−1.84 pp
philosophy	51.30%	48.70%	−2.60 pp
physics	32.80%	34.50%	+1.70 pp
psychology	68.55%	67.54%	−1.01 pp

Summary averages

Metric	FT BF16	FT W4A16	Δ
HELM overall avg (23 suites)	61.77%	61.19%	−0.58 pp
MMLU Pro avg (14 subjects)	50.34%	50.02%	−0.32 pp
Non-MMLU avg (9 scenarios)	79.54%	78.56%	−0.98 pp
SBVR unresolvable overall (1,432 items)	90.92%	91.97%	+1.05 pp

BBQ bias

Metric	FT BF16	FT W4A16
Ambiguous accuracy	88.67%	88.67%
Non-ambiguous accuracy	95.29%	95.29%
Bias gap (	amb − non-amb	)

Original model

This model is a quantized version of gratex/Mistral-Small-3.2-24B-Instruct-2506-sbvr-compliance-unresolvable-pattern-finetune-v1. Please refer to the original model card for detailed information about the fine-tuning methodology, training configuration, and HELM evaluation results.