sh111111111111111/Qwen3-4B-Instruct-2507-BitClass-MX1-GGUF

Qwen3-4B-Instruct-2507 — BitClass MX-1 (Mixed-Precision, 3.54 bpw)

A GGUF-quantized version of Qwen3-4B-Instruct-2507 using BitClass, our learned mixed-precision quantization.

This is the MX-1 (compact) variant at 3.54 bits per weight, optimized for size and throughput. For a higher-quality variant, see MX-2 (4.00 bpw).

Model

File	Bits/Weight	Size	Perplexity ↓	Throughput (GPU)	Throughput (CPU)
Qwen3-4B-Instruct-2507-Q3_K_S-3.54bpw.gguf	3.54	1.78 GB	3.337	93.3 tok/s	11.4 tok/s

Benchmark Results

All models evaluated using lm-evaluation-harness v0.4.11 (0-shot) on identical hardware. Higher is better for all metrics.

Model	BPW	Size	ARC-C ↑	GSM8K ↑	IFEval ↑	TruthfulQA ↑	Avg
Unsloth Q5_K_M	5.75	2.69 GB	58.79	72.55	57.49	62.49	62.83
Unsloth Q4_K_M	4.97	2.33 GB	57.76	66.87	55.27	60.75	60.16
Ours MX-2	4.00	2.01 GB	57.42	51.40	53.60	60.21	55.66
Ours MX-1	3.54	1.78 GB	53.75	53.60	50.46	60.14	54.49
ByteShape KQ 3.34	3.34	1.69 GB	55.72	45.56	51.20	58.41	52.72
Unsloth Q3_K_S	3.75	1.76 GB	55.89	41.24	52.13	60.10	52.34

ARC-C: acc_norm, GSM8K: exact_match (flexible-extract), IFEval: prompt_level_strict_acc, TruthfulQA: acc (mc2). All values ×100.

MX-1 at a glance:

• +2.15 average over Unsloth Q3_K_S at comparable size (1.78 vs 1.76 GB)
• GSM8K 53.60 — massively beats both Q3_K_S (41.24, +12.36) and ByteShape (45.56, +8.04)
• TruthfulQA 60.14 — on par with Q4_K_M (60.75) at smaller size (1.78 vs 2.33 GB)
• 93.3 tok/s GPU throughput — fastest in our tests

Perplexity Comparison

Model	BPW	Size	PPL ↓	Source
Unsloth Q5_K_M	5.75	2.69 GB	2.907	unsloth
Unsloth Q3_K_S	3.75	1.76 GB	3.007	unsloth
Unsloth Q4_K_M	4.97	2.33 GB	2.956	unsloth
ByteShape KQ 3.34	3.34	1.69 GB	3.175	byteshape
ByteShape KQ 3.19	3.19	1.61 GB	3.192	byteshape
Ours MX-1	3.54	1.78 GB	3.337	This repo
ByteShape IQ 3.07	3.07	1.55 GB	3.423	byteshape

Comparable perplexity to ByteShape's IQ3_S 3.07bpw, but with 2.1x higher CPU throughput (11.4 vs 5.4 tok/s) and dramatically better task scores — especially GSM8K where MX-1 scores 53.60 versus Q3_K_S's 41.24.

Quantization Labels

The filename label Q3_K_S indicates the base quantization type. The actual model uses a mix of quantization types across tensor groups, with an average effective bits per weight of 3.54.

Running with Ollama

ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-BitClass-MX1-GGUF:Qwen3-4B-Instruct-2507-Q3_K_S-3.54bpw.gguf

Running with llama.cpp

# Chat
llama-cli -m Qwen3-4B-Instruct-2507-Q3_K_S-3.54bpw.gguf -cnv

# Server (OpenAI-compatible API)
llama-server -m Qwen3-4B-Instruct-2507-Q3_K_S-3.54bpw.gguf --port 8080

Evaluation Details

• Perplexity & Throughput: llama.cpp b8514, measured on both NVIDIA GB10 GPU (-ngl 999) and CPU
• Task benchmarks: lm-evaluation-harness v0.4.11, 0-shot, via llama-cpp-python with logits_all=True
• All models benchmarked in the same session on identical hardware for fair comparison

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.

License

Apache 2.0, inherited from Qwen3-4B-Instruct-2507.

Acknowledgments

• Base model by Qwen Team
• Importance matrix data from Unsloth
• Quantization infrastructure built on llama.cpp