littlebit-qwen3-4b-mlx-4bit

Qwen3-4B in 4-bit MLX format for Apple Silicon. This is the first LittleBit release: a small, fast model for running locally on a Mac.

What this is: standard MLX 4-bit post-training quantization (group size 64, about 4.5 bits per weight). It is not the sub-1-bit LittleBit-2 method. The sub-1-bit (0.55 bpw) Qwen3 builds are in progress; the training recipe is in LittleBitLLM/littlebit.

Measured on an M1 Max (64 GB)

	Value
Size on disk	2.1 GB (BF16 original: ~8 GB)
Bits per weight	4.5
Generation speed	~74 tokens/s
Peak memory	2.4 GB

These numbers come from one short generation run. We have not run benchmark evals on this build. Expect quality close to the BF16 Qwen3-4B, as is usual for 4-bit group quantization.

Usage

pip install mlx-lm
mlx_lm.generate --model LittleBitLLM/littlebit-qwen3-4b-mlx-4bit --prompt "Explain quantization in two sentences."

from mlx_lm import load, generate

model, tokenizer = load("LittleBitLLM/littlebit-qwen3-4b-mlx-4bit")
messages = [{"role": "user", "content": "Explain quantization in two sentences."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

Qwen3 thinks before it answers by default. Add /no_think to a message for a direct answer.

Build

mlx_lm.convert --hf-path Qwen/Qwen3-4B -q --q-bits 4 --q-group-size 64

License

Apache 2.0, inherited from Qwen3-4B by the Qwen team.

surplus-launcher: 0x9370BE7C8Dd5Ba50920a66D977b9c39b839706A2