sahilchachra/LightOnOCR-3-4B-MXFP8

LightOnOCR-3-4B — MLX MXFP8

MLX MXFP8 quantization of lightonai/LightOnOCR-3-4B, LightOn's OCR / document-understanding vision-language model. It's a standard qwen3_5 VL model (hybrid GatedDeltaNet linear-attention + full-attention text backbone, 32 layers, full attention every 4th, hidden 2560; Qwen3-VL-style vision tower, depth 24, hidden 1024) — the same architecture family already supported natively by mlx-vlm 0.6.12, so no custom patch was needed for this one (unlike some other qwen3_5-labeled checkpoints with custom attention). Runs on Apple Silicon via mlx-vlm. Stays image-text-to-text — the vision tower is kept in bf16; only the text backbone is quantized.

Precision	MXFP8 (E4M3 + E8M0 shared scale, group size 32)
Bits per weight	8.821 bpw
On-disk size	4.7 GB (1 shard)
Quantized	text backbone (32 layers, hidden 2560; embeddings tied, no separate lm_head)
Kept in bf16	vision tower (Qwen3-VL-style ViT, depth 24, hidden 1024)

Quantizations

Variant	Bits	Size
LightOnOCR-3-4B-MXFP4	5.116 bpw	2.7 GB	smaller / fastest
LightOnOCR-3-4B-MXFP8	8.821 bpw	4.7 GB	← this repo

Verification

Tested with deterministic greedy decoding via native mlx-vlm, comparing against the unquantized bf16 reference:

• Text coherence: both quantized builds answer correctly — "What is the capital of France?" → "Paris" (MXFP4) / "The capital of France is Paris." (MXFP8); "What is 25 + 17?" → "42" on both.
• Vision smoke test (solid-color and shape images, bf16 vision tower through the quantized text backbone): both builds correctly identify color ("red", "blue") and shape+color ("green circle").
• Real OCR task (synthetic invoice image: "Invoice #A-9921", "Total Due: $482.50", "Due Date: 2026-11-01", "Bill To: Sahil Chachra") — bf16, MXFP4, and MXFP8 all produce byte-identical, fully correct output, reading all four lines with no hallucination or omission.
• LM Studio (tested via lms load + the OpenAI-compatible chat completions API on localhost:1234): both builds load cleanly (qwen3_5 architecture recognized, format: mlx present) and reproduce the same correct OCR output through the API as the native mlx-vlm test.

Usage (mlx-vlm)

pip install -U mlx-vlm   # needs the qwen3_5 architecture (>= 0.6.12)

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor = load("sahilchachra/LightOnOCR-3-4B-MXFP8")
config = model.config

prompt = apply_chat_template(processor, config, "Read all the text in this image.", num_images=1)
print(generate(model, processor, prompt, image=["invoice.png"], max_tokens=300, verbose=True))

Run in LM Studio

Loads and runs out of the box — the qwen3_5 architecture is recognized natively by LM Studio's bundled MLX runtime (tested on 0.4.20). Works through both the UI and the OpenAI-compatible /v1/chat/completions API with image inputs (image_url with a base64 data URL). No patch file or extra setup required.

Notes & limitations

• tie_word_embeddings=true — no separate lm_head tensor; embeddings are quantized once and reused for the output projection.
• Inherits all capabilities and limitations of the base model, including its OCR specialization — it is tuned for document/table/form understanding rather than general chit-chat. See the original model card.
• Quantized by @sahilchachra with MLX. Apache-2.0.