MERaLiON-3-10B-MLX-4bit

4-bit MLX quantization of MERaLiON/MERaLiON-3-10B, the instruction-tuned Singaporean multilingual speech-language model from A*STAR / I²R — in a loadable single-tree layout with a working MLX inference harness.

2026-07-07 republish: this repo previously shipped a split component layout (encoder.safetensors / adaptor.safetensors / decoder-*.safetensors) that no MLX runner could load directly. It now ships the standard model.safetensors.index.json tree produced by pipelines/mlx_direct_quantize, verified with audio transcription through the MLX harness below.

Quality caveat (4-bit): output stays coherent but is measurably less faithful than the 8-bit variant: on the code-switched Singlish sample it paraphrases and translates the Mandarin tail instead of transcribing it verbatim (There's like two quarters, ma. Then he will teach. vs 8-bit's there's like two quarters ma then 他会教). For faithful code-switched ASR prefer the 8-bit pack; this variant suits lower-memory speech understanding (summaries, Q&A over audio).

At a glance

Source	MERaLiON/MERaLiON-3-10B @ 3d5c2f772641b1cfeba35743df3db32a07db8c48 (bf16, ~19 GB)
Format	MLX (Apple Silicon native), single safetensors tree
Quantization	4-bit affine, group-size 64 (decoder 2-D tensors only)
Disk size	6.1 GB (6,601,088,576 bytes)
Decoder	Gemma-2-9B-it, 295 tensors quantized (169 decoder 1-D preserved bf16)
Encoder	Whisper-large-v3 derived, 487 tensors preserved bf16
Adaptor + bridges	speech MLP adaptor, ln_speech, speech_encoder_layer_weighted_sum — 7 tensors preserved bf16

Unlike the 3B-ASR sibling, MERaLiON-3-10B mixes its speech encoder's per-layer outputs with learned softmax weights (speech_config.use_weighted_layer_sum=true); the 1-D weight vector rides through unquantized with the rest of the speech tower.

Verification (2026-07-07, M-series Mac, 128 GB)

• Composite-load self-test: PASS (decoder quantized, speech tower bf16, finite logits, decode bitwise-matches the mlx_lm gemma2 forward).
• Audio transcription on the upstream MERaLiON sample clips runs end to end and stays coherent, with the fidelity caveat above.

Usage

Runs on the MERaLiON-3 MLX harness in majek/pipelines/meralion3_mlx (Whisper encoder + weighted layer sum + adaptor in MLX, Gemma-2 decoder via mlx_lm.models.gemma2):

# one-shot transcription
python -m pipelines.meralion3_mlx.generate \
    --path majentik/MERaLiON-3-10B-MLX-4bit \
    --wav your_clip.wav

# local HTTP endpoint (POST wav bytes or multipart "file")
python -m pipelines.meralion3_mlx_serve \
    --model majentik/MERaLiON-3-10B-MLX-4bit --port 8321

Stock mlx-lm alone cannot load this model (composite audio architecture, model_type=meralion3); the harness routes the speech tower and decoder from the same checkpoint tree. The prompt template is the upstream one: Instruction: <TextHere> \nFollow the text instruction based on the following audio: <SpeechHere> inside Gemma-2 chat turns.

Quantization recipe

Only the Gemma-2-9B decoder's 2-D tensors are quantized (where >95 % of parameters live). The Whisper encoder, speech adaptor, layer-sum weights, norms, embeddings and all 1-D tensors are preserved bf16 — they are small and disproportionately sensitive to quantization noise.

Reproduce:

python -m pipelines.mlx_direct_quantize --model meralion3-10b \
    --base-dir /tmp/mlx-direct-release/meralion3-10b/base \
    --out-dir /tmp/mlx-direct-release/meralion3-10b/4bit \
    --bits 4 --mode affine --group-size 64

Intended use

Speech understanding for Singapore-context English, Mandarin, Malay, Tamil (plus Indonesian, Thai, Vietnamese) locally on Apple Silicon, where the 8-bit pack does not fit alongside other workloads.

Licence

Derivative of MERaLiON-3-10B under the MERaLiON-3-Public-Licence, which permits derivatives with notices. This repository redistributes quantized weights with the required attribution; upstream model, training data and evaluation details are documented on the base model card.