| --- |
| base_model: mistralai/Voxtral-Mini-4B-Realtime-2602 |
| language: |
| - ar |
| - de |
| - en |
| - es |
| - fr |
| - hi |
| - it |
| - nl |
| - pt |
| - zh |
| - ja |
| - ko |
| - ru |
| library_name: mlx |
| license: apache-2.0 |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - mlx |
| - mlx-audio |
| - speech-to-text |
| - streaming |
| - realtime |
| - voxtral |
| - fp16 |
| --- |
| |
| # Voxtral Mini 4B Realtime 4bit (float16) |
|
|
| This is a **4-bit quantized, float16-base** [MLX](https://github.com/ml-explore/mlx) conversion of [mistralai/Voxtral-Mini-4B-Realtime-2602](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602). |
|
|
| ## Which variant should you pick? |
|
|
| | Chip | Recommended | Why | |
| |----------------|--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| |
| | **M1 / M2** | **This repo (`-4bit-fp16`)** | Metal on M1/M2 has no native `bfloat16` ALU; bf16 ops fall back to a slower path. `float16` stays on the fast GPU path. | |
| | **M3 / M4+** | [`iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit`](https://huggingface.co/iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit) (bf16) | bf16 is natively supported and gives the same speed as fp16 with a wider dynamic range (slightly safer numerics). | |
|
|
| Only the non-quantized weights differ between the two repos (norms, biases, scales, some embeddings). The quantized mat-mul weights are bit-identical. Transcription output is byte-identical on a 20 s French clip at temperature 0 (verified locally). |
|
|
| ## Conversion |
|
|
| Source model: |
|
|
| - `mistralai/Voxtral-Mini-4B-Realtime-2602` |
|
|
| Local conversion command: |
|
|
| ```bash |
| python -m mlx_audio.convert \ |
| --hf-path mistralai/Voxtral-Mini-4B-Realtime-2602 \ |
| --mlx-path /path/to/Voxtral-Mini-4B-Realtime-2602-4bit-fp16 \ |
| --quantize \ |
| --q-group-size 64 \ |
| --q-bits 4 \ |
| --dtype float16 \ |
| --model-domain stt |
| ``` |
|
|
| Quantization config: |
|
|
| - bits: `4` |
| - group size: `64` |
| - mode: `affine` |
| - non-quant dtype: `float16` |
|
|
| ## Files |
|
|
| Only the MLX runtime artifacts needed for inference: |
|
|
| - `model.safetensors` |
| - `model.safetensors.index.json` |
| - `config.json` |
| - `generation_config.json` |
| - `params.json` |
| - `processor_config.json` |
| - `tekken.json` |
|
|
| ## Usage |
|
|
| ```bash |
| pip install "mlx-audio[stt]" |
| ``` |
|
|
| ```python |
| from mlx_audio.stt.utils import load_model |
| |
| model = load_model("iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit-fp16") |
| result = model.generate("audio.wav") |
| print(result.text) |
| ``` |
|
|
| ## Notes |
|
|
| - Base model license remains Apache 2.0. |
| - On M3/M4, prefer the `-4bit` (bf16) repo; there is no speed benefit to fp16 there and bf16's wider exponent range is slightly more robust. |
| - Transcription quality was verified identical to the bf16 variant at `temperature=0` on a 20 s French parliamentary audio clip. |
|
|
