Ministral-3-3B-Reasoning-2512 — LiteRT-LM (blockwise int4)

mistralai/Ministral-3-3B-Reasoning-2512 converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Text-only conversion (the Ministral-3 text decoder; the Pixtral vision tower is dropped).


File	`model.litertlm` (~2.2 GB; embedding externalized so every section is <2 GiB → loads on iOS)
Quantization	int4 weights — blockwise (block 32), symmetric, OCTAV clipping; tied embedding/lm_head at INT8
Compute	integer
Context (KV cache)	4096
Base model	mistralai/Ministral-3-3B-Reasoning-2512 (Apache-2.0)

Usage

Run with the LiteRT-LM runtime:

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total?"

The .litertlm bundle carries the tokenizer and the prompt template (Ministral's native Mistral [INST] … [/INST] format, stop token </s>), so no separate tokenizer files are needed. This is a reasoning model: it works the problem step-by-step before giving the final answer (best evaluated with a generous token budget — see below), and terminates cleanly at </s>.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this model on a phone is the official Google AI Edge Gallery app — it runs .litertlm models fully on-device and can import your own:

Install a recent Gallery (package com.google.ai.edge.gallery, APK from the repo's releases — 1.0.15+ supports .litertlm).
Download model.litertlm from this repo and push it to the device:
```
adb push model.litertlm /sdcard/Download/
```
In the app, tap the + button (bottom-right), pick the file, and choose the GPU backend (CPU also works).
Chat. The bundle already carries the tokenizer and the native Mistral [INST] chat template, so nothing else needs configuring.

Device RAM (important — this is a ~2.2 GB / 3B model): GPU on Android needs roughly 2× the model size (weights plus the ML Drift GPU weight cache), so GPU is only offered on ~12 GB+ devices. On an 8 GB phone only CPU is selectable; free RAM first (close apps / reboot) or the app is OOM-killed on load. Because this is a reasoning model that emits long chains of thought, give it a high max-tokens.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Device	Backend	Prefill (256)	Decode	TTFT	Load	Peak footprint
Apple M4 Max (macOS)	CPU	123 tok/s	22.4 tok/s	2.41 s	—	—
Apple M4 Max (macOS)	GPU (Metal)	1241 tok/s	95.8 tok/s	0.23 s	—	—

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The model loads and generates on an iPhone 17 Pro, but no on-device timing was taken, so none is quoted.

Accuracy note

Measured on GSM8K (n=150, greedy, 0-shot chain-of-thought, max-tokens 2048 — a reasoning model needs the budget to finish its work; scoring it at 512 tokens falsely penalises it):

Configuration	GSM8K
This model — LiteRT int4 (block32 + OCTAV)	90.7%

90.7% is a strong on-device GSM8K for a 3B, and above the same-family Ministral-3-3B-Instruct LiteRT build (85%) — the reasoning traces help. No reasoning collapse, non-degenerate; the model also passes the local quality gate 8/8 with a clean stop at </s>. Blockwise-32 + OCTAV optimal-clipping (data-free) is what preserves the accuracy versus a naive min-max int4.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

file	GPU backend	delegation	peak
`model.litertlm`	runs	2274 / 2274 ops across 2 subgraphs on `LiteRT GPU`	1159 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with litert-torch using a blockwise int4 recipe (INT4 weights, block size 32, symmetric, OCTAV optimal-clipping) with the tied embedding/lm_head kept at INT8, KV cache 4096, and Ministral's native Mistral chat template. Ministral-3 is a standard dense decoder (Ministral3ForCausalLM), so it rides the existing converter and runtime with no custom graph code; only the text decoder is exported (the vision tower is dropped first, with a strict missing=0 / unexpected=0 weight check).

externalize_embedder=True (required for iPhone). This 3B's weights would otherwise be a single >2 GiB TFLite section, which exceeds the ~2 GiB single-section mmap limit on iOS — engine creation fails with "Failed to map section: Cannot allocate memory." Externalizing the (tied) embedding into its own section drops the main weights section under 2 GiB (total ~2.2 GB), so the model loads on iPhone (Metal GPU) as well as Android/desktop. Same weights, so GSM8K is unchanged.

Template note (important for any Mistral/Ministral): the model must be exported with its native Mistral [INST] … [/INST] template and real EOS </s> — not ChatML. Mistral's tokenizer has no <|im_end|> token, so under a ChatML template the int4 model never hits a registered stop token and runs away after the correct answer. With the Mistral template it stops cleanly.

Reproduce

# 1. extract the dense text decoder from the multimodal checkpoint (missing=0/unexpected=0)
python scripts/extract_text_backbone.py \
    mistralai/Ministral-3-3B-Reasoning-2512 \
    src_models/ministral-3-3b-reasoning-text

# 2. convert: blockwise-32 int4 + OCTAV, int8 embeddings, embedding externalized for iOS
EXTERNALIZE_EMBEDDER=1 FORCE_SPM=1 CACHE=4096 python scripts/export_simple_template.py \
    src_models/ministral-3-3b-reasoning-text \
    out/ministral-3-3b-reasoning-ext \
    templates/mistral_simple.jinja \
    BOCTAV4

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: # HOW YOU SHOULD THINK AND ANSWER First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your th…. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in model.litertlm now emits the block exactly once when no system message is given. In model.litertlm, the block is not emitted when you pass a system message. The restored block adds 123 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

2026-08-30 — tokenizer section replaced (weights unchanged)

The tokenizer in model.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (é, ñ, ü, °, ·, …) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and any character without a whole-character vocabulary entry (emoji, most of Latin Extended-A) became the token the conversion had reused as UNK — the end-of-text token for this vocabulary. model.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most bundles in this collection use).

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token — only the file's own sha256 differs. Verified on the LiteRT-LM runtime: the default turn, 7 probe strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). Prompts containing accented letters, symbols or emoji reach the model differently from before, so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too — the on-device gate has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

2026-08-31 — thought channel declared (metadata only, weights unchanged)

model.litertlm now declares the reasoning channel in its metadata (LlmMetadata.channels: channel name thought, markers [THINK]…[/THINK] exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.