DeepSeek-R1-Distill-Qwen-7B — LiteRT-LM (blockwise int4)

deepseek-ai/DeepSeek-R1-Distill-Qwen-7B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

A reasoning model: it emits a <think> … </think> chain before the answer. MIT-licensed (distilled onto an Apache-2.0 Qwen2.5 base). Converted with the official upstream litert-torch — no fork, no custom code.

File	DeepSeek-R1-Distill-Qwen-7B_q4_block32_ekv4096.litertlm (~4.2 GB)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8
Compute	integer
Context (KV cache)	4096
Base model	deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
Platforms	Desktop (Mac) ✓ · high-RAM (12 GB+) Android ✓ · iPhone / 8 GB phones ✗ (4 GB exceeds the budget)

Usage

litert_lm_main --model_path DeepSeek-R1-Distill-Qwen-7B_q4_block32_ekv4096.litertlm --backend gpu \
  --input_prompt "If a train travels 60 km in 45 minutes, what is its speed in km/h?"

The .litertlm bundle carries the tokenizer and the DeepSeek prompt template (<｜User｜> / <｜Assistant｜>, stop token <｜end▁of▁sentence｜>). The assistant opens a <think> block, reasons step by step, then gives the final answer (commonly in \boxed{}).

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Device	Backend	Prefill (256)	Decode	TTFT	Load	Peak footprint
Apple M4 Max (macOS)	CPU	64 tok/s	18.2 tok/s	4.69 s	—	—
Apple M4 Max (macOS)	GPU (Metal)	644 tok/s	65.9 tok/s	0.43 s	—	—

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

At 4.5 GB this build does not fit an 8 GB phone — its weights section exceeds the iOS mmap budget — so there is no on-device speed row. It does load and generate on a 12 GB Android phone: see the Galaxy S26 section below.

Accuracy note

GSM8K (n=100, greedy, 0-shot, identical prompt + answer-extraction; max_new_tokens=2048 to fit the reasoning chain).

Configuration	GSM8K
bf16 (reference)	88.0%
This model — LiteRT int4 (BOCTAV4)	87.0%

LiteRT int4 is at parity — −1.0 pt vs bf16. The reasoning behavior is fully preserved through 4-bit quantization; the shallow-wide Qwen2 (28 layers) absorbs int4 rounding cleanly.

Galaxy S26 — GPU backend

The published bundle runs on the Android GPU backend and generates.

file	GPU backend	delegation	peak
DeepSeek-R1-Distill-Qwen-7B_q4_block32_ekv4096.litertlm	runs	2375 / 2375 ops across 2 subgraphs on LiteRT GPU	1537 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with the official upstream litert-torch export_hf (clean git worktree at upstream/main, dev-fork patches excluded). Qwen2ForCausalLM rides the stock converter with no custom code. int4 recipe = blockwise (block 32) + OCTAV with INT8 embedding (externalized into its own bundle section); KV cache 4096.

Training data & PII

This is a weights-exact format conversion of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B; no new training was performed. It is a Qwen2.5-7B-family base supervised-fine-tuned by DeepSeek on ~800K reasoning traces generated by DeepSeek-R1; the distillation set is model-generated and the Qwen base pretraining corpus is web-derived and not fully disclosed. Web-derived data may incidentally contain PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for details.

2026-08-31 — thought channel declared (metadata only, weights unchanged)

DeepSeek-R1-Distill-Qwen-7B_q4_block32_ekv4096.litertlm now declares the reasoning channel in its metadata (LlmMetadata.channels: channel name thought, markers <think>…</think> exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
DeepSeek-R1-Distill-Qwen-7B_q4_block32_ekv4096.litertlm	11.0 (10.4–11.0)	1.8 (1.7–1.8)	28.3 s	5.4 GB

License

MIT (model weights), inherited from deepseek-ai/DeepSeek-R1-Distill-Qwen-7B; the Qwen2.5 base is Apache-2.0. Commercial use and derivatives permitted.