Qwen3.5-4B — LiteRT-LM

Qwen/Qwen3.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0). Same conversion rail as our Qwen3.5-0.8B — the GPU-delegable rank-≤4 gated-delta kernel — plus two 4B-specific fixes described in the conversion notes.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 4B model has 24 linear-attention + 8 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 8 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

File	Recipe	Size
Qwen3.5-4B_int8.litertlm	int8 dynamic on linears + embedding (convs and the delta rule stay float), fp32 activations declared	4.10 GB
Qwen3.5-4B_mixed_int4.litertlm	Mixed INT4: int4 blockwise-32 on linears, int8 on embedding + lm_head (convs and the delta rule stay float), fp32 activations declared	2.57 GB

The Mixed INT4 variant exists for RAM-constrained devices (requested in LiteRT-LM #1658 for 8 GB Android phones): 2.57 GB of weights instead of 4.10 GB, measured ~1.7 GB CPU peak on the 138-token probe. If you have the memory for int8, int8 answers harder prompts slightly better (measured below).

Correctness

All gates below were run on this exact file (or its float parent, same graph rail):

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): the composite 8-question probe (all 8 questions in one 138-token prompt) answers 8/8, word-for-word identical to the HF fp32 reference run greedy with the same template.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.

Mixed INT4 correctness (all gates re-run on Qwen3.5-4B_mixed_int4.litertlm):

• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). Prompt-length robustness: 40/40 CPU, 20/20 GPU.
• iPhone 17 Pro: the composite 8-question probe answers 8/8 on both Metal GPU and CPU (the int8 file answers 6 of 8 on CPU on this probe).
• GSM8K, n=100 (0-shot CoT, greedy, 2048-token budget, identical harness for both files, measured on this exact export): int8 97/100, Mixed INT4 93/100. That 4-point gap is the real cost of int4 on this model — invisible to the sanity gates, visible on math word problems. We also measured blockwise-128 (90/100) and OCTAV variants (90–92/100); blockwise-32 min-max won and is what ships.

Usage

litert-lm run ./Qwen3.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-4B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: 1024, 256, 64, 16, 4, 1 (not the 0.8B's full 1–1024 ladder). Every exported signature is charged engine memory even if never called, and with this model's 248k vocabulary the full ladder's Metal programs push a 12 GB iPhone past its jetsam ceiling during engine creation. The reduced ladder keeps GPU peak at ~4.9 GB on iPhone 17 Pro with no correctness cost (the runtime just plans slightly coarser chunks).

Chat template note: the bundle ships a simplified ChatML template rather than the stock Qwen3.5 template. Thinking is disabled the way the stock template's non-thinking mode does it (an empty <think>\n\n</think> block opens each assistant turn), and — deliberately — that block is kept in history renders too: the stock template strips it from past turns, which breaks LiteRT-LM's incremental conversation rendering (the engine requires each turn's render to be a string-extension of the previous one) and kills multi-turn on turn 2. Tool-calling and vision sections are not included.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	672 tok/s	62.0 tok/s	0.40 s
CPU	243 tok/s	19.9 tok/s	1.11 s

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	87.1 tok/s	9.3 tok/s	1.93 s	4.98 GB
iPhone 17 Pro	CPU	24.0 tok/s	5.4 tok/s	6.88 s	1.70 GB

(iPhone GPU: model load 66 s on first launch.)

Mixed INT4 (litert-lm benchmark 0.16.0, Apple M4 Max, same flags; iPhone rows are single cold runs of the 138-token quality probe, unplugged):

Device	Backend	Prefill	Decode	TTFT	Peak memory
Apple M4 Max	GPU	669 tok/s	68.5 tok/s	0.40 s	—
Apple M4 Max	CPU	100 tok/s	20.1 tok/s	2.61 s	—
iPhone 17 Pro	GPU (Metal)	88.7 tok/s	11.4 tok/s	1.89 s	5.74 GB
iPhone 17 Pro	CPU	46.7 tok/s	8.9 tok/s	3.12 s	1.68 GB
Pixel 8a (8 GB)	CPU (XNNPACK)	22.1 tok/s	6.3 tok/s	11.9 s	4.43 GB

(iPhone GPU: model load 65 s on first launch. Pixel 8a: litert_lm_main v0.16.0, 260-token prompt / 204 decoded tokens, warm weight cache; peak is process VmHWM; engine init 30–50 s; TTFT scales with prompt length — 4.2 s on a 24-token prompt.)

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). We have not verified GPU execution on Qualcomm Adreno devices — if you are on a Snapdragon phone, use the CPU backend unless you have confirmed the GPU path on your own device.
• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (and 8 GB-class Android phones): the int8 file is not attempted, and not expected to fit. The 0.8B's fp32-expanded GPU programs already exceeded this phone's ~3.8 GB available memory at 963 MB on disk; the int8 file is 4.3× larger, and its 4.1 GB of int8 weights alone leave no CPU headroom on an 8 GB phone either. Treat this release as Apple-hardware-first; an Android row can follow on 12 GB+ devices.
• The Mixed INT4 file runs on a Pixel 8a (8 GB) on CPU — measured, table above. Peak process memory 4.43 GB on a 7.75 GB-RAM phone, coherent long-form generation and correct QA output. Use the CPU backend on 8 GB phones: the GPU path expands to ~5.7 GB on Metal and Android GPU budgets are similar or tighter (the int8 2B already killed a Pixel 8a during GPU engine creation — do not try GPU on 8 GB devices). First run writes a 2.66 GB XNNPACK weight cache next to the model (one-time) — leave ~3 GB of storage free.
• Mixed INT4 CPU prefill is slower than int8's (Mac: 100 vs 243 tok/s prefill, TTFT 2.6 vs 1.1 s at 256 tokens; iPhone CPU is the exception — 46.7 vs 24.0, faster). Decode is equal or faster everywhere.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: individual questions gate 8/8 on every backend. On the harder composite probe (8 questions in one prompt), GPU matches HF fp32 word-for-word; the CPU int8 path answers 6 of the 8 (every answer it gives is correct — it skips two). If you need maximum fidelity on complex prompts on CPU, ask for a float variant.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Rank-4 head interleave (4B-specific): the 4B is the first Qwen3.5 with grouped value heads (32 value vs 16 key heads). Upstream expands q/k with repeat_interleave, whose rank-5 lowering the GPU delegate rejects outright — the 0.8B (ratio 1) never traces that branch. The patch re-expresses it as concat + rank-4 reshapes, bitwise-identical to the stock op.
• Reduced prefill ladder (4B-specific): see the signature note above.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• Mixed INT4 recipe: post-hoc ai-edge-quantizer on the same float export — int4 BLOCKWISE-32 min-max on every FULLY_CONNECTED, overridden to int8 channelwise on the lm_head and the embedding lookup (one shared 248320×2560 vocab buffer, kept single). Block size and algorithm were chosen by measurement: b32/b128 × min-max/OCTAV all built from the same export and scored on GSM8K n=100 (93 / 90 / 92 / 90) — blockwise-32 min-max won. Convs and the delta rule stay float, prefer_activation_type fp32 as in the int8 file.