huluhuluu/llama-3.2-3b-instruct-mllm-qnn-htp

Llama-3.2-3B-Instruct — MLLM QNN AOT context binaries (v75 + v79)

Two pre-compiled QNN context binaries of Llama-3.2-3B-Instruct for the MLLM runtime on the Qualcomm Hexagon NPU, plus the 160-request latency measurement.

Completion status differs per SoC — read this before using either file.

Target	Artifact	160-request run
SM8650 / HTP v75 (Redmi K70 Pro)	v75-sm8650/…-v75.bin	complete, 160/160 validated
SM8750 / HTP v79 (Realme RMX5090)	v79-sm8750/…-v79.bin	NOT complete — uploaded as a backup

The v79 context itself is fully built and passed its on-device smoke test. What is missing is its 160-request measurement: two attempts ended at 60/160 and 24/160 when the phone powered off (ro.boot.bootreason=shutdown,battery). Both phones only negotiate 500 mA @ 5 V (2.5 W) over USB, which is below the draw of sustained HTP inference, so a ~2.7 h group runs at a net battery loss. This is a harness/power limitation, not a defect in the context binary. The partial result is included under results/realme-sm8750-INCOMPLETE/ as evidence only and must not be merged with a later run.

Artifacts

File	Bytes	SHA-256
v75-sm8650/llama3.2-3b-lpbq-sha-ctx2048-v75.bin	2,531,520,512	20d682c678e8bc3301defdc4968681d199b1031b4a8efdcd8bcbbddbffd911af
v79-sm8750/llama3.2-3b-lpbq-sha-ctx2048-v79.bin	2,529,255,424	6fa07cfa27c0a8821fe77a5ae0c584568ebfeb9680a79f12f0123405f0e44d1f

Each binary contains two restored graphs, model.0.s32 (prefill, AR 32) and model.0.s1 (decode), each with 59 input and 57 output tensors.

Quantization and export

Item	Value
Source	meta-llama/Llama-3.2-3B-Instruct (local licensed checkout)
Method	PTQ via pymllm/mobile/backends/qualcomm/transformers/llama/train.py
Linear implementation	QNN_LPBQ_w4a16o16_G32 — LPBQ, 4-bit weights / 16-bit activations / 16-bit output, group size 32
Attention layout	SHA (split-head attention)
Calibration	128 samples, max_seq_length 2048, Wikipedia (English), datasets==2.21.0, host CUDA fp32
Sanity check	8 tokens post-calibration
Context length	2048 (max_cache_length: 2048)
Prefill AR length	32
Compiler	QNN SDK 2.40 (QNN Backend Build Id: v2.40)
Custom ops	libQnnLLaMAPackage_HTP.so / _CPU.so, registers DequantizeAdd:4097

Model config (config_3B.json, included in this repo):

hidden_size 3072 | intermediate_size 8192 | head_dim 128
num_attention_heads 24 | num_key_value_heads 8 | num_hidden_layers 28
rope_theta 500000.0 | rms_norm_eps 1e-05 | vocab_size 128256
bos_token_id 128000 | eos_token_id 128009 | tie_word_embeddings true

Measured latency (SM8650 / v75 only)

Serial FIFO, one resident mllm-llama-aot-jsonl-runner, 160 requests executed contiguously with no restart. Greedy, context 2048, AR 32. Total wall time 187.6 min (70.4 s/request).

Metric	Value
Requests	160/160 status=ok
TTFT P50 (model-side)	400.017 ms
TPOT P50	56.613 ms
TPOT samples	197,785
TTFT SLO attainment	0.1187
TPOT SLO attainment	0.2543
Result SHA-256	84d1736a7216eae3415d5e282029c5b0c828c44d2659cdf79cdbec7d1ccc0ae3

Per-dataset P50 (ms):

Dataset	TTFT	TPOT
gsm8k	1376.379	56.550
humaneval	337.159	56.642
math500	266.758	56.625
mt_bench	528.468	56.603

Smoke reference (32 tokens, before the full run): Redmi TTFT 1181.3 ms / mean decode interval 46.54 ms; Realme TTFT 931.6 ms / 40.98 ms. Both had 32/32 strictly increasing token timestamps.

Partial Realme evidence

results/realme-sm8750-INCOMPLETE/result_full.interrupted.jsonl — 60 rows, all status=ok with complete token traces, SHA-256 d038844a9d94c2b47d4b0265217e5b5bdd80624c1f9ea2efe177791e74082919. Wall time 60.1 min for those 60 requests (60.1 s/request). It is a valid prefix, not a complete run, and no summary statistic in this card is derived from it.

Workload

160 requests: 40 each from gsm8k / humaneval / math500 / mt_bench, replayed as pre-tokenized input_ids (max token id 128009 < vocab 128256). max_new_tokens is per-request from the workload (601–1822). Included as workload/llama3.2-3b-inst.jsonl.

Validation applied to every completed result: exact workload request order, status=ok, completion_tokens == output_tokens == len(token_trace), contiguous token_index, strictly increasing device_commit_ns.

Runtime requirements

LD_LIBRARY_PATH=<dir>
ADSP_LIBRARY_PATH=<dir>:/odm/lib64/se_algo:/odm/lib/rfsa/adsp:/vendor/lib/rfsa/adsp:/system/lib/rfsa/adsp:/dsp
MLLM_QNN_IO_MEM_ESTIMATION=1

Omitting the skel directory from ADSP_LIBRARY_PATH produces Failed to create device: 14001, which is a library-path fault, not a permission or signing problem.

Launch the device process under setsid nohup — otherwise reaping the host-side adb wrapper kills inference mid-run.

Scope

Built for a serial-arrival, single-request latency comparison across on-device inference frameworks. Batch size 1; no continuous batching, no speculative decoding. Accuracy was not evaluated — only latency and token-trace integrity.