Sync model repo (text/metadata)

README.md

@@ -0,0 +1,313 @@
+---
+library_name: onnxruntime-genai
+display_name: TinyLlama-1.1B-Chat INT4 — ONNX GenAI (Vivo X300)
+license: apache-2.0
+tags:
+  - text-generation
+  - tinyllama
+  - llama
+  - int4
+  - quantized
+  - onnx
+  - onnxruntime-genai
+  - mlas
+  - kleidiai
+  - arm
+  - android
+  - vivo-x300
+  - edge-ai
+  - k_quant_last
+pipeline_tag: text-generation
+datasets:
+  - Rowan/hellaswag
+metrics:
+  - accuracy
+base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+base_model_relation: quantized
+model-index:
+  - name: tinyllama-1.1b-chat-onnx-genai-exp10-kquantlast-emb-int8-asym
+    results:
+      - task:
+          type: text-generation
+          name: Commonsense Reasoning (HellaSwag)
+        dataset:
+          type: Rowan/hellaswag
+          name: HellaSwag
+          split: validation
+          args:
+            eval_samples: 8000
+        metrics:
+          - type: accuracy
+            value: 59.00
+            name: HellaSwag Accuracy (acc_norm, character-normalized)
+          - type: accuracy
+            value: 45.98
+            name: HellaSwag Accuracy (raw, acc)
+---
+
+# TinyLlama-1.1B-Chat INT4 + INT8 Embeddings — ONNX (GenAI-Builder, k_quant_last + asym, MLAS + KleidiAI on Vivo X300)
+
+Mixed-precision quantized version of [`TinyLlama/TinyLlama-1.1B-Chat-v1.0`](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0): **INT4 weights (GPTQ)** for the transformer-block Linears, **INT8 weights for the LM head** (`k_quant_last` protection), **INT8 per-token dynamic activations**, and an **INT8 per-row embedding table** — benchmarked on a **Vivo X300 (Android 16 / OriginOS 6)** smartphone with the MediaTek Dimensity 9500 (C1-Ultra / C1-Premium / C1-Pro, aarch64). The decode loop is dispatched almost entirely (99.84 % of runtime) to ONNX Runtime's MLAS execution provider, with `MatMulNBits` hitting the **KleidiAI fast-path** (`qsi8d32p x qsi4c32p` SDOT kernel) for W4A8-dynamic GEMM acceleration on the phone's ARMv9 cores.
+
+## Key Highlights
+
+Paired on-device benchmark on the Vivo X300 smartphone (Android 16 / OriginOS 6, `onnxruntime-genai` 1.27.0, `intra_threads = 4`, 30 measured runs, 5 warm-ups, ~154-token prompt, 30 decoded tokens) — the FP32 baseline (`optimum-cli` ONNX export, 4197 MB) and the optimized INT4+INT8-emb bundle were run in the same session on the same phone:
+
+- **Bundle size — 727.86 MB** vs FP32 4196.99 MB (**5.77× compression**).
+- **55.65 tok/s decode throughput** on-device vs FP32 7.82 tok/s (**+47.83 tok/s, 7.12× speedup**).
+- **510.14 ms time-to-first-token** for a ~154-token prompt vs FP32 905.95 ms (−395.81 ms).
+- **883.66 ms cold-start model load** vs FP32 5900.94 ms (−5017 ms); **890.47 MB peak USS memory** vs FP32 4086.28 MB (**4.59× memory reduction**).
+- **1047.62 ms E2E p50 latency** vs FP32 4757.49 ms (−3710 ms).
+- **99.84 % MLAS time coverage** — 5 376 / 5 592 operators (96.14 %) dispatched to MLAS; `MatMulNBits` (83.96 % of decode time) uses the KleidiAI fast-path.
+- **Minimal accuracy loss** — HellaSwag `acc_norm` **59.0 %** (Δ **-0.80 pp** vs the 59.8 % FP32 baseline; identical bundle to the Graviton measurement — INT8/INT4 weights are hardware-independent).
+
+## Model Details
+
+### Model Description
+
+W4A8-dynamic quantized re-pack of TinyLlama's `TinyLlama-1.1B-Chat-v1.0` for efficient ARM-CPU inference via `onnxruntime-genai`. Every Linear projection in attention and the FFN is re-packed to per-row group-wise **INT4 weights** (group size 32) using **GPTQ** as the weight-calibration algorithm — recipe `exp10_kquantlast_emb_int8_asym`, asymmetric per-group `(scale, zero_point)`. The **final LM-head projection** is protected by the `k_quant_last` algorithm and kept at **INT8 weights** rather than being compressed to 4 bits (the output-vocabulary projection is the most sensitive layer in the network). Activations are quantized to **INT8 per-token dynamic** at runtime: each row gets its own `delta_t = max(|X[t, :]|) / 127`, so the loudest token and the median token both use the full INT8 grid. The token embedding table is stored **INT8 per-row** (one `(scale, zero_point)` per vocabulary token).
+
+- **Developed by:** TinyLlama project (base model); ONNX INT4 re-pack by Marvik using the `onnxruntime-genai` model-builder toolchain.
+- **Model type:** Causal Language Model (decoder-only Transformer, 22 layers, hidden size 2048, 32 / 4 GQA, 2048-token context).
+- **License:** Apache-2.0 (inherited from the TinyLlama-1.1B-Chat-v1.0 base model).
+- **Base model:** [`TinyLlama/TinyLlama-1.1B-Chat-v1.0`](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) — quantized, not fine-tuned.
+- **Deployment target for this card:** Vivo X300 (MediaTek Dimensity 9500, 8-core aarch64 — 2x C1-Ultra + 3x C1-Premium + 3x C1-Pro, 16 GB LPDDR, Android 16 / OriginOS 6).
+
+### Model Sources
+
+- **Base model:** https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
+- **TinyLlama project:** https://github.com/jzhang38/TinyLlama
+- **ONNX Runtime GenAI:** https://github.com/microsoft/onnxruntime-genai
+
+## How to Get Started with the Model
+
+### Install dependencies
+
+```bash
+# onnxruntime + the genai wrapper (Python 3.12)
+pip install "onnxruntime>=1.22" "onnxruntime-genai>=0.14" tokenizers jinja2
+```
+
+> **On-device Android note.** On-device benchmarking of this bundle was done with a cross-compiled `bench_genai` C runner pushed to the phone via ADB (see the sibling Android runner in the source repo). The Python `example.py` in this card is meant to be used off-device (Linux / macOS / aarch64 server) to reproduce the same accuracy from the same bundle. `onnxruntime-genai >= 0.14.0` ships `manylinux_2_28_aarch64` wheels for Linux aarch64; Android on-device inference is done via the ONNX Runtime C API + KleidiAI-enabled shared library.
+
+### Download the model
+
+```python
+from huggingface_hub import snapshot_download
+
+local_dir = snapshot_download(
+    repo_id="Arm/tinyllama-1-1b-chat-onnx-genai-int4-kquantlast-emb-int8-vivo-x300",
+    allow_patterns=[
+        "model.onnx",
+        "model.onnx.data",
+        "genai_config.json",
+        "tokenizer.json",
+        "tokenizer_config.json",
+        "tokenizer.model",
+        "special_tokens_map.json",
+        "chat_template.jinja",
+    ],
+)
+```
+
+### Run inference
+
+The bundled `example.py` runs a single chat-style generation end-to-end and saves the result to `predictions.json` next to the script:
+
+```bash
+python example.py
+```
+
+Core inference loop (equivalent to what `example.py` does):
+
+```python
+import onnxruntime_genai as og
+
+# Load the model directory (must contain model.onnx + model.onnx.data + genai_config.json + tokenizer)
+model = og.Model("./model_dir")
+tokenizer = og.Tokenizer(model)
+
+prompt = (
+    "<|user|>\nExplain in one paragraph what gravity is and why it matters.</s>\n"
+    "<|assistant|>"
+)
+input_ids = tokenizer.encode(prompt)
+
+params = og.GeneratorParams(model)
+params.set_search_options(max_length=256, do_sample=False, temperature=0.0)
+generator = og.Generator(model, params)
+generator.append_tokens(input_ids)
+
+while not generator.is_done():
+    generator.generate_next_token()
+
+print(tokenizer.decode(generator.get_sequence(0)))
+```
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+- **HellaSwag** — [`Rowan/hellaswag`](https://huggingface.co/datasets/Rowan/hellaswag) validation split, full 8 000 questions, zero-shot. Accuracy is bundle-only (identical `model.onnx` + `model.onnx.data` used on the Vivo phone and on the Graviton server), so `acc_norm` = 59.00 % / raw `acc` = 45.98 % applies verbatim to this deployment.
+- **On-device latency & memory** — measured on a **Vivo X300** smartphone: 30 measured runs + 5 warm-ups, ~154-token prompt (rendered from the standard "gravity" prompt via the bundled Zephyr chat template), 30 decoded tokens per run, `intra_op_num_threads = 4`.
+
+#### Metrics
+
+- **HellaSwag Accuracy (`acc_norm`)** — character-length-normalized accuracy. For each (context, 4 endings) tuple, the model scores each ending's log-likelihood under the context, divides by the ending's character length, and picks the argmax. This is the `lm_eval` equivalent of `acc_norm` and the standard HellaSwag headline number.
+- **HellaSwag Accuracy (raw `acc`)** — same task, without length normalization. Reported alongside `acc_norm` because raw accuracy is more sensitive to logit-scale drift introduced by quantization.
+- **Decode Throughput (tok/s)** — end-to-end tokens/sec over a 30-token continuation on the Vivo X300.
+- **TTFT (ms)** — wall-clock latency from "submit prompt" to "first token emitted" for a ~154-token prompt.
+- **Per-token Decode Latency (ms)** — median autoregressive cost per generated token.
+- **E2E Latency (p50 / p90 / p99)** — end-to-end run wall-clock (prefill + 30 decoded tokens), across the 30 measured runs.
+- **Peak Memory (USS, MB)** — peak unique-set-size of the `bench_genai` process on the phone (mmapped weights + KV cache + ORT scratch).
+- **MLAS Operator / Time Coverage** — fraction of ONNX operators (and runtime) dispatched to ORT's MLAS execution provider.
+
+### Results
+
+#### Accuracy (HellaSwag, zero-shot, 8 000 validation samples)
+
+Accuracy is deterministic per bundle (INT8/INT4 arithmetic is hardware-independent), so these numbers are the same on Vivo X300 and on Graviton — they characterize the exp10 recipe, not the device.
+
+| Metric | FP32 (Original) | INT4 + INT8-emb (Optimized) | Delta |
+|:---|:---:|:---:|:---:|
+| HellaSwag Accuracy (`acc_norm`) | 59.80 % | **59.00 %** | **-0.80 pp** |
+| HellaSwag Accuracy (raw `acc`)  | 46.76 % | **45.98 %** | **-0.78 pp** |
+| Standard error (per side, pp)   | 0.55 | 0.55 | — |
+
+The 95 % CIs (approx. +/- 1.10 pp at n = 8 000) overlap, so the small drop on both metrics is within statistical noise.
+
+#### On-device Performance (Vivo X300, MLAS + KleidiAI, intra_threads = 4, 30 measured runs, paired)
+
+| Metric | FP32 (Original) | INT4 + INT8-emb (Optimized) | Delta |
+|:---|:---:|:---:|:---:|
+| Decode Throughput               | 7.82 tok/s | **55.65 tok/s** | **+47.83 tok/s (7.12×)** |
+| Time-to-First-Token (~154-token prompt) | 905.95 ms | **510.14 ms** | −395.81 ms |
+| Per-token Decode (median)       | 127.87 ms | **17.97 ms** | −109.90 ms (7.12×) |
+| Decode Total (30 tokens)        | 3.84 s | **0.54 s** | −3.30 s |
+| E2E Latency p50                 | 4757.49 ms | **1047.62 ms** | −3709.86 ms |
+| E2E Latency p90                 | 4798.12 ms | **1218.37 ms** | −3579.75 ms |
+| E2E Latency p99                 | 4804.92 ms | **1251.55 ms** | −3553.37 ms |
+| Time-to-First-Inference (cold)  | 4414.16 ms | **1009.50 ms** | −3404.66 ms |
+| Model Load Time (cold)          | 5900.94 ms | **883.66 ms** | −5017.28 ms |
+| Init Memory (USS)               | 4063.73 MB | **863.54 MB** | −3200.19 MB |
+| Median Memory (USS)             | 4067.23 MB | **866.41 MB** | −3200.82 MB (4.69×) |
+| Peak Memory (USS)               | 4086.28 MB | **890.47 MB** | −3195.80 MB (4.59×) |
+| CPU utilisation (peak)          | 800.00 % (8 active threads) | 800.00 % (8 active threads) | — |
+
+The FP32 baseline is the `optimum-cli` ONNX export of `TinyLlama/TinyLlama-1.1B-Chat-v1.0` (4196.99 MB). Both bundles were pushed via ADB to the same phone and benched back-to-back in the same session — the numbers above are directly comparable. The optimized bundle beats FP32 on every latency axis: prefill (TTFT), per-token decode, cold-start load, and end-to-end p50/p90/p99. `MatMulNBits` on the KleidiAI fast-path is what makes the 7.12× decode speedup possible.
+
+#### Runtime Backend Coverage (optimized model, on-device)
+
+| Metric | Value |
+|:---|---:|
+| Total operators (full decode) | 5 592 |
+| MLAS operators                | 5 376 (**96.14 %**) |
+| Default-CPU operators         | 216 |
+| MLAS Time Coverage            | **99.84 %** |
+| KleidiAI fast-path operators  | `MatMulNBits` (2 664 ops, 83.96 % of decode time) |
+
+Top ops by runtime share: `MatMulNBits` 83.96 %, `GroupQueryAttention` 11.60 %, `SkipSimplifiedLayerNormalization` 1.68 %, `QuickGelu` 1.54 %, `Mul` 1.02 % — everything remaining is in the noise floor (< 0.1 % each: `Cast`, `Gather`, `SimplifiedLayerNormalization`, `ReduceSum`, `Sub`, `Unsqueeze`, `Shape`). Non-MLAS ops (216 total, 0.16 % of runtime): `Cast`, `Gather`, `ReduceSum`, `Shape`, `Unsqueeze`.
+
+#### Model Size
+
+| Artifact | Size |
+|:---|---:|
+| `model.onnx` (graph)         | ~193 KB |
+| `model.onnx.data` (external INT4 weights + INT8 embedding table) | **~727 MB** |
+| Tokenizer (`tokenizer.json` + `.model`) | ~4 MB |
+| **Total bundle**             | **~728 MB** |
+
+Parameter count: 1 100 M (unchanged — quantization repacks weights, does not prune them).
+
+## Technical Specifications
+
+### Objective
+
+Causal language modelling / instruction following — given a sequence of input tokens, predict the next-token distribution; iterate to generate a continuation. Fine-tuned for chat-style turns with a 32k SentencePiece vocabulary and a 2048-token context window.
+
+### Quantization
+
+- **Method:** Post-Training Quantization (PTQ) — weights packed by the ONNX Runtime GenAI `model_builder` with `int4_algo_config=k_quant_last`, `int4_block_size=32`, `int4_is_symmetric=false`. The INT4 weight quantization is calibrated with **GPTQ** on **WikiText2** (256 sequences × 512 tokens each, ~131k tokens total). Activations are quantized at runtime per-token (dynamic INT8), and the embedding table is stored INT8 per-row.
+- **Bit-width:** INT4 weights (packed 2-per-byte) + INT8 per-token dynamic activations (W4A8 dyn) + INT8 per-row embeddings + **INT8 LM-head weights** (protected).
+- **Weight calibration algorithm:** **GPTQ** — layer-wise Hessian-based error compensation over 256 WikiText2 sequences of 512 tokens each (~131k calibration tokens total); produces the per-group `(scale, zero_point)` values for the INT4-packed `MatMulNBits` tensors.
+- **Granularity (weights):** Per-row, group-wise (group size = 32).
+- **Granularity (activations):** Per-row (per-token) dynamic — `(scale, zero_point)` recomputed at runtime, no offline calibration.
+- **Symmetry:** Asymmetric weights (each group stores `(scale, zero_point)`); asymmetric activations (per-token min/max).
+- **Last-layer protection (`k_quant_last`):** the **final LM-head projection** (vocab-size output MatMul, ~64 MB) is kept at **INT8 weights** rather than compressed to INT4. This is the single most quantization-sensitive layer in the network — protecting it recovers most of the `acc_norm` drop that a pure-INT4 export would incur, at the cost of ~32 MB extra bundle size relative to full INT4.
+- **Embedding table:** Stored as **INT8 per-row** (one `(scale, zero_point)` per vocabulary token). The lookup is a `Gather` op (not a MatMul), so quantizing it is a pure storage choice — no kernel changes downstream.
+- **Backend:** ONNX Runtime CPU EP with MLAS + KleidiAI fast-path for `MatMulNBits`.
+- **Layers kept at higher precision:** LM head at INT8; LayerNorms, residual adds, GeLU stay in FP16/FP32 inside the MLAS-fused `GroupQueryAttention` / `SkipSimplifiedLayerNormalization` ops because they are already cheap.
+
+### Export Pipeline
+
+1. Load the FP32 base model from `TinyLlama/TinyLlama-1.1B-Chat-v1.0`.
+2. Export to FP32 ONNX via `optimum-cli export onnx` (the ~4.10 GB baseline).
+3. Re-pack with the ONNX Runtime GenAI model builder using `--extra_options int4_is_symmetric=false int4_block_size=32 int4_algo_config=k_quant_last`. GPTQ calibration data (256 WikiText2 sequences × 512 tokens each, ~131k tokens) is fed to the builder so the per-group `(scale, zero_point)` values minimise the layer-wise reconstruction loss rather than being derived from bare min/max. The builder packs transformer-block Linear weights into `MatMulNBits` (4-bit, group = 32, asymmetric, GPTQ-calibrated), keeps the final LM-head projection at INT8 via `k_quant_last`, fuses attention into a single `GroupQueryAttention` op, wires up the KV-cache inputs/outputs, and writes `model.onnx` + `model.onnx.data` + `genai_config.json`.
+4. Apply INT8 per-row surgery to the token embedding table (saves ~96 MB on a 32 000 x 2 048 table at no measurable accuracy cost).
+5. Ship the tokenizer files alongside (`tokenizer.json`, `tokenizer.model`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja`).
+
+### On-device Runner
+
+For the on-phone benchmark reported in this card, the same bundle was pushed to the Vivo X300 via ADB and invoked from a small C runner (`bench_genai`) linked against the ORT + `onnxruntime-genai` shared libraries with KleidiAI enabled. The runner does 5 warm-up + 30 measured generations of 30 tokens each from the rendered chat prompt, samples per-thread CPU / RSS every 200 ms while running, and reports median TTFT / tokens-per-second / p90/p99 E2E latency / peak USS memory / MLAS coverage from the ORT profiling JSON.
+
+### Preprocessing
+
+| Property | Value |
+|---|---|
+| Input shape | `[1, T]` (`int64` `input_ids`) |
+| Tokenizer | `tokenizer.json` (Llama SentencePiece BPE, 32 000 vocab) |
+| Chat template | `chat_template.jinja` (Zephyr-style `<|user|>` / `<|assistant|>` markers) |
+| `position_ids` input | **Not exposed** — the GenAI graph derives positions internally |
+
+**Steps**:
+1. Apply the chat template to the user prompt (wraps in `<|user|>\n...</s>\n<|assistant|>`).
+2. Tokenize with the bundled SentencePiece BPE tokenizer to obtain `input_ids`.
+3. Hand to `onnxruntime-genai`'s `Generator`; the runtime owns the autoregressive loop and the KV cache.
+
+**Normalization**: None required — `input_ids` are integer token indices, not real-valued features.
+
+### Postprocessing
+
+| Property | Value |
+|---|---|
+| Output | One token-id per decode step (greedy argmax over the 32 000-token vocabulary) |
+| Decoder | Same SentencePiece BPE tokenizer used for input |
+| HellaSwag scoring | Per-ending log-likelihood under the context, normalized by ending character length (`acc_norm`) or raw (`acc`) |
+
+**Steps**:
+1. Iterate `Generator.generate_next_token()` until EOS (`</s>`) is emitted or the `max_length` budget is hit.
+2. `Tokenizer.decode(output_ids, skip_special_tokens=True)` to obtain the human-readable continuation.
+
+## Known Limitations
+
+- **Evaluation scope is HellaSwag-only.** HellaSwag is a four-choice commonsense reasoning benchmark; it does not measure code, math, multi-turn instruction following, or long-context behaviour. Quantization-sensitive benchmarks (MMLU, GSM8K, HumanEval) may show larger gaps than the -0.80 pp seen here.
+- **Latency & memory are Vivo X300-specific.** The MediaTek Dimensity 9500 (C1-Ultra / C1-Premium / C1-Pro, ARMv9) is a recent premium mobile SoC; older Android SoCs (Cortex-A76, A78, Snapdragon 7-series) or non-Android aarch64 devices (Raspberry Pi, Cortex-A55) will see materially different absolute numbers. Relative INT4-vs-FP32 speedup is expected to remain favourable because both paths share the same MLAS + KleidiAI code paths.
+- **Fixed `intra_op_num_threads = 4`** in the published Vivo runtime numbers. Batch > 1 was not measured; larger thread counts on the phone's big cores may further improve throughput at the cost of memory / battery.
+- **2 048-token context window.** Inherited from the TinyLlama base model — long-context inputs (> 2 048 tokens) are not supported.
+- **`onnxruntime-genai` aarch64 wheel ships from 0.14.0.** Older versions (<= 0.13.2) require a source build on aarch64 to enable KleidiAI. On Android the runtime is built from source via the ORT NDK toolchain rather than pip-installed.
+
+## Citation
+
+If you use this model, please also cite the upstream papers:
+
+```bibtex
+@misc{zhang2024tinyllama,
+  title={TinyLlama: An Open-Source Small Language Model},
+  author={Peiyuan Zhang and Guangtao Zeng and Tianduo Wang and Wei Lu},
+  year={2024},
+  eprint={2401.02385},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL}
+}
+
+@misc{zellers2019hellaswag,
+  title={HellaSwag: Can a Machine Really Finish Your Sentence?},
+  author={Rowan Zellers and Ari Holtzman and Yonatan Bisk and Ali Farhadi and Yejin Choi},
+  year={2019},
+  eprint={1905.07830},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL}
+}
+```

benchmarks/README.md

@@ -0,0 +1,49 @@
+# Benchmarks
+
+Machine-readable measurement records for this model, using the delivered
+`nlp-llm` metadata shape from the ARM Model Optimization Pipeline.
+
+These complement the eval block in the model card. The card carries the headline
+numbers in human-readable form; these YAMLs carry the same data with target
+hardware, runtime config, dataset, and benchmark workload context.
+
+## Files
+
+- `tinyllama-1-1b-chat-onnx-genai-vivo-x300-int4.yaml` — optimized INT4/INT8 ONNX
+  Runtime GenAI measurements on the Vivo X300 smartphone via MLAS + KleidiAI.
+- `tinyllama-1-1b-chat-onnx-genai-vivo-x300-fp32.yaml` — FP32 runtime baseline on
+  the same target/workload, from the delivery baseline metadata.
+
+## Reading these
+
+Each record has three main areas:
+
+- `context.{model,target,runtime,dataset,benchmark}` — everything needed to
+  understand the measurement setup.
+- `performance` — LLM runtime metrics: E2E latency percentiles, TTFT, decode
+  throughput, model-load / time-to-first-inference, and peak/average memory.
+- `accuracy` — HellaSwag zero-shot accuracy (`acc_norm`, character-normalized).
+
+The two records share target/runtime/dataset/workload context. The disambiguator
+is `weight_dtype`.
+
+## Notes specific to this model
+
+- This is the `TinyLlama/TinyLlama-1.1B-Chat-v1.0` chat model, re-packed to
+  W4A8-dynamic INT4 (GPTQ) with INT8 per-row embeddings and an INT8 LM head via
+  the `onnxruntime-genai` model builder (recipe `exp10_kquantlast_emb_int8_asym`).
+- The runtime metrics were measured on the **Vivo X300 premium smartphone**
+  (C1-Ultra / C1-Premium / C1-Pro, Android 16 / OriginOS 6), not a server CPU or
+  Raspberry Pi. The AWS Graviton G4 cut of this same INT4 model is a separate repo
+  (`tinyllama-1-1b-chat-onnx-genai-int4-kquantlast-emb-int8-graviton-g4`); the optimized
+  weights are byte-identical across the two targets — only the benchmark records differ.
+- The optimized `model.onnx.data` sidecar is about 728 MB and is gitignored because it
+  exceeds GitHub's 100 MB per-file limit. The FP32 baseline ONNX artifact (~4.10 GB) is
+  not present in this repository.
+
+## Provenance
+
+Source: `models/tinyllama_1_1b_chat_vivo_x300_onnx/` (`metadata.yaml`,
+`baseline_metadata.yaml`, `report.json`, and `huggingface/README.md`). The optimized
+record maps from the delivered `metadata.yaml`; the FP32 baseline maps from the
+delivered `baseline_metadata.yaml`.

benchmarks/tinyllama-1-1b-chat-onnx-genai-vivo-x300-fp32.yaml

@@ -0,0 +1,57 @@
+version: 1.0.0
+task: llm-generative
+created_at: '2026-07-10T13:41:20Z'
+context:
+  model:
+    id: Arm/tinyllama-1-1b-chat-onnx-genai-int4-kquantlast-emb-int8-vivo-x300
+    filename: model.onnx
+    base_model_id: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+    profile: Baseline
+    weight_dtype: fp32
+    model_size_mb: 4196.994
+    format: onnx
+  target:
+    name: Vivo X300
+    hardware_class: Premium smartphone
+    cpu_architecture: arm64
+    cpu_model: C1-Ultra, C1-Premium, C1-Pro
+    cpu_core_count: 8
+    cpu_clock_ghz: 4.21
+    system_memory_gb: 16
+    os: android
+    os_version: Android 16 / OriginOS 6
+  runtime:
+    name: onnxruntime
+    execution_backend: cpu
+    version: 1.27.0
+    config:
+      n_threads: 8
+      optimisations:
+        - MLAS
+        - KleidiAI
+  dataset:
+    name: HellaSwag
+    reference_url: https://huggingface.co/datasets/Rowan/hellaswag
+    slice: validation
+    sample_count: 8000
+  benchmark:
+    batch_size: 1
+    num_runs: 30
+    warmup_runs: 5
+    prompt_length_tokens: 154
+    generation_length_tokens: 30
+accuracy:
+  benchmark_name: HellaSwag
+  accuracy_pct: 59.8
+  shot_count: 0
+performance:
+  end_to_end_latency_ms:
+    p50: 4757.487
+    p90: 4798.116
+    p99: 4804.921
+  peak_memory_mb: 4086.28
+  average_memory_mb: 4067.23
+  ttft_ms: 905.95
+  tokens_per_second: 7.82
+  model_load_time_ms: 5900.936
+  time_to_first_inference_ms: 4414.16

benchmarks/tinyllama-1-1b-chat-onnx-genai-vivo-x300-int4.yaml

@@ -0,0 +1,70 @@
+version: 1.0.0
+task: llm-generative
+created_at: '2026-07-09T02:55:51Z'
+context:
+  model:
+    id: Arm/tinyllama-1-1b-chat-onnx-genai-int4-kquantlast-emb-int8-vivo-x300
+    filename: model.onnx
+    base_model_id: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+    profile: Arm-Optimized
+    weight_dtype: int4
+    quantization:
+      method: GPTQ
+      weight_bits: 4
+      activation_bits: 8
+      symmetric: false
+      mode: dynamic
+      weight_granularity: per-group
+      variant: W4A8_dyn_emb_int8
+    calibration:
+      dataset_name: WikiText2
+      sample_count: 256
+      selection: random
+    model_size_mb: 727.861
+    format: onnx
+  target:
+    name: Vivo X300
+    hardware_class: Premium smartphone
+    cpu_architecture: arm64
+    cpu_model: C1-Ultra, C1-Premium, C1-Pro
+    cpu_core_count: 8
+    cpu_clock_ghz: 4.21
+    system_memory_gb: 16
+    os: android
+    os_version: Android 16 / OriginOS 6
+  runtime:
+    name: onnxruntime
+    execution_backend: cpu
+    version: 1.27.0
+    config:
+      n_threads: 8
+      optimisations:
+        - MLAS
+        - KleidiAI
+  dataset:
+    name: HellaSwag
+    reference_url: https://huggingface.co/datasets/Rowan/hellaswag
+    slice: validation
+    sample_count: 8000
+  benchmark:
+    batch_size: 1
+    num_runs: 30
+    warmup_runs: 5
+    prompt_length_tokens: 154
+    generation_length_tokens: 30
+accuracy:
+  benchmark_name: HellaSwag
+  accuracy_pct: 59.0
+  shot_count: 0
+performance:
+  end_to_end_latency_ms:
+    p50: 1047.625
+    p90: 1218.37
+    p99: 1251.551
+  peak_memory_mb: 890.47
+  average_memory_mb: 866.41
+  delegation_pct: 99.84
+  ttft_ms: 510.14
+  tokens_per_second: 55.65
+  model_load_time_ms: 883.655
+  time_to_first_inference_ms: 1009.498

chat_template.jinja

@@ -0,0 +1,15 @@
+{% for message in messages %}
+{% if message['role'] == 'user' %}
+{{ '<|user|>
+' + message['content'] + eos_token }}
+{% elif message['role'] == 'system' %}
+{{ '<|system|>
+' + message['content'] + eos_token }}
+{% elif message['role'] == 'assistant' %}
+{{ '<|assistant|>
+'  + message['content'] + eos_token }}
+{% endif %}
+{% if loop.last and add_generation_prompt %}
+{{ '<|assistant|>' }}
+{% endif %}
+{% endfor %}

config.yaml

@@ -0,0 +1,64 @@
+input:
+  shape: [1, "T"]
+  dtype: int64
+  name: input_ids
+  preprocessing:
+    - apply_chat_template: chat_template.jinja
+    - tokenize:
+        tokenizer: tokenizer.json
+        add_special_tokens: true
+  notes: |
+    `T` is the runtime sequence length (variable, up to 2048). `position_ids`
+    is NOT exposed as an input — the ONNX Runtime GenAI graph derives positions
+    internally. KV-cache I/O is wired up internally by
+    `onnxruntime-genai.Generator` and is not exposed at the Python API surface.
+
+output:
+  format: "Token ids streamed one at a time by onnxruntime-genai.Generator"
+  postprocessing:
+    decode:
+      tokenizer: tokenizer.json
+    stop_on:
+      - eos_token
+      - max_length
+
+generation:
+  default_max_length: 256
+  default_decode_tokens: 30
+  do_sample: false
+  temperature: 0.0
+  top_p: 1.0
+  intra_op_num_threads: 4
+
+model:
+  format: onnx
+  graph_layout: onnxruntime-genai
+  fused_attention: GroupQueryAttention
+  num_layers: 22
+  hidden_size: 2048
+  num_attention_heads: 32
+  num_kv_heads: 4
+  head_size: 64
+  vocab_size: 32000
+  max_context_length: 2048
+
+quantization:
+  weight_dtype: int4
+  weight_symmetric: false
+  weight_group_size: 32
+  weight_calibration_algorithm: GPTQ    # per-group (scale, zero_point) fit by GPTQ
+  weight_algo_config: k_quant_last      # LM-head protection (see last_layer_dtype)
+  last_layer_dtype: int8                # final LM-head projection kept INT8
+  last_layer_protection: k_quant_last
+  activation_dtype: int8
+  activation_mode: per_token_dynamic
+  activation_symmetric: false
+  embedding_dtype: int8
+  embedding_granularity: per_row  # one (scale, zero_point) per vocabulary token
+  scheme: W4A8_dyn_emb_int8_lmhead_int8
+
+runtime:
+  execution_provider: cpu
+  mlas: true
+  kleidiai: true
+  onnxruntime_version: 1.27.0

example.py

@@ -0,0 +1,166 @@
+"""Minimal inference example for TinyLlama-1.1B-Chat exp10 W4A8 (ONNX Runtime GenAI).
+
+Loads the ONNX Runtime GenAI model + tokenizer + Jinja chat template *from this
+folder* and runs a single chat-style generation, then saves the response to
+``predictions.json`` next to this script. Requires:
+
+    onnxruntime>=1.22
+    onnxruntime-genai>=0.14.0   # Linux-aarch64 wheels available since 0.14.0
+    tokenizers                  # HF tokenizers library (loads tokenizer.json)
+    jinja2                      # renders the bundled chat template
+
+The folder is expected to contain the standard GenAI bundle:
+
+    model.onnx
+    model.onnx.data            # external-data weights side-car
+    genai_config.json
+    tokenizer.json
+    tokenizer_config.json
+    chat_template.jinja        # TinyLlama Zephyr-style chat template
+
+Run it from inside this directory:
+
+    python example.py
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import onnxruntime_genai as og
+from jinja2 import Environment
+from tokenizers import Tokenizer
+
+
+# -- Configuration -------------------------------------------------------------
+# HellaSwag-style commonsense continuation prompt. Small chat models
+# (~1B) generate cleanest output on well-scoped everyday scenarios (see
+# TinyLlama's 59% HellaSwag acc_norm) vs open-ended factual prose which
+# tends to hallucinate and loop.
+DEFAULT_PROMPT = (
+    "A woman is in the kitchen making pancakes. She pours the batter onto "
+    "a hot pan and waits for bubbles to appear on the surface. Once the "
+    "bubbles pop, she"
+)
+MAX_LENGTH = 256  # absolute token budget (prompt + decoded)
+DO_SAMPLE = False  # greedy decode for reproducibility
+TEMPERATURE = 0.0  # ignored unless do_sample=True
+
+
+def _raise_exception(msg: str) -> None:
+    """Bridge for the chat template's ``raise_exception`` helper."""
+    raise RuntimeError(msg)
+
+
+def render_chat_template(
+    template_path: Path, bundle_dir: Path, user_prompt: str
+) -> str:
+    """Render ``chat_template.jinja`` with a single user turn.
+
+    Reads BOS/EOS tokens from ``tokenizer_config.json`` so the rendered string
+    matches the actual tokenizer's special tokens. ``add_generation_prompt=True``
+    appends the assistant header so the model continues from there.
+    """
+    template_src = template_path.read_text(encoding="utf-8")
+    with (bundle_dir / "tokenizer_config.json").open() as f:
+        tok_cfg = json.load(f)
+
+    env = Environment(trim_blocks=True, lstrip_blocks=True, autoescape=False)
+    env.globals["raise_exception"] = _raise_exception
+    template = env.from_string(template_src)
+
+    return template.render(
+        messages=[{"role": "user", "content": user_prompt}],
+        bos_token=tok_cfg.get("bos_token", "<s>"),
+        eos_token=tok_cfg.get("eos_token", "</s>"),
+        add_generation_prompt=True,
+    )
+
+
+def load_model(bundle_dir: Path) -> tuple[og.Model, Tokenizer]:
+    """Load the ONNX Runtime GenAI model and the bundled HF tokenizer."""
+    model = og.Model(str(bundle_dir))
+    tokenizer = Tokenizer.from_file(str(bundle_dir / "tokenizer.json"))
+    return model, tokenizer
+
+
+def generate(
+    model: og.Model,
+    tokenizer: Tokenizer,
+    chat_prompt: str,
+) -> tuple[str, int, int]:
+    """Run greedy decode for one chat turn.
+
+    Returns ``(decoded_response, prompt_token_count, generated_token_count)``.
+    """
+    # The chat template already contains the BOS marker; don't double-add it.
+    input_ids = tokenizer.encode(chat_prompt, add_special_tokens=False).ids
+
+    params = og.GeneratorParams(model)
+    params.set_search_options(
+        max_length=MAX_LENGTH,
+        do_sample=DO_SAMPLE,
+        temperature=TEMPERATURE,
+    )
+
+    generator = og.Generator(model, params)
+    generator.append_tokens(input_ids)
+    while not generator.is_done():
+        generator.generate_next_token()
+
+    full_ids = list(generator.get_sequence(0))
+    response_ids = full_ids[len(input_ids) :]
+    decoded = tokenizer.decode(response_ids, skip_special_tokens=True)
+    return decoded, len(input_ids), len(response_ids)
+
+
+def save_results(
+    bundle_dir: Path,
+    user_prompt: str,
+    response: str,
+    prompt_tokens: int,
+    generated_tokens: int,
+) -> Path:
+    """Persist the prompt/response pair as ``predictions.json``."""
+    output_path = bundle_dir / "predictions.json"
+    payload = {
+        "prompt": user_prompt,
+        "response": response,
+        "prompt_tokens": prompt_tokens,
+        "generated_tokens": generated_tokens,
+        "max_length": MAX_LENGTH,
+        "do_sample": DO_SAMPLE,
+        "temperature": TEMPERATURE,
+    }
+    output_path.write_text(json.dumps(payload, indent=2, ensure_ascii=False))
+    return output_path
+
+
+def main() -> None:
+    bundle_dir = Path(__file__).resolve().parent
+    template_path = bundle_dir / "chat_template.jinja"
+
+    print(f"Loading model from: {bundle_dir}")
+    model, tokenizer = load_model(bundle_dir)
+
+    chat_prompt = render_chat_template(template_path, bundle_dir, DEFAULT_PROMPT)
+    print(f"\nPrompt: {DEFAULT_PROMPT}\n")
+    print("Generating...")
+    response, prompt_tokens, generated_tokens = generate(model, tokenizer, chat_prompt)
+
+    print("\n--- Response ---")
+    print(response)
+    print("--- /Response ---")
+    print(
+        f"\nPrompt tokens: {prompt_tokens} | " f"Generated tokens: {generated_tokens}"
+    )
+
+    saved = save_results(
+        bundle_dir, DEFAULT_PROMPT, response, prompt_tokens, generated_tokens
+    )
+    print(f"Saved: {saved}")
+
+
+if __name__ == "__main__":
+    main()

genai_config.json

@@ -0,0 +1,49 @@
+{
+    "model": {
+        "bos_token_id": 1,
+        "context_length": 2048,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": []
+            },
+            "filename": "model.onnx",
+            "head_size": 64,
+            "hidden_size": 2048,
+            "inputs": {
+                "input_ids": "input_ids",
+                "attention_mask": "attention_mask",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "logits": "logits",
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 32,
+            "num_hidden_layers": 22,
+            "num_key_value_heads": 4
+        },
+        "eos_token_id": 2,
+        "pad_token_id": 0,
+        "type": "llama",
+        "vocab_size": 32000
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 2048,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": true,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 50,
+        "top_p": 1.0
+    }
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 2048,
+  "pad_token_id": 0,
+  "transformers_version": "4.35.0"
+}

metadata.yaml

@@ -0,0 +1,68 @@
+version: 1.0.0
+task: llm-generative
+created_at: '2026-07-09T02:55:51Z'
+context:
+  model:
+    id: Arm/tinyllama-1-1b-chat-onnx-genai-int4-kquantlast-emb-int8-vivo-x300
+    base_model_id: TinyLlama/TinyLlama-1.1B-Chat-v1.0
+    profile: Arm-Optimized
+    weight_dtype: int4
+    quantization:
+      method: GPTQ
+      weight_bits: 4
+      activation_bits: 8
+      symmetric: false
+      mode: dynamic
+      weight_granularity: per-group
+    calibration:
+      dataset_name: WikiText2
+      sample_count: 256
+      selection: random
+    model_size_mb: 727.861
+    parameter_count: 1100179456
+    format: onnx
+    filename: model.onnx
+  target:
+    name: Vivo X300
+    hardware_class: Premium smartphone
+    cpu_architecture: arm64
+    cpu_model: C1-Ultra, C1-Premium, C1-Pro
+    cpu_core_count: 8
+    system_memory_gb: 16
+    os: android
+    os_version: Android 16 / OriginOS 6
+  runtime:
+    name: onnxruntime
+    execution_backend: cpu
+    config:
+      optimisations:
+      - MLAS
+      - KleidiAI
+      n_threads: 8
+    version: 1.27.0
+  dataset:
+    name: HellaSwag
+    sample_count: 8000
+    reference_url: https://huggingface.co/datasets/Rowan/hellaswag
+  benchmark:
+    batch_size: 1
+    num_runs: 30
+    warmup_runs: 5
+    prompt_length_tokens: 154
+    generation_length_tokens: 30
+performance:
+  end_to_end_latency_ms:
+    p50: 1047.625
+    p90: 1218.37
+    p99: 1251.551
+  model_load_time_ms: 883.655
+  time_to_first_inference_ms: 1009.498
+  ttft_ms: 510.14
+  tokens_per_second: 55.65
+  peak_memory_mb: 890.47
+  average_memory_mb: 866.41
+  delegation_pct: 99.84
+accuracy:
+  benchmark_name: HellaSwag
+  accuracy_pct: 59.0
+  shot_count: 0

predictions.json

@@ -0,0 +1,9 @@
+{
+  "prompt": "A woman is in the kitchen making pancakes. She pours the batter onto a hot pan and waits for bubbles to appear on the surface. Once the bubbles pop, she",
+  "response": "A man is in the kitchen making pancakes. He pours the batter onto a hot pan and waits for bubbles to appear on the surface. Once the bubbles pop, he uses a spatula to smooth out the surface of the pancake and then flips it over.",
+  "prompt_tokens": 56,
+  "generated_tokens": 64,
+  "max_length": 256,
+  "do_sample": false,
+  "temperature": 0.0
+}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 2048,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}