aorabdel commited on
Commit
2f11eca
·
verified ·
1 Parent(s): a1c27c8

Sync model repo (text/metadata)

Browse files
Files changed (2) hide show
  1. README.md +0 -1
  2. benchmarks/README.md +0 -49
README.md CHANGED
@@ -1,6 +1,5 @@
1
  ---
2
  library_name: onnxruntime-genai
3
- display_name: TinyLlama-1.1B-Chat INT4 — ONNX GenAI (Vivo X300)
4
  license: apache-2.0
5
  tags:
6
  - text-generation
 
1
  ---
2
  library_name: onnxruntime-genai
 
3
  license: apache-2.0
4
  tags:
5
  - text-generation
benchmarks/README.md DELETED
@@ -1,49 +0,0 @@
1
- # Benchmarks
2
-
3
- Machine-readable measurement records for this model, using the delivered
4
- `nlp-llm` metadata shape from the ARM Model Optimization Pipeline.
5
-
6
- These complement the eval block in the model card. The card carries the headline
7
- numbers in human-readable form; these YAMLs carry the same data with target
8
- hardware, runtime config, dataset, and benchmark workload context.
9
-
10
- ## Files
11
-
12
- - `tinyllama-1-1b-chat-onnx-genai-vivo-x300-int4.yaml` — optimized INT4/INT8 ONNX
13
- Runtime GenAI measurements on the Vivo X300 smartphone via MLAS + KleidiAI.
14
- - `tinyllama-1-1b-chat-onnx-genai-vivo-x300-fp32.yaml` — FP32 runtime baseline on
15
- the same target/workload, from the delivery baseline metadata.
16
-
17
- ## Reading these
18
-
19
- Each record has three main areas:
20
-
21
- - `context.{model,target,runtime,dataset,benchmark}` — everything needed to
22
- understand the measurement setup.
23
- - `performance` — LLM runtime metrics: E2E latency percentiles, TTFT, decode
24
- throughput, model-load / time-to-first-inference, and peak/average memory.
25
- - `accuracy` — HellaSwag zero-shot accuracy (`acc_norm`, character-normalized).
26
-
27
- The two records share target/runtime/dataset/workload context. The disambiguator
28
- is `weight_dtype`.
29
-
30
- ## Notes specific to this model
31
-
32
- - This is the `TinyLlama/TinyLlama-1.1B-Chat-v1.0` chat model, re-packed to
33
- W4A8-dynamic INT4 (GPTQ) with INT8 per-row embeddings and an INT8 LM head via
34
- the `onnxruntime-genai` model builder (recipe `exp10_kquantlast_emb_int8_asym`).
35
- - The runtime metrics were measured on the **Vivo X300 premium smartphone**
36
- (C1-Ultra / C1-Premium / C1-Pro, Android 16 / OriginOS 6), not a server CPU or
37
- Raspberry Pi. The AWS Graviton G4 cut of this same INT4 model is a separate repo
38
- (`tinyllama-1-1b-chat-onnx-genai-int4-kquantlast-emb-int8-graviton-g4`); the optimized
39
- weights are byte-identical across the two targets — only the benchmark records differ.
40
- - The optimized `model.onnx.data` sidecar is about 728 MB and is gitignored because it
41
- exceeds GitHub's 100 MB per-file limit. The FP32 baseline ONNX artifact (~4.10 GB) is
42
- not present in this repository.
43
-
44
- ## Provenance
45
-
46
- Source: `models/tinyllama_1_1b_chat_vivo_x300_onnx/` (`metadata.yaml`,
47
- `baseline_metadata.yaml`, `report.json`, and `huggingface/README.md`). The optimized
48
- record maps from the delivered `metadata.yaml`; the FP32 baseline maps from the
49
- delivered `baseline_metadata.yaml`.