On-device performance: speed, temperature and accuracy
Model: Union command parser, 6.44M params (d=256, 5 encoder + 3 decoder layers, 8 heads), character-level
encoder-decoder with grammar-constrained decoding over 371 actions. Weights: model.npz (23.9 MB, float32).
Inference: pure NumPy on CPU, one prompt at a time, greedy decoding. Measured 2026-09-27.
Summary
| Laptop (Intel i7-1360P) | Raspberry Pi 5 (16 GB) | |
|---|---|---|
| Accuracy, exact (2,340 prompts) | 97.82% | 97.82% |
| Accuracy, action only | 98.46% | 98.46% |
| Model load time | 0.13 s | 0.20 s |
| Latency p50 / p95 / p99 (10 min sustained) | 18.4 / 168.6 / 239.6 ms | 30.2 / 243.9 / 281.6 ms |
| Mean latency (10 min sustained) | 45.8 ms | 72.0 ms |
| Throughput, first pass (cool) | 29.6 prompts/s | 13.9 prompts/s |
| Throughput, last 5 minutes (hot) | 18.5 prompts/s | 13.9 prompts/s |
| Temperature idle → peak | 52.6 → 91.9 °C | 38.5 → 59.8 °C |
| Thermal throttling | slowed down by ~39% once hot (see below) | none (get_throttled=0x0, 2400 MHz throughout) |
| Best BLAS thread setting | 4 threads | 2 threads |
- Accuracy is identical on both devices. The same predictions come out on x86-64 and on ARM64.
- The Pi 5 holds its speed. Its latency did not change across 10 minutes of continuous load. With the active cooler, the SoC stayed below 60 °C and the fan never went above state 2.
- The laptop is about 2.1x faster than the Pi when cool, but only about 1.3x faster once hot. After it reached 88–92 °C, its throughput fell from ~30 to ~18 prompts/s.
Accuracy per evaluation set
The numbers come from the runtime, so they include the post-decode "snap" repair. They are the same on both devices.
| Set | n | Exact | Action | What it tests |
|---|---|---|---|---|
| curated | 371 | 99.46% | 99.73% | one held-out prompt per action (all 371 actions) |
| wifi_holdout | 461 | 100.00% | 100.00% | unseen Wi-Fi connect phrasings (SSID/password extraction) |
| gap11_holdout | 696 | 99.28% | 100.00% | unseen templates: password-first Wi-Fi, CPU clock vs usage, board↔BCM pins, fan vs mouse speed |
| B_golden_final | 384 | 97.40% | 97.40% | external golden set, including out-of-scope prompts (unknown) |
| A_practical-holdout | 428 | 92.06% | 94.16% | external practical set; some misses are label-convention differences |
| All | 2,340 | 97.82% | 98.46% |
"Exact" means the action and every slot value match. "Action" means only the action label matches.
Speed vs temperature under sustained load
Each device ran the same 2,340 prompts in a fixed shuffled order (seed 0), repeated for 10 minutes. Every pass is the same workload, so any change between passes comes from the device, not from the prompts.
Laptop, Intel Core i7-1360P (Windows 11, on AC power, Balanced power plan, 4 BLAS threads)
| Pass | Temp (°C) min–max | Avg clock (MHz) | Mean (ms) | p50 (ms) | p95 (ms) | Prompts/s |
|---|---|---|---|---|---|---|
| 1 | 51.9 – 90.9 | 3067 | 33.8 | 14.6 | 111.0 | 29.6 |
| 2 | 83.9 – 91.9 | 2838 | 37.5 | 15.6 | 121.4 | 26.7 |
| 3 | 81.9 – 91.9 | 2872 | 42.4 | 17.0 | 150.4 | 23.6 |
| 4 | 81.9 – 89.9 | 2853 | 55.8 | 22.5 | 202.0 | 17.9 |
| 5 | 81.9 – 86.9 | 2959 | 55.2 | 22.7 | 196.2 | 18.1 |
| 6 (partial, 1,411) | 79.9 – 82.9 | 2954 | 52.5 | 21.3 | 184.8 | 19.0 |
Raspberry Pi 5, 16 GB (Raspberry Pi OS, kernel 6.12, ondemand governor, active cooler (PWM fan), 2 BLAS threads)
| Pass | Temp (°C) min–max | Clock (MHz) | Mean (ms) | p50 (ms) | p95 (ms) | Prompts/s | Throttled |
|---|---|---|---|---|---|---|---|
| 1 | 41.1 – 57.6 | 2400 | 72.2 | 30.4 | 244.5 | 13.86 | 0x0 |
| 2 | 57.1 – 59.8 | 2400 | 72.0 | 30.2 | 243.9 | 13.90 | 0x0 |
| 3 | 56.5 – 59.3 | 2400 | 71.9 | 29.9 | 243.8 | 13.91 | 0x0 |
| 4 (partial, 1,314) | 56.5 – 58.7 | 2400 | 71.9 | 30.5 | 242.5 | 13.90 | 0x0 |
Per-minute view
| Minute | Laptop °C | Laptop MHz | Laptop mean ms | Laptop prompts/s | Pi °C | Pi MHz | Pi mean ms | Pi prompts/s | Pi fan state |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 88.9 | 3145 | 33.2 | 30.1 | 54.9 | 2400 | 72.2 | 13.9 | 1 |
| 2 | 91.9 | 3094 | 34.4 | 29.1 | 57.6 | 2400 | 74.5 | 13.4 | 1 |
| 3 | 91.9 | 2745 | 39.2 | 25.5 | 57.6 | 2400 | 68.9 | 14.5 | 1 |
| 4 | 91.9 | 2819 | 39.9 | 25.0 | 58.7 | 2400 | 73.5 | 13.6 | 1 |
| 5 | 88.9 | 2703 | 61.8 | 16.2 | 59.8 | 2400 | 72.4 | 13.8 | 1 |
| 6 | 89.9 | 2966 | 56.3 | 17.7 | 59.8 | 2400 | 70.5 | 14.2 | 1→2 |
| 7 | 84.9 | 2936 | 52.0 | 19.3 | 58.2 | 2400 | 71.5 | 14.0 | 2 |
| 8 | 86.9 | 2815 | 59.7 | 16.8 | 58.2 | 2400 | 72.5 | 13.8 | 2 |
| 9 | 84.9 | 2982 | 49.5 | 20.3 | 58.2 | 2400 | 71.7 | 14.0 | 2 |
| 10 | 81.9 | 2969 | 53.5 | 18.7 | 58.7 | 2400 | 72.5 | 13.8 | 2 |
About the laptop slowdown: from pass 1 to pass 5, throughput fell by 39% and mean latency rose by 63%, but the Windows all-core frequency counter fell by only 3–10%. So the clock drop alone does not explain the slowdown. The likely causes are the package power limit and the scheduler moving work between P-cores and E-cores. This counter cannot tell those apart.
Latency depends on output length
The model decodes one token at a time, so latency grows with the length of the command it writes. Short commands
(volume_up, set_volume value=50) finish in about 10–20 ms on the laptop and 20–35 ms on the Pi 5. Commands with long text slots (connect_wifi ssid=… password=…, web searches, file paths) make up the p95/p99 tail: about 110–200 ms on the laptop and about 245 ms on
the Pi 5.
BLAS threads matter more than core count
We ran 300 mixed prompts per setting, with OPENBLAS_NUM_THREADS and OMP_NUM_THREADS set before NumPy loads.
| Threads | Laptop mean / p95 (ms) | Pi 5 mean / p95 (ms) |
|---|---|---|
| NumPy default (all cores) | 130.1 / 519.9 | 82.0 / 286.1 |
| 1 | 32.6 / 113.0 | 82.3 / 295.2 |
| 2 | 31.6 / 108.2 | 66.2 / 232.6 |
| 4 | 30.7 / 99.6 | 81.2 / 280.5 |
The matrices in this model are small. With OpenBLAS's default setting of one thread per core (16 on the laptop),
thread synchronisation costs more than the arithmetic, which makes the laptop about 4x slower. Set
OPENBLAS_NUM_THREADS=4 on x86 laptops and OPENBLAS_NUM_THREADS=2 on the Raspberry Pi 5.
Test setup
| Laptop | Raspberry Pi 5 | |
|---|---|---|
| CPU | Intel Core i7-1360P (4P + 8E cores, 16 threads) | Broadcom BCM2712, 4× Cortex-A76 @ 2.4 GHz |
| RAM | 32 GB | 16 GB |
| OS | Windows 11 (build 26200) | Raspberry Pi OS (Debian 13), kernel 6.12.47 |
| Python / NumPy | 3.13.14 / 2.5.3 (OpenBLAS 0.3.34) | 3.13.5 / 2.2.4 |
| Cooling / power | built-in fan, AC power, Balanced plan | active cooler (PWM fan), ondemand governor |
| Temperature source | ACPI thermal zone (\_TZ.TZ01), not the CPU core sensor |
SoC sensor (vcgencmd measure_temp) |
| Clock source | % Processor Performance × base frequency (all-core average) |
vcgencmd measure_clock arm |
Method: a 30 s idle reading, 20 warm-up prompts, then 10 minutes of continuous single-stream inference. Temperature, clock, throttle flags and fan state were sampled every 10 s. Accuracy was measured on the first full pass. Both devices ran at the same time, and nothing else heavy ran on either one (the laptop also ran the SSH session that drove the Pi).
Reproduce
# from the agent folder; <data> holds curated / wifi_holdout / gap11_holdout / B_golden_final / A_practical-holdout .jsonl
cd eval
sh bench_run_all.sh python ../model <data> <out_dir> default 1 2 4 # thread probe, picks the fastest, 2 min cooldown, 10 min run
Raw results are in eval/bench_results/ (*_threads.json = thread probe, the other files = full run with per-pass
and per-minute data).