sraivante commited on
Commit
e8e336e
·
verified ·
1 Parent(s): 73a8477

Add verified Raspberry Pi 5 edge performance measurements

Browse files

Record actual 16 GB Pi 5 latency, RAM and correctness with frozen evaluation inputs, raw per-call measurements, runtime hashes and reproducible settings. Preserve weights and existing training data.

README.md CHANGED
@@ -90,7 +90,7 @@ web UI and Docker image.
90
  <!-- rpi5-measurements -->
91
  ## Raspberry Pi 5 edge-device performance
92
 
93
- Measured **2026-09-26** on a **Raspberry Pi 5 Model B, 16 GB RAM**, Debian 13.2/aarch64, Python 3.13.5. Measurement update: `edge-rpi5-20260926`; weights and training data are unchanged.
94
 
95
  | Threads | Median ms | p95 ms | p99 ms | Serial requests/s | Process RSS MiB | CPU °C min–mean–max | Timed correct |
96
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -101,7 +101,7 @@ Batch 1; 20 warmups and 500 timed calls per row, cycling over 10 frozen model-sp
101
 
102
  On-device quality check: **192/200** on training-set diagnostic, NOT held-out accuracy. For v3 this is explicitly a training-set diagnostic, not held-out accuracy; the earlier 96.73% result was not rerun.
103
 
104
- [Full Pi methodology, temperature plot and reproduction](edge_benchmarks/rpi5-2026-09-26/REPORT.md) · [Every latency/temperature reading](edge_benchmarks/rpi5-2026-09-26/readings.csv) · [Raw measurements](edge_benchmarks/rpi5-2026-09-26/results) · [Versioned evaluation data](https://huggingface.co/datasets/sraivante/home-commands-json-v3/tree/edge-rpi5-20260926/edge_benchmarks/rpi5-2026-09-26).
105
  <!-- /rpi5-measurements -->
106
 
107
  ## Why a tiny model and not an LLM
 
90
  <!-- rpi5-measurements -->
91
  ## Raspberry Pi 5 edge-device performance
92
 
93
+ Measured **2026-09-26** on a **Raspberry Pi 5 Model B, 16 GB RAM**, Debian 13.2/aarch64, Python 3.13.5. Measurement update: `edge-rpi5-20260926.1`; weights and training data are unchanged.
94
 
95
  | Threads | Median ms | p95 ms | p99 ms | Serial requests/s | Process RSS MiB | CPU °C min–mean–max | Timed correct |
96
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
 
101
 
102
  On-device quality check: **192/200** on training-set diagnostic, NOT held-out accuracy. For v3 this is explicitly a training-set diagnostic, not held-out accuracy; the earlier 96.73% result was not rerun.
103
 
104
+ [Full Pi methodology, temperature plot and reproduction](edge_benchmarks/rpi5-2026-09-26/REPORT.md) · [Every latency/temperature reading](edge_benchmarks/rpi5-2026-09-26/readings.csv) · [Raw measurements](edge_benchmarks/rpi5-2026-09-26/results) · [Versioned evaluation data](https://huggingface.co/datasets/sraivante/home-commands-json-v3/tree/edge-rpi5-20260926.1/edge_benchmarks/rpi5-2026-09-26).
105
  <!-- /rpi5-measurements -->
106
 
107
  ## Why a tiny model and not an LLM
edge_benchmarks/rpi5-2026-09-26/REPORT.md CHANGED
@@ -1,6 +1,6 @@
1
  # Raspberry Pi 5 measurements — Tiny Home Robotics JSON 1M v3
2
 
3
- Measured on 2026-09-26; measurement revision `edge-rpi5-20260926`. The model weights and training data were not changed.
4
 
5
  Hardware: **Raspberry Pi 5 Model B Rev 1.1**, 15.84 GiB OS-visible RAM (16 GB model), four CPU cores, aarch64, Debian 13.2, kernel `6.12.47+rpt-rpi-2712`. Python 3.13.5. The device's governor and library versions are recorded in each JSON result. Cooling and power-supply model were not inspected.
6
 
@@ -39,12 +39,12 @@ Timed-correct counts above describe the repeated ten-input latency set, not an i
39
  - Published model tested: [`sraivante/superfast-tiny-home-robotics-json-1m-v3@2b2e71d37ea9fa0fe6ab247b6a74e8db774fb6f2`](https://huggingface.co/sraivante/superfast-tiny-home-robotics-json-1m-v3/tree/2b2e71d37ea9fa0fe6ab247b6a74e8db774fb6f2). All tested runtime assets were checked against this revision.
40
  - Learned weight SHA-256: `fc749ecd89c1ae30a159d6d701036298b5583645b7fbf114311e006929c8778b` (2,146,783 bytes).
41
  - Associated preserved training dataset: [`sraivante/home-commands-json-v3@da10aca352a34cb2834a049867f6f2e9fcc30e3e`](https://huggingface.co/datasets/sraivante/home-commands-json-v3/tree/da10aca352a34cb2834a049867f6f2e9fcc30e3e). Existing source/split hashes and historical provenance remain in that snapshot. No fine-tuning was performed for this measurement update.
42
- - The [versioned evaluation inputs and Pi predictions](https://huggingface.co/datasets/sraivante/home-commands-json-v3/tree/edge-rpi5-20260926/edge_benchmarks/rpi5-2026-09-26) record the new diagnostic snapshot. `provenance.json` records its hashes, counts, source revisions, and historical identity limitations.
43
  - [Runtime identity check](runtime_identity.json); [raw per-call records](results); [exact environment](environment.lock.txt).
44
 
45
  ## Reproduce
46
 
47
- Use a 64-bit Raspberry Pi in an isolated environment. Download this model repository at `edge-rpi5-20260926`, then run:
48
 
49
  ```bash
50
  python3 -m venv .venv
 
1
  # Raspberry Pi 5 measurements — Tiny Home Robotics JSON 1M v3
2
 
3
+ Measured on 2026-09-26; measurement revision `edge-rpi5-20260926.1`. The model weights and training data were not changed.
4
 
5
  Hardware: **Raspberry Pi 5 Model B Rev 1.1**, 15.84 GiB OS-visible RAM (16 GB model), four CPU cores, aarch64, Debian 13.2, kernel `6.12.47+rpt-rpi-2712`. Python 3.13.5. The device's governor and library versions are recorded in each JSON result. Cooling and power-supply model were not inspected.
6
 
 
39
  - Published model tested: [`sraivante/superfast-tiny-home-robotics-json-1m-v3@2b2e71d37ea9fa0fe6ab247b6a74e8db774fb6f2`](https://huggingface.co/sraivante/superfast-tiny-home-robotics-json-1m-v3/tree/2b2e71d37ea9fa0fe6ab247b6a74e8db774fb6f2). All tested runtime assets were checked against this revision.
40
  - Learned weight SHA-256: `fc749ecd89c1ae30a159d6d701036298b5583645b7fbf114311e006929c8778b` (2,146,783 bytes).
41
  - Associated preserved training dataset: [`sraivante/home-commands-json-v3@da10aca352a34cb2834a049867f6f2e9fcc30e3e`](https://huggingface.co/datasets/sraivante/home-commands-json-v3/tree/da10aca352a34cb2834a049867f6f2e9fcc30e3e). Existing source/split hashes and historical provenance remain in that snapshot. No fine-tuning was performed for this measurement update.
42
+ - The [versioned evaluation inputs and Pi predictions](https://huggingface.co/datasets/sraivante/home-commands-json-v3/tree/edge-rpi5-20260926.1/edge_benchmarks/rpi5-2026-09-26) record the new diagnostic snapshot. `provenance.json` records its hashes, counts, source revisions, and historical identity limitations.
43
  - [Runtime identity check](runtime_identity.json); [raw per-call records](results); [exact environment](environment.lock.txt).
44
 
45
  ## Reproduce
46
 
47
+ Use a 64-bit Raspberry Pi in an isolated environment. Download this model repository at `edge-rpi5-20260926.1`, then run:
48
 
49
  ```bash
50
  python3 -m venv .venv
edge_benchmarks/rpi5-2026-09-26/provenance.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "measurement_revision": "edge-rpi5-20260926",
3
  "model": {
4
  "repo_id": "sraivante/superfast-tiny-home-robotics-json-1m-v3",
5
  "revision": "2b2e71d37ea9fa0fe6ab247b6a74e8db774fb6f2",
 
1
  {
2
+ "measurement_revision": "edge-rpi5-20260926.1",
3
  "model": {
4
  "repo_id": "sraivante/superfast-tiny-home-robotics-json-1m-v3",
5
  "revision": "2b2e71d37ea9fa0fe6ab247b6a74e8db774fb6f2",