v2: compact (4.3 GB)

README.md

@@ -1,57 +1,71 @@
----
-license: apache-2.0
-base_model: google/gemma-4-E4B-it
-pipeline_tag: image-text-to-text
-library_name: openvino
-tags:
-  - openvino
-  - npu
-  - intel
-  - intel-npu
-  - gemma4
-  - onw
-language:
-  - en
-  - ja
----
-
-# Gemma 4 E4B-it for onw — every component on the Intel NPU
-
-[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) (text + image) packaged for
-**[onw](https://huggingface.co/ryugyosoft/onw)** (俺のNPUがこんなに動くわけない), the engine that runs LLMs entirely
-on the Intel NPU. Same NPU graphs as the standalone
-[ryugyosoft/gemma-4-E4B-it-npu](https://huggingface.co/ryugyosoft/gemma-4-E4B-it-npu), run by the shared engine.
-
-```bash
-hf download ryugyosoft/onw --local-dir onw && cd onw
-start.bat ryugyosoft/gemma-4-E4B-it-onw          # Windows
-bash start.sh ryugyosoft/gemma-4-E4B-it-onw      # Ubuntu
-```
-
-| NPU 3720 (Core Ultra 9 285HX) | |
-|---|---|
-| decode | **6.7-6.8 tok/s** (prompt lookup decoding on, exact) |
-| image + question (284 tokens) | vision 1.9 s + prefill 2.0 s (64-token blocks, >= 24 GB RAM) |
-| follow-up turn | only new tokens are processed (turn 2 of an image chat: 1.7 s vs 4.8 s) |
-| memory | ~8 GB with the 64-token block, ~6 GB without |
-
-What changed vs the standalone repo: the token embedding and the 2.7 GB per-layer embedding table are looked up
-on the host (they are table reads; no NPU worker process any more), the LM head is one shared INT8 input for all
-block sizes, and prompt lookup decoding / prefix reuse come from the engine.
-
-**Download with `hf download` or let `onw` fetch it** — a `git clone` without Git LFS gets pointer files, which
-`onw` detects and reports.
-
-## Files
-
-| file | what |
-|---|---|
-| `seg00_S{1,16,64}.xml` | the static Gemma 4 decoder (head_dim-512 attention split into 256-wide heads, host-built masks) for 1 / 16 / 64-token blocks |
-| `seg01_S*.xml` | LM head + logit softcapping (shared INT8 weight) |
-| `vision.xml` | static vision encoder (280 soft tokens) |
-| `shared.bin` | INT8 token embedding (= tied head), per-layer embedding table + id map |
-| `engine.json`, tokenizer / processor files | metadata for onw |
-
-## License
-
-Apache 2.0, same as the base model. The weights are re-quantized / restructured from it.
+---
+license: apache-2.0
+base_model: google/gemma-4-E4B-it
+pipeline_tag: image-text-to-text
+library_name: openvino
+tags:
+  - openvino
+  - npu
+  - intel
+  - intel-npu
+  - gemma4
+  - onw
+language:
+  - en
+  - ja
+---
+
+# Gemma 4 E4B-it for onw — every component on the Intel NPU
+
+[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) (text + image) packaged for
+**[onw](https://huggingface.co/ryugyosoft/onw)** (俺のNPUがこんなに動くわけない), the engine that runs LLMs entirely
+on the Intel NPU. Same NPU graphs as the standalone
+[ryugyosoft/gemma-4-E4B-it-npu](https://huggingface.co/ryugyosoft/gemma-4-E4B-it-npu), run by the shared engine.
+
+## Version 2 (compact)
+
+| | |
+|---|---|
+| download | 9.2 GB → **4.3 GB** |
+| memory in use (working set, NPU 3720) | 16.1 GB → 14.8 GB (without the 64-token block) |
+| what changed | decoder weights stored once for the 1 / 16 / 64-token graphs, INT4 per-layer embedding table (host lookup) |
+| checked | same answers as v1 on our image chat check (NPU 3720), decode 7.2 tok/s |
+
+Most of the download saving comes from storing identical weights once; in memory the NPU still keeps one
+compiled copy per block size, so the footprint shrinks less. Needs onw 0.3 or newer. The previous version
+stays available: `hf download ryugyosoft/gemma-4-E4B-it-onw --revision v1 --local-dir ...`.
+
+
+```bash
+hf download ryugyosoft/onw --local-dir onw && cd onw
+start.bat ryugyosoft/gemma-4-E4B-it-onw          # Windows
+bash start.sh ryugyosoft/gemma-4-E4B-it-onw      # Ubuntu
+```
+
+| NPU 3720 (Core Ultra 9 285HX) | |
+|---|---|
+| decode | **6.7-6.8 tok/s** (prompt lookup decoding on, exact) |
+| image + question (284 tokens) | vision 1.9 s + prefill 2.0 s (64-token blocks, >= 24 GB RAM) |
+| follow-up turn | only new tokens are processed (turn 2 of an image chat: 1.7 s vs 4.8 s) |
+| memory | ~8 GB with the 64-token block, ~6 GB without |
+
+What changed vs the standalone repo: the token embedding and the 2.7 GB per-layer embedding table are looked up
+on the host (they are table reads; no NPU worker process any more), the LM head is one shared INT8 input for all
+block sizes, and prompt lookup decoding / prefix reuse come from the engine.
+
+**Download with `hf download` or let `onw` fetch it** — a `git clone` without Git LFS gets pointer files, which
+`onw` detects and reports.
+
+## Files
+
+| file | what |
+|---|---|
+| `seg00_S{1,16,64}.xml` + `seg00.bin` | the static Gemma 4 decoder (head_dim-512 attention split into 256-wide heads, host-built masks) for 1 / 16 / 64-token blocks |
+| `seg01_S*.xml` | LM head + logit softcapping (shared INT8 weight) |
+| `vision.xml` | static vision encoder (280 soft tokens) |
+| `shared.bin` | INT8 token embedding (= tied head), INT4 per-layer embedding table + id map |
+| `engine.json`, tokenizer / processor files | metadata for onw |
+
+## License
+
+Apache 2.0, same as the base model. The weights are re-quantized / restructured from it.

engine.json

@@ -105,7 +105,8 @@
    "router_layer": null,
    "S": 1,
    "slots": 0,
-   "head": false
+   "head": false,
+   "bin": "seg00.bin"
   },
   "seg01_S1.xml": {
    "index": 1,
@@ -113,7 +114,8 @@
    "router_layer": null,
    "S": 1,
    "slots": 0,
-   "head": true
+   "head": true,
+   "bin": "seg01.bin"
   },
   "seg00_S16.xml": {
    "index": 0,
@@ -121,7 +123,8 @@
    "router_layer": null,
    "S": 16,
    "slots": 0,
-   "head": false
+   "head": false,
+   "bin": "seg00.bin"
   },
   "seg01_S16.xml": {
    "index": 1,
@@ -129,7 +132,8 @@
    "router_layer": null,
    "S": 16,
    "slots": 0,
-   "head": true
+   "head": true,
+   "bin": "seg01.bin"
   },
   "seg00_S64.xml": {
    "index": 0,
@@ -137,7 +141,8 @@
    "router_layer": null,
    "S": 64,
    "slots": 0,
-   "head": false
+   "head": false,
+   "bin": "seg00.bin"
   },
   "seg01_S64.xml": {
    "index": 1,
@@ -145,7 +150,8 @@
    "router_layer": null,
    "S": 64,
    "slots": 0,
-   "head": true
+   "head": true,
+   "bin": "seg01.bin"
   }
  },
  "masks": {
@@ -193,5 +199,11 @@
   "NPU_USE_NPUW": "YES",
   "NPUW_DEVICES": "NPU",
   "NPUW_FOLD": "YES"
+ },
+ "tables": {
+  "ple": {
+   "bits": 4,
+   "group": 32
+  }
  }
 }

seg00_S16.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:38b18a93e134fd55cdcd54772c26b4d966a371c6763eb8ab020e0ce91f87a971
-size 2050670280

seg00_S64.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:38b18a93e134fd55cdcd54772c26b4d966a371c6763eb8ab020e0ce91f87a971
-size 2050670280

seg01_S16.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5e48af94438b6e5de715ec5e3e80e252dc3f66d9ea35b3bdb8dac0650fa8ad71
-size 6

seg01_S64.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:5e48af94438b6e5de715ec5e3e80e252dc3f66d9ea35b3bdb8dac0650fa8ad71
-size 6

shared.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:402bac867a19a87a5b21800b3a093bc56306b49e52a7bd5a3a9cdf471a7175c6
-size 3492282368
+oid sha256:9800fc61111c52fff3c69497b1cff3e2623af746ae55a38c60733580c57c3e3a
+size 2258632704

shared.bin.json

@@ -1 +1 @@
-{"tensors": {"embed_q": {"at": [0, 671088640], "type": "i8", "shape": [262144, 2560]}, "embed_s": {"at": [671088640, 524288], "type": "f16", "shape": [262144]}, "head_q": {"alias": "embed_q"}, "head_s": {"at": [671612928, 524288], "type": "f16", "shape": [262144]}, "ple_q": {"at": [672137216, 2818572288], "type": "i8", "shape": [262144, 10752]}, "ple_s": {"at": [3490709504, 524288], "type": "f16", "shape": [262144]}, "ple_map": {"at": [3491233792, 1048576], "type": "i32", "shape": [262144]}}}
+{"tensors": {"ple_q4": {"at": [0, 1409286144], "type": "u8", "shape": [262144, 5376]}, "ple_s4": {"at": [1409286144, 176160768], "type": "f16", "shape": [262144, 336]}, "embed_q": {"at": [1585446912, 671088640], "type": "i8", "shape": [262144, 2560]}, "embed_s": {"at": [2256535552, 524288], "type": "f16", "shape": [262144]}, "head_q": {"alias": "embed_q"}, "head_s": {"at": [2257059840, 524288], "type": "f16", "shape": [262144]}, "ple_map": {"at": [2257584128, 1048576], "type": "i32", "shape": [262144]}}}