gemma-4-E4B-it for onw — entirely on the Intel NPU (text + image)

日本語 | English

google/gemma-4-E4B-it converted for onw (俺のNPUがこんなに動くわけない - "there's no way my NPU runs this well"), the engine that runs LLMs entirely on the Intel NPU. This repo holds only the model; the engine is a separate download.

Text + image. The same validated NPU graphs as the standalone ryugyosoft/gemma-4-E4B-it-npu, run by the shared engine.

To our knowledge, onw is the first public implementation that runs all of Gemma 4 - vision encoder and per-layer embeddings included - on the NPU (survey by Fable 5.1, September 2026; Intel's OpenVINO GenAI runs the vision encoder on the CPU and the other public ports are text-only).

NPU 3720 (Core Ultra 9 285HX)
input	text + image
decode	7.2 tok/s (prompt lookup on, same output as plain greedy)
prompt processing	image + question, 284 tokens: vision 1.9 s + prefill 2.0 s (64-token blocks, >= 24 GB RAM)
download	4.3 GB
memory in use (working set after loading)	~6 GB (8.5 GB while reading an image)
first start (NPU compile) / later	~4 min / ~20 s

Supported PCs: Intel Core Ultra with an NPU (series 1 / 2: Meteor Lake, Arrow Lake, Lunar Lake; series 3: Panther Lake), Windows 11 or Ubuntu 22.04+. The numbers above are measured on the test machine (NPU 3720) and vary with the NPU and memory.

Use

Install onw (the engine) if you have not yet - one line, see Install in onw. On Windows open "PowerShell" from the Start menu, paste this and press Enter (Ubuntu: the curl ... | bash line in the onw README).
```
irm https://huggingface.co/ryugyosoft/onw/resolve/main/install.ps1 | iex
```
In the onw window (from the onw icon in the task tray), Models tab: pick gemma-4-E4B-it and press Download (4.3 GB).
Server tab: pick it and press Load. The first load prepares it for the NPU (~4 min).
"Chat" to try it. Other apps can use it as an OpenAI-compatible API (base URL http://localhost:8000/v1).

Server tab Settings tab

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
r = c.chat.completions.create(model="gemma-4-E4B-it-onw", messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)

Advanced: hf download ryugyosoft/gemma-4-E4B-it-onw and onw serve <folder> works too. Do not git clone without Git LFS (you would get pointer files instead of the weights; onw detects that and stops).

Versions

v2 (current): decoder weights stored once for the 1 / 16 / 64-token graphs, INT4 per-layer embedding table (9.2 → 4.3 GB; memory 16.1 → 14.8 GB). Same answers as v1 on our checks. Needs onw 0.3 or newer.
v1: hf download ryugyosoft/gemma-4-E4B-it-onw --revision v1 --local-dir ...

Most of the download saving comes from storing identical weights once; in memory the NPU keeps one compiled copy per block size, so the footprint shrinks less.

How it runs

The static Gemma 4 decoder (head_dim-512 attention split into 256-wide heads, host-built masks) is imported as is.
The token embedding and the 2.7 GB per-layer embedding table are looked up on the host (table reads: no NPU memory, no worker process).
The LM head is one shared INT8 input for all block sizes; the logit softcap is arranged to avoid an NPU compiler bug.
Details: onw technical notes.

Files

file	what
`seg00_S{1,16,64}.xml` + `seg00.bin`	static decoder for 1 / 16 / 64-token blocks (shared weights)
`seg01_S*.xml` + `seg01.bin`	LM head + logit softcapping
`vision.xml`	static vision encoder (280 soft tokens)
`shared.bin`	INT8 token embedding (= tied head), INT4 per-layer embedding table + id map
`engine.json`, tokenizer / config files	onw metadata, the original repo's configs

Quality

Channel-wise / group-wise INT4 (round-to-nearest): on our checks the first token matches the bf16 model; answers are fluent, word choices may diverge after a few tokens.

License

Apache 2.0, same as the base model. The weights are re-quantized / restructured from it.