# gemma-4-E4B-it for onw — entirely on the Intel NPU (text + image)

[日本語](README.md) | **English**

[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) converted for **[onw](https://huggingface.co/ryugyosoft/onw)** (俺のNPUがこんなに動くわけない - "there's no way my NPU runs this well"), the engine that runs LLMs entirely on the Intel NPU. This repo holds only the model; the engine is a separate download.

Text + image. The same validated NPU graphs as the standalone [ryugyosoft/gemma-4-E4B-it-npu](https://huggingface.co/ryugyosoft/gemma-4-E4B-it-npu), run by the shared engine.

**To our knowledge, onw is the first public implementation that runs all of Gemma 4 - vision encoder and per-layer embeddings included - on the NPU** (survey by Fable 5.1, September 2026; Intel's OpenVINO GenAI runs the vision encoder on the CPU and the other public ports are text-only).

| NPU 3720 (Core Ultra 9 285HX) | |
|---|---|
| input | text + image |
| decode | **7.2 tok/s (prompt lookup on, same output as plain greedy)** |
| prompt processing | image + question, 284 tokens: vision 1.9 s + prefill 2.0 s (64-token blocks, >= 24 GB RAM) |
| download | **4.3 GB** |
| memory in use (working set after loading) | ~6 GB (8.5 GB while reading an image) |
| first start (NPU compile) / later | ~4 min / ~20 s |

**Supported PCs**: Intel Core Ultra with an NPU (series 1 / 2: Meteor Lake, Arrow Lake, Lunar Lake; series 3: Panther Lake), Windows 11 or Ubuntu 22.04+. The numbers above are measured on the test machine (NPU 3720) and vary with the NPU and memory.

## Use

1. **Install onw (the engine)** if you have not yet - one line, see [Install in onw](https://huggingface.co/ryugyosoft/onw). On Windows open "PowerShell" from the Start menu, paste this and press Enter (Ubuntu: the `curl ... | bash` line in the onw README).

   ```powershell
   irm https://huggingface.co/ryugyosoft/onw/resolve/main/install.ps1 | iex
   ```

2. In the **onw window** (from the onw icon in the task tray), Models tab: pick **gemma-4-E4B-it** and press Download (4.3 GB).
3. Server tab: pick it and press Load. The first load prepares it for the NPU (~4 min).
4. "Chat" to try it. Other apps can use it as an **OpenAI-compatible API** (base URL `http://localhost:8000/v1`).

<img src="https://huggingface.co/ryugyosoft/onw/resolve/main/images/onw-models.png" width="640" alt="The Models tab of the onw window">

<img src="https://huggingface.co/ryugyosoft/onw/resolve/main/images/onw-server.png" width="49%" alt="Server tab"> <img src="https://huggingface.co/ryugyosoft/onw/resolve/main/images/onw-settings.png" width="49%" alt="Settings tab">

```python
from openai import OpenAI
c = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
r = c.chat.completions.create(model="gemma-4-E4B-it-onw", messages=[{"role": "user", "content": "Hello"}])
print(r.choices[0].message.content)
```

Advanced: `hf download ryugyosoft/gemma-4-E4B-it-onw` and `onw serve <folder>` works too. Do not `git clone` without Git LFS (you would get pointer files instead of the weights; onw detects that and stops).

## Versions

- **v2 (current)**: decoder weights stored once for the 1 / 16 / 64-token graphs, INT4 per-layer embedding table (9.2 → 4.3 GB; memory 16.1 → 14.8 GB). Same answers as v1 on our checks. Needs onw 0.3 or newer.
- v1: `hf download ryugyosoft/gemma-4-E4B-it-onw --revision v1 --local-dir ...`

Most of the download saving comes from storing identical weights once; in memory the NPU keeps one compiled copy per block size, so the footprint shrinks less.

## How it runs

- The static Gemma 4 decoder (head_dim-512 attention split into 256-wide heads, host-built masks) is imported as is.
- The token embedding and the 2.7 GB per-layer embedding table are looked up on the host (table reads: no NPU memory, no worker process).
- The LM head is one shared INT8 input for all block sizes; the logit softcap is arranged to avoid an NPU compiler bug.
- Details: [onw technical notes](https://huggingface.co/ryugyosoft/onw/blob/main/TECHNICAL_en.md).

## Files

| file | what |
|---|---|
| `seg00_S{1,16,64}.xml` + `seg00.bin` | static decoder for 1 / 16 / 64-token blocks (shared weights) |
| `seg01_S*.xml` + `seg01.bin` | LM head + logit softcapping |
| `vision.xml` | static vision encoder (280 soft tokens) |
| `shared.bin` | INT8 token embedding (= tied head), INT4 per-layer embedding table + id map |
| `engine.json`, tokenizer / config files | onw metadata, the original repo's configs |

## Quality

Channel-wise / group-wise INT4 (round-to-nearest): on our checks the first token matches the bf16 model; answers are fluent, word choices may diverge after a few tokens.

## License

Apache 2.0, same as the base model. The weights are re-quantized / restructured from it.