---
license: gemma
base_model: google/gemma-4-E2B-it
pipeline_tag: image-text-to-text
tags:
  - onnx
  - onnxruntime-genai
  - openvino
  - mobius
  - int4
  - multimodal
---

# gemma-4-E2B-it — ONNX / OpenVINO

ONNX and OpenVINO exports of
[google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it), built with
[mobius](https://github.com/onnxruntime/mobius) and packaged for
[ONNX Runtime GenAI](https://github.com/microsoft/onnxruntime-genai). INT4
variants are quantized with [Olive](https://github.com/microsoft/Olive).

Gemma 4 E2B is an any-to-any multimodal model, so **every** variant is a
multi-component package with four ONNX models — `decoder`, `embedding`,
`vision_encoder`, `audio_encoder` — plus a shared `genai_config.json`,
tokenizer, and image / audio processor configs.

## Variants

Each variant folder is `<precision>/<target>` (or a top-level folder for the
OpenVINO packages). Pick the **precision** for your quality/size budget and the
**target** for your runtime/EP.

### Precisions

| Precision | Folder prefix | Notes |
|---|---|---|
| FP16 | `f16/` | Full-precision half. |
| BF16 | `bf16/` | BFloat16 (no INT4). |
| INT4 · K-Quant (Q4_K_M) | `Q4_K_M/` | Olive K-Quant, block 32. Recommended INT4. |
| INT4 · NF4 | `NF4/` | NF4 4-bit weights. |
| INT4 · OpenVINO | `openvino-*/` | K-Quant + `MatMulNBitsToQDQ`, compiled for OpenVINO. |

### Targets (execution-provider builds)

| Target (subfolder) | Runtime / EP | Provider in `genai_config` |
|---|---|---|
| `cuda` | ONNX Runtime GenAI + CUDA | `cuda` (fused GroupQueryAttention) |
| `default` | ONNX Runtime GenAI (CPU / DML) | empty → portable ONNX |
| `onnx-standard` | any conformant ONNX runtime | empty → pure `ai.onnx`, no `com.microsoft` ops |
| `webgpu` | ONNX Runtime Web / WebGPU | `webgpu` |

### Full matrix (present in this repo)

| Folder | Precision | Target | Size |
|---|---|---|---|
| `f16/cuda` | FP16 | CUDA | ~11.3 GB |
| `f16/default` | FP16 | CPU / DML | ~11.3 GB |
| `f16/onnx-standard` | FP16 | portable ONNX | ~11.3 GB |
| `f16/webgpu` | FP16 | WebGPU | ~11.3 GB |
| `bf16/cuda` | BF16 | CUDA | ~11.3 GB |
| `bf16/default` | BF16 | CPU / DML | ~11.3 GB |
| `bf16/onnx-standard` | BF16 | portable ONNX | ~11.3 GB |
| `Q4_K_M/cuda` | INT4 (K-Quant) | CUDA | ~8.1 GB |
| `Q4_K_M/default` | INT4 (K-Quant) | CPU / DML | ~8.1 GB |
| `NF4/cuda` | INT4 (NF4) | CUDA | ~7.9 GB |
| `NF4/default` | INT4 (NF4) | CPU / DML | ~7.9 GB |
| `openvino-epcontext/` | INT4 | OpenVINO EP (ORT) | ~7.4 GB |
| `openvino-ir/` | INT4 | OpenVINO runtime | ~7.4 GB |

## Usage — ONNX Runtime GenAI (the `f16` / `bf16` / `Q4_K_M` / `NF4` variants)

```bash
pip install onnxruntime-genai        # CPU / WebGPU / DML
pip install onnxruntime-genai-cuda   # for the */cuda variants
```

```python
import onnxruntime_genai as og
model = og.Model("Q4_K_M/cuda")   # or f16/default, NF4/cuda, ...
# ... standard ORT GenAI generation loop ...
```

## OpenVINO

Both OpenVINO packages are INT4 and require **OpenVINO ≥ 2026.3** (the release
that ships the opset-24 `RMSNormalization` / `RotaryEmbedding` ONNX-frontend
translators, [openvinotoolkit/openvino#35545](https://github.com/openvinotoolkit/openvino/pull/35545)).

### `openvino-epcontext/` — compiled graph for ONNX Runtime ✅ verified

Each component's `model.onnx` is a single `com.microsoft::EPContext` node
(`source="OpenVINOExecutionProvider"`) that points at the compiled OpenVINO IR
(`model.xml` / `model.bin`) next to it, so ONNX Runtime's OpenVINO EP loads the
precompiled graph instead of recompiling.

**Verified end-to-end**: the decoder loads via the OpenVINO EP and a forward
pass returns `logits (1, seq, 262144)` fp16 (all finite), using ONNX Runtime
1.28 built against OpenVINO 2026.3.

```python
import onnxruntime as ort
sess = ort.InferenceSession(
    "openvino-epcontext/decoder/model.onnx",
    providers=[("OpenVINOExecutionProvider", {"device_type": "NPU"})],  # or GPU / CPU
)
```

### `openvino-ir/` — OpenVINO IR

Native OpenVINO IR (`openvino_model.xml` / `.bin`) per component:

```python
import openvino as ov
decoder = ov.Core().compile_model("openvino-ir/decoder/openvino_model.xml", "NPU")
```

### `runtime-wheels/`

The exact wheels used to verify the OpenVINO packages (Linux, cp312) — OpenVINO
2026.3 + ONNX Runtime 1.28 with the OpenVINO EP — since no official release
ships OpenVINO ≥ 2026.3 yet:

```bash
pip install runtime-wheels/openvino-2026.3.0-*.whl \
            runtime-wheels/onnxruntime_openvino-1.28.0-*.whl
```

## Build recipe

Built via [microsoft/olive-recipes `google-gemma-4-E2B-it`](https://github.com/microsoft/olive-recipes/tree/main/google-gemma-4-E2B-it):
`MobiusBuilder` for the FP16 / BF16 exports; `MobiusBuilder` →
`OnnxKQuantQuantization` for INT4; the OpenVINO packages additionally run
`MatMulNBitsToQDQ` → `OpenVINOConversion` → `OpenVINOEncapsulation`.

## License

Inherits the [Gemma license](https://ai.google.dev/gemma/terms).