README.md

---
library_name: transformers
license: gemma
base_model:
  - google/gemma-4-E4B-it
pipeline_tag: image-text-to-text
tags:
  - gemma4
  - multimodal
  - vision
  - video
  - audio
  - axera
  - ax650
  - npu
  - quantized
  - gptq
  - 4-bit
  - W4A16
  - long-context
language:
  - en
  - multilingual
---

<p align="center">
  <img src="assets/gemma4_axera_banner.jpg" alt="Gemma4-Axera Banner">
</p>

# Gemma 4 E4B GPTQ-INT4 8K-Context on AXERA NPU

Ready-to-run deployment package for the GPTQ INT4 (W4A16) conversion of
[`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it) on AX650 / NPU3,
compiled for an extended 8K-token context.

- Runtime: packaged `axllm` OpenAI-compatible server
- Target: AX650 / AX650N, AArch64
- Core context layout: 256-token prefill chunk, 6400-token max prefill, 8191-token decode context
- Packaged modalities: text, single image, video, fixed-duration audio
- Included runtime assets: `bin/axllm`, 43 text axmodel files, 3 vision axmodel files,
  2 audio axmodel files, tokenizer, embedding weights, sample assets

The text runtime extends the context window to 8191 tokens with up to 6400 tokens of prefill
capacity, using 25 warm-prefill history buckets (256 to 6144 in 256-token steps). The model has 42
transformer layers with sliding/full attention (512-token sliding window, full attention every 6th
layer) and 18 KV-shared layers.

## Supported Platform

- AX650 / AX650N
- NPU3 runtime environment
- 10 GiB CMM configuration required

## Compiled Runtime Profile

| Setting | Value |
| --- | ---: |
| Transformer layers | 42 |
| Prefill graph length | 256 tokens |
| Warm-prefill history buckets | 256 to 6144 tokens, in 256-token steps |
| Number of prefill groups | 25 |
| Maximum compiled prefill capacity | 6400 tokens |
| Maximum KV-cache length | 8191 tokens |
| Sliding-attention window | 512 tokens |
| Full-attention layers | 7 (every 6th layer) |
| KV-shared layers | 18 |
| Text quantization | W4A16 GPTQ (AutoRound), group size 128 |
| Post stage | INT8 (s8) weights |
| Vision / audio encoder quantization | U16 activations, FP32 conv bias |
| Hidden-state interface | BF16 |
| Validated server concurrency | 1 request |

## Text Performance

Measured on AX650 with the packaged runtime, one request at a time, greedy decoding
(`temperature=0`). TTFT means time to first token.

| Prompt tokens | Prefill chunks | TTFT (s) | Decode (tok/s) |
| ---: | ---: | ---: | ---: |
| 497 | 2 | 4.2 | 3.3 |
| 1009 | 4 | 8.6 | 3.3 |
| 2033 | 8 | 14.9 | 3.3 |
| 4081 | 16 | 37.1 | 3.0 |
| 5105 | 20 | 53.6 | 2.9 |
| 6129 | 24 | 70.7 | 3.1 |

Decode holds at roughly 2.9 to 3.4 tok/s and does not degrade as the KV cache fills.

TTFT also depends on the prompt's content, not only its length: repeated wording is cheaper than
highly varied text, and follow-up requests that reuse earlier wording are cheaper again. The table
above is the slower end of that range. Image, video and audio tokens always pay full price, so a
30 s audio clip adds 750 tokens at full cost.

## Runtime Footprint

CMM and Linux OS memory are separate pools on AX650. The values below were measured with the
packaged server fully loaded on an otherwise idle board.

| Item | Measured value |
| --- | ---: |
| CMM used, text runtime only | 6955 MiB |
| CMM used, text + `t70` vision encoder | 7126 MiB |
| CMM used, text + `t70` vision + both audio encoders (packaged default) | 7778 MiB |
| CMM remaining at the packaged default | 2461 MiB |
| OS RSS after init | 946 MiB |
| OS RSS at peak (during 6k-token request) | 1132 MiB |

CMM does not grow during inference. The KV cache is pre-allocated at initialization for the full
8191-token context. Weights are memory-mapped (`mmap`) into page cache rather than anonymous
memory, so the OS resident cost is small.

Only one vision encoder is loaded at a time, selected by `filename_image_encoder_axmodel` in
`config.json`; both audio encoders are loaded when both files are present. Switching the vision
profile to `t140` or `t280` adds roughly 19 MiB or 126 MiB over the `t70` default. This package
cannot be loaded alongside another multi-GB model on a 10 GiB CMM device.

## Vision Encoder

Three fixed-resolution vision encoders are packaged. Each is a fixed shape: the resolution and the
soft-token count are compiled in and are not user-configurable at runtime. Soft tokens are the
number of image embeddings inserted into the text sequence, equal to
`(height / patch_size) * (width / patch_size) / 9` with `patch_size = 16` and 3x3 pooling.

| Model | Resolution | Soft tokens | Encoder latency | Encoder CMM |
| --- | --- | ---: | ---: | ---: |
| `gemma4_vision_h336_w480_t70.axmodel` | 336x480 | 70 | 87.4 ms | 180.9 MiB |
| `gemma4_vision_h480_w672_t140.axmodel` | 480x672 | 140 | 256.6 ms | 199.4 MiB |
| `gemma4_vision_h672_w960_t280.axmodel` | 672x960 | 280 | 744.3 ms | 306.4 MiB |

Latency is a single-model average from `ax_run_model -w 1 -r 5` on AX650, excluding text prefill
and decode. The package default is the `t70` profile. To switch profiles, edit `config.json`:

```json
{
  "filename_image_encoder_axmodel": "gemma4_vision_h480_w672_t140.axmodel",
  "vision_width": 672,
  "vision_height": 480
}
```

Change the resolution pair together with the encoder file. A mismatch makes the runtime preprocess
images to a shape the encoder was not compiled for.

The runtime caches image embeddings under `vision_cache/` keyed by the media path. Delete that
directory after switching profiles, otherwise an entry produced by the previous encoder is reused.

Both the vision and the audio encoders are converted from the unquantized upstream weights, because
the source INT4 checkpoint leaves both encoder towers in full precision.

Higher profiles resolve finer detail at a higher latency cost. Use `t70` for the fastest response
and `t140` or `t280` when small features such as text in the image matter.

## Audio Encoder

Two fixed-duration audio encoders are packaged. The duration is compiled in, and `axllm` selects
the profile from the clip length. Audio longer than 30 s is truncated to the 30 s window.

| Model | Audio duration | Mel frames | Audio tokens | Encoder latency | Encoder CMM |
| --- | --- | ---: | ---: | ---: | ---: |
| `gemma4_audio_5s.axmodel` | 5 s | 499 | 125 | 29.5 ms | 335.7 MiB |
| `gemma4_audio_30s.axmodel` | 30 s | 2999 | 750 | 173.1 ms | 360.5 MiB |

Latency is a single-model average from `ax_run_model -w 1 -r 5` on AX650, excluding text prefill
and decode. Both encoders are loaded at startup when both files are present.

### Audio Input Format

Gemma 4's audio front end is defined at **16 kHz mono** (`sampling_rate: 16000` in
`gemma_4_e4b_it_tokenizer/processor_config.json`). Supply audio at that rate and the runtime feeds it straight to the mel
front end. Any other rate is resampled on the CPU first, and that resampling — not the NPU encoder —
dominates the time before the first token. Measured on AX650 for a 30 s clip:

| Input | Feature extraction (read + resample + mel) | NPU encode |
| --- | ---: | ---: |
| 16 kHz mono (native) | 0.94 s | 0.18 s |
| 48 kHz stereo | 2.6 s | 0.18 s |
| 44.1 kHz stereo | 3.0 s | 0.18 s |

Convert once before sending, for example:

```bash
ffmpeg -i input.wav -ar 16000 -ac 1 -c:a pcm_s16le input_16k.wav
```

The packaged sample clips are already 16 kHz mono.

**WAV only.** The audio path accepts uncompressed `RIFF/WAVE` PCM. Compressed containers are
rejected with `unsupported wav container (expect RIFF/WAVE)`, even when `ffmpeg` is installed on the
board, so decode `mp3 / flac / m4a / ogg` to 16 kHz mono WAV first with the command above.

A 30 s clip's 750 audio tokens span several prefill chunks, so attention within the audio block is
not fully bidirectional across chunk boundaries.

## Video

Video is handled by the same vision encoder, one frame at a time, using the frame count in
`gemma_4_e4b_it_tokenizer/processor_config.json` (`num_frames: 32`). At the default `t70` profile that is
32 x 70 = 2240 soft tokens per clip, so a video request costs about 2360 input tokens including the
text part and uses 10 prefill chunks.

Only the picture is used. A video's audio track is ignored; audio is a separate input, so send it as
its own `audio_url` part if you need both.

## Package Layout

```text
.
├── README.md
├── bin/axllm
├── serve.sh
├── config.json
├── post_config.json
├── gemma4_tokenizer.txt
├── assets/
│   ├── gemma4_axera_banner.jpg
│   ├── sample.png
│   ├── red-panda-openai.mp4
│   ├── gemma4_audio_test_5s.wav
│   ├── gemma4_audio_test_chunk0_30s.wav
│   └── gemma4_audio_test_chunk1_30s.wav
├── gemma4_text_p256_l0_together.axmodel ... gemma4_text_p256_l41_together.axmodel
├── gemma4_text_post.axmodel
├── gemma4_vision_h336_w480_t70.axmodel
├── gemma4_vision_h480_w672_t140.axmodel
├── gemma4_vision_h672_w960_t280.axmodel
├── gemma4_audio_5s.axmodel
├── gemma4_audio_30s.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── model.embed_tokens_per_layer.weight.npy
├── model.per_layer_model_projection.weight.npy
├── model.per_layer_projection_norm.weight.npy
└── gemma_4_e4b_it_tokenizer/
    ├── config.json
    ├── tokenizer.json
    ├── tokenizer_config.json
    ├── generation_config.json
    ├── processor_config.json
    └── chat_template.jinja
```

The root directory is the direct `axllm` runtime layout: all axmodel files, the embedding weights,
and the tokenizer text file live at the root. The tokenizer subdirectory holds the Hugging Face
tokenizer and model metadata without source weight shards.

## Sample Assets

The package ships the media used in the examples below.

`assets/sample.png`

![sample](assets/sample.png)

Three WAV clips of English speech at 16 kHz mono, one per audio profile length:

- `assets/gemma4_audio_test_5s.wav` (5.00 s)
- `assets/gemma4_audio_test_chunk0_30s.wav` (30.00 s)
- `assets/gemma4_audio_test_chunk1_30s.wav` (30.00 s)

One video clip for the video example:

- `assets/red-panda-openai.mp4` (30.1 s, 360x640)

## Download

```bash
mkdir -p AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K
cd AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K
hf download AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K --local-dir .
```

## Packaged Runtime

The package ships a prebuilt AArch64 `bin/axllm`, so no separate install step is needed, and the
figures in this README refer to that binary. Replacing it with a different build of
[`AXERA-TECH/ax-llm`](https://github.com/AXERA-TECH/ax-llm) may change the TTFT figures.

## Run on the Board

From the package root:

```bash
chmod +x ./bin/axllm ./serve.sh
./serve.sh --port 8000
```

Or equivalently:

```bash
chmod +x ./bin/axllm
./bin/axllm serve . --port 8000
```

Check service health and the loaded model:

```bash
curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models
```

The health endpoint returns `"status": "healthy"`, and the model list contains
`AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K`.

**Server timeout:** the default `--server_timeout_ms` is 300 seconds, which covers a 6000-token
prompt (about 71 s of TTFT) plus a few hundred output tokens. A long prompt combined with a long
generation can still exceed it, since decode runs at roughly 3 tok/s. Raise it when you need both,
and raise the client-side HTTP timeout to match:

```bash
./serve.sh --port 8000 --server_timeout_ms 600000
```

## Text Request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 64,
    "temperature": 0,
    "stream": false
  }'
```

Measured response:

```text
The capital of France is **Paris**.
```

## Image Request

Pass the image as a board-side file path or as a base64 `data:` URI.

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "./assets/sample.png"}},
        {"type": "text", "text": "What animal is in this image and what colour is it? Answer in one sentence."}
      ]}
    ],
    "max_tokens": 96,
    "temperature": 0,
    "stream": false
  }'
```

Measured response with the default `t70` profile:

```text
The animal in the image is a red lobster.
```

One image per message is supported. Mixing image and audio in the same message is not supported.

## Audio Request

Pass the audio as a board-side file path or as a base64 `data:` URI. The clip length selects the
encoder profile.

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": "./assets/gemma4_audio_test_5s.wav"}},
        {"type": "text", "text": "Transcribe the speech in this audio in its original language. Output only the transcription."}
      ]}
    ],
    "max_tokens": 128,
    "temperature": 0,
    "stream": false
  }'
```

Measured response for `assets/gemma4_audio_test_5s.wav` (5 s profile, 125 audio tokens):

```text
When I was seventeen, I read a quote that went something like, if you
```

Measured response for `assets/gemma4_audio_test_chunk0_30s.wav` (30 s profile, 750 audio tokens):

```text
No one wants to die. Even people who want to go to heaven don't want to die to get there. And yet,
death is the destination we all share. No one has ever escaped it, and that is as it should be
because death is very likely the single best invention of life. It's life's change agent. It clears
out the old to make way for the new. Right now, the new is you. But someday, not too long from now,
you will gradually become
```

End to end on AX650 with the packaged clips this takes about 9 s for the 5 s clip and 34 to 44 s for
a 30 s clip, almost all of it text decode rather than audio processing. Both transcriptions above
stop at the `max_tokens` limit, not at a sentence boundary.

One audio file per message is supported. Mixing audio and image in the same message is not
supported.

## Video Request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": [
        {"type": "video_url", "video_url": {"url": "./assets/red-panda-openai.mp4"}},
        {"type": "text", "text": "What animal is in this video? Answer with the animal name only."}
      ]}
    ],
    "max_tokens": 48,
    "temperature": 0,
    "stream": false
  }'
```

Measured response for the packaged 30 s clip at the default `t70` profile:

```text
Red Panda
```

A video request of this length costs about 2360 input tokens and takes roughly 40 s end to end.

## Multi-turn Conversation

Send the whole message list each turn, as an OpenAI client normally does. The runtime keeps the KV
cache from the previous turns and only processes the new tokens, so follow-up turns are far cheaper
than the first. With a 3223-token document in the opening message, the first turn takes about 22 s to
the first token and each short follow-up about 3 s, with the earlier turns still available to the
model.

## Context Limits

| Limit | Value | Behaviour |
| --- | ---: | --- |
| Input tokens per request | 6400 | Above it the server returns HTTP 400 with `input_num_token(N) > prefill_max_token_num(6400)` |
| Total context (prompt + generation + history) | 8191 | KV cache is pre-allocated for this at startup |

A request over the 6400-token cap is rejected outright rather than silently truncated, so keep
prompts within the cap.

## Transcription Endpoint

The OpenAI-compatible `/v1/audio/transcriptions` and `/v1/audio/translations` endpoints are also
available. They take a multipart upload, so the audio file does not have to exist on the board, and
they work with any OpenAI-compatible client or web UI without extra parameters.

```bash
curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F model=AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K \
  -F file=@./assets/gemma4_audio_test_5s.wav
```
Measured response:

```json
{
  "text": "When I was seventeen, I read a quote that went something like, if you"
}
```

The 30 s clips select the 750-token encoder through the same endpoint. Measured response for
`assets/gemma4_audio_test_chunk0_30s.wav`:

```json
{
  "text": "No one wants to die. Even people who want to go to heaven don't want to die to get there. And yet, death is the destination we all share. No one has ever escaped it, and that is as it should be because death is very likely the single best invention of life. It's life's change agent. It clears out the old to make way for the new. Right now, the new is you. But someday, not too long from now, you will gradually become"
}
```

`response_format` accepts `json` (default), `verbose_json`, `text`, `srt` and `vtt`. The response is
always wrapped as `{"text": ...}`; `srt` and `vtt` put the same text in a single subtitle cue
spanning the whole clip, so there is no word-level or segment-level timing.

`/v1/audio/translations` follows the OpenAI definition and translates into English. With the packaged
English clips its output is the same as the transcription.

## Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

- Original Hugging Face model: [`google/gemma-4-E4B-it`](https://huggingface.co/google/gemma-4-E4B-it)
- Source INT4 checkpoint for the text weights: [`Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ`](https://huggingface.co/Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ)
- AXERA conversion and deployment workflow: [`AXERA-TECH/gemma-4-E2B-it.axera`](https://github.com/AXERA-TECH/gemma-4-E2B-it.axera)

## Discussion

- [GitHub Issues](https://github.com/AXERA-TECH/ax-llm/issues)
- QQ group: `139953715`