AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K

Gemma 4 E4B GPTQ-INT4 8K-Context on AXERA NPU

Ready-to-run deployment package for the GPTQ INT4 (W4A16) conversion of google/gemma-4-E4B-it on AX650 / NPU3, compiled for an extended 8K-token context.

• Runtime: packaged axllm OpenAI-compatible server
• Target: AX650 / AX650N, AArch64
• Core context layout: 256-token prefill chunk, 6400-token max prefill, 8191-token decode context
• Packaged modalities: text, single image, video, fixed-duration audio
• Included runtime assets: bin/axllm, 43 text axmodel files, 3 vision axmodel files, 2 audio axmodel files, tokenizer, embedding weights, sample assets

The text runtime extends the context window to 8191 tokens with up to 6400 tokens of prefill capacity, using 25 warm-prefill history buckets (256 to 6144 in 256-token steps). The model has 42 transformer layers with sliding/full attention (512-token sliding window, full attention every 6th layer) and 18 KV-shared layers.

Supported Platform

• AX650 / AX650N
• NPU3 runtime environment
• 10 GiB CMM configuration required

Compiled Runtime Profile

Setting	Value
Transformer layers	42
Prefill graph length	256 tokens
Warm-prefill history buckets	256 to 6144 tokens, in 256-token steps
Number of prefill groups	25
Maximum compiled prefill capacity	6400 tokens
Maximum KV-cache length	8191 tokens
Sliding-attention window	512 tokens
Full-attention layers	7 (every 6th layer)
KV-shared layers	18
Text quantization	W4A16 GPTQ (AutoRound), group size 128
Post stage	INT8 (s8) weights
Vision / audio encoder quantization	U16 activations, FP32 conv bias
Hidden-state interface	BF16
Validated server concurrency	1 request

Text Performance

Measured on AX650 with the packaged runtime, one request at a time, greedy decoding (temperature=0). TTFT means time to first token.

Prompt tokens	Prefill chunks	TTFT (s)	Decode (tok/s)
497	2	4.2	3.3
1009	4	8.6	3.3
2033	8	14.9	3.3
4081	16	37.1	3.0
5105	20	53.6	2.9
6129	24	70.7	3.1

Decode holds at roughly 2.9 to 3.4 tok/s and does not degrade as the KV cache fills.

TTFT also depends on the prompt's content, not only its length: repeated wording is cheaper than highly varied text, and follow-up requests that reuse earlier wording are cheaper again. The table above is the slower end of that range. Image, video and audio tokens always pay full price, so a 30 s audio clip adds 750 tokens at full cost.

Runtime Footprint

CMM and Linux OS memory are separate pools on AX650. The values below were measured with the packaged server fully loaded on an otherwise idle board.

Item	Measured value
CMM used, text runtime only	6955 MiB
CMM used, text + t70 vision encoder	7126 MiB
CMM used, text + t70 vision + both audio encoders (packaged default)	7778 MiB
CMM remaining at the packaged default	2461 MiB
OS RSS after init	946 MiB
OS RSS at peak (during 6k-token request)	1132 MiB

CMM does not grow during inference. The KV cache is pre-allocated at initialization for the full 8191-token context. Weights are memory-mapped (mmap) into page cache rather than anonymous memory, so the OS resident cost is small.

Only one vision encoder is loaded at a time, selected by filename_image_encoder_axmodel in config.json; both audio encoders are loaded when both files are present. Switching the vision profile to t140 or t280 adds roughly 19 MiB or 126 MiB over the t70 default. This package cannot be loaded alongside another multi-GB model on a 10 GiB CMM device.

Vision Encoder

Three fixed-resolution vision encoders are packaged. Each is a fixed shape: the resolution and the soft-token count are compiled in and are not user-configurable at runtime. Soft tokens are the number of image embeddings inserted into the text sequence, equal to (height / patch_size) * (width / patch_size) / 9 with patch_size = 16 and 3x3 pooling.

Model	Resolution	Soft tokens	Encoder latency	Encoder CMM
gemma4_vision_h336_w480_t70.axmodel	336x480	70	87.4 ms	180.9 MiB
gemma4_vision_h480_w672_t140.axmodel	480x672	140	256.6 ms	199.4 MiB
gemma4_vision_h672_w960_t280.axmodel	672x960	280	744.3 ms	306.4 MiB

Latency is a single-model average from ax_run_model -w 1 -r 5 on AX650, excluding text prefill and decode. The package default is the t70 profile. To switch profiles, edit config.json:

{
  "filename_image_encoder_axmodel": "gemma4_vision_h480_w672_t140.axmodel",
  "vision_width": 672,
  "vision_height": 480
}

Change the resolution pair together with the encoder file. A mismatch makes the runtime preprocess images to a shape the encoder was not compiled for.

The runtime caches image embeddings under vision_cache/ keyed by the media path. Delete that directory after switching profiles, otherwise an entry produced by the previous encoder is reused.

Both the vision and the audio encoders are converted from the unquantized upstream weights, because the source INT4 checkpoint leaves both encoder towers in full precision.

Higher profiles resolve finer detail at a higher latency cost. Use t70 for the fastest response and t140 or t280 when small features such as text in the image matter.

Audio Encoder

Two fixed-duration audio encoders are packaged. The duration is compiled in, and axllm selects the profile from the clip length. Audio longer than 30 s is truncated to the 30 s window.

Model	Audio duration	Mel frames	Audio tokens	Encoder latency	Encoder CMM
gemma4_audio_5s.axmodel	5 s	499	125	29.5 ms	335.7 MiB
gemma4_audio_30s.axmodel	30 s	2999	750	173.1 ms	360.5 MiB

Latency is a single-model average from ax_run_model -w 1 -r 5 on AX650, excluding text prefill and decode. Both encoders are loaded at startup when both files are present.

Audio Input Format

Gemma 4's audio front end is defined at 16 kHz mono (sampling_rate: 16000 in gemma_4_e4b_it_tokenizer/processor_config.json). Supply audio at that rate and the runtime feeds it straight to the mel front end. Any other rate is resampled on the CPU first, and that resampling — not the NPU encoder — dominates the time before the first token. Measured on AX650 for a 30 s clip:

Input	Feature extraction (read + resample + mel)	NPU encode
16 kHz mono (native)	0.94 s	0.18 s
48 kHz stereo	2.6 s	0.18 s
44.1 kHz stereo	3.0 s	0.18 s

Convert once before sending, for example:

ffmpeg -i input.wav -ar 16000 -ac 1 -c:a pcm_s16le input_16k.wav

The packaged sample clips are already 16 kHz mono.

WAV only. The audio path accepts uncompressed RIFF/WAVE PCM. Compressed containers are rejected with unsupported wav container (expect RIFF/WAVE), even when ffmpeg is installed on the board, so decode mp3 / flac / m4a / ogg to 16 kHz mono WAV first with the command above.

A 30 s clip's 750 audio tokens span several prefill chunks, so attention within the audio block is not fully bidirectional across chunk boundaries.

Video

Video is handled by the same vision encoder, one frame at a time, using the frame count in gemma_4_e4b_it_tokenizer/processor_config.json (num_frames: 32). At the default t70 profile that is 32 x 70 = 2240 soft tokens per clip, so a video request costs about 2360 input tokens including the text part and uses 10 prefill chunks.

Only the picture is used. A video's audio track is ignored; audio is a separate input, so send it as its own audio_url part if you need both.

Package Layout

.
├── README.md
├── bin/axllm
├── serve.sh
├── config.json
├── post_config.json
├── gemma4_tokenizer.txt
├── assets/
│   ├── gemma4_axera_banner.jpg
│   ├── sample.png
│   ├── red-panda-openai.mp4
│   ├── gemma4_audio_test_5s.wav
│   ├── gemma4_audio_test_chunk0_30s.wav
│   └── gemma4_audio_test_chunk1_30s.wav
├── gemma4_text_p256_l0_together.axmodel ... gemma4_text_p256_l41_together.axmodel
├── gemma4_text_post.axmodel
├── gemma4_vision_h336_w480_t70.axmodel
├── gemma4_vision_h480_w672_t140.axmodel
├── gemma4_vision_h672_w960_t280.axmodel
├── gemma4_audio_5s.axmodel
├── gemma4_audio_30s.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── model.embed_tokens_per_layer.weight.npy
├── model.per_layer_model_projection.weight.npy
├── model.per_layer_projection_norm.weight.npy
└── gemma_4_e4b_it_tokenizer/
    ├── config.json
    ├── tokenizer.json
    ├── tokenizer_config.json
    ├── generation_config.json
    ├── processor_config.json
    └── chat_template.jinja

The root directory is the direct axllm runtime layout: all axmodel files, the embedding weights, and the tokenizer text file live at the root. The tokenizer subdirectory holds the Hugging Face tokenizer and model metadata without source weight shards.

Sample Assets

The package ships the media used in the examples below.

assets/sample.png

Three WAV clips of English speech at 16 kHz mono, one per audio profile length:

• assets/gemma4_audio_test_5s.wav (5.00 s)
• assets/gemma4_audio_test_chunk0_30s.wav (30.00 s)
• assets/gemma4_audio_test_chunk1_30s.wav (30.00 s)

One video clip for the video example:

• assets/red-panda-openai.mp4 (30.1 s, 360x640)

Download

mkdir -p AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K
cd AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K
hf download AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K --local-dir .

Packaged Runtime

The package ships a prebuilt AArch64 bin/axllm, so no separate install step is needed, and the figures in this README refer to that binary. Replacing it with a different build of AXERA-TECH/ax-llm may change the TTFT figures.

Run on the Board

From the package root:

chmod +x ./bin/axllm ./serve.sh
./serve.sh --port 8000

Or equivalently:

chmod +x ./bin/axllm
./bin/axllm serve . --port 8000

Check service health and the loaded model:

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models

The health endpoint returns "status": "healthy", and the model list contains AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K.

Server timeout: the default --server_timeout_ms is 300 seconds, which covers a 6000-token prompt (about 71 s of TTFT) plus a few hundred output tokens. A long prompt combined with a long generation can still exceed it, since decode runs at roughly 3 tok/s. Raise it when you need both, and raise the client-side HTTP timeout to match:

./serve.sh --port 8000 --server_timeout_ms 600000

Text Request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 64,
    "temperature": 0,
    "stream": false
  }'

Measured response:

The capital of France is **Paris**.

Image Request

Pass the image as a board-side file path or as a base64 data: URI.

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "./assets/sample.png"}},
        {"type": "text", "text": "What animal is in this image and what colour is it? Answer in one sentence."}
      ]}
    ],
    "max_tokens": 96,
    "temperature": 0,
    "stream": false
  }'

Measured response with the default t70 profile:

The animal in the image is a red lobster.

One image per message is supported. Mixing image and audio in the same message is not supported.

Audio Request

Pass the audio as a board-side file path or as a base64 data: URI. The clip length selects the encoder profile.

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": [
        {"type": "audio_url", "audio_url": {"url": "./assets/gemma4_audio_test_5s.wav"}},
        {"type": "text", "text": "Transcribe the speech in this audio in its original language. Output only the transcription."}
      ]}
    ],
    "max_tokens": 128,
    "temperature": 0,
    "stream": false
  }'

Measured response for assets/gemma4_audio_test_5s.wav (5 s profile, 125 audio tokens):

When I was seventeen, I read a quote that went something like, if you

Measured response for assets/gemma4_audio_test_chunk0_30s.wav (30 s profile, 750 audio tokens):

No one wants to die. Even people who want to go to heaven don't want to die to get there. And yet,
death is the destination we all share. No one has ever escaped it, and that is as it should be
because death is very likely the single best invention of life. It's life's change agent. It clears
out the old to make way for the new. Right now, the new is you. But someday, not too long from now,
you will gradually become

End to end on AX650 with the packaged clips this takes about 9 s for the 5 s clip and 34 to 44 s for a 30 s clip, almost all of it text decode rather than audio processing. Both transcriptions above stop at the max_tokens limit, not at a sentence boundary.

One audio file per message is supported. Mixing audio and image in the same message is not supported.

Video Request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K",
    "messages": [
      {"role": "user", "content": [
        {"type": "video_url", "video_url": {"url": "./assets/red-panda-openai.mp4"}},
        {"type": "text", "text": "What animal is in this video? Answer with the animal name only."}
      ]}
    ],
    "max_tokens": 48,
    "temperature": 0,
    "stream": false
  }'

Measured response for the packaged 30 s clip at the default t70 profile:

Red Panda

A video request of this length costs about 2360 input tokens and takes roughly 40 s end to end.

Multi-turn Conversation

Send the whole message list each turn, as an OpenAI client normally does. The runtime keeps the KV cache from the previous turns and only processes the new tokens, so follow-up turns are far cheaper than the first. With a 3223-token document in the opening message, the first turn takes about 22 s to the first token and each short follow-up about 3 s, with the earlier turns still available to the model.

Context Limits

Limit	Value	Behaviour
Input tokens per request	6400	Above it the server returns HTTP 400 with input_num_token(N) > prefill_max_token_num(6400)
Total context (prompt + generation + history)	8191	KV cache is pre-allocated for this at startup

A request over the 6400-token cap is rejected outright rather than silently truncated, so keep prompts within the cap.

Transcription Endpoint

The OpenAI-compatible /v1/audio/transcriptions and /v1/audio/translations endpoints are also available. They take a multipart upload, so the audio file does not have to exist on the board, and they work with any OpenAI-compatible client or web UI without extra parameters.

curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F model=AXERA-TECH/gemma-4-E4B-it-GPTQ-INT4-P6K-C8K \
  -F file=@./assets/gemma4_audio_test_5s.wav

Measured response:

{
  "text": "When I was seventeen, I read a quote that went something like, if you"
}

The 30 s clips select the 750-token encoder through the same endpoint. Measured response for assets/gemma4_audio_test_chunk0_30s.wav:

{
  "text": "No one wants to die. Even people who want to go to heaven don't want to die to get there. And yet, death is the destination we all share. No one has ever escaped it, and that is as it should be because death is very likely the single best invention of life. It's life's change agent. It clears out the old to make way for the new. Right now, the new is you. But someday, not too long from now, you will gradually become"
}

response_format accepts json (default), verbose_json, text, srt and vtt. The response is always wrapped as {"text": ...}; srt and vtt put the same text in a single subtitle cue spanning the whole clip, so there is no word-level or segment-level timing.

/v1/audio/translations follows the OpenAI definition and translates into English. With the packaged English clips its output is the same as the transcription.

Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

• Original Hugging Face model: google/gemma-4-E4B-it
• Source INT4 checkpoint for the text weights: Vishva007/gemma-4-E4B-it-W4A16-AutoRound-GPTQ
• AXERA conversion and deployment workflow: AXERA-TECH/gemma-4-E2B-it.axera

Discussion

• GitHub Issues
• QQ group: 139953715