README.md

metadata

base_model: google/gemma-4-E2B-it-assistant
language:
  - en
library_name: gguf
license: gemma
tags:
  - gguf
  - quantized
  - mtp
  - gemma-4

Gemma 4 E2B IT Assistant — MTP GGUF

This repository contains a GGUF conversion of google/gemma-4-E2B-it-assistant to use as MTP, including the centroid-based embedding tensors (masked_embedding.centroids and masked_embedding.token_ordering) used by the E2B/E4B sparse variants.

Do not use with llama.cpp as they do not offer support. This model is only compatible with ik_llama.cpp.

Use only with PR 1744, which enables the gemma 4 MTP to work, otherwise you will encounter errors such as the model failing to load.

New: PR is merged.

Usage

The assistant is a draft model that requires a Gemma 4 target GGUF alongside it.

./build/bin/llama-server \
  --model google_gemma-4-E2B-it-Q8_0.gguf \
  --ctx-size 32768 ctk q8_0 -ctv q8_0 --n-gpu-layers 99 \
  -b 1024 -ub 1024 --jinja \
  --spec-type mtp -md gemma-4-E2B-it-assistant-F16.gguf -ngld 99 \
  --draft-max 3 --draft-p-min 0.0

Note: --draft-max 3 is a good starting point.