webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode-GGUF

Sakura Gemma 4 E4B DualMode — Generation + 768d Embeddings — Q5_K_M GGUF

Research prototype. Community model. Not official from Google.

In 30 seconds

• What: one Gemma 4 E4B-it model that can chat and produce 768-dimensional search vectors for RAG, so you do not need a second embedding model next to it.
• How: the same backbone is used in two modes. Generate is the normal Gemma 4. Embed switches to bidirectional attention with a tiny adapter (+8.8 MB) and a 768d head.
• One file: Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf holds backbone, adapter, head and metadata.
• Checked: generation is bitwise identical to plain Gemma 4; switching modes leaks no state (36 switches, 0 differences); retrieval keeps 66-74 % of the teacher's MRR on external benchmarks.
• Catch: it needs a small llama.cpp patch (provided); stock llama.cpp cannot load this file.
• Which size? The smaller E2B DualMode GGUF retrieves better (71-79 % of the teacher) with the same recipe and is smaller (3.6 GB). Use E4B if you want the stronger chat model.

This is the GGUF version of webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode. EmbeddingGemma 2 (google/embeddinggemma-2) was used only as the teacher during training; it is not in this file and not needed at runtime. The retrieval is EmbeddingGemma-2-distilled, not index-compatible with EmbeddingGemma 2 vectors.

Mode	What runs
GENERATE	the original Gemma 4 E4B-it causal path and LM head (the embed adapter is not part of this graph)
EMBED	same backbone + bidirectional attention + embedding-only LoRA (q_proj/o_proj, layers 24-40, rank 16) + 768d projection + mean pooling + L2 normalization

A small llama.cpp patch is required. Stock llama.cpp cannot load Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf (unmodified llama.cpp b86d2f0: done_getting_tensors: wrong number of tensors; expected 789, got 720) because the file contains the embedding head and the embed-only LoRA as extra tensors. The patch is in patches/ and applies cleanly to llama.cpp commit f498f864fbc0472004ee1c3616c1188c68eb157f; build instructions are in docs/LLAMA_CPP_RUNTIME.md. For plain generation with stock llama.cpp use the unchanged base file Gemma4-E4B-it-Q5_K_M.gguf from this repo.

Files

File	Size	Purpose
Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf	5.77 GB	the dual-mode model (backbone Q5_K_M, head + LoRA F16, metadata). Needs the patched runtime.
Gemma4-E4B-it-Q5_K_M.gguf	5.76 GB	unchanged Gemma 4 E4B-it base for stock llama.cpp (generation only)
patches/llama.cpp-dualmode.patch	28 KB	runtime patch for llama.cpp f498f864 (same patch as for E2B)
docs/LLAMA_CPP_RUNTIME.md		pinned commit, build, run, GGUF tensor/metadata layout
assets/overview.svg		the diagram above
tools/		make_dualmode_gguf.py, dual_client.py (JSON-lines client), parity_texts.py

SHA256 Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf: 3ca221c803d9fddc6427f2308dcfa134cebcf09c67d4a30c8b88ca4f23f06c5e

The dual file is 9.2 MB larger than the base file of the same quantization (head + LoRA, 4,603,904 parameters in F16). Only the backbone is quantized.

Quick start

# build the patched runtime (see docs/LLAMA_CPP_RUNTIME.md)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout f498f864fbc0472004ee1c3616c1188c68eb157f
git apply ../patches/llama.cpp-dualmode.patch
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF && cmake --build build --target llama-sakura-dualmode llama-server -j 8

# one process, one backbone, two contexts (JSON lines on stdin/stdout)
build/bin/llama-sakura-dualmode -m Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf
{"op":"generate","prompt":"<bos><start_of_turn>user\nWhat is the capital of France?<end_of_turn>\n<start_of_turn>model\n","n":32}
{"op":"embed","text":"task: search result | query: What is the capital of France?"}
{"op":"embed","text":"title: none | text: Paris is the capital and most populous city of France."}

# or: embeddings over HTTP (embed mode only; llama-server --embeddings)
build/bin/llama-server -m Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf --embeddings --ubatch-size 512

Prefixes are part of the text: queries task: search result | query: , documents title: none | text: (or title: <title> | text: ). Truncate to about 128 tokens (training length). llama-server --embeddings serves the embed mode only; the single-process dual mode is llama-sakura-dualmode.

Measured results (E4B adapter, webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode)

SciFact test (300 queries / 500 docs), NFCorpus test (249 queries / 500 docs), max 128 tokens, same inputs as the Hugging Face model. Teacher = EmbeddingGemma 2 on the same texts.

Model	SciFact MRR	SciFact nDCG@10	NFCorpus MRR	NFCorpus nDCG@10
Teacher EmbeddingGemma 2	0.8971	0.9137	0.4609	0.3099
HF BF16 (reference)	0.6679	0.6984	0.3031	0.1836
DualMode GGUF Q5_K_M (this file)	0.6555	0.6867	0.2979	0.1795

Q5_K_M versus the BF16 model: MRR 98.2 % (SciFact) / 98.3 % (NFCorpus), nDCG@10 98.3 % / 97.7 %. Versus the teacher: 73.1 % / 64.6 % MRR. Mean cosine to the HF vectors: 0.963 (min 0.878). All embeddings are finite with L2 norm 1.

Runtime parity (F16, not shipped): a GGUF F16 build of this model matches an fp16 Hugging Face reference with cosine 1.0000 on 24 of 25 test texts (the 25th is a text longer than 128 tokens that the Hugging Face reference truncates and the server does not). The Q5_K_M cosine of 0.963 is therefore quantization noise of the backbone, not a runtime error.

Generation and state isolation (this file, patched runtime)

Check	Result
generate vs. the plain base GGUF of the same quant (3 prompts, 32 tokens)	tokens identical; raw logit bytes identical
36 embed/generate switches in one process on one model	0 embedding mismatches (bitwise), 0 generation mismatches

Performance (CPU only, 8 threads, AMD Ryzen AI Max+ 392, Windows 11; no GPU backend was built)

Metric	Value
File size	5.77 GB
Load time (mmap, warm file cache)	1.1 s
Resident memory after load / after use	234 MB / 3.62 GB (weights are memory-mapped, so this is not the model size)
GPU / UMA memory	0 (CPU build)
Embed latency	median 862 ms, min 250 ms
Generation	12.8 tokens/s (64 tokens incl. prefill)

Known limitations

• Needs the patched runtime for both modes. Only the CPU backend was tested.
• No single server process for chat + embeddings yet; llama-sakura-dualmode is the one-process path.
• Embed mode was trained on <=128 tokens, without BOS (the runtime masks the BOS token that llama.cpp always adds).
• Quality: research grade. About 66-74 % of the teacher's MRR on external retrieval, below the E2B model; not a drop-in replacement for EmbeddingGemma 2.
• Only Q5_K_M was built and measured for E4B.
• The patch is a private-fork style patch for this release, not an upstream proposal.

Provenance

Backbone google/gemma-4-E4B-it, teacher google/embeddinggemma-2 (914f7f89142e33e77833254d9c9b90c3cef7303b), distillation recipe: see the HF model card. Gemma terms of use apply. The similarly named webmp3/Sakura-EmbeddingGemma-2-AutoRound-GGUF is a separate project (quantized EmbeddingGemma 2 only).

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura Gemma 4 E4B DualMode — 生成 + 768 维嵌入 — Q5_K_M GGUF

研究原型。社区模型。并非 Google 官方发布。

30 秒了解

• 是什么: 一个 Gemma 4 E4B-it 模型,既能 聊天,又能为 RAG 生成 768 维的搜索向量,因此无需在它旁边再放第二个嵌入模型。
• 如何实现: 同一个主干以两种模式使用。Generate(生成)是正常的 Gemma 4。Embed(嵌入)切换为双向注意力,并加上一个很小的适配器(+8.8 MB)和一个 768 维的头。
• 一个文件: Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf 包含主干、适配器、头和元数据。
• 已检查: 生成结果与原版 Gemma 4 逐位一致;切换模式不会泄漏状态(36 次切换,0 处差异);在外部基准测试上,检索保留教师模型 MRR 的 66-74 %。
• 注意事项: 需要一个小型 llama.cpp 补丁(已提供);原版 llama.cpp 无法加载此文件。
• 选哪个尺寸? 较小的 E2B DualMode GGUF 在相同配方下检索效果更好(教师的 71-79 %),而且更小(3.6 GB)。如果你想要更强的聊天模型,请使用 E4B。

这是 webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode 的 GGUF 版本。EmbeddingGemma 2(google/embeddinggemma-2)在训练中 仅作为教师模型 使用;它不在此文件中,运行时也不需要。该检索是 EmbeddingGemma-2 蒸馏 的,与 EmbeddingGemma 2 的向量在索引上 不 兼容。

模式	运行内容
GENERATE	原始 Gemma 4 E4B-it 的因果路径和 LM 头(嵌入适配器不属于此计算图)
EMBED	同一个主干 + 双向注意力 + 仅用于嵌入的 LoRA(q_proj/o_proj,24-40 层,rank 16)+ 768 维投影 + 均值池化 + L2 归一化

需要一个小型 llama.cpp 补丁。 原版 llama.cpp 无法加载 Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf(未修改的 llama.cpp b86d2f0:done_getting_tensors: wrong number of tensors; expected 789, got 720),因为该文件额外包含嵌入头和仅用于嵌入的 LoRA 张量。补丁位于 patches/,可干净地应用到 llama.cpp 提交 f498f864fbc0472004ee1c3616c1188c68eb157f 上;构建说明见 docs/LLAMA_CPP_RUNTIME.md。若只用原版 llama.cpp 进行普通生成,请使用本仓库中未改动的基础文件 Gemma4-E4B-it-Q5_K_M.gguf。

文件

文件	大小	用途
Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf	5.77 GB	双模式模型(主干 Q5_K_M,头 + LoRA 为 F16,含元数据)。需要打过补丁的运行时。
Gemma4-E4B-it-Q5_K_M.gguf	5.76 GB	供 原版 llama.cpp 使用的未改动 Gemma 4 E4B-it 基础文件(仅生成)
patches/llama.cpp-dualmode.patch	28 KB	针对 llama.cpp f498f864 的运行时补丁(与 E2B 使用同一个补丁)
docs/LLAMA_CPP_RUNTIME.md		固定的提交、构建、运行、GGUF 张量/元数据布局
assets/overview.svg		上面的示意图
tools/		make_dualmode_gguf.py、dual_client.py(JSON-lines 客户端)、parity_texts.py

Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf 的 SHA256:3ca221c803d9fddc6427f2308dcfa134cebcf09c67d4a30c8b88ca4f23f06c5e

双模式文件比相同量化的基础文件大 9.2 MB(头 + LoRA,4,603,904 个 F16 参数)。只有主干被量化。

快速开始

# build the patched runtime (see docs/LLAMA_CPP_RUNTIME.md)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout f498f864fbc0472004ee1c3616c1188c68eb157f
git apply ../patches/llama.cpp-dualmode.patch
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF && cmake --build build --target llama-sakura-dualmode llama-server -j 8

# one process, one backbone, two contexts (JSON lines on stdin/stdout)
build/bin/llama-sakura-dualmode -m Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf
{"op":"generate","prompt":"<bos><start_of_turn>user\nWhat is the capital of France?<end_of_turn>\n<start_of_turn>model\n","n":32}
{"op":"embed","text":"task: search result | query: What is the capital of France?"}
{"op":"embed","text":"title: none | text: Paris is the capital and most populous city of France."}

# or: embeddings over HTTP (embed mode only; llama-server --embeddings)
build/bin/llama-server -m Sakura-Gemma4-E4B-DualMode-Q5_K_M.gguf --embeddings --ubatch-size 512

前缀是文本的一部分:查询 task: search result | query: ,文档 title: none | text: (或 title: <title> | text: )。请截断到约 128 个 token(训练长度)。llama-server --embeddings 只提供嵌入模式;单进程的双模式是 llama-sakura-dualmode。

测量结果(E4B 适配器,webmp3/Sakura-Gemma4-E4B-EmbeddingGemma2-DualMode)

SciFact test(300 个查询 / 500 篇文档)、NFCorpus test(249 个查询 / 500 篇文档),最多 128 个 token,输入与 Hugging Face 模型相同。教师 = 在相同文本上的 EmbeddingGemma 2。

模型	SciFact MRR	SciFact nDCG@10	NFCorpus MRR	NFCorpus nDCG@10
教师 EmbeddingGemma 2	0.8971	0.9137	0.4609	0.3099
HF BF16(参照)	0.6679	0.6984	0.3031	0.1836
DualMode GGUF Q5_K_M(本文件)	0.6555	0.6867	0.2979	0.1795

Q5_K_M 相对于 BF16 模型:MRR 为 98.2 %(SciFact)/ 98.3 %(NFCorpus),nDCG@10 为 98.3 % / 97.7 %。相对于教师:MRR 为 73.1 % / 64.6 %。与 HF 向量的平均余弦:0.963(min 0.878)。所有嵌入都是有限值,L2 范数为 1。

运行时对照(F16,未发布): 该模型的 GGUF F16 构建在 25 个测试文本中的 24 个上与 fp16 的 Hugging Face 参照的余弦为 1.0000(第 25 个文本超过 128 个 token,Hugging Face 参照会截断它,而服务器不会)。因此 Q5_K_M 的余弦 0.963 是主干的量化噪声,而不是运行时错误。

生成与状态隔离(本文件,打过补丁的运行时)

检查项	结果
generate 与相同量化的普通基础 GGUF 比较(3 个提示,32 个 token)	token 一致;原始 logit 字节一致
在一个进程、一个模型上进行 36 次 embed/generate 切换	0 处嵌入不一致(逐位),0 处生成不一致

性能(仅 CPU,8 线程,AMD Ryzen AI Max+ 392,Windows 11;未构建 GPU 后端)

指标	数值
文件大小	5.77 GB
加载时间(mmap,文件缓存已预热)	1.1 s
加载后 / 使用后的常驻内存	234 MB / 3.62 GB(权重通过内存映射,因此这并不是模型大小)
GPU / UMA 内存	0(CPU 构建)
嵌入延迟	中位数 862 ms,最小 250 ms
生成	12.8 tokens/s(64 个 token,含预填充)

已知局限

• 两种模式都需要打过补丁的运行时。只测试了 CPU 后端。
• 目前还没有同时提供聊天和嵌入的单一服务器进程;llama-sakura-dualmode 是单进程方案。
• 嵌入模式是在 <=128 个 token 上、不带 BOS 训练的(运行时会屏蔽 llama.cpp 总是添加的 BOS token)。
• 质量:研究级。在外部检索上约为教师模型 MRR 的 66-74 %,低于 E2B 模型;不能直接替代 EmbeddingGemma 2。
• 对于 E4B,只构建并测量了 Q5_K_M。
• 该补丁是为本次发布准备的私有分支式补丁,并非提交给上游的提案。

出处

主干 google/gemma-4-E4B-it,教师 google/embeddinggemma-2(914f7f89142e33e77833254d9c9b90c3cef7303b),蒸馏配方:见 HF 模型卡。适用 Gemma 使用条款。名称相近的 webmp3/Sakura-EmbeddingGemma-2-AutoRound-GGUF 是一个独立的项目(仅量化 EmbeddingGemma 2)。