ai-babai/giga-embeddings-0826-3b-gguf

Giga Embeddings 0826 3B GGUF — Russian text embeddings for llama.cpp and Ollama

Русская карточка · Original model · Smaller 480M GGUF · Original paper

Local Russian and English text embeddings for semantic search, RAG, text similarity, clustering, and classification with stock llama.cpp and Ollama. This repository contains one BF16 reference and three direct quantizations of the bidirectional Giga Embeddings 3B 0826 model.

Embedding model only: use llama-server --embeddings or Ollama POST /api/embed. This is not a chat or generative model.

Start with Q8_0. It is the recommended quality/size default. Choose Q4_K_M only when download size and memory matter most; it is explicitly an experimental lightweight option. BF16 is the high-precision reference. Q6_K is included for research, but is not a better default than Q8_0 or Q4_K_M.

This is an independent ai-babai GGUF conversion, not an official ai-sage release.

Choose a quant in 10 seconds

Variant	Best for	File size	Saving vs BF16	Metal memory	CUDA peak VRAM
BF16	high-precision reference	6.31 GB	—	8.00 GB	not measured
Q8_0	recommended default	3.35 GB	46.8%	5.04 GB	5.38 GB
Q6_K	research / owner review	2.59 GB	58.9%	4.28 GB	4.62 GB
Q4_K_M	lightweight / experimental	1.96 GB	68.9%	3.65 GB	3.99 GB

Apple Silicon uses unified memory. Metal allocation and process RSS are different views of the same shared memory and must not be added together. Peak process RSS in the same Mac runs was 8.21 / 5.20 / 4.45 / 3.84 GB for BF16 / Q8_0 / Q6_K / Q4_K_M. All capacities in this section use decimal GB (1 GB = 10^9 bytes); measured MiB values were converted to GB.

SHA256SUMS · Machine-readable manifest

Quick start

Download the recommended file:

hf download ai-babai/giga-embeddings-0826-3b-gguf \
  giga-embeddings-0826-3b-q8_0.gguf \
  --local-dir .

Run a recent stock llama.cpp server:

llama-server \
  -m giga-embeddings-0826-3b-q8_0.gguf \
  --embeddings \
  -c 2048 -b 2048 -ub 2048 -np 1 \
  --cache-type-k f32 --cache-type-v f32 \
  --flash-attn auto -ngl 99 \
  --host 127.0.0.1 --port 8080

Use -ngl 0 for CPU-only execution.

For retrieval, prepend an instruction to the query and embed documents as plain text:

curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-3b-q8_0.gguf",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации.",
      "Париж — столица Франции."
    ]
  }'

Compare the returned normalized 2048-dimensional embeddings with cosine similarity (equivalent to their dot product after normalization).

For symmetric tasks such as semantic similarity or deduplication, use the same instruction for both sides or no instruction. The GGUF metadata selects the required mean pooling; do not replace it with CLS or last-token pooling.

Quick start with Ollama

After downloading giga-embeddings-0826-3b-q8_0.gguf, create a Modelfile next to it:

FROM ./giga-embeddings-0826-3b-q8_0.gguf

Import the embedding model and call the embeddings API:

ollama create giga-embeddings-0826-3b-q8 -f Modelfile

curl http://127.0.0.1:11434/api/embed \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-3b-q8",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'

BF16, Q8_0, Q6_K, and Q4_K_M were compatibility-tested with Ollama 0.33.3 on Apple M4 Pro Metal. Each imports as a Qwen3 embedding-only model and returns finite, normalized 2048-dimensional vectors. This was a compatibility smoke test, not an Ollama quality or performance benchmark.

Quality at a glance

The original authors report 74.57 Russian MTEB, 71.93 English MTEB, 76.93 code MTEB, and 63.9 multilingual MTEB for the source BF16 model. Those numbers belong to the original model; we did not rerun the complete MTEB suites for these GGUF files.

Our giga-embeddings-external-retrieval-v1 evaluation used complete pinned RuBQ (e19b6ffa60b3bc248e0b41f4cc37c26a55c2a67b) and SciFact (d56462d0e63a25450459c4f213e49ffdb866f7f9) test splits, instruction-prefixed queries, title + "\n" + text documents, and a 512-token contract. Values below are equal-task macro averages. This is not a complete MTEB run, a leaderboard submission, or a cross-model comparison.

Variant	NDCG@10	MRR@10	Recall@10	NDCG change vs BF16
BF16	0.778758	0.767285	0.895762	reference
Q8_0	0.778431	0.767302	0.893750	−0.0327 points
Q6_K	0.779334	0.768964	0.895146	+0.0576 points¹
Q4_K_M	0.778297	0.769349	0.888652	−0.0461 points

¹The small positive aggregate difference is not evidence that Q6_K improves the model. A separate frozen representation comparison found one reproducible long-code outlier for Q6_K.

The stricter frozen multilingual/code holdout compared each quantized GGUF with the BF16 GGUF:

Variant	Min / mean vector cosine	Top-1 agreement	Mean top-10 overlap
Q8_0	0.993540 / 0.999734	100.00%	99.06%
Q6_K	0.974783 / 0.997845	99.61%	97.07%
Q4_K_M	0.950085 / 0.982052	96.88%	91.37%

These agreement numbers measure preservation versus our BF16 GGUF, not absolute retrieval accuracy. All four files passed functional runtime validation. The larger representation drift is why Q4_K_M is labeled experimental even though its full RuBQ+SciFact NDCG loss was small; Q6_K's lower minimum comes from one reproducible long-code sample.

Measured speed

Median total throughput, after two warmups and across five repetitions, context 2048 and parallelism 1:

Backend / workload	BF16	Q8_0	Q6_K	Q4_K_M
Apple M4 Pro Metal, 1×512	976 tok/s	834 tok/s	907 tok/s	799 tok/s
Apple M4 Pro Metal, 16×1024	864 tok/s	788 tok/s	661 tok/s	732 tok/s
Apple M4 Pro CPU, 1×512	243 tok/s	345 tok/s	149 tok/s	189 tok/s
Apple M4 Pro CPU, 16×1024	298 tok/s	275 tok/s	140 tok/s	205 tok/s
RTX PRO 4500 CUDA, 1×512	not measured	10,031 tok/s	8,460 tok/s	9,393 tok/s
RTX PRO 4500 CUDA, 16×1024	not measured	11,185 tok/s	9,141 tok/s	10,263 tok/s

The Mac had an Apple M4 Pro and 48 GB unified memory. On Metal, BF16 was faster than every quantized variant in the measured matrix; quantization primarily saves storage and memory. Q6_K was also unusually slow on the tested CPU and CUDA paths.

Tested runtimes

Validation used clean stock ggml-org/llama.cpp commit e750b887a82719c27200b71545f63ed78ec24719 (Linux build 10763).

Runtime/backend	BF16	Q8_0	Q6_K	Q4_K_M
macOS Apple Silicon / Metal, CLI + server	tested	tested	tested	tested
macOS Apple Silicon / CPU, server resource API	tested	tested	tested	tested
Clean stock Linux / CUDA, CLI + server	tested	tested	tested	tested
Clean stock Linux x86 CPU, CLI + server	not tested	tested	not tested	tested
Ollama 0.33.3 / Apple Metal	tested	tested	tested	tested

All tested Linux lanes passed llama-embedding, llama-server --embeddings, single/batch/repeat/permutation checks, and finite unit-norm 2048-dimensional output checks. A separate fresh server process reproduced the same eight exact test embeddings on each tested Linux lane (cosine 1.0, maximum component delta 0).

Numerical and scope limitations

• Intended use is dense retrieval/RAG, semantic similarity, clustering, and classification in Russian and English. This is not a generative model or a cross-encoder reranker.
• Functional portability is not bit-identical cross-backend arithmetic. BF16 passed the frozen Mac Metal → CUDA numerical parity gate. Q8_0 and Q6_K retained 8/8 top-1 agreement but narrowly missed its mean-cosine threshold: about 0.99987 observed versus 0.99990 required. Q4_K_M showed larger drift, with about 0.99929–0.99937 mean cosine and 6/8 to 8/8 top-1 agreement across tested lanes.
• The portability sample contains eight exact prompts. Within-backend repeat, permutation, and fresh-process checks passed; the observed differences are backend arithmetic drift, not evidence of corrupt files.
• Runtime/resource testing used context 2048. A 4096-token workload and true cold-cache startup were not benchmarked.
• External retrieval used a 512-token contract; 10.3% of SciFact corpus documents were truncated.
• Ollama preserves Q8_0/Q6_K/Q4_K_M as byte-identical GGUF model layers. Its BF16 import performs an internal same-size COPY rewrite while preserving the model type and metadata; BF16 is functionally compatible but is not executed byte-for-byte after import.
• Windows, Linux Ollama, LM Studio, Jan, and older llama.cpp/Ollama builds were not tested.

Artifact integrity and provenance

File	Bytes	SHA-256
giga-embeddings-0826-3b-bf16.gguf	6,307,610,848	61820afd79134c8b3691fba0442aa203916d9a0c6101438a5e0a0bee61217919
giga-embeddings-0826-3b-q8_0.gguf	3,354,067,168	429f2d04a968ffe73137fe65c2e458a08236056168b905d208b4d81ecab08c22
giga-embeddings-0826-3b-q6_k.gguf	2,591,068,384	e7956ee5c0f0e6f776cc67c643f1cd99575697b928c373b644a31fb34b3dc247
giga-embeddings-0826-3b-q4_k_m.gguf	1,960,915,168	9f81d6e5015fc981d1c4ac9d66b8179efa4af21c7b2ce6d39acf04d0cdc9f5b5

• Source: ai-sage/Giga-Embeddings-instruct-3B-0826
• Exact source revision: ed7db5c91b900b39381b27b6e9c0a3d31137cd29
• Source license: MIT
• Source model.safetensors SHA-256: de8519bef7ee360043970b0081088c5294e2a9196ad2d0570a33c4f51bb2e134
• Source tokenizer.json SHA-256: 6fb1280bd7fd529f425929b5df823a5a44485cd7fa9679d1ec1acaac4962e8ca
• Source tokenizer_config.json SHA-256: 843eeba481465c1485a5b5f24bd24d6c12c4e502c16f093c3ab6a0f058c2c5f2
• Converter: local bidirectional-model patch 409723a88b12071974ed5924a2dc1c8b2b2064f7 on top of upstream llama.cpp e750b887a82719c27200b71545f63ed78ec24719
• Clean stock Linux validation runtime: ggml-org/llama.cpp@e750b887a82719c27200b71545f63ed78ec24719, build 10763
• Validation harness: 7ba4e00e76b27316b1b3709476d807958bcd1e9d
• Q8_0, Q6_K, and Q4_K_M were quantized directly from the accepted BF16 GGUF; no cascade or requantization was used
• Architecture preserved: bidirectional Qwen3, 398/398 tensors, mean pooling, 2048-dimensional output

Machine-readable provenance and hashes are also available in manifest.json and SHA256SUMS.

Русские текстовые эмбеддинги

Локальная GGUF-версия Giga Embeddings 0826 3B для семантического поиска, RAG, сравнения текстов, кластеризации и классификации на русском и английском языках. Полное описание на русском.

Citation

Please cite both this GGUF release and the original Giga-Embeddings work:

@software{popkov2026gigaembeddingsgguf,
  author  = {Maksim Popkov},
  title   = {Giga Embeddings 0826 GGUF},
  year    = {2026},
  url     = {https://huggingface.co/ai-babai/giga-embeddings-0826-3b-gguf}
}

@misc{kolodin2026gigaembeddings,
  title         = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
  author        = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
  year          = {2026},
  eprint        = {2608.23806},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.23806}
}