README.md

---
license: mit
thumbnail: https://huggingface.co/ai-babai/giga-embeddings-0826-3b-gguf/resolve/main/assets/giga-embeddings-0826-gguf-choice.png
language:
  - ru
  - en
pipeline_tag: sentence-similarity
library_name: gguf
base_model: ai-sage/Giga-Embeddings-instruct-3B-0826
base_model_relation: quantized
quantized_by: ai-babai
tags:
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - embeddings
  - text-embeddings
  - russian-text-embeddings
  - russian-embeddings
  - multilingual-embeddings
  - semantic-search
  - retrieval
  - rag
  - russian
  - local-ai
  - gguf
  - llama-cpp
  - ollama
  - apple-silicon
  - cuda
  - cpu
  - quantized
  - q8
  - q6
  - q4
  - arxiv:2608.23806
inference: false
---

# Giga Embeddings 0826 3B GGUF — Russian text embeddings for llama.cpp and Ollama

[Русская карточка](README.ru.md) ·
[GGUF collection](https://huggingface.co/collections/ai-babai/giga-embeddings-0826-gguf-llamacpp-and-ollama) ·
[Original model](https://huggingface.co/ai-sage/Giga-Embeddings-instruct-3B-0826) ·
[Smaller 480M GGUF](https://huggingface.co/ai-babai/giga-embeddings-0826-480m-gguf) ·
[Original paper](https://arxiv.org/abs/2608.23806)

Local Russian and English text embeddings for semantic search, RAG, text
similarity, clustering, and classification with stock `llama.cpp` and Ollama.
This repository contains one BF16 reference and three direct quantizations of
the bidirectional Giga Embeddings 3B 0826 model.

**Embedding model only:** use `llama-server --embeddings` or Ollama
`POST /api/embed`. This is not a chat or generative model.

**Start with `Q8_0`.** It is the recommended quality/size default. Choose
`Q4_K_M` only when download size and memory matter most; it is explicitly an
experimental lightweight option. BF16 is the high-precision reference. Q6_K is
included for research, but is not a better default than Q8_0 or Q4_K_M.

This is an independent `ai-babai` GGUF conversion, not an official `ai-sage`
release.

![Quant chooser for Russian text embeddings in llama.cpp and Ollama: Q8_0, Q4_K_M, BF16, and Q6_K](assets/giga-embeddings-0826-gguf-choice.png)

## Choose a quant in 10 seconds

| Variant | Best for | File size | Saving vs BF16 | Metal memory | CUDA peak VRAM |
|---|---|---:|---:|---:|---:|
| [BF16](giga-embeddings-0826-3b-bf16.gguf) | high-precision reference | 6.31 GB | — | 8.00 GB | not measured |
| [**Q8_0**](giga-embeddings-0826-3b-q8_0.gguf) | **recommended default** | **3.35 GB** | **46.8%** | **5.04 GB** | **5.38 GB** |
| [Q6_K](giga-embeddings-0826-3b-q6_k.gguf) | research / owner review | 2.59 GB | 58.9% | 4.28 GB | 4.62 GB |
| [Q4_K_M](giga-embeddings-0826-3b-q4_k_m.gguf) | lightweight / experimental | 1.96 GB | 68.9% | 3.65 GB | 3.99 GB |

Apple Silicon uses unified memory. Metal allocation and process RSS are
different views of the same shared memory and must not be added together. Peak
process RSS in the same Mac runs was 8.21 / 5.20 / 4.45 / 3.84 GB for BF16 /
Q8_0 / Q6_K / Q4_K_M. All capacities in this section use decimal GB
(`1 GB = 10^9 bytes`); measured MiB values were converted to GB.

[SHA256SUMS](SHA256SUMS) · [Machine-readable manifest](manifest.json)

## Quick start

Download the recommended file:

```bash
hf download ai-babai/giga-embeddings-0826-3b-gguf \
  giga-embeddings-0826-3b-q8_0.gguf \
  --local-dir .
```

Run a recent stock `llama.cpp` server:

```bash
llama-server \
  -m giga-embeddings-0826-3b-q8_0.gguf \
  --embeddings \
  -c 2048 -b 2048 -ub 2048 -np 1 \
  --cache-type-k f32 --cache-type-v f32 \
  --flash-attn auto -ngl 99 \
  --host 127.0.0.1 --port 8080
```

Use `-ngl 0` for CPU-only execution.

For retrieval, prepend an instruction to the **query** and embed documents as
plain text:

```bash
curl http://127.0.0.1:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-3b-q8_0.gguf",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации.",
      "Париж — столица Франции."
    ]
  }'
```

Compare the returned normalized 2048-dimensional embeddings with cosine
similarity (equivalent to their dot product after normalization).

For symmetric tasks such as semantic similarity or deduplication, use the same
instruction for both sides or no instruction. The GGUF metadata selects the
required mean pooling; do not replace it with CLS or last-token pooling.

## Quick start with Ollama

After downloading `giga-embeddings-0826-3b-q8_0.gguf`, create a `Modelfile`
next to it:

```dockerfile
FROM ./giga-embeddings-0826-3b-q8_0.gguf
```

Import the embedding model and call the embeddings API:

```bash
ollama create giga-embeddings-0826-3b-q8 -f Modelfile

curl http://127.0.0.1:11434/api/embed \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "giga-embeddings-0826-3b-q8",
    "input": [
      "Instruct: Given a query, retrieve relevant passages\nQuery: Где находится Москва?",
      "Москва — столица Российской Федерации."
    ]
  }'
```

BF16, Q8_0, Q6_K, and Q4_K_M were compatibility-tested with Ollama 0.33.3
on Apple M4 Pro Metal. Each imports as a Qwen3 embedding-only model and
returns finite, normalized 2048-dimensional vectors. This was a compatibility
smoke test, not an Ollama quality or performance benchmark.

## Quality at a glance

The original authors report 74.57 Russian MTEB, 71.93 English MTEB, 76.93 code
MTEB, and 63.9 multilingual MTEB for the source BF16 model. Those numbers belong
to the original model; we did not rerun the complete MTEB suites for these GGUF
files.

Our `giga-embeddings-external-retrieval-v1` evaluation used complete pinned
RuBQ (`e19b6ffa60b3bc248e0b41f4cc37c26a55c2a67b`) and SciFact
(`d56462d0e63a25450459c4f213e49ffdb866f7f9`) test splits, instruction-prefixed
queries, `title + "\n" + text` documents, and a 512-token contract. Values
below are equal-task macro averages. This is not a complete MTEB run, a
leaderboard submission, or a cross-model comparison.

| Variant | NDCG@10 | MRR@10 | Recall@10 | NDCG change vs BF16 |
|---|---:|---:|---:|---:|
| BF16 | 0.778758 | 0.767285 | 0.895762 | reference |
| **Q8_0** | **0.778431** | **0.767302** | **0.893750** | **−0.0327 points** |
| Q6_K | 0.779334 | 0.768964 | 0.895146 | +0.0576 points¹ |
| Q4_K_M | 0.778297 | 0.769349 | 0.888652 | −0.0461 points |

¹The small positive aggregate difference is not evidence that Q6_K improves the
model. A separate frozen representation comparison found one reproducible
long-code outlier for Q6_K.

The stricter frozen multilingual/code holdout compared each quantized GGUF with
the BF16 GGUF:

| Variant | Min / mean vector cosine | Top-1 agreement | Mean top-10 overlap |
|---|---:|---:|---:|
| Q8_0 | 0.993540 / 0.999734 | 100.00% | 99.06% |
| Q6_K | 0.974783 / 0.997845 | 99.61% | 97.07% |
| Q4_K_M | 0.950085 / 0.982052 | 96.88% | 91.37% |

These agreement numbers measure preservation versus our BF16 GGUF, not absolute
retrieval accuracy. All four files passed functional runtime validation. The
larger representation drift is why Q4_K_M is labeled experimental even though
its full RuBQ+SciFact NDCG loss was small; Q6_K's lower minimum comes from one
reproducible long-code sample.

## Measured speed

Median total throughput, after two warmups and across five repetitions, context
2048 and parallelism 1:

| Backend / workload | BF16 | Q8_0 | Q6_K | Q4_K_M |
|---|---:|---:|---:|---:|
| Apple M4 Pro Metal, 1×512 | 976 tok/s | 834 tok/s | 907 tok/s | 799 tok/s |
| Apple M4 Pro Metal, 16×1024 | 864 tok/s | 788 tok/s | 661 tok/s | 732 tok/s |
| Apple M4 Pro CPU, 1×512 | 243 tok/s | 345 tok/s | 149 tok/s | 189 tok/s |
| Apple M4 Pro CPU, 16×1024 | 298 tok/s | 275 tok/s | 140 tok/s | 205 tok/s |
| RTX PRO 4500 CUDA, 1×512 | not measured | 10,031 tok/s | 8,460 tok/s | 9,393 tok/s |
| RTX PRO 4500 CUDA, 16×1024 | not measured | 11,185 tok/s | 9,141 tok/s | 10,263 tok/s |

The Mac had an Apple M4 Pro and 48 GB unified memory. On Metal, BF16 was faster
than every quantized variant in the measured matrix; quantization primarily
saves storage and memory. Q6_K was also unusually slow on the tested CPU and
CUDA paths.

## Tested runtimes

Validation used clean stock `ggml-org/llama.cpp` commit
`e750b887a82719c27200b71545f63ed78ec24719` (Linux build 10763).

| Runtime/backend | BF16 | Q8_0 | Q6_K | Q4_K_M |
|---|---|---|---|---|
| macOS Apple Silicon / Metal, CLI + server | tested | tested | tested | tested |
| macOS Apple Silicon / CPU, server resource API | tested | tested | tested | tested |
| Clean stock Linux / CUDA, CLI + server | tested | tested | tested | tested |
| Clean stock Linux x86 CPU, CLI + server | not tested | tested | not tested | tested |
| Ollama 0.33.3 / Apple Metal | tested | tested | tested | tested |

All tested Linux lanes passed `llama-embedding`, `llama-server --embeddings`,
single/batch/repeat/permutation checks, and finite unit-norm 2048-dimensional
output checks. A separate fresh server process reproduced the same eight exact
test embeddings on each tested Linux lane (cosine 1.0, maximum component delta
0).

## Numerical and scope limitations

- Intended use is dense retrieval/RAG, semantic similarity, clustering, and
  classification in Russian and English. This is not a generative model or a
  cross-encoder reranker.
- Functional portability is not bit-identical cross-backend arithmetic. BF16
  passed the frozen Mac Metal → CUDA numerical parity gate. Q8_0 and Q6_K
  retained 8/8 top-1 agreement but narrowly missed its mean-cosine threshold:
  about 0.99987 observed versus 0.99990 required. Q4_K_M showed larger drift,
  with about 0.99929–0.99937 mean cosine and 6/8 to 8/8 top-1 agreement across
  tested lanes.
- The portability sample contains eight exact prompts. Within-backend repeat,
  permutation, and fresh-process checks passed; the observed differences are
  backend arithmetic drift, not evidence of corrupt files.
- Runtime/resource testing used context 2048. A 4096-token workload and true
  cold-cache startup were not benchmarked.
- External retrieval used a 512-token contract; 10.3% of SciFact corpus
  documents were truncated.
- Ollama preserves Q8_0/Q6_K/Q4_K_M as byte-identical GGUF model layers. Its
  BF16 import performs an internal same-size COPY rewrite while preserving the
  model type and metadata; BF16 is functionally compatible but is not executed
  byte-for-byte after import.
- Windows, Linux Ollama, LM Studio, Jan, and older `llama.cpp`/Ollama builds
  were not tested.

## Artifact integrity and provenance

| File | Bytes | SHA-256 |
|---|---:|---|
| `giga-embeddings-0826-3b-bf16.gguf` | 6,307,610,848 | `61820afd79134c8b3691fba0442aa203916d9a0c6101438a5e0a0bee61217919` |
| `giga-embeddings-0826-3b-q8_0.gguf` | 3,354,067,168 | `429f2d04a968ffe73137fe65c2e458a08236056168b905d208b4d81ecab08c22` |
| `giga-embeddings-0826-3b-q6_k.gguf` | 2,591,068,384 | `e7956ee5c0f0e6f776cc67c643f1cd99575697b928c373b644a31fb34b3dc247` |
| `giga-embeddings-0826-3b-q4_k_m.gguf` | 1,960,915,168 | `9f81d6e5015fc981d1c4ac9d66b8179efa4af21c7b2ce6d39acf04d0cdc9f5b5` |

- Source: `ai-sage/Giga-Embeddings-instruct-3B-0826`
- Exact source revision: `ed7db5c91b900b39381b27b6e9c0a3d31137cd29`
- Source license: MIT
- Source `model.safetensors` SHA-256:
  `de8519bef7ee360043970b0081088c5294e2a9196ad2d0570a33c4f51bb2e134`
- Source `tokenizer.json` SHA-256:
  `6fb1280bd7fd529f425929b5df823a5a44485cd7fa9679d1ec1acaac4962e8ca`
- Source `tokenizer_config.json` SHA-256:
  `843eeba481465c1485a5b5f24bd24d6c12c4e502c16f093c3ab6a0f058c2c5f2`
- Converter: local bidirectional-model patch
  `409723a88b12071974ed5924a2dc1c8b2b2064f7` on top of upstream `llama.cpp`
  `e750b887a82719c27200b71545f63ed78ec24719`
- Clean stock Linux validation runtime:
  `ggml-org/llama.cpp@e750b887a82719c27200b71545f63ed78ec24719`, build 10763
- Validation harness: `7ba4e00e76b27316b1b3709476d807958bcd1e9d`
- Q8_0, Q6_K, and Q4_K_M were quantized directly from the accepted BF16 GGUF;
  no cascade or requantization was used
- Architecture preserved: bidirectional Qwen3, 398/398 tensors, mean pooling,
  2048-dimensional output

Machine-readable provenance and hashes are also available in `manifest.json`
and `SHA256SUMS`.

## Русские текстовые эмбеддинги

Локальная GGUF-версия Giga Embeddings 0826 3B для семантического поиска, RAG,
сравнения текстов, кластеризации и классификации на русском и английском
языках. [Полное описание на русском](README.ru.md).

## Citation

Please cite both this GGUF release and the original Giga-Embeddings work:

```bibtex
@software{popkov2026gigaembeddingsgguf,
  author  = {Maksim Popkov},
  title   = {Giga Embeddings 0826 GGUF},
  year    = {2026},
  url     = {https://huggingface.co/ai-babai/giga-embeddings-0826-3b-gguf}
}

@misc{kolodin2026gigaembeddings,
  title         = {Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings},
  author        = {Egor Kolodin and Egor Krasnoperov and Evgeniy Kosarev and Fyodor Minkin},
  year          = {2026},
  eprint        = {2608.23806},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2608.23806}
}
```