Instructions to use ai-babai/giga-embeddings-0826-3b-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ai-babai/giga-embeddings-0826-3b-gguf with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ai-babai/giga-embeddings-0826-3b-gguf")

sentences = [
    "Это счастливый человек",
    "Это счастливая собака",
    "Это очень счастливый человек",
    "Сегодня солнечный день"
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ai-babai/giga-embeddings-0826-3b-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M

Use Docker

docker model run hf.co/ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M

LM Studio
Jan
Ollama
How to use ai-babai/giga-embeddings-0826-3b-gguf with Ollama:
```
ollama run hf.co/ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ai-babai/giga-embeddings-0826-3b-gguf with Docker Model Runner:
```
docker model run hf.co/ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M
```

Lemonade

How to use ai-babai/giga-embeddings-0826-3b-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ai-babai/giga-embeddings-0826-3b-gguf:Q4_K_M

Run and chat with the model

lemonade run user.giga-embeddings-0826-3b-gguf-Q4_K_M

List all available models

lemonade list

Atomic Chat

ai-babai commited on 13 days ago

Commit

8300cc3

verified ·

1 Parent(s): 33b2ff4

Polish model card and add quantization chooser

Browse files

Files changed (6) hide show

.gitattributes +1 -0
README.md +21 -16
README.ru.md +22 -16
assets/giga-embeddings-0826-gguf-choice.png +3 -0
assets/giga-embeddings-0826-gguf-choice.svg +126 -0
manifest.json +12 -0

.gitattributes CHANGED Viewed

@@ -37,3 +37,4 @@ giga-embeddings-0826-3b-bf16.gguf filter=lfs diff=lfs merge=lfs -text
 giga-embeddings-0826-3b-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
 giga-embeddings-0826-3b-q6_k.gguf filter=lfs diff=lfs merge=lfs -text
 giga-embeddings-0826-3b-q8_0.gguf filter=lfs diff=lfs merge=lfs -text

 giga-embeddings-0826-3b-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
 giga-embeddings-0826-3b-q6_k.gguf filter=lfs diff=lfs merge=lfs -text
 giga-embeddings-0826-3b-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+assets/giga-embeddings-0826-gguf-choice.png filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -46,19 +46,22 @@ included for research, but is not a better default than Q8_0 or Q4_K_M.
 This is an independent `ai-babai` GGUF conversion, not an official `ai-sage`
 release.
 ## Choose a file
-| Variant | Best for | Download | Saving vs BF16 | Measured Metal allocation | RTX PRO 4500 peak VRAM |
 |---|---|---:|---:|---:|---:|
-| **Q8_0** | **recommended default** | **3.354 GB** | **46.8%** | **4,810 MiB** | **5,128 MiB** |
-| **Q4_K_M** | **lightweight / experimental** | **1.961 GB** | **68.9%** | **3,482 MiB** | **3,802 MiB** |
-| BF16 | high-precision reference | 6.308 GB | — | 7,627 MiB | not measured |
-| Q6_K | research / owner review | 2.591 GB | 58.9% | 4,083 MiB | 4,402 MiB |
 Apple Silicon uses unified memory. Metal allocation and process RSS are
 different views of the same shared memory and must not be added together. Peak
-process RSS in the same Mac runs was 7,827 / 4,955 / 4,245 / 3,658 MiB for
-BF16 / Q8_0 / Q6_K / Q4_K_M.
 ## Quick start
@@ -129,21 +132,23 @@ leaderboard submission, or a cross-model comparison.
 | Q4_K_M | 0.778297 | 0.769349 | 0.888652 | −0.0461 points |
 ¹The small positive aggregate difference is not evidence that Q6_K improves the
-model. Q6_K failed a separate frozen representation gate on one reproducible
-long-code sample.
 The stricter frozen multilingual/code holdout compared each quantized GGUF with
 the BF16 GGUF:
-| Variant | Min / mean vector cosine | Top-1 agreement | Mean top-10 overlap | Gate |
-|---|---:|---:|---:|---|
-| Q8_0 | 0.993540 / 0.999734 | 100.00% | 99.06% | PASS |
-| Q6_K | 0.974783 / 0.997845 | 99.61% | 97.07% | FAIL: one long-code outlier |
-| Q4_K_M | 0.950085 / 0.982052 | 96.88% | 91.37% | FAIL: representation/ranking |
 These agreement numbers measure preservation versus our BF16 GGUF, not absolute
-retrieval accuracy. They are why Q4_K_M is labeled experimental even though its
-full RuBQ+SciFact NDCG loss was small.
 ## Measured speed

 This is an independent `ai-babai` GGUF conversion, not an official `ai-sage`
 release.
+![GGUF quantization chooser comparing file size, memory, speed, and quality](assets/giga-embeddings-0826-gguf-choice.png)
 ## Choose a file
+| Variant | Best for | File size | Saving vs BF16 | Metal memory | CUDA peak VRAM |
 |---|---|---:|---:|---:|---:|
+| **Q8_0** | **recommended default** | **3.35 GB** | **46.8%** | **5.04 GB** | **5.38 GB** |
+| **Q4_K_M** | **lightweight / experimental** | **1.96 GB** | **68.9%** | **3.65 GB** | **3.99 GB** |
+| BF16 | high-precision reference | 6.31 GB | — | 8.00 GB | not measured |
+| Q6_K | research / owner review | 2.59 GB | 58.9% | 4.28 GB | 4.62 GB |
 Apple Silicon uses unified memory. Metal allocation and process RSS are
 different views of the same shared memory and must not be added together. Peak
+process RSS in the same Mac runs was 8.21 / 5.20 / 4.45 / 3.84 GB for BF16 /
+Q8_0 / Q6_K / Q4_K_M. All capacities in this section use decimal GB
+(`1 GB = 10^9 bytes`); measured MiB values were converted to GB.
 ## Quick start
 | Q4_K_M | 0.778297 | 0.769349 | 0.888652 | −0.0461 points |
 ¹The small positive aggregate difference is not evidence that Q6_K improves the
+model. A separate frozen representation comparison found one reproducible
+long-code outlier for Q6_K.
 The stricter frozen multilingual/code holdout compared each quantized GGUF with
 the BF16 GGUF:
+| Variant | Min / mean vector cosine | Top-1 agreement | Mean top-10 overlap |
+|---|---:|---:|---:|
+| Q8_0 | 0.993540 / 0.999734 | 100.00% | 99.06% |
+| Q6_K | 0.974783 / 0.997845 | 99.61% | 97.07% |
+| Q4_K_M | 0.950085 / 0.982052 | 96.88% | 91.37% |
 These agreement numbers measure preservation versus our BF16 GGUF, not absolute
+retrieval accuracy. All four files passed functional runtime validation. The
+larger representation drift is why Q4_K_M is labeled experimental even though
+its full RuBQ+SciFact NDCG loss was small; Q6_K's lower minimum comes from one
+reproducible long-code sample.
 ## Measured speed

README.ru.md CHANGED Viewed

@@ -16,18 +16,22 @@ Giga Embeddings 3B 0826 для stock `llama.cpp`.
 Это независимая GGUF-конверсия `ai-babai`, а не официальный релиз `ai-sage`.
 ## Какой файл выбрать
-| Вариант | Для чего | Загрузка | Экономия к BF16 | Metal allocation | Peak VRAM RTX PRO 4500 |
 |---|---|---:|---:|---:|---:|
-| **Q8_0** | **рекомендуемый default** | **3,354 ГБ** | **46,8%** | **4 810 MiB** | **5 128 MiB** |
-| **Q4_K_M** | **лёгкий / экспериментальный** | **1,961 ГБ** | **68,9%** | **3 482 MiB** | **3 802 MiB** |
-| BF16 | высокоточный reference | 6,308 ГБ | — | 7 627 MiB | не измерялось |
-| Q6_K | исследование / owner review | 2,591 ГБ | 58,9% | 4 083 MiB | 4 402 MiB |
 На Apple Silicon unified memory общая: Metal allocation и RSS — разные срезы
 одной памяти, их нельзя складывать. Peak RSS в тех же Mac-тестах составил
-7 827 / 4 955 / 4 245 / 3 658 MiB для BF16 / Q8_0 / Q6_K / Q4_K_M.
 ## Быстрый старт
@@ -91,20 +95,22 @@ macro-average двух задач. Это не полный MTEB, не leaderboa
 | Q6_K | 0,779334 | 0,768964 | 0,895146 | +0,0576 points¹ |
 | Q4_K_M | 0,778297 | 0,769349 | 0,888652 | −0,0461 points |
-¹Небольшой положительный delta не доказывает улучшение Q6_K. На отдельном
-frozen representation gate у него воспроизводимо провалился один длинный
-пример с кодом.
 Более строгий frozen multilingual/code holdout сравнивал кванты с нашим BF16:
-| Вариант | Min / mean cosine | Top-1 | Mean top-10 overlap | Gate |
-|---|---:|---:|---:|---|
-| Q8_0 | 0,993540 / 0,999734 | 100,00% | 99,06% | PASS |
-| Q6_K | 0,974783 / 0,997845 | 99,61% | 97,07% | FAIL: один long-code outlier |
-| Q4_K_M | 0,950085 / 0,982052 | 96,88% | 91,37% | FAIL: representation/ranking |
-Поэтому Q4 помечен экспериментальным, хотя его потеря полного
-RuBQ+SciFact NDCG невелика.
 ## Измеренная скорость

 Это независимая GGUF-конверсия `ai-babai`, а не официальный релиз `ai-sage`.
+![Выбор GGUF-кванта по размеру файла, памяти, скорости и качеству](assets/giga-embeddings-0826-gguf-choice.png)
 ## Какой файл выбрать
+| Вариант | Для чего | Размер файла | Экономия к BF16 | Память Metal | Peak VRAM CUDA |
 |---|---|---:|---:|---:|---:|
+| **Q8_0** | **рекомендуемый default** | **3,35 ГБ** | **46,8%** | **5,04 ГБ** | **5,38 ГБ** |
+| **Q4_K_M** | **лёгкий / экспериментальный** | **1,96 ГБ** | **68,9%** | **3,65 ГБ** | **3,99 ГБ** |
+| BF16 | высокоточный reference | 6,31 ГБ | — | 8,00 ГБ | не измерялось |
+| Q6_K | исследование / owner review | 2,59 ГБ | 58,9% | 4,28 ГБ | 4,62 ГБ |
 На Apple Silicon unified memory общая: Metal allocation и RSS — разные срезы
 одной памяти, их нельзя складывать. Peak RSS в тех же Mac-тестах составил
+8,21 / 5,20 / 4,45 / 3,84 ГБ для BF16 / Q8_0 / Q6_K / Q4_K_M. В этом разделе
+используются десятичные гигабайты (`1 ГБ = 10^9 байт`); исходные измерения в
+MiB пересчитаны в ГБ.
 ## Быстрый старт
 | Q6_K | 0,779334 | 0,768964 | 0,895146 | +0,0576 points¹ |
 | Q4_K_M | 0,778297 | 0,769349 | 0,888652 | −0,0461 points |
+¹Небольшой положительный delta не доказывает улучшение Q6_K. Отдельное
+сравнение сохранения представлений выявило один воспроизводимый длинный
+пример с кодом, на котором Q6_K отклоняется сильнее.
 Более строгий frozen multilingual/code holdout сравнивал кванты с нашим BF16:
+| Вариант | Min / mean cosine | Top-1 | Mean top-10 overlap |
+|---|---:|---:|---:|
+| Q8_0 | 0,993540 / 0,999734 | 100,00% | 99,06% |
+| Q6_K | 0,974783 / 0,997845 | 99,61% | 97,07% |
+| Q4_K_M | 0,950085 / 0,982052 | 96,88% | 91,37% |
+Все четыре файла прошли функциональную runtime-проверку. Q4 помечен
+экспериментальным из-за большего отклонения представлений, хотя его потеря
+полного RuBQ+SciFact NDCG невелика. Более низкий минимум Q6_K относится к
+одному воспроизводимому длинному примеру с кодом.
 ## Измеренная скорость

assets/giga-embeddings-0826-gguf-choice.png ADDED Viewed

Git LFS Details

SHA256: 81533d6d89915720ab35021afcdba363be353fd6f42a21663549a2dcd836c65a
Pointer size: 131 Bytes
Size of remote file: 404 kB

assets/giga-embeddings-0826-gguf-choice.svg ADDED Viewed

manifest.json CHANGED Viewed

@@ -32,6 +32,18 @@
     "pooling": "mean",
     "embedding_dimension": 2048
   },
   "artifacts": [
     {
       "file": "giga-embeddings-0826-3b-bf16.gguf",

     "pooling": "mean",
     "embedding_dimension": 2048
   },
+  "card_assets": [
+    {
+      "file": "assets/giga-embeddings-0826-gguf-choice.svg",
+      "role": "editable quantization chooser source",
+      "sha256": "7205e7e3d10223a570761aadd9c2a753552778effe5f454040352d6ff7c67b90"
+    },
+    {
+      "file": "assets/giga-embeddings-0826-gguf-choice.png",
+      "role": "rendered quantization chooser",
+      "sha256": "81533d6d89915720ab35021afcdba363be353fd6f42a21663549a2dcd836c65a"
+    }
+  ],
   "artifacts": [
     {
       "file": "giga-embeddings-0826-3b-bf16.gguf",