---
license: apache-2.0
base_model: ibm-granite/granite-embedding-311m-multilingual-r2
tags:
  - gguf
  - llama.cpp
  - embeddings
  - granite
  - modernbert
  - edge
---

# granite-embedding-311m-multilingual-r2-GGUF

F16 GGUF conversion of [`ibm-granite/granite-embedding-311m-multilingual-r2`](https://huggingface.co/ibm-granite/granite-embedding-311m-multilingual-r2) for local serving with [llama.cpp](https://github.com/ggml-org/llama.cpp). Converted and independently verified by ATF (Agent Taskflow) for edge-local embedding serving via `atf-serve`.

This is a format conversion only — no weights were modified, retrained, or fine-tuned. All model weights are © IBM, licensed Apache-2.0 (same as the base model). This repository is not affiliated with or endorsed by IBM.

## Why this exists

IBM does not publish a GGUF for this model. This repo documents its own build end-to-end — source checksum, conversion command, and independent correctness verification — rather than asking you to trust an unverified re-hosted binary.

## Conversion details

- **Source**: `ibm-granite/granite-embedding-311m-multilingual-r2`, `model.safetensors` (bf16, 623,341,952 bytes)
- **Tool**: `llama.cpp` built from source at commit `11924d4c17abc27383376a1ac6a24fa3e36c1c0c` (2026-08-02). This model's tokenizer (`granite-embed-multi-311m`, maps to `LLAMA_VOCAB_PRE_TYPE_GEMMA4`) is **not** recognized by llama.cpp release `b9204` or earlier — the registration landed upstream after that tag. A current build (or any release ≥ the commit that added it) is required both to *convert* and to *serve* this model; older binaries fail with `unknown pre-tokenizer type: 'granite-embed-multi-311m'` at load time, not at conversion time.
- **Command**:
  ```
  python3 convert_hf_to_gguf.py <model-dir> \
    --outfile granite-embedding-311m-multilingual-r2-f16.gguf \
    --outtype f16
  ```
- **Output**: F16, 768-dim, 638,121,344 bytes.

## Verification (independent, not vendor-claimed)

Embedded the same test sentence through both this GGUF (via `llama-server --embedding --pooling cls`) and the original HF model (via `sentence-transformers`, loaded directly from the source safetensors), then computed cosine similarity between the two output vectors.

| Check | Result |
|---|---|
| Output dimension | 768 (matches source `hidden_size`) |
| Cosine similarity vs. HF reference pipeline | **0.999970** |
| Required pooling mode | `cls` (matches source `classifier_pooling: "cls"` / `pooling_mode_cls_token: true` in `config.json`; mean pooling is **not** correct for this model) |

## Usage

```
llama-server --model granite-embedding-311m-multilingual-r2-f16.gguf \
  --embedding --pooling cls --port 8089
```

Requires a llama.cpp build that includes `granite-embed-multi-311m` tokenizer support (see Conversion details above — current upstream `master` has it; check your pinned release tag if serving fails with an `unknown pre-tokenizer type` error).

```
curl http://127.0.0.1:8089/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "your text here", "model": "granite-embedding-311m"}'
```

---

Converted by [ATF](https://atf.ai) — agent orchestration with edge-local model serving.