---
language:
- multilingual
license: mit
base_model:
- hotchpotch/mmBERT-L4H384-pruned
base_model_relation: finetune
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- embeddings
- feature-extraction
- modernbert
- matryoshka
- multilingual
- retrieval
---

# bekko-embedding-v1-a8m-pt

bekko-embedding-v1-a8m-pt is the pretrained model used as the base for [bekko-embedding-v1-a8m](https://huggingface.co/hotchpotch/bekko-embedding-v1-a8m). It is a starting point for additional training on a downstream task, not a model intended for direct use.

> [!IMPORTANT]
> Fine-tune this checkpoint for your target task before using it. It has not received the final supervised retrieval fine-tuning and should not be used directly as a general-purpose embedding or retrieval model. For general-purpose retrieval, use [bekko-embedding-v1-a8m](https://huggingface.co/hotchpotch/bekko-embedding-v1-a8m).

## Model Details

| Item | Value |
| --- | --- |
| Model type | Pretrained Sentence Transformer dense embedding model |
| Architecture | mmBERT (ModernBERT-style) encoder, 4 layers, hidden size 384 |
| Base model | [hotchpotch/mmBERT-L4H384-pruned](https://huggingface.co/hotchpotch/mmBERT-L4H384-pruned) |
| Original backbone | [jhu-clsp/mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) |
| Active parameters | 7,671,168 |
| Total parameters | 105,975,168 |
| Embedding dimension | 384 |
| Supported truncate dimensions | 256, 128, 64 |
| Max sequence length | 8192 tokens |
| Pooling | Mean pooling |
| Similarity | Cosine similarity |
| Query/document prefixes | None |

The model was pretrained with multilingual text pairs using Matryoshka representation learning and quantization-aware training. The 4-layer backbone retains layers 0, 1, 2, and 18 from mmBERT-small.

## Intended Use

Use this checkpoint as initialization for further training on a specific downstream task. Appropriate uses include:

- fine-tuning for retrieval, classification, reranking, or another target task,
- reproducing or adapting the bekko fine-tuning pipeline,
- pretrain-versus-fine-tuned comparisons and ablation studies.

## License

MIT