Update README.md

README.md

@@ -7,11 +7,15 @@ base_model:
 - BSC-LT/MrBERT
 datasets:
 - proxectonos/corpusnos
+pipeline_tag: fill-mask
 ---
 # MrBERT-nos-gl
 
-MrBERT-nos-gl is a domain-adapted encoder model fine-tuned from [BSC-LT/MrBERT](https://huggingface.co/BSC-LT/MrBERT) on a large-scale Galician and Portuguese corpus. It inherits MrBERT's ModernBERT architecture — with efficient long-context modeling and strong bidirectional representations — and extends its capabilities with significantly improved coverage of two closely related Iberian languages that are underrepresented in most multilingual encoders.
-Galician (galego) and European/Brazilian Portuguese share substantial lexical and morphological overlap, making joint continued pre-training an effective strategy for mutual reinforcement. MrBERT-gl is designed as a general-purpose encoder for both languages, suitable for tasks such as named entity recognition, text classification, semantic similarity, question answering, and cross-lingual retrieval.
+MrBERT-nos-gl is a domain-adapted encoder model obtained by continued pre-training of [BSC-LT/MrBERT](https://huggingface.co/BSC-LT/MrBERT) on [CorpusNÓS](https://huggingface.co/datasets/proxectonos/corpusnos), a large-scale Galician corpus (~1.9B tokens). It inherits MrBERT's ModernBERT architecture — with efficient long-context modeling (up to 1,024 tokens) via RoPE and sliding-window attention — and extends its coverage of Galician and Portuguese, two closely related Iberian languages underrepresented in most multilingual encoders.
+
+The model is designed as a general-purpose encoder suitable for fine-tuning on downstream tasks such as named entity recognition, part-of-speech tagging, text classification, semantic similarity, question answering, and cross-lingual retrieval. It is the foundation for the [MrBERT-nos-gl model collection](https://huggingface.co/collections/proxectonos/mrbert-nos-gl).
+
+Developed as part of **Proxecto Nós**, an initiative to build language technology for the Galician language.
 
 ## Technical Description
 
@@ -48,7 +52,6 @@ MrBERT-nos-gl starts from the MrBERT base checkpoint and continues pre-training
 | Eval MLM Mask Probability       | 15%                                                |
 | Peak Learning Rate              | 1e-5                                               |
 | LR Scheduler                    | Warmup–Stable–Decay (WSD)                          |
-| Warmup Budget                   | 3,000,000,000 tokens                               |
 | Decay Budget                    | 0 tokens (no decay phase)                          |
 | Final LR Factor (α_f)           | 0.0                                                |
 | Optimizer                       | Decoupled StableAdamW                              |
@@ -65,4 +68,47 @@ MrBERT-nos-gl starts from the MrBERT base checkpoint and continues pre-training
 | Count Padding Tokens in Budget  | No                                                 |
 | Batch Size Warmup               | From microbatch size up over 30M tokens            |
 | Attention Dropout               | 0.0 (train) / 0.1 (output projection)             |
-| Seed                            | 17                                                 |
+| Seed                            | 17                                                 |
+
+## Usage
+
+This model is a **base encoder** intended for fine-tuning, not for direct text generation. Load it with the `fill-mask` pipeline for masked language modeling, or use it as a backbone for downstream task fine-tuning.
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Masked language modeling
+
+```python
+from transformers import pipeline
+
+mlm = pipeline("fill-mask", model="proxectonos/MrBERT-nos-gl")
+
+results = mlm("A lingua galega é unha das linguas [MASK] de Europa.")
+for r in results:
+    print(f"{r['token_str']:<20} {r['score']*100:.1f}%")
+```
+
+### Feature extraction / embeddings
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("proxectonos/MrBERT-nos-gl")
+model = AutoModel.from_pretrained("proxectonos/MrBERT-nos-gl")
+
+inputs = tokenizer("A lingua galega é unha das linguas romances de Europa.", return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# Mean-pool the last hidden state for a sentence embedding
+embeddings = outputs.last_hidden_state.mean(dim=1)
+```
+
+## Acknowledgements
+
+This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project Desarrollo de Modelos ALIA. (Esta publicación del proyecto Desarrollo de Modelos ALIA está financiada por el Ministerio para la Transformación Digital y de la Función Pública y por el Plan de Recuperación, Transformación y Resiliencia – Financiado por la Unión Europea – NextGenerationEU)