File size: 3,035 Bytes
70e59ed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
language:
- es
library_name: transformers
pipeline_tag: feature-extraction
base_model: DT4H/CardioBERTa.es
tags:
- biomedical
- clinical
- cardiology
- entity-linking
- concept-normalization
- umls
- metric-learning
---
# DT4H_CardioBERTa_grandparents_es_translations_only
`DT4H_CardioBERTa_grandparents_es_translations_only` is a Spanish biomedical terminology encoder for **clinical concept normalization and entity linking**. It is initialized from [`DT4H/CardioBERTa.es`] and specialized using CUI-supervised terminology pairs and metric learning.
## Backbone
The backbone belongs to the **CardioBERTa** family from *CardioLM - a multilingual suite of small language models for the cardiology domain*. CardioBERTa comprises language-specific encoder models adapted to cardiology through continued pretraining on monolingual biomedical and cardiology-related corpora using Masked Language Modeling (MLM). The family covers Czech, Dutch, English, Italian, Romanian, Spanish and Swedish.
## Training
| | |
|---|---|
| Language | Spanish (`es`) |
| Triplet collection | `translations_only` |
| Strategy | `grandparents` |
| Objective | Multi-Similarity Loss |
| Mining | All triplets, margin 0.2 |
| Pooling | CLS |
| Epochs | 1 |
| Batch size | 256 |
| Learning rate | 2e-5 |
| Max. length | 25 |
CUI-supervised terminology pairs enriched with grandparent-level ontology relations.
### Terminology statistics
| Strategy | Triplets | CUIs | Unique terms | Unique positives | Terms/CUI | Δ terms |
|---|---:|---:|---:|---:|---:|---:|
| synonyms | 69,277 | 69,277 | 136,233 | 68,712 | 2.00 | 0 |
| parents | 1,593,029 | 476,344 | 529,722 | 418,463 | 3.92 | +393,489 |
| grandparents | 4,701,649 | 476,968 | 530,009 | 468,324 | 9.82 | +393,776 |
This model uses **4,701,649 triplets**, covering **476,968 CUIs** and **530,009 unique normalized terms**.
The training terminology is **not distributed** with this repository because it contains resources subject to UMLS licensing conditions. Only aggregate statistics are released.
## Intended use
The model is intended for terminology embedding, biomedical candidate retrieval, concept normalization and entity linking, particularly in cardiology and clinical NLP pipelines. It is not intended for direct clinical decision-making.
## Usage
```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
model_id = "DT4H/DT4H_CardioBERTa_grandparents_es_translations_only"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)
inputs = tokenizer(
"clinical concept",
return_tensors="pt",
truncation=True,
max_length=25,
)
with torch.no_grad():
output = model(**inputs)
embedding = F.normalize(
output.last_hidden_state[:, 0, :],
p=2,
dim=1,
)
```
## Reference
**Danu et al.** *CardioLM - a multilingual suite of small language models for the cardiology domain*.
Developed within the **DataTools4Heart (DT4H)** project, Grant Agreement 101057849.
|