Release CardioBERTa-based clinical concept disambiguation model

README.md

@@ -0,0 +1,91 @@
+---
+language:
+- sv
+library_name: transformers
+pipeline_tag: feature-extraction
+base_model: DT4H/CardioBERTa.sv
+tags:
+- biomedical
+- clinical
+- cardiology
+- entity-linking
+- concept-normalization
+- umls
+- metric-learning
+---
+
+# DT4H_CardioBERTa_sv_translations_only
+
+`DT4H_CardioBERTa_sv_translations_only` is a Swedish biomedical terminology encoder for **clinical concept normalization and entity linking**. It is initialized from [`DT4H/CardioBERTa.sv`] and specialized using CUI-supervised terminology pairs and metric learning.
+
+## Backbone
+
+The backbone belongs to the **CardioBERTa** family from *CardioLM - a multilingual suite of small language models for the cardiology domain*. CardioBERTa comprises language-specific encoder models adapted to cardiology through continued pretraining on monolingual biomedical and cardiology-related corpora using Masked Language Modeling (MLM). The family covers Czech, Dutch, English, Italian, Romanian, Spanish and Swedish.
+
+## Training
+
+| | |
+|---|---|
+| Language | Swedish (`sv`) |
+| Triplet collection | `translations_only` |
+| Strategy | `synonyms` |
+| Objective | Multi-Similarity Loss |
+| Mining | All triplets, margin 0.2 |
+| Pooling | CLS |
+| Epochs | 1 |
+| Batch size | 256 |
+| Learning rate | 2e-5 |
+| Max. length | 25 |
+
+CUI-supervised synonym pairs.
+
+### Terminology statistics
+
+| Strategy | Triplets | CUIs | Unique terms | Unique positives | Terms/CUI | Δ terms |
+|---|---:|---:|---:|---:|---:|---:|
+| synonyms | 71,919 | 71,919 | 141,369 | 71,328 | 2.00 | 0 |
+| parents | 1,609,325 | 476,238 | 531,269 | 406,618 | 3.94 | +389,900 |
+| grandparents | 4,725,159 | 476,971 | 531,556 | 460,891 | 9.80 | +390,187 |
+
+This model uses **71,919 triplets**, covering **71,919 CUIs** and **141,369 unique normalized terms**.
+
+The training terminology is **not distributed** with this repository because it contains resources subject to UMLS licensing conditions. Only aggregate statistics are released.
+
+## Intended use
+
+The model is intended for terminology embedding, biomedical candidate retrieval, concept normalization and entity linking, particularly in cardiology and clinical NLP pipelines. It is not intended for direct clinical decision-making.
+
+## Usage
+
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoModel, AutoTokenizer
+
+model_id = "DT4H/DT4H_CardioBERTa_sv_translations_only"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModel.from_pretrained(model_id)
+
+inputs = tokenizer(
+    "clinical concept",
+    return_tensors="pt",
+    truncation=True,
+    max_length=25,
+)
+
+with torch.no_grad():
+    output = model(**inputs)
+
+embedding = F.normalize(
+    output.last_hidden_state[:, 0, :],
+    p=2,
+    dim=1,
+)
+```
+
+## Reference
+
+**Danu et al.** *CardioLM - a multilingual suite of small language models for the cardiology domain*.
+
+Developed within the **DataTools4Heart (DT4H)** project, Grant Agreement 101057849.

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "/mnt/home/users/gaias_001_uma/fgd/SapBERT/models/baseline/CardioBERTa.sv",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 50325
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59362d150e06b7cd489ecd87069258c49339b1938789d4ee4c02092ffcb83b6d
+size 498786152

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": false,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_config.json

@@ -0,0 +1,22 @@
+{
+  "language": "sv",
+  "strategy": "synonyms",
+  "triplets_type": "translations_only",
+  "backbone": "DT4H/CardioBERTa.sv",
+  "training": {
+    "objective": "Multi-Similarity Loss",
+    "pairwise": true,
+    "miner": {
+      "enabled": true,
+      "type_of_triplets": "all",
+      "margin": 0.2
+    },
+    "pooling": "cls",
+    "epochs": 1,
+    "batch_size": 256,
+    "learning_rate": 2e-05,
+    "max_length": 25,
+    "random_seed": 33
+  },
+  "training_data_availability": "Training terminology is not redistributed because it contains resources subject to UMLS licensing conditions."
+}

training_data_stats.json

@@ -0,0 +1,31 @@
+{
+  "triplets": 71919,
+  "malformed_rows": 0,
+  "unique_triplets": 71919,
+  "duplicate_triplets": 0,
+  "unique_cuis": 71919,
+  "unique_anchors": 71096,
+  "unique_positives": 71328,
+  "unique_terms": 141369,
+  "unique_anchor_positive_pairs": 71821,
+  "ambiguous_terms_across_cuis": 2268,
+  "self_pairs_exact": 0,
+  "self_pairs_normalized": 37,
+  "cuis_with_multiple_terms": 71882,
+  "terms_per_cui_mean": 1.999485532334988,
+  "terms_per_cui_median": 2,
+  "terms_per_cui_p95": 2.0,
+  "terms_per_cui_max": 2,
+  "triplets_per_cui_mean": 1,
+  "triplets_per_cui_median": 1,
+  "triplets_per_cui_p95": 1.0,
+  "triplets_per_cui_max": 1,
+  "anchor_words_mean": 4.662286739248321,
+  "positive_words_mean": 5.321333722660215,
+  "cuis_common_with_synonyms": 71919,
+  "cuis_added_vs_synonyms": 0,
+  "cuis_missing_vs_synonyms": 0,
+  "terms_common_with_synonyms": 141369,
+  "terms_added_vs_synonyms": 0,
+  "terms_missing_vs_synonyms": 0
+}

triplet_comparison.csv

@@ -0,0 +1,4 @@
+strategy,triplets,unique_triplets,unique_cuis,unique_anchors,unique_positives,unique_terms,unique_anchor_positive_pairs,terms_per_cui_mean,terms_per_cui_median,cuis_added_vs_synonyms,cuis_missing_vs_synonyms,terms_added_vs_synonyms,terms_missing_vs_synonyms
+synonyms,71919,71919,71919,71096,71328,141369,71821,1.999485532334988,2,0,0,0,0
+parents,1609325,1609086,476238,389862,406618,531269,1592302,3.935698117327891,3.0,404319,0,389900,0
+grandparents,4725159,4724753,476971,461115,460891,531556,4665781,9.803621184516459,7,405052,0,390187,0