| --- |
| license: llama3.1 |
|
|
| base_model: |
| - meta-llama/Llama-3.1-8B-Instruct |
|
|
| language: |
| - fr |
|
|
| tags: |
| - biomedical-entity-linking |
| - entity-linking |
| - entity-disambiguation |
| - named-entity-linking |
| - biomedical |
| - healthcare |
| - umls |
| - quaero |
| - emea |
| - text-generation |
| - constrained-decoding |
| - causal-lm |
| - llm |
|
|
| library_name: transformers |
| pipeline_tag: text-generation |
|
|
| datasets: |
| - AnonymousARR42/EMEA |
|
|
| finetuning_task: |
| - entity-linking |
|
|
| metrics: |
| - recall |
|
|
| model-index: |
| - name: LongBEL-8B-QUAERO-EMEA |
| results: |
| - task: |
| type: entity-linking |
| name: Biomedical Entity Linking |
| dataset: |
| type: AnonymousARR42/EMEA |
| name: QUAERO-EMEA |
| metrics: |
| - type: recall |
| name: Recall@1 |
| value: 0.754 |
| --- |
| |
| # LongBEL: Long-Context and Document-Consistent Biomedical Entity Linking |
|
|
| ## LongBEL |
|
|
| **LongBEL** is a novel document-level framework for biomedical entity linking (BEL). Instead of normalizing each mention independently, LongBEL conditions each prediction on the document context and on previous normalizations produced in the same document. This design enforces document-level consistency and is enhanced by our **robust memory** mechanism. The method is introduced in our paper, currently under review. |
|
|
| ## LongBEL (QUAERO-EMEA Edition) |
|
|
| This is a **finetuned version of LLaMA-3-8B** trained on **QUAERO-EMEA**, applying the LongBEL framework to enable long context and robust memory predictions. |
|
|
| | Field | Value | |
| |---|---| |
| | Base model | `meta-llama/Llama-3.1-8B-Instruct` | |
| | Task | Biomedical Entity Linking | |
| | Dataset | QUAERO-EMEA | |
| | Knowledge base | UMLS 2014AA | |
| | Input | BigBio-like documents with mention spans and semantic groups | |
| | Output | Ranked UMLS concept predictions | |
| | Decoding | Semantic-guided constrained decoding | |
| | Main metric | Recall@1 | |
|
|
|
|
| ## Intended Use |
|
|
| This model is intended for research on biomedical entity linking and document-level consistency. |
|
|
| It assumes that mention spans and semantic groups are already provided. It does **not** perform named entity recognition. In a full pipeline, a NER model should first detect mentions and assign semantic groups, then LongBEL can normalize these mentions to UMLS concepts. |
|
|
| ## Usage |
|
|
| ### Loading the model |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| "AnonymousARR42/LongBEL_8B_QUAERO_EMEA", |
| trust_remote_code=True, |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| ) |
| ```` |
|
|
| ### Inference example |
|
|
| The model expects BigBio-like documents. Each entity should include a mention text, character offsets, and a semantic group in the `type` field. |
|
|
| ```python |
| num_beams = 5 |
| |
| bigbio_pages = [ |
| { |
| "id": "001", |
| "document_id": "doc_001", |
| "passages": [ |
| { |
| "id": "0", |
| "type": "paragraph", |
| "text": [ |
| "Une femme enceinte de 29 ans s'est présentée avec une hypertension sévère, " |
| "des céphalées et une douleur épigastrique. Les analyses biologiques ont montré " |
| "une protéinurie et une légère élévation des enzymes hépatiques. Elle a été " |
| "hospitalisée pendant la nuit avec une suspicion de PET et un traitement urgent " |
| "a été débuté." |
| ], |
| "offsets": [[0, 321]], |
| } |
| ], |
| "entities": [ |
| { |
| "id": "T1", |
| "type": "Living Beings", |
| "text": ["femme enceinte"], |
| "offsets": [[4, 18]], |
| }, |
| { |
| "id": "T2", |
| "type": "Disorders", |
| "text": ["hypertension sévère"], |
| "offsets": [[54, 73]], |
| }, |
| { |
| "id": "T3", |
| "type": "Disorders", |
| "text": ["protéinurie"], |
| "offsets": [[158, 169]], |
| }, |
| { |
| "id": "T4", |
| "type": "Disorders", |
| "text": ["PET"], |
| "offsets": [[280, 283]], |
| }, |
| ], |
| "events": [], |
| "coreferences": [], |
| "relations": [], |
| } |
| ] |
| |
| predictions = model.sample( |
| bigbio_pages=bigbio_pages, |
| num_beams=num_beams, |
| ) |
| |
| for i in range(0, len(predictions), num_beams): |
| mention = predictions[i]["mention"] |
| print(f"## Mention {(i // num_beams) + 1}: {mention}") |
| |
| for j in range(num_beams): |
| pred = predictions[i + j] |
| print( |
| f" - Beam {j + 1}:\n" |
| f" Predicted concept name: {pred['pred_concept_name']}\n" |
| f" Predicted code: {pred['pred_concept_code']}\n" |
| f" Beam score: {pred['beam_score']:.3f}\n" |
| ) |
| ``` |
|
|
|
|
| **Example Output:** |
|
|
| ```text |
| ## Mention 1: femme enceinte |
| - Beam 1: |
| Predicted concept name: Femmes enceintes |
| Predicted code: C0033011 |
| Beam score: 1.000 |
| |
| - Beam 2: |
| Predicted concept name: Femmes (mariage) |
| Predicted code: C0242665 |
| Beam score: 0.000 |
| |
| - Beam 3: |
| Predicted concept name: Femmes en postpartum |
| Predicted code: C0032804 |
| Beam score: 0.000 |
| |
| - Beam 4: |
| Predicted concept name: Femmes en post-partum |
| Predicted code: C0032804 |
| Beam score: 0.000 |
| |
| - Beam 5: |
| Predicted concept name: Females |
| Predicted code: C0086287 |
| Beam score: 0.000 |
| |
| ## Mention 2: hypertension sévère |
| - Beam 1: |
| Predicted concept name: Hypotension, non précisée |
| Predicted code: C0020649 |
| Beam score: 0.029 |
| |
| - Beam 2: |
| Predicted concept name: Pression sanguine augmentée |
| Predicted code: C0497247 |
| Beam score: 0.007 |
| |
| - Beam 3: |
| Predicted concept name: Hypertension; encephalopathy |
| Predicted code: C1396475 |
| Beam score: 0.001 |
| |
| - Beam 4: |
| Predicted concept name: Hypertension;in pregnancy |
| Predicted code: C0565599 |
| Beam score: 0.001 |
| |
| - Beam 5: |
| Predicted concept name: Pression sanguine non mesurable |
| Predicted code: C0858911 |
| Beam score: 0.000 |
| |
| ## Mention 3: protéinurie |
| - Beam 1: |
| Predicted concept name: Protéinurie |
| Predicted code: C0033687 |
| Beam score: 1.000 |
| |
| - Beam 2: |
| Predicted concept name: Protéinurie - aggravée |
| Predicted code: C0856146 |
| Beam score: 0.002 |
| |
| - Beam 3: |
| Predicted concept name: Protéine urinaire positive |
| Predicted code: C0033687 |
| Beam score: 0.000 |
| |
| - Beam 4: |
| Predicted concept name: Protéine urinaire de Bence Jones présente |
| Predicted code: C0854075 |
| Beam score: 0.000 |
| |
| - Beam 5: |
| Predicted concept name: Protéine urinaire de Bence Jones absente |
| Predicted code: C0855589 |
| Beam score: 0.000 |
| |
| ## Mention 4: PET |
| - Beam 1: |
| Predicted concept name: Petechial hemorrhage |
| Predicted code: C0031256 |
| Beam score: 0.073 |
| |
| - Beam 2: |
| Predicted concept name: Petechial; hemorrhage |
| Predicted code: C0031256 |
| Beam score: 0.022 |
| |
| - Beam 3: |
| Predicted concept name: Petechia |
| Predicted code: C0031256 |
| Beam score: 0.008 |
| |
| - Beam 4: |
| Predicted concept name: PET - Pre-eclamptic toxaemia |
| Predicted code: C0032914 |
| Beam score: 0.001 |
| |
| - Beam 5: |
| Predicted concept name: Petechial hemorrhages |
| Predicted code: C0031256 |
| Beam score: 0.000 |
| ``` |
|
|
| ### Saliency map example |
|
|
| The model can also return token-level saliency maps during inference. |
|
|
| ```python |
| predictions, saliency_maps = model.sample( |
| bigbio_pages=bigbio_pages, |
| num_beams=num_beams, |
| with_saliency_maps=True, |
| ) |
| |
| model.display_saliency_map(saliency_maps[3]) |
| ```` |
|
|
| Example saliency map for the mention `PET`: |
|
|
| <p align="center"> |
| <img src="saliency_map.png" alt="Saliency map for PET prediction" width="900"> |
| </p> |
|
|
| ## Evaluation |
|
|
| Entity linking performance is reported using Recall@1 with bootstrap confidence intervals. The best result is shown in **bold**, and the second-best result is <u>underlined</u> and ⭐ marks the main LongBEL-8B model. |
|
|
| | Model | MM-ST21PV<br>(English) | QUAERO-EMEA<br>(French) | SympTEMIST<br>(Spanish) | DisTEMIST<br>(Spanish) | MedProcNER<br>(Spanish) | |
| | :--- | :---: | :---: | :---: | :---: | :---: | |
| | **Context-Free BEL** ||||| | |
| | SciSpacy | 53.8 ± 1.0 | 37.1 ± 4.3 | 9.8 ± 1.3 | 21.1 ± 1.9 | 10.3 ± 1.2 | |
| | SapBERT | 65.6 ± 1.0 | 59.7 ± 3.8 | 34.2 ± 2.0 | 38.6 ± 2.6 | 30.4 ± 2.1 | |
| | CODER-all | 62.9 ± 1.1 | 66.9 ± 4.0 | 42.2 ± 2.2 | 47.0 ± 2.6 | 42.7 ± 2.1 | |
| | SapBERT-all | 64.6 ± 1.1 | 67.9 ± 3.9 | 49.8 ± 2.4 | 49.6 ± 2.6 | 45.1 ± 2.2 | |
| | BERGAMOT | 60.9 ± 1.1 | 63.8 ± 4.9 | 48.0 ± 2.7 | 48.9 ± 2.4 | 42.3 ± 2.2 | |
| | **Local-Context BEL** ||||| | |
| | ArboEL | 76.9 ± 0.9 | 63.0 ± 3.9 | 55.4 ± 2.5 | 54.7 ± 2.6 | 59.7 ± 2.6 | |
| | GENRE / mBART-large | 69.6 ± 1.0 | 69.3 ± 5.4 | 59.8 ± 2.7 | 58.7 ± 2.7 | 66.0 ± 2.3 | |
| | GENRE / Llama-1B | 73.1 ± 1.0 | 75.1 ± 3.6 | 60.5 ± 2.4 | 62.5 ± 2.3 | 67.4 ± 2.1 | |
| | GENRE / Llama-8B | 75.0 ± 0.9 | 73.8 ± 4.0 | 61.7 ± 2.5 | 63.2 ± 2.5 | 68.3 ± 2.2 | |
| | **Global-Context BEL: LongBEL** ||||| | |
| | LongBEL-1B | 77.6 ± 0.9 | 74.5 ± 3.7 | 59.8 ± 2.5 | 61.9 ± 2.4 | 66.6 ± 2.1 | |
| | LongBEL-1B + Ensemble | 78.6 ± 0.8 | <u>77.2 ± 3.0</u> | 61.8 ± 2.5 | <u>64.3 ± 2.2</u> | <u>69.0 ± 2.0</u> | |
| | **⭐ LongBEL-8B** | <u>79.3 ± 0.8</u> | 75.4 ± 3.4 | <u>62.0 ± 2.6</u> | 63.6 ± 2.1 | <u>69.0 ± 2.1</u> | |
| | LongBEL-8B + Ensemble | **80.0 ± 0.8** | **77.6 ± 3.0** | **63.3 ± 2.5** | **65.8 ± 2.2** | **71.0 ± 2.0** | |
|
|
| The score reported for this checkpoint is the **single LongBEL-8B model**. The ensemble result requires fusing several LongBEL input configurations and is not produced by this checkpoint alone. |
|
|
| ## Speed and Memory |
|
|
| Measured on a single NVIDIA H100 80GB GPU. |
|
|
| | Model | Model memory | Candidate memory | Speed | |
| | ----------------------- | -----------: | ---------------: | --------------: | |
| | GENRE-Llama-8B baseline | 28.6 GB | 5.4 GB | 38.2 mentions/s | |
| | LongBEL-8B | 28.6 GB | 5.4 GB | 15.2 mentions/s | |
|
|
| LongBEL has the same model memory footprint as the sentence-level Llama-8B baseline, but it is slower because it processes longer contexts and updates document-level memory during inference. |
|
|
| ## Limitations |
|
|
| This model assumes that mention spans and semantic groups are given. It does not perform mention detection. |
|
|
| LongBEL is most useful when concepts recur within a document. When most concepts appear only once, the memory mechanism has less information to exploit. |
|
|
| Because LongBEL uses previous predictions as memory, early mistakes can still influence later predictions. Robust memory training reduces this risk but does not remove it completely. |
|
|
| This model is intended for research use. It should not be used for clinical decision-making without additional validation and human oversight. |
|
|
| ## Reproducibility |
|
|
| Code and evaluation scripts are available in this [GitHub repository](https://anonymous.4open.science/r/LongBEL-31AD). |
|
|
| Trained model checkpoints and processed datasets are available in the anonymous Hugging Face collection associated with LongBEL. |
