---
license: mit
language:
- en
metrics:
- accuracy
- mse
- f1
base_model:
- dmis-lab/biobert-base-cased-v1.2
- google-bert/bert-base-cased
- kelingwang/bert-causation-rating-pubmed
pipeline_tag: text-classification
model-index:
- name: bert-causation-rating-dgt
  results:
  - task:
      type: text-classification
    dataset:
      name: rating_dgt
      type: dataset
    metrics:
    - name: off by 1 accuracy
      type: accuracy
      value: 76.92307692307693
    - name: mean squared error for ordinal data
      type: mse
      value: 0.9230769230769231
    - name: weighted F1 score
      type: f1
      value: 0.7767905245021951
    - name: Kendall's tau coefficient
      type: Kendall's tau
      value: 0.7686967495426562
    source:
      name: Keling Wang
      url: https://github.com/Keling-Wang
datasets:
- kelingwang/causation_strength_rating
library_name: transformers
---
# Model description
This `bert-causation-rating-dgt` model is a fine-tuned [biobert-base-cased-v1.2](https://huggingface.co/dmis-lab/biobert-base-cased-v1.2) model on a small set of manually annotated texts with causation labels. This model is tasked with classifying a sentence into different levels of strength of causation expressed in this sentence.
Before tuning on this dataset, the `biobert-base-cased-v1.2` model is fine-tuned on a dataset containing causation labels from a published paper. This model starts from pre-trained [`kelingwang/bert-causation-rating-pubmed`](https://huggingface.co/kelingwang/bert-causation-rating-pubmed). For more information please view the link and my [GitHub page](https://github.com/Keling-Wang/causation_rating).
The sentences in the dataset were rated independently by at least two researchers and any discrepancies were resolved by the two researchers or with a third one.

# Intended use and limitations

This model is primarily used to rate for the strength of expressed causation in a sentence extracted from a clinical guideline in the field of diabetes mellitus management. 
This model predicts strength of causation (SoC) labels based on the text inputs as: 
 * -1: No correlation or variable relationships mentioned in the sentence.
 * 0: There is correlational relationships but not causation in the sentence.
 * 1: The sentence expresses weak causation.
 * 2: The sentence expresses moderate causation.
 * 3: The sentence expresses strong causation.
*NOTE:* The model output is five one-hot logits and will be 0-index based, and the labels will be 0 to 4. It is good to use [this `python` module](https://github.com/Keling-Wang/causation_rating/blob/main/tests/prediction_from_pretrained.py) if one wants to make predictions.

# Performance and hyperparameters
## Test metrics
This model achieves the following results on the test dataset. The test dataset is a 25% held-out stratified split of the entire dataset with `SEED=114514`.
 * Loss: 7.3786
 * Off-by-1 accuracy: 76.92
 * Off-by-2 accuracy: 91.61
 * MSE for ordinal data: 0.9231
 * Weighted F1: 0.7768
 * Kendall's Tau: 0.7687

## Hyperparameter tuning
   
This performance is achieved with the following hyperparameters:
 * Learning rate: 6.67197e-05
 * Weight decay: 0.048376
 * Warmup ratio: 0.357855
 * Power of polynomial learning rate scheduler: 1.056534
 * Power to the distance measure used in the loss function \alpha: 2.25

# Training settings

The following training configurations apply:
 * Pre-trained model: `kelingwang/bert-causation-rating-pubmed`
 * `seed`: 114514
 * `batch_size`: 128
 * `epoch`: 6
 * `max_length` in `torch.utils.data.Dataset`: 128
 * Loss function: the [OLL loss](https://aclanthology.org/2022.coling-1.407/) with a tunable hyperparameter \alpha (Power to the distance measure used in the loss function).
 * `lr`: 6.67197e-05
 * `weight_decay`: 0.048376
 * `warmup_ratio`: 0.357855
 * `lr_scheduler_type`: polynomial
 * `lr_scheduler_kwargs`: `{"power": 1.056534, "lr_end": 1e-8}`
 * Power to the distance measure used in the loss function \alpha: 2.25

# Framework versions and devices

This model is run on a NVIDIA P100 CPU provided by Kaggle. 
Framework versions are: 
 * python==3.10.14
 * cuda==12.4
 * NVIDIA-SMI==550.90.07
 * torch=2.4.0
 * transformers==4.45.1
 * scikit-learn==1.2.2
 * optuna==4.0.0
 * nlpaug==1.1.11