---
library_name: transformers
license: mit
base_model: FacebookAI/xlm-roberta-large
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: panoncology-RE-sp
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# panoncology-RE-sp

This model is a fine-tuned version of [FacebookAI/xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large).
It achieves the following results on the evaluation set:
- Loss: 0.1095
- F1 Macro: 0.9791
- F1 Weighted: 0.9878
- Precision Macro: 0.9834
- Recall Macro: 0.9749
- Accuracy: 0.9878

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 15
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Weighted | Precision Macro | Recall Macro | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:-----------:|:---------------:|:------------:|:--------:|
| 6.4623        | 1.0   | 65   | 1.5947          | 0.1778   | 0.1949      | 0.1840          | 0.2552       | 0.3520   |
| 5.4292        | 2.0   | 130  | 1.0449          | 0.6247   | 0.7446      | 0.6485          | 0.7039       | 0.7408   |
| 1.8966        | 3.0   | 195  | 0.3466          | 0.8689   | 0.9233      | 0.8803          | 0.8765       | 0.9304   |
| 0.9579        | 4.0   | 260  | 0.2411          | 0.9352   | 0.9573      | 0.9319          | 0.9387       | 0.9574   |
| 0.5551        | 5.0   | 325  | 0.1844          | 0.9308   | 0.9528      | 0.9279          | 0.9346       | 0.9536   |
| 0.5579        | 6.0   | 390  | 0.2231          | 0.9218   | 0.9431      | 0.9051          | 0.9512       | 0.9381   |
| 0.2832        | 7.0   | 455  | 0.1940          | 0.9505   | 0.9666      | 0.9511          | 0.9504       | 0.9671   |
| 0.4714        | 8.0   | 520  | 0.1696          | 0.9655   | 0.9750      | 0.9608          | 0.9705       | 0.9749   |
| 0.1448        | 9.0   | 585  | 0.2455          | 0.9573   | 0.9705      | 0.9602          | 0.9553       | 0.9710   |
| 0.1025        | 10.0  | 650  | 0.2173          | 0.9637   | 0.9749      | 0.9605          | 0.9672       | 0.9749   |
| 0.1946        | 11.0  | 715  | 0.1978          | 0.9653   | 0.9770      | 0.9593          | 0.9717       | 0.9768   |


### Framework versions

- Transformers 5.9.0
- Pytorch 2.11.0+cu128
- Datasets 4.8.5
- Tokenizers 0.22.2