---
model_name: transformer_multi_head_robbertv2_lora
base_model: pdelobelle/robbert-v2-dutch-base
language: nl
library_name: transformers
tags:
  - dutch
  - regression
  - multi-head
  - robbert-v2
  - lora
  - text-quality
license: mit
datasets:
  - proprietary
metrics:
  - rmse
  - r2
pipeline_tag: text-classification
---

# transformer_multi_head_robbertv2_lora

This is a **multi-head transformer regression model** using **RobBERT-v2** with **LoRA parameter-efficient fine-tuning**, designed to predict **four separate text quality scores** for Dutch texts.

The final **aggregate metric** recomputes a combined score from the four heads and compares it to the actual aggregate, providing robust quality tracking.

---

## 📈 Training & Evaluation

| Epoch | Train Loss | Val Loss | RMSE (delta_cola_to_final) | R² (delta_cola_to_final) | RMSE (delta_perplexity_to_final_large) | R² (delta_perplexity_to_final_large) | RMSE (iter_to_final_simplified) | R² (iter_to_final_simplified) | RMSE (robbert_delta_blurb_to_final) | R² (robbert_delta_blurb_to_final) | Mean RMSE |
|-------|-------------|-----------|----------------------------|--------------------------|----------------------------------------|--------------------------------------|---------------------------------|---------------------------------|-------------------------------------|-----------------------------------|-----------|
| 1     | 0.0363      | 0.0221    | 0.1543                     | 0.3456                   | 0.1210                                 | 0.4855                               | 0.1765                          | 0.7058                          | 0.1377                              | 0.6308                           | 0.1474    |
| 2     | 0.0237      | 0.0199    | 0.1549                     | 0.3401                   | 0.1157                                 | 0.5297                               | 0.1621                          | 0.7517                          | 0.1257                              | 0.6922                           | 0.1396    |
| 3     | 0.0212      | 0.0187    | 0.1543                     | 0.3457                   | 0.1074                                 | 0.5947                               | 0.1547                          | 0.7739                          | 0.1243                              | 0.6991                           | 0.1352    |
| 4     | 0.0201      | 0.0185    | 0.1533                     | 0.3544                   | 0.1091                                 | 0.5818                               | 0.1531                          | 0.7784                          | 0.1234                              | 0.7032                           | 0.1347    |
| 5     | 0.0196      | 0.0182    | 0.1508                     | 0.3752                   | 0.1081                                 | 0.5896                               | 0.1528                          | 0.7794                          | 0.1233                              | 0.7041                           | 0.1337    |

**Final aggregate performance**  
✅ **Aggregate RMSE:** `0.0872`  
✅ **Aggregate R²:** `0.7970`

---

## 🧾 Notes

- This model uses **LoRA fine-tuning** to train only ~0.75% of RobBERT-v2’s parameters.
- It has **four parallel regression heads** for:
  - `delta_cola_to_final`
  - `delta_perplexity_to_final_large`
  - `iter_to_final_simplified`
  - `robbert_delta_blurb_to_final`
- The final test set results confirm robust performance with individual and aggregate metrics.
- Fine-tuned on a proprietary dataset of Dutch text variations.
- **Base:** RobBERT-v2 Dutch Base (`pdelobel