---
language: en
tags:
  - token-classification
  - hallucination-detection
  - modernbert
  - rag
  - ragtruth
base_model: answerdotai/ModernBERT-large
library_name: transformers
pipeline_tag: token-classification
model_name: raghal-modernbert-large-en-v1
---

# raghal-modernbert-large-en-v1

**RAGHal** — token-level hallucination detector for Retrieval-Augmented Generation (RAG).  
Trained from [answerdotai/ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) with a newly initialized token-classification head.

## Overview

The model marks answer tokens that are **not supported** by the given context. Predictions are aggregated into character spans of hallucinated text.

ModernBERT supports long context (up to **8192** tokens), so full RAG contexts can usually be scored in one pass.

## Model details

| | |
|---|---|
| **Name** | `raghal-modernbert-large-en-v1` |
| **Architecture** | ModernBERT-large token classification (2 labels: clean / hallucinated) |
| **Base model** | [answerdotai/ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) |
| **Initialization** | Pretrained encoder + newly initialized classifier head |
| **Max context** | 8192 tokens |
| **Language** | English |
| **Tasks** | Hallucination / attribution detection for RAG answers |

### Training data (auto-annotated, no human train labels)

This model is **not** trained on RAGTruth human span labels.  
We keep the original RAGTruth **train answers and prompts**, and produce **automatic** token-level hallucination spans with our own annotation stack. Human labels are used only for **evaluation** (RAGTruth test).

**Corpus (answers unchanged):**

| Item | Detail |
|---|---|
| Source | [RAGTruth](https://github.com/ParticleMedia/RAGTruth) **train** split |
| Responses | Original answers from RAGTruth generators: GPT-4, GPT-3.5-turbo, Mistral-7B-Instruct, Llama-2-7B/13B/70B-chat (**no re-generation**) |
| Inputs | 15 090 responses (2 515 sources × 6 generators) |
| Prompt | Original RAGTruth `source_info` prompt (QA / Summary / Data2txt) |
| Train labels | Auto-annotated spans only |
| Final train size | **14 633** samples after postprocessing |

**Tasks in the final train set:**

| Task | Task definition | Samples |
|---|---|---:|
| **QA** | Answer from retrieved passages; mark unsupported answer spans | 4 925 |
| **Summary** | Summarize a document; mark unsupported summary spans | 4 758 |
| **Data2txt** | Generate text from structured JSON; mark unsupported claims | 4 950 |
| Validation | RAGTruth **human** test set | 2 700 |

#### Automatic annotation pipeline

Faithfulness labeling (unsupported / contradicts **SOURCE**), not open-world factuality. Spans are written as `[HAL]…[/HAL]` tags, then converted to character offsets.

```
RAGTruth train responses
        │
        ▼
 GPT-OSS-120B  Pass 1   (T=0.6)
   task-specific system + few-shot prompts
        │
        ▼
 GPT-OSS-120B  Critic   (T=0.3)
   removal-only: drop false-positive tags, never add new ones
        │
        ├─ Summary only ──► DeBERTa-large-MNLI filter (entailment thr=0.5)
        │
        ▼
 Postprocess (snap spans to word boundaries; Data2txt: merge adjacent spans)
        │
        ▼
 Token-classification JSON  (prompt, answer, char-span labels)
```

| Stage | Tool / model | Role |
|---|---|---|
| Pass 1 annotator | `openai/gpt-oss-120b` (vLLM) | Propose hallucinated spans |
| Critic | same model, stricter prompt | Remove over-tagged spans |
| NLI filter (Summary) | `microsoft/deberta-large-mnli` | Drop spans entailed by the source document |
| Inference runtime | vLLM, multi-GPU | Batch annotation |
| Span postprocess | custom rules | Word-boundary snap; merge adjacent Data2txt spans |

**Task-specific annotation configs:**

| Task | Prompt pack | Critic | NLI |
|---|---|---|---|
| **QA** | system + 5 human-gold few-shots (refusal / synthesis / contradiction) | yes | no |
| **Summary** | system + few-shots | yes | yes (DeBERTa-MNLI @ 0.5) |
| **Data2txt** | system + aligned Data2txt rules (null fields, subjective descriptors) | full re-annot (pass1 → critic) | no |

Final mix = QA (system) + Summary (system + critic + NLI) + Data2txt (aligned critic re-annotation).  
**No manual span editing** on the train set.

### Training hyperparameters

| Parameter | Value |
|---|---|
| Optimizer | AdamW |
| Peak learning rate | 1e-5 |
| LR schedule | warmup ratio 0.05 + cosine |
| Batch size | 8 (DataParallel, 2× GPU) |
| Gradient accumulation | 1 |
| Max epochs | 10 |
| Eval | 2× / epoch |
| Early stopping | patience 3 validations, min 6 epochs |
| Class weights | disabled (uniform CE) |
| Stopped at | ~8.0 epochs |
| Best val metric | example-level Hal F1 |

## Usage

```bash
pip install transformers torch
```

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "YOUR_ORG/raghal-modernbert-large-en-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)
```

Format inputs as `prompt + answer` (context and question in the prompt; answer is the sequence to label) and aggregate token predictions into character spans.

## Performance

Values: **precision / recall / F1 (%)**.  
Evaluated on [RAGTruth](https://github.com/ParticleMedia/RAGTruth) test (2700) and zero-shot [PsiloQA](https://huggingface.co/datasets/s-nlp/PsiloQA) English test (1098).

### RAGTruth test — example-level

| Task | P | R | F1 |
|---|---:|---:|---:|
| QA | 71.43 | 62.50 | **66.67** |
| Summary | 61.40 | 51.47 | **56.00** |
| Data2txt | 89.12 | 87.74 | **88.42** |
| **Whole** | **80.93** | **75.61** | **78.18** |

### RAGTruth test — span-level

| Task | P | R | F1 |
|---|---:|---:|---:|
| QA | 70.54 | 55.34 | **62.02** |
| Summary | 64.83 | 31.82 | **42.68** |
| Data2txt | 54.16 | 54.61 | **54.38** |
| **Whole** | **61.29** | **50.11** | **55.13** |

### PsiloQA English test (zero-shot)

| Metric | Value |
|---|---:|
| AP | **76.32%** |
| IoU | **51.02%** |

### Comparison

| Benchmark | **raghal** (this) | [lettucedect-large](https://huggingface.co/KRLabsOrg/lettucedect-large-modernbert-en-v1) | ModernBERT-large SFT on [PsiloQA](https://huggingface.co/datasets/s-nlp/PsiloQA) en |
|---|---:|---:|---:|
| RAGTruth ex F1 (whole) | 78.18 | **79.22** | 57.09 |
| RAGTruth span F1 (whole) | 55.13 | **58.93** | 23.58 |
| PsiloQA AP | 76.32 | 71.72 | **83.88** |
| PsiloQA IoU | 51.02 | 47.13 | **67.23** |

The PsiloQA column is the same ModernBERT-large architecture trained only on PsiloQA English train (in-domain on PsiloQA, poor transfer to RAGTruth).

## Limitations

- English-only.
- Train labels are **automatic** (LLM teacher + critic + optional NLI), not human gold — residual annotation noise is possible.
- Tuned for RAGTruth-style QA / Summary / Data2txt prompts.
- Summary span recall is the weakest subtask.
- Not a multilingual detector.

## Citation

```bibtex
@inproceedings{modernbert,
  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author={Warner, Benjamin and others},
  booktitle={ACL},
  year={2025}
}
```

```bibtex
@inproceedings{nie2024ragtruth,
  title={RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models},
  author={Nie, Fuxiang and Yao, Yufeng and Zhu, Jingheng and others},
  booktitle={ACL},
  year={2024},
}
```