README.md

---
license: apache-2.0
language:
  - en
tags:
  - modernbert
  - hallucination-detection
  - rag
  - fact-checking
  - long-context
  - 32k
  - amd
  - rocm
  - mi300x
datasets:
  - vllm-sr/longcontext-haldetect
  - vllm-sr/dart-halspans
  - vllm-sr/e2e-halspans
base_model:
  - vllm-sr/modernbert-base-32k
pipeline_tag: token-classification
model-index:
  - name: modernbert-base-32k-haldetect
    results:
      - task:
          type: token-classification
          name: Hallucination Detection
        dataset:
          name: RAGTruth Test Set
          type: ragtruth
        metrics:
          - name: Example-Level F1
            type: f1
            value: 76.56
          - name: Token-Level F1
            type: f1
            value: 53.77
      - task:
          type: token-classification
          name: Long-Context Hallucination Detection
        dataset:
          name: Long-Context Benchmark (8K-24K tokens)
          type: vllm-sr/longcontext-haldetect
        metrics:
          - name: Hallucination F1
            type: f1
            value: 49.86
---

# 🥬 ModernBERT-base-32k Hallucination Detector

A hallucination detection model fine-tuned on RAGTruth dataset with Data2txt augmentation using extended 32K context ModernBERT. **Specifically designed for long documents that exceed 8K tokens.**

## 🚀 Why 32K Context Matters

| Scenario | 8K Model | 32K Model |
|----------|----------|-----------|
| 15K-token legal contract | ❌ Truncates 47% | ✅ Full context |
| Multi-document RAG | ❌ Loses evidence | ✅ Sees all docs |
| Long-form summarization | ❌ Misses details | ✅ Complete view |

## Performance

### RAGTruth Benchmark (Standard, <3K tokens)

Evaluated on RAGTruth test set (2,700 samples):

| Metric | This Model | LettuceDetect BASE | LettuceDetect LARGE |
|--------|------------|-------------------|---------------------|
| **Example-Level F1** | **76.56%** ✅ | 75.99% | 79.22% |
| Token-Level F1 | 53.77% | 56.27% | - |
| Context Window | **32K** | 8K | 8K |

✅ **Exceeds LettuceDetect BASE** on short documents while supporting **4x longer context**

### Long-Context Benchmark (8K-24K tokens)

Evaluated on [vllm-sr/longcontext-haldetect](https://huggingface.co/datasets/vllm-sr/longcontext-haldetect) (337 test samples, avg 17,550 tokens):

| Metric | 32K ModernBERT | 8K LettuceDetect | Improvement |
|--------|----------------|------------------|-------------|
| **Samples Truncated** | 0 (0%) | 320 (95%) | **-95%** |
| Hallucination Recall | 0.547 | 0.056 | **+877%** |
| **Hallucination F1** | **0.499** | 0.101 | **+393%** |

## Model Description

This model detects hallucinations in LLM-generated text by classifying each token as either **Supported** (grounded in context) or **Hallucinated** (not supported by context).

### Key Features

- **32K Context Window**: Built on [vllm-sr/modernbert-base-32k](https://huggingface.co/vllm-sr/modernbert-base-32k) with YaRN RoPE scaling
- **Token-Level Classification**: Identifies specific spans that are hallucinated
- **RAG Optimized**: Trained on RAGTruth benchmark for RAG applications
- **Data2txt Augmentation**: Enhanced with DART and E2E datasets for better structured data handling
- **Long Document Support**: Handles legal contracts, financial reports, research papers

## Usage

```python
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

model_name = "vllm-sr/modernbert-base-32k-haldetect"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(model_name, trust_remote_code=True)

# Format: context + question + answer
text = """Context: The Eiffel Tower is located in Paris, France. It was completed in 1889.
Question: Where is the Eiffel Tower and when was it built?
Answer: The Eiffel Tower is located in London, England and was completed in 1920."""

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=24000)
with torch.no_grad():
    outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)

# 0 = Supported, 1 = Hallucinated
# Tokens for "London, England" and "1920" will be marked as hallucinated
```

### With LettuceDetect Library

```python
from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(
    method="transformer",
    model_path="vllm-sr/modernbert-base-32k-haldetect",
    max_length=24000  # Use extended context
)

context = "The Eiffel Tower is located in Paris, France. It was completed in 1889."
question = "Where is the Eiffel Tower?"
answer = "The Eiffel Tower is located in London, England."

spans = detector.predict(context, question, answer)
# Returns: [{"text": "London, England", "start": 35, "end": 50, "confidence": 0.95}]
```

## Training Details

### Datasets

| Dataset | Samples | Task Type | Description |
|---------|---------|-----------|-------------|
| **[RAGTruth](https://github.com/ParticleMedia/RAGTruth)** | 17,790 | QA, Summary, Data2txt | Human-annotated hallucination spans |
| **[DART](https://huggingface.co/datasets/vllm-sr/dart-halspans)** | 2,000 | Data2txt | LLM-generated structured data responses |
| **[E2E](https://huggingface.co/datasets/vllm-sr/e2e-halspans)** | 1,500 | Data2txt | LLM-generated restaurant descriptions |
| **Total** | 21,290 | Mixed | Balanced task distribution |

The DART and E2E datasets were synthetically generated using Qwen2.5-72B-Instruct to create both faithful and intentionally hallucinated responses from structured data, then LLM-annotated for span-level hallucinations.

### Configuration

```yaml
base_model: vllm-sr/modernbert-base-32k
max_length: 8192
batch_size: 32
learning_rate: 1e-5
epochs: 6
loss: CrossEntropyLoss (weighted)
scheduler: None (constant LR)
early_stopping_patience: 4
```

### Hardware

- **AMD Instinct MI300X GPU** (192GB HBM3) - Trained entirely on AMD ROCm
- Training time: ~17 minutes (6 epochs)
- Framework: PyTorch 2.9 + HuggingFace Transformers on ROCm 7.0

## When to Use This Model

| Use Case | Recommended Model |
|----------|-------------------|
| Documents > 8K tokens | ✅ **This model** |
| Multi-document RAG | ✅ **This model** |
| Legal/Financial docs | ✅ **This model** |
| Structured data (tables, lists) | ✅ **This model** |
| Short QA (<3K tokens) | Either model works |
| Speed critical | 8K model (faster) |

## Limitations

- Trained primarily on English text
- Best performance on RAG-style prompts (context + question + answer format)
- Longer contexts require more GPU memory

## Related Resources

### Datasets
- **Long-Context Benchmark**: [vllm-sr/longcontext-haldetect](https://huggingface.co/datasets/vllm-sr/longcontext-haldetect) - 3,366 samples, 8K-24K tokens
- **DART Hallucination Spans**: [vllm-sr/dart-halspans](https://huggingface.co/datasets/vllm-sr/dart-halspans) - 2,000 Data2txt samples
- **E2E Hallucination Spans**: [vllm-sr/e2e-halspans](https://huggingface.co/datasets/vllm-sr/e2e-halspans) - 1,500 restaurant descriptions

### Models
- **Base Model**: [vllm-sr/modernbert-base-32k](https://huggingface.co/vllm-sr/modernbert-base-32k) - Extended ModernBERT
- **Combined Model**: [modernbert-base-32k-haldetect-combined](https://huggingface.co/vllm-sr/modernbert-base-32k-haldetect-combined) - Trained on RAGTruth + HaluEval

## Citation

```bibtex
@misc{modernbert-32k-haldetect,
  title={ModernBERT-32K Hallucination Detector with Data2txt Augmentation},
  author={LLM Semantic Router Team},
  year={2026},
  url={https://huggingface.co/vllm-sr/modernbert-base-32k-haldetect}
}
```

## Acknowledgments

- Built on [LettuceDetect](https://github.com/KRLabTech/LettuceDetect) framework
- Uses [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) architecture
- Trained on [RAGTruth](https://github.com/ParticleMedia/RAGTruth) dataset
- Data2txt augmentation from [DART](https://github.com/Yale-LILY/dart) and [E2E](https://github.com/tuetschek/e2e-dataset) datasets