# ModernBERT-base-32k Hallucination Detector

A hallucination detection model fine-tuned on RAGTruth dataset using extended 32K context ModernBERT.

## Model Description

This model detects hallucinations in LLM-generated text by classifying each token as either **Supported** (grounded in context) or **Hallucinated** (not supported by context).

### Key Features

- **32K Context Window**: Built on [`llm-semantic-router/modernbert-base-32k`](https://huggingface.co/llm-semantic-router/modernbert-base-32k) with YaRN RoPE scaling
- **Token-Level Classification**: Identifies specific spans that are hallucinated
- **RAG Optimized**: Trained on RAGTruth benchmark for RAG applications

## Performance

| Metric | This Model | LettuceDetect BASE | LettuceDetect LARGE |
|--------|------------|-------------------|---------------------|
| **Example-Level F1** | **77.49%** | 75.99% | 79.22% |
| Token-Level F1 | 51.47% | 56.27% | - |

**Beats LettuceDetect BASE** while supporting 4x longer context (32K vs 8K tokens).

## Usage

```python
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_name = "llm-semantic-router/modernbert-base-32k-haldetect"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

# Format: context + question + answer
text = """Context: The Eiffel Tower is located in Paris, France.
Question: Where is the Eiffel Tower?
Answer: The Eiffel Tower is located in London, England."""

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=8192)
outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)

# 0 = Supported, 1 = Hallucinated
```

### With LettuceDetect Library

```python
from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(
    method="transformer",
    model_path="llm-semantic-router/modernbert-base-32k-haldetect"
)

context = "The Eiffel Tower is located in Paris, France."
question = "Where is the Eiffel Tower?"
answer = "The Eiffel Tower is located in London, England."

spans = detector.predict(context, question, answer)
# Returns: [{"text": "London, England", "start": 35, "end": 50, "confidence": 0.95}]
```

## Training Details

### Dataset
- **RAGTruth**: ~13,500 samples (QA, Data-to-Text, Summarization)
- Train/Dev/Test split from original RAGTruth

### Configuration
```yaml
base_model: llm-semantic-router/modernbert-base-32k
max_length: 8192
batch_size: 8
learning_rate: 1e-5
epochs: 6
loss: CrossEntropyLoss
scheduler: None (constant LR)
```

### Hardware
- AMD MI300X GPU (196GB VRAM)
- Training time: ~20 minutes

## Limitations

- Trained primarily on English text
- Best performance on RAG-style prompts (context + question + answer format)
- Token-level F1 is lower than example-level F1

## Citation

```bibtex
@misc{modernbert-32k-haldetect,
  title={ModernBERT-base-32k Hallucination Detector},
  author={llm-semantic-router},
  year={2025},
  url={https://huggingface.co/llm-semantic-router/modernbert-base-32k-haldetect}
}
```

## Acknowledgments

- Built on [LettuceDetect](https://github.com/KRLabsOrg/LettuceDetect) framework
- Uses [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) architecture
- Trained on [RAGTruth](https://github.com/ParticleMedia/RAGTruth) dataset