---
library_name: transformers
license: apache-2.0
pipeline_tag: token-classification
base_model: vllm-sr/Vela-1.0-Encoder-307M
base_model_relation: finetune
datasets:
- KRLabsOrg/lettucedetect-code-hallucination
- KRLabsOrg/lettucedetect-prose-hallucination
tags:
- modernbert
- semantic-router
- vela
- hallucination-detection
---

<div align="center">
  <img src="https://vllm-sr.ai/img/vllm-sr-logo.social.png" alt="vLLM Semantic Router" width="560" />
  <p>
    <a href="https://vllm-sr.ai/"><strong>Docs</strong></a> |
    <a href="https://vllm-sr.ai/blog/"><strong>Blog</strong></a> |
    <a href="https://vllm-dev.slack.com/archives/C09CTGF8KCN"><strong>Slack</strong></a> |
    <a href="https://github.com/vllm-project/semantic-router"><strong>GitHub</strong></a>
  </p>
</div>

# Vela Halu

Vela Halu finds answer spans unsupported by the supplied evidence, for grounded answers and response verification.

**307M parameters · Supported input length: 8,192 tokens, including special tokens.**

Labels are `supported` (0) and `hallucinated` (1). Spans use Unicode character offsets in the answer. Evidence support is distinct from real-world factual truth; an empty result does not guarantee correctness.

Answers requiring arithmetic or multi-step reasoning beyond explicit context may be incorrectly flagged.

## Evaluation

Reported scores on the 10,698-example fixed evaluation set. Each cell is **Vela Halu / LettuceDetect**, on a 0-100 scale; higher is better. Bold Vela scores exceed the paired baseline.

| Source | Samples | Span F1 | Span P | Span R | Example F1 | IoU |
|---|---:|---:|---:|---:|---:|---:|
| ALL | 10,698 | **64.7** / 64.2 | **70.0** / 68.4 | 60.1 / 60.5 | **87.5** / 86.9 | **67.9** / 67.1 |
| ACL | 440 | **60.3** / 57.9 | **72.0** / 70.5 | **51.9** / 49.2 | 85.9 / 87.3 | **65.1** / 63.7 |
| Code agent | 2,015 | **53.0** / 50.8 | **65.5** / 61.9 | **44.6** / 43.0 | **78.3** / 77.0 | 58.0 / 58.1 |
| README | 641 | **76.7** / 75.1 | **81.8** / 78.9 | **72.2** / 71.6 | **91.5** / 90.0 | **78.4** / 76.8 |
| Tool output | 617 | **64.3** / 58.8 | **78.0** / 73.6 | **54.7** / 49.0 | **82.7** / 76.3 | **70.3** / 64.5 |
| Wikipedia | 1,388 | **73.4** / 70.8 | **77.1** / 74.1 | **70.1** / 67.8 | **93.0** / 91.7 | **79.1** / 76.8 |
| PsiloQA (multilingual) | 2,897 | 70.8 / 71.4 | **70.5** / 69.6 | 71.1 / 73.3 | 94.1 / 94.3 | 62.5 / 62.7 |
| RAGTruth | 2,700 | 52.5 / 52.8 | **67.4** / 66.8 | 43.1 / 43.7 | **74.9** / 74.3 | **72.8** / 72.4 |

LettuceDetect scores are from its [published model card](https://huggingface.co/KRLabsOrg/lettucedect-v2-mmbert-base/blob/8831ce063dff760b01efe4519b4188f5ee2456f2/README.md#performance), rounded to three decimal places on the 0-1 scale before conversion.

Span metrics measure character overlap; example F1 measures whether an answer contains any hallucinated span. IoU is mean character-set intersection over union. Vela evaluation uses bfloat16, an 8,192-token limit, `only_first` pair truncation, and token scores strictly above 0.5. [Full scores](./scores.json) include all source groups and truncation statistics.

## Quick start

With PyTorch and Transformers 4.57.6:

```python
import torch
from transformers import AutoConfig, AutoModelForTokenClassification, AutoTokenizer

model_id = "vllm-sr/Vela-1.0-Encoder-307M-Halu"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
config = AutoConfig.from_pretrained(model_id)
config.reference_compile = False
model = AutoModelForTokenClassification.from_pretrained(
    model_id, config=config, attn_implementation="sdpa"
).eval()

context = "The museum opens at 10:00 on Tuesday."
question = "When does the museum open on Tuesday?"
answer = "The museum opens at 09:00 on Tuesday."
prompt = f"User request: {question}\n\n{context}"
inputs = tokenizer(prompt, answer, return_offsets_mapping=True,
                   return_tensors="pt", truncation=False)
assert inputs.input_ids.shape[1] <= 8192
sequence_ids = inputs.sequence_ids(0)
offsets = inputs.pop("offset_mapping")[0].tolist()
with torch.inference_mode():
    scores = model(**inputs).logits.float().softmax(-1)[0, :, 1].tolist()

spans, current = [], None
for sequence, (start, end), score in zip(sequence_ids, offsets, scores):
    if sequence != 1 or end <= start:
        continue
    if score > 0.5:
        if current is None:
            current = {"start": start, "end": end}
        else:
            current["end"] = max(current["end"], end)
    elif current is not None:
        spans.append(current)
        current = None
if current is not None:
    spans.append(current)
print([{**span, "text": answer[span["start"]:span["end"]]} for span in spans])
```

Keep the complete evidence, request and answer within the supported input length. The example checks this limit before inference.

[Explore the Vela model collection](https://huggingface.co/collections/vllm-sr/vela-10-router-models-6aa555ba70cc6997d6d67798)