ESNLIR — XLM-RoBERTa (premise only)

Diagnostic probe, not a usable NLI model. XLM-RoBERTa trained and evaluated on the premise alone, to test whether ESNLIR labels can be guessed without the hypothesis. It exists to measure annotation artifacts.

ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation Johan R. Portela, Nicolás Pérez-Terán, Rubén Manrique — Universidad de los Andes, Bogotá Applied Informatics, Springer, 2026, pp. 345–361 — doi:10.1007/978-3-032-07175-0_23

Part of the ESNLIR collection.

ESNLIR labels sentence pairs with four relations. Three come from the discourse connector linking the sentences; neutral is built by pairing sentences from different paragraphs.

id	label	meaning
0	`contrasting`	contradiction, comparison or opposition
1	`entailment`	generalization, specification or equivalence
2	`neutral`	no relation
3	`reasoning`	cause, reason or condition — the class ESNLIR adds

Results on the ESNLIR test set (80,216 pairs, class-balanced)

accuracy	macro F1	contrasting	entailment	neutral	reasoning
0.3901	0.3804	0.446	0.449	0.199	0.466

Stress tests

Four label-preserving perturbations of the test set (see the dataset card):

test	accuracy	macro F1	contrasting	entailment	neutral	reasoning
`test_length_mismatch`	0.3552	0.3297	0.199	0.444	0.153	0.624
`test_negation`	0.3901	0.3804	0.446	0.449	0.199	0.466
`test_overlap`	0.3901	0.3804	0.446	0.449	0.199	0.466
`test_spelling`	0.3824	0.3700	0.464	0.438	0.172	0.455

Why this matters

Following Gururangan et al.-style artifact probing, a partial-input model that scores well would mean the labels leak into one sentence. At 0.390 against the full model's 0.676, the gap is ~29 points, and the score sits near the XGBoost bag-of-words baseline (0.350). ESNLIR is therefore largely free of annotation artifacts: both sentences are genuinely required.

Do not use this model for inference. Use ESNLIR-XLM-RoBERTa instead.

Note the stress-test rows are largely identical to the baseline row: negation and overlap perturb only the hypothesis, which this model never sees.

Training


base model	`FacebookAI/xlm-roberta-base`
data	`Flaglab/ESNLIR-dataset`, premise only (`only_premise: true`)
max samples	1,000,000 (subsampled from the 4.4M train split, stratified)
epochs	6, early stopping on validation macro F1 (patience 3)
batch size	64
learning rate	2e-5
max sequence length	256
class weighting	balanced
seed	42

Trained with jd-rodriguezp1234/esnlir (auto_nli/model/bert_based/run.py).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "Flaglab/ESNLIR-XLM-RoBERTa-premise-only"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)

# premise only, by design
inputs = tok(
    "La evidencia que apoya la superioridad del liderazgo transformacional es abrumadora",
    return_tensors="pt", truncation=True, max_length=256,
)
with torch.no_grad():
    logits = model(**inputs).logits
print(model.config.id2label[int(logits.argmax(-1))])

The tokenizer bundled here is an unmodified copy of the one from FacebookAI/xlm-roberta-base; vocabulary size matches this model's embedding table exactly.

Citation

@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}