README.md

metadata

license: cc-by-4.0
language:
  - es
base_model: FacebookAI/xlm-roberta-base
pipeline_tag: text-classification
library_name: transformers
datasets:
  - Flaglab/ESNLIR-dataset
metrics:
  - accuracy
  - f1
tags:
  - nli
  - spanish
  - causal
  - esnlir
  - text-classification
  - artifact-detection
  - partial-input

ESNLIR — XLM-RoBERTa (premise only)

Diagnostic probe, not a usable NLI model. XLM-RoBERTa trained and evaluated on the premise alone, to test whether ESNLIR labels can be guessed without the hypothesis. It exists to measure annotation artifacts.

ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation Johan R. Portela, Nicolás Pérez-Terán, Rubén Manrique — Universidad de los Andes, Bogotá Applied Informatics, Springer, 2026, pp. 345–361 — doi:10.1007/978-3-032-07175-0_23

Part of the ESNLIR collection.

ESNLIR labels sentence pairs with four relations. Three come from the discourse connector linking the sentences; neutral is built by pairing sentences from different paragraphs.

id	label	meaning
0	contrasting	contradiction, comparison or opposition
1	entailment	generalization, specification or equivalence
2	neutral	no relation
3	reasoning	cause, reason or condition — the class ESNLIR adds

Results on the ESNLIR test set (80,216 pairs, class-balanced)

accuracy	macro F1	contrasting	entailment	neutral	reasoning
0.3901	0.3804	0.446	0.449	0.199	0.466

Stress tests

Four label-preserving perturbations of the test set (see the dataset card):

test	accuracy	macro F1	contrasting	entailment	neutral	reasoning
test_length_mismatch	0.3552	0.3297	0.199	0.444	0.153	0.624
test_negation	0.3901	0.3804	0.446	0.449	0.199	0.466
test_overlap	0.3901	0.3804	0.446	0.449	0.199	0.466
test_spelling	0.3824	0.3700	0.464	0.438	0.172	0.455

Why this matters

Following Gururangan et al.-style artifact probing, a partial-input model that scores well would mean the labels leak into one sentence. At 0.390 against the full model's 0.676, the gap is ~29 points, and the score sits near the XGBoost bag-of-words baseline (0.350). ESNLIR is therefore largely free of annotation artifacts: both sentences are genuinely required.

Do not use this model for inference. Use ESNLIR-XLM-RoBERTa instead.

Note the stress-test rows are largely identical to the baseline row: negation and overlap perturb only the hypothesis, which this model never sees.

Training

base model	FacebookAI/xlm-roberta-base
data	Flaglab/ESNLIR-dataset, premise only (only_premise: true)
max samples	1,000,000 (subsampled from the 4.4M train split, stratified)
epochs	6, early stopping on validation macro F1 (patience 3)
batch size	64
learning rate	2e-5
max sequence length	256
class weighting	balanced
seed	42

Trained with jd-rodriguezp1234/esnlir (auto_nli/model/bert_based/run.py).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "Flaglab/ESNLIR-XLM-RoBERTa-premise-only"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)

# premise only, by design
inputs = tok(
    "La evidencia que apoya la superioridad del liderazgo transformacional es abrumadora",
    return_tensors="pt", truncation=True, max_length=256,
)
with torch.no_grad():
    logits = model(**inputs).logits
print(model.config.id2label[int(logits.argmax(-1))])

The tokenizer bundled here is an unmodified copy of the one from FacebookAI/xlm-roberta-base; vocabulary size matches this model's embedding table exactly.

Citation

@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}