jorgeortizfuentes/chilean-spanish-judgment-subtypes-flair-tulio

Judgment Subtypes — Flair BiLSTM-CRF (Chilean Spanish)

This model tags the five Judgment subtypes of Appraisal Theory in Systemic Functional Linguistics: normality, capacity and tenacity under Social Esteem, veracity and propriety under Social Sanction. It is a Flair BiLSTM-CRF tagger over stacked embeddings: Spanish fastText vectors, the es-forward and es-backward contextual string embeddings, and TULIO with first-subtoken pooling, fine-tuned on the Chilean Spanish Attitude Corpus, and it is the flair entry for Judgment Subtypes in Table 3 of the accompanying paper. The released weights are the seed-42 run whose predictions produce the value printed there.

Content warning. The training corpus was collected around episodes of political conflict in Chile and contains insults, hate speech and threats. The model was fitted on that language and will reproduce its distribution.

Labels

The model emits BIO token labels over the classes capacity, normality, propriety, tenacity, veracity, plus O. A span is a maximal contiguous run of the same non-O class.

Usage

from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load("jorgeortizfuentes/chilean-spanish-judgment-subtypes-flair-tulio")

sentence = Sentence("Ese diputado es un sinvergüenza y el proyecto es lamentable.")
tagger.predict(sentence)
for span in sentence.get_spans("ner"):
    print(span.text, span.get_label("ner").value, span.score)

The tagger emits BIO tags under the internal label type ner, which is the name Flair used during training; the labels themselves are the Attitude categories, not named entities. Flair applies its own tokenizer to a raw string. To reproduce the reported scores, build the Sentence from the corpus tokens with Sentence(tokens) instead of from raw text.

Evaluation

Scores are strict span level: a predicted span counts only when its class and both boundaries match the gold span. They come from the 382-text test split of the corpus, recomputed from the saved predictions of this run.

Quantity	Value
Observed run (seed 42), micro F1	0.356
Observed run, micro precision / recall	0.435 / 0.301
Three retrainings (seeds 1, 2, 3), mean ± SD	0.356 ± 0.016
Per-seed F1	0.339, 0.371, 0.356
Expert span-level agreement on this task	0.600

The three retrainings change only the seed. Their spread describes run-to-run variation of the recipe, not the uncertainty of this checkpoint, and it is not a significance test. The released weights are the seed-42 run, so its F1 is the one this file reproduces.

Every score stays below expert agreement. The task is not solved.

Per class

Label	Precision	Recall	F1	Gold spans
capacity	0.433	0.349	0.386	129
normality	0.333	0.119	0.176	67
propriety	0.452	0.434	0.442	226
tenacity	0.333	0.036	0.065	56
veracity	0.000	0.000	0.000	30

Training data

The corpus holds 2,546 Chilean Spanish texts annotated by three linguists trained in SFL, split 1,782 / 382 / 382. It is gated: the source texts are public posts whose authors did not consent to redistribution, so access is granted for non-commercial research under the corpus LICENSE. The weights released here are not gated; they do not redistribute the texts.

The corpus is dominated by tweets (2,420 of 2,546 texts). Letters to the editor, opinion columns and consumer complaints are present in small numbers.

Training configuration

Base model: dccuchile/tulio-chilean-spanish-bert (DOI 10.57967/hf/1846).

Parameter	Value
label_type	subtypes_jugdment_tags
embedding_type	combined
bert_model	dccuchile/tulio-chilean-spanish-bert
max_epochs	100
learning_rate	0.05
bert_learning_rate	3e-05
mini_batch_size	8
eval_batch_size	8
patience	3
anneal_factor	0.5
hidden_size	256
seed	42
use_crf	True
label_encoding	BIO
best_model_metric	strict_span_level_micro_f1

Validation scores of this run: accuracy 0.9108, f1 0.3428, precision 0.4462, recall 0.2783.

Intended use

Research on Appraisal, span-level sequence labeling and evaluative language in Chilean Spanish. The fixed splits allow comparison with the paper.

It is not fit for moderating users, profiling individuals, making decisions about people, or as a general-purpose sentiment classifier. Attitude analysis identifies evaluative language and its category; it does not identify who is right, and a Judgment label is not a finding about the person being judged.

Limitations

Scores are far below what sequence labeling reaches on more established tasks, and below the agreement of the experts who annotated the data. Judgment Subtypes is the harder of the two tasks for every architecture tested.

The model was trained on Chilean Spanish from 2010–2022, mostly Twitter. Under Twitter-only training, both supervised architectures in the paper score lower outside Twitter, particularly on Judgment Subtypes; cross-genre transfer is not established. Performance on other varieties of Spanish is untested.

Provenance

Field	Value
Task column in the corpus	subtypes_jugdment_tags
Hyperparameter set	combined2_aggressive_bert
Seed	42
Selection	best validation F1 during training
Predictions behind the reported score	results/flair/flair_combined_dccuchile_tulio-chilean-spanish-bert_subtypes_jugdment_tags_combined2_aggressive_bert_predictions.parquet in the code repository

File	Bytes	SHA-256
pytorch_model.bin	1,938,413,483	85a9ac8324ad9df2…

Citation

@inproceedings{ortizfuentes-attitude-2026,
  title     = {Attitude Analysis in Systemic Functional Linguistics: A New Corpus and Benchmark Task},
  author    = {Ortiz-Fuentes, Jorge and Bravo-Marquez, Felipe and Quiroz, Beatriz},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  note      = {To appear}
}

Cite TULIO as well when you use this model; the encoder is dccuchile/tulio-chilean-spanish-bert.

License

The fine-tuned weights are released under CC BY 4.0, the license declared by the TULIO encoder they derive from. The training corpus keeps its own license and access conditions.