jorgeortizfuentes/chilean-spanish-judgment-subtypes-bert-tulio

Judgment Subtypes — TULIO token classifier (Chilean Spanish)

This model tags the five Judgment subtypes of Appraisal Theory in Systemic Functional Linguistics: normality, capacity and tenacity under Social Esteem, veracity and propriety under Social Sanction. It is a TULIO encoder with a token-classification head, fine-tuned on the Chilean Spanish Attitude Corpus, and it is the transformers entry for Judgment Subtypes in Table 3 of the accompanying paper. The released weights are the seed-42 run whose predictions produce the value printed there.

Content warning. The training corpus was collected around episodes of political conflict in Chile and contains insults, hate speech and threats. The model was fitted on that language and will reproduce its distribution.

Labels

The model emits flat token labels over the classes capacity, normality, propriety, tenacity, veracity, plus O. A span is a maximal contiguous run of the same non-O class.

Usage

from transformers import pipeline

tagger = pipeline(
    "token-classification",
    model="jorgeortizfuentes/chilean-spanish-judgment-subtypes-bert-tulio",
    aggregation_strategy="simple",
)
print(tagger("Ese diputado es un sinvergüenza y el proyecto es lamentable."))

The labels are flat token classes without BIO prefixes, so a span is a maximal contiguous run of the same non-O class; aggregation_strategy="simple" reproduces that grouping. Training and evaluation used pre-tokenized words (is_split_into_words=True, max_length=512), with the label of a word on its first sub-token. To reproduce the reported scores, tokenize the corpus tokens field the same way and read the prediction of each word's first sub-token.

Evaluation

Scores are strict span level: a predicted span counts only when its class and both boundaries match the gold span. They come from the 382-text test split of the corpus, recomputed from the saved predictions of this run.

Quantity	Value
Observed run (seed 42), micro F1	0.363
Observed run, micro precision / recall	0.342 / 0.388
Three retrainings (seeds 1, 2, 3), mean ± SD	0.346 ± 0.012
Per-seed F1	0.335, 0.358, 0.344
Expert span-level agreement on this task	0.600

The three retrainings change only the seed. Their spread describes run-to-run variation of the recipe, not the uncertainty of this checkpoint, and it is not a significance test. The released weights are the seed-42 run, so its F1 is the one this file reproduces.

Every score stays below expert agreement. The task is not solved.

Per class

Label	Precision	Recall	F1	Gold spans
capacity	0.383	0.442	0.410	129
normality	0.260	0.299	0.278	67
propriety	0.377	0.456	0.413	226
tenacity	0.197	0.214	0.205	56
veracity	0.312	0.167	0.217	30

Training data

The corpus holds 2,546 Chilean Spanish texts annotated by three linguists trained in SFL, split 1,782 / 382 / 382. It is gated: the source texts are public posts whose authors did not consent to redistribution, so access is granted for non-commercial research under the corpus LICENSE. The weights released here are not gated; they do not redistribute the texts.

The corpus is dominated by tweets (2,420 of 2,546 texts). Letters to the editor, opinion columns and consumer complaints are present in small numbers.

Training configuration

Base model: dccuchile/tulio-chilean-spanish-bert (DOI 10.57967/hf/1846).

Parameter	Value
label_type	subtypes_jugdment_tags
model_name	dccuchile/tulio-chilean-spanish-bert
max_epochs	10
learning_rate	2e-05
per_device_train_batch_size	16
per_device_eval_batch_size	32
effective_batch_size	16
weight_decay	0.01
warmup_steps	500
warmup_ratio	0.1
evaluation_strategy	epoch
eval_steps	500
patience	3
max_length	512
gradient_accumulation_steps	1
fp16	False
bf16	False
max_grad_norm	1.0
lr_scheduler_type	linear
label_smoothing_factor	0.1
adam_epsilon	1e-08
adam_beta1	0.9
adam_beta2	0.999
seed	42
dataloader_num_workers	0
hidden_dropout_prob	None
attention_probs_dropout_prob	None
save_total_limit	2
logging_steps	100

Validation scores of this run: accuracy 0.9060, f1 0.3309, precision 0.3185, recall 0.3442.

Intended use

Research on Appraisal, span-level sequence labeling and evaluative language in Chilean Spanish. The fixed splits allow comparison with the paper.

It is not fit for moderating users, profiling individuals, making decisions about people, or as a general-purpose sentiment classifier. Attitude analysis identifies evaluative language and its category; it does not identify who is right, and a Judgment label is not a finding about the person being judged.

Limitations

Scores are far below what sequence labeling reaches on more established tasks, and below the agreement of the experts who annotated the data. Judgment Subtypes is the harder of the two tasks for every architecture tested.

The model was trained on Chilean Spanish from 2010–2022, mostly Twitter. Under Twitter-only training, both supervised architectures in the paper score lower outside Twitter, particularly on Judgment Subtypes; cross-genre transfer is not established. Performance on other varieties of Spanish is untested.

Provenance

Field	Value
Task column in the corpus	subtypes_jugdment_tags
Hyperparameter set	set1_base
Seed	42
Selection	best validation F1 during training
Predictions behind the reported score	results/transformers/dccuchile_tulio-chilean-spanish-bert_subtypes_jugdment_tags_set1_base_predictions.parquet in the code repository

File	Bytes	SHA-256
config.json	922	1e126a2186684f11…
model.safetensors	437,082,952	e06d8b81b7be3c7a…
special_tokens_map.json	695	5d5b662e421ea9fa…
tokenizer.json	729,619	7853d7e22cc6f760…
tokenizer_config.json	1,294	be7d13e2311a8fe6…
vocab.txt	241,796	b8f1c939e21273bd…

Citation

@inproceedings{ortizfuentes-attitude-2026,
  title     = {Attitude Analysis in Systemic Functional Linguistics: A New Corpus and Benchmark Task},
  author    = {Ortiz-Fuentes, Jorge and Bravo-Marquez, Felipe and Quiroz, Beatriz},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  note      = {To appear}
}

Cite TULIO as well when you use this model; the encoder is dccuchile/tulio-chilean-spanish-bert.

License

The fine-tuned weights are released under CC BY 4.0, the license declared by the TULIO encoder they derive from. The training corpus keeps its own license and access conditions.