README.md

---
language:
  - es
language_details: es-CL
license: cc-by-4.0
library_name: flair
pipeline_tag: token-classification
base_model: dccuchile/tulio-chilean-spanish-bert
base_model_relation: finetune
datasets:
  - jorgeortizfuentes/chilean-spanish-attitude-corpus
metrics:
  - f1
  - precision
  - recall
tags:
  - appraisal-theory
  - systemic-functional-linguistics
  - attitude
  - span-classification
  - chilean-spanish
model-index:
  - name: chilean-spanish-judgment-subtypes-flair-tulio
    results:
      - task:
          type: token-classification
          name: Judgment Subtypes (strict span level)
        dataset:
          name: Chilean Spanish Attitude Corpus
          type: jorgeortizfuentes/chilean-spanish-attitude-corpus
          split: test
        metrics:
          - type: f1
            value: 0.3558
            name: Strict span-level micro F1
          - type: precision
            value: 0.4347
            name: Strict span-level micro precision
          - type: recall
            value: 0.3012
            name: Strict span-level micro recall
---

# Judgment Subtypes — Flair BiLSTM-CRF (Chilean Spanish)

This model tags the five Judgment subtypes of Appraisal Theory in Systemic Functional Linguistics: normality, capacity and tenacity under Social Esteem, veracity and propriety under Social Sanction. It is a Flair BiLSTM-CRF tagger over stacked embeddings: Spanish fastText vectors, the `es-forward` and `es-backward` contextual string embeddings, and TULIO with first-subtoken pooling,
fine-tuned on the [Chilean Spanish Attitude Corpus](https://huggingface.co/datasets/jorgeortizfuentes/chilean-spanish-attitude-corpus),
and it is the `flair` entry for *Judgment Subtypes* in Table 3 of the
accompanying paper. The released weights are the seed-42 run whose predictions
produce the value printed there.

> **Content warning.** The training corpus was collected around episodes of
> political conflict in Chile and contains insults, hate speech and threats. The
> model was fitted on that language and will reproduce its distribution.

## Labels

The model emits `BIO` token labels over the classes
`capacity`, `normality`, `propriety`, `tenacity`, `veracity`, plus `O`. A span is a maximal contiguous run of the same
non-`O` class.

## Usage

```python
from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load("jorgeortizfuentes/chilean-spanish-judgment-subtypes-flair-tulio")

sentence = Sentence("Ese diputado es un sinvergüenza y el proyecto es lamentable.")
tagger.predict(sentence)
for span in sentence.get_spans("ner"):
    print(span.text, span.get_label("ner").value, span.score)
```

The tagger emits BIO tags under the internal label type `ner`, which is the
name Flair used during training; the labels themselves are the Attitude
categories, not named entities. Flair applies its own tokenizer to a raw
string. To reproduce the reported scores, build the `Sentence` from the
corpus tokens with `Sentence(tokens)` instead of from raw text.

## Evaluation

Scores are strict span level: a predicted span counts only when its class and
both boundaries match the gold span. They come from the 382-text test split of
the corpus, recomputed from the saved predictions of this run.

| Quantity | Value |
|---|---|
| Observed run (seed 42), micro F1 | **0.356** |
| Observed run, micro precision / recall | 0.435 / 0.301 |
| Three retrainings (seeds 1, 2, 3), mean ± SD | 0.356 ± 0.016 |
| Per-seed F1 | 0.339, 0.371, 0.356 |
| Expert span-level agreement on this task | 0.600 |

The three retrainings change only the seed. Their spread describes run-to-run
variation of the recipe, not the uncertainty of this checkpoint, and it is not a
significance test. The released weights are the seed-42 run, so its F1 is the
one this file reproduces.

Every score stays below expert agreement. The task is not solved.

### Per class

| Label | Precision | Recall | F1 | Gold spans |
|---|---|---|---|---|
| `capacity` | 0.433 | 0.349 | 0.386 | 129 |
| `normality` | 0.333 | 0.119 | 0.176 | 67 |
| `propriety` | 0.452 | 0.434 | 0.442 | 226 |
| `tenacity` | 0.333 | 0.036 | 0.065 | 56 |
| `veracity` | 0.000 | 0.000 | 0.000 | 30 |

## Training data

The corpus holds 2,546 Chilean Spanish texts annotated by three linguists
trained in SFL, split 1,782 / 382 / 382. It is
[gated](https://huggingface.co/datasets/jorgeortizfuentes/chilean-spanish-attitude-corpus): the source texts are
public posts whose authors did not consent to redistribution, so access is
granted for non-commercial research under the corpus LICENSE. The weights
released here are not gated; they do not redistribute the texts.

The corpus is dominated by tweets (2,420 of 2,546 texts). Letters to the
editor, opinion columns and consumer complaints are present in small numbers.

## Training configuration

Base model: [dccuchile/tulio-chilean-spanish-bert](https://huggingface.co/dccuchile/tulio-chilean-spanish-bert) (DOI 10.57967/hf/1846).

| Parameter | Value |
|---|---|
| `label_type` | subtypes_jugdment_tags |
| `embedding_type` | combined |
| `bert_model` | dccuchile/tulio-chilean-spanish-bert |
| `max_epochs` | 100 |
| `learning_rate` | 0.05 |
| `bert_learning_rate` | 3e-05 |
| `mini_batch_size` | 8 |
| `eval_batch_size` | 8 |
| `patience` | 3 |
| `anneal_factor` | 0.5 |
| `hidden_size` | 256 |
| `seed` | 42 |
| `use_crf` | True |
| `label_encoding` | BIO |
| `best_model_metric` | strict_span_level_micro_f1 |

Validation scores of this run: accuracy 0.9108, f1 0.3428, precision 0.4462, recall 0.2783.

## Intended use

Research on Appraisal, span-level sequence labeling and evaluative language in
Chilean Spanish. The fixed splits allow comparison with the paper.

It is not fit for moderating users, profiling individuals, making decisions
about people, or as a general-purpose sentiment classifier. Attitude analysis
identifies evaluative language and its category; it does not identify who is
right, and a Judgment label is not a finding about the person being judged.

## Limitations

Scores are far below what sequence labeling reaches on more established tasks,
and below the agreement of the experts who annotated the data. Judgment
Subtypes is the harder of the two tasks for every architecture tested.

The model was trained on Chilean Spanish from 2010–2022, mostly Twitter.
Under Twitter-only training, both supervised architectures in the paper score
lower outside Twitter, particularly on Judgment Subtypes; cross-genre transfer
is not established. Performance on other varieties of Spanish is untested.

## Provenance

| Field | Value |
|---|---|
| Task column in the corpus | `subtypes_jugdment_tags` |
| Hyperparameter set | `combined2_aggressive_bert` |
| Seed | 42 |
| Selection | best validation F1 during training |
| Predictions behind the reported score | `results/flair/flair_combined_dccuchile_tulio-chilean-spanish-bert_subtypes_jugdment_tags_combined2_aggressive_bert_predictions.parquet` in the code repository |

| File | Bytes | SHA-256 |
|---|---|---|
| `pytorch_model.bin` | 1,938,413,483 | `85a9ac8324ad9df2…` |

## Citation

```bibtex
@inproceedings{ortizfuentes-attitude-2026,
  title     = {Attitude Analysis in Systemic Functional Linguistics: A New Corpus and Benchmark Task},
  author    = {Ortiz-Fuentes, Jorge and Bravo-Marquez, Felipe and Quiroz, Beatriz},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  note      = {To appear}
}
```

Cite TULIO as well when you use this model; the encoder is
[dccuchile/tulio-chilean-spanish-bert](https://huggingface.co/dccuchile/tulio-chilean-spanish-bert).

## License

The fine-tuned weights are released under CC BY 4.0, the license declared by the
TULIO encoder they derive from. The training corpus keeps its own license and
access conditions.