lfm2.5-350M-datause-multitask-tiered
LoRA SFT of LiquidAI/LFM2.5-350M for the tiered data-use workflow: provenance attributes (producer/year/geography/acronym + data_type/specificity/bibtex) and usage/impact classification (usage_action/impact_label/usage_summary). Trained exclusively on T1∪T2-anchored rows of rafmacalaba/data-use-sft-tiered — see the dataset card for the tier policy.
Training
- base model:
LiquidAI/LFM2.5-350M - dataset:
rafmacalaba/data-use-sft-tiered(provenance + usage_impact configs) - epochs: 5 | lr: 0.0002
- LoRA: r=16 alpha=32 dropout=0.05
- completion-only masking (loss on the assistant JSON turn)
Holdout
n=8804. Full per-row gold/pred pairs are in holdout_predictions.jsonl on this repo.
Metrics recomputed offline from holdout_predictions.jsonl with string-aware JSON extraction (training/score_holdout_predictions.py). An earlier in-trainer extraction bug suppressed predictions containing bibtex values; the numbers below are the corrected ones.
Provenance — verbatim attrs (exact match)
| attribute | tp | fp | fn | precision | recall | f1 |
|---|---|---|---|---|---|---|
| producer | 1040 | 348 | 336 | 0.7493 | 0.7558 | 0.7525 |
| year | 1001 | 225 | 291 | 0.8165 | 0.7748 | 0.7951 |
| geography | 1569 | 391 | 360 | 0.8005 | 0.8134 | 0.8069 |
| acronym | 1178 | 203 | 178 | 0.8530 | 0.8687 | 0.8608 |
| overall | 4788 | 1167 | 1165 | 0.8040 | 0.8043 | 0.8042 |
bibtex (gold rows with producer+year):
exact: 195 (36.9%) | semantic (author+title+year, key ignored): 310 = 58.6% (key-only diffs: 115)
format: 98.9% of 364 emitted are canonical
@datasetentriesomitted: 165 (all missing producer/year inputs) | hallucinated on gold-none rows: 175
data_type macro-F1: 0.4905 (n=1198)
specificity macro-F1: 0.6307 (n=3325)
Usage/impact
- usage_action macro-F1: 0.5441 (n=5479)
- impact_label macro-F1: 0.6519 (n=5479)
Top usage_action confusions: analyze -> inform (241), inform -> analyze (239), curate -> analyze (143), source -> analyze (141), analyze -> source (127) Top impact_label confusions: policy -> evidence-base (320), evidence-base -> none (263), none -> evidence-base (262), evidence-base -> policy (214), policy -> none (72)
usage_summary: mean_sim=0.5587 grounded_rate=0.6623