README.md

---
license: other
license_name: lfm1.0
license_link: LICENSE
base_model:
- LiquidAI/LFM2.5-VL-450M
base_model_relation: finetune
pipeline_tag: image-text-to-text
library_name: transformers
language:
- en
tags:
- calibration
- classification
- system-one
- jev-compatible
datasets:
- allenai/ai2_arc
- allenai/sciq
- allenai/openbookqa
- tau/commonsense_qa
- uoft-cs/cifar10
- mteb/stsbenchmark-sts
---

<div align="center">
  <img
    src="https://huggingface.co/buckets/nullsilver/main/resolve/nullsilver-banner-light-1.png"
    alt="alpha-sys-1 banner"
    style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;"
  />
  <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;">
    <a href="https://github.com/nullsilver-labs/alpha-sys-1/blob/main/docs/USAGE.md"><strong>docs</strong></a> •
    <a href="https://nullsilver.com"><strong>nullsilver.com</strong></a>
  </div>
</div>

# alpha-sys-1-450M

alpha-sys-1 is a multimodal, Jev-compatible **System One model**. It takes a state, which
may contain text, an image, or both, together with a question that has a fixed set of
answers, and returns a probability distribution over those answers in one forward pass. It
generates no text.

The model is trained for calibrated probabilities: across a large group of similar
examples where it assigns an answer a probability near 80%, that answer should be correct
in roughly 80% of cases. Calibration degrades when the input differs substantially from
the training data, so check the probabilities on data from the intended application.

| | |
|---|---|
| Base | [LiquidAI/LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) |
| Tuning | LoRA rank 32, lr 1e-4, merged into the base weights |
| Checkpoint | `alpha-sys-1-260919`, revision `260919`, seed 1 of 3 |
| Input | text, one image, or both; English |
| Output | probabilities over the answer space |
| Sizes | [450M](https://huggingface.co/nullsilver/alpha-sys-1-450M) · [1.6B](https://huggingface.co/nullsilver/alpha-sys-1-1.6B) · [3B](https://huggingface.co/nullsilver/alpha-sys-1-3B) |

## Question types

Questions follow TypeSafe's System One format: a request contains one `state` and any
number of `questions`, so a question written for Jev runs here as is. `images` is an extra
field for multimodal inputs.

| type | answer space | returns |
|---|---|---|
| `choice` | named options, up to 26 | `probabilities` over the options, `choice` (argmax) |
| `noul` | a statement | `noul`, P(true) |
| `score` | ordered levels, lowest first | `probabilities` over the levels, `score` (expected level index) |

The answer is read from the next-token logits for the answer labels (`A`, `B`, … or
`No`/`Yes`), renormalised over the valid labels. Each question is answered independently:
one question's answer is never context for another.

> [!NOTE]
> `confidence` is `1 - H(p)/log(n)`, computed from the distribution. It is not a separate
> prediction.

> [!TIP]
> A `noul` probability near 0.5 means the model is uncertain.

## Usage

`alpha_sys_1.py` in this repository renders questions the way the model was trained on
them, batches the questions on one state, and returns answers in the System One shape.

```python
from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("alpha_sys_1", hf_hub_download("nullsilver/alpha-sys-1-450M", "alpha_sys_1.py", revision="260919"))
alpha_sys_1 = importlib.util.module_from_spec(spec); spec.loader.exec_module(alpha_sys_1)

m = alpha_sys_1.SystemOne("nullsilver/alpha-sys-1-450M", revision="260919")
m.system_one({
    "state": {"subject": "Duplicate charge on invoice #4411",
              "body": "We were billed twice for March. Refund the duplicate today or we cancel our plan."},
    "questions": {
        "department": {"type": "choice", "instructions": "Which department should handle this email?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
                                    "sales": "pricing, new contracts", "other": "everything else"}},
        "urgency": {"type": "score", "instructions": "How urgent is this request?",
                    "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
        "churn_risk": {"type": "noul", "instructions": "The user threatens to cancel or leave."}}})
# {"department": {"choice": "billing", "probabilities": {"billing": 0.96, "technical": 0.01, "sales": 0.02, "other": 0.02}, "confidence": 0.84},
#  "urgency": {"score": 0.56, "probabilities": [0.60, 0.24, 0.16], ...},
#  "churn_risk": {"noul": 0.48}}
```

Email triage is not one of the training environments; the output above is what this
checkpoint returns on it, not a tuned result.

Without the client, use this prompt format. A different format gives less reliable
probabilities.

```python
import string, torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo, rev = "nullsilver/alpha-sys-1-450M", "260919"
processor = AutoProcessor.from_pretrained(repo, revision=rev)
processor.tokenizer.padding_side = "left"
model = AutoModelForImageTextToText.from_pretrained(
    repo, revision=rev, dtype=torch.bfloat16, device_map="auto").eval()

def render(state, q):
    parts = [state] if state else []
    if q["type"] == "noul":
        c = q.get("criteria") or {}
        clar = "".join(f"\n{lab} means: {c[k]}" for lab, k in (("Yes", "true"), ("No", "false")) if c.get(k))
        parts.append(f"Statement: {q['instructions']}{clar}\nIs the statement true? Answer with Yes or No only.")
        return "\n\n".join(parts), ["No", "Yes"]
    crit = q["criteria"]
    items = list(crit.items()) if isinstance(crit, dict) else [(o, None) for o in crit]
    labels = list(string.ascii_uppercase[:len(items)])
    lines = [f"{lab}. {o}" + (f": {d}" if d else "") for lab, (o, d) in zip(labels, items)]
    parts.append(q["instructions"] + "\n" + "\n".join(lines) + "\nAnswer with the letter only.")
    return "\n\n".join(parts), labels

@torch.inference_mode()
def ask(q, state="", image=None):
    text, labels = render(state, q)
    content = ([{"type": "image", "image": image}] if image is not None else []) + [{"type": "text", "text": text}]
    inputs = processor.apply_chat_template(
        [[{"role": "user", "content": content}]], add_generation_prompt=True, tokenize=True,
        return_dict=True, processor_kwargs={"return_tensors": "pt"}).to(model.device)
    logits = model(**inputs, logits_to_keep=1).logits[0, -1].float()
    ids = [processor.tokenizer.encode(lab, add_special_tokens=False)[0] for lab in labels]
    return torch.softmax(logits[ids], -1).tolist()

p = ask({"type": "noul", "instructions": "The message conveys urgency"},
        state="Our API integration started returning 500 errors an hour before launch.")
urgent = p[1]  # P(Yes)
```

> [!NOTE]
> - A dict `state` is rendered one field per line, as `key: value`.
> - Training images smaller than 256 px were upscaled to 256 px.
> - For several questions on one state, batch them with `padding_side="left"`.
> - In bfloat16, probabilities move by up to a few hundredths with batch composition and
>   padding.

## Training

Training uses cross-entropy between the model's distribution and a target `y_soft`. The
target is one-hot when a dataset provides one answer, and the annotator distribution when
several annotations are available. Options are shuffled on every draw, the vision tower is
frozen, and environments are sampled in proportion to the square root of their size.

| environment | modality | type | label |
|---|---|---|---|
| mcq (ARC-Easy, SciQ, OpenBookQA, CommonsenseQA) | text | choice | one-hot |
| ChaosNLI (100-annotator items) | text | choice | annotator distribution |
| CivilComments-WILDS | text | noul | annotator share |
| STS-B | text | score | annotator mean |
| Folktables (ACS income, California 2014) | tabular as text | noul | outcome |
| CIFAR-10 | image | choice | one-hot |
| Camelyon17-WILDS | image | noul | outcome |

Three random seeds were trained. The released checkpoint is the seed with the lowest mean
development loss across environments.

## Evaluation

Each test split was read once per checkpoint. Reported intervals are 95% clustered
bootstrap intervals, clustered on the relevant dataset group: question, comment, hospital,
or state-year.

> [!IMPORTANT]
> Compare models on NLL and Brier score. ECE is reported alongside them and is misleading
> on its own: a model that always predicts the base rate can have a low ECE.

The tables carry two reference points. The **base rate** is the constant predictor: it
answers every question with the label frequencies of the training split (for example
"toxic" 14% of the time on CivilComments, whatever the comment says), or uniformly when
the options are shuffled. Any model should beat it. **Base + T** is the untuned
LFM2.5-VL-450M, read the same way as the tuned model, with its label logits divided by
one scalar temperature chosen to minimise NLL on the environment's development split.

**Trained environments.**

| environment | NLL | NLL, base + T | Brier | ECE | AUROC | acc |
|---|---|---|---|---|---|---|
| mcq | **0.727** | 0.929 | 0.380 | 0.067 | 0.816 | 0.722 |
| ChaosNLI | **0.902** | 0.941 | 0.245 | 0.079 | 0.591 | 0.575 |
| CivilComments | **0.324** | 0.581 | 0.044 | 0.073 | 0.906 | 0.940 |
| STS-B | **1.107** | 1.782 | 0.325 | 0.046 | 0.622 | 0.562 |
| Folktables | **0.436** | 0.693 | 0.286 | 0.013 | 0.763 | 0.789 |
| CIFAR-10 (+C) | **0.509** | 0.804 | 0.212 | 0.040 | 0.918 | 0.851 |
| Camelyon17 | **0.202** | 0.693 | 0.110 | 0.033 | 0.887 | 0.930 |

> [!NOTE]
> ChaosNLI, CivilComments and STS-B have soft labels from multiple annotations, so
> top-label ECE does not fully measure calibration. On these, use NLL and KL divergence to
> the annotator distribution.

**Unseen tasks.** Not in training.

| task | type | NLL | NLL, base + T | base rate |
|---|---|---|---|---|
| BoolQ | noul | 0.661 | **0.646** | 0.665 |
| Yelp review stars | score | 1.467 | **1.455** | 1.609 |

**Distribution shift.** CIFAR-10-C.

| | clean | sev. 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| accuracy | 0.960 | 0.907 | 0.872 | 0.841 | 0.795 | 0.724 |
| mean confidence | 0.962 | 0.928 | 0.903 | 0.882 | 0.856 | 0.811 |

**Other System One models.** NLL on the text environments, same test splits, same
readout. The other alpha-sys-1 sizes on the table are their released seeds. `Qwen3.8-27B` is the open 27B
generalist, read at its first answer token with reasoning off, plus a dev-fitted
temperature.

| environment | alpha-sys-1-450M (this) | alpha-sys-1-1.6B | alpha-sys-1-3B | Qwen3.8-27B + T | base rate |
|---|---|---|---|---|---|
| mcq | 0.727 | 0.427 | 0.289 | **0.159** | 1.439 |
| ChaosNLI | 0.902 | 0.784 | 0.735 | **0.706** | 0.938 |
| CivilComments | 0.324 | 0.323 | **0.319** | 0.473 | 0.425 |
| STS-B | 1.107 | 1.033 | **0.961** | 1.347 | 1.727 |
| Folktables | **0.436** | 0.448 | 0.441 | 0.472 | 0.683 |
| BoolQ (unseen) | 0.661 | 0.479 | 0.405 | **0.316** | 0.665 |
| Yelp review stars (unseen) | 1.467 | 1.110 | 0.973 | **0.858** | 1.609 |

Per-hospital, per-state-year and per-identity-group tables, the three-seed gate tables and
the full comparison against other System One models (hosted and open) are in the
[repository](https://github.com/nullsilver-labs/alpha-sys-1) under `runs/`.

## Limitations

> [!WARNING]
> On a task that differs substantially from the training environments, do not assume this
> model stays calibrated; measure it against the base model's calibration. In
> leave-one-domain-out tests at 1.6B, a model tuned on the other environments beat the
> untuned base with a transferred temperature on one held-out environment out of three,
> and on the two unseen tasks above the trained-on-all checkpoints match the base model and do not beat it.
> With a few hundred labelled examples from your own task, fit a temperature on them:
> divide the label logits by one scalar chosen to minimise NLL on those examples
> (`alpha_sys_1.fit_temperature`), then pass it as `SystemOne(..., temperature=T)`.

- Knowledge depends on model size: on MCQ, the untuned 1.6B base outperforms this tuned
  model.
- When the model does not know an answer, its distribution is close to uniform.
- Under strong distribution shift, such as CIFAR-10-C at severity 5, confidence remains
  higher than accuracy.
- Reversing the option order changes the top answer on 15% of MCQ items, mostly
  among low-confidence examples.
- The answer space is capped at 26 options.
- Fine-tuning used English data only and at most one image per question.

## Related work

The interface follows TypeSafe's Jev (a hosted System One model, the `state` / `questions`
request shape). Reading an answer distribution from the label-token logits of one forward
pass is the readout of Kadavath et al. (2022, *Language Models (Mostly) Know What They
Know*) and of the LLM-as-a-Verifier line of work, which scores rubric levels from the
logits of letter tokens. That calibration improves with size, and that a temperature
fitted on one domain transfers badly to another, is Jiang et al. (2021, *How Can We Know
When Language Models Know?*). Training on a proper scoring rule against annotator
distributions is why a fixed answer space and calibration are non-conflicting (Kalai and
Vempala, 2024, *Calibrated Language Models Must Hallucinate*). Base models: Liquid AI's
LFM2.5-VL.

## License

This model is derived from LiquidAI/LFM2.5-VL-450M and is released under the [LFM Open License
v1.0](LICENSE).

## Citation

```bibtex
@misc{alphasys1,
  title  = {alpha-sys-1: a small calibrated System One model},
  author = {Nullsilver},
  year   = {2026},
  url    = {https://huggingface.co/collections/nullsilver/alpha-sys-1}
}
```