Hanish/lq-decide-0.6b

LQ-Decide 0.6B

A small model that answers typed decisions instead of generating text. You give it a state and a question with a fixed set of options; it returns a probability over those options from a single forward pass. No tokens are sampled, so the output cannot be malformed and there is nothing to parse or repair.

Built by Hanish Keloth for LocalQuill, a private on-device AI keyboard and offline chat app. It is trained from Qwen/Qwen3-0.6B and is small enough to run on a phone.

This is a v1 and it is not state of the art. The numbers below are measured on public held-out fixtures and are printed in full, including the family it is bad at. Read them before you use it.

Results

Evaluated on the two labelled fixtures from the SemIf project, which were never seen in training. Balanced accuracy, three options per item, chance is 0.333.

	authored144	perturbations108
LQ-Decide 0.6B (bf16)	0.757	0.557
LQ-Decide 0.6B (Q8_0 GGUF)	0.750	0.586
LQ-Decide 0.6B (Q4_K_M GGUF)	0.703	—
Untrained Qwen3-0.6B, same prompt	0.512	—
SemIf, Qwen3.5-4B, published	0.813	—
Jev (TypeSafe, closed), published	0.883	—

Per family, Q8_0:

family	authored144	perturbations108
rule application	0.812	0.722
evidence interpretation	0.792	0.639
candidate selection	0.646	0.389

Known weakness

Candidate selection is weak, and on the perturbation set it is close to chance. That set contains modified variants of the same items, so the gap between 0.646 and 0.389 means the model is leaning on surface wording rather than the distinction being asked about. If your task is "pick which of these two statements satisfies X", this model is not reliable yet. Rule application and evidence interpretation are considerably better.

The model is also below the open 4B alternative on the same fixture. Use LQ-Decide if size and latency matter to you; use a larger model if accuracy matters more.

Calibration

Probabilities are temperature-scaled with a single constant fitted on a held-out split, shipped in calibration.json.

	expected calibration error
raw	0.172
after T = 2.05	0.049

Averaging the logits over all option orderings costs nothing in accuracy and improves calibration further on some splits. Apply the temperature before reading the probabilities as confidence.

Latency

Measured with llama.cpp on an Apple M5 Max, 147-token prompts, one decision per request:

quant	median	p95	file size
Q4_K_M	36 ms	47 ms	378 MB
Q8_0	37 ms	49 ms	610 MB

Phone numbers are not published yet. Every latency figure in this space, including this one, comes from a desktop. Do not assume it transfers to a handset.

Use

import torch, json
from transformers import AutoTokenizer, AutoModelForCausalLM

LETTERS = "ABCDEFGHIJKLMNOP"
SYSTEM = ("Apply the supplied criterion to the supplied evidence. Choose exactly one listed option. "
          "Respond with only its uppercase letter, with no explanation or reasoning.")

tok = AutoTokenizer.from_pretrained("Hanish/lq-decide-0.6b")
model = AutoModelForCausalLM.from_pretrained("Hanish/lq-decide-0.6b", dtype=torch.bfloat16).eval()
T = json.load(open("calibration.json"))["temperature"]   # 2.05

def decide(state, question, options):
    payload = {"evidence": state, "criterion": question,
               "options": [{"letter": LETTERS[i], "description": d} for i, d in enumerate(options)]}
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(payload, ensure_ascii=False)}]
    text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    ids = tok(text, return_tensors="pt")
    with torch.no_grad():
        logits = model(**ids).logits[0, -1]
    sel = torch.tensor([logits[tok.encode(LETTERS[i], add_special_tokens=False)[0]]
                        for i in range(len(options))]).float()
    return torch.softmax(sel / T, -1).tolist()

print(decide(
    "The optician ordered replacement lenses. The workshop confirms they have not yet been fitted.",
    "Assess the claim: the replacement lenses have been fitted.",
    ["The evidence establishes the claim",
     "The evidence does not establish either",
     "The evidence establishes the opposite"]))

The prompt format matters. The model was trained on exactly this template, with options shuffled and the answer letter re-based during training so it reads descriptions rather than positions.

GGUF builds are in this repository. With llama-server, request one token with logprobs and read the probability mass on the option letters; nothing is generated.

Training

LoRA rank 32 on all projections, merged. Loss on the single answer-letter token only. Options were permuted and the letter set re-based every epoch. 117,096 rows, two epochs, about two hours on one L40S.

Data, all licence-checked, non-commercial sources excluded:

source	rows	licence
MNLI, SNLI, WANLI	60,000	CC-BY / CC-BY-SA
CLINC150, MASSIVE, Banking77	36,757	CC-BY 3.0 / 4.0
BoolQ	9,427	CC-BY-SA 3.0
SMS Spam	5,574	CC-BY 4.0 (UCI)
Synthetic decision families	29,280	generated with Ornith-1.5-35B-A3B (MIT)

The synthetic portion covers three families with fifteen mechanisms each. The intended answer was fixed before generation and the teacher only wrote the surface text, so there is no teacher-labelling noise. The evaluation fixtures were never generated from, trained on, or shown to the teacher.

Limitations

• English only.
• Three to sixteen options; not tested beyond that.
• Weak on candidate selection, as shown above.
• Below larger open models on the same fixture.
• Not compared against Laya or other recent decision models, because those publish on their own benchmarks and this one has not been run on them. Do not read the table above as a ranking against anything not in it.

Credits

Base model Qwen/Qwen3-0.6B (Apache-2.0, Alibaba). Evaluation fixtures from the SemIf project (MIT). Synthetic data generated with Ornith-1.5-35B-A3B (MIT). The interface pattern follows TypeSafe's Jev, which is a separate closed product; this model is not affiliated with or endorsed by it.