mohit67890/imajev-2b

Decisions for real-world cases.

Small open models that read the photos, records and text a business already has and answer in the options you set, with a probability on each and an explicit can't tell. Your system acts when it is sure and hands the rest to a person.

imajev-2b is the smallest, most cautious size of the family · other sizes: imajev-4b · imajev-9b
Live demo · Website · Code and results · Technical report

imajev-2b is the smallest and most cautious size: it automates the fewest decisions, and is right most often when it does.

What sets it apart

• A photo read against your record. Checks a photo against your own fields and names the one that is wrong. Trained on 72k photo-vs-record and two-photo decisions.
• Two photos, one decision. A reference and a target in the same request: shipped against returned, a known-good part against the one on the line.
• A trained can't tell. Every answer carries a probability for unknown, so the app can stop instead of guessing.
• Open, small and local. Apache-2.0, MLX on a Mac or PyTorch on one GPU; photos and customer data never leave your network.
• Jev's contract, now with images. TypeSafe's Jev request and response (POST /v1/systemone), plus images, unknown_probability and abstained. Jev itself is text-only and hosted; its state limit (32k tokens) is larger than imajev's (32 KB).

One request, every answer typed

The exact script we ran against imajev-4b and its output (rounded, usage shortened); 1.15 s on a Mac Studio (four option orders averaged, calibration file applied). Swap the adapter for this size and the request is unchanged.

import json, requests

URL = "http://127.0.0.1:8765/v1/systemone"

listing = {
    "title": "Men's suede boat shoes",
    "color": "red",
    "product_type": "shoe",
}

questions = {
    "contradicted_field": {
        "type": "choice",
        "instructions":
            "Which field of `listing` does this photo contradict?",
        "criteria": {
            "listing.color": None,
            "listing.product_type": None,
            "none of these": "the photo agrees with every field",
        },
    },
    "color_matches": {
        "type": "noul",
        "instructions":
            "The product in the photo matches `listing.color`.",
    },
    "type_matches": {
        "type": "noul",
        "instructions": "The photo shows the kind of product "
                        "given in `listing.product_type`.",
    },
}

request = {"state": {"listing": listing}, "questions": questions}
with open("listing.jpg", "rb") as photo:
    r = requests.post(URL, files={"image": photo},
                      data={"request": json.dumps(request)})
print(json.dumps(r.json(), indent=2))

Result

{
  "model": "imajev-4b",
  "answers": {
    "contradicted_field": {
      "type": "choice",
      "choice": "listing.color",
      "probabilities": {
        "listing.color": 0.95,
        "listing.product_type": 0.006,
        "none of these": 0.043
      },
      "confidence": 0.919,
      "unknown_probability": 0.007,
      "abstained": false
    },
    "color_matches": {
      "type": "noul",
      "noul": 0.082,
      "unknown_probability": 0.022,
      "abstained": false
    },
    "type_matches": {
      "type": "noul",
      "noul": 0.989,
      "unknown_probability": 0.004,
      "abstained": false
    }
  },
  "usage": {
    "total_ms": 1152.9,
    "input_tokens": 224
  }
}

Automate what is clear, route the rest

imajev-2b on the 279 ImajevBench test questions (photos, records and text; 21 whose honest answer is can't tell), raw probabilities, scored with the benchmark's own rule:

Act automatically when at least…	Decisions automated	Automatic decisions right
80% sure	49%	92.7%
90% sure	38%	95.3%
99% sure	21%	100%

The rest go to a person. The benchmark is built to be hard; measure on a few hundred of your own cases before choosing a threshold. Other sizes at 90%: 2B 38% automated at 95.3% right, 4B 58% at 94.5%, 9B 70% at 91.8%.

How it was made

About a million training decisions across the family, in four stages, for about $676 of rented GPU time for the whole project. The 2B went through all four stages: about 500k human-labelled decisions, about 490k more: 416k on new photo and text sources labelled by our 9B, plus 72k photo-vs-record and two-photo decisions (most labelled by the 9B, 17k by construction), then about 23k hard questions kept only when open-weight teachers agreed, then a soft-target continuation on 39,515 rows carrying Qwen3.6-35B-A3B's full probability distributions (with the strict slice of the Eikos decisions set (caiovicentino1/eikos-decisions, CC-BY-4.0; attribution and per-source licences in docs/eikos-decisions-usage.md) and 5k replayed image decisions). The shipped adapter is the weight-space average of two adapters: the hard-question adapter and that continuation. Every teacher is open-weight; no Jev outputs, paid-API outputs or JevBench items were used.

This adapter

This repository holds the 2B adapter, the latency tier. It is a LoRA (rank 16, alpha 32) on the language layers of Qwen3.5-2B (revision 15852e8c) plus a 255-code decision readout, in PEFT format at the root and in MLX format under mlx/; the weights are the element-wise average (0.5 / 0.5, LoRA matrices and readout) of the hard-question adapter and its soft-target continuation. Code, server and evaluation harness: https://github.com/mohit67890/imajev. Other tiers: https://huggingface.co/mohit67890/imajev-4b (recommended default), https://huggingface.co/mohit67890/imajev-9b (quality).

Which size? The 2B is for tight latency or memory budgets. The 4B is 11 points higher on ImajevBench and 10 points higher on JevBench hard at about 1.5× the latency; start there unless the 2B's footprint is the point.

Technical specification

Base model	Qwen/Qwen3.5-2B, revision 15852e8c (Apache-2.0)
LoRA	rank 16, alpha 32, dropout 0, no bias, on every language-model projection: q,k,v,o, gate,up,down and the DeltaNet in_proj_qkv, in_proj_z, out_proj; vision encoder frozen, no LoRA
Decision readout	one bias-free linear layer, 255 × 2048, float32
Trainable parameters	15,630,336 LoRA + 522,240 readout = 16,152,576
Files	adapter_model.safetensors 62.6 MB (F32); readout 2.1 MB
Precision	base weights bfloat16; LoRA and readout float32 (MLX copies under mlx/ converted from the same files)
Request limits	0–2 images (resized to at most 400,000 pixels), state up to 32 KB, 1–8 questions, 2–254 options per choice, 2–10 levels per score, at most 4,096 tokens (longer requests are refused, not truncated); English only
Calibration	one temperature, 1.646, fitted on 150 template-generated JevBench-style items (none from JevBench), also used to pick checkpoints

Training path. One trainer for every stage (PyTorch + PEFT): cross-entropy on the readout logits (soft targets where a record carries a distribution), AdamW with weight decay 0, linear warm-up then cosine decay to 10% of the peak rate, gradient clipping 1.0, seed 0, 4 GPUs. The soft-target stage adds a rationale loss (weight 0.3, at most 192 tokens) and permutes the options of every question.

Stage	Started from	Epochs	Peak LR	Steps (kept / total)	Hardware	Time
Initial run (594,214 records, earlier image mixture)	Qwen3.5-2B	1	2e-4	2,614 / 2,614	4×H100	1.3 h
Stage 1	initial run	1	1e-4	2,495 / 2,495	4×H100	1.3 h
Stage 2, first pass	stage 1	0.4	5e-5	1,100 / 1,866	4×H100	1.0 h
Stage 2, pairs pass	first pass	1	5e-5	1,361 / 1,361 (chosen by hand on probe results)	4×H100	not recorded
Stage 3, round 1	stage 2	2	3e-5	250 / 256	4×H200	8 min
Stage 3, round 2	round 1	2	2e-5	80 / 220	4×H100	9 min
Stage 4, soft-target continuation	round 2	2	2e-5	best on dev / 620	4×H100	49 min
Weight-space average	½ round 2 + ½ stage 4, element-wise (LoRA and readout)	–	–	–	–	–

Data this size saw.

• Initial run: 594,214 records from an earlier image mixture.
• Stage 1: 504,000 decisions from 36 licence-admitted sources (15 text, 21 image), including 4,000 photo-vs-listing contradictions.
• Stage 2, two passes: 616,964 decisions labelled by the 9B, 475,305 kept (a label is kept only when two option orders agree and the top probability is at least 0.6, or unknown at least 0.5); the pairs pass trained on the 71,630 photo-vs-record and two-photo decisions (17,162 labelled by construction). Targets were blended 0.5 : 0.5 with the untuned 2B's own distribution.
• Stage 3, round 1: 14,112 training records: kept teacher questions (9,368 of 13,386 kept on two-answerer agreement) plus the training share of 8,532 human reasoning items from 10 licensed sets.
• Stage 3, round 2: 7,812 training records: 3,598 new (4,852 of 8,097 kept on three-answerer agreement) + 4,214 replayed from round 1.
• Stage 4: 39,515 records: the stage-3 teacher questions relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions.

Compute. $499.07 of rented GPU time on RunPod through stage 3 plus about $177 for stage 4 (8×H100, 6 h 20 min, all three sizes): about $676 for the whole project, every run included.

Full specification: https://github.com/mohit67890/imajev/blob/main/docs/technical-specification.md

Results (2026-09-24; every number reproducible from the code repository's results/)

Benchmark	imajev-2b	Notes
JevBench public hard (111)	60.4% served (4 option rotations + calibration.json; ECE 0.123), 58.6% raw (ECE 0.176)	same protocol, our runs: imajev-4b 70.3%, JevK5 v0.2.0 73.9%, Eikos-4B 73.9%, Hopper 67.6%, Qwen3.5-4B base (generation) 48.6%, mojev 0.85B 33.3%; the previous imajev-2b 56.8% (ECE 0.138)
JevBench public original (72) / easy (48)	93.1% / 100%
ImajevBench v2.0-lite test (279: text, photo, photo+state)	71.7% (200/279), 95% CI [0.65, 0.78]	Qwen3.5-2B base 60.2%; the previous imajev-2b 70.3%; an earlier imajev-2b 63.1%
· text / visual / joint tracks	19/37 · 99/120 · 82/122
· correct Unknown / false abstention	5/21 · 4/258	abstains too rarely on this benchmark's Unknown items
ImajevBench private-1 hidden split (202; aggregates only)	74.3% (150/202); text 25/30 · visual 67/84 · joint 58/88; ECE 0.089	the previous imajev-2b 70.8%
MLX (Mac) vs PyTorch on ImajevBench	69.9% vs 71.7%, 97.1% argmax agreement	parity check of the mlx/ weights against the pod run
MMLU-1000, text-only / with an unrelated photo	59.8% / 54.9%	measured on the previous imajev-2b, not re-run; an earlier imajev-2b: 45.2% text-only
Irrelevance panel (2,823: MMLU with and without an unrelated photo, ABO, VizWiz)	68.9%	the previous imajev-2b 67.7% (false abstention 2.3%, correct abstention 91.1%; Mac Studio, MLX)
Hard-question test (435): correct on Unknown-gold rows / false abstention	12/14 · 1.19%	ship gates
State probe (200) / pairs probe (60)	68.5% / 100%	authored, templated
Reasoning dev (6,240 items; also used for checkpoint selection)	not measured for the shipped average; 62.7% for the soft-target checkpoint it averages, 58.9% for the previous imajev-2b	64.5% before the last part of the hard-question stage (see Limitations)
p50 latency, JevBench hard item, 1×H100, serial	238 ms served (4 rotations + calibration), 83 ms raw	shared pod, under load

Our pre-registered test against the untuned base model on ImajevBench (paired cluster sign-flip over 89 evidence clusters): the shipped imajev-2b vs the untuned Qwen3.5-2B, +11.5 points (200 vs 168) [+3.5, +18.9], p = 0.005. The previous imajev-2b (previous adapter) gave +10.0 [+1.8, +17.8], p = 0.019; the earlier imajev-2b the test was registered with gave +2.9 points, p = 0.572. We report all three.

How this version was chosen. We compared the soft-target checkpoint on its own with its average with the previous imajev-2b. The checkpoint alone gains on the hidden ImajevBench split (74.8% vs 70.8%) but loses one JevBench hard item and three photo+record items on the public test split; the average is ahead of the previous version on every tier (JevBench hard 56.8% → 60.4%, ImajevBench 70.3% → 71.7%, hidden split 70.8% → 74.3%) and passes every ship gate. The ImajevBench test split was one of the no-regression gates for that choice, so the 2B's public ImajevBench number is not a pure held-out estimate; the hidden split is.

Older panels measured on an earlier imajev-2b, before the hard-question stage (not re-run): held-out photo sources 54.6%, real two-image pairs 51.4%.

Calibration

calibration.json (schema 1.1) applies one temperature (1.646) to every question type × option-count bucket, fitted by negative log-likelihood on 150 template-generated JevBench-style items (none from JevBench), also used to pick checkpoints. Temperature scaling never changes an answer, only its probability. unknown offsets are 0: bounded offsets were tested and changed no panel by more than 0.1 points.

Checked through the released server (ECE, uncalibrated → with calibration.json); the off-distribution rows were measured on the previous imajev-2b with its own temperature (1.61) and have not been re-run for this version:

Panel	ECE
JevBench public hard (served with 4 rotations, 111)	0.176 raw → 0.123
MMLU-1000, text-only	0.141 → 0.038
typed-decisions test (2,000)	0.128 → 0.039
SST-5 (2,210)	0.221 → 0.105
Photo-only verification (ABO + VizWiz, 823)	0.023 → 0.120

Pooled ECE over all 231 public JevBench items in the served configuration: 0.025. An ECE-fit temperature (2.20) lowers the hard-tier ECE to 0.090 but raises the pooled ECE to 0.075, so the NLL fit is shipped.

On photo-only verification the raw probabilities are already calibrated and the temperature over-softens them; if your traffic is mostly photo-against-record checks, serve without --calibration or fit your own temperature on a held-out sample.

How to use

git clone https://github.com/mohit67890/imajev && cd imajev
python3.11 -m venv .venv && . .venv/bin/activate
pip install -e ".[serve,mlx]"            # Apple silicon;  elsewhere: pip install -e ".[serve,torch]"
python scripts/download_model.py         # pinned Qwen3.5-2B
hf download mohit67890/imajev-2b --local-dir adapters/imajev-2b
# Mac (MLX)
PYTHONPATH=src:scripts python scripts/playground/server.py --adapter adapters/imajev-2b/mlx \
  --calibration adapters/imajev-2b/calibration.json --model-name imajev-2b --port 8765
# Linux / CUDA (PyTorch + PEFT)
PYTHONPATH=src:scripts python scripts/playground/server.py --backend torch --adapter adapters/imajev-2b \
  --calibration adapters/imajev-2b/calibration.json --model-name imajev-2b --port 8765

curl -s http://127.0.0.1:8765/v1/systemone \
  -F 'request={"state":{"listing":{"title":"Blue ceramic mug, 350 ml","colour":"blue"}},
               "questions":{"matches":{"type":"noul","instructions":"Does the photo show the listed item?"},
                            "wrong_field":{"type":"choice","instructions":"Which listing field does the photo contradict?",
                                           "criteria":{"title":null,"colour":null,"none":null}}}}' \
  -F image=@photo.jpg

The numbers above were measured with --rotations 4 (four option orders averaged: +1.8 hard on JevBench for the 2B, at about 3× the latency).

Training

• Base: Qwen3.5-2B (Apache-2.0). LoRA r16/α32 on every language-model projection including the DeltaNet projections; 255 single-token option codes read at the decision position through a float32 readout head; vision tower frozen.
• Licence-checked decisions and new photo sources: 504,000 human-labelled image and text decisions from licence-verified sources, then 616,964 decisions on new photo and text sources labelled by an earlier version of imajev-9b, 475,305 kept with a keep rule (two option orders must agree; top probability ≥ 0.6 or unknown ≥ 0.5), then 72,000 state-grounded and two-image (reference vs target) decisions (17,162 labelled by construction).
• Hard-question stage: 2 epochs, lr 3e-5, on 17,898 hard typed questions (documents written by Qwen3.6-27B, kept only when two answerers of different families agree) plus licence-verified human reasoning sets.
• Last part of the hard-question stage: 2 epochs, lr 2e-5, on 9,066 rows (new questions kept only when three open-weight answerers agree unanimously, plus 30% replay of the earlier hard questions). Best checkpoint chosen on held-out dev rows.
• Soft-target stage: 2 epochs, lr 2e-5, on 39,515 rows: the hard questions relabelled with Qwen3.6-35B-A3B's full probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions; soft-target cross-entropy plus a rationale loss (0.3) with option permutation. The shipped adapter is the weight-space average of the previous adapter and this stage's best checkpoint (0.5 / 0.5, LoRA matrices and readout).

Data and licence posture

• Adapter, readout and code: Apache-2.0. Base model Qwen3.5-2B: Apache-2.0.
• No JevBench items (8-gram contamination lint), no outputs from Jev, and no outputs from any paid API were used in training. All teacher models are open-weight.
• Per-source licence receipts are in the code repository. New photo sources: PD12M (CC0), Wikimedia Commons (CC-BY-4.0, CC-BY-3.0 or CC0, checked per file), Open Images (CC-BY-2.0). 16 of the 21 image sources from the first training stage are used under their annotation licences only, with photos under upstream terms (not redistributed); for abo, vizwiz, vizwiz_quality and defects the grant covers the images too.

Limitations

• Single pass, no reasoning at inference: multi-step arithmetic and judging trail larger and reasoning models by a wide margin.
• The last part of the hard-question stage cost 5.6 points on our reasoning dev set, also used for checkpoint selection (64.5% → 58.9%); the soft-target checkpoint recovers to 62.7%, and the shipped average was not measured on that set. MMLU did not move, so the loss looks specific to that set's item styles, but it is a measured regression.
• Abstains too rarely on ImajevBench's Unknown items (5/21 correct, vs 14/21 for the 4B and 15/21 for the 9B).
• Over-confident without calibration.json. Counting is weak. At most two images, 32 KB state, 254 options, 8 questions per request. English only. No free text.

Intended use

Typed decisions inside applications where latency or memory is tight: photo-against-record checks, routing, extraction into fixed option sets, abstention when evidence is missing. Not a safety classifier, not a certificate of correctness, and not for decisions about people without human review.

Citation

@software{imajev2026,
  author = {Garg, Mohit},
  title = {imajev: an open Jev-style typed decision model family for images and text},
  year = {2026},
  url = {https://github.com/mohit67890/imajev}
}

ImajevBench, the photo-and-text benchmark released alongside: https://huggingface.co/datasets/mohit67890/imajev-bench.