mohit67890/imajev-4b

Decisions for real-world cases.

Small open models that read the photos, records and text a business already has and answer in the options you set, with a probability on each and an explicit can't tell. Your system acts when it is sure and hands the rest to a person.

imajev-4b is the recommended default of the family · other sizes: imajev-2b · imajev-9b
Live demo · Website · Code and results · Technical report

imajev-4b is the recommended default: 83.9% on ImajevBench against the 9B's 82.1% at under half the size, and the only size trained in the phase-3 hard-data stage.

What sets it apart

• A photo read against your record. Checks a photo against your own fields and names the one that is wrong. Trained on 72k photo-vs-record and two-photo decisions.
• Two photos, one decision. A reference and a target in the same request: shipped against returned, a known-good part against the one on the line.
• A trained can't tell. Every answer carries a probability for unknown, so the app can stop instead of guessing.
• Open, small and local. Apache-2.0, MLX on a Mac or PyTorch on one GPU; photos and customer data never leave your network.
• Jev's contract, now with images. TypeSafe's Jev request and response (POST /v1/systemone), plus images, unknown_probability and abstained. Jev itself is text-only and hosted; its state limit (32k tokens) is larger than imajev's (32 KB).

One request, every answer typed

The exact script we ran against imajev-4b and its output (rounded, usage shortened); 1.15 s on a Mac Studio (four option orders averaged, calibration file applied). Swap the adapter for this size and the request is unchanged.

import json, requests

URL = "http://127.0.0.1:8765/v1/systemone"

listing = {
    "title": "Men's suede boat shoes",
    "color": "red",
    "product_type": "shoe",
}

questions = {
    "contradicted_field": {
        "type": "choice",
        "instructions":
            "Which field of `listing` does this photo contradict?",
        "criteria": {
            "listing.color": None,
            "listing.product_type": None,
            "none of these": "the photo agrees with every field",
        },
    },
    "color_matches": {
        "type": "noul",
        "instructions":
            "The product in the photo matches `listing.color`.",
    },
    "type_matches": {
        "type": "noul",
        "instructions": "The photo shows the kind of product "
                        "given in `listing.product_type`.",
    },
}

request = {"state": {"listing": listing}, "questions": questions}
with open("listing.jpg", "rb") as photo:
    r = requests.post(URL, files={"image": photo},
                      data={"request": json.dumps(request)})
print(json.dumps(r.json(), indent=2))

Result

{
  "model": "imajev-4b",
  "answers": {
    "contradicted_field": {
      "type": "choice",
      "choice": "listing.color",
      "probabilities": {
        "listing.color": 0.95,
        "listing.product_type": 0.006,
        "none of these": 0.043
      },
      "confidence": 0.919,
      "unknown_probability": 0.007,
      "abstained": false
    },
    "color_matches": {
      "type": "noul",
      "noul": 0.082,
      "unknown_probability": 0.022,
      "abstained": false
    },
    "type_matches": {
      "type": "noul",
      "noul": 0.989,
      "unknown_probability": 0.004,
      "abstained": false
    }
  },
  "usage": {
    "total_ms": 1152.9,
    "input_tokens": 224
  }
}

Automate what is clear, route the rest

imajev-4b on the 279 ImajevBench test questions (photos, records and text; 21 whose honest answer is can't tell), raw probabilities, scored with the benchmark's own rule:

Act automatically when at least…	Decisions automated	Automatic decisions right
80% sure	64%	91.6%
90% sure	58%	94.5%
99% sure	42%	99.1%

The rest go to a person. The benchmark is built to be hard; measure on a few hundred of your own cases before choosing a threshold. Other sizes at 90%: 2B 38% automated at 95.3% right, 4B 58% at 94.5%, 9B 70% at 91.8%.

Checked demos. Every clickable combination in the five playground apps (business checks, text only, wardrobe, stylist, tracing pad) was run on imajev-4b (four option orders) and compared with the right answer: 130 of 145 pass without the calibration file, 118 with it. Only passing combinations are shown as demos; the misses are listed in reports/scenarios/.

How it was made

About a million training decisions across the family, in four stages, for about $676 of rented GPU time for the whole project. The 4B was trained on stages 1 and 2 in one run (867k decisions: the 504k human-labelled set, 296k labelled by our 9B and 66k photo-vs-record and two-photo decisions), then on about 23k hard questions kept only when open-weight teachers agreed, then a soft-target continuation on 39,515 rows carrying Qwen3.6-35B-A3B's full probability distributions (with the strict slice of the Eikos decisions set (caiovicentino1/eikos-decisions, CC-BY-4.0; attribution and per-source licences in docs/eikos-decisions-usage.md) and 5k replayed image decisions). The shipped adapter is the weight-space average of two adapters: the hard-question adapter and that continuation. Every teacher is open-weight; no Jev outputs, paid-API outputs or JevBench items were used.

This adapter

This repository holds the 4B adapter: the recommended default tier — the best accuracy per millisecond in the family. It is a LoRA (rank 64, alpha 128) on the language layers of Qwen3.5-4B (revision 851bf6e8) plus a 256-code decision readout (255 option codes and unknown), in PEFT format at the root and in MLX format under mlx/. It is the last checkpoint of the phase-3 run: the previous release (a rank-16 weight-space average) expanded to rank 64 and trained for two rounds on the decisions that release got wrong. Code, server and evaluation harness: https://github.com/mohit67890/imajev. Other tiers: https://huggingface.co/mohit67890/imajev-2b (latency), https://huggingface.co/mohit67890/imajev-9b (quality); both are still the previous-generation adapters.

Technical specification

Base model	Qwen/Qwen3.5-4B, revision 851bf6e8 (Apache-2.0)
LoRA	rank 64, alpha 128 (scale 2, as before), dropout 0, no bias, on every language-model projection: q,k,v,o, gate,up,down and the DeltaNet in_proj_qkv, in_proj_z, out_proj; vision encoder frozen, no LoRA
Decision readout	one bias-free linear layer, 256 × 2560, float32 (255 option codes + unknown)
Trainable parameters	121,896,960 LoRA + 655,360 readout = 122,552,320
Files	adapter_model.safetensors 487.6 MB (F32); readout 2.6 MB
Precision	base weights bfloat16; LoRA and readout float32 (MLX copies under mlx/ converted from the same files)
Request limits	0–2 images (resized to at most 400,000 pixels), state up to 32 KB, 1–8 questions, 2–255 options per choice, 2–10 levels per score, at most 4,096 tokens (longer requests are refused, not truncated); English only
Calibration	one temperature, 1.305, fitted on 150 template-generated JevBench-style items (none from JevBench); calibration-rot4.json is the same fit for the 4-rotation serving mode

Training path. One trainer for every stage (PyTorch + PEFT): cross-entropy on the readout logits (soft targets where a record carries a distribution), AdamW with weight decay 0, linear warm-up then cosine decay to 10% of the peak rate, gradient clipping 1.0, seed 0, 4 GPUs. The soft-target stage adds a rationale loss (weight 0.3, at most 192 tokens) and permutes the options of every question.

Stage	Started from	Epochs	Peak LR	Steps (kept / total)	Hardware	Time
First run (stages 1 and 2 combined)	Qwen3.5-4B	0.5	1.5e-4	1,900 / 2,595	4×H200	1.9 h (2 h 06 min wall)
Stage 3, round 1	first run	2	3e-5	250 / 303	4×H200	12 min
Stage 3, round 2	round 1	2	2e-5	266 / 266	4×H100	22 min (27 min wall)
Stage 4, soft-target continuation	round 2	2	2e-5	260 / 747	4×H100	1.4 h (out of memory at step 625, resumed at the same budget)
Weight-space average (= the previous release)	½ round 2 + ½ stage 4, element-wise (LoRA and readout)	–	–	–	–	–
Stage 5 (phase 3), round 1	previous release, LoRA rank 16 → 64 (identical output at step 0), readout 255 → 256 codes	2	2e-5	2,640 / 2,640	8×H100	3 h 13 min
Stage 5, round 2	round 1	1	1e-5	291 / 291	8×H100	23 min

Data this size saw.

• First run: 866,854 decisions: 504,000 stage-1 decisions with their original labels (36 licence-admitted sources); 296,482 stage-2 decisions labelled by the 9B, with unknown targets capped at 15%; 66,372 photo-vs-record and two-photo decisions. 17.15% of its training targets are unknown; no base-model blend.

• Stage 3, round 1: 14,112 training records: kept teacher questions (9,368 of 13,386 kept on two-answerer agreement) plus the training share of 8,532 human reasoning items from 10 licensed sets.

• Stage 3, round 2: 7,812 training records: 3,598 new (4,852 of 8,097 kept on three-answerer agreement) + 4,214 replayed from round 1.

• Stage 4: 39,515 records: the stage-3 teacher questions relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions.

• Stage 5 (phase 3), round 1: 125,424 rows: 58,000 hard text and 32,427 hard image decisions the previous release got wrong, mined from a 210,565-candidate pool (our generators, public datasets, earlier pools, trap variants), labelled by Qwen3.6-35B-A3B (thinking) with distribution targets and kept only after an independent review (Kimi-K2.5 plus a 220-item blind human-style review; families over 5% estimated label error dropped), including 20,270 constructed chart, document, inventory, safety, geometry and screenshot decisions with answers by construction; 19,998 earlier text and 14,999 earlier photo decisions replayed. 22% of targets are unknown.

• Stage 5, round 2: 26,494 rows: the 13,247 labelled items the round-1 model still failed plus 13,247 replayed.

Compute. $499.07 of rented GPU time on RunPod through stage 3, about $177 for stage 4 (all three sizes) and, for stage 5 (4B only), $177 of training (8×H100, 6 h 20 min incl. the pilot and the full evaluation) plus about $414 of mining, teacher labelling and review (RunPod and Azure, open-weight teachers): about $1,270 for the whole project, every run included.

Full specification: https://github.com/mohit67890/imajev/blob/main/docs/technical-specification.md

Results (2026-09-26; every number reproducible from the repo's reports/)

Benchmark	imajev-4b (this version)	Previous version (same pod, same protocol)	Notes
JevBench public hard (111)	72.1% with 4 option rotations + calibration (ECE 0.082); single pass 71.2% raw (ECE 0.113), 71.2% calibrated (ECE 0.082)	70.3% served; 69.4% raw	unchanged within noise (111 items, ±3.7 pts); same protocol, our runs: JevK5 v0.2.0 73.9% / 0.073, Eikos-4B 73.9% / 0.054, Hopper 67.6% / 0.050, Qwen3.5-4B base (generation) 48.6%; a frozen Qwen3.6-35B-A3B with thinking 97.3% at seconds per decision
JevBench public original / easy	98.6% / 100%	98.6% / 100%
JevBench public, served from a Mac (MLX, 4 rotations + calibration)	71.2% hard (ECE 0.073), 98.6% original, 100% easy	70.3%	parity check of the mlx/ weights
ImajevBench v2.0-lite test (279 items: text, photo, photo+state)	83.9% (234/279); tracks text 25/37 · visual 109/120 · joint 100/122	82.4% (230/279); 26/37 · 107/120 · 97/122	Qwen3.5-4B base 70.6%; imajev-9b 82.1%; frontier APIs by structured generation 91–99.6% (a different interface)
MLX (Mac) vs PyTorch on ImajevBench public test	83.9% vs 83.9% (234/279 both), 98.6% argmax agreement (275/279); ECE 0.059 vs 0.070	82.1% vs 82.4%	parity check of the mlx/ weights against the pod run
ImajevBench private-1 hidden split (202; aggregates only)	85.6% (173/202); text 20/30 · visual 83/84 · joint 70/88; ECE 0.029	84.2%	run on a Mac (MLX weights), the previous version on the pod
Fresh held-out hard set (4,297 decisions from every generator family and the unflagged pool, never mined or trained on)	87.2%	68.4%	same generators as the training data: an in-distribution number
Human-verified slice (785 decisions, blind-reviewed labels)	76.2%	48.2%
Constructed image sets, held-out (charts 300 / documents 300 / inventory 250 / safety 250 / geometry 250 / screenshots 250)	95.7 / 90.3 / 67.6 / 92.4 / 88.4 / 95.6%	84.0 / 76.0 / 54.8 / 72.0 / 66.4 / 72.8%	answers by construction; inventory (shelf counts) stays the weakest
DecisionBench 1.0 (Hanno-Labs), full suite (23,900 rows), the benchmark's own harness, 4 rotations + calibration, 64k-token serving limit	79.7% primary, every row scored (coverage 100%); ECE 0.069; reasoning family 80.6%, ordinal scoring 46.2%	77.5% primary (79.3% scored; 537 rows unsupported); ECE 0.024; reasoning 68.0%, ordinal 40.3%	the previous version is 3rd of 55 on the public board (2026-09-25) and had the lowest ECE there; this version would also be 3rd (Bosun v3.1 1.7B 84.9 and 0.6B 81.2 ahead) with a wider lead over 4th; its record (revision c9e5f132) is submitted in the registry's PR #68 alongside the 1.0 record; it trades calibration (ECE 0.024 → 0.069) for accuracy on this suite
fastino/fast-decisions dev (1,700 rows, 17 domains; their board scores a held-out test split)	60.4% domain macro (59.4% pooled heads)	59.0% (58.4%)	not comparable to their board
Irrelevance panel (2,823)	80.0%	80.2%
Hard-question test (435): correct on Unknown-gold rows / false abstention	11/14 · 0.48%	14/14 · 0.24%	this version answers 3 of the 14 unknown-gold items (at 0.83, 0.60 and 0.51 confidence); the previous release's 14/14 ship gate was overridden for this release, see Limits
State probe / pairs probe	76.5% / 96.7%	71.5% / 98.3%
S1-Bench, typed conversion (212 of the 220 English items; our derivative, not an S1-Bench score)	99.1% (ECE 0.019, abstention 0)	98.6% (0.006)	saturated: a no-regression check on simple questions
MMLU-1000, text-only / with an unrelated photo; typed-decisions test (2,000); two-image / held-out photo sources	74.5% / 72.9%; 67.0%; 41.8% / 56.7%		measured on earlier versions, not re-run

Every number in this table was produced in one evaluation on 2026-09-26 (pod ctr4dy9bzom5ji, 8×H100) with the previous release re-measured under the identical protocol; the full table for all eight phase-3 checkpoints, with item counts and gate results, is reports/phase3/train-results/benchmarks.md in the repo. The paired cluster test of this version against the previous one on ImajevBench (89 evidence clusters) is pending; the previous release beat its untuned base by +11.8 points [+5.8, +18.0], p = 0.0006. JevBench is text-only; imajev's image capability shows only on ImajevBench and in use. The official JevBench leaderboard adds 308 sealed items and a four-axis score that only its maintainers can run; a measurement has been requested.

Calibration

calibration.json (schema 1.0) applies one temperature (1.305) to every question type × option-count bucket, fitted by negative log-likelihood on 150 template-generated JevBench-style items (none from JevBench). calibration-rot4.json is the same fit for the 4-rotation serving mode. A per-type fit on the flagged half of our held-out set was tried and rejected: it lowers hard-tier ECE but raises the pooled ECE over all public JevBench items (0.059 single / 0.039 rot4 against a 0.03 guard). Temperature scaling never changes an answer, only its probability. unknown offsets are 0.

Checked through the evaluation server (ECE, uncalibrated → with calibration.json); the off-distribution rows were measured on the previous version with its own temperature and have not been re-run for this version:

Panel	ECE
JevBench public hard, single pass (111)	0.113 raw → 0.082
JevBench public hard, 4 rotations (111)	0.079 raw → 0.082
Pooled over all 231 public JevBench items, single pass	0.064 raw → 0.046
MMLU-1000, text-only (previous version)	0.150 → 0.035
typed-decisions test (2,000; previous version)	0.149 → 0.047
SST-5 (2,210; previous version)	0.220 → 0.020
Photo-only verification (ABO + VizWiz, 823; previous version)	0.038 → 0.062

On photo-only verification the previous version's raw probabilities were already calibrated and the temperature over-softened them; if your traffic is mostly photo-against-record checks, serve without --calibration or fit your own temperature on a held-out sample.

Serving

git clone https://github.com/mohit67890/imajev && cd imajev
python3.11 -m venv .venv && . .venv/bin/activate
pip install -e ".[serve,mlx]"            # Apple silicon;  elsewhere: pip install -e ".[serve,torch]"
python scripts/download_model.py --model 4b
hf download mohit67890/imajev-4b --local-dir adapters/imajev-4b
# Mac (MLX)
PYTHONPATH=src:scripts python scripts/playground/server.py --model-bundle artifacts/model-qwen4b.json \
  --adapter adapters/imajev-4b/mlx --rotations 4 --calibration adapters/imajev-4b/calibration-rot4.json --model-name imajev-4b --port 8765
# Linux / CUDA (PyTorch + PEFT)
PYTHONPATH=src:scripts python scripts/playground/server.py --backend torch --model-bundle artifacts/model-qwen4b.json \
  --adapter adapters/imajev-4b --rotations 4 --calibration adapters/imajev-4b/calibration-rot4.json --model-name imajev-4b --port 8765

Then POST /v1/systemone with a Jev-shaped request. One forward pass per question: p50 96 ms raw on one H100 for a JevBench hard item, serially; 350 ms with the --rotations 4 (four option orders averaged, +0.9 hard) and calibration.json used for the numbers above (shared pod, under load).

Training data and provenance

Synthetic documents and typed questions written by Qwen3.6-27B, answered independently by Qwen3.6-27B (thinking), gpt-oss-20b and, in the last part of the hard-question stage, Qwen3.6-35B-A3B (thinking); a question is kept only when every answerer agrees with the intended answer. In the soft-target stage the same questions were relabelled with Qwen3.6-35B-A3B's probability distributions (thinking mode), joined by 9,880 new hard, judge and programmatic questions, the strict slice of the Eikos decisions set (10,570 rows, open-weight teachers only) and 5,000 replayed image decisions; the shipped adapter is the weight-space average of the hard-question adapter and that continuation's best checkpoint. Plus licensed image datasets and human-written states from earlier training stages (see the repo's datasheets). No JevBench items (8-gram lint), no outputs from Jev or any paid API. All teachers are open-weight, Apache-2.0.

Limits

Single-pass: no reasoning at inference, so multi-step arithmetic and answer-quality judging trail reasoning models (a frozen Qwen3.6-35B-A3B with thinking scores 97% on JevBench hard at seconds per decision); the phase-3 stage moved our own hard held-out sets by 19 to 28 points but left JevBench hard unchanged within noise. Over-confident without calibration.json. On the previous release's 14-item unknown-gold check this version abstains on 11 (the previous release on all 14) while its false-abstention rate on answerable items stays at 0.5%: it is slightly less conservative on borderline "cannot tell" cases, and that ship gate was overridden for this release. Two-image comparisons are the weakest visual task (41.8% on real pairs, measured on an earlier imajev-4b); shelf inventory counts are the weakest constructed image family (67.6%). English only. The 2B and 9B tiers are still the previous generation.