--- language: en license: other license_name: research-use-derived-data library_name: peft base_model: jaredpalmer/kev-0.8b base_model_relation: adapter pipeline_tag: visual-question-answering tags: - decision-model - calibration - typesafe - kev - jev - vision - qwen3.5 datasets: - Jacqkues/kev-vision-decisions-full - Jacqkues/sev-ood - Jacqkues/sev-generic-bench --- # Sev-0.8B **Sev** is a *decision model that reads images*: one image plus a state (text or JSON) and a set of typed questions go in, a calibrated probability distribution per question comes out, in a single forward pass, with no text generation. Question types are the TypeSafe `/v1/systemone` ones: `choice` (one of named options), `noul` (yes / no) and `score` (ordered levels). New questions and options are given at request time; nothing is retrained. It is [Kev-0.8B](https://huggingface.co/jaredpalmer/kev-0.8b) (a Jev-style text decision model: LoRA + pointer head on `Qwen/Qwen3.5-0.8B-Base`) plugged into the **full Qwen3.5 vision-language model** and fine-tuned on [kev-vision-decisions-full](https://huggingface.co/datasets/Jacqkues/kev-vision-decisions-full) (gated: research use, per-source licences). The vision tower is frozen; the Kev LoRA (r = 16) and pointer head are trained further. - **Status**: a research project, heavily vibe-coded with Claude Code. Not for production. - **Document version**: [Sev-0.8B-docindex](https://huggingface.co/Jacqkues/sev-0.8b-docindex) (page indexing: 21 attributes) - **Code**: `kev-vision` (`VisionDecisionModel`, training, dataset builder, evaluation) - **Served temperature**: T = 1.32, fitted on the validation split (hard-label questions only) ## Use ```python from kev.api import SystemOneRequest, to_record from kev_vision.model import VisionDecisionModel from PIL import Image model = VisionDecisionModel("Jacqkues/sev-0.8b", device="cuda").eval() # Ampere/Ada GPU, MPS or CPU (see limits) request = {"state": "A document page is attached.", "questions": {"doc_type": {"type": "choice", "instructions": "What type of document is this?", "criteria": {"invoice": "An invoice", "receipt": "A receipt", "letter": "A letter"}}, "handwritten": {"type": "noul", "instructions": "Is it mostly handwritten?"}}} rec, meta = to_record(SystemOneRequest.model_validate(request)); rec["image"] = Image.open("page.jpg") for m, p in zip(meta, model.probs(rec)): print(m["id"], dict(zip(m["keys"], p.tolist()))) ``` ## How it works ``` [state] image tokens state text | [q] instructions [opt] option ... [decide] Qwen3.5 (vision frozen, LoRA on the language model) → pointer head: · each option → softmax / T ``` Every question is its own causal row over the shared state (Qwen3.5 is a Gated DeltaNet hybrid), positions are Qwen's 3D M-RoPE, and each option is scored at its end token. On text-only input Sev's code reproduces Kev exactly (max |Δp| 5.7e-7). ## Training - **Start**: `jaredpalmer/kev-0.8b` (adapter + head). A controlled comparison on 20k records picked it over a bare Qwen3.5-0.8B-Base with a fresh LoRA of the same rank: better on 14 of 21 test sets, clearly on text (0.827 vs 0.766). - **Data**: kev-vision-decisions-full, ~126k training records: VQAv2 yes/no + counting, A-OKVQA, ScienceQA, EuroSAT, Oxford Pets, FairFace age, AVA aesthetics (soft targets), ChartNet, document types (DocLayNet, CORD receipts, invoices, RVL-CDIP), control (Pong, Breakout, Freeway, Space Invaders, Ms. Pac-Man from JAT; LeRobot PushT, upweighted ×2), text replay from Kev's sources, and blank / noise images with a uniform target. Yes/no balanced per source, option order randomised, images shared between train and held-out splits removed. - **Recipe**: 1 epoch (136,502 passages), batch 8, AdamW lr 5e-5 OneCycle, bf16 autocast, Kev's augmentation (option permutation, none-of-the-above, distractors) redrawn each pass. ~3 h on one A100 80 GB (Modal). - **Validation**: loss 0.6635, accuracy 0.763, Brier 0.31. ## Results: test splits (200 records each, served temperature) | test set | before (Kev-0.8B, zero-shot) | **Sev-0.8B** | Brier | ECE | |---|---|---|---|---| | `ai2d_ood` | 0.585 | **0.665** | 0.446 | 0.077 | | `aokvqa` | 0.77 | **0.810** | 0.267 | 0.028 | | `ava` | 0.23 | **0.640** | 0.706 | 0.397 | | `breakout` | 0.37 | **0.445** | 0.650 | 0.068 | | `chartnet` | 0.825 | **0.971** | 0.043 | 0.016 | | `countbench_ood` | 0.555 | **0.730** | 0.399 | 0.106 | | `documents` | 0.866 | **0.985** | 0.025 | 0.012 | | `eurosat` | 0.588 | **0.940** | 0.087 | 0.029 | | `fairface_age` | 0.295 | **0.595** | 0.544 | 0.059 | | `freeway` | 0.862 | **0.814** | 0.359 | 0.176 | | `mspacman` | 0.05 | **0.380** | 0.768 | 0.062 | | `pets` | 0.815 | **0.975** | 0.036 | 0.025 | | `pong` | 0.462 | **0.519** | 0.548 | 0.048 | | `pope_ood` | 0.855 | **0.880** | 0.178 | 0.051 | | `pusht` | 0.213 | **0.647** | 0.485 | 0.084 | | `rvl_cdip` | 0.567 | **0.861** | 0.202 | 0.031 | | `scienceqa` | 0.619 | **0.942** | 0.102 | 0.051 | | `spaceinvaders` | 0.35 | **0.490** | 0.667 | 0.086 | | `text` | 0.805 | **0.830** | 0.243 | 0.057 | | `text_full` | 0.844 | **0.851** | 0.214 | 0.053 | | `vqav2` | 0.796 | **0.853** | 0.210 | 0.042 | Blank images (the image carries no answer): mean confidence **0.245** against a uniform 0.238. ## Results: sev-ood (never trained on) Unseen Atari games, unseen classes, questions the image cannot answer, and corrupted images ([Jacqkues/sev-ood](https://huggingface.co/datasets/Jacqkues/sev-ood), gated, 500 records per set). *Confident errors* = wrong at p ≥ 0.9; *coverage* = share of decisions that could be automated with at most 5% of them wrong. | set | accuracy | ECE | confident errors | coverage at 5% error | |---|---|---|---|---| | `corrupted_eurosat` | **0.555** | 0.263 | 0.187 | 0.00 | | `corrupted_pets` | **0.938** | 0.018 | 0.005 | 0.98 | | `corrupted_vqav2` | **0.767** | 0.039 | 0.025 | 0.21 | | `ood_beamrider` | **0.350** | 0.081 | 0.000 | 0.00 | | `ood_boxing` | **0.162** | 0.038 | 0.000 | 0.00 | | `ood_cars` | **0.872** | 0.054 | 0.000 | 0.75 | | `ood_flowers` | **0.734** | 0.083 | 0.022 | 0.53 | | `ood_food101` | **0.886** | 0.010 | 0.012 | 0.81 | | `ood_qbert` | **0.278** | 0.011 | 0.000 | 0.00 | | `ood_seaquest` | **0.160** | 0.027 | 0.000 | 0.00 | | `unknowable` (uniform target) | mean confidence 0.396 vs chance 0.227 | | answered at ≥ 0.9: **1.2%** | | ## Results: generic vision benchmarks (never trained on) Five public benchmarks, cast as typed questions: MMStar (A-D, vision-indispensable), MMBench-en dev (1,000 sampled), MME (yes/no), RealWorldQA (A-F or yes/no) and HallusionBench (image split, yes/no). ScienceQA-sourced items were dropped (Sev trained on ScienceQA). **Contamination control**: every benchmark image was hashed (256-bit average hash) against every training image; the 305 of 6,001 items within 6 bits of a training image (mostly COCO photos via VQAv2 / A-OKVQA, and ScienceQA) are removed below. Removing them changes no conclusion: they were slightly easier for all three models, including the baseline that never saw them. The baseline is **Qwen3.5-0.8B** (the post-trained VLM of the same family) in zero-shot, scored by its next-token probability over the option letters (or Yes / No), so the same calibration metrics apply. All models see images capped at 448 x 448 pixels. Accuracy on the 5,696 clean items; gap = Sev minus Qwen, paired bootstrap 95% interval. | benchmark (chance) | n | Qwen3.5-0.8B | **Sev-0.8B** | gap [95% CI] | ECE Qwen → Sev | |---|---|---|---|---|---| | MMStar (0.25) | 1,036 | 0.460 | **0.526** | +6.6 [+3.3, +9.9] | 0.18 → **0.08** | | MMBench dev (0.26) | 927 | 0.774 | **0.825** | +5.1 [+2.6, +7.6] | **0.02** → 0.06 | | MME (0.50) | 2,206 | 0.711 | **0.771** | +6.0 [+4.2, +7.9] | 0.12 → **0.01** | | RealWorldQA (0.39) | 586 | 0.570 | **0.631** | +6.1 [+2.2, +10.2] | 0.15 → **0.05** | | HallusionBench (0.50) | 941 | 0.644 | 0.629 | −1.5 [−4.8, +1.9] | 0.15 → **0.10** | | **all** | 5,696 | 0.650 | **0.698** | **+4.7 [+3.5, +6.0]** | 0.12 → **0.02** | Confident errors (wrong at p ≥ 0.9), all clean items: 4.5% for Qwen, **1.5%** for Sev. HallusionBench (visual illusions, edited charts) is the exception: no model of this size is reliably better than chance-level guessing there. ## Results: docbench (document pages, never trained on) DocLayNet test + validation pages, three yes/no questions (table of contents? figure? table?), 676 items, labels from independent rules. See [Sev-0.8B-docindex](https://huggingface.co/Jacqkues/sev-0.8b-docindex) for the document model. | model | accuracy | AUROC | ECE | |---|---|---|---| | Kev-0.8B | 0.886 | 0.934 | 0.236 | | Sev-0.8B | 0.831 | 0.924 | **0.044** | | Sev-0.8B-docindex | **0.928** | **0.976** | **0.018** | ## Limits - **Control does not transfer**: unseen games stay near chance, and on trained games Sev is still below simply repeating the previous action (Pong 0.52 vs ~0.69). It is cautious there (no confident errors), not competent. - **Degraded satellite images**: on blurred / noised EuroSAT tiles Sev is over-confident (18.7% confident errors). - **Aesthetics (AVA) is poorly calibrated** (ECE 0.40): a subjective target. - Zero-shot classification of unseen classes is good (cars 0.87, food 0.89, flowers 0.73) but 1-4 points below the 20k-record Kev start: more task-specific training erodes a little general knowledge. - **Hardware**: on Hopper GPUs (H100) flash-linear-attention refuses the Gated DeltaNet backward with torch 2.8's Triton (incorrect results, fla #640): train on Ampere/Ada (A100, L4). On Macs the DeltaNet layers run a slow PyTorch fallback. - It answers the questions it is asked; it does not verify identity, and it must not be used to infer sensitive attributes (the training data asks apparent age only; gender and ethnicity were deliberately never asked). ## Licence The adapter and head are derived from Kev-0.8B (Apache-2.0) and Qwen3.5 (Apache-2.0), but were trained on data that includes research-only / non-commercial sources (RVL-CDIP, AVA, ScienceQA, AG News, Yelp; see the dataset card). **Use for research only.** A commercially usable version would have to be retrained on permissively licensed sources only.