Hugging Face's logo

Jacqkues
/
sev-0.8b

Visual Question Answering
PEFT
Safetensors
English
decision-model
calibration
typesafe
kev
jev
vision
qwen3.5
Model card Files
xet
 Community
sev-0.8b / README.md
Jacqkues's picture
Jacqkues
Model card
4a0a64e verified
preview code
|
Raw History Blame
7.79 kB
metadata 
language: en
license: other
license_name: research-use-derived-data
library_name: peft
base_model: jaredpalmer/kev-0.8b
base_model_relation: adapter
pipeline_tag: visual-question-answering
tags:
  - decision-model
  - calibration
  - typesafe
  - kev
  - jev
  - vision
  - qwen3.5
datasets:
  - Jacqkues/kev-vision-decisions-full
  - Jacqkues/sev-ood

Sev-0.8B

Sev is a decision model that reads images: one image plus a state (text or JSON) and a set of typed questions go in, a calibrated probability distribution per question comes out, in a single forward pass, with no text generation. Question types are the TypeSafe /v1/systemone ones: choice (one of named options), noul (yes / no) and score (ordered levels). New questions and options are given at request time; nothing is retrained.

It is Kev-0.8B (a Jev-style text decision model: LoRA + pointer head on Qwen/Qwen3.5-0.8B-Base) plugged into the full Qwen3.5 vision-language model and fine-tuned on kev-vision-decisions-full. The vision tower is frozen; the Kev LoRA (r = 16) and pointer head are trained further.

  • Demo: https://jacq-dumora--sev-demo-ui.modal.run (ID-photo check, document type, photo content, or your own questions)
  • Code: kev-vision (VisionDecisionModel, training, dataset builder, evaluation)
  • Served temperature: T = 1.32, fitted on the validation split (hard-label questions only)

Use 

from kev.api import SystemOneRequest, to_record
from kev_vision.model import VisionDecisionModel
from PIL import Image

model = VisionDecisionModel("Jacqkues/sev-0.8b", device="cuda").eval()     # Ampere/Ada GPU, MPS or CPU (see limits)
request = {"state": "A document page is attached.",
           "questions": {"doc_type": {"type": "choice", "instructions": "What type of document is this?",
                                      "criteria": {"invoice": "An invoice", "receipt": "A receipt", "letter": "A letter"}},
                         "handwritten": {"type": "noul", "instructions": "Is it mostly handwritten?"}}}
rec, meta = to_record(SystemOneRequest.model_validate(request)); rec["image"] = Image.open("page.jpg")
for m, p in zip(meta, model.probs(rec)):
    print(m["id"], dict(zip(m["keys"], p.tolist())))

How it works 

[state] <vision_start> image tokens <vision_end> state text  |  [q] instructions [opt] option ... [decide]
Qwen3.5 (vision frozen, LoRA on the language model)          →  pointer head: <decide> · each option  →  softmax / T

Every question is its own causal row over the shared state (Qwen3.5 is a Gated DeltaNet hybrid), positions are Qwen's 3D M-RoPE, and each option is scored at its end token. On text-only input Sev's code reproduces Kev exactly (max |Δp| 5.7e-7).

Training

  • Start: jaredpalmer/kev-0.8b (adapter + head). A controlled comparison on 20k records picked it over a bare Qwen3.5-0.8B-Base with a fresh LoRA of the same rank: better on 14 of 21 test sets, clearly on text (0.827 vs 0.766).
  • Data: kev-vision-decisions-full, ~126k training records: VQAv2 yes/no + counting, A-OKVQA, ScienceQA, EuroSAT, Oxford Pets, FairFace age, AVA aesthetics (soft targets), ChartNet, document types (DocLayNet, CORD receipts, invoices, RVL-CDIP), control (Pong, Breakout, Freeway, Space Invaders, Ms. Pac-Man from JAT; LeRobot PushT, upweighted ×2), text replay from Kev's sources, and blank / noise images with a uniform target. Yes/no balanced per source, option order randomised, images shared between train and held-out splits removed.
  • Recipe: 1 epoch (136,502 passages), batch 8, AdamW lr 5e-5 OneCycle, bf16 autocast, Kev's augmentation (option permutation, none-of-the-above, distractors) redrawn each pass. ~3 h on one A100 80 GB (Modal).
  • Validation: loss 0.6635, accuracy 0.763, Brier 0.31.

Results: test splits (200 records each, served temperature)

test set before (Kev-0.8B, zero-shot) Sev-0.8B Brier ECE
ai2d_ood 0.585 0.665 0.446 0.077
aokvqa 0.77 0.810 0.267 0.028
ava 0.23 0.640 0.706 0.397
breakout 0.37 0.445 0.650 0.068
chartnet 0.825 0.971 0.043 0.016
countbench_ood 0.555 0.730 0.399 0.106
documents 0.866 0.985 0.025 0.012
eurosat 0.588 0.940 0.087 0.029
fairface_age 0.295 0.595 0.544 0.059
freeway 0.862 0.814 0.359 0.176
mspacman 0.05 0.380 0.768 0.062
pets 0.815 0.975 0.036 0.025
pong 0.462 0.519 0.548 0.048
pope_ood 0.855 0.880 0.178 0.051
pusht 0.213 0.647 0.485 0.084
rvl_cdip 0.567 0.861 0.202 0.031
scienceqa 0.619 0.942 0.102 0.051
spaceinvaders 0.35 0.490 0.667 0.086
text 0.805 0.830 0.243 0.057
text_full 0.844 0.851 0.214 0.053
vqav2 0.796 0.853 0.210 0.042

Blank images (the image carries no answer): mean confidence 0.245 against a uniform 0.238.

Results: sev-ood (never trained on)

Unseen Atari games, unseen classes, questions the image cannot answer, and corrupted images (Jacqkues/sev-ood, 500 records per set). Confident errors = wrong at p ≥ 0.9; coverage = share of decisions that could be automated with at most 5% of them wrong.

set accuracy ECE confident errors coverage at 5% error
corrupted_eurosat 0.555 0.263 0.187 0.00
corrupted_pets 0.938 0.018 0.005 0.98
corrupted_vqav2 0.767 0.039 0.025 0.21
ood_beamrider 0.350 0.081 0.000 0.00
ood_boxing 0.162 0.038 0.000 0.00
ood_cars 0.872 0.054 0.000 0.75
ood_flowers 0.734 0.083 0.022 0.53
ood_food101 0.886 0.010 0.012 0.81
ood_qbert 0.278 0.011 0.000 0.00
ood_seaquest 0.160 0.027 0.000 0.00
unknowable (uniform target) mean confidence 0.396 vs chance 0.227 answered at ≥ 0.9: 1.2%

Limits

  • Control does not transfer: unseen games stay near chance, and on trained games Sev is still below simply repeating the previous action (Pong 0.52 vs ~0.69). It is cautious there (no confident errors), not competent.
  • Degraded satellite images: on blurred / noised EuroSAT tiles Sev is over-confident (18.7% confident errors).
  • Aesthetics (AVA) is poorly calibrated (ECE 0.40): a subjective target.
  • Zero-shot classification of unseen classes is good (cars 0.87, food 0.89, flowers 0.73) but 1-4 points below the 20k-record Kev start: more task-specific training erodes a little general knowledge.
  • Hardware: on Hopper GPUs (H100) flash-linear-attention refuses the Gated DeltaNet backward with torch 2.8's Triton (incorrect results, fla #640): train on Ampere/Ada (A100, L4). On Macs the DeltaNet layers run a slow PyTorch fallback.
  • It answers the questions it is asked; it does not verify identity, and it must not be used to infer sensitive attributes (the training data asks apparent age only; gender and ethnicity were deliberately never asked).

Licence

The adapter and head are derived from Kev-0.8B (Apache-2.0) and Qwen3.5 (Apache-2.0), but were trained on data that includes research-only / non-commercial sources (RVL-CDIP, AVA, ScienceQA, AG News, Yelp; see the dataset card). Use for research only unless you retrain on the public tier (Jacqkues/kev-vision-decisions).