README.md

---
license: apache-2.0
base_model: akhilaaa3/Jev-Omni
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- apple-silicon
- multimodal
- text-classification
- image-classification
- typed-decision
---

# Jev-Omni-MLX-4bit

An Apple Silicon MLX conversion of [`akhilaaa3/Jev-Omni`](https://huggingface.co/akhilaaa3/Jev-Omni) for local inference on a Mac mini with 16GB unified memory.

This repository keeps the upstream Jev-Omni unified multimodal checkpoint and its trained 256-way decision head, then converts the language-model weights to **4-bit affine quantization with group size 64**. The vision weights remain BF16 and the decision head remains FP32.

It is an independent conversion. It is not an official TypeSafe Jev release and does not claim to reproduce TypeSafe's proprietary system. It is also not a new fine-tune.

## Hardware and speed

Measured on:

- Mac mini, Apple M4, 10 CPU cores, 16GB unified memory
- macOS 26.5.1
- Python 3.13.12
- MLX 0.32.2
- MLX-VLM 0.7.1
- single request, batch size 1, no token generation

Ten warm requests were measured after one warm-up request. The image test used the recommended 20 visual-token budget and a 3-option question.

| Mode | Median | P95 | Peak Metal memory |
|---|---:|---:|---:|
| Text decision | ~963 ms | ~998 ms | ~7.0 GB |
| Image decision, 20 visual tokens | ~994 ms | ~1,021 ms | ~7.0 GB |

The first request includes MLX graph and memory warm-up. On the same machine, a 70-token image request is slower (roughly 1.8 seconds warm in an earlier run). Lowering visual tokens reduces latency but can lose small details; validate on your own game frames.

The published Jev-Omni H200 numbers are not transferable to this Mac mini. This model card reports local measurements only.

![Local warm latency](assets/local-latency.svg)

## Results

The public JevBench files were evaluated with the same typed-choice mapping used by the runtime. Temperature scaling changes probabilities only; it does not change the selected option.

| Benchmark | Accuracy / state macro | Micro accuracy | ECE-10 |
|---|---:|---:|---:|
| JevBench public · 195 groups / 231 decisions | **85.90%** | **87.88%** | 0.04497 raw / **0.03069** scaled |
| DecisionBench Medium · 293 questions | — | — | Full run not published |

![JevBench public accuracy](assets/jevbench-accuracy.svg)

The JevBench result is our local public-set measurement, not a claim that the 4-bit MLX conversion reproduces the upstream card's protocol. The upstream [Jev-Omni card](https://huggingface.co/akhilaaa3/Jev-Omni) reports its own merged-model result separately. Dataset revisions, item filtering and scoring splits must match before comparing the numbers.

## Validation

- Upstream unified verification cases: 4/4 argmax decisions matched after 4-bit conversion.
- Six simple red/blue/green circle and square image checks: 6/6 color decisions matched.
- Maximum absolute probability difference on the four upstream text cases: 0.244 in this small check.
- Public JevBench v1.2 public items (195 groups / 231 decisions) were re-run locally. Raw micro accuracy was **87.88%** and group-macro accuracy was **85.90%**. Raw ECE-10 was **0.04497**.
- A single global temperature was fit on even source rows (116 items) and checked on odd rows (115 items): `T=1.11517`. On all 231 items, ECE-10 was **0.03069** after scaling; the held-out ECE was **0.06774** versus raw **0.06261**, so this is a published post-hoc calibration artifact, not a universal confidence guarantee.
- DecisionBench medium was started with the original full states; an exact checkpoint (68/293 questions, 0 errors) is under `artifacts/benchmarks/runs/` in the source project. The partial aggregate is deliberately not presented as the official complete score because long states take tens of seconds to minutes on a 16GB Mac.
- The runtime supports the published temperature file through `--calibration calibration.json`. Accuracy/argmax is unchanged by temperature scaling; only the returned probability distribution changes.

## Installation

This release is intended for Apple Silicon. Download the repository and install the small MLX runtime:

```bash
hf download Ruiruiz30/Jev-Omni-MLX-4bit \
  --local-dir Jev-Omni-MLX-4bit

cd Jev-Omni-MLX-4bit
python3.13 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## Image decision

```bash
python -m omni_mlx.classifier \
  --model . \
  --calibration calibration.json \
  --image /path/to/frame.png \
  --state "A kart is approaching a right turn." \
  --question "Which steering action is best?" \
  --options "Turn left" "Hold center" "Turn right" \
  --image-tokens 20
```

The classifier returns candidate probabilities and the selected option. It does not generate a free-form explanation. Use `--image-tokens 70` when the scene contains small or dense visual details.

Omit `--calibration` to inspect the raw quantized probabilities. The included calibration file was fit only on the public JevBench split described above; it is not trained on a user's game or on private benchmark items.

## Benchmark artifacts

- [`benchmarks/jevbench-4bit-report.json`](benchmarks/jevbench-4bit-report.json) contains the raw and temperature-scaled aggregate metrics.
- [`calibration.json`](calibration.json) is the small runtime file consumed by `--calibration`.
- The benchmark runner and raw checkpoints remain in the source project so the long DecisionBench run can be resumed without putting the full benchmark text into this model repository.

## Calibration

Temperature scaling was fit on 116 even-indexed public JevBench rows and checked on 115 odd-indexed rows. The fitted temperature is `T=1.11516790625`. On the full public set, ECE-10 moves from `0.04497` to `0.03069`; on the held-out split it moves from `0.06261` to `0.06774`. This is a transparent post-hoc calibration file, not a guarantee of calibrated confidence on game footage.

![JevBench calibration](assets/jevbench-calibration.svg)

## Local conversion code

`omni_mlx/convert.py` contains the conversion path used for this release. The original unquantized checkpoint is not bundled here; it can be obtained from the upstream repository under its own license and terms.

## License and attribution

Apache-2.0. See [LICENSE](LICENSE) and [NOTICE.md](NOTICE.md). The upstream model card, Gemma 4 terms, and dataset rights remain authoritative for their respective components.