Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,178 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-0.8B-Base
+language: [en]
+pipeline_tag: text-classification
+tags: [decision-model, calibrated, structured-output, system-one, one-pass, jevcontrol]
+---
+
+# jev-control-core: a small, fast decision model for agent decision sites
+
+A single-forward-pass typed-decision model (open-spark-Jev's "System One" class, the same family as
+[spark-s1](https://huggingface.co/abhishek085/spark-s1-4b-v6)), purpose-built for a narrower target than
+spark-s1's JevBench-style general decision benchmark: the decision *sites* inside a real agent
+harness — guardrail/injection gates, tool routing, context ranking, answer sufficiency, triage,
+moderation, claim verification, next-action selection, entity matching, escalate-to-human. These are
+the ten patterns [JevControl](https://github.com/abhishek085/JevControl)'s own in-app Guide lists as
+"where a decision model tends to fit" — short state, 2-5 options, embedded in a larger agent loop, not
+an adversarial benchmark question.
+
+Base model: [Qwen/Qwen3.5-0.8B-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) (24 layers, hybrid
+gated-delta-net linear attention + full attention, hidden size 1,024). Qwen ships this size only as a
+vision-language checkpoint (`Qwen3_5ForConditionalGeneration`); this repository is the extracted
+text-only decoder (`Qwen3_5ForCausalLM`, 0.75B), full-parameter fine-tuned -- 0.8B is small enough and
+the target distribution narrow enough that a full tune is cheap and reaches higher accuracy than a
+rank-16 LoRA adapter would on this task.
+
+**Two sizes, one family:** `jev-control-core` (this repository, 0.8B dense) and
+[`jev-control-es`](https://huggingface.co/abhishek085/jev-control-es) (149M, ModernBERT encoder,
+Laya-style option-marker readout) -- pick by latency budget.
+
+## How it works
+
+Same readout as spark-s1: the state and a typed question (Choice/Score/Noul, each with an explicit
+option list) are rendered into one prompt; the answer-slot letter logits are read from a single forward
+pass, restricted to the valid option letters, and softmaxed. No decoding, no parsing, no output outside
+the options you defined. Served through the same `/v1/systemone` gateway as spark-s1
+(`open_spark_jev.serve.gateway`), so it's a drop-in smaller sibling, not a different serving stack.
+
+## Training
+
+Fully fine-tuned (all 0.75B parameters, no LoRA) for 3 epochs on 20,000 rows from
+`os_datagen.control`'s ten decision-site families (2,000 rows/family) -- code-generated, code-verified
+gold, zero overlap with any benchmark (`scripts/tools/benchmark_overlap.py`, 0/26,000 rows flagged).
+lr 3e-5, batch 16 x grad-accum 2, cosine schedule, `lambda_brier` 0.5 (same KL + Brier-regularised loss
+as spark-s1's `sft.py`). Every choice-type family's option list is rendered in a per-row-random order,
+and every family draws from substantially widened prompt/topic pools rather than a handful of fixed
+templates (see History below for why both of these matter more than they sound).
+
+## Evaluation
+
+**Own held-out splits** (`os_datagen.control`'s `test_locked`/`challenge`, same families as training,
+different generated instances, scored both at the option order the row was written in and averaged over
+3 random re-permutations of that order):
+
+| split | accuracy | accuracy (mean over 3 option-order permutations) | ECE (calibrated) |
+|---|---|---|---|
+| test_locked | 1.000 | 1.000 | 0.000 |
+| challenge | 1.000 | 1.000 | 0.000 |
+
+**Latency**, isolated single-decision calls, in-process HF backend, one NVIDIA GB10: **18.5 ms** per
+decision (median). Faster than decider-4b (32-35 ms) and well under spark-s1-4b-v6 (75 ms).
+
+**Real harness (JevControl, `support_desk` demo, 203 tasks, 4 decision sites/task, against a Gemma-4-E4B
+baseline that decides by prompting):**
+
+| arm | accuracy | p50 latency | verdict |
+|---|---|---|---|
+| Gemma 4 (prompted) | 0.892 | 2,933 ms | baseline |
+| **jev-control-core** | **0.837** | **4,563 ms*** | close (Δ -0.054) |
+
+Per decision site: injection 0.911, route 0.988, sufficiency 0.915, relevance 0.770.
+
+*\*p50 wall-clock for the harness run as a whole, not model latency alone -- this run's decider shared the
+box with other GPU work; the isolated 18.5ms/decision figure above is the honest per-call latency number.*
+
+## Why the real-harness number moved from 0.438 to 0.837 (three fixes, kept here on purpose)
+
+The first trained version of this model scored **0.438** on the real harness despite 0.956 in-distribution
+accuracy -- a large, genuine transfer gap. Three separate, diagnosed issues accounted for essentially all of
+it, and all three are worth knowing if you retrain this family on your own decision sites:
+
+1. **Templated, fixed-vocabulary synthetic state.** The `route`/`sufficient`/`relevance` families originally
+   rendered abstract, symbolic state (e.g. `"Retrieved so far: the price: known."`) instead of realistic
+   customer-message and article prose. Rewriting them to use varied greetings/sign-offs and real-looking KB
+   article text over a 12-topic corpus took the real-harness number to 0.635.
+2. **Fixed option order.** Four choice-type families (`tool_routing`, `moderation_class`, `next_action`,
+   `escalate_human`) always rendered their options in the same order every training row. The model could
+   solve every training example by learning "the answer is at position N" without ever reading the option
+   text -- invisible on our own eval (which never varies the order) but fatal the moment a real caller
+   enumerates its own options in its own order. The `route` site's real-harness confusion matrix showed the
+   unmistakable signature: a clean positional shift (`kb`->`orders` 94/203, `orders`->`human` 48/203) rather
+   than random noise. Randomizing option order per training row (`_shuffled()` in
+   `datagen-pipeline/src/os_datagen/control/families.py`) fixed `route` specifically from 0.152 to 0.978 and
+   took the overall real-harness number to 0.813.
+3. **Narrow templates within each family.** Even with (1) and (2) fixed, training loss collapsed to 0.0000
+   within the first ~100 of ~560 steps -- the 5,000-row dataset was diverse enough to fix the two bugs above
+   but still narrow enough (e.g. one family's negative case was drawn from just 8 topics) to overfit near-
+   instantly rather than learn a robust rule. Widening every family's template/topic/signal pools and scaling
+   to 20,000 rows (2,000/family) pushed the loss curve out past the first epoch and took this model's
+   real-harness number to 0.837. (The same fix made the smaller `jev-control-es` sibling's `injection` site
+   *worse*, not better -- see that model's card for why more data alone isn't a universal fix.)
+
+## Usage
+
+**As an API (recommended -- restricted-option decoding, calibration and abstain-threshold logic all live
+server-side).** Run the OpenAI-compatible gateway this repository ships with:
+
+```bash
+python -m open_spark_jev.serve.gateway --backend hf --default-model jev-control-core --port 8620
+```
+
+```bash
+curl -s http://localhost:8620/v1/systemone -H "Content-Type: application/json" -d '{
+  "model": "jev-control-core",
+  "state": "Customer: my order #A1006 never arrived.",
+  "question": {"type": "choice", "prompt": "route to:",
+               "options": ["kb", "orders", "human"]}
+}'
+```
+
+**Direct, CUDA or CPU (`transformers`):**
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tok = AutoTokenizer.from_pretrained("abhishek085/jev-control-core")
+model = AutoModelForCausalLM.from_pretrained("abhishek085/jev-control-core", torch_dtype="bfloat16", device_map="cuda")
+```
+
+Restricted-option decoding (reading only the answer-slot letter logits, softmaxed over the valid options)
+is what actually makes this a calibrated decision model rather than a free-generation chat model -- that
+logic lives in `open_spark_jev.serve.gateway` / `open_spark_jev/eval/osdg.py`, not in `transformers` alone,
+so prefer the gateway path above unless you're reimplementing that readout yourself.
+
+**Apple Silicon (verified path -- GGUF + llama.cpp Metal):** this repo includes an f16 GGUF build
+(`jev-control-core-f16.gguf`, see Files below), verified to give byte-identical top-logit ordering and
+confidence to the safetensors checkpoint on a real decision prompt. This is the Metal path to use today.
+
+**MLX:** not currently supported. `mlx-lm` does not ship a model class for Qwen3.5's hybrid
+gated-delta-net + full-attention architecture as of this writing, and this checkpoint was built and
+tested on Linux/CUDA hardware with no Apple Silicon available to verify an MLX conversion -- so rather
+than claim untested support, use the GGUF/llama.cpp Metal path above on Apple Silicon.
+
+## JevControl
+
+This model is purpose-built for the decision sites [JevControl](https://github.com/abhishek085/JevControl)'s
+own in-app Guide documents -- JevControl is the tool used to produce the real-harness numbers on this card
+(`support_desk` demo, 203 tasks) and is the recommended way to measure this model's savings against your
+own agent harness before adopting it.
+
+## Limitations
+
+* Not evaluated on JevBench: this model is intentionally scoped to JevControl-shaped decision sites, not
+  general benchmark decisions -- use spark-s1 for that.
+* No reinforcement-learning stage.
+* The remaining 0.079 real-harness gap to the Gemma-4 baseline is concentrated in `sufficiency` (0.815) and
+  `relevance` (0.801) -- both require judging whether a short article's prose actually contains an answer,
+  the hardest reading-comprehension step of the four sites, and the most likely to still carry some
+  synthetic-corpus vocabulary bias even after the fixes above.
+* Per-question-type temperatures were fitted on the synthetic calibration split only; refit on your own
+  labelled traffic before trusting confidence-gated escalation (`python -m open_spark_jev.eval.osdg` or
+  JevControl's own `python -m decider.calibrate`-equivalent path).
+
+## Files
+
+`config.json`, `tokenizer.json`/`tokenizer_config.json`, `model.safetensors` (bf16), `calibration.json`
+(per-type temperatures), `chat_template.jinja`, `generation_config.json`. A GGUF build
+(`jev-control-core-f16.gguf`, f16, no quantisation) is included for llama.cpp / Metal serving; verified to
+give byte-identical top-logit ordering and confidence to the safetensors checkpoint on a real decision
+prompt. Convert with `--no-mtp` if rebuilding from the HF checkpoint (this size of Qwen3.5 was extracted
+from a vision-language release and no longer carries the speculative-decoding head llama.cpp's converter
+otherwise expects).
+
+## Reproduction
+
+Data generation, training config and the extraction/GGUF scripts are in
+[abhishek085/open-spark-jev](https://github.com/abhishek085/open-spark-jev):
+`datagen-pipeline/src/os_datagen/control/`, `configs/train/sft_control_jev.yaml`,
+`scripts/tools/extract_qwen35_text.py`.