Archive detailed training artifacts and simplify model Files view

README.md

@@ -14,6 +14,8 @@ tags:
 
 # Jet
 
+The new **[Jet 4B release](https://huggingface.co/michaljach/jet-4b)** is available separately. This repository contains the Qwen3-0.6B model.
+
 Jet is a small decision model built on **Qwen3-0.6B**. Give it a state and named,
 typed questions; it returns choices, scores, and probabilities without generating
 free-form text. Answers always follow the requested type, but decisions can still
@@ -84,11 +86,11 @@ These are single-order adapter measurements. They are not a Decision Index
 leaderboard score. Partial public benchmark reconstructions do not cover the
 complete current suite. Cross-dataset text overlap inherited during training
 also prevents a certified decontamination claim. Performance varies by task;
-see the [evaluation report](docs/training/jet-v5/summary.md) for full methodology,
+see the [evaluation report](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/summary.md) for full methodology,
 limitations, and the separate two-order results.
 
 To reproduce training, source acquisition, audits, and evaluation, use the
-[recorded protocol](docs/training/jet-v5/protocol.md) and scripts under `scripts/`.
+[recorded protocol](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/protocol.md) and scripts under `scripts/`.
 Dataset sources, revisions, exclusion checks, configuration, and checkpoint hashes
 are recorded under `docs/training/`.
 
@@ -124,3 +126,5 @@ quantization can change probabilities.
 - `src/data/`, `scripts/`: data builders and reproducible experiments
 - `src/decision_index_engine.py`, `src/decision_index_ensemble.py`: benchmark adapters
 - `deploy/huggingface/`: hosted demo and API deployment
+
+Detailed experiment files are preserved in the [training archive](https://huggingface.co/michaljach/jet/tree/training-archive-20260924/docs/training).

TRAINING_HISTORY.md

@@ -13,7 +13,7 @@ step 2,750 by validation NLL. The fused release reached about 67.7% accuracy on
 the 3,683-row general test. A Qwen3-1.7B comparison was evaluated but not retained.
 
 The complete original tables, earlier 1,500-row experiment, ordinal evaluation,
-and Kev transfer evaluation are preserved in [initial results](docs/training/initial-results.md).
+and Kev transfer evaluation are preserved in [initial results](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/initial-results.md).
 
 ## V3 — broader decision training
 
@@ -22,7 +22,7 @@ arithmetic and code behavior. Trained on 47,313 rows for two epochs / 4,888
 steps with rank-16 LoRA and learning rate `1e-4`.
 New-family accuracy reached 80.6%; general-test accuracy regressed to 65.2%.
 
-[Experiment report](docs/training/jet-v3/summary.md)
+[Experiment report](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v3/summary.md)
 
 ## Warm start and V4 — transfer and routing
 
@@ -31,8 +31,8 @@ V4 reached 66.4% general-test accuracy and 93.3% on domain-held-out tool routing
 It improved several partial language/relevance diagnostics but regressed on code.
 A CPU-fused bf16/q8 release package was prepared separately.
 
-[Training and comparisons](docs/training/jet-next/summary.md) ·
-[Export preparation](docs/training/jet-v4-release.md)
+[Training and comparisons](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-next/summary.md) ·
+[Export preparation](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v4-release.md)
 
 ## V5 — wider intent and adversarial language tasks
 
@@ -58,12 +58,14 @@ benchmark adapter. This increases inference work and is not part of standard
 serving. Gains and regressions on fixed public benchmark samples are recorded
 separately; no overall leaderboard result or win has been established.
 
-[Results](docs/training/jet-v5/summary.md) ·
-[Protocol](docs/training/jet-v5/protocol.md) ·
-[Completion audit](docs/training/jet-v5/completion-audit.json)
+[Results](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/summary.md) ·
+[Protocol](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/protocol.md) ·
+[Completion audit](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/completion-audit.json)
 
 ## Release — 2026-09-24
 
 The current release is the V5 R2 selected checkpoint, fused with native MLX CUDA,
 with a calibrated bf16 model and validated q8 ONNX export. Export and publication
-receipts are recorded in [release notes](docs/training/jet-v5-release.md).
+receipts are recorded in [release notes](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5-release.md).
+
+Detailed experiment files are preserved in the [training archive](https://huggingface.co/michaljach/jet/tree/training-archive-20260924/docs/training).

docs/training/initial-results.md

@@ -1,77 +0,0 @@
-# Initial training results
-
-
-The released model is `jet` (`adapters/jet`, `models/jet`, and [michaljach/jet](https://huggingface.co/michaljach/jet)
-on Hugging Face). It is Qwen3-0.6B trained on `train_v2` (public data plus varied-scale score questions,
-no Claude distillation yet) on an RTX 4080: 2 epochs, 2,910 steps, best checkpoint at step 2,750 by
-validation NLL (0.485), then calibrated and fused. A Qwen3-1.7B variant was trained the same way for
-comparison (best at step 2,500, val NLL 0.420) and not kept: `jet` is better calibrated and about 1.8×
-faster, for half a point of accuracy. The test set has 3,683 rows. `emotion` makes up two-thirds of them
-and is held out of training entirely, so it measures transfer to a task the model never saw.
-
-| model                     | accuracy  | NLL      | Brier     | ECE       | trained tasks acc | held-out `emotion` acc | ms / question |
-| ------------------------- | --------- | -------- | --------- | --------- | ----------------- | ---------------------- | ------------- |
-| Qwen3-0.6B, untrained     | 46.7%     | 2.63     | 0.885     | 0.428     | 46.6%             | 46.7%                  | 6.1           |
-| **jet** (0.6B, fused)     | 67.7%     | 0.87     | **0.427** | **0.061** | 83.2%             | 60.3%                  | **6.1**       |
-| Qwen3-1.7B, untrained     | 60.5%     | 5.67     | 0.741     | 0.375     | 62.1%             | 59.8%                  | 11.1          |
-| 1.7B variant (fused)      | **68.2%** | **0.86** | 0.430     | 0.100     | **83.8%**         | **60.9%**              | 11.1          |
-
-Latency is batched eval time (batch size 4) on the 4080. Per source, `jet`: dbpedia 99%, civil_comments 92%,
-massive 92%, banking77 89%, ag_news 88%, mnli 82%, boolq 81%, yelp 72%, emotion (held out) 60%, stsb 56%.
-By type: noul 79%, choice 62%, score 61%.
-
-The larger base model learns the trained tasks better: test NLL drops on 8 of 9 of them (mnli 0.48 → 0.36,
-massive 0.33 → 0.18, yelp 0.69 → 0.58), and on its in-distribution validation data it needs almost no
-temperature correction (T 1.00–1.09, against 1.03–1.30 for `jet`). It does not transfer better. Untrained
-Qwen3-1.7B already reaches 59.8% on `emotion` zero-shot and training only takes it to 60.9%, while it becomes
-more overconfident there (ECE 0.095 → 0.144). The temperatures are fit on trained tasks, so they can't correct
-that, and it is why the 1.7B variant's overall ECE is worse. Varied training tasks, such as the
-Claude-distilled set, still look like the lever for generalization, not model size.
-
-On the ordinal set (`score_eval.jsonl`, 2,400 score questions from held-out splits):
-
-| model        | exact level | within one level | Spearman | ECE   | Spearman per source (stsb / yelp / amazon / sst5) |
-| ------------ | ----------- | ---------------- | -------- | ----- | ------------------------------------------------- |
-| `jet`        | 52.1%       | 91.4%            | 0.83     | 0.035 | 0.90 / 0.89 / 0.79 / 0.74                          |
-| 1.7B variant | 53.9%       | 93.5%            | 0.86     | 0.071 | 0.93 / 0.91 / 0.83 / 0.79                          |
-
-Fusing leaves the metrics unchanged (`jet` test accuracy 67.6% → 67.7%, 1.7B variant 68.0% → 68.2%) and
-cuts batched eval time by about a third (`jet` 9.1 → 6.1 ms, 1.7B variant 16.5 → 11.1 ms per question).
-
-### Against Kev and Jev, on Kev's out-of-domain suite
-
-![jet next to the Kev family and Jev](docs/jet-vs-kev.png)
-
-[Kev](https://github.com/jaredpalmer/kev) publishes a frozen out-of-domain suite (`transfer-v4`, 656 clean rows,
-11 sources) and per-source numbers for its family and for Jev. `jet-bench-kev` converts those rows to jet's format
-and scores them. `jet` gets **54.9%** (Brier 0.622), up from 51.7% for untrained Qwen3-0.6B. That is below
-Kev-0.8B (65.2%) and the Kev-0.5B prototype (56.1%), and far below Kev-4B/9B (≈80%) and Jev (85.7%). It holds up
-on sources that look like its training mix (SciQ 90%, QNLI 76%, TweetEval 72%) and is at or below chance on
-everything else. On the policy, rule and PAWS rows it mostly picks the same answer for every row (always "yes" on
-authorization and PAWS, always "late but accepted" on deadline). MMLU is at chance (29%). jet has never trained on
-rule-following or knowledge questions. Kev's policy and rule families are exactly the varied tasks that
-distillation is meant to add.
-
-```sh
-uv run jet-bench-kev --base-model michaljach/jet --name jet      # → docs/bench/kev-transfer-v4/jet.json
-uv run jet-bench-kev --name qwen3-0.6b-untrained                 # untrained baseline
-uv sync --extra plot && uv run jet-plot-kev                       # → docs/jet-vs-kev.png
-```
-
-### Earlier run (1,500-row test split)
-
-Before the held-out sets were versioned: public data only, 2 epochs, 2,378 steps, best checkpoint at step
-2,250. These rows come from a different split, so compare them with each other, not with the table above.
-
-| model                   | accuracy | NLL  | Brier | ECE   | trained tasks acc | held-out `emotion` acc | 1 question |
-| ----------------------- | -------- | ---- | ----- | ----- | ----------------- | ---------------------- | ---------- |
-| Qwen3-0.6B, untrained   | 46.6%    | 2.66 | 0.886 | 0.428 | 44.6%             | 47.6%                  | 82 ms      |
-| Jet step 1,000          | 65.8%    | 0.92 | 0.462 | 0.101 | 79.9%             | 58.8%                  | 59 ms      |
-| Jet final (fused)       | 66.7%    | 0.86 | 0.441 | 0.079 | 82.7%             | 58.7%                  | 59 ms      |
-
-Latency is for one question on an M2 Pro. Ten questions about one ~640-token state take about 500 ms
-together, because the state is encoded once.
-
-The second epoch helped the trained tasks (79.9% → 82.7%) but not the unseen one (58.8% → 58.7%). More
-varied training tasks, such as the Claude-distilled set, are the likely lever for generalization.
-Jet has not been benchmarked against the hosted Jev API yet; `jet-bench-jev` does that.

docs/training/jet-next.md

@@ -1,64 +0,0 @@
-# Qwen3-0.6B transfer experiment
-
-User target: best possible local result while keeping Qwen3-0.6B. No hosted API calls, publication, or submission. This is a bounded follow-up experiment, not a claim that a leaderboard score has been beaten.
-
-## Protocol fixed before post-training testing
-
-1. Warm-start released Jet's local rank-16 adapter on immutable `train_v3.jsonl`, one epoch, LR 2e-5, seed 230923, 4096 batch tokens, 1024 state tokens. Fresh AdamW state and schedule. Select by `selection_v3` NLL. This ablation isolates warm-start/lower learning rate from new data.
-2. Continue its best validation checkpoint on `train_v4_tools_focus.jsonl`, one epoch, LR 1e-5, seed 230924, 3072 batch tokens, 2048 state tokens. Fresh optimizer. This mixture retains 18,000 V3 rows sampled without replacement, balanced by source, and adds all 9,000 V4 transfer rows plus 3,000 native intent-routing rows. Select by `selection_v4_tools` NLL. Capped inverse-frequency weights apply only to new stance/contract classes; validation is unweighted.
-3. Fit temperatures on independent calibration files. Compare both candidates with released Jet and V3 on unchanged old test sets and the 500-request partial diagnostic. Evaluate new template/document/article-held-out tests at 2048 state tokens. No test results choose a checkpoint, epoch, or temperature.
-
-All runs use base revision `42096995f6402fde107068cf530136fe64b604f8`, all-layer LoRA rank 16, scale 10, dropout .05, gradient checkpointing, BF16 base, cache limit 1 GB. All trained checkpoint directories are new. Initial weights remain eligible if fine-tuning never improves validation; final optimizer/weights are retained separately in `last/`.
-
-The local released adapter was compared tensor by tensor after CPU BF16 fusion against the unchanged published fused model. Tensor names match, 120 tensors match exactly, 190 differ by at most 0.00048828125 (consistent with fusion rounding). This is a close numerical reconstruction, not bit-identical fused inference. See `jet-next/baseline-adapter-check.json`.
-
-## New data
-
-- [VAST](https://github.com/emilyallaway/zero-shot-stance), revision `e7c4775182b184730f350995f8260579c9e066fe`, `data/VAST/vast_train.csv`. Native development/test posts are excluded. Related articles and identical documents stay together. 3,000 training examples; 240 each validation/calibration/test.
-- [ContractNLI](https://github.com/stanfordnlp/contract-nli), revision `eced6528dd3c1d14d73f9a87df8f7bdbc03126f9`, archive `resources/contract-nli.zip`, member `contract-nli/train.json`. Full contracts only, fitting 2048 state tokens. Native development/test document text is excluded; one duplicate train/test document was removed. 3,000 training, 240 validation, 238 calibration, 240 test examples. These are sampled from native training documents, not the native evaluation labels.
-- Local executable Python: 3,000 training, 240 each validation/calibration/test. Eighteen complete templates: twelve train, two validation, two calibration, two test. Loops, early exits, accumulators, filtering, ordering, slicing and list mutation. Counterfactual inputs supply plausible distractors. Options are shuffled and anonymous keys reassigned; every answer is checked by execution.
-
-- [Schema-Guided Dialogue](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), revision `e852981ae34990f4358979625854259302feaa78`, native `train/` only. 3,000 training and 240 each validation/calibration/test, split by entire service domain. Native same-service intents plus NONE are the candidate set; complete dialogue prefixes determine the native active intent. All 3,720 annotations verified. Native intent routing is distinct from ToolRet ranking or BFCL execution. ToolRet public evaluation queries are exclusion-only.
-
-Builder: `scripts/build_v4_transfer.py`; audit: `scripts/audit_v4_transfer.py`; retention mixture: `scripts/prepare_v4_focus.py`. Data hashes, source hashes, source revisions, group identities, class weights and exclusions are in `data/train_v4.provenance.json`. Existing V3, validation and test files are unchanged. V4 uses 56,313 rows before retention subsampling; initial focus mixture uses 27,000; the final tools-augmented focus mixture uses 30,000. `scripts/build_v4_tools.py` and `scripts/audit_v4_tools.py` create/audit this extension, with provenance in `data/train_v4_tools.provenance.json`.
-
-Limitations: exact normalized matching is not semantic paraphrase detection; contracts longer than 2048 state tokens do not train; code templates share Python primitives; V3 tool tests remain small/easy and do not establish tool-ranking quality. Neither the frozen suite nor all its content is available for exclusion auditing.
-
-## Scoring target
-
-The official leaderboard snapshot at Space commit `eec4fe1b0a8f55eb150ae537715ff35cf211f47b` reports 19 panel benchmarks, with five equally weighted categories. The installed pinned reproduction kit's `index-panel.json` is an older proposal and differs from this deployed panel. Consequently, use native benchmark metrics from partial diagnostics, not its category totals or a reconstructed headline score.
-
-Actual deployed panel: knowledge 24/25/30/43/44/31; language 11/40/41; retrieval 36/37; tools 1/2/6; arts 20/21/22/23/50. Snapshot archived as `jet-next/leaderboard-snapshot.json`. [Official source](https://huggingface.co/spaces/multimodalart/jev-decision-index/tree/eec4fe1b0a8f55eb150ae537715ff35cf211f47b).
-
-The public frozen-suite dataset still returned repository-not-found at this experiment's start. Partial results cannot establish a leaderboard win.
-
-## Reproduction
-
-After the V3 prerequisites and sources are available:
-
-```sh
-.venv/bin/python scripts/build_v4_transfer.py
-.venv/bin/python scripts/audit_v4_transfer.py
-.venv/bin/python scripts/prepare_v4_focus.py
-.venv/bin/python scripts/build_v4_tools.py
-.venv/bin/python scripts/audit_v4_tools.py
-bash train_cuda.sh --init-adapter adapters/jet \
-  --base-revision 42096995f6402fde107068cf530136fe64b604f8 \
-  --train data/train_v3.jsonl --val data/selection_v3.jsonl \
-  --out adapters/jet-v3-warm-20260924 --epochs 1 --lr 2e-5 \
-  --eval-every 250 --val-limit 10000 --seed 230923
-# prepare_v4_focus also creates the 64-row training-only smoke file.
-bash scripts/run_next_cuda.sh
-```
-
-Commands refuse existing dataset/checkpoint outputs. Logs are in `logs/jet-next/`; validation metrics and exact configurations are in each adapter directory. Warm-run source snapshot is preserved separately because weighted-example support was added while that already-loaded process was running.
-
-## Additional evaluation coverage
-
-Amazon ESCI was rebuilt in a separate directory with the pinned official normalizer and original test partition: 5,000 selected requests, uncompressed SHA256 `db1d2d3d4dde242dc04262149153bfd596044ad8986e0ae5de8443383973e386`. A deterministic complete-group 500-request sample is reserved for comparison after training. Original eight-benchmark artifacts remain unchanged. This remains a partial public reconstruction, not the frozen suite. `scripts/evaluate_next_extra.sh` compares the same rows across released Jet, V3, warm-start and V4.
-
-The initial diagnostic contains only 19 English-A sarcasm examples. All 1,400 available English-A rows were therefore selected, solely by track ID, for a separate fixed post-training evaluation. Its metric is positive-class F1 from the official static scorer, whose headline track is `iSarcasmEval-A-En`. No aggregate score is inferred from this single track. Sample provenance is in `jet-next/isarcasm-english-sample.json`.
-
-Evaluation context is explicit: original test/score_eval/V3 heldout data use 4096 state tokens, matching the earlier baseline inference default; all new V4 heldouts use 2048. Calibration uses 4096. Training context remains 1024 for the warm ablation and 2048 for V4. The distinction between training and inference limits is intentional and recorded.
-
-After the fixed 55-case CRUXEval diagnostic showed a regression, evaluation was expanded to all 570 available eligible CRUXEval requests after official exclusions, across all four models. This is a targeted regression check, not another training or checkpoint-selection round. No training settings, weights, or calibration were adjusted using that result. `scripts/evaluate_next_code.sh` records these runs separately.

docs/training/jet-next/baseline-adapter-check.json

@@ -1,6 +0,0 @@
-{
-  "keys_match": true,
-  "exact_tensors": 120,
-  "different_tensors": 190,
-  "max_abs_difference": 0.00048828125
-}

docs/training/jet-next/baseline-score_eval.json

@@ -1,65 +0,0 @@
-{
-  "overall": {
-    "n": 2400,
-    "acc": 0.5170833333333333,
-    "nll": 1.0893632032843228,
-    "brier": 0.5287154496496189,
-    "ece": 0.03900318566709756
-  },
-  "by_source": {
-    "stsb": {
-      "n": 600,
-      "acc": 0.5216666666666666,
-      "nll": 1.1038147784073833,
-      "brier": 0.33254204642705393,
-      "ece": 0.04424463902910554,
-      "mae": 0.10265430813162432,
-      "within1": 0.88,
-      "spearman": 0.9022189976505913
-    },
-    "yelp": {
-      "n": 600,
-      "acc": 0.6033333333333334,
-      "nll": 0.8791397150792182,
-      "brier": 0.5038029301388466,
-      "ece": 0.04722331101695697,
-      "mae": 0.12401153839202986,
-      "within1": 0.97,
-      "spearman": 0.8922031203071111
-    },
-    "sst5": {
-      "n": 600,
-      "acc": 0.475,
-      "nll": 1.198184583062927,
-      "brier": 0.6437840936114044,
-      "ece": 0.031442425797382975,
-      "mae": 0.1737434748756156,
-      "within1": 0.9066666666666666,
-      "spearman": 0.7429955533970433
-    },
-    "amazon": {
-      "n": 600,
-      "acc": 0.4683333333333333,
-      "nll": 1.1763137365877627,
-      "brier": 0.6347327284211705,
-      "ece": 0.09294485206405323,
-      "mae": 0.17271815862875275,
-      "within1": 0.8966666666666666,
-      "spearman": 0.7921592534430197
-    }
-  },
-  "by_type": {
-    "score": {
-      "n": 2400,
-      "acc": 0.5170833333333333,
-      "nll": 1.0893632032843228,
-      "brier": 0.5287154496496189,
-      "ece": 0.03900318566709756,
-      "mae": 0.14328187000700565,
-      "within1": 0.9133333333333333,
-      "spearman": 0.832342527754484
-    }
-  },
-  "ms_per_question": 8.335110699583007,
-  "max_state_tokens": 4096
-}

docs/training/jet-next/baseline-test.json

@@ -1,112 +0,0 @@
-{
-  "overall": {
-    "n": 3683,
-    "acc": 0.6760792831930491,
-    "nll": 0.871394003472334,
-    "brier": 0.42748423587682505,
-    "ece": 0.06219506579315473
-  },
-  "by_source": {
-    "dbpedia": {
-      "n": 90,
-      "acc": 0.9888888888888889,
-      "nll": 0.04892235172964218,
-      "brier": 0.020441859866866866,
-      "ece": 0.002619326776928345
-    },
-    "boolq": {
-      "n": 145,
-      "acc": 0.8,
-      "nll": 0.4566301124234652,
-      "brier": 0.2894317164731869,
-      "ece": 0.04909670681789005
-    },
-    "emotion": {
-      "n": 2500,
-      "acc": 0.6036,
-      "nll": 1.061848826275766,
-      "brier": 0.5326494158453977,
-      "ece": 0.09432493007183075
-    },
-    "ag_news": {
-      "n": 135,
-      "acc": 0.8814814814814815,
-      "nll": 0.3042688087249796,
-      "brier": 0.1724398602199645,
-      "ece": 0.030876177770120094
-    },
-    "stsb": {
-      "n": 117,
-      "acc": 0.5641025641025641,
-      "nll": 1.133929574747437,
-      "brier": 0.3139340576398371,
-      "ece": 0.06830696112070328,
-      "mae": 0.09924938618105988,
-      "within1": 0.8803418803418803,
-      "spearman": 0.8588398930237986
-    },
-    "mnli": {
-      "n": 142,
-      "acc": 0.8169014084507042,
-      "nll": 0.47927640531290794,
-      "brier": 0.268308599548532,
-      "ece": 0.08220711266490774
-    },
-    "yelp": {
-      "n": 146,
-      "acc": 0.7191780821917808,
-      "nll": 0.6889766353144221,
-      "brier": 0.3981368921297408,
-      "ece": 0.08682229931223884
-    },
-    "banking77": {
-      "n": 140,
-      "acc": 0.8857142857142857,
-      "nll": 0.38170325982251335,
-      "brier": 0.15168387780774686,
-      "ece": 0.053875945082732635
-    },
-    "massive": {
-      "n": 119,
-      "acc": 0.9159663865546218,
-      "nll": 0.33522919206459373,
-      "brier": 0.11903663498667257,
-      "ece": 0.026440266551089928
-    },
-    "civil_comments": {
-      "n": 149,
-      "acc": 0.9194630872483222,
-      "nll": 0.3247184777021575,
-      "brier": 0.04936214179418669,
-      "ece": 0.07150244792835826
-    }
-  },
-  "by_type": {
-    "choice": {
-      "n": 2267,
-      "acc": 0.6241729157476842,
-      "nll": 1.0613521105481916,
-      "brier": 0.5002384772375326,
-      "ece": 0.08481803025029859
-    },
-    "noul": {
-      "n": 1185,
-      "acc": 0.7881856540084389,
-      "nll": 0.48529899323149145,
-      "brier": 0.29418788896638004,
-      "ece": 0.03554945349190307
-    },
-    "score": {
-      "n": 231,
-      "acc": 0.6103896103896104,
-      "nll": 0.9877903601581737,
-      "brier": 0.39727776801601805,
-      "ece": 0.05583026837476918,
-      "mae": 0.10864421681236895,
-      "within1": 0.9307359307359307,
-      "spearman": 0.8929563754198246
-    }
-  },
-  "ms_per_question": 7.412202802334466,
-  "max_state_tokens": 4096
-}

docs/training/jet-next/baseline-test_v3_new.json

@@ -1,92 +0,0 @@
-{
-  "overall": {
-    "n": 1364,
-    "acc": 0.4567448680351906,
-    "nll": 1.3997251274893088,
-    "brier": 0.7614422661756503,
-    "ece": 0.28511070977907377
-  },
-  "by_source": {
-    "v3:stance": {
-      "n": 160,
-      "acc": 0.46875,
-      "nll": 1.3125342436600476,
-      "brier": 0.7248338334245858,
-      "ece": 0.18823765367269515
-    },
-    "v3:entailment": {
-      "n": 160,
-      "acc": 0.66875,
-      "nll": 0.7707340998575092,
-      "brier": 0.445276258289386,
-      "ece": 0.10283493064343929
-    },
-    "v3:sarcasm_irony": {
-      "n": 160,
-      "acc": 0.50625,
-      "nll": 1.2612553351966198,
-      "brier": 0.7811341016274713,
-      "ece": 0.33411078229546554
-    },
-    "v3:tool_selection": {
-      "n": 59,
-      "acc": 1.0,
-      "nll": 0.01032785861358299,
-      "brier": 0.00037024526307215803,
-      "ece": 0.010188460350036621
-    },
-    "v3:code_behavior": {
-      "n": 160,
-      "acc": 0.2,
-      "nll": 1.8659685906022787,
-      "brier": 0.9681135361388087,
-      "ece": 0.350784395635128
-    },
-    "v3:document_relevance": {
-      "n": 160,
-      "acc": 0.54375,
-      "nll": 1.0387783478654455,
-      "brier": 0.6880096149279901,
-      "ece": 0.33221421800553796
-    },
-    "v3:arithmetic": {
-      "n": 160,
-      "acc": 0.20625,
-      "nll": 2.169761631079018,
-      "brier": 0.9876360526115023,
-      "ece": 0.32759805452078583
-    },
-    "v3:product_relevance": {
-      "n": 160,
-      "acc": 0.25625,
-      "nll": 2.6546468422631735,
-      "brier": 1.2787821977853926,
-      "ece": 0.6251898461952805
-    },
-    "v3:boolean_rules": {
-      "n": 160,
-      "acc": 0.51875,
-      "nll": 0.8461553640197963,
-      "brier": 0.615775613120263,
-      "ece": 0.2303109876811505
-    },
-    "v3:tool_relevance": {
-      "n": 25,
-      "acc": 1.0,
-      "nll": 0.05768870040774345,
-      "brier": 0.010224533000073113,
-      "ece": 0.05483078479766846
-    }
-  },
-  "by_type": {
-    "choice": {
-      "n": 1364,
-      "acc": 0.4567448680351906,
-      "nll": 1.3997251274893088,
-      "brier": 0.7614422661756503,
-      "ece": 0.28511070977907377
-    }
-  },
-  "ms_per_question": 9.546318121700155,
-  "max_state_tokens": 4096
-}

docs/training/jet-next/baseline-test_v4_new.json

@@ -1,43 +0,0 @@
-{
-  "overall": {
-    "n": 720,
-    "acc": 0.3416666666666667,
-    "nll": 1.4211787942176064,
-    "brier": 0.8029851960497246,
-    "ece": 0.23449580069217418
-  },
-  "by_source": {
-    "v4:stance_transfer": {
-      "n": 240,
-      "acc": 0.32916666666666666,
-      "nll": 1.603874272511651,
-      "brier": 0.8979469405703165,
-      "ece": 0.3367636108150085
-    },
-    "v4:code_transfer": {
-      "n": 240,
-      "acc": 0.3,
-      "nll": 1.3962707759191593,
-      "brier": 0.7587345977402417,
-      "ece": 0.11225773791472118
-    },
-    "v4:contract_entailment": {
-      "n": 240,
-      "acc": 0.3958333333333333,
-      "nll": 1.2633913342220089,
-      "brier": 0.7522740498386156,
-      "ece": 0.2594969267646472
-    }
-  },
-  "by_type": {
-    "choice": {
-      "n": 720,
-      "acc": 0.3416666666666667,
-      "nll": 1.4211787942176064,
-      "brier": 0.8029851960497246,
-      "ece": 0.23449580069217418
-    }
-  },
-  "ms_per_question": 21.310045005556177,
-  "max_state_tokens": 2048
-}

docs/training/jet-next/baseline-test_v4_tools.json

@@ -1,29 +0,0 @@
-{
-  "overall": {
-    "n": 240,
-    "acc": 0.3625,
-    "nll": 1.7029224928080415,
-    "brier": 0.8800043840538005,
-    "ece": 0.3612365548809369
-  },
-  "by_source": {
-    "v4:tool_intent_routing": {
-      "n": 240,
-      "acc": 0.3625,
-      "nll": 1.7029224928080415,
-      "brier": 0.8800043840538005,
-      "ece": 0.3612365548809369
-    }
-  },
-  "by_type": {
-    "choice": {
-      "n": 240,
-      "acc": 0.3625,
-      "nll": 1.7029224928080415,
-      "brier": 0.8800043840538005,
-      "ece": 0.3612365548809369
-    }
-  },
-  "ms_per_question": 21.0819619958329,
-  "max_state_tokens": 2048
-}

docs/training/jet-next/build-v4-snapshot.py

@@ -1,169 +0,0 @@
-"""Add training-partition stance/contracts and executable multi-step Python tasks.
-
-Never uses evaluation labels. Existing files are read-only exclusion/retention inputs.
-New holdouts are grouped by article/document or whole program template.
-"""
-import csv
-import hashlib
-import json
-import random
-import zipfile
-from collections import Counter, defaultdict
-from pathlib import Path
-
-from data.decision_training import ROOT, atoms, choice, digest, normalized, read, statekey, write
-from format import Question, render_state, build_prompt
-from transformers import AutoTokenizer
-
-SEED = 230924
-RAW = ROOT / 'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw'
-# Distinct complete templates are assigned to each split before generating inputs.
-PROGRAMS = [
-    'out = []\nfor v in x:\n    if v > k:\n        out.append(v * 2)\nreturn out',
-    'out = 0\nfor i, v in enumerate(x):\n    out += v if i % 2 == 0 else -v\nreturn out + k',
-    'out = []\nfor v in x:\n    if v not in out:\n        out.append(v)\nreturn out[k:]',
-    'out = []\nfor i in range(1, len(x)):\n    out.append(x[i] - x[i-1])\nreturn out[:k]',
-    'out = 1\nfor v in x:\n    if v != 0:\n        out *= v\nreturn out + k',
-    'out = []\nfor v in x:\n    if v >= k:\n        out.insert(0, v)\nreturn out',
-    'out = 0\nfor v in x:\n    if v == k:\n        break\n    out += v\nreturn out',
-    'out = []\nfor v in x:\n    if v % 2:\n        continue\n    out.append(v + k)\nreturn out',
-    'out = []\nfor a, b in zip(x, x[1:]):\n    if a < b:\n        out.append(a + b + k)\nreturn out',
-    'out = []\nfor v in x:\n    out.append(sum(x[:len(out)+1]))\nreturn out[k:]',
-    'out = sorted(x)\nif k % 2:\n    out.reverse()\nreturn out[:k]',
-    'out = []\nfor v in x:\n    if x.count(v) > k and v not in out:\n        out.append(v)\nreturn out',
-    'out = 0\nfor v in reversed(x):\n    out = out * 2 + v\nreturn out - k',
-    'out = []\nfor i, v in enumerate(x):\n    if v > sum(x[:i]):\n        out.append(i)\nreturn out[k:]',
-    'out = []\nfor v in x:\n    if out and out[-1] == v:\n        out.pop()\n    else:\n        out.append(v)\nreturn out + [k]',
-    'out = 0\nfor a in x:\n    for b in x:\n        if a + b == k:\n            out += 1\nreturn out',
-    'out = list(x)\nfor i in range(k):\n    out = out[1:] + out[:1]\nreturn out',
-    'out = []\nfor v in x:\n    out.append(v)\n    if v == k:\n        out.append(-v)\nreturn out',
-]
-
-def program_rows():
-    safe = dict(enumerate=enumerate, range=range, len=len, zip=zip, sum=sum,
-                sorted=sorted, reversed=reversed, list=list)
-    for tid, body in enumerate(PROGRAMS):
-        split = 'train' if tid < 12 else 'val' if tid < 14 else 'calibration' if tid < 16 else 'test'
-        code = 'def f(x, k):\n' + '\n'.join('    '+line for line in body.splitlines())
-        ns = {}; exec(code, {'__builtins__': {}, **safe}, ns); fn = ns['f']
-        rng = random.Random(SEED + tid)
-        for i in range(300 if split == 'train' else 120):
-            x = [rng.randint(-4, 7) for _ in range(rng.randint(3, 8))]; k = rng.randint(0, 4)
-            result = fn(x.copy(), k)
-            # Counterfactual inputs/parameters produce plausible same-type distractors.
-            alternatives = [fn(x.copy(), k+1), fn(x[::-1], k), fn(x[1:], k), fn(x+[k], k)]
-            alternatives += ([result+[k], result+[k+1], result+[k+2]] if isinstance(result, list)
-                             else [result-1, result+1, result+2])
-            vals = list(dict.fromkeys([repr(result)] + [repr(v) for v in alternatives]))[:4]
-            options = {f'candidate_{j}': v for j,v in enumerate(vals)}
-            r = choice({'code':code,'arguments':{'x':x,'k':k}}, 'What does f(**arguments) return?',
-                       options,'candidate_0','code_behavior',f'local-v4:{tid}:{i}',f'v4-template:{tid}')
-            # choice randomizes semantic keys again for this family.
-            r['family']='code_transfer'; r['source']='v4:code_transfer'
-            r['provenance'].update(split=split, oracle={'result':result,'template':tid}, original_split='generated')
-            yield r
-
-def main():
-    data=ROOT/'data'; outputs=['train_v4.jsonl','selection_v4.jsonl','calibration_v4.jsonl','test_v4_new.jsonl']
-    if any((data/p).exists() for p in outputs): raise SystemExit('Refusing to overwrite existing v4 files')
-    tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8')
-    old_train=read(data/'train_v3.jsonl')
-    protected=[data/p for p in ['val.jsonl','test.jsonl','score_eval.jsonl','val_v3_new.jsonl','calibration_v3_new.jsonl','test_v3_new.jsonl']]
-    protected += [ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl']
-    held_atoms={a for p in protected for r in read(p) for a in atoms(r['state'])}
-    held_states={statekey(r['state']) for p in protected for r in read(p)}
-    seen={digest([r['state'],r['question']]) for r in old_train}
-    old_atoms={a for r in old_train for a in atoms(r['state'])}
-    sources=[]; candidates=list(program_rows()); excluded=Counter()
-    vast=RAW/'vast/data/VAST/vast_train.csv'
-    sources.append(dict(repository='https://github.com/emilyallaway/zero-shot-stance',revision='e7c4775182b184730f350995f8260579c9e066fe',file='data/VAST/vast_train.csv',sha256=hashlib.sha256(vast.read_bytes()).hexdigest(),split='train'))
-    # Native development/test posts are exclusion-only, including all topic variants.
-    blocked_posts={normalized(r['post']) for f in ['vast_dev.csv','vast_test.csv'] for r in csv.DictReader((vast.parent/f).open())}
-    for r in csv.DictReader(vast.open()):
-        if normalized(r['post']) in blocked_posts: excluded['vast_native_heldout_post']+=1; continue
-        row=choice({'text':r['post'],'target':r['new_topic']},'What stance does the author express towards the target?',
-            dict(against='opposes the target',favor='supports the target',none='no clear stance towards the target'),
-            ['against','favor','none'][int(r['label'])],'stance_transfer','vast-train:'+r['new_id'],'vast-article:'+r['arc_id'])
-        candidates.append(row)
-    archive=RAW/'repos/contractnli/resources/contract-nli.zip'
-    sources.append(dict(repository='https://github.com/stanfordnlp/contract-nli',revision='eced6528dd3c1d14d73f9a87df8f7bdbc03126f9',file='resources/contract-nli.zip!contract-nli/train.json',sha256=hashlib.sha256(archive.read_bytes()).hexdigest(),split='train'))
-    with zipfile.ZipFile(archive) as z:
-        contracts=json.loads(z.read('contract-nli/train.json'))
-        held_contracts={normalized(d['text']) for s in ['dev','test'] for d in json.loads(z.read(f'contract-nli/{s}.json'))['documents']}
-    for d in contracts['documents']:
-        if normalized(d['text']) in held_contracts: excluded['contract_native_heldout_text']+=1; continue
-        # Keep the full contract. Never assign a full-document label to a cropped context.
-        for lid,a in d['annotation_sets'][0]['annotations'].items():
-            row=choice({'contract':d['text'],'hypothesis':contracts['labels'][lid]['hypothesis']},
-                'Does the full contract entail, contradict, or leave the hypothesis undetermined?',
-                dict(Entailment='entailed by the contract',Contradiction='contradicted by the contract',NotMentioned='not determined by the contract'),
-                a['choice'],'contract_entailment',f'contractnli-train:{d["id"]}:{lid}',f'contractnli-document:{d["id"]}')
-            candidates.append(row)
-    # Same normalized document/post stays grouped even if source IDs differ.
-    from data.decision_training import Components
-    components=Components()
-    for r in candidates:
-        group=r['provenance']['group']; s=r['state']
-        text=s.get('contract',s.get('text'))
-        if text: components.union(group,'document:'+digest(normalized(text)))
-    forced_train=set()
-    for r in candidates:
-        s=r['state']; text=s.get('contract',s.get('text'))
-        if text and digest(normalized(text)) in old_atoms:
-            forced_train.add(components.find(r['provenance']['group']))
-    bins=defaultdict(list)
-    for r in candidates:
-        p=r['provenance'];s=r['state'];text=s.get('contract',s.get('text'))
-        # Global hypotheses recur by design; only document content is an exclusion key.
-        if statekey(s) in held_states or (text and digest(normalized(text)) in held_atoms):
-            excluded['existing_evaluation_content']+=1;continue
-        signature=digest([s,r['question']])
-        if signature in seen: excluded['duplicate']+=1;continue
-        seen.add(signature)
-        if len(tokenizer.encode(render_state(s),add_special_tokens=False))>2048:
-            excluded['over_2048_full_state']+=1;continue
-        if len(tokenizer.encode(build_prompt(tokenizer,s,Question.from_dict(r['question'])),add_special_tokens=False))>2304:
-            excluded['over_2304_full_prompt']+=1;continue
-        group=components.find(p['group']); b=int(digest([SEED,group])[:8],16)/2**32
-        split=p.get('split') or ('train' if group in forced_train or b<.8 else 'val' if b<.87 else 'calibration' if b<.94 else 'test')
-        p.update(component=group,split=split);r['source']='v4:'+r['family'];bins[(split,r['family'])].append(r)
-    splits=defaultdict(list);counts={}
-    for (split,family),rows in sorted(bins.items()):
-        random.Random(SEED).shuffle(rows)
-        classes=defaultdict(list)
-        for r in rows:classes[r['gold'] if family!='code_transfer' else 'all'].append(r)
-        selected=[];limit=3000 if split=='train' else 240
-        while classes and len(selected)<limit:
-            for key in list(classes):
-                selected.append(classes[key].pop())
-                if not classes[key]:del classes[key]
-                if len(selected)==limit:break
-        splits[split].extend(selected);counts[split+':'+family]=len(selected)
-    owners={}
-    for split,rows in splits.items():
-        for r in rows:
-            for key in [r['provenance']['component'],r['provenance']['origin'],statekey(r['state'])]:
-                assert owners.setdefault(key,split)==split,(key,split)
-    payload={outputs[0]:old_train+splits['train'],outputs[1]:read(data/'selection_v3.jsonl')+splits['val'],
-             outputs[2]:read(data/'calibration_v3.jsonl')+splits['calibration'],outputs[3]:splits['test']}
-    class_weights={}
-    for family in ['contract_entailment','stance_transfer']:
-        rows=[r for r in splits['train'] if r['family']==family]
-        counts_by_class=Counter(r['gold'] for r in rows)
-        raw={k:min(3.0,len(rows)/(len(counts_by_class)*n)) for k,n in counts_by_class.items()}
-        mean=sum(raw[k]*n for k,n in counts_by_class.items())/len(rows)
-        weights={k:w/mean for k,w in raw.items()};class_weights[family]=weights
-        for r in rows:r['weight']=weights[r['gold']]
-    hashes={}
-    for name,rows in payload.items():
-        random.Random(SEED).shuffle(rows);write(data/name,rows);hashes[name]=hashlib.sha256((data/name).read_bytes()).hexdigest()
-    report=dict(seed=SEED,sources=sources,counts=counts,excluded=dict(excluded),files=hashes,class_weights=class_weights,
-        code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
-        retained_train_v3_sha256=hashlib.sha256((data/'train_v3.jsonl').read_bytes()).hexdigest(),
-        protected_sha256={str(p.relative_to(ROOT)):hashlib.sha256(p.read_bytes()).hexdigest() for p in protected},
-        limitations=['Full frozen suite remains unavailable.', 'ContractNLI restricted to complete contracts fitting 2048 state tokens.',
-        'Source article/document groups are held together. Code holdouts use disjoint complete templates, but share Python primitives.',
-        'Exact normalized content exclusion is not a semantic paraphrase detector.', 'V3 tool tests remain small and easy; do not interpret their accuracy as strong tool retrieval evidence.'])
-    (data/'train_v4.provenance.json').write_text(json.dumps(report,indent=2));print(json.dumps(report,indent=2))
-
-if __name__=='__main__': main()

docs/training/jet-next/check-baseline-adapter.py

@@ -1,18 +0,0 @@
-import json
-from pathlib import Path
-import mlx.core as mx
-from huggingface_hub import snapshot_download
-mx.set_default_device(mx.cpu)
-base=Path(snapshot_download('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8'))
-b=mx.load(str(base/'model.safetensors')); a=mx.load('adapters/jet/adapters.safetensors'); pub=mx.load('models/jet/model.safetensors')
-report={'keys_match':b.keys()==pub.keys(),'exact_tensors':0,'different_tensors':0,'max_abs_difference':0.0}
-for key,weight in b.items():
- prefix=key.removesuffix('.weight')
- if prefix+'.lora_a' in a:
-  delta=((10*a[prefix+'.lora_b'].T)@a[prefix+'.lora_a'].T).astype(weight.dtype)
-  weight=weight+delta
- diff=float(mx.max(mx.abs(weight.astype(mx.float32)-pub[key].astype(mx.float32))).item())
- report['exact_tensors' if diff==0 else 'different_tensors']+=1
- report['max_abs_difference']=max(report['max_abs_difference'],diff)
-Path('docs/training/jet-next/baseline-adapter-check.json').write_text(json.dumps(report,indent=2))
-print(report)

docs/training/jet-next/code/scripts/audit_v4_tools.py

@@ -1,31 +0,0 @@
-"""Validate every SGD example against native train annotations and check isolation."""
-import json
-from collections import Counter
-from functools import lru_cache
-from pathlib import Path
-from data.decision_training import ROOT, read, atoms, normalized, digest, statekey
-
-src=ROOT/'work/jet-next/sgd/train'
-schema={s['service_name']:s for s in json.loads((src/'schema.json').read_text())}
-@lru_cache(None)
-def source(name):return {d['dialogue_id']:d for d in json.loads((src/name).read_text())}
-parents=read(ROOT/'data/train_v4_focus.jsonl');parent_atoms={a for r in parents for a in atoms(r['state'])}
-owners={};counts=Counter();options=Counter()
-for split,name in dict(train='train_v4_tools_focus.jsonl',val='selection_v4_tools.jsonl',calibration='calibration_v4_tools.jsonl',test='test_v4_tools.jsonl').items():
-    for r in read(ROOT/'data'/name):
-        if r['source']!='v4:tool_intent_routing':continue
-        p=r['provenance'];_,filename,did,turn=p['origin'].split(':');turn=int(turn)
-        d=source(filename)[did];native=d['turns'][turn]['frames'][0]
-        assert native['service']==p['native_service']
-        assert native['state']['active_intent']==r['gold']
-        assert r['state']['conversation']==[dict(role=t['speaker'].lower(),text=t['utterance']) for t in d['turns'][:turn+1]]
-        opts={i['name']:i['description'] for i in schema[p['native_service']]['intents']}
-        opts['NONE']='No active intent; no service action is currently requested.'
-        assert r['question']['criteria']==opts
-        assert list(r['question']['criteria'])[r['target'].index(1.0)]==r['gold']
-        if split!='train': assert not any(digest(normalized(t['text'])) in parent_atoms for t in r['state']['conversation'])
-        for key in [p['component'],p['origin'],*p['identities'],statekey(r['state'])]:assert owners.setdefault(key,split)==split
-        counts[split]+=1;options[len(opts)]+=1
-report=dict(status='passed',native_annotations_checked=sum(counts.values()),counts=dict(counts),option_counts=dict(options),parent_training_overlap=0,
-    checks=['Whole domains and dialogues held together','Native intent targets and complete conversation prefixes match source','Held-out conversation utterances absent from parent training content','Options are the native same-service catalog plus NONE'])
-(ROOT/'docs/training/jet-next/v4-tool-audit.json').write_text(json.dumps(report,indent=2));print(report)

docs/training/jet-next/code/scripts/audit_v4_transfer.py

@@ -1,43 +0,0 @@
-"""Check V4 isolation, public annotations and every generated executable target."""
-import ast
-import csv
-import json
-import zipfile
-from collections import Counter
-from pathlib import Path
-from data.decision_training import read, digest, statekey
-from build_v4_transfer import ROOT, RAW, PROGRAMS
-
-def main():
-    paths={s:ROOT/'data'/p for s,p in dict(train='train_v4.jsonl',val='selection_v4.jsonl',calibration='calibration_v4.jsonl',test='test_v4_new.jsonl').items()}
-    owners={};identities={};counts=Counter();positions=Counter()
-    vast={r['new_id']:r for r in csv.DictReader((RAW/'vast/data/VAST/vast_train.csv').open())}
-    z=zipfile.ZipFile(RAW/'repos/contractnli/resources/contract-nli.zip')
-    c=json.loads(z.read('contract-nli/train.json'));contracts={str(d['id']):d for d in c['documents']}
-    safe=dict(enumerate=enumerate,range=range,len=len,zip=zip,sum=sum,sorted=sorted,reversed=reversed,list=list)
-    for split,path in paths.items():
-        for r in read(path):
-            key=statekey(r['state']); assert owners.setdefault(key,split)==split,('state overlap',split)
-            if not r.get('source','').startswith('v4:'):continue
-            p=r['provenance']; assert p['split']==split
-            for key in [p['component'],p['origin']]: assert identities.setdefault(key,split)==split,key
-            q=r['question'];idx=r['target'].index(1.0);gold=list(q['criteria'])[idx];chosen=list(q['criteria'].values())[idx]
-            assert gold==r['gold']; assert sum(r['target'])==1.0
-            family=r['family'];counts[split+':'+family]+=1;positions[(family,idx)]+=1
-            if family=='code_transfer':
-                ns={};exec(r['state']['code'],{'__builtins__':{},**safe},ns)
-                out=ns['f'](**r['state']['arguments']);assert ast.literal_eval(chosen)==out==p['oracle']['result']
-                assert len(set(q['criteria'].values()))==len(q['criteria'])
-            elif family=='stance_transfer':
-                source=vast[p['origin'].split(':')[-1]]
-                assert r['state']['text']==source['post'] and r['state']['target']==source['new_topic']
-                assert gold==['against','favor','none'][int(source['label'])]
-            elif family=='contract_entailment':
-                _,did,lid=p['origin'].split(':');source=contracts[did]
-                assert r['state']['contract']==source['text']
-                assert r['state']['hypothesis']==c['labels'][lid]['hypothesis']
-                assert gold==source['annotation_sets'][0]['annotations'][lid]['choice']
-    report=dict(status='passed',counts=dict(counts),gold_positions={str(k):v for k,v in positions.items()},checks=['No state or component crosses a split','All VAST and ContractNLI targets checked against native train annotations','Every generated program executed and selected answer verified'])
-    out=ROOT/'docs/training/jet-next/v4-data-audit.json';out.write_text(json.dumps(report,indent=2));print(json.dumps(report,indent=2))
-
-if __name__=='__main__':main()

docs/training/jet-next/code/scripts/bench_index_cuda.sh

@@ -1,17 +0,0 @@
-#!/usr/bin/env bash
-# Decision Index CLI with Jet's MLX CUDA environment. No cloud job or upload.
-set -euo pipefail
-cd "$(dirname "$0")/.."
-uv sync --extra cuda --extra benchmark --inexact
-export CUDA_HOME="$PWD/.venv/cuda_home"
-mkdir -p "$CUDA_HOME/include"
-shopt -s nullglob
-for pkg in .venv/lib/python*/site-packages/nvidia/{cuda_runtime,cuda_cccl,cuda_nvrtc}/include; do
-    for header in "$PWD/$pkg"/*; do
-        [ "$(basename "$header")" = __init__.py ] || ln -sfn "$header" "$CUDA_HOME/include/"
-    done
-done
-export MLX_USE_CUDA_GRAPHS=0
-export MLX_CUDA_SDPA_CACHE_SIZE="${MLX_CUDA_SDPA_CACHE_SIZE:-2048}"
-export MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache"
-exec .venv/bin/python -m decision_index "$@"

docs/training/jet-next/code/scripts/build_v4_tools.py

@@ -1,89 +0,0 @@
-"""Native SGD training intent routing with whole-domain heldouts and ToolRet exclusion."""
-import hashlib
-import json
-import random
-from collections import Counter, defaultdict
-from pathlib import Path
-import pyarrow.parquet as pq
-from huggingface_hub import snapshot_download
-from transformers import AutoTokenizer
-from data.decision_training import ROOT, choice, read, write, normalized, statekey, atoms, digest
-from format import render_state
-
-SEED=230924
-REV='e852981ae34990f4358979625854259302feaa78'
-
-def main():
-    data=ROOT/'data'; prefix='v4_tools'
-    files={s:data/p for s,p in dict(train='train_v4_tools_focus.jsonl',val='selection_v4_tools.jsonl',calibration='calibration_v4_tools.jsonl',test='test_v4_tools.jsonl').items()}
-    if any(p.exists() for p in files.values()):raise SystemExit('Refusing to overwrite tool experiment')
-    src=ROOT/'work/jet-next/sgd/train'
-    if not src.exists():
-        from decision_index.suite.build.acquire import git_fetch
-        git_fetch(src.parent,'https://github.com/google-research-datasets/dstc8-schema-guided-dialogue.git',REV,['train'])
-    schemas={s['service_name']:s for s in json.loads((src/'schema.json').read_text())}
-    domains=sorted({s.split('_')[0] for s,v in schemas.items() if len(v['intents'])>=2})
-    random.Random(SEED).shuffle(domains)
-    splits={d:'val' if i<2 else 'calibration' if i<4 else 'test' if i<6 else 'train' for i,d in enumerate(domains)}
-    queries_path=Path(snapshot_download('mangopy/ToolRet-Queries',repo_type='dataset',revision='b8c76ad3349ff17497b6bdb28bb5b8f61a0f6445',allow_patterns=['*/*.parquet']))
-    blocked={normalized(r['query']) for p in queries_path.rglob('*.parquet') for r in pq.read_table(p).to_pylist()}
-    protected=[data/p for p in ['val.jsonl','test.jsonl','score_eval.jsonl','val_v3_new.jsonl','calibration_v3_new.jsonl','test_v3_new.jsonl','selection_v4.jsonl','calibration_v4.jsonl','test_v4_new.jsonl']]
-    held_atoms={a for p in protected for r in read(p) for a in atoms(r['state'])}
-    tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8')
-    bins=defaultdict(list);excluded=Counter();seen=set();source_hashes={p.name:hashlib.sha256(p.read_bytes()).hexdigest() for p in src.glob('*.json')}
-    for p in sorted(src.glob('dialogues_*.json')):
-        for d in json.loads(p.read_text()):
-            first=d['turns'][0]
-            if len(first['frames'])!=1:continue
-            service=first['frames'][0]['service'];schema=schemas[service];domain=service.split('_')[0]
-            if domain not in splits:continue
-            # Long exact query overlap excludes the entire dialogue, without inspecting answers.
-            if any(len(normalized(t['utterance']))>=32 and (normalized(t['utterance']) in blocked or digest(normalized(t['utterance'])) in held_atoms) for t in d['turns']):
-                excluded['dialogue_with_evaluation_query']+=1;continue
-            history=[]
-            for turn,t in enumerate(d['turns'][:12]):
-                history.append({'role':t['speaker'].lower(),'text':t['utterance']})
-                if any(f['service']!=service for f in t['frames']):break
-                if t['speaker']!='USER' or len(t['frames'])!=1:continue
-                gold=t['frames'][0]['state']['active_intent']
-                options={i['name']:i['description'] for i in schema['intents']}
-                options['NONE']='No active intent; no service action is currently requested.'
-                if gold not in options:continue
-                state={'service':schema['description'],'conversation':list(history)}
-                if len(tokenizer.encode(render_state(state),add_special_tokens=False))>1024:
-                    excluded['over_capacity']+=1;continue
-                key=statekey(state)
-                if key in seen:excluded['duplicate_state']+=1;continue
-                seen.add(key)
-                r=choice(state,'Which service intent is active after the last user turn?',options,gold,
-                    'tool_intent_routing',f'sgd:{p.name}:{d["dialogue_id"]}:{turn}',f'sgd-domain:{domain}',
-                    identities=[f'sgd-dialogue:{d["dialogue_id"]}'])
-                r['source']='v4:tool_intent_routing';r['provenance'].update(split=splits[domain],component=f'sgd-domain:{domain}',native_service=service,source_file=p.name)
-                bins[splits[domain]].append(r)
-    selected={};counts={};rng=random.Random(SEED)
-    for split,rows in bins.items():
-        rng.shuffle(rows);classes=defaultdict(list)
-        for r in rows:classes[r['gold']].append(r)
-        out=[];limit=3000 if split=='train' else 240
-        while classes and len(out)<limit:
-            for k in list(classes):
-                out.append(classes[k].pop())
-                if not classes[k]:del classes[k]
-                if len(out)==limit:break
-        selected[split]=out;counts[split]=len(out)
-    assert all(counts.get(s,0)>=200 for s in ['val','calibration','test']),counts
-    parents=dict(train=data/'train_v4_focus.jsonl',val=data/'selection_v4.jsonl',calibration=data/'calibration_v4.jsonl')
-    hashes={};owners={}
-    for split,rows in selected.items():
-        for r in rows:
-            p=r['provenance']
-            for key in [p['component'],p['origin'],*p['identities'],statekey(r['state'])]:assert owners.setdefault(key,split)==split
-        payload=(read(parents[split]) if split in parents else [])+rows;rng.shuffle(payload);write(files[split],payload)
-        hashes[files[split].name]=hashlib.sha256(files[split].read_bytes()).hexdigest()
-    report=dict(seed=SEED,repository='https://github.com/google-research-datasets/dstc8-schema-guided-dialogue',revision=REV,native_split='train',source_files=source_hashes,
-        domain_splits=splits,counts=counts,exclusions=dict(excluded),toolret_query_revision='b8c76ad3349ff17497b6bdb28bb5b8f61a0f6445',toolret_queries=len(blocked),files=hashes,
-        parents={p.name:hashlib.sha256(p.read_bytes()).hexdigest() for p in parents.values()},code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
-        limitations=['Intent routing within one native service; not a ToolRet ranking score or BFCL multi-tool execution task.', 'Six heldout domains are entirely absent from added SGD training data; base-model pretraining coverage is unknown.', 'Exact long-query exclusion does not detect paraphrases.'])
-    (data/'train_v4_tools.provenance.json').write_text(json.dumps(report,indent=2));print(json.dumps(report,indent=2))
-
-if __name__=='__main__':main()

docs/training/jet-next/code/scripts/build_v4_transfer.py

@@ -1,169 +0,0 @@
-"""Add training-partition stance/contracts and executable multi-step Python tasks.
-
-Never uses evaluation labels. Existing files are read-only exclusion/retention inputs.
-New holdouts are grouped by article/document or whole program template.
-"""
-import csv
-import hashlib
-import json
-import random
-import zipfile
-from collections import Counter, defaultdict
-from pathlib import Path
-
-from data.decision_training import ROOT, atoms, choice, digest, normalized, read, statekey, write
-from format import Question, render_state, build_prompt
-from transformers import AutoTokenizer
-
-SEED = 230924
-RAW = ROOT / 'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw'
-# Distinct complete templates are assigned to each split before generating inputs.
-PROGRAMS = [
-    'out = []\nfor v in x:\n    if v > k:\n        out.append(v * 2)\nreturn out',
-    'out = 0\nfor i, v in enumerate(x):\n    out += v if i % 2 == 0 else -v\nreturn out + k',
-    'out = []\nfor v in x:\n    if v not in out:\n        out.append(v)\nreturn out[k:]',
-    'out = []\nfor i in range(1, len(x)):\n    out.append(x[i] - x[i-1])\nreturn out[:k]',
-    'out = 1\nfor v in x:\n    if v != 0:\n        out *= v\nreturn out + k',
-    'out = []\nfor v in x:\n    if v >= k:\n        out.insert(0, v)\nreturn out',
-    'out = 0\nfor v in x:\n    if v == k:\n        break\n    out += v\nreturn out',
-    'out = []\nfor v in x:\n    if v % 2:\n        continue\n    out.append(v + k)\nreturn out',
-    'out = []\nfor a, b in zip(x, x[1:]):\n    if a < b:\n        out.append(a + b + k)\nreturn out',
-    'out = []\nfor v in x:\n    out.append(sum(x[:len(out)+1]))\nreturn out[k:]',
-    'out = sorted(x)\nif k % 2:\n    out.reverse()\nreturn out[:k]',
-    'out = []\nfor v in x:\n    if x.count(v) > k and v not in out:\n        out.append(v)\nreturn out',
-    'out = 0\nfor v in reversed(x):\n    out = out * 2 + v\nreturn out - k',
-    'out = []\nfor i, v in enumerate(x):\n    if v > sum(x[:i]):\n        out.append(i)\nreturn out[k:]',
-    'out = []\nfor v in x:\n    if out and out[-1] == v:\n        out.pop()\n    else:\n        out.append(v)\nreturn out + [k]',
-    'out = 0\nfor a in x:\n    for b in x:\n        if a + b == k:\n            out += 1\nreturn out',
-    'out = list(x)\nfor i in range(k):\n    out = out[1:] + out[:1]\nreturn out',
-    'out = []\nfor v in x:\n    out.append(v)\n    if v == k:\n        out.append(-v)\nreturn out',
-]
-
-def program_rows():
-    safe = dict(enumerate=enumerate, range=range, len=len, zip=zip, sum=sum,
-                sorted=sorted, reversed=reversed, list=list)
-    for tid, body in enumerate(PROGRAMS):
-        split = 'train' if tid < 12 else 'val' if tid < 14 else 'calibration' if tid < 16 else 'test'
-        code = 'def f(x, k):\n' + '\n'.join('    '+line for line in body.splitlines())
-        ns = {}; exec(code, {'__builtins__': {}, **safe}, ns); fn = ns['f']
-        rng = random.Random(SEED + tid)
-        for i in range(300 if split == 'train' else 120):
-            x = [rng.randint(-4, 7) for _ in range(rng.randint(3, 8))]; k = rng.randint(0, 4)
-            result = fn(x.copy(), k)
-            # Counterfactual inputs/parameters produce plausible same-type distractors.
-            alternatives = [fn(x.copy(), k+1), fn(x[::-1], k), fn(x[1:], k), fn(x+[k], k)]
-            alternatives += ([result+[k], result+[k+1], result+[k+2]] if isinstance(result, list)
-                             else [result-1, result+1, result+2])
-            vals = list(dict.fromkeys([repr(result)] + [repr(v) for v in alternatives]))[:4]
-            options = {f'candidate_{j}': v for j,v in enumerate(vals)}
-            r = choice({'code':code,'arguments':{'x':x,'k':k}}, 'What does f(**arguments) return?',
-                       options,'candidate_0','code_behavior',f'local-v4:{tid}:{i}',f'v4-template:{tid}')
-            # choice randomizes semantic keys again for this family.
-            r['family']='code_transfer'; r['source']='v4:code_transfer'
-            r['provenance'].update(split=split, oracle={'result':result,'template':tid}, original_split='generated')
-            yield r
-
-def main():
-    data=ROOT/'data'; outputs=['train_v4.jsonl','selection_v4.jsonl','calibration_v4.jsonl','test_v4_new.jsonl']
-    if any((data/p).exists() for p in outputs): raise SystemExit('Refusing to overwrite existing v4 files')
-    tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8')
-    old_train=read(data/'train_v3.jsonl')
-    protected=[data/p for p in ['val.jsonl','test.jsonl','score_eval.jsonl','val_v3_new.jsonl','calibration_v3_new.jsonl','test_v3_new.jsonl']]
-    protected += [ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl']
-    held_atoms={a for p in protected for r in read(p) for a in atoms(r['state'])}
-    held_states={statekey(r['state']) for p in protected for r in read(p)}
-    seen={digest([r['state'],r['question']]) for r in old_train}
-    old_atoms={a for r in old_train for a in atoms(r['state'])}
-    sources=[]; candidates=list(program_rows()); excluded=Counter()
-    vast=RAW/'vast/data/VAST/vast_train.csv'
-    sources.append(dict(repository='https://github.com/emilyallaway/zero-shot-stance',revision='e7c4775182b184730f350995f8260579c9e066fe',file='data/VAST/vast_train.csv',sha256=hashlib.sha256(vast.read_bytes()).hexdigest(),split='train'))
-    # Native development/test posts are exclusion-only, including all topic variants.
-    blocked_posts={normalized(r['post']) for f in ['vast_dev.csv','vast_test.csv'] for r in csv.DictReader((vast.parent/f).open())}
-    for r in csv.DictReader(vast.open()):
-        if normalized(r['post']) in blocked_posts: excluded['vast_native_heldout_post']+=1; continue
-        row=choice({'text':r['post'],'target':r['new_topic']},'What stance does the author express towards the target?',
-            dict(against='opposes the target',favor='supports the target',none='no clear stance towards the target'),
-            ['against','favor','none'][int(r['label'])],'stance_transfer','vast-train:'+r['new_id'],'vast-article:'+r['arc_id'])
-        candidates.append(row)
-    archive=RAW/'repos/contractnli/resources/contract-nli.zip'
-    sources.append(dict(repository='https://github.com/stanfordnlp/contract-nli',revision='eced6528dd3c1d14d73f9a87df8f7bdbc03126f9',file='resources/contract-nli.zip!contract-nli/train.json',sha256=hashlib.sha256(archive.read_bytes()).hexdigest(),split='train'))
-    with zipfile.ZipFile(archive) as z:
-        contracts=json.loads(z.read('contract-nli/train.json'))
-        held_contracts={normalized(d['text']) for s in ['dev','test'] for d in json.loads(z.read(f'contract-nli/{s}.json'))['documents']}
-    for d in contracts['documents']:
-        if normalized(d['text']) in held_contracts: excluded['contract_native_heldout_text']+=1; continue
-        # Keep the full contract. Never assign a full-document label to a cropped context.
-        for lid,a in d['annotation_sets'][0]['annotations'].items():
-            row=choice({'contract':d['text'],'hypothesis':contracts['labels'][lid]['hypothesis']},
-                'Does the full contract entail, contradict, or leave the hypothesis undetermined?',
-                dict(Entailment='entailed by the contract',Contradiction='contradicted by the contract',NotMentioned='not determined by the contract'),
-                a['choice'],'contract_entailment',f'contractnli-train:{d["id"]}:{lid}',f'contractnli-document:{d["id"]}')
-            candidates.append(row)
-    # Same normalized document/post stays grouped even if source IDs differ.
-    from data.decision_training import Components
-    components=Components()
-    for r in candidates:
-        group=r['provenance']['group']; s=r['state']
-        text=s.get('contract',s.get('text'))
-        if text: components.union(group,'document:'+digest(normalized(text)))
-    forced_train=set()
-    for r in candidates:
-        s=r['state']; text=s.get('contract',s.get('text'))
-        if text and digest(normalized(text)) in old_atoms:
-            forced_train.add(components.find(r['provenance']['group']))
-    bins=defaultdict(list)
-    for r in candidates:
-        p=r['provenance'];s=r['state'];text=s.get('contract',s.get('text'))
-        # Global hypotheses recur by design; only document content is an exclusion key.
-        if statekey(s) in held_states or (text and digest(normalized(text)) in held_atoms):
-            excluded['existing_evaluation_content']+=1;continue
-        signature=digest([s,r['question']])
-        if signature in seen: excluded['duplicate']+=1;continue
-        seen.add(signature)
-        if len(tokenizer.encode(render_state(s),add_special_tokens=False))>2048:
-            excluded['over_2048_full_state']+=1;continue
-        if len(tokenizer.encode(build_prompt(tokenizer,s,Question.from_dict(r['question'])),add_special_tokens=False))>2304:
-            excluded['over_2304_full_prompt']+=1;continue
-        group=components.find(p['group']); b=int(digest([SEED,group])[:8],16)/2**32
-        split=p.get('split') or ('train' if group in forced_train or b<.8 else 'val' if b<.87 else 'calibration' if b<.94 else 'test')
-        p.update(component=group,split=split);r['source']='v4:'+r['family'];bins[(split,r['family'])].append(r)
-    splits=defaultdict(list);counts={}
-    for (split,family),rows in sorted(bins.items()):
-        random.Random(SEED).shuffle(rows)
-        classes=defaultdict(list)
-        for r in rows:classes[r['gold'] if family!='code_transfer' else 'all'].append(r)
-        selected=[];limit=3000 if split=='train' else 240
-        while classes and len(selected)<limit:
-            for key in list(classes):
-                selected.append(classes[key].pop())
-                if not classes[key]:del classes[key]
-                if len(selected)==limit:break
-        splits[split].extend(selected);counts[split+':'+family]=len(selected)
-    owners={}
-    for split,rows in splits.items():
-        for r in rows:
-            for key in [r['provenance']['component'],r['provenance']['origin'],statekey(r['state'])]:
-                assert owners.setdefault(key,split)==split,(key,split)
-    payload={outputs[0]:old_train+splits['train'],outputs[1]:read(data/'selection_v3.jsonl')+splits['val'],
-             outputs[2]:read(data/'calibration_v3.jsonl')+splits['calibration'],outputs[3]:splits['test']}
-    class_weights={}
-    for family in ['contract_entailment','stance_transfer']:
-        rows=[r for r in splits['train'] if r['family']==family]
-        counts_by_class=Counter(r['gold'] for r in rows)
-        raw={k:min(3.0,len(rows)/(len(counts_by_class)*n)) for k,n in counts_by_class.items()}
-        mean=sum(raw[k]*n for k,n in counts_by_class.items())/len(rows)
-        weights={k:w/mean for k,w in raw.items()};class_weights[family]=weights
-        for r in rows:r['weight']=weights[r['gold']]
-    hashes={}
-    for name,rows in payload.items():
-        random.Random(SEED).shuffle(rows);write(data/name,rows);hashes[name]=hashlib.sha256((data/name).read_bytes()).hexdigest()
-    report=dict(seed=SEED,sources=sources,counts=counts,excluded=dict(excluded),files=hashes,class_weights=class_weights,
-        code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
-        retained_train_v3_sha256=hashlib.sha256((data/'train_v3.jsonl').read_bytes()).hexdigest(),
-        protected_sha256={str(p.relative_to(ROOT)):hashlib.sha256(p.read_bytes()).hexdigest() for p in protected},
-        limitations=['Full frozen suite remains unavailable.', 'ContractNLI restricted to complete contracts fitting 2048 state tokens.',
-        'Source article/document groups are held together. Code holdouts use disjoint complete templates, but share Python primitives.',
-        'Exact normalized content exclusion is not a semantic paraphrase detector.', 'V3 tool tests remain small and easy; do not interpret their accuracy as strong tool retrieval evidence.'])
-    (data/'train_v4.provenance.json').write_text(json.dumps(report,indent=2));print(json.dumps(report,indent=2))
-
-if __name__=='__main__': main()

docs/training/jet-next/code/scripts/evaluate_next_code.sh

@@ -1,27 +0,0 @@
-#!/usr/bin/env bash
-# Larger fixed CRUXEval check after a regression on the original 55-case sample.
-set -euo pipefail
-cd "$(dirname "$0")/.."
-export PYTHONUNBUFFERED=1
-if [[ -n "${1:-}" ]]; then while kill -0 "$1" 2>/dev/null; do sleep 10; done; fi
-[[ -f docs/training/jet-next/isarcasm-en-v4.json ]]
-for name in baseline v3 warm v4; do
-  if [[ "$name" == baseline ]]; then
-    ARGS=(--option model=michaljach/jet --option revision=8a97cfea2df622bb03f5dc9b02567e21abd2551c)
-  else
-    case "$name" in
-      v3) RUN=adapters/jet-v3-decision-20260923;;
-      warm) RUN=adapters/jet-v3-warm-20260924;;
-      v4) RUN=adapters/jet-v4-transfer-20260924;;
-    esac
-    ARGS=(--option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$RUN")
-  fi
-  bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \
-    --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \
-    --out "artifacts/decision-index/runs/cruxeval-full-cuda-next-$name" \
-    --option max_tokens=8192 --option max_cached_tokens=8192 "${ARGS[@]}" > "logs/jet-next/$name-cruxeval-full.log" 2>&1
-  uv run --no-sync jet-bench-index report --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \
-    --results "artifacts/decision-index/runs/cruxeval-full-cuda-next-$name/results.jsonl" \
-    --out "docs/training/jet-next/cruxeval-full-$name.json" > "logs/jet-next/$name-cruxeval-full-report.log" 2>&1
-done
-printf 'Full available CRUXEval comparison completed.\n'

docs/training/jet-next/code/scripts/evaluate_next_extra.sh

@@ -1,46 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-cd "$(dirname "$0")/.."
-export PYTHONUNBUFFERED=1 CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0
-export MLX_CUDA_SDPA_CACHE_SIZE=2048 MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache"
-if [[ -n "${1:-}" ]]; then while kill -0 "$1" 2>/dev/null; do sleep 10; done; fi
-[[ -f docs/training/jet-next/baseline-test_v4_tools.json ]]
-for dataset in test score_eval test_v3_new; do
-  uv run --no-sync jet-eval --base-model models/jet --data "data/$dataset.jsonl" \
-    --batch-size 2 --max-state-tokens 4096 --json "docs/training/jet-next/baseline-$dataset.json" \
-    > "logs/jet-next/baseline-$dataset.log" 2>&1
-done
-for dataset in test score_eval test_v3_new test_v4_new test_v4_tools; do
-  CONTEXT=4096
-  if [[ "$dataset" == test_v4_new || "$dataset" == test_v4_tools ]]; then CONTEXT=2048; fi
-  uv run --no-sync jet-eval --adapter adapters/jet-v3-decision-20260923 --data "data/$dataset.jsonl" \
-    --batch-size 2 --max-state-tokens "$CONTEXT" --json "docs/training/jet-next/v3-$dataset.json" \
-    > "logs/jet-next/v3-$dataset.log" 2>&1
-done
-for name in baseline v3 warm v4; do
-  if [[ "$name" == baseline ]]; then
-    ARGS=(--option model=michaljach/jet --option revision=8a97cfea2df622bb03f5dc9b02567e21abd2551c)
-  else
-    case "$name" in
-      v3) RUN=adapters/jet-v3-decision-20260923;;
-      warm) RUN=adapters/jet-v3-warm-20260924;;
-      v4) RUN=adapters/jet-v4-transfer-20260924;;
-    esac
-    ARGS=(--option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$RUN")
-  fi
-  bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \
-    --rows artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz \
-    --out "artifacts/decision-index/runs/esci-cuda-next-$name" \
-    --option max_tokens=8192 --option max_cached_tokens=8192 "${ARGS[@]}" > "logs/jet-next/$name-esci.log" 2>&1
-  uv run --no-sync jet-bench-index report --rows artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz \
-    --results "artifacts/decision-index/runs/esci-cuda-next-$name/results.jsonl" \
-    --out "docs/training/jet-next/esci-$name.json" > "logs/jet-next/$name-esci-report.log" 2>&1
-  bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \
-    --rows artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz \
-    --out "artifacts/decision-index/runs/isarcasm-en-cuda-next-$name" \
-    --option max_tokens=8192 --option max_cached_tokens=8192 "${ARGS[@]}" > "logs/jet-next/$name-isarcasm-en.log" 2>&1
-  uv run --no-sync jet-bench-index report --rows artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz \
-    --results "artifacts/decision-index/runs/isarcasm-en-cuda-next-$name/results.jsonl" \
-    --out "docs/training/jet-next/isarcasm-en-$name.json" > "logs/jet-next/$name-isarcasm-en-report.log" 2>&1
-done
-printf 'Additional heldouts and ESCI diagnostics completed.\n'

docs/training/jet-next/code/scripts/finalize_next.py

@@ -1,54 +0,0 @@
-"""Require completed runs, verify preserved files, and archive reproducibility metadata."""
-import hashlib
-import csv
-import json
-import shutil
-import subprocess
-from pathlib import Path
-ROOT=Path(__file__).resolve().parents[1];OUT=ROOT/'docs/training/jet-next'
-def sha(p):return hashlib.sha256(p.read_bytes()).hexdigest()
-
-def main():
-    for name in ['warm','v4']:
-        for prefix in ['diagnostic','esci','isarcasm-en','cruxeval-full']:
-            r=json.loads((OUT/f'{prefix}-{name}.json').read_text())
-            assert r['complete_for_supplied_rows'] and r['pending_requests']==0
-            assert r['summary']['counts']=={'ok':r['planned_requests']},r['summary']['counts']
-    runs={}
-    for name,directory in [('warm','jet-v3-warm-20260924'),('v4','jet-v4-transfer-20260924')]:
-        p=ROOT/'adapters'/directory;c=json.loads((p/'training_config.json').read_text())
-        metrics=[json.loads(l) for l in (p/'metrics.jsonl').read_text().splitlines()]
-        assert metrics[-1]['step']==c['total_steps']
-        for filename,expected in c['data_sha256'].items():assert sha(ROOT/filename)==expected
-        assert sha(ROOT/c['init_adapter']/'adapters.safetensors')==c['init_adapter_sha256']
-        best=min(metrics,key=lambda m:m['nll'])
-        runs[name]=dict(directory=str(p.relative_to(ROOT)),config=c,best=best,
-            weights_sha256=sha(p/'adapters.safetensors'),last_weights_sha256=sha(p/'last/adapters.safetensors'),
-            calibration=json.loads((p/'calibration.json').read_text()))
-        for f in ['training_config.json','metrics.jsonl','calibration.json']:
-            shutil.copy2(p/f,OUT/f'{name}-{f.replace("_", "-")}')
-    for name in ['protected','published']:
-        subprocess.run(['sha256sum','-c',str(ROOT/f'work/jet-v3/{name}.sha256')],cwd=ROOT,check=True,stdout=subprocess.DEVNULL)
-    current=subprocess.check_output(['git','diff','--','README.md','src/data/distill.py'],cwd=ROOT)
-    assert current==(ROOT/'work/jet-v3/preexisting.patch').read_bytes(),'Preexisting user changes differ'
-    code=['src/train.py','src/evaluate.py','src/model.py','src/inference.py','src/format.py','src/decision_index_engine.py','src/bench_decision_index.py','src/data/decision_training.py','train_cuda.sh','scripts/bench_index_cuda.sh','scripts/evaluate_next_code.sh']
-    code += ['scripts/'+n for n in ['build_v4_transfer.py','audit_v4_transfer.py','prepare_v4_focus.py','build_v4_tools.py','audit_v4_tools.py','run_next_cuda.sh','evaluate_next_extra.sh','summarize_next.py','finalize_next.py']]
-    for relative in code:
-        dest=OUT/'code'/relative;dest.parent.mkdir(parents=True,exist_ok=True);shutil.copy2(ROOT/relative,dest)
-    hashes={p:sha(ROOT/p) for p in code+['uv.lock','pyproject.toml']}
-    corpus={p.name:sha(p) for p in (ROOT/'data').glob('*v4*.jsonl')}
-    samples=list(csv.DictReader((ROOT/'logs/jet-next/gpu.csv').open(),skipinitialspace=True))
-    peak=max(int(r['memory.used [MiB]'].split()[0]) for r in samples)
-    hardware=subprocess.check_output(['nvidia-smi','--query-gpu=name,memory.total,driver_version','--format=csv,noheader'],text=True).strip()
-    manifest=dict(status='completed',base='mlx-community/Qwen3-0.6B-bf16',base_revision='42096995f6402fde107068cf530136fe64b604f8',
-        git_head=subprocess.check_output(['git','rev-parse','HEAD'],cwd=ROOT,text=True).strip(),
-        branch=subprocess.check_output(['git','branch','--show-current'],cwd=ROOT,text=True).strip(),
-        runs=runs,hardware=hardware,monitored_device_peak_mib=peak,monitor_scope='V4 training and subsequent evaluations, including desktop GPU use',code_sha256=hashes,data_sha256=corpus,protected_original_files='unchanged',published_model='unchanged',preexisting_user_changes='unchanged',
-        warm_source_snapshot_sha256=sha(OUT/'train-warm-snapshot.py'),
-        limitations=['No official full-suite or leaderboard score','Native metrics only; installed panel proposal differs from deployed panel','Partial diagnostic differences are descriptive, not significance tests'])
-    (OUT/'run-manifest.json').write_text(json.dumps(manifest,indent=2))
-    destination=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs');destination.mkdir(parents=True,exist_ok=True)
-    shutil.copy2(OUT/'run-manifest.json',destination/'jet-next-run-manifest.json')
-    shutil.copy2(OUT/'comparison.json',destination/'jet-next-results.json')
-    print('Completed runs verified; published model, original files and prior changes preserved.')
-if __name__=='__main__':main()

docs/training/jet-next/code/scripts/prepare_v4_focus.py

@@ -1,35 +0,0 @@
-"""Create a bounded retention mixture from V4 training rows only."""
-import hashlib
-import json
-import random
-from collections import defaultdict
-from pathlib import Path
-from data.decision_training import ROOT, read, write
-
-p=ROOT/'data/train_v4_focus.jsonl'
-if p.exists(): raise SystemExit('Refusing to overwrite a training mixture')
-rows=read(ROOT/'data/train_v4.jsonl'); buckets=defaultdict(list); new=[]
-for r in rows:
-    if r['source'].startswith('v4:'):new.append(r)
-    else:buckets[r['source']].append(r)
-rng=random.Random(230924); selected=[]
-# Round robin by original source, preserving rare families without replacement.
-for rows in buckets.values():rng.shuffle(rows)
-while buckets and len(selected)<18000:
-    for key in list(buckets):
-        selected.append(buckets[key].pop())
-        if not buckets[key]:del buckets[key]
-        if len(selected)==18000:break
-rows=selected+new;rng.shuffle(rows);write(p,rows)
-smoke=sorted([r for r in rows if r['source']=='v4:contract_entailment'],
-             key=lambda r:len(r['state']['contract']),reverse=True)[:32]
-smoke += [r for r in rows if r['source']=='v4:stance_transfer'][:16]
-smoke += [r for r in rows if r['source']=='v4:code_transfer'][:16]
-smoke_path=ROOT/'work/jet-next/v4-smoke.jsonl'
-smoke_path.parent.mkdir(parents=True,exist_ok=True);write(smoke_path,smoke)
-report=dict(seed=230924,retained_v3=len(selected),new_v4=len(new),total=len(rows),
-    source_sha256=hashlib.sha256((ROOT/'data/train_v4.jsonl').read_bytes()).hexdigest(),
-    output_sha256=hashlib.sha256(p.read_bytes()).hexdigest(),
-    smoke_sha256=hashlib.sha256(smoke_path.read_bytes()).hexdigest(),
-    note='Without-replacement source-balanced subsample of training only; all 9000 V4 additions retained. Existing heldouts unchanged.')
-(ROOT/'data/train_v4_focus.provenance.json').write_text(json.dumps(report,indent=2));print(report)

docs/training/jet-next/code/scripts/run_next_cuda.sh

@@ -1,62 +0,0 @@
-#!/usr/bin/env bash
-# Continue after the one-epoch V3 warm-start ablation; all outputs are local.
-set -euo pipefail
-cd "$(dirname "$0")/.."
-export PYTHONUNBUFFERED=1 CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0
-export MLX_CUDA_SDPA_CACHE_SIZE=2048 MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache"
-WARM=adapters/jet-v3-warm-20260924
-NEXT=adapters/jet-v4-transfer-20260924
-# Waiting PID is supplied only by the live parent session, never discovered by pattern.
-if [[ -n "${1:-}" ]]; then
-    while kill -0 "$1" 2>/dev/null; do sleep 10; done
-fi
-.venv/bin/python - <<'PY'
-import json
-from pathlib import Path
-p=Path('adapters/jet-v3-warm-20260924')
-config=json.loads((p/'training_config.json').read_text())
-metrics=[json.loads(x) for x in (p/'metrics.jsonl').read_text().splitlines()]
-assert metrics[-1]['step']==config['total_steps'], 'Warm-start run did not finish'
-PY
-bash train_cuda.sh --init-adapter "$WARM" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \
-  --train work/jet-next/v4-smoke.jsonl --val work/jet-next/v4-smoke.jsonl \
-  --out adapters/jet-v4-transfer-smoke --epochs 1 --lr 1e-5 --max-batch-tokens 3072 \
-  --max-state-tokens 2048 --use-example-weights --val-limit 64 --eval-every 10 --seed 230924 \
-  > logs/jet-next/v4-smoke.log 2>&1
-bash train_cuda.sh --init-adapter "$WARM" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \
-  --train data/train_v4_tools_focus.jsonl --val data/selection_v4_tools.jsonl --out "$NEXT" \
-  --epochs 1 --rank 16 --lr 1e-5 --grad-checkpoint --max-batch-tokens 3072 \
-  --max-state-tokens 2048 --use-example-weights --val-limit 10000 --eval-every 250 --seed 230924 \
-  > logs/jet-next/v4-train.log 2>&1
-for name in warm v4; do
-  if [[ "$name" == warm ]]; then RUN="$WARM"; CAL=data/calibration_v3.jsonl;
-  else RUN="$NEXT"; CAL=data/calibration_v4_tools.jsonl; fi
-  CONTEXT=4096
-  uv run --no-sync jet-calibrate --adapter "$RUN" --data "$CAL" --limit 10000 \
-    --batch-size 2 --max-state-tokens "$CONTEXT" > "logs/jet-next/$name-calibration.log" 2>&1
-  for dataset in test score_eval test_v3_new test_v4_new test_v4_tools; do
-    CONTEXT=4096
-    if [[ "$dataset" == test_v4_new || "$dataset" == test_v4_tools ]]; then CONTEXT=2048; fi
-    uv run --no-sync jet-eval --adapter "$RUN" --data "data/$dataset.jsonl" --batch-size 2 \
-      --max-state-tokens "$CONTEXT" --json "docs/training/jet-next/$name-$dataset.json" \
-      > "logs/jet-next/$name-$dataset.log" 2>&1
-  done
-  bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \
-    --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \
-    --out "artifacts/decision-index/runs/diagnostic-cuda-next-$name" \
-    --option model=mlx-community/Qwen3-0.6B-bf16 \
-    --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$RUN" \
-    --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-next/$name-diagnostic.log" 2>&1
-  uv run --no-sync jet-bench-index report --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \
-    --results "artifacts/decision-index/runs/diagnostic-cuda-next-$name/results.jsonl" \
-    --out "docs/training/jet-next/diagnostic-$name.json" > "logs/jet-next/$name-report.log" 2>&1
-done
-uv run --no-sync jet-eval --base-model models/jet --data data/test_v4_new.jsonl --batch-size 2 \
-  --max-state-tokens 2048 --json docs/training/jet-next/baseline-test_v4_new.json \
-  > logs/jet-next/baseline-test_v4_new.log 2>&1
-uv run --no-sync jet-eval --base-model models/jet --data data/test_v4_tools.jsonl --batch-size 2 \
-  --max-state-tokens 2048 --json docs/training/jet-next/baseline-test_v4_tools.json \
-  > logs/jet-next/baseline-test_v4_tools.log 2>&1
-sha256sum -c work/jet-v3/protected.sha256 > logs/jet-next/protected-check.log
-sha256sum -c work/jet-v3/published.sha256 > logs/jet-next/published-check.log
-printf 'Next checkpoint training and evaluation completed.\n'

docs/training/jet-next/code/scripts/summarize_next.py

@@ -1,61 +0,0 @@
-"""Summarize completed, fixed-test comparisons without a reconstructed index."""
-import hashlib
-import json
-from pathlib import Path
-
-ROOT=Path(__file__).resolve().parents[1];OLD=ROOT/'docs/training/jet-v3';OUT=ROOT/'docs/training/jet-next'
-MODELS=['baseline','v3','warm','v4'];LABELS=['Released Jet','V3','Warm start','V4 transfer']
-def load(path):return json.loads(path.read_text())
-def pct(x):return '—' if x is None else f'{100*x:.1f}'
-def main():
-    data={}
-    for dataset in ['test','score_eval','test_v3_new','test_v4_new','test_v4_tools']:
-        data[dataset]={m:load(OUT/f'{m}-{dataset}.json') for m in MODELS}
-    diagnostics={m:load((OLD if m in ['baseline','v3'] else OUT)/f'diagnostic-{m}.json') for m in MODELS}
-    esci={m:load(OUT/f'esci-{m}.json') for m in MODELS}
-    sarcasm={m:load(OUT/f'isarcasm-en-{m}.json') for m in MODELS}
-    code={m:load(OUT/f'cruxeval-full-{m}.json') for m in MODELS}
-    for group in [diagnostics,esci,sarcasm,code]:
-        assert len({r['rows_sha256'] for r in group.values()})==1
-        assert all(r['complete_for_supplied_rows'] and r['pending_requests']==0 for r in group.values())
-    results=dict(local_tests=data,diagnostics=diagnostics,esci=esci,isarcasm_english=sarcasm,cruxeval_full=code)
-    (OUT/'comparison.json').write_text(json.dumps(results,indent=2))
-    lines=['# Jet Qwen3-0.6B follow-up results','',
-        'Two more local checkpoints were trained: a lower-learning-rate warm start from released Jet, then targeted transfer training. No weights were published and no leaderboard entry was submitted.',
-        '', 'V4 substantially improves contract entailment, product relevance, stance, and English sarcasm over released Jet on the evaluations below. Confirmed regressions remain in CRUXEval and original-test accuracy. The warm checkpoint preserves original-test accuracy more closely. Neither candidate is an across-the-board replacement.',
-        '', '**These results do not establish a leaderboard win.** The official frozen suite remains unavailable. Diagnostics are partial public reconstructions; the installed kit also contains an older panel definition. Native benchmark metrics below are comparable across these four models on identical rows, but are not an overall index.',
-        '', '| Held-out accuracy (%) | Released Jet | V3 | Warm start | V4 transfer |', '|---|---:|---:|---:|---:|']
-    names=dict(test='Original test (3,683)',score_eval='Ordinal test (2,400)',test_v3_new='V3 new families (1,364)',test_v4_new='Harder transfer tasks (720)',test_v4_tools='SGD domain-held-out routing (240)')
-    for ds,reports in data.items():lines.append('| '+names[ds]+' | '+' | '.join(pct(reports[m]['overall']['acc']) for m in MODELS)+' |')
-    lines += ['', '| Partial benchmark — native metric (%) | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|']
-    for group in [diagnostics,esci]:
-        tables={m:{r['catalog_id']:r for r in report['summary']['benchmarks']} for m,report in group.items()}
-        for bid,base in tables['baseline'].items():
-            lines.append('| '+base['dataset']+' — '+str(base.get('metric') or 'unavailable')+' | '+' | '.join(pct(tables[m][bid].get('score')) for m in MODELS)+' |')
-    english={m:next(t for t in r['panel_metrics_on_sample']['40']['tracks'] if t['track']=='iSarcasmEval-A-En')['raw'] for m,r in sarcasm.items()}
-    lines.append('| iSarcasmEval English A, all 1,400 cases — positive F1 | '+' | '.join(pct(english[m]) for m in MODELS)+' |')
-    lines.append('| CRUXEval, all '+str(code['baseline']['planned_requests'])+' available eligible cases — accuracy | '+' | '.join(pct(code[m]['summary']['benchmarks'][0]['score']) for m in MODELS)+' |')
-    lines += ['', 'iSarcasmEval has no aggregate native score in the pinned summary report. A missing value is not zero. The separate English-A row uses the native positive-class F1 scorer on all 1,400 available cases (the official headline track), without combining other sarcasm tracks. The 500-request eight-benchmark sample and separate 500-request ESCI sample are small; differences are descriptive, not statistical significance claims.',
-        '', '| V4 task family — accuracy (%) | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|']
-    for ds in ['test_v4_new','test_v4_tools']:
-        for family in sorted(data[ds]['baseline']['by_source']):
-            lines.append('| '+family.removeprefix('v4:')+' | '+' | '.join(pct(data[ds][m]['by_source'][family]['acc']) for m in MODELS)+' |')
-    lines += ['', '| Original family — accuracy (%) | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|']
-    for family in sorted(data['test']['baseline']['by_source']):
-        lines.append('| '+family+' | '+' | '.join(pct(data['test'][m]['by_source'][family]['acc']) for m in MODELS)+' |')
-    lines += ['', '| Calibration metric | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|']
-    for ds in ['test','test_v3_new','test_v4_new','test_v4_tools']:
-        for metric in ['nll','ece']:
-            lines.append('| '+ds+' '+metric.upper()+' | '+' | '.join(f"{data[ds][m]['overall'][metric]:.4f}" for m in MODELS)+' |')
-    for metric in ['spearman','mae']:
-        lines.append('| Ordinal test '+metric+' | '+' | '.join(f"{data['score_eval'][m]['by_type']['score'][metric]:.4f}" for m in MODELS)+' |')
-    lines += ['', 'Both candidates were selected by held-out validation NLL and calibrated on separate data before these test runs. Original three test sets use the same 4,096-token state limit for all models; new V4 tests use 2,048 for all models. Benchmark adapters preserve complete prompts up to 8,192 tokens.', '', '## Training and provenance', '']
-    for name,directory in [('Warm start','jet-v3-warm-20260924'),('V4 transfer','jet-v4-transfer-20260924')]:
-        p=ROOT/'adapters'/directory;c=load(p/'training_config.json');metrics=[json.loads(l) for l in (p/'metrics.jsonl').read_text().splitlines()]
-        best=min(metrics,key=lambda r:r['nll'])
-        lines.append(f"- {name}: `{p.relative_to(ROOT)}`; {c['train_examples']:,} rows, {c['total_steps']:,} steps, LR {c['lr']}, best validation step {best['step']}, NLL {best['nll']:.4f}, accuracy {best['acc']:.4f}. Weight SHA256 `{hashlib.sha256((p/'adapters.safetensors').read_bytes()).hexdigest()}`.")
-    lines += ['', 'The published model remains unchanged. V4 retains 18,000 earlier training examples and adds 3,000 examples each for full-contract entailment, stance, executable multi-step code, and native service-intent routing. Sources are pinned; original training labels were verified; related documents/articles/domains stay together. No hosted model APIs were used.', '', 'Full experimental protocol and reproduction details follow.', '', (ROOT/'docs/training/jet-next.md').read_text()]
-    (OUT/'summary.md').write_text('\n'.join(lines)+'\n')
-    destination=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs/jet-next-report.md');destination.parent.mkdir(parents=True,exist_ok=True);destination.write_text('\n'.join(lines)+'\n')
-    print(destination)
-if __name__=='__main__':main()

docs/training/jet-next/code/src/bench_decision_index.py

@@ -1,158 +0,0 @@
-"""Prepare, audit, and report Decision Index diagnostics without training on them."""
-from __future__ import annotations
-
-import argparse
-from collections import Counter, defaultdict
-import gzip
-import hashlib
-import json
-from pathlib import Path
-
-import numpy as np
-from decision_index.engines import Unsupported
-from decision_index.scoring.index import score_panel
-from decision_index.scoring.report import benchmark_summary, load_results
-from decision_index.suite.io import Suite, atomic_json, read_jsonl, sha256_file
-from decision_index.suite.sample import stratified_sample
-
-from decision_index_engine import load_assets, prepare_request
-
-
-def write_rows(path, rows):
-    path = Path(path)
-    path.parent.mkdir(parents=True, exist_ok=True)
-    opener = gzip.open if path.suffix == ".gz" else open
-    with opener(path, "wt", encoding="utf-8") as f:
-        for row in rows:
-            f.write(json.dumps(row, ensure_ascii=False) + "\n")
-
-
-def prepare(suite, out, n, allow_partial=False):
-    verification = suite.verify(strict=not allow_partial)
-    # Even partial rebuilds must use the published exclusions.
-    if not verification.get("exclusions_match"):
-        raise ValueError("Missing or mismatched official exclusions file")
-    out.mkdir(parents=True, exist_ok=True)
-    tracks = defaultdict(list)
-    for row in suite.rows(apply_exclusions=True):
-        key = row["_evaluation"]["track"]
-        # Deterministic, label-independent selection, independent of file order.
-        rank = hashlib.sha256(row["_evaluation"]["run_id"].encode()).hexdigest()
-        tracks[key].append((rank, row))
-        tracks[key] = sorted(tracks[key], key=lambda x: x[0])[:2]
-    smoke = [row for key in sorted(tracks) for _, row in tracks[key]]
-    write_rows(out / "compatibility.jsonl.gz", smoke)
-    sample = stratified_sample(suite, n, out / "diagnostic.jsonl.gz")
-    report = {"verification": verification, "full_frozen_suite": verification["match"],
-              "compatibility_requests": len(smoke), "tracks": len(tracks),
-              "diagnostic": sample, "note": "Compatibility is two requests per available track; diagnostic sampling keeps complete source groups. Neither is a leaderboard score."}
-    atomic_json(out / "preparation.json", report)
-    return report
-
-
-def audit(rows_path, out):
-    path, tokenizer, config = load_assets()
-    limits = [4096, 6144, 8192, 16384, 32768, config["max_position_embeddings"]]
-    stats = defaultdict(lambda: {"requests": 0, "fields": 0, "lengths": [],
-                                 "options": Counter(), "questions_per_request": Counter(),
-                                 "supported_requests": Counter(), "capacity_failures": Counter()})
-    groups = defaultdict(lambda: {limit: True for limit in limits})
-    for i, row in enumerate(read_jsonl(rows_path)):
-        e = row["_evaluation"]
-        s = stats[e["dataset"]]
-        s["requests"] += 1
-        s["fields"] += len(row["questions"])
-        s["questions_per_request"][len(row["questions"])] += 1
-        for q in row["questions"].values():
-            s["options"][len(q["criteria"])] += 1
-        try:
-            prepared = prepare_request(tokenizer, row["state"], row["questions"], 10**12)
-            lengths = [len(p[2]) for p in prepared]
-            s["lengths"].extend(lengths)
-            maximum = max(lengths)
-        except Unsupported as exc:
-            s["capacity_failures"][str(exc)] += 1
-            maximum = float("inf")
-        group = groups[(e["dataset"], e["group_id"])]
-        for limit in limits:
-            supported = maximum <= limit
-            s["supported_requests"][limit] += int(supported)
-            group[limit] &= supported
-        if (i + 1) % 500 == 0:
-            print(f"Audited {i + 1} requests", flush=True)
-    for name, s in stats.items():
-        lengths = s.pop("lengths")
-        s["prompt_tokens"] = dict(zip(["p50", "p95", "max"], map(float, np.percentile(lengths, [50, 95, 100])))) if lengths else None
-        selected = [g for (dataset, _), g in groups.items() if dataset == name]
-        s["source_groups"] = len(selected)
-        s["supported_complete_groups"] = {limit: sum(g[limit] for g in selected) for limit in limits}
-    report = {"rows_sha256": sha256_file(rows_path),
-              "rows_uncompressed_sha256": sha256_file(rows_path, gunzip=Path(rows_path).suffix == ".gz"),
-              "model_revision": path.name,
-              "scope": "Only the supplied rows; token capacity, not measured accuracy or validated long-context quality.",
-              "limits": limits, "benchmarks": dict(stats)}
-    out.parent.mkdir(parents=True, exist_ok=True)
-    atomic_json(out, report)
-    return report
-
-
-class SampleSuite:
-    """Read exactly the diagnostic rows, with no invented full-suite denominator."""
-    def __init__(self, rows_path):
-        self.rows_path = rows_path
-
-    def rows(self, apply_exclusions=False):
-        yield from read_jsonl(self.rows_path)
-
-
-def report(rows_path, results_path, out):
-    suite = SampleSuite(rows_path)
-    results = load_results(results_path)
-    summary = benchmark_summary(suite, results, "jet")
-    panel = score_panel(suite, results)
-    planned = {r["_evaluation"]["run_id"] for r in suite.rows()}
-    present = planned & results.keys()
-    environment_path = Path(results_path).with_name("environment.json")
-    data = {"scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-            "rows_sha256": sha256_file(rows_path), "results_sha256": sha256_file(results_path),
-            "planned_requests": len(planned), "processed_requests": len(present),
-            "pending_requests": len(planned - present),
-            "complete_for_supplied_rows": planned == present,
-            "environment": json.loads(environment_path.read_text()) if environment_path.exists() else None,
-            "summary": summary, "panel_metrics_on_sample": panel}
-    out.parent.mkdir(parents=True, exist_ok=True)
-    atomic_json(out, data)
-    return data
-
-
-def main():
-    ap = argparse.ArgumentParser(description=__doc__)
-    sub = ap.add_subparsers(dest="command", required=True)
-    p = sub.add_parser("prepare")
-    p.add_argument("--suite-dir", type=Path, required=True)
-    p.add_argument("--out", type=Path, required=True)
-    p.add_argument("--n", type=int, default=500)
-    p.add_argument("--allow-partial", action="store_true", help="Explicitly permit a nonmatching partial rebuilt corpus")
-    p = sub.add_parser("audit")
-    p.add_argument("--rows", type=Path, required=True)
-    p.add_argument("--out", type=Path, required=True)
-    p = sub.add_parser("report")
-    p.add_argument("--rows", type=Path, required=True)
-    p.add_argument("--results", type=Path, required=True)
-    p.add_argument("--out", type=Path, required=True)
-    args = ap.parse_args()
-    if args.command == "prepare":
-        if args.n < 1:
-            ap.error("--n must be positive")
-        data = prepare(Suite(args.suite_dir), args.out, args.n, args.allow_partial)
-        print(json.dumps(data, indent=2))
-    elif args.command == "audit":
-        audit(args.rows, args.out)
-        print(f"Wrote {args.out}")
-    else:
-        report(args.rows, args.results, args.out)
-        print(f"Wrote {args.out}")
-
-
-if __name__ == "__main__":
-    main()

docs/training/jet-next/code/src/data/decision_training.py

@@ -1,280 +0,0 @@
-"""Reproducible v3 data from pinned TRAIN files and locally executable generators.
-
-Run after scripts/rebuild_index_diagnostic.py and scripts/download_v3_sources.py:
-  uv run --no-sync python -m data.decision_training
-No benchmark normalizer supplies training rows. Evaluation is exclusion-only.
-"""
-from __future__ import annotations
-import ast
-import csv
-import hashlib
-import json
-import random
-import re
-from collections import Counter, defaultdict
-from pathlib import Path
-
-import pyarrow.parquet as pq
-from format import Question, render_state, build_prompt
-
-SEED = 230923
-ROOT = Path(__file__).resolve().parents[2]
-
-def digest(x):
-    return hashlib.sha256(json.dumps(x, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
-
-def normalized(s):
-    return ' '.join(s.casefold().split())
-
-def atoms(value):
-    if isinstance(value, str):
-        if len(normalized(value)) >= 32:
-            yield digest(normalized(value))
-    elif isinstance(value, dict):
-        for v in value.values(): yield from atoms(v)
-    elif isinstance(value, list):
-        for v in value: yield from atoms(v)
-
-def statekey(value):
-    return digest(normalized(value) if isinstance(value, str) else value)
-
-def choice(state, instruction, options, gold, family, origin, group, identities=(), oracle=None):
-    rng = random.Random(SEED + int(digest([origin, family])[:12], 16))
-    keys = list(options); rng.shuffle(keys)
-    if family in ('arithmetic', 'code_behavior'):
-        # Anonymous keys must not reveal the right answer (e.g. output_0).
-        gold_position = keys.index(gold)
-        options = {f'option_{i}': options[k] for i, k in enumerate(keys)}
-        keys = list(options)
-        gold = keys[gold_position]
-    result = dict(state=state, question=dict(type='choice', instructions=instruction,
-        criteria={k: options[k] for k in keys}), target=[float(k == gold) for k in keys],
-        source='v3:'+family, family=family, provenance=dict(origin=origin, original_split='train',
-        group=group, identities=list(identities)), gold=gold)
-    if oracle is not None: result['provenance']['oracle'] = oracle
-    return result
-
-def public_candidates(sources):
-    for spec in sources:
-        repo, file = spec['repo'], spec['file']
-        if 'glaive' in repo:
-            yield from tool_candidates(spec)
-            continue
-        table = pq.read_table(spec['path'])
-        indices = list(range(len(table))); random.Random(SEED).shuffle(indices)
-        # Bounded deterministic subset, not a new partition of the public evaluation files.
-        for idx in indices[:40000]:
-            r = table.slice(idx, 1).to_pylist()[0]
-            origin = f'{repo}@{spec["revision"]}/{file}:{r.get("example_id", r.get("idx",idx))}'
-            if 'esci' in repo:
-                if r['product_locale'] != 'us': continue
-                product = {k: r[k] for k in ['product_title','product_description','product_bullet_point','product_brand','product_color'] if r[k]}
-                if len(json.dumps(product)) > 5000: continue
-                options = dict(Exact='exact match to the requested product', Substitute='a substitute that can satisfy the same need', Complement='an accessory or complementary product', Irrelevant='irrelevant to the search intent')
-                yield choice(dict(search_query=r['query'],product=product),'How relevant is the product to the search query?',options,r['esci_label'],'product_relevance',origin,f'esci-query:{r["query_id"]}',[f'esci-product:{r["product_id"]}', 'query:'+normalized(r['query'])])
-            elif 'snli' in repo:
-                if r['label'] < 0: continue
-                options=dict(entailment='the premise implies the hypothesis',neutral='the premise does not determine whether the hypothesis is true',contradiction='the premise contradicts the hypothesis')
-                yield choice(dict(premise=r['premise'],hypothesis=r['hypothesis']),'What is the relation of the hypothesis to the premise?',options,list(options)[r['label']],'entailment',origin,'snli-premise:'+normalized(r['premise']))
-            elif 'glue' in repo:
-                yield choice(dict(query=r['question'],document=r['sentence']),'Does this document sentence contain the answer to the query?',dict(yes='contains the answer',no='does not contain the answer'),'yes' if r['label']==0 else 'no','document_relevance',origin,'qnli-query:'+normalized(r['question']),['qnli-doc:'+normalized(r['sentence'])])
-            elif 'irony' in file:
-                yield choice(r['text'],'Does the text express verbal irony?',dict(yes='ironic',no='not ironic'),'yes' if r['label'] else 'no','sarcasm_irony',origin,'tweet:'+normalized(r['text']))
-            elif 'stance' in file:
-                topic={'stance_abortion':'legalization of abortion','stance_atheism':'atheism','stance_climate':'climate change is a real concern','stance_feminist':'the feminist movement','stance_hillary':'Hillary Clinton'}[file.split('/')[0]]
-                options=dict(none='no clear stance towards the target',against='opposes the target',favor='supports the target')
-                yield choice(dict(text=r['text'],target=topic),'What stance does the author express towards the target?',options,list(options)[r['label']],'stance',origin,'tweet:'+normalized(r['text']))
-    # Actual author-labelled sarcasm, separate official TRAIN partition. Rephrases stay grouped.
-    file=ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw/repos/isarcasm/train/train.En.csv'
-    for idx,r in enumerate(csv.DictReader(file.open())):
-        if not r.get('tweet'): continue
-        origin=f'iSarcasmEval@dfc708b53bde1bb571abfb5692f63231c2232195/train/train.En.csv:{idx}'
-        group='isarcasm:'+str(idx)
-        yield choice(r['tweet'],'Is the author being sarcastic?',dict(yes='sarcastic',no='not sarcastic'),'yes' if int(r['sarcastic']) else 'no','sarcasm_irony',origin,group)
-        if r.get('rephrase'):
-            yield choice(r['rephrase'],'Is the author being sarcastic?',dict(yes='sarcastic',no='not sarcastic'),'no','sarcasm_irony',origin+':rephrase',group)
-
-def tool_candidates(spec):
-    records=json.load(open(spec['path']))
-    parsed=[]
-    relevance=[]
-    for idx,r in enumerate(records):
-        # Use only the first request and its immediate labelled function call.
-        match=re.search(r'USER:\s*(.*?)\n+ASSISTANT:\s*(.*?)(?:<\|endoftext\|>|\n\n)',r['chat'],re.S)
-        if not match: continue
-        try:
-            positive=match[2].startswith('<functioncall>')
-            call=ast.literal_eval(match[2].replace('<functioncall>', '', 1).strip()) if positive else None
-            tools=[]; rem=r['system'][r['system'].index('{'):]
-            while rem.strip():
-                item,end=json.JSONDecoder().raw_decode(rem.lstrip());tools.append(item);rem=rem.lstrip()[end:]
-            names=[x['name'] for x in tools]
-            if len(set(names)) != len(names): continue
-            if positive and call['name'] not in names: continue
-            if not positive and not re.search(r"(don't|do not) have (the )?(capability|function|ability)|only (have|allows|function)|cannot (assist|help|perform)", match[2], re.I): continue
-            if len(match[1]) > 1800: continue
-        except (ValueError,SyntaxError,KeyError,TypeError): continue
-        if len(tools)==1:
-            relevance.append((idx,match[1].strip(),tools,positive))
-        if positive: parsed.append((idx,match[1].strip(),tools,call['name']))
-    catalog={t['name']:t['description'] for _,_,ts,_ in parsed for t in ts}
-    names=sorted(catalog)
-    for idx,query,tools,gold in parsed:
-        origin=f'{spec["repo"]}@{spec["revision"]}/{spec["file"]}:{idx}'
-        group='tool-catalog:'+digest(sorted(t['name'] for t in tools))
-        rng=random.Random(SEED+idx)
-        options={t['name']:t['description'] for t in tools}
-        # Selection uses source-labelled calls plus catalog distractors. Binary relevance
-        # uses the native single-tool call/refusal annotation, without added distractors.
-        others=[n for n in names if n not in options]
-        for n in rng.sample(others,min(5,len(others))): options[n]=catalog[n]
-        if sum(len(x) for x in options.values())>2200: continue
-        yield choice(query,'Select the tool that directly fulfills the user request.',options,gold,'tool_selection',origin,group)
-    for idx,query,tools,positive in relevance:
-        tool=tools[0]
-        origin=f'{spec["repo"]}@{spec["revision"]}/{spec["file"]}:{idx}'
-        group='tool-catalog:'+digest([tool['name']])
-        yield choice(dict(request=query,tool=tool),'Is this tool relevant for fulfilling the user request?',
-            dict(yes='the tool can fulfill the request',no='the request requires a different capability'),
-            'yes' if positive else 'no','tool_relevance',origin,group)
-
-def generators(n=10000):
-    rng=random.Random(SEED)
-    for i in range(n):
-        values={x:bool(rng.getrandbits(1)) for x in ['active','verified','paid','blocked','admin']}
-        def rule(depth):
-            if depth==0 or rng.random()<0.22:
-                name=rng.choice(list(values));return f'not {name}' if rng.random()<0.3 else name
-            return f'({rule(depth-1)} {rng.choice(["and","or"])} {rule(depth-1)})'
-        expr=rule(rng.randint(2,4)); result=bool(eval(expr,{'__builtins__':{}},values))
-        origin=f'generator:boolean:{i}'; group='boolean-rule:'+expr
-        yield choice(dict(facts=values,allow_if=expr),'Under this exact Boolean rule, is access allowed?',dict(allow='access is allowed',deny='access is denied'),'allow' if result else 'deny','boolean_rules',origin,group,oracle=dict(expression=expr,variables=values,result=result))
-        a,b,c=[rng.randint(-30,50) for _ in range(3)];d=rng.randint(1,12)
-        expr=rng.choice([f'({a} + {b}) * {c}',f'{a} * {b} - {c}',f'({a} - {b}) // {d}',f'{a} + {b} * {c}',f'({a} * {b}) % {d}',f'abs({a} - {b}) + {c}'])
-        result=eval(expr,{'__builtins__':{},'abs':abs})
-        choices={result,result+1,result-1,result+rng.choice([3,5,10])}
-        options={f'value_{j}':str(v) for j,v in enumerate(sorted(choices))};gold=next(k for k,v in options.items() if v==str(result))
-        yield choice(dict(expression=expr),'Evaluate the Python integer expression. // is floor division and % is modulo.',options,gold,'arithmetic',f'generator:arithmetic:{i}','arithmetic:'+expr,oracle=dict(expression=expr,result=result))
-        k=rng.randint(-8,12);m=rng.randint(2,8);inp=[rng.randint(-10,15) for _ in range(rng.randint(3,7))]
-        body=rng.choice([f'return sum(v + {k} for v in x if v % {m} != 0)',f'return [v * {k} for v in x][::{m}]',f'return sorted(set(x + [{k}]))[:{m}]',f'return sum(x[-{m}:]) + {k}',f'return [v for v in x if v > {k}][::-1]',f'return len([v for v in x if v % {m} == {k%m}])'])
-        code='def f(x):\n    '+body
-        ns={};exec(code,{'__builtins__':{},'sum':sum,'sorted':sorted,'set':set,'len':len},ns)
-        result=ns['f'](inp)
-        if isinstance(result,list): bad=[result[::-1],result+[k],result[1:],[]]
-        else: bad=[result-1,result+1,result+3,0]
-        outputs=list(dict.fromkeys([repr(result)]+[repr(x) for x in bad]))[:4]
-        if len(outputs)<2: continue
-        options={f'output_{j}':v for j,v in enumerate(outputs)}
-        yield choice(dict(code=code,input=inp),'What value does f return for the supplied input?',options,'output_0','code_behavior',f'generator:code:{i}','program:'+code,oracle=dict(result=result,executable=True))
-
-class Components:
-    def __init__(self):self.parent={}
-    def find(self,x):
-        self.parent.setdefault(x,x)
-        while x!=self.parent[x]:
-            self.parent[x]=self.parent[self.parent[x]];x=self.parent[x]
-        return x
-    def union(self,a,b):
-        a,b=self.find(a),self.find(b)
-        if a!=b:self.parent[max(a,b)]=min(a,b)
-
-def read(path):return [json.loads(line) for line in path.open() if line.strip()]
-def write(path,rows):
-    path.write_text(''.join(json.dumps(r,ensure_ascii=False)+'\n' for r in rows))
-
-def main():
-    sources=json.load(open(ROOT/'work/jet-v3/sources.json'))
-    from transformers import AutoTokenizer
-    tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16')
-    data=ROOT/'data'; protected=[data/x for x in ['val.jsonl','test.jsonl','score_eval.jsonl']]
-    suite=ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl'
-    assert suite.exists(), 'Rebuild the evaluation corpus first, for exclusion-only use.'
-    held=[r for p in protected+[suite] for r in read(p)]
-    blocked_states={statekey(r['state']) for r in held}
-    blocked_atoms={a for r in held for a in atoms(r['state'])}
-    legacy=read(data/'train_v2.jsonl')
-    legacy_keys={statekey(r['state']) for r in legacy}
-    legacy_atoms={a for r in legacy for a in atoms(r['state'])}
-    stats=Counter();seen=set();candidates=[]
-    for r in list(public_candidates(sources))+list(generators()):
-        if statekey(r['state']) in blocked_states or set(atoms(r['state'])) & blocked_atoms:
-            stats['excluded_evaluation_content']+=1;continue
-        # Original identity + family permits multiple tasks from one source row, kept together.
-        identity=(r['provenance']['origin'],r['family'])
-        semantic=digest([r['state'],r['question']['instructions'],r['question']['criteria']])
-        if identity in seen or semantic in seen:
-            stats['duplicate']+=1;continue
-        if len(tokenizer.encode(render_state(r['state']), add_special_tokens=False)) > 1024:
-            stats['over_state_capacity']+=1;continue
-        if len(tokenizer.encode(build_prompt(tokenizer,r['state'],Question.from_dict(r['question'])), add_special_tokens=False)) > 2048:
-            stats['over_prompt_capacity']+=1;continue
-        seen.add(identity);seen.add(semantic);candidates.append(r)
-    components=Components()
-    for r in candidates:
-        p=r['provenance'];key=p['group']
-        links=['origin:'+p['origin'],'state:'+statekey(r['state'])]+p['identities']
-        if r['family'] in ('tool_selection', 'tool_relevance'):
-            query=r['state'] if isinstance(r['state'],str) else r['state']['request']
-            links.append('tool-query:'+normalized(query))
-        # Entire same tweet/premise/query/product/program components stay in one partition.
-        for link in links:components.union(key,link)
-    forced_train=set()
-    for r in candidates:
-        if statekey(r['state']) in legacy_keys or set(atoms(r['state'])) & legacy_atoms:
-            forced_train.add(components.find(r['provenance']['group']))
-    bins=defaultdict(list)
-    for r in candidates:
-        group=components.find(r['provenance']['group']);b=int(digest([SEED,group])[:8],16)/2**32
-        split='train' if group in forced_train or b<.76 else 'val' if b<.84 else 'calibration' if b<.92 else 'test'
-        r['provenance']['component']=group;r['provenance']['split']=split
-        bins[(split,r['family'])].append(r)
-    out=defaultdict(list);counts={}
-    for (split,family),rows in sorted(bins.items()):
-        # Equal family budget, with round-robin classes; no duplicate oversampling.
-        random.Random(SEED).shuffle(rows);classes=defaultdict(list)
-        for r in rows:classes[r['gold'] if family not in ['tool_selection','arithmetic','code_behavior'] else 'all'].append(r)
-        selected=[];limit=2000 if split=='train' else 160
-        while classes and len(selected)<limit:
-            for key in list(classes):
-                selected.append(classes[key].pop())
-                if not classes[key]:del classes[key]
-                if len(selected)==limit:break
-        out[split].extend(selected);counts[split+':'+family]=len(selected)
-    # Remove legacy content colliding with known protected evaluation; never alter train_v2.
-    clean_legacy=[];seen_legacy=set()
-    for r in legacy:
-        if statekey(r['state']) in blocked_states:
-            stats['legacy_evaluation_overlap']+=1;continue
-        signature=digest([r['state'],r['question']])
-        if signature in seen_legacy:stats['legacy_duplicate']+=1;continue
-        seen_legacy.add(signature);clean_legacy.append(r)
-    out['train']=clean_legacy+out['train']
-    # Split existing val by state into checkpoint-selection and calibration (file untouched).
-    original_val=read(data/'val.jsonl')
-    selection=[r for r in original_val if int(statekey(r['state'])[:8],16)%2==0]
-    calibration=[r for r in original_val if int(statekey(r['state'])[:8],16)%2!=0]
-    files={'train_v3.jsonl':out['train'],'val_v3_new.jsonl':out['val'],
-        'calibration_v3_new.jsonl':out['calibration'],'test_v3_new.jsonl':out['test'],
-        'selection_v3.jsonl':selection+out['val'],'calibration_v3.jsonl':calibration+out['calibration']}
-    hashes={}
-    for name,rows in files.items():
-        random.Random(SEED).shuffle(rows)
-        for r in rows:
-            Question.from_dict(r['question']); assert abs(sum(r['target'])-1)<1e-6
-        write(data/name,rows);hashes[name]=hashlib.sha256((data/name).read_bytes()).hexdigest()
-    # A group and original source ID may occur many times, but only in one split.
-    owners={}
-    for split,rows in out.items():
-        for r in rows:
-            if 'provenance' not in r:continue
-            for key in [r['provenance']['component'],r['provenance']['origin'],statekey(r['state'])]:
-                assert owners.setdefault(key,split)==split, (key,split)
-    provenance=dict(seed=SEED,sources=[{**s,'sha256':hashlib.sha256(Path(s['path']).read_bytes()).hexdigest()} for s in sources],
-        isarcasm=dict(repository='https://github.com/iabufarha/iSarcasmEval',revision='dfc708b53bde1bb571abfb5692f63231c2232195',file='train/train.En.csv',split='train'),
-        protected_sha256={str(p.relative_to(ROOT)):hashlib.sha256(p.read_bytes()).hexdigest() for p in protected+[suite]},
-        counts=counts,filters=dict(stats),files=hashes,legacy_rows=len(clean_legacy),generator_code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
-        limitations=['Official frozen suite unavailable; exclusions cover all 23,113 rebuilt requests and existing held-outs.', 'ESCI uses only the first pinned training shard; official ESCI evaluation uses test.', 'New held-outs are grouped samples of training partitions, not leaderboard evaluation.', 'Glaive labels are public synthetic annotations; added distractors are not exhaustively verified for semantic equivalence.', 'Legacy train_v2 lacks original source IDs; retained legacy provenance is its immutable file hash.'])
-    (data/'train_v3.provenance.json').write_text(json.dumps(provenance,indent=2))
-    print(json.dumps(dict(counts=counts,filters=dict(stats),total_train=len(out['train'])),indent=2))
-
-if __name__=='__main__':main()

docs/training/jet-next/code/src/decision_index_engine.py

@@ -1,133 +0,0 @@
-"""Jet's native MLX readout through the official Decision Index engine API.
-
-Install with ``uv sync --extra benchmark``. No benchmark identity or gold labels
-are accepted by this adapter. The full native prompt is preserved, or rejected.
-"""
-from __future__ import annotations
-
-import hashlib
-import json
-import math
-from pathlib import Path
-
-import numpy as np
-from decision_index.engines import Engine, Unsupported
-from huggingface_hub import snapshot_download
-from transformers import AutoTokenizer
-
-from format import MAX_CHOICE_OPTIONS, Question, build_prompt, label_token_ids
-
-MODEL_ID = "michaljach/jet"
-REVISION = "8a97cfea2df622bb03f5dc9b02567e21abd2551c"
-DEFAULT_MAX_TOKENS = 8192
-
-
-def as_text(value):
-    """Mechanical JSON rendering for structured instructions/descriptions."""
-    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
-
-
-def prepare_request(tokenizer, state, questions, max_tokens):
-    prepared = []
-    for name, original in questions.items():
-        if original["type"] != "choice":
-            raise Unsupported(f"Only suite choice questions supported: {original['type']}")
-        criteria = original["criteria"]
-        if not 2 <= len(criteria) <= MAX_CHOICE_OPTIONS:
-            raise Unsupported(f"Option count {len(criteria)} outside 2..{MAX_CHOICE_OPTIONS}")
-        q = Question.from_dict({
-            "type": "choice", "instructions": as_text(original["instructions"]),
-            "criteria": {key: as_text(value) for key, value in criteria.items()},
-        })
-        # Do not call inference.encode: its application default truncates states.
-        ids = tokenizer.encode(build_prompt(tokenizer, state, q), add_special_tokens=False)
-        if len(ids) > max_tokens:
-            raise Unsupported(f"Full prompt has {len(ids)} tokens; declared limit {max_tokens}")
-        prepared.append((name, q, ids, label_token_ids(tokenizer, q)))
-    return prepared
-
-
-def load_assets(model=MODEL_ID, revision=REVISION):
-    path = Path(snapshot_download(model, revision=revision, allow_patterns=[
-        "config.json", "calibration.json", "tokenizer*", "special_tokens_map.json",
-        "added_tokens.json", "vocab.json", "merges.txt", "chat_template*",
-    ]))
-    return path, AutoTokenizer.from_pretrained(path), json.loads((path / "config.json").read_text())
-
-
-class JetEngine(Engine):
-    name = "jet"
-    latency = "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-
-    def __init__(self, model=MODEL_ID, revision=REVISION, max_tokens=DEFAULT_MAX_TOKENS,
-                 max_cached_tokens=32768, adapter=None, **options):
-        super().__init__(**options)
-        import mlx.core as mx
-        from mlx_lm import load
-
-        self.mx = mx
-        mx.set_cache_limit(1_000_000_000)
-        path, self.tokenizer, config = load_assets(model, revision)
-        self.max_tokens = int(max_tokens)
-        self.max_cached_tokens = int(max_cached_tokens)
-        if not 1 <= self.max_tokens <= config["max_position_embeddings"]:
-            raise ValueError("max_tokens must be positive and within the checkpoint context window")
-        if self.max_cached_tokens < 1:
-            raise ValueError("max_cached_tokens must be positive")
-        calibration = Path(adapter) / "calibration.json" if adapter else path / "calibration.json"
-        self.temperature = float(json.loads(calibration.read_text())["choice"])
-        if not math.isfinite(self.temperature) or self.temperature <= 0:
-            raise ValueError("Invalid published choice temperature")
-        # Pin weights to the SAME resolved snapshot as the tokenizer/config.
-        snapshot_download(model, revision=path.name, allow_patterns=["*.safetensors", "*.safetensors.index.json"])
-        self.model, _ = load(str(path), adapter_path=adapter)
-        self.model.eval()
-        self.provenance = {
-            "model": model, "revision": path.name, "backend": "mlx",
-            "max_tokens": self.max_tokens, "checkpoint_context": config["max_position_embeddings"],
-            "max_cached_tokens": self.max_cached_tokens, "choice_temperature": self.temperature,
-            "adapter_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
-            "format_sha256": hashlib.sha256(Path(__file__).with_name("format.py").read_bytes()).hexdigest(),
-            "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-        }
-        if adapter:
-            self.provenance["lora_adapter"] = str(Path(adapter).resolve())
-            self.provenance["lora_weights_sha256"] = hashlib.sha256(
-                (Path(adapter) / "adapters.safetensors").read_bytes()).hexdigest()
-            self.provenance["calibration_sha256"] = hashlib.sha256(calibration.read_bytes()).hexdigest()
-
-    def __call__(self, state, questions):
-        from model import shared_prefix_label_logits
-
-        prepared = prepare_request(self.tokenizer, state, questions, self.max_tokens)
-        if not prepared:
-            raise ValueError("At least one question is required")
-        logits = shared_prefix_label_logits(
-            self.model, [p[2] for p in prepared], [p[3] for p in prepared],
-            max_cached_tokens=self.max_cached_tokens,
-        )
-        answers, raw = {}, {}
-        for i, (name, q, ids, labels) in enumerate(prepared):
-            z = logits[i, :len(labels)].astype(np.float64) / self.temperature
-            p = np.exp(z - z.max())
-            p /= p.sum()
-            if not np.isfinite(p).all():
-                raise ValueError("Non-finite model probabilities")
-            keys = q.keys
-            answers[name] = {
-                "type": "choice", "choice": keys[int(np.argmax(p))],
-                "probabilities": dict(zip(keys, map(float, p))),
-                "confidence": float(p.max()),
-            }
-            raw[name] = {"label_logits": logits[i, :len(labels)].tolist(), "prompt_tokens": len(ids)}
-        return {"model": "jet", "answers": answers,
-                "usage": {"input_tokens": sum(len(p[2]) for p in prepared)}}, raw
-
-    def synchronize(self):
-        self.mx.synchronize()
-
-    def runtime(self):
-        import platform
-        from importlib.metadata import version
-        return {"platform": platform.platform(), "mlx": version("mlx"),
-                "mlx_lm": version("mlx-lm"), "device": str(self.mx.default_device())}

docs/training/jet-next/code/src/evaluate.py

@@ -1,190 +0,0 @@
-"""Evaluate and calibrate a Jet model.
-
-    jet-eval --adapter adapters/jet --data data/test.jsonl
-    jet-eval --data data/test.jsonl                # untrained base model, as a baseline
-    jet-calibrate --adapter adapters/jet --data data/val.jsonl
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import time
-from collections import defaultdict
-from pathlib import Path
-
-import mlx.core as mx
-import numpy as np
-
-from format import Question
-from model import DEFAULT_BASE_MODEL, DEFAULT_MAX_STATE_TOKENS, Jet
-
-TEMPERATURE_GRID = np.exp(np.linspace(np.log(0.25), np.log(8.0), 121))
-
-
-def load_data(path: Path, limit: int | None) -> list[dict]:
-    rows = [json.loads(line) for line in path.open()]
-    return rows[:limit] if limit else rows
-
-
-def adapter_base(adapter: str | None, fallback: str) -> str:
-    if adapter and (cfg := Path(adapter) / "adapter_config.json").exists():
-        config = json.loads(cfg.read_text())
-        base = config.get("base_model", fallback)
-        if config.get("base_revision"):
-            from huggingface_hub import snapshot_download
-            return snapshot_download(base, revision=config["base_revision"])
-        return base
-    return fallback
-
-
-def collect_logits(jet: Jet, rows: list[dict], batch_size: int) -> tuple[list[np.ndarray], float]:
-    """Raw label logits for every row, batched by prompt length. Returns (logits, ms per question)."""
-    # Batch shapes vary, so MLX's freed-buffer cache otherwise grows until CUDA runs out (as in jet-train).
-    mx.set_cache_limit(1_000_000_000)
-    items = [(r["state"], Question.from_dict(r["question"])) for r in rows]
-    order = sorted(range(len(items)), key=lambda i: len(str(items[i][0])))
-    logits: list[np.ndarray | None] = [None] * len(items)
-    started = time.perf_counter()
-    for start in range(0, len(order), batch_size):
-        idx = order[start : start + batch_size]
-        out, _ = jet.raw_logits([items[i] for i in idx])
-        for i, z in zip(idx, out):
-            logits[i] = z
-        print(f"\r  {min(start + batch_size, len(order))}/{len(order)}", end="", flush=True)
-    print()
-    return logits, (time.perf_counter() - started) * 1000 / max(len(items), 1)
-
-
-def softmax(z: np.ndarray, t: float) -> np.ndarray:
-    z = z / t
-    e = np.exp(z - z.max())
-    return e / e.sum()
-
-
-def nll(logits: list[np.ndarray], targets: list[np.ndarray], t: float) -> float:
-    return float(np.mean([-(tg * np.log(np.clip(softmax(z, t), 1e-12, 1))).sum() for z, tg in zip(logits, targets)]))
-
-
-def metrics(probs: list[np.ndarray], targets: list[np.ndarray], bins: int = 10) -> dict[str, float]:
-    top = np.array([p.max() for p in probs])
-    hit = np.array([p.argmax() == t.argmax() for p, t in zip(probs, targets)], dtype=float)
-    # ECE over the model's top prediction, scored against the (possibly soft) target's argmax.
-    ece = 0.0
-    for lo in np.linspace(0, 1, bins, endpoint=False):
-        sel = (top > lo) & (top <= lo + 1 / bins)
-        if sel.any():
-            ece += sel.mean() * abs(top[sel].mean() - hit[sel].mean())
-    return {
-        "n": len(probs),
-        "acc": float(hit.mean()),
-        "nll": float(np.mean([-(t * np.log(np.clip(p, 1e-12, 1))).sum() for p, t in zip(probs, targets)])),
-        "brier": float(np.mean([((p - t) ** 2).sum() for p, t in zip(probs, targets)])),
-        "ece": float(ece),
-    }
-
-
-def rank(x: np.ndarray) -> np.ndarray:
-    """Average ranks (ties share their mean rank), for Spearman correlation."""
-    order = np.argsort(x, kind="stable")
-    ranks = np.empty(len(x))
-    ranks[order] = np.arange(len(x))
-    for v in np.unique(x):
-        tie = x == v
-        ranks[tie] = ranks[tie].mean()
-    return ranks
-
-
-def ordinal_metrics(probs: list[np.ndarray], targets: list[np.ndarray]) -> dict[str, float]:
-    """Score questions: compare expected levels, normalized to 0-1 so scales of any size mix.
-
-    mae: mean |E[pred] - E[target]| on the 0-1 scale; within1: argmax level within one of the
-    target's expected level; spearman: rank correlation of expected scores.
-    """
-    pred = np.array([np.dot(np.arange(len(p)), p) / (len(p) - 1) for p in probs])
-    true = np.array([np.dot(np.arange(len(t)), t) / (len(t) - 1) for t in targets])
-    within = [abs(int(p.argmax()) - np.dot(np.arange(len(t)), t)) <= 1 for p, t in zip(probs, targets)]
-    spearman = float(np.corrcoef(rank(pred), rank(true))[0, 1]) if len(pred) > 2 and true.std() > 0 else float("nan")
-    return {"mae": float(np.abs(pred - true).mean()), "within1": float(np.mean(within)), "spearman": spearman}
-
-
-def print_table(title: str, groups: dict[str, dict[str, float]]) -> None:
-    print(f"\n{title}")
-    print(f"{'':<42}{'n':>6}{'acc':>8}{'nll':>8}{'brier':>8}{'ece':>8}{'mae':>8}{'±1':>8}{'rho':>8}")
-    for name, m in sorted(groups.items()):
-        ordinal = "".join(f"{m[k]:>8.3f}" for k in ("mae", "within1", "spearman") if k in m)
-        print(f"{name[:41]:<42}{m['n']:>6}{m['acc']:>8.3f}{m['nll']:>8.3f}{m['brier']:>8.3f}{m['ece']:>8.3f}{ordinal}")
-
-
-def main_eval() -> None:
-    ap = argparse.ArgumentParser(description="Evaluate a Jet model.")
-    ap.add_argument("--adapter", default=None, help="adapter dir; omit to evaluate the base model")
-    ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL)
-    ap.add_argument("--data", type=Path, default=Path("data/test.jsonl"))
-    ap.add_argument("--limit", type=int, default=None)
-    ap.add_argument("--batch-size", type=int, default=16)
-    ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS)
-    ap.add_argument("--json", type=Path, default=None, help="also write metrics as JSON")
-    args = ap.parse_args()
-
-    jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens)
-    rows = load_data(args.data, args.limit)
-    logits, ms = collect_logits(jet, rows, args.batch_size)
-    probs = [softmax(z, jet.temperatures[r["question"]["type"]]) for z, r in zip(logits, rows)]
-    targets = [np.array(r["target"]) for r in rows]
-
-    by_source, by_type = defaultdict(list), defaultdict(list)
-    for i, r in enumerate(rows):
-        source = "distill (all)" if r["source"].startswith("distill:") else r["source"]
-        by_source[source].append(i)
-        by_type[r["question"]["type"]].append(i)
-
-    def group(ix: list[int]) -> dict[str, float]:
-        m = metrics([probs[i] for i in ix], [targets[i] for i in ix])
-        scored = [i for i in ix if rows[i]["question"]["type"] == "score"]
-        if len(scored) == len(ix):
-            m |= ordinal_metrics([probs[i] for i in scored], [targets[i] for i in scored])
-        return m
-
-    overall = metrics(probs, targets)
-    print_table("by source", {k: group(v) for k, v in by_source.items()})
-    print_table("by type", {k: group(v) for k, v in by_type.items()})
-    print_table("overall", {"all": overall})
-    print(f"\n{ms:.1f} ms/question (batched, batch size {args.batch_size}); temperatures {jet.temperatures}")
-    if args.json:
-        args.json.write_text(json.dumps({
-            "overall": overall,
-            "by_source": {k: group(v) for k, v in by_source.items()},
-            "by_type": {k: group(v) for k, v in by_type.items()},
-            "ms_per_question": ms,
-            "max_state_tokens": args.max_state_tokens,
-        }, indent=2))
-
-
-def main_calibrate() -> None:
-    ap = argparse.ArgumentParser(description="Fit per-type softmax temperatures on held-out data.")
-    ap.add_argument("--adapter", required=True)
-    ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL)
-    ap.add_argument("--data", type=Path, default=Path("data/val.jsonl"))
-    ap.add_argument("--limit", type=int, default=3000)
-    ap.add_argument("--batch-size", type=int, default=16)
-    ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS)
-    args = ap.parse_args()
-
-    jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens)
-    rows = load_data(args.data, args.limit)
-    logits, _ = collect_logits(jet, rows, args.batch_size)
-
-    temperatures = {}
-    for qtype in ("choice", "score", "noul"):
-        ix = [i for i, r in enumerate(rows) if r["question"]["type"] == qtype]
-        if not ix:
-            temperatures[qtype] = 1.0
-            continue
-        zs, ts = [logits[i] for i in ix], [np.array(rows[i]["target"]) for i in ix]
-        scores = [nll(zs, ts, t) for t in TEMPERATURE_GRID]
-        best = float(TEMPERATURE_GRID[int(np.argmin(scores))])
-        temperatures[qtype] = round(best, 4)
-        print(f"{qtype:>6}: n={len(ix)} T={best:.3f} nll {nll(zs, ts, 1.0):.4f} -> {min(scores):.4f}")
-    (Path(args.adapter) / "calibration.json").write_text(json.dumps(temperatures, indent=2))
-    print(f"wrote {Path(args.adapter) / 'calibration.json'}")

docs/training/jet-next/code/src/format.py

@@ -1,143 +0,0 @@
-"""Prompt format shared by training and inference.
-
-Every question is reduced to "pick one label token". The model is run once over
-the prompt and we read the next-token logits restricted to the label tokens:
-
-- choice: one label per option (A..Z, then single-token AA, AB, ...), up to 255 options
-- score:  one digit per level (0..9), 2-10 levels, ordered low -> high
-- noul:   "no" / "yes"
-"""
-
-from __future__ import annotations
-
-import itertools
-import json
-import string
-from dataclasses import dataclass
-from typing import Any, Literal
-
-QuestionType = Literal["choice", "score", "noul"]
-
-MAX_CHOICE_OPTIONS = 255
-MIN_SCORE_LEVELS, MAX_SCORE_LEVELS = 2, 10
-
-SYSTEM_PROMPT = (
-    "You are Jet, a decision model. Read the state and the question, then answer "
-    "with exactly one label from the allowed labels."
-)
-
-
-@dataclass(frozen=True)
-class Question:
-    type: QuestionType
-    instructions: str
-    # choice: {key: description}; score: [level descriptions, low -> high]; noul: None
-    criteria: dict[str, str] | list[str] | None = None
-
-    @property
-    def keys(self) -> list[str]:
-        """Answer keys in label order (option keys, level descriptions, or no/yes)."""
-        if self.type == "choice":
-            return list(self.criteria)
-        if self.type == "score":
-            return list(self.criteria)
-        return ["no", "yes"]
-
-    @classmethod
-    def from_dict(cls, d: dict[str, Any]) -> Question:
-        q = cls(type=d["type"], instructions=d["instructions"], criteria=d.get("criteria"))
-        q.validate()
-        return q
-
-    def to_dict(self) -> dict[str, Any]:
-        d: dict[str, Any] = {"type": self.type, "instructions": self.instructions}
-        if self.criteria is not None:
-            d["criteria"] = self.criteria
-        return d
-
-    def validate(self) -> None:
-        if self.type not in ("choice", "score", "noul"):
-            raise ValueError(f"unknown question type {self.type!r}")
-        if not isinstance(self.instructions, str) or not self.instructions.strip():
-            raise ValueError("instructions must be a non-empty string")
-        if self.type == "choice":
-            if not isinstance(self.criteria, dict) or not 2 <= len(self.criteria) <= MAX_CHOICE_OPTIONS:
-                raise ValueError(f"choice criteria must be an object with 2-{MAX_CHOICE_OPTIONS} options")
-        elif self.type == "score":
-            if not isinstance(self.criteria, list) or not MIN_SCORE_LEVELS <= len(self.criteria) <= MAX_SCORE_LEVELS:
-                raise ValueError(f"score criteria must be a list of {MIN_SCORE_LEVELS}-{MAX_SCORE_LEVELS} levels")
-        elif self.criteria is not None:
-            raise ValueError("noul questions take no criteria")
-
-
-_CHOICE_LABELS: dict[int, list[str]] = {}
-
-
-def choice_labels(tokenizer, n: int) -> list[str]:
-    """A..Z, then the two-letter labels AA, AB, ... that are single tokens for this tokenizer."""
-    key = id(tokenizer)
-    if key not in _CHOICE_LABELS:
-        letters = string.ascii_uppercase
-        pairs = ("".join(p) for p in itertools.product(letters, repeat=2))
-        candidates = list(letters) + [p for p in pairs if len(tokenizer.encode(p, add_special_tokens=False)) == 1]
-        if len(candidates) < MAX_CHOICE_OPTIONS:
-            raise ValueError(f"tokenizer only has {len(candidates)} single-token choice labels")
-        _CHOICE_LABELS[key] = candidates[:MAX_CHOICE_OPTIONS]
-    return _CHOICE_LABELS[key][:n]
-
-
-def labels_for(tokenizer, q: Question) -> list[str]:
-    if q.type == "choice":
-        return choice_labels(tokenizer, len(q.criteria))
-    if q.type == "score":
-        return [str(i) for i in range(len(q.criteria))]
-    return ["no", "yes"]
-
-
-def render_state(state: Any) -> str:
-    if isinstance(state, str):
-        return state
-    if isinstance(state, list) and all(isinstance(s, str) for s in state):
-        return "\n\n".join(state)
-    return json.dumps(state, ensure_ascii=False, indent=2)
-
-
-def render_user(state: Any, q: Question, labels: list[str]) -> str:
-    parts = [f"<state>\n{render_state(state)}\n</state>", ""]
-    if q.type == "choice":
-        opts = "\n".join(f"{lab}: {key}: {desc}" for lab, (key, desc) in zip(labels, q.criteria.items()))
-        parts += [
-            f"Question: {q.instructions}",
-            f"Options:\n{opts}",
-            f"Answer with the label of the best option ({labels[0]}-{labels[-1]}).",
-        ]
-    elif q.type == "score":
-        levels = "\n".join(f"{lab}: {desc}" for lab, desc in zip(labels, q.criteria))
-        parts += [
-            f"Question: {q.instructions}",
-            f"Scale (lowest to highest):\n{levels}",
-            f"Answer with the level number (0-{len(labels) - 1}).",
-        ]
-    else:
-        parts += [f"Question: {q.instructions}", "Answer yes or no."]
-    return "\n".join(parts)
-
-
-def build_prompt(tokenizer, state: Any, q: Question) -> str:
-    return tokenizer.apply_chat_template(
-        [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": render_user(state, q, labels_for(tokenizer, q))}],
-        add_generation_prompt=True,
-        tokenize=False,
-        enable_thinking=False,
-    )
-
-
-def _label_token_id(tokenizer, label: str) -> int:
-    ids = tokenizer.encode(label, add_special_tokens=False)
-    if len(ids) != 1:
-        raise ValueError(f"label {label!r} is not a single token for this tokenizer")
-    return ids[0]
-
-
-def label_token_ids(tokenizer, q: Question) -> list[int]:
-    return [_label_token_id(tokenizer, lab) for lab in labels_for(tokenizer, q)]

docs/training/jet-next/code/src/inference.py

@@ -1,41 +0,0 @@
-"""Backend-independent Jet prompt encoding and typed answers."""
-from __future__ import annotations
-import math
-from typing import Any
-import numpy as np
-from format import Question, build_prompt, render_state
-
-DEFAULT_MAX_STATE_TOKENS = 4096
-
-def encode(tokenizer, state: Any, q: Question, max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS) -> list[int]:
-    """Tokenize a prompt, truncating the middle of an over-long state."""
-    text = render_state(state)
-    ids = tokenizer.encode(text, add_special_tokens=False)
-    if len(ids) > max_state_tokens:
-        half = max_state_tokens // 2
-        text = tokenizer.decode(ids[:half]) + "\n[...]\n" + tokenizer.decode(ids[-half:])
-    return tokenizer.encode(build_prompt(tokenizer, text, q), add_special_tokens=False)
-
-def summarize(q: Question, probs: np.ndarray) -> dict[str, Any]:
-    """Turn a label distribution into the typed answer for the question."""
-    probs = probs.astype(np.float64)
-    k = len(probs)
-    entropy = -float(np.sum(probs * np.log(np.clip(probs, 1e-12, 1.0))))
-    confidence = round(1.0 - entropy / math.log(k), 4)
-    if q.type == "choice":
-        keys = q.keys
-        return {
-            "type": "choice",
-            "choice": keys[int(np.argmax(probs))],
-            "probabilities": {key: round(float(p), 4) for key, p in zip(keys, probs)},
-            "confidence": confidence,
-        }
-    if q.type == "score":
-        return {
-            "type": "score",
-            "score": round(float(np.dot(np.arange(k), probs)), 4),
-            "level": q.criteria[int(np.argmax(probs))],
-            "probabilities": [round(float(p), 4) for p in probs],
-            "confidence": confidence,
-        }
-    return {"type": "noul", "probability": round(float(probs[1]), 4), "confidence": confidence}

docs/training/jet-next/code/src/model.py

@@ -1,151 +0,0 @@
-"""Jet inference: one forward pass per batch of questions, no generation."""
-
-from __future__ import annotations
-
-import json
-import time
-from pathlib import Path
-from typing import Any
-
-import mlx.core as mx
-import numpy as np
-from huggingface_hub import snapshot_download
-from mlx_lm import load
-from mlx_lm.models.cache import make_prompt_cache
-
-from format import Question, label_token_ids
-from inference import DEFAULT_MAX_STATE_TOKENS, encode, summarize
-
-DEFAULT_BASE_MODEL = "mlx-community/Qwen3-0.6B-bf16"
-NEG_INF = -1e9
-
-
-def label_logits(model, tokens: mx.array, lengths: mx.array, label_ids: mx.array, label_mask: mx.array) -> mx.array:
-    """Next-token logits after each prompt, restricted to its label tokens.
-
-    tokens: (B, L) right-padded prompts; lengths: (B,) true lengths;
-    label_ids / label_mask: (B, K) label token ids padded to the widest question.
-    Returns (B, K) logits with padded labels set to -inf.
-    """
-    hidden = model.model(tokens)
-    last = hidden[mx.arange(tokens.shape[0]), lengths - 1][:, None, :]
-    logits = head_logits(model, last)[:, 0, :]
-    picked = mx.take_along_axis(logits, label_ids, axis=-1).astype(mx.float32)
-    return mx.where(label_mask, picked, NEG_INF)
-
-
-def head_logits(model, hidden: mx.array) -> mx.array:
-    if getattr(model.args, "tie_word_embeddings", False):
-        return model.model.embed_tokens.as_linear(hidden)
-    return model.lm_head(hidden)
-
-
-def shared_prefix_label_logits(
-    model, seqs: list[list[int]], label_lists: list[list[int]], max_cached_tokens: int = 32_768
-) -> np.ndarray:
-    """Like label_logits, but encodes the prompts' common prefix (system + state) once.
-
-    The prefix KV cache is replicated across a chunk of question suffixes, so each
-    question only pays for its own few dozen tokens. Chunks are sized so that
-    replicated prefix tokens stay under max_cached_tokens.
-    """
-    common = min(len(s) for s in seqs) - 1  # every suffix keeps at least one token
-    for i in range(common):
-        if any(s[i] != seqs[0][i] for s in seqs):
-            common = i
-            break
-    prefix_cache = make_prompt_cache(model)
-    if common:
-        model.model(mx.array(seqs[0][:common])[None], cache=prefix_cache)
-    prefix_state = [c.state for c in prefix_cache] if common else None
-    if prefix_state:
-        mx.eval(prefix_state)
-
-    out = np.full((len(seqs), max(map(len, label_lists))), NEG_INF, dtype=np.float32)
-    chunk = max(1, max_cached_tokens // max(common, 1))
-    for start in range(0, len(seqs), chunk):
-        suffixes = [s[common:] for s in seqs[start : start + chunk]]
-        b = len(suffixes)
-        cache = make_prompt_cache(model)
-        if prefix_state:
-            for c, (k, v) in zip(cache, prefix_state):
-                c.state = (mx.repeat(k, b, axis=0), mx.repeat(v, b, axis=0))
-        tokens, lengths = pad_batch(suffixes)
-        hidden = model.model(tokens, cache=cache)
-        last = hidden[mx.arange(b), lengths - 1][:, None, :]
-        logits = head_logits(model, last)[:, 0, :]
-        ids, mask = pad_labels(label_lists[start : start + chunk])
-        picked = mx.where(mask, mx.take_along_axis(logits, ids, axis=-1).astype(mx.float32), NEG_INF)
-        out[start : start + b, : picked.shape[1]] = np.array(picked)
-    return out
-
-
-def pad_batch(seqs: list[list[int]], pad: int = 0) -> tuple[mx.array, mx.array]:
-    width = max(len(s) for s in seqs)
-    arr = np.full((len(seqs), width), pad, dtype=np.int32)
-    for i, s in enumerate(seqs):
-        arr[i, : len(s)] = s
-    return mx.array(arr), mx.array([len(s) for s in seqs], dtype=mx.int32)
-
-
-def pad_labels(label_lists: list[list[int]]) -> tuple[mx.array, mx.array]:
-    width = max(len(ls) for ls in label_lists)
-    ids = np.zeros((len(label_lists), width), dtype=np.int32)
-    mask = np.zeros((len(label_lists), width), dtype=bool)
-    for i, ls in enumerate(label_lists):
-        ids[i, : len(ls)] = ls
-        mask[i, : len(ls)] = True
-    return mx.array(ids), mx.array(mask)
-
-
-class Jet:
-    def __init__(
-        self,
-        base_model: str = DEFAULT_BASE_MODEL,
-        adapter_path: str | None = None,
-        max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS,
-    ):
-        self.model, self.tokenizer = load(base_model, adapter_path=adapter_path)
-        self.model.eval()
-        self.max_state_tokens = max_state_tokens
-        self.temperatures = {"choice": 1.0, "score": 1.0, "noul": 1.0}
-        # Calibration lives next to the adapter, or inside a fused model dir. A Hub repo id has no dir
-        # of its own, so look in its local snapshot (fetching calibration.json if the repo has one).
-        model_dir = Path(base_model) if Path(base_model).exists() else Path(snapshot_download(base_model, allow_patterns=["calibration.json"]))
-        for directory in (adapter_path, model_dir):
-            if directory and (calib := Path(directory) / "calibration.json").exists():
-                self.temperatures.update(json.loads(calib.read_text()))
-                break
-
-    def raw_logits(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]:
-        """Uncalibrated label logits for (state, question) pairs, plus the number of
-        input tokens. With shared_state, the common state prefix is encoded once."""
-        seqs = [encode(self.tokenizer, s, q, self.max_state_tokens) for s, q in items]
-        label_lists = [label_token_ids(self.tokenizer, q) for _, q in items]
-        if shared_state and len(items) > 1:
-            out = shared_prefix_label_logits(self.model, seqs, label_lists)
-        else:
-            tokens, lengths = pad_batch(seqs)
-            ids, mask = pad_labels(label_lists)
-            out = np.array(label_logits(self.model, tokens, lengths, ids, mask))
-        return [out[i, : len(q.keys)] for i, (_, q) in enumerate(items)], sum(map(len, seqs))
-
-    def probabilities(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]:
-        logits, n_tokens = self.raw_logits(items, shared_state)
-        result = []
-        for (_, q), z in zip(items, logits):
-            z = z / self.temperatures[q.type]
-            z = np.exp(z - z.max())
-            result.append(z / z.sum())
-        return result, n_tokens
-
-    def decide(self, state: Any, questions: dict[str, Question]) -> dict[str, Any]:
-        """Answer every question about one state together (shared batch)."""
-        started = time.perf_counter()
-        items = [(state, q) for q in questions.values()]
-        probs, input_tokens = self.probabilities(items, shared_state=True)
-        return {
-            "answers": {name: summarize(q, p) for (name, q), p in zip(questions.items(), probs)},
-            "usage": {"input_tokens": input_tokens},
-            "latency_ms": round((time.perf_counter() - started) * 1000, 1),
-        }

docs/training/jet-next/code/src/train.py

@@ -1,304 +0,0 @@
-"""LoRA fine-tuning of a small LM into a Jet decision model (MLX, Apple Silicon).
-
-The loss is cross-entropy between the soft target and the model's next-token
-distribution restricted to the question's label tokens, read at the last prompt
-position only - exactly what inference computes. Score questions add a ranked
-probability score term, so mass far from the target level costs more than mass
-next to it (cross-entropy alone treats the levels as unordered).
-"""
-
-from __future__ import annotations
-
-import argparse
-import hashlib
-import json
-import math
-import random
-import time
-from functools import partial
-from pathlib import Path
-
-import mlx.core as mx
-import mlx.nn as nn
-import mlx.optimizers as optim
-import numpy as np
-from mlx.utils import tree_flatten, tree_unflatten
-from mlx_lm import load
-from huggingface_hub import snapshot_download
-from mlx_lm.tuner.trainer import grad_checkpoint
-from mlx_lm.tuner.utils import linear_to_lora_layers
-
-from format import Question, label_token_ids
-from model import DEFAULT_BASE_MODEL, encode, label_logits, pad_batch, pad_labels
-
-
-class Example:
-    __slots__ = ("tokens", "labels", "target", "ordinal", "weight")
-
-    def __init__(self, tokens: list[int], labels: list[int], target: list[float], ordinal: bool, weight: float = 1.0):
-        self.tokens, self.labels, self.target, self.ordinal = tokens, labels, target, ordinal
-        self.weight = weight
-
-
-def load_examples(path: Path, tokenizer, max_state_tokens: int, smoothing: float, use_weights: bool = False) -> list[Example]:
-    out = []
-    for line in path.open():
-        ex = json.loads(line)
-        q = Question.from_dict(ex["question"])
-        k = len(q.keys)
-        target = [(1 - smoothing) * t + smoothing / k for t in ex["target"]]
-        weight = float(ex.get("weight", 1.0)) if use_weights else 1.0
-        if not math.isfinite(weight) or weight <= 0:
-            raise ValueError("training weights must be finite and positive")
-        out.append(Example(encode(tokenizer, ex["state"], q, max_state_tokens), label_token_ids(tokenizer, q), target, q.type == "score", weight))
-    return out
-
-
-def make_batches(examples: list[Example], max_batch_tokens: int, rng: random.Random | None) -> list[list[Example]]:
-    """Group similar lengths together; each batch's padded size stays under max_batch_tokens."""
-    order = sorted(examples, key=lambda e: len(e.tokens))
-    batches, cur = [], []
-    for ex in order:
-        width = max(len(ex.tokens), max((len(e.tokens) for e in cur), default=0))
-        if cur and width * (len(cur) + 1) > max_batch_tokens:
-            batches.append(cur)
-            cur = []
-        cur.append(ex)
-    if cur:
-        batches.append(cur)
-    if rng:
-        rng.shuffle(batches)
-    return batches
-
-
-def to_arrays(batch: list[Example]):
-    tokens, lengths = pad_batch([e.tokens for e in batch])
-    ids, mask = pad_labels([e.labels for e in batch])
-    target = np.zeros(ids.shape, dtype=np.float32)
-    for i, e in enumerate(batch):
-        target[i, : len(e.target)] = e.target
-    ordinal = mx.array([e.ordinal for e in batch])
-    levels = mx.array([len(e.labels) for e in batch], dtype=mx.float32)
-    return tokens, lengths, ids, mask, mx.array(target), ordinal, levels, mx.array([e.weight for e in batch])
-
-
-def ranked_probability_score(logp: mx.array, target: mx.array, levels: mx.array) -> mx.array:
-    """Squared distance between predicted and target CDFs, divided by (levels - 1) so it stays
-    in 0-1 for any scale size. Padded labels have zero mass in both, so they add nothing."""
-    cdf_gap = mx.cumsum(mx.exp(logp), axis=-1) - mx.cumsum(target, axis=-1)
-    return (cdf_gap**2).sum(axis=-1) / (levels - 1)
-
-
-def loss_fn(model, tokens, lengths, ids, mask, target, ordinal, levels, weights, ordinal_weight=0.0):
-    logits = label_logits(model, tokens, lengths, ids, mask)
-    logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True)
-    loss = -(target * logp).sum(axis=-1)
-    if ordinal_weight:
-        loss = loss + ordinal_weight * mx.where(ordinal, ranked_probability_score(logp, target, levels), 0.0)
-    return (loss * weights).mean()
-
-
-def evaluate(model, batches: list[list[Example]]) -> dict[str, float]:
-    total, nll, correct = 0, 0.0, 0
-    for batch in batches:
-        tokens, lengths, ids, mask, target, *_ = to_arrays(batch)
-        logits = label_logits(model, tokens, lengths, ids, mask)
-        logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True)
-        nll += float(-(target * logp).sum())
-        correct += int((mx.argmax(logits, axis=-1) == mx.argmax(target, axis=-1)).sum())
-        total += len(batch)
-    return {"nll": nll / total, "acc": correct / total}
-
-
-def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | None = None) -> None:
-    out.mkdir(parents=True, exist_ok=True)
-    mx.save_safetensors(str(out / "adapters.safetensors"), dict(tree_flatten(model.trainable_parameters())))
-    (out / "adapter_config.json").write_text(json.dumps(config, indent=2))
-    if optimizer is not None:
-        # Adam moments + step, so --resume continues exactly instead of restarting cold.
-        mx.save_safetensors(str(out / "optimizer.safetensors"), dict(tree_flatten(optimizer.state)))
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.")
-    ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL)
-    ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit")
-    ap.add_argument("--train", type=Path, default=Path("data/train.jsonl"))
-    ap.add_argument("--val", type=Path, default=Path("data/val.jsonl"))
-    ap.add_argument("--out", type=Path, default=Path("adapters/jet"))
-    ap.add_argument("--epochs", type=float, default=2.0)
-    ap.add_argument("--lr", type=float, default=1e-4)
-    ap.add_argument("--warmup", type=int, default=100)
-    ap.add_argument("--rank", type=int, default=16)
-    ap.add_argument("--lora-scale", type=float, default=10.0)
-    ap.add_argument("--dropout", type=float, default=0.05)
-    ap.add_argument("--lora-layers", type=int, default=-1, help="-1 = all layers")
-    ap.add_argument("--max-batch-tokens", type=int, default=4096)
-    ap.add_argument("--max-state-tokens", type=int, default=1024)
-    ap.add_argument("--smoothing", type=float, default=0.02)
-    ap.add_argument("--use-example-weights", action="store_true",
-                    help="use positive per-row weights for training only; validation remains unweighted")
-    ap.add_argument("--ordinal-weight", type=float, default=2.0,
-                    help="weight of the ranked probability score term on score questions; 0 = plain cross-entropy")
-    ap.add_argument("--grad-checkpoint", action=argparse.BooleanOptionalAction, default=True)
-    ap.add_argument("--eval-every", type=int, default=250)
-    ap.add_argument("--val-limit", type=int, default=1000)
-    ap.add_argument("--cache-limit-gb", type=float, default=1.0,
-                    help="cap on MLX's freed-buffer cache; variable batch shapes otherwise grow it toward all of RAM")
-    ap.add_argument("--resume", type=Path, default=None, help="adapter dir to continue from")
-    ap.add_argument("--init-adapter", type=Path, default=None,
-                    help="warm-start adapter weights with a fresh optimizer and schedule")
-    ap.add_argument("--start-step", type=int, default=None,
-                    help="step the resumed adapter was saved at; read from its optimizer state when present")
-    ap.add_argument("--seed", type=int, default=0)
-    args = ap.parse_args()
-    if args.init_adapter and (args.resume or args.start_step is not None):
-        ap.error("--init-adapter cannot be combined with --resume or --start-step")
-    if (args.out / "adapters.safetensors").exists():
-        ap.error("output already contains an adapter; choose a new directory")
-
-    args.out.mkdir(parents=True, exist_ok=True)
-    run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()}
-    run_config["data_sha256"] = {
-        str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val)
-    }
-    if args.init_adapter:
-        source_config = json.loads((args.init_adapter / "adapter_config.json").read_text())
-        expected = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout}
-        if source_config.get("lora_parameters") != expected:
-            ap.error("initial adapter LoRA parameters do not match requested parameters")
-        if source_config.get("base_model") != args.base_model:
-            ap.error("initial adapter uses a different base model")
-        if source_config.get("base_revision") not in (None, args.base_revision):
-            ap.error("initial adapter uses a different base revision")
-        run_config["init_adapter_sha256"] = hashlib.sha256(
-            (args.init_adapter / "adapters.safetensors").read_bytes()).hexdigest()
-    (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2))
-
-    mx.set_cache_limit(int(args.cache_limit_gb * 1e9))
-    mx.random.seed(args.seed)
-    rng = random.Random(args.seed)
-    model_path = args.base_model
-    if args.base_revision:
-        model_path = snapshot_download(args.base_model, revision=args.base_revision)
-    model, tokenizer = load(model_path)
-    model.freeze()
-    num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers
-    lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout}
-    linear_to_lora_layers(model, num_layers, lora_parameters)
-    if args.init_adapter:
-        if source_config.get("num_layers") != num_layers:
-            ap.error("initial adapter layer count does not match")
-        weights = mx.load(str(args.init_adapter / "adapters.safetensors"))
-        expected_weights = dict(tree_flatten(model.trainable_parameters()))
-        if weights.keys() != expected_weights.keys() or any(
-            weights[k].shape != expected_weights[k].shape for k in expected_weights
-        ):
-            ap.error("initial adapter tensor names or shapes do not match")
-        model.load_weights(list(weights.items()), strict=False)
-        print(f"initialized weights from {args.init_adapter}; optimizer starts fresh", flush=True)
-    if args.resume:
-        model.load_weights(str(args.resume / "adapters.safetensors"), strict=False)
-        print(f"resumed weights from {args.resume}")
-    if args.grad_checkpoint:
-        grad_checkpoint(model.layers[0])
-    n_train = sum(v.size for _, v in tree_flatten(model.trainable_parameters()))
-    print(f"trainable params: {n_train / 1e6:.2f}M")
-
-    train = load_examples(args.train, tokenizer, args.max_state_tokens, args.smoothing, args.use_example_weights)
-    val = load_examples(args.val, tokenizer, args.max_state_tokens, 0.0)
-    rng.shuffle(val)
-    val_batches = make_batches(val[: args.val_limit], args.max_batch_tokens, None)
-    steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None))
-    total_steps = math.ceil(steps_per_epoch * args.epochs)
-    print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches")
-    run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps)
-    (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2))
-
-    warmup = max(1, min(args.warmup, total_steps // 10))
-    base_schedule = optim.join_schedules(
-        [optim.linear_schedule(1e-7, args.lr, warmup), optim.cosine_decay(args.lr, total_steps - warmup, args.lr * 0.05)],
-        [warmup],
-    )
-    optimizer_state = args.resume / "optimizer.safetensors" if args.resume else None
-    if optimizer_state is not None and optimizer_state.exists():
-        schedule = base_schedule
-        optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01)
-        optimizer.init(model.trainable_parameters())
-        optimizer.state = tree_unflatten(list(mx.load(str(optimizer_state)).items()))
-        start_step = int(optimizer.state["step"].item())
-        print(f"restored optimizer state at step {start_step}")
-    else:
-        start_step = args.start_step or 0
-        if args.resume:
-            # Cold Adam moments (MLX AdamW has no bias correction by default) make the first
-            # updates several times too large; ramp the LR back up to protect the checkpoint.
-            def schedule(step, start=start_step):
-                return base_schedule(step) * mx.minimum(1.0, (step - start + 1).astype(mx.float32) / warmup)
-            print(f"no optimizer state in {args.resume}; re-warming the LR over {warmup} steps")
-        else:
-            schedule = base_schedule
-        optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01)
-        optimizer.state["step"] = mx.array(start_step, mx.uint64)
-    value_and_grad = nn.value_and_grad(model, partial(loss_fn, ordinal_weight=args.ordinal_weight))
-
-    adapter_config = {
-        "fine_tune_type": "lora",
-        "num_layers": num_layers,
-        "lora_parameters": lora_parameters,
-        "base_model": args.base_model,
-        "base_revision": args.base_revision,
-    }
-    model.eval()
-    best = evaluate(model, val_batches)
-    # Keep the initial checkpoint eligible; a regression must not replace it merely
-    # because no trained checkpoint improved the validation loss.
-    save_adapter(model, args.out, adapter_config, optimizer)
-    (args.out / "metrics.jsonl").write_text(json.dumps({"step": 0, **best,
-        "improved": False, "initial": True}) + "\n")
-    print(f"step 0 val nll {best['nll']:.4f} acc {best['acc']:.3f}")
-    model.train()
-
-    saved = False
-    step, seen_tokens, started = start_step, 0, time.perf_counter()
-    losses: list[float] = []
-    while step < total_steps:
-        for batch in make_batches(train, args.max_batch_tokens, rng):
-            if step >= total_steps:
-                break
-            arrays = to_arrays(batch)
-            loss, grads = value_and_grad(model, *arrays)
-            optimizer.update(model, grads)
-            mx.eval(model.trainable_parameters(), optimizer.state, loss)
-            step += 1
-            losses.append(loss.item())
-            seen_tokens += int(arrays[1].sum())
-            if step % 25 == 0:
-                elapsed = time.perf_counter() - started
-                print(
-                    f"step {step}/{total_steps} loss {np.mean(losses):.4f} lr {optimizer.learning_rate.item():.2e} "
-                    f"{seen_tokens / elapsed:.0f} tok/s peak mem {mx.get_peak_memory() / 1e9:.1f}GB cache {mx.get_cache_memory() / 1e9:.1f}GB",
-                    flush=True,
-                )
-                losses = []
-            if step % args.eval_every == 0 or step == total_steps:
-                model.eval()
-                metrics = evaluate(model, val_batches)
-                model.train()
-                improved = metrics["nll"] < best["nll"]
-                print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{'  *saved*' if improved else ''}", flush=True)
-                with (args.out / "metrics.jsonl").open("a") as f:
-                    f.write(json.dumps({"step": step, **metrics, "improved": improved,
-                                        "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n")
-                save_adapter(model, args.out / "last", adapter_config, optimizer)
-                if improved:
-                    best = metrics
-                    save_adapter(model, args.out, adapter_config, optimizer)
-                    saved = True
-    if not saved:
-        print("validation never improved; retaining initial weights (final weights in last/)")
-    print(f"best val nll {best['nll']:.4f} acc {best['acc']:.3f}; adapter in {args.out}")
-
-
-if __name__ == "__main__":
-    main()

docs/training/jet-next/code/train_cuda.sh

@@ -1,21 +0,0 @@
-#!/usr/bin/env bash
-# Run jet-train on Linux/CUDA. Extra args go to jet-train.
-set -euo pipefail
-cd "$(dirname "$0")"
-uv sync --extra cuda --inexact --quiet
-
-# MLX JIT-compiles kernels and wants one CUDA_HOME/include; the pip wheels spread the headers
-# over several packages, so link them into a single tree.
-export CUDA_HOME=$PWD/.venv/cuda_home
-if [ ! -d "$CUDA_HOME/include" ]; then
-    mkdir -p "$CUDA_HOME/include"
-    for pkg in .venv/lib/python*/site-packages/nvidia/{cuda_runtime,cuda_cccl,cuda_nvrtc}/include; do
-        for f in "$PWD/$pkg"/*; do
-            [ "$(basename "$f")" = __init__.py ] || ln -sfn "$f" "$CUDA_HOME/include/"
-        done
-    done
-fi
-export MLX_USE_CUDA_GRAPHS=0                      # batch shapes vary every step, so graphs just thrash
-export MLX_CUDA_SDPA_CACHE_SIZE=${MLX_CUDA_SDPA_CACHE_SIZE:-2048}  # same reason; the default 256 aborts mid-run
-export MLX_PTX_CACHE_DIR=$PWD/.venv/ptx_cache     # keep JIT-compiled kernels across runs
-exec uv run --extra cuda --no-sync jet-train "$@"

docs/training/jet-next/completion-audit.json

@@ -1,20 +0,0 @@
-{
-  "status": "passed",
-  "compatibility": {
-    "ok": 26,
-    "total": 26
-  },
-  "tests": {
-    "adapter_including_gpu_golden_and_prefix_parity": 8,
-    "executable_v3_data": 3
-  },
-  "selected_optimizer_steps": {
-    "warm": 2444,
-    "v4": 3750
-  },
-  "original_datasets": "unchanged",
-  "all_v3_datasets": "unchanged",
-  "published_weights_and_calibration": "unchanged",
-  "preexisting_user_changes": "unchanged",
-  "active_training_or_evaluation_jobs": 0
-}

docs/training/jet-next/cruxeval-full-baseline.json

@@ -1,148 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16",
-  "results_sha256": "d61110ea9c6307f5fffa1a2a56ae634f716187298fa65cf5aaa8e2dfb57a8743",
-  "planned_requests": 570,
-  "processed_requests": 570,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "michaljach/jet",
-      "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c"
-    },
-    "model_source": {
-      "model": "michaljach/jet",
-      "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.0293,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts."
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 2.082998502999544,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 570
-    },
-    "successful_request_latency_ms": {
-      "median": 13.089024500004598,
-      "p95": 46.50721500001964,
-      "mean": 22.395500426424213
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 43,
-        "dataset": "CRUXEval",
-        "requests": 570,
-        "answered": 570,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 570,
-        "metric": "accuracy",
-        "score": 0.387719298245614,
-        "reference_same_cases": null,
-        "median_ms": 13.089024500004598,
-        "detail": {
-          "field_accuracy": 0.387719298245614,
-          "case_exact_accuracy": 0.387719298245614,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "43": {
-      "catalog_id": 43,
-      "raw": 0.387719298245614,
-      "skill": 0.028559562416605518,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.387719298245614,
-          "random": 0.36971874129768867,
-          "skill": 0.028559562416605518,
-          "planned_cases": 570,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/cruxeval-full-sample.json

@@ -1,6 +0,0 @@
-{
-  "requests": 570,
-  "selection": "All available CRUXEval rows after official exclusions; no label-based selection",
-  "source_sha256": "8dd94acf6c313c50aec3fc48188c840d994d14f764b3fff9fe7a89e2e41b9211",
-  "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16"
-}

docs/training/jet-next/cruxeval-full-v3.json

@@ -1,152 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16",
-  "results_sha256": "f7748f31af62f0c601fd96d83c2d2b2f57033986547cfa5b4ac7f7c4f90f46b5",
-  "planned_requests": 570,
-  "processed_requests": 570,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v3-decision-20260923"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.0,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923",
-      "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5",
-      "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.8621119829986128,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 570
-    },
-    "successful_request_latency_ms": {
-      "median": 25.944673499907367,
-      "p95": 62.15157100086799,
-      "mean": 35.532386945570785
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 43,
-        "dataset": "CRUXEval",
-        "requests": 570,
-        "answered": 570,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 570,
-        "metric": "accuracy",
-        "score": 0.3473684210526316,
-        "reference_same_cases": null,
-        "median_ms": 25.944673499907367,
-        "detail": {
-          "field_accuracy": 0.3473684210526316,
-          "case_exact_accuracy": 0.3473684210526316,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "43": {
-      "catalog_id": 43,
-      "raw": 0.3473684210526316,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.3473684210526316,
-          "random": 0.36971874129768867,
-          "skill": 0.0,
-          "planned_cases": 570,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/cruxeval-full-v4.json

@@ -1,152 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16",
-  "results_sha256": "c2ec7dfde1a861c361c3eaca34133d624726eaf1df28347fae25ac568df65736",
-  "planned_requests": 570,
-  "processed_requests": 570,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v4-transfer-20260924"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.1554,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924",
-      "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad",
-      "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.8805886459995236,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 570
-    },
-    "successful_request_latency_ms": {
-      "median": 25.62000750185689,
-      "p95": 61.887538999144454,
-      "mean": 35.259878875436506
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 43,
-        "dataset": "CRUXEval",
-        "requests": 570,
-        "answered": 570,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 570,
-        "metric": "accuracy",
-        "score": 0.34385964912280703,
-        "reference_same_cases": null,
-        "median_ms": 25.62000750185689,
-        "detail": {
-          "field_accuracy": 0.34385964912280703,
-          "case_exact_accuracy": 0.34385964912280703,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "43": {
-      "catalog_id": 43,
-      "raw": 0.34385964912280703,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.34385964912280703,
-          "random": 0.36971874129768867,
-          "skill": 0.0,
-          "planned_cases": 570,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/cruxeval-full-warm.json

@@ -1,152 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16",
-  "results_sha256": "9b65aed86040c13dd85316232fbdae3a77500a0dcfad7c9243c0bda1be44f933",
-  "planned_requests": 570,
-  "processed_requests": 570,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v3-warm-20260924"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 0.9439,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924",
-      "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4",
-      "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.9001195040000312,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 570
-    },
-    "successful_request_latency_ms": {
-      "median": 26.362825999967754,
-      "p95": 62.52290100019309,
-      "mean": 36.53820609660327
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 43,
-        "dataset": "CRUXEval",
-        "requests": 570,
-        "answered": 570,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 570,
-        "metric": "accuracy",
-        "score": 0.3508771929824561,
-        "reference_same_cases": null,
-        "median_ms": 26.362825999967754,
-        "detail": {
-          "field_accuracy": 0.3508771929824561,
-          "case_exact_accuracy": 0.3508771929824561,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "43": {
-      "catalog_id": 43,
-      "raw": 0.3508771929824561,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.3508771929824561,
-          "random": 0.36971874129768867,
-          "skill": 0.0,
-          "planned_cases": 570,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/diagnostic-v4.json

@@ -1,515 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d",
-  "results_sha256": "07a78ecfc6f9a4f45c618eee9c3b710d50ea08b9f963fdc43213f01d4374d76e",
-  "planned_requests": 500,
-  "processed_requests": 500,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v4-transfer-20260924",
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.1554,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924",
-      "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad",
-      "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.8496340909987339,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 500
-    },
-    "successful_request_latency_ms": {
-      "median": 60.23746450227918,
-      "p95": 435.2597789984429,
-      "mean": 103.62341392398957
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 11,
-        "dataset": "ContractNLI",
-        "requests": 56,
-        "answered": 56,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 56,
-        "metric": "macro-F1",
-        "score": 0.6536335738829072,
-        "reference_same_cases": null,
-        "median_ms": 410.0216540009569,
-        "detail": {
-          "field_accuracy": 0.6691176470588235,
-          "case_exact_accuracy": 0,
-          "macro_f1": 0.6536335738829072,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 30,
-        "dataset": "GSM8K",
-        "requests": 112,
-        "answered": 112,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 112,
-        "metric": "accuracy",
-        "score": 0.375,
-        "reference_same_cases": null,
-        "median_ms": 54.77364450052846,
-        "tracks": {
-          "GSM8K-10choice": {
-            "metric": "accuracy",
-            "score": 0.2857142857142857,
-            "scored_requests": 56
-          },
-          "GSM8K-4choice": {
-            "metric": "accuracy",
-            "score": 0.4642857142857143,
-            "scored_requests": 56
-          }
-        },
-        "detail": {
-          "field_accuracy": 0.375,
-          "case_exact_accuracy": 0.17857142857142858,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 40,
-        "dataset": "iSarcasmEval",
-        "requests": 56,
-        "answered": 56,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 56,
-        "metric": "see separate subtask tracks",
-        "score": null,
-        "reference_same_cases": null,
-        "median_ms": 61.44454200148175,
-        "tracks": {
-          "iSarcasmEval-A-Ar": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 12
-          },
-          "iSarcasmEval-A-En": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 19
-          },
-          "iSarcasmEval-B-En": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 23
-          },
-          "iSarcasmEval-C-Ar": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 1
-          },
-          "iSarcasmEval-C-En": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 1
-          }
-        },
-        "detail": {
-          "field_accuracy": 0.52046783625731,
-          "case_exact_accuracy": 0.35714285714285715,
-          "macro_f1": 0.12538452073335796,
-          "custom_metrics": {},
-          "positive_f1_by_field": {
-            "sarcastic": 0.125,
-            "sarcasm": 0.2222222222222222,
-            "irony": 0.11764705882352941,
-            "satire": 0.0,
-            "understatement": 0.0,
-            "overstatement": 0.0,
-            "rhetorical_question": 0.0
-          },
-          "category_macro_f1": 0.06640989729225023
-        }
-      },
-      {
-        "catalog_id": 41,
-        "dataset": "VAST",
-        "requests": 56,
-        "answered": 56,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 56,
-        "metric": "macro-F1",
-        "score": 0.4431124431124431,
-        "reference_same_cases": null,
-        "median_ms": 28.437076000045636,
-        "detail": {
-          "field_accuracy": 0.4642857142857143,
-          "case_exact_accuracy": 0.4642857142857143,
-          "macro_f1": 0.4431124431124431,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 42,
-        "dataset": "NLI4CT",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "macro-F1",
-        "score": 0.5203488372093024,
-        "reference_same_cases": null,
-        "median_ms": 71.75399799962179,
-        "detail": {
-          "field_accuracy": 0.6727272727272727,
-          "case_exact_accuracy": 0.6727272727272727,
-          "macro_f1": 0.5203488372093024,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 43,
-        "dataset": "CRUXEval",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "accuracy",
-        "score": 0.32727272727272727,
-        "reference_same_cases": null,
-        "median_ms": 55.303547997027636,
-        "detail": {
-          "field_accuracy": 0.32727272727272727,
-          "case_exact_accuracy": 0.32727272727272727,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 44,
-        "dataset": "CLadder",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "accuracy",
-        "score": 0.4727272727272727,
-        "reference_same_cases": null,
-        "median_ms": 54.30907100162585,
-        "detail": {
-          "field_accuracy": 0.4727272727272727,
-          "case_exact_accuracy": 0.4727272727272727,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 50,
-        "dataset": "Habermas Machine",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "accuracy",
-        "score": 0.32727272727272727,
-        "reference_same_cases": null,
-        "median_ms": 80.21276100043906,
-        "detail": {
-          "field_accuracy": 0.32727272727272727,
-          "case_exact_accuracy": 0.32727272727272727,
-          "custom_metrics": {
-            "normalized_rank_regret": 0.2815151515151515
-          }
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "11": {
-      "catalog_id": 11,
-      "raw": 0.6536335738829072,
-      "skill": 0.4990948536692409,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.6536335738829072,
-          "random": 0.30851893087083776,
-          "skill": 0.4990948536692409,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"The contract contradicts the hypothesis.\"",
-            "\"The contract entails the hypothesis.\"",
-            "\"The hypothesis is neither entailed nor contradicted by the contract.\""
-          ]
-        }
-      ]
-    },
-    "30": {
-      "catalog_id": 30,
-      "raw": 0.375,
-      "skill": 0.24603174603174605,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "GSM8K-4choice",
-          "raw": 0.4642857142857143,
-          "random": 0.25,
-          "skill": 0.28571428571428575,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        },
-        {
-          "track": "GSM8K-10choice",
-          "raw": 0.2857142857142857,
-          "random": 0.1,
-          "skill": 0.2063492063492063,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "40": {
-      "catalog_id": 40,
-      "raw": 0.0625,
-      "skill": 0.00878466561850422,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "iSarcasmEval-A-En",
-          "raw": 0.25,
-          "random": 0.2226862339379383,
-          "skill": 0.03513866247401688,
-          "planned_cases": 19,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "no",
-            "yes"
-          ]
-        },
-        {
-          "track": "iSarcasmEval-A-Ar",
-          "raw": 0.0,
-          "random": 0.22250656045660705,
-          "skill": 0.0,
-          "planned_cases": 12,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "no",
-            "yes"
-          ]
-        },
-        {
-          "track": "iSarcasmEval-C-En",
-          "raw": 0.0,
-          "random": 0.5,
-          "skill": 0.0,
-          "planned_cases": 1,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        },
-        {
-          "track": "iSarcasmEval-C-Ar",
-          "raw": 0.0,
-          "random": 0.5,
-          "skill": 0.0,
-          "planned_cases": 1,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "41": {
-      "catalog_id": 41,
-      "raw": 0.4431124431124431,
-      "skill": 0.16467966294233655,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.4431124431124431,
-          "random": 0.3333245556439576,
-          "skill": 0.16467966294233655,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"against\"",
-            "\"favor\"",
-            "\"neutral\""
-          ]
-        }
-      ]
-    },
-    "43": {
-      "catalog_id": 43,
-      "raw": 0.32727272727272727,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.32727272727272727,
-          "random": 0.34943722943722944,
-          "skill": 0.0,
-          "planned_cases": 55,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "44": {
-      "catalog_id": 44,
-      "raw": 0.4727272727272727,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.4727272727272727,
-          "random": 0.5,
-          "skill": 0.0,
-          "planned_cases": 55,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "50": {
-      "catalog_id": 50,
-      "raw": 0.32727272727272727,
-      "skill": 0.02503293807641634,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.32727272727272727,
-          "random": 0.31,
-          "skill": 0.02503293807641634,
-          "planned_cases": 55,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/diagnostic-warm.json

@@ -1,515 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d",
-  "results_sha256": "7ae5d3f7b866c5a4475cdbedbd2482ebec1a635e5b479c9916a3a0f60053ac5f",
-  "planned_requests": 500,
-  "processed_requests": 500,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v3-warm-20260924",
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 0.9439,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924",
-      "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4",
-      "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.864510400002473,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 500
-    },
-    "successful_request_latency_ms": {
-      "median": 61.95938749988272,
-      "p95": 433.2520369971462,
-      "mean": 104.73630839803081
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 11,
-        "dataset": "ContractNLI",
-        "requests": 56,
-        "answered": 56,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 56,
-        "metric": "macro-F1",
-        "score": 0.30832722516635036,
-        "reference_same_cases": null,
-        "median_ms": 411.9378854993556,
-        "detail": {
-          "field_accuracy": 0.4422268907563025,
-          "case_exact_accuracy": 0,
-          "macro_f1": 0.30832722516635036,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 30,
-        "dataset": "GSM8K",
-        "requests": 112,
-        "answered": 112,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 112,
-        "metric": "accuracy",
-        "score": 0.39285714285714285,
-        "reference_same_cases": null,
-        "median_ms": 56.87792700155114,
-        "tracks": {
-          "GSM8K-10choice": {
-            "metric": "accuracy",
-            "score": 0.35714285714285715,
-            "scored_requests": 56
-          },
-          "GSM8K-4choice": {
-            "metric": "accuracy",
-            "score": 0.42857142857142855,
-            "scored_requests": 56
-          }
-        },
-        "detail": {
-          "field_accuracy": 0.39285714285714285,
-          "case_exact_accuracy": 0.21428571428571427,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 40,
-        "dataset": "iSarcasmEval",
-        "requests": 56,
-        "answered": 56,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 56,
-        "metric": "see separate subtask tracks",
-        "score": null,
-        "reference_same_cases": null,
-        "median_ms": 63.550472499628086,
-        "tracks": {
-          "iSarcasmEval-A-Ar": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 12
-          },
-          "iSarcasmEval-A-En": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 19
-          },
-          "iSarcasmEval-B-En": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 23
-          },
-          "iSarcasmEval-C-Ar": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 1
-          },
-          "iSarcasmEval-C-En": {
-            "metric": "see separate subtask tracks",
-            "score": null,
-            "scored_requests": 1
-          }
-        },
-        "detail": {
-          "field_accuracy": 0.7192982456140351,
-          "case_exact_accuracy": 0.5892857142857143,
-          "macro_f1": 0.37563731374902903,
-          "custom_metrics": {},
-          "positive_f1_by_field": {
-            "sarcastic": 0.0,
-            "sarcasm": 0.0,
-            "irony": 0.14285714285714285,
-            "satire": 0.0,
-            "understatement": 0.0,
-            "overstatement": 0.0,
-            "rhetorical_question": 0.0
-          },
-          "category_macro_f1": 0.02040816326530612
-        }
-      },
-      {
-        "catalog_id": 41,
-        "dataset": "VAST",
-        "requests": 56,
-        "answered": 56,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 56,
-        "metric": "macro-F1",
-        "score": 0.39442285094459006,
-        "reference_same_cases": null,
-        "median_ms": 29.223100502349553,
-        "detail": {
-          "field_accuracy": 0.44642857142857145,
-          "case_exact_accuracy": 0.44642857142857145,
-          "macro_f1": 0.39442285094459006,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 42,
-        "dataset": "NLI4CT",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "macro-F1",
-        "score": 0.5083014048531289,
-        "reference_same_cases": null,
-        "median_ms": 71.3829269989219,
-        "detail": {
-          "field_accuracy": 0.6181818181818182,
-          "case_exact_accuracy": 0.6181818181818182,
-          "macro_f1": 0.5083014048531289,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 43,
-        "dataset": "CRUXEval",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "accuracy",
-        "score": 0.4,
-        "reference_same_cases": null,
-        "median_ms": 55.5151049993583,
-        "detail": {
-          "field_accuracy": 0.4,
-          "case_exact_accuracy": 0.4,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 44,
-        "dataset": "CLadder",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "accuracy",
-        "score": 0.41818181818181815,
-        "reference_same_cases": null,
-        "median_ms": 54.88006499945186,
-        "detail": {
-          "field_accuracy": 0.41818181818181815,
-          "case_exact_accuracy": 0.41818181818181815,
-          "custom_metrics": {}
-        }
-      },
-      {
-        "catalog_id": 50,
-        "dataset": "Habermas Machine",
-        "requests": 55,
-        "answered": 55,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 55,
-        "metric": "accuracy",
-        "score": 0.2545454545454545,
-        "reference_same_cases": null,
-        "median_ms": 78.45878399893991,
-        "detail": {
-          "field_accuracy": 0.2545454545454545,
-          "case_exact_accuracy": 0.2545454545454545,
-          "custom_metrics": {
-            "normalized_rank_regret": 0.3190909090909091
-          }
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "11": {
-      "catalog_id": 11,
-      "raw": 0.30832722516635036,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.30832722516635036,
-          "random": 0.30851893087083776,
-          "skill": 0.0,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"The contract contradicts the hypothesis.\"",
-            "\"The contract entails the hypothesis.\"",
-            "\"The hypothesis is neither entailed nor contradicted by the contract.\""
-          ]
-        }
-      ]
-    },
-    "30": {
-      "catalog_id": 30,
-      "raw": 0.39285714285714285,
-      "skill": 0.26190476190476186,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "GSM8K-4choice",
-          "raw": 0.42857142857142855,
-          "random": 0.25,
-          "skill": 0.23809523809523805,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        },
-        {
-          "track": "GSM8K-10choice",
-          "raw": 0.35714285714285715,
-          "random": 0.1,
-          "skill": 0.2857142857142857,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "40": {
-      "catalog_id": 40,
-      "raw": 0.25,
-      "skill": 0.25,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "iSarcasmEval-A-En",
-          "raw": 0.0,
-          "random": 0.2226862339379383,
-          "skill": 0.0,
-          "planned_cases": 19,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "no",
-            "yes"
-          ]
-        },
-        {
-          "track": "iSarcasmEval-A-Ar",
-          "raw": 0.0,
-          "random": 0.22250656045660705,
-          "skill": 0.0,
-          "planned_cases": 12,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "no",
-            "yes"
-          ]
-        },
-        {
-          "track": "iSarcasmEval-C-En",
-          "raw": 0.0,
-          "random": 0.5,
-          "skill": 0.0,
-          "planned_cases": 1,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        },
-        {
-          "track": "iSarcasmEval-C-Ar",
-          "raw": 1.0,
-          "random": 0.5,
-          "skill": 1.0,
-          "planned_cases": 1,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "41": {
-      "catalog_id": 41,
-      "raw": 0.39442285094459006,
-      "skill": 0.09164623628765685,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.39442285094459006,
-          "random": 0.3333245556439576,
-          "skill": 0.09164623628765685,
-          "planned_cases": 56,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"against\"",
-            "\"favor\"",
-            "\"neutral\""
-          ]
-        }
-      ]
-    },
-    "43": {
-      "catalog_id": 43,
-      "raw": 0.4,
-      "skill": 0.07772158637210544,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.4,
-          "random": 0.34943722943722944,
-          "skill": 0.07772158637210544,
-          "planned_cases": 55,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "44": {
-      "catalog_id": 44,
-      "raw": 0.41818181818181815,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.41818181818181815,
-          "random": 0.5,
-          "skill": 0.0,
-          "planned_cases": 55,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "50": {
-      "catalog_id": 50,
-      "raw": 0.2545454545454545,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.2545454545454545,
-          "random": 0.31,
-          "skill": 0.0,
-          "planned_cases": 55,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": []
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/environment-freeze.txt

@@ -1,82 +0,0 @@
-aiohappyeyeballs==2.7.1
-aiohttp==3.14.3
-aiosignal==1.4.0
-annotated-doc==0.0.5
-annotated-types==0.8.0
-anthropic==1.8.0
-anyio==4.15.1
-attrs==26.1.0
-certifi==2026.7.22
-charset-normalizer==3.5.1
-chess==1.11.2
-click==8.5.0
-datasets==5.0.1
-decision-index @ git+https://github.com/apolinario/decision-index.git@52a698928a9ae5bdf16b75687c903871db29c6e5
-dill==0.4.1
-docstring-parser==0.18.0
-fastapi==0.141.1
-filelock==4.0.1
-frozenlist==1.8.0
-fsspec==2026.6.0
-h11==0.16.0
-hf-xet==1.6.0
-httpcore==1.0.9
-httpcore2==2.13.0
-httpx==0.28.1
-httpx2==2.13.0
-huggingface-hub==1.32.0
-idna==3.20
--e file:///home/jach/dev/jet
-jinja2==3.1.6
-jiter==0.17.0
-markdown-it-py==4.2.0
-markupsafe==3.0.3
-mdurl==0.1.2
-mido==1.3.3
-mlx==0.32.2
-mlx-cuda-12==0.32.2
-mlx-lm==0.31.3
-multidict==6.9.1
-multiprocess==0.70.19
-numpy==2.5.3
-nvidia-cublas-cu12==12.9.2.10
-nvidia-cuda-cccl-cu12==12.9.27
-nvidia-cuda-nvrtc-cu12==12.9.86
-nvidia-cuda-runtime-cu12==12.9.79
-nvidia-cudnn-cu12==9.26.0.51
-nvidia-cufft-cu12==11.4.1.4
-nvidia-nccl-cu12==2.32.3
-nvidia-nvjitlink-cu12==12.9.86
-packaging==26.3
-pandas==3.0.6
-propcache==0.5.4
-protobuf==7.36.2
-pyarrow==25.0.1
-pydantic==2.13.5
-pydantic-core==2.46.5
-pygments==2.21.0
-python-chess==1.999
-python-dateutil==2.9.0.post0
-pyyaml==6.0.3
-regex==2026.9.10
-requests==2.34.2
-rich==15.0.0
-safetensors==0.8.0
-scipy==1.18.1
-sentencepiece==0.2.2
-shellingham==1.5.4
-six==1.17.0
-sniffio==1.3.1
-starlette==1.6.0
-tiktoken==0.14.0
-tokenizers==0.23.2
-tqdm==4.70.1
-transformers==5.17.0
-truststore==0.10.4
-typer==0.27.2
-typing-extensions==4.16.0
-typing-inspection==0.4.4
-urllib3==2.8.0
-uvicorn==0.53.0
-xxhash==4.0.1
-yarl==1.25.1

docs/training/jet-next/esci-baseline.json

@@ -1,154 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea",
-  "results_sha256": "1f2b23b88ea5ef287c7d7b3e02d9fa8aa303da49f28529924d4d8d63e57aeb6c",
-  "planned_requests": 500,
-  "processed_requests": 500,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "michaljach/jet",
-      "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c"
-    },
-    "model_source": {
-      "model": "michaljach/jet",
-      "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.0293,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts."
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 2.5189373569992313,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 500
-    },
-    "successful_request_latency_ms": {
-      "median": 47.89422899921192,
-      "p95": 59.1946660024405,
-      "mean": 42.08971909406682
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 37,
-        "dataset": "Amazon ESCI",
-        "requests": 500,
-        "answered": 500,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 500,
-        "metric": "macro-F1",
-        "score": 0.19511369134010642,
-        "reference_same_cases": null,
-        "median_ms": 47.89422899921192,
-        "detail": {
-          "field_accuracy": 0.598,
-          "case_exact_accuracy": 0.598,
-          "macro_f1": 0.19511369134010642,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "37": {
-      "catalog_id": 37,
-      "raw": 0.19511369134010642,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.19511369134010642,
-          "random": 0.20273782411694521,
-          "skill": 0.0,
-          "planned_cases": 500,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"Complement: a product that complements the requested product.\"",
-            "\"Exact: the product satisfies the search query.\"",
-            "\"Irrelevant: the product does not address the requested product need.\"",
-            "\"Substitute: a product that could substitute for the requested product.\""
-          ]
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/esci-coverage.json

@@ -1,49 +0,0 @@
-{
-  "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea",
-  "rows_uncompressed_sha256": "f2bf36ce7a136f68dbd5d81ca01a1685297da258263511939dfaf8b818fefa9a",
-  "model_revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c",
-  "scope": "Only the supplied rows; token capacity, not measured accuracy or validated long-context quality.",
-  "limits": [
-    4096,
-    6144,
-    8192,
-    16384,
-    32768,
-    40960
-  ],
-  "benchmarks": {
-    "Amazon ESCI": {
-      "requests": 500,
-      "fields": 500,
-      "options": {
-        "4": 500
-      },
-      "questions_per_request": {
-        "1": 500
-      },
-      "supported_requests": {
-        "4096": 500,
-        "6144": 500,
-        "8192": 500,
-        "16384": 500,
-        "32768": 500,
-        "40960": 500
-      },
-      "capacity_failures": {},
-      "prompt_tokens": {
-        "p50": 400.0,
-        "p95": 1007.2499999999998,
-        "max": 2402.0
-      },
-      "source_groups": 500,
-      "supported_complete_groups": {
-        "4096": 500,
-        "6144": 500,
-        "8192": 500,
-        "16384": 500,
-        "32768": 500,
-        "40960": 500
-      }
-    }
-  }
-}

docs/training/jet-next/esci-training-overlap.json

@@ -1,7 +0,0 @@
-{
-  "training_esci_ids": 948,
-  "evaluation_esci_ids": 5000,
-  "overlap": 0,
-  "source": "Original example_id identities, train versus test",
-  "suite_sha256": "db1d2d3d4dde242dc04262149153bfd596044ad8986e0ae5de8443383973e386"
-}

docs/training/jet-next/esci-v3.json

@@ -1,158 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea",
-  "results_sha256": "73fca06d9aa2dfad6a21b3c755aa2684200b67bf081c7d80e1196dc5ec483bed",
-  "planned_requests": 500,
-  "processed_requests": 500,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v3-decision-20260923"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.0,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923",
-      "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5",
-      "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.879909809002129,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 500
-    },
-    "successful_request_latency_ms": {
-      "median": 62.32302250100474,
-      "p95": 78.50551000228734,
-      "mean": 57.14209690994903
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 37,
-        "dataset": "Amazon ESCI",
-        "requests": 500,
-        "answered": 500,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 500,
-        "metric": "macro-F1",
-        "score": 0.413930360808208,
-        "reference_same_cases": null,
-        "median_ms": 62.32302250100474,
-        "detail": {
-          "field_accuracy": 0.55,
-          "case_exact_accuracy": 0.55,
-          "macro_f1": 0.413930360808208,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "37": {
-      "catalog_id": 37,
-      "raw": 0.413930360808208,
-      "skill": 0.26489722337240446,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.413930360808208,
-          "random": 0.20273782411694521,
-          "skill": 0.26489722337240446,
-          "planned_cases": 500,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"Complement: a product that complements the requested product.\"",
-            "\"Exact: the product satisfies the search query.\"",
-            "\"Irrelevant: the product does not address the requested product need.\"",
-            "\"Substitute: a product that could substitute for the requested product.\""
-          ]
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/esci-v4.json

@@ -1,158 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea",
-  "results_sha256": "d6118b06389f9283e3bb081ffd69474bb7633415c7143926971e4fa5121cbb0a",
-  "planned_requests": 500,
-  "processed_requests": 500,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v4-transfer-20260924"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.1554,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924",
-      "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad",
-      "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.8783724759996403,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 500
-    },
-    "successful_request_latency_ms": {
-      "median": 63.90887149973423,
-      "p95": 79.99267199920723,
-      "mean": 58.1968331539465
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 37,
-        "dataset": "Amazon ESCI",
-        "requests": 500,
-        "answered": 500,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 500,
-        "metric": "macro-F1",
-        "score": 0.4134568939832098,
-        "reference_same_cases": null,
-        "median_ms": 63.90887149973423,
-        "detail": {
-          "field_accuracy": 0.532,
-          "case_exact_accuracy": 0.532,
-          "macro_f1": 0.4134568939832098,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "37": {
-      "catalog_id": 37,
-      "raw": 0.4134568939832098,
-      "skill": 0.26430335746565453,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.4134568939832098,
-          "random": 0.20273782411694521,
-          "skill": 0.26430335746565453,
-          "planned_cases": 500,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"Complement: a product that complements the requested product.\"",
-            "\"Exact: the product satisfies the search query.\"",
-            "\"Irrelevant: the product does not address the requested product need.\"",
-            "\"Substitute: a product that could substitute for the requested product.\""
-          ]
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/esci-warm.json

@@ -1,158 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea",
-  "results_sha256": "360a9c1c12feb61d36783462013d09cc093b0d5d83c633ba416c2180db28997c",
-  "planned_requests": 500,
-  "processed_requests": 500,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v3-warm-20260924"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 0.9439,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924",
-      "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4",
-      "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.8837185640004463,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 500
-    },
-    "successful_request_latency_ms": {
-      "median": 62.575644500611816,
-      "p95": 80.28408699829015,
-      "mean": 57.501974687904294
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 37,
-        "dataset": "Amazon ESCI",
-        "requests": 500,
-        "answered": 500,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 500,
-        "metric": "macro-F1",
-        "score": 0.4147424168042725,
-        "reference_same_cases": null,
-        "median_ms": 62.575644500611816,
-        "detail": {
-          "field_accuracy": 0.56,
-          "case_exact_accuracy": 0.56,
-          "macro_f1": 0.4147424168042725,
-          "custom_metrics": {}
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "37": {
-      "catalog_id": 37,
-      "raw": 0.4147424168042725,
-      "skill": 0.2659157791507029,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "all",
-          "raw": 0.4147424168042725,
-          "random": 0.20273782411694521,
-          "skill": 0.2659157791507029,
-          "planned_cases": 500,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "\"Complement: a product that complements the requested product.\"",
-            "\"Exact: the product satisfies the search query.\"",
-            "\"Irrelevant: the product does not address the requested product need.\"",
-            "\"Substitute: a product that could substitute for the requested product.\""
-          ]
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/isarcasm-en-baseline.json

@@ -1,156 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d",
-  "results_sha256": "fce07a83cfbc84cab673c9a0218e6d2bd3e32bbb08bfdce2ca586e68a322e9b5",
-  "planned_requests": 1400,
-  "processed_requests": 1400,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "michaljach/jet",
-      "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c"
-    },
-    "model_source": {
-      "model": "michaljach/jet",
-      "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.0293,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts."
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 2.120845952998934,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 1400
-    },
-    "successful_request_latency_ms": {
-      "median": 11.024953000742244,
-      "p95": 40.01360600159387,
-      "mean": 12.74496764784739
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 40,
-        "dataset": "iSarcasmEval",
-        "requests": 1400,
-        "answered": 1400,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 1400,
-        "metric": "see separate subtask tracks",
-        "score": null,
-        "reference_same_cases": null,
-        "median_ms": 11.024953000742244,
-        "detail": {
-          "field_accuracy": 0.6535714285714286,
-          "case_exact_accuracy": 0.6535714285714286,
-          "macro_f1": 0.4811847326191155,
-          "custom_metrics": {},
-          "positive_f1_by_field": {
-            "sarcastic": 0.1821247892074199
-          },
-          "category_macro_f1": 0.1821247892074199
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "40": {
-      "catalog_id": 40,
-      "raw": 0.1821247892074199,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "iSarcasmEval-A-En",
-          "raw": 0.1821247892074199,
-          "random": 0.2226862339379383,
-          "skill": 0.0,
-          "planned_cases": 1400,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "no",
-            "yes"
-          ]
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}

docs/training/jet-next/isarcasm-en-v3.json

@@ -1,160 +0,0 @@
-{
-  "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.",
-  "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d",
-  "results_sha256": "f9aae1d9e7c2e7fbeb3751e8fb4b798572f22ac82b345a4377f21bdd29655571",
-  "planned_requests": 1400,
-  "processed_requests": 1400,
-  "pending_requests": 0,
-  "complete_for_supplied_rows": true,
-  "environment": {
-    "engine": "decision_index_engine:JetEngine",
-    "engine_options": {
-      "max_tokens": 8192,
-      "max_cached_tokens": 8192,
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "adapter": "adapters/jet-v3-decision-20260923"
-    },
-    "model_source": {
-      "model": "mlx-community/Qwen3-0.6B-bf16",
-      "revision": "42096995f6402fde107068cf530136fe64b604f8",
-      "backend": "mlx",
-      "max_tokens": 8192,
-      "checkpoint_context": 40960,
-      "max_cached_tokens": 8192,
-      "choice_temperature": 1.0,
-      "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba",
-      "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828",
-      "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.",
-      "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923",
-      "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5",
-      "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5"
-    },
-    "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44",
-    "mlx": "0.32.2",
-    "mlx_lm": "0.31.3",
-    "device": "Device(gpu, 0)",
-    "loaded_seconds": 1.878869845000736,
-    "frozen_corpus_sha256": null,
-    "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz",
-    "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8",
-    "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading."
-  },
-  "summary": {
-    "engine": "jet",
-    "counts": {
-      "ok": 1400
-    },
-    "successful_request_latency_ms": {
-      "median": 25.58769000097527,
-      "p95": 54.116885003168136,
-      "mean": 26.95643487361719
-    },
-    "benchmarks": [
-      {
-        "catalog_id": 40,
-        "dataset": "iSarcasmEval",
-        "requests": 1400,
-        "answered": 1400,
-        "unsupported": 0,
-        "errors": 0,
-        "abstained": 0,
-        "pending": 0,
-        "scored_requests": 1400,
-        "metric": "see separate subtask tracks",
-        "score": null,
-        "reference_same_cases": null,
-        "median_ms": 25.58769000097527,
-        "detail": {
-          "field_accuracy": 0.7371428571428571,
-          "case_exact_accuracy": 0.7371428571428571,
-          "macro_f1": 0.5162822838386456,
-          "custom_metrics": {},
-          "positive_f1_by_field": {
-            "sarcastic": 0.1894273127753304
-          },
-          "category_macro_f1": 0.1894273127753304
-        }
-      }
-    ],
-    "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage."
-  },
-  "panel_metrics_on_sample": {
-    "40": {
-      "catalog_id": 40,
-      "raw": 0.1894273127753304,
-      "skill": 0.0,
-      "coverage": 1,
-      "pending": 0,
-      "tracks": [
-        {
-          "track": "iSarcasmEval-A-En",
-          "raw": 0.1894273127753304,
-          "random": 0.2226862339379383,
-          "skill": 0.0,
-          "planned_cases": 1400,
-          "coverage": 1,
-          "pending": 0,
-          "label_universe": [
-            "no",
-            "yes"
-          ]
-        }
-      ]
-    },
-    "7": {
-      "catalog_id": 7,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "16": {
-      "catalog_id": 16,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "13": {
-      "catalog_id": 13,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "15": {
-      "catalog_id": 15,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "17": {
-      "catalog_id": 17,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    },
-    "19": {
-      "catalog_id": 19,
-      "raw": 0.0,
-      "skill": 0.0,
-      "coverage": 0.0,
-      "pending": 1.0,
-      "status": "unrun; provisional lower bound",
-      "tracks": []
-    }
-  }
-}