diff --git a/README.md b/README.md index ffc415ef6e3ba2f54b6df4d94f4f989b47a687d0..a692d57e24053a80cf409522c31a3a6a2937c013 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ tags: # Jet +The new **[Jet 4B release](https://huggingface.co/michaljach/jet-4b)** is available separately. This repository contains the Qwen3-0.6B model. + Jet is a small decision model built on **Qwen3-0.6B**. Give it a state and named, typed questions; it returns choices, scores, and probabilities without generating free-form text. Answers always follow the requested type, but decisions can still @@ -84,11 +86,11 @@ These are single-order adapter measurements. They are not a Decision Index leaderboard score. Partial public benchmark reconstructions do not cover the complete current suite. Cross-dataset text overlap inherited during training also prevents a certified decontamination claim. Performance varies by task; -see the [evaluation report](docs/training/jet-v5/summary.md) for full methodology, +see the [evaluation report](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/summary.md) for full methodology, limitations, and the separate two-order results. To reproduce training, source acquisition, audits, and evaluation, use the -[recorded protocol](docs/training/jet-v5/protocol.md) and scripts under `scripts/`. +[recorded protocol](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/protocol.md) and scripts under `scripts/`. Dataset sources, revisions, exclusion checks, configuration, and checkpoint hashes are recorded under `docs/training/`. @@ -124,3 +126,5 @@ quantization can change probabilities. - `src/data/`, `scripts/`: data builders and reproducible experiments - `src/decision_index_engine.py`, `src/decision_index_ensemble.py`: benchmark adapters - `deploy/huggingface/`: hosted demo and API deployment + +Detailed experiment files are preserved in the [training archive](https://huggingface.co/michaljach/jet/tree/training-archive-20260924/docs/training). diff --git a/TRAINING_HISTORY.md b/TRAINING_HISTORY.md index c13eccc4cb8f8b2d9a7685ec900feb78091bca38..c85a16a22c1e406b63197d59df7a962e2ee3d9b3 100644 --- a/TRAINING_HISTORY.md +++ b/TRAINING_HISTORY.md @@ -13,7 +13,7 @@ step 2,750 by validation NLL. The fused release reached about 67.7% accuracy on the 3,683-row general test. A Qwen3-1.7B comparison was evaluated but not retained. The complete original tables, earlier 1,500-row experiment, ordinal evaluation, -and Kev transfer evaluation are preserved in [initial results](docs/training/initial-results.md). +and Kev transfer evaluation are preserved in [initial results](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/initial-results.md). ## V3 — broader decision training @@ -22,7 +22,7 @@ arithmetic and code behavior. Trained on 47,313 rows for two epochs / 4,888 steps with rank-16 LoRA and learning rate `1e-4`. New-family accuracy reached 80.6%; general-test accuracy regressed to 65.2%. -[Experiment report](docs/training/jet-v3/summary.md) +[Experiment report](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v3/summary.md) ## Warm start and V4 — transfer and routing @@ -31,8 +31,8 @@ V4 reached 66.4% general-test accuracy and 93.3% on domain-held-out tool routing It improved several partial language/relevance diagnostics but regressed on code. A CPU-fused bf16/q8 release package was prepared separately. -[Training and comparisons](docs/training/jet-next/summary.md) · -[Export preparation](docs/training/jet-v4-release.md) +[Training and comparisons](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-next/summary.md) · +[Export preparation](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v4-release.md) ## V5 — wider intent and adversarial language tasks @@ -58,12 +58,14 @@ benchmark adapter. This increases inference work and is not part of standard serving. Gains and regressions on fixed public benchmark samples are recorded separately; no overall leaderboard result or win has been established. -[Results](docs/training/jet-v5/summary.md) · -[Protocol](docs/training/jet-v5/protocol.md) · -[Completion audit](docs/training/jet-v5/completion-audit.json) +[Results](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/summary.md) · +[Protocol](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/protocol.md) · +[Completion audit](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5/completion-audit.json) ## Release — 2026-09-24 The current release is the V5 R2 selected checkpoint, fused with native MLX CUDA, with a calibrated bf16 model and validated q8 ONNX export. Export and publication -receipts are recorded in [release notes](docs/training/jet-v5-release.md). +receipts are recorded in [release notes](https://huggingface.co/michaljach/jet/blob/ee7d33ecae2b8bee30891657dd57c92d34e0ce53/docs/training/jet-v5-release.md). + +Detailed experiment files are preserved in the [training archive](https://huggingface.co/michaljach/jet/tree/training-archive-20260924/docs/training). diff --git a/docs/training/initial-results.md b/docs/training/initial-results.md deleted file mode 100644 index 241d5ddc466bd41af042de77fb9b0a110ef18839..0000000000000000000000000000000000000000 --- a/docs/training/initial-results.md +++ /dev/null @@ -1,77 +0,0 @@ -# Initial training results - - -The released model is `jet` (`adapters/jet`, `models/jet`, and [michaljach/jet](https://huggingface.co/michaljach/jet) -on Hugging Face). It is Qwen3-0.6B trained on `train_v2` (public data plus varied-scale score questions, -no Claude distillation yet) on an RTX 4080: 2 epochs, 2,910 steps, best checkpoint at step 2,750 by -validation NLL (0.485), then calibrated and fused. A Qwen3-1.7B variant was trained the same way for -comparison (best at step 2,500, val NLL 0.420) and not kept: `jet` is better calibrated and about 1.8× -faster, for half a point of accuracy. The test set has 3,683 rows. `emotion` makes up two-thirds of them -and is held out of training entirely, so it measures transfer to a task the model never saw. - -| model | accuracy | NLL | Brier | ECE | trained tasks acc | held-out `emotion` acc | ms / question | -| ------------------------- | --------- | -------- | --------- | --------- | ----------------- | ---------------------- | ------------- | -| Qwen3-0.6B, untrained | 46.7% | 2.63 | 0.885 | 0.428 | 46.6% | 46.7% | 6.1 | -| **jet** (0.6B, fused) | 67.7% | 0.87 | **0.427** | **0.061** | 83.2% | 60.3% | **6.1** | -| Qwen3-1.7B, untrained | 60.5% | 5.67 | 0.741 | 0.375 | 62.1% | 59.8% | 11.1 | -| 1.7B variant (fused) | **68.2%** | **0.86** | 0.430 | 0.100 | **83.8%** | **60.9%** | 11.1 | - -Latency is batched eval time (batch size 4) on the 4080. Per source, `jet`: dbpedia 99%, civil_comments 92%, -massive 92%, banking77 89%, ag_news 88%, mnli 82%, boolq 81%, yelp 72%, emotion (held out) 60%, stsb 56%. -By type: noul 79%, choice 62%, score 61%. - -The larger base model learns the trained tasks better: test NLL drops on 8 of 9 of them (mnli 0.48 → 0.36, -massive 0.33 → 0.18, yelp 0.69 → 0.58), and on its in-distribution validation data it needs almost no -temperature correction (T 1.00–1.09, against 1.03–1.30 for `jet`). It does not transfer better. Untrained -Qwen3-1.7B already reaches 59.8% on `emotion` zero-shot and training only takes it to 60.9%, while it becomes -more overconfident there (ECE 0.095 → 0.144). The temperatures are fit on trained tasks, so they can't correct -that, and it is why the 1.7B variant's overall ECE is worse. Varied training tasks, such as the -Claude-distilled set, still look like the lever for generalization, not model size. - -On the ordinal set (`score_eval.jsonl`, 2,400 score questions from held-out splits): - -| model | exact level | within one level | Spearman | ECE | Spearman per source (stsb / yelp / amazon / sst5) | -| ------------ | ----------- | ---------------- | -------- | ----- | ------------------------------------------------- | -| `jet` | 52.1% | 91.4% | 0.83 | 0.035 | 0.90 / 0.89 / 0.79 / 0.74 | -| 1.7B variant | 53.9% | 93.5% | 0.86 | 0.071 | 0.93 / 0.91 / 0.83 / 0.79 | - -Fusing leaves the metrics unchanged (`jet` test accuracy 67.6% → 67.7%, 1.7B variant 68.0% → 68.2%) and -cuts batched eval time by about a third (`jet` 9.1 → 6.1 ms, 1.7B variant 16.5 → 11.1 ms per question). - -### Against Kev and Jev, on Kev's out-of-domain suite - -![jet next to the Kev family and Jev](docs/jet-vs-kev.png) - -[Kev](https://github.com/jaredpalmer/kev) publishes a frozen out-of-domain suite (`transfer-v4`, 656 clean rows, -11 sources) and per-source numbers for its family and for Jev. `jet-bench-kev` converts those rows to jet's format -and scores them. `jet` gets **54.9%** (Brier 0.622), up from 51.7% for untrained Qwen3-0.6B. That is below -Kev-0.8B (65.2%) and the Kev-0.5B prototype (56.1%), and far below Kev-4B/9B (≈80%) and Jev (85.7%). It holds up -on sources that look like its training mix (SciQ 90%, QNLI 76%, TweetEval 72%) and is at or below chance on -everything else. On the policy, rule and PAWS rows it mostly picks the same answer for every row (always "yes" on -authorization and PAWS, always "late but accepted" on deadline). MMLU is at chance (29%). jet has never trained on -rule-following or knowledge questions. Kev's policy and rule families are exactly the varied tasks that -distillation is meant to add. - -```sh -uv run jet-bench-kev --base-model michaljach/jet --name jet # → docs/bench/kev-transfer-v4/jet.json -uv run jet-bench-kev --name qwen3-0.6b-untrained # untrained baseline -uv sync --extra plot && uv run jet-plot-kev # → docs/jet-vs-kev.png -``` - -### Earlier run (1,500-row test split) - -Before the held-out sets were versioned: public data only, 2 epochs, 2,378 steps, best checkpoint at step -2,250. These rows come from a different split, so compare them with each other, not with the table above. - -| model | accuracy | NLL | Brier | ECE | trained tasks acc | held-out `emotion` acc | 1 question | -| ----------------------- | -------- | ---- | ----- | ----- | ----------------- | ---------------------- | ---------- | -| Qwen3-0.6B, untrained | 46.6% | 2.66 | 0.886 | 0.428 | 44.6% | 47.6% | 82 ms | -| Jet step 1,000 | 65.8% | 0.92 | 0.462 | 0.101 | 79.9% | 58.8% | 59 ms | -| Jet final (fused) | 66.7% | 0.86 | 0.441 | 0.079 | 82.7% | 58.7% | 59 ms | - -Latency is for one question on an M2 Pro. Ten questions about one ~640-token state take about 500 ms -together, because the state is encoded once. - -The second epoch helped the trained tasks (79.9% → 82.7%) but not the unseen one (58.8% → 58.7%). More -varied training tasks, such as the Claude-distilled set, are the likely lever for generalization. -Jet has not been benchmarked against the hosted Jev API yet; `jet-bench-jev` does that. diff --git a/docs/training/jet-next.md b/docs/training/jet-next.md deleted file mode 100644 index d513cd9e0ec12a85abd16c92b56d629a2a1e1d26..0000000000000000000000000000000000000000 --- a/docs/training/jet-next.md +++ /dev/null @@ -1,64 +0,0 @@ -# Qwen3-0.6B transfer experiment - -User target: best possible local result while keeping Qwen3-0.6B. No hosted API calls, publication, or submission. This is a bounded follow-up experiment, not a claim that a leaderboard score has been beaten. - -## Protocol fixed before post-training testing - -1. Warm-start released Jet's local rank-16 adapter on immutable `train_v3.jsonl`, one epoch, LR 2e-5, seed 230923, 4096 batch tokens, 1024 state tokens. Fresh AdamW state and schedule. Select by `selection_v3` NLL. This ablation isolates warm-start/lower learning rate from new data. -2. Continue its best validation checkpoint on `train_v4_tools_focus.jsonl`, one epoch, LR 1e-5, seed 230924, 3072 batch tokens, 2048 state tokens. Fresh optimizer. This mixture retains 18,000 V3 rows sampled without replacement, balanced by source, and adds all 9,000 V4 transfer rows plus 3,000 native intent-routing rows. Select by `selection_v4_tools` NLL. Capped inverse-frequency weights apply only to new stance/contract classes; validation is unweighted. -3. Fit temperatures on independent calibration files. Compare both candidates with released Jet and V3 on unchanged old test sets and the 500-request partial diagnostic. Evaluate new template/document/article-held-out tests at 2048 state tokens. No test results choose a checkpoint, epoch, or temperature. - -All runs use base revision `42096995f6402fde107068cf530136fe64b604f8`, all-layer LoRA rank 16, scale 10, dropout .05, gradient checkpointing, BF16 base, cache limit 1 GB. All trained checkpoint directories are new. Initial weights remain eligible if fine-tuning never improves validation; final optimizer/weights are retained separately in `last/`. - -The local released adapter was compared tensor by tensor after CPU BF16 fusion against the unchanged published fused model. Tensor names match, 120 tensors match exactly, 190 differ by at most 0.00048828125 (consistent with fusion rounding). This is a close numerical reconstruction, not bit-identical fused inference. See `jet-next/baseline-adapter-check.json`. - -## New data - -- [VAST](https://github.com/emilyallaway/zero-shot-stance), revision `e7c4775182b184730f350995f8260579c9e066fe`, `data/VAST/vast_train.csv`. Native development/test posts are excluded. Related articles and identical documents stay together. 3,000 training examples; 240 each validation/calibration/test. -- [ContractNLI](https://github.com/stanfordnlp/contract-nli), revision `eced6528dd3c1d14d73f9a87df8f7bdbc03126f9`, archive `resources/contract-nli.zip`, member `contract-nli/train.json`. Full contracts only, fitting 2048 state tokens. Native development/test document text is excluded; one duplicate train/test document was removed. 3,000 training, 240 validation, 238 calibration, 240 test examples. These are sampled from native training documents, not the native evaluation labels. -- Local executable Python: 3,000 training, 240 each validation/calibration/test. Eighteen complete templates: twelve train, two validation, two calibration, two test. Loops, early exits, accumulators, filtering, ordering, slicing and list mutation. Counterfactual inputs supply plausible distractors. Options are shuffled and anonymous keys reassigned; every answer is checked by execution. - -- [Schema-Guided Dialogue](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), revision `e852981ae34990f4358979625854259302feaa78`, native `train/` only. 3,000 training and 240 each validation/calibration/test, split by entire service domain. Native same-service intents plus NONE are the candidate set; complete dialogue prefixes determine the native active intent. All 3,720 annotations verified. Native intent routing is distinct from ToolRet ranking or BFCL execution. ToolRet public evaluation queries are exclusion-only. - -Builder: `scripts/build_v4_transfer.py`; audit: `scripts/audit_v4_transfer.py`; retention mixture: `scripts/prepare_v4_focus.py`. Data hashes, source hashes, source revisions, group identities, class weights and exclusions are in `data/train_v4.provenance.json`. Existing V3, validation and test files are unchanged. V4 uses 56,313 rows before retention subsampling; initial focus mixture uses 27,000; the final tools-augmented focus mixture uses 30,000. `scripts/build_v4_tools.py` and `scripts/audit_v4_tools.py` create/audit this extension, with provenance in `data/train_v4_tools.provenance.json`. - -Limitations: exact normalized matching is not semantic paraphrase detection; contracts longer than 2048 state tokens do not train; code templates share Python primitives; V3 tool tests remain small/easy and do not establish tool-ranking quality. Neither the frozen suite nor all its content is available for exclusion auditing. - -## Scoring target - -The official leaderboard snapshot at Space commit `eec4fe1b0a8f55eb150ae537715ff35cf211f47b` reports 19 panel benchmarks, with five equally weighted categories. The installed pinned reproduction kit's `index-panel.json` is an older proposal and differs from this deployed panel. Consequently, use native benchmark metrics from partial diagnostics, not its category totals or a reconstructed headline score. - -Actual deployed panel: knowledge 24/25/30/43/44/31; language 11/40/41; retrieval 36/37; tools 1/2/6; arts 20/21/22/23/50. Snapshot archived as `jet-next/leaderboard-snapshot.json`. [Official source](https://huggingface.co/spaces/multimodalart/jev-decision-index/tree/eec4fe1b0a8f55eb150ae537715ff35cf211f47b). - -The public frozen-suite dataset still returned repository-not-found at this experiment's start. Partial results cannot establish a leaderboard win. - -## Reproduction - -After the V3 prerequisites and sources are available: - -```sh -.venv/bin/python scripts/build_v4_transfer.py -.venv/bin/python scripts/audit_v4_transfer.py -.venv/bin/python scripts/prepare_v4_focus.py -.venv/bin/python scripts/build_v4_tools.py -.venv/bin/python scripts/audit_v4_tools.py -bash train_cuda.sh --init-adapter adapters/jet \ - --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train data/train_v3.jsonl --val data/selection_v3.jsonl \ - --out adapters/jet-v3-warm-20260924 --epochs 1 --lr 2e-5 \ - --eval-every 250 --val-limit 10000 --seed 230923 -# prepare_v4_focus also creates the 64-row training-only smoke file. -bash scripts/run_next_cuda.sh -``` - -Commands refuse existing dataset/checkpoint outputs. Logs are in `logs/jet-next/`; validation metrics and exact configurations are in each adapter directory. Warm-run source snapshot is preserved separately because weighted-example support was added while that already-loaded process was running. - -## Additional evaluation coverage - -Amazon ESCI was rebuilt in a separate directory with the pinned official normalizer and original test partition: 5,000 selected requests, uncompressed SHA256 `db1d2d3d4dde242dc04262149153bfd596044ad8986e0ae5de8443383973e386`. A deterministic complete-group 500-request sample is reserved for comparison after training. Original eight-benchmark artifacts remain unchanged. This remains a partial public reconstruction, not the frozen suite. `scripts/evaluate_next_extra.sh` compares the same rows across released Jet, V3, warm-start and V4. - -The initial diagnostic contains only 19 English-A sarcasm examples. All 1,400 available English-A rows were therefore selected, solely by track ID, for a separate fixed post-training evaluation. Its metric is positive-class F1 from the official static scorer, whose headline track is `iSarcasmEval-A-En`. No aggregate score is inferred from this single track. Sample provenance is in `jet-next/isarcasm-english-sample.json`. - -Evaluation context is explicit: original test/score_eval/V3 heldout data use 4096 state tokens, matching the earlier baseline inference default; all new V4 heldouts use 2048. Calibration uses 4096. Training context remains 1024 for the warm ablation and 2048 for V4. The distinction between training and inference limits is intentional and recorded. - -After the fixed 55-case CRUXEval diagnostic showed a regression, evaluation was expanded to all 570 available eligible CRUXEval requests after official exclusions, across all four models. This is a targeted regression check, not another training or checkpoint-selection round. No training settings, weights, or calibration were adjusted using that result. `scripts/evaluate_next_code.sh` records these runs separately. diff --git a/docs/training/jet-next/baseline-adapter-check.json b/docs/training/jet-next/baseline-adapter-check.json deleted file mode 100644 index 82e9ac65c618dd77af0af03623c5dba84c1630ca..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/baseline-adapter-check.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "keys_match": true, - "exact_tensors": 120, - "different_tensors": 190, - "max_abs_difference": 0.00048828125 -} \ No newline at end of file diff --git a/docs/training/jet-next/baseline-score_eval.json b/docs/training/jet-next/baseline-score_eval.json deleted file mode 100644 index b9462fe851a1d9b0463026e944bdd3e68e7ea608..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/baseline-score_eval.json +++ /dev/null @@ -1,65 +0,0 @@ -{ - "overall": { - "n": 2400, - "acc": 0.5170833333333333, - "nll": 1.0893632032843228, - "brier": 0.5287154496496189, - "ece": 0.03900318566709756 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5216666666666666, - "nll": 1.1038147784073833, - "brier": 0.33254204642705393, - "ece": 0.04424463902910554, - "mae": 0.10265430813162432, - "within1": 0.88, - "spearman": 0.9022189976505913 - }, - "yelp": { - "n": 600, - "acc": 0.6033333333333334, - "nll": 0.8791397150792182, - "brier": 0.5038029301388466, - "ece": 0.04722331101695697, - "mae": 0.12401153839202986, - "within1": 0.97, - "spearman": 0.8922031203071111 - }, - "sst5": { - "n": 600, - "acc": 0.475, - "nll": 1.198184583062927, - "brier": 0.6437840936114044, - "ece": 0.031442425797382975, - "mae": 0.1737434748756156, - "within1": 0.9066666666666666, - "spearman": 0.7429955533970433 - }, - "amazon": { - "n": 600, - "acc": 0.4683333333333333, - "nll": 1.1763137365877627, - "brier": 0.6347327284211705, - "ece": 0.09294485206405323, - "mae": 0.17271815862875275, - "within1": 0.8966666666666666, - "spearman": 0.7921592534430197 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.5170833333333333, - "nll": 1.0893632032843228, - "brier": 0.5287154496496189, - "ece": 0.03900318566709756, - "mae": 0.14328187000700565, - "within1": 0.9133333333333333, - "spearman": 0.832342527754484 - } - }, - "ms_per_question": 8.335110699583007, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/baseline-test.json b/docs/training/jet-next/baseline-test.json deleted file mode 100644 index 159f20937830f575025d70b83a8d755bb6d79385..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/baseline-test.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.6760792831930491, - "nll": 0.871394003472334, - "brier": 0.42748423587682505, - "ece": 0.06219506579315473 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04892235172964218, - "brier": 0.020441859866866866, - "ece": 0.002619326776928345 - }, - "boolq": { - "n": 145, - "acc": 0.8, - "nll": 0.4566301124234652, - "brier": 0.2894317164731869, - "ece": 0.04909670681789005 - }, - "emotion": { - "n": 2500, - "acc": 0.6036, - "nll": 1.061848826275766, - "brier": 0.5326494158453977, - "ece": 0.09432493007183075 - }, - "ag_news": { - "n": 135, - "acc": 0.8814814814814815, - "nll": 0.3042688087249796, - "brier": 0.1724398602199645, - "ece": 0.030876177770120094 - }, - "stsb": { - "n": 117, - "acc": 0.5641025641025641, - "nll": 1.133929574747437, - "brier": 0.3139340576398371, - "ece": 0.06830696112070328, - "mae": 0.09924938618105988, - "within1": 0.8803418803418803, - "spearman": 0.8588398930237986 - }, - "mnli": { - "n": 142, - "acc": 0.8169014084507042, - "nll": 0.47927640531290794, - "brier": 0.268308599548532, - "ece": 0.08220711266490774 - }, - "yelp": { - "n": 146, - "acc": 0.7191780821917808, - "nll": 0.6889766353144221, - "brier": 0.3981368921297408, - "ece": 0.08682229931223884 - }, - "banking77": { - "n": 140, - "acc": 0.8857142857142857, - "nll": 0.38170325982251335, - "brier": 0.15168387780774686, - "ece": 0.053875945082732635 - }, - "massive": { - "n": 119, - "acc": 0.9159663865546218, - "nll": 0.33522919206459373, - "brier": 0.11903663498667257, - "ece": 0.026440266551089928 - }, - "civil_comments": { - "n": 149, - "acc": 0.9194630872483222, - "nll": 0.3247184777021575, - "brier": 0.04936214179418669, - "ece": 0.07150244792835826 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6241729157476842, - "nll": 1.0613521105481916, - "brier": 0.5002384772375326, - "ece": 0.08481803025029859 - }, - "noul": { - "n": 1185, - "acc": 0.7881856540084389, - "nll": 0.48529899323149145, - "brier": 0.29418788896638004, - "ece": 0.03554945349190307 - }, - "score": { - "n": 231, - "acc": 0.6103896103896104, - "nll": 0.9877903601581737, - "brier": 0.39727776801601805, - "ece": 0.05583026837476918, - "mae": 0.10864421681236895, - "within1": 0.9307359307359307, - "spearman": 0.8929563754198246 - } - }, - "ms_per_question": 7.412202802334466, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/baseline-test_v3_new.json b/docs/training/jet-next/baseline-test_v3_new.json deleted file mode 100644 index d15c0433857cebb6fb305d76b5b76202d0c3377f..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/baseline-test_v3_new.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "overall": { - "n": 1364, - "acc": 0.4567448680351906, - "nll": 1.3997251274893088, - "brier": 0.7614422661756503, - "ece": 0.28511070977907377 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.46875, - "nll": 1.3125342436600476, - "brier": 0.7248338334245858, - "ece": 0.18823765367269515 - }, - "v3:entailment": { - "n": 160, - "acc": 0.66875, - "nll": 0.7707340998575092, - "brier": 0.445276258289386, - "ece": 0.10283493064343929 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.50625, - "nll": 1.2612553351966198, - "brier": 0.7811341016274713, - "ece": 0.33411078229546554 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.01032785861358299, - "brier": 0.00037024526307215803, - "ece": 0.010188460350036621 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.2, - "nll": 1.8659685906022787, - "brier": 0.9681135361388087, - "ece": 0.350784395635128 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.54375, - "nll": 1.0387783478654455, - "brier": 0.6880096149279901, - "ece": 0.33221421800553796 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.20625, - "nll": 2.169761631079018, - "brier": 0.9876360526115023, - "ece": 0.32759805452078583 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.25625, - "nll": 2.6546468422631735, - "brier": 1.2787821977853926, - "ece": 0.6251898461952805 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.51875, - "nll": 0.8461553640197963, - "brier": 0.615775613120263, - "ece": 0.2303109876811505 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.05768870040774345, - "brier": 0.010224533000073113, - "ece": 0.05483078479766846 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.4567448680351906, - "nll": 1.3997251274893088, - "brier": 0.7614422661756503, - "ece": 0.28511070977907377 - } - }, - "ms_per_question": 9.546318121700155, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/baseline-test_v4_new.json b/docs/training/jet-next/baseline-test_v4_new.json deleted file mode 100644 index e252e016ce8da7ba7f558c5de6b841e0b78356a0..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/baseline-test_v4_new.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "overall": { - "n": 720, - "acc": 0.3416666666666667, - "nll": 1.4211787942176064, - "brier": 0.8029851960497246, - "ece": 0.23449580069217418 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.32916666666666666, - "nll": 1.603874272511651, - "brier": 0.8979469405703165, - "ece": 0.3367636108150085 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.3, - "nll": 1.3962707759191593, - "brier": 0.7587345977402417, - "ece": 0.11225773791472118 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.3958333333333333, - "nll": 1.2633913342220089, - "brier": 0.7522740498386156, - "ece": 0.2594969267646472 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.3416666666666667, - "nll": 1.4211787942176064, - "brier": 0.8029851960497246, - "ece": 0.23449580069217418 - } - }, - "ms_per_question": 21.310045005556177, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/baseline-test_v4_tools.json b/docs/training/jet-next/baseline-test_v4_tools.json deleted file mode 100644 index 1f7c39dbaa698c2f89a4957ee16ad06585b3539a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/baseline-test_v4_tools.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "overall": { - "n": 240, - "acc": 0.3625, - "nll": 1.7029224928080415, - "brier": 0.8800043840538005, - "ece": 0.3612365548809369 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.3625, - "nll": 1.7029224928080415, - "brier": 0.8800043840538005, - "ece": 0.3612365548809369 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.3625, - "nll": 1.7029224928080415, - "brier": 0.8800043840538005, - "ece": 0.3612365548809369 - } - }, - "ms_per_question": 21.0819619958329, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/build-v4-snapshot.py b/docs/training/jet-next/build-v4-snapshot.py deleted file mode 100644 index 0acbecc3729aa311771abdc666bd0cd1694abad2..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/build-v4-snapshot.py +++ /dev/null @@ -1,169 +0,0 @@ -"""Add training-partition stance/contracts and executable multi-step Python tasks. - -Never uses evaluation labels. Existing files are read-only exclusion/retention inputs. -New holdouts are grouped by article/document or whole program template. -""" -import csv -import hashlib -import json -import random -import zipfile -from collections import Counter, defaultdict -from pathlib import Path - -from data.decision_training import ROOT, atoms, choice, digest, normalized, read, statekey, write -from format import Question, render_state, build_prompt -from transformers import AutoTokenizer - -SEED = 230924 -RAW = ROOT / 'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw' -# Distinct complete templates are assigned to each split before generating inputs. -PROGRAMS = [ - 'out = []\nfor v in x:\n if v > k:\n out.append(v * 2)\nreturn out', - 'out = 0\nfor i, v in enumerate(x):\n out += v if i % 2 == 0 else -v\nreturn out + k', - 'out = []\nfor v in x:\n if v not in out:\n out.append(v)\nreturn out[k:]', - 'out = []\nfor i in range(1, len(x)):\n out.append(x[i] - x[i-1])\nreturn out[:k]', - 'out = 1\nfor v in x:\n if v != 0:\n out *= v\nreturn out + k', - 'out = []\nfor v in x:\n if v >= k:\n out.insert(0, v)\nreturn out', - 'out = 0\nfor v in x:\n if v == k:\n break\n out += v\nreturn out', - 'out = []\nfor v in x:\n if v % 2:\n continue\n out.append(v + k)\nreturn out', - 'out = []\nfor a, b in zip(x, x[1:]):\n if a < b:\n out.append(a + b + k)\nreturn out', - 'out = []\nfor v in x:\n out.append(sum(x[:len(out)+1]))\nreturn out[k:]', - 'out = sorted(x)\nif k % 2:\n out.reverse()\nreturn out[:k]', - 'out = []\nfor v in x:\n if x.count(v) > k and v not in out:\n out.append(v)\nreturn out', - 'out = 0\nfor v in reversed(x):\n out = out * 2 + v\nreturn out - k', - 'out = []\nfor i, v in enumerate(x):\n if v > sum(x[:i]):\n out.append(i)\nreturn out[k:]', - 'out = []\nfor v in x:\n if out and out[-1] == v:\n out.pop()\n else:\n out.append(v)\nreturn out + [k]', - 'out = 0\nfor a in x:\n for b in x:\n if a + b == k:\n out += 1\nreturn out', - 'out = list(x)\nfor i in range(k):\n out = out[1:] + out[:1]\nreturn out', - 'out = []\nfor v in x:\n out.append(v)\n if v == k:\n out.append(-v)\nreturn out', -] - -def program_rows(): - safe = dict(enumerate=enumerate, range=range, len=len, zip=zip, sum=sum, - sorted=sorted, reversed=reversed, list=list) - for tid, body in enumerate(PROGRAMS): - split = 'train' if tid < 12 else 'val' if tid < 14 else 'calibration' if tid < 16 else 'test' - code = 'def f(x, k):\n' + '\n'.join(' '+line for line in body.splitlines()) - ns = {}; exec(code, {'__builtins__': {}, **safe}, ns); fn = ns['f'] - rng = random.Random(SEED + tid) - for i in range(300 if split == 'train' else 120): - x = [rng.randint(-4, 7) for _ in range(rng.randint(3, 8))]; k = rng.randint(0, 4) - result = fn(x.copy(), k) - # Counterfactual inputs/parameters produce plausible same-type distractors. - alternatives = [fn(x.copy(), k+1), fn(x[::-1], k), fn(x[1:], k), fn(x+[k], k)] - alternatives += ([result+[k], result+[k+1], result+[k+2]] if isinstance(result, list) - else [result-1, result+1, result+2]) - vals = list(dict.fromkeys([repr(result)] + [repr(v) for v in alternatives]))[:4] - options = {f'candidate_{j}': v for j,v in enumerate(vals)} - r = choice({'code':code,'arguments':{'x':x,'k':k}}, 'What does f(**arguments) return?', - options,'candidate_0','code_behavior',f'local-v4:{tid}:{i}',f'v4-template:{tid}') - # choice randomizes semantic keys again for this family. - r['family']='code_transfer'; r['source']='v4:code_transfer' - r['provenance'].update(split=split, oracle={'result':result,'template':tid}, original_split='generated') - yield r - -def main(): - data=ROOT/'data'; outputs=['train_v4.jsonl','selection_v4.jsonl','calibration_v4.jsonl','test_v4_new.jsonl'] - if any((data/p).exists() for p in outputs): raise SystemExit('Refusing to overwrite existing v4 files') - tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8') - old_train=read(data/'train_v3.jsonl') - protected=[data/p for p in ['val.jsonl','test.jsonl','score_eval.jsonl','val_v3_new.jsonl','calibration_v3_new.jsonl','test_v3_new.jsonl']] - protected += [ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl'] - held_atoms={a for p in protected for r in read(p) for a in atoms(r['state'])} - held_states={statekey(r['state']) for p in protected for r in read(p)} - seen={digest([r['state'],r['question']]) for r in old_train} - old_atoms={a for r in old_train for a in atoms(r['state'])} - sources=[]; candidates=list(program_rows()); excluded=Counter() - vast=RAW/'vast/data/VAST/vast_train.csv' - sources.append(dict(repository='https://github.com/emilyallaway/zero-shot-stance',revision='e7c4775182b184730f350995f8260579c9e066fe',file='data/VAST/vast_train.csv',sha256=hashlib.sha256(vast.read_bytes()).hexdigest(),split='train')) - # Native development/test posts are exclusion-only, including all topic variants. - blocked_posts={normalized(r['post']) for f in ['vast_dev.csv','vast_test.csv'] for r in csv.DictReader((vast.parent/f).open())} - for r in csv.DictReader(vast.open()): - if normalized(r['post']) in blocked_posts: excluded['vast_native_heldout_post']+=1; continue - row=choice({'text':r['post'],'target':r['new_topic']},'What stance does the author express towards the target?', - dict(against='opposes the target',favor='supports the target',none='no clear stance towards the target'), - ['against','favor','none'][int(r['label'])],'stance_transfer','vast-train:'+r['new_id'],'vast-article:'+r['arc_id']) - candidates.append(row) - archive=RAW/'repos/contractnli/resources/contract-nli.zip' - sources.append(dict(repository='https://github.com/stanfordnlp/contract-nli',revision='eced6528dd3c1d14d73f9a87df8f7bdbc03126f9',file='resources/contract-nli.zip!contract-nli/train.json',sha256=hashlib.sha256(archive.read_bytes()).hexdigest(),split='train')) - with zipfile.ZipFile(archive) as z: - contracts=json.loads(z.read('contract-nli/train.json')) - held_contracts={normalized(d['text']) for s in ['dev','test'] for d in json.loads(z.read(f'contract-nli/{s}.json'))['documents']} - for d in contracts['documents']: - if normalized(d['text']) in held_contracts: excluded['contract_native_heldout_text']+=1; continue - # Keep the full contract. Never assign a full-document label to a cropped context. - for lid,a in d['annotation_sets'][0]['annotations'].items(): - row=choice({'contract':d['text'],'hypothesis':contracts['labels'][lid]['hypothesis']}, - 'Does the full contract entail, contradict, or leave the hypothesis undetermined?', - dict(Entailment='entailed by the contract',Contradiction='contradicted by the contract',NotMentioned='not determined by the contract'), - a['choice'],'contract_entailment',f'contractnli-train:{d["id"]}:{lid}',f'contractnli-document:{d["id"]}') - candidates.append(row) - # Same normalized document/post stays grouped even if source IDs differ. - from data.decision_training import Components - components=Components() - for r in candidates: - group=r['provenance']['group']; s=r['state'] - text=s.get('contract',s.get('text')) - if text: components.union(group,'document:'+digest(normalized(text))) - forced_train=set() - for r in candidates: - s=r['state']; text=s.get('contract',s.get('text')) - if text and digest(normalized(text)) in old_atoms: - forced_train.add(components.find(r['provenance']['group'])) - bins=defaultdict(list) - for r in candidates: - p=r['provenance'];s=r['state'];text=s.get('contract',s.get('text')) - # Global hypotheses recur by design; only document content is an exclusion key. - if statekey(s) in held_states or (text and digest(normalized(text)) in held_atoms): - excluded['existing_evaluation_content']+=1;continue - signature=digest([s,r['question']]) - if signature in seen: excluded['duplicate']+=1;continue - seen.add(signature) - if len(tokenizer.encode(render_state(s),add_special_tokens=False))>2048: - excluded['over_2048_full_state']+=1;continue - if len(tokenizer.encode(build_prompt(tokenizer,s,Question.from_dict(r['question'])),add_special_tokens=False))>2304: - excluded['over_2304_full_prompt']+=1;continue - group=components.find(p['group']); b=int(digest([SEED,group])[:8],16)/2**32 - split=p.get('split') or ('train' if group in forced_train or b<.8 else 'val' if b<.87 else 'calibration' if b<.94 else 'test') - p.update(component=group,split=split);r['source']='v4:'+r['family'];bins[(split,r['family'])].append(r) - splits=defaultdict(list);counts={} - for (split,family),rows in sorted(bins.items()): - random.Random(SEED).shuffle(rows) - classes=defaultdict(list) - for r in rows:classes[r['gold'] if family!='code_transfer' else 'all'].append(r) - selected=[];limit=3000 if split=='train' else 240 - while classes and len(selected)=2}) - random.Random(SEED).shuffle(domains) - splits={d:'val' if i<2 else 'calibration' if i<4 else 'test' if i<6 else 'train' for i,d in enumerate(domains)} - queries_path=Path(snapshot_download('mangopy/ToolRet-Queries',repo_type='dataset',revision='b8c76ad3349ff17497b6bdb28bb5b8f61a0f6445',allow_patterns=['*/*.parquet'])) - blocked={normalized(r['query']) for p in queries_path.rglob('*.parquet') for r in pq.read_table(p).to_pylist()} - protected=[data/p for p in ['val.jsonl','test.jsonl','score_eval.jsonl','val_v3_new.jsonl','calibration_v3_new.jsonl','test_v3_new.jsonl','selection_v4.jsonl','calibration_v4.jsonl','test_v4_new.jsonl']] - held_atoms={a for p in protected for r in read(p) for a in atoms(r['state'])} - tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8') - bins=defaultdict(list);excluded=Counter();seen=set();source_hashes={p.name:hashlib.sha256(p.read_bytes()).hexdigest() for p in src.glob('*.json')} - for p in sorted(src.glob('dialogues_*.json')): - for d in json.loads(p.read_text()): - first=d['turns'][0] - if len(first['frames'])!=1:continue - service=first['frames'][0]['service'];schema=schemas[service];domain=service.split('_')[0] - if domain not in splits:continue - # Long exact query overlap excludes the entire dialogue, without inspecting answers. - if any(len(normalized(t['utterance']))>=32 and (normalized(t['utterance']) in blocked or digest(normalized(t['utterance'])) in held_atoms) for t in d['turns']): - excluded['dialogue_with_evaluation_query']+=1;continue - history=[] - for turn,t in enumerate(d['turns'][:12]): - history.append({'role':t['speaker'].lower(),'text':t['utterance']}) - if any(f['service']!=service for f in t['frames']):break - if t['speaker']!='USER' or len(t['frames'])!=1:continue - gold=t['frames'][0]['state']['active_intent'] - options={i['name']:i['description'] for i in schema['intents']} - options['NONE']='No active intent; no service action is currently requested.' - if gold not in options:continue - state={'service':schema['description'],'conversation':list(history)} - if len(tokenizer.encode(render_state(state),add_special_tokens=False))>1024: - excluded['over_capacity']+=1;continue - key=statekey(state) - if key in seen:excluded['duplicate_state']+=1;continue - seen.add(key) - r=choice(state,'Which service intent is active after the last user turn?',options,gold, - 'tool_intent_routing',f'sgd:{p.name}:{d["dialogue_id"]}:{turn}',f'sgd-domain:{domain}', - identities=[f'sgd-dialogue:{d["dialogue_id"]}']) - r['source']='v4:tool_intent_routing';r['provenance'].update(split=splits[domain],component=f'sgd-domain:{domain}',native_service=service,source_file=p.name) - bins[splits[domain]].append(r) - selected={};counts={};rng=random.Random(SEED) - for split,rows in bins.items(): - rng.shuffle(rows);classes=defaultdict(list) - for r in rows:classes[r['gold']].append(r) - out=[];limit=3000 if split=='train' else 240 - while classes and len(out)=200 for s in ['val','calibration','test']),counts - parents=dict(train=data/'train_v4_focus.jsonl',val=data/'selection_v4.jsonl',calibration=data/'calibration_v4.jsonl') - hashes={};owners={} - for split,rows in selected.items(): - for r in rows: - p=r['provenance'] - for key in [p['component'],p['origin'],*p['identities'],statekey(r['state'])]:assert owners.setdefault(key,split)==split - payload=(read(parents[split]) if split in parents else [])+rows;rng.shuffle(payload);write(files[split],payload) - hashes[files[split].name]=hashlib.sha256(files[split].read_bytes()).hexdigest() - report=dict(seed=SEED,repository='https://github.com/google-research-datasets/dstc8-schema-guided-dialogue',revision=REV,native_split='train',source_files=source_hashes, - domain_splits=splits,counts=counts,exclusions=dict(excluded),toolret_query_revision='b8c76ad3349ff17497b6bdb28bb5b8f61a0f6445',toolret_queries=len(blocked),files=hashes, - parents={p.name:hashlib.sha256(p.read_bytes()).hexdigest() for p in parents.values()},code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), - limitations=['Intent routing within one native service; not a ToolRet ranking score or BFCL multi-tool execution task.', 'Six heldout domains are entirely absent from added SGD training data; base-model pretraining coverage is unknown.', 'Exact long-query exclusion does not detect paraphrases.']) - (data/'train_v4_tools.provenance.json').write_text(json.dumps(report,indent=2));print(json.dumps(report,indent=2)) - -if __name__=='__main__':main() diff --git a/docs/training/jet-next/code/scripts/build_v4_transfer.py b/docs/training/jet-next/code/scripts/build_v4_transfer.py deleted file mode 100644 index 0acbecc3729aa311771abdc666bd0cd1694abad2..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/scripts/build_v4_transfer.py +++ /dev/null @@ -1,169 +0,0 @@ -"""Add training-partition stance/contracts and executable multi-step Python tasks. - -Never uses evaluation labels. Existing files are read-only exclusion/retention inputs. -New holdouts are grouped by article/document or whole program template. -""" -import csv -import hashlib -import json -import random -import zipfile -from collections import Counter, defaultdict -from pathlib import Path - -from data.decision_training import ROOT, atoms, choice, digest, normalized, read, statekey, write -from format import Question, render_state, build_prompt -from transformers import AutoTokenizer - -SEED = 230924 -RAW = ROOT / 'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw' -# Distinct complete templates are assigned to each split before generating inputs. -PROGRAMS = [ - 'out = []\nfor v in x:\n if v > k:\n out.append(v * 2)\nreturn out', - 'out = 0\nfor i, v in enumerate(x):\n out += v if i % 2 == 0 else -v\nreturn out + k', - 'out = []\nfor v in x:\n if v not in out:\n out.append(v)\nreturn out[k:]', - 'out = []\nfor i in range(1, len(x)):\n out.append(x[i] - x[i-1])\nreturn out[:k]', - 'out = 1\nfor v in x:\n if v != 0:\n out *= v\nreturn out + k', - 'out = []\nfor v in x:\n if v >= k:\n out.insert(0, v)\nreturn out', - 'out = 0\nfor v in x:\n if v == k:\n break\n out += v\nreturn out', - 'out = []\nfor v in x:\n if v % 2:\n continue\n out.append(v + k)\nreturn out', - 'out = []\nfor a, b in zip(x, x[1:]):\n if a < b:\n out.append(a + b + k)\nreturn out', - 'out = []\nfor v in x:\n out.append(sum(x[:len(out)+1]))\nreturn out[k:]', - 'out = sorted(x)\nif k % 2:\n out.reverse()\nreturn out[:k]', - 'out = []\nfor v in x:\n if x.count(v) > k and v not in out:\n out.append(v)\nreturn out', - 'out = 0\nfor v in reversed(x):\n out = out * 2 + v\nreturn out - k', - 'out = []\nfor i, v in enumerate(x):\n if v > sum(x[:i]):\n out.append(i)\nreturn out[k:]', - 'out = []\nfor v in x:\n if out and out[-1] == v:\n out.pop()\n else:\n out.append(v)\nreturn out + [k]', - 'out = 0\nfor a in x:\n for b in x:\n if a + b == k:\n out += 1\nreturn out', - 'out = list(x)\nfor i in range(k):\n out = out[1:] + out[:1]\nreturn out', - 'out = []\nfor v in x:\n out.append(v)\n if v == k:\n out.append(-v)\nreturn out', -] - -def program_rows(): - safe = dict(enumerate=enumerate, range=range, len=len, zip=zip, sum=sum, - sorted=sorted, reversed=reversed, list=list) - for tid, body in enumerate(PROGRAMS): - split = 'train' if tid < 12 else 'val' if tid < 14 else 'calibration' if tid < 16 else 'test' - code = 'def f(x, k):\n' + '\n'.join(' '+line for line in body.splitlines()) - ns = {}; exec(code, {'__builtins__': {}, **safe}, ns); fn = ns['f'] - rng = random.Random(SEED + tid) - for i in range(300 if split == 'train' else 120): - x = [rng.randint(-4, 7) for _ in range(rng.randint(3, 8))]; k = rng.randint(0, 4) - result = fn(x.copy(), k) - # Counterfactual inputs/parameters produce plausible same-type distractors. - alternatives = [fn(x.copy(), k+1), fn(x[::-1], k), fn(x[1:], k), fn(x+[k], k)] - alternatives += ([result+[k], result+[k+1], result+[k+2]] if isinstance(result, list) - else [result-1, result+1, result+2]) - vals = list(dict.fromkeys([repr(result)] + [repr(v) for v in alternatives]))[:4] - options = {f'candidate_{j}': v for j,v in enumerate(vals)} - r = choice({'code':code,'arguments':{'x':x,'k':k}}, 'What does f(**arguments) return?', - options,'candidate_0','code_behavior',f'local-v4:{tid}:{i}',f'v4-template:{tid}') - # choice randomizes semantic keys again for this family. - r['family']='code_transfer'; r['source']='v4:code_transfer' - r['provenance'].update(split=split, oracle={'result':result,'template':tid}, original_split='generated') - yield r - -def main(): - data=ROOT/'data'; outputs=['train_v4.jsonl','selection_v4.jsonl','calibration_v4.jsonl','test_v4_new.jsonl'] - if any((data/p).exists() for p in outputs): raise SystemExit('Refusing to overwrite existing v4 files') - tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8') - old_train=read(data/'train_v3.jsonl') - protected=[data/p for p in ['val.jsonl','test.jsonl','score_eval.jsonl','val_v3_new.jsonl','calibration_v3_new.jsonl','test_v3_new.jsonl']] - protected += [ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl'] - held_atoms={a for p in protected for r in read(p) for a in atoms(r['state'])} - held_states={statekey(r['state']) for p in protected for r in read(p)} - seen={digest([r['state'],r['question']]) for r in old_train} - old_atoms={a for r in old_train for a in atoms(r['state'])} - sources=[]; candidates=list(program_rows()); excluded=Counter() - vast=RAW/'vast/data/VAST/vast_train.csv' - sources.append(dict(repository='https://github.com/emilyallaway/zero-shot-stance',revision='e7c4775182b184730f350995f8260579c9e066fe',file='data/VAST/vast_train.csv',sha256=hashlib.sha256(vast.read_bytes()).hexdigest(),split='train')) - # Native development/test posts are exclusion-only, including all topic variants. - blocked_posts={normalized(r['post']) for f in ['vast_dev.csv','vast_test.csv'] for r in csv.DictReader((vast.parent/f).open())} - for r in csv.DictReader(vast.open()): - if normalized(r['post']) in blocked_posts: excluded['vast_native_heldout_post']+=1; continue - row=choice({'text':r['post'],'target':r['new_topic']},'What stance does the author express towards the target?', - dict(against='opposes the target',favor='supports the target',none='no clear stance towards the target'), - ['against','favor','none'][int(r['label'])],'stance_transfer','vast-train:'+r['new_id'],'vast-article:'+r['arc_id']) - candidates.append(row) - archive=RAW/'repos/contractnli/resources/contract-nli.zip' - sources.append(dict(repository='https://github.com/stanfordnlp/contract-nli',revision='eced6528dd3c1d14d73f9a87df8f7bdbc03126f9',file='resources/contract-nli.zip!contract-nli/train.json',sha256=hashlib.sha256(archive.read_bytes()).hexdigest(),split='train')) - with zipfile.ZipFile(archive) as z: - contracts=json.loads(z.read('contract-nli/train.json')) - held_contracts={normalized(d['text']) for s in ['dev','test'] for d in json.loads(z.read(f'contract-nli/{s}.json'))['documents']} - for d in contracts['documents']: - if normalized(d['text']) in held_contracts: excluded['contract_native_heldout_text']+=1; continue - # Keep the full contract. Never assign a full-document label to a cropped context. - for lid,a in d['annotation_sets'][0]['annotations'].items(): - row=choice({'contract':d['text'],'hypothesis':contracts['labels'][lid]['hypothesis']}, - 'Does the full contract entail, contradict, or leave the hypothesis undetermined?', - dict(Entailment='entailed by the contract',Contradiction='contradicted by the contract',NotMentioned='not determined by the contract'), - a['choice'],'contract_entailment',f'contractnli-train:{d["id"]}:{lid}',f'contractnli-document:{d["id"]}') - candidates.append(row) - # Same normalized document/post stays grouped even if source IDs differ. - from data.decision_training import Components - components=Components() - for r in candidates: - group=r['provenance']['group']; s=r['state'] - text=s.get('contract',s.get('text')) - if text: components.union(group,'document:'+digest(normalized(text))) - forced_train=set() - for r in candidates: - s=r['state']; text=s.get('contract',s.get('text')) - if text and digest(normalized(text)) in old_atoms: - forced_train.add(components.find(r['provenance']['group'])) - bins=defaultdict(list) - for r in candidates: - p=r['provenance'];s=r['state'];text=s.get('contract',s.get('text')) - # Global hypotheses recur by design; only document content is an exclusion key. - if statekey(s) in held_states or (text and digest(normalized(text)) in held_atoms): - excluded['existing_evaluation_content']+=1;continue - signature=digest([s,r['question']]) - if signature in seen: excluded['duplicate']+=1;continue - seen.add(signature) - if len(tokenizer.encode(render_state(s),add_special_tokens=False))>2048: - excluded['over_2048_full_state']+=1;continue - if len(tokenizer.encode(build_prompt(tokenizer,s,Question.from_dict(r['question'])),add_special_tokens=False))>2304: - excluded['over_2304_full_prompt']+=1;continue - group=components.find(p['group']); b=int(digest([SEED,group])[:8],16)/2**32 - split=p.get('split') or ('train' if group in forced_train or b<.8 else 'val' if b<.87 else 'calibration' if b<.94 else 'test') - p.update(component=group,split=split);r['source']='v4:'+r['family'];bins[(split,r['family'])].append(r) - splits=defaultdict(list);counts={} - for (split,family),rows in sorted(bins.items()): - random.Random(SEED).shuffle(rows) - classes=defaultdict(list) - for r in rows:classes[r['gold'] if family!='code_transfer' else 'all'].append(r) - selected=[];limit=3000 if split=='train' else 240 - while classes and len(selected)/dev/null; do sleep 10; done; fi -[[ -f docs/training/jet-next/isarcasm-en-v4.json ]] -for name in baseline v3 warm v4; do - if [[ "$name" == baseline ]]; then - ARGS=(--option model=michaljach/jet --option revision=8a97cfea2df622bb03f5dc9b02567e21abd2551c) - else - case "$name" in - v3) RUN=adapters/jet-v3-decision-20260923;; - warm) RUN=adapters/jet-v3-warm-20260924;; - v4) RUN=adapters/jet-v4-transfer-20260924;; - esac - ARGS=(--option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$RUN") - fi - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/cruxeval-full-cuda-next-$name" \ - --option max_tokens=8192 --option max_cached_tokens=8192 "${ARGS[@]}" > "logs/jet-next/$name-cruxeval-full.log" 2>&1 - uv run --no-sync jet-bench-index report --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/cruxeval-full-cuda-next-$name/results.jsonl" \ - --out "docs/training/jet-next/cruxeval-full-$name.json" > "logs/jet-next/$name-cruxeval-full-report.log" 2>&1 -done -printf 'Full available CRUXEval comparison completed.\n' diff --git a/docs/training/jet-next/code/scripts/evaluate_next_extra.sh b/docs/training/jet-next/code/scripts/evaluate_next_extra.sh deleted file mode 100644 index 54997dc221585f4b26462ca11f0c3f302b460092..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/scripts/evaluate_next_extra.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail -cd "$(dirname "$0")/.." -export PYTHONUNBUFFERED=1 CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 -export MLX_CUDA_SDPA_CACHE_SIZE=2048 MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" -if [[ -n "${1:-}" ]]; then while kill -0 "$1" 2>/dev/null; do sleep 10; done; fi -[[ -f docs/training/jet-next/baseline-test_v4_tools.json ]] -for dataset in test score_eval test_v3_new; do - uv run --no-sync jet-eval --base-model models/jet --data "data/$dataset.jsonl" \ - --batch-size 2 --max-state-tokens 4096 --json "docs/training/jet-next/baseline-$dataset.json" \ - > "logs/jet-next/baseline-$dataset.log" 2>&1 -done -for dataset in test score_eval test_v3_new test_v4_new test_v4_tools; do - CONTEXT=4096 - if [[ "$dataset" == test_v4_new || "$dataset" == test_v4_tools ]]; then CONTEXT=2048; fi - uv run --no-sync jet-eval --adapter adapters/jet-v3-decision-20260923 --data "data/$dataset.jsonl" \ - --batch-size 2 --max-state-tokens "$CONTEXT" --json "docs/training/jet-next/v3-$dataset.json" \ - > "logs/jet-next/v3-$dataset.log" 2>&1 -done -for name in baseline v3 warm v4; do - if [[ "$name" == baseline ]]; then - ARGS=(--option model=michaljach/jet --option revision=8a97cfea2df622bb03f5dc9b02567e21abd2551c) - else - case "$name" in - v3) RUN=adapters/jet-v3-decision-20260923;; - warm) RUN=adapters/jet-v3-warm-20260924;; - v4) RUN=adapters/jet-v4-transfer-20260924;; - esac - ARGS=(--option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$RUN") - fi - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/esci-cuda-next-$name" \ - --option max_tokens=8192 --option max_cached_tokens=8192 "${ARGS[@]}" > "logs/jet-next/$name-esci.log" 2>&1 - uv run --no-sync jet-bench-index report --rows artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/esci-cuda-next-$name/results.jsonl" \ - --out "docs/training/jet-next/esci-$name.json" > "logs/jet-next/$name-esci-report.log" 2>&1 - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/isarcasm-en-cuda-next-$name" \ - --option max_tokens=8192 --option max_cached_tokens=8192 "${ARGS[@]}" > "logs/jet-next/$name-isarcasm-en.log" 2>&1 - uv run --no-sync jet-bench-index report --rows artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/isarcasm-en-cuda-next-$name/results.jsonl" \ - --out "docs/training/jet-next/isarcasm-en-$name.json" > "logs/jet-next/$name-isarcasm-en-report.log" 2>&1 -done -printf 'Additional heldouts and ESCI diagnostics completed.\n' diff --git a/docs/training/jet-next/code/scripts/finalize_next.py b/docs/training/jet-next/code/scripts/finalize_next.py deleted file mode 100644 index a24e1dcebafd595d6e6fcc11e7a14be8bdec2524..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/scripts/finalize_next.py +++ /dev/null @@ -1,54 +0,0 @@ -"""Require completed runs, verify preserved files, and archive reproducibility metadata.""" -import hashlib -import csv -import json -import shutil -import subprocess -from pathlib import Path -ROOT=Path(__file__).resolve().parents[1];OUT=ROOT/'docs/training/jet-next' -def sha(p):return hashlib.sha256(p.read_bytes()).hexdigest() - -def main(): - for name in ['warm','v4']: - for prefix in ['diagnostic','esci','isarcasm-en','cruxeval-full']: - r=json.loads((OUT/f'{prefix}-{name}.json').read_text()) - assert r['complete_for_supplied_rows'] and r['pending_requests']==0 - assert r['summary']['counts']=={'ok':r['planned_requests']},r['summary']['counts'] - runs={} - for name,directory in [('warm','jet-v3-warm-20260924'),('v4','jet-v4-transfer-20260924')]: - p=ROOT/'adapters'/directory;c=json.loads((p/'training_config.json').read_text()) - metrics=[json.loads(l) for l in (p/'metrics.jsonl').read_text().splitlines()] - assert metrics[-1]['step']==c['total_steps'] - for filename,expected in c['data_sha256'].items():assert sha(ROOT/filename)==expected - assert sha(ROOT/c['init_adapter']/'adapters.safetensors')==c['init_adapter_sha256'] - best=min(metrics,key=lambda m:m['nll']) - runs[name]=dict(directory=str(p.relative_to(ROOT)),config=c,best=best, - weights_sha256=sha(p/'adapters.safetensors'),last_weights_sha256=sha(p/'last/adapters.safetensors'), - calibration=json.loads((p/'calibration.json').read_text())) - for f in ['training_config.json','metrics.jsonl','calibration.json']: - shutil.copy2(p/f,OUT/f'{name}-{f.replace("_", "-")}') - for name in ['protected','published']: - subprocess.run(['sha256sum','-c',str(ROOT/f'work/jet-v3/{name}.sha256')],cwd=ROOT,check=True,stdout=subprocess.DEVNULL) - current=subprocess.check_output(['git','diff','--','README.md','src/data/distill.py'],cwd=ROOT) - assert current==(ROOT/'work/jet-v3/preexisting.patch').read_bytes(),'Preexisting user changes differ' - code=['src/train.py','src/evaluate.py','src/model.py','src/inference.py','src/format.py','src/decision_index_engine.py','src/bench_decision_index.py','src/data/decision_training.py','train_cuda.sh','scripts/bench_index_cuda.sh','scripts/evaluate_next_code.sh'] - code += ['scripts/'+n for n in ['build_v4_transfer.py','audit_v4_transfer.py','prepare_v4_focus.py','build_v4_tools.py','audit_v4_tools.py','run_next_cuda.sh','evaluate_next_extra.sh','summarize_next.py','finalize_next.py']] - for relative in code: - dest=OUT/'code'/relative;dest.parent.mkdir(parents=True,exist_ok=True);shutil.copy2(ROOT/relative,dest) - hashes={p:sha(ROOT/p) for p in code+['uv.lock','pyproject.toml']} - corpus={p.name:sha(p) for p in (ROOT/'data').glob('*v4*.jsonl')} - samples=list(csv.DictReader((ROOT/'logs/jet-next/gpu.csv').open(),skipinitialspace=True)) - peak=max(int(r['memory.used [MiB]'].split()[0]) for r in samples) - hardware=subprocess.check_output(['nvidia-smi','--query-gpu=name,memory.total,driver_version','--format=csv,noheader'],text=True).strip() - manifest=dict(status='completed',base='mlx-community/Qwen3-0.6B-bf16',base_revision='42096995f6402fde107068cf530136fe64b604f8', - git_head=subprocess.check_output(['git','rev-parse','HEAD'],cwd=ROOT,text=True).strip(), - branch=subprocess.check_output(['git','branch','--show-current'],cwd=ROOT,text=True).strip(), - runs=runs,hardware=hardware,monitored_device_peak_mib=peak,monitor_scope='V4 training and subsequent evaluations, including desktop GPU use',code_sha256=hashes,data_sha256=corpus,protected_original_files='unchanged',published_model='unchanged',preexisting_user_changes='unchanged', - warm_source_snapshot_sha256=sha(OUT/'train-warm-snapshot.py'), - limitations=['No official full-suite or leaderboard score','Native metrics only; installed panel proposal differs from deployed panel','Partial diagnostic differences are descriptive, not significance tests']) - (OUT/'run-manifest.json').write_text(json.dumps(manifest,indent=2)) - destination=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs');destination.mkdir(parents=True,exist_ok=True) - shutil.copy2(OUT/'run-manifest.json',destination/'jet-next-run-manifest.json') - shutil.copy2(OUT/'comparison.json',destination/'jet-next-results.json') - print('Completed runs verified; published model, original files and prior changes preserved.') -if __name__=='__main__':main() diff --git a/docs/training/jet-next/code/scripts/prepare_v4_focus.py b/docs/training/jet-next/code/scripts/prepare_v4_focus.py deleted file mode 100644 index b4a8b217e658c8426572653467bb354d15998ba2..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/scripts/prepare_v4_focus.py +++ /dev/null @@ -1,35 +0,0 @@ -"""Create a bounded retention mixture from V4 training rows only.""" -import hashlib -import json -import random -from collections import defaultdict -from pathlib import Path -from data.decision_training import ROOT, read, write - -p=ROOT/'data/train_v4_focus.jsonl' -if p.exists(): raise SystemExit('Refusing to overwrite a training mixture') -rows=read(ROOT/'data/train_v4.jsonl'); buckets=defaultdict(list); new=[] -for r in rows: - if r['source'].startswith('v4:'):new.append(r) - else:buckets[r['source']].append(r) -rng=random.Random(230924); selected=[] -# Round robin by original source, preserving rare families without replacement. -for rows in buckets.values():rng.shuffle(rows) -while buckets and len(selected)<18000: - for key in list(buckets): - selected.append(buckets[key].pop()) - if not buckets[key]:del buckets[key] - if len(selected)==18000:break -rows=selected+new;rng.shuffle(rows);write(p,rows) -smoke=sorted([r for r in rows if r['source']=='v4:contract_entailment'], - key=lambda r:len(r['state']['contract']),reverse=True)[:32] -smoke += [r for r in rows if r['source']=='v4:stance_transfer'][:16] -smoke += [r for r in rows if r['source']=='v4:code_transfer'][:16] -smoke_path=ROOT/'work/jet-next/v4-smoke.jsonl' -smoke_path.parent.mkdir(parents=True,exist_ok=True);write(smoke_path,smoke) -report=dict(seed=230924,retained_v3=len(selected),new_v4=len(new),total=len(rows), - source_sha256=hashlib.sha256((ROOT/'data/train_v4.jsonl').read_bytes()).hexdigest(), - output_sha256=hashlib.sha256(p.read_bytes()).hexdigest(), - smoke_sha256=hashlib.sha256(smoke_path.read_bytes()).hexdigest(), - note='Without-replacement source-balanced subsample of training only; all 9000 V4 additions retained. Existing heldouts unchanged.') -(ROOT/'data/train_v4_focus.provenance.json').write_text(json.dumps(report,indent=2));print(report) diff --git a/docs/training/jet-next/code/scripts/run_next_cuda.sh b/docs/training/jet-next/code/scripts/run_next_cuda.sh deleted file mode 100644 index 7de46da6e704f44800ead126fb724a22cb13b3f4..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/scripts/run_next_cuda.sh +++ /dev/null @@ -1,62 +0,0 @@ -#!/usr/bin/env bash -# Continue after the one-epoch V3 warm-start ablation; all outputs are local. -set -euo pipefail -cd "$(dirname "$0")/.." -export PYTHONUNBUFFERED=1 CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 -export MLX_CUDA_SDPA_CACHE_SIZE=2048 MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" -WARM=adapters/jet-v3-warm-20260924 -NEXT=adapters/jet-v4-transfer-20260924 -# Waiting PID is supplied only by the live parent session, never discovered by pattern. -if [[ -n "${1:-}" ]]; then - while kill -0 "$1" 2>/dev/null; do sleep 10; done -fi -.venv/bin/python - <<'PY' -import json -from pathlib import Path -p=Path('adapters/jet-v3-warm-20260924') -config=json.loads((p/'training_config.json').read_text()) -metrics=[json.loads(x) for x in (p/'metrics.jsonl').read_text().splitlines()] -assert metrics[-1]['step']==config['total_steps'], 'Warm-start run did not finish' -PY -bash train_cuda.sh --init-adapter "$WARM" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train work/jet-next/v4-smoke.jsonl --val work/jet-next/v4-smoke.jsonl \ - --out adapters/jet-v4-transfer-smoke --epochs 1 --lr 1e-5 --max-batch-tokens 3072 \ - --max-state-tokens 2048 --use-example-weights --val-limit 64 --eval-every 10 --seed 230924 \ - > logs/jet-next/v4-smoke.log 2>&1 -bash train_cuda.sh --init-adapter "$WARM" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train data/train_v4_tools_focus.jsonl --val data/selection_v4_tools.jsonl --out "$NEXT" \ - --epochs 1 --rank 16 --lr 1e-5 --grad-checkpoint --max-batch-tokens 3072 \ - --max-state-tokens 2048 --use-example-weights --val-limit 10000 --eval-every 250 --seed 230924 \ - > logs/jet-next/v4-train.log 2>&1 -for name in warm v4; do - if [[ "$name" == warm ]]; then RUN="$WARM"; CAL=data/calibration_v3.jsonl; - else RUN="$NEXT"; CAL=data/calibration_v4_tools.jsonl; fi - CONTEXT=4096 - uv run --no-sync jet-calibrate --adapter "$RUN" --data "$CAL" --limit 10000 \ - --batch-size 2 --max-state-tokens "$CONTEXT" > "logs/jet-next/$name-calibration.log" 2>&1 - for dataset in test score_eval test_v3_new test_v4_new test_v4_tools; do - CONTEXT=4096 - if [[ "$dataset" == test_v4_new || "$dataset" == test_v4_tools ]]; then CONTEXT=2048; fi - uv run --no-sync jet-eval --adapter "$RUN" --data "data/$dataset.jsonl" --batch-size 2 \ - --max-state-tokens "$CONTEXT" --json "docs/training/jet-next/$name-$dataset.json" \ - > "logs/jet-next/$name-$dataset.log" 2>&1 - done - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/diagnostic-cuda-next-$name" \ - --option model=mlx-community/Qwen3-0.6B-bf16 \ - --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$RUN" \ - --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-next/$name-diagnostic.log" 2>&1 - uv run --no-sync jet-bench-index report --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/diagnostic-cuda-next-$name/results.jsonl" \ - --out "docs/training/jet-next/diagnostic-$name.json" > "logs/jet-next/$name-report.log" 2>&1 -done -uv run --no-sync jet-eval --base-model models/jet --data data/test_v4_new.jsonl --batch-size 2 \ - --max-state-tokens 2048 --json docs/training/jet-next/baseline-test_v4_new.json \ - > logs/jet-next/baseline-test_v4_new.log 2>&1 -uv run --no-sync jet-eval --base-model models/jet --data data/test_v4_tools.jsonl --batch-size 2 \ - --max-state-tokens 2048 --json docs/training/jet-next/baseline-test_v4_tools.json \ - > logs/jet-next/baseline-test_v4_tools.log 2>&1 -sha256sum -c work/jet-v3/protected.sha256 > logs/jet-next/protected-check.log -sha256sum -c work/jet-v3/published.sha256 > logs/jet-next/published-check.log -printf 'Next checkpoint training and evaluation completed.\n' diff --git a/docs/training/jet-next/code/scripts/summarize_next.py b/docs/training/jet-next/code/scripts/summarize_next.py deleted file mode 100644 index b41c6d8447567282f28c0911129bb6037dad20ec..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/scripts/summarize_next.py +++ /dev/null @@ -1,61 +0,0 @@ -"""Summarize completed, fixed-test comparisons without a reconstructed index.""" -import hashlib -import json -from pathlib import Path - -ROOT=Path(__file__).resolve().parents[1];OLD=ROOT/'docs/training/jet-v3';OUT=ROOT/'docs/training/jet-next' -MODELS=['baseline','v3','warm','v4'];LABELS=['Released Jet','V3','Warm start','V4 transfer'] -def load(path):return json.loads(path.read_text()) -def pct(x):return '—' if x is None else f'{100*x:.1f}' -def main(): - data={} - for dataset in ['test','score_eval','test_v3_new','test_v4_new','test_v4_tools']: - data[dataset]={m:load(OUT/f'{m}-{dataset}.json') for m in MODELS} - diagnostics={m:load((OLD if m in ['baseline','v3'] else OUT)/f'diagnostic-{m}.json') for m in MODELS} - esci={m:load(OUT/f'esci-{m}.json') for m in MODELS} - sarcasm={m:load(OUT/f'isarcasm-en-{m}.json') for m in MODELS} - code={m:load(OUT/f'cruxeval-full-{m}.json') for m in MODELS} - for group in [diagnostics,esci,sarcasm,code]: - assert len({r['rows_sha256'] for r in group.values()})==1 - assert all(r['complete_for_supplied_rows'] and r['pending_requests']==0 for r in group.values()) - results=dict(local_tests=data,diagnostics=diagnostics,esci=esci,isarcasm_english=sarcasm,cruxeval_full=code) - (OUT/'comparison.json').write_text(json.dumps(results,indent=2)) - lines=['# Jet Qwen3-0.6B follow-up results','', - 'Two more local checkpoints were trained: a lower-learning-rate warm start from released Jet, then targeted transfer training. No weights were published and no leaderboard entry was submitted.', - '', 'V4 substantially improves contract entailment, product relevance, stance, and English sarcasm over released Jet on the evaluations below. Confirmed regressions remain in CRUXEval and original-test accuracy. The warm checkpoint preserves original-test accuracy more closely. Neither candidate is an across-the-board replacement.', - '', '**These results do not establish a leaderboard win.** The official frozen suite remains unavailable. Diagnostics are partial public reconstructions; the installed kit also contains an older panel definition. Native benchmark metrics below are comparable across these four models on identical rows, but are not an overall index.', - '', '| Held-out accuracy (%) | Released Jet | V3 | Warm start | V4 transfer |', '|---|---:|---:|---:|---:|'] - names=dict(test='Original test (3,683)',score_eval='Ordinal test (2,400)',test_v3_new='V3 new families (1,364)',test_v4_new='Harder transfer tasks (720)',test_v4_tools='SGD domain-held-out routing (240)') - for ds,reports in data.items():lines.append('| '+names[ds]+' | '+' | '.join(pct(reports[m]['overall']['acc']) for m in MODELS)+' |') - lines += ['', '| Partial benchmark — native metric (%) | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|'] - for group in [diagnostics,esci]: - tables={m:{r['catalog_id']:r for r in report['summary']['benchmarks']} for m,report in group.items()} - for bid,base in tables['baseline'].items(): - lines.append('| '+base['dataset']+' — '+str(base.get('metric') or 'unavailable')+' | '+' | '.join(pct(tables[m][bid].get('score')) for m in MODELS)+' |') - english={m:next(t for t in r['panel_metrics_on_sample']['40']['tracks'] if t['track']=='iSarcasmEval-A-En')['raw'] for m,r in sarcasm.items()} - lines.append('| iSarcasmEval English A, all 1,400 cases — positive F1 | '+' | '.join(pct(english[m]) for m in MODELS)+' |') - lines.append('| CRUXEval, all '+str(code['baseline']['planned_requests'])+' available eligible cases — accuracy | '+' | '.join(pct(code[m]['summary']['benchmarks'][0]['score']) for m in MODELS)+' |') - lines += ['', 'iSarcasmEval has no aggregate native score in the pinned summary report. A missing value is not zero. The separate English-A row uses the native positive-class F1 scorer on all 1,400 available cases (the official headline track), without combining other sarcasm tracks. The 500-request eight-benchmark sample and separate 500-request ESCI sample are small; differences are descriptive, not statistical significance claims.', - '', '| V4 task family — accuracy (%) | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|'] - for ds in ['test_v4_new','test_v4_tools']: - for family in sorted(data[ds]['baseline']['by_source']): - lines.append('| '+family.removeprefix('v4:')+' | '+' | '.join(pct(data[ds][m]['by_source'][family]['acc']) for m in MODELS)+' |') - lines += ['', '| Original family — accuracy (%) | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|'] - for family in sorted(data['test']['baseline']['by_source']): - lines.append('| '+family+' | '+' | '.join(pct(data['test'][m]['by_source'][family]['acc']) for m in MODELS)+' |') - lines += ['', '| Calibration metric | Released Jet | V3 | Warm start | V4 transfer |','|---|---:|---:|---:|---:|'] - for ds in ['test','test_v3_new','test_v4_new','test_v4_tools']: - for metric in ['nll','ece']: - lines.append('| '+ds+' '+metric.upper()+' | '+' | '.join(f"{data[ds][m]['overall'][metric]:.4f}" for m in MODELS)+' |') - for metric in ['spearman','mae']: - lines.append('| Ordinal test '+metric+' | '+' | '.join(f"{data['score_eval'][m]['by_type']['score'][metric]:.4f}" for m in MODELS)+' |') - lines += ['', 'Both candidates were selected by held-out validation NLL and calibrated on separate data before these test runs. Original three test sets use the same 4,096-token state limit for all models; new V4 tests use 2,048 for all models. Benchmark adapters preserve complete prompts up to 8,192 tokens.', '', '## Training and provenance', ''] - for name,directory in [('Warm start','jet-v3-warm-20260924'),('V4 transfer','jet-v4-transfer-20260924')]: - p=ROOT/'adapters'/directory;c=load(p/'training_config.json');metrics=[json.loads(l) for l in (p/'metrics.jsonl').read_text().splitlines()] - best=min(metrics,key=lambda r:r['nll']) - lines.append(f"- {name}: `{p.relative_to(ROOT)}`; {c['train_examples']:,} rows, {c['total_steps']:,} steps, LR {c['lr']}, best validation step {best['step']}, NLL {best['nll']:.4f}, accuracy {best['acc']:.4f}. Weight SHA256 `{hashlib.sha256((p/'adapters.safetensors').read_bytes()).hexdigest()}`.") - lines += ['', 'The published model remains unchanged. V4 retains 18,000 earlier training examples and adds 3,000 examples each for full-contract entailment, stance, executable multi-step code, and native service-intent routing. Sources are pinned; original training labels were verified; related documents/articles/domains stay together. No hosted model APIs were used.', '', 'Full experimental protocol and reproduction details follow.', '', (ROOT/'docs/training/jet-next.md').read_text()] - (OUT/'summary.md').write_text('\n'.join(lines)+'\n') - destination=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs/jet-next-report.md');destination.parent.mkdir(parents=True,exist_ok=True);destination.write_text('\n'.join(lines)+'\n') - print(destination) -if __name__=='__main__':main() diff --git a/docs/training/jet-next/code/src/bench_decision_index.py b/docs/training/jet-next/code/src/bench_decision_index.py deleted file mode 100644 index 99ba55deb3e787e94486dce131b33133dc8039c9..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/bench_decision_index.py +++ /dev/null @@ -1,158 +0,0 @@ -"""Prepare, audit, and report Decision Index diagnostics without training on them.""" -from __future__ import annotations - -import argparse -from collections import Counter, defaultdict -import gzip -import hashlib -import json -from pathlib import Path - -import numpy as np -from decision_index.engines import Unsupported -from decision_index.scoring.index import score_panel -from decision_index.scoring.report import benchmark_summary, load_results -from decision_index.suite.io import Suite, atomic_json, read_jsonl, sha256_file -from decision_index.suite.sample import stratified_sample - -from decision_index_engine import load_assets, prepare_request - - -def write_rows(path, rows): - path = Path(path) - path.parent.mkdir(parents=True, exist_ok=True) - opener = gzip.open if path.suffix == ".gz" else open - with opener(path, "wt", encoding="utf-8") as f: - for row in rows: - f.write(json.dumps(row, ensure_ascii=False) + "\n") - - -def prepare(suite, out, n, allow_partial=False): - verification = suite.verify(strict=not allow_partial) - # Even partial rebuilds must use the published exclusions. - if not verification.get("exclusions_match"): - raise ValueError("Missing or mismatched official exclusions file") - out.mkdir(parents=True, exist_ok=True) - tracks = defaultdict(list) - for row in suite.rows(apply_exclusions=True): - key = row["_evaluation"]["track"] - # Deterministic, label-independent selection, independent of file order. - rank = hashlib.sha256(row["_evaluation"]["run_id"].encode()).hexdigest() - tracks[key].append((rank, row)) - tracks[key] = sorted(tracks[key], key=lambda x: x[0])[:2] - smoke = [row for key in sorted(tracks) for _, row in tracks[key]] - write_rows(out / "compatibility.jsonl.gz", smoke) - sample = stratified_sample(suite, n, out / "diagnostic.jsonl.gz") - report = {"verification": verification, "full_frozen_suite": verification["match"], - "compatibility_requests": len(smoke), "tracks": len(tracks), - "diagnostic": sample, "note": "Compatibility is two requests per available track; diagnostic sampling keeps complete source groups. Neither is a leaderboard score."} - atomic_json(out / "preparation.json", report) - return report - - -def audit(rows_path, out): - path, tokenizer, config = load_assets() - limits = [4096, 6144, 8192, 16384, 32768, config["max_position_embeddings"]] - stats = defaultdict(lambda: {"requests": 0, "fields": 0, "lengths": [], - "options": Counter(), "questions_per_request": Counter(), - "supported_requests": Counter(), "capacity_failures": Counter()}) - groups = defaultdict(lambda: {limit: True for limit in limits}) - for i, row in enumerate(read_jsonl(rows_path)): - e = row["_evaluation"] - s = stats[e["dataset"]] - s["requests"] += 1 - s["fields"] += len(row["questions"]) - s["questions_per_request"][len(row["questions"])] += 1 - for q in row["questions"].values(): - s["options"][len(q["criteria"])] += 1 - try: - prepared = prepare_request(tokenizer, row["state"], row["questions"], 10**12) - lengths = [len(p[2]) for p in prepared] - s["lengths"].extend(lengths) - maximum = max(lengths) - except Unsupported as exc: - s["capacity_failures"][str(exc)] += 1 - maximum = float("inf") - group = groups[(e["dataset"], e["group_id"])] - for limit in limits: - supported = maximum <= limit - s["supported_requests"][limit] += int(supported) - group[limit] &= supported - if (i + 1) % 500 == 0: - print(f"Audited {i + 1} requests", flush=True) - for name, s in stats.items(): - lengths = s.pop("lengths") - s["prompt_tokens"] = dict(zip(["p50", "p95", "max"], map(float, np.percentile(lengths, [50, 95, 100])))) if lengths else None - selected = [g for (dataset, _), g in groups.items() if dataset == name] - s["source_groups"] = len(selected) - s["supported_complete_groups"] = {limit: sum(g[limit] for g in selected) for limit in limits} - report = {"rows_sha256": sha256_file(rows_path), - "rows_uncompressed_sha256": sha256_file(rows_path, gunzip=Path(rows_path).suffix == ".gz"), - "model_revision": path.name, - "scope": "Only the supplied rows; token capacity, not measured accuracy or validated long-context quality.", - "limits": limits, "benchmarks": dict(stats)} - out.parent.mkdir(parents=True, exist_ok=True) - atomic_json(out, report) - return report - - -class SampleSuite: - """Read exactly the diagnostic rows, with no invented full-suite denominator.""" - def __init__(self, rows_path): - self.rows_path = rows_path - - def rows(self, apply_exclusions=False): - yield from read_jsonl(self.rows_path) - - -def report(rows_path, results_path, out): - suite = SampleSuite(rows_path) - results = load_results(results_path) - summary = benchmark_summary(suite, results, "jet") - panel = score_panel(suite, results) - planned = {r["_evaluation"]["run_id"] for r in suite.rows()} - present = planned & results.keys() - environment_path = Path(results_path).with_name("environment.json") - data = {"scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": sha256_file(rows_path), "results_sha256": sha256_file(results_path), - "planned_requests": len(planned), "processed_requests": len(present), - "pending_requests": len(planned - present), - "complete_for_supplied_rows": planned == present, - "environment": json.loads(environment_path.read_text()) if environment_path.exists() else None, - "summary": summary, "panel_metrics_on_sample": panel} - out.parent.mkdir(parents=True, exist_ok=True) - atomic_json(out, data) - return data - - -def main(): - ap = argparse.ArgumentParser(description=__doc__) - sub = ap.add_subparsers(dest="command", required=True) - p = sub.add_parser("prepare") - p.add_argument("--suite-dir", type=Path, required=True) - p.add_argument("--out", type=Path, required=True) - p.add_argument("--n", type=int, default=500) - p.add_argument("--allow-partial", action="store_true", help="Explicitly permit a nonmatching partial rebuilt corpus") - p = sub.add_parser("audit") - p.add_argument("--rows", type=Path, required=True) - p.add_argument("--out", type=Path, required=True) - p = sub.add_parser("report") - p.add_argument("--rows", type=Path, required=True) - p.add_argument("--results", type=Path, required=True) - p.add_argument("--out", type=Path, required=True) - args = ap.parse_args() - if args.command == "prepare": - if args.n < 1: - ap.error("--n must be positive") - data = prepare(Suite(args.suite_dir), args.out, args.n, args.allow_partial) - print(json.dumps(data, indent=2)) - elif args.command == "audit": - audit(args.rows, args.out) - print(f"Wrote {args.out}") - else: - report(args.rows, args.results, args.out) - print(f"Wrote {args.out}") - - -if __name__ == "__main__": - main() diff --git a/docs/training/jet-next/code/src/data/decision_training.py b/docs/training/jet-next/code/src/data/decision_training.py deleted file mode 100644 index 658ab80d47588f88d2cc0aa62496192f8e0b10b9..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/data/decision_training.py +++ /dev/null @@ -1,280 +0,0 @@ -"""Reproducible v3 data from pinned TRAIN files and locally executable generators. - -Run after scripts/rebuild_index_diagnostic.py and scripts/download_v3_sources.py: - uv run --no-sync python -m data.decision_training -No benchmark normalizer supplies training rows. Evaluation is exclusion-only. -""" -from __future__ import annotations -import ast -import csv -import hashlib -import json -import random -import re -from collections import Counter, defaultdict -from pathlib import Path - -import pyarrow.parquet as pq -from format import Question, render_state, build_prompt - -SEED = 230923 -ROOT = Path(__file__).resolve().parents[2] - -def digest(x): - return hashlib.sha256(json.dumps(x, sort_keys=True, ensure_ascii=False).encode()).hexdigest() - -def normalized(s): - return ' '.join(s.casefold().split()) - -def atoms(value): - if isinstance(value, str): - if len(normalized(value)) >= 32: - yield digest(normalized(value)) - elif isinstance(value, dict): - for v in value.values(): yield from atoms(v) - elif isinstance(value, list): - for v in value: yield from atoms(v) - -def statekey(value): - return digest(normalized(value) if isinstance(value, str) else value) - -def choice(state, instruction, options, gold, family, origin, group, identities=(), oracle=None): - rng = random.Random(SEED + int(digest([origin, family])[:12], 16)) - keys = list(options); rng.shuffle(keys) - if family in ('arithmetic', 'code_behavior'): - # Anonymous keys must not reveal the right answer (e.g. output_0). - gold_position = keys.index(gold) - options = {f'option_{i}': options[k] for i, k in enumerate(keys)} - keys = list(options) - gold = keys[gold_position] - result = dict(state=state, question=dict(type='choice', instructions=instruction, - criteria={k: options[k] for k in keys}), target=[float(k == gold) for k in keys], - source='v3:'+family, family=family, provenance=dict(origin=origin, original_split='train', - group=group, identities=list(identities)), gold=gold) - if oracle is not None: result['provenance']['oracle'] = oracle - return result - -def public_candidates(sources): - for spec in sources: - repo, file = spec['repo'], spec['file'] - if 'glaive' in repo: - yield from tool_candidates(spec) - continue - table = pq.read_table(spec['path']) - indices = list(range(len(table))); random.Random(SEED).shuffle(indices) - # Bounded deterministic subset, not a new partition of the public evaluation files. - for idx in indices[:40000]: - r = table.slice(idx, 1).to_pylist()[0] - origin = f'{repo}@{spec["revision"]}/{file}:{r.get("example_id", r.get("idx",idx))}' - if 'esci' in repo: - if r['product_locale'] != 'us': continue - product = {k: r[k] for k in ['product_title','product_description','product_bullet_point','product_brand','product_color'] if r[k]} - if len(json.dumps(product)) > 5000: continue - options = dict(Exact='exact match to the requested product', Substitute='a substitute that can satisfy the same need', Complement='an accessory or complementary product', Irrelevant='irrelevant to the search intent') - yield choice(dict(search_query=r['query'],product=product),'How relevant is the product to the search query?',options,r['esci_label'],'product_relevance',origin,f'esci-query:{r["query_id"]}',[f'esci-product:{r["product_id"]}', 'query:'+normalized(r['query'])]) - elif 'snli' in repo: - if r['label'] < 0: continue - options=dict(entailment='the premise implies the hypothesis',neutral='the premise does not determine whether the hypothesis is true',contradiction='the premise contradicts the hypothesis') - yield choice(dict(premise=r['premise'],hypothesis=r['hypothesis']),'What is the relation of the hypothesis to the premise?',options,list(options)[r['label']],'entailment',origin,'snli-premise:'+normalized(r['premise'])) - elif 'glue' in repo: - yield choice(dict(query=r['question'],document=r['sentence']),'Does this document sentence contain the answer to the query?',dict(yes='contains the answer',no='does not contain the answer'),'yes' if r['label']==0 else 'no','document_relevance',origin,'qnli-query:'+normalized(r['question']),['qnli-doc:'+normalized(r['sentence'])]) - elif 'irony' in file: - yield choice(r['text'],'Does the text express verbal irony?',dict(yes='ironic',no='not ironic'),'yes' if r['label'] else 'no','sarcasm_irony',origin,'tweet:'+normalized(r['text'])) - elif 'stance' in file: - topic={'stance_abortion':'legalization of abortion','stance_atheism':'atheism','stance_climate':'climate change is a real concern','stance_feminist':'the feminist movement','stance_hillary':'Hillary Clinton'}[file.split('/')[0]] - options=dict(none='no clear stance towards the target',against='opposes the target',favor='supports the target') - yield choice(dict(text=r['text'],target=topic),'What stance does the author express towards the target?',options,list(options)[r['label']],'stance',origin,'tweet:'+normalized(r['text'])) - # Actual author-labelled sarcasm, separate official TRAIN partition. Rephrases stay grouped. - file=ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw/repos/isarcasm/train/train.En.csv' - for idx,r in enumerate(csv.DictReader(file.open())): - if not r.get('tweet'): continue - origin=f'iSarcasmEval@dfc708b53bde1bb571abfb5692f63231c2232195/train/train.En.csv:{idx}' - group='isarcasm:'+str(idx) - yield choice(r['tweet'],'Is the author being sarcastic?',dict(yes='sarcastic',no='not sarcastic'),'yes' if int(r['sarcastic']) else 'no','sarcasm_irony',origin,group) - if r.get('rephrase'): - yield choice(r['rephrase'],'Is the author being sarcastic?',dict(yes='sarcastic',no='not sarcastic'),'no','sarcasm_irony',origin+':rephrase',group) - -def tool_candidates(spec): - records=json.load(open(spec['path'])) - parsed=[] - relevance=[] - for idx,r in enumerate(records): - # Use only the first request and its immediate labelled function call. - match=re.search(r'USER:\s*(.*?)\n+ASSISTANT:\s*(.*?)(?:<\|endoftext\|>|\n\n)',r['chat'],re.S) - if not match: continue - try: - positive=match[2].startswith('') - call=ast.literal_eval(match[2].replace('', '', 1).strip()) if positive else None - tools=[]; rem=r['system'][r['system'].index('{'):] - while rem.strip(): - item,end=json.JSONDecoder().raw_decode(rem.lstrip());tools.append(item);rem=rem.lstrip()[end:] - names=[x['name'] for x in tools] - if len(set(names)) != len(names): continue - if positive and call['name'] not in names: continue - if not positive and not re.search(r"(don't|do not) have (the )?(capability|function|ability)|only (have|allows|function)|cannot (assist|help|perform)", match[2], re.I): continue - if len(match[1]) > 1800: continue - except (ValueError,SyntaxError,KeyError,TypeError): continue - if len(tools)==1: - relevance.append((idx,match[1].strip(),tools,positive)) - if positive: parsed.append((idx,match[1].strip(),tools,call['name'])) - catalog={t['name']:t['description'] for _,_,ts,_ in parsed for t in ts} - names=sorted(catalog) - for idx,query,tools,gold in parsed: - origin=f'{spec["repo"]}@{spec["revision"]}/{spec["file"]}:{idx}' - group='tool-catalog:'+digest(sorted(t['name'] for t in tools)) - rng=random.Random(SEED+idx) - options={t['name']:t['description'] for t in tools} - # Selection uses source-labelled calls plus catalog distractors. Binary relevance - # uses the native single-tool call/refusal annotation, without added distractors. - others=[n for n in names if n not in options] - for n in rng.sample(others,min(5,len(others))): options[n]=catalog[n] - if sum(len(x) for x in options.values())>2200: continue - yield choice(query,'Select the tool that directly fulfills the user request.',options,gold,'tool_selection',origin,group) - for idx,query,tools,positive in relevance: - tool=tools[0] - origin=f'{spec["repo"]}@{spec["revision"]}/{spec["file"]}:{idx}' - group='tool-catalog:'+digest([tool['name']]) - yield choice(dict(request=query,tool=tool),'Is this tool relevant for fulfilling the user request?', - dict(yes='the tool can fulfill the request',no='the request requires a different capability'), - 'yes' if positive else 'no','tool_relevance',origin,group) - -def generators(n=10000): - rng=random.Random(SEED) - for i in range(n): - values={x:bool(rng.getrandbits(1)) for x in ['active','verified','paid','blocked','admin']} - def rule(depth): - if depth==0 or rng.random()<0.22: - name=rng.choice(list(values));return f'not {name}' if rng.random()<0.3 else name - return f'({rule(depth-1)} {rng.choice(["and","or"])} {rule(depth-1)})' - expr=rule(rng.randint(2,4)); result=bool(eval(expr,{'__builtins__':{}},values)) - origin=f'generator:boolean:{i}'; group='boolean-rule:'+expr - yield choice(dict(facts=values,allow_if=expr),'Under this exact Boolean rule, is access allowed?',dict(allow='access is allowed',deny='access is denied'),'allow' if result else 'deny','boolean_rules',origin,group,oracle=dict(expression=expr,variables=values,result=result)) - a,b,c=[rng.randint(-30,50) for _ in range(3)];d=rng.randint(1,12) - expr=rng.choice([f'({a} + {b}) * {c}',f'{a} * {b} - {c}',f'({a} - {b}) // {d}',f'{a} + {b} * {c}',f'({a} * {b}) % {d}',f'abs({a} - {b}) + {c}']) - result=eval(expr,{'__builtins__':{},'abs':abs}) - choices={result,result+1,result-1,result+rng.choice([3,5,10])} - options={f'value_{j}':str(v) for j,v in enumerate(sorted(choices))};gold=next(k for k,v in options.items() if v==str(result)) - yield choice(dict(expression=expr),'Evaluate the Python integer expression. // is floor division and % is modulo.',options,gold,'arithmetic',f'generator:arithmetic:{i}','arithmetic:'+expr,oracle=dict(expression=expr,result=result)) - k=rng.randint(-8,12);m=rng.randint(2,8);inp=[rng.randint(-10,15) for _ in range(rng.randint(3,7))] - body=rng.choice([f'return sum(v + {k} for v in x if v % {m} != 0)',f'return [v * {k} for v in x][::{m}]',f'return sorted(set(x + [{k}]))[:{m}]',f'return sum(x[-{m}:]) + {k}',f'return [v for v in x if v > {k}][::-1]',f'return len([v for v in x if v % {m} == {k%m}])']) - code='def f(x):\n '+body - ns={};exec(code,{'__builtins__':{},'sum':sum,'sorted':sorted,'set':set,'len':len},ns) - result=ns['f'](inp) - if isinstance(result,list): bad=[result[::-1],result+[k],result[1:],[]] - else: bad=[result-1,result+1,result+3,0] - outputs=list(dict.fromkeys([repr(result)]+[repr(x) for x in bad]))[:4] - if len(outputs)<2: continue - options={f'output_{j}':v for j,v in enumerate(outputs)} - yield choice(dict(code=code,input=inp),'What value does f return for the supplied input?',options,'output_0','code_behavior',f'generator:code:{i}','program:'+code,oracle=dict(result=result,executable=True)) - -class Components: - def __init__(self):self.parent={} - def find(self,x): - self.parent.setdefault(x,x) - while x!=self.parent[x]: - self.parent[x]=self.parent[self.parent[x]];x=self.parent[x] - return x - def union(self,a,b): - a,b=self.find(a),self.find(b) - if a!=b:self.parent[max(a,b)]=min(a,b) - -def read(path):return [json.loads(line) for line in path.open() if line.strip()] -def write(path,rows): - path.write_text(''.join(json.dumps(r,ensure_ascii=False)+'\n' for r in rows)) - -def main(): - sources=json.load(open(ROOT/'work/jet-v3/sources.json')) - from transformers import AutoTokenizer - tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16') - data=ROOT/'data'; protected=[data/x for x in ['val.jsonl','test.jsonl','score_eval.jsonl']] - suite=ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl' - assert suite.exists(), 'Rebuild the evaluation corpus first, for exclusion-only use.' - held=[r for p in protected+[suite] for r in read(p)] - blocked_states={statekey(r['state']) for r in held} - blocked_atoms={a for r in held for a in atoms(r['state'])} - legacy=read(data/'train_v2.jsonl') - legacy_keys={statekey(r['state']) for r in legacy} - legacy_atoms={a for r in legacy for a in atoms(r['state'])} - stats=Counter();seen=set();candidates=[] - for r in list(public_candidates(sources))+list(generators()): - if statekey(r['state']) in blocked_states or set(atoms(r['state'])) & blocked_atoms: - stats['excluded_evaluation_content']+=1;continue - # Original identity + family permits multiple tasks from one source row, kept together. - identity=(r['provenance']['origin'],r['family']) - semantic=digest([r['state'],r['question']['instructions'],r['question']['criteria']]) - if identity in seen or semantic in seen: - stats['duplicate']+=1;continue - if len(tokenizer.encode(render_state(r['state']), add_special_tokens=False)) > 1024: - stats['over_state_capacity']+=1;continue - if len(tokenizer.encode(build_prompt(tokenizer,r['state'],Question.from_dict(r['question'])), add_special_tokens=False)) > 2048: - stats['over_prompt_capacity']+=1;continue - seen.add(identity);seen.add(semantic);candidates.append(r) - components=Components() - for r in candidates: - p=r['provenance'];key=p['group'] - links=['origin:'+p['origin'],'state:'+statekey(r['state'])]+p['identities'] - if r['family'] in ('tool_selection', 'tool_relevance'): - query=r['state'] if isinstance(r['state'],str) else r['state']['request'] - links.append('tool-query:'+normalized(query)) - # Entire same tweet/premise/query/product/program components stay in one partition. - for link in links:components.union(key,link) - forced_train=set() - for r in candidates: - if statekey(r['state']) in legacy_keys or set(atoms(r['state'])) & legacy_atoms: - forced_train.add(components.find(r['provenance']['group'])) - bins=defaultdict(list) - for r in candidates: - group=components.find(r['provenance']['group']);b=int(digest([SEED,group])[:8],16)/2**32 - split='train' if group in forced_train or b<.76 else 'val' if b<.84 else 'calibration' if b<.92 else 'test' - r['provenance']['component']=group;r['provenance']['split']=split - bins[(split,r['family'])].append(r) - out=defaultdict(list);counts={} - for (split,family),rows in sorted(bins.items()): - # Equal family budget, with round-robin classes; no duplicate oversampling. - random.Random(SEED).shuffle(rows);classes=defaultdict(list) - for r in rows:classes[r['gold'] if family not in ['tool_selection','arithmetic','code_behavior'] else 'all'].append(r) - selected=[];limit=2000 if split=='train' else 160 - while classes and len(selected) max_tokens: - raise Unsupported(f"Full prompt has {len(ids)} tokens; declared limit {max_tokens}") - prepared.append((name, q, ids, label_token_ids(tokenizer, q))) - return prepared - - -def load_assets(model=MODEL_ID, revision=REVISION): - path = Path(snapshot_download(model, revision=revision, allow_patterns=[ - "config.json", "calibration.json", "tokenizer*", "special_tokens_map.json", - "added_tokens.json", "vocab.json", "merges.txt", "chat_template*", - ])) - return path, AutoTokenizer.from_pretrained(path), json.loads((path / "config.json").read_text()) - - -class JetEngine(Engine): - name = "jet" - latency = "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - - def __init__(self, model=MODEL_ID, revision=REVISION, max_tokens=DEFAULT_MAX_TOKENS, - max_cached_tokens=32768, adapter=None, **options): - super().__init__(**options) - import mlx.core as mx - from mlx_lm import load - - self.mx = mx - mx.set_cache_limit(1_000_000_000) - path, self.tokenizer, config = load_assets(model, revision) - self.max_tokens = int(max_tokens) - self.max_cached_tokens = int(max_cached_tokens) - if not 1 <= self.max_tokens <= config["max_position_embeddings"]: - raise ValueError("max_tokens must be positive and within the checkpoint context window") - if self.max_cached_tokens < 1: - raise ValueError("max_cached_tokens must be positive") - calibration = Path(adapter) / "calibration.json" if adapter else path / "calibration.json" - self.temperature = float(json.loads(calibration.read_text())["choice"]) - if not math.isfinite(self.temperature) or self.temperature <= 0: - raise ValueError("Invalid published choice temperature") - # Pin weights to the SAME resolved snapshot as the tokenizer/config. - snapshot_download(model, revision=path.name, allow_patterns=["*.safetensors", "*.safetensors.index.json"]) - self.model, _ = load(str(path), adapter_path=adapter) - self.model.eval() - self.provenance = { - "model": model, "revision": path.name, "backend": "mlx", - "max_tokens": self.max_tokens, "checkpoint_context": config["max_position_embeddings"], - "max_cached_tokens": self.max_cached_tokens, "choice_temperature": self.temperature, - "adapter_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), - "format_sha256": hashlib.sha256(Path(__file__).with_name("format.py").read_bytes()).hexdigest(), - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - } - if adapter: - self.provenance["lora_adapter"] = str(Path(adapter).resolve()) - self.provenance["lora_weights_sha256"] = hashlib.sha256( - (Path(adapter) / "adapters.safetensors").read_bytes()).hexdigest() - self.provenance["calibration_sha256"] = hashlib.sha256(calibration.read_bytes()).hexdigest() - - def __call__(self, state, questions): - from model import shared_prefix_label_logits - - prepared = prepare_request(self.tokenizer, state, questions, self.max_tokens) - if not prepared: - raise ValueError("At least one question is required") - logits = shared_prefix_label_logits( - self.model, [p[2] for p in prepared], [p[3] for p in prepared], - max_cached_tokens=self.max_cached_tokens, - ) - answers, raw = {}, {} - for i, (name, q, ids, labels) in enumerate(prepared): - z = logits[i, :len(labels)].astype(np.float64) / self.temperature - p = np.exp(z - z.max()) - p /= p.sum() - if not np.isfinite(p).all(): - raise ValueError("Non-finite model probabilities") - keys = q.keys - answers[name] = { - "type": "choice", "choice": keys[int(np.argmax(p))], - "probabilities": dict(zip(keys, map(float, p))), - "confidence": float(p.max()), - } - raw[name] = {"label_logits": logits[i, :len(labels)].tolist(), "prompt_tokens": len(ids)} - return {"model": "jet", "answers": answers, - "usage": {"input_tokens": sum(len(p[2]) for p in prepared)}}, raw - - def synchronize(self): - self.mx.synchronize() - - def runtime(self): - import platform - from importlib.metadata import version - return {"platform": platform.platform(), "mlx": version("mlx"), - "mlx_lm": version("mlx-lm"), "device": str(self.mx.default_device())} diff --git a/docs/training/jet-next/code/src/evaluate.py b/docs/training/jet-next/code/src/evaluate.py deleted file mode 100644 index 0fae3a5346f00745f77e1703a3e4806ee49ba073..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/evaluate.py +++ /dev/null @@ -1,190 +0,0 @@ -"""Evaluate and calibrate a Jet model. - - jet-eval --adapter adapters/jet --data data/test.jsonl - jet-eval --data data/test.jsonl # untrained base model, as a baseline - jet-calibrate --adapter adapters/jet --data data/val.jsonl -""" - -from __future__ import annotations - -import argparse -import json -import time -from collections import defaultdict -from pathlib import Path - -import mlx.core as mx -import numpy as np - -from format import Question -from model import DEFAULT_BASE_MODEL, DEFAULT_MAX_STATE_TOKENS, Jet - -TEMPERATURE_GRID = np.exp(np.linspace(np.log(0.25), np.log(8.0), 121)) - - -def load_data(path: Path, limit: int | None) -> list[dict]: - rows = [json.loads(line) for line in path.open()] - return rows[:limit] if limit else rows - - -def adapter_base(adapter: str | None, fallback: str) -> str: - if adapter and (cfg := Path(adapter) / "adapter_config.json").exists(): - config = json.loads(cfg.read_text()) - base = config.get("base_model", fallback) - if config.get("base_revision"): - from huggingface_hub import snapshot_download - return snapshot_download(base, revision=config["base_revision"]) - return base - return fallback - - -def collect_logits(jet: Jet, rows: list[dict], batch_size: int) -> tuple[list[np.ndarray], float]: - """Raw label logits for every row, batched by prompt length. Returns (logits, ms per question).""" - # Batch shapes vary, so MLX's freed-buffer cache otherwise grows until CUDA runs out (as in jet-train). - mx.set_cache_limit(1_000_000_000) - items = [(r["state"], Question.from_dict(r["question"])) for r in rows] - order = sorted(range(len(items)), key=lambda i: len(str(items[i][0]))) - logits: list[np.ndarray | None] = [None] * len(items) - started = time.perf_counter() - for start in range(0, len(order), batch_size): - idx = order[start : start + batch_size] - out, _ = jet.raw_logits([items[i] for i in idx]) - for i, z in zip(idx, out): - logits[i] = z - print(f"\r {min(start + batch_size, len(order))}/{len(order)}", end="", flush=True) - print() - return logits, (time.perf_counter() - started) * 1000 / max(len(items), 1) - - -def softmax(z: np.ndarray, t: float) -> np.ndarray: - z = z / t - e = np.exp(z - z.max()) - return e / e.sum() - - -def nll(logits: list[np.ndarray], targets: list[np.ndarray], t: float) -> float: - return float(np.mean([-(tg * np.log(np.clip(softmax(z, t), 1e-12, 1))).sum() for z, tg in zip(logits, targets)])) - - -def metrics(probs: list[np.ndarray], targets: list[np.ndarray], bins: int = 10) -> dict[str, float]: - top = np.array([p.max() for p in probs]) - hit = np.array([p.argmax() == t.argmax() for p, t in zip(probs, targets)], dtype=float) - # ECE over the model's top prediction, scored against the (possibly soft) target's argmax. - ece = 0.0 - for lo in np.linspace(0, 1, bins, endpoint=False): - sel = (top > lo) & (top <= lo + 1 / bins) - if sel.any(): - ece += sel.mean() * abs(top[sel].mean() - hit[sel].mean()) - return { - "n": len(probs), - "acc": float(hit.mean()), - "nll": float(np.mean([-(t * np.log(np.clip(p, 1e-12, 1))).sum() for p, t in zip(probs, targets)])), - "brier": float(np.mean([((p - t) ** 2).sum() for p, t in zip(probs, targets)])), - "ece": float(ece), - } - - -def rank(x: np.ndarray) -> np.ndarray: - """Average ranks (ties share their mean rank), for Spearman correlation.""" - order = np.argsort(x, kind="stable") - ranks = np.empty(len(x)) - ranks[order] = np.arange(len(x)) - for v in np.unique(x): - tie = x == v - ranks[tie] = ranks[tie].mean() - return ranks - - -def ordinal_metrics(probs: list[np.ndarray], targets: list[np.ndarray]) -> dict[str, float]: - """Score questions: compare expected levels, normalized to 0-1 so scales of any size mix. - - mae: mean |E[pred] - E[target]| on the 0-1 scale; within1: argmax level within one of the - target's expected level; spearman: rank correlation of expected scores. - """ - pred = np.array([np.dot(np.arange(len(p)), p) / (len(p) - 1) for p in probs]) - true = np.array([np.dot(np.arange(len(t)), t) / (len(t) - 1) for t in targets]) - within = [abs(int(p.argmax()) - np.dot(np.arange(len(t)), t)) <= 1 for p, t in zip(probs, targets)] - spearman = float(np.corrcoef(rank(pred), rank(true))[0, 1]) if len(pred) > 2 and true.std() > 0 else float("nan") - return {"mae": float(np.abs(pred - true).mean()), "within1": float(np.mean(within)), "spearman": spearman} - - -def print_table(title: str, groups: dict[str, dict[str, float]]) -> None: - print(f"\n{title}") - print(f"{'':<42}{'n':>6}{'acc':>8}{'nll':>8}{'brier':>8}{'ece':>8}{'mae':>8}{'±1':>8}{'rho':>8}") - for name, m in sorted(groups.items()): - ordinal = "".join(f"{m[k]:>8.3f}" for k in ("mae", "within1", "spearman") if k in m) - print(f"{name[:41]:<42}{m['n']:>6}{m['acc']:>8.3f}{m['nll']:>8.3f}{m['brier']:>8.3f}{m['ece']:>8.3f}{ordinal}") - - -def main_eval() -> None: - ap = argparse.ArgumentParser(description="Evaluate a Jet model.") - ap.add_argument("--adapter", default=None, help="adapter dir; omit to evaluate the base model") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--data", type=Path, default=Path("data/test.jsonl")) - ap.add_argument("--limit", type=int, default=None) - ap.add_argument("--batch-size", type=int, default=16) - ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS) - ap.add_argument("--json", type=Path, default=None, help="also write metrics as JSON") - args = ap.parse_args() - - jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens) - rows = load_data(args.data, args.limit) - logits, ms = collect_logits(jet, rows, args.batch_size) - probs = [softmax(z, jet.temperatures[r["question"]["type"]]) for z, r in zip(logits, rows)] - targets = [np.array(r["target"]) for r in rows] - - by_source, by_type = defaultdict(list), defaultdict(list) - for i, r in enumerate(rows): - source = "distill (all)" if r["source"].startswith("distill:") else r["source"] - by_source[source].append(i) - by_type[r["question"]["type"]].append(i) - - def group(ix: list[int]) -> dict[str, float]: - m = metrics([probs[i] for i in ix], [targets[i] for i in ix]) - scored = [i for i in ix if rows[i]["question"]["type"] == "score"] - if len(scored) == len(ix): - m |= ordinal_metrics([probs[i] for i in scored], [targets[i] for i in scored]) - return m - - overall = metrics(probs, targets) - print_table("by source", {k: group(v) for k, v in by_source.items()}) - print_table("by type", {k: group(v) for k, v in by_type.items()}) - print_table("overall", {"all": overall}) - print(f"\n{ms:.1f} ms/question (batched, batch size {args.batch_size}); temperatures {jet.temperatures}") - if args.json: - args.json.write_text(json.dumps({ - "overall": overall, - "by_source": {k: group(v) for k, v in by_source.items()}, - "by_type": {k: group(v) for k, v in by_type.items()}, - "ms_per_question": ms, - "max_state_tokens": args.max_state_tokens, - }, indent=2)) - - -def main_calibrate() -> None: - ap = argparse.ArgumentParser(description="Fit per-type softmax temperatures on held-out data.") - ap.add_argument("--adapter", required=True) - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--data", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--limit", type=int, default=3000) - ap.add_argument("--batch-size", type=int, default=16) - ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS) - args = ap.parse_args() - - jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens) - rows = load_data(args.data, args.limit) - logits, _ = collect_logits(jet, rows, args.batch_size) - - temperatures = {} - for qtype in ("choice", "score", "noul"): - ix = [i for i, r in enumerate(rows) if r["question"]["type"] == qtype] - if not ix: - temperatures[qtype] = 1.0 - continue - zs, ts = [logits[i] for i in ix], [np.array(rows[i]["target"]) for i in ix] - scores = [nll(zs, ts, t) for t in TEMPERATURE_GRID] - best = float(TEMPERATURE_GRID[int(np.argmin(scores))]) - temperatures[qtype] = round(best, 4) - print(f"{qtype:>6}: n={len(ix)} T={best:.3f} nll {nll(zs, ts, 1.0):.4f} -> {min(scores):.4f}") - (Path(args.adapter) / "calibration.json").write_text(json.dumps(temperatures, indent=2)) - print(f"wrote {Path(args.adapter) / 'calibration.json'}") diff --git a/docs/training/jet-next/code/src/format.py b/docs/training/jet-next/code/src/format.py deleted file mode 100644 index 16ad75d3b250cde3629d60f849bd84cb61fa246a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/format.py +++ /dev/null @@ -1,143 +0,0 @@ -"""Prompt format shared by training and inference. - -Every question is reduced to "pick one label token". The model is run once over -the prompt and we read the next-token logits restricted to the label tokens: - -- choice: one label per option (A..Z, then single-token AA, AB, ...), up to 255 options -- score: one digit per level (0..9), 2-10 levels, ordered low -> high -- noul: "no" / "yes" -""" - -from __future__ import annotations - -import itertools -import json -import string -from dataclasses import dataclass -from typing import Any, Literal - -QuestionType = Literal["choice", "score", "noul"] - -MAX_CHOICE_OPTIONS = 255 -MIN_SCORE_LEVELS, MAX_SCORE_LEVELS = 2, 10 - -SYSTEM_PROMPT = ( - "You are Jet, a decision model. Read the state and the question, then answer " - "with exactly one label from the allowed labels." -) - - -@dataclass(frozen=True) -class Question: - type: QuestionType - instructions: str - # choice: {key: description}; score: [level descriptions, low -> high]; noul: None - criteria: dict[str, str] | list[str] | None = None - - @property - def keys(self) -> list[str]: - """Answer keys in label order (option keys, level descriptions, or no/yes).""" - if self.type == "choice": - return list(self.criteria) - if self.type == "score": - return list(self.criteria) - return ["no", "yes"] - - @classmethod - def from_dict(cls, d: dict[str, Any]) -> Question: - q = cls(type=d["type"], instructions=d["instructions"], criteria=d.get("criteria")) - q.validate() - return q - - def to_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"type": self.type, "instructions": self.instructions} - if self.criteria is not None: - d["criteria"] = self.criteria - return d - - def validate(self) -> None: - if self.type not in ("choice", "score", "noul"): - raise ValueError(f"unknown question type {self.type!r}") - if not isinstance(self.instructions, str) or not self.instructions.strip(): - raise ValueError("instructions must be a non-empty string") - if self.type == "choice": - if not isinstance(self.criteria, dict) or not 2 <= len(self.criteria) <= MAX_CHOICE_OPTIONS: - raise ValueError(f"choice criteria must be an object with 2-{MAX_CHOICE_OPTIONS} options") - elif self.type == "score": - if not isinstance(self.criteria, list) or not MIN_SCORE_LEVELS <= len(self.criteria) <= MAX_SCORE_LEVELS: - raise ValueError(f"score criteria must be a list of {MIN_SCORE_LEVELS}-{MAX_SCORE_LEVELS} levels") - elif self.criteria is not None: - raise ValueError("noul questions take no criteria") - - -_CHOICE_LABELS: dict[int, list[str]] = {} - - -def choice_labels(tokenizer, n: int) -> list[str]: - """A..Z, then the two-letter labels AA, AB, ... that are single tokens for this tokenizer.""" - key = id(tokenizer) - if key not in _CHOICE_LABELS: - letters = string.ascii_uppercase - pairs = ("".join(p) for p in itertools.product(letters, repeat=2)) - candidates = list(letters) + [p for p in pairs if len(tokenizer.encode(p, add_special_tokens=False)) == 1] - if len(candidates) < MAX_CHOICE_OPTIONS: - raise ValueError(f"tokenizer only has {len(candidates)} single-token choice labels") - _CHOICE_LABELS[key] = candidates[:MAX_CHOICE_OPTIONS] - return _CHOICE_LABELS[key][:n] - - -def labels_for(tokenizer, q: Question) -> list[str]: - if q.type == "choice": - return choice_labels(tokenizer, len(q.criteria)) - if q.type == "score": - return [str(i) for i in range(len(q.criteria))] - return ["no", "yes"] - - -def render_state(state: Any) -> str: - if isinstance(state, str): - return state - if isinstance(state, list) and all(isinstance(s, str) for s in state): - return "\n\n".join(state) - return json.dumps(state, ensure_ascii=False, indent=2) - - -def render_user(state: Any, q: Question, labels: list[str]) -> str: - parts = [f"\n{render_state(state)}\n", ""] - if q.type == "choice": - opts = "\n".join(f"{lab}: {key}: {desc}" for lab, (key, desc) in zip(labels, q.criteria.items())) - parts += [ - f"Question: {q.instructions}", - f"Options:\n{opts}", - f"Answer with the label of the best option ({labels[0]}-{labels[-1]}).", - ] - elif q.type == "score": - levels = "\n".join(f"{lab}: {desc}" for lab, desc in zip(labels, q.criteria)) - parts += [ - f"Question: {q.instructions}", - f"Scale (lowest to highest):\n{levels}", - f"Answer with the level number (0-{len(labels) - 1}).", - ] - else: - parts += [f"Question: {q.instructions}", "Answer yes or no."] - return "\n".join(parts) - - -def build_prompt(tokenizer, state: Any, q: Question) -> str: - return tokenizer.apply_chat_template( - [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": render_user(state, q, labels_for(tokenizer, q))}], - add_generation_prompt=True, - tokenize=False, - enable_thinking=False, - ) - - -def _label_token_id(tokenizer, label: str) -> int: - ids = tokenizer.encode(label, add_special_tokens=False) - if len(ids) != 1: - raise ValueError(f"label {label!r} is not a single token for this tokenizer") - return ids[0] - - -def label_token_ids(tokenizer, q: Question) -> list[int]: - return [_label_token_id(tokenizer, lab) for lab in labels_for(tokenizer, q)] diff --git a/docs/training/jet-next/code/src/inference.py b/docs/training/jet-next/code/src/inference.py deleted file mode 100644 index da8630b4dfb9168d59dd22c64e1010ab892bdfbe..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/inference.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Backend-independent Jet prompt encoding and typed answers.""" -from __future__ import annotations -import math -from typing import Any -import numpy as np -from format import Question, build_prompt, render_state - -DEFAULT_MAX_STATE_TOKENS = 4096 - -def encode(tokenizer, state: Any, q: Question, max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS) -> list[int]: - """Tokenize a prompt, truncating the middle of an over-long state.""" - text = render_state(state) - ids = tokenizer.encode(text, add_special_tokens=False) - if len(ids) > max_state_tokens: - half = max_state_tokens // 2 - text = tokenizer.decode(ids[:half]) + "\n[...]\n" + tokenizer.decode(ids[-half:]) - return tokenizer.encode(build_prompt(tokenizer, text, q), add_special_tokens=False) - -def summarize(q: Question, probs: np.ndarray) -> dict[str, Any]: - """Turn a label distribution into the typed answer for the question.""" - probs = probs.astype(np.float64) - k = len(probs) - entropy = -float(np.sum(probs * np.log(np.clip(probs, 1e-12, 1.0)))) - confidence = round(1.0 - entropy / math.log(k), 4) - if q.type == "choice": - keys = q.keys - return { - "type": "choice", - "choice": keys[int(np.argmax(probs))], - "probabilities": {key: round(float(p), 4) for key, p in zip(keys, probs)}, - "confidence": confidence, - } - if q.type == "score": - return { - "type": "score", - "score": round(float(np.dot(np.arange(k), probs)), 4), - "level": q.criteria[int(np.argmax(probs))], - "probabilities": [round(float(p), 4) for p in probs], - "confidence": confidence, - } - return {"type": "noul", "probability": round(float(probs[1]), 4), "confidence": confidence} diff --git a/docs/training/jet-next/code/src/model.py b/docs/training/jet-next/code/src/model.py deleted file mode 100644 index 5e28e2317876fcd76b3edafaee434bf6740ed293..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/model.py +++ /dev/null @@ -1,151 +0,0 @@ -"""Jet inference: one forward pass per batch of questions, no generation.""" - -from __future__ import annotations - -import json -import time -from pathlib import Path -from typing import Any - -import mlx.core as mx -import numpy as np -from huggingface_hub import snapshot_download -from mlx_lm import load -from mlx_lm.models.cache import make_prompt_cache - -from format import Question, label_token_ids -from inference import DEFAULT_MAX_STATE_TOKENS, encode, summarize - -DEFAULT_BASE_MODEL = "mlx-community/Qwen3-0.6B-bf16" -NEG_INF = -1e9 - - -def label_logits(model, tokens: mx.array, lengths: mx.array, label_ids: mx.array, label_mask: mx.array) -> mx.array: - """Next-token logits after each prompt, restricted to its label tokens. - - tokens: (B, L) right-padded prompts; lengths: (B,) true lengths; - label_ids / label_mask: (B, K) label token ids padded to the widest question. - Returns (B, K) logits with padded labels set to -inf. - """ - hidden = model.model(tokens) - last = hidden[mx.arange(tokens.shape[0]), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - picked = mx.take_along_axis(logits, label_ids, axis=-1).astype(mx.float32) - return mx.where(label_mask, picked, NEG_INF) - - -def head_logits(model, hidden: mx.array) -> mx.array: - if getattr(model.args, "tie_word_embeddings", False): - return model.model.embed_tokens.as_linear(hidden) - return model.lm_head(hidden) - - -def shared_prefix_label_logits( - model, seqs: list[list[int]], label_lists: list[list[int]], max_cached_tokens: int = 32_768 -) -> np.ndarray: - """Like label_logits, but encodes the prompts' common prefix (system + state) once. - - The prefix KV cache is replicated across a chunk of question suffixes, so each - question only pays for its own few dozen tokens. Chunks are sized so that - replicated prefix tokens stay under max_cached_tokens. - """ - common = min(len(s) for s in seqs) - 1 # every suffix keeps at least one token - for i in range(common): - if any(s[i] != seqs[0][i] for s in seqs): - common = i - break - prefix_cache = make_prompt_cache(model) - if common: - model.model(mx.array(seqs[0][:common])[None], cache=prefix_cache) - prefix_state = [c.state for c in prefix_cache] if common else None - if prefix_state: - mx.eval(prefix_state) - - out = np.full((len(seqs), max(map(len, label_lists))), NEG_INF, dtype=np.float32) - chunk = max(1, max_cached_tokens // max(common, 1)) - for start in range(0, len(seqs), chunk): - suffixes = [s[common:] for s in seqs[start : start + chunk]] - b = len(suffixes) - cache = make_prompt_cache(model) - if prefix_state: - for c, (k, v) in zip(cache, prefix_state): - c.state = (mx.repeat(k, b, axis=0), mx.repeat(v, b, axis=0)) - tokens, lengths = pad_batch(suffixes) - hidden = model.model(tokens, cache=cache) - last = hidden[mx.arange(b), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - ids, mask = pad_labels(label_lists[start : start + chunk]) - picked = mx.where(mask, mx.take_along_axis(logits, ids, axis=-1).astype(mx.float32), NEG_INF) - out[start : start + b, : picked.shape[1]] = np.array(picked) - return out - - -def pad_batch(seqs: list[list[int]], pad: int = 0) -> tuple[mx.array, mx.array]: - width = max(len(s) for s in seqs) - arr = np.full((len(seqs), width), pad, dtype=np.int32) - for i, s in enumerate(seqs): - arr[i, : len(s)] = s - return mx.array(arr), mx.array([len(s) for s in seqs], dtype=mx.int32) - - -def pad_labels(label_lists: list[list[int]]) -> tuple[mx.array, mx.array]: - width = max(len(ls) for ls in label_lists) - ids = np.zeros((len(label_lists), width), dtype=np.int32) - mask = np.zeros((len(label_lists), width), dtype=bool) - for i, ls in enumerate(label_lists): - ids[i, : len(ls)] = ls - mask[i, : len(ls)] = True - return mx.array(ids), mx.array(mask) - - -class Jet: - def __init__( - self, - base_model: str = DEFAULT_BASE_MODEL, - adapter_path: str | None = None, - max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS, - ): - self.model, self.tokenizer = load(base_model, adapter_path=adapter_path) - self.model.eval() - self.max_state_tokens = max_state_tokens - self.temperatures = {"choice": 1.0, "score": 1.0, "noul": 1.0} - # Calibration lives next to the adapter, or inside a fused model dir. A Hub repo id has no dir - # of its own, so look in its local snapshot (fetching calibration.json if the repo has one). - model_dir = Path(base_model) if Path(base_model).exists() else Path(snapshot_download(base_model, allow_patterns=["calibration.json"])) - for directory in (adapter_path, model_dir): - if directory and (calib := Path(directory) / "calibration.json").exists(): - self.temperatures.update(json.loads(calib.read_text())) - break - - def raw_logits(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - """Uncalibrated label logits for (state, question) pairs, plus the number of - input tokens. With shared_state, the common state prefix is encoded once.""" - seqs = [encode(self.tokenizer, s, q, self.max_state_tokens) for s, q in items] - label_lists = [label_token_ids(self.tokenizer, q) for _, q in items] - if shared_state and len(items) > 1: - out = shared_prefix_label_logits(self.model, seqs, label_lists) - else: - tokens, lengths = pad_batch(seqs) - ids, mask = pad_labels(label_lists) - out = np.array(label_logits(self.model, tokens, lengths, ids, mask)) - return [out[i, : len(q.keys)] for i, (_, q) in enumerate(items)], sum(map(len, seqs)) - - def probabilities(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - logits, n_tokens = self.raw_logits(items, shared_state) - result = [] - for (_, q), z in zip(items, logits): - z = z / self.temperatures[q.type] - z = np.exp(z - z.max()) - result.append(z / z.sum()) - return result, n_tokens - - def decide(self, state: Any, questions: dict[str, Question]) -> dict[str, Any]: - """Answer every question about one state together (shared batch).""" - started = time.perf_counter() - items = [(state, q) for q in questions.values()] - probs, input_tokens = self.probabilities(items, shared_state=True) - return { - "answers": {name: summarize(q, p) for (name, q), p in zip(questions.items(), probs)}, - "usage": {"input_tokens": input_tokens}, - "latency_ms": round((time.perf_counter() - started) * 1000, 1), - } diff --git a/docs/training/jet-next/code/src/train.py b/docs/training/jet-next/code/src/train.py deleted file mode 100644 index 816c0765fa1a66b174e71ec7b2b95cf360664903..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/src/train.py +++ /dev/null @@ -1,304 +0,0 @@ -"""LoRA fine-tuning of a small LM into a Jet decision model (MLX, Apple Silicon). - -The loss is cross-entropy between the soft target and the model's next-token -distribution restricted to the question's label tokens, read at the last prompt -position only - exactly what inference computes. Score questions add a ranked -probability score term, so mass far from the target level costs more than mass -next to it (cross-entropy alone treats the levels as unordered). -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import random -import time -from functools import partial -from pathlib import Path - -import mlx.core as mx -import mlx.nn as nn -import mlx.optimizers as optim -import numpy as np -from mlx.utils import tree_flatten, tree_unflatten -from mlx_lm import load -from huggingface_hub import snapshot_download -from mlx_lm.tuner.trainer import grad_checkpoint -from mlx_lm.tuner.utils import linear_to_lora_layers - -from format import Question, label_token_ids -from model import DEFAULT_BASE_MODEL, encode, label_logits, pad_batch, pad_labels - - -class Example: - __slots__ = ("tokens", "labels", "target", "ordinal", "weight") - - def __init__(self, tokens: list[int], labels: list[int], target: list[float], ordinal: bool, weight: float = 1.0): - self.tokens, self.labels, self.target, self.ordinal = tokens, labels, target, ordinal - self.weight = weight - - -def load_examples(path: Path, tokenizer, max_state_tokens: int, smoothing: float, use_weights: bool = False) -> list[Example]: - out = [] - for line in path.open(): - ex = json.loads(line) - q = Question.from_dict(ex["question"]) - k = len(q.keys) - target = [(1 - smoothing) * t + smoothing / k for t in ex["target"]] - weight = float(ex.get("weight", 1.0)) if use_weights else 1.0 - if not math.isfinite(weight) or weight <= 0: - raise ValueError("training weights must be finite and positive") - out.append(Example(encode(tokenizer, ex["state"], q, max_state_tokens), label_token_ids(tokenizer, q), target, q.type == "score", weight)) - return out - - -def make_batches(examples: list[Example], max_batch_tokens: int, rng: random.Random | None) -> list[list[Example]]: - """Group similar lengths together; each batch's padded size stays under max_batch_tokens.""" - order = sorted(examples, key=lambda e: len(e.tokens)) - batches, cur = [], [] - for ex in order: - width = max(len(ex.tokens), max((len(e.tokens) for e in cur), default=0)) - if cur and width * (len(cur) + 1) > max_batch_tokens: - batches.append(cur) - cur = [] - cur.append(ex) - if cur: - batches.append(cur) - if rng: - rng.shuffle(batches) - return batches - - -def to_arrays(batch: list[Example]): - tokens, lengths = pad_batch([e.tokens for e in batch]) - ids, mask = pad_labels([e.labels for e in batch]) - target = np.zeros(ids.shape, dtype=np.float32) - for i, e in enumerate(batch): - target[i, : len(e.target)] = e.target - ordinal = mx.array([e.ordinal for e in batch]) - levels = mx.array([len(e.labels) for e in batch], dtype=mx.float32) - return tokens, lengths, ids, mask, mx.array(target), ordinal, levels, mx.array([e.weight for e in batch]) - - -def ranked_probability_score(logp: mx.array, target: mx.array, levels: mx.array) -> mx.array: - """Squared distance between predicted and target CDFs, divided by (levels - 1) so it stays - in 0-1 for any scale size. Padded labels have zero mass in both, so they add nothing.""" - cdf_gap = mx.cumsum(mx.exp(logp), axis=-1) - mx.cumsum(target, axis=-1) - return (cdf_gap**2).sum(axis=-1) / (levels - 1) - - -def loss_fn(model, tokens, lengths, ids, mask, target, ordinal, levels, weights, ordinal_weight=0.0): - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - loss = -(target * logp).sum(axis=-1) - if ordinal_weight: - loss = loss + ordinal_weight * mx.where(ordinal, ranked_probability_score(logp, target, levels), 0.0) - return (loss * weights).mean() - - -def evaluate(model, batches: list[list[Example]]) -> dict[str, float]: - total, nll, correct = 0, 0.0, 0 - for batch in batches: - tokens, lengths, ids, mask, target, *_ = to_arrays(batch) - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - nll += float(-(target * logp).sum()) - correct += int((mx.argmax(logits, axis=-1) == mx.argmax(target, axis=-1)).sum()) - total += len(batch) - return {"nll": nll / total, "acc": correct / total} - - -def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | None = None) -> None: - out.mkdir(parents=True, exist_ok=True) - mx.save_safetensors(str(out / "adapters.safetensors"), dict(tree_flatten(model.trainable_parameters()))) - (out / "adapter_config.json").write_text(json.dumps(config, indent=2)) - if optimizer is not None: - # Adam moments + step, so --resume continues exactly instead of restarting cold. - mx.save_safetensors(str(out / "optimizer.safetensors"), dict(tree_flatten(optimizer.state))) - - -def main() -> None: - ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit") - ap.add_argument("--train", type=Path, default=Path("data/train.jsonl")) - ap.add_argument("--val", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--out", type=Path, default=Path("adapters/jet")) - ap.add_argument("--epochs", type=float, default=2.0) - ap.add_argument("--lr", type=float, default=1e-4) - ap.add_argument("--warmup", type=int, default=100) - ap.add_argument("--rank", type=int, default=16) - ap.add_argument("--lora-scale", type=float, default=10.0) - ap.add_argument("--dropout", type=float, default=0.05) - ap.add_argument("--lora-layers", type=int, default=-1, help="-1 = all layers") - ap.add_argument("--max-batch-tokens", type=int, default=4096) - ap.add_argument("--max-state-tokens", type=int, default=1024) - ap.add_argument("--smoothing", type=float, default=0.02) - ap.add_argument("--use-example-weights", action="store_true", - help="use positive per-row weights for training only; validation remains unweighted") - ap.add_argument("--ordinal-weight", type=float, default=2.0, - help="weight of the ranked probability score term on score questions; 0 = plain cross-entropy") - ap.add_argument("--grad-checkpoint", action=argparse.BooleanOptionalAction, default=True) - ap.add_argument("--eval-every", type=int, default=250) - ap.add_argument("--val-limit", type=int, default=1000) - ap.add_argument("--cache-limit-gb", type=float, default=1.0, - help="cap on MLX's freed-buffer cache; variable batch shapes otherwise grow it toward all of RAM") - ap.add_argument("--resume", type=Path, default=None, help="adapter dir to continue from") - ap.add_argument("--init-adapter", type=Path, default=None, - help="warm-start adapter weights with a fresh optimizer and schedule") - ap.add_argument("--start-step", type=int, default=None, - help="step the resumed adapter was saved at; read from its optimizer state when present") - ap.add_argument("--seed", type=int, default=0) - args = ap.parse_args() - if args.init_adapter and (args.resume or args.start_step is not None): - ap.error("--init-adapter cannot be combined with --resume or --start-step") - if (args.out / "adapters.safetensors").exists(): - ap.error("output already contains an adapter; choose a new directory") - - args.out.mkdir(parents=True, exist_ok=True) - run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()} - run_config["data_sha256"] = { - str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val) - } - if args.init_adapter: - source_config = json.loads((args.init_adapter / "adapter_config.json").read_text()) - expected = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - if source_config.get("lora_parameters") != expected: - ap.error("initial adapter LoRA parameters do not match requested parameters") - if source_config.get("base_model") != args.base_model: - ap.error("initial adapter uses a different base model") - if source_config.get("base_revision") not in (None, args.base_revision): - ap.error("initial adapter uses a different base revision") - run_config["init_adapter_sha256"] = hashlib.sha256( - (args.init_adapter / "adapters.safetensors").read_bytes()).hexdigest() - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - mx.set_cache_limit(int(args.cache_limit_gb * 1e9)) - mx.random.seed(args.seed) - rng = random.Random(args.seed) - model_path = args.base_model - if args.base_revision: - model_path = snapshot_download(args.base_model, revision=args.base_revision) - model, tokenizer = load(model_path) - model.freeze() - num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers - lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - linear_to_lora_layers(model, num_layers, lora_parameters) - if args.init_adapter: - if source_config.get("num_layers") != num_layers: - ap.error("initial adapter layer count does not match") - weights = mx.load(str(args.init_adapter / "adapters.safetensors")) - expected_weights = dict(tree_flatten(model.trainable_parameters())) - if weights.keys() != expected_weights.keys() or any( - weights[k].shape != expected_weights[k].shape for k in expected_weights - ): - ap.error("initial adapter tensor names or shapes do not match") - model.load_weights(list(weights.items()), strict=False) - print(f"initialized weights from {args.init_adapter}; optimizer starts fresh", flush=True) - if args.resume: - model.load_weights(str(args.resume / "adapters.safetensors"), strict=False) - print(f"resumed weights from {args.resume}") - if args.grad_checkpoint: - grad_checkpoint(model.layers[0]) - n_train = sum(v.size for _, v in tree_flatten(model.trainable_parameters())) - print(f"trainable params: {n_train / 1e6:.2f}M") - - train = load_examples(args.train, tokenizer, args.max_state_tokens, args.smoothing, args.use_example_weights) - val = load_examples(args.val, tokenizer, args.max_state_tokens, 0.0) - rng.shuffle(val) - val_batches = make_batches(val[: args.val_limit], args.max_batch_tokens, None) - steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None)) - total_steps = math.ceil(steps_per_epoch * args.epochs) - print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches") - run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps) - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - warmup = max(1, min(args.warmup, total_steps // 10)) - base_schedule = optim.join_schedules( - [optim.linear_schedule(1e-7, args.lr, warmup), optim.cosine_decay(args.lr, total_steps - warmup, args.lr * 0.05)], - [warmup], - ) - optimizer_state = args.resume / "optimizer.safetensors" if args.resume else None - if optimizer_state is not None and optimizer_state.exists(): - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.init(model.trainable_parameters()) - optimizer.state = tree_unflatten(list(mx.load(str(optimizer_state)).items())) - start_step = int(optimizer.state["step"].item()) - print(f"restored optimizer state at step {start_step}") - else: - start_step = args.start_step or 0 - if args.resume: - # Cold Adam moments (MLX AdamW has no bias correction by default) make the first - # updates several times too large; ramp the LR back up to protect the checkpoint. - def schedule(step, start=start_step): - return base_schedule(step) * mx.minimum(1.0, (step - start + 1).astype(mx.float32) / warmup) - print(f"no optimizer state in {args.resume}; re-warming the LR over {warmup} steps") - else: - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.state["step"] = mx.array(start_step, mx.uint64) - value_and_grad = nn.value_and_grad(model, partial(loss_fn, ordinal_weight=args.ordinal_weight)) - - adapter_config = { - "fine_tune_type": "lora", - "num_layers": num_layers, - "lora_parameters": lora_parameters, - "base_model": args.base_model, - "base_revision": args.base_revision, - } - model.eval() - best = evaluate(model, val_batches) - # Keep the initial checkpoint eligible; a regression must not replace it merely - # because no trained checkpoint improved the validation loss. - save_adapter(model, args.out, adapter_config, optimizer) - (args.out / "metrics.jsonl").write_text(json.dumps({"step": 0, **best, - "improved": False, "initial": True}) + "\n") - print(f"step 0 val nll {best['nll']:.4f} acc {best['acc']:.3f}") - model.train() - - saved = False - step, seen_tokens, started = start_step, 0, time.perf_counter() - losses: list[float] = [] - while step < total_steps: - for batch in make_batches(train, args.max_batch_tokens, rng): - if step >= total_steps: - break - arrays = to_arrays(batch) - loss, grads = value_and_grad(model, *arrays) - optimizer.update(model, grads) - mx.eval(model.trainable_parameters(), optimizer.state, loss) - step += 1 - losses.append(loss.item()) - seen_tokens += int(arrays[1].sum()) - if step % 25 == 0: - elapsed = time.perf_counter() - started - print( - f"step {step}/{total_steps} loss {np.mean(losses):.4f} lr {optimizer.learning_rate.item():.2e} " - f"{seen_tokens / elapsed:.0f} tok/s peak mem {mx.get_peak_memory() / 1e9:.1f}GB cache {mx.get_cache_memory() / 1e9:.1f}GB", - flush=True, - ) - losses = [] - if step % args.eval_every == 0 or step == total_steps: - model.eval() - metrics = evaluate(model, val_batches) - model.train() - improved = metrics["nll"] < best["nll"] - print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{' *saved*' if improved else ''}", flush=True) - with (args.out / "metrics.jsonl").open("a") as f: - f.write(json.dumps({"step": step, **metrics, "improved": improved, - "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n") - save_adapter(model, args.out / "last", adapter_config, optimizer) - if improved: - best = metrics - save_adapter(model, args.out, adapter_config, optimizer) - saved = True - if not saved: - print("validation never improved; retaining initial weights (final weights in last/)") - print(f"best val nll {best['nll']:.4f} acc {best['acc']:.3f}; adapter in {args.out}") - - -if __name__ == "__main__": - main() diff --git a/docs/training/jet-next/code/train_cuda.sh b/docs/training/jet-next/code/train_cuda.sh deleted file mode 100644 index 70cd9d07bdab4eb045f37bfdcf2d9bed856d2ec1..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/code/train_cuda.sh +++ /dev/null @@ -1,21 +0,0 @@ -#!/usr/bin/env bash -# Run jet-train on Linux/CUDA. Extra args go to jet-train. -set -euo pipefail -cd "$(dirname "$0")" -uv sync --extra cuda --inexact --quiet - -# MLX JIT-compiles kernels and wants one CUDA_HOME/include; the pip wheels spread the headers -# over several packages, so link them into a single tree. -export CUDA_HOME=$PWD/.venv/cuda_home -if [ ! -d "$CUDA_HOME/include" ]; then - mkdir -p "$CUDA_HOME/include" - for pkg in .venv/lib/python*/site-packages/nvidia/{cuda_runtime,cuda_cccl,cuda_nvrtc}/include; do - for f in "$PWD/$pkg"/*; do - [ "$(basename "$f")" = __init__.py ] || ln -sfn "$f" "$CUDA_HOME/include/" - done - done -fi -export MLX_USE_CUDA_GRAPHS=0 # batch shapes vary every step, so graphs just thrash -export MLX_CUDA_SDPA_CACHE_SIZE=${MLX_CUDA_SDPA_CACHE_SIZE:-2048} # same reason; the default 256 aborts mid-run -export MLX_PTX_CACHE_DIR=$PWD/.venv/ptx_cache # keep JIT-compiled kernels across runs -exec uv run --extra cuda --no-sync jet-train "$@" diff --git a/docs/training/jet-next/comparison.json b/docs/training/jet-next/comparison.json deleted file mode 100644 index 83da3eda3728c6a763ff91d6906cad6a170e64c2..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/comparison.json +++ /dev/null @@ -1,5308 +0,0 @@ -{ - "local_tests": { - "test": { - "baseline": { - "overall": { - "n": 3683, - "acc": 0.6760792831930491, - "nll": 0.871394003472334, - "brier": 0.42748423587682505, - "ece": 0.06219506579315473 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04892235172964218, - "brier": 0.020441859866866866, - "ece": 0.002619326776928345 - }, - "boolq": { - "n": 145, - "acc": 0.8, - "nll": 0.4566301124234652, - "brier": 0.2894317164731869, - "ece": 0.04909670681789005 - }, - "emotion": { - "n": 2500, - "acc": 0.6036, - "nll": 1.061848826275766, - "brier": 0.5326494158453977, - "ece": 0.09432493007183075 - }, - "ag_news": { - "n": 135, - "acc": 0.8814814814814815, - "nll": 0.3042688087249796, - "brier": 0.1724398602199645, - "ece": 0.030876177770120094 - }, - "stsb": { - "n": 117, - "acc": 0.5641025641025641, - "nll": 1.133929574747437, - "brier": 0.3139340576398371, - "ece": 0.06830696112070328, - "mae": 0.09924938618105988, - "within1": 0.8803418803418803, - "spearman": 0.8588398930237986 - }, - "mnli": { - "n": 142, - "acc": 0.8169014084507042, - "nll": 0.47927640531290794, - "brier": 0.268308599548532, - "ece": 0.08220711266490774 - }, - "yelp": { - "n": 146, - "acc": 0.7191780821917808, - "nll": 0.6889766353144221, - "brier": 0.3981368921297408, - "ece": 0.08682229931223884 - }, - "banking77": { - "n": 140, - "acc": 0.8857142857142857, - "nll": 0.38170325982251335, - "brier": 0.15168387780774686, - "ece": 0.053875945082732635 - }, - "massive": { - "n": 119, - "acc": 0.9159663865546218, - "nll": 0.33522919206459373, - "brier": 0.11903663498667257, - "ece": 0.026440266551089928 - }, - "civil_comments": { - "n": 149, - "acc": 0.9194630872483222, - "nll": 0.3247184777021575, - "brier": 0.04936214179418669, - "ece": 0.07150244792835826 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6241729157476842, - "nll": 1.0613521105481916, - "brier": 0.5002384772375326, - "ece": 0.08481803025029859 - }, - "noul": { - "n": 1185, - "acc": 0.7881856540084389, - "nll": 0.48529899323149145, - "brier": 0.29418788896638004, - "ece": 0.03554945349190307 - }, - "score": { - "n": 231, - "acc": 0.6103896103896104, - "nll": 0.9877903601581737, - "brier": 0.39727776801601805, - "ece": 0.05583026837476918, - "mae": 0.10864421681236895, - "within1": 0.9307359307359307, - "spearman": 0.8929563754198246 - } - }, - "ms_per_question": 7.412202802334466, - "max_state_tokens": 4096 - }, - "v3": { - "overall": { - "n": 3683, - "acc": 0.6519142003801249, - "nll": 0.9362977022113222, - "brier": 0.4648460641984677, - "ece": 0.087802886825329 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04086569703973105, - "brier": 0.018036308573590402, - "ece": 0.02379309203889635 - }, - "boolq": { - "n": 145, - "acc": 0.7793103448275862, - "nll": 0.5092739206128594, - "brier": 0.31537677875818465, - "ece": 0.08913842850718003 - }, - "emotion": { - "n": 2500, - "acc": 0.5688, - "nll": 1.1567023090774193, - "brier": 0.5870578081436417, - "ece": 0.13159714596271516 - }, - "ag_news": { - "n": 135, - "acc": 0.8962962962962963, - "nll": 0.30371399552871786, - "brier": 0.16402837572263823, - "ece": 0.050474644148791284 - }, - "stsb": { - "n": 117, - "acc": 0.5726495726495726, - "nll": 1.0607522075833626, - "brier": 0.2886478136405242, - "ece": 0.06050514436175681, - "mae": 0.09295228923149568, - "within1": 0.905982905982906, - "spearman": 0.8851879649671098 - }, - "mnli": { - "n": 142, - "acc": 0.8380281690140845, - "nll": 0.46939173813181284, - "brier": 0.2641611997591568, - "ece": 0.06754905090365615 - }, - "yelp": { - "n": 146, - "acc": 0.6917808219178082, - "nll": 0.6570821626403701, - "brier": 0.38738836694199724, - "ece": 0.11636449886511446 - }, - "banking77": { - "n": 140, - "acc": 0.8642857142857143, - "nll": 0.47842126280551645, - "brier": 0.1862706174014704, - "ece": 0.06801908761262894 - }, - "massive": { - "n": 119, - "acc": 0.8991596638655462, - "nll": 0.31506088688605693, - "brier": 0.1227447798011435, - "ece": 0.05545356293686299 - }, - "civil_comments": { - "n": 149, - "acc": 0.9463087248322147, - "nll": 0.31501872988091356, - "brier": 0.04269061276457159, - "ece": 0.053070764413615995 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.5963828848698721, - "nll": 1.1364487257573137, - "brier": 0.5443700049236206, - "ece": 0.09375659254452287 - }, - "noul": { - "n": 1185, - "acc": 0.7670886075949367, - "nll": 0.5571621690939522, - "brier": 0.331423905411652, - "ece": 0.09775889085817938 - }, - "score": { - "n": 231, - "acc": 0.6060606060606061, - "nll": 0.9169610631001566, - "brier": 0.3688481617675371, - "ece": 0.06983209030452746, - "mae": 0.10237847252706574, - "within1": 0.948051948051948, - "spearman": 0.9093660800040722 - } - }, - "ms_per_question": 11.374087093130628, - "max_state_tokens": 4096 - }, - "warm": { - "overall": { - "n": 3683, - "acc": 0.6749932120553896, - "nll": 0.8770703999656424, - "brier": 0.43323420329822226, - "ece": 0.07087006142548558 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.02723309464054182, - "brier": 0.012118271812111808, - "ece": 0.019888099696901113 - }, - "boolq": { - "n": 145, - "acc": 0.7931034482758621, - "nll": 0.45604896231971936, - "brier": 0.29052021521305826, - "ece": 0.061517162158571434 - }, - "emotion": { - "n": 2500, - "acc": 0.6012, - "nll": 1.0810718602698297, - "brier": 0.5453604910102474, - "ece": 0.10244272117614747 - }, - "ag_news": { - "n": 135, - "acc": 0.8888888888888888, - "nll": 0.299911898188293, - "brier": 0.16838452547589777, - "ece": 0.04407136572731866 - }, - "stsb": { - "n": 117, - "acc": 0.5641025641025641, - "nll": 1.0637444151019242, - "brier": 0.2787832582420156, - "ece": 0.07576155611592482, - "mae": 0.0916897660563103, - "within1": 0.8974358974358975, - "spearman": 0.8805116134565336 - }, - "mnli": { - "n": 142, - "acc": 0.8028169014084507, - "nll": 0.47346722350364, - "brier": 0.2732897038041701, - "ece": 0.04862175105323253 - }, - "yelp": { - "n": 146, - "acc": 0.6917808219178082, - "nll": 0.6881789257175812, - "brier": 0.39966799591362356, - "ece": 0.06584085483257086 - }, - "banking77": { - "n": 140, - "acc": 0.9285714285714286, - "nll": 0.3419254706257821, - "brier": 0.13463227703766398, - "ece": 0.03366895892790385 - }, - "massive": { - "n": 119, - "acc": 0.9243697478991597, - "nll": 0.2586406250181021, - "brier": 0.09246301809645702, - "ece": 0.048050142511600245 - }, - "civil_comments": { - "n": 149, - "acc": 0.9261744966442953, - "nll": 0.32008108970125426, - "brier": 0.04445927595205485, - "ece": 0.10419578600249835 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6259373621526246, - "nll": 1.0736314924175012, - "brier": 0.5102127508525695, - "ece": 0.09648422277678359 - }, - "noul": { - "n": 1185, - "acc": 0.7848101265822784, - "nll": 0.48621275175856077, - "brier": 0.29606669919747736, - "ece": 0.02945000119350126 - }, - "score": { - "n": 231, - "acc": 0.5930735930735931, - "nll": 0.95309774428178, - "brier": 0.38142954985093946, - "ece": 0.03230119448203545, - "mae": 0.10350282123406596, - "within1": 0.9393939393939394, - "spearman": 0.9028200815430442 - } - }, - "ms_per_question": 11.407006713820156, - "max_state_tokens": 4096 - }, - "v4": { - "overall": { - "n": 3683, - "acc": 0.6644040184632093, - "nll": 0.8783917908240401, - "brier": 0.4363018875628721, - "ece": 0.044080123370055624 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04609997218164305, - "brier": 0.00993532707109211, - "ece": 0.03980768852763706 - }, - "boolq": { - "n": 145, - "acc": 0.7862068965517242, - "nll": 0.4650379801875558, - "brier": 0.2991545819338941, - "ece": 0.056202566623687736 - }, - "emotion": { - "n": 2500, - "acc": 0.5852, - "nll": 1.0812736734218895, - "brier": 0.5494403625547004, - "ece": 0.06851770701408388 - }, - "ag_news": { - "n": 135, - "acc": 0.8888888888888888, - "nll": 0.3087087308918988, - "brier": 0.16917814412184332, - "ece": 0.03841419992623507 - }, - "stsb": { - "n": 117, - "acc": 0.5470085470085471, - "nll": 1.0753559749796122, - "brier": 0.28199056470137557, - "ece": 0.03365990544995692, - "mae": 0.09383068554079495, - "within1": 0.8717948717948718, - "spearman": 0.877984058590352 - }, - "mnli": { - "n": 142, - "acc": 0.8309859154929577, - "nll": 0.4340047932059412, - "brier": 0.24689615776249432, - "ece": 0.07052275685357377 - }, - "yelp": { - "n": 146, - "acc": 0.6986301369863014, - "nll": 0.7119902010420209, - "brier": 0.41549555641030894, - "ece": 0.11680830300670779 - }, - "banking77": { - "n": 140, - "acc": 0.9071428571428571, - "nll": 0.3257405191460358, - "brier": 0.13279354980220526, - "ece": 0.07537455537489482 - }, - "massive": { - "n": 119, - "acc": 0.9327731092436975, - "nll": 0.2788916040219742, - "brier": 0.099814120778048, - "ece": 0.05662691129856755 - }, - "civil_comments": { - "n": 149, - "acc": 0.9328859060402684, - "nll": 0.32543574588733154, - "brier": 0.0470122361134796, - "ece": 0.10539027988510645 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6109395677106307, - "nll": 1.0674977012510223, - "brier": 0.508642577204092, - "ece": 0.054254766210197196 - }, - "noul": { - "n": 1185, - "acc": 0.7805907172995781, - "nll": 0.49928341122037284, - "brier": 0.30730912987911574, - "ece": 0.034595404093778585 - }, - "score": { - "n": 231, - "acc": 0.5930735930735931, - "nll": 0.9673109721763195, - "brier": 0.38807710158281145, - "ece": 0.06311224291334937, - "mae": 0.1062169110937673, - "within1": 0.9177489177489178, - "spearman": 0.899445205722314 - } - }, - "ms_per_question": 11.422907029324417, - "max_state_tokens": 4096 - } - }, - "score_eval": { - "baseline": { - "overall": { - "n": 2400, - "acc": 0.5170833333333333, - "nll": 1.0893632032843228, - "brier": 0.5287154496496189, - "ece": 0.03900318566709756 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5216666666666666, - "nll": 1.1038147784073833, - "brier": 0.33254204642705393, - "ece": 0.04424463902910554, - "mae": 0.10265430813162432, - "within1": 0.88, - "spearman": 0.9022189976505913 - }, - "yelp": { - "n": 600, - "acc": 0.6033333333333334, - "nll": 0.8791397150792182, - "brier": 0.5038029301388466, - "ece": 0.04722331101695697, - "mae": 0.12401153839202986, - "within1": 0.97, - "spearman": 0.8922031203071111 - }, - "sst5": { - "n": 600, - "acc": 0.475, - "nll": 1.198184583062927, - "brier": 0.6437840936114044, - "ece": 0.031442425797382975, - "mae": 0.1737434748756156, - "within1": 0.9066666666666666, - "spearman": 0.7429955533970433 - }, - "amazon": { - "n": 600, - "acc": 0.4683333333333333, - "nll": 1.1763137365877627, - "brier": 0.6347327284211705, - "ece": 0.09294485206405323, - "mae": 0.17271815862875275, - "within1": 0.8966666666666666, - "spearman": 0.7921592534430197 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.5170833333333333, - "nll": 1.0893632032843228, - "brier": 0.5287154496496189, - "ece": 0.03900318566709756, - "mae": 0.14328187000700565, - "within1": 0.9133333333333333, - "spearman": 0.832342527754484 - } - }, - "ms_per_question": 8.335110699583007, - "max_state_tokens": 4096 - }, - "v3": { - "overall": { - "n": 2400, - "acc": 0.5016666666666667, - "nll": 1.1098948827101054, - "brier": 0.5388255751601235, - "ece": 0.049056352948149064 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5316666666666666, - "nll": 1.0845926729567896, - "brier": 0.3220883925843873, - "ece": 0.032908969024817145, - "mae": 0.1033457961170546, - "within1": 0.8766666666666667, - "spearman": 0.9018376493733692 - }, - "yelp": { - "n": 600, - "acc": 0.6066666666666667, - "nll": 0.8713869097207984, - "brier": 0.5001777113619436, - "ece": 0.04452881038188934, - "mae": 0.12427177343595153, - "within1": 0.9633333333333334, - "spearman": 0.88508184570763 - }, - "sst5": { - "n": 600, - "acc": 0.415, - "nll": 1.2602221814046304, - "brier": 0.6752547946764169, - "ece": 0.04730295836925506, - "mae": 0.18144155930979952, - "within1": 0.8866666666666667, - "spearman": 0.737770481866935 - }, - "amazon": { - "n": 600, - "acc": 0.4533333333333333, - "nll": 1.2233777667582035, - "brier": 0.6577814020177467, - "ece": 0.09444272140661875, - "mae": 0.18028308809317725, - "within1": 0.9033333333333333, - "spearman": 0.7926277614470995 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.5016666666666667, - "nll": 1.1098948827101054, - "brier": 0.5388255751601235, - "ece": 0.049056352948149064, - "mae": 0.14733555423899575, - "within1": 0.9075, - "spearman": 0.8294005319526306 - } - }, - "ms_per_question": 11.946628592916872, - "max_state_tokens": 4096 - }, - "warm": { - "overall": { - "n": 2400, - "acc": 0.5304166666666666, - "nll": 1.092109727657017, - "brier": 0.5292645190337113, - "ece": 0.05937186859548092 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5566666666666666, - "nll": 1.0754084892512186, - "brier": 0.3166843313684106, - "ece": 0.04638583054145179, - "mae": 0.09808268856668535, - "within1": 0.9, - "spearman": 0.9082776913459185 - }, - "yelp": { - "n": 600, - "acc": 0.6083333333333333, - "nll": 0.8788605593300114, - "brier": 0.5081122332933897, - "ece": 0.08311119998494783, - "mae": 0.12012921416409275, - "within1": 0.975, - "spearman": 0.8964504716876638 - }, - "sst5": { - "n": 600, - "acc": 0.45166666666666666, - "nll": 1.2218405024707317, - "brier": 0.655538649626939, - "ece": 0.04943077882130942, - "mae": 0.17551595646558174, - "within1": 0.9183333333333333, - "spearman": 0.7543010013520536 - }, - "amazon": { - "n": 600, - "acc": 0.505, - "nll": 1.192329359576106, - "brier": 0.6367228618461059, - "ece": 0.09014196003476778, - "mae": 0.16920178736618255, - "within1": 0.8966666666666666, - "spearman": 0.7976749967006868 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.5304166666666666, - "nll": 1.092109727657017, - "brier": 0.5292645190337113, - "ece": 0.05937186859548092, - "mae": 0.1407324116406356, - "within1": 0.9225, - "spearman": 0.8367095771732885 - } - }, - "ms_per_question": 12.109397532082463, - "max_state_tokens": 4096 - }, - "v4": { - "overall": { - "n": 2400, - "acc": 0.52, - "nll": 1.1051732249609856, - "brier": 0.5357743376369629, - "ece": 0.06524991735816002 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5366666666666666, - "nll": 1.1102153950019593, - "brier": 0.332459629641885, - "ece": 0.06996035655339557, - "mae": 0.10243597520344581, - "within1": 0.89, - "spearman": 0.9011497264456896 - }, - "yelp": { - "n": 600, - "acc": 0.595, - "nll": 0.9026956470062335, - "brier": 0.5201132048387901, - "ece": 0.09163652310768762, - "mae": 0.12338489234049727, - "within1": 0.97, - "spearman": 0.8902715344208428 - }, - "sst5": { - "n": 600, - "acc": 0.475, - "nll": 1.2161626872668663, - "brier": 0.6505152737815312, - "ece": 0.0305570506552855, - "mae": 0.1754747062223032, - "within1": 0.9116666666666666, - "spearman": 0.758046670545616 - }, - "amazon": { - "n": 600, - "acc": 0.47333333333333333, - "nll": 1.1916191705688834, - "brier": 0.6400092422856453, - "ece": 0.10654552747805911, - "mae": 0.17166901100873172, - "within1": 0.8983333333333333, - "spearman": 0.7888142015605857 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.52, - "nll": 1.1051732249609856, - "brier": 0.5357743376369629, - "ece": 0.06524991735816002, - "mae": 0.1432411461937445, - "within1": 0.9175, - "spearman": 0.8328310991663338 - } - }, - "ms_per_question": 11.977573064583945, - "max_state_tokens": 4096 - } - }, - "test_v3_new": { - "baseline": { - "overall": { - "n": 1364, - "acc": 0.4567448680351906, - "nll": 1.3997251274893088, - "brier": 0.7614422661756503, - "ece": 0.28511070977907377 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.46875, - "nll": 1.3125342436600476, - "brier": 0.7248338334245858, - "ece": 0.18823765367269515 - }, - "v3:entailment": { - "n": 160, - "acc": 0.66875, - "nll": 0.7707340998575092, - "brier": 0.445276258289386, - "ece": 0.10283493064343929 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.50625, - "nll": 1.2612553351966198, - "brier": 0.7811341016274713, - "ece": 0.33411078229546554 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.01032785861358299, - "brier": 0.00037024526307215803, - "ece": 0.010188460350036621 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.2, - "nll": 1.8659685906022787, - "brier": 0.9681135361388087, - "ece": 0.350784395635128 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.54375, - "nll": 1.0387783478654455, - "brier": 0.6880096149279901, - "ece": 0.33221421800553796 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.20625, - "nll": 2.169761631079018, - "brier": 0.9876360526115023, - "ece": 0.32759805452078583 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.25625, - "nll": 2.6546468422631735, - "brier": 1.2787821977853926, - "ece": 0.6251898461952805 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.51875, - "nll": 0.8461553640197963, - "brier": 0.615775613120263, - "ece": 0.2303109876811505 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.05768870040774345, - "brier": 0.010224533000073113, - "ece": 0.05483078479766846 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.4567448680351906, - "nll": 1.3997251274893088, - "brier": 0.7614422661756503, - "ece": 0.28511070977907377 - } - }, - "ms_per_question": 9.546318121700155, - "max_state_tokens": 4096 - }, - "v3": { - "overall": { - "n": 1364, - "acc": 0.8042521994134897, - "nll": 0.44222923351265353, - "brier": 0.259567315985772, - "ece": 0.021087675303482965 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.625, - "nll": 0.7615659017814324, - "brier": 0.45959500191353253, - "ece": 0.09703730791807175 - }, - "v3:entailment": { - "n": 160, - "acc": 0.8125, - "nll": 0.5015351219626609, - "brier": 0.2794180829289865, - "ece": 0.0648006197065115 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.675, - "nll": 0.618581273034215, - "brier": 0.4265876277825605, - "ece": 0.061761666089296355 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.013145773430844233, - "brier": 0.00039104018188636664, - "ece": 0.012993097305297852 - }, - "v3:code_behavior": { - "n": 160, - "acc": 1.0, - "nll": 0.021853731421288103, - "brier": 0.0033133787000630367, - "ece": 0.02084268778562546 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.79375, - "nll": 0.4674972024862655, - "brier": 0.29483375706962633, - "ece": 0.07397291809320453 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.975, - "nll": 0.08550979029969312, - "brier": 0.03868921824095032, - "ece": 0.03651268482208252 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.50625, - "nll": 1.1571399623062462, - "brier": 0.6228013072055809, - "ece": 0.06916055623441936 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.94375, - "nll": 0.15053857543389312, - "brier": 0.08741509523600818, - "ece": 0.049381273239851003 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.0059849793091416355, - "brier": 8.770325969450155e-05, - "ece": 0.0059629082679748535 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.8042521994134897, - "nll": 0.44222923351265353, - "brier": 0.259567315985772, - "ece": 0.021087675303482965 - } - }, - "ms_per_question": 13.624932868769761, - "max_state_tokens": 4096 - }, - "warm": { - "overall": { - "n": 1364, - "acc": 0.7749266862170088, - "nll": 0.5070664367182426, - "brier": 0.30008040549776915, - "ece": 0.02237014299875829 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.63125, - "nll": 0.7567455511540174, - "brier": 0.4507593102169916, - "ece": 0.06857692748308183 - }, - "v3:entailment": { - "n": 160, - "acc": 0.79375, - "nll": 0.5101304793381132, - "brier": 0.28997449331401376, - "ece": 0.03158894721418622 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.7, - "nll": 0.5968082959763705, - "brier": 0.4076295926973904, - "ece": 0.057135591283440576 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.011557995653474483, - "brier": 0.00022427398406391338, - "ece": 0.011465013027191162 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.9, - "nll": 0.2162705672410084, - "brier": 0.13338298348556035, - "ece": 0.040077064931392674 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.8375, - "nll": 0.4227919111261144, - "brier": 0.2553973171422433, - "ece": 0.055019789934158306 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.84375, - "nll": 0.4120700732804835, - "brier": 0.24532611174360155, - "ece": 0.06690083071589471 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.525, - "nll": 1.1109857523348183, - "brier": 0.591940409718284, - "ece": 0.052010057494044304 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.85, - "nll": 0.2916829534515273, - "brier": 0.18366851220511013, - "ece": 0.04999278411269187 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.0063601806294173, - "brier": 0.00015376200744249418, - "ece": 0.0063211917877197266 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.7749266862170088, - "nll": 0.5070664367182426, - "brier": 0.30008040549776915, - "ece": 0.02237014299875829 - } - }, - "ms_per_question": 13.352808236071233, - "max_state_tokens": 4096 - }, - "v4": { - "overall": { - "n": 1364, - "acc": 0.7719941348973607, - "nll": 0.5133160767353705, - "brier": 0.299812832129049, - "ece": 0.04058140996963746 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.625, - "nll": 0.7707278399262577, - "brier": 0.45656743231427815, - "ece": 0.10337576027959586 - }, - "v3:entailment": { - "n": 160, - "acc": 0.80625, - "nll": 0.5095306556206196, - "brier": 0.27964219746474067, - "ece": 0.046038990654051294 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.7, - "nll": 0.5887704980559647, - "brier": 0.4007791939368809, - "ece": 0.05541613362729548 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.030981391526253548, - "brier": 0.001263318868803046, - "ece": 0.03043532371520996 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.875, - "nll": 0.24078694187337532, - "brier": 0.13652739831088206, - "ece": 0.06979165971279144 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.84375, - "nll": 0.4296516477130353, - "brier": 0.25673140257791316, - "ece": 0.07205855362117289 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.8125, - "nll": 0.4019184489734471, - "brier": 0.24345378337085438, - "ece": 0.07234472986310721 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.5375, - "nll": 1.109215586911887, - "brier": 0.5870259634345372, - "ece": 0.08422903288155792 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.85625, - "nll": 0.31212113342480735, - "brier": 0.19465629563358502, - "ece": 0.07933795303106309 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.011983446683734656, - "brier": 0.0003512193510441135, - "ece": 0.011894524097442627 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.7719941348973607, - "nll": 0.5133160767353705, - "brier": 0.299812832129049, - "ece": 0.04058140996963746 - } - }, - "ms_per_question": 13.481722373900375, - "max_state_tokens": 4096 - } - }, - "test_v4_new": { - "baseline": { - "overall": { - "n": 720, - "acc": 0.3416666666666667, - "nll": 1.4211787942176064, - "brier": 0.8029851960497246, - "ece": 0.23449580069217418 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.32916666666666666, - "nll": 1.603874272511651, - "brier": 0.8979469405703165, - "ece": 0.3367636108150085 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.3, - "nll": 1.3962707759191593, - "brier": 0.7587345977402417, - "ece": 0.11225773791472118 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.3958333333333333, - "nll": 1.2633913342220089, - "brier": 0.7522740498386156, - "ece": 0.2594969267646472 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.3416666666666667, - "nll": 1.4211787942176064, - "brier": 0.8029851960497246, - "ece": 0.23449580069217418 - } - }, - "ms_per_question": 21.310045005556177, - "max_state_tokens": 2048 - }, - "v3": { - "overall": { - "n": 720, - "acc": 0.5041666666666667, - "nll": 1.2512630294710916, - "brier": 0.6757878953089775, - "ece": 0.18835306784345046 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.5333333333333333, - "nll": 1.00347384273385, - "brier": 0.6163914967164069, - "ece": 0.15025270320475104 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.525, - "nll": 1.6918137950861516, - "brier": 0.7644781774214823, - "ece": 0.29563305079936986 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.45416666666666666, - "nll": 1.0585014505932728, - "brier": 0.6464940117890435, - "ece": 0.12545051338771981 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.5041666666666667, - "nll": 1.2512630294710916, - "brier": 0.6757878953089775, - "ece": 0.18835306784345046 - } - }, - "ms_per_question": 30.59652488194236, - "max_state_tokens": 2048 - }, - "warm": { - "overall": { - "n": 720, - "acc": 0.48333333333333334, - "nll": 1.3513476976581538, - "brier": 0.7107875032531545, - "ece": 0.1748247489333153 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6, - "nll": 0.9012368252500892, - "brier": 0.537829477895586, - "ece": 0.058517271776994055 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.38333333333333336, - "nll": 2.0515175121137874, - "brier": 0.9352148454889156, - "ece": 0.3826701547950506 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.4666666666666667, - "nll": 1.1012887556105853, - "brier": 0.6593181863749621, - "ece": 0.11634314979116123 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.48333333333333334, - "nll": 1.3513476976581538, - "brier": 0.7107875032531545, - "ece": 0.1748247489333153 - } - }, - "ms_per_question": 30.290553562498545, - "max_state_tokens": 2048 - }, - "v4": { - "overall": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6, - "nll": 0.8641331787531574, - "brier": 0.5189303984919411, - "ece": 0.08732014621297518 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.5708333333333333, - "nll": 1.0104112304126223, - "brier": 0.553216138337243, - "ece": 0.1417993993808826 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.8083333333333333, - "nll": 0.468007056050313, - "brier": 0.2578825642308249, - "ece": 0.0798518454035123 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - } - }, - "ms_per_question": 30.677324276388795, - "max_state_tokens": 2048 - } - }, - "test_v4_tools": { - "baseline": { - "overall": { - "n": 240, - "acc": 0.3625, - "nll": 1.7029224928080415, - "brier": 0.8800043840538005, - "ece": 0.3612365548809369 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.3625, - "nll": 1.7029224928080415, - "brier": 0.8800043840538005, - "ece": 0.3612365548809369 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.3625, - "nll": 1.7029224928080415, - "brier": 0.8800043840538005, - "ece": 0.3612365548809369 - } - }, - "ms_per_question": 21.0819619958329, - "max_state_tokens": 2048 - }, - "v3": { - "overall": { - "n": 240, - "acc": 0.31666666666666665, - "nll": 1.7665268682098636, - "brier": 0.8797549399674246, - "ece": 0.3770314007997513 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.31666666666666665, - "nll": 1.7665268682098636, - "brier": 0.8797549399674246, - "ece": 0.3770314007997513 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.31666666666666665, - "nll": 1.7665268682098636, - "brier": 0.8797549399674246, - "ece": 0.3770314007997513 - } - }, - "ms_per_question": 26.11035827082257, - "max_state_tokens": 2048 - }, - "warm": { - "overall": { - "n": 240, - "acc": 0.2875, - "nll": 1.7748789598233998, - "brier": 0.9634518656932426, - "ece": 0.4559627714256445 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.2875, - "nll": 1.7748789598233998, - "brier": 0.9634518656932426, - "ece": 0.4559627714256445 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.2875, - "nll": 1.7748789598233998, - "brier": 0.9634518656932426, - "ece": 0.4559627714256445 - } - }, - "ms_per_question": 25.927134495835464, - "max_state_tokens": 2048 - }, - "v4": { - "overall": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "ms_per_question": 25.963092025009853, - "max_state_tokens": 2048 - } - } - }, - "diagnostics": { - "baseline": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "301aec4322e27832ed7b5d22651d86951534808e0548df29addd7f6d99c021ad", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.1049274939996394, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 48.596005499348394, - "p95": 373.8532560000749, - "mean": 84.14850932991249 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.285649774536176, - "reference_same_cases": null, - "median_ms": 353.8561925006434, - "detail": { - "field_accuracy": 0.47794117647058826, - "case_exact_accuracy": 0, - "macro_f1": 0.285649774536176, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.20535714285714285, - "reference_same_cases": null, - "median_ms": 43.758601999797975, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.125, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.20535714285714285, - "case_exact_accuracy": 0.07142857142857142, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 47.17806650023704, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.5730994152046783, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.35085995085995086, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.2222222222222222, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.031746031746031744 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.36556995679627785, - "reference_same_cases": null, - "median_ms": 15.092844499122293, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.36556995679627785, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.25058158856763046, - "reference_same_cases": null, - "median_ms": 57.2153459997935, - "detail": { - "field_accuracy": 0.2545454545454545, - "case_exact_accuracy": 0.2545454545454545, - "macro_f1": 0.25058158856763046, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 43.94272399986221, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.38181818181818183, - "reference_same_cases": null, - "median_ms": 42.82419500123069, - "detail": { - "field_accuracy": 0.38181818181818183, - "case_exact_accuracy": 0.38181818181818183, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 62.874530000044615, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.3003030303030303 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.285649774536176, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.285649774536176, - "random": 0.30851893087083776, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.20535714285714285, - "skill": 0.03769841269841268, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.2857142857142857, - "random": 0.25, - "skill": 0.047619047619047596, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.125, - "random": 0.1, - "skill": 0.02777777777777777, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.25, - "skill": 0.25, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.0, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.36556995679627785, - "skill": 0.04836746489660628, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.36556995679627785, - "random": 0.3333245556439576, - "skill": 0.04836746489660628, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.4, - "skill": 0.07772158637210544, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4, - "random": 0.34943722943722944, - "skill": 0.07772158637210544, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.38181818181818183, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.38181818181818183, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2909090909090909, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2909090909090909, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v3": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "f83fe3495f56fe326735f93a47a7ae6553ec22178cbd277a266a34d6fcd16657", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8959890799997083, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 62.191346499275824, - "p95": 441.3005799997336, - "mean": 104.96575838203717 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.28266566725189984, - "reference_same_cases": null, - "median_ms": 416.59302300013223, - "detail": { - "field_accuracy": 0.39705882352941174, - "case_exact_accuracy": 0, - "macro_f1": 0.28266566725189984, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.4642857142857143, - "reference_same_cases": null, - "median_ms": 57.63679649953701, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.5, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.42857142857142855, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.4642857142857143, - "case_exact_accuracy": 0.19642857142857142, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 61.89103449924005, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.6608187134502924, - "case_exact_accuracy": 0.39285714285714285, - "macro_f1": 0.3659194459631857, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.09523809523809523, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.013605442176870748 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.32318840579710145, - "reference_same_cases": null, - "median_ms": 28.75634900010482, - "detail": { - "field_accuracy": 0.4107142857142857, - "case_exact_accuracy": 0.4107142857142857, - "macro_f1": 0.32318840579710145, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5877061469265368, - "reference_same_cases": null, - "median_ms": 72.34201500068593, - "detail": { - "field_accuracy": 0.7272727272727273, - "case_exact_accuracy": 0.7272727272727273, - "macro_f1": 0.5877061469265368, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.38181818181818183, - "reference_same_cases": null, - "median_ms": 56.003140000029816, - "detail": { - "field_accuracy": 0.38181818181818183, - "case_exact_accuracy": 0.38181818181818183, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.43636363636363634, - "reference_same_cases": null, - "median_ms": 54.70588299976953, - "detail": { - "field_accuracy": 0.43636363636363634, - "case_exact_accuracy": 0.43636363636363634, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 81.98549199914851, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.29515151515151516 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.28266566725189984, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.28266566725189984, - "random": 0.30851893087083776, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.4642857142857143, - "skill": 0.3412698412698413, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.42857142857142855, - "random": 0.25, - "skill": 0.23809523809523805, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.5, - "random": 0.1, - "skill": 0.4444444444444445, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.25, - "skill": 0.25, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.0, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.32318840579710145, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32318840579710145, - "random": 0.3333245556439576, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.38181818181818183, - "skill": 0.04977375565610862, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.38181818181818183, - "random": 0.34943722943722944, - "skill": 0.04977375565610862, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.43636363636363634, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.43636363636363634, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2909090909090909, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2909090909090909, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "warm": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "7ae5d3f7b866c5a4475cdbedbd2482ebec1a635e5b479c9916a3a0f60053ac5f", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.864510400002473, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 61.95938749988272, - "p95": 433.2520369971462, - "mean": 104.73630839803081 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.30832722516635036, - "reference_same_cases": null, - "median_ms": 411.9378854993556, - "detail": { - "field_accuracy": 0.4422268907563025, - "case_exact_accuracy": 0, - "macro_f1": 0.30832722516635036, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.39285714285714285, - "reference_same_cases": null, - "median_ms": 56.87792700155114, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.35714285714285715, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.42857142857142855, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.39285714285714285, - "case_exact_accuracy": 0.21428571428571427, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 63.550472499628086, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.7192982456140351, - "case_exact_accuracy": 0.5892857142857143, - "macro_f1": 0.37563731374902903, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.14285714285714285, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.02040816326530612 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.39442285094459006, - "reference_same_cases": null, - "median_ms": 29.223100502349553, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.39442285094459006, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5083014048531289, - "reference_same_cases": null, - "median_ms": 71.3829269989219, - "detail": { - "field_accuracy": 0.6181818181818182, - "case_exact_accuracy": 0.6181818181818182, - "macro_f1": 0.5083014048531289, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 55.5151049993583, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.41818181818181815, - "reference_same_cases": null, - "median_ms": 54.88006499945186, - "detail": { - "field_accuracy": 0.41818181818181815, - "case_exact_accuracy": 0.41818181818181815, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2545454545454545, - "reference_same_cases": null, - "median_ms": 78.45878399893991, - "detail": { - "field_accuracy": 0.2545454545454545, - "case_exact_accuracy": 0.2545454545454545, - "custom_metrics": { - "normalized_rank_regret": 0.3190909090909091 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.30832722516635036, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.30832722516635036, - "random": 0.30851893087083776, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.39285714285714285, - "skill": 0.26190476190476186, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.42857142857142855, - "random": 0.25, - "skill": 0.23809523809523805, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.35714285714285715, - "random": 0.1, - "skill": 0.2857142857142857, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.25, - "skill": 0.25, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.0, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.39442285094459006, - "skill": 0.09164623628765685, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.39442285094459006, - "random": 0.3333245556439576, - "skill": 0.09164623628765685, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.4, - "skill": 0.07772158637210544, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4, - "random": 0.34943722943722944, - "skill": 0.07772158637210544, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.41818181818181815, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.41818181818181815, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2545454545454545, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2545454545454545, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v4": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "07a78ecfc6f9a4f45c618eee9c3b710d50ea08b9f963fdc43213f01d4374d76e", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8496340909987339, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 60.23746450227918, - "p95": 435.2597789984429, - "mean": 103.62341392398957 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.6536335738829072, - "reference_same_cases": null, - "median_ms": 410.0216540009569, - "detail": { - "field_accuracy": 0.6691176470588235, - "case_exact_accuracy": 0, - "macro_f1": 0.6536335738829072, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.375, - "reference_same_cases": null, - "median_ms": 54.77364450052846, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.4642857142857143, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.375, - "case_exact_accuracy": 0.17857142857142858, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 61.44454200148175, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.52046783625731, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.12538452073335796, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.125, - "sarcasm": 0.2222222222222222, - "irony": 0.11764705882352941, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.06640989729225023 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.4431124431124431, - "reference_same_cases": null, - "median_ms": 28.437076000045636, - "detail": { - "field_accuracy": 0.4642857142857143, - "case_exact_accuracy": 0.4642857142857143, - "macro_f1": 0.4431124431124431, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5203488372093024, - "reference_same_cases": null, - "median_ms": 71.75399799962179, - "detail": { - "field_accuracy": 0.6727272727272727, - "case_exact_accuracy": 0.6727272727272727, - "macro_f1": 0.5203488372093024, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 55.303547997027636, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4727272727272727, - "reference_same_cases": null, - "median_ms": 54.30907100162585, - "detail": { - "field_accuracy": 0.4727272727272727, - "case_exact_accuracy": 0.4727272727272727, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 80.21276100043906, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": { - "normalized_rank_regret": 0.2815151515151515 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.6536335738829072, - "skill": 0.4990948536692409, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.6536335738829072, - "random": 0.30851893087083776, - "skill": 0.4990948536692409, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.375, - "skill": 0.24603174603174605, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.4642857142857143, - "random": 0.25, - "skill": 0.28571428571428575, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.2857142857142857, - "random": 0.1, - "skill": 0.2063492063492063, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.0625, - "skill": 0.00878466561850422, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.25, - "random": 0.2226862339379383, - "skill": 0.03513866247401688, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.4431124431124431, - "skill": 0.16467966294233655, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4431124431124431, - "random": 0.3333245556439576, - "skill": 0.16467966294233655, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.32727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32727272727272727, - "random": 0.34943722943722944, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.4727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4727272727272727, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.32727272727272727, - "skill": 0.02503293807641634, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32727272727272727, - "random": 0.31, - "skill": 0.02503293807641634, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - } - }, - "esci": { - "baseline": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "1f2b23b88ea5ef287c7d7b3e02d9fa8aa303da49f28529924d4d8d63e57aeb6c", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.5189373569992313, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 47.89422899921192, - "p95": 59.1946660024405, - "mean": 42.08971909406682 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.19511369134010642, - "reference_same_cases": null, - "median_ms": 47.89422899921192, - "detail": { - "field_accuracy": 0.598, - "case_exact_accuracy": 0.598, - "macro_f1": 0.19511369134010642, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.19511369134010642, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.19511369134010642, - "random": 0.20273782411694521, - "skill": 0.0, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v3": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "73fca06d9aa2dfad6a21b3c755aa2684200b67bf081c7d80e1196dc5ec483bed", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.879909809002129, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 62.32302250100474, - "p95": 78.50551000228734, - "mean": 57.14209690994903 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.413930360808208, - "reference_same_cases": null, - "median_ms": 62.32302250100474, - "detail": { - "field_accuracy": 0.55, - "case_exact_accuracy": 0.55, - "macro_f1": 0.413930360808208, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.413930360808208, - "skill": 0.26489722337240446, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.413930360808208, - "random": 0.20273782411694521, - "skill": 0.26489722337240446, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "warm": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "360a9c1c12feb61d36783462013d09cc093b0d5d83c633ba416c2180db28997c", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8837185640004463, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 62.575644500611816, - "p95": 80.28408699829015, - "mean": 57.501974687904294 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.4147424168042725, - "reference_same_cases": null, - "median_ms": 62.575644500611816, - "detail": { - "field_accuracy": 0.56, - "case_exact_accuracy": 0.56, - "macro_f1": 0.4147424168042725, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.4147424168042725, - "skill": 0.2659157791507029, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4147424168042725, - "random": 0.20273782411694521, - "skill": 0.2659157791507029, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v4": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "d6118b06389f9283e3bb081ffd69474bb7633415c7143926971e4fa5121cbb0a", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8783724759996403, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 63.90887149973423, - "p95": 79.99267199920723, - "mean": 58.1968331539465 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.4134568939832098, - "reference_same_cases": null, - "median_ms": 63.90887149973423, - "detail": { - "field_accuracy": 0.532, - "case_exact_accuracy": 0.532, - "macro_f1": 0.4134568939832098, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.4134568939832098, - "skill": 0.26430335746565453, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4134568939832098, - "random": 0.20273782411694521, - "skill": 0.26430335746565453, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - } - }, - "isarcasm_english": { - "baseline": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "fce07a83cfbc84cab673c9a0218e6d2bd3e32bbb08bfdce2ca586e68a322e9b5", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.120845952998934, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 11.024953000742244, - "p95": 40.01360600159387, - "mean": 12.74496764784739 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 11.024953000742244, - "detail": { - "field_accuracy": 0.6535714285714286, - "case_exact_accuracy": 0.6535714285714286, - "macro_f1": 0.4811847326191155, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.1821247892074199 - }, - "category_macro_f1": 0.1821247892074199 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.1821247892074199, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.1821247892074199, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v3": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "f9aae1d9e7c2e7fbeb3751e8fb4b798572f22ac82b345a4377f21bdd29655571", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.878869845000736, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 25.58769000097527, - "p95": 54.116885003168136, - "mean": 26.95643487361719 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 25.58769000097527, - "detail": { - "field_accuracy": 0.7371428571428571, - "case_exact_accuracy": 0.7371428571428571, - "macro_f1": 0.5162822838386456, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.1894273127753304 - }, - "category_macro_f1": 0.1894273127753304 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.1894273127753304, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.1894273127753304, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "warm": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "3299a2f218a80735f0d00ad55108953983d942220e7992f4e9c4b61a96bfb0d3", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8737670730006357, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 25.724428500325303, - "p95": 55.542318001243984, - "mean": 27.1736307021482 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 25.724428500325303, - "detail": { - "field_accuracy": 0.7857142857142857, - "case_exact_accuracy": 0.7857142857142857, - "macro_f1": 0.526335733232285, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.17582417582417584 - }, - "category_macro_f1": 0.17582417582417584 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.17582417582417584, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.17582417582417584, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v4": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "46041659385e21e09f5e92f6bd0731c493a4595641c6c0022e56e1180c09437e", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8742223579974961, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 25.532456500513945, - "p95": 54.09158000111347, - "mean": 26.608081711388618 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 25.532456500513945, - "detail": { - "field_accuracy": 0.735, - "case_exact_accuracy": 0.735, - "macro_f1": 0.5388976333215262, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.23819301848049282 - }, - "category_macro_f1": 0.23819301848049282 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.23819301848049282, - "skill": 0.019949195832608534, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.23819301848049282, - "random": 0.2226862339379383, - "skill": 0.019949195832608534, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - } - }, - "cruxeval_full": { - "baseline": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "d61110ea9c6307f5fffa1a2a56ae634f716187298fa65cf5aaa8e2dfb57a8743", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.082998502999544, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 13.089024500004598, - "p95": 46.50721500001964, - "mean": 22.395500426424213 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.387719298245614, - "reference_same_cases": null, - "median_ms": 13.089024500004598, - "detail": { - "field_accuracy": 0.387719298245614, - "case_exact_accuracy": 0.387719298245614, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.387719298245614, - "skill": 0.028559562416605518, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.387719298245614, - "random": 0.36971874129768867, - "skill": 0.028559562416605518, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v3": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "f7748f31af62f0c601fd96d83c2d2b2f57033986547cfa5b4ac7f7c4f90f46b5", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8621119829986128, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 25.944673499907367, - "p95": 62.15157100086799, - "mean": 35.532386945570785 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.3473684210526316, - "reference_same_cases": null, - "median_ms": 25.944673499907367, - "detail": { - "field_accuracy": 0.3473684210526316, - "case_exact_accuracy": 0.3473684210526316, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.3473684210526316, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3473684210526316, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "warm": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "9b65aed86040c13dd85316232fbdae3a77500a0dcfad7c9243c0bda1be44f933", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9001195040000312, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 26.362825999967754, - "p95": 62.52290100019309, - "mean": 36.53820609660327 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.3508771929824561, - "reference_same_cases": null, - "median_ms": 26.362825999967754, - "detail": { - "field_accuracy": 0.3508771929824561, - "case_exact_accuracy": 0.3508771929824561, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.3508771929824561, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3508771929824561, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - }, - "v4": { - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "c2ec7dfde1a861c361c3eaca34133d624726eaf1df28347fae25ac568df65736", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8805886459995236, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 25.62000750185689, - "p95": 61.887538999144454, - "mean": 35.259878875436506 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.34385964912280703, - "reference_same_cases": null, - "median_ms": 25.62000750185689, - "detail": { - "field_accuracy": 0.34385964912280703, - "case_exact_accuracy": 0.34385964912280703, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.34385964912280703, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.34385964912280703, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } - } - } -} \ No newline at end of file diff --git a/docs/training/jet-next/completion-audit.json b/docs/training/jet-next/completion-audit.json deleted file mode 100644 index e7e751a355a1d1f43d4ae4a86464729c9f577678..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/completion-audit.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "status": "passed", - "compatibility": { - "ok": 26, - "total": 26 - }, - "tests": { - "adapter_including_gpu_golden_and_prefix_parity": 8, - "executable_v3_data": 3 - }, - "selected_optimizer_steps": { - "warm": 2444, - "v4": 3750 - }, - "original_datasets": "unchanged", - "all_v3_datasets": "unchanged", - "published_weights_and_calibration": "unchanged", - "preexisting_user_changes": "unchanged", - "active_training_or_evaluation_jobs": 0 -} \ No newline at end of file diff --git a/docs/training/jet-next/cruxeval-full-baseline.json b/docs/training/jet-next/cruxeval-full-baseline.json deleted file mode 100644 index 589d1f9f8037f192a54adbb45d38953aeae8874a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/cruxeval-full-baseline.json +++ /dev/null @@ -1,148 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "d61110ea9c6307f5fffa1a2a56ae634f716187298fa65cf5aaa8e2dfb57a8743", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.082998502999544, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 13.089024500004598, - "p95": 46.50721500001964, - "mean": 22.395500426424213 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.387719298245614, - "reference_same_cases": null, - "median_ms": 13.089024500004598, - "detail": { - "field_accuracy": 0.387719298245614, - "case_exact_accuracy": 0.387719298245614, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.387719298245614, - "skill": 0.028559562416605518, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.387719298245614, - "random": 0.36971874129768867, - "skill": 0.028559562416605518, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/cruxeval-full-sample.json b/docs/training/jet-next/cruxeval-full-sample.json deleted file mode 100644 index 58425481586e53bc0a0ca7b0730666e6fc890320..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/cruxeval-full-sample.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "requests": 570, - "selection": "All available CRUXEval rows after official exclusions; no label-based selection", - "source_sha256": "8dd94acf6c313c50aec3fc48188c840d994d14f764b3fff9fe7a89e2e41b9211", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16" -} \ No newline at end of file diff --git a/docs/training/jet-next/cruxeval-full-v3.json b/docs/training/jet-next/cruxeval-full-v3.json deleted file mode 100644 index 4c64e31bca9ae8d9150df273bcc8f43daa5a3875..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/cruxeval-full-v3.json +++ /dev/null @@ -1,152 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "f7748f31af62f0c601fd96d83c2d2b2f57033986547cfa5b4ac7f7c4f90f46b5", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8621119829986128, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 25.944673499907367, - "p95": 62.15157100086799, - "mean": 35.532386945570785 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.3473684210526316, - "reference_same_cases": null, - "median_ms": 25.944673499907367, - "detail": { - "field_accuracy": 0.3473684210526316, - "case_exact_accuracy": 0.3473684210526316, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.3473684210526316, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3473684210526316, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/cruxeval-full-v4.json b/docs/training/jet-next/cruxeval-full-v4.json deleted file mode 100644 index 99b6272e4cd183cbce5fbe4f74a3750a0bf5b021..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/cruxeval-full-v4.json +++ /dev/null @@ -1,152 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "c2ec7dfde1a861c361c3eaca34133d624726eaf1df28347fae25ac568df65736", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8805886459995236, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 25.62000750185689, - "p95": 61.887538999144454, - "mean": 35.259878875436506 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.34385964912280703, - "reference_same_cases": null, - "median_ms": 25.62000750185689, - "detail": { - "field_accuracy": 0.34385964912280703, - "case_exact_accuracy": 0.34385964912280703, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.34385964912280703, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.34385964912280703, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/cruxeval-full-warm.json b/docs/training/jet-next/cruxeval-full-warm.json deleted file mode 100644 index b5c7783ca4c1dc4bef129cdf78974f745f8e01d6..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/cruxeval-full-warm.json +++ /dev/null @@ -1,152 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "9b65aed86040c13dd85316232fbdae3a77500a0dcfad7c9243c0bda1be44f933", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9001195040000312, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 26.362825999967754, - "p95": 62.52290100019309, - "mean": 36.53820609660327 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.3508771929824561, - "reference_same_cases": null, - "median_ms": 26.362825999967754, - "detail": { - "field_accuracy": 0.3508771929824561, - "case_exact_accuracy": 0.3508771929824561, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.3508771929824561, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3508771929824561, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/diagnostic-v4.json b/docs/training/jet-next/diagnostic-v4.json deleted file mode 100644 index 56e7e02b8b7ea0140298ad2f3210b619f0d0066a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/diagnostic-v4.json +++ /dev/null @@ -1,515 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "07a78ecfc6f9a4f45c618eee9c3b710d50ea08b9f963fdc43213f01d4374d76e", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8496340909987339, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 60.23746450227918, - "p95": 435.2597789984429, - "mean": 103.62341392398957 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.6536335738829072, - "reference_same_cases": null, - "median_ms": 410.0216540009569, - "detail": { - "field_accuracy": 0.6691176470588235, - "case_exact_accuracy": 0, - "macro_f1": 0.6536335738829072, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.375, - "reference_same_cases": null, - "median_ms": 54.77364450052846, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.4642857142857143, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.375, - "case_exact_accuracy": 0.17857142857142858, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 61.44454200148175, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.52046783625731, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.12538452073335796, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.125, - "sarcasm": 0.2222222222222222, - "irony": 0.11764705882352941, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.06640989729225023 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.4431124431124431, - "reference_same_cases": null, - "median_ms": 28.437076000045636, - "detail": { - "field_accuracy": 0.4642857142857143, - "case_exact_accuracy": 0.4642857142857143, - "macro_f1": 0.4431124431124431, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5203488372093024, - "reference_same_cases": null, - "median_ms": 71.75399799962179, - "detail": { - "field_accuracy": 0.6727272727272727, - "case_exact_accuracy": 0.6727272727272727, - "macro_f1": 0.5203488372093024, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 55.303547997027636, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4727272727272727, - "reference_same_cases": null, - "median_ms": 54.30907100162585, - "detail": { - "field_accuracy": 0.4727272727272727, - "case_exact_accuracy": 0.4727272727272727, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 80.21276100043906, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": { - "normalized_rank_regret": 0.2815151515151515 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.6536335738829072, - "skill": 0.4990948536692409, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.6536335738829072, - "random": 0.30851893087083776, - "skill": 0.4990948536692409, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.375, - "skill": 0.24603174603174605, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.4642857142857143, - "random": 0.25, - "skill": 0.28571428571428575, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.2857142857142857, - "random": 0.1, - "skill": 0.2063492063492063, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.0625, - "skill": 0.00878466561850422, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.25, - "random": 0.2226862339379383, - "skill": 0.03513866247401688, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.4431124431124431, - "skill": 0.16467966294233655, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4431124431124431, - "random": 0.3333245556439576, - "skill": 0.16467966294233655, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.32727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32727272727272727, - "random": 0.34943722943722944, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.4727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4727272727272727, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.32727272727272727, - "skill": 0.02503293807641634, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32727272727272727, - "random": 0.31, - "skill": 0.02503293807641634, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/diagnostic-warm.json b/docs/training/jet-next/diagnostic-warm.json deleted file mode 100644 index de51c7d6715cc162ccf83ad2e6be37aa492e9431..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/diagnostic-warm.json +++ /dev/null @@ -1,515 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "7ae5d3f7b866c5a4475cdbedbd2482ebec1a635e5b479c9916a3a0f60053ac5f", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.864510400002473, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 61.95938749988272, - "p95": 433.2520369971462, - "mean": 104.73630839803081 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.30832722516635036, - "reference_same_cases": null, - "median_ms": 411.9378854993556, - "detail": { - "field_accuracy": 0.4422268907563025, - "case_exact_accuracy": 0, - "macro_f1": 0.30832722516635036, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.39285714285714285, - "reference_same_cases": null, - "median_ms": 56.87792700155114, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.35714285714285715, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.42857142857142855, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.39285714285714285, - "case_exact_accuracy": 0.21428571428571427, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 63.550472499628086, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.7192982456140351, - "case_exact_accuracy": 0.5892857142857143, - "macro_f1": 0.37563731374902903, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.14285714285714285, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.02040816326530612 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.39442285094459006, - "reference_same_cases": null, - "median_ms": 29.223100502349553, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.39442285094459006, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5083014048531289, - "reference_same_cases": null, - "median_ms": 71.3829269989219, - "detail": { - "field_accuracy": 0.6181818181818182, - "case_exact_accuracy": 0.6181818181818182, - "macro_f1": 0.5083014048531289, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 55.5151049993583, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.41818181818181815, - "reference_same_cases": null, - "median_ms": 54.88006499945186, - "detail": { - "field_accuracy": 0.41818181818181815, - "case_exact_accuracy": 0.41818181818181815, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2545454545454545, - "reference_same_cases": null, - "median_ms": 78.45878399893991, - "detail": { - "field_accuracy": 0.2545454545454545, - "case_exact_accuracy": 0.2545454545454545, - "custom_metrics": { - "normalized_rank_regret": 0.3190909090909091 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.30832722516635036, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.30832722516635036, - "random": 0.30851893087083776, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.39285714285714285, - "skill": 0.26190476190476186, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.42857142857142855, - "random": 0.25, - "skill": 0.23809523809523805, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.35714285714285715, - "random": 0.1, - "skill": 0.2857142857142857, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.25, - "skill": 0.25, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.0, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.39442285094459006, - "skill": 0.09164623628765685, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.39442285094459006, - "random": 0.3333245556439576, - "skill": 0.09164623628765685, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.4, - "skill": 0.07772158637210544, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4, - "random": 0.34943722943722944, - "skill": 0.07772158637210544, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.41818181818181815, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.41818181818181815, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2545454545454545, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2545454545454545, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/environment-freeze.txt b/docs/training/jet-next/environment-freeze.txt deleted file mode 100644 index 9220e676c708088caf1af48cf841d79e3153cc44..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/environment-freeze.txt +++ /dev/null @@ -1,82 +0,0 @@ -aiohappyeyeballs==2.7.1 -aiohttp==3.14.3 -aiosignal==1.4.0 -annotated-doc==0.0.5 -annotated-types==0.8.0 -anthropic==1.8.0 -anyio==4.15.1 -attrs==26.1.0 -certifi==2026.7.22 -charset-normalizer==3.5.1 -chess==1.11.2 -click==8.5.0 -datasets==5.0.1 -decision-index @ git+https://github.com/apolinario/decision-index.git@52a698928a9ae5bdf16b75687c903871db29c6e5 -dill==0.4.1 -docstring-parser==0.18.0 -fastapi==0.141.1 -filelock==4.0.1 -frozenlist==1.8.0 -fsspec==2026.6.0 -h11==0.16.0 -hf-xet==1.6.0 -httpcore==1.0.9 -httpcore2==2.13.0 -httpx==0.28.1 -httpx2==2.13.0 -huggingface-hub==1.32.0 -idna==3.20 --e file:///home/jach/dev/jet -jinja2==3.1.6 -jiter==0.17.0 -markdown-it-py==4.2.0 -markupsafe==3.0.3 -mdurl==0.1.2 -mido==1.3.3 -mlx==0.32.2 -mlx-cuda-12==0.32.2 -mlx-lm==0.31.3 -multidict==6.9.1 -multiprocess==0.70.19 -numpy==2.5.3 -nvidia-cublas-cu12==12.9.2.10 -nvidia-cuda-cccl-cu12==12.9.27 -nvidia-cuda-nvrtc-cu12==12.9.86 -nvidia-cuda-runtime-cu12==12.9.79 -nvidia-cudnn-cu12==9.26.0.51 -nvidia-cufft-cu12==11.4.1.4 -nvidia-nccl-cu12==2.32.3 -nvidia-nvjitlink-cu12==12.9.86 -packaging==26.3 -pandas==3.0.6 -propcache==0.5.4 -protobuf==7.36.2 -pyarrow==25.0.1 -pydantic==2.13.5 -pydantic-core==2.46.5 -pygments==2.21.0 -python-chess==1.999 -python-dateutil==2.9.0.post0 -pyyaml==6.0.3 -regex==2026.9.10 -requests==2.34.2 -rich==15.0.0 -safetensors==0.8.0 -scipy==1.18.1 -sentencepiece==0.2.2 -shellingham==1.5.4 -six==1.17.0 -sniffio==1.3.1 -starlette==1.6.0 -tiktoken==0.14.0 -tokenizers==0.23.2 -tqdm==4.70.1 -transformers==5.17.0 -truststore==0.10.4 -typer==0.27.2 -typing-extensions==4.16.0 -typing-inspection==0.4.4 -urllib3==2.8.0 -uvicorn==0.53.0 -xxhash==4.0.1 -yarl==1.25.1 diff --git a/docs/training/jet-next/esci-baseline.json b/docs/training/jet-next/esci-baseline.json deleted file mode 100644 index 59b1232c8425567c3303cbed8d046f7ab8d81085..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/esci-baseline.json +++ /dev/null @@ -1,154 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "1f2b23b88ea5ef287c7d7b3e02d9fa8aa303da49f28529924d4d8d63e57aeb6c", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.5189373569992313, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 47.89422899921192, - "p95": 59.1946660024405, - "mean": 42.08971909406682 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.19511369134010642, - "reference_same_cases": null, - "median_ms": 47.89422899921192, - "detail": { - "field_accuracy": 0.598, - "case_exact_accuracy": 0.598, - "macro_f1": 0.19511369134010642, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.19511369134010642, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.19511369134010642, - "random": 0.20273782411694521, - "skill": 0.0, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/esci-coverage.json b/docs/training/jet-next/esci-coverage.json deleted file mode 100644 index 7e4676c8c1ceb55806ad0da55f53a49336b85f0d..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/esci-coverage.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "rows_uncompressed_sha256": "f2bf36ce7a136f68dbd5d81ca01a1685297da258263511939dfaf8b818fefa9a", - "model_revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "scope": "Only the supplied rows; token capacity, not measured accuracy or validated long-context quality.", - "limits": [ - 4096, - 6144, - 8192, - 16384, - 32768, - 40960 - ], - "benchmarks": { - "Amazon ESCI": { - "requests": 500, - "fields": 500, - "options": { - "4": 500 - }, - "questions_per_request": { - "1": 500 - }, - "supported_requests": { - "4096": 500, - "6144": 500, - "8192": 500, - "16384": 500, - "32768": 500, - "40960": 500 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 400.0, - "p95": 1007.2499999999998, - "max": 2402.0 - }, - "source_groups": 500, - "supported_complete_groups": { - "4096": 500, - "6144": 500, - "8192": 500, - "16384": 500, - "32768": 500, - "40960": 500 - } - } - } -} diff --git a/docs/training/jet-next/esci-training-overlap.json b/docs/training/jet-next/esci-training-overlap.json deleted file mode 100644 index 7fa12cea7d5d75b1e70684e21ac241da3ec11d75..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/esci-training-overlap.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "training_esci_ids": 948, - "evaluation_esci_ids": 5000, - "overlap": 0, - "source": "Original example_id identities, train versus test", - "suite_sha256": "db1d2d3d4dde242dc04262149153bfd596044ad8986e0ae5de8443383973e386" -} \ No newline at end of file diff --git a/docs/training/jet-next/esci-v3.json b/docs/training/jet-next/esci-v3.json deleted file mode 100644 index 94e150dc21dcf6072583677ffdb2b668204847a3..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/esci-v3.json +++ /dev/null @@ -1,158 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "73fca06d9aa2dfad6a21b3c755aa2684200b67bf081c7d80e1196dc5ec483bed", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.879909809002129, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 62.32302250100474, - "p95": 78.50551000228734, - "mean": 57.14209690994903 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.413930360808208, - "reference_same_cases": null, - "median_ms": 62.32302250100474, - "detail": { - "field_accuracy": 0.55, - "case_exact_accuracy": 0.55, - "macro_f1": 0.413930360808208, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.413930360808208, - "skill": 0.26489722337240446, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.413930360808208, - "random": 0.20273782411694521, - "skill": 0.26489722337240446, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/esci-v4.json b/docs/training/jet-next/esci-v4.json deleted file mode 100644 index 136c6be51a8c8309378aab656126d069a8a91131..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/esci-v4.json +++ /dev/null @@ -1,158 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "d6118b06389f9283e3bb081ffd69474bb7633415c7143926971e4fa5121cbb0a", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8783724759996403, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 63.90887149973423, - "p95": 79.99267199920723, - "mean": 58.1968331539465 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.4134568939832098, - "reference_same_cases": null, - "median_ms": 63.90887149973423, - "detail": { - "field_accuracy": 0.532, - "case_exact_accuracy": 0.532, - "macro_f1": 0.4134568939832098, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.4134568939832098, - "skill": 0.26430335746565453, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4134568939832098, - "random": 0.20273782411694521, - "skill": 0.26430335746565453, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/esci-warm.json b/docs/training/jet-next/esci-warm.json deleted file mode 100644 index 1e03b2d2e475825ec24e54e2c5c702494a04c50a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/esci-warm.json +++ /dev/null @@ -1,158 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "360a9c1c12feb61d36783462013d09cc093b0d5d83c633ba416c2180db28997c", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8837185640004463, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 62.575644500611816, - "p95": 80.28408699829015, - "mean": 57.501974687904294 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.4147424168042725, - "reference_same_cases": null, - "median_ms": 62.575644500611816, - "detail": { - "field_accuracy": 0.56, - "case_exact_accuracy": 0.56, - "macro_f1": 0.4147424168042725, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.4147424168042725, - "skill": 0.2659157791507029, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4147424168042725, - "random": 0.20273782411694521, - "skill": 0.2659157791507029, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/isarcasm-en-baseline.json b/docs/training/jet-next/isarcasm-en-baseline.json deleted file mode 100644 index cca407f1c878b90d5294531ad5c1fbe06a7fd929..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/isarcasm-en-baseline.json +++ /dev/null @@ -1,156 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "fce07a83cfbc84cab673c9a0218e6d2bd3e32bbb08bfdce2ca586e68a322e9b5", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.120845952998934, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 11.024953000742244, - "p95": 40.01360600159387, - "mean": 12.74496764784739 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 11.024953000742244, - "detail": { - "field_accuracy": 0.6535714285714286, - "case_exact_accuracy": 0.6535714285714286, - "macro_f1": 0.4811847326191155, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.1821247892074199 - }, - "category_macro_f1": 0.1821247892074199 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.1821247892074199, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.1821247892074199, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/isarcasm-en-v3.json b/docs/training/jet-next/isarcasm-en-v3.json deleted file mode 100644 index f4bee19472933aa6cf4bdaf7402c62b7e3f7c2c7..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/isarcasm-en-v3.json +++ /dev/null @@ -1,160 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "f9aae1d9e7c2e7fbeb3751e8fb4b798572f22ac82b345a4377f21bdd29655571", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.878869845000736, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 25.58769000097527, - "p95": 54.116885003168136, - "mean": 26.95643487361719 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 25.58769000097527, - "detail": { - "field_accuracy": 0.7371428571428571, - "case_exact_accuracy": 0.7371428571428571, - "macro_f1": 0.5162822838386456, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.1894273127753304 - }, - "category_macro_f1": 0.1894273127753304 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.1894273127753304, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.1894273127753304, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/isarcasm-en-v4.json b/docs/training/jet-next/isarcasm-en-v4.json deleted file mode 100644 index 7b5522d9324a48f7cb2558f01aed2102a05f5ff5..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/isarcasm-en-v4.json +++ /dev/null @@ -1,160 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "46041659385e21e09f5e92f6bd0731c493a4595641c6c0022e56e1180c09437e", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8742223579974961, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 25.532456500513945, - "p95": 54.09158000111347, - "mean": 26.608081711388618 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 25.532456500513945, - "detail": { - "field_accuracy": 0.735, - "case_exact_accuracy": 0.735, - "macro_f1": 0.5388976333215262, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.23819301848049282 - }, - "category_macro_f1": 0.23819301848049282 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.23819301848049282, - "skill": 0.019949195832608534, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.23819301848049282, - "random": 0.2226862339379383, - "skill": 0.019949195832608534, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/isarcasm-en-warm.json b/docs/training/jet-next/isarcasm-en-warm.json deleted file mode 100644 index 5499e3ac70cc59bbb5ea86c4a68a3d40c723ebb2..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/isarcasm-en-warm.json +++ /dev/null @@ -1,160 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "3299a2f218a80735f0d00ad55108953983d942220e7992f4e9c4b61a96bfb0d3", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-warm-20260924" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 0.9439, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-warm-20260924", - "lora_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration_sha256": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8737670730006357, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 25.724428500325303, - "p95": 55.542318001243984, - "mean": 27.1736307021482 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 25.724428500325303, - "detail": { - "field_accuracy": 0.7857142857142857, - "case_exact_accuracy": 0.7857142857142857, - "macro_f1": 0.526335733232285, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.17582417582417584 - }, - "category_macro_f1": 0.17582417582417584 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.17582417582417584, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.17582417582417584, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-next/isarcasm-english-sample.json b/docs/training/jet-next/isarcasm-english-sample.json deleted file mode 100644 index 451d5835bad9ca686d0a5a33434cd74bff04a443..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/isarcasm-english-sample.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "requests": 1400, - "selection": "All available English track-A rows; no label-based selection", - "source_sha256": "8dd94acf6c313c50aec3fc48188c840d994d14f764b3fff9fe7a89e2e41b9211", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "official_exclusions_checked": true, - "exclusions_sha256": "331df32d4b719c7db43214d0e5d85859d39c3b2eb7d0b3812214cce150155e81", - "excluded_english_track_a": 0 -} \ No newline at end of file diff --git a/docs/training/jet-next/leaderboard-snapshot.json b/docs/training/jet-next/leaderboard-snapshot.json deleted file mode 100644 index e64fd19b704264216f6e9b7645fbbe16568e906e..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/leaderboard-snapshot.json +++ /dev/null @@ -1,27174 +0,0 @@ -{ - "generated_utc": "2026-09-22T11:12:47+00:00", - "suite": { - "requests": 132422, - "scoreable": 131980, - "benchmarks": 37, - "jev_benchmarks": 43, - "panel": 19, - "panel_full": 25, - "categories": 5, - "dropped": [ - "MiniWoB++", - "Boxoban", - "RTFM", - "ScienceWorld", - "Hanabi", - "Codenames" - ], - "hardware": "1 x NVIDIA RTX PRO 6000", - "jev_version": "jev-1.13.0", - "panel_id": "core25-observed-protocol-v1", - "calibration_note": "33 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT, RouterBench and ForecastBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option.", - "formulas": [ - { - "id": "balanced_raw", - "normalization": "raw", - "formula": "100*sum_c(0.2*category[c])" - }, - { - "id": "balanced_skill", - "normalization": "skill", - "formula": "100*sum_c(0.2*category[c])" - }, - { - "id": "breadth_skill", - "normalization": "skill", - "shift": 0.1, - "formula": "100*(product_c((0.1+0.9*category[c])**0.2)-0.1)/0.9" - } - ] - }, - "categories": [ - { - "id": "knowledge", - "label": "Knowledge & Reasoning", - "color": "#5976bd", - "icon": "M4 4h6a3 3 0 0 1 3 3v14a4 4 0 0 0-4-2H4z M20 4h-4a3 3 0 0 0-3 3v14a4 4 0 0 1 4-2h3z", - "description": "Academic knowledge, commonsense, mathematics, and code.", - "panel": [ - 24, - 25, - 30, - 43, - 44, - 31 - ], - "panel_full": [ - 24, - 25, - 30, - 43, - 44, - 31 - ], - "suite": [ - 24, - 25, - 26, - 27, - 28, - 29, - 30, - 32, - 33, - 34, - 43, - 44, - 45, - 31 - ], - "group": [ - 24, - 25, - 26, - 27, - 28, - 29, - 30, - 32, - 33, - 34, - 43, - 44, - 45, - 51, - 52 - ], - "priority": [ - 24, - 25, - 30, - 45, - 28 - ] - }, - { - "id": "language", - "label": "Language Understanding", - "color": "#407e9e", - "icon": "M4 4h16v12H9l-5 4z M8 8h8 M8 12h5", - "description": "Read a text and judge what it means: entailment, sentiment, sarcasm and stance.", - "panel": [ - 11, - 40, - 41 - ], - "panel_full": [ - 11, - 40, - 41 - ], - "suite": [ - 11, - 12, - 38, - 39, - 40, - 41, - 42 - ], - "group": [ - 11, - 12, - 38, - 39, - 40, - 41, - 42 - ], - "priority": [ - 40, - 42, - 39, - 38, - 11 - ] - }, - { - "id": "retrieval", - "label": "Retrieval & Classification", - "color": "#3e8e5e", - "icon": "M11 4a7 7 0 1 0 0 14 7 7 0 0 0 0-14z M20 20l-4.3-4.3", - "description": "Pick the right item from a large set: rank documents and products by relevance, route a message to one of many intents, sort out spam and phishing.", - "panel": [ - 36, - 37 - ], - "panel_full": [ - 36, - 37 - ], - "suite": [ - 36, - 37, - 4, - 5, - 10 - ], - "group": [ - 36, - 37, - 4, - 5, - 10, - 54, - 55, - 56 - ], - "priority": [ - 36, - 37, - 4, - 5, - 10, - 55 - ] - }, - { - "id": "tools", - "label": "Tools & Automation", - "color": "#b77a38", - "icon": "M5 4v11a4 4 0 0 0 4 4h10 M5 9h9a4 4 0 0 0 4-4 M2 7l3-3 3 3 m8 9 3 3-3 3", - "description": "Tool choice, model routing, household control, and browser or simulator actions.", - "panel": [ - 1, - 2, - 6 - ], - "panel_full": [ - 1, - 2, - 6, - 7, - 16 - ], - "suite": [ - 1, - 2, - 3, - 6, - 9 - ], - "group": [ - 1, - 2, - 3, - 6, - 9, - 53, - 7, - 16 - ], - "priority": [ - 1, - 9, - 6 - ] - }, - { - "id": "arts", - "label": "Arts & Human Judgment", - "color": "#b6698a", - "icon": "M12 3a9 9 0 1 0 0 18h2a2 2 0 0 0 0-4h-1a2 2 0 0 1 0-4h4a4 4 0 0 0 4-4c0-4-5-6-9-6 M7 8h.01 M12 6h.01 M17 8h.01 M6 13h.01", - "description": "Poetry, music, humor, human preferences, and forecasting.", - "panel": [ - 20, - 21, - 22, - 23, - 50 - ], - "panel_full": [ - 20, - 21, - 22, - 23, - 50 - ], - "suite": [ - 20, - 21, - 22, - 23, - 48, - 50 - ], - "group": [ - 20, - 21, - 22, - 23, - 48, - 50 - ], - "priority": [ - 20, - 22, - 21, - 23, - 48 - ] - }, - { - "id": "games", - "label": "Games & Strategy", - "color": "#ad6f48", - "icon": "M7 8h10a4 4 0 0 1 4 4l1 5a2 2 0 0 1-3 2l-3-3H8l-3 3a2 2 0 0 1-3-2l1-5a4 4 0 0 1 4-4z M7 10v4 M5 12h4 M16 11h.01 M18 13h.01", - "description": "Move selection, spatial planning, rules, and decisions against other players.", - "panel": [], - "panel_full": [ - 31 - ], - "suite": [], - "jev_only": true, - "folded_into": "knowledge", - "group": [ - 31, - 13, - 15, - 17, - 19 - ], - "priority": [ - 31 - ] - } - ], - "benchmarks": { - "1": { - "id": 1, - "dataset": "BFCL", - "short": "BFCL", - "metric": "case exact accuracy", - "explainer": "Choose which supplied tools are needed for a request. Tests tool selection, not arguments or execution.", - "baseline": { - "value": 0.2592, - "label": "Random whole-case guesses" - }, - "lower": false, - "cases": 1694, - "group": "tools", - "in_suite": true, - "interactive": false - }, - "2": { - "id": 2, - "dataset": "ToolRet", - "short": "ToolRet", - "metric": "nDCG@10", - "explainer": "Rank candidate tools by relevance to a user request after an initial keyword search.", - "baseline": { - "value": 0.0937, - "label": "Random ranking" - }, - "lower": false, - "cases": 7961, - "group": "tools", - "in_suite": true, - "interactive": false - }, - "3": { - "id": 3, - "dataset": "API-Bank", - "short": "API-Bank", - "metric": "accuracy", - "explainer": "Choose the next API to call from 53 tools using the conversation and earlier tool results.", - "baseline": { - "value": 0.0189, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 508, - "group": "tools", - "in_suite": true, - "interactive": false - }, - "4": { - "id": 4, - "dataset": "BANKING77", - "short": "BANKING77", - "metric": "macro-F1", - "explainer": "Identify the intent of a banking support message, such as a failed transfer or a missing card.", - "baseline": { - "value": 0.0127, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 3080, - "group": "retrieval", - "in_suite": true, - "interactive": false - }, - "5": { - "id": 5, - "dataset": "CLINC150+OOS", - "short": "CLINC150", - "metric": "macro-F1", - "explainer": "Classify a request into 150 assistant intents, or recognize that it is outside the supported scope.", - "baseline": { - "value": 0.006, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 5500, - "group": "retrieval", - "in_suite": true, - "interactive": false - }, - "6": { - "id": 6, - "dataset": "RouterBench", - "short": "RouterBench", - "metric": "selected quality (quality objective)", - "explainer": "Route a prompt to a model to maximize measured quality or balance quality with cost.", - "baseline": { - "value": 0.5735, - "label": "Random model selection" - }, - "lower": false, - "cases": 5001, - "group": "tools", - "in_suite": true, - "interactive": false - }, - "9": { - "id": 9, - "dataset": "Home appliance simulator", - "short": "Home appliances", - "metric": "case exact accuracy", - "explainer": "Translate a household command into device decisions. A request passes only if every required field is correct.", - "baseline": { - "value": 0.0, - "label": "Random whole-case guesses" - }, - "lower": false, - "cases": 160, - "group": "tools", - "in_suite": true, - "interactive": false - }, - "10": { - "id": 10, - "dataset": "SGD/SGD-X", - "short": "SGD", - "metric": "macro-F1", - "explainer": "Infer the user's current intent from a service dialogue. The service is supplied; this track evaluates SGD, not SGD-X or slot extraction.", - "baseline": { - "value": 0.399, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 2500, - "group": "retrieval", - "in_suite": true, - "interactive": false - }, - "11": { - "id": 11, - "dataset": "ContractNLI", - "short": "ContractNLI", - "metric": "macro-F1", - "explainer": "Read a contract and decide whether each of 17 legal statements is supported, contradicted, or unstated.", - "baseline": { - "value": 0.3085, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 123, - "group": "language", - "in_suite": true, - "interactive": false - }, - "12": { - "id": 12, - "dataset": "ANLI", - "short": "ANLI", - "metric": "macro-F1", - "explainer": "Decide whether a claim follows from, conflicts with, or is neutral to a premise designed to fool inference models.", - "baseline": { - "value": 0.3324, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 3200, - "group": "language", - "in_suite": true, - "interactive": false - }, - "20": { - "id": 20, - "dataset": "BPoMP", - "short": "BPoMP", - "metric": "accuracy", - "explainer": "Distinguish an original limerick from a perturbed version of a poem. This tests sensitivity to poetry structure, not aesthetic taste.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 811, - "group": "arts", - "in_suite": true, - "interactive": false - }, - "21": { - "id": 21, - "dataset": "Humicroedit", - "short": "Humicroedit", - "metric": "accuracy", - "explainer": "Choose the funnier of two edited news headlines, using human humor ratings as the answer key.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 2628, - "group": "arts", - "in_suite": true, - "interactive": false - }, - "22": { - "id": 22, - "dataset": "POP909-CL", - "short": "POP909", - "metric": "accuracy", - "explainer": "Recognize the annotated chord at a beat from symbolic musical notes, across 129 pitch-class categories.", - "baseline": { - "value": 0.0078, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 2000, - "group": "arts", - "in_suite": true, - "interactive": false - }, - "23": { - "id": 23, - "dataset": "cfcolor", - "short": "cfcolor", - "metric": "accuracy", - "explainer": "Predict which color palette a person preferred, using their earlier ratings and the palettes' RGB values.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 5000, - "group": "arts", - "in_suite": true, - "interactive": false - }, - "24": { - "id": 24, - "dataset": "MMLU", - "short": "MMLU", - "metric": "accuracy", - "explainer": "Answer four-choice questions across 57 academic subjects, from history and law to science and medicine.", - "baseline": { - "value": 0.25, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 14042, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "25": { - "id": 25, - "dataset": "GPQA Diamond", - "short": "GPQA Diamond", - "metric": "accuracy", - "explainer": "Answer 198 graduate-level science questions in physics, chemistry, and biology, with four options each.", - "baseline": { - "value": 0.25, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 198, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "26": { - "id": 26, - "dataset": "ARC-Easy", - "short": "ARC-Easy", - "metric": "accuracy", - "explainer": "Answer the easier split of school science multiple-choice questions.", - "baseline": { - "value": 0.2502, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 2376, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "27": { - "id": 27, - "dataset": "ARC-Challenge", - "short": "ARC-Challenge", - "metric": "accuracy", - "explainer": "Answer the more difficult split of school science questions that simple retrieval methods struggle with.", - "baseline": { - "value": 0.2502, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 1172, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "28": { - "id": 28, - "dataset": "WinoGrande", - "short": "WinoGrande", - "metric": "accuracy", - "explainer": "Use commonsense context to choose which of two words correctly completes an ambiguous sentence.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 1267, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "29": { - "id": 29, - "dataset": "HellaSwag", - "short": "HellaSwag", - "metric": "accuracy", - "explainer": "Choose the most plausible continuation of an everyday scene from four candidate endings.", - "baseline": { - "value": 0.25, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 10042, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "30": { - "id": 30, - "dataset": "GSM8K", - "short": "GSM8K", - "metric": "accuracy", - "explainer": "Solve grade-school arithmetic word problems by selecting a numeric answer from four or ten choices.", - "baseline": { - "value": 0.175, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 1319, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "31": { - "id": 31, - "dataset": "ChessBench", - "short": "ChessBench", - "metric": "accuracy", - "explainer": "Choose the highest-valued legal chess move using published reference values. All legal moves are offered and ties count.", - "baseline": { - "value": 0.0819, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 5000, - "group": "games", - "in_suite": true, - "interactive": false - }, - "32": { - "id": 32, - "dataset": "MuSR", - "short": "MuSR", - "metric": "accuracy", - "explainer": "Combine clues across long narratives to solve murder mysteries, track objects, or allocate people to teams.", - "baseline": { - "value": 0.371, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 756, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "33": { - "id": 33, - "dataset": "SATA-Bench", - "short": "SATA-Bench", - "metric": "case exact accuracy", - "explainer": "Select every correct answer to a question. The entire answer set must match; partial selections do not pass.", - "baseline": { - "value": 0.0131, - "label": "Random whole-case guesses" - }, - "lower": false, - "cases": 1650, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "34": { - "id": 34, - "dataset": "SimpleBench", - "short": "SimpleBench", - "metric": "accuracy", - "explainer": "Answer deceptively simple commonsense and trick questions. Only the ten public examples are evaluated here.", - "baseline": { - "value": 0.1667, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 10, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "36": { - "id": 36, - "dataset": "BRIGHT", - "short": "BRIGHT", - "metric": "nDCG@10", - "explainer": "Rank documents that answer reasoning-intensive queries, within a candidate pool retrieved by keyword search.", - "baseline": { - "value": 0.0448, - "label": "Random ranking" - }, - "lower": false, - "cases": 1384, - "group": "retrieval", - "in_suite": true, - "interactive": false - }, - "37": { - "id": 37, - "dataset": "Amazon ESCI", - "short": "Amazon ESCI", - "metric": "macro-F1", - "explainer": "Judge whether a product is an exact match, substitute, complement, or irrelevant to a shopping query.", - "baseline": { - "value": 0.2027, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 5000, - "group": "retrieval", - "in_suite": true, - "interactive": false - }, - "38": { - "id": 38, - "dataset": "ACOS", - "short": "ACOS", - "metric": "case exact accuracy", - "explainer": "Identify which aspect-and-sentiment combinations appear in a review. Every combination must be right for the review to pass.", - "baseline": { - "value": 0.0, - "label": "Random whole-case guesses" - }, - "lower": false, - "cases": 1399, - "group": "language", - "in_suite": true, - "interactive": false - }, - "39": { - "id": 39, - "dataset": "FinEntity", - "short": "FinEntity", - "metric": "macro-F1", - "explainer": "Judge the sentiment expressed toward a specified company or financial entity in a document.", - "baseline": { - "value": 0.3201, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 979, - "group": "language", - "in_suite": true, - "interactive": false - }, - "40": { - "id": 40, - "dataset": "iSarcasmEval", - "short": "iSarcasmEval", - "metric": "Sarcasm F1 \u00b7 track A, English", - "explainer": "Decide whether a single English text is sarcastic, scored with F1 for the sarcastic class. Headline track; the Arabic and paired-text tracks are listed under each result.", - "baseline": { - "value": 0.2227, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 4600, - "group": "language", - "in_suite": true, - "interactive": false - }, - "41": { - "id": 41, - "dataset": "VAST", - "short": "VAST", - "metric": "macro-F1", - "explainer": "Classify a text's stance toward a supplied topic as supportive, opposed, or neutral.", - "baseline": { - "value": 0.3333, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 3006, - "group": "language", - "in_suite": true, - "interactive": false - }, - "42": { - "id": 42, - "dataset": "NLI4CT", - "short": "NLI4CT", - "metric": "macro-F1", - "explainer": "Determine whether clinical-trial report sections support or contradict a statement, including deliberately altered statements.", - "baseline": { - "value": 0.486, - "label": "Random choices \u00b7 F1" - }, - "lower": false, - "cases": 5500, - "group": "language", - "in_suite": true, - "interactive": false - }, - "43": { - "id": 43, - "dataset": "CRUXEval", - "short": "CRUXEval", - "metric": "accuracy", - "explainer": "Predict a short Python function's output by selecting among candidate values drawn from prior model generations.", - "baseline": { - "value": 0.3697, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 570, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "44": { - "id": 44, - "dataset": "CLadder", - "short": "CLadder", - "metric": "accuracy", - "explainer": "Answer yes/no questions about statistical association, intervention, and counterfactuals from supplied causal information.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 5000, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "45": { - "id": 45, - "dataset": "HLE", - "short": "HLE", - "metric": "accuracy", - "explainer": "Answer the text-only multiple-choice subset of expert-level Humanity's Last Exam. Images and free-response questions are excluded.", - "baseline": { - "value": 0.1641, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 513, - "group": "knowledge", - "in_suite": true, - "interactive": false - }, - "48": { - "id": 48, - "dataset": "ForecastBench", - "short": "ForecastBench", - "metric": "Brier (lower is better)", - "explainer": "Assign probabilities to events resolved in July\u2013September 2026 using frozen earlier context. Lower Brier loss is better.", - "baseline": { - "value": 0.25, - "label": "Always predict p = 0.5" - }, - "lower": true, - "cases": 10139, - "group": "arts", - "in_suite": true, - "interactive": false - }, - "50": { - "id": 50, - "dataset": "Habermas Machine", - "short": "Habermas", - "metric": "accuracy", - "explainer": "Choose the consensus statement a human discussion group preferred, based on members' views and their recorded rankings.", - "baseline": { - "value": 0.311, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 1676, - "group": "arts", - "in_suite": true, - "interactive": false - }, - "51": { - "id": 51, - "dataset": "OpenBookQA", - "short": "OpenBookQA", - "metric": "accuracy", - "explainer": "Answer elementary-science questions from four original choices, without the supporting fact.", - "baseline": { - "value": 0.25, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 500, - "group": "knowledge", - "in_suite": false, - "interactive": false - }, - "52": { - "id": 52, - "dataset": "CommonsenseQA", - "short": "CommonsenseQA", - "metric": "accuracy", - "explainer": "Answer everyday commonsense questions from five original choices.", - "baseline": { - "value": 0.2, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 1221, - "group": "knowledge", - "in_suite": false, - "interactive": false - }, - "53": { - "id": 53, - "dataset": "Support-ticket calibration", - "short": "Support tickets", - "metric": "accuracy", - "explainer": "Judge a support ticket for queue, anger and priority, against deliberately noisy labels.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": 300, - "group": "tools", - "in_suite": false, - "interactive": false - }, - "54": { - "id": 54, - "dataset": "Phishing difficulty gradient", - "short": "Phishing gradient", - "metric": "accuracy", - "explainer": "Judge generated emails as phishing across four difficulty tiers, from obvious to misleading.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 800, - "group": "retrieval", - "in_suite": false, - "interactive": false - }, - "55": { - "id": 55, - "dataset": "Email spam classification", - "short": "Email spam", - "metric": "accuracy", - "explainer": "Detect spam in real emails, scored on the structured-focus question wording.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 3000, - "group": "retrieval", - "in_suite": false, - "interactive": false - }, - "56": { - "id": 56, - "dataset": "PhishNChips phishing decisions", - "short": "PhishNChips", - "metric": "accuracy", - "explainer": "Judge whether an email and its links are malicious, scored on the main verdict.", - "baseline": { - "value": 0.5, - "label": "Uniform random choice" - }, - "lower": false, - "cases": 2000, - "group": "retrieval", - "in_suite": false, - "interactive": false - }, - "7": { - "id": 7, - "dataset": "MiniWoB++", - "short": "MiniWoB++", - "metric": "episode success", - "explainer": "Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": null, - "group": "tools", - "in_suite": false, - "interactive": true - }, - "16": { - "id": 16, - "dataset": "ScienceWorld", - "short": "ScienceWorld", - "metric": "episode success", - "explainer": "Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": null, - "group": "tools", - "in_suite": false, - "interactive": true - }, - "13": { - "id": 13, - "dataset": "Boxoban", - "short": "Boxoban", - "metric": "episode success", - "explainer": "Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": null, - "group": "games", - "in_suite": false, - "interactive": true - }, - "15": { - "id": 15, - "dataset": "RTFM", - "short": "RTFM", - "metric": "episode success", - "explainer": "Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": null, - "group": "games", - "in_suite": false, - "interactive": true - }, - "17": { - "id": 17, - "dataset": "Hanabi", - "short": "Hanabi", - "metric": "episode success", - "explainer": "Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": null, - "group": "games", - "in_suite": false, - "interactive": true - }, - "19": { - "id": 19, - "dataset": "Codenames", - "short": "Codenames", - "metric": "episode success", - "explainer": "Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.", - "baseline": { - "value": null, - "label": null - }, - "lower": false, - "cases": null, - "group": "games", - "in_suite": false, - "interactive": true - } - }, - "jev": { - "engine": "jev", - "name": "Jev", - "version": "jev-1.13.0", - "edition": "Full release v1", - "results": { - "7": { - "score": 1.0, - "metric": "episode success", - "cases": 9, - "median_ms": null, - "p95_ms": null, - "usd": null - }, - "16": { - "score": 0.067, - "metric": "episode success", - "cases": 10, - "median_ms": null, - "p95_ms": null, - "usd": null - }, - "13": { - "score": 0.0, - "metric": "levels solved", - "cases": 100, - "median_ms": null, - "p95_ms": null, - "usd": null - }, - "15": { - "score": 0.17, - "metric": "episode success", - "cases": 100, - "median_ms": null, - "p95_ms": null, - "usd": null - }, - "17": { - "score": 0.0, - "metric": "episode success", - "cases": 100, - "median_ms": null, - "p95_ms": null, - "usd": null - }, - "19": { - "score": 0.0, - "metric": "episode success", - "cases": 66, - "median_ms": null, - "p95_ms": null, - "usd": null - }, - "1": { - "score": 0.9575, - "metric": "case exact accuracy", - "cases": 1694, - "median_ms": 265.7, - "p95_ms": 382.1, - "usd": 0.1 - }, - "2": { - "score": 0.4502, - "metric": "nDCG@10", - "cases": 7961, - "median_ms": 328.1, - "p95_ms": 600.6, - "usd": 3.781 - }, - "3": { - "score": 0.8819, - "metric": "accuracy", - "cases": 508, - "median_ms": 351.4, - "p95_ms": 461.7, - "usd": 0.216 - }, - "4": { - "score": 0.7974, - "metric": "macro-F1", - "cases": 3080, - "median_ms": 265.6, - "p95_ms": 439.1, - "usd": 0.222 - }, - "5": { - "score": 0.8927, - "metric": "macro-F1", - "cases": 5500, - "median_ms": 247.5, - "p95_ms": 399.3, - "usd": 0.644 - }, - "6": { - "score": 0.7992, - "metric": "selected quality (quality objective)", - "cases": 5001, - "median_ms": 238.9, - "p95_ms": 370.9, - "usd": 1.3 - }, - "9": { - "score": 0.525, - "metric": "case exact accuracy", - "cases": 160, - "median_ms": 275.6, - "p95_ms": 360.6, - "usd": 0.023 - }, - "10": { - "score": 0.4295, - "metric": "macro-F1", - "cases": 2500, - "median_ms": 265.2, - "p95_ms": 457.0, - "usd": 0.19 - }, - "11": { - "score": 0.7169, - "metric": "macro-F1", - "cases": 123, - "median_ms": 306.2, - "p95_ms": 415.5, - "usd": 0.023 - }, - "12": { - "score": 0.7479, - "metric": "macro-F1", - "cases": 3200, - "median_ms": 259.2, - "p95_ms": 434.6, - "usd": 0.058 - }, - "20": { - "score": 0.906, - "metric": "accuracy", - "cases": 811, - "median_ms": 235.3, - "p95_ms": 392.4, - "usd": 0.091 - }, - "21": { - "score": 0.6187, - "metric": "accuracy", - "cases": 2628, - "median_ms": 261.9, - "p95_ms": 433.0, - "usd": 0.038 - }, - "22": { - "score": 0.181, - "metric": "accuracy", - "cases": 2000, - "median_ms": 301.8, - "p95_ms": 476.3, - "usd": 0.641 - }, - "23": { - "score": 0.6474, - "metric": "accuracy", - "cases": 5000, - "median_ms": 239.9, - "p95_ms": 400.8, - "usd": 0.192 - }, - "24": { - "score": 0.9173, - "metric": "accuracy", - "cases": 14042, - "median_ms": 232.5, - "p95_ms": 444.6, - "usd": 0.246 - }, - "25": { - "score": 0.7828, - "metric": "accuracy", - "cases": 198, - "median_ms": 253.8, - "p95_ms": 344.9, - "usd": 0.004 - }, - "26": { - "score": 0.9933, - "metric": "accuracy", - "cases": 2376, - "median_ms": 260.8, - "p95_ms": 411.8, - "usd": 0.036 - }, - "27": { - "score": 0.9778, - "metric": "accuracy", - "cases": 1172, - "median_ms": 258.1, - "p95_ms": 372.6, - "usd": 0.018 - }, - "28": { - "score": 0.9195, - "metric": "accuracy", - "cases": 1267, - "median_ms": 259.3, - "p95_ms": 374.6, - "usd": 0.018 - }, - "29": { - "score": 0.9452, - "metric": "accuracy", - "cases": 10042, - "median_ms": 231.0, - "p95_ms": 364.6, - "usd": 0.213 - }, - "30": { - "score": 0.7987, - "metric": "accuracy", - "cases": 1319, - "median_ms": 264.2, - "p95_ms": 412.1, - "usd": 0.055 - }, - "31": { - "score": 0.1722, - "metric": "accuracy", - "cases": 5000, - "median_ms": 241.0, - "p95_ms": 396.1, - "usd": 0.302 - }, - "32": { - "score": 0.6601, - "metric": "accuracy", - "cases": 756, - "median_ms": 255.7, - "p95_ms": 366.7, - "usd": 0.041 - }, - "33": { - "score": 0.2642, - "metric": "case exact accuracy", - "cases": 1650, - "median_ms": 267.3, - "p95_ms": 381.3, - "usd": 0.11 - }, - "34": { - "score": 0.5, - "metric": "accuracy", - "cases": 10, - "median_ms": 240.0, - "p95_ms": 403.6, - "usd": 0.0 - }, - "36": { - "score": 0.1869, - "metric": "nDCG@10", - "cases": 1384, - "median_ms": 404.8, - "p95_ms": 620.9, - "usd": 0.889 - }, - "37": { - "score": 0.5521, - "metric": "macro-F1", - "cases": 5000, - "median_ms": 236.5, - "p95_ms": 396.7, - "usd": 0.162 - }, - "38": { - "score": 0.0558, - "metric": "case exact accuracy", - "cases": 1399, - "median_ms": 342.2, - "p95_ms": 531.6, - "usd": 1.658 - }, - "39": { - "score": 0.8698, - "metric": "macro-F1", - "cases": 979, - "median_ms": 259.2, - "p95_ms": 371.1, - "usd": 0.023 - }, - "40": { - "score": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "cases": 1400, - "median_ms": 243.7, - "p95_ms": 401.3, - "usd": 0.08, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.4449, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5051, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.7, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.95, - "headline": false - } - ] - }, - "41": { - "score": 0.6463, - "metric": "macro-F1", - "cases": 3006, - "median_ms": 262.6, - "p95_ms": 437.5, - "usd": 0.057 - }, - "42": { - "score": 0.8406, - "metric": "macro-F1", - "cases": 5500, - "median_ms": 238.5, - "p95_ms": 399.4, - "usd": 0.221 - }, - "43": { - "score": 0.7298, - "metric": "accuracy", - "cases": 570, - "median_ms": 250.9, - "p95_ms": 360.9, - "usd": 0.011 - }, - "44": { - "score": 0.7264, - "metric": "accuracy", - "cases": 5000, - "median_ms": 238.0, - "p95_ms": 392.3, - "usd": 0.09 - }, - "45": { - "score": 0.2008, - "metric": "accuracy", - "cases": 513, - "median_ms": 254.2, - "p95_ms": 347.5, - "usd": 0.016 - }, - "48": { - "score": 0.1736, - "metric": "Brier (lower is better)", - "cases": 10139, - "median_ms": 233.9, - "p95_ms": 368.3, - "usd": 0.499 - }, - "50": { - "score": 0.4594, - "metric": "accuracy", - "cases": 1676, - "median_ms": 264.0, - "p95_ms": 377.9, - "usd": 0.084 - }, - "51": { - "score": 0.94, - "metric": "accuracy", - "cases": 500, - "median_ms": 1411.9, - "p95_ms": 2198.7, - "usd": 0.0 - }, - "52": { - "score": 0.8747, - "metric": "accuracy", - "cases": 1221, - "median_ms": 1387.5, - "p95_ms": 2570.8, - "usd": 0.0 - }, - "53": { - "score": 0.905, - "metric": "accuracy", - "cases": 300, - "median_ms": 1383.9, - "p95_ms": 2379.8, - "usd": 0.0 - }, - "54": { - "score": 0.9463, - "metric": "accuracy", - "cases": 800, - "median_ms": 1840.8, - "p95_ms": 3083.4, - "usd": 0.0 - }, - "55": { - "score": 0.9743, - "metric": "accuracy", - "cases": 3000, - "median_ms": 1382.0, - "p95_ms": 2622.4, - "usd": 0.0 - }, - "56": { - "score": 0.6106, - "metric": "accuracy", - "cases": 2000, - "median_ms": 1372.4, - "p95_ms": 2612.8, - "usd": 0.0 - } - }, - "latency": { - "median": 252.8, - "p95": 436.6, - "note": "Remote hosted API measured over HTTP. total_wall_ms includes client retries and network; http_wall_ms is the request itself. Local GPU reproductions have no network leg, so these are not a controlled speed comparison." - }, - "interactive": [ - { - "track": "miniwob", - "success_rate": 1.0, - "episodes": 9 - }, - { - "track": "minigrid_babyai", - "success_rate": 0.0833, - "episodes": 12 - }, - { - "track": "scienceworld", - "success_rate": 0.0, - "episodes": 10 - }, - { - "track": "hanabi", - "success_rate": null, - "episodes": 100 - }, - { - "track": "leduc", - "success_rate": null, - "episodes": 300 - }, - { - "track": "codenames", - "success_rate": 0.0, - "episodes": 66 - }, - { - "track": "rtfm", - "success_rate": 0.17, - "episodes": 100 - } - ], - "references": [ - { - "catalog_id": 25, - "model": "Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)", - "score": 0.9374, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)", - "score": 0.5913, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "GPT-6 Astra (max)", - "score": 0.9606, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "GPT-6 Astra (max)", - "score": 0.5468, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Claude Opus 5 (Adaptive Reasoning, Max Effort)", - "score": 0.9323, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Claude Opus 5 (Adaptive Reasoning, Max Effort)", - "score": 0.5487, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)", - "score": 0.9263, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)", - "score": 0.5547, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Muse Spark 1.3 (max)", - "score": 0.9354, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Muse Spark 1.3 (max)", - "score": 0.487, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "GPT-5.6 Sol (max)", - "score": 0.9414, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "GPT-5.6 Sol (max)", - "score": 0.4949, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Qwen3.8 Max (0902)", - "score": 0.9283, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Qwen3.8 Max (0902)", - "score": 0.431, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "GLM-5.3 (max)", - "score": 0.9172, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "GLM-5.3 (max)", - "score": 0.4226, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Grok 4.6 (high)", - "score": 0.9495, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Grok 4.6 (high)", - "score": 0.4291, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Kimi K3 (max)", - "score": 0.9354, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Kimi K3 (max)", - "score": 0.469, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Step 5 Preview", - "score": 0.4648, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "GPT-5.6 Terra (max)", - "score": 0.9253, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "GPT-5.6 Terra (max)", - "score": 0.4291, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "GLM 5.3 Flash", - "score": 0.9121, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "GLM 5.3 Flash", - "score": 0.3985, - "metric": "Accuracy" - }, - { - "catalog_id": 25, - "model": "Gemini 3.8 Flash (high)", - "score": 0.9525, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "Gemini 3.8 Flash (high)", - "score": 0.4782, - "metric": "Accuracy" - }, - { - "catalog_id": 45, - "model": "DeepSeek V4.1 Flash (Reasoning, Max Effort)", - "score": 0.3925, - "metric": "Accuracy" - }, - { - "catalog_id": 34, - "model": "Claude Fable 5", - "score": 0.819, - "metric": "AVG@5 accuracy" - }, - { - "catalog_id": 34, - "model": "Claude Opus 5", - "score": 0.806, - "metric": "AVG@5 accuracy" - } - ], - "summary_notes": [ - "Metrics exclude failed/incomplete cases; coverage is explicit. No silent replacement of failures.", - "Different primary metrics are not averaged across benchmarks.", - "Latency includes HTTPS and concurrent service scheduling; not isolated model inference.", - "Full raw HTTP responses/payloads and all retry timings are in http-attempts.jsonl.", - "BPoMP/RouterBench/GSM8K and chunked tasks retain all linked requests for sampled cases." - ], - "calibration": { - "acc": 0.7472, - "conf": 0.8106, - "ece": 0.0651, - "brier": 0.3476, - "over95": 0.0201, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0002, - "conf": 0.082, - "acc": 0.4 - }, - { - "lo": 0.1, - "w": 0.0181, - "conf": 0.1649, - "acc": 0.0998 - }, - { - "lo": 0.2, - "w": 0.0303, - "conf": 0.2424, - "acc": 0.2013 - }, - { - "lo": 0.3, - "w": 0.0266, - "conf": 0.3493, - "acc": 0.3792 - }, - { - "lo": 0.4, - "w": 0.0307, - "conf": 0.4512, - "acc": 0.3783 - }, - { - "lo": 0.5, - "w": 0.0799, - "conf": 0.5464, - "acc": 0.4796 - }, - { - "lo": 0.6, - "w": 0.0784, - "conf": 0.6436, - "acc": 0.5217 - }, - { - "lo": 0.7, - "w": 0.1128, - "conf": 0.7555, - "acc": 0.7005 - }, - { - "lo": 0.8, - "w": 0.1005, - "conf": 0.8488, - "acc": 0.708 - }, - { - "lo": 0.9, - "w": 0.5225, - "conf": 0.9808, - "acc": 0.9342 - } - ], - "areas": { - "knowledge": { - "ece": 0.0516, - "acc": 0.769, - "conf": 0.7993 - }, - "language": { - "ece": 0.0837, - "acc": 0.7898, - "conf": 0.8734 - }, - "retrieval": { - "ece": 0.1536, - "acc": 0.6685, - "conf": 0.8206 - }, - "tools": { - "ece": 0.0196, - "acc": 0.9458, - "conf": 0.9534 - }, - "arts": { - "ece": 0.0934, - "acc": 0.5701, - "conf": 0.6605 - } - } - }, - "scores": { - "balanced_raw": 59.51, - "balanced_skill": 46.26, - "breadth_skill": 44.79 - }, - "frozen_scores": { - "balanced_raw": 52.49, - "balanced_skill": 41.75, - "breadth_skill": 35.57 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.6888, - "skill": 0.5811, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.6228, - "skill": 0.4745, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.3701, - "skill": 0.2943, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7343, - "skill": 0.6194, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.5595, - "skill": 0.3438, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9575, - "skill": 0.9426, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4461, - "skill": 0.3891, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7992, - "skill": 0.5266, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.786, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8125, - "headline": false - } - ] - }, - "11": { - "raw": 0.7169, - "skill": 0.5906, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.9092, - "skill": 0.8184, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.6187, - "skill": 0.2374, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.1658, - "skill": 0.1593, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.6441, - "skill": 0.2883, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.9199, - "skill": 0.8932, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.7857, - "skill": 0.7143, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.7987, - "skill": 0.7565, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.8218, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.7756, - "headline": false - } - ] - }, - "31": { - "raw": 0.1722, - "skill": 0.0984, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1881, - "skill": 0.1503, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.5521, - "skill": 0.4382, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.5051, - "skill": 0.3633, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.4449, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5051, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.7, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.95, - "headline": false - } - ] - }, - "41": { - "raw": 0.6463, - "skill": 0.4695, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.7298, - "skill": 0.5713, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.7264, - "skill": 0.4528, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4594, - "skill": 0.2154, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 1.0, - "skill": 1.0, - "coverage": 1.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.067, - "skill": 0.067, - "coverage": 1.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 1.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.17, - "skill": 0.17, - "coverage": 1.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 1.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 1.0, - "random": null, - "tracks": [] - } - }, - "coverage": 1.0, - "pending": 0.0 - }, - "models": [ - { - "engine": "akash-gemma", - "name": "system-one-gemma", - "long_name": "system-one-gemma", - "variant": null, - "meta": { - "kind": "LoRA + head", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 268098176, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/akash-kamat/system-one-gemma", - "base_model": "google/gemma-3-270m", - "base_url": "https://huggingface.co/google/gemma-3-270m", - "weights_repo": null, - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 84222, - "unsupported": 48200 - }, - "calibration": { - "acc": 0.3613, - "conf": 0.5497, - "ece": 0.234, - "brier": 0.7692, - "over95": 0.0178, - "n": 66496, - "benchmarks": 28, - "rel": [ - { - "lo": 0.0, - "w": 0.0293, - "conf": 0.0558, - "acc": 0.0622 - }, - { - "lo": 0.1, - "w": 0.0497, - "conf": 0.1654, - "acc": 0.0307 - }, - { - "lo": 0.2, - "w": 0.0671, - "conf": 0.2547, - "acc": 0.5915 - }, - { - "lo": 0.3, - "w": 0.1075, - "conf": 0.3571, - "acc": 0.2441 - }, - { - "lo": 0.4, - "w": 0.1332, - "conf": 0.449, - "acc": 0.2831 - }, - { - "lo": 0.5, - "w": 0.2063, - "conf": 0.5477, - "acc": 0.4257 - }, - { - "lo": 0.6, - "w": 0.1424, - "conf": 0.6459, - "acc": 0.3856 - }, - { - "lo": 0.7, - "w": 0.1149, - "conf": 0.7466, - "acc": 0.3498 - }, - { - "lo": 0.8, - "w": 0.0813, - "conf": 0.8477, - "acc": 0.4908 - }, - { - "lo": 0.9, - "w": 0.0682, - "conf": 0.9509, - "acc": 0.4606 - } - ], - "areas": { - "knowledge": { - "ece": 0.2371, - "acc": 0.3592, - "conf": 0.4966 - }, - "language": { - "ece": 0.3453, - "acc": 0.3021, - "conf": 0.6474 - }, - "retrieval": { - "ece": 0.215, - "acc": 0.3579, - "conf": 0.5707 - }, - "tools": { - "ece": 0.1536, - "acc": 0.76, - "conf": 0.6064 - }, - "arts": { - "ece": 0.1467, - "acc": 0.3745, - "conf": 0.5212 - } - } - }, - "gaps": { - "share": 0.364, - "reasons": [ - { - "reason": "prompt longer than its 384-token context window", - "share": 0.364 - } - ] - }, - "latency": { - "median": 32.1, - "p95": 1272.8, - "mean": 162.3 - }, - "results": { - "1": { - "score": 0.8065, - "jev": 0.9896, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 672, - "unsupported": 1022, - "errors": 0, - "median_ms": 32.0 - }, - "2": { - "score": 0.0926, - "jev": 0.472, - "metric": "nDCG@10", - "requests": 7704, - "answered": 848, - "unsupported": 6856, - "errors": 0, - "median_ms": 1021.4 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.5347, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 47.7 - }, - "5": { - "score": 0.2486, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 85.5 - }, - "6": { - "score": null, - "jev": null, - "metric": null, - "requests": 10000, - "answered": 0, - "unsupported": 10000, - "errors": 0, - "median_ms": null - }, - "9": { - "score": null, - "jev": null, - "metric": null, - "requests": 160, - "answered": 0, - "unsupported": 160, - "errors": 0, - "median_ms": null - }, - "10": { - "score": null, - "jev": null, - "metric": null, - "requests": 2500, - "answered": 0, - "unsupported": 2500, - "errors": 0, - "median_ms": null - }, - "11": { - "score": 0.188, - "jev": 0.5043, - "metric": "macro-F1", - "requests": 123, - "answered": 1, - "unsupported": 122, - "errors": 0, - "median_ms": 583.2 - }, - "12": { - "score": 0.2389, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 32.4 - }, - "20": { - "score": 0.5776, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 31.9 - }, - "21": { - "score": 0.4726, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 29.1 - }, - "22": { - "score": 0, - "jev": 0.1667, - "metric": "accuracy", - "requests": 2000, - "answered": 12, - "unsupported": 1988, - "errors": 0, - "median_ms": 203.6 - }, - "23": { - "score": 0.5111, - "jev": 0.6056, - "metric": "accuracy", - "requests": 5000, - "answered": 180, - "unsupported": 4820, - "errors": 0, - "median_ms": 31.9 - }, - "24": { - "score": 0.3076, - "jev": 0.9176, - "metric": "accuracy", - "requests": 14033, - "answered": 13817, - "unsupported": 216, - "errors": 0, - "median_ms": 31.6 - }, - "25": { - "score": 0.2448, - "jev": 0.7865, - "metric": "accuracy", - "requests": 196, - "answered": 192, - "unsupported": 4, - "errors": 0, - "median_ms": 32.3 - }, - "26": { - "score": 0.4465, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 31.5 - }, - "27": { - "score": 0.3097, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 31.7 - }, - "28": { - "score": 0.4988, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 29.1 - }, - "29": { - "score": 0.2841, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 32.4 - }, - "30": { - "score": 0.1892, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 32.0 - }, - "31": { - "score": 0.0912, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 64.1 - }, - "32": { - "score": null, - "jev": null, - "metric": null, - "requests": 752, - "answered": 0, - "unsupported": 752, - "errors": 0, - "median_ms": null - }, - "33": { - "score": 0.0015, - "jev": 0.3309, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 686, - "unsupported": 964, - "errors": 0, - "median_ms": 239.9 - }, - "34": { - "score": 0.2, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 33.4 - }, - "36": { - "score": 0.0071, - "jev": 0.031, - "metric": "nDCG@10", - "requests": 1297, - "answered": 88, - "unsupported": 1209, - "errors": 0, - "median_ms": 1034.1 - }, - "37": { - "score": 0.2143, - "jev": 0.5432, - "metric": "macro-F1", - "requests": 5000, - "answered": 3212, - "unsupported": 1788, - "errors": 0, - "median_ms": 32.6 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 1880.2 - }, - "39": { - "score": 0.13, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 59.3 - }, - "40": { - "score": 0.2506, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 28.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2466, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2506, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.335, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.345, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3489, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 30.2 - }, - "42": { - "score": 0.418, - "jev": 0.8661, - "metric": "macro-F1", - "requests": 5500, - "answered": 2433, - "unsupported": 3067, - "errors": 0, - "median_ms": 29.8 - }, - "43": { - "score": 0.3409, - "jev": 0.7293, - "metric": "accuracy", - "requests": 570, - "answered": 569, - "unsupported": 1, - "errors": 0, - "median_ms": 32.1 - }, - "44": { - "score": 0.502, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 29.9 - }, - "45": { - "score": 0.1221, - "jev": 0.2207, - "metric": "accuracy", - "requests": 501, - "answered": 426, - "unsupported": 75, - "errors": 0, - "median_ms": 33.0 - }, - "48": { - "score": 0.5227, - "jev": 0.2151, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 79, - "unsupported": 10060, - "errors": 0, - "median_ms": 32.8 - }, - "50": { - "score": 0.1818, - "jev": 0.4545, - "metric": "accuracy", - "requests": 1676, - "answered": 11, - "unsupported": 1665, - "errors": 0, - "median_ms": 35.1 - } - }, - "scores": { - "balanced_raw": 17.09, - "balanced_skill": 1.83, - "breadth_skill": 1.79 - }, - "frozen_scores": { - "balanced_raw": 15.22, - "balanced_skill": 1.44, - "breadth_skill": 1.42 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2773, - "skill": 0.018, - "coverage": 0.9937, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2004, - "skill": 0.0198, - "coverage": 0.6694, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.0554, - "skill": 0.0, - "coverage": 0.3551, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.11, - "skill": 0.0273, - "coverage": 0.1689, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.2113, - "skill": 0.0262, - "coverage": 0.4097, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.32, - "skill": 0.082, - "coverage": 0.3967, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0102, - "skill": 0.0, - "coverage": 0.1101, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0, - "headline": false - } - ] - }, - "11": { - "raw": 0.0016, - "skill": 0.0, - "coverage": 0.0081, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5654, - "skill": 0.1309, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4726, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.006, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.0173, - "skill": 0.0, - "coverage": 0.036, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.3015, - "skill": 0.0686, - "coverage": 0.9846, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2398, - "skill": 0.0, - "coverage": 0.9796, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.1892, - "skill": 0.0254, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2881, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.0902, - "headline": false - } - ] - }, - "31": { - "raw": 0.0912, - "skill": 0.0102, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0005, - "skill": 0.0, - "coverage": 0.0678, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1102, - "skill": 0.0, - "coverage": 0.6424, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2506, - "skill": 0.036, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2466, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2506, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.335, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.345, - "headline": false - } - ] - }, - "41": { - "raw": 0.3489, - "skill": 0.0234, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3404, - "skill": 0.0, - "coverage": 0.9982, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.502, - "skill": 0.004, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.0012, - "skill": 0.0, - "coverage": 0.0066, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.4494, - "pending": 0.24 - }, - { - "engine": "decider-2b-fp8-http", - "name": "Decider 2B", - "long_name": "Decider 2B", - "variant": "FP8", - "meta": { - "kind": "full fine-tune", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 2274069824, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 3763692048, - "model_url": "https://huggingface.co/Mapika/decider-2b", - "code_url": null, - "base_model": "Qwen/Qwen3.5-2B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-2B-Base", - "weights_repo": "Mapika/decider-2b", - "served_checkpoint": "Mapika/decider-2b" - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.5681, - "conf": 0.6566, - "ece": 0.0885, - "brier": 0.5407, - "over95": 0.0094, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.05, - "conf": 0.0575, - "acc": 0.0288 - }, - { - "lo": 0.1, - "w": 0.0289, - "conf": 0.1499, - "acc": 0.1282 - }, - { - "lo": 0.2, - "w": 0.0265, - "conf": 0.2553, - "acc": 0.1883 - }, - { - "lo": 0.3, - "w": 0.0572, - "conf": 0.3525, - "acc": 0.2861 - }, - { - "lo": 0.4, - "w": 0.0973, - "conf": 0.4548, - "acc": 0.237 - }, - { - "lo": 0.5, - "w": 0.1327, - "conf": 0.5491, - "acc": 0.4997 - }, - { - "lo": 0.6, - "w": 0.1201, - "conf": 0.6498, - "acc": 0.5915 - }, - { - "lo": 0.7, - "w": 0.1258, - "conf": 0.7494, - "acc": 0.6396 - }, - { - "lo": 0.8, - "w": 0.1284, - "conf": 0.8508, - "acc": 0.742 - }, - { - "lo": 0.9, - "w": 0.233, - "conf": 0.9603, - "acc": 0.8816 - } - ], - "areas": { - "knowledge": { - "ece": 0.103, - "acc": 0.5141, - "conf": 0.6171 - }, - "language": { - "ece": 0.0617, - "acc": 0.6785, - "conf": 0.7402 - }, - "retrieval": { - "ece": 0.1054, - "acc": 0.6542, - "conf": 0.7537 - }, - "tools": { - "ece": 0.0671, - "acc": 0.6702, - "conf": 0.7337 - }, - "arts": { - "ece": 0.0917, - "acc": 0.4347, - "conf": 0.5263 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [ - { - "reason": "runtime error", - "share": 0.0 - } - ] - }, - "latency": { - "median": 49.7, - "p95": 1640.9, - "mean": 427.8 - }, - "results": { - "1": { - "score": 0.8294, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 432.2 - }, - "2": { - "score": 0.4296, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 1451.7 - }, - "3": { - "score": 0.3169, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 1683.9 - }, - "4": { - "score": 0.7277, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 67.0 - }, - "5": { - "score": 0.7369, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 478.5 - }, - "6": { - "score": 0.7988, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 981.5 - }, - "9": { - "score": 0.025, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 1037.4 - }, - "10": { - "score": 0.4885, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 460.1 - }, - "11": { - "score": 0.634, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 1296.2 - }, - "12": { - "score": 0.4937, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 22.1 - }, - "20": { - "score": 0.6492, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 21.8 - }, - "21": { - "score": 0.5993, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 14.0 - }, - "22": { - "score": 0.0025, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 1637.8 - }, - "23": { - "score": 0.5696, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 40.0 - }, - "24": { - "score": 0.619, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 19.4 - }, - "25": { - "score": 0.3622, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 31.5 - }, - "26": { - "score": 0.9377, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 17.0 - }, - "27": { - "score": 0.8515, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 17.3 - }, - "28": { - "score": 0.6772, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 12.3 - }, - "29": { - "score": 0.8581, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 29.0 - }, - "30": { - "score": 0.2111, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 25.5 - }, - "31": { - "score": 0.0988, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 121.5 - }, - "32": { - "score": 0.5492, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 122.4 - }, - "33": { - "score": 0.2467, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 714.4 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 322.7 - }, - "36": { - "score": 0.1431, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 3248.3 - }, - "37": { - "score": 0.3468, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 55.3 - }, - "38": { - "score": 0.0029, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 689.6 - }, - "39": { - "score": 0.7612, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 38.0 - }, - "40": { - "score": 0.2243, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 15.8, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2174, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2243, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.375, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.55, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.5104, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "42": { - "score": 0.6547, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 188.0 - }, - "43": { - "score": 0.3298, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 24.9 - }, - "44": { - "score": 0.5726, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 22.5 - }, - "45": { - "score": 0.1198, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 49.7 - }, - "48": { - "score": 0.2706, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 92.4 - }, - "50": { - "score": 0.4218, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 105.3 - } - }, - "scores": { - "balanced_raw": 44.0, - "balanced_skill": 24.8, - "breadth_skill": 22.12 - }, - "frozen_scores": { - "balanced_raw": 33.87, - "balanced_skill": 17.36, - "breadth_skill": 14.97 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.3698, - "skill": 0.1472, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4562, - "skill": 0.2461, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.245, - "skill": 0.142, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.6859, - "skill": 0.5554, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4432, - "skill": 0.1491, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8294, - "skill": 0.7697, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4296, - "skill": 0.3709, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7988, - "skill": 0.5256, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7853, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8123, - "headline": false - } - ] - }, - "11": { - "raw": 0.634, - "skill": 0.4706, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.6481, - "skill": 0.2962, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5993, - "skill": 0.1986, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.002, - "skill": 0.0, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5449, - "skill": 0.0897, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.6439, - "skill": 0.5253, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3622, - "skill": 0.1497, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2111, - "skill": 0.0446, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2904, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1319, - "headline": false - } - ] - }, - "31": { - "raw": 0.0988, - "skill": 0.0185, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1431, - "skill": 0.1033, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.3468, - "skill": 0.1808, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2243, - "skill": 0.0021, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2174, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2243, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.375, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.55, - "headline": false - } - ] - }, - "41": { - "raw": 0.5104, - "skill": 0.2656, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3298, - "skill": 0.0, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5726, - "skill": 0.1452, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4218, - "skill": 0.1609, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "decider-35b-nvfp4", - "name": "Decider 35B-A3B", - "long_name": "Decider 35B-A3B", - "variant": "NVFP4", - "meta": { - "kind": "full fine-tune", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 35951822704, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 21013206328, - "model_url": "https://huggingface.co/Mapika/decider-35b-a3b-nvfp4", - "code_url": null, - "base_model": "Qwen/Qwen3.5-35B-A3B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base", - "weights_repo": "Mapika/decider-35b-a3b-nvfp4", - "served_checkpoint": "Mapika/decider-35b-a3b-nvfp4" - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.6748, - "conf": 0.6734, - "ece": 0.031, - "brier": 0.4153, - "over95": 0.0037, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0433, - "conf": 0.0708, - "acc": 0.0496 - }, - { - "lo": 0.1, - "w": 0.0304, - "conf": 0.1412, - "acc": 0.1385 - }, - { - "lo": 0.2, - "w": 0.0288, - "conf": 0.2567, - "acc": 0.3389 - }, - { - "lo": 0.3, - "w": 0.0611, - "conf": 0.3522, - "acc": 0.3491 - }, - { - "lo": 0.4, - "w": 0.0708, - "conf": 0.449, - "acc": 0.4597 - }, - { - "lo": 0.5, - "w": 0.1586, - "conf": 0.5586, - "acc": 0.4731 - }, - { - "lo": 0.6, - "w": 0.1169, - "conf": 0.6518, - "acc": 0.6765 - }, - { - "lo": 0.7, - "w": 0.1025, - "conf": 0.7497, - "acc": 0.7962 - }, - { - "lo": 0.8, - "w": 0.1033, - "conf": 0.8493, - "acc": 0.8918 - }, - { - "lo": 0.9, - "w": 0.2843, - "conf": 0.9704, - "acc": 0.9742 - } - ], - "areas": { - "knowledge": { - "ece": 0.062, - "acc": 0.6122, - "conf": 0.6664 - }, - "language": { - "ece": 0.0494, - "acc": 0.7905, - "conf": 0.7419 - }, - "retrieval": { - "ece": 0.0538, - "acc": 0.7811, - "conf": 0.7372 - }, - "tools": { - "ece": 0.065, - "acc": 0.7947, - "conf": 0.7368 - }, - "arts": { - "ece": 0.0347, - "acc": 0.5312, - "conf": 0.5078 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 99.3, - "p95": 366.5, - "mean": 138.4 - }, - "results": { - "1": { - "score": 0.9215, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 196.7 - }, - "2": { - "score": 0.4403, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 364.2 - }, - "3": { - "score": 0.565, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 222.5 - }, - "4": { - "score": 0.8979, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 98.0 - }, - "5": { - "score": 0.9468, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 99.6 - }, - "6": { - "score": 0.7963, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 206.0 - }, - "9": { - "score": 0.2188, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 597.7 - }, - "10": { - "score": 0.6414, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 102.6 - }, - "11": { - "score": 0.7496, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 905.5 - }, - "12": { - "score": 0.6603, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 99.1 - }, - "20": { - "score": 0.8906, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 99.5 - }, - "21": { - "score": 0.6195, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 38.9 - }, - "22": { - "score": 0.0665, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 182.4 - }, - "23": { - "score": 0.649, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 97.6 - }, - "24": { - "score": 0.8261, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 39.3 - }, - "25": { - "score": 0.4082, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 101.4 - }, - "26": { - "score": 0.9899, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 39.7 - }, - "27": { - "score": 0.9633, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 39.7 - }, - "28": { - "score": 0.8863, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 39.5 - }, - "29": { - "score": 0.9665, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 96.5 - }, - "30": { - "score": 0.4606, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 99.8 - }, - "31": { - "score": 0.1088, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 97.9 - }, - "32": { - "score": 0.6184, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 99.9 - }, - "33": { - "score": 0.3473, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 204.5 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 100.8 - }, - "36": { - "score": 0.1675, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 497.9 - }, - "37": { - "score": 0.5065, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 98.7 - }, - "38": { - "score": 0.0836, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 341.1 - }, - "39": { - "score": 0.8524, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 195.6 - }, - "40": { - "score": 0.5481, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 39.2, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2937, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5481, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.535, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.845, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.5452, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 100.4 - }, - "42": { - "score": 0.7847, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 100.1 - }, - "43": { - "score": 0.5982, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 99.8 - }, - "44": { - "score": 0.6672, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 99.1 - }, - "45": { - "score": 0.0878, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 98.1 - }, - "48": { - "score": 0.2007, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 99.3 - }, - "50": { - "score": 0.4642, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 99.7 - } - }, - "scores": { - "balanced_raw": 54.34, - "balanced_skill": 39.37, - "breadth_skill": 37.99 - }, - "frozen_scores": { - "balanced_raw": 41.71, - "balanced_skill": 28.83, - "breadth_skill": 24.4 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.5127, - "skill": 0.3446, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.6143, - "skill": 0.4581, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.337, - "skill": 0.2549, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7194, - "skill": 0.5987, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.5337, - "skill": 0.3121, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9215, - "skill": 0.894, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4403, - "skill": 0.3826, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7963, - "skill": 0.5196, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7832, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8094, - "headline": false - } - ] - }, - "11": { - "raw": 0.7496, - "skill": 0.6379, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.8918, - "skill": 0.7836, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.6195, - "skill": 0.239, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0635, - "skill": 0.0562, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.6297, - "skill": 0.2594, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.8331, - "skill": 0.7775, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.4082, - "skill": 0.2109, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4606, - "skill": 0.3531, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5724, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3487, - "headline": false - } - ] - }, - "31": { - "raw": 0.1088, - "skill": 0.0293, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1675, - "skill": 0.1288, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.5065, - "skill": 0.381, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.5481, - "skill": 0.4187, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2937, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5481, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.535, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.845, - "headline": false - } - ] - }, - "41": { - "raw": 0.5452, - "skill": 0.3178, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.5982, - "skill": 0.3626, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6672, - "skill": 0.3344, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4642, - "skill": 0.2224, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "decision-kai", - "name": "Decision 1.0 Kai", - "long_name": "Decision 1.0 Kai", - "variant": null, - "meta": { - "kind": "head / adapter", - "technique": "encoder", - "technique_name": "encoder", - "served_params": 307797035, - "params_estimated": true, - "params_from_name": false, - "trained_bytes": 2287687932, - "model_url": "https://huggingface.co/llm-semantic-router/decision-1.0-kai", - "code_url": null, - "base_model": "jhu-clsp/mmBERT-base", - "base_url": "https://huggingface.co/jhu-clsp/mmBERT-base", - "weights_repo": "llm-semantic-router/decision-1.0-kai", - "served_checkpoint": null - }, - "completed": 131980, - "counts": { - "ok": 107970, - "unsupported": 24010 - }, - "calibration": { - "acc": 0.3636, - "conf": 0.5386, - "ece": 0.1815, - "brier": 0.7468, - "over95": 0.0018, - "n": 69940, - "benchmarks": 29, - "rel": [ - { - "lo": 0.0, - "w": 0.0725, - "conf": 0.0717, - "acc": 0.0636 - }, - { - "lo": 0.1, - "w": 0.0364, - "conf": 0.1291, - "acc": 0.2197 - }, - { - "lo": 0.2, - "w": 0.0762, - "conf": 0.275, - "acc": 0.1547 - }, - { - "lo": 0.3, - "w": 0.1177, - "conf": 0.3521, - "acc": 0.2999 - }, - { - "lo": 0.4, - "w": 0.1168, - "conf": 0.4486, - "acc": 0.2909 - }, - { - "lo": 0.5, - "w": 0.1736, - "conf": 0.5473, - "acc": 0.4386 - }, - { - "lo": 0.6, - "w": 0.1134, - "conf": 0.6479, - "acc": 0.4175 - }, - { - "lo": 0.7, - "w": 0.0875, - "conf": 0.7523, - "acc": 0.469 - }, - { - "lo": 0.8, - "w": 0.1545, - "conf": 0.8527, - "acc": 0.5158 - }, - { - "lo": 0.9, - "w": 0.0515, - "conf": 0.9285, - "acc": 0.5002 - } - ], - "areas": { - "knowledge": { - "ece": 0.1972, - "acc": 0.3045, - "conf": 0.501 - }, - "language": { - "ece": 0.2551, - "acc": 0.4455, - "conf": 0.7007 - }, - "retrieval": { - "ece": 0.0513, - "acc": 0.33, - "conf": 0.2889 - }, - "tools": { - "ece": 0.2572, - "acc": 0.4382, - "conf": 0.6905 - }, - "arts": { - "ece": 0.1306, - "acc": 0.4241, - "conf": 0.5547 - } - } - }, - "gaps": { - "share": 0.1819, - "reasons": [ - { - "reason": "prompt longer than its 1,024-token context window", - "share": 0.1271 - }, - { - "reason": "ran out of GPU memory on the longest prompts", - "share": 0.0304 - }, - { - "reason": "rejects requests with an empty state", - "share": 0.0244 - } - ] - }, - "latency": { - "median": 24.7, - "p95": 139.3, - "mean": 38.8 - }, - "results": { - "1": { - "score": 0.4932, - "jev": 0.964, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1332, - "unsupported": 362, - "errors": 0, - "median_ms": 25.4 - }, - "2": { - "score": 0.1237, - "jev": 0.4485, - "metric": "nDCG@10", - "requests": 7704, - "answered": 6146, - "unsupported": 1558, - "errors": 0, - "median_ms": 151.0 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.0442, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 26.1 - }, - "5": { - "score": 0.0572, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 27.6 - }, - "6": { - "score": null, - "jev": null, - "metric": null, - "requests": 10000, - "answered": 0, - "unsupported": 10000, - "errors": 0, - "median_ms": null - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 150.6 - }, - "10": { - "score": 0.502, - "jev": 0.4858, - "metric": "macro-F1", - "requests": 2500, - "answered": 1130, - "unsupported": 1370, - "errors": 0, - "median_ms": 25.9 - }, - "11": { - "score": 0.2536, - "jev": 0.675, - "metric": "macro-F1", - "requests": 123, - "answered": 16, - "unsupported": 107, - "errors": 0, - "median_ms": 160.6 - }, - "12": { - "score": 0.2196, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "20": { - "score": 0.4176, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "21": { - "score": null, - "jev": null, - "metric": null, - "requests": 2628, - "answered": 0, - "unsupported": 2628, - "errors": 0, - "median_ms": null - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.4882, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "24": { - "score": 0.3081, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "25": { - "score": null, - "jev": null, - "metric": null, - "requests": 196, - "answered": 0, - "unsupported": 196, - "errors": 0, - "median_ms": null - }, - "26": { - "score": 0.4844, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "27": { - "score": 0.3225, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "28": { - "score": 0.4988, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - }, - "29": { - "score": 0.3292, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 24.6 - }, - "30": { - "score": 0.3343, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 24.3 - }, - "31": { - "score": 0.0755, - "jev": 0.1708, - "metric": "accuracy", - "requests": 5000, - "answered": 4977, - "unsupported": 23, - "errors": 0, - "median_ms": 25.5 - }, - "32": { - "score": 0.3776, - "jev": 0.6527, - "metric": "accuracy", - "requests": 752, - "answered": 429, - "unsupported": 323, - "errors": 0, - "median_ms": 26.2 - }, - "33": { - "score": 0.005, - "jev": 0.2709, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1388, - "unsupported": 262, - "errors": 0, - "median_ms": 37.7 - }, - "34": { - "score": 0.3, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 24.4 - }, - "36": { - "score": 0.0128, - "jev": 0.1082, - "metric": "nDCG@10", - "requests": 1297, - "answered": 542, - "unsupported": 755, - "errors": 0, - "median_ms": 156.9 - }, - "37": { - "score": 0.2076, - "jev": 0.5485, - "metric": "macro-F1", - "requests": 5000, - "answered": 4885, - "unsupported": 115, - "errors": 0, - "median_ms": 24.6 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 136.9 - }, - "39": { - "score": 0.3976, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 24.5 - }, - "40": { - "score": 0.0, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 400, - "errors": 0, - "median_ms": 23.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.0, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.0, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.0, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3868, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "42": { - "score": 0.2534, - "jev": 0.8535, - "metric": "macro-F1", - "requests": 5500, - "answered": 4860, - "unsupported": 640, - "errors": 0, - "median_ms": 24.8 - }, - "43": { - "score": 0.3649, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 24.3 - }, - "44": { - "score": 0.4994, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.3 - }, - "45": { - "score": 0.1731, - "jev": 0.2115, - "metric": "accuracy", - "requests": 501, - "answered": 468, - "unsupported": 33, - "errors": 0, - "median_ms": 24.6 - }, - "48": { - "score": 0.5311, - "jev": 0.1749, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 7708, - "unsupported": 2431, - "errors": 0, - "median_ms": 25.3 - }, - "50": { - "score": 0.3747, - "jev": 0.4749, - "metric": "accuracy", - "requests": 1676, - "answered": 1377, - "unsupported": 299, - "errors": 0, - "median_ms": 26.0 - } - }, - "scores": { - "balanced_raw": 18.37, - "balanced_skill": 2.64, - "breadth_skill": 2.43 - }, - "frozen_scores": { - "balanced_raw": 15.75, - "balanced_skill": 2.13, - "breadth_skill": 1.97 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2646, - "skill": 0.0456, - "coverage": 0.8326, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.1454, - "skill": 0.0267, - "coverage": 0.71, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1018, - "skill": 0.0, - "coverage": 0.6974, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.1622, - "skill": 0.0598, - "coverage": 0.528, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.2447, - "skill": 0.0, - "coverage": 0.5643, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3878, - "skill": 0.1736, - "coverage": 0.7863, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0987, - "skill": 0.0059, - "coverage": 0.7978, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0, - "headline": false - } - ] - }, - "11": { - "raw": 0.0493, - "skill": 0.0, - "coverage": 0.1301, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.4229, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.4926, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.314, - "skill": 0.0853, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.3343, - "skill": 0.1884, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.3525, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3161, - "headline": false - } - ] - }, - "31": { - "raw": 0.0752, - "skill": 0.0, - "coverage": 0.9954, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0053, - "skill": 0.0, - "coverage": 0.4179, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1982, - "skill": 0.0, - "coverage": 0.977, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.0, - "skill": 0.0, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.0, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.0, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.0, - "headline": false - } - ] - }, - "41": { - "raw": 0.3868, - "skill": 0.0802, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3649, - "skill": 0.0, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4994, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3079, - "skill": 0.0, - "coverage": 0.8216, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.497, - "pending": 0.24 - }, - { - "engine": "decision-lex", - "name": "Decision 1.0 Lex", - "long_name": "Decision 1.0 Lex", - "variant": null, - "meta": { - "kind": "head / adapter", - "technique": "encoder", - "technique_name": "encoder", - "served_params": 307797035, - "params_estimated": true, - "params_from_name": false, - "trained_bytes": 2287687932, - "model_url": "https://huggingface.co/llm-semantic-router/decision-1.0-lex", - "code_url": null, - "base_model": "jhu-clsp/mmBERT-base", - "base_url": "https://huggingface.co/jhu-clsp/mmBERT-base", - "weights_repo": "llm-semantic-router/decision-1.0-lex", - "served_checkpoint": null - }, - "completed": 131980, - "counts": { - "ok": 107970, - "unsupported": 24010 - }, - "calibration": { - "acc": 0.3495, - "conf": 0.4957, - "ece": 0.1748, - "brier": 0.7417, - "over95": 0.0005, - "n": 69940, - "benchmarks": 29, - "rel": [ - { - "lo": 0.0, - "w": 0.0962, - "conf": 0.0497, - "acc": 0.1922 - }, - { - "lo": 0.1, - "w": 0.0182, - "conf": 0.1532, - "acc": 0.1851 - }, - { - "lo": 0.2, - "w": 0.0749, - "conf": 0.247, - "acc": 0.1341 - }, - { - "lo": 0.3, - "w": 0.1311, - "conf": 0.3555, - "acc": 0.2951 - }, - { - "lo": 0.4, - "w": 0.1332, - "conf": 0.4443, - "acc": 0.3452 - }, - { - "lo": 0.5, - "w": 0.1946, - "conf": 0.5475, - "acc": 0.4289 - }, - { - "lo": 0.6, - "w": 0.1412, - "conf": 0.6476, - "acc": 0.4304 - }, - { - "lo": 0.7, - "w": 0.1283, - "conf": 0.7415, - "acc": 0.4878 - }, - { - "lo": 0.8, - "w": 0.0654, - "conf": 0.8433, - "acc": 0.3829 - }, - { - "lo": 0.9, - "w": 0.017, - "conf": 0.9201, - "acc": 0.0673 - } - ], - "areas": { - "knowledge": { - "ece": 0.1452, - "acc": 0.2948, - "conf": 0.4392 - }, - "language": { - "ece": 0.2553, - "acc": 0.3511, - "conf": 0.6064 - }, - "retrieval": { - "ece": 0.1608, - "acc": 0.4036, - "conf": 0.3309 - }, - "tools": { - "ece": 0.2383, - "acc": 0.4448, - "conf": 0.6732 - }, - "arts": { - "ece": 0.1462, - "acc": 0.4474, - "conf": 0.584 - } - } - }, - "gaps": { - "share": 0.1819, - "reasons": [ - { - "reason": "prompt longer than its 1,024-token context window", - "share": 0.1271 - }, - { - "reason": "ran out of GPU memory on the longest prompts", - "share": 0.0304 - }, - { - "reason": "rejects requests with an empty state", - "share": 0.0244 - } - ] - }, - "latency": { - "median": 24.5, - "p95": 138.9, - "mean": 38.6 - }, - "results": { - "1": { - "score": 0.4925, - "jev": 0.964, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1332, - "unsupported": 362, - "errors": 0, - "median_ms": 25.2 - }, - "2": { - "score": 0.1603, - "jev": 0.4485, - "metric": "nDCG@10", - "requests": 7704, - "answered": 6146, - "unsupported": 1558, - "errors": 0, - "median_ms": 150.6 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.2356, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 26.0 - }, - "5": { - "score": 0.1121, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 27.5 - }, - "6": { - "score": null, - "jev": null, - "metric": null, - "requests": 10000, - "answered": 0, - "unsupported": 10000, - "errors": 0, - "median_ms": null - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 150.1 - }, - "10": { - "score": 0.4386, - "jev": 0.4858, - "metric": "macro-F1", - "requests": 2500, - "answered": 1130, - "unsupported": 1370, - "errors": 0, - "median_ms": 25.8 - }, - "11": { - "score": 0.2536, - "jev": 0.675, - "metric": "macro-F1", - "requests": 123, - "answered": 16, - "unsupported": 107, - "errors": 0, - "median_ms": 159.8 - }, - "12": { - "score": 0.2256, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 24.0 - }, - "20": { - "score": 0.4798, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.0 - }, - "21": { - "score": null, - "jev": null, - "metric": null, - "requests": 2628, - "answered": 0, - "unsupported": 2628, - "errors": 0, - "median_ms": null - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.512, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "24": { - "score": 0.3017, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - }, - "25": { - "score": null, - "jev": null, - "metric": null, - "requests": 196, - "answered": 0, - "unsupported": 196, - "errors": 0, - "median_ms": null - }, - "26": { - "score": 0.476, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 23.8 - }, - "27": { - "score": 0.3038, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 23.8 - }, - "28": { - "score": 0.5067, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 23.6 - }, - "29": { - "score": 0.3331, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 24.3 - }, - "30": { - "score": 0.2343, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "31": { - "score": 0.0645, - "jev": 0.1708, - "metric": "accuracy", - "requests": 5000, - "answered": 4977, - "unsupported": 23, - "errors": 0, - "median_ms": 25.3 - }, - "32": { - "score": 0.338, - "jev": 0.6527, - "metric": "accuracy", - "requests": 752, - "answered": 429, - "unsupported": 323, - "errors": 0, - "median_ms": 25.7 - }, - "33": { - "score": 0.0007, - "jev": 0.2709, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1388, - "unsupported": 262, - "errors": 0, - "median_ms": 37.4 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "36": { - "score": 0.028, - "jev": 0.1082, - "metric": "nDCG@10", - "requests": 1297, - "answered": 542, - "unsupported": 755, - "errors": 0, - "median_ms": 156.6 - }, - "37": { - "score": 0.1932, - "jev": 0.5485, - "metric": "macro-F1", - "requests": 5000, - "answered": 4885, - "unsupported": 115, - "errors": 0, - "median_ms": 24.5 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 134.6 - }, - "39": { - "score": 0.2178, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 24.4 - }, - "40": { - "score": 0.1896, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 400, - "errors": 0, - "median_ms": 24.0, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2054, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.1896, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.0, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.0, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4039, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "42": { - "score": 0.3064, - "jev": 0.8535, - "metric": "macro-F1", - "requests": 5500, - "answered": 4860, - "unsupported": 640, - "errors": 0, - "median_ms": 24.5 - }, - "43": { - "score": 0.3684, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 24.0 - }, - "44": { - "score": 0.4838, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "45": { - "score": 0.1496, - "jev": 0.2115, - "metric": "accuracy", - "requests": 501, - "answered": 468, - "unsupported": 33, - "errors": 0, - "median_ms": 24.4 - }, - "48": { - "score": 0.4497, - "jev": 0.1749, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 7708, - "unsupported": 2431, - "errors": 0, - "median_ms": 24.9 - }, - "50": { - "score": 0.3413, - "jev": 0.4749, - "metric": "accuracy", - "requests": 1676, - "answered": 1377, - "unsupported": 299, - "errors": 0, - "median_ms": 25.9 - } - }, - "scores": { - "balanced_raw": 19.57, - "balanced_skill": 2.61, - "breadth_skill": 2.37 - }, - "frozen_scores": { - "balanced_raw": 15.89, - "balanced_skill": 1.86, - "breadth_skill": 1.75 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2431, - "skill": 0.0247, - "coverage": 0.8326, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2143, - "skill": 0.0353, - "coverage": 0.71, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1002, - "skill": 0.0, - "coverage": 0.6974, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.1717, - "skill": 0.0703, - "coverage": 0.528, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.2494, - "skill": 0.0, - "coverage": 0.5643, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3872, - "skill": 0.1728, - "coverage": 0.7863, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.1279, - "skill": 0.0381, - "coverage": 0.7978, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0, - "headline": false - } - ] - }, - "11": { - "raw": 0.0493, - "skill": 0.0, - "coverage": 0.1301, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.48, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.4868, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.3079, - "skill": 0.0772, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2343, - "skill": 0.071, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2964, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1721, - "headline": false - } - ] - }, - "31": { - "raw": 0.0642, - "skill": 0.0, - "coverage": 0.9954, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0117, - "skill": 0.0, - "coverage": 0.4179, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1887, - "skill": 0.0, - "coverage": 0.977, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.1896, - "skill": 0.0, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2054, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.1896, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.0, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.0, - "headline": false - } - ] - }, - "41": { - "raw": 0.4039, - "skill": 0.1059, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3684, - "skill": 0.0, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4838, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.2804, - "skill": 0.0, - "coverage": 0.8216, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.497, - "pending": 0.24 - }, - { - "engine": "decision-nox-full", - "name": "Decision 1.0 Nox", - "long_name": "Decision 1.0 Nox", - "variant": null, - "meta": { - "kind": "head / adapter", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 4659865088, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 8422078976, - "model_url": "https://huggingface.co/llm-semantic-router/decision-1.0-nox", - "code_url": null, - "base_model": "Qwen/Qwen3.5-4B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-4B-Base", - "weights_repo": "llm-semantic-router/decision-1.0-nox", - "served_checkpoint": null - }, - "completed": 131980, - "counts": { - "ok": 131968, - "unsupported": 12 - }, - "calibration": { - "acc": 0.5923, - "conf": 0.7463, - "ece": 0.156, - "brier": 0.5585, - "over95": 0.0538, - "n": 72590, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0229, - "conf": 0.0794, - "acc": 0.1088 - }, - { - "lo": 0.1, - "w": 0.029, - "conf": 0.1377, - "acc": 0.1498 - }, - { - "lo": 0.2, - "w": 0.0137, - "conf": 0.2497, - "acc": 0.1856 - }, - { - "lo": 0.3, - "w": 0.0404, - "conf": 0.3555, - "acc": 0.2417 - }, - { - "lo": 0.4, - "w": 0.0659, - "conf": 0.4531, - "acc": 0.3459 - }, - { - "lo": 0.5, - "w": 0.1012, - "conf": 0.549, - "acc": 0.4702 - }, - { - "lo": 0.6, - "w": 0.0901, - "conf": 0.6483, - "acc": 0.5659 - }, - { - "lo": 0.7, - "w": 0.1235, - "conf": 0.7427, - "acc": 0.432 - }, - { - "lo": 0.8, - "w": 0.1099, - "conf": 0.8522, - "acc": 0.6584 - }, - { - "lo": 0.9, - "w": 0.4034, - "conf": 0.9754, - "acc": 0.8084 - } - ], - "areas": { - "knowledge": { - "ece": 0.1747, - "acc": 0.5125, - "conf": 0.6872 - }, - "language": { - "ece": 0.1158, - "acc": 0.6711, - "conf": 0.7866 - }, - "retrieval": { - "ece": 0.1375, - "acc": 0.7181, - "conf": 0.8544 - }, - "tools": { - "ece": 0.0796, - "acc": 0.8517, - "conf": 0.9299 - }, - "arts": { - "ece": 0.2494, - "acc": 0.4492, - "conf": 0.6586 - } - } - }, - "gaps": { - "share": 0.0001, - "reasons": [ - { - "reason": "prompt longer than its 16,384-token context window", - "share": 0.0001 - } - ] - }, - "latency": { - "median": 48.8, - "p95": 619.5, - "mean": 167.5 - }, - "results": { - "1": { - "score": 0.8695, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 62.1 - }, - "2": { - "score": 0.4302, - "jev": 0.446, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7693, - "unsupported": 11, - "errors": 0, - "median_ms": 699.8 - }, - "3": { - "score": 0.7362, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 323.8 - }, - "4": { - "score": 0.8308, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 76.8 - }, - "5": { - "score": 0.8673, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 140.0 - }, - "6": { - "score": 0.7352, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 188.3 - }, - "9": { - "score": 0.0938, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 749.5 - }, - "10": { - "score": 0.6546, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 50.5 - }, - "11": { - "score": 0.6527, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 1567.5 - }, - "12": { - "score": 0.5753, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 47.3 - }, - "20": { - "score": 0.5062, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 47.3 - }, - "21": { - "score": 0.5099, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 46.9 - }, - "22": { - "score": 0.2166, - "jev": 0.1811, - "metric": "accuracy", - "requests": 2000, - "answered": 1999, - "unsupported": 1, - "errors": 0, - "median_ms": 327.0 - }, - "23": { - "score": 0.5878, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 48.0 - }, - "24": { - "score": 0.5912, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 47.3 - }, - "25": { - "score": 0.2857, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 47.7 - }, - "26": { - "score": 0.9432, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 47.0 - }, - "27": { - "score": 0.8097, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 47.0 - }, - "28": { - "score": 0.6354, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 46.9 - }, - "29": { - "score": 0.7218, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 47.7 - }, - "30": { - "score": 0.4356, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 47.7 - }, - "31": { - "score": 0.0706, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 53.1 - }, - "32": { - "score": 0.6011, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 51.0 - }, - "33": { - "score": 0.2552, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 146.8 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 48.0 - }, - "36": { - "score": 0.1296, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 1190.2 - }, - "37": { - "score": 0.3345, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 48.0 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 501.2 - }, - "39": { - "score": 0.7952, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 47.8 - }, - "40": { - "score": 0.3583, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 47.2, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3734, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3583, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.415, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.485, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4846, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 47.1 - }, - "42": { - "score": 0.7427, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 48.0 - }, - "43": { - "score": 0.4825, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 46.7 - }, - "44": { - "score": 0.6296, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 46.7 - }, - "45": { - "score": 0.1038, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 47.5 - }, - "48": { - "score": 0.2156, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 48.5 - }, - "50": { - "score": 0.3783, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 49.3 - } - }, - "scores": { - "balanced_raw": 45.31, - "balanced_skill": 25.36, - "breadth_skill": 22.54 - }, - "frozen_scores": { - "balanced_raw": 34.98, - "balanced_skill": 17.76, - "breadth_skill": 14.97 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4178, - "skill": 0.2135, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4985, - "skill": 0.2997, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2321, - "skill": 0.1272, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.6781, - "skill": 0.5255, - "coverage": 0.9995, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4388, - "skill": 0.1021, - "coverage": 0.9999, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8695, - "skill": 0.8239, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4296, - "skill": 0.3709, - "coverage": 0.9986, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7353, - "skill": 0.3816, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.6962, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.7743, - "headline": false - } - ] - }, - "11": { - "raw": 0.6527, - "skill": 0.4978, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.508, - "skill": 0.0159, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5099, - "skill": 0.0198, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.2122, - "skill": 0.206, - "coverage": 0.9995, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5856, - "skill": 0.1712, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.6027, - "skill": 0.4703, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2857, - "skill": 0.0476, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4356, - "skill": 0.3248, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5679, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3033, - "headline": false - } - ] - }, - "31": { - "raw": 0.0706, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1296, - "skill": 0.0892, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.3345, - "skill": 0.1653, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.3583, - "skill": 0.1744, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3734, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3583, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.415, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.485, - "headline": false - } - ] - }, - "41": { - "raw": 0.4846, - "skill": 0.2269, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.4825, - "skill": 0.1789, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6296, - "skill": 0.2592, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3783, - "skill": 0.0977, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.7599, - "pending": 0.24 - }, - { - "engine": "decision-sol", - "name": "Decision 1.0 Sol", - "long_name": "Decision 1.0 Sol", - "variant": null, - "meta": { - "kind": "head / adapter", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 2274069824, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 3772109600, - "model_url": "https://huggingface.co/llm-semantic-router/decision-1.0-sol", - "code_url": null, - "base_model": "Qwen/Qwen3.5-2B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-2B-Base", - "weights_repo": "llm-semantic-router/decision-1.0-sol", - "served_checkpoint": null - }, - "completed": 131980, - "counts": { - "ok": 131968, - "unsupported": 12 - }, - "calibration": { - "acc": 0.5267, - "conf": 0.6632, - "ece": 0.1365, - "brier": 0.6234, - "over95": 0.0204, - "n": 72590, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0498, - "conf": 0.0548, - "acc": 0.0271 - }, - { - "lo": 0.1, - "w": 0.0124, - "conf": 0.154, - "acc": 0.0972 - }, - { - "lo": 0.2, - "w": 0.023, - "conf": 0.2603, - "acc": 0.2369 - }, - { - "lo": 0.3, - "w": 0.0689, - "conf": 0.3553, - "acc": 0.3098 - }, - { - "lo": 0.4, - "w": 0.0816, - "conf": 0.4488, - "acc": 0.3618 - }, - { - "lo": 0.5, - "w": 0.1604, - "conf": 0.5493, - "acc": 0.4737 - }, - { - "lo": 0.6, - "w": 0.1358, - "conf": 0.6471, - "acc": 0.5209 - }, - { - "lo": 0.7, - "w": 0.1076, - "conf": 0.7488, - "acc": 0.6045 - }, - { - "lo": 0.8, - "w": 0.1107, - "conf": 0.8509, - "acc": 0.6708 - }, - { - "lo": 0.9, - "w": 0.2498, - "conf": 0.9635, - "acc": 0.7276 - } - ], - "areas": { - "knowledge": { - "ece": 0.1639, - "acc": 0.4541, - "conf": 0.618 - }, - "language": { - "ece": 0.1345, - "acc": 0.6272, - "conf": 0.7599 - }, - "retrieval": { - "ece": 0.1411, - "acc": 0.5887, - "conf": 0.7297 - }, - "tools": { - "ece": 0.0506, - "acc": 0.7543, - "conf": 0.79 - }, - "arts": { - "ece": 0.1221, - "acc": 0.4028, - "conf": 0.525 - } - } - }, - "gaps": { - "share": 0.0001, - "reasons": [ - { - "reason": "prompt longer than its 16,384-token context window", - "share": 0.0001 - } - ] - }, - "latency": { - "median": 37.8, - "p95": 315.6, - "mean": 88.8 - }, - "results": { - "1": { - "score": 0.8849, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 38.8 - }, - "2": { - "score": 0.3844, - "jev": 0.446, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7693, - "unsupported": 11, - "errors": 0, - "median_ms": 306.7 - }, - "3": { - "score": 0.622, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 140.2 - }, - "4": { - "score": 0.7719, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 40.3 - }, - "5": { - "score": 0.5694, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 68.4 - }, - "6": { - "score": 0.6887, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 84.0 - }, - "9": { - "score": 0.0063, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 327.8 - }, - "10": { - "score": 0.6154, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 38.4 - }, - "11": { - "score": 0.5057, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 732.3 - }, - "12": { - "score": 0.4786, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 36.7 - }, - "20": { - "score": 0.5128, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 36.8 - }, - "21": { - "score": 0.5563, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 36.1 - }, - "22": { - "score": 0.0235, - "jev": 0.1811, - "metric": "accuracy", - "requests": 2000, - "answered": 1999, - "unsupported": 1, - "errors": 0, - "median_ms": 140.8 - }, - "23": { - "score": 0.5568, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 37.2 - }, - "24": { - "score": 0.4752, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 36.4 - }, - "25": { - "score": 0.2908, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 36.7 - }, - "26": { - "score": 0.8864, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 36.1 - }, - "27": { - "score": 0.6809, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 36.2 - }, - "28": { - "score": 0.5841, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 36.1 - }, - "29": { - "score": 0.5698, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 36.9 - }, - "30": { - "score": 0.318, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 37.1 - }, - "31": { - "score": 0.065, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 38.6 - }, - "32": { - "score": 0.5492, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 39.2 - }, - "33": { - "score": 0.1152, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 68.3 - }, - "34": { - "score": 0, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 37.0 - }, - "36": { - "score": 0.1081, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 502.9 - }, - "37": { - "score": 0.3201, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 37.6 - }, - "38": { - "score": 0.015, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 311.2 - }, - "39": { - "score": 0.6133, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 37.3 - }, - "40": { - "score": 0.2477, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 36.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2819, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2477, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.295, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.28, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4905, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 36.7 - }, - "42": { - "score": 0.6789, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 37.8 - }, - "43": { - "score": 0.3754, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 36.6 - }, - "44": { - "score": 0.5564, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 36.7 - }, - "45": { - "score": 0.1277, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 37.0 - }, - "48": { - "score": 0.3355, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 37.9 - }, - "50": { - "score": 0.3132, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 38.3 - } - }, - "scores": { - "balanced_raw": 40.41, - "balanced_skill": 18.65, - "breadth_skill": 15.68 - }, - "frozen_scores": { - "balanced_raw": 30.81, - "balanced_skill": 12.5, - "breadth_skill": 10.45 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.3481, - "skill": 0.1115, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4146, - "skill": 0.1843, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2141, - "skill": 0.1069, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.6525, - "skill": 0.4776, - "coverage": 0.9995, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3914, - "skill": 0.0523, - "coverage": 0.9999, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8849, - "skill": 0.8446, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.3839, - "skill": 0.3204, - "coverage": 0.9986, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.6887, - "skill": 0.2678, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.6618, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.7157, - "headline": false - } - ] - }, - "11": { - "raw": 0.5057, - "skill": 0.2852, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5151, - "skill": 0.0303, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5563, - "skill": 0.1126, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0212, - "skill": 0.0135, - "coverage": 0.9995, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.551, - "skill": 0.102, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.4832, - "skill": 0.3109, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2908, - "skill": 0.0544, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.318, - "skill": 0.1817, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4549, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1812, - "headline": false - } - ] - }, - "31": { - "raw": 0.065, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1081, - "skill": 0.0666, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.3201, - "skill": 0.1472, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2477, - "skill": 0.0321, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2819, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2477, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.295, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.28, - "headline": false - } - ] - }, - "41": { - "raw": 0.4905, - "skill": 0.2357, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3754, - "skill": 0.0091, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5564, - "skill": 0.1128, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3132, - "skill": 0.0033, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.7599, - "pending": 0.24 - }, - { - "engine": "gliner-base", - "name": "GLiNER 2.5 base", - "long_name": "GLiNER 2.5 base", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "GLi family", - "technique_name": "GLiNER2", - "served_params": 193581591, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 774366564, - "model_url": "https://huggingface.co/fastino/gliner2.5-base-v1", - "code_url": null, - "base_model": null, - "base_url": null, - "weights_repo": "fastino/gliner2.5-base-v1", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 105440, - "unsupported": 26982 - }, - "calibration": { - "acc": 0.357, - "conf": 0.734, - "ece": 0.3771, - "brier": 0.9531, - "over95": 0.172, - "n": 17845, - "benchmarks": 27, - "rel": [ - { - "lo": 0.0, - "w": 0.0049, - "conf": 0.0837, - "acc": 0.074 - }, - { - "lo": 0.1, - "w": 0.0184, - "conf": 0.1498, - "acc": 0.0388 - }, - { - "lo": 0.2, - "w": 0.0454, - "conf": 0.2619, - "acc": 0.1638 - }, - { - "lo": 0.3, - "w": 0.0574, - "conf": 0.3534, - "acc": 0.1473 - }, - { - "lo": 0.4, - "w": 0.0674, - "conf": 0.4524, - "acc": 0.2601 - }, - { - "lo": 0.5, - "w": 0.1183, - "conf": 0.5476, - "acc": 0.3406 - }, - { - "lo": 0.6, - "w": 0.0959, - "conf": 0.6496, - "acc": 0.3649 - }, - { - "lo": 0.7, - "w": 0.0902, - "conf": 0.7515, - "acc": 0.4349 - }, - { - "lo": 0.8, - "w": 0.1241, - "conf": 0.8515, - "acc": 0.4261 - }, - { - "lo": 0.9, - "w": 0.3781, - "conf": 0.9725, - "acc": 0.4102 - } - ], - "areas": { - "knowledge": { - "ece": 0.3752, - "acc": 0.3264, - "conf": 0.7 - }, - "language": { - "ece": 0.5583, - "acc": 0.3308, - "conf": 0.8891 - }, - "retrieval": { - "ece": 0.2449, - "acc": 0.3619, - "conf": 0.6068 - }, - "tools": { - "ece": 0.4175, - "acc": 0.3636, - "conf": 0.7811 - }, - "arts": { - "ece": 0.3191, - "acc": 0.4742, - "conf": 0.7933 - } - } - }, - "gaps": { - "share": 0.2038, - "reasons": [ - { - "reason": "prompt text with parentheses, which the native GLiNER prompt syntax cannot carry", - "share": 0.1837 - }, - { - "reason": "prompt longer than its context window (no truncation allowed)", - "share": 0.02 - } - ] - }, - "latency": { - "median": 14.4, - "p95": 68.2, - "mean": 25.5 - }, - "results": { - "1": { - "score": 0.389, - "jev": 0.958, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1689, - "unsupported": 5, - "errors": 0, - "median_ms": 18.2 - }, - "2": { - "score": 0.0521, - "jev": 0.4901, - "metric": "nDCG@10", - "requests": 7704, - "answered": 5, - "unsupported": 7699, - "errors": 0, - "median_ms": 224.1 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.2381, - "jev": 0.7973, - "metric": "macro-F1", - "requests": 3080, - "answered": 3078, - "unsupported": 2, - "errors": 0, - "median_ms": 26.6 - }, - "5": { - "score": 0.2688, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5499, - "unsupported": 1, - "errors": 0, - "median_ms": 55.5 - }, - "6": { - "score": 0.6699, - "jev": 0.7978, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 9953, - "unsupported": 47, - "errors": 0, - "median_ms": 67.7 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 122.1 - }, - "10": { - "score": 0.459, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 38.7 - }, - "11": { - "score": null, - "jev": null, - "metric": null, - "requests": 123, - "answered": 0, - "unsupported": 123, - "errors": 0, - "median_ms": null - }, - "12": { - "score": 0.3037, - "jev": 0.7388, - "metric": "macro-F1", - "requests": 3200, - "answered": 1905, - "unsupported": 1295, - "errors": 0, - "median_ms": 13.9 - }, - "20": { - "score": 0.5286, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 4403, - "unsupported": 597, - "errors": 0, - "median_ms": 14.0 - }, - "21": { - "score": 0.4918, - "jev": 0.6189, - "metric": "accuracy", - "requests": 2628, - "answered": 2611, - "unsupported": 17, - "errors": 0, - "median_ms": 13.6 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.503, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 14.4 - }, - "24": { - "score": 0.338, - "jev": 0.9225, - "metric": "accuracy", - "requests": 14033, - "answered": 11439, - "unsupported": 2594, - "errors": 0, - "median_ms": 13.9 - }, - "25": { - "score": 0.3636, - "jev": 0.7818, - "metric": "accuracy", - "requests": 196, - "answered": 55, - "unsupported": 141, - "errors": 0, - "median_ms": 14.1 - }, - "26": { - "score": 0.4824, - "jev": 0.9931, - "metric": "accuracy", - "requests": 2376, - "answered": 2332, - "unsupported": 44, - "errors": 0, - "median_ms": 13.8 - }, - "27": { - "score": 0.2865, - "jev": 0.9779, - "metric": "accuracy", - "requests": 1172, - "answered": 1131, - "unsupported": 41, - "errors": 0, - "median_ms": 13.8 - }, - "28": { - "score": 0.5083, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 13.4 - }, - "29": { - "score": 0.352, - "jev": 0.936, - "metric": "accuracy", - "requests": 10042, - "answered": 8153, - "unsupported": 1889, - "errors": 0, - "median_ms": 14.3 - }, - "30": { - "score": 0.2237, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 14.2 - }, - "31": { - "score": 0.0742, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 25.7 - }, - "32": { - "score": 0.348, - "jev": 0.66, - "metric": "accuracy", - "requests": 752, - "answered": 750, - "unsupported": 2, - "errors": 0, - "median_ms": 19.5 - }, - "33": { - "score": 0.0015, - "jev": 0.3125, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1296, - "unsupported": 354, - "errors": 0, - "median_ms": 17.5 - }, - "34": { - "score": 0, - "jev": 0.6667, - "metric": "accuracy", - "requests": 10, - "answered": 3, - "unsupported": 7, - "errors": 0, - "median_ms": 14.4 - }, - "36": { - "score": null, - "jev": null, - "metric": null, - "requests": 1297, - "answered": 0, - "unsupported": 1297, - "errors": 0, - "median_ms": null - }, - "37": { - "score": 0.2005, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 14.5 - }, - "38": { - "score": null, - "jev": null, - "metric": null, - "requests": 5479, - "answered": 0, - "unsupported": 5479, - "errors": 0, - "median_ms": null - }, - "39": { - "score": null, - "jev": null, - "metric": null, - "requests": 979, - "answered": 0, - "unsupported": 979, - "errors": 0, - "median_ms": null - }, - "40": { - "score": 0.2384, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 13, - "errors": 0, - "median_ms": 13.3, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2591, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2384, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.415, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3632, - "jev": 0.6461, - "metric": "macro-F1", - "requests": 3006, - "answered": 2311, - "unsupported": 695, - "errors": 0, - "median_ms": 14.0 - }, - "42": { - "score": 0.3947, - "jev": 0.8391, - "metric": "macro-F1", - "requests": 5500, - "answered": 5199, - "unsupported": 301, - "errors": 0, - "median_ms": 14.6 - }, - "43": { - "score": 0.4144, - "jev": 0.7293, - "metric": "accuracy", - "requests": 570, - "answered": 543, - "unsupported": 27, - "errors": 0, - "median_ms": 13.9 - }, - "44": { - "score": 0.5164, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 13.9 - }, - "45": { - "score": 0.1667, - "jev": 0.2094, - "metric": "accuracy", - "requests": 501, - "answered": 234, - "unsupported": 267, - "errors": 0, - "median_ms": 14.2 - }, - "48": { - "score": 0.4664, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 15.7 - }, - "50": { - "score": 0.3553, - "jev": 0.468, - "metric": "accuracy", - "requests": 1676, - "answered": 1545, - "unsupported": 131, - "errors": 0, - "median_ms": 18.4 - } - }, - "scores": { - "balanced_raw": 24.7, - "balanced_skill": 3.4, - "breadth_skill": 2.77 - }, - "frozen_scores": { - "balanced_raw": 20.9, - "balanced_skill": 2.42, - "breadth_skill": 2.15 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2647, - "skill": 0.0282, - "coverage": 0.8414, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.1633, - "skill": 0.0067, - "coverage": 0.5896, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1003, - "skill": 0.0, - "coverage": 0.5, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3505, - "skill": 0.1277, - "coverage": 0.6628, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3563, - "skill": 0.0072, - "coverage": 0.7609, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3878, - "skill": 0.1736, - "coverage": 0.997, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0006, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.6636, - "skill": 0.2095, - "coverage": 0.9906, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.6309, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.6962, - "headline": false - } - ] - }, - "11": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.4594, - "skill": 0.0, - "coverage": 0.889, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4886, - "skill": 0.0, - "coverage": 0.9935, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5059, - "skill": 0.0119, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.2769, - "skill": 0.0359, - "coverage": 0.8152, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.102, - "skill": 0.0, - "coverage": 0.2806, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2237, - "skill": 0.0608, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.3108, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1365, - "headline": false - } - ] - }, - "31": { - "raw": 0.0742, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2005, - "skill": 0.0, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2384, - "skill": 0.0202, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2591, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2384, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.415, - "headline": false - } - ] - }, - "41": { - "raw": 0.2514, - "skill": 0.0, - "coverage": 0.7688, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3947, - "skill": 0.0397, - "coverage": 0.9526, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5164, - "skill": 0.0328, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3276, - "skill": 0.0241, - "coverage": 0.9218, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.5437, - "pending": 0.24 - }, - { - "engine": "gliner-multi", - "name": "GLiNER 2.5 multilingual", - "long_name": "GLiNER 2.5 multilingual", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "GLi family", - "technique_name": "GLiNER2", - "served_params": 287355159, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 1149461028, - "model_url": "https://huggingface.co/fastino/gliner2.5-multi-v1", - "code_url": null, - "base_model": null, - "base_url": null, - "weights_repo": "fastino/gliner2.5-multi-v1", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 105115, - "unsupported": 27307 - }, - "calibration": { - "acc": 0.3328, - "conf": 0.5656, - "ece": 0.2328, - "brier": 0.8282, - "over95": 0.0502, - "n": 17739, - "benchmarks": 28, - "rel": [ - { - "lo": 0.0, - "w": 0.003, - "conf": 0.0847, - "acc": 0.087 - }, - { - "lo": 0.1, - "w": 0.0179, - "conf": 0.1518, - "acc": 0.0265 - }, - { - "lo": 0.2, - "w": 0.0791, - "conf": 0.2715, - "acc": 0.2119 - }, - { - "lo": 0.3, - "w": 0.1865, - "conf": 0.3494, - "acc": 0.2653 - }, - { - "lo": 0.4, - "w": 0.1712, - "conf": 0.4542, - "acc": 0.2108 - }, - { - "lo": 0.5, - "w": 0.144, - "conf": 0.5452, - "acc": 0.3736 - }, - { - "lo": 0.6, - "w": 0.1116, - "conf": 0.6486, - "acc": 0.4229 - }, - { - "lo": 0.7, - "w": 0.1017, - "conf": 0.7432, - "acc": 0.4552 - }, - { - "lo": 0.8, - "w": 0.0622, - "conf": 0.8462, - "acc": 0.4274 - }, - { - "lo": 0.9, - "w": 0.1228, - "conf": 0.9706, - "acc": 0.4556 - } - ], - "areas": { - "knowledge": { - "ece": 0.1788, - "acc": 0.3263, - "conf": 0.505 - }, - "language": { - "ece": 0.308, - "acc": 0.4051, - "conf": 0.7127 - }, - "retrieval": { - "ece": 0.2284, - "acc": 0.237, - "conf": 0.4654 - }, - "tools": { - "ece": 0.4176, - "acc": 0.3449, - "conf": 0.7625 - }, - "arts": { - "ece": 0.2453, - "acc": 0.3639, - "conf": 0.6069 - } - } - }, - "gaps": { - "share": 0.2062, - "reasons": [ - { - "reason": "prompt text with parentheses, which the native GLiNER prompt syntax cannot carry", - "share": 0.1837 - }, - { - "reason": "prompt longer than its context window (no truncation allowed)", - "share": 0.0225 - } - ] - }, - "latency": { - "median": 15.2, - "p95": 83.1, - "mean": 29.2 - }, - "results": { - "1": { - "score": 0.3792, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1672, - "unsupported": 22, - "errors": 0, - "median_ms": 22.5 - }, - "2": { - "score": 0.2648, - "jev": 0.4325, - "metric": "nDCG@10", - "requests": 7704, - "answered": 9, - "unsupported": 7695, - "errors": 0, - "median_ms": 218.7 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.0892, - "jev": 0.7973, - "metric": "macro-F1", - "requests": 3080, - "answered": 3078, - "unsupported": 2, - "errors": 0, - "median_ms": 27.2 - }, - "5": { - "score": 0.1278, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5499, - "unsupported": 1, - "errors": 0, - "median_ms": 55.9 - }, - "6": { - "score": 0.5912, - "jev": 0.8022, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 9691, - "unsupported": 309, - "errors": 0, - "median_ms": 81.6 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 162.2 - }, - "10": { - "score": 0.4188, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 56.7 - }, - "11": { - "score": null, - "jev": null, - "metric": null, - "requests": 123, - "answered": 0, - "unsupported": 123, - "errors": 0, - "median_ms": null - }, - "12": { - "score": 0.315, - "jev": 0.7388, - "metric": "macro-F1", - "requests": 3200, - "answered": 1905, - "unsupported": 1295, - "errors": 0, - "median_ms": 14.9 - }, - "20": { - "score": 0.4949, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 4403, - "unsupported": 597, - "errors": 0, - "median_ms": 14.7 - }, - "21": { - "score": 0.5216, - "jev": 0.6189, - "metric": "accuracy", - "requests": 2628, - "answered": 2611, - "unsupported": 17, - "errors": 0, - "median_ms": 14.4 - }, - "22": { - "score": 0.0909, - "jev": 0.1818, - "metric": "accuracy", - "requests": 2000, - "answered": 11, - "unsupported": 1989, - "errors": 0, - "median_ms": 206.0 - }, - "23": { - "score": 0.4958, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 15.1 - }, - "24": { - "score": 0.2886, - "jev": 0.9225, - "metric": "accuracy", - "requests": 14033, - "answered": 11439, - "unsupported": 2594, - "errors": 0, - "median_ms": 14.6 - }, - "25": { - "score": 0.2364, - "jev": 0.7818, - "metric": "accuracy", - "requests": 196, - "answered": 55, - "unsupported": 141, - "errors": 0, - "median_ms": 14.8 - }, - "26": { - "score": 0.3533, - "jev": 0.9931, - "metric": "accuracy", - "requests": 2376, - "answered": 2332, - "unsupported": 44, - "errors": 0, - "median_ms": 14.4 - }, - "27": { - "score": 0.2617, - "jev": 0.9779, - "metric": "accuracy", - "requests": 1172, - "answered": 1131, - "unsupported": 41, - "errors": 0, - "median_ms": 14.5 - }, - "28": { - "score": 0.5114, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 14.3 - }, - "29": { - "score": 0.3032, - "jev": 0.936, - "metric": "accuracy", - "requests": 10042, - "answered": 8153, - "unsupported": 1889, - "errors": 0, - "median_ms": 15.1 - }, - "30": { - "score": 0.1945, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 14.8 - }, - "31": { - "score": 0.0738, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 21.6 - }, - "32": { - "score": 0.34, - "jev": 0.66, - "metric": "accuracy", - "requests": 752, - "answered": 750, - "unsupported": 2, - "errors": 0, - "median_ms": 29.2 - }, - "33": { - "score": 0, - "jev": 0.3115, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1268, - "unsupported": 382, - "errors": 0, - "median_ms": 19.8 - }, - "34": { - "score": 0.3333, - "jev": 0.6667, - "metric": "accuracy", - "requests": 10, - "answered": 3, - "unsupported": 7, - "errors": 0, - "median_ms": 15.2 - }, - "36": { - "score": null, - "jev": null, - "metric": null, - "requests": 1297, - "answered": 0, - "unsupported": 1297, - "errors": 0, - "median_ms": null - }, - "37": { - "score": 0.1498, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 15.2 - }, - "38": { - "score": null, - "jev": null, - "metric": null, - "requests": 5479, - "answered": 0, - "unsupported": 5479, - "errors": 0, - "median_ms": null - }, - "39": { - "score": null, - "jev": null, - "metric": null, - "requests": 979, - "answered": 0, - "unsupported": 979, - "errors": 0, - "median_ms": null - }, - "40": { - "score": 0.2376, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 13, - "errors": 0, - "median_ms": 14.3, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.221, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2376, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.33, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.35, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.2556, - "jev": 0.6461, - "metric": "macro-F1", - "requests": 3006, - "answered": 2311, - "unsupported": 695, - "errors": 0, - "median_ms": 14.7 - }, - "42": { - "score": 0.4116, - "jev": 0.8385, - "metric": "macro-F1", - "requests": 5500, - "answered": 5179, - "unsupported": 321, - "errors": 0, - "median_ms": 15.3 - }, - "43": { - "score": 0.4052, - "jev": 0.7293, - "metric": "accuracy", - "requests": 570, - "answered": 543, - "unsupported": 27, - "errors": 0, - "median_ms": 14.7 - }, - "44": { - "score": 0.4926, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 14.6 - }, - "45": { - "score": 0.1496, - "jev": 0.2094, - "metric": "accuracy", - "requests": 501, - "answered": 234, - "unsupported": 267, - "errors": 0, - "median_ms": 14.8 - }, - "48": { - "score": 0.3338, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10126, - "unsupported": 13, - "errors": 0, - "median_ms": 18.1 - }, - "50": { - "score": 0.3618, - "jev": 0.468, - "metric": "accuracy", - "requests": 1676, - "answered": 1545, - "unsupported": 131, - "errors": 0, - "median_ms": 23.4 - } - }, - "scores": { - "balanced_raw": 22.42, - "balanced_skill": 1.61, - "breadth_skill": 1.49 - }, - "frozen_scores": { - "balanced_raw": 19.09, - "balanced_skill": 1.12, - "breadth_skill": 1.07 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2423, - "skill": 0.0085, - "coverage": 0.8414, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.1393, - "skill": 0.0064, - "coverage": 0.5896, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.0749, - "skill": 0.0, - "coverage": 0.5, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3097, - "skill": 0.0518, - "coverage": 0.6421, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.355, - "skill": 0.0139, - "coverage": 0.762, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3743, - "skill": 0.1553, - "coverage": 0.987, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0003, - "skill": 0.0, - "coverage": 0.0012, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.5547, - "skill": 0.0, - "coverage": 0.9382, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.5125, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.5968, - "headline": false - } - ] - }, - "11": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.4375, - "skill": 0.0, - "coverage": 0.889, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5183, - "skill": 0.0365, - "coverage": 0.9935, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0004, - "skill": 0.0, - "coverage": 0.0055, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.4851, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.2408, - "skill": 0.0, - "coverage": 0.8152, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.0663, - "skill": 0.0, - "coverage": 0.2806, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.1945, - "skill": 0.025, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2805, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1084, - "headline": false - } - ] - }, - "31": { - "raw": 0.0738, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1498, - "skill": 0.0, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2376, - "skill": 0.0192, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.221, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2376, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.33, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.35, - "headline": false - } - ] - }, - "41": { - "raw": 0.1804, - "skill": 0.0, - "coverage": 0.7688, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.386, - "skill": 0.0258, - "coverage": 0.9526, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4926, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3335, - "skill": 0.0327, - "coverage": 0.9218, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.5415, - "pending": 0.24 - }, - { - "engine": "gliner-small", - "name": "GLiNER 2.5 small", - "long_name": "GLiNER 2.5 small", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "GLi family", - "technique_name": "GLiNER2", - "served_params": 73881879, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 295567700, - "model_url": "https://huggingface.co/fastino/gliner2.5-small-v1", - "code_url": null, - "base_model": null, - "base_url": null, - "weights_repo": "fastino/gliner2.5-small-v1", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 105440, - "unsupported": 26982 - }, - "calibration": { - "acc": 0.3567, - "conf": 0.5175, - "ece": 0.1616, - "brier": 0.7588, - "over95": 0.0262, - "n": 17845, - "benchmarks": 27, - "rel": [ - { - "lo": 0.0, - "w": 0.0661, - "conf": 0.0497, - "acc": 0.0559 - }, - { - "lo": 0.1, - "w": 0.0611, - "conf": 0.1459, - "acc": 0.0905 - }, - { - "lo": 0.2, - "w": 0.0745, - "conf": 0.2686, - "acc": 0.2494 - }, - { - "lo": 0.3, - "w": 0.1422, - "conf": 0.3488, - "acc": 0.2816 - }, - { - "lo": 0.4, - "w": 0.0903, - "conf": 0.4479, - "acc": 0.3342 - }, - { - "lo": 0.5, - "w": 0.1979, - "conf": 0.5469, - "acc": 0.4847 - }, - { - "lo": 0.6, - "w": 0.1293, - "conf": 0.647, - "acc": 0.4687 - }, - { - "lo": 0.7, - "w": 0.0826, - "conf": 0.7472, - "acc": 0.4595 - }, - { - "lo": 0.8, - "w": 0.071, - "conf": 0.8482, - "acc": 0.3797 - }, - { - "lo": 0.9, - "w": 0.0851, - "conf": 0.9575, - "acc": 0.4379 - } - ], - "areas": { - "knowledge": { - "ece": 0.1547, - "acc": 0.3154, - "conf": 0.4702 - }, - "language": { - "ece": 0.4122, - "acc": 0.3287, - "conf": 0.7409 - }, - "retrieval": { - "ece": 0.0742, - "acc": 0.3214, - "conf": 0.3478 - }, - "tools": { - "ece": 0.0995, - "acc": 0.4913, - "conf": 0.5864 - }, - "arts": { - "ece": 0.1007, - "acc": 0.4867, - "conf": 0.5832 - } - } - }, - "gaps": { - "share": 0.2038, - "reasons": [ - { - "reason": "prompt text with parentheses, which the native GLiNER prompt syntax cannot carry", - "share": 0.1837 - }, - { - "reason": "prompt longer than its context window (no truncation allowed)", - "share": 0.02 - } - ] - }, - "latency": { - "median": 14.5, - "p95": 38.4, - "mean": 18.9 - }, - "results": { - "1": { - "score": 0.4079, - "jev": 0.958, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1689, - "unsupported": 5, - "errors": 0, - "median_ms": 15.9 - }, - "2": { - "score": 0.3425, - "jev": 0.4901, - "metric": "nDCG@10", - "requests": 7704, - "answered": 5, - "unsupported": 7699, - "errors": 0, - "median_ms": 125.3 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.0852, - "jev": 0.7973, - "metric": "macro-F1", - "requests": 3080, - "answered": 3078, - "unsupported": 2, - "errors": 0, - "median_ms": 19.3 - }, - "5": { - "score": 0.1358, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5499, - "unsupported": 1, - "errors": 0, - "median_ms": 31.9 - }, - "6": { - "score": 0.5704, - "jev": 0.7978, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 9953, - "unsupported": 47, - "errors": 0, - "median_ms": 38.0 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 71.2 - }, - "10": { - "score": 0.479, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 19.7 - }, - "11": { - "score": null, - "jev": null, - "metric": null, - "requests": 123, - "answered": 0, - "unsupported": 123, - "errors": 0, - "median_ms": null - }, - "12": { - "score": 0.2633, - "jev": 0.7388, - "metric": "macro-F1", - "requests": 3200, - "answered": 1905, - "unsupported": 1295, - "errors": 0, - "median_ms": 14.1 - }, - "20": { - "score": 0.4839, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 4403, - "unsupported": 597, - "errors": 0, - "median_ms": 14.0 - }, - "21": { - "score": 0.5025, - "jev": 0.6189, - "metric": "accuracy", - "requests": 2628, - "answered": 2611, - "unsupported": 17, - "errors": 0, - "median_ms": 13.4 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.5182, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 14.6 - }, - "24": { - "score": 0.314, - "jev": 0.9225, - "metric": "accuracy", - "requests": 14033, - "answered": 11439, - "unsupported": 2594, - "errors": 0, - "median_ms": 13.6 - }, - "25": { - "score": 0.2727, - "jev": 0.7818, - "metric": "accuracy", - "requests": 196, - "answered": 55, - "unsupported": 141, - "errors": 0, - "median_ms": 14.0 - }, - "26": { - "score": 0.4314, - "jev": 0.9931, - "metric": "accuracy", - "requests": 2376, - "answered": 2332, - "unsupported": 44, - "errors": 0, - "median_ms": 13.5 - }, - "27": { - "score": 0.2776, - "jev": 0.9779, - "metric": "accuracy", - "requests": 1172, - "answered": 1131, - "unsupported": 41, - "errors": 0, - "median_ms": 13.5 - }, - "28": { - "score": 0.4988, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 13.4 - }, - "29": { - "score": 0.3245, - "jev": 0.936, - "metric": "accuracy", - "requests": 10042, - "answered": 8153, - "unsupported": 1889, - "errors": 0, - "median_ms": 14.4 - }, - "30": { - "score": 0.2346, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 14.3 - }, - "31": { - "score": 0.0808, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 16.4 - }, - "32": { - "score": 0.3773, - "jev": 0.66, - "metric": "accuracy", - "requests": 752, - "answered": 750, - "unsupported": 2, - "errors": 0, - "median_ms": 16.7 - }, - "33": { - "score": 0.0015, - "jev": 0.3125, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1296, - "unsupported": 354, - "errors": 0, - "median_ms": 17.6 - }, - "34": { - "score": 0.3333, - "jev": 0.6667, - "metric": "accuracy", - "requests": 10, - "answered": 3, - "unsupported": 7, - "errors": 0, - "median_ms": 14.4 - }, - "36": { - "score": null, - "jev": null, - "metric": null, - "requests": 1297, - "answered": 0, - "unsupported": 1297, - "errors": 0, - "median_ms": null - }, - "37": { - "score": 0.2082, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 14.6 - }, - "38": { - "score": null, - "jev": null, - "metric": null, - "requests": 5479, - "answered": 0, - "unsupported": 5479, - "errors": 0, - "median_ms": null - }, - "39": { - "score": null, - "jev": null, - "metric": null, - "requests": 979, - "answered": 0, - "unsupported": 979, - "errors": 0, - "median_ms": null - }, - "40": { - "score": 0.2373, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 13, - "errors": 0, - "median_ms": 13.3, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2475, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2373, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.455, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3729, - "jev": 0.6461, - "metric": "macro-F1", - "requests": 3006, - "answered": 2311, - "unsupported": 695, - "errors": 0, - "median_ms": 13.9 - }, - "42": { - "score": 0.4608, - "jev": 0.8391, - "metric": "macro-F1", - "requests": 5500, - "answered": 5199, - "unsupported": 301, - "errors": 0, - "median_ms": 14.6 - }, - "43": { - "score": 0.4088, - "jev": 0.7293, - "metric": "accuracy", - "requests": 570, - "answered": 543, - "unsupported": 27, - "errors": 0, - "median_ms": 13.9 - }, - "44": { - "score": 0.4922, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 13.8 - }, - "45": { - "score": 0.141, - "jev": 0.2094, - "metric": "accuracy", - "requests": 501, - "answered": 234, - "unsupported": 267, - "errors": 0, - "median_ms": 14.2 - }, - "48": { - "score": 0.5084, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 15.3 - }, - "50": { - "score": 0.3333, - "jev": 0.468, - "metric": "accuracy", - "requests": 1676, - "answered": 1545, - "unsupported": 131, - "errors": 0, - "median_ms": 16.6 - } - }, - "scores": { - "balanced_raw": 23.93, - "balanced_skill": 2.02, - "breadth_skill": 1.82 - }, - "frozen_scores": { - "balanced_raw": 20.29, - "balanced_skill": 1.44, - "breadth_skill": 1.35 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2556, - "skill": 0.0201, - "coverage": 0.8414, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.1648, - "skill": 0.0063, - "coverage": 0.5896, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1041, - "skill": 0.0034, - "coverage": 0.5, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.324, - "skill": 0.0684, - "coverage": 0.6628, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3482, - "skill": 0.0031, - "coverage": 0.7609, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.4067, - "skill": 0.1991, - "coverage": 0.997, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0002, - "skill": 0.0, - "coverage": 0.0006, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.5651, - "skill": 0.0062, - "coverage": 0.9906, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.503, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.6272, - "headline": false - } - ] - }, - "11": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.427, - "skill": 0.0, - "coverage": 0.889, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4992, - "skill": 0.0, - "coverage": 0.9935, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5076, - "skill": 0.0153, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.2601, - "skill": 0.0135, - "coverage": 0.8152, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.0765, - "skill": 0.0, - "coverage": 0.2806, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2346, - "skill": 0.0756, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.3343, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.135, - "headline": false - } - ] - }, - "31": { - "raw": 0.0808, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2082, - "skill": 0.0068, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2373, - "skill": 0.0188, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2475, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2373, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.455, - "headline": false - } - ] - }, - "41": { - "raw": 0.257, - "skill": 0.0, - "coverage": 0.7688, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3895, - "skill": 0.0313, - "coverage": 0.9526, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4922, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3073, - "skill": 0.0, - "coverage": 0.9218, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.5437, - "pending": 0.24 - }, - { - "engine": "harsha", - "name": "Qwen-2.5-1B-RLCD", - "long_name": "Qwen-2.5-1B-RLCD", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 1543714304, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD", - "base_model": "Qwen/Qwen2.5-1.5B", - "base_url": "https://huggingface.co/Qwen/Qwen2.5-1.5B", - "weights_repo": null, - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 132042, - "unsupported": 380 - }, - "calibration": { - "acc": 0.3841, - "conf": 0.5684, - "ece": 0.1871, - "brier": 0.775, - "over95": 0.0341, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.1385, - "conf": 0.0392, - "acc": 0.0474 - }, - { - "lo": 0.1, - "w": 0.0097, - "conf": 0.1432, - "acc": 0.1705 - }, - { - "lo": 0.2, - "w": 0.0184, - "conf": 0.2576, - "acc": 0.121 - }, - { - "lo": 0.3, - "w": 0.0594, - "conf": 0.3546, - "acc": 0.2552 - }, - { - "lo": 0.4, - "w": 0.1002, - "conf": 0.4547, - "acc": 0.3506 - }, - { - "lo": 0.5, - "w": 0.1656, - "conf": 0.5484, - "acc": 0.4176 - }, - { - "lo": 0.6, - "w": 0.1492, - "conf": 0.6405, - "acc": 0.5332 - }, - { - "lo": 0.7, - "w": 0.1455, - "conf": 0.7495, - "acc": 0.475 - }, - { - "lo": 0.8, - "w": 0.0912, - "conf": 0.8516, - "acc": 0.4813 - }, - { - "lo": 0.9, - "w": 0.1222, - "conf": 0.9578, - "acc": 0.5039 - } - ], - "areas": { - "knowledge": { - "ece": 0.1474, - "acc": 0.4218, - "conf": 0.5641 - }, - "language": { - "ece": 0.3192, - "acc": 0.4037, - "conf": 0.7223 - }, - "retrieval": { - "ece": 0.2027, - "acc": 0.1929, - "conf": 0.3787 - }, - "tools": { - "ece": 0.2339, - "acc": 0.347, - "conf": 0.5761 - }, - "arts": { - "ece": 0.1175, - "acc": 0.4261, - "conf": 0.5121 - } - } - }, - "gaps": { - "share": 0.0029, - "reasons": [ - { - "reason": "prompt longer than its context window (no truncation allowed)", - "share": 0.0029 - } - ] - }, - "latency": { - "median": 42.4, - "p95": 294.9, - "mean": 73.3 - }, - "results": { - "1": { - "score": 0.474, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 43.5 - }, - "2": { - "score": 0.0613, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 257.5 - }, - "3": { - "score": 0.0433, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 129.0 - }, - "4": { - "score": 0.0259, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 46.5 - }, - "5": { - "score": 0.0116, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 54.4 - }, - "6": { - "score": 0.5886, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 63.9 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 75.3 - }, - "10": { - "score": 0.4644, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 45.3 - }, - "11": { - "score": 0.2668, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 96.9 - }, - "12": { - "score": 0.2819, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 41.3 - }, - "20": { - "score": 0.6808, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 41.3 - }, - "21": { - "score": 0.5068, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 40.4 - }, - "22": { - "score": 0.029, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 111.5 - }, - "23": { - "score": 0.5024, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 42.2 - }, - "24": { - "score": 0.3688, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 40.8 - }, - "25": { - "score": 0.2806, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 41.2 - }, - "26": { - "score": 0.5648, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 40.8 - }, - "27": { - "score": 0.4369, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 40.9 - }, - "28": { - "score": 0.5043, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 40.6 - }, - "29": { - "score": 0.4255, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 41.9 - }, - "30": { - "score": 0.1748, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 41.6 - }, - "31": { - "score": 0.0882, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 43.9 - }, - "32": { - "score": 0.3777, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 46.3 - }, - "33": { - "score": 0.0236, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 45.3 - }, - "34": { - "score": 0.5, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 40.9 - }, - "36": { - "score": 0.0547, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 339.2 - }, - "37": { - "score": 0.0892, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 42.3 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 295.6 - }, - "39": { - "score": 0.4332, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 41.9 - }, - "40": { - "score": 0.2431, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 40.6, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2768, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2431, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.495, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.475, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3142, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 41.2 - }, - "42": { - "score": 0.4373, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 42.4 - }, - "43": { - "score": 0.3579, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 41.3 - }, - "44": { - "score": 0.503, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 41.6 - }, - "45": { - "score": 0.1717, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 41.8 - }, - "48": { - "score": 0.2466, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 44.0 - }, - "50": { - "score": 0.3848, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 45.6 - } - }, - "scores": { - "balanced_raw": 28.81, - "balanced_skill": 5.29, - "breadth_skill": 4.69 - }, - "frozen_scores": { - "balanced_raw": 24.46, - "balanced_skill": 4.43, - "breadth_skill": 3.98 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2979, - "skill": 0.0391, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2747, - "skill": 0.0087, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.072, - "skill": 0.0054, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3746, - "skill": 0.1091, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.421, - "skill": 0.102, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.474, - "skill": 0.29, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0613, - "skill": 0.0, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.5886, - "skill": 0.0372, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.5345, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.6427, - "headline": false - } - ] - }, - "11": { - "raw": 0.2668, - "skill": 0.0, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.6735, - "skill": 0.347, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5068, - "skill": 0.0137, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0298, - "skill": 0.0222, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.51, - "skill": 0.02, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.3832, - "skill": 0.1776, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2806, - "skill": 0.0408, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.1748, - "skill": 0.003, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2441, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1054, - "headline": false - } - ] - }, - "31": { - "raw": 0.0882, - "skill": 0.0069, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0547, - "skill": 0.0108, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.0892, - "skill": 0.0, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2431, - "skill": 0.0262, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2768, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2431, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.495, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.475, - "headline": false - } - ] - }, - "41": { - "raw": 0.3142, - "skill": 0.0, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3579, - "skill": 0.0, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.503, - "skill": 0.006, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3848, - "skill": 0.1072, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "jeff-uncapped", - "name": "jeff", - "long_name": "jeff GLiFormer-large", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "GLi family", - "technique_name": "GLiFormer", - "served_params": 575683963, - "params_estimated": true, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/logan-markewich/jeff", - "base_model": "knowledgator/gliformer-large-v1", - "base_url": "https://huggingface.co/knowledgator/gliformer-large-v1", - "weights_repo": null, - "served_checkpoint": "knowledgator/gliformer-large-v1" - }, - "completed": 132422, - "counts": { - "ok": 131356, - "unsupported": 1066 - }, - "calibration": { - "acc": 0.4011, - "conf": 0.4136, - "ece": 0.1131, - "brier": 0.6841, - "over95": 0, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.144, - "conf": 0.037, - "acc": 0.2279 - }, - { - "lo": 0.1, - "w": 0.0297, - "conf": 0.146, - "acc": 0.1938 - }, - { - "lo": 0.2, - "w": 0.0948, - "conf": 0.257, - "acc": 0.4826 - }, - { - "lo": 0.3, - "w": 0.1957, - "conf": 0.3492, - "acc": 0.3442 - }, - { - "lo": 0.4, - "w": 0.0991, - "conf": 0.4394, - "acc": 0.4299 - }, - { - "lo": 0.5, - "w": 0.2285, - "conf": 0.5454, - "acc": 0.4335 - }, - { - "lo": 0.6, - "w": 0.1294, - "conf": 0.646, - "acc": 0.5012 - }, - { - "lo": 0.7, - "w": 0.0662, - "conf": 0.7403, - "acc": 0.5444 - }, - { - "lo": 0.8, - "w": 0.0125, - "conf": 0.8286, - "acc": 0.549 - }, - { - "lo": 0.9, - "w": 0.0002, - "conf": 0.9121, - "acc": 0.0968 - } - ], - "areas": { - "knowledge": { - "ece": 0.1148, - "acc": 0.3864, - "conf": 0.4008 - }, - "language": { - "ece": 0.0988, - "acc": 0.4166, - "conf": 0.5154 - }, - "retrieval": { - "ece": 0.3082, - "acc": 0.553, - "conf": 0.2595 - }, - "tools": { - "ece": 0.1054, - "acc": 0.3014, - "conf": 0.3827 - }, - "arts": { - "ece": 0.096, - "acc": 0.3592, - "conf": 0.4491 - } - } - }, - "gaps": { - "share": 0.0081, - "reasons": [ - { - "reason": "prompt longer than its 16,384-token context window", - "share": 0.0081 - } - ] - }, - "latency": { - "median": 22.4, - "p95": 157.9, - "mean": 45.1 - }, - "results": { - "1": { - "score": 0.2149, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 32.1 - }, - "2": { - "score": 0.1359, - "jev": 0.4454, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7345, - "unsupported": 359, - "errors": 0, - "median_ms": 205.8 - }, - "3": { - "score": 0.0315, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 434.5 - }, - "4": { - "score": 0.6099, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 22.5 - }, - "5": { - "score": 0.443, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 24.4 - }, - "6": { - "score": 0.5384, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 46.4 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 50.1 - }, - "10": { - "score": 0.4479, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 48.2 - }, - "11": { - "score": 0.2126, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 79.4 - }, - "12": { - "score": 0.2934, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 18.8 - }, - "20": { - "score": 0.4814, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 18.9 - }, - "21": { - "score": 0.4665, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 18.3 - }, - "22": { - "score": 0.024, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 113.7 - }, - "23": { - "score": 0.5044, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 25.8 - }, - "24": { - "score": 0.316, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 18.6 - }, - "25": { - "score": 0.2449, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 18.9 - }, - "26": { - "score": 0.5059, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 18.5 - }, - "27": { - "score": 0.3166, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 18.5 - }, - "28": { - "score": 0.5036, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 18.3 - }, - "29": { - "score": 0.3455, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 19.0 - }, - "30": { - "score": 0.2002, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 20.1 - }, - "31": { - "score": 0.086, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 22.1 - }, - "32": { - "score": 0.4787, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 21.2 - }, - "33": { - "score": 0.0158, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 29.1 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 19.1 - }, - "36": { - "score": 0.03, - "jev": 0.1588, - "metric": "nDCG@10", - "requests": 1297, - "answered": 987, - "unsupported": 310, - "errors": 0, - "median_ms": 173.7 - }, - "37": { - "score": 0.2128, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 23.2 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 101.8 - }, - "39": { - "score": 0.7487, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 20.3 - }, - "40": { - "score": 0.2832, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 18.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0679, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2832, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.52, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.37, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4047, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 18.8 - }, - "42": { - "score": 0.5504, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 26.1 - }, - "43": { - "score": 0.3895, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 20.1 - }, - "44": { - "score": 0.4956, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 18.7 - }, - "45": { - "score": 0.1657, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 19.2 - }, - "48": { - "score": 0.2906, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 32.2 - }, - "50": { - "score": 0.3252, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 28.8 - } - }, - "scores": { - "balanced_raw": 27.23, - "balanced_skill": 2.37, - "breadth_skill": 2.23 - }, - "frozen_scores": { - "balanced_raw": 22.33, - "balanced_skill": 1.6, - "breadth_skill": 1.55 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2887, - "skill": 0.026, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.3001, - "skill": 0.0616, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1178, - "skill": 0.0063, - "coverage": 0.8805, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.2943, - "skill": 0.0133, - "coverage": 0.9845, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3608, - "skill": 0.0111, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.2149, - "skill": 0.0, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.1295, - "skill": 0.0399, - "coverage": 0.9534, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.5385, - "skill": 0.0, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.4941, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.5828, - "headline": false - } - ] - }, - "11": { - "raw": 0.2126, - "skill": 0.0, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.4807, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4665, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0204, - "skill": 0.0127, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.511, - "skill": 0.0221, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.316, - "skill": 0.088, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2449, - "skill": 0.0, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2002, - "skill": 0.0318, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2851, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1152, - "headline": false - } - ] - }, - "31": { - "raw": 0.086, - "skill": 0.0045, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0228, - "skill": 0.0, - "coverage": 0.761, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2128, - "skill": 0.0127, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2832, - "skill": 0.0778, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0679, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2832, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.52, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.37, - "headline": false - } - ] - }, - "41": { - "raw": 0.4047, - "skill": 0.107, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3895, - "skill": 0.0313, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4956, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3252, - "skill": 0.0206, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.7486, - "pending": 0.24 - }, - { - "engine": "jevfire-uncapped", - "name": "Jevfire", - "long_name": "Jevfire Qwen3.8-27B", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 27781427952, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/kikoncuo/jevfire", - "base_model": "Qwen/Qwen3.8-27B", - "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", - "weights_repo": null, - "served_checkpoint": "Qwen/Qwen3.8-27B-FP8" - }, - "completed": 132422, - "counts": { - "ok": 132113, - "unsupported": 309 - }, - "calibration": { - "acc": 0.6685, - "conf": 0.7316, - "ece": 0.0631, - "brier": 0.4325, - "over95": 0.0147, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0164, - "conf": 0.171, - "acc": 0.0521 - }, - { - "lo": 0.2, - "w": 0.0325, - "conf": 0.2552, - "acc": 0.1665 - }, - { - "lo": 0.3, - "w": 0.0437, - "conf": 0.3573, - "acc": 0.2594 - }, - { - "lo": 0.4, - "w": 0.097, - "conf": 0.4404, - "acc": 0.2962 - }, - { - "lo": 0.5, - "w": 0.1427, - "conf": 0.5496, - "acc": 0.5063 - }, - { - "lo": 0.6, - "w": 0.0974, - "conf": 0.6485, - "acc": 0.5991 - }, - { - "lo": 0.7, - "w": 0.1013, - "conf": 0.7494, - "acc": 0.7231 - }, - { - "lo": 0.8, - "w": 0.1019, - "conf": 0.8542, - "acc": 0.7528 - }, - { - "lo": 0.9, - "w": 0.3673, - "conf": 0.9739, - "acc": 0.9302 - } - ], - "areas": { - "knowledge": { - "ece": 0.0776, - "acc": 0.5981, - "conf": 0.6757 - }, - "language": { - "ece": 0.0431, - "acc": 0.7578, - "conf": 0.798 - }, - "retrieval": { - "ece": 0.0794, - "acc": 0.7266, - "conf": 0.804 - }, - "tools": { - "ece": 0.0466, - "acc": 0.9297, - "conf": 0.9266 - }, - "arts": { - "ece": 0.0845, - "acc": 0.5374, - "conf": 0.6201 - } - } - }, - "gaps": { - "share": 0.0023, - "reasons": [ - { - "reason": "prompt longer than its 16,384-token context window", - "share": 0.0023 - } - ] - }, - "latency": { - "median": 82.5, - "p95": 1562.9, - "mean": 312.5 - }, - "results": { - "1": { - "score": 0.9557, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 191.0 - }, - "2": { - "score": 0.4445, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7684, - "unsupported": 20, - "errors": 0, - "median_ms": 1538.2 - }, - "3": { - "score": 0.8406, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 746.6 - }, - "4": { - "score": 0.7517, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 117.3 - }, - "5": { - "score": 0.8128, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 267.3 - }, - "6": { - "score": 0.7995, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 309.9 - }, - "9": { - "score": 0.475, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 1751.4 - }, - "10": { - "score": 0.6497, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 119.3 - }, - "11": { - "score": 0.7547, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 1616.2 - }, - "12": { - "score": 0.6355, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 73.6 - }, - "20": { - "score": 0.9492, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 73.7 - }, - "21": { - "score": 0.5784, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 73.2 - }, - "22": { - "score": 0.1321, - "jev": 0.1811, - "metric": "accuracy", - "requests": 2000, - "answered": 1999, - "unsupported": 1, - "errors": 0, - "median_ms": 752.3 - }, - "23": { - "score": 0.6046, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 82.6 - }, - "24": { - "score": 0.7971, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 73.5 - }, - "25": { - "score": 0.4694, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 73.3 - }, - "26": { - "score": 0.9848, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 73.6 - }, - "27": { - "score": 0.9599, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 73.2 - }, - "28": { - "score": 0.7451, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 73.8 - }, - "29": { - "score": 0.9046, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 74.2 - }, - "30": { - "score": 0.5489, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 74.8 - }, - "31": { - "score": 0.1732, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 116.1 - }, - "32": { - "score": 0.5944, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 117.6 - }, - "33": { - "score": 0.2752, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 404.7 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 74.8 - }, - "36": { - "score": 0.1933, - "jev": 0.1882, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1296, - "unsupported": 1, - "errors": 0, - "median_ms": 1949.1 - }, - "37": { - "score": 0.5351, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 75.5 - }, - "38": { - "score": 0.0322, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 1513.6 - }, - "39": { - "score": 0.8032, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 145.0 - }, - "40": { - "score": 0.5324, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 73.4, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3969, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5324, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.72, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.935, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.6081, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 73.8 - }, - "42": { - "score": 0.7522, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 81.9 - }, - "43": { - "score": 0.5368, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 74.0 - }, - "44": { - "score": 0.6178, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 74.3 - }, - "45": { - "score": 0.1198, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 75.8 - }, - "48": { - "score": 0.2129, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 98.7 - }, - "50": { - "score": 0.395, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 107.1 - } - }, - "scores": { - "balanced_raw": 55.74, - "balanced_skill": 40.86, - "breadth_skill": 39.45 - }, - "frozen_scores": { - "balanced_raw": 43.01, - "balanced_skill": 30.32, - "breadth_skill": 26.11 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.5247, - "skill": 0.3471, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.6317, - "skill": 0.4853, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.3641, - "skill": 0.2863, - "coverage": 0.9996, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7329, - "skill": 0.6179, - "coverage": 0.9991, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.5335, - "skill": 0.3063, - "coverage": 0.9999, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9557, - "skill": 0.9402, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4434, - "skill": 0.3861, - "coverage": 0.9974, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7996, - "skill": 0.5273, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7866, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8125, - "headline": false - } - ] - }, - "11": { - "raw": 0.7547, - "skill": 0.6453, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.9508, - "skill": 0.9016, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5784, - "skill": 0.1568, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.1287, - "skill": 0.1219, - "coverage": 0.9995, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.6146, - "skill": 0.2292, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.8022, - "skill": 0.7363, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.4694, - "skill": 0.2925, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.5489, - "skill": 0.4539, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5959, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.5019, - "headline": false - } - ] - }, - "31": { - "raw": 0.1732, - "skill": 0.0995, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1932, - "skill": 0.1557, - "coverage": 0.9992, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.5351, - "skill": 0.4169, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.5324, - "skill": 0.3985, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3969, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5324, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.72, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.935, - "headline": false - } - ] - }, - "41": { - "raw": 0.6081, - "skill": 0.4121, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.5368, - "skill": 0.2652, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6178, - "skill": 0.2356, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.395, - "skill": 0.1219, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.7598, - "pending": 0.24 - }, - { - "engine": "joshua-diffusion-full", - "name": "JoshuaSP diffusiongemma (open-jev)", - "long_name": "JoshuaSP diffgemma", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "diffusion", - "technique_name": "diffusion", - "served_params": 25823778864, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/JoshuaSP/open-jev", - "base_model": "google/diffusiongemma-26B-A4B-it", - "base_url": "https://huggingface.co/google/diffusiongemma-26B-A4B-it", - "weights_repo": null, - "served_checkpoint": "google/diffusiongemma-26B-A4B-it" - }, - "completed": 132107, - "counts": { - "ok": 132107 - }, - "calibration": { - "acc": 0.6598, - "conf": 0.8986, - "ece": 0.2388, - "brier": 0.5677, - "over95": 0.1628, - "n": 72590, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.002, - "conf": 0.0894, - "acc": 0.0 - }, - { - "lo": 0.1, - "w": 0.0143, - "conf": 0.1478, - "acc": 0.0 - }, - { - "lo": 0.2, - "w": 0.0117, - "conf": 0.2486, - "acc": 0.0881 - }, - { - "lo": 0.3, - "w": 0.015, - "conf": 0.3542, - "acc": 0.1417 - }, - { - "lo": 0.4, - "w": 0.0185, - "conf": 0.4495, - "acc": 0.1841 - }, - { - "lo": 0.5, - "w": 0.0359, - "conf": 0.5496, - "acc": 0.4329 - }, - { - "lo": 0.6, - "w": 0.0372, - "conf": 0.6459, - "acc": 0.4039 - }, - { - "lo": 0.7, - "w": 0.0422, - "conf": 0.7526, - "acc": 0.4219 - }, - { - "lo": 0.8, - "w": 0.065, - "conf": 0.8528, - "acc": 0.5294 - }, - { - "lo": 0.9, - "w": 0.7582, - "conf": 0.9876, - "acc": 0.7524 - } - ], - "areas": { - "knowledge": { - "ece": 0.2943, - "acc": 0.5937, - "conf": 0.888 - }, - "language": { - "ece": 0.1927, - "acc": 0.7809, - "conf": 0.9736 - }, - "retrieval": { - "ece": 0.2289, - "acc": 0.6582, - "conf": 0.886 - }, - "tools": { - "ece": 0.0448, - "acc": 0.9328, - "conf": 0.9743 - }, - "arts": { - "ece": 0.2749, - "acc": 0.513, - "conf": 0.788 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 269.1, - "p95": 1680.9, - "mean": 529.8 - }, - "results": { - "1": { - "score": 0.9652, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 275.1 - }, - "2": { - "score": 0.4139, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 1576.6 - }, - "3": { - "score": 0.8366, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 1438.7 - }, - "4": { - "score": 0.7428, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 317.1 - }, - "5": { - "score": 0.8349, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 466.6 - }, - "6": { - "score": 0.79, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 491.0 - }, - "9": { - "score": 0.3937, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 879.5 - }, - "10": { - "score": 0.4065, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 293.2 - }, - "11": { - "score": 0.7596, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 637.9 - }, - "12": { - "score": 0.6637, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 256.3 - }, - "20": { - "score": 0.8694, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 257.5 - }, - "21": { - "score": 0.6298, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 250.0 - }, - "22": { - "score": 0.025, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 1210.7 - }, - "23": { - "score": 0.5824, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 261.7 - }, - "24": { - "score": 0.7926, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 252.9 - }, - "25": { - "score": 0.449, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 253.8 - }, - "26": { - "score": 0.9823, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 250.1 - }, - "27": { - "score": 0.9445, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 251.6 - }, - "28": { - "score": 0.7364, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 252.2 - }, - "29": { - "score": 0.8328, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 260.3 - }, - "30": { - "score": 0.5027, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 253.8 - }, - "31": { - "score": 0.1416, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 279.2 - }, - "32": { - "score": 0.6117, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 287.1 - }, - "33": { - "score": 0.2745, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 287.7 - }, - "34": { - "score": 0.2, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 260.8 - }, - "36": { - "score": 0.1661, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 2212.4 - }, - "37": { - "score": 0.5337, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 262.7 - }, - "38": { - "score": 0.01, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 1671.1 - }, - "39": { - "score": 0.8904, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 262.6 - }, - "40": { - "score": 0.66, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 250.8, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.49, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.66, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.8, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.945, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.5569, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 257.9 - }, - "42": { - "score": 0.7843, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 267.9 - }, - "43": { - "score": 0.6474, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 246.1 - }, - "44": { - "score": 0.6778, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 256.4 - }, - "45": { - "score": 0.0978, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 260.7 - }, - "48": { - "score": 0.2959, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 278.2 - }, - "50": { - "score": 0.4499, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 274.3 - } - }, - "scores": { - "balanced_raw": 55.56, - "balanced_skill": 40.84, - "breadth_skill": 39.19 - }, - "frozen_scores": { - "balanced_raw": 42.7, - "balanced_skill": 30.27, - "breadth_skill": 25.61 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.5356, - "skill": 0.3755, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.6588, - "skill": 0.5167, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.3499, - "skill": 0.2713, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.723, - "skill": 0.6036, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.5105, - "skill": 0.2747, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9652, - "skill": 0.953, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4139, - "skill": 0.3536, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.79, - "skill": 0.5041, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.779, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8011, - "headline": false - } - ] - }, - "11": { - "raw": 0.7596, - "skill": 0.6523, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.8729, - "skill": 0.7459, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.6298, - "skill": 0.2595, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0263, - "skill": 0.0187, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5739, - "skill": 0.1477, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7954, - "skill": 0.7272, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.449, - "skill": 0.2653, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.5027, - "skill": 0.3995, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5686, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.4367, - "headline": false - } - ] - }, - "31": { - "raw": 0.1416, - "skill": 0.0651, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1661, - "skill": 0.1274, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.5337, - "skill": 0.4152, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.66, - "skill": 0.5626, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.49, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.66, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.8, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.945, - "headline": false - } - ] - }, - "41": { - "raw": 0.5569, - "skill": 0.3353, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.6474, - "skill": 0.4405, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6778, - "skill": 0.3556, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4499, - "skill": 0.2016, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "kev-0.5b-raised", - "name": "Kev 0.5B", - "long_name": "Kev 0.5B", - "variant": null, - "meta": { - "kind": "LoRA + head", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 494032768, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 37074383, - "model_url": "https://huggingface.co/jaredpalmer/kev-0.5b", - "code_url": "https://github.com/jaredpalmer/kev", - "base_model": "Qwen/Qwen2.5-0.5B", - "base_url": "https://huggingface.co/Qwen/Qwen2.5-0.5B", - "weights_repo": "jaredpalmer/kev-0.5b", - "served_checkpoint": "jaredpalmer/kev-0.5b" - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.3987, - "conf": 0.5922, - "ece": 0.1936, - "brier": 0.7513, - "over95": 0.0452, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.021, - "conf": 0.0765, - "acc": 0.0783 - }, - { - "lo": 0.1, - "w": 0.0364, - "conf": 0.1561, - "acc": 0.1027 - }, - { - "lo": 0.2, - "w": 0.0857, - "conf": 0.2401, - "acc": 0.0768 - }, - { - "lo": 0.3, - "w": 0.0802, - "conf": 0.3528, - "acc": 0.2485 - }, - { - "lo": 0.4, - "w": 0.1056, - "conf": 0.45, - "acc": 0.3719 - }, - { - "lo": 0.5, - "w": 0.1812, - "conf": 0.5482, - "acc": 0.4686 - }, - { - "lo": 0.6, - "w": 0.1447, - "conf": 0.6477, - "acc": 0.4949 - }, - { - "lo": 0.7, - "w": 0.1203, - "conf": 0.7481, - "acc": 0.4871 - }, - { - "lo": 0.8, - "w": 0.0974, - "conf": 0.8475, - "acc": 0.506 - }, - { - "lo": 0.9, - "w": 0.1275, - "conf": 0.964, - "acc": 0.495 - } - ], - "areas": { - "knowledge": { - "ece": 0.1776, - "acc": 0.3268, - "conf": 0.5044 - }, - "language": { - "ece": 0.1545, - "acc": 0.5085, - "conf": 0.6571 - }, - "retrieval": { - "ece": 0.265, - "acc": 0.5111, - "conf": 0.7722 - }, - "tools": { - "ece": 0.2432, - "acc": 0.3744, - "conf": 0.5648 - }, - "arts": { - "ece": 0.2483, - "acc": 0.371, - "conf": 0.6193 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 16.8, - "p95": 85.6, - "mean": 34.4 - }, - "results": { - "1": { - "score": 0.3878, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 17.8 - }, - "2": { - "score": 0.3098, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 94.5 - }, - "3": { - "score": 0.3346, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 76.0 - }, - "4": { - "score": 0.7576, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 19.5 - }, - "5": { - "score": 0.016, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 22.1 - }, - "6": { - "score": 0.4967, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 23.3 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 26.4 - }, - "10": { - "score": 0.4644, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 18.3 - }, - "11": { - "score": 0.3664, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 34.8 - }, - "12": { - "score": 0.2901, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 16.2 - }, - "20": { - "score": 0.5086, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 16.2 - }, - "21": { - "score": 0.4859, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 16.0 - }, - "22": { - "score": 0.0135, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 79.3 - }, - "23": { - "score": 0.5098, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 16.7 - }, - "24": { - "score": 0.3662, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 16.1 - }, - "25": { - "score": 0.1888, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 16.4 - }, - "26": { - "score": 0.6372, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 15.9 - }, - "27": { - "score": 0.3891, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 16.0 - }, - "28": { - "score": 0.5091, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 15.9 - }, - "29": { - "score": 0.3798, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 16.3 - }, - "30": { - "score": 0.1653, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 16.5 - }, - "31": { - "score": 0.0878, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 17.8 - }, - "32": { - "score": 0.3976, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 18.0 - }, - "33": { - "score": 0.0164, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 19.1 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 16.7 - }, - "36": { - "score": 0.0779, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 141.3 - }, - "37": { - "score": 0.209, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 16.6 - }, - "38": { - "score": 0.0029, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 75.6 - }, - "39": { - "score": 0.6744, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 16.8 - }, - "40": { - "score": 0.2742, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 15.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0256, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2742, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.42, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.345, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3375, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 16.4 - }, - "42": { - "score": 0.4004, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 17.0 - }, - "43": { - "score": 0.3947, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 16.3 - }, - "44": { - "score": 0.497, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 16.3 - }, - "45": { - "score": 0.1397, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 16.8 - }, - "48": { - "score": 0.4466, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 17.4 - }, - "50": { - "score": 0.2906, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 17.8 - } - }, - "scores": { - "balanced_raw": 30.34, - "balanced_skill": 5.22, - "breadth_skill": 4.71 - }, - "frozen_scores": { - "balanced_raw": 23.93, - "balanced_skill": 3.39, - "breadth_skill": 3.14 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2842, - "skill": 0.0347, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.3261, - "skill": 0.0521, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1435, - "skill": 0.0215, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3981, - "skill": 0.1374, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.365, - "skill": 0.0153, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3878, - "skill": 0.1736, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.3098, - "skill": 0.2387, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.4967, - "skill": 0.0, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.4347, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.5587, - "headline": false - } - ] - }, - "11": { - "raw": 0.3664, - "skill": 0.0838, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5168, - "skill": 0.0337, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4859, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0127, - "skill": 0.005, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5188, - "skill": 0.0376, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.3716, - "skill": 0.1621, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.1888, - "skill": 0.0, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.1653, - "skill": 0.0, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2358, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.0948, - "headline": false - } - ] - }, - "31": { - "raw": 0.0878, - "skill": 0.0065, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0779, - "skill": 0.0351, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.209, - "skill": 0.0079, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2742, - "skill": 0.0663, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0256, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2742, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.42, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.345, - "headline": false - } - ] - }, - "41": { - "raw": 0.3375, - "skill": 0.0063, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3947, - "skill": 0.0397, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.497, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.2906, - "skill": 0.0, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "kev-0.6b-raised", - "name": "Kev 0.6B", - "long_name": "Kev 0.6B", - "variant": null, - "meta": { - "kind": "LoRA + head", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 596049920, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 42522215, - "model_url": "https://huggingface.co/jaredpalmer/kev-0.6b", - "code_url": "https://github.com/jaredpalmer/kev", - "base_model": "Qwen/Qwen3-0.6B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3-0.6B-Base", - "weights_repo": "jaredpalmer/kev-0.6b", - "served_checkpoint": "jaredpalmer/kev-0.6b" - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.4258, - "conf": 0.652, - "ece": 0.2262, - "brier": 0.7426, - "over95": 0.033, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.002, - "conf": 0.0888, - "acc": 0.0189 - }, - { - "lo": 0.1, - "w": 0.0454, - "conf": 0.1396, - "acc": 0.0502 - }, - { - "lo": 0.2, - "w": 0.0585, - "conf": 0.2753, - "acc": 0.0513 - }, - { - "lo": 0.3, - "w": 0.0662, - "conf": 0.3552, - "acc": 0.2537 - }, - { - "lo": 0.4, - "w": 0.0808, - "conf": 0.451, - "acc": 0.3167 - }, - { - "lo": 0.5, - "w": 0.1422, - "conf": 0.5505, - "acc": 0.4386 - }, - { - "lo": 0.6, - "w": 0.1437, - "conf": 0.6489, - "acc": 0.4751 - }, - { - "lo": 0.7, - "w": 0.133, - "conf": 0.7487, - "acc": 0.4734 - }, - { - "lo": 0.8, - "w": 0.1445, - "conf": 0.8518, - "acc": 0.4926 - }, - { - "lo": 0.9, - "w": 0.1837, - "conf": 0.9538, - "acc": 0.6171 - } - ], - "areas": { - "knowledge": { - "ece": 0.2209, - "acc": 0.3571, - "conf": 0.578 - }, - "language": { - "ece": 0.2938, - "acc": 0.5015, - "conf": 0.7932 - }, - "retrieval": { - "ece": 0.0939, - "acc": 0.6308, - "conf": 0.7237 - }, - "tools": { - "ece": 0.3404, - "acc": 0.3866, - "conf": 0.7185 - }, - "arts": { - "ece": 0.1923, - "acc": 0.3719, - "conf": 0.5642 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 22.8, - "p95": 180.9, - "mean": 133.9 - }, - "results": { - "1": { - "score": 0.4486, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 23.8 - }, - "2": { - "score": 0.3549, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 194.4 - }, - "3": { - "score": 0.4016, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 160.9 - }, - "4": { - "score": 0.7005, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 25.5 - }, - "5": { - "score": 0.5679, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 28.1 - }, - "6": { - "score": 0.6425, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 41.3 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 38.6 - }, - "10": { - "score": 0.5854, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - }, - "11": { - "score": 0.3132, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 63.0 - }, - "12": { - "score": 0.2963, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 21.5 - }, - "20": { - "score": 0.4876, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 22.0 - }, - "21": { - "score": 0.4726, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 21.6 - }, - "22": { - "score": 0.0155, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 168.1 - }, - "23": { - "score": 0.5318, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 21.8 - }, - "24": { - "score": 0.4224, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 21.7 - }, - "25": { - "score": 0.2296, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 21.8 - }, - "26": { - "score": 0.734, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 21.3 - }, - "27": { - "score": 0.5179, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 22.2 - }, - "28": { - "score": 0.5083, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 21.6 - }, - "29": { - "score": 0.4167, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 22.4 - }, - "30": { - "score": 0.2604, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 22.3 - }, - "31": { - "score": 0.0808, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - }, - "32": { - "score": 0.4588, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - }, - "33": { - "score": 0.0079, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 25.0 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 22.3 - }, - "36": { - "score": 0.0912, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 323.6 - }, - "37": { - "score": 0.2091, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 22.7 - }, - "38": { - "score": 0.0014, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 138.5 - }, - "39": { - "score": 0.5313, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 22.7 - }, - "40": { - "score": 0.0632, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 21.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0408, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.0632, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.37, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.235, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.381, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 22.2 - }, - "42": { - "score": 0.3856, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 22.9 - }, - "43": { - "score": 0.3175, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 21.7 - }, - "44": { - "score": 0.5032, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 21.8 - }, - "45": { - "score": 0.1537, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 22.7 - }, - "48": { - "score": 0.3466, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 23.5 - }, - "50": { - "score": 0.3717, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - } - }, - "scores": { - "balanced_raw": 31.3, - "balanced_skill": 7.55, - "breadth_skill": 6.29 - }, - "frozen_scores": { - "balanced_raw": 25.29, - "balanced_skill": 5.38, - "breadth_skill": 4.73 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.3041, - "skill": 0.059, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2525, - "skill": 0.0261, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1501, - "skill": 0.0285, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.482, - "skill": 0.2321, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3763, - "skill": 0.0317, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.4486, - "skill": 0.2557, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.3549, - "skill": 0.2885, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.6425, - "skill": 0.1521, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.6366, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.6484, - "headline": false - } - ] - }, - "11": { - "raw": 0.3132, - "skill": 0.0068, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.4902, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4726, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0155, - "skill": 0.0078, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5314, - "skill": 0.0628, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.433, - "skill": 0.244, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2296, - "skill": 0.0, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2604, - "skill": 0.1034, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.326, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1948, - "headline": false - } - ] - }, - "31": { - "raw": 0.0808, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0912, - "skill": 0.049, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2091, - "skill": 0.008, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.0632, - "skill": 0.0, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0408, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.0632, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.37, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.235, - "headline": false - } - ] - }, - "41": { - "raw": 0.381, - "skill": 0.0715, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3175, - "skill": 0.0, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5032, - "skill": 0.0064, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3717, - "skill": 0.0882, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "kev-4b-raised", - "name": "Kev 4B", - "long_name": "Kev 4B", - "variant": null, - "meta": { - "kind": "LoRA + head", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 4659865088, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 135172799, - "model_url": "https://huggingface.co/jaredpalmer/kev-4b", - "code_url": "https://github.com/jaredpalmer/kev", - "base_model": "Qwen/Qwen3.5-4B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-4B-Base", - "weights_repo": "jaredpalmer/kev-4b", - "served_checkpoint": "jaredpalmer/kev-4b" - }, - "completed": 132422, - "counts": { - "ok": 132411, - "error": 11 - }, - "calibration": { - "acc": 0.5975, - "conf": 0.7972, - "ece": 0.1998, - "brier": 0.5873, - "over95": 0.0998, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0119, - "conf": 0.0808, - "acc": 0.0527 - }, - { - "lo": 0.1, - "w": 0.0153, - "conf": 0.1418, - "acc": 0.1444 - }, - { - "lo": 0.2, - "w": 0.0166, - "conf": 0.2604, - "acc": 0.1604 - }, - { - "lo": 0.3, - "w": 0.0384, - "conf": 0.3554, - "acc": 0.1695 - }, - { - "lo": 0.4, - "w": 0.0512, - "conf": 0.4513, - "acc": 0.2605 - }, - { - "lo": 0.5, - "w": 0.0837, - "conf": 0.5466, - "acc": 0.4282 - }, - { - "lo": 0.6, - "w": 0.0776, - "conf": 0.6493, - "acc": 0.4725 - }, - { - "lo": 0.7, - "w": 0.0835, - "conf": 0.7496, - "acc": 0.5142 - }, - { - "lo": 0.8, - "w": 0.1119, - "conf": 0.8536, - "acc": 0.6469 - }, - { - "lo": 0.9, - "w": 0.51, - "conf": 0.9781, - "acc": 0.7536 - } - ], - "areas": { - "knowledge": { - "ece": 0.2101, - "acc": 0.538, - "conf": 0.7477 - }, - "language": { - "ece": 0.1905, - "acc": 0.695, - "conf": 0.8855 - }, - "retrieval": { - "ece": 0.1279, - "acc": 0.7092, - "conf": 0.8362 - }, - "tools": { - "ece": 0.1915, - "acc": 0.7713, - "conf": 0.9571 - }, - "arts": { - "ece": 0.2512, - "acc": 0.4336, - "conf": 0.6849 - } - } - }, - "gaps": { - "share": 0.0001, - "reasons": [ - { - "reason": "ran out of GPU memory on the longest prompts", - "share": 0.0001 - } - ] - }, - "latency": { - "median": 53.8, - "p95": 627.1, - "mean": 314.7 - }, - "results": { - "1": { - "score": 0.9227, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 105.4 - }, - "2": { - "score": 0.4274, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 737.2 - }, - "3": { - "score": 0.5118, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 296.1 - }, - "4": { - "score": 0.8224, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 55.8 - }, - "5": { - "score": 0.7936, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 63.6 - }, - "6": { - "score": 0.8007, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 127.2 - }, - "9": { - "score": 0.275, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 179.2 - }, - "10": { - "score": 0.5693, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 107.9 - }, - "11": { - "score": 0.6224, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 155.8 - }, - "12": { - "score": 0.4621, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 51.5 - }, - "20": { - "score": 0.59, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 51.2 - }, - "21": { - "score": 0.5293, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 50.5 - }, - "22": { - "score": 0.0115, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 317.0 - }, - "23": { - "score": 0.5884, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 57.1 - }, - "24": { - "score": 0.6932, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 52.0 - }, - "25": { - "score": 0.3878, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 52.2 - }, - "26": { - "score": 0.971, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 51.1 - }, - "27": { - "score": 0.8985, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 51.8 - }, - "28": { - "score": 0.6851, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 50.8 - }, - "29": { - "score": 0.7537, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 51.8 - }, - "30": { - "score": 0.4011, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 51.6 - }, - "31": { - "score": 0.0982, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 54.1 - }, - "32": { - "score": 0.5824, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 54.3 - }, - "33": { - "score": 0.2685, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 109.0 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 51.8 - }, - "36": { - "score": 0.1545, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 1333.1 - }, - "37": { - "score": 0.3848, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 51.7 - }, - "38": { - "score": 0.01, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 311.4 - }, - "39": { - "score": 0.8179, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 53.5 - }, - "40": { - "score": 0.4231, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 51.1, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2552, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4231, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.495, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.755, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.5094, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 51.5 - }, - "42": { - "score": 0.7202, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 104.7 - }, - "43": { - "score": 0.4684, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 51.6 - }, - "44": { - "score": 0.6022, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 51.4 - }, - "45": { - "score": 0.1118, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 52.1 - }, - "48": { - "score": 0.1859, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 105.6 - }, - "50": { - "score": 0.4177, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 105.8 - } - }, - "scores": { - "balanced_raw": 47.43, - "balanced_skill": 28.86, - "breadth_skill": 25.67 - }, - "frozen_scores": { - "balanced_raw": 36.32, - "balanced_skill": 20.15, - "breadth_skill": 16.82 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4429, - "skill": 0.2401, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.5183, - "skill": 0.3253, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2696, - "skill": 0.1718, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7169, - "skill": 0.598, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4239, - "skill": 0.1078, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9227, - "skill": 0.8956, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4274, - "skill": 0.3684, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.8007, - "skill": 0.53, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7879, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8135, - "headline": false - } - ] - }, - "11": { - "raw": 0.6224, - "skill": 0.4539, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5953, - "skill": 0.1905, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5293, - "skill": 0.0586, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0114, - "skill": 0.0037, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5657, - "skill": 0.1314, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7, - "skill": 0.6, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3878, - "skill": 0.1837, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4011, - "skill": 0.2783, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4943, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3078, - "headline": false - } - ] - }, - "31": { - "raw": 0.0982, - "skill": 0.0178, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1545, - "skill": 0.1152, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.3848, - "skill": 0.2283, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.4231, - "skill": 0.2578, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2552, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4231, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.495, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.755, - "headline": false - } - ] - }, - "41": { - "raw": 0.5094, - "skill": 0.2641, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.4684, - "skill": 0.1566, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6022, - "skill": 0.2044, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4177, - "skill": 0.1548, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "kev-8b-raised", - "name": "Kev 8B", - "long_name": "Kev 8B", - "variant": null, - "meta": { - "kind": "LoRA + head", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 8190735360, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 183046367, - "model_url": "https://huggingface.co/jaredpalmer/kev-8b", - "code_url": "https://github.com/jaredpalmer/kev", - "base_model": "Qwen/Qwen3-8B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3-8B-Base", - "weights_repo": "jaredpalmer/kev-8b", - "served_checkpoint": "jaredpalmer/kev-8b" - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.6006, - "conf": 0.7651, - "ece": 0.1645, - "brier": 0.5671, - "over95": 0.0769, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0116, - "conf": 0.082, - "acc": 0.0418 - }, - { - "lo": 0.1, - "w": 0.0316, - "conf": 0.1484, - "acc": 0.0986 - }, - { - "lo": 0.2, - "w": 0.0227, - "conf": 0.2511, - "acc": 0.1388 - }, - { - "lo": 0.3, - "w": 0.0324, - "conf": 0.3555, - "acc": 0.2986 - }, - { - "lo": 0.4, - "w": 0.0471, - "conf": 0.4502, - "acc": 0.3684 - }, - { - "lo": 0.5, - "w": 0.0999, - "conf": 0.5484, - "acc": 0.4584 - }, - { - "lo": 0.6, - "w": 0.0999, - "conf": 0.6498, - "acc": 0.5223 - }, - { - "lo": 0.7, - "w": 0.1005, - "conf": 0.7509, - "acc": 0.5478 - }, - { - "lo": 0.8, - "w": 0.1154, - "conf": 0.8515, - "acc": 0.651 - }, - { - "lo": 0.9, - "w": 0.4387, - "conf": 0.9745, - "acc": 0.7718 - } - ], - "areas": { - "knowledge": { - "ece": 0.2037, - "acc": 0.5305, - "conf": 0.7342 - }, - "language": { - "ece": 0.1714, - "acc": 0.6855, - "conf": 0.857 - }, - "retrieval": { - "ece": 0.1276, - "acc": 0.6869, - "conf": 0.7788 - }, - "tools": { - "ece": 0.1146, - "acc": 0.8233, - "conf": 0.935 - }, - "arts": { - "ece": 0.1348, - "acc": 0.4752, - "conf": 0.61 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 38.3, - "p95": 652.7, - "mean": 264.7 - }, - "results": { - "1": { - "score": 0.8855, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 54.3 - }, - "2": { - "score": 0.4293, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 705.1 - }, - "3": { - "score": 0.6673, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 615.9 - }, - "4": { - "score": 0.8266, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 64.1 - }, - "5": { - "score": 0.7679, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 108.4 - }, - "6": { - "score": 0.7993, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 159.0 - }, - "9": { - "score": 0.0063, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 151.3 - }, - "10": { - "score": 0.5888, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 75.9 - }, - "11": { - "score": 0.4852, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 231.2 - }, - "12": { - "score": 0.4931, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 29.9 - }, - "20": { - "score": 0.74, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 30.0 - }, - "21": { - "score": 0.559, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 29.7 - }, - "22": { - "score": 0.0535, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 629.9 - }, - "23": { - "score": 0.5762, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 29.9 - }, - "24": { - "score": 0.6942, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 30.1 - }, - "25": { - "score": 0.3316, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 30.1 - }, - "26": { - "score": 0.9701, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 29.5 - }, - "27": { - "score": 0.9096, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 29.9 - }, - "28": { - "score": 0.6851, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 29.3 - }, - "29": { - "score": 0.795, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 30.1 - }, - "30": { - "score": 0.4117, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 29.7 - }, - "31": { - "score": 0.1064, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 61.9 - }, - "32": { - "score": 0.5479, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 73.7 - }, - "33": { - "score": 0.1109, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 62.6 - }, - "34": { - "score": 0, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 30.2 - }, - "36": { - "score": 0.1477, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 1112.1 - }, - "37": { - "score": 0.4023, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 38.3 - }, - "38": { - "score": 0.0043, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 494.4 - }, - "39": { - "score": 0.8675, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 30.4 - }, - "40": { - "score": 0.3896, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 29.6, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3735, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3896, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.515, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.585, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4704, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 30.2 - }, - "42": { - "score": 0.7035, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 45.3 - }, - "43": { - "score": 0.4404, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 29.9 - }, - "44": { - "score": 0.5994, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 29.7 - }, - "45": { - "score": 0.1337, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 29.9 - }, - "48": { - "score": 0.2377, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 55.8 - }, - "50": { - "score": 0.4171, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 62.3 - } - }, - "scores": { - "balanced_raw": 46.55, - "balanced_skill": 27.89, - "breadth_skill": 25.64 - }, - "frozen_scores": { - "balanced_raw": 36.07, - "balanced_skill": 19.95, - "breadth_skill": 17.19 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4348, - "skill": 0.2272, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4484, - "skill": 0.2253, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.275, - "skill": 0.1792, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7047, - "skill": 0.5808, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4646, - "skill": 0.1823, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8855, - "skill": 0.8454, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4293, - "skill": 0.3705, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7993, - "skill": 0.5265, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7871, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8115, - "headline": false - } - ] - }, - "11": { - "raw": 0.4852, - "skill": 0.2555, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.7368, - "skill": 0.4735, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.559, - "skill": 0.118, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0473, - "skill": 0.0398, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5631, - "skill": 0.1261, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7196, - "skill": 0.6261, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3316, - "skill": 0.1088, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4117, - "skill": 0.2904, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4966, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3268, - "headline": false - } - ] - }, - "31": { - "raw": 0.1064, - "skill": 0.0267, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1477, - "skill": 0.1081, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.4023, - "skill": 0.2503, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.3896, - "skill": 0.2148, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3735, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3896, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.515, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.585, - "headline": false - } - ] - }, - "41": { - "raw": 0.4704, - "skill": 0.2055, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.4404, - "skill": 0.1121, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5994, - "skill": 0.1988, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4171, - "skill": 0.154, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "kev-9b-raised", - "name": "Kev 9B", - "long_name": "Kev 9B", - "variant": null, - "meta": { - "kind": "LoRA + head", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 9653104368, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 181572591, - "model_url": "https://huggingface.co/jaredpalmer/kev-9b", - "code_url": "https://github.com/jaredpalmer/kev", - "base_model": "Qwen/Qwen3.5-9B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-9B-Base", - "weights_repo": "jaredpalmer/kev-9b", - "served_checkpoint": "jaredpalmer/kev-9b" - }, - "completed": 132422, - "counts": { - "ok": 132406, - "error": 16 - }, - "calibration": { - "acc": 0.6323, - "conf": 0.7962, - "ece": 0.1639, - "brier": 0.5326, - "over95": 0.0835, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0129, - "conf": 0.0886, - "acc": 0.0389 - }, - { - "lo": 0.1, - "w": 0.0313, - "conf": 0.138, - "acc": 0.0733 - }, - { - "lo": 0.2, - "w": 0.0158, - "conf": 0.2498, - "acc": 0.1809 - }, - { - "lo": 0.3, - "w": 0.0294, - "conf": 0.3541, - "acc": 0.2577 - }, - { - "lo": 0.4, - "w": 0.036, - "conf": 0.4538, - "acc": 0.3623 - }, - { - "lo": 0.5, - "w": 0.0909, - "conf": 0.5482, - "acc": 0.4134 - }, - { - "lo": 0.6, - "w": 0.071, - "conf": 0.65, - "acc": 0.4996 - }, - { - "lo": 0.7, - "w": 0.0865, - "conf": 0.7519, - "acc": 0.5815 - }, - { - "lo": 0.8, - "w": 0.1103, - "conf": 0.8548, - "acc": 0.6541 - }, - { - "lo": 0.9, - "w": 0.5159, - "conf": 0.9783, - "acc": 0.7958 - } - ], - "areas": { - "knowledge": { - "ece": 0.2082, - "acc": 0.5672, - "conf": 0.7754 - }, - "language": { - "ece": 0.1307, - "acc": 0.7554, - "conf": 0.8854 - }, - "retrieval": { - "ece": 0.116, - "acc": 0.7455, - "conf": 0.8567 - }, - "tools": { - "ece": 0.1603, - "acc": 0.7877, - "conf": 0.9434 - }, - "arts": { - "ece": 0.1379, - "acc": 0.4586, - "conf": 0.5927 - } - } - }, - "gaps": { - "share": 0.0001, - "reasons": [ - { - "reason": "ran out of GPU memory on the longest prompts", - "share": 0.0001 - } - ] - }, - "latency": { - "median": 54.1, - "p95": 948.2, - "mean": 395.3 - }, - "results": { - "1": { - "score": 0.9451, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 106.0 - }, - "2": { - "score": 0.4454, - "jev": 0.4462, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7703, - "unsupported": 0, - "errors": 1, - "median_ms": 1117.4 - }, - "3": { - "score": 0.563, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 445.8 - }, - "4": { - "score": 0.8483, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 62.6 - }, - "5": { - "score": 0.7903, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 101.0 - }, - "6": { - "score": 0.8005, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 173.0 - }, - "9": { - "score": 0.25, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 233.6 - }, - "10": { - "score": 0.6398, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 121.5 - }, - "11": { - "score": 0.5777, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 232.2 - }, - "12": { - "score": 0.5626, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 51.0 - }, - "20": { - "score": 0.67, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 49.4 - }, - "21": { - "score": 0.5575, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 48.9 - }, - "22": { - "score": 0.1075, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 449.8 - }, - "23": { - "score": 0.5632, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 54.5 - }, - "24": { - "score": 0.7529, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 49.6 - }, - "25": { - "score": 0.3878, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 50.5 - }, - "26": { - "score": 0.9773, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 50.0 - }, - "27": { - "score": 0.9369, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 49.9 - }, - "28": { - "score": 0.7316, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 49.4 - }, - "29": { - "score": 0.8192, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 50.4 - }, - "30": { - "score": 0.4871, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 49.9 - }, - "31": { - "score": 0.1116, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 58.0 - }, - "32": { - "score": 0.5785, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 69.6 - }, - "33": { - "score": 0.2673, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 126.1 - }, - "34": { - "score": 0, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 50.4 - }, - "36": { - "score": 0.1544, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 2001.9 - }, - "37": { - "score": 0.4922, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 52.3 - }, - "38": { - "score": 0.0343, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 457.9 - }, - "39": { - "score": 0.8838, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 53.1 - }, - "40": { - "score": 0.4708, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 50.8, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3118, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4708, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.545, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.705, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.5542, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 51.4 - }, - "42": { - "score": 0.7486, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 104.9 - }, - "43": { - "score": 0.5123, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 51.4 - }, - "44": { - "score": 0.62, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 51.0 - }, - "45": { - "score": 0.1038, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 51.7 - }, - "48": { - "score": 0.176, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 105.8 - }, - "50": { - "score": 0.3944, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 104.9 - } - }, - "scores": { - "balanced_raw": 50.48, - "balanced_skill": 32.96, - "breadth_skill": 30.54 - }, - "frozen_scores": { - "balanced_raw": 38.55, - "balanced_skill": 23.23, - "breadth_skill": 19.59 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4801, - "skill": 0.2915, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.5342, - "skill": 0.3466, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.3233, - "skill": 0.2391, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7303, - "skill": 0.6146, - "coverage": 1.0, - "pending": 0.0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4558, - "skill": 0.1565, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9451, - "skill": 0.9259, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4454, - "skill": 0.3883, - "coverage": 0.9999, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.8005, - "skill": 0.5296, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7873, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8137, - "headline": false - } - ] - }, - "11": { - "raw": 0.5777, - "skill": 0.3892, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.6722, - "skill": 0.3444, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5575, - "skill": 0.1149, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.1012, - "skill": 0.0942, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5539, - "skill": 0.1079, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7619, - "skill": 0.6825, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3878, - "skill": 0.1837, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4871, - "skill": 0.384, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5853, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3889, - "headline": false - } - ] - }, - "31": { - "raw": 0.1116, - "skill": 0.0324, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1544, - "skill": 0.1151, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.4922, - "skill": 0.363, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.4708, - "skill": 0.3192, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3118, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4708, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.545, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.705, - "headline": false - } - ] - }, - "41": { - "raw": 0.5542, - "skill": 0.3314, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.5123, - "skill": 0.2262, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.62, - "skill": 0.24, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3944, - "skill": 0.1211, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "laya", - "name": "Laya", - "long_name": "Laya", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "encoder", - "technique_name": "encoder", - "served_params": 421293830, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 2329053944, - "model_url": "https://huggingface.co/convaiinnovations/laya", - "code_url": "https://github.com/NandhaKishorM/laya", - "base_model": null, - "base_url": null, - "weights_repo": "convaiinnovations/laya", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 65599, - "unsupported": 66823 - }, - "calibration": { - "acc": 0.492, - "conf": 0.5729, - "ece": 0.1308, - "brier": 0.6452, - "over95": 0.0143, - "n": 63375, - "benchmarks": 25, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0078, - "conf": 0.1704, - "acc": 0.1205 - }, - { - "lo": 0.2, - "w": 0.0602, - "conf": 0.2691, - "acc": 0.1763 - }, - { - "lo": 0.3, - "w": 0.1957, - "conf": 0.3459, - "acc": 0.4614 - }, - { - "lo": 0.4, - "w": 0.1354, - "conf": 0.4392, - "acc": 0.4565 - }, - { - "lo": 0.5, - "w": 0.1987, - "conf": 0.5475, - "acc": 0.4712 - }, - { - "lo": 0.6, - "w": 0.1164, - "conf": 0.6465, - "acc": 0.5095 - }, - { - "lo": 0.7, - "w": 0.0854, - "conf": 0.7495, - "acc": 0.5565 - }, - { - "lo": 0.8, - "w": 0.1016, - "conf": 0.8541, - "acc": 0.6217 - }, - { - "lo": 0.9, - "w": 0.0989, - "conf": 0.9447, - "acc": 0.6547 - } - ], - "areas": { - "knowledge": { - "ece": 0.1859, - "acc": 0.4007, - "conf": 0.502 - }, - "language": { - "ece": 0.1271, - "acc": 0.6437, - "conf": 0.6819 - }, - "retrieval": { - "ece": 0.0945, - "acc": 0.286, - "conf": 0.3805 - }, - "tools": { - "ece": 0.0575, - "acc": 0.7949, - "conf": 0.8231 - }, - "arts": { - "ece": 0.1056, - "acc": 0.5012, - "conf": 0.6068 - } - } - }, - "gaps": { - "share": 0.5046, - "reasons": [ - { - "reason": "instructions longer than its native budget (no truncation allowed)", - "share": 0.1948 - }, - { - "reason": "answer options longer than its per-option budget (no truncation allowed)", - "share": 0.1874 - }, - { - "reason": "prompt longer than its context window (no truncation allowed)", - "share": 0.1224 - } - ] - }, - "latency": { - "median": 18.9, - "p95": 74.8, - "mean": 23.4 - }, - "results": { - "1": { - "score": 0.8374, - "jev": 0.9863, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 732, - "unsupported": 962, - "errors": 0, - "median_ms": 19.0 - }, - "2": { - "score": 0.0524, - "jev": 0.4065, - "metric": "nDCG@10", - "requests": 7704, - "answered": 34, - "unsupported": 7670, - "errors": 0, - "median_ms": 66.2 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": null, - "jev": null, - "metric": null, - "requests": 3080, - "answered": 0, - "unsupported": 3080, - "errors": 0, - "median_ms": null - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": null, - "jev": null, - "metric": null, - "requests": 10000, - "answered": 0, - "unsupported": 10000, - "errors": 0, - "median_ms": null - }, - "9": { - "score": null, - "jev": null, - "metric": null, - "requests": 160, - "answered": 0, - "unsupported": 160, - "errors": 0, - "median_ms": null - }, - "10": { - "score": null, - "jev": null, - "metric": null, - "requests": 2500, - "answered": 0, - "unsupported": 2500, - "errors": 0, - "median_ms": null - }, - "11": { - "score": 0.2636, - "jev": 0.5909, - "metric": "macro-F1", - "requests": 123, - "answered": 2, - "unsupported": 121, - "errors": 0, - "median_ms": 62.1 - }, - "12": { - "score": 0.4873, - "jev": 0.7476, - "metric": "macro-F1", - "requests": 3200, - "answered": 3169, - "unsupported": 31, - "errors": 0, - "median_ms": 18.8 - }, - "20": { - "score": 0.51, - "jev": 0.8942, - "metric": "accuracy", - "requests": 5000, - "answered": 2271, - "unsupported": 2729, - "errors": 0, - "median_ms": 18.7 - }, - "21": { - "score": 0.4722, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 17.0 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.5234, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 19.5 - }, - "24": { - "score": 0.3029, - "jev": 0.9233, - "metric": "accuracy", - "requests": 14033, - "answered": 12029, - "unsupported": 2004, - "errors": 0, - "median_ms": 18.4 - }, - "25": { - "score": 0.2623, - "jev": 0.8197, - "metric": "accuracy", - "requests": 196, - "answered": 122, - "unsupported": 74, - "errors": 0, - "median_ms": 18.9 - }, - "26": { - "score": 0.4699, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2375, - "unsupported": 1, - "errors": 0, - "median_ms": 17.0 - }, - "27": { - "score": 0.2858, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 18.3 - }, - "28": { - "score": 0.5051, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 17.0 - }, - "29": { - "score": 0.3324, - "jev": 0.8679, - "metric": "accuracy", - "requests": 10042, - "answered": 3315, - "unsupported": 6727, - "errors": 0, - "median_ms": 18.8 - }, - "30": { - "score": 0.2165, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 19.0 - }, - "31": { - "score": 0.3904, - "jev": 0.4795, - "metric": "accuracy", - "requests": 5000, - "answered": 438, - "unsupported": 4562, - "errors": 0, - "median_ms": 19.2 - }, - "32": { - "score": null, - "jev": null, - "metric": null, - "requests": 752, - "answered": 0, - "unsupported": 752, - "errors": 0, - "median_ms": null - }, - "33": { - "score": 0.0045, - "jev": 0.3163, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 898, - "unsupported": 752, - "errors": 0, - "median_ms": 25.6 - }, - "34": { - "score": 0.4286, - "jev": 0.5714, - "metric": "accuracy", - "requests": 10, - "answered": 7, - "unsupported": 3, - "errors": 0, - "median_ms": 18.9 - }, - "36": { - "score": 0.024, - "jev": 0.0558, - "metric": "nDCG@10", - "requests": 1297, - "answered": 7, - "unsupported": 1290, - "errors": 0, - "median_ms": 82.3 - }, - "37": { - "score": 0.2501, - "jev": 0.5511, - "metric": "macro-F1", - "requests": 5000, - "answered": 3316, - "unsupported": 1684, - "errors": 0, - "median_ms": 19.3 - }, - "38": { - "score": 0.0021, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 76.3 - }, - "39": { - "score": 0.6098, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 19.9 - }, - "40": { - "score": 0.149, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 25, - "errors": 0, - "median_ms": 17.2, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0184, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.149, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.525, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.38, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4066, - "jev": 0.6586, - "metric": "macro-F1", - "requests": 3006, - "answered": 2237, - "unsupported": 769, - "errors": 0, - "median_ms": 18.8 - }, - "42": { - "score": 0.4797, - "jev": 0.848, - "metric": "macro-F1", - "requests": 5500, - "answered": 3209, - "unsupported": 2291, - "errors": 0, - "median_ms": 19.4 - }, - "43": { - "score": 0.4089, - "jev": 0.7375, - "metric": "accuracy", - "requests": 570, - "answered": 560, - "unsupported": 10, - "errors": 0, - "median_ms": 18.8 - }, - "44": { - "score": 0.5356, - "jev": 0.7395, - "metric": "accuracy", - "requests": 5000, - "answered": 4472, - "unsupported": 528, - "errors": 0, - "median_ms": 19.0 - }, - "45": { - "score": 0.1549, - "jev": 0.2389, - "metric": "accuracy", - "requests": 501, - "answered": 226, - "unsupported": 275, - "errors": 0, - "median_ms": 19.1 - }, - "48": { - "score": 0.3935, - "jev": 0.1554, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 2428, - "unsupported": 7711, - "errors": 0, - "median_ms": 19.9 - }, - "50": { - "score": null, - "jev": null, - "metric": null, - "requests": 1676, - "answered": 0, - "unsupported": 1676, - "errors": 0, - "median_ms": null - } - }, - "scores": { - "balanced_raw": 16.39, - "balanced_skill": 1.46, - "breadth_skill": 1.34 - }, - "frozen_scores": { - "balanced_raw": 15.06, - "balanced_skill": 1.23, - "breadth_skill": 1.17 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.262, - "skill": 0.023, - "coverage": 0.7407, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.1414, - "skill": 0.0, - "coverage": 0.5868, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.0632, - "skill": 0.0, - "coverage": 0.3343, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.1207, - "skill": 0.0462, - "coverage": 0.1455, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.232, - "skill": 0.0038, - "coverage": 0.4804, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3619, - "skill": 0.1385, - "coverage": 0.4321, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0002, - "skill": 0.0, - "coverage": 0.0044, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0, - "headline": false - } - ] - }, - "11": { - "raw": 0.0048, - "skill": 0.0, - "coverage": 0.0163, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.1781, - "skill": 0.0, - "coverage": 0.402, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4722, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5095, - "skill": 0.019, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.2775, - "skill": 0.0367, - "coverage": 0.8572, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.1633, - "skill": 0.0, - "coverage": 0.6224, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2165, - "skill": 0.0506, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2911, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1418, - "headline": false - } - ] - }, - "31": { - "raw": 0.0342, - "skill": 0.0, - "coverage": 0.0876, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0001, - "skill": 0.0, - "coverage": 0.0054, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1262, - "skill": 0.0, - "coverage": 0.6632, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.149, - "skill": 0.0, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0184, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.149, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.525, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.38, - "headline": false - } - ] - }, - "41": { - "raw": 0.2705, - "skill": 0.0, - "coverage": 0.7442, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.4018, - "skill": 0.0508, - "coverage": 0.9825, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.479, - "skill": 0.0, - "coverage": 0.8944, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.3872, - "pending": 0.24 - }, - { - "engine": "lfm2600", - "name": "LFM2.5-2.6B-RLCD", - "long_name": "LFM2.5-2.6B-RLCD", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 2697198592, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 5394427456, - "model_url": "https://huggingface.co/monotykamary/LFM2.5-2.6B-RLCD", - "code_url": null, - "base_model": "LiquidAI/LFM2.5-2.6B-Base", - "base_url": "https://huggingface.co/LiquidAI/LFM2.5-2.6B-Base", - "weights_repo": "monotykamary/LFM2.5-2.6B-RLCD", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 112416, - "unsupported": 20006 - }, - "calibration": { - "acc": 0.405, - "conf": 0.6571, - "ece": 0.2521, - "brier": 0.7753, - "over95": 0.0474, - "n": 20276, - "benchmarks": 29, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.005, - "conf": 0.1788, - "acc": 0.1184 - }, - { - "lo": 0.2, - "w": 0.0423, - "conf": 0.2587, - "acc": 0.128 - }, - { - "lo": 0.3, - "w": 0.0871, - "conf": 0.3511, - "acc": 0.1703 - }, - { - "lo": 0.4, - "w": 0.1054, - "conf": 0.4473, - "acc": 0.2831 - }, - { - "lo": 0.5, - "w": 0.1879, - "conf": 0.5506, - "acc": 0.4268 - }, - { - "lo": 0.6, - "w": 0.1363, - "conf": 0.6495, - "acc": 0.4591 - }, - { - "lo": 0.7, - "w": 0.1316, - "conf": 0.7475, - "acc": 0.4624 - }, - { - "lo": 0.8, - "w": 0.1313, - "conf": 0.8384, - "acc": 0.3841 - }, - { - "lo": 0.9, - "w": 0.1732, - "conf": 0.9652, - "acc": 0.5794 - } - ], - "areas": { - "knowledge": { - "ece": 0.2278, - "acc": 0.4006, - "conf": 0.6285 - }, - "language": { - "ece": 0.2974, - "acc": 0.3856, - "conf": 0.683 - }, - "retrieval": { - "ece": 0.2867, - "acc": 0.3718, - "conf": 0.6584 - }, - "tools": { - "ece": 0.4374, - "acc": 0.3577, - "conf": 0.7951 - }, - "arts": { - "ece": 0.1048, - "acc": 0.5404, - "conf": 0.6452 - } - } - }, - "gaps": { - "share": 0.1511, - "reasons": [ - { - "reason": "prompt longer than its 4,096-token context window", - "share": 0.0839 - }, - { - "reason": "more questions per request than the model accepts", - "share": 0.0414 - }, - { - "reason": "answer options longer than its per-option budget (no truncation allowed)", - "share": 0.0258 - } - ] - }, - "latency": { - "median": 38.7, - "p95": 94.1, - "mean": 47.2 - }, - "results": { - "1": { - "score": 0.3877, - "jev": 0.9574, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1692, - "unsupported": 2, - "errors": 0, - "median_ms": 43.2 - }, - "2": { - "score": null, - "jev": null, - "metric": null, - "requests": 7704, - "answered": 0, - "unsupported": 7704, - "errors": 0, - "median_ms": null - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": 0.1778, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 66.4 - }, - "5": { - "score": 0.0732, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 94.6 - }, - "6": { - "score": 0.5914, - "jev": 0.7979, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 9956, - "unsupported": 44, - "errors": 0, - "median_ms": 69.7 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 107.2 - }, - "10": { - "score": 0.5294, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 48.7 - }, - "11": { - "score": 0.3085, - "jev": 0.7139, - "metric": "macro-F1", - "requests": 123, - "answered": 78, - "unsupported": 45, - "errors": 0, - "median_ms": 109.0 - }, - "12": { - "score": 0.3486, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 38.2 - }, - "20": { - "score": 0.6, - "jev": 0.9056, - "metric": "accuracy", - "requests": 5000, - "answered": 4958, - "unsupported": 42, - "errors": 0, - "median_ms": 38.1 - }, - "21": { - "score": 0.4981, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 37.8 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.512, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 38.3 - }, - "24": { - "score": 0.4911, - "jev": 0.9183, - "metric": "accuracy", - "requests": 14033, - "answered": 13960, - "unsupported": 73, - "errors": 0, - "median_ms": 38.2 - }, - "25": { - "score": 0.3495, - "jev": 0.7903, - "metric": "accuracy", - "requests": 196, - "answered": 186, - "unsupported": 10, - "errors": 0, - "median_ms": 38.4 - }, - "26": { - "score": 0.8131, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 38.2 - }, - "27": { - "score": 0.6596, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 38.3 - }, - "28": { - "score": 0.5083, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 37.7 - }, - "29": { - "score": 0.4511, - "jev": 0.9424, - "metric": "accuracy", - "requests": 10042, - "answered": 9410, - "unsupported": 632, - "errors": 0, - "median_ms": 38.5 - }, - "30": { - "score": 0.2051, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 39.0 - }, - "31": { - "score": 0.1048, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 55.7 - }, - "32": { - "score": 0.4707, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 50.9 - }, - "33": { - "score": 0.0006, - "jev": 0.2626, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1626, - "unsupported": 24, - "errors": 0, - "median_ms": 52.8 - }, - "34": { - "score": 0.2, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 39.7 - }, - "36": { - "score": null, - "jev": null, - "metric": null, - "requests": 1297, - "answered": 0, - "unsupported": 1297, - "errors": 0, - "median_ms": null - }, - "37": { - "score": 0.2226, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 37.2 - }, - "38": { - "score": null, - "jev": null, - "metric": null, - "requests": 5479, - "answered": 0, - "unsupported": 5479, - "errors": 0, - "median_ms": null - }, - "39": { - "score": 0.3685, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 39.6 - }, - "40": { - "score": 0.2629, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 6, - "errors": 0, - "median_ms": 36.6, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2773, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2629, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.535, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3628, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 37.7 - }, - "42": { - "score": 0.3342, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 39.7 - }, - "43": { - "score": 0.3673, - "jev": 0.7293, - "metric": "accuracy", - "requests": 570, - "answered": 569, - "unsupported": 1, - "errors": 0, - "median_ms": 38.0 - }, - "44": { - "score": 0.5158, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 38.0 - }, - "45": { - "score": 0.1353, - "jev": 0.2151, - "metric": "accuracy", - "requests": 501, - "answered": 451, - "unsupported": 50, - "errors": 0, - "median_ms": 38.9 - }, - "48": { - "score": 0.3034, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 42.0 - }, - "50": { - "score": 0, - "jev": 0, - "metric": "accuracy", - "requests": 1676, - "answered": 1, - "unsupported": 1675, - "errors": 0, - "median_ms": 42.2 - } - }, - "scores": { - "balanced_raw": 27.27, - "balanced_skill": 4.76, - "breadth_skill": 4.53 - }, - "frozen_scores": { - "balanced_raw": 23.03, - "balanced_skill": 4.16, - "breadth_skill": 3.84 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.3372, - "skill": 0.0892, - "coverage": 0.9903, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2724, - "skill": 0.032, - "coverage": 0.878, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1113, - "skill": 0.0125, - "coverage": 0.5, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3245, - "skill": 0.067, - "coverage": 0.6633, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3182, - "skill": 0.0371, - "coverage": 0.5981, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3872, - "skill": 0.1728, - "coverage": 0.9988, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.5862, - "skill": 0.0283, - "coverage": 0.9912, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.5441, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.6283, - "headline": false - } - ] - }, - "11": { - "raw": 0.1914, - "skill": 0.0, - "coverage": 0.6341, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5928, - "skill": 0.1855, - "coverage": 0.9901, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4981, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.4999, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.4994, - "skill": 0.3325, - "coverage": 0.9948, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3316, - "skill": 0.1088, - "coverage": 0.949, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.2051, - "skill": 0.0375, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2866, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1236, - "headline": false - } - ] - }, - "31": { - "raw": 0.1048, - "skill": 0.025, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2226, - "skill": 0.0249, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2629, - "skill": 0.0518, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2773, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2629, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.535, - "headline": false - } - ] - }, - "41": { - "raw": 0.3628, - "skill": 0.0442, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3667, - "skill": 0.0, - "coverage": 0.9982, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5158, - "skill": 0.0316, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0006, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.582, - "pending": 0.24 - }, - { - "engine": "lfm350-sdpa", - "name": "LFM2.5-350M-RLCD", - "long_name": "LFM2.5-350M-RLCD", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 354483968, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 708984464, - "model_url": "https://huggingface.co/notnotsamuel/LFM2.5-350M-RLCD", - "code_url": null, - "base_model": null, - "base_url": null, - "weights_repo": "notnotsamuel/LFM2.5-350M-RLCD", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.3004, - "conf": 0.8497, - "ece": 0.5495, - "brier": 1.187, - "over95": 0.3275, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0002, - "conf": 0.192, - "acc": 0.5766 - }, - { - "lo": 0.2, - "w": 0.0073, - "conf": 0.262, - "acc": 0.0486 - }, - { - "lo": 0.3, - "w": 0.018, - "conf": 0.3519, - "acc": 0.0377 - }, - { - "lo": 0.4, - "w": 0.0306, - "conf": 0.4506, - "acc": 0.0665 - }, - { - "lo": 0.5, - "w": 0.0796, - "conf": 0.5496, - "acc": 0.2987 - }, - { - "lo": 0.6, - "w": 0.0825, - "conf": 0.6515, - "acc": 0.3012 - }, - { - "lo": 0.7, - "w": 0.0916, - "conf": 0.7502, - "acc": 0.2712 - }, - { - "lo": 0.8, - "w": 0.1154, - "conf": 0.8504, - "acc": 0.2794 - }, - { - "lo": 0.9, - "w": 0.5747, - "conf": 0.9801, - "acc": 0.3332 - } - ], - "areas": { - "knowledge": { - "ece": 0.5045, - "acc": 0.3452, - "conf": 0.8492 - }, - "language": { - "ece": 0.6054, - "acc": 0.2487, - "conf": 0.854 - }, - "retrieval": { - "ece": 0.6173, - "acc": 0.1868, - "conf": 0.8041 - }, - "tools": { - "ece": 0.5431, - "acc": 0.2504, - "conf": 0.7935 - }, - "arts": { - "ece": 0.5467, - "acc": 0.3684, - "conf": 0.9152 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 26.4, - "p95": 378.6, - "mean": 82.5 - }, - "results": { - "1": { - "score": 0.3837, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 28.8 - }, - "2": { - "score": 0.0858, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 305.5 - }, - "3": { - "score": 0.0236, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 146.8 - }, - "4": { - "score": 0.0018, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 91.7 - }, - "5": { - "score": 0.0029, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 172.7 - }, - "6": { - "score": 0.5035, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 49.1 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 83.6 - }, - "10": { - "score": 0.3421, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 28.0 - }, - "11": { - "score": 0.2021, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 109.8 - }, - "12": { - "score": 0.1962, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 25.1 - }, - "20": { - "score": 0.5848, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 25.0 - }, - "21": { - "score": 0.4958, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 24.6 - }, - "22": { - "score": 0.051, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 266.2 - }, - "23": { - "score": 0.4814, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 25.4 - }, - "24": { - "score": 0.2389, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 25.1 - }, - "25": { - "score": 0.2704, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 25.2 - }, - "26": { - "score": 0.2647, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 24.7 - }, - "27": { - "score": 0.2304, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 24.8 - }, - "28": { - "score": 0.5201, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 24.2 - }, - "29": { - "score": 0.2515, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 25.8 - }, - "30": { - "score": 0.1827, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 25.5 - }, - "31": { - "score": 0.0796, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 46.3 - }, - "32": { - "score": 0.3617, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 28.2 - }, - "33": { - "score": 0.0133, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 33.5 - }, - "34": { - "score": 0.4, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 25.7 - }, - "36": { - "score": 0.044, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 301.1 - }, - "37": { - "score": 0.1238, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 26.3 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 389.1 - }, - "39": { - "score": 0.3315, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 26.5 - }, - "40": { - "score": 0.2495, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 24.4, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2505, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2495, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.51, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.455, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.2532, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 25.1 - }, - "42": { - "score": 0.4288, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 25.9 - }, - "43": { - "score": 0.3228, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 24.9 - }, - "44": { - "score": 0.4974, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 24.5 - }, - "45": { - "score": 0.1357, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 25.9 - }, - "48": { - "score": 0.6215, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 26.7 - }, - "50": { - "score": 0.3037, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 29.4 - } - }, - "scores": { - "balanced_raw": 25.79, - "balanced_skill": 2.33, - "breadth_skill": 2.16 - }, - "frozen_scores": { - "balanced_raw": 21.83, - "balanced_skill": 1.77, - "breadth_skill": 1.66 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2656, - "skill": 0.0061, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2349, - "skill": 0.0115, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.0839, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3243, - "skill": 0.056, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.3805, - "skill": 0.0428, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3837, - "skill": 0.168, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0858, - "skill": 0.0, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.5036, - "skill": 0.0, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.41, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.5972, - "headline": false - } - ] - }, - "11": { - "raw": 0.2021, - "skill": 0.0, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5865, - "skill": 0.1731, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.4958, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0485, - "skill": 0.0411, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.4679, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.2408, - "skill": 0.0, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2704, - "skill": 0.0272, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.1827, - "skill": 0.0096, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.2593, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.1061, - "headline": false - } - ] - }, - "31": { - "raw": 0.0796, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.044, - "skill": 0.0, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1238, - "skill": 0.0, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2495, - "skill": 0.0345, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2505, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2495, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.51, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.455, - "headline": false - } - ] - }, - "41": { - "raw": 0.2532, - "skill": 0.0, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3228, - "skill": 0.0, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4974, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3037, - "skill": 0.0, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "mini-jev", - "name": "mini-jev", - "long_name": "mini-jev Qwen3-4B", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 4022468096, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/r-ms/mini-jev", - "base_model": "Qwen/Qwen3-4B-Instruct-2507", - "base_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507", - "weights_repo": null, - "served_checkpoint": "Qwen/Qwen3-4B-Instruct-2507" - }, - "completed": 132422, - "counts": { - "ok": 117922, - "unsupported": 14500 - }, - "calibration": { - "acc": 0.6176, - "conf": 0.9482, - "ece": 0.3305, - "brier": 0.7012, - "over95": 0.264, - "n": 70200, - "benchmarks": 29, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.2, - "w": 0.0008, - "conf": 0.2684, - "acc": 0.0 - }, - { - "lo": 0.3, - "w": 0.0018, - "conf": 0.3573, - "acc": 0.081 - }, - { - "lo": 0.4, - "w": 0.0065, - "conf": 0.4561, - "acc": 0.3247 - }, - { - "lo": 0.5, - "w": 0.0279, - "conf": 0.5543, - "acc": 0.3071 - }, - { - "lo": 0.6, - "w": 0.0292, - "conf": 0.6506, - "acc": 0.4033 - }, - { - "lo": 0.7, - "w": 0.0352, - "conf": 0.7533, - "acc": 0.3752 - }, - { - "lo": 0.8, - "w": 0.0471, - "conf": 0.8553, - "acc": 0.3883 - }, - { - "lo": 0.9, - "w": 0.8516, - "conf": 0.9901, - "acc": 0.6618 - } - ], - "areas": { - "knowledge": { - "ece": 0.3452, - "acc": 0.5857, - "conf": 0.9309 - }, - "language": { - "ece": 0.3356, - "acc": 0.6383, - "conf": 0.9739 - }, - "retrieval": { - "ece": 0.3419, - "acc": 0.6439, - "conf": 0.9857 - }, - "tools": { - "ece": 0.1914, - "acc": 0.7885, - "conf": 0.9798 - }, - "arts": { - "ece": 0.3342, - "acc": 0.5947, - "conf": 0.9289 - } - } - }, - "gaps": { - "share": 0.1095, - "reasons": [ - { - "reason": "supports only 2\u201326 answer options per question", - "share": 0.1095 - }, - { - "reason": "ran out of GPU memory on the longest prompts", - "share": 0.0 - } - ] - }, - "latency": { - "median": 64.5, - "p95": 834.3, - "mean": 224.8 - }, - "results": { - "1": { - "score": 0.7739, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 70.0 - }, - "2": { - "score": 0.3092, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 950.5 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": null, - "jev": null, - "metric": null, - "requests": 3080, - "answered": 0, - "unsupported": 3080, - "errors": 0, - "median_ms": null - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": 0.7938, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 147.8 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 241.2 - }, - "10": { - "score": 0.5875, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 78.7 - }, - "11": { - "score": 0.5444, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 379.9 - }, - "12": { - "score": 0.5194, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 63.2 - }, - "20": { - "score": 0.8098, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 63.2 - }, - "21": { - "score": 0.5822, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 62.4 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.5418, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 64.3 - }, - "24": { - "score": 0.6734, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 63.0 - }, - "25": { - "score": 0.3061, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 63.4 - }, - "26": { - "score": 0.9579, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 62.5 - }, - "27": { - "score": 0.884, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 62.8 - }, - "28": { - "score": 0.5951, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 62.7 - }, - "29": { - "score": 0.7721, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 63.8 - }, - "30": { - "score": 0.4814, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 63.8 - }, - "31": { - "score": 0.1688, - "jev": 0.2746, - "metric": "accuracy", - "requests": 5000, - "answered": 1588, - "unsupported": 3412, - "errors": 0, - "median_ms": 65.2 - }, - "32": { - "score": 0.5957, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 83.0 - }, - "33": { - "score": 0.2079, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 85.7 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 63.5 - }, - "36": { - "score": 0.1321, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 1562.8 - }, - "37": { - "score": 0.2921, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 64.8 - }, - "38": { - "score": 0.0014, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 362.7 - }, - "39": { - "score": 0.6397, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 64.5 - }, - "40": { - "score": 0.3613, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 62.6, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2807, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3613, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.75, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.865, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4306, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 62.7 - }, - "42": { - "score": 0.7075, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 65.9 - }, - "43": { - "score": 0.3877, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 63.4 - }, - "44": { - "score": 0.5442, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 62.8 - }, - "45": { - "score": 0.1198, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 64.1 - }, - "48": { - "score": 0.4745, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 70.4 - }, - "50": { - "score": 0.4684, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 69.8 - } - }, - "scores": { - "balanced_raw": 43.48, - "balanced_skill": 24.31, - "breadth_skill": 22.35 - }, - "frozen_scores": { - "balanced_raw": 34.83, - "balanced_skill": 19.01, - "breadth_skill": 16.54 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4125, - "skill": 0.1949, - "coverage": 0.8863, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4454, - "skill": 0.2217, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2121, - "skill": 0.1019, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.6256, - "skill": 0.4823, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4782, - "skill": 0.2148, - "coverage": 0.8, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.7739, - "skill": 0.6948, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.3092, - "skill": 0.2381, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7938, - "skill": 0.5141, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7791, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8085, - "headline": false - } - ] - }, - "11": { - "raw": 0.5444, - "skill": 0.3411, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.8024, - "skill": 0.6049, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5822, - "skill": 0.1644, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5381, - "skill": 0.0761, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7019, - "skill": 0.6026, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3061, - "skill": 0.0748, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4814, - "skill": 0.3748, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5588, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.4041, - "headline": false - } - ] - }, - "31": { - "raw": 0.0536, - "skill": 0.0, - "coverage": 0.3176, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1321, - "skill": 0.0918, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2921, - "skill": 0.112, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.3613, - "skill": 0.1783, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2807, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3613, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.75, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.865, - "headline": false - } - ] - }, - "41": { - "raw": 0.4306, - "skill": 0.1459, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3877, - "skill": 0.0286, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5442, - "skill": 0.0884, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4684, - "skill": 0.2284, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.6927, - "pending": 0.24 - }, - { - "engine": "nanojev-8k", - "name": "NanoJev", - "long_name": "NanoJev", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 596049920, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 21465353520, - "model_url": "https://huggingface.co/C-Tianyu/NanoJev", - "code_url": "https://github.com/TianyuCodings/NanoJev", - "base_model": "Qwen/Qwen3-0.6B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3-0.6B-Base", - "weights_repo": "C-Tianyu/NanoJev", - "served_checkpoint": null - }, - "completed": 132129, - "counts": { - "ok": 131930, - "unsupported": 199 - }, - "calibration": { - "acc": 0.2785, - "conf": 0.45, - "ece": 0.1714, - "brier": 0.8089, - "over95": 0.0281, - "n": 72590, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.1461, - "conf": 0.0325, - "acc": 0.0187 - }, - { - "lo": 0.1, - "w": 0.0182, - "conf": 0.1591, - "acc": 0.0811 - }, - { - "lo": 0.2, - "w": 0.1047, - "conf": 0.2594, - "acc": 0.1677 - }, - { - "lo": 0.3, - "w": 0.19, - "conf": 0.3491, - "acc": 0.2788 - }, - { - "lo": 0.4, - "w": 0.1198, - "conf": 0.445, - "acc": 0.3403 - }, - { - "lo": 0.5, - "w": 0.1803, - "conf": 0.5405, - "acc": 0.4577 - }, - { - "lo": 0.6, - "w": 0.0521, - "conf": 0.6425, - "acc": 0.4354 - }, - { - "lo": 0.7, - "w": 0.0364, - "conf": 0.7524, - "acc": 0.343 - }, - { - "lo": 0.8, - "w": 0.0696, - "conf": 0.8573, - "acc": 0.3621 - }, - { - "lo": 0.9, - "w": 0.0827, - "conf": 0.9369, - "acc": 0.243 - } - ], - "areas": { - "knowledge": { - "ece": 0.1509, - "acc": 0.2902, - "conf": 0.4411 - }, - "language": { - "ece": 0.3489, - "acc": 0.2407, - "conf": 0.5896 - }, - "retrieval": { - "ece": 0.0323, - "acc": 0.2341, - "conf": 0.2465 - }, - "tools": { - "ece": 0.2611, - "acc": 0.2265, - "conf": 0.4877 - }, - "arts": { - "ece": 0.0539, - "acc": 0.3658, - "conf": 0.4196 - } - } - }, - "gaps": { - "share": 0.0015, - "reasons": [ - { - "reason": "prompt longer than its 8,192-token context window", - "share": 0.0015 - } - ] - }, - "latency": { - "median": 27.6, - "p95": 955.7, - "mean": 278.4 - }, - "results": { - "1": { - "score": 0.3884, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 35.9 - }, - "2": { - "score": 0.0999, - "jev": 0.4455, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7653, - "unsupported": 51, - "errors": 0, - "median_ms": 558.7 - }, - "3": { - "score": 0.0374, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 9550.7 - }, - "4": { - "score": 0.0124, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 40.9 - }, - "5": { - "score": 0.0004, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 87.1 - }, - "6": { - "score": 0.4689, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 747.1 - }, - "9": { - "score": 0, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 703.9 - }, - "10": { - "score": 0.4577, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 45.8 - }, - "11": { - "score": 0.2374, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 1600.1 - }, - "12": { - "score": 0.2874, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 26.3 - }, - "20": { - "score": 0.488, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 26.4 - }, - "21": { - "score": 0.5027, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 25.3 - }, - "22": { - "score": 0.008, - "jev": 0.1811, - "metric": "accuracy", - "requests": 2000, - "answered": 1999, - "unsupported": 1, - "errors": 0, - "median_ms": 4378.3 - }, - "23": { - "score": 0.533, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 26.5 - }, - "24": { - "score": 0.2716, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 26.5 - }, - "25": { - "score": 0.2041, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 26.2 - }, - "26": { - "score": 0.2917, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 26.4 - }, - "27": { - "score": 0.279, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 26.6 - }, - "28": { - "score": 0.5004, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 24.0 - }, - "29": { - "score": 0.2729, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 26.9 - }, - "30": { - "score": 0.1562, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 26.6 - }, - "31": { - "score": 0.071, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 67.3 - }, - "32": { - "score": 0.5173, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 29.3 - }, - "33": { - "score": 0.0012, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 77.4 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 27.0 - }, - "36": { - "score": 0.0465, - "jev": 0.1874, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1291, - "unsupported": 6, - "errors": 0, - "median_ms": 1001.4 - }, - "37": { - "score": 0.2313, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 27.6 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 173.1 - }, - "39": { - "score": 0.2365, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 26.3 - }, - "40": { - "score": 0.2511, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 24.0, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.25, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2511, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.49, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.505, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.179, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 24.1 - }, - "42": { - "score": 0.3358, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 27.5 - }, - "43": { - "score": 0.3965, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 26.1 - }, - "44": { - "score": 0.4982, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 23.9 - }, - "45": { - "score": 0.1257, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 27.4 - }, - "48": { - "score": 0.4306, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 28.0 - }, - "50": { - "score": 0.2995, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 30.9 - } - }, - "scores": { - "balanced_raw": 26.19, - "balanced_skill": 2.28, - "breadth_skill": 2.16 - }, - "frozen_scores": { - "balanced_raw": 21.75, - "balanced_skill": 1.46, - "breadth_skill": 1.4 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.2663, - "skill": 0.0127, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.2225, - "skill": 0.0122, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.1388, - "skill": 0.0189, - "coverage": 0.9977, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.3189, - "skill": 0.0603, - "coverage": 0.9978, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.363, - "skill": 0.0098, - "coverage": 0.9999, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3884, - "skill": 0.1744, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0993, - "skill": 0.0065, - "coverage": 0.9934, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.4689, - "skill": 0.0, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.3756, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.5622, - "headline": false - } - ] - }, - "11": { - "raw": 0.2374, - "skill": 0.0, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.483, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5027, - "skill": 0.0053, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0081, - "skill": 0.0003, - "coverage": 0.9995, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5216, - "skill": 0.0432, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.2717, - "skill": 0.029, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.2041, - "skill": 0.0, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.1562, - "skill": 0.0047, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.257, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.0553, - "headline": false - } - ] - }, - "31": { - "raw": 0.071, - "skill": 0.0, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0463, - "skill": 0.002, - "coverage": 0.9954, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.2313, - "skill": 0.0358, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.2511, - "skill": 0.0366, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.25, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.2511, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.49, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.505, - "headline": false - } - ] - }, - "41": { - "raw": 0.179, - "skill": 0.0, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3965, - "skill": 0.0425, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.4982, - "skill": 0.0, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.2995, - "skill": 0.0, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.7595, - "pending": 0.24 - }, - { - "engine": "nimble", - "name": "Bespoke Nimble 9B", - "long_name": "Bespoke Nimble 9B", - "variant": null, - "meta": { - "kind": "LoRA", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 9653104368, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 173188512, - "model_url": "https://huggingface.co/bespokelabs/Bespoke-Nimble-9B", - "code_url": "https://github.com/bespokelabsai/nimble", - "base_model": "Qwen/Qwen3.5-9B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-9B-Base", - "weights_repo": "bespokelabs/Bespoke-Nimble-9B", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 91939, - "unsupported": 40483 - }, - "calibration": { - "acc": 0.5999, - "conf": 0.8079, - "ece": 0.208, - "brier": 0.5869, - "over95": 0.0557, - "n": 16835, - "benchmarks": 26, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0063, - "conf": 0.1738, - "acc": 0.1215 - }, - { - "lo": 0.2, - "w": 0.0191, - "conf": 0.2519, - "acc": 0.1183 - }, - { - "lo": 0.3, - "w": 0.0313, - "conf": 0.3529, - "acc": 0.2332 - }, - { - "lo": 0.4, - "w": 0.0425, - "conf": 0.4529, - "acc": 0.2838 - }, - { - "lo": 0.5, - "w": 0.0894, - "conf": 0.5518, - "acc": 0.4339 - }, - { - "lo": 0.6, - "w": 0.0835, - "conf": 0.6495, - "acc": 0.4063 - }, - { - "lo": 0.7, - "w": 0.096, - "conf": 0.7532, - "acc": 0.5218 - }, - { - "lo": 0.8, - "w": 0.1624, - "conf": 0.862, - "acc": 0.4599 - }, - { - "lo": 0.9, - "w": 0.4696, - "conf": 0.9708, - "acc": 0.8092 - } - ], - "areas": { - "knowledge": { - "ece": 0.2157, - "acc": 0.5555, - "conf": 0.7711 - }, - "language": { - "ece": 0.2207, - "acc": 0.6415, - "conf": 0.8621 - }, - "retrieval": { - "ece": 0.2677, - "acc": 0.628, - "conf": 0.8956 - }, - "tools": { - "ece": 0.0066, - "acc": 0.9537, - "conf": 0.9519 - }, - "arts": { - "ece": 0.1932, - "acc": 0.6006, - "conf": 0.7889 - } - } - }, - "gaps": { - "share": 0.3057, - "reasons": [ - { - "reason": "prompt longer than its 2,048-token context window", - "share": 0.1962 - }, - { - "reason": "supports at most 26 answer options per question", - "share": 0.1095 - } - ] - }, - "latency": { - "median": 83.5, - "p95": 136.9, - "mean": 108.0 - }, - "results": { - "1": { - "score": 0.9196, - "jev": 0.9605, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1518, - "unsupported": 176, - "errors": 0, - "median_ms": 168.7 - }, - "2": { - "score": null, - "jev": null, - "metric": null, - "requests": 7704, - "answered": 0, - "unsupported": 7704, - "errors": 0, - "median_ms": null - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": null, - "jev": null, - "metric": null, - "requests": 3080, - "answered": 0, - "unsupported": 3080, - "errors": 0, - "median_ms": null - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": 1.0, - "jev": 1.0, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 2, - "unsupported": 9998, - "errors": 0, - "median_ms": 302.4 - }, - "9": { - "score": null, - "jev": null, - "metric": null, - "requests": 160, - "answered": 0, - "unsupported": 160, - "errors": 0, - "median_ms": null - }, - "10": { - "score": 0.6342, - "jev": 0.4294, - "metric": "macro-F1", - "requests": 2500, - "answered": 2499, - "unsupported": 1, - "errors": 0, - "median_ms": 100.9 - }, - "11": { - "score": null, - "jev": null, - "metric": null, - "requests": 123, - "answered": 0, - "unsupported": 123, - "errors": 0, - "median_ms": null - }, - "12": { - "score": 0.6734, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 82.7 - }, - "20": { - "score": 0.8682, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 81.7 - }, - "21": { - "score": 0.5247, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 80.5 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.5342, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 83.2 - }, - "24": { - "score": 0.7394, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 82.3 - }, - "25": { - "score": 0.4103, - "jev": 0.7897, - "metric": "accuracy", - "requests": 196, - "answered": 195, - "unsupported": 1, - "errors": 0, - "median_ms": 83.6 - }, - "26": { - "score": 0.9739, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 81.4 - }, - "27": { - "score": 0.9283, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 81.3 - }, - "28": { - "score": 0.7388, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 80.9 - }, - "29": { - "score": 0.7844, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 83.7 - }, - "30": { - "score": 0.3647, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 85.5 - }, - "31": { - "score": 0.2078, - "jev": 0.2746, - "metric": "accuracy", - "requests": 5000, - "answered": 1588, - "unsupported": 3412, - "errors": 0, - "median_ms": 99.3 - }, - "32": { - "score": 0.5492, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 100.9 - }, - "33": { - "score": 0.3295, - "jev": 0.2752, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1308, - "unsupported": 342, - "errors": 0, - "median_ms": 702.1 - }, - "34": { - "score": 0, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 84.1 - }, - "36": { - "score": null, - "jev": null, - "metric": null, - "requests": 1297, - "answered": 0, - "unsupported": 1297, - "errors": 0, - "median_ms": null - }, - "37": { - "score": 0.4282, - "jev": 0.5517, - "metric": "macro-F1", - "requests": 5000, - "answered": 4979, - "unsupported": 21, - "errors": 0, - "median_ms": 85.6 - }, - "38": { - "score": null, - "jev": null, - "metric": null, - "requests": 5479, - "answered": 0, - "unsupported": 5479, - "errors": 0, - "median_ms": null - }, - "39": { - "score": 0.6392, - "jev": 0.869, - "metric": "macro-F1", - "requests": 979, - "answered": 978, - "unsupported": 1, - "errors": 0, - "median_ms": 165.1 - }, - "40": { - "score": 0.4069, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 81.4, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.364, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4069, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.56, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.76, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4872, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 83.2 - }, - "42": { - "score": 0.7756, - "jev": 0.8455, - "metric": "macro-F1", - "requests": 5500, - "answered": 5368, - "unsupported": 132, - "errors": 0, - "median_ms": 85.0 - }, - "43": { - "score": 0.5211, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 81.9 - }, - "44": { - "score": 0.66, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 81.6 - }, - "45": { - "score": 0.0896, - "jev": 0.2057, - "metric": "accuracy", - "requests": 501, - "answered": 491, - "unsupported": 10, - "errors": 0, - "median_ms": 85.3 - }, - "48": { - "score": 0.2318, - "jev": 0.1734, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10014, - "unsupported": 125, - "errors": 0, - "median_ms": 86.2 - }, - "50": { - "score": 0.4242, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 93.9 - } - }, - "scores": { - "balanced_raw": 34.21, - "balanced_skill": 20.39, - "breadth_skill": 19.92 - }, - "frozen_scores": { - "balanced_raw": 29.4, - "balanced_skill": 16.68, - "breadth_skill": 14.27 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4612, - "skill": 0.2783, - "coverage": 0.8854, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.298, - "skill": 0.1559, - "coverage": 0.6667, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2113, - "skill": 0.1379, - "coverage": 0.4979, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.2748, - "skill": 0.2542, - "coverage": 0.2988, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.465, - "skill": 0.1933, - "coverage": 0.8, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8241, - "skill": 0.7625, - "coverage": 0.8961, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0002, - "skill": 0.0, - "coverage": 0.0002, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0002, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0002, - "headline": false - } - ] - }, - "11": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.8663, - "skill": 0.7326, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5247, - "skill": 0.0495, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5099, - "skill": 0.0199, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7472, - "skill": 0.6629, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.4082, - "skill": 0.2109, - "coverage": 0.9949, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.3647, - "skill": 0.236, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4776, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.2517, - "headline": false - } - ] - }, - "31": { - "raw": 0.066, - "skill": 0.0, - "coverage": 0.3176, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.4227, - "skill": 0.2759, - "coverage": 0.9958, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.4069, - "skill": 0.237, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.364, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4069, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.56, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.76, - "headline": false - } - ] - }, - "41": { - "raw": 0.4872, - "skill": 0.2308, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.5211, - "skill": 0.2401, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.66, - "skill": 0.32, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4242, - "skill": 0.1644, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.5282, - "pending": 0.24 - }, - { - "engine": "openvons", - "name": "openvons", - "long_name": "openvons Qwen3-4B", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 4022468096, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/genai-craft/openvons", - "base_model": "Qwen/Qwen3-4B-Instruct-2507", - "base_url": "https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507", - "weights_repo": null, - "served_checkpoint": "Qwen/Qwen3-4B-Instruct-2507" - }, - "completed": 132422, - "counts": { - "ok": 132422 - }, - "calibration": { - "acc": 0.5956, - "conf": 0.9527, - "ece": 0.3571, - "brier": 0.7519, - "over95": 0.2958, - "n": 72595, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.2, - "w": 0.0007, - "conf": 0.2424, - "acc": 0.0 - }, - { - "lo": 0.3, - "w": 0.0025, - "conf": 0.3383, - "acc": 0.0431 - }, - { - "lo": 0.4, - "w": 0.0112, - "conf": 0.4512, - "acc": 0.1647 - }, - { - "lo": 0.5, - "w": 0.0216, - "conf": 0.5416, - "acc": 0.3145 - }, - { - "lo": 0.6, - "w": 0.0268, - "conf": 0.6423, - "acc": 0.2856 - }, - { - "lo": 0.7, - "w": 0.0246, - "conf": 0.7429, - "acc": 0.3081 - }, - { - "lo": 0.8, - "w": 0.0374, - "conf": 0.8415, - "acc": 0.2981 - }, - { - "lo": 0.9, - "w": 0.8751, - "conf": 0.9917, - "acc": 0.6405 - } - ], - "areas": { - "knowledge": { - "ece": 0.332, - "acc": 0.6052, - "conf": 0.9373 - }, - "language": { - "ece": 0.3384, - "acc": 0.6438, - "conf": 0.9822 - }, - "retrieval": { - "ece": 0.4285, - "acc": 0.54, - "conf": 0.9685 - }, - "tools": { - "ece": 0.28, - "acc": 0.7086, - "conf": 0.9886 - }, - "arts": { - "ece": 0.4422, - "acc": 0.478, - "conf": 0.9202 - } - } - }, - "gaps": { - "share": 0.0, - "reasons": [] - }, - "latency": { - "median": 18.0, - "p95": 295.9, - "mean": 69.7 - }, - "results": { - "1": { - "score": 0.8406, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 37.9 - }, - "2": { - "score": 0.4207, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 238.2 - }, - "3": { - "score": 0.435, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 337.5 - }, - "4": { - "score": 0.3255, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 73.2 - }, - "5": { - "score": 0.4172, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 100.6 - }, - "6": { - "score": 0.798, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 81.0 - }, - "9": { - "score": 0.0688, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 113.6 - }, - "10": { - "score": 0.6232, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 37.2 - }, - "11": { - "score": 0.5355, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 192.0 - }, - "12": { - "score": 0.5379, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 16.1 - }, - "20": { - "score": 0.7622, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 13.6 - }, - "21": { - "score": 0.5906, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 12.6 - }, - "22": { - "score": 0.056, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 283.6 - }, - "23": { - "score": 0.5326, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 14.8 - }, - "24": { - "score": 0.6918, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 13.3 - }, - "25": { - "score": 0.3878, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 16.1 - }, - "26": { - "score": 0.963, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 13.0 - }, - "27": { - "score": 0.8951, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 13.2 - }, - "28": { - "score": 0.6259, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 12.4 - }, - "29": { - "score": 0.8076, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 16.2 - }, - "30": { - "score": 0.4587, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 15.8 - }, - "31": { - "score": 0.109, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 66.5 - }, - "32": { - "score": 0.6011, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 32.7 - }, - "33": { - "score": 0.2612, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 55.7 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 16.1 - }, - "36": { - "score": 0.1587, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 326.6 - }, - "37": { - "score": 0.33, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 19.1 - }, - "38": { - "score": 0.0114, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 282.9 - }, - "39": { - "score": 0.6046, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 26.6 - }, - "40": { - "score": 0.3333, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 13.1, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3208, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3333, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.735, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.87, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4249, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 16.5 - }, - "42": { - "score": 0.7, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 15.4 - }, - "43": { - "score": 0.4263, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 16.3 - }, - "44": { - "score": 0.5578, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 13.6 - }, - "45": { - "score": 0.1138, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 16.8 - }, - "48": { - "score": 0.3874, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 26.5 - }, - "50": { - "score": 0.4588, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 31.6 - } - }, - "scores": { - "balanced_raw": 45.59, - "balanced_skill": 26.54, - "breadth_skill": 24.27 - }, - "frozen_scores": { - "balanced_raw": 36.41, - "balanced_skill": 20.63, - "breadth_skill": 17.89 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4431, - "skill": 0.2324, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4312, - "skill": 0.2026, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2444, - "skill": 0.1396, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.6864, - "skill": 0.5564, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4742, - "skill": 0.196, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8406, - "skill": 0.7848, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4207, - "skill": 0.3611, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.798, - "skill": 0.5234, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7851, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8108, - "headline": false - } - ] - }, - "11": { - "raw": 0.5355, - "skill": 0.3282, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.7561, - "skill": 0.5122, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5906, - "skill": 0.1811, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.051, - "skill": 0.0435, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5143, - "skill": 0.0287, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7191, - "skill": 0.6255, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3878, - "skill": 0.1837, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.4587, - "skill": 0.3503, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.5663, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.351, - "headline": false - } - ] - }, - "31": { - "raw": 0.109, - "skill": 0.0296, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1587, - "skill": 0.1196, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.33, - "skill": 0.1597, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.3333, - "skill": 0.1423, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3208, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3333, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.735, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.87, - "headline": false - } - ] - }, - "41": { - "raw": 0.4249, - "skill": 0.1374, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.4263, - "skill": 0.0898, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5578, - "skill": 0.1156, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4588, - "skill": 0.2146, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.76, - "pending": 0.24 - }, - { - "engine": "pngwn-space-uncapped-full", - "name": "open-jev (pngwn)", - "long_name": "open-jev (pngwn)", - "variant": null, - "meta": { - "kind": "LoRA", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 4659865088, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 121956232, - "model_url": "https://huggingface.co/pngwn/system-one-qwen3.5-4b-scorer-v2b", - "code_url": null, - "base_model": "Qwen/Qwen3.5-4B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-4B-Base", - "weights_repo": "pngwn/system-one-qwen3.5-4b-scorer-v2b", - "served_checkpoint": null - }, - "completed": 132200, - "counts": { - "ok": 132160, - "unsupported": 40 - }, - "calibration": { - "acc": 0.579, - "conf": 0.6017, - "ece": 0.0683, - "brier": 0.5553, - "over95": 0.0063, - "n": 72590, - "benchmarks": 33, - "rel": [ - { - "lo": 0.0, - "w": 0.0981, - "conf": 0.0513, - "acc": 0.2456 - }, - { - "lo": 0.1, - "w": 0.0253, - "conf": 0.1465, - "acc": 0.294 - }, - { - "lo": 0.2, - "w": 0.0274, - "conf": 0.2455, - "acc": 0.223 - }, - { - "lo": 0.3, - "w": 0.0425, - "conf": 0.3567, - "acc": 0.3158 - }, - { - "lo": 0.4, - "w": 0.0765, - "conf": 0.4535, - "acc": 0.4316 - }, - { - "lo": 0.5, - "w": 0.184, - "conf": 0.5494, - "acc": 0.5337 - }, - { - "lo": 0.6, - "w": 0.1471, - "conf": 0.6477, - "acc": 0.6172 - }, - { - "lo": 0.7, - "w": 0.1455, - "conf": 0.7442, - "acc": 0.5654 - }, - { - "lo": 0.8, - "w": 0.1138, - "conf": 0.8501, - "acc": 0.8406 - }, - { - "lo": 0.9, - "w": 0.1398, - "conf": 0.9663, - "acc": 0.9163 - } - ], - "areas": { - "knowledge": { - "ece": 0.1154, - "acc": 0.5516, - "conf": 0.6669 - }, - "language": { - "ece": 0.0205, - "acc": 0.6776, - "conf": 0.6788 - }, - "retrieval": { - "ece": 0.2241, - "acc": 0.5681, - "conf": 0.3573 - }, - "tools": { - "ece": 0.1496, - "acc": 0.7304, - "conf": 0.6205 - }, - "arts": { - "ece": 0.0824, - "acc": 0.436, - "conf": 0.4955 - } - } - }, - "gaps": { - "share": 0.0003, - "reasons": [ - { - "reason": "doc_0: the question is 828 tokens; the limit is 96.", - "share": 0.0 - }, - { - "reason": "doc_11: the question is 138 tokens; the limit is 96.", - "share": 0.0 - }, - { - "reason": "doc_0: the question is 128 tokens; the limit is 96.", - "share": 0.0 - } - ] - }, - "latency": { - "median": 138.5, - "p95": 2079.3, - "mean": 523.5 - }, - "results": { - "1": { - "score": 0.8973, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 148.0 - }, - "2": { - "score": 0.4154, - "jev": 0.4466, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7664, - "unsupported": 40, - "errors": 0, - "median_ms": 2695.8 - }, - "3": { - "score": 0.4843, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 688.0 - }, - "4": { - "score": 0.5048, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 296.8 - }, - "5": { - "score": 0.5762, - "jev": 0.8927, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 576.4 - }, - "6": { - "score": 0.7988, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 305.6 - }, - "9": { - "score": 0.0437, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 269.9 - }, - "10": { - "score": 0.5904, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 176.1 - }, - "11": { - "score": 0.6368, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 491.0 - }, - "12": { - "score": 0.5879, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 123.7 - }, - "20": { - "score": 0.519, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 121.7 - }, - "21": { - "score": 0.5521, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 117.6 - }, - "22": { - "score": 0.056, - "jev": 0.181, - "metric": "accuracy", - "requests": 2000, - "answered": 2000, - "unsupported": 0, - "errors": 0, - "median_ms": 1080.2 - }, - "23": { - "score": 0.5828, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 138.8 - }, - "24": { - "score": 0.8058, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 118.3 - }, - "25": { - "score": 0.4388, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 125.7 - }, - "26": { - "score": 0.9621, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 116.0 - }, - "27": { - "score": 0.9121, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 116.8 - }, - "28": { - "score": 0.708, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 116.0 - }, - "29": { - "score": 0.7713, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 122.8 - }, - "30": { - "score": 0.3669, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 121.6 - }, - "31": { - "score": 0.1004, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 235.6 - }, - "32": { - "score": 0.5811, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 210.4 - }, - "33": { - "score": 0.3073, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 154.7 - }, - "34": { - "score": 0, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 168.9 - }, - "36": { - "score": 0.1623, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 4192.5 - }, - "37": { - "score": 0.3067, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 133.1 - }, - "38": { - "score": 0.0021, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 855.7 - }, - "39": { - "score": 0.7463, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 121.6 - }, - "40": { - "score": 0.3756, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 117.2, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3459, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3756, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.675, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.3828, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 125.3 - }, - "42": { - "score": 0.7434, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 144.9 - }, - "43": { - "score": 0.4825, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 121.3 - }, - "44": { - "score": 0.6316, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 123.5 - }, - "45": { - "score": 0.1098, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 129.5 - }, - "48": { - "score": 0.2185, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 158.4 - }, - "50": { - "score": 0.4206, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 154.6 - } - }, - "scores": { - "balanced_raw": 46.15, - "balanced_skill": 27.07, - "breadth_skill": 23.84 - }, - "frozen_scores": { - "balanced_raw": 36.29, - "balanced_skill": 19.99, - "breadth_skill": 16.56 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.476, - "skill": 0.2887, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4651, - "skill": 0.2486, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2345, - "skill": 0.1269, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7031, - "skill": 0.58, - "coverage": 0.9983, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4289, - "skill": 0.1093, - "coverage": 1, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.8973, - "skill": 0.8613, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4132, - "skill": 0.3528, - "coverage": 0.9948, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7989, - "skill": 0.5257, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7854, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8123, - "headline": false - } - ] - }, - "11": { - "raw": 0.6368, - "skill": 0.4747, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.5395, - "skill": 0.079, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5521, - "skill": 0.1043, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0535, - "skill": 0.0461, - "coverage": 1, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.579, - "skill": 0.158, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.8357, - "skill": 0.7809, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.4388, - "skill": 0.2517, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.3669, - "skill": 0.2372, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4655, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.2684, - "headline": false - } - ] - }, - "31": { - "raw": 0.1004, - "skill": 0.0202, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1623, - "skill": 0.1234, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.3067, - "skill": 0.1304, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.3756, - "skill": 0.1967, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3459, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3756, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.5, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.675, - "headline": false - } - ] - }, - "41": { - "raw": 0.3828, - "skill": 0.0742, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.4825, - "skill": 0.1789, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6316, - "skill": 0.2632, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4206, - "skill": 0.1592, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.7598, - "pending": 0.24 - }, - { - "engine": "razorback-one-read", - "name": "razorback16 openjev diffusiongemma (NVFP4, vLLM)", - "long_name": "razorback16 diffgemma", - "variant": "openjev harness", - "meta": { - "kind": "inference technique", - "technique": "diffusion", - "technique_name": "diffusion", - "served_params": 25823778864, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/vllm-project/vllm/pull/57250", - "base_model": "google/diffusiongemma-26B-A4B-it", - "base_url": "https://huggingface.co/google/diffusiongemma-26B-A4B-it", - "weights_repo": null, - "served_checkpoint": "nvidia/diffusiongemma-26B-A4B-it-NVFP4" - }, - "completed": 132422, - "counts": { - "ok": 124899, - "unsupported": 7523 - }, - "calibration": { - "acc": 0.6322, - "conf": 0.8524, - "ece": 0.2201, - "brier": 0.5797, - "over95": 0.099, - "n": 71361, - "benchmarks": 31, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0022, - "conf": 0.1694, - "acc": 0.1411 - }, - { - "lo": 0.2, - "w": 0.0079, - "conf": 0.2546, - "acc": 0.1141 - }, - { - "lo": 0.3, - "w": 0.0149, - "conf": 0.3531, - "acc": 0.1704 - }, - { - "lo": 0.4, - "w": 0.0322, - "conf": 0.4535, - "acc": 0.3425 - }, - { - "lo": 0.5, - "w": 0.0636, - "conf": 0.5509, - "acc": 0.4247 - }, - { - "lo": 0.6, - "w": 0.1033, - "conf": 0.649, - "acc": 0.3057 - }, - { - "lo": 0.7, - "w": 0.0789, - "conf": 0.7516, - "acc": 0.5409 - }, - { - "lo": 0.8, - "w": 0.1154, - "conf": 0.8535, - "acc": 0.5746 - }, - { - "lo": 0.9, - "w": 0.5816, - "conf": 0.9805, - "acc": 0.7735 - } - ], - "areas": { - "knowledge": { - "ece": 0.2475, - "acc": 0.569, - "conf": 0.8165 - }, - "language": { - "ece": 0.1751, - "acc": 0.7286, - "conf": 0.9037 - }, - "retrieval": { - "ece": 0.3071, - "acc": 0.5577, - "conf": 0.8648 - }, - "tools": { - "ece": 0.1026, - "acc": 0.7498, - "conf": 0.8524 - }, - "arts": { - "ece": 0.2261, - "acc": 0.6526, - "conf": 0.8787 - } - } - }, - "gaps": { - "share": 0.0568, - "reasons": [ - { - "reason": "supports at most 128 answer options per question", - "share": 0.0566 - }, - { - "reason": "prompt longer than its context window (no truncation allowed)", - "share": 0.0002 - } - ] - }, - "latency": { - "median": 40.6, - "p95": 340.8, - "mean": 90.4 - }, - "results": { - "1": { - "score": 0.9658, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 57.0 - }, - "2": { - "score": 0.3234, - "jev": 0.4461, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7704, - "unsupported": 0, - "errors": 0, - "median_ms": 358.9 - }, - "3": { - "score": 0.6161, - "jev": 0.8819, - "metric": "accuracy", - "requests": 508, - "answered": 508, - "unsupported": 0, - "errors": 0, - "median_ms": 359.3 - }, - "4": { - "score": 0.6915, - "jev": 0.7974, - "metric": "macro-F1", - "requests": 3080, - "answered": 3080, - "unsupported": 0, - "errors": 0, - "median_ms": 76.8 - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": 0.7988, - "jev": 0.7992, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 10000, - "unsupported": 0, - "errors": 0, - "median_ms": 120.7 - }, - "9": { - "score": 0.0063, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 172.9 - }, - "10": { - "score": 0.3584, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 64.6 - }, - "11": { - "score": 0.5404, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 107.5 - }, - "12": { - "score": 0.564, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 35.9 - }, - "20": { - "score": 0.8826, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 34.9 - }, - "21": { - "score": 0.6065, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 33.3 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.6022, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 43.6 - }, - "24": { - "score": 0.7772, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 34.9 - }, - "25": { - "score": 0.4796, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 37.1 - }, - "26": { - "score": 0.9752, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 34.2 - }, - "27": { - "score": 0.9334, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 34.4 - }, - "28": { - "score": 0.708, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 33.4 - }, - "29": { - "score": 0.8081, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 39.1 - }, - "30": { - "score": 0.3965, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 37.5 - }, - "31": { - "score": 0.1428, - "jev": 0.1722, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 62.1 - }, - "32": { - "score": 0.641, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 59.3 - }, - "33": { - "score": 0.3109, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 62.1 - }, - "34": { - "score": 0.2, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 37.6 - }, - "36": { - "score": 0.0848, - "jev": 0.1881, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1297, - "unsupported": 0, - "errors": 0, - "median_ms": 502.5 - }, - "37": { - "score": 0.5675, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 39.5 - }, - "38": { - "score": 0.0029, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 102.6 - }, - "39": { - "score": 0.8508, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 38.5 - }, - "40": { - "score": 0.5367, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 32.7, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3684, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5367, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.79, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.935, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.5763, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 36.4 - }, - "42": { - "score": 0.738, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 44.6 - }, - "43": { - "score": 0.493, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 34.9 - }, - "44": { - "score": 0.6222, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 34.4 - }, - "45": { - "score": 0.1038, - "jev": 0.2036, - "metric": "accuracy", - "requests": 501, - "answered": 501, - "unsupported": 0, - "errors": 0, - "median_ms": 38.7 - }, - "48": { - "score": 0.2226, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 46.3 - }, - "50": { - "score": 0.4982, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 59.4 - } - }, - "scores": { - "balanced_raw": 51.52, - "balanced_skill": 35.63, - "breadth_skill": 34.37 - }, - "frozen_scores": { - "balanced_raw": 40.05, - "balanced_skill": 26.69, - "breadth_skill": 23.11 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4862, - "skill": 0.2985, - "coverage": 1, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.5511, - "skill": 0.3679, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.3261, - "skill": 0.2499, - "coverage": 1, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.696, - "skill": 0.5778, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.5163, - "skill": 0.2877, - "coverage": 0.8, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9658, - "skill": 0.9538, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.3234, - "skill": 0.2538, - "coverage": 1, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7989, - "skill": 0.5257, - "coverage": 1, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7854, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.8123, - "headline": false - } - ] - }, - "11": { - "raw": 0.5404, - "skill": 0.3353, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.8836, - "skill": 0.7673, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.6065, - "skill": 0.2131, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5932, - "skill": 0.1864, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7832, - "skill": 0.711, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.4796, - "skill": 0.3061, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.3965, - "skill": 0.2673, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4405, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.3525, - "headline": false - } - ] - }, - "31": { - "raw": 0.1428, - "skill": 0.0664, - "coverage": 1, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0848, - "skill": 0.0422, - "coverage": 1, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.5675, - "skill": 0.4575, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.5367, - "skill": 0.4039, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.3684, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.5367, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.79, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.935, - "headline": false - } - ] - }, - "41": { - "raw": 0.5763, - "skill": 0.3645, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.493, - "skill": 0.1956, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.6222, - "skill": 0.2444, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.4982, - "skill": 0.2717, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.72, - "pending": 0.24 - }, - { - "engine": "semif", - "name": "SemIf", - "long_name": "SemIf Qwen3.5-4B Base", - "variant": null, - "meta": { - "kind": "inference technique", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 4659865088, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": null, - "model_url": null, - "code_url": "https://github.com/TheoLeeCJ/openjev", - "base_model": "Qwen/Qwen3.5-4B-Base", - "base_url": "https://huggingface.co/Qwen/Qwen3.5-4B-Base", - "weights_repo": null, - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 113935, - "unsupported": 18487 - }, - "calibration": { - "acc": 0.6202, - "conf": 0.739, - "ece": 0.1188, - "brier": 0.508, - "over95": 0.0243, - "n": 69607, - "benchmarks": 29, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0004, - "conf": 0.1774, - "acc": 0.0 - }, - { - "lo": 0.2, - "w": 0.0166, - "conf": 0.2633, - "acc": 0.1842 - }, - { - "lo": 0.3, - "w": 0.0553, - "conf": 0.3561, - "acc": 0.2891 - }, - { - "lo": 0.4, - "w": 0.0669, - "conf": 0.4501, - "acc": 0.3225 - }, - { - "lo": 0.5, - "w": 0.1611, - "conf": 0.5495, - "acc": 0.5093 - }, - { - "lo": 0.6, - "w": 0.1095, - "conf": 0.6494, - "acc": 0.5582 - }, - { - "lo": 0.7, - "w": 0.1511, - "conf": 0.7456, - "acc": 0.4769 - }, - { - "lo": 0.8, - "w": 0.1118, - "conf": 0.8531, - "acc": 0.6857 - }, - { - "lo": 0.9, - "w": 0.3274, - "conf": 0.9688, - "acc": 0.8789 - } - ], - "areas": { - "knowledge": { - "ece": 0.1444, - "acc": 0.5452, - "conf": 0.6896 - }, - "language": { - "ece": 0.1384, - "acc": 0.6874, - "conf": 0.822 - }, - "retrieval": { - "ece": 0.1798, - "acc": 0.6178, - "conf": 0.7891 - }, - "tools": { - "ece": 0.0383, - "acc": 0.919, - "conf": 0.8897 - }, - "arts": { - "ece": 0.0496, - "acc": 0.6169, - "conf": 0.6665 - } - } - }, - "gaps": { - "share": 0.1396, - "reasons": [ - { - "reason": "supports only 2\u201316 answer options per question", - "share": 0.1157 - }, - { - "reason": "shared state prefix does not match every prompt (no truncation allowed)", - "share": 0.0179 - }, - { - "reason": "prompt longer than its 4,096-token context window", - "share": 0.006 - } - ] - }, - "latency": { - "median": 110.4, - "p95": 958.6, - "mean": 237.5 - }, - "results": { - "1": { - "score": 0.9326, - "jev": 0.9574, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1692, - "unsupported": 2, - "errors": 0, - "median_ms": 115.4 - }, - "2": { - "score": 0.4061, - "jev": 0.445, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7374, - "unsupported": 330, - "errors": 0, - "median_ms": 1134.5 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": null, - "jev": null, - "metric": null, - "requests": 3080, - "answered": 0, - "unsupported": 3080, - "errors": 0, - "median_ms": null - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": 0.7944, - "jev": 0.7982, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 9936, - "unsupported": 64, - "errors": 0, - "median_ms": 206.8 - }, - "9": { - "score": 0.1, - "jev": 0.525, - "metric": "case exact accuracy", - "requests": 160, - "answered": 160, - "unsupported": 0, - "errors": 0, - "median_ms": 354.7 - }, - "10": { - "score": 0.568, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 118.9 - }, - "11": { - "score": 0.6096, - "jev": 0.7161, - "metric": "macro-F1", - "requests": 123, - "answered": 108, - "unsupported": 15, - "errors": 0, - "median_ms": 456.8 - }, - "12": { - "score": 0.5491, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 108.7 - }, - "20": { - "score": 0.8522, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 108.3 - }, - "21": { - "score": 0.5993, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 107.6 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.566, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 110.1 - }, - "24": { - "score": 0.7144, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 108.3 - }, - "25": { - "score": 0.3673, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 109.2 - }, - "26": { - "score": 0.9676, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 108.1 - }, - "27": { - "score": 0.9181, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 108.4 - }, - "28": { - "score": 0.6598, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 107.3 - }, - "29": { - "score": 0.8272, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 109.5 - }, - "30": { - "score": 0.3131, - "jev": 0.7987, - "metric": "accuracy", - "requests": 2638, - "answered": 2638, - "unsupported": 0, - "errors": 0, - "median_ms": 111.0 - }, - "31": { - "score": 0.3316, - "jev": 0.3914, - "metric": "accuracy", - "requests": 5000, - "answered": 787, - "unsupported": 4213, - "errors": 0, - "median_ms": 110.9 - }, - "32": { - "score": 0.5465, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 116.3 - }, - "33": { - "score": 0.2005, - "jev": 0.2637, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1646, - "unsupported": 4, - "errors": 0, - "median_ms": 134.3 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 109.8 - }, - "36": { - "score": 0.159, - "jev": 0.1822, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1254, - "unsupported": 43, - "errors": 0, - "median_ms": 1533.8 - }, - "37": { - "score": 0.3635, - "jev": 0.5565, - "metric": "macro-F1", - "requests": 5000, - "answered": 4722, - "unsupported": 278, - "errors": 0, - "median_ms": 112.4 - }, - "38": { - "score": 0.0107, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 469.5 - }, - "39": { - "score": 0.6596, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 111.5 - }, - "40": { - "score": 0.3748, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4534, - "unsupported": 47, - "errors": 0, - "median_ms": 108.5, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2662, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3748, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.645, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.84, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.4396, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 109.4 - }, - "42": { - "score": 0.7202, - "jev": 0.8428, - "metric": "macro-F1", - "requests": 5500, - "answered": 3926, - "unsupported": 1574, - "errors": 0, - "median_ms": 111.3 - }, - "43": { - "score": 0.3895, - "jev": 0.7298, - "metric": "accuracy", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "median_ms": 108.9 - }, - "44": { - "score": 0.5596, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 107.9 - }, - "45": { - "score": 0.1095, - "jev": 0.2045, - "metric": "accuracy", - "requests": 501, - "answered": 484, - "unsupported": 17, - "errors": 0, - "median_ms": 111.3 - }, - "48": { - "score": 0.2252, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 112.8 - }, - "50": { - "score": 0.4525, - "jev": 0.4675, - "metric": "accuracy", - "requests": 1676, - "answered": 1200, - "unsupported": 476, - "errors": 0, - "median_ms": 117.8 - } - }, - "scores": { - "balanced_raw": 44.77, - "balanced_skill": 26.01, - "breadth_skill": 23.38 - }, - "frozen_scores": { - "balanced_raw": 35.11, - "balanced_skill": 19.42, - "breadth_skill": 16.67 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4015, - "skill": 0.1856, - "coverage": 0.8596, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.4375, - "skill": 0.2098, - "coverage": 0.9546, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.2319, - "skill": 0.1245, - "coverage": 0.9556, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.7015, - "skill": 0.5747, - "coverage": 0.9811, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4659, - "skill": 0.206, - "coverage": 0.7432, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9315, - "skill": 0.9076, - "coverage": 0.9988, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.3887, - "skill": 0.3258, - "coverage": 0.9572, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.7843, - "skill": 0.4908, - "coverage": 0.9872, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.7718, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.7967, - "headline": false - } - ] - }, - "11": { - "raw": 0.4982, - "skill": 0.2743, - "coverage": 0.878, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.8517, - "skill": 0.7034, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5993, - "skill": 0.1986, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.5545, - "skill": 0.109, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.7275, - "skill": 0.6367, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.3673, - "skill": 0.1565, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.3131, - "skill": 0.1702, - "coverage": 1, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.4003, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.2259, - "headline": false - } - ] - }, - "31": { - "raw": 0.0522, - "skill": 0.0, - "coverage": 0.1574, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1537, - "skill": 0.1144, - "coverage": 0.9668, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.3101, - "skill": 0.1347, - "coverage": 0.9444, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.3748, - "skill": 0.1957, - "coverage": 0.9857, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.2662, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.3748, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.645, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.84, - "headline": false - } - ] - }, - "41": { - "raw": 0.4396, - "skill": 0.1595, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3895, - "skill": 0.0313, - "coverage": 1, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.5596, - "skill": 0.1192, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.324, - "skill": 0.0189, - "coverage": 0.716, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.664, - "pending": 0.24 - }, - { - "engine": "solomon-v11-bf16-encoding", - "name": "Solomon v1.1", - "long_name": "Solomon v1.1", - "variant": null, - "meta": { - "kind": "head / adapter", - "technique": "autoregressive", - "technique_name": "autoregressive", - "served_params": 27781427952, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 903468342, - "model_url": "https://huggingface.co/DoccyHealth/Solomon", - "code_url": null, - "base_model": "Qwen/Qwen3.8-27B", - "base_url": "https://huggingface.co/Qwen/Qwen3.8-27B", - "weights_repo": "DoccyHealth/Solomon", - "served_checkpoint": null - }, - "completed": 132410, - "counts": { - "ok": 104787, - "unsupported": 27623 - }, - "calibration": { - "acc": 0.7017, - "conf": 0.7274, - "ece": 0.1012, - "brier": 0.4456, - "over95": 0.0291, - "n": 69333, - "benchmarks": 28, - "rel": [ - { - "lo": 0.0, - "w": 0, - "conf": null, - "acc": null - }, - { - "lo": 0.1, - "w": 0.0044, - "conf": 0.1867, - "acc": 0.3613 - }, - { - "lo": 0.2, - "w": 0.0123, - "conf": 0.2605, - "acc": 0.2044 - }, - { - "lo": 0.3, - "w": 0.1279, - "conf": 0.3483, - "acc": 0.5935 - }, - { - "lo": 0.4, - "w": 0.0695, - "conf": 0.4484, - "acc": 0.5093 - }, - { - "lo": 0.5, - "w": 0.127, - "conf": 0.5513, - "acc": 0.5625 - }, - { - "lo": 0.6, - "w": 0.1007, - "conf": 0.6462, - "acc": 0.621 - }, - { - "lo": 0.7, - "w": 0.0789, - "conf": 0.751, - "acc": 0.653 - }, - { - "lo": 0.8, - "w": 0.0982, - "conf": 0.8521, - "acc": 0.7032 - }, - { - "lo": 0.9, - "w": 0.3811, - "conf": 0.97, - "acc": 0.8706 - } - ], - "areas": { - "knowledge": { - "ece": 0.1264, - "acc": 0.6819, - "conf": 0.6315 - }, - "language": { - "ece": 0.0916, - "acc": 0.7966, - "conf": 0.8783 - }, - "retrieval": { - "ece": 0.2617, - "acc": 0.5834, - "conf": 0.8451 - }, - "tools": { - "ece": 0.0072, - "acc": 0.991, - "conf": 0.9929 - }, - "arts": { - "ece": 0.0823, - "acc": 0.5919, - "conf": 0.6736 - } - } - }, - "gaps": { - "share": 0.2086, - "reasons": [ - { - "reason": "supports only 2\u20138 answer options per question", - "share": 0.2051 - }, - { - "reason": "request shape outside the model's schema", - "share": 0.0031 - }, - { - "reason": "request larger than its native request budget", - "share": 0.0004 - } - ] - }, - "latency": { - "median": 223.2, - "p95": 5301.0, - "mean": 1005.3 - }, - "results": { - "1": { - "score": 0.9799, - "jev": 0.9575, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1694, - "unsupported": 0, - "errors": 0, - "median_ms": 504.8 - }, - "2": { - "score": 0.441, - "jev": 0.4455, - "metric": "nDCG@10", - "requests": 7704, - "answered": 7654, - "unsupported": 50, - "errors": 0, - "median_ms": 4227.1 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": null, - "jev": null, - "metric": null, - "requests": 3080, - "answered": 0, - "unsupported": 3080, - "errors": 0, - "median_ms": null - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": null, - "jev": null, - "metric": null, - "requests": 10000, - "answered": 0, - "unsupported": 10000, - "errors": 0, - "median_ms": null - }, - "9": { - "score": null, - "jev": null, - "metric": null, - "requests": 160, - "answered": 0, - "unsupported": 160, - "errors": 0, - "median_ms": null - }, - "10": { - "score": 0.592, - "jev": 0.4295, - "metric": "macro-F1", - "requests": 2500, - "answered": 2500, - "unsupported": 0, - "errors": 0, - "median_ms": 458.5 - }, - "11": { - "score": 0.7153, - "jev": 0.7169, - "metric": "macro-F1", - "requests": 123, - "answered": 123, - "unsupported": 0, - "errors": 0, - "median_ms": 2779.2 - }, - "12": { - "score": 0.6708, - "jev": 0.7479, - "metric": "macro-F1", - "requests": 3200, - "answered": 3200, - "unsupported": 0, - "errors": 0, - "median_ms": 215.2 - }, - "20": { - "score": 0.9414, - "jev": 0.906, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 217.9 - }, - "21": { - "score": 0.5422, - "jev": 0.6187, - "metric": "accuracy", - "requests": 2628, - "answered": 2628, - "unsupported": 0, - "errors": 0, - "median_ms": 218.3 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.502, - "jev": 0.6474, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 257.6 - }, - "24": { - "score": 0.6995, - "jev": 0.9173, - "metric": "accuracy", - "requests": 14033, - "answered": 14033, - "unsupported": 0, - "errors": 0, - "median_ms": 215.6 - }, - "25": { - "score": 0.4286, - "jev": 0.7857, - "metric": "accuracy", - "requests": 196, - "answered": 196, - "unsupported": 0, - "errors": 0, - "median_ms": 221.6 - }, - "26": { - "score": 0.8657, - "jev": 0.9933, - "metric": "accuracy", - "requests": 2376, - "answered": 2376, - "unsupported": 0, - "errors": 0, - "median_ms": 214.6 - }, - "27": { - "score": 0.8456, - "jev": 0.9778, - "metric": "accuracy", - "requests": 1172, - "answered": 1172, - "unsupported": 0, - "errors": 0, - "median_ms": 212.6 - }, - "28": { - "score": 0.8106, - "jev": 0.9195, - "metric": "accuracy", - "requests": 1267, - "answered": 1267, - "unsupported": 0, - "errors": 0, - "median_ms": 212.4 - }, - "29": { - "score": 0.9099, - "jev": 0.9452, - "metric": "accuracy", - "requests": 10042, - "answered": 10042, - "unsupported": 0, - "errors": 0, - "median_ms": 216.8 - }, - "30": { - "score": null, - "jev": null, - "metric": null, - "requests": 2638, - "answered": 1319, - "unsupported": 1319, - "errors": 0, - "median_ms": 218.9 - }, - "31": { - "score": 0.4019, - "jev": 0.4533, - "metric": "accuracy", - "requests": 5000, - "answered": 525, - "unsupported": 4475, - "errors": 0, - "median_ms": 218.4 - }, - "32": { - "score": 0.6064, - "jev": 0.6609, - "metric": "accuracy", - "requests": 752, - "answered": 752, - "unsupported": 0, - "errors": 0, - "median_ms": 348.2 - }, - "33": { - "score": 0.2055, - "jev": 0.2642, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 1650, - "unsupported": 0, - "errors": 0, - "median_ms": 1070.7 - }, - "34": { - "score": 0.1, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 10, - "unsupported": 0, - "errors": 0, - "median_ms": 216.3 - }, - "36": { - "score": 0.1863, - "jev": 0.1873, - "metric": "nDCG@10", - "requests": 1297, - "answered": 1292, - "unsupported": 5, - "errors": 0, - "median_ms": 4641.0 - }, - "37": { - "score": 0.5618, - "jev": 0.5521, - "metric": "macro-F1", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 222.6 - }, - "38": { - "score": 0.0029, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 7653.2 - }, - "39": { - "score": 0.9186, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 333.5 - }, - "40": { - "score": 0.4822, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4600, - "unsupported": 0, - "errors": 0, - "median_ms": 222.2, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.4583, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4822, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.715, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.9, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.694, - "jev": 0.6463, - "metric": "macro-F1", - "requests": 3006, - "answered": 3006, - "unsupported": 0, - "errors": 0, - "median_ms": 218.9 - }, - "42": { - "score": 0.8176, - "jev": 0.8406, - "metric": "macro-F1", - "requests": 5500, - "answered": 5500, - "unsupported": 0, - "errors": 0, - "median_ms": 239.3 - }, - "43": { - "score": 0.6933, - "jev": 0.734, - "metric": "accuracy", - "requests": 570, - "answered": 564, - "unsupported": 6, - "errors": 0, - "median_ms": 220.4 - }, - "44": { - "score": 0.7044, - "jev": 0.7264, - "metric": "accuracy", - "requests": 5000, - "answered": 5000, - "unsupported": 0, - "errors": 0, - "median_ms": 220.5 - }, - "45": { - "score": 0.1474, - "jev": 0.2138, - "metric": "accuracy", - "requests": 501, - "answered": 407, - "unsupported": 94, - "errors": 0, - "median_ms": 223.1 - }, - "48": { - "score": 0.2385, - "jev": 0.1736, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 10139, - "unsupported": 0, - "errors": 0, - "median_ms": 286.7 - }, - "50": { - "score": 0.3729, - "jev": 0.4594, - "metric": "accuracy", - "requests": 1676, - "answered": 1676, - "unsupported": 0, - "errors": 0, - "median_ms": 268.6 - } - }, - "scores": { - "balanced_raw": 47.51, - "balanced_skill": 34.85, - "breadth_skill": 33.69 - }, - "frozen_scores": { - "balanced_raw": 36.54, - "balanced_skill": 25.45, - "breadth_skill": 21.14 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.4261, - "skill": 0.2904, - "coverage": 0.6824, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.6305, - "skill": 0.4877, - "coverage": 1, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.3737, - "skill": 0.2991, - "coverage": 0.9981, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.4727, - "skill": 0.4511, - "coverage": 0.6645, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.4728, - "skill": 0.2144, - "coverage": 0.8, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.9799, - "skill": 0.9729, - "coverage": 1, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.4381, - "skill": 0.3803, - "coverage": 0.9935, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0, - "headline": false - } - ] - }, - "11": { - "raw": 0.7153, - "skill": 0.5882, - "coverage": 1, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.9437, - "skill": 0.8874, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.5422, - "skill": 0.0845, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.505, - "skill": 0.01, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.6953, - "skill": 0.5938, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.4286, - "skill": 0.2381, - "coverage": 1, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.0, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.0, - "headline": false - } - ] - }, - "31": { - "raw": 0.0422, - "skill": 0.0, - "coverage": 0.105, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.1856, - "skill": 0.1478, - "coverage": 0.9961, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.5618, - "skill": 0.4504, - "coverage": 1, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.4822, - "skill": 0.3338, - "coverage": 1, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.4583, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.4822, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.715, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.9, - "headline": false - } - ] - }, - "41": { - "raw": 0.694, - "skill": 0.541, - "coverage": 1, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.686, - "skill": 0.5018, - "coverage": 0.9895, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.7044, - "skill": 0.4088, - "coverage": 1, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3729, - "skill": 0.0899, - "coverage": 1, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.6034, - "pending": 0.24 - }, - { - "engine": "verdict", - "name": "Verdict", - "long_name": "Verdict", - "variant": null, - "meta": { - "kind": "full fine-tune", - "technique": "GLi family", - "technique_name": "GLiClass", - "served_params": 151378176, - "params_estimated": false, - "params_from_name": false, - "trained_bytes": 605529340, - "model_url": "https://huggingface.co/heman10x/rlcd-modernbert-151m", - "code_url": "https://github.com/Heman10x-NGU/Verdict-open-jev", - "base_model": "knowledgator/gliclass-modern-base-v2.0", - "base_url": "https://huggingface.co/knowledgator/gliclass-modern-base-v2.0", - "weights_repo": "heman10x/rlcd-modernbert-151m", - "served_checkpoint": null - }, - "completed": 132422, - "counts": { - "ok": 68131, - "unsupported": 33898, - "abstained": 30393 - }, - "calibration": { - "acc": 0.3675, - "conf": 0.706, - "ece": 0.3386, - "brier": 0.8745, - "over95": 0.0928, - "n": 61069, - "benchmarks": 28, - "rel": [ - { - "lo": 0.0, - "w": 0.0036, - "conf": 0.084, - "acc": 0.0909 - }, - { - "lo": 0.1, - "w": 0.0141, - "conf": 0.1476, - "acc": 0.1047 - }, - { - "lo": 0.2, - "w": 0.0101, - "conf": 0.2559, - "acc": 0.1309 - }, - { - "lo": 0.3, - "w": 0.0367, - "conf": 0.3591, - "acc": 0.3041 - }, - { - "lo": 0.4, - "w": 0.0669, - "conf": 0.4587, - "acc": 0.3014 - }, - { - "lo": 0.5, - "w": 0.1936, - "conf": 0.5435, - "acc": 0.3614 - }, - { - "lo": 0.6, - "w": 0.1527, - "conf": 0.6488, - "acc": 0.4404 - }, - { - "lo": 0.7, - "w": 0.1625, - "conf": 0.7501, - "acc": 0.4059 - }, - { - "lo": 0.8, - "w": 0.1255, - "conf": 0.853, - "acc": 0.3844 - }, - { - "lo": 0.9, - "w": 0.2344, - "conf": 0.9556, - "acc": 0.3482 - } - ], - "areas": { - "knowledge": { - "ece": 0.3251, - "acc": 0.346, - "conf": 0.6709 - }, - "language": { - "ece": 0.3259, - "acc": 0.4161, - "conf": 0.742 - }, - "retrieval": { - "ece": 0.3737, - "acc": 0.2898, - "conf": 0.6635 - }, - "tools": { - "ece": 0.614, - "acc": 0.3048, - "conf": 0.9188 - }, - "arts": { - "ece": 0.2501, - "acc": 0.4398, - "conf": 0.6896 - } - } - }, - "gaps": { - "share": 0.4855, - "reasons": [ - { - "reason": "chose its own abstain option instead of an answer", - "share": 0.2295 - }, - { - "reason": "prompt longer than its 1,024-token context window", - "share": 0.1447 - }, - { - "reason": "supports only 2\u201324 answer options per question", - "share": 0.1113 - } - ] - }, - "latency": { - "median": 11.8, - "p95": 167.8, - "mean": 33.3 - }, - "results": { - "1": { - "score": 0.4962, - "jev": 0.9645, - "metric": "case exact accuracy", - "requests": 1694, - "answered": 1324, - "unsupported": 370, - "errors": 0, - "median_ms": 12.6 - }, - "2": { - "score": 0.1346, - "jev": 0.4488, - "metric": "nDCG@10", - "requests": 7704, - "answered": 6108, - "unsupported": 1577, - "errors": 0, - "median_ms": 170.1 - }, - "3": { - "score": null, - "jev": null, - "metric": null, - "requests": 508, - "answered": 0, - "unsupported": 508, - "errors": 0, - "median_ms": null - }, - "4": { - "score": null, - "jev": null, - "metric": null, - "requests": 3080, - "answered": 0, - "unsupported": 3080, - "errors": 0, - "median_ms": null - }, - "5": { - "score": null, - "jev": null, - "metric": null, - "requests": 5500, - "answered": 0, - "unsupported": 5500, - "errors": 0, - "median_ms": null - }, - "6": { - "score": 0.0, - "jev": 1.0, - "metric": "selected quality (quality objective)", - "requests": 10000, - "answered": 2, - "unsupported": 9998, - "errors": 0, - "median_ms": 23.4 - }, - "9": { - "score": 0, - "jev": 0.5758, - "metric": "case exact accuracy", - "requests": 160, - "answered": 33, - "unsupported": 0, - "errors": 0, - "median_ms": 152.1 - }, - "10": { - "score": 0.0688, - "jev": 0.5214, - "metric": "macro-F1", - "requests": 2500, - "answered": 889, - "unsupported": 1611, - "errors": 0, - "median_ms": 15.3 - }, - "11": { - "score": 0.3042, - "jev": 0.6448, - "metric": "macro-F1", - "requests": 123, - "answered": 3, - "unsupported": 110, - "errors": 0, - "median_ms": 101.8 - }, - "12": { - "score": 0.2138, - "jev": 0.7596, - "metric": "macro-F1", - "requests": 3200, - "answered": 1786, - "unsupported": 0, - "errors": 0, - "median_ms": 11.2 - }, - "20": { - "score": 0.4378, - "jev": 0.8983, - "metric": "accuracy", - "requests": 5000, - "answered": 3385, - "unsupported": 0, - "errors": 0, - "median_ms": 11.2 - }, - "21": { - "score": 0.4653, - "jev": 0.6238, - "metric": "accuracy", - "requests": 2628, - "answered": 101, - "unsupported": 0, - "errors": 0, - "median_ms": 10.9 - }, - "22": { - "score": null, - "jev": null, - "metric": null, - "requests": 2000, - "answered": 0, - "unsupported": 2000, - "errors": 0, - "median_ms": null - }, - "23": { - "score": 0.546, - "jev": 0.6339, - "metric": "accuracy", - "requests": 5000, - "answered": 2958, - "unsupported": 0, - "errors": 0, - "median_ms": 11.7 - }, - "24": { - "score": 0.3153, - "jev": 0.9118, - "metric": "accuracy", - "requests": 14033, - "answered": 6654, - "unsupported": 0, - "errors": 0, - "median_ms": 11.1 - }, - "25": { - "score": 0.3267, - "jev": 0.7723, - "metric": "accuracy", - "requests": 196, - "answered": 101, - "unsupported": 1, - "errors": 0, - "median_ms": 11.5 - }, - "26": { - "score": 0.517, - "jev": 0.9932, - "metric": "accuracy", - "requests": 2376, - "answered": 1031, - "unsupported": 0, - "errors": 0, - "median_ms": 10.6 - }, - "27": { - "score": 0.2863, - "jev": 0.9861, - "metric": "accuracy", - "requests": 1172, - "answered": 503, - "unsupported": 0, - "errors": 0, - "median_ms": 10.9 - }, - "28": { - "score": 0.4952, - "jev": 0.9198, - "metric": "accuracy", - "requests": 1267, - "answered": 1260, - "unsupported": 0, - "errors": 0, - "median_ms": 9.8 - }, - "29": { - "score": 0.3439, - "jev": 0.9552, - "metric": "accuracy", - "requests": 10042, - "answered": 8483, - "unsupported": 0, - "errors": 0, - "median_ms": 11.7 - }, - "30": { - "score": 0.2143, - "jev": 0.8071, - "metric": "accuracy", - "requests": 2638, - "answered": 465, - "unsupported": 0, - "errors": 0, - "median_ms": 11.3 - }, - "31": { - "score": 0.1868, - "jev": 0.2954, - "metric": "accuracy", - "requests": 5000, - "answered": 1317, - "unsupported": 3641, - "errors": 0, - "median_ms": 12.0 - }, - "32": { - "score": 0.4617, - "jev": 0.6885, - "metric": "accuracy", - "requests": 752, - "answered": 366, - "unsupported": 378, - "errors": 0, - "median_ms": 14.6 - }, - "33": { - "score": 0.0053, - "jev": 0.2434, - "metric": "case exact accuracy", - "requests": 1650, - "answered": 189, - "unsupported": 253, - "errors": 0, - "median_ms": 38.1 - }, - "34": { - "score": 0.125, - "jev": 0.5, - "metric": "accuracy", - "requests": 10, - "answered": 8, - "unsupported": 0, - "errors": 0, - "median_ms": 11.5 - }, - "36": { - "score": 0.0247, - "jev": 0.1089, - "metric": "nDCG@10", - "requests": 1297, - "answered": 547, - "unsupported": 725, - "errors": 0, - "median_ms": 169.4 - }, - "37": { - "score": 0.2049, - "jev": 0.5467, - "metric": "macro-F1", - "requests": 5000, - "answered": 3827, - "unsupported": 340, - "errors": 0, - "median_ms": 11.7 - }, - "38": { - "score": 0, - "jev": 0.0558, - "metric": "case exact accuracy", - "requests": 5479, - "answered": 5479, - "unsupported": 0, - "errors": 0, - "median_ms": 80.0 - }, - "39": { - "score": 0.4063, - "jev": 0.8698, - "metric": "macro-F1", - "requests": 979, - "answered": 979, - "unsupported": 0, - "errors": 0, - "median_ms": 12.0 - }, - "40": { - "score": 0.1502, - "jev": 0.5051, - "metric": "Sarcasm F1 \u00b7 track A, English", - "requests": 4600, - "answered": 4183, - "unsupported": 0, - "errors": 0, - "median_ms": 9.9, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0163, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.1502, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.28, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.09, - "headline": false - } - ], - "jev_note": "Jev on all track cases" - }, - "41": { - "score": 0.2989, - "jev": 0.6528, - "metric": "macro-F1", - "requests": 3006, - "answered": 2555, - "unsupported": 0, - "errors": 0, - "median_ms": 11.4 - }, - "42": { - "score": 0.4939, - "jev": 0.8554, - "metric": "macro-F1", - "requests": 5500, - "answered": 3757, - "unsupported": 596, - "errors": 0, - "median_ms": 12.0 - }, - "43": { - "score": 0.3854, - "jev": 0.7388, - "metric": "accuracy", - "requests": 570, - "answered": 467, - "unsupported": 0, - "errors": 0, - "median_ms": 11.2 - }, - "44": { - "score": 0.5102, - "jev": 0.6131, - "metric": "accuracy", - "requests": 5000, - "answered": 1225, - "unsupported": 0, - "errors": 0, - "median_ms": 11.2 - }, - "45": { - "score": 0.1559, - "jev": 0.2102, - "metric": "accuracy", - "requests": 501, - "answered": 295, - "unsupported": 33, - "errors": 0, - "median_ms": 11.5 - }, - "48": { - "score": 0.3757, - "jev": 0.1708, - "metric": "Brier (lower is better)", - "requests": 10139, - "answered": 7117, - "unsupported": 2406, - "errors": 0, - "median_ms": 13.0 - }, - "50": { - "score": 0.3971, - "jev": 0.4766, - "metric": "accuracy", - "requests": 1676, - "answered": 1322, - "unsupported": 350, - "errors": 0, - "median_ms": 15.6 - } - }, - "scores": { - "balanced_raw": 13.38, - "balanced_skill": 1.27, - "breadth_skill": 1.06 - }, - "frozen_scores": { - "balanced_raw": 10.51, - "balanced_skill": 0.77, - "breadth_skill": 0.68 - }, - "categories": [ - { - "id": "knowledge", - "raw": 0.1353, - "skill": 0.0, - "coverage": 0.395, - "pending": 0, - "n": 6 - }, - { - "id": "language", - "raw": 0.1339, - "skill": 0.0, - "coverage": 0.5945, - "pending": 0, - "n": 3 - }, - { - "id": "retrieval", - "raw": 0.0796, - "skill": 0.0, - "coverage": 0.5936, - "pending": 0, - "n": 2 - }, - { - "id": "tools", - "raw": 0.1649, - "skill": 0.0628, - "coverage": 0.5249, - "pending": 0, - "n": 3 - }, - { - "id": "arts", - "raw": 0.1551, - "skill": 0.0007, - "coverage": 0.3834, - "pending": 0, - "n": 5 - } - ], - "benchmarks": { - "1": { - "raw": 0.3878, - "skill": 0.1736, - "coverage": 0.7816, - "random": 0.2592, - "tracks": [] - }, - "2": { - "raw": 0.1067, - "skill": 0.0148, - "coverage": 0.7928, - "random": 0.0933, - "tracks": [] - }, - "6": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0002, - "random": 0.5245, - "tracks": [ - { - "track": "RouterBench-0shot", - "score": 0.0, - "headline": false - }, - { - "track": "RouterBench-5shot", - "score": 0.0, - "headline": false - } - ] - }, - "11": { - "raw": 0.0089, - "skill": 0.0, - "coverage": 0.0244, - "random": 0.3085, - "tracks": [] - }, - "20": { - "raw": 0.1573, - "skill": 0.0, - "coverage": 0.4982, - "random": 0.5, - "tracks": [] - }, - "21": { - "raw": 0.0179, - "skill": 0.0, - "coverage": 0.0384, - "random": 0.5, - "tracks": [] - }, - "22": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0, - "random": 0.0078, - "tracks": [] - }, - "23": { - "raw": 0.287, - "skill": 0.0, - "coverage": 0.5916, - "random": 0.5, - "tracks": [] - }, - "24": { - "raw": 0.1419, - "skill": 0.0, - "coverage": 0.4742, - "random": 0.25, - "tracks": [] - }, - "25": { - "raw": 0.1684, - "skill": 0.0, - "coverage": 0.5153, - "random": 0.25, - "tracks": [] - }, - "30": { - "raw": 0.0114, - "skill": 0.0, - "coverage": 0.0531, - "random": 0.25, - "tracks": [ - { - "track": "GSM8K-4choice", - "score": 0.0136, - "headline": false - }, - { - "track": "GSM8K-10choice", - "score": 0.0091, - "headline": false - } - ] - }, - "31": { - "raw": 0.0492, - "skill": 0.0, - "coverage": 0.2634, - "random": 0.0819, - "tracks": [] - }, - "36": { - "raw": 0.0104, - "skill": 0.0, - "coverage": 0.4217, - "random": 0.0444, - "tracks": [] - }, - "37": { - "raw": 0.1489, - "skill": 0.0, - "coverage": 0.7654, - "random": 0.2027, - "tracks": [] - }, - "40": { - "raw": 0.1502, - "skill": 0.0, - "coverage": 0.9093, - "random": 0.2227, - "tracks": [ - { - "track": "A \u00b7 Arabic", - "score": 0.0163, - "headline": false - }, - { - "track": "A \u00b7 English", - "score": 0.1502, - "headline": true - }, - { - "track": "C \u00b7 Arabic pairs", - "score": 0.28, - "headline": false - }, - { - "track": "C \u00b7 English pairs", - "score": 0.09, - "headline": false - } - ] - }, - "41": { - "raw": 0.2426, - "skill": 0.0, - "coverage": 0.85, - "random": 0.3333, - "tracks": [] - }, - "43": { - "raw": 0.3158, - "skill": 0.0, - "coverage": 0.8193, - "random": 0.3697, - "tracks": [] - }, - "44": { - "raw": 0.125, - "skill": 0.0, - "coverage": 0.245, - "random": 0.5, - "tracks": [] - }, - "50": { - "raw": 0.3132, - "skill": 0.0033, - "coverage": 0.7888, - "random": 0.311, - "tracks": [] - }, - "7": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "16": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "13": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "15": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "17": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - }, - "19": { - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "random": null, - "tracks": [] - } - }, - "coverage": 0.3446, - "pending": 0.24 - } - ] -} \ No newline at end of file diff --git a/docs/training/jet-next/run-manifest.json b/docs/training/jet-next/run-manifest.json deleted file mode 100644 index 7d52f5469902cd33b63040559ef53dec9f1048fe..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/run-manifest.json +++ /dev/null @@ -1,159 +0,0 @@ -{ - "status": "completed", - "base": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "git_head": "829955484f94e1dff420880de656ce99775a7b61", - "branch": "codex/decision-index-training", - "runs": { - "warm": { - "directory": "adapters/jet-v3-warm-20260924", - "config": { - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v3.jsonl", - "val": "data/selection_v3.jsonl", - "out": "adapters/jet-v3-warm-20260924", - "epochs": 1.0, - "lr": 2e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 4096, - "max_state_tokens": 1024, - "smoothing": 0.02, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet", - "start_step": null, - "seed": 230923, - "data_sha256": { - "data/train_v3.jsonl": "ec5666923b48289d5ab3b0887f43e52ec403c838cce7a8eee9479888ccc58be4", - "data/selection_v3.jsonl": "f38f2fd8e503d749ea3a72a811348f663c30b9b5702f171c66f344fecc1dbee4" - }, - "init_adapter_sha256": "e70670a173d6d95c329acb72f692577eb47171c106443aa0171fb9584399fbce", - "train_examples": 47313, - "steps_per_epoch": 2444, - "total_steps": 2444 - }, - "best": { - "step": 2444, - "nll": 0.5131154033423817, - "acc": 0.7806882383153569, - "improved": true, - "peak_memory_gb": 4.074936008 - }, - "weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "last_weights_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "calibration": { - "choice": 0.9439, - "score": 1.0293, - "noul": 1.2599 - } - }, - "v4": { - "directory": "adapters/jet-v4-transfer-20260924", - "config": { - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v4_tools_focus.jsonl", - "val": "data/selection_v4_tools.jsonl", - "out": "adapters/jet-v4-transfer-20260924", - "epochs": 1.0, - "lr": 1e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 3072, - "max_state_tokens": 2048, - "smoothing": 0.02, - "use_example_weights": true, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet-v3-warm-20260924", - "start_step": null, - "seed": 230924, - "data_sha256": { - "data/train_v4_tools_focus.jsonl": "6f55f89bbd2d3211dd0cc0afca1398c74f997df0d76db0e15eba3a7390200601", - "data/selection_v4_tools.jsonl": "b430db40826a3da1cc9ca52723e1689be95bed5d0cf1cdc70b520101254a862a" - }, - "init_adapter_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "train_examples": 30000, - "steps_per_epoch": 4033, - "total_steps": 4033 - }, - "best": { - "step": 3750, - "nll": 0.554768383523464, - "acc": 0.7602339181286549, - "improved": true, - "peak_memory_gb": 3.4478726 - }, - "weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "last_weights_sha256": "2c650a6fb492f673aba7c2f362d83306d5b0dd0ad5c7fdb715d8fbde4a9c02fc", - "calibration": { - "choice": 1.1554, - "score": 1.1554, - "noul": 1.3348 - } - } - }, - "hardware": "NVIDIA GeForce RTX 4080 SUPER, 16376 MiB, 615.71.09", - "monitored_device_peak_mib": 10861, - "monitor_scope": "V4 training and subsequent evaluations, including desktop GPU use", - "code_sha256": { - "src/train.py": "040ef12d706db397d6031a745faf9fc922497e6eb59f0d45b0ac18cd13b8cbfd", - "src/evaluate.py": "25dcaf7bcf9ddcfdd8349eeea047ef7cc33388cf74f32a65c8a1931b4f40f72c", - "src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "src/bench_decision_index.py": "809bbfb59ecf007cb3b38a501ebeb36ec6fa9c68649cd21c360e5c15bde5536d", - "src/data/decision_training.py": "785b8407f93c9ba952f98d29f987be73c853707f4f3211ccd6a4865c4cee3061", - "train_cuda.sh": "464ebe6e844f2a02983f76f66edb7fd7f9354ed407c1ff2dfa043b97cfea6dcf", - "scripts/bench_index_cuda.sh": "415a46bf5bb39a4fdb9760339abff2304fb9762969da929965b6d26636ca99fc", - "scripts/evaluate_next_code.sh": "7755cc0e69d834db4686af24d4c1b991e8a4ae2f4593dac9ff159c9aec01236f", - "scripts/build_v4_transfer.py": "37262ee1a8978d19c58a031e11598d504ed2d4b981ee2f004d6c0a8ccff9af9b", - "scripts/audit_v4_transfer.py": "a878e78f59fbd1a1dd2032279222073b4c0dba01e509b231a77a74d21c24d011", - "scripts/prepare_v4_focus.py": "bd47eec38af66c0f27cd8973c8f4053bfaa558874b26a82b919a262d1916848c", - "scripts/build_v4_tools.py": "35b3694f83c326b3e236cf2528ebba341af0a0be34bb85b365409e6003019626", - "scripts/audit_v4_tools.py": "5e5c38a42274220bffe681133fa3f0ac6ac000bd72e1631e2a12b33b0f52d789", - "scripts/run_next_cuda.sh": "8cb6f179a0467565eb0cdf9348b55e305f628694dd82be8e15ae4c0fb4faa161", - "scripts/evaluate_next_extra.sh": "21b11259db33cbbf5c6e39b8cc4526770a0c2d71c5b7a5b0040c94447745a889", - "scripts/summarize_next.py": "584fe4addd24ce9b03de0b67635009bdaba9af39932f2d35a3505652b764f1df", - "scripts/finalize_next.py": "33b9456de4fdcca0f42ce81ef8ce2d8e35b36536e759ea47191179993bfb5b73", - "uv.lock": "158790ff4bb087ca458bbfd16025251923382fd6ebbee2965410fbf66b1c322b", - "pyproject.toml": "82c7478890e1e40bf184ed0d7428d7b2cbdf6ed968a1f45d4a554ec70f5192bc" - }, - "data_sha256": { - "train_v4.jsonl": "f997660ef5a798a4aeeb0ace8c9c76756751d71f87523a31ab9e7a084df8cd5f", - "selection_v4.jsonl": "43c9aef75410fe06efccf9ee8fb71196f2d9de0136785d1684a4d7ed2914a158", - "calibration_v4.jsonl": "e08bd6f26e3150485ad67099ecdb30700474ba3173570beb901136dc8a669a77", - "test_v4_new.jsonl": "36b31bc98595c814f53c745c02e902f263af5b63d6933af8137b6cd7411915f6", - "train_v4_focus.jsonl": "266817085e61b7ed176e9d02254abf7e55fa4ffcf4d0d2beb35c5c47d839c863", - "train_v4_tools_focus.jsonl": "6f55f89bbd2d3211dd0cc0afca1398c74f997df0d76db0e15eba3a7390200601", - "test_v4_tools.jsonl": "20f0e11691f8e51760bfef7cc9982ceb8719faeadb5340c14905b393cf58d68d", - "calibration_v4_tools.jsonl": "cbbe6269a2824a6239aaba9c89b64e43aac74788961f4e272fe369a73af0ab97", - "selection_v4_tools.jsonl": "b430db40826a3da1cc9ca52723e1689be95bed5d0cf1cdc70b520101254a862a" - }, - "protected_original_files": "unchanged", - "published_model": "unchanged", - "preexisting_user_changes": "unchanged", - "warm_source_snapshot_sha256": "d1552430ccc972c13e6371ba3030c86ff8439d7a96f46729b1add5e2fa344624", - "limitations": [ - "No official full-suite or leaderboard score", - "Native metrics only; installed panel proposal differs from deployed panel", - "Partial diagnostic differences are descriptive, not significance tests" - ] -} \ No newline at end of file diff --git a/docs/training/jet-next/summary.md b/docs/training/jet-next/summary.md deleted file mode 100644 index 58e31aa161903636ea63044ae52c692901723434..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/summary.md +++ /dev/null @@ -1,143 +0,0 @@ -# Jet Qwen3-0.6B follow-up results - -Two more local checkpoints were trained: a lower-learning-rate warm start from released Jet, then targeted transfer training. No weights were published and no leaderboard entry was submitted. - -V4 substantially improves contract entailment, product relevance, stance, and English sarcasm over released Jet on the evaluations below. Confirmed regressions remain in CRUXEval and original-test accuracy. The warm checkpoint preserves original-test accuracy more closely. Neither candidate is an across-the-board replacement. - -**These results do not establish a leaderboard win.** The official frozen suite remains unavailable. Diagnostics are partial public reconstructions; the installed kit also contains an older panel definition. Native benchmark metrics below are comparable across these four models on identical rows, but are not an overall index. - -| Held-out accuracy (%) | Released Jet | V3 | Warm start | V4 transfer | -|---|---:|---:|---:|---:| -| Original test (3,683) | 67.6 | 65.2 | 67.5 | 66.4 | -| Ordinal test (2,400) | 51.7 | 50.2 | 53.0 | 52.0 | -| V3 new families (1,364) | 45.7 | 80.4 | 77.5 | 77.2 | -| Harder transfer tasks (720) | 34.2 | 50.4 | 48.3 | 66.0 | -| SGD domain-held-out routing (240) | 36.2 | 31.7 | 28.7 | 93.3 | - -| Partial benchmark — native metric (%) | Released Jet | V3 | Warm start | V4 transfer | -|---|---:|---:|---:|---:| -| ContractNLI — macro-F1 | 28.6 | 28.3 | 30.8 | 65.4 | -| GSM8K — accuracy | 20.5 | 46.4 | 39.3 | 37.5 | -| iSarcasmEval — see separate subtask tracks | — | — | — | — | -| VAST — macro-F1 | 36.6 | 32.3 | 39.4 | 44.3 | -| NLI4CT — macro-F1 | 25.1 | 58.8 | 50.8 | 52.0 | -| CRUXEval — accuracy | 40.0 | 38.2 | 40.0 | 32.7 | -| CLadder — accuracy | 38.2 | 43.6 | 41.8 | 47.3 | -| Habermas Machine — accuracy | 29.1 | 29.1 | 25.5 | 32.7 | -| Amazon ESCI — macro-F1 | 19.5 | 41.4 | 41.5 | 41.3 | -| iSarcasmEval English A, all 1,400 cases — positive F1 | 18.2 | 18.9 | 17.6 | 23.8 | -| CRUXEval, all 570 available eligible cases — accuracy | 38.8 | 34.7 | 35.1 | 34.4 | - -iSarcasmEval has no aggregate native score in the pinned summary report. A missing value is not zero. The separate English-A row uses the native positive-class F1 scorer on all 1,400 available cases (the official headline track), without combining other sarcasm tracks. The 500-request eight-benchmark sample and separate 500-request ESCI sample are small; differences are descriptive, not statistical significance claims. - -| V4 task family — accuracy (%) | Released Jet | V3 | Warm start | V4 transfer | -|---|---:|---:|---:|---:| -| code_transfer | 30.0 | 52.5 | 38.3 | 57.1 | -| contract_entailment | 39.6 | 45.4 | 46.7 | 80.8 | -| stance_transfer | 32.9 | 53.3 | 60.0 | 60.0 | -| tool_intent_routing | 36.2 | 31.7 | 28.7 | 93.3 | - -| Original family — accuracy (%) | Released Jet | V3 | Warm start | V4 transfer | -|---|---:|---:|---:|---:| -| ag_news | 88.1 | 89.6 | 88.9 | 88.9 | -| banking77 | 88.6 | 86.4 | 92.9 | 90.7 | -| boolq | 80.0 | 77.9 | 79.3 | 78.6 | -| civil_comments | 91.9 | 94.6 | 92.6 | 93.3 | -| dbpedia | 98.9 | 98.9 | 98.9 | 98.9 | -| emotion | 60.4 | 56.9 | 60.1 | 58.5 | -| massive | 91.6 | 89.9 | 92.4 | 93.3 | -| mnli | 81.7 | 83.8 | 80.3 | 83.1 | -| stsb | 56.4 | 57.3 | 56.4 | 54.7 | -| yelp | 71.9 | 69.2 | 69.2 | 69.9 | - -| Calibration metric | Released Jet | V3 | Warm start | V4 transfer | -|---|---:|---:|---:|---:| -| test NLL | 0.8714 | 0.9363 | 0.8771 | 0.8784 | -| test ECE | 0.0622 | 0.0878 | 0.0709 | 0.0441 | -| test_v3_new NLL | 1.3997 | 0.4422 | 0.5071 | 0.5133 | -| test_v3_new ECE | 0.2851 | 0.0211 | 0.0224 | 0.0406 | -| test_v4_new NLL | 1.4212 | 1.2513 | 1.3513 | 0.7809 | -| test_v4_new ECE | 0.2345 | 0.1884 | 0.1748 | 0.0463 | -| test_v4_tools NLL | 1.7029 | 1.7665 | 1.7749 | 0.2137 | -| test_v4_tools ECE | 0.3612 | 0.3770 | 0.4560 | 0.0977 | -| Ordinal test spearman | 0.8323 | 0.8294 | 0.8367 | 0.8328 | -| Ordinal test mae | 0.1433 | 0.1473 | 0.1407 | 0.1432 | - -Both candidates were selected by held-out validation NLL and calibrated on separate data before these test runs. Original three test sets use the same 4,096-token state limit for all models; new V4 tests use 2,048 for all models. Benchmark adapters preserve complete prompts up to 8,192 tokens. - -## Training and provenance - -- Warm start: `adapters/jet-v3-warm-20260924`; 47,313 rows, 2,444 steps, LR 2e-05, best validation step 2444, NLL 0.5131, accuracy 0.7807. Weight SHA256 `93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4`. -- V4 transfer: `adapters/jet-v4-transfer-20260924`; 30,000 rows, 4,033 steps, LR 1e-05, best validation step 3750, NLL 0.5548, accuracy 0.7602. Weight SHA256 `4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad`. - -The published model remains unchanged. V4 retains 18,000 earlier training examples and adds 3,000 examples each for full-contract entailment, stance, executable multi-step code, and native service-intent routing. Sources are pinned; original training labels were verified; related documents/articles/domains stay together. No hosted model APIs were used. - -Full experimental protocol and reproduction details follow. - -# Qwen3-0.6B transfer experiment - -User target: best possible local result while keeping Qwen3-0.6B. No hosted API calls, publication, or submission. This is a bounded follow-up experiment, not a claim that a leaderboard score has been beaten. - -## Protocol fixed before post-training testing - -1. Warm-start released Jet's local rank-16 adapter on immutable `train_v3.jsonl`, one epoch, LR 2e-5, seed 230923, 4096 batch tokens, 1024 state tokens. Fresh AdamW state and schedule. Select by `selection_v3` NLL. This ablation isolates warm-start/lower learning rate from new data. -2. Continue its best validation checkpoint on `train_v4_tools_focus.jsonl`, one epoch, LR 1e-5, seed 230924, 3072 batch tokens, 2048 state tokens. Fresh optimizer. This mixture retains 18,000 V3 rows sampled without replacement, balanced by source, and adds all 9,000 V4 transfer rows plus 3,000 native intent-routing rows. Select by `selection_v4_tools` NLL. Capped inverse-frequency weights apply only to new stance/contract classes; validation is unweighted. -3. Fit temperatures on independent calibration files. Compare both candidates with released Jet and V3 on unchanged old test sets and the 500-request partial diagnostic. Evaluate new template/document/article-held-out tests at 2048 state tokens. No test results choose a checkpoint, epoch, or temperature. - -All runs use base revision `42096995f6402fde107068cf530136fe64b604f8`, all-layer LoRA rank 16, scale 10, dropout .05, gradient checkpointing, BF16 base, cache limit 1 GB. All trained checkpoint directories are new. Initial weights remain eligible if fine-tuning never improves validation; final optimizer/weights are retained separately in `last/`. - -The local released adapter was compared tensor by tensor after CPU BF16 fusion against the unchanged published fused model. Tensor names match, 120 tensors match exactly, 190 differ by at most 0.00048828125 (consistent with fusion rounding). This is a close numerical reconstruction, not bit-identical fused inference. See `jet-next/baseline-adapter-check.json`. - -## New data - -- [VAST](https://github.com/emilyallaway/zero-shot-stance), revision `e7c4775182b184730f350995f8260579c9e066fe`, `data/VAST/vast_train.csv`. Native development/test posts are excluded. Related articles and identical documents stay together. 3,000 training examples; 240 each validation/calibration/test. -- [ContractNLI](https://github.com/stanfordnlp/contract-nli), revision `eced6528dd3c1d14d73f9a87df8f7bdbc03126f9`, archive `resources/contract-nli.zip`, member `contract-nli/train.json`. Full contracts only, fitting 2048 state tokens. Native development/test document text is excluded; one duplicate train/test document was removed. 3,000 training, 240 validation, 238 calibration, 240 test examples. These are sampled from native training documents, not the native evaluation labels. -- Local executable Python: 3,000 training, 240 each validation/calibration/test. Eighteen complete templates: twelve train, two validation, two calibration, two test. Loops, early exits, accumulators, filtering, ordering, slicing and list mutation. Counterfactual inputs supply plausible distractors. Options are shuffled and anonymous keys reassigned; every answer is checked by execution. - -- [Schema-Guided Dialogue](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), revision `e852981ae34990f4358979625854259302feaa78`, native `train/` only. 3,000 training and 240 each validation/calibration/test, split by entire service domain. Native same-service intents plus NONE are the candidate set; complete dialogue prefixes determine the native active intent. All 3,720 annotations verified. Native intent routing is distinct from ToolRet ranking or BFCL execution. ToolRet public evaluation queries are exclusion-only. - -Builder: `scripts/build_v4_transfer.py`; audit: `scripts/audit_v4_transfer.py`; retention mixture: `scripts/prepare_v4_focus.py`. Data hashes, source hashes, source revisions, group identities, class weights and exclusions are in `data/train_v4.provenance.json`. Existing V3, validation and test files are unchanged. V4 uses 56,313 rows before retention subsampling; initial focus mixture uses 27,000; the final tools-augmented focus mixture uses 30,000. `scripts/build_v4_tools.py` and `scripts/audit_v4_tools.py` create/audit this extension, with provenance in `data/train_v4_tools.provenance.json`. - -Limitations: exact normalized matching is not semantic paraphrase detection; contracts longer than 2048 state tokens do not train; code templates share Python primitives; V3 tool tests remain small/easy and do not establish tool-ranking quality. Neither the frozen suite nor all its content is available for exclusion auditing. - -## Scoring target - -The official leaderboard snapshot at Space commit `eec4fe1b0a8f55eb150ae537715ff35cf211f47b` reports 19 panel benchmarks, with five equally weighted categories. The installed pinned reproduction kit's `index-panel.json` is an older proposal and differs from this deployed panel. Consequently, use native benchmark metrics from partial diagnostics, not its category totals or a reconstructed headline score. - -Actual deployed panel: knowledge 24/25/30/43/44/31; language 11/40/41; retrieval 36/37; tools 1/2/6; arts 20/21/22/23/50. Snapshot archived as `jet-next/leaderboard-snapshot.json`. [Official source](https://huggingface.co/spaces/multimodalart/jev-decision-index/tree/eec4fe1b0a8f55eb150ae537715ff35cf211f47b). - -The public frozen-suite dataset still returned repository-not-found at this experiment's start. Partial results cannot establish a leaderboard win. - -## Reproduction - -After the V3 prerequisites and sources are available: - -```sh -.venv/bin/python scripts/build_v4_transfer.py -.venv/bin/python scripts/audit_v4_transfer.py -.venv/bin/python scripts/prepare_v4_focus.py -.venv/bin/python scripts/build_v4_tools.py -.venv/bin/python scripts/audit_v4_tools.py -bash train_cuda.sh --init-adapter adapters/jet \ - --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train data/train_v3.jsonl --val data/selection_v3.jsonl \ - --out adapters/jet-v3-warm-20260924 --epochs 1 --lr 2e-5 \ - --eval-every 250 --val-limit 10000 --seed 230923 -# prepare_v4_focus also creates the 64-row training-only smoke file. -bash scripts/run_next_cuda.sh -``` - -Commands refuse existing dataset/checkpoint outputs. Logs are in `logs/jet-next/`; validation metrics and exact configurations are in each adapter directory. Warm-run source snapshot is preserved separately because weighted-example support was added while that already-loaded process was running. - -## Additional evaluation coverage - -Amazon ESCI was rebuilt in a separate directory with the pinned official normalizer and original test partition: 5,000 selected requests, uncompressed SHA256 `db1d2d3d4dde242dc04262149153bfd596044ad8986e0ae5de8443383973e386`. A deterministic complete-group 500-request sample is reserved for comparison after training. Original eight-benchmark artifacts remain unchanged. This remains a partial public reconstruction, not the frozen suite. `scripts/evaluate_next_extra.sh` compares the same rows across released Jet, V3, warm-start and V4. - -The initial diagnostic contains only 19 English-A sarcasm examples. All 1,400 available English-A rows were therefore selected, solely by track ID, for a separate fixed post-training evaluation. Its metric is positive-class F1 from the official static scorer, whose headline track is `iSarcasmEval-A-En`. No aggregate score is inferred from this single track. Sample provenance is in `jet-next/isarcasm-english-sample.json`. - -Evaluation context is explicit: original test/score_eval/V3 heldout data use 4096 state tokens, matching the earlier baseline inference default; all new V4 heldouts use 2048. Calibration uses 4096. Training context remains 1024 for the warm ablation and 2048 for V4. The distinction between training and inference limits is intentional and recorded. - -After the fixed 55-case CRUXEval diagnostic showed a regression, evaluation was expanded to all 570 available eligible CRUXEval requests after official exclusions, across all four models. This is a targeted regression check, not another training or checkpoint-selection round. No training settings, weights, or calibration were adjusted using that result. `scripts/evaluate_next_code.sh` records these runs separately. - - -Final verification: 26/26 V4 compatibility requests passed; 8 adapter/GPU tests and 3 executable-data tests passed. All V4 source-annotation audits passed. Saved optimizer steps match the selected checkpoints (warm 2444, V4 3750). Original files, V3 datasets, published weights/calibration and preexisting user changes are unchanged. No training/evaluation jobs remain active. diff --git a/docs/training/jet-next/train-v4-snapshot.py b/docs/training/jet-next/train-v4-snapshot.py deleted file mode 100644 index 816c0765fa1a66b174e71ec7b2b95cf360664903..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/train-v4-snapshot.py +++ /dev/null @@ -1,304 +0,0 @@ -"""LoRA fine-tuning of a small LM into a Jet decision model (MLX, Apple Silicon). - -The loss is cross-entropy between the soft target and the model's next-token -distribution restricted to the question's label tokens, read at the last prompt -position only - exactly what inference computes. Score questions add a ranked -probability score term, so mass far from the target level costs more than mass -next to it (cross-entropy alone treats the levels as unordered). -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import random -import time -from functools import partial -from pathlib import Path - -import mlx.core as mx -import mlx.nn as nn -import mlx.optimizers as optim -import numpy as np -from mlx.utils import tree_flatten, tree_unflatten -from mlx_lm import load -from huggingface_hub import snapshot_download -from mlx_lm.tuner.trainer import grad_checkpoint -from mlx_lm.tuner.utils import linear_to_lora_layers - -from format import Question, label_token_ids -from model import DEFAULT_BASE_MODEL, encode, label_logits, pad_batch, pad_labels - - -class Example: - __slots__ = ("tokens", "labels", "target", "ordinal", "weight") - - def __init__(self, tokens: list[int], labels: list[int], target: list[float], ordinal: bool, weight: float = 1.0): - self.tokens, self.labels, self.target, self.ordinal = tokens, labels, target, ordinal - self.weight = weight - - -def load_examples(path: Path, tokenizer, max_state_tokens: int, smoothing: float, use_weights: bool = False) -> list[Example]: - out = [] - for line in path.open(): - ex = json.loads(line) - q = Question.from_dict(ex["question"]) - k = len(q.keys) - target = [(1 - smoothing) * t + smoothing / k for t in ex["target"]] - weight = float(ex.get("weight", 1.0)) if use_weights else 1.0 - if not math.isfinite(weight) or weight <= 0: - raise ValueError("training weights must be finite and positive") - out.append(Example(encode(tokenizer, ex["state"], q, max_state_tokens), label_token_ids(tokenizer, q), target, q.type == "score", weight)) - return out - - -def make_batches(examples: list[Example], max_batch_tokens: int, rng: random.Random | None) -> list[list[Example]]: - """Group similar lengths together; each batch's padded size stays under max_batch_tokens.""" - order = sorted(examples, key=lambda e: len(e.tokens)) - batches, cur = [], [] - for ex in order: - width = max(len(ex.tokens), max((len(e.tokens) for e in cur), default=0)) - if cur and width * (len(cur) + 1) > max_batch_tokens: - batches.append(cur) - cur = [] - cur.append(ex) - if cur: - batches.append(cur) - if rng: - rng.shuffle(batches) - return batches - - -def to_arrays(batch: list[Example]): - tokens, lengths = pad_batch([e.tokens for e in batch]) - ids, mask = pad_labels([e.labels for e in batch]) - target = np.zeros(ids.shape, dtype=np.float32) - for i, e in enumerate(batch): - target[i, : len(e.target)] = e.target - ordinal = mx.array([e.ordinal for e in batch]) - levels = mx.array([len(e.labels) for e in batch], dtype=mx.float32) - return tokens, lengths, ids, mask, mx.array(target), ordinal, levels, mx.array([e.weight for e in batch]) - - -def ranked_probability_score(logp: mx.array, target: mx.array, levels: mx.array) -> mx.array: - """Squared distance between predicted and target CDFs, divided by (levels - 1) so it stays - in 0-1 for any scale size. Padded labels have zero mass in both, so they add nothing.""" - cdf_gap = mx.cumsum(mx.exp(logp), axis=-1) - mx.cumsum(target, axis=-1) - return (cdf_gap**2).sum(axis=-1) / (levels - 1) - - -def loss_fn(model, tokens, lengths, ids, mask, target, ordinal, levels, weights, ordinal_weight=0.0): - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - loss = -(target * logp).sum(axis=-1) - if ordinal_weight: - loss = loss + ordinal_weight * mx.where(ordinal, ranked_probability_score(logp, target, levels), 0.0) - return (loss * weights).mean() - - -def evaluate(model, batches: list[list[Example]]) -> dict[str, float]: - total, nll, correct = 0, 0.0, 0 - for batch in batches: - tokens, lengths, ids, mask, target, *_ = to_arrays(batch) - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - nll += float(-(target * logp).sum()) - correct += int((mx.argmax(logits, axis=-1) == mx.argmax(target, axis=-1)).sum()) - total += len(batch) - return {"nll": nll / total, "acc": correct / total} - - -def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | None = None) -> None: - out.mkdir(parents=True, exist_ok=True) - mx.save_safetensors(str(out / "adapters.safetensors"), dict(tree_flatten(model.trainable_parameters()))) - (out / "adapter_config.json").write_text(json.dumps(config, indent=2)) - if optimizer is not None: - # Adam moments + step, so --resume continues exactly instead of restarting cold. - mx.save_safetensors(str(out / "optimizer.safetensors"), dict(tree_flatten(optimizer.state))) - - -def main() -> None: - ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit") - ap.add_argument("--train", type=Path, default=Path("data/train.jsonl")) - ap.add_argument("--val", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--out", type=Path, default=Path("adapters/jet")) - ap.add_argument("--epochs", type=float, default=2.0) - ap.add_argument("--lr", type=float, default=1e-4) - ap.add_argument("--warmup", type=int, default=100) - ap.add_argument("--rank", type=int, default=16) - ap.add_argument("--lora-scale", type=float, default=10.0) - ap.add_argument("--dropout", type=float, default=0.05) - ap.add_argument("--lora-layers", type=int, default=-1, help="-1 = all layers") - ap.add_argument("--max-batch-tokens", type=int, default=4096) - ap.add_argument("--max-state-tokens", type=int, default=1024) - ap.add_argument("--smoothing", type=float, default=0.02) - ap.add_argument("--use-example-weights", action="store_true", - help="use positive per-row weights for training only; validation remains unweighted") - ap.add_argument("--ordinal-weight", type=float, default=2.0, - help="weight of the ranked probability score term on score questions; 0 = plain cross-entropy") - ap.add_argument("--grad-checkpoint", action=argparse.BooleanOptionalAction, default=True) - ap.add_argument("--eval-every", type=int, default=250) - ap.add_argument("--val-limit", type=int, default=1000) - ap.add_argument("--cache-limit-gb", type=float, default=1.0, - help="cap on MLX's freed-buffer cache; variable batch shapes otherwise grow it toward all of RAM") - ap.add_argument("--resume", type=Path, default=None, help="adapter dir to continue from") - ap.add_argument("--init-adapter", type=Path, default=None, - help="warm-start adapter weights with a fresh optimizer and schedule") - ap.add_argument("--start-step", type=int, default=None, - help="step the resumed adapter was saved at; read from its optimizer state when present") - ap.add_argument("--seed", type=int, default=0) - args = ap.parse_args() - if args.init_adapter and (args.resume or args.start_step is not None): - ap.error("--init-adapter cannot be combined with --resume or --start-step") - if (args.out / "adapters.safetensors").exists(): - ap.error("output already contains an adapter; choose a new directory") - - args.out.mkdir(parents=True, exist_ok=True) - run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()} - run_config["data_sha256"] = { - str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val) - } - if args.init_adapter: - source_config = json.loads((args.init_adapter / "adapter_config.json").read_text()) - expected = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - if source_config.get("lora_parameters") != expected: - ap.error("initial adapter LoRA parameters do not match requested parameters") - if source_config.get("base_model") != args.base_model: - ap.error("initial adapter uses a different base model") - if source_config.get("base_revision") not in (None, args.base_revision): - ap.error("initial adapter uses a different base revision") - run_config["init_adapter_sha256"] = hashlib.sha256( - (args.init_adapter / "adapters.safetensors").read_bytes()).hexdigest() - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - mx.set_cache_limit(int(args.cache_limit_gb * 1e9)) - mx.random.seed(args.seed) - rng = random.Random(args.seed) - model_path = args.base_model - if args.base_revision: - model_path = snapshot_download(args.base_model, revision=args.base_revision) - model, tokenizer = load(model_path) - model.freeze() - num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers - lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - linear_to_lora_layers(model, num_layers, lora_parameters) - if args.init_adapter: - if source_config.get("num_layers") != num_layers: - ap.error("initial adapter layer count does not match") - weights = mx.load(str(args.init_adapter / "adapters.safetensors")) - expected_weights = dict(tree_flatten(model.trainable_parameters())) - if weights.keys() != expected_weights.keys() or any( - weights[k].shape != expected_weights[k].shape for k in expected_weights - ): - ap.error("initial adapter tensor names or shapes do not match") - model.load_weights(list(weights.items()), strict=False) - print(f"initialized weights from {args.init_adapter}; optimizer starts fresh", flush=True) - if args.resume: - model.load_weights(str(args.resume / "adapters.safetensors"), strict=False) - print(f"resumed weights from {args.resume}") - if args.grad_checkpoint: - grad_checkpoint(model.layers[0]) - n_train = sum(v.size for _, v in tree_flatten(model.trainable_parameters())) - print(f"trainable params: {n_train / 1e6:.2f}M") - - train = load_examples(args.train, tokenizer, args.max_state_tokens, args.smoothing, args.use_example_weights) - val = load_examples(args.val, tokenizer, args.max_state_tokens, 0.0) - rng.shuffle(val) - val_batches = make_batches(val[: args.val_limit], args.max_batch_tokens, None) - steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None)) - total_steps = math.ceil(steps_per_epoch * args.epochs) - print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches") - run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps) - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - warmup = max(1, min(args.warmup, total_steps // 10)) - base_schedule = optim.join_schedules( - [optim.linear_schedule(1e-7, args.lr, warmup), optim.cosine_decay(args.lr, total_steps - warmup, args.lr * 0.05)], - [warmup], - ) - optimizer_state = args.resume / "optimizer.safetensors" if args.resume else None - if optimizer_state is not None and optimizer_state.exists(): - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.init(model.trainable_parameters()) - optimizer.state = tree_unflatten(list(mx.load(str(optimizer_state)).items())) - start_step = int(optimizer.state["step"].item()) - print(f"restored optimizer state at step {start_step}") - else: - start_step = args.start_step or 0 - if args.resume: - # Cold Adam moments (MLX AdamW has no bias correction by default) make the first - # updates several times too large; ramp the LR back up to protect the checkpoint. - def schedule(step, start=start_step): - return base_schedule(step) * mx.minimum(1.0, (step - start + 1).astype(mx.float32) / warmup) - print(f"no optimizer state in {args.resume}; re-warming the LR over {warmup} steps") - else: - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.state["step"] = mx.array(start_step, mx.uint64) - value_and_grad = nn.value_and_grad(model, partial(loss_fn, ordinal_weight=args.ordinal_weight)) - - adapter_config = { - "fine_tune_type": "lora", - "num_layers": num_layers, - "lora_parameters": lora_parameters, - "base_model": args.base_model, - "base_revision": args.base_revision, - } - model.eval() - best = evaluate(model, val_batches) - # Keep the initial checkpoint eligible; a regression must not replace it merely - # because no trained checkpoint improved the validation loss. - save_adapter(model, args.out, adapter_config, optimizer) - (args.out / "metrics.jsonl").write_text(json.dumps({"step": 0, **best, - "improved": False, "initial": True}) + "\n") - print(f"step 0 val nll {best['nll']:.4f} acc {best['acc']:.3f}") - model.train() - - saved = False - step, seen_tokens, started = start_step, 0, time.perf_counter() - losses: list[float] = [] - while step < total_steps: - for batch in make_batches(train, args.max_batch_tokens, rng): - if step >= total_steps: - break - arrays = to_arrays(batch) - loss, grads = value_and_grad(model, *arrays) - optimizer.update(model, grads) - mx.eval(model.trainable_parameters(), optimizer.state, loss) - step += 1 - losses.append(loss.item()) - seen_tokens += int(arrays[1].sum()) - if step % 25 == 0: - elapsed = time.perf_counter() - started - print( - f"step {step}/{total_steps} loss {np.mean(losses):.4f} lr {optimizer.learning_rate.item():.2e} " - f"{seen_tokens / elapsed:.0f} tok/s peak mem {mx.get_peak_memory() / 1e9:.1f}GB cache {mx.get_cache_memory() / 1e9:.1f}GB", - flush=True, - ) - losses = [] - if step % args.eval_every == 0 or step == total_steps: - model.eval() - metrics = evaluate(model, val_batches) - model.train() - improved = metrics["nll"] < best["nll"] - print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{' *saved*' if improved else ''}", flush=True) - with (args.out / "metrics.jsonl").open("a") as f: - f.write(json.dumps({"step": step, **metrics, "improved": improved, - "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n") - save_adapter(model, args.out / "last", adapter_config, optimizer) - if improved: - best = metrics - save_adapter(model, args.out, adapter_config, optimizer) - saved = True - if not saved: - print("validation never improved; retaining initial weights (final weights in last/)") - print(f"best val nll {best['nll']:.4f} acc {best['acc']:.3f}; adapter in {args.out}") - - -if __name__ == "__main__": - main() diff --git a/docs/training/jet-next/train-warm-snapshot.py b/docs/training/jet-next/train-warm-snapshot.py deleted file mode 100644 index 9c4c09b64792567806e2b58a78f7f7342f887e75..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/train-warm-snapshot.py +++ /dev/null @@ -1,296 +0,0 @@ -"""LoRA fine-tuning of a small LM into a Jet decision model (MLX, Apple Silicon). - -The loss is cross-entropy between the soft target and the model's next-token -distribution restricted to the question's label tokens, read at the last prompt -position only - exactly what inference computes. Score questions add a ranked -probability score term, so mass far from the target level costs more than mass -next to it (cross-entropy alone treats the levels as unordered). -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import random -import time -from functools import partial -from pathlib import Path - -import mlx.core as mx -import mlx.nn as nn -import mlx.optimizers as optim -import numpy as np -from mlx.utils import tree_flatten, tree_unflatten -from mlx_lm import load -from huggingface_hub import snapshot_download -from mlx_lm.tuner.trainer import grad_checkpoint -from mlx_lm.tuner.utils import linear_to_lora_layers - -from format import Question, label_token_ids -from model import DEFAULT_BASE_MODEL, encode, label_logits, pad_batch, pad_labels - - -class Example: - __slots__ = ("tokens", "labels", "target", "ordinal") - - def __init__(self, tokens: list[int], labels: list[int], target: list[float], ordinal: bool): - self.tokens, self.labels, self.target, self.ordinal = tokens, labels, target, ordinal - - -def load_examples(path: Path, tokenizer, max_state_tokens: int, smoothing: float) -> list[Example]: - out = [] - for line in path.open(): - ex = json.loads(line) - q = Question.from_dict(ex["question"]) - k = len(q.keys) - target = [(1 - smoothing) * t + smoothing / k for t in ex["target"]] - out.append(Example(encode(tokenizer, ex["state"], q, max_state_tokens), label_token_ids(tokenizer, q), target, q.type == "score")) - return out - - -def make_batches(examples: list[Example], max_batch_tokens: int, rng: random.Random | None) -> list[list[Example]]: - """Group similar lengths together; each batch's padded size stays under max_batch_tokens.""" - order = sorted(examples, key=lambda e: len(e.tokens)) - batches, cur = [], [] - for ex in order: - width = max(len(ex.tokens), max((len(e.tokens) for e in cur), default=0)) - if cur and width * (len(cur) + 1) > max_batch_tokens: - batches.append(cur) - cur = [] - cur.append(ex) - if cur: - batches.append(cur) - if rng: - rng.shuffle(batches) - return batches - - -def to_arrays(batch: list[Example]): - tokens, lengths = pad_batch([e.tokens for e in batch]) - ids, mask = pad_labels([e.labels for e in batch]) - target = np.zeros(ids.shape, dtype=np.float32) - for i, e in enumerate(batch): - target[i, : len(e.target)] = e.target - ordinal = mx.array([e.ordinal for e in batch]) - levels = mx.array([len(e.labels) for e in batch], dtype=mx.float32) - return tokens, lengths, ids, mask, mx.array(target), ordinal, levels - - -def ranked_probability_score(logp: mx.array, target: mx.array, levels: mx.array) -> mx.array: - """Squared distance between predicted and target CDFs, divided by (levels - 1) so it stays - in 0-1 for any scale size. Padded labels have zero mass in both, so they add nothing.""" - cdf_gap = mx.cumsum(mx.exp(logp), axis=-1) - mx.cumsum(target, axis=-1) - return (cdf_gap**2).sum(axis=-1) / (levels - 1) - - -def loss_fn(model, tokens, lengths, ids, mask, target, ordinal, levels, ordinal_weight=0.0): - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - loss = -(target * logp).sum(axis=-1) - if ordinal_weight: - loss = loss + ordinal_weight * mx.where(ordinal, ranked_probability_score(logp, target, levels), 0.0) - return loss.mean() - - -def evaluate(model, batches: list[list[Example]]) -> dict[str, float]: - total, nll, correct = 0, 0.0, 0 - for batch in batches: - tokens, lengths, ids, mask, target, *_ = to_arrays(batch) - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - nll += float(-(target * logp).sum()) - correct += int((mx.argmax(logits, axis=-1) == mx.argmax(target, axis=-1)).sum()) - total += len(batch) - return {"nll": nll / total, "acc": correct / total} - - -def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | None = None) -> None: - out.mkdir(parents=True, exist_ok=True) - mx.save_safetensors(str(out / "adapters.safetensors"), dict(tree_flatten(model.trainable_parameters()))) - (out / "adapter_config.json").write_text(json.dumps(config, indent=2)) - if optimizer is not None: - # Adam moments + step, so --resume continues exactly instead of restarting cold. - mx.save_safetensors(str(out / "optimizer.safetensors"), dict(tree_flatten(optimizer.state))) - - -def main() -> None: - ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit") - ap.add_argument("--train", type=Path, default=Path("data/train.jsonl")) - ap.add_argument("--val", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--out", type=Path, default=Path("adapters/jet")) - ap.add_argument("--epochs", type=float, default=2.0) - ap.add_argument("--lr", type=float, default=1e-4) - ap.add_argument("--warmup", type=int, default=100) - ap.add_argument("--rank", type=int, default=16) - ap.add_argument("--lora-scale", type=float, default=10.0) - ap.add_argument("--dropout", type=float, default=0.05) - ap.add_argument("--lora-layers", type=int, default=-1, help="-1 = all layers") - ap.add_argument("--max-batch-tokens", type=int, default=4096) - ap.add_argument("--max-state-tokens", type=int, default=1024) - ap.add_argument("--smoothing", type=float, default=0.02) - ap.add_argument("--ordinal-weight", type=float, default=2.0, - help="weight of the ranked probability score term on score questions; 0 = plain cross-entropy") - ap.add_argument("--grad-checkpoint", action=argparse.BooleanOptionalAction, default=True) - ap.add_argument("--eval-every", type=int, default=250) - ap.add_argument("--val-limit", type=int, default=1000) - ap.add_argument("--cache-limit-gb", type=float, default=1.0, - help="cap on MLX's freed-buffer cache; variable batch shapes otherwise grow it toward all of RAM") - ap.add_argument("--resume", type=Path, default=None, help="adapter dir to continue from") - ap.add_argument("--init-adapter", type=Path, default=None, - help="warm-start adapter weights with a fresh optimizer and schedule") - ap.add_argument("--start-step", type=int, default=None, - help="step the resumed adapter was saved at; read from its optimizer state when present") - ap.add_argument("--seed", type=int, default=0) - args = ap.parse_args() - if args.init_adapter and (args.resume or args.start_step is not None): - ap.error("--init-adapter cannot be combined with --resume or --start-step") - if (args.out / "adapters.safetensors").exists(): - ap.error("output already contains an adapter; choose a new directory") - - args.out.mkdir(parents=True, exist_ok=True) - run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()} - run_config["data_sha256"] = { - str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val) - } - if args.init_adapter: - source_config = json.loads((args.init_adapter / "adapter_config.json").read_text()) - expected = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - if source_config.get("lora_parameters") != expected: - ap.error("initial adapter LoRA parameters do not match requested parameters") - if source_config.get("base_model") != args.base_model: - ap.error("initial adapter uses a different base model") - run_config["init_adapter_sha256"] = hashlib.sha256( - (args.init_adapter / "adapters.safetensors").read_bytes()).hexdigest() - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - mx.set_cache_limit(int(args.cache_limit_gb * 1e9)) - mx.random.seed(args.seed) - rng = random.Random(args.seed) - model_path = args.base_model - if args.base_revision: - model_path = snapshot_download(args.base_model, revision=args.base_revision) - model, tokenizer = load(model_path) - model.freeze() - num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers - lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - linear_to_lora_layers(model, num_layers, lora_parameters) - if args.init_adapter: - if source_config.get("num_layers") != num_layers: - ap.error("initial adapter layer count does not match") - weights = mx.load(str(args.init_adapter / "adapters.safetensors")) - expected_weights = dict(tree_flatten(model.trainable_parameters())) - if weights.keys() != expected_weights.keys() or any( - weights[k].shape != expected_weights[k].shape for k in expected_weights - ): - ap.error("initial adapter tensor names or shapes do not match") - model.load_weights(list(weights.items()), strict=False) - print(f"initialized weights from {args.init_adapter}; optimizer starts fresh", flush=True) - if args.resume: - model.load_weights(str(args.resume / "adapters.safetensors"), strict=False) - print(f"resumed weights from {args.resume}") - if args.grad_checkpoint: - grad_checkpoint(model.layers[0]) - n_train = sum(v.size for _, v in tree_flatten(model.trainable_parameters())) - print(f"trainable params: {n_train / 1e6:.2f}M") - - train = load_examples(args.train, tokenizer, args.max_state_tokens, args.smoothing) - val = load_examples(args.val, tokenizer, args.max_state_tokens, 0.0) - rng.shuffle(val) - val_batches = make_batches(val[: args.val_limit], args.max_batch_tokens, None) - steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None)) - total_steps = math.ceil(steps_per_epoch * args.epochs) - print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches") - run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps) - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - warmup = max(1, min(args.warmup, total_steps // 10)) - base_schedule = optim.join_schedules( - [optim.linear_schedule(1e-7, args.lr, warmup), optim.cosine_decay(args.lr, total_steps - warmup, args.lr * 0.05)], - [warmup], - ) - optimizer_state = args.resume / "optimizer.safetensors" if args.resume else None - if optimizer_state is not None and optimizer_state.exists(): - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.init(model.trainable_parameters()) - optimizer.state = tree_unflatten(list(mx.load(str(optimizer_state)).items())) - start_step = int(optimizer.state["step"].item()) - print(f"restored optimizer state at step {start_step}") - else: - start_step = args.start_step or 0 - if args.resume: - # Cold Adam moments (MLX AdamW has no bias correction by default) make the first - # updates several times too large; ramp the LR back up to protect the checkpoint. - def schedule(step, start=start_step): - return base_schedule(step) * mx.minimum(1.0, (step - start + 1).astype(mx.float32) / warmup) - print(f"no optimizer state in {args.resume}; re-warming the LR over {warmup} steps") - else: - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.state["step"] = mx.array(start_step, mx.uint64) - value_and_grad = nn.value_and_grad(model, partial(loss_fn, ordinal_weight=args.ordinal_weight)) - - adapter_config = { - "fine_tune_type": "lora", - "num_layers": num_layers, - "lora_parameters": lora_parameters, - "base_model": args.base_model, - "base_revision": args.base_revision, - } - model.eval() - best = evaluate(model, val_batches) - # Keep the initial checkpoint eligible; a regression must not replace it merely - # because no trained checkpoint improved the validation loss. - save_adapter(model, args.out, adapter_config, optimizer) - (args.out / "metrics.jsonl").write_text(json.dumps({"step": 0, **best, - "improved": False, "initial": True}) + "\n") - print(f"step 0 val nll {best['nll']:.4f} acc {best['acc']:.3f}") - model.train() - - saved = False - step, seen_tokens, started = start_step, 0, time.perf_counter() - losses: list[float] = [] - while step < total_steps: - for batch in make_batches(train, args.max_batch_tokens, rng): - if step >= total_steps: - break - arrays = to_arrays(batch) - loss, grads = value_and_grad(model, *arrays) - optimizer.update(model, grads) - mx.eval(model.trainable_parameters(), optimizer.state, loss) - step += 1 - losses.append(loss.item()) - seen_tokens += int(arrays[1].sum()) - if step % 25 == 0: - elapsed = time.perf_counter() - started - print( - f"step {step}/{total_steps} loss {np.mean(losses):.4f} lr {optimizer.learning_rate.item():.2e} " - f"{seen_tokens / elapsed:.0f} tok/s peak mem {mx.get_peak_memory() / 1e9:.1f}GB cache {mx.get_cache_memory() / 1e9:.1f}GB", - flush=True, - ) - losses = [] - if step % args.eval_every == 0 or step == total_steps: - model.eval() - metrics = evaluate(model, val_batches) - model.train() - improved = metrics["nll"] < best["nll"] - print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{' *saved*' if improved else ''}", flush=True) - with (args.out / "metrics.jsonl").open("a") as f: - f.write(json.dumps({"step": step, **metrics, "improved": improved, - "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n") - save_adapter(model, args.out / "last", adapter_config, optimizer) - if improved: - best = metrics - save_adapter(model, args.out, adapter_config, optimizer) - saved = True - if not saved: - print("validation never improved; retaining initial weights (final weights in last/)") - print(f"best val nll {best['nll']:.4f} acc {best['acc']:.3f}; adapter in {args.out}") - - -if __name__ == "__main__": - main() diff --git a/docs/training/jet-next/train_v4.provenance.json b/docs/training/jet-next/train_v4.provenance.json deleted file mode 100644 index 4a394db8c8d3e52470214fc998be88a957c89bfb..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/train_v4.provenance.json +++ /dev/null @@ -1,74 +0,0 @@ -{ - "seed": 230924, - "sources": [ - { - "repository": "https://github.com/emilyallaway/zero-shot-stance", - "revision": "e7c4775182b184730f350995f8260579c9e066fe", - "file": "data/VAST/vast_train.csv", - "sha256": "b2f0d69a73231b8ef22f10e4df7d759cf228505ce5252506068da6aa7fe42109", - "split": "train" - }, - { - "repository": "https://github.com/stanfordnlp/contract-nli", - "revision": "eced6528dd3c1d14d73f9a87df8f7bdbc03126f9", - "file": "resources/contract-nli.zip!contract-nli/train.json", - "sha256": "e03fc77bbf8b53e2976a250e81d8a294bc3d5e5fb014521e477dee9340d6287b", - "split": "train" - } - ], - "counts": { - "calibration:code_transfer": 240, - "calibration:contract_entailment": 238, - "calibration:stance_transfer": 240, - "test:code_transfer": 240, - "test:contract_entailment": 240, - "test:stance_transfer": 240, - "train:code_transfer": 3000, - "train:contract_entailment": 3000, - "train:stance_transfer": 3000, - "val:code_transfer": 240, - "val:contract_entailment": 240, - "val:stance_transfer": 240 - }, - "excluded": { - "contract_native_heldout_text": 1, - "duplicate": 3639, - "over_2048_full_state": 3332 - }, - "files": { - "train_v4.jsonl": "f997660ef5a798a4aeeb0ace8c9c76756751d71f87523a31ab9e7a084df8cd5f", - "selection_v4.jsonl": "43c9aef75410fe06efccf9ee8fb71196f2d9de0136785d1684a4d7ed2914a158", - "calibration_v4.jsonl": "e08bd6f26e3150485ad67099ecdb30700474ba3173570beb901136dc8a669a77", - "test_v4_new.jsonl": "36b31bc98595c814f53c745c02e902f263af5b63d6933af8137b6cd7411915f6" - }, - "class_weights": { - "contract_entailment": { - "NotMentioned": 0.724112961622013, - "Entailment": 0.794912559618442, - "Contradiction": 2.770083102493075 - }, - "stance_transfer": { - "against": 0.8465209540742631, - "favor": 0.8471187796068014, - "none": 3.598560575769692 - } - }, - "code_sha256": "37262ee1a8978d19c58a031e11598d504ed2d4b981ee2f004d6c0a8ccff9af9b", - "retained_train_v3_sha256": "ec5666923b48289d5ab3b0887f43e52ec403c838cce7a8eee9479888ccc58be4", - "protected_sha256": { - "data/val.jsonl": "5a73ff6b450e2e1c490afa296fce786aff4af456da81a5ff5ce67846325c04f2", - "data/test.jsonl": "83ef28ce0dd483d6b51d808adac72c84b282da4fea719582f34b83436d3f239e", - "data/score_eval.jsonl": "6a53d16895fe2b3579b1e157b012417d2b8600d6c5605a31cfff6862a6368cf2", - "data/val_v3_new.jsonl": "9c7810c969295bad4bed794025bc1ca505d515a782fd8fefc68d1ef88b39051c", - "data/calibration_v3_new.jsonl": "9e022e0cbc76c9253ad4304bb705118b00ff09d45b4ce0d29d4c7eeb3b8bbb07", - "data/test_v3_new.jsonl": "3f4ff9211aea1762ba5c2dcc330047abbb82c08b0c16c952409229006f027677", - "artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl": "8dd94acf6c313c50aec3fc48188c840d994d14f764b3fff9fe7a89e2e41b9211" - }, - "limitations": [ - "Full frozen suite remains unavailable.", - "ContractNLI restricted to complete contracts fitting 2048 state tokens.", - "Source article/document groups are held together. Code holdouts use disjoint complete templates, but share Python primitives.", - "Exact normalized content exclusion is not a semantic paraphrase detector.", - "V3 tool tests remain small and easy; do not interpret their accuracy as strong tool retrieval evidence." - ] -} \ No newline at end of file diff --git a/docs/training/jet-next/train_v4_focus.provenance.json b/docs/training/jet-next/train_v4_focus.provenance.json deleted file mode 100644 index 811a44a798f8c97098986b9d6f814ce18f657a2f..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/train_v4_focus.provenance.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "seed": 230924, - "retained_v3": 18000, - "new_v4": 9000, - "total": 27000, - "source_sha256": "f997660ef5a798a4aeeb0ace8c9c76756751d71f87523a31ab9e7a084df8cd5f", - "output_sha256": "266817085e61b7ed176e9d02254abf7e55fa4ffcf4d0d2beb35c5c47d839c863", - "note": "Without-replacement source-balanced subsample of training only; all 9000 V4 additions retained. Existing heldouts unchanged." -} \ No newline at end of file diff --git a/docs/training/jet-next/train_v4_tools.provenance.json b/docs/training/jet-next/train_v4_tools.provenance.json deleted file mode 100644 index dbd7d1c3a99f9712cb8c02c768f20319ea255fb8..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/train_v4_tools.provenance.json +++ /dev/null @@ -1,179 +0,0 @@ -{ - "seed": 230924, - "repository": "https://github.com/google-research-datasets/dstc8-schema-guided-dialogue", - "revision": "e852981ae34990f4358979625854259302feaa78", - "native_split": "train", - "source_files": { - "dialogues_001.json": "5ce28a6e656bc88c1da29c983d243d5a81eb37de92698da3864d1cf02894ca24", - "dialogues_002.json": "56a06f9762593b27f5b34225de4ce37da5e83c9c5747e56bca2899f147d88b3e", - "dialogues_003.json": "aae3d8d141c8c36bc5268ca573044b4e82d4534947c169d951a1f229b9852029", - "dialogues_004.json": "4f4aa821e01d7d9efd44c2e89264d37e428045e7058bf3b3c6be0eaec09eddce", - "dialogues_005.json": "e19563efc34dead580435331d39ec3183c79230fa361b2e2d79035af242dd90b", - "dialogues_006.json": "6b4829cb18185f063b81d271b847aa6915ef970cb592a2366edbec56d4b4c746", - "dialogues_007.json": "59adf47264e106df1ce4a59d2c88e95b5a12497e6f5e3e5464381c2a7526837c", - "dialogues_008.json": "36614c125ce310dd162b62d624b8bcd117053cf727978458b6c36b6592f6e757", - "dialogues_009.json": "d8cc086374bbce168a03169341687eaa9f886fe25f8b8dd5bd6832f6f441951d", - "dialogues_010.json": "d8658cd35bd4b6d274720d14308ec721f238d70cc37ea947c38f9d83a0267c1b", - "dialogues_011.json": "66f7eb72a46fa379d85e53ae607e5920a4fa14a4a99765d91ee67f3675fdf4d7", - "dialogues_012.json": "483b09fc9ad3d5b7285d2e1bc44057f1286ea3e73d7ff191fcec5e1acb364412", - "dialogues_013.json": "c6592a97753a8d3be8bae0099b3877e21f293a12d8daf9bcdcbec7289d5f1be3", - "dialogues_014.json": "5cd6cc68c676a2546a3312eda0f29be170ab5f0d6c602c9548ffa2c095037c7b", - "dialogues_015.json": "b698e8db9b0898186453ec9fcec5f80ab043cfe301447d28dc577e99bd3acfd3", - "dialogues_016.json": "5a7d85a3058f38bf40290782721865c3d816f43841e0cd302b21167eefecf449", - "dialogues_017.json": "0eb666f375ba2cc6362c88c02cfd255b702a2b8216e30fce200f630bcca99fa0", - "dialogues_018.json": "ceadf059b7c4ec5e31b7b53ee696198c7a5baa2ff03544b81b4adbe9cb7eca05", - "dialogues_019.json": "96adb06dd0605bb3536bc5e795d045d745a035e9d1c2ed05f7e89937fe883fb9", - "dialogues_020.json": "b2a3cd6d8e677c6a9d342f5703a947525b17b906348cd944a4da1c449ff4f3a4", - "dialogues_021.json": "f13f15b1b62f62d4ae6b71db4f37a1843861d8c1c8907fd0fde19451c08576e9", - "dialogues_022.json": "01a403f462a62b324f94de598c279b5adbaee55ac10c7aed72d798aa5924c510", - "dialogues_023.json": "73bf6f4be084de8fafa347257f39cf7c312bc488178dd2ce667f6e5f8baddb82", - "dialogues_024.json": "fc951b44990efac041bbbd993172b954d3d55af8230a00ce78ba8bbd50daf82a", - "dialogues_025.json": "8305b0d85317d77c5b6ebf128ba4d5e2410e248071a49eb2e05b1ee347c1833d", - "dialogues_026.json": "2b2e2c497c870d39c575892c7aae38b08cd4e39bd3a795be1e8550330e747800", - "dialogues_027.json": "63b5dd62e100afa001b596cda967f5bc86f93dc1072bd00bdc57a198ff691268", - "dialogues_028.json": "225af9eebcb34f90ab1c9634c5771c9e6179019fdb903884ff70a5833b77620c", - "dialogues_029.json": "87b61fd3495d851df7fa66ce43084431a7cd19154c1de32c0d6949bf6c74bae9", - "dialogues_030.json": "82b282eb98a4a26832b5e87d85fdfa8e19b1073a5a903123f7a28d056c4ec2b7", - "dialogues_031.json": "45e31767734dc89d825b8f84c37fdd258250bb1e0b63d8a8e775c99dde07547e", - "dialogues_032.json": "70a0916baba1e2fbeaef8bc8c2086cd3f31555a8eaec6d0eed80e483ed2b9a59", - "dialogues_033.json": "74c23ae93203ed5672015ce13d01c01d80d4f3a3ccd64c8f62f52e0b7c411871", - "dialogues_034.json": "782a499577aba5dc0f11edb6ed803cb8023bd4828a914cb4ff712966604e36cb", - "dialogues_035.json": "d6cf92ad829d275f15a62409ab8d24187a5e120b09d8f545d16d898ed4a36fd7", - "dialogues_036.json": "34223f6b06febfc5b86b0b20324e98b42e9ab4ca7e9c9797c6ba4e49e86e5cce", - "dialogues_037.json": "6e6bd6787f30640305ce0dc1b1dac8fa3d9b53ab0dd28c224519ff727cea916f", - "dialogues_038.json": "936ee784006908d323d558b84e1646dd932a3b4978bc03deafa1da6e2776e6f0", - "dialogues_039.json": "155da94cb95bdbc95e52062459e46b98cce4c8e049b956c5f285ce7a821a08c3", - "dialogues_040.json": "c807c28a3d1dbbe8ecaddd96ed97ac7b67918b12181e07eb89fb046ef6084891", - "dialogues_041.json": "d24455e00ea88b20a94cfd54581b30b224c207b20a197484ef9b27022af47f05", - "dialogues_042.json": "51d94d5d602cbf51c819c8f79a25f45174ebc3191c8ea3b2278d5b6c96742d51", - "dialogues_043.json": "98c9cc7865bc70e67123cc4e64daad8843d5750552bae2559d7bb95aae782c0a", - "dialogues_044.json": "8d3024399540122bef422925f2508ead00bcd85e4f07e16236de87747555fe5f", - "dialogues_045.json": "056df88c7bbff91bcc9b8fe009bfc6f01400aa59feae549cace7d5caa823dcb0", - "dialogues_046.json": "79c92a6fbb92481deb5fe00127359c6029724b49ade4913704a08ce397938eeb", - "dialogues_047.json": "fbe4e810316077148a2d96c8197928486a547946b44b9fce96eafa4f6fa4e0e7", - "dialogues_048.json": "882d7c255da334163b024a6fd511c5eaeedfd507da4db350e4ed2cde393cc787", - "dialogues_049.json": "fa48932032d150414a97d12f9cde4d1ac00bcc593bb30c43a45f2b802f293ed5", - "dialogues_050.json": "40543b657e832ab34b4bb4733ecbf1e7d1f15566674e49191c70261d29490aba", - "dialogues_051.json": "e8eef63422e756f420c60ddfd3bcae3fc5caddf3370c31fa7fa3648a3579bbdc", - "dialogues_052.json": "6736abeb95425a570f6ae5b1dd75967849957cabbdc550a9372a9a7646aba898", - "dialogues_053.json": "2825a26dbed8e7aba44677fd2f387d131991921506a5c8647a75c1c905571491", - "dialogues_054.json": "68e7ed4999fc4f21615e9f61a613309ab0560ed53e5e5837ccb96956ee2c5c2f", - "dialogues_055.json": "c1e4d78a6137b9e3232f0d8a0045f81ac0bbda937cc6187621406f6279885034", - "dialogues_056.json": "b4c4293c6cc1eea766025b0ba658e52070d6ef743afb73b1fb5531d280776977", - "dialogues_057.json": "7586422916e6f7c943b97b304f2724e6eefbe4ee1dd42b9d45413c1fc457cfc2", - "dialogues_058.json": "7ff43d0030fa5f32eb92aa87f3b4086d760d55ebefe0318b17ccf6c11795556e", - "dialogues_059.json": "d5597d0c8e9a904aeeb722e8d8c3a8d15a92eacf1713241fe9c05bd470b625a5", - "dialogues_060.json": "12d510c5644954820f48eb80601d599722672f6759aefdd7774d55d6b509f333", - "dialogues_061.json": "1b1b9c6c8a8113f49e18998512f7d5fea2141be7bec48aac201f306586ac15ce", - "dialogues_062.json": "4c4d76ffc33419e835087d081ff1d469f0b9e65b8d5c372b43d4680c286bbd45", - "dialogues_063.json": "ab015c005d3e1d26bdcd970aee1992d7084453d57f82e2be9795846157eb8fa9", - "dialogues_064.json": "ac2fac08f9f1673be6bd1cf6cec44dbd8d35861f658567313ef8d7b7f8c1908c", - "dialogues_065.json": "a9bd32fb9cce88b4bd0a4a11b136423401ba6e14fb5be306fe1eda9e485e8760", - "dialogues_066.json": "9b6145e988296f133c5414db17798098b97ef01bda44b7b1740ade97c268770e", - "dialogues_067.json": "d16cfeda3585639f9ccbac0b5d1fa5c755d9b87bafa95ac737ddd40d7b1fe7a7", - "dialogues_068.json": "ba2f245f26ba2c13463d34f8a8d1635347974a9f0e423bda53ca4bc15fb82a17", - "dialogues_069.json": "319035b2843a089f352f2e7fbb06544c7d68d86ebbba57ebc3bbf3725c07a9cd", - "dialogues_070.json": "ca37b41fefd5af07b4d9fba75c3268d6831f2ce49f5bf4e6d5412e39c764f0a8", - "dialogues_071.json": "59a78ab61ef8e419c7c2791e028db3704254a196f73e79a890274f09e7e65801", - "dialogues_072.json": "c0e8dd91f70c150ef02118b15152da7f973dcf261112a1604bfa4823d276829c", - "dialogues_073.json": "e2b9aa0843a8cdfabaf24fecae2bf4b280ea8aed75670d6e955d25df58dc8494", - "dialogues_074.json": "0c4fd25b8d66f46ee5a8332410a093473dc57655e1edc2e59ec1d53e2e1a7756", - "dialogues_075.json": "4e1ce624c664a402ee31f69083fc263346aef3ed70d866dc940f4565c1c3e9b8", - "dialogues_076.json": "72567237e51a2282a34d66acbc6beeb23b400f29da404d98de6f796686c720ec", - "dialogues_077.json": "2ec1a3ed1a85498992f4fcfecbd8e9a6b4a72968b8da4cd3502d170edffd9437", - "dialogues_078.json": "288a96936c4426a40e50eb6e220a93cce32177827b416f656c4c78e43aebad16", - "dialogues_079.json": "a2817b6b744c4a03fb4f5cf36100a233ea0af7d96b784b237ea9ab1dd3dd4aad", - "dialogues_080.json": "4b166ba32851d5164271d83ed671135682c344aca1ffa4095c1b38cd63a89600", - "dialogues_081.json": "b34c8b63682242361d30ca5ced6d1a55719f4ed8f151f2a52854b61831e5a45c", - "dialogues_082.json": "11751fd14c9580880d1b3157e10103477cd9a7333a689eda89c6f00a21660da6", - "dialogues_083.json": "c4b2d5f1bbdbe6b53fd96d3fde936c66ecfe35674cae3c79e207c0b1c397616e", - "dialogues_084.json": "f728459cbdbb4f69f87dfbc00af0584b9a44b841a568717e7dab7c3146f492cb", - "dialogues_085.json": "f22be96c2ff0a95c4c883a64f9677e0a0106fac25ea42a523df91700ead7bfa3", - "dialogues_086.json": "c1ebb9b572cd00cc4ec09e7ad775aadec6310e2fe1cc064e0ed18480992a04a4", - "dialogues_087.json": "1b95dda470d3f0809bbbeb86cb99a9a322a81fd1a96e96735d953d5409331810", - "dialogues_088.json": "cbb055520fdd1d7cfffe7b911829a00774cc8190c2d07cb29787cca7f5d51fbf", - "dialogues_089.json": "357d12d7b5fcc9a2414cafc3a9bf6b31449b1f11474d16be5b7747e1999423b4", - "dialogues_090.json": "78310f5e8e4c5a48cb600d36ba762448d13ab15e820aa5f56cc6f25994b6902a", - "dialogues_091.json": "c80f229312047567f61db69223c79e1921662f88f63f9718a1ec51b5204612b0", - "dialogues_092.json": "b9656f9135cefd1fe1bd175f4b45d74c2c2a558137a1f2d509771ccf7e8aff4b", - "dialogues_093.json": "fb6ec8f4c975d7bf7e5a18645ef16614a393dab96e623074410ff5bd9a60d8ca", - "dialogues_094.json": "309f412040022ca48c75c7b980fd65ff3f93e9c486305e2cd29a2bafbb57d7b0", - "dialogues_095.json": "0411f940d9e3f659c6ab1037e14094612d088b9ee98a62043658fe1b76fa700f", - "dialogues_096.json": "8d004b5c6a865428bf25093356f46a0a84b4bd3b749f4172f9d0e606a6d70217", - "dialogues_097.json": "ac32e0ccd38e439655ef8a6fc5436e5d14c5dbcf44faeb1fac48b17e3a7bad34", - "dialogues_098.json": "b08094174ff819f259ea64c199e8d3c5141006b11e00b428be21d5b7e452bc0b", - "dialogues_099.json": "fb7110d90b179fe43eaee15393422ae7fb0d522190a69495023f0cf9cadebcbb", - "dialogues_100.json": "4cdafb648909b34c0f609e61f5c4db5511a880e3d4ba7c726bf4e78e3fe09590", - "dialogues_101.json": "142dc7b5452d6aaafb25fa9637b3f5126ac83194b907016af5e85659220d9cba", - "dialogues_102.json": "15da667bc211a0abb82203184c09e19756cff59feecf1e8578632854254af05e", - "dialogues_103.json": "8c5b4c3df6b7fb43e74b745bca2cfff0b2555be20d13859e6c6c502bb1f27326", - "dialogues_104.json": "4e02e3fc67c89d50365feaa2a1424bf5a8de5edaef02f3191cc35cf46918f8b2", - "dialogues_105.json": "fdf5795a36fe9b59721df5a3d35fe37efdc5d555573ee4f9a939409d737c7376", - "dialogues_106.json": "98b72cb9cea0b320fbc58e80977d48879005159156d4e7c19b7905a6723f3d5f", - "dialogues_107.json": "ae315294f293e65edbdd58f0476c8693605a791872db840788e334f637df9b4c", - "dialogues_108.json": "584a1c096b49fac0b66d8ba08f39ca2cef41d93c98851f1b10e86cf995170b1e", - "dialogues_109.json": "ec28252137a7ceae1a3843bf39d3c9a78a4bd139b13c70165982b0c05c391502", - "dialogues_110.json": "48f2d6256d96385350e1b58e7cc3b499f1be78584790250e89d9d0d11cc03dd0", - "dialogues_111.json": "740087b7e2a5305c4ef16a7227dcc61f9af3cf054690ee682a823070a9e2b2c1", - "dialogues_112.json": "0120d9160a30b1b15c21613122250d2a036609b0ba1e3ed33b5b396c4161e10d", - "dialogues_113.json": "895cbdca09acff80c6a194dc44a293fd6875390ad5899720bc98f004eca7d99b", - "dialogues_114.json": "619616f5a4d46200105dc2ea3ce1b1c131051edf57ca64b495f693cd8eafb4f4", - "dialogues_115.json": "3bb6c375846127053c01d51a0e6509eddfaad47f6d2a2bb3a7cfee65120bec32", - "dialogues_116.json": "c0c409d8a0cf74e37a59d1843680501c245f91f9b28fea79c15a1e66195906ef", - "dialogues_117.json": "b84563e3e581a1da03e48b780de326ad68f3004d225509d6ae89ea84b271a4c4", - "dialogues_118.json": "f8116cfa196a5e43b94c6c3657d8cf4e62c112a2f683d179ade4f13baac588fc", - "dialogues_119.json": "b374904ff28881f08e7212296848c0bf9cc18a45195f4d982123f27aa3bd3a23", - "dialogues_120.json": "d4565f5378c2205129b0c0a1feada548a62220c0a8e2b1caf6d234ffe463eab3", - "dialogues_121.json": "952e6284dbacb2bebca8ba36ac3c942edb50d248b2395706facf6c95f3b84d1c", - "dialogues_122.json": "d4d3b718ac0527099412df7774c873939060d65e82b09ba6a1bec593f5559349", - "dialogues_123.json": "469b268f44aad6fb2bba4d271163d6c774234a80123e66204357b37d0619e2c2", - "dialogues_124.json": "8dd0418beb880f5f021c8466fc92233ef5992e8adcc6380a180f4107dfccf7f1", - "dialogues_125.json": "45d59601652c9b648175125718b55e5d65b3567c09518c9d054f67bdc8b2eaba", - "dialogues_126.json": "ae4c17bab75be5eb1ec8738dedd72bd5d33657243068e657642259c8091269ac", - "dialogues_127.json": "ba42de75a55e27a239b238d94ece3c98e5b64a755e155a027d8a31e5fbf791dc", - "schema.json": "97b6c21673aa7e5c2d1ceef52aa35170c565223cdcbf33bb24c69739bda04a82" - }, - "domain_splits": { - "Services": "val", - "Banks": "val", - "RentalCars": "calibration", - "Buses": "calibration", - "Movies": "test", - "Media": "test", - "Calendar": "train", - "Flights": "train", - "Homes": "train", - "Hotels": "train", - "Events": "train", - "Music": "train", - "Restaurants": "train" - }, - "counts": { - "train": 3000, - "test": 240, - "calibration": 240, - "val": 240 - }, - "exclusions": { - "duplicate_state": 1890 - }, - "toolret_query_revision": "b8c76ad3349ff17497b6bdb28bb5b8f61a0f6445", - "toolret_queries": 7477, - "files": { - "train_v4_tools_focus.jsonl": "6f55f89bbd2d3211dd0cc0afca1398c74f997df0d76db0e15eba3a7390200601", - "test_v4_tools.jsonl": "20f0e11691f8e51760bfef7cc9982ceb8719faeadb5340c14905b393cf58d68d", - "calibration_v4_tools.jsonl": "cbbe6269a2824a6239aaba9c89b64e43aac74788961f4e272fe369a73af0ab97", - "selection_v4_tools.jsonl": "b430db40826a3da1cc9ca52723e1689be95bed5d0cf1cdc70b520101254a862a" - }, - "parents": { - "train_v4_focus.jsonl": "266817085e61b7ed176e9d02254abf7e55fa4ffcf4d0d2beb35c5c47d839c863", - "selection_v4.jsonl": "43c9aef75410fe06efccf9ee8fb71196f2d9de0136785d1684a4d7ed2914a158", - "calibration_v4.jsonl": "e08bd6f26e3150485ad67099ecdb30700474ba3173570beb901136dc8a669a77" - }, - "code_sha256": "35b3694f83c326b3e236cf2528ebba341af0a0be34bb85b365409e6003019626", - "limitations": [ - "Intent routing within one native service; not a ToolRet ranking score or BFCL multi-tool execution task.", - "Six heldout domains are entirely absent from added SGD training data; base-model pretraining coverage is unknown.", - "Exact long-query exclusion does not detect paraphrases." - ] -} \ No newline at end of file diff --git a/docs/training/jet-next/v3-score_eval.json b/docs/training/jet-next/v3-score_eval.json deleted file mode 100644 index d1bf87ebdceeed4f7623d46b3e6804f1628e4faf..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v3-score_eval.json +++ /dev/null @@ -1,65 +0,0 @@ -{ - "overall": { - "n": 2400, - "acc": 0.5016666666666667, - "nll": 1.1098948827101054, - "brier": 0.5388255751601235, - "ece": 0.049056352948149064 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5316666666666666, - "nll": 1.0845926729567896, - "brier": 0.3220883925843873, - "ece": 0.032908969024817145, - "mae": 0.1033457961170546, - "within1": 0.8766666666666667, - "spearman": 0.9018376493733692 - }, - "yelp": { - "n": 600, - "acc": 0.6066666666666667, - "nll": 0.8713869097207984, - "brier": 0.5001777113619436, - "ece": 0.04452881038188934, - "mae": 0.12427177343595153, - "within1": 0.9633333333333334, - "spearman": 0.88508184570763 - }, - "sst5": { - "n": 600, - "acc": 0.415, - "nll": 1.2602221814046304, - "brier": 0.6752547946764169, - "ece": 0.04730295836925506, - "mae": 0.18144155930979952, - "within1": 0.8866666666666667, - "spearman": 0.737770481866935 - }, - "amazon": { - "n": 600, - "acc": 0.4533333333333333, - "nll": 1.2233777667582035, - "brier": 0.6577814020177467, - "ece": 0.09444272140661875, - "mae": 0.18028308809317725, - "within1": 0.9033333333333333, - "spearman": 0.7926277614470995 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.5016666666666667, - "nll": 1.1098948827101054, - "brier": 0.5388255751601235, - "ece": 0.049056352948149064, - "mae": 0.14733555423899575, - "within1": 0.9075, - "spearman": 0.8294005319526306 - } - }, - "ms_per_question": 11.946628592916872, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/v3-test.json b/docs/training/jet-next/v3-test.json deleted file mode 100644 index ee86e3d095ac11e013545c5482438477d7dbcf04..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v3-test.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.6519142003801249, - "nll": 0.9362977022113222, - "brier": 0.4648460641984677, - "ece": 0.087802886825329 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04086569703973105, - "brier": 0.018036308573590402, - "ece": 0.02379309203889635 - }, - "boolq": { - "n": 145, - "acc": 0.7793103448275862, - "nll": 0.5092739206128594, - "brier": 0.31537677875818465, - "ece": 0.08913842850718003 - }, - "emotion": { - "n": 2500, - "acc": 0.5688, - "nll": 1.1567023090774193, - "brier": 0.5870578081436417, - "ece": 0.13159714596271516 - }, - "ag_news": { - "n": 135, - "acc": 0.8962962962962963, - "nll": 0.30371399552871786, - "brier": 0.16402837572263823, - "ece": 0.050474644148791284 - }, - "stsb": { - "n": 117, - "acc": 0.5726495726495726, - "nll": 1.0607522075833626, - "brier": 0.2886478136405242, - "ece": 0.06050514436175681, - "mae": 0.09295228923149568, - "within1": 0.905982905982906, - "spearman": 0.8851879649671098 - }, - "mnli": { - "n": 142, - "acc": 0.8380281690140845, - "nll": 0.46939173813181284, - "brier": 0.2641611997591568, - "ece": 0.06754905090365615 - }, - "yelp": { - "n": 146, - "acc": 0.6917808219178082, - "nll": 0.6570821626403701, - "brier": 0.38738836694199724, - "ece": 0.11636449886511446 - }, - "banking77": { - "n": 140, - "acc": 0.8642857142857143, - "nll": 0.47842126280551645, - "brier": 0.1862706174014704, - "ece": 0.06801908761262894 - }, - "massive": { - "n": 119, - "acc": 0.8991596638655462, - "nll": 0.31506088688605693, - "brier": 0.1227447798011435, - "ece": 0.05545356293686299 - }, - "civil_comments": { - "n": 149, - "acc": 0.9463087248322147, - "nll": 0.31501872988091356, - "brier": 0.04269061276457159, - "ece": 0.053070764413615995 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.5963828848698721, - "nll": 1.1364487257573137, - "brier": 0.5443700049236206, - "ece": 0.09375659254452287 - }, - "noul": { - "n": 1185, - "acc": 0.7670886075949367, - "nll": 0.5571621690939522, - "brier": 0.331423905411652, - "ece": 0.09775889085817938 - }, - "score": { - "n": 231, - "acc": 0.6060606060606061, - "nll": 0.9169610631001566, - "brier": 0.3688481617675371, - "ece": 0.06983209030452746, - "mae": 0.10237847252706574, - "within1": 0.948051948051948, - "spearman": 0.9093660800040722 - } - }, - "ms_per_question": 11.374087093130628, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/v3-test_v3_new.json b/docs/training/jet-next/v3-test_v3_new.json deleted file mode 100644 index 94c0dd265ae9ecdaf15768a2c206e046460238e7..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v3-test_v3_new.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "overall": { - "n": 1364, - "acc": 0.8042521994134897, - "nll": 0.44222923351265353, - "brier": 0.259567315985772, - "ece": 0.021087675303482965 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.625, - "nll": 0.7615659017814324, - "brier": 0.45959500191353253, - "ece": 0.09703730791807175 - }, - "v3:entailment": { - "n": 160, - "acc": 0.8125, - "nll": 0.5015351219626609, - "brier": 0.2794180829289865, - "ece": 0.0648006197065115 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.675, - "nll": 0.618581273034215, - "brier": 0.4265876277825605, - "ece": 0.061761666089296355 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.013145773430844233, - "brier": 0.00039104018188636664, - "ece": 0.012993097305297852 - }, - "v3:code_behavior": { - "n": 160, - "acc": 1.0, - "nll": 0.021853731421288103, - "brier": 0.0033133787000630367, - "ece": 0.02084268778562546 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.79375, - "nll": 0.4674972024862655, - "brier": 0.29483375706962633, - "ece": 0.07397291809320453 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.975, - "nll": 0.08550979029969312, - "brier": 0.03868921824095032, - "ece": 0.03651268482208252 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.50625, - "nll": 1.1571399623062462, - "brier": 0.6228013072055809, - "ece": 0.06916055623441936 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.94375, - "nll": 0.15053857543389312, - "brier": 0.08741509523600818, - "ece": 0.049381273239851003 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.0059849793091416355, - "brier": 8.770325969450155e-05, - "ece": 0.0059629082679748535 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.8042521994134897, - "nll": 0.44222923351265353, - "brier": 0.259567315985772, - "ece": 0.021087675303482965 - } - }, - "ms_per_question": 13.624932868769761, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/v3-test_v4_new.json b/docs/training/jet-next/v3-test_v4_new.json deleted file mode 100644 index d0c2dab124ff9fbf3df55e5a8ffb94b01ebe068d..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v3-test_v4_new.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "overall": { - "n": 720, - "acc": 0.5041666666666667, - "nll": 1.2512630294710916, - "brier": 0.6757878953089775, - "ece": 0.18835306784345046 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.5333333333333333, - "nll": 1.00347384273385, - "brier": 0.6163914967164069, - "ece": 0.15025270320475104 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.525, - "nll": 1.6918137950861516, - "brier": 0.7644781774214823, - "ece": 0.29563305079936986 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.45416666666666666, - "nll": 1.0585014505932728, - "brier": 0.6464940117890435, - "ece": 0.12545051338771981 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.5041666666666667, - "nll": 1.2512630294710916, - "brier": 0.6757878953089775, - "ece": 0.18835306784345046 - } - }, - "ms_per_question": 30.59652488194236, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/v3-test_v4_tools.json b/docs/training/jet-next/v3-test_v4_tools.json deleted file mode 100644 index 6d18bb5bb73f1d090ee5bc8f4742e2bf32b8a08f..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v3-test_v4_tools.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "overall": { - "n": 240, - "acc": 0.31666666666666665, - "nll": 1.7665268682098636, - "brier": 0.8797549399674246, - "ece": 0.3770314007997513 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.31666666666666665, - "nll": 1.7665268682098636, - "brier": 0.8797549399674246, - "ece": 0.3770314007997513 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.31666666666666665, - "nll": 1.7665268682098636, - "brier": 0.8797549399674246, - "ece": 0.3770314007997513 - } - }, - "ms_per_question": 26.11035827082257, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-calibration.json b/docs/training/jet-next/v4-calibration.json deleted file mode 100644 index 4cc81e8c48a7b815a6469c43212f7fa6370ada3e..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-calibration.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "choice": 1.1554, - "score": 1.1554, - "noul": 1.3348 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-data-audit.json b/docs/training/jet-next/v4-data-audit.json deleted file mode 100644 index 34b4232a05c4a9420356c9e5e942c78768071966..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-data-audit.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "status": "passed", - "counts": { - "train:code_transfer": 3000, - "train:contract_entailment": 3000, - "train:stance_transfer": 3000, - "val:stance_transfer": 240, - "val:code_transfer": 240, - "val:contract_entailment": 240, - "calibration:stance_transfer": 240, - "calibration:contract_entailment": 238, - "calibration:code_transfer": 240, - "test:stance_transfer": 240, - "test:code_transfer": 240, - "test:contract_entailment": 240 - }, - "gold_positions": { - "('code_transfer', 1)": 948, - "('contract_entailment', 0)": 1247, - "('code_transfer', 3)": 912, - "('contract_entailment', 2)": 1232, - "('contract_entailment', 1)": 1239, - "('stance_transfer', 0)": 1238, - "('stance_transfer', 2)": 1281, - "('code_transfer', 0)": 935, - "('code_transfer', 2)": 925, - "('stance_transfer', 1)": 1201 - }, - "checks": [ - "No state or component crosses a split", - "All VAST and ContractNLI targets checked against native train annotations", - "Every generated program executed and selected answer verified" - ] -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-metrics.jsonl b/docs/training/jet-next/v4-metrics.jsonl deleted file mode 100644 index f69c3d9d03698026116bd24f91a583e1b79aa95a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-metrics.jsonl +++ /dev/null @@ -1,18 +0,0 @@ -{"step": 0, "nll": 0.7978558958622447, "acc": 0.6756105951152391, "improved": false, "initial": true} -{"step": 250, "nll": 0.6485306280849313, "acc": 0.7227382180942552, "improved": true, "peak_memory_gb": 3.418119256} -{"step": 500, "nll": 0.641251611758804, "acc": 0.7268661850705195, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 750, "nll": 0.6265733636528674, "acc": 0.7340901272789818, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 1000, "nll": 0.6166972800715091, "acc": 0.7364981080151359, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 1250, "nll": 0.5989821272942878, "acc": 0.7471620227038184, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 1500, "nll": 0.5876775383334165, "acc": 0.7536979704162367, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 1750, "nll": 0.581943311845473, "acc": 0.7506019951840385, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 2000, "nll": 0.586809283316566, "acc": 0.7471620227038184, "improved": false, "peak_memory_gb": 3.4478726} -{"step": 2250, "nll": 0.5838194228055185, "acc": 0.7450980392156863, "improved": false, "peak_memory_gb": 3.4478726} -{"step": 2500, "nll": 0.5626733705765363, "acc": 0.7585139318885449, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 2750, "nll": 0.5607419161604654, "acc": 0.7629858961128311, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 3000, "nll": 0.5676812559704539, "acc": 0.7561059511523908, "improved": false, "peak_memory_gb": 3.4478726} -{"step": 3250, "nll": 0.5616303123392762, "acc": 0.7550739594083248, "improved": false, "peak_memory_gb": 3.4478726} -{"step": 3500, "nll": 0.5558042203104697, "acc": 0.7581699346405228, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 3750, "nll": 0.554768383523464, "acc": 0.7602339181286549, "improved": true, "peak_memory_gb": 3.4478726} -{"step": 4000, "nll": 0.5552222224334937, "acc": 0.7571379428964569, "improved": false, "peak_memory_gb": 3.4478726} -{"step": 4033, "nll": 0.5548539117393848, "acc": 0.7585139318885449, "improved": false, "peak_memory_gb": 3.4478726} diff --git a/docs/training/jet-next/v4-score_eval.json b/docs/training/jet-next/v4-score_eval.json deleted file mode 100644 index 3fc9752a0b7e8097412ab4eb223446f89d7080b9..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-score_eval.json +++ /dev/null @@ -1,65 +0,0 @@ -{ - "overall": { - "n": 2400, - "acc": 0.52, - "nll": 1.1051732249609856, - "brier": 0.5357743376369629, - "ece": 0.06524991735816002 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5366666666666666, - "nll": 1.1102153950019593, - "brier": 0.332459629641885, - "ece": 0.06996035655339557, - "mae": 0.10243597520344581, - "within1": 0.89, - "spearman": 0.9011497264456896 - }, - "yelp": { - "n": 600, - "acc": 0.595, - "nll": 0.9026956470062335, - "brier": 0.5201132048387901, - "ece": 0.09163652310768762, - "mae": 0.12338489234049727, - "within1": 0.97, - "spearman": 0.8902715344208428 - }, - "sst5": { - "n": 600, - "acc": 0.475, - "nll": 1.2161626872668663, - "brier": 0.6505152737815312, - "ece": 0.0305570506552855, - "mae": 0.1754747062223032, - "within1": 0.9116666666666666, - "spearman": 0.758046670545616 - }, - "amazon": { - "n": 600, - "acc": 0.47333333333333333, - "nll": 1.1916191705688834, - "brier": 0.6400092422856453, - "ece": 0.10654552747805911, - "mae": 0.17166901100873172, - "within1": 0.8983333333333333, - "spearman": 0.7888142015605857 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.52, - "nll": 1.1051732249609856, - "brier": 0.5357743376369629, - "ece": 0.06524991735816002, - "mae": 0.1432411461937445, - "within1": 0.9175, - "spearman": 0.8328310991663338 - } - }, - "ms_per_question": 11.977573064583945, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-smoke-config.json b/docs/training/jet-next/v4-smoke-config.json deleted file mode 100644 index 37fd233dab48104945cb76d5b651394ce89ee87c..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-smoke-config.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "work/jet-next/v4-smoke.jsonl", - "val": "work/jet-next/v4-smoke.jsonl", - "out": "adapters/jet-v4-transfer-smoke", - "epochs": 1.0, - "lr": 1e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 3072, - "max_state_tokens": 2048, - "smoothing": 0.02, - "use_example_weights": true, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 10, - "val_limit": 64, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet-v3-warm-20260924", - "start_step": null, - "seed": 230924, - "data_sha256": { - "work/jet-next/v4-smoke.jsonl": "6b69acef7d542c101c5854b193f20aa1ed78cefac71f2e38cf03c09244b14b19" - }, - "init_adapter_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "train_examples": 64, - "steps_per_epoch": 35, - "total_steps": 35 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-smoke-metrics.jsonl b/docs/training/jet-next/v4-smoke-metrics.jsonl deleted file mode 100644 index 872d55a4a44affd4d216647260a8809f174f2583..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-smoke-metrics.jsonl +++ /dev/null @@ -1,5 +0,0 @@ -{"step": 0, "nll": 1.313755914568901, "acc": 0.390625, "improved": false, "initial": true} -{"step": 10, "nll": 1.0998961925506592, "acc": 0.453125, "improved": true, "peak_memory_gb": 3.256545968} -{"step": 20, "nll": 1.0723880529403687, "acc": 0.453125, "improved": true, "peak_memory_gb": 3.256545968} -{"step": 30, "nll": 1.0399636775255203, "acc": 0.453125, "improved": true, "peak_memory_gb": 3.256545968} -{"step": 35, "nll": 1.033509224653244, "acc": 0.453125, "improved": true, "peak_memory_gb": 3.256578736} diff --git a/docs/training/jet-next/v4-test.json b/docs/training/jet-next/v4-test.json deleted file mode 100644 index 147688e10562a6c83a302cd4e3b9b46ac58a055c..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-test.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.6644040184632093, - "nll": 0.8783917908240401, - "brier": 0.4363018875628721, - "ece": 0.044080123370055624 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04609997218164305, - "brier": 0.00993532707109211, - "ece": 0.03980768852763706 - }, - "boolq": { - "n": 145, - "acc": 0.7862068965517242, - "nll": 0.4650379801875558, - "brier": 0.2991545819338941, - "ece": 0.056202566623687736 - }, - "emotion": { - "n": 2500, - "acc": 0.5852, - "nll": 1.0812736734218895, - "brier": 0.5494403625547004, - "ece": 0.06851770701408388 - }, - "ag_news": { - "n": 135, - "acc": 0.8888888888888888, - "nll": 0.3087087308918988, - "brier": 0.16917814412184332, - "ece": 0.03841419992623507 - }, - "stsb": { - "n": 117, - "acc": 0.5470085470085471, - "nll": 1.0753559749796122, - "brier": 0.28199056470137557, - "ece": 0.03365990544995692, - "mae": 0.09383068554079495, - "within1": 0.8717948717948718, - "spearman": 0.877984058590352 - }, - "mnli": { - "n": 142, - "acc": 0.8309859154929577, - "nll": 0.4340047932059412, - "brier": 0.24689615776249432, - "ece": 0.07052275685357377 - }, - "yelp": { - "n": 146, - "acc": 0.6986301369863014, - "nll": 0.7119902010420209, - "brier": 0.41549555641030894, - "ece": 0.11680830300670779 - }, - "banking77": { - "n": 140, - "acc": 0.9071428571428571, - "nll": 0.3257405191460358, - "brier": 0.13279354980220526, - "ece": 0.07537455537489482 - }, - "massive": { - "n": 119, - "acc": 0.9327731092436975, - "nll": 0.2788916040219742, - "brier": 0.099814120778048, - "ece": 0.05662691129856755 - }, - "civil_comments": { - "n": 149, - "acc": 0.9328859060402684, - "nll": 0.32543574588733154, - "brier": 0.0470122361134796, - "ece": 0.10539027988510645 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6109395677106307, - "nll": 1.0674977012510223, - "brier": 0.508642577204092, - "ece": 0.054254766210197196 - }, - "noul": { - "n": 1185, - "acc": 0.7805907172995781, - "nll": 0.49928341122037284, - "brier": 0.30730912987911574, - "ece": 0.034595404093778585 - }, - "score": { - "n": 231, - "acc": 0.5930735930735931, - "nll": 0.9673109721763195, - "brier": 0.38807710158281145, - "ece": 0.06311224291334937, - "mae": 0.1062169110937673, - "within1": 0.9177489177489178, - "spearman": 0.899445205722314 - } - }, - "ms_per_question": 11.422907029324417, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-test_v3_new.json b/docs/training/jet-next/v4-test_v3_new.json deleted file mode 100644 index 32416a3322e1bc3bb5062f713064274fb48f84fd..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-test_v3_new.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "overall": { - "n": 1364, - "acc": 0.7719941348973607, - "nll": 0.5133160767353705, - "brier": 0.299812832129049, - "ece": 0.04058140996963746 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.625, - "nll": 0.7707278399262577, - "brier": 0.45656743231427815, - "ece": 0.10337576027959586 - }, - "v3:entailment": { - "n": 160, - "acc": 0.80625, - "nll": 0.5095306556206196, - "brier": 0.27964219746474067, - "ece": 0.046038990654051294 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.7, - "nll": 0.5887704980559647, - "brier": 0.4007791939368809, - "ece": 0.05541613362729548 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.030981391526253548, - "brier": 0.001263318868803046, - "ece": 0.03043532371520996 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.875, - "nll": 0.24078694187337532, - "brier": 0.13652739831088206, - "ece": 0.06979165971279144 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.84375, - "nll": 0.4296516477130353, - "brier": 0.25673140257791316, - "ece": 0.07205855362117289 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.8125, - "nll": 0.4019184489734471, - "brier": 0.24345378337085438, - "ece": 0.07234472986310721 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.5375, - "nll": 1.109215586911887, - "brier": 0.5870259634345372, - "ece": 0.08422903288155792 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.85625, - "nll": 0.31212113342480735, - "brier": 0.19465629563358502, - "ece": 0.07933795303106309 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.011983446683734656, - "brier": 0.0003512193510441135, - "ece": 0.011894524097442627 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.7719941348973607, - "nll": 0.5133160767353705, - "brier": 0.299812832129049, - "ece": 0.04058140996963746 - } - }, - "ms_per_question": 13.481722373900375, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-test_v4_new.json b/docs/training/jet-next/v4-test_v4_new.json deleted file mode 100644 index 2a627a91693fff4f1d525edf461b24e8baf4b989..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-test_v4_new.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "overall": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6, - "nll": 0.8641331787531574, - "brier": 0.5189303984919411, - "ece": 0.08732014621297518 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.5708333333333333, - "nll": 1.0104112304126223, - "brier": 0.553216138337243, - "ece": 0.1417993993808826 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.8083333333333333, - "nll": 0.468007056050313, - "brier": 0.2578825642308249, - "ece": 0.0798518454035123 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - } - }, - "ms_per_question": 30.677324276388795, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-test_v4_tools.json b/docs/training/jet-next/v4-test_v4_tools.json deleted file mode 100644 index a9de5c83f013ec3cd34053b544fbb2ddd9bac644..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-test_v4_tools.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "overall": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "ms_per_question": 25.963092025009853, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-tool-audit.json b/docs/training/jet-next/v4-tool-audit.json deleted file mode 100644 index 59b2d426c10d2774261b7a8d42667bfc4bb20789..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-tool-audit.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "status": "passed", - "native_annotations_checked": 3720, - "counts": { - "train": 3000, - "val": 240, - "calibration": 240, - "test": 240 - }, - "option_counts": { - "3": 2377, - "5": 504, - "4": 839 - }, - "parent_training_overlap": 0, - "checks": [ - "Whole domains and dialogues held together", - "Native intent targets and complete conversation prefixes match source", - "Held-out conversation utterances absent from parent training content", - "Options are the native same-service catalog plus NONE" - ] -} \ No newline at end of file diff --git a/docs/training/jet-next/v4-training-config.json b/docs/training/jet-next/v4-training-config.json deleted file mode 100644 index 63e6d0cb952d689e97b2b4e5c8d54be7ad66217b..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/v4-training-config.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v4_tools_focus.jsonl", - "val": "data/selection_v4_tools.jsonl", - "out": "adapters/jet-v4-transfer-20260924", - "epochs": 1.0, - "lr": 1e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 3072, - "max_state_tokens": 2048, - "smoothing": 0.02, - "use_example_weights": true, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet-v3-warm-20260924", - "start_step": null, - "seed": 230924, - "data_sha256": { - "data/train_v4_tools_focus.jsonl": "6f55f89bbd2d3211dd0cc0afca1398c74f997df0d76db0e15eba3a7390200601", - "data/selection_v4_tools.jsonl": "b430db40826a3da1cc9ca52723e1689be95bed5d0cf1cdc70b520101254a862a" - }, - "init_adapter_sha256": "93fd8eeade6eff0d0a8aa06119f71223dc1bee36c49f1a9298d2c37816b609d4", - "train_examples": 30000, - "steps_per_epoch": 4033, - "total_steps": 4033 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-calibration.json b/docs/training/jet-next/warm-calibration.json deleted file mode 100644 index 53ff6eb63e74c19c69b3d9547120349efa73bed0..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-calibration.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "choice": 0.9439, - "score": 1.0293, - "noul": 1.2599 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-metrics.jsonl b/docs/training/jet-next/warm-metrics.jsonl deleted file mode 100644 index 505c5fb13205d5378d352168925dda0e22c89e49..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-metrics.jsonl +++ /dev/null @@ -1,11 +0,0 @@ -{"step": 0, "nll": 1.1268778682428564, "acc": 0.5731895223420647, "improved": false, "initial": true} -{"step": 250, "nll": 0.7733177955051416, "acc": 0.6255778120184899, "improved": true, "peak_memory_gb": 3.884384936} -{"step": 500, "nll": 0.6916508070921739, "acc": 0.7021058038007191, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 750, "nll": 0.6277072637829342, "acc": 0.7303543913713405, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 1000, "nll": 0.5759907600263724, "acc": 0.743708269131998, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 1250, "nll": 0.5443521160680206, "acc": 0.7596302003081664, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 1500, "nll": 0.5442286748793043, "acc": 0.7616846430405753, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 1750, "nll": 0.5310963410013568, "acc": 0.7760657421674371, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 2000, "nll": 0.5192416464908954, "acc": 0.7786337955829481, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 2250, "nll": 0.5145194000259815, "acc": 0.7791474062660503, "improved": true, "peak_memory_gb": 4.074936008} -{"step": 2444, "nll": 0.5131154033423817, "acc": 0.7806882383153569, "improved": true, "peak_memory_gb": 4.074936008} diff --git a/docs/training/jet-next/warm-score_eval.json b/docs/training/jet-next/warm-score_eval.json deleted file mode 100644 index 68a27895829e78423eb07e047963e214d37e1c9b..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-score_eval.json +++ /dev/null @@ -1,65 +0,0 @@ -{ - "overall": { - "n": 2400, - "acc": 0.5304166666666666, - "nll": 1.092109727657017, - "brier": 0.5292645190337113, - "ece": 0.05937186859548092 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.5566666666666666, - "nll": 1.0754084892512186, - "brier": 0.3166843313684106, - "ece": 0.04638583054145179, - "mae": 0.09808268856668535, - "within1": 0.9, - "spearman": 0.9082776913459185 - }, - "yelp": { - "n": 600, - "acc": 0.6083333333333333, - "nll": 0.8788605593300114, - "brier": 0.5081122332933897, - "ece": 0.08311119998494783, - "mae": 0.12012921416409275, - "within1": 0.975, - "spearman": 0.8964504716876638 - }, - "sst5": { - "n": 600, - "acc": 0.45166666666666666, - "nll": 1.2218405024707317, - "brier": 0.655538649626939, - "ece": 0.04943077882130942, - "mae": 0.17551595646558174, - "within1": 0.9183333333333333, - "spearman": 0.7543010013520536 - }, - "amazon": { - "n": 600, - "acc": 0.505, - "nll": 1.192329359576106, - "brier": 0.6367228618461059, - "ece": 0.09014196003476778, - "mae": 0.16920178736618255, - "within1": 0.8966666666666666, - "spearman": 0.7976749967006868 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.5304166666666666, - "nll": 1.092109727657017, - "brier": 0.5292645190337113, - "ece": 0.05937186859548092, - "mae": 0.1407324116406356, - "within1": 0.9225, - "spearman": 0.8367095771732885 - } - }, - "ms_per_question": 12.109397532082463, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-test.json b/docs/training/jet-next/warm-test.json deleted file mode 100644 index 8dd1a53d40d33732ef92e6943825643811fc2c65..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-test.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.6749932120553896, - "nll": 0.8770703999656424, - "brier": 0.43323420329822226, - "ece": 0.07087006142548558 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.02723309464054182, - "brier": 0.012118271812111808, - "ece": 0.019888099696901113 - }, - "boolq": { - "n": 145, - "acc": 0.7931034482758621, - "nll": 0.45604896231971936, - "brier": 0.29052021521305826, - "ece": 0.061517162158571434 - }, - "emotion": { - "n": 2500, - "acc": 0.6012, - "nll": 1.0810718602698297, - "brier": 0.5453604910102474, - "ece": 0.10244272117614747 - }, - "ag_news": { - "n": 135, - "acc": 0.8888888888888888, - "nll": 0.299911898188293, - "brier": 0.16838452547589777, - "ece": 0.04407136572731866 - }, - "stsb": { - "n": 117, - "acc": 0.5641025641025641, - "nll": 1.0637444151019242, - "brier": 0.2787832582420156, - "ece": 0.07576155611592482, - "mae": 0.0916897660563103, - "within1": 0.8974358974358975, - "spearman": 0.8805116134565336 - }, - "mnli": { - "n": 142, - "acc": 0.8028169014084507, - "nll": 0.47346722350364, - "brier": 0.2732897038041701, - "ece": 0.04862175105323253 - }, - "yelp": { - "n": 146, - "acc": 0.6917808219178082, - "nll": 0.6881789257175812, - "brier": 0.39966799591362356, - "ece": 0.06584085483257086 - }, - "banking77": { - "n": 140, - "acc": 0.9285714285714286, - "nll": 0.3419254706257821, - "brier": 0.13463227703766398, - "ece": 0.03366895892790385 - }, - "massive": { - "n": 119, - "acc": 0.9243697478991597, - "nll": 0.2586406250181021, - "brier": 0.09246301809645702, - "ece": 0.048050142511600245 - }, - "civil_comments": { - "n": 149, - "acc": 0.9261744966442953, - "nll": 0.32008108970125426, - "brier": 0.04445927595205485, - "ece": 0.10419578600249835 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6259373621526246, - "nll": 1.0736314924175012, - "brier": 0.5102127508525695, - "ece": 0.09648422277678359 - }, - "noul": { - "n": 1185, - "acc": 0.7848101265822784, - "nll": 0.48621275175856077, - "brier": 0.29606669919747736, - "ece": 0.02945000119350126 - }, - "score": { - "n": 231, - "acc": 0.5930735930735931, - "nll": 0.95309774428178, - "brier": 0.38142954985093946, - "ece": 0.03230119448203545, - "mae": 0.10350282123406596, - "within1": 0.9393939393939394, - "spearman": 0.9028200815430442 - } - }, - "ms_per_question": 11.407006713820156, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-test_v3_new.json b/docs/training/jet-next/warm-test_v3_new.json deleted file mode 100644 index a1d7af183a2be4d1331a868d6567ede5d430a1a6..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-test_v3_new.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "overall": { - "n": 1364, - "acc": 0.7749266862170088, - "nll": 0.5070664367182426, - "brier": 0.30008040549776915, - "ece": 0.02237014299875829 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.63125, - "nll": 0.7567455511540174, - "brier": 0.4507593102169916, - "ece": 0.06857692748308183 - }, - "v3:entailment": { - "n": 160, - "acc": 0.79375, - "nll": 0.5101304793381132, - "brier": 0.28997449331401376, - "ece": 0.03158894721418622 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.7, - "nll": 0.5968082959763705, - "brier": 0.4076295926973904, - "ece": 0.057135591283440576 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.011557995653474483, - "brier": 0.00022427398406391338, - "ece": 0.011465013027191162 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.9, - "nll": 0.2162705672410084, - "brier": 0.13338298348556035, - "ece": 0.040077064931392674 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.8375, - "nll": 0.4227919111261144, - "brier": 0.2553973171422433, - "ece": 0.055019789934158306 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.84375, - "nll": 0.4120700732804835, - "brier": 0.24532611174360155, - "ece": 0.06690083071589471 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.525, - "nll": 1.1109857523348183, - "brier": 0.591940409718284, - "ece": 0.052010057494044304 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.85, - "nll": 0.2916829534515273, - "brier": 0.18366851220511013, - "ece": 0.04999278411269187 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.0063601806294173, - "brier": 0.00015376200744249418, - "ece": 0.0063211917877197266 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.7749266862170088, - "nll": 0.5070664367182426, - "brier": 0.30008040549776915, - "ece": 0.02237014299875829 - } - }, - "ms_per_question": 13.352808236071233, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-test_v4_new.json b/docs/training/jet-next/warm-test_v4_new.json deleted file mode 100644 index 28a9a67a13ed3565a0f9ec9b42f6336b82687d6a..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-test_v4_new.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "overall": { - "n": 720, - "acc": 0.48333333333333334, - "nll": 1.3513476976581538, - "brier": 0.7107875032531545, - "ece": 0.1748247489333153 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6, - "nll": 0.9012368252500892, - "brier": 0.537829477895586, - "ece": 0.058517271776994055 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.38333333333333336, - "nll": 2.0515175121137874, - "brier": 0.9352148454889156, - "ece": 0.3826701547950506 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.4666666666666667, - "nll": 1.1012887556105853, - "brier": 0.6593181863749621, - "ece": 0.11634314979116123 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.48333333333333334, - "nll": 1.3513476976581538, - "brier": 0.7107875032531545, - "ece": 0.1748247489333153 - } - }, - "ms_per_question": 30.290553562498545, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-test_v4_tools.json b/docs/training/jet-next/warm-test_v4_tools.json deleted file mode 100644 index 7be32ed3167a2631aa249941e70675c958f1db35..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-test_v4_tools.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "overall": { - "n": 240, - "acc": 0.2875, - "nll": 1.7748789598233998, - "brier": 0.9634518656932426, - "ece": 0.4559627714256445 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.2875, - "nll": 1.7748789598233998, - "brier": 0.9634518656932426, - "ece": 0.4559627714256445 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.2875, - "nll": 1.7748789598233998, - "brier": 0.9634518656932426, - "ece": 0.4559627714256445 - } - }, - "ms_per_question": 25.927134495835464, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-next/warm-training-config.json b/docs/training/jet-next/warm-training-config.json deleted file mode 100644 index e8fa0fe7d0c328c1810e49d84c057c0432693c0f..0000000000000000000000000000000000000000 --- a/docs/training/jet-next/warm-training-config.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v3.jsonl", - "val": "data/selection_v3.jsonl", - "out": "adapters/jet-v3-warm-20260924", - "epochs": 1.0, - "lr": 2e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 4096, - "max_state_tokens": 1024, - "smoothing": 0.02, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet", - "start_step": null, - "seed": 230923, - "data_sha256": { - "data/train_v3.jsonl": "ec5666923b48289d5ab3b0887f43e52ec403c838cce7a8eee9479888ccc58be4", - "data/selection_v3.jsonl": "f38f2fd8e503d749ea3a72a811348f663c30b9b5702f171c66f344fecc1dbee4" - }, - "init_adapter_sha256": "e70670a173d6d95c329acb72f692577eb47171c106443aa0171fb9584399fbce", - "train_examples": 47313, - "steps_per_epoch": 2444, - "total_steps": 2444 -} \ No newline at end of file diff --git a/docs/training/jet-v3.md b/docs/training/jet-v3.md deleted file mode 100644 index db7e1e10eb33685f7a343dd77d507a7262b22dc9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3.md +++ /dev/null @@ -1,120 +0,0 @@ -# Jet v3: Decision Index data experiment - -Local Arch / RTX 4080 SUPER experiment, 2026-09-23. Published Jet is unchanged. -This uses Qwen3-0.6B, not a larger backbone, and starts fresh LoRA weights from the -same frozen base. No hosted API calls, weight publication, or submission. - -## Reproduction - -```bash -uv sync --extra cuda --extra benchmark --inexact -uv run --no-sync python scripts/rebuild_index_diagnostic.py -uv run --no-sync python scripts/download_v3_sources.py -uv run --no-sync python -m data.decision_training -uv run --no-sync python scripts/audit_v3_data.py -uv run --no-sync python scripts/audit_v3_sources.py -bash scripts/run_v3_cuda.sh -``` - -The run script refuses to overwrite its completed adapter. The initial CUDA -smoke used 64 rows / 9 steps with 2,048 batch tokens. A second smoke used 128 -mixed rows, including long product descriptions, at 4,096 batch tokens. -Neither smoke adapter initializes the full experiment. - -## Data and isolation - -`data/train_v3.jsonl` retains `train_v2` except exact held-out-state collisions -and duplicates, then adds approximately 2,000 examples per family. The original -train, validation, test and ordinal evaluation files are unchanged. Inherited -`train_v2` has no original row IDs, so its provenance is the file SHA256; new -rows carry pinned source revision, training filename, original ID (or original -row offset where the source has no ID), and a connected-component split ID. - -Public training inputs: - -- Product relevance: [Amazon ESCI](https://github.com/amazon-science/esci-data), - via the pinned first training shard of `tasksource/esci`. English only, - original example/query/product IDs and four native relevance labels. - This is a bounded shard sample, not a representative sample of all queries. -- Document relevance: [GLUE QNLI](https://huggingface.co/datasets/nyu-mll/glue), - training partition only, answer-containing sentence classification. -- Entailment: [SNLI](https://huggingface.co/datasets/stanfordnlp/snli), training - partition, with missing labels rejected; shared premises stay grouped. -- Stance and irony: [TweetEval](https://github.com/cardiffnlp/tweeteval), five - stance training partitions and the irony training partition. Irony is reported - as irony, not relabelled as author-intended sarcasm. -- Sarcasm: [iSarcasmEval](https://github.com/iabufarha/iSarcasmEval), English - **training** CSV only. Author rephrases stay with their original tweets. - All available benchmark evaluation text is excluded before splitting. -- Tool selection and relevance: [Glaive function calling v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2), - public synthetic training data. Use the first user request and its immediate - function-call annotation, or an explicit single-tool refusal. No assistant - answer or function response enters the model input. Selection adds catalog - distractors; these are not exhaustively checked for equivalent capabilities. - Shared native catalogs and repeated requests stay grouped. No source tools - are executed. -- Boolean rules, arithmetic and Python code behavior: deterministic local - generators, with labels evaluated from the generated expressions/programs. - Only our own bounded generated code is executed. No CRUXEval/GSM8K or other - evaluation examples generate training data. Programs with different inputs - stay grouped; arithmetic hold-outs assess new numeric instances, not unseen - operator families. - -Source revisions and downloaded-file hashes are in -`data/train_v3.provenance.json`. Licensing follows each source: ESCI Apache-2.0, -SNLI CC-BY-SA-4.0, Glaive Apache-2.0; GLUE, TweetEval and iSarcasm retain their -upstream/component terms. This local experiment does not redistribute weights -or the collected datasets. The gated xLAM candidate was not used. - -Before sampling, deduplication ignores option order. Connected groups include -source IDs, states, query/product relations, repeated tool requests/catalogs, -SNLI premises, tweets/rephrases, and generated programs/rules. Entire groups -are assigned deterministically to training, checkpoint selection, calibration, -or final evaluation. Family sampling is without replacement and round-robins -native classes. Choice order is randomized; generated numeric/output keys are -reassigned after shuffling so `option_0` cannot leak the answer. - -New rows whose full state exceeds 1,024 tokens or complete prompt exceeds 2,048 -are excluded. Existing train_v2 retains the prior 1,024-state-token training -policy. `selection_v3.jsonl` combines one state-grouped half of existing val -with new-family selection rows. `calibration_v3.jsonl` combines the other half -with separate new-family calibration rows. `test_v3_new.jsonl` is untouched by -checkpoint selection and temperature fitting. - -The frozen Decision Index corpus remains unavailable. The rebuilt partial -corpus has 23,113 requests and SHA256 -`8dd94acf6c313c50aec3fc48188c840d994d14f764b3fff9fe7a89e2e41b9211`. -It is used only for exclusion and diagnostic evaluation. A separate audit verifies all -2,000 selected product IDs, queries and labels against the original Amazon -training partition, and finds zero normalized-query overlap with all 35 pinned -ToolRet query configurations. The published -normalizer reads ESCI's official test split; our ESCI input is train. This is -not a claim that unavailable evaluation content can be exhaustively audited. - -## Training and evaluation protocol - -- Base: `mlx-community/Qwen3-0.6B-bf16`, pinned revision - `42096995f6402fde107068cf530136fe64b604f8`. -- Two epochs; rank 16 on all 28 layers; LR 1e-4; LoRA scale 10; dropout .05; - gradient checkpointing; 4,096 batch tokens; 1,024 state tokens; seed 230923. -- Preserve existing smoothing .02 and ordinal-loss weight 2.0. AdamW weight - decay .01, warmup 100 steps followed by cosine decay to 5% of peak LR. -- Validate every 250 steps on all checkpoint-selection rows. Root adapter is - best validation NLL; `last/` stores latest weights and optimizer state. -- Full resolved arguments, dataset hashes, validation metrics, optimizer - checkpoints and per-type calibration are saved with the new adapter. -- Temperature fitting uses calibration only, batch size 4. Compare original - test, ordinal evaluation, and separate new-family test against the released - fused baseline at `8a97cfea2df622bb03f5dc9b02567e21abd2551c`. -- Run the same 500-request complete-group partial diagnostic on both models, - preserving full prompts at an 8,192-token limit. Official native metrics - apply only to that sample. No overall leaderboard score is reported. - -All command logs are under `logs/jet-v3/`; machine-readable results are under -`docs/training/jet-v3/`. The completed scorecard is in `docs/training/jet-v3/summary.md`. All 4,888 steps -completed; the new adapter is calibrated. New-family accuracy rose from 45.9% -to 80.6%, while original test accuracy fell from 67.7% to 65.2% and ordinal -accuracy fell from 52.1% to 50.5%. Keep the released model as the default. Both -500-request partial diagnostics finished without errors; they are not leaderboard -scores. Completion checks confirmed original user edits, published model weights, -and evaluation files remain unchanged. diff --git a/docs/training/jet-v3/adapter_config.json b/docs/training/jet-v3/adapter_config.json deleted file mode 100644 index fbe650c61eb1aac8612ebf5284ae55f1bbf8715e..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/adapter_config.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "fine_tune_type": "lora", - "num_layers": 28, - "lora_parameters": { - "rank": 16, - "scale": 10.0, - "dropout": 0.05 - }, - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8" -} \ No newline at end of file diff --git a/docs/training/jet-v3/baseline-score_eval.json b/docs/training/jet-v3/baseline-score_eval.json deleted file mode 100644 index 4a8c9f69a16a8c32c6bcd8fa17fc932329849a45..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/baseline-score_eval.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "overall": { - "n": 2400, - "acc": 0.52125, - "nll": 1.0903444933540307, - "brier": 0.5290462652704264, - "ece": 0.03454353142529726 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.525, - "nll": 1.1053056762888263, - "brier": 0.33305591800356504, - "ece": 0.042865492701530436, - "mae": 0.102795440373399, - "within1": 0.8833333333333333, - "spearman": 0.9019142986107999 - }, - "yelp": { - "n": 600, - "acc": 0.6033333333333334, - "nll": 0.8807060158190628, - "brier": 0.5043648978801348, - "ece": 0.04826747432351114, - "mae": 0.12410087461073999, - "within1": 0.97, - "spearman": 0.891761260001728 - }, - "sst5": { - "n": 600, - "acc": 0.4866666666666667, - "nll": 1.1988302002350488, - "brier": 0.6439570196420256, - "ece": 0.036009422938028994, - "mae": 0.17386579125653953, - "within1": 0.9083333333333333, - "spearman": 0.7423420876712294 - }, - "amazon": { - "n": 600, - "acc": 0.47, - "nll": 1.1765360810731849, - "brier": 0.6348072255559799, - "ece": 0.09157247965534528, - "mae": 0.1727618505602004, - "within1": 0.9016666666666666, - "spearman": 0.7919591933802752 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.52125, - "nll": 1.0903444933540307, - "brier": 0.5290462652704264, - "ece": 0.03454353142529726, - "mae": 0.14338098920021972, - "within1": 0.9158333333333334, - "spearman": 0.8319325374979925 - } - }, - "ms_per_question": 6.545865319583299 -} \ No newline at end of file diff --git a/docs/training/jet-v3/baseline-test.json b/docs/training/jet-v3/baseline-test.json deleted file mode 100644 index 2a5892cc58575e7efdcec84143ed21dd326e1ae6..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/baseline-test.json +++ /dev/null @@ -1,111 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.676622318761879, - "nll": 0.8710569258689598, - "brier": 0.4271373951303378, - "ece": 0.0614988670833338 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04856743158953678, - "brier": 0.02051652638874066, - "ece": 0.0028781566354963246 - }, - "boolq": { - "n": 145, - "acc": 0.8137931034482758, - "nll": 0.4547928252996042, - "brier": 0.2877616179971535, - "ece": 0.033534014636072584 - }, - "emotion": { - "n": 2500, - "acc": 0.6032, - "nll": 1.061520102331415, - "brier": 0.5323164863764696, - "ece": 0.0945529906153679 - }, - "ag_news": { - "n": 135, - "acc": 0.8814814814814815, - "nll": 0.30449030542815175, - "brier": 0.17246954337666645, - "ece": 0.04308961409109611 - }, - "stsb": { - "n": 117, - "acc": 0.5641025641025641, - "nll": 1.1319686002356557, - "brier": 0.313126957621089, - "ece": 0.06744724868709204, - "mae": 0.09893415102229701, - "within1": 0.8803418803418803, - "spearman": 0.8583073814644547 - }, - "mnli": { - "n": 142, - "acc": 0.823943661971831, - "nll": 0.48070815917183185, - "brier": 0.2682452474690198, - "ece": 0.08867672715388554 - }, - "yelp": { - "n": 146, - "acc": 0.7191780821917808, - "nll": 0.6878386074421953, - "brier": 0.39726907064692013, - "ece": 0.0848420409307088 - }, - "banking77": { - "n": 140, - "acc": 0.8857142857142857, - "nll": 0.3826026692570719, - "brier": 0.15188631481491133, - "ece": 0.048383180584226367 - }, - "massive": { - "n": 119, - "acc": 0.9159663865546218, - "nll": 0.3343984374229624, - "brier": 0.11869874499716762, - "ece": 0.027953259834722355 - }, - "civil_comments": { - "n": 149, - "acc": 0.9194630872483222, - "nll": 0.32481257138715025, - "brier": 0.049552360791724696, - "ece": 0.0649635215733675 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.623731804146449, - "nll": 1.0611654770329015, - "brier": 0.4999820134671009, - "ece": 0.08503175119655376 - }, - "noul": { - "n": 1185, - "acc": 0.790717299578059, - "nll": 0.4849385924236856, - "brier": 0.29378105357893325, - "ece": 0.03384284862486117 - }, - "score": { - "n": 231, - "acc": 0.6103896103896104, - "nll": 0.9860964914273742, - "brier": 0.39635174564536996, - "ece": 0.05825626927536799, - "mae": 0.10844739449580373, - "within1": 0.9264069264069265, - "spearman": 0.8927321003363274 - } - }, - "ms_per_question": 6.126005038283936 -} \ No newline at end of file diff --git a/docs/training/jet-v3/baseline-test_v3_new.json b/docs/training/jet-v3/baseline-test_v3_new.json deleted file mode 100644 index f0075eb742966806034d34f79e77dcf39f3107c7..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/baseline-test_v3_new.json +++ /dev/null @@ -1,91 +0,0 @@ -{ - "overall": { - "n": 1364, - "acc": 0.45894428152492667, - "nll": 1.4013935283572028, - "brier": 0.762386327881818, - "ece": 0.28340858491984283 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.48125, - "nll": 1.3094110070029275, - "brier": 0.723692375483714, - "ece": 0.17523168660700322 - }, - "v3:entailment": { - "n": 160, - "acc": 0.66875, - "nll": 0.7749338110908865, - "brier": 0.44728858764816976, - "ece": 0.09622256122529507 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.50625, - "nll": 1.2614889068703632, - "brier": 0.7816077215117401, - "ece": 0.3372890591621399 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.010400141747194832, - "brier": 0.00038297277889314164, - "ece": 0.010256767272949219 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.19375, - "nll": 1.8700578375719488, - "brier": 0.9692866911493991, - "ece": 0.35744852144271133 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.54375, - "nll": 1.041868308855919, - "brier": 0.6888522365721684, - "ece": 0.332965100184083 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.21875, - "nll": 2.166966852918267, - "brier": 0.9870833563145507, - "ece": 0.31585227102041247 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.25625, - "nll": 2.658379727520514, - "brier": 1.279929787567725, - "ece": 0.6257376797497273 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.51875, - "nll": 0.8509478795807809, - "brier": 0.6198447547883315, - "ece": 0.23628756739199164 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.0575388516113162, - "brier": 0.010346962844694527, - "ece": 0.05463370561599731 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.45894428152492667, - "nll": 1.4013935283572028, - "brier": 0.762386327881818, - "ece": 0.28340858491984283 - } - }, - "ms_per_question": 7.297233795454618 -} \ No newline at end of file diff --git a/docs/training/jet-v3/calibration.json b/docs/training/jet-v3/calibration.json deleted file mode 100644 index 1e5e965a4979a01d00a8a43075ef89d425bcc21c..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/calibration.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "choice": 1.0, - "score": 1.1225, - "noul": 0.917 -} \ No newline at end of file diff --git a/docs/training/jet-v3/code.patch b/docs/training/jet-v3/code.patch deleted file mode 100644 index eee0e70d010a2f324df8a0eea471b6a0eb932bd9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/code.patch +++ /dev/null @@ -1,143 +0,0 @@ -diff --git a/src/decision_index_engine.py b/src/decision_index_engine.py -index 6587cd3..71ce9ff 100644 ---- a/src/decision_index_engine.py -+++ b/src/decision_index_engine.py -@@ -60,12 +60,13 @@ class JetEngine(Engine): - latency = "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - - def __init__(self, model=MODEL_ID, revision=REVISION, max_tokens=DEFAULT_MAX_TOKENS, -- max_cached_tokens=32768, **options): -+ max_cached_tokens=32768, adapter=None, **options): - super().__init__(**options) - import mlx.core as mx - from mlx_lm import load - - self.mx = mx -+ mx.set_cache_limit(1_000_000_000) - path, self.tokenizer, config = load_assets(model, revision) - self.max_tokens = int(max_tokens) - self.max_cached_tokens = int(max_cached_tokens) -@@ -73,12 +74,13 @@ class JetEngine(Engine): - raise ValueError("max_tokens must be positive and within the checkpoint context window") - if self.max_cached_tokens < 1: - raise ValueError("max_cached_tokens must be positive") -- self.temperature = float(json.loads((path / "calibration.json").read_text())["choice"]) -+ calibration = Path(adapter) / "calibration.json" if adapter else path / "calibration.json" -+ self.temperature = float(json.loads(calibration.read_text())["choice"]) - if not math.isfinite(self.temperature) or self.temperature <= 0: - raise ValueError("Invalid published choice temperature") - # Pin weights to the SAME resolved snapshot as the tokenizer/config. - snapshot_download(model, revision=path.name, allow_patterns=["*.safetensors", "*.safetensors.index.json"]) -- self.model, _ = load(str(path)) -+ self.model, _ = load(str(path), adapter_path=adapter) - self.model.eval() - self.provenance = { - "model": model, "revision": path.name, "backend": "mlx", -@@ -88,6 +90,11 @@ class JetEngine(Engine): - "format_sha256": hashlib.sha256(Path(__file__).with_name("format.py").read_bytes()).hexdigest(), - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - } -+ if adapter: -+ self.provenance["lora_adapter"] = str(Path(adapter).resolve()) -+ self.provenance["lora_weights_sha256"] = hashlib.sha256( -+ (Path(adapter) / "adapters.safetensors").read_bytes()).hexdigest() -+ self.provenance["calibration_sha256"] = hashlib.sha256(calibration.read_bytes()).hexdigest() - - def __call__(self, state, questions): - from model import shared_prefix_label_logits -diff --git a/src/evaluate.py b/src/evaluate.py -index 87eb76d..b29953d 100644 ---- a/src/evaluate.py -+++ b/src/evaluate.py -@@ -29,7 +29,12 @@ def load_data(path: Path, limit: int | None) -> list[dict]: - - def adapter_base(adapter: str | None, fallback: str) -> str: - if adapter and (cfg := Path(adapter) / "adapter_config.json").exists(): -- return json.loads(cfg.read_text()).get("base_model", fallback) -+ config = json.loads(cfg.read_text()) -+ base = config.get("base_model", fallback) -+ if config.get("base_revision"): -+ from huggingface_hub import snapshot_download -+ return snapshot_download(base, revision=config["base_revision"]) -+ return base - return fallback - - -diff --git a/src/train.py b/src/train.py -index 91b4e6b..7cd2cd0 100644 ---- a/src/train.py -+++ b/src/train.py -@@ -10,6 +10,7 @@ next to it (cross-entropy alone treats the levels as unordered). - from __future__ import annotations - - import argparse -+import hashlib - import json - import math - import random -@@ -23,6 +24,7 @@ import mlx.optimizers as optim - import numpy as np - from mlx.utils import tree_flatten, tree_unflatten - from mlx_lm import load -+from huggingface_hub import snapshot_download - from mlx_lm.tuner.trainer import grad_checkpoint - from mlx_lm.tuner.utils import linear_to_lora_layers - -@@ -116,6 +118,7 @@ def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | No - def main() -> None: - ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) -+ ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit") - ap.add_argument("--train", type=Path, default=Path("data/train.jsonl")) - ap.add_argument("--val", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--out", type=Path, default=Path("adapters/jet")) -@@ -142,10 +145,20 @@ def main() -> None: - ap.add_argument("--seed", type=int, default=0) - args = ap.parse_args() - -+ args.out.mkdir(parents=True, exist_ok=True) -+ run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()} -+ run_config["data_sha256"] = { -+ str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val) -+ } -+ (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) -+ - mx.set_cache_limit(int(args.cache_limit_gb * 1e9)) - mx.random.seed(args.seed) - rng = random.Random(args.seed) -- model, tokenizer = load(args.base_model) -+ model_path = args.base_model -+ if args.base_revision: -+ model_path = snapshot_download(args.base_model, revision=args.base_revision) -+ model, tokenizer = load(model_path) - model.freeze() - num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers - lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} -@@ -165,6 +178,8 @@ def main() -> None: - steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None)) - total_steps = math.ceil(steps_per_epoch * args.epochs) - print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches") -+ run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps) -+ (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - warmup = max(1, min(args.warmup, total_steps // 10)) - base_schedule = optim.join_schedules( -@@ -198,6 +213,7 @@ def main() -> None: - "num_layers": num_layers, - "lora_parameters": lora_parameters, - "base_model": args.base_model, -+ "base_revision": args.base_revision, - } - model.eval() - best = evaluate(model, val_batches) -@@ -232,6 +248,10 @@ def main() -> None: - model.train() - improved = metrics["nll"] < best["nll"] - print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{' *saved*' if improved else ''}", flush=True) -+ with (args.out / "metrics.jsonl").open("a") as f: -+ f.write(json.dumps({"step": step, **metrics, "improved": improved, -+ "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n") -+ save_adapter(model, args.out / "last", adapter_config, optimizer) - if improved: - best = metrics - save_adapter(model, args.out, adapter_config, optimizer) diff --git a/docs/training/jet-v3/comparison.json b/docs/training/jet-v3/comparison.json deleted file mode 100644 index b19565e7c79b8d62709879697e4e50af66623b96..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/comparison.json +++ /dev/null @@ -1,540 +0,0 @@ -{ - "test": { - "baseline": { - "overall": { - "n": 3683, - "acc": 0.676622318761879, - "nll": 0.8710569258689598, - "brier": 0.4271373951303378, - "ece": 0.0614988670833338 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04856743158953678, - "brier": 0.02051652638874066, - "ece": 0.0028781566354963246 - }, - "boolq": { - "n": 145, - "acc": 0.8137931034482758, - "nll": 0.4547928252996042, - "brier": 0.2877616179971535, - "ece": 0.033534014636072584 - }, - "emotion": { - "n": 2500, - "acc": 0.6032, - "nll": 1.061520102331415, - "brier": 0.5323164863764696, - "ece": 0.0945529906153679 - }, - "ag_news": { - "n": 135, - "acc": 0.8814814814814815, - "nll": 0.30449030542815175, - "brier": 0.17246954337666645, - "ece": 0.04308961409109611 - }, - "stsb": { - "n": 117, - "acc": 0.5641025641025641, - "nll": 1.1319686002356557, - "brier": 0.313126957621089, - "ece": 0.06744724868709204, - "mae": 0.09893415102229701, - "within1": 0.8803418803418803, - "spearman": 0.8583073814644547 - }, - "mnli": { - "n": 142, - "acc": 0.823943661971831, - "nll": 0.48070815917183185, - "brier": 0.2682452474690198, - "ece": 0.08867672715388554 - }, - "yelp": { - "n": 146, - "acc": 0.7191780821917808, - "nll": 0.6878386074421953, - "brier": 0.39726907064692013, - "ece": 0.0848420409307088 - }, - "banking77": { - "n": 140, - "acc": 0.8857142857142857, - "nll": 0.3826026692570719, - "brier": 0.15188631481491133, - "ece": 0.048383180584226367 - }, - "massive": { - "n": 119, - "acc": 0.9159663865546218, - "nll": 0.3343984374229624, - "brier": 0.11869874499716762, - "ece": 0.027953259834722355 - }, - "civil_comments": { - "n": 149, - "acc": 0.9194630872483222, - "nll": 0.32481257138715025, - "brier": 0.049552360791724696, - "ece": 0.0649635215733675 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.623731804146449, - "nll": 1.0611654770329015, - "brier": 0.4999820134671009, - "ece": 0.08503175119655376 - }, - "noul": { - "n": 1185, - "acc": 0.790717299578059, - "nll": 0.4849385924236856, - "brier": 0.29378105357893325, - "ece": 0.03384284862486117 - }, - "score": { - "n": 231, - "acc": 0.6103896103896104, - "nll": 0.9860964914273742, - "brier": 0.39635174564536996, - "ece": 0.05825626927536799, - "mae": 0.10844739449580373, - "within1": 0.9264069264069265, - "spearman": 0.8927321003363274 - } - }, - "ms_per_question": 6.126005038283936 - }, - "v3": { - "overall": { - "n": 3683, - "acc": 0.65164268259571, - "nll": 0.9367224135821249, - "brier": 0.4652083129310561, - "ece": 0.08906644675602512 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04082268442531737, - "brier": 0.017851146073772996, - "ece": 0.023841043313344322 - }, - "boolq": { - "n": 145, - "acc": 0.7724137931034483, - "nll": 0.5099232494638398, - "brier": 0.31584783904712616, - "ece": 0.08301583117452158 - }, - "emotion": { - "n": 2500, - "acc": 0.5692, - "nll": 1.1573365865381435, - "brier": 0.5875020244710764, - "ece": 0.13260239474773405 - }, - "ag_news": { - "n": 135, - "acc": 0.8962962962962963, - "nll": 0.3023664565205022, - "brier": 0.16289228421711804, - "ece": 0.04908419781261021 - }, - "stsb": { - "n": 117, - "acc": 0.5726495726495726, - "nll": 1.0586881882199628, - "brier": 0.28848051574213823, - "ece": 0.05020417234836481, - "mae": 0.09248589382678049, - "within1": 0.905982905982906, - "spearman": 0.8859154807594529 - }, - "mnli": { - "n": 142, - "acc": 0.8380281690140845, - "nll": 0.4709366592381832, - "brier": 0.26544458521928466, - "ece": 0.07175782679671974 - }, - "yelp": { - "n": 146, - "acc": 0.6917808219178082, - "nll": 0.6570527735525066, - "brier": 0.3875032489726316, - "ece": 0.0966695281740737 - }, - "banking77": { - "n": 140, - "acc": 0.8571428571428571, - "nll": 0.4772076248224559, - "brier": 0.18612738216737845, - "ece": 0.06723336721105234 - }, - "massive": { - "n": 119, - "acc": 0.8991596638655462, - "nll": 0.31824351055547595, - "brier": 0.12467704403489516, - "ece": 0.06691118699161946 - }, - "civil_comments": { - "n": 149, - "acc": 0.9463087248322147, - "nll": 0.31426526626399415, - "brier": 0.04226127270500813, - "ece": 0.053159262509954085 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.595941773268637, - "nll": 1.1370602582882419, - "brier": 0.5447489371027728, - "ece": 0.09578652206987445 - }, - "noul": { - "n": 1185, - "acc": 0.7670886075949367, - "nll": 0.5576139499214292, - "brier": 0.3319211231389647, - "ece": 0.09861372347119488 - }, - "score": { - "n": 231, - "acc": 0.6060606060606061, - "nll": 0.9154134763057489, - "brier": 0.36835430819662557, - "ece": 0.06402145177771, - "mae": 0.10210627521034452, - "within1": 0.948051948051948, - "spearman": 0.9098766844970014 - } - }, - "ms_per_question": 9.170050761607317 - } - }, - "score_eval": { - "baseline": { - "overall": { - "n": 2400, - "acc": 0.52125, - "nll": 1.0903444933540307, - "brier": 0.5290462652704264, - "ece": 0.03454353142529726 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.525, - "nll": 1.1053056762888263, - "brier": 0.33305591800356504, - "ece": 0.042865492701530436, - "mae": 0.102795440373399, - "within1": 0.8833333333333333, - "spearman": 0.9019142986107999 - }, - "yelp": { - "n": 600, - "acc": 0.6033333333333334, - "nll": 0.8807060158190628, - "brier": 0.5043648978801348, - "ece": 0.04826747432351114, - "mae": 0.12410087461073999, - "within1": 0.97, - "spearman": 0.891761260001728 - }, - "sst5": { - "n": 600, - "acc": 0.4866666666666667, - "nll": 1.1988302002350488, - "brier": 0.6439570196420256, - "ece": 0.036009422938028994, - "mae": 0.17386579125653953, - "within1": 0.9083333333333333, - "spearman": 0.7423420876712294 - }, - "amazon": { - "n": 600, - "acc": 0.47, - "nll": 1.1765360810731849, - "brier": 0.6348072255559799, - "ece": 0.09157247965534528, - "mae": 0.1727618505602004, - "within1": 0.9016666666666666, - "spearman": 0.7919591933802752 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.52125, - "nll": 1.0903444933540307, - "brier": 0.5290462652704264, - "ece": 0.03454353142529726, - "mae": 0.14338098920021972, - "within1": 0.9158333333333334, - "spearman": 0.8319325374979925 - } - }, - "ms_per_question": 6.545865319583299 - }, - "v3": { - "overall": { - "n": 2400, - "acc": 0.505, - "nll": 1.1094650883266746, - "brier": 0.5384935784289648, - "ece": 0.04999333718170723 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.535, - "nll": 1.0843378790252087, - "brier": 0.3219786909605596, - "ece": 0.032326187988122285, - "mae": 0.10335274777949478, - "within1": 0.8766666666666667, - "spearman": 0.9015603001158942 - }, - "yelp": { - "n": 600, - "acc": 0.61, - "nll": 0.8702803689117233, - "brier": 0.49920965441985893, - "ece": 0.0410090483476718, - "mae": 0.12408573205873835, - "within1": 0.965, - "spearman": 0.8851276258163018 - }, - "sst5": { - "n": 600, - "acc": 0.4216666666666667, - "nll": 1.2604459897180398, - "brier": 0.6751992408589816, - "ece": 0.05601734429597855, - "mae": 0.18132481554833552, - "within1": 0.8866666666666667, - "spearman": 0.7381838925517012 - }, - "amazon": { - "n": 600, - "acc": 0.4533333333333333, - "nll": 1.2227961156517266, - "brier": 0.6575867274764592, - "ece": 0.09584452390670777, - "mae": 0.18021441891537204, - "within1": 0.905, - "spearman": 0.792213578142597 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.505, - "nll": 1.1094650883266746, - "brier": 0.5384935784289648, - "ece": 0.04999333718170723, - "mae": 0.14724442857548517, - "within1": 0.9083333333333333, - "spearman": 0.8293814583040202 - } - }, - "ms_per_question": 9.288973581249897 - } - }, - "test_v3_new": { - "baseline": { - "overall": { - "n": 1364, - "acc": 0.45894428152492667, - "nll": 1.4013935283572028, - "brier": 0.762386327881818, - "ece": 0.28340858491984283 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.48125, - "nll": 1.3094110070029275, - "brier": 0.723692375483714, - "ece": 0.17523168660700322 - }, - "v3:entailment": { - "n": 160, - "acc": 0.66875, - "nll": 0.7749338110908865, - "brier": 0.44728858764816976, - "ece": 0.09622256122529507 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.50625, - "nll": 1.2614889068703632, - "brier": 0.7816077215117401, - "ece": 0.3372890591621399 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.010400141747194832, - "brier": 0.00038297277889314164, - "ece": 0.010256767272949219 - }, - "v3:code_behavior": { - "n": 160, - "acc": 0.19375, - "nll": 1.8700578375719488, - "brier": 0.9692866911493991, - "ece": 0.35744852144271133 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.54375, - "nll": 1.041868308855919, - "brier": 0.6888522365721684, - "ece": 0.332965100184083 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.21875, - "nll": 2.166966852918267, - "brier": 0.9870833563145507, - "ece": 0.31585227102041247 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.25625, - "nll": 2.658379727520514, - "brier": 1.279929787567725, - "ece": 0.6257376797497273 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.51875, - "nll": 0.8509478795807809, - "brier": 0.6198447547883315, - "ece": 0.23628756739199164 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.0575388516113162, - "brier": 0.010346962844694527, - "ece": 0.05463370561599731 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.45894428152492667, - "nll": 1.4013935283572028, - "brier": 0.762386327881818, - "ece": 0.28340858491984283 - } - }, - "ms_per_question": 7.297233795454618 - }, - "v3": { - "overall": { - "n": 1364, - "acc": 0.8064516129032258, - "nll": 0.44169222164381555, - "brier": 0.25937597216388103, - "ece": 0.019281816119847035 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.61875, - "nll": 0.7616191643755883, - "brier": 0.46012666295995486, - "ece": 0.10276228729635477 - }, - "v3:entailment": { - "n": 160, - "acc": 0.8125, - "nll": 0.5009652316919528, - "brier": 0.2791422625106856, - "ece": 0.06446061916649343 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.6875, - "nll": 0.6180527405813336, - "brier": 0.4261957125154504, - "ece": 0.04085877276957036 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.013040701393038034, - "brier": 0.00037752764778995764, - "ece": 0.012892603874206543 - }, - "v3:code_behavior": { - "n": 160, - "acc": 1.0, - "nll": 0.02178114914568141, - "brier": 0.003290951900657952, - "ece": 0.02077661231160164 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.7875, - "nll": 0.46873742507304994, - "brier": 0.2962765927932337, - "ece": 0.0709937993437052 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.98125, - "nll": 0.08353415211022366, - "brier": 0.03711475835478075, - "ece": 0.04531667511910201 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.51875, - "nll": 1.1555677497759462, - "brier": 0.6220968388351862, - "ece": 0.044088254123926175 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.94375, - "nll": 0.1494387712678872, - "brier": 0.08678391005763345, - "ece": 0.04387272968888283 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.005894699860364199, - "brier": 8.486047603611497e-05, - "ece": 0.005873322486877441 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.8064516129032258, - "nll": 0.44169222164381555, - "brier": 0.25937597216388103, - "ece": 0.019281816119847035 - } - }, - "ms_per_question": 10.176085403225422 - } - } -} \ No newline at end of file diff --git a/docs/training/jet-v3/completion-audit.json b/docs/training/jet-v3/completion-audit.json deleted file mode 100644 index f4ac9902ad0475a4e45bbe2514336319d0381420..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/completion-audit.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "original_user_diff_unchanged": true, - "published_weight_hashes_unchanged": true, - "original_data_hashes_unchanged": true, - "training_code_hashes_match": true, - "diagnostic_requests_per_model": 500, - "diagnostic_errors": 0, - "gpu_peak_observed_mib": 13938, - "tests": { - "data_tests": 3, - "gpu_adapter_tests": 8, - "generated_label_checks": 7440 - }, - "adapter_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5" -} \ No newline at end of file diff --git a/docs/training/jet-v3/data-audit.json b/docs/training/jet-v3/data-audit.json deleted file mode 100644 index add5c2f69d07448c3e98db07133ba789d7c9efa1..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/data-audit.json +++ /dev/null @@ -1,80 +0,0 @@ -{ - "status": "passed", - "counts": { - "train": 47313, - "val": 1395, - "calibration": 1378, - "test": 1364 - }, - "max_new_state_tokens": 1024, - "max_new_prompt_tokens": 1145, - "executed_generated_labels": 7440, - "positions": { - "product_relevance": { - "0": 616, - "1": 613, - "3": 624, - "2": 627 - }, - "boolean_rules": { - "1": 1223, - "0": 1257 - }, - "tool_selection": { - "3": 351, - "4": 386, - "5": 343, - "2": 356, - "0": 345, - "1": 336, - "6": 90 - }, - "entailment": { - "1": 852, - "2": 825, - "0": 803 - }, - "code_behavior": { - "2": 655, - "1": 677, - "3": 469, - "0": 679 - }, - "tool_relevance": { - "1": 1069, - "0": 1021 - }, - "stance": { - "2": 797, - "1": 835, - "0": 830 - }, - "arithmetic": { - "0": 650, - "1": 605, - "3": 601, - "2": 624 - }, - "sarcasm_irony": { - "0": 1248, - "1": 1232 - }, - "document_relevance": { - "1": 1257, - "0": 1223 - } - }, - "source_ids": { - "product_relevance": 2480, - "boolean_rules": 2480, - "tool_selection": 2207, - "entailment": 2480, - "code_behavior": 2480, - "tool_relevance": 2090, - "stance": 2462, - "arithmetic": 2480, - "sarcasm_irony": 2480, - "document_relevance": 2480 - }, - "protected_inputs_unchanged": true -} \ No newline at end of file diff --git a/docs/training/jet-v3/data-provenance.json b/docs/training/jet-v3/data-provenance.json deleted file mode 100644 index 71f7753f812e2c3a1f8f0f84968ae4ec84bc22c7..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/data-provenance.json +++ /dev/null @@ -1,162 +0,0 @@ -{ - "seed": 230923, - "sources": [ - { - "repo": "tasksource/esci", - "revision": "8113b17a5d4099e20243282c926f1bc1a08a4d13", - "file": "data/train-00000-of-00011-2d36455632bef8a2.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--tasksource--esci/snapshots/8113b17a5d4099e20243282c926f1bc1a08a4d13/data/train-00000-of-00011-2d36455632bef8a2.parquet", - "split": "train", - "sha256": "bd6e1217eef98968103d9731ae52e5e1e640b8af8810956ad144cb13481bf3b9" - }, - { - "repo": "stanfordnlp/snli", - "revision": "cdb5c3d5eed6ead6e5a341c8e56e669bb666725b", - "file": "plain_text/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--stanfordnlp--snli/snapshots/cdb5c3d5eed6ead6e5a341c8e56e669bb666725b/plain_text/train-00000-of-00001.parquet", - "split": "train", - "sha256": "ef9a7b25d97390a62aeda7abe26aec8640600f50b818eaeb9107097d60ac6620" - }, - { - "repo": "nyu-mll/glue", - "revision": "bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c", - "file": "qnli/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--nyu-mll--glue/snapshots/bcdcba79d07bc864c1c254ccfcedcce55bcc9a8c/qnli/train-00000-of-00001.parquet", - "split": "train", - "sha256": "ebc7cb70a5bbde0b0336c3d51f31bb4df4673e908e8874b090b52169b1365c6c" - }, - { - "repo": "glaiveai/glaive-function-calling-v2", - "revision": "e7f4b6456019f5d8bcb991ef0dd67d8ff23221ac", - "file": "glaive-function-calling-v2.json", - "path": "/home/jach/.cache/huggingface/hub/datasets--glaiveai--glaive-function-calling-v2/snapshots/e7f4b6456019f5d8bcb991ef0dd67d8ff23221ac/glaive-function-calling-v2.json", - "split": "train", - "sha256": "e9b5d671812b5ca2fbd7b625a37d5c99a19576c37252cdc806defe256aea6dad" - }, - { - "repo": "cardiffnlp/tweet_eval", - "revision": "b3a375baf0f409c77e6bc7aa35102b7b3534f8be", - "file": "irony/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--cardiffnlp--tweet_eval/snapshots/b3a375baf0f409c77e6bc7aa35102b7b3534f8be/irony/train-00000-of-00001.parquet", - "split": "train", - "sha256": "541de8c14f53024660fa5ae1534489b51106c5f4905da294e47ded7e09d73ad0" - }, - { - "repo": "cardiffnlp/tweet_eval", - "revision": "b3a375baf0f409c77e6bc7aa35102b7b3534f8be", - "file": "stance_abortion/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--cardiffnlp--tweet_eval/snapshots/b3a375baf0f409c77e6bc7aa35102b7b3534f8be/stance_abortion/train-00000-of-00001.parquet", - "split": "train", - "sha256": "5d90ce50ba5ed14d7600bc9ca919c2ac2b9f58141b80d63b96fa17027607bd20" - }, - { - "repo": "cardiffnlp/tweet_eval", - "revision": "b3a375baf0f409c77e6bc7aa35102b7b3534f8be", - "file": "stance_atheism/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--cardiffnlp--tweet_eval/snapshots/b3a375baf0f409c77e6bc7aa35102b7b3534f8be/stance_atheism/train-00000-of-00001.parquet", - "split": "train", - "sha256": "4d915d95b5a313db91f3737d095aa89db6b9e749c7838e00fe844ff5a6f98661" - }, - { - "repo": "cardiffnlp/tweet_eval", - "revision": "b3a375baf0f409c77e6bc7aa35102b7b3534f8be", - "file": "stance_climate/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--cardiffnlp--tweet_eval/snapshots/b3a375baf0f409c77e6bc7aa35102b7b3534f8be/stance_climate/train-00000-of-00001.parquet", - "split": "train", - "sha256": "248ba3695bba19f1fcef9c65421d81c65fa3a76570f1d7d3766872022e96b224" - }, - { - "repo": "cardiffnlp/tweet_eval", - "revision": "b3a375baf0f409c77e6bc7aa35102b7b3534f8be", - "file": "stance_feminist/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--cardiffnlp--tweet_eval/snapshots/b3a375baf0f409c77e6bc7aa35102b7b3534f8be/stance_feminist/train-00000-of-00001.parquet", - "split": "train", - "sha256": "f6ae1f6ee44c1e0ac0523bab6278febf8a62387305c24aad3ea9701868b81e1a" - }, - { - "repo": "cardiffnlp/tweet_eval", - "revision": "b3a375baf0f409c77e6bc7aa35102b7b3534f8be", - "file": "stance_hillary/train-00000-of-00001.parquet", - "path": "/home/jach/.cache/huggingface/hub/datasets--cardiffnlp--tweet_eval/snapshots/b3a375baf0f409c77e6bc7aa35102b7b3534f8be/stance_hillary/train-00000-of-00001.parquet", - "split": "train", - "sha256": "61b7730fdffe10728554b85b3d36af454df6185fcebf214697215965a28dc364" - } - ], - "isarcasm": { - "repository": "https://github.com/iabufarha/iSarcasmEval", - "revision": "dfc708b53bde1bb571abfb5692f63231c2232195", - "file": "train/train.En.csv", - "split": "train" - }, - "protected_sha256": { - "data/val.jsonl": "5a73ff6b450e2e1c490afa296fce786aff4af456da81a5ff5ce67846325c04f2", - "data/test.jsonl": "83ef28ce0dd483d6b51d808adac72c84b282da4fea719582f34b83436d3f239e", - "data/score_eval.jsonl": "6a53d16895fe2b3579b1e157b012417d2b8600d6c5605a31cfff6862a6368cf2", - "artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl": "8dd94acf6c313c50aec3fc48188c840d994d14f764b3fff9fe7a89e2e41b9211" - }, - "counts": { - "calibration:arithmetic": 160, - "calibration:boolean_rules": 160, - "calibration:code_behavior": 160, - "calibration:document_relevance": 160, - "calibration:entailment": 160, - "calibration:product_relevance": 160, - "calibration:sarcasm_irony": 160, - "calibration:stance": 160, - "calibration:tool_relevance": 27, - "calibration:tool_selection": 71, - "test:arithmetic": 160, - "test:boolean_rules": 160, - "test:code_behavior": 160, - "test:document_relevance": 160, - "test:entailment": 160, - "test:product_relevance": 160, - "test:sarcasm_irony": 160, - "test:stance": 160, - "test:tool_relevance": 25, - "test:tool_selection": 59, - "train:arithmetic": 2000, - "train:boolean_rules": 2000, - "train:code_behavior": 2000, - "train:document_relevance": 2000, - "train:entailment": 2000, - "train:product_relevance": 2000, - "train:sarcasm_irony": 2000, - "train:stance": 1982, - "train:tool_relevance": 2000, - "train:tool_selection": 2000, - "val:arithmetic": 160, - "val:boolean_rules": 160, - "val:code_behavior": 160, - "val:document_relevance": 160, - "val:entailment": 160, - "val:product_relevance": 160, - "val:sarcasm_irony": 160, - "val:stance": 160, - "val:tool_relevance": 38, - "val:tool_selection": 77 - }, - "filters": { - "over_state_capacity": 537, - "duplicate": 16830, - "excluded_evaluation_content": 46, - "legacy_evaluation_overlap": 46 - }, - "files": { - "train_v3.jsonl": "ec5666923b48289d5ab3b0887f43e52ec403c838cce7a8eee9479888ccc58be4", - "val_v3_new.jsonl": "9c7810c969295bad4bed794025bc1ca505d515a782fd8fefc68d1ef88b39051c", - "calibration_v3_new.jsonl": "9e022e0cbc76c9253ad4304bb705118b00ff09d45b4ce0d29d4c7eeb3b8bbb07", - "test_v3_new.jsonl": "3f4ff9211aea1762ba5c2dcc330047abbb82c08b0c16c952409229006f027677", - "selection_v3.jsonl": "f38f2fd8e503d749ea3a72a811348f663c30b9b5702f171c66f344fecc1dbee4", - "calibration_v3.jsonl": "692519ec1469cc83571b3ebb2bb5e2f2c17b46cf87b80e87f69576cd4539d8cd" - }, - "legacy_rows": 27331, - "generator_code_sha256": "785b8407f93c9ba952f98d29f987be73c853707f4f3211ccd6a4865c4cee3061", - "limitations": [ - "Official frozen suite unavailable; exclusions cover all 23,113 rebuilt requests and existing held-outs.", - "ESCI uses only the first pinned training shard; official ESCI evaluation uses test.", - "New held-outs are grouped samples of training partitions, not leaderboard evaluation.", - "Glaive labels are public synthetic annotations; added distractors are not exhaustively verified for semantic equivalence.", - "Legacy train_v2 lacks original source IDs; retained legacy provenance is its immutable file hash." - ] -} \ No newline at end of file diff --git a/docs/training/jet-v3/diagnostic-baseline.json b/docs/training/jet-v3/diagnostic-baseline.json deleted file mode 100644 index 575af7898abd305a40d01491c89db729dac153d2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/diagnostic-baseline.json +++ /dev/null @@ -1,509 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "301aec4322e27832ed7b5d22651d86951534808e0548df29addd7f6d99c021ad", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.1049274939996394, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 48.596005499348394, - "p95": 373.8532560000749, - "mean": 84.14850932991249 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.285649774536176, - "reference_same_cases": null, - "median_ms": 353.8561925006434, - "detail": { - "field_accuracy": 0.47794117647058826, - "case_exact_accuracy": 0, - "macro_f1": 0.285649774536176, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.20535714285714285, - "reference_same_cases": null, - "median_ms": 43.758601999797975, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.125, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.20535714285714285, - "case_exact_accuracy": 0.07142857142857142, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 47.17806650023704, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.5730994152046783, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.35085995085995086, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.2222222222222222, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.031746031746031744 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.36556995679627785, - "reference_same_cases": null, - "median_ms": 15.092844499122293, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.36556995679627785, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.25058158856763046, - "reference_same_cases": null, - "median_ms": 57.2153459997935, - "detail": { - "field_accuracy": 0.2545454545454545, - "case_exact_accuracy": 0.2545454545454545, - "macro_f1": 0.25058158856763046, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 43.94272399986221, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.38181818181818183, - "reference_same_cases": null, - "median_ms": 42.82419500123069, - "detail": { - "field_accuracy": 0.38181818181818183, - "case_exact_accuracy": 0.38181818181818183, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 62.874530000044615, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.3003030303030303 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.285649774536176, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.285649774536176, - "random": 0.30851893087083776, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.20535714285714285, - "skill": 0.03769841269841268, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.2857142857142857, - "random": 0.25, - "skill": 0.047619047619047596, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.125, - "random": 0.1, - "skill": 0.02777777777777777, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.25, - "skill": 0.25, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.0, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.36556995679627785, - "skill": 0.04836746489660628, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.36556995679627785, - "random": 0.3333245556439576, - "skill": 0.04836746489660628, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.4, - "skill": 0.07772158637210544, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4, - "random": 0.34943722943722944, - "skill": 0.07772158637210544, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.38181818181818183, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.38181818181818183, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2909090909090909, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2909090909090909, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v3/diagnostic-v3.json b/docs/training/jet-v3/diagnostic-v3.json deleted file mode 100644 index 523792f05e652ecfcee3e5380c41cfb54a230d85..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/diagnostic-v3.json +++ /dev/null @@ -1,515 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "f83fe3495f56fe326735f93a47a7ae6553ec22178cbd277a266a34d6fcd16657", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "max_tokens": 8192, - "max_cached_tokens": 8192, - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v3-decision-20260923" - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v3-decision-20260923", - "lora_weights_sha256": "d17a7f6c998e46348263d76760d9766a2f65e731caa5351e14861c4be43535c5", - "calibration_sha256": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.8959890799997083, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 62.191346499275824, - "p95": 441.3005799997336, - "mean": 104.96575838203717 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.28266566725189984, - "reference_same_cases": null, - "median_ms": 416.59302300013223, - "detail": { - "field_accuracy": 0.39705882352941174, - "case_exact_accuracy": 0, - "macro_f1": 0.28266566725189984, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.4642857142857143, - "reference_same_cases": null, - "median_ms": 57.63679649953701, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.5, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.42857142857142855, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.4642857142857143, - "case_exact_accuracy": 0.19642857142857142, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 61.89103449924005, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.6608187134502924, - "case_exact_accuracy": 0.39285714285714285, - "macro_f1": 0.3659194459631857, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.09523809523809523, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.013605442176870748 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.32318840579710145, - "reference_same_cases": null, - "median_ms": 28.75634900010482, - "detail": { - "field_accuracy": 0.4107142857142857, - "case_exact_accuracy": 0.4107142857142857, - "macro_f1": 0.32318840579710145, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5877061469265368, - "reference_same_cases": null, - "median_ms": 72.34201500068593, - "detail": { - "field_accuracy": 0.7272727272727273, - "case_exact_accuracy": 0.7272727272727273, - "macro_f1": 0.5877061469265368, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.38181818181818183, - "reference_same_cases": null, - "median_ms": 56.003140000029816, - "detail": { - "field_accuracy": 0.38181818181818183, - "case_exact_accuracy": 0.38181818181818183, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.43636363636363634, - "reference_same_cases": null, - "median_ms": 54.70588299976953, - "detail": { - "field_accuracy": 0.43636363636363634, - "case_exact_accuracy": 0.43636363636363634, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 81.98549199914851, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.29515151515151516 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.28266566725189984, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.28266566725189984, - "random": 0.30851893087083776, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.4642857142857143, - "skill": 0.3412698412698413, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.42857142857142855, - "random": 0.25, - "skill": 0.23809523809523805, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.5, - "random": 0.1, - "skill": 0.4444444444444445, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.25, - "skill": 0.25, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.0, - "random": 0.2226862339379383, - "skill": 0.0, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.32318840579710145, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32318840579710145, - "random": 0.3333245556439576, - "skill": 0.0, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.38181818181818183, - "skill": 0.04977375565610862, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.38181818181818183, - "random": 0.34943722943722944, - "skill": 0.04977375565610862, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.43636363636363634, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.43636363636363634, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2909090909090909, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2909090909090909, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v3/environment-freeze.txt b/docs/training/jet-v3/environment-freeze.txt deleted file mode 100644 index 9220e676c708088caf1af48cf841d79e3153cc44..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/environment-freeze.txt +++ /dev/null @@ -1,82 +0,0 @@ -aiohappyeyeballs==2.7.1 -aiohttp==3.14.3 -aiosignal==1.4.0 -annotated-doc==0.0.5 -annotated-types==0.8.0 -anthropic==1.8.0 -anyio==4.15.1 -attrs==26.1.0 -certifi==2026.7.22 -charset-normalizer==3.5.1 -chess==1.11.2 -click==8.5.0 -datasets==5.0.1 -decision-index @ git+https://github.com/apolinario/decision-index.git@52a698928a9ae5bdf16b75687c903871db29c6e5 -dill==0.4.1 -docstring-parser==0.18.0 -fastapi==0.141.1 -filelock==4.0.1 -frozenlist==1.8.0 -fsspec==2026.6.0 -h11==0.16.0 -hf-xet==1.6.0 -httpcore==1.0.9 -httpcore2==2.13.0 -httpx==0.28.1 -httpx2==2.13.0 -huggingface-hub==1.32.0 -idna==3.20 --e file:///home/jach/dev/jet -jinja2==3.1.6 -jiter==0.17.0 -markdown-it-py==4.2.0 -markupsafe==3.0.3 -mdurl==0.1.2 -mido==1.3.3 -mlx==0.32.2 -mlx-cuda-12==0.32.2 -mlx-lm==0.31.3 -multidict==6.9.1 -multiprocess==0.70.19 -numpy==2.5.3 -nvidia-cublas-cu12==12.9.2.10 -nvidia-cuda-cccl-cu12==12.9.27 -nvidia-cuda-nvrtc-cu12==12.9.86 -nvidia-cuda-runtime-cu12==12.9.79 -nvidia-cudnn-cu12==9.26.0.51 -nvidia-cufft-cu12==11.4.1.4 -nvidia-nccl-cu12==2.32.3 -nvidia-nvjitlink-cu12==12.9.86 -packaging==26.3 -pandas==3.0.6 -propcache==0.5.4 -protobuf==7.36.2 -pyarrow==25.0.1 -pydantic==2.13.5 -pydantic-core==2.46.5 -pygments==2.21.0 -python-chess==1.999 -python-dateutil==2.9.0.post0 -pyyaml==6.0.3 -regex==2026.9.10 -requests==2.34.2 -rich==15.0.0 -safetensors==0.8.0 -scipy==1.18.1 -sentencepiece==0.2.2 -shellingham==1.5.4 -six==1.17.0 -sniffio==1.3.1 -starlette==1.6.0 -tiktoken==0.14.0 -tokenizers==0.23.2 -tqdm==4.70.1 -transformers==5.17.0 -truststore==0.10.4 -typer==0.27.2 -typing-extensions==4.16.0 -typing-inspection==0.4.4 -urllib3==2.8.0 -uvicorn==0.53.0 -xxhash==4.0.1 -yarl==1.25.1 diff --git a/docs/training/jet-v3/esci-original-split-audit.json b/docs/training/jet-v3/esci-original-split-audit.json deleted file mode 100644 index e30e3b82065f3b6c6b3c8b691a69d34dd915f898..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/esci-original-split-audit.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "verified_rows": 2000, - "official_split": "train", - "labels_and_queries_match": true, - "official_examples_sha256": "4a735b693b4a424a6fc67f5be6e4c811495c488bbf66d02a602d308b2744263a" -} \ No newline at end of file diff --git a/docs/training/jet-v3/metrics.jsonl b/docs/training/jet-v3/metrics.jsonl deleted file mode 100644 index 16cf9ecaec454b4c917df628444bee5e1cd7fc04..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/metrics.jsonl +++ /dev/null @@ -1,20 +0,0 @@ -{"step": 250, "nll": 1.1396356397980474, "acc": 0.4067796610169492, "improved": true, "peak_memory_gb": 3.844020168} -{"step": 500, "nll": 0.872410734065937, "acc": 0.5829481253210067, "improved": true, "peak_memory_gb": 3.844022328} -{"step": 750, "nll": 0.9326960652316603, "acc": 0.559322033898305, "improved": false, "peak_memory_gb": 3.873621528} -{"step": 1000, "nll": 0.7486243336152957, "acc": 0.6343091936312275, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 1250, "nll": 0.7108680260261022, "acc": 0.6877247046738573, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 1500, "nll": 0.6501422261481538, "acc": 0.7185413456599897, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 1750, "nll": 0.6395669230817218, "acc": 0.7226502311248074, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 2000, "nll": 0.6394625997078008, "acc": 0.7272727272727273, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 2250, "nll": 0.590340985233011, "acc": 0.7452491011813046, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 2500, "nll": 0.5810151624263221, "acc": 0.7483307652799178, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 2750, "nll": 0.5433845906975212, "acc": 0.765793528505393, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 3000, "nll": 0.5288521645922629, "acc": 0.7770929635336415, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 3250, "nll": 0.5026491355700069, "acc": 0.7822290703646636, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 3500, "nll": 0.4989109680967326, "acc": 0.7837699024139703, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 3750, "nll": 0.5065731091198091, "acc": 0.7981510015408321, "improved": false, "peak_memory_gb": 4.066629744} -{"step": 4000, "nll": 0.48433677220748644, "acc": 0.7935285053929122, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 4250, "nll": 0.47090714909691167, "acc": 0.810477657935285, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 4500, "nll": 0.4653407073841379, "acc": 0.8120184899845917, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 4750, "nll": 0.4574972794615555, "acc": 0.8120184899845917, "improved": true, "peak_memory_gb": 4.066629744} -{"step": 4888, "nll": 0.4572592325313433, "acc": 0.8058551617873652, "improved": true, "peak_memory_gb": 4.066629744} diff --git a/docs/training/jet-v3/run-manifest.json b/docs/training/jet-v3/run-manifest.json deleted file mode 100644 index 1c476646b7ff60c82dc400c4078795bbf3ac8b75..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/run-manifest.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "git_head": "829955484f94e1dff420880de656ce99775a7b61", - "branch": "codex/decision-index-training", - "python": "3.12.14", - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "files": { - "src/train.py": "cd7467a8775ce38aa49dfb72e53db2552348fd840a3e54698ce7660ffb90fdd4", - "src/evaluate.py": "e38a9667107ca7b4b8cf53400fe8da22c98a044a970fa9fffe81c921c9eea16e", - "src/data/decision_training.py": "785b8407f93c9ba952f98d29f987be73c853707f4f3211ccd6a4865c4cee3061", - "src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "train_cuda.sh": "464ebe6e844f2a02983f76f66edb7fd7f9354ed407c1ff2dfa043b97cfea6dcf", - "scripts/run_v3_cuda.sh": "11232f55dcb76f97613a409e98550c577cdb9c9798dc33dde357443eef996e71", - "scripts/download_v3_sources.py": "ef189800b1b4456cd075a07644385c6cc4b5d61cf77a92c73c807716a7c13e3a", - "uv.lock": "158790ff4bb087ca458bbfd16025251923382fd6ebbee2965410fbf66b1c322b" - }, - "gpu": "name, memory.total [MiB], driver_version\nNVIDIA GeForce RTX 4080 SUPER, 16376 MiB, 615.71.09\n", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "baseline_revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c" -} \ No newline at end of file diff --git a/docs/training/jet-v3/summary.md b/docs/training/jet-v3/summary.md deleted file mode 100644 index 57692e3aab123e4d518293c59d8060d6b2429325..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/summary.md +++ /dev/null @@ -1,136 +0,0 @@ -# Jet v3 completed experiment - -Qwen3-0.6B, Arch Linux, RTX 4080 SUPER (16 GB). Local training only; released Jet unchanged. - -Completed **4,888 steps / 2 epochs**, with 47,313 training rows. Best checkpoint: step 4,888, selection NLL 0.4573, accuracy 80.6%. - -New adapter: `/home/jach/dev/jet/adapters/jet-v3-decision-20260923`. -Calibration: `{"choice": 1.0, "score": 1.1225, "noul": 0.917}`. - -## Independent evaluation - -| Set | Rows | Released Jet accuracy | v3 accuracy | Change | Baseline → v3 NLL | Baseline → v3 ECE | -|---|---:|---:|---:|---:|---:|---:| -| test | 3683 | 67.7% | 65.2% | -2.5 pp | 0.871 → 0.937 | 0.061 → 0.089 | -| score_eval | 2400 | 52.1% | 50.5% | -1.6 pp | 1.090 → 1.109 | 0.035 → 0.050 | -| test_v3_new | 1364 | 45.9% | 80.6% | +34.8 pp | 1.401 → 0.442 | 0.283 → 0.019 | - -Excluding the local Boolean, arithmetic, and code generators, the 884-row public-source subset improves from **54.0% to 71.5%**. The Glaive part is itself publicly released synthetic data. - -### test: source breakdown - -| Source | Rows | Released Jet | v3 | Change | NLL before → after | -|---|---:|---:|---:|---:|---:| -| ag_news | 135 | 88.1% | 89.6% | +1.5 pp | 0.304 → 0.302 | -| banking77 | 140 | 88.6% | 85.7% | -2.9 pp | 0.383 → 0.477 | -| boolq | 145 | 81.4% | 77.2% | -4.1 pp | 0.455 → 0.510 | -| civil_comments | 149 | 91.9% | 94.6% | +2.7 pp | 0.325 → 0.314 | -| dbpedia | 90 | 98.9% | 98.9% | +0.0 pp | 0.049 → 0.041 | -| emotion | 2500 | 60.3% | 56.9% | -3.4 pp | 1.062 → 1.157 | -| massive | 119 | 91.6% | 89.9% | -1.7 pp | 0.334 → 0.318 | -| mnli | 142 | 82.4% | 83.8% | +1.4 pp | 0.481 → 0.471 | -| stsb | 117 | 56.4% | 57.3% | +0.9 pp | 1.132 → 1.059 | -| yelp | 146 | 71.9% | 69.2% | -2.7 pp | 0.688 → 0.657 | - -Batched latency: released fused model 6.1 ms/question; v3 unfused adapter 9.2 ms/question. This is not a fusion-controlled latency comparison. - -### score_eval: source breakdown - -| Source | Rows | Released Jet | v3 | Change | NLL before → after | -|---|---:|---:|---:|---:|---:| -| amazon | 600 | 47.0% | 45.3% | -1.7 pp | 1.177 → 1.223 | -| sst5 | 600 | 48.7% | 42.2% | -6.5 pp | 1.199 → 1.260 | -| stsb | 600 | 52.5% | 53.5% | +1.0 pp | 1.105 → 1.084 | -| yelp | 600 | 60.3% | 61.0% | +0.7 pp | 0.881 → 0.870 | - -Batched latency: released fused model 6.5 ms/question; v3 unfused adapter 9.3 ms/question. This is not a fusion-controlled latency comparison. - -Ordinal within-one accuracy: 91.6% → 90.8%; Spearman: 0.832 → 0.829; normalized MAE: 0.143 → 0.147. - -### test_v3_new: source breakdown - -| Source | Rows | Released Jet | v3 | Change | NLL before → after | -|---|---:|---:|---:|---:|---:| -| v3:arithmetic | 160 | 21.9% | 98.1% | +76.2 pp | 2.167 → 0.084 | -| v3:boolean_rules | 160 | 51.9% | 94.4% | +42.5 pp | 0.851 → 0.149 | -| v3:code_behavior | 160 | 19.4% | 100.0% | +80.6 pp | 1.870 → 0.022 | -| v3:document_relevance | 160 | 54.4% | 78.8% | +24.4 pp | 1.042 → 0.469 | -| v3:entailment | 160 | 66.9% | 81.2% | +14.4 pp | 0.775 → 0.501 | -| v3:product_relevance | 160 | 25.6% | 51.9% | +26.3 pp | 2.658 → 1.156 | -| v3:sarcasm_irony | 160 | 50.6% | 68.8% | +18.1 pp | 1.261 → 0.618 | -| v3:stance | 160 | 48.1% | 61.9% | +13.8 pp | 1.309 → 0.762 | -| v3:tool_relevance | 25 | 100.0% | 100.0% | +0.0 pp | 0.058 → 0.006 | -| v3:tool_selection | 59 | 100.0% | 100.0% | +0.0 pp | 0.010 → 0.013 | - -Batched latency: released fused model 7.3 ms/question; v3 unfused adapter 10.2 ms/question. This is not a fusion-controlled latency comparison. - -## Partial Decision Index diagnostic - -**Not a leaderboard score.** Same 500 complete-group requests from the eight-benchmark public rebuild, complete native prompts up to 8,192 tokens. Frozen official suite remains unavailable. - -| Benchmark | Metric | Released Jet | v3 | Requests | -|---|---|---:|---:|---:| -| ContractNLI | macro-F1 | 0.2856 | 0.2827 | 56 | -| GSM8K | accuracy | 0.2054 | 0.4643 | 112 | -| iSarcasmEval | see separate subtask tracks | not provided | not provided | 56 | -| VAST | macro-F1 | 0.3656 | 0.3232 | 56 | -| NLI4CT | macro-F1 | 0.2506 | 0.5877 | 55 | -| CRUXEval | accuracy | 0.4000 | 0.3818 | 55 | -| CLadder | accuracy | 0.3818 | 0.4364 | 55 | -| Habermas Machine | accuracy | 0.2909 | 0.2909 | 55 | - -The pinned official summary provides no native iSarcasmEval subtask scores (null). Its secondary field accuracy rises from 57.3% to 66.1%, but category positive-F1 falls from 0.0317 to 0.0136; neither is substituted for a native headline score. - -The synthetic code held-out gain does not transfer to CRUXEval here (40.0% → 38.2%). Likewise, the stance hold-out gain does not transfer to VAST (macro-F1 0.366 → 0.323). Keep these generator/domain limitations explicit. - -Recommendation: retain released Jet as the default. Keep v3 as a data-expansion experiment; the gains on new tasks come with regressions on existing tests and mixed out-of-domain transfer. Do not tune against these final test results or promote this as a leaderboard result. - -## Audit and limits - -- New training examples come from pinned public training partitions and deterministic executable generators. Original IDs, hashes, grouping, and configuration are saved. -- All original data/evaluation files and published model files were preserved. 46 original training rows sharing held-out states were removed only from train_v3 (21 validation overlaps, 25 test overlaps). These are state-level overlaps, not necessarily identical questions. -- New families use separate checkpoint-selection, calibration, and final-test components. Existing validation was divided by state into selection and calibration without editing it. -- New training is balanced at 2,000 rows per family except stance (1,982). Tool tests are small after strict connected grouping; treat their changes as noisy diagnostics. -- Public Glaive annotations and distractor selection can contain semantic ambiguity. Product data uses one ESCI training shard. Synthetic tests assess these generator distributions, not general reasoning mastery. -- Training uses the existing 1,024-token state policy; overlong new rows were excluded. The Decision Index adapter preserves complete evaluation prompts. -- This single run changes the data mixture and its held-out selection/calibration mix. It does not establish a replicated causal gain or a full-suite score. - -## Exact configuration - -```json -{ - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v3.jsonl", - "val": "data/selection_v3.jsonl", - "out": "adapters/jet-v3-decision-20260923", - "epochs": 2.0, - "lr": 0.0001, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 4096, - "max_state_tokens": 1024, - "smoothing": 0.02, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "start_step": null, - "seed": 230923, - "data_sha256": { - "data/train_v3.jsonl": "ec5666923b48289d5ab3b0887f43e52ec403c838cce7a8eee9479888ccc58be4", - "data/selection_v3.jsonl": "f38f2fd8e503d749ea3a72a811348f663c30b9b5702f171c66f344fecc1dbee4" - }, - "train_examples": 47313, - "steps_per_epoch": 2444, - "total_steps": 4888 -} -``` - -Logs: `/home/jach/dev/jet/logs/jet-v3`. Data provenance and score JSON: `/home/jach/dev/jet/docs/training/jet-v3`. -Implementation and source descriptions: `/home/jach/dev/jet/docs/training/jet-v3.md`. diff --git a/docs/training/jet-v3/toolret-overlap-audit.json b/docs/training/jet-v3/toolret-overlap-audit.json deleted file mode 100644 index 0b762095b28471937e33ecd7ed572888a3adab6f..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/toolret-overlap-audit.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "repository": "mangopy/ToolRet-Queries", - "revision": "b8c76ad3349ff17497b6bdb28bb5b8f61a0f6445", - "queries": 7477, - "overlap": [] -} \ No newline at end of file diff --git a/docs/training/jet-v3/training_config.json b/docs/training/jet-v3/training_config.json deleted file mode 100644 index 77caf63c1f48b88f53e571811e03306cc3a67a25..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/training_config.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v3.jsonl", - "val": "data/selection_v3.jsonl", - "out": "adapters/jet-v3-decision-20260923", - "epochs": 2.0, - "lr": 0.0001, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 4096, - "max_state_tokens": 1024, - "smoothing": 0.02, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "start_step": null, - "seed": 230923, - "data_sha256": { - "data/train_v3.jsonl": "ec5666923b48289d5ab3b0887f43e52ec403c838cce7a8eee9479888ccc58be4", - "data/selection_v3.jsonl": "f38f2fd8e503d749ea3a72a811348f663c30b9b5702f171c66f344fecc1dbee4" - }, - "train_examples": 47313, - "steps_per_epoch": 2444, - "total_steps": 4888 -} \ No newline at end of file diff --git a/docs/training/jet-v3/v3-score_eval.json b/docs/training/jet-v3/v3-score_eval.json deleted file mode 100644 index 3425a6d600f3bf5e8ff9951bf6fa91ec5d10b309..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/v3-score_eval.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "overall": { - "n": 2400, - "acc": 0.505, - "nll": 1.1094650883266746, - "brier": 0.5384935784289648, - "ece": 0.04999333718170723 - }, - "by_source": { - "stsb": { - "n": 600, - "acc": 0.535, - "nll": 1.0843378790252087, - "brier": 0.3219786909605596, - "ece": 0.032326187988122285, - "mae": 0.10335274777949478, - "within1": 0.8766666666666667, - "spearman": 0.9015603001158942 - }, - "yelp": { - "n": 600, - "acc": 0.61, - "nll": 0.8702803689117233, - "brier": 0.49920965441985893, - "ece": 0.0410090483476718, - "mae": 0.12408573205873835, - "within1": 0.965, - "spearman": 0.8851276258163018 - }, - "sst5": { - "n": 600, - "acc": 0.4216666666666667, - "nll": 1.2604459897180398, - "brier": 0.6751992408589816, - "ece": 0.05601734429597855, - "mae": 0.18132481554833552, - "within1": 0.8866666666666667, - "spearman": 0.7381838925517012 - }, - "amazon": { - "n": 600, - "acc": 0.4533333333333333, - "nll": 1.2227961156517266, - "brier": 0.6575867274764592, - "ece": 0.09584452390670777, - "mae": 0.18021441891537204, - "within1": 0.905, - "spearman": 0.792213578142597 - } - }, - "by_type": { - "score": { - "n": 2400, - "acc": 0.505, - "nll": 1.1094650883266746, - "brier": 0.5384935784289648, - "ece": 0.04999333718170723, - "mae": 0.14724442857548517, - "within1": 0.9083333333333333, - "spearman": 0.8293814583040202 - } - }, - "ms_per_question": 9.288973581249897 -} \ No newline at end of file diff --git a/docs/training/jet-v3/v3-test.json b/docs/training/jet-v3/v3-test.json deleted file mode 100644 index 3d23fd9d4323169683be17e8de5801795abb0568..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/v3-test.json +++ /dev/null @@ -1,111 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.65164268259571, - "nll": 0.9367224135821249, - "brier": 0.4652083129310561, - "ece": 0.08906644675602512 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04082268442531737, - "brier": 0.017851146073772996, - "ece": 0.023841043313344322 - }, - "boolq": { - "n": 145, - "acc": 0.7724137931034483, - "nll": 0.5099232494638398, - "brier": 0.31584783904712616, - "ece": 0.08301583117452158 - }, - "emotion": { - "n": 2500, - "acc": 0.5692, - "nll": 1.1573365865381435, - "brier": 0.5875020244710764, - "ece": 0.13260239474773405 - }, - "ag_news": { - "n": 135, - "acc": 0.8962962962962963, - "nll": 0.3023664565205022, - "brier": 0.16289228421711804, - "ece": 0.04908419781261021 - }, - "stsb": { - "n": 117, - "acc": 0.5726495726495726, - "nll": 1.0586881882199628, - "brier": 0.28848051574213823, - "ece": 0.05020417234836481, - "mae": 0.09248589382678049, - "within1": 0.905982905982906, - "spearman": 0.8859154807594529 - }, - "mnli": { - "n": 142, - "acc": 0.8380281690140845, - "nll": 0.4709366592381832, - "brier": 0.26544458521928466, - "ece": 0.07175782679671974 - }, - "yelp": { - "n": 146, - "acc": 0.6917808219178082, - "nll": 0.6570527735525066, - "brier": 0.3875032489726316, - "ece": 0.0966695281740737 - }, - "banking77": { - "n": 140, - "acc": 0.8571428571428571, - "nll": 0.4772076248224559, - "brier": 0.18612738216737845, - "ece": 0.06723336721105234 - }, - "massive": { - "n": 119, - "acc": 0.8991596638655462, - "nll": 0.31824351055547595, - "brier": 0.12467704403489516, - "ece": 0.06691118699161946 - }, - "civil_comments": { - "n": 149, - "acc": 0.9463087248322147, - "nll": 0.31426526626399415, - "brier": 0.04226127270500813, - "ece": 0.053159262509954085 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.595941773268637, - "nll": 1.1370602582882419, - "brier": 0.5447489371027728, - "ece": 0.09578652206987445 - }, - "noul": { - "n": 1185, - "acc": 0.7670886075949367, - "nll": 0.5576139499214292, - "brier": 0.3319211231389647, - "ece": 0.09861372347119488 - }, - "score": { - "n": 231, - "acc": 0.6060606060606061, - "nll": 0.9154134763057489, - "brier": 0.36835430819662557, - "ece": 0.06402145177771, - "mae": 0.10210627521034452, - "within1": 0.948051948051948, - "spearman": 0.9098766844970014 - } - }, - "ms_per_question": 9.170050761607317 -} \ No newline at end of file diff --git a/docs/training/jet-v3/v3-test_v3_new.json b/docs/training/jet-v3/v3-test_v3_new.json deleted file mode 100644 index 97bf8b54bed34315d795c01fc9e2c0bb37e17e92..0000000000000000000000000000000000000000 --- a/docs/training/jet-v3/v3-test_v3_new.json +++ /dev/null @@ -1,91 +0,0 @@ -{ - "overall": { - "n": 1364, - "acc": 0.8064516129032258, - "nll": 0.44169222164381555, - "brier": 0.25937597216388103, - "ece": 0.019281816119847035 - }, - "by_source": { - "v3:stance": { - "n": 160, - "acc": 0.61875, - "nll": 0.7616191643755883, - "brier": 0.46012666295995486, - "ece": 0.10276228729635477 - }, - "v3:entailment": { - "n": 160, - "acc": 0.8125, - "nll": 0.5009652316919528, - "brier": 0.2791422625106856, - "ece": 0.06446061916649343 - }, - "v3:sarcasm_irony": { - "n": 160, - "acc": 0.6875, - "nll": 0.6180527405813336, - "brier": 0.4261957125154504, - "ece": 0.04085877276957036 - }, - "v3:tool_selection": { - "n": 59, - "acc": 1.0, - "nll": 0.013040701393038034, - "brier": 0.00037752764778995764, - "ece": 0.012892603874206543 - }, - "v3:code_behavior": { - "n": 160, - "acc": 1.0, - "nll": 0.02178114914568141, - "brier": 0.003290951900657952, - "ece": 0.02077661231160164 - }, - "v3:document_relevance": { - "n": 160, - "acc": 0.7875, - "nll": 0.46873742507304994, - "brier": 0.2962765927932337, - "ece": 0.0709937993437052 - }, - "v3:arithmetic": { - "n": 160, - "acc": 0.98125, - "nll": 0.08353415211022366, - "brier": 0.03711475835478075, - "ece": 0.04531667511910201 - }, - "v3:product_relevance": { - "n": 160, - "acc": 0.51875, - "nll": 1.1555677497759462, - "brier": 0.6220968388351862, - "ece": 0.044088254123926175 - }, - "v3:boolean_rules": { - "n": 160, - "acc": 0.94375, - "nll": 0.1494387712678872, - "brier": 0.08678391005763345, - "ece": 0.04387272968888283 - }, - "v3:tool_relevance": { - "n": 25, - "acc": 1.0, - "nll": 0.005894699860364199, - "brier": 8.486047603611497e-05, - "ece": 0.005873322486877441 - } - }, - "by_type": { - "choice": { - "n": 1364, - "acc": 0.8064516129032258, - "nll": 0.44169222164381555, - "brier": 0.25937597216388103, - "ece": 0.019281816119847035 - } - }, - "ms_per_question": 10.176085403225422 -} \ No newline at end of file diff --git a/docs/training/jet-v4-release.md b/docs/training/jet-v4-release.md deleted file mode 100644 index f8127f94910c2241c60a68f6b4d353fc5e46c697..0000000000000000000000000000000000000000 --- a/docs/training/jet-v4-release.md +++ /dev/null @@ -1,52 +0,0 @@ -# V4 release preparation — 2026-09-24 - -The training branch and locally cached `origin/main` were merged in a separate -writable checkout. The original `/home/jach/dev/jet` checkout and its model files -were preserved. Remote main could not be refreshed because network access was -unavailable. Before a GitHub push, fetch and incorporate any newer main commits. - -The merge retains the Decision Index and training changes together with main's -`jet-golden` and ONNX exporter. Golden prompt rendering now uses the shared -backend-independent `inference.py`; all 36 previous reference prompts and token -sequences match. The ONNX launcher uses Python safe-path mode to avoid the local -`onnx_model.py` shadowing ONNX Runtime's module of the same name. - -## Export - -Candidate: `adapters/jet-v4-transfer-20260924`, selected step 3,750. The original -adapter remains under the original checkout. New artifacts are in -`models/jet-v4` in the release checkout. No training data or previous weights -were overwritten. - -CUDA initialization was unavailable during export. `scripts/export_release_cpu.py` -implements the installed MLX LoRA fusion formula on CPU, retaining bf16 weights. -It consumes all 392 adapter tensors and fuses 196 linear layers. Reference -inference uses PyTorch fp32 and the newly fused bf16 weights, with the existing -calibration. This is not evidence of CUDA/CPU numerical parity. - -`scripts/run_export_cpu.py` exports fp32 ONNX with an 8 GiB resident-memory -watchdog. Actual export peak: 6,591 MiB. `src/onnx_web.py` slices logits to the last -position and performs the existing 8-bit weight-only quantization. The resulting -model is about 791 MB. Logs are under `release-logs/`; final runtime validation is -also copied into the model package as `export-validation.txt`. - -The model package includes a model card, calibration, training report, run -manifest, reference vectors, fusion provenance, and per-file SHA256 manifest. -Evaluation gains and regressions are stated in its model card. Partial Decision -Index rebuild results are not an official leaderboard score. - -## Upload - -`scripts/upload_release.py --model models/jet-v4 --repo michaljach/jet ---receipt release-logs/huggingface-receipt.json` checks every manifest hash, -requires a passing ONNX reference check, and uploads an explicit file list in an -atomic model-repository commit with a parent-revision guard. It does not select -hardware, start a Space, or change account billing. Remote upload completion is -established only by a receipt containing the new Hugging Face commit revision. - -The serving backends still pin the previous published revision. Update those -pins to the uploaded commit only after upload succeeds and before deploying the -Space. An upload alone does not update an already pinned serving endpoint. - -Validation passed: all 36 q8 reference answers agree with CPU fp32. Maximum -probability difference: 0.0384; maximum label-logit difference: 0.248. diff --git a/docs/training/jet-v5-release.md b/docs/training/jet-v5-release.md deleted file mode 100644 index 5107f78b97b0da9935de36c585b96d3dc66b5dde..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5-release.md +++ /dev/null @@ -1,28 +0,0 @@ -# Jet release — 2026-09-24 - -Checkpoint: `jet-v5-panel02-r2-20260924`, selected step 3,500. The backbone remains -Qwen3-0.6B. Training configuration, calibration and data audits are preserved in -[the experiment directory](jet-v5/). - -## Export validation - -Native MLX CUDA fused 196 LoRA linear layers into bf16 weights. On 36 fixed -reference cases, fusion preserved all selected answers; the maximum probability -difference from the unfused adapter was 0.02582. - -The CPU ONNX exporter used at most 6,550 MiB resident memory. The q8 export is -791 MB. Against native fused MLX CUDA reference vectors, all 36 selected answers -agree; maximum probability difference is 0.0249, maximum label-logit difference -0.254. These checks establish export compatibility on reference cases, not exact -numerical equivalence or a full benchmark evaluation of the exported model. - -The model package includes calibration, reference vectors, an explicit per-file -SHA256 manifest, training reports and history. Standard serving remains -single-order; the two-order benchmark policy is separately available in source. -No leaderboard entry was submitted. - -## Publication - -Publication receipts and hosting status are recorded after the remote operations -complete. Hosting uses free hardware only; the account's free quota may prevent -the Space from running even after its code and weights have been uploaded. diff --git a/docs/training/jet-v5/aborted-attempt.json b/docs/training/jet-v5/aborted-attempt.json deleted file mode 100644 index ff26bfe4735bc056d40e38fa02b8d528508b2ea9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/aborted-attempt.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "reason": "Two retained MASSIVE utterances also appear in native CLINC evaluation. Discovered by independent original-source check.", - "adapter": "adapters/jet-v5-panel02-20260924", - "eligible_for_release": false, - "overlap_utterances": [ - "what's up with my order", - "turn down the volume" - ], - "inherited_checkpoint_caveat": "V4 inherited legitimate MASSIVE training that may include these utterances. Removing them from continuation data does not erase prior exposure." -} diff --git a/docs/training/jet-v5/baseline-expanded.json b/docs/training/jet-v5/baseline-expanded.json deleted file mode 100644 index d09261233e4ecfbf68cbfc4a0bac821848910f8f..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/baseline-expanded.json +++ /dev/null @@ -1,270 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "9908538b8b244efa20a2d8428104d5f56a131b37569d0e0b10867ced7beb104b", - "results_sha256": "6cb8a48a410aaded44152e7e4b0124ff39ad19bc4465e162e11c8965289b9fc7", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "michaljach/jet", - "revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0293, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts." - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.640739116000077, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 13.238662000730983, - "p95": 52.205200000571494, - "mean": 23.389769003577253 - }, - "benchmarks": [ - { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.14481551509913024, - "reference_same_cases": null, - "median_ms": 39.677124499576166, - "detail": { - "field_accuracy": 0.265, - "case_exact_accuracy": 0.265, - "macro_f1": 0.14481551509913024, - "custom_metrics": {} - } - }, - { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3147420023997653, - "reference_same_cases": null, - "median_ms": 14.061678999951255, - "detail": { - "field_accuracy": 0.32, - "case_exact_accuracy": 0.32, - "macro_f1": 0.3147420023997653, - "custom_metrics": {} - } - }, - { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.295, - "reference_same_cases": null, - "median_ms": 13.026751999859698, - "detail": { - "field_accuracy": 0.295, - "case_exact_accuracy": 0.295, - "custom_metrics": {} - } - }, - { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.605, - "reference_same_cases": null, - "median_ms": 12.673159500081965, - "detail": { - "field_accuracy": 0.605, - "case_exact_accuracy": 0.605, - "custom_metrics": {} - } - }, - { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.37, - "reference_same_cases": null, - "median_ms": 12.696092999703978, - "detail": { - "field_accuracy": 0.37, - "case_exact_accuracy": 0.37, - "custom_metrics": {} - } - }, - { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.545, - "reference_same_cases": null, - "median_ms": 12.078891500095779, - "detail": { - "field_accuracy": 0.545, - "case_exact_accuracy": 0.545, - "custom_metrics": {} - } - }, - { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.36, - "reference_same_cases": null, - "median_ms": 15.943470500133117, - "detail": { - "field_accuracy": 0.36, - "case_exact_accuracy": 0.36, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "24": { - "catalog_id": 24, - "raw": 0.2741441303794245, - "skill": 0.03219217383923264, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2741441303794245, - "random": 0.25, - "skill": 0.03219217383923264, - "planned_cases": 200, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/baseline-test_v5_new.json b/docs/training/jet-v5/baseline-test_v5_new.json deleted file mode 100644 index e746633b9c73716e3c002a8f5e54837a781c0415..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/baseline-test_v5_new.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "overall": { - "n": 600, - "acc": 0.56, - "nll": 1.3715643659847168, - "brier": 0.6246184740195994, - "ece": 0.16034251227974894 - }, - "by_source": { - "v5:adversarial_entailment": { - "n": 150, - "acc": 0.6333333333333333, - "nll": 0.8615019588172436, - "brier": 0.4993069695071258, - "ece": 0.08038273334503174 - }, - "v5:tool_response_preference": { - "n": 150, - "acc": 0.64, - "nll": 1.0372949434661616, - "brier": 0.599172491305248, - "ece": 0.25046542525291443 - }, - "v5:commonsense_completion": { - "n": 150, - "acc": 0.28, - "nll": 1.9893529394765694, - "brier": 0.984350499832082, - "ece": 0.4115457673867543 - }, - "v5:wide_intent": { - "n": 150, - "acc": 0.6866666666666666, - "nll": 1.5981076221788924, - "brier": 0.41564393543394196, - "ece": 0.10220249791940052 - } - }, - "by_type": { - "choice": { - "n": 600, - "acc": 0.56, - "nll": 1.3715643659847168, - "brier": 0.6246184740195994, - "ece": 0.16034251227974894 - } - }, - "ms_per_question": 25.214695985000617, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/code-selected.json b/docs/training/jet-v5/code-selected.json deleted file mode 100644 index 8f2761b2aa33425c6b76857e858f531c32bf6a97..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code-selected.json +++ /dev/null @@ -1,154 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "baa924a42552263d6dad17c9239dfff308eb09c27dd3d3afa7ceae184e0b5ae8", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_ensemble:TwoOrderJetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.961390269999356, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 57.45168149996971, - "p95": 98.6306089998834, - "mean": 67.34930491578972 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.3684210526315789, - "reference_same_cases": null, - "median_ms": 57.45168149996971, - "detail": { - "field_accuracy": 0.3684210526315789, - "case_exact_accuracy": 0.3684210526315789, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.3684210526315789, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3684210526315789, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/code-single.json b/docs/training/jet-v5/code-single.json deleted file mode 100644 index 2b0a5eb1bfcb20a9d1abf12a85878ceea2040ae4..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code-single.json +++ /dev/null @@ -1,152 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16", - "results_sha256": "0d692f6b5310afdc45f272eeeca289fe75a6573b528579911142401e910ef0b1", - "planned_requests": 570, - "processed_requests": 570, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9353613180001048, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 570 - }, - "successful_request_latency_ms": { - "median": 28.505754499747127, - "p95": 68.51539299987053, - "mean": 38.86297063333283 - }, - "benchmarks": [ - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 570, - "answered": 570, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 570, - "metric": "accuracy", - "score": 0.3649122807017544, - "reference_same_cases": null, - "median_ms": 28.505754499747127, - "detail": { - "field_accuracy": 0.3649122807017544, - "case_exact_accuracy": 0.3649122807017544, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "43": { - "catalog_id": 43, - "raw": 0.3649122807017544, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3649122807017544, - "random": 0.36971874129768867, - "skill": 0.0, - "planned_cases": 570, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/code/scripts/audit_v5.py b/docs/training/jet-v5/code/scripts/audit_v5.py deleted file mode 100644 index d7497dc7c03a2139ac78b15ee136f1ba316fc886..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code/scripts/audit_v5.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Independently match every new V5 label back to its pinned native train row.""" -import json,hashlib -from collections import defaultdict,Counter -from pathlib import Path -import pyarrow.parquet as pq -from data.decision_training import ROOT,read,statekey - -def main(): - specs=json.loads((ROOT/'work/jet-v5/sources-complete.json').read_text()); native=defaultdict(list);labels=None - for s in specs: - if s['role']!='train':continue - if s['file'].endswith('.parquet'): - t=pq.read_table(s['path']);rows=t.to_pylist() - if 'clinc' in s['repo']:labels=json.loads(t.schema.metadata[b'huggingface'])['info']['features']['intent']['names'] - else:rows=[json.loads(x) for x in open(s['path'])] - for i,r in enumerate(rows): - key=f'{s["repo"]}@{s["revision"]}/{s["file"]}:{r.get("uid",r.get("ind",i))}' - native[key].append(r) - owners={}; counts=Counter(); origins=set() - for split,file in [('train','train_v5'),('val','selection_v5'),('calibration','calibration_v5'),('test','test_v5_new')]: - for row in read(ROOT/f'data/{file}.jsonl'): - if not row['source'].startswith('v5:'):continue - p=row['provenance']; assert p['origin'] not in origins; origins.add(p['origin']) - answer=row['question']['criteria'][row['gold']] - candidates=native[p['origin']];family=row['family'] - def matches(r): - if family=='wide_intent': - correct=labels[r['intent']]; correct='outside all supported intents' if correct=='oos' else correct.replace('_',' ') - return row['state']==r['text'] and answer==correct and len(row['question']['criteria'])==151 - if family=='commonsense_completion':return row['state']==r['ctx'] and answer==r['endings'][int(r['label'])] - if family=='adversarial_entailment':return row['state']==dict(premise=r['premise'],hypothesis=r['hypothesis']) and answer==['the premise entails the hypothesis','the premise does not determine the hypothesis','the premise contradicts the hypothesis'][r['label']] - return row['state']['messages']==r['messages'] and answer==r['chosen_response']['content'] - assert any(matches(r) for r in candidates),(family,p['origin']) - assert row['target'][list(row['question']['criteria']).index(row['gold'])]==1 - for key in [p['component'],statekey(row['state'])]+p['content_keys']: - assert owners.setdefault(key,split)==split,(key,split) - counts[split+':'+family]+=1 - report={'verified_native_labels':sum(counts.values()),'counts':dict(counts),'source_id_duplicates':0,'cross_split_related_groups':0} - (ROOT/'docs/training/jet-v5/data-audit.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps(report,indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/code/scripts/build_v5.py b/docs/training/jet-v5/code/scripts/build_v5.py deleted file mode 100644 index 89a043bbf116e2de6eb374f6662763491b6353f2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code/scripts/build_v5.py +++ /dev/null @@ -1,143 +0,0 @@ -"""V5: native TRAIN-only coverage for the expanded Decision Index panel. - -Native evaluation files are exclusion-only. All new heldouts come from source -training partitions, grouped before sampling. Never modifies earlier files. -""" -import gzip, hashlib, json, random -from collections import Counter, defaultdict -from pathlib import Path -import pyarrow.parquet as pq -from transformers import AutoTokenizer -from data.decision_training import ROOT, Components, atoms, choice, digest, normalized, read, statekey, write -from format import Question, render_state, build_prompt -SEED=240925 - -def sha(p): - with Path(p).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() - -def records(spec): - if spec['file'].endswith('.parquet'):return pq.read_table(spec['path']).to_pylist() - return [json.loads(s) for s in open(spec['path'])] - -def texts(repo,r): - if 'clinc' in repo:return [r['text']] - if 'hellaswag' in repo:return [r['ctx']] - if 'anli' in repo:return [r['premise'],r['hypothesis']] - return [r[k] for k in ['question','orig_question'] if r.get(k)]+[m['content'] for m in r.get('messages',[]) if m['role']=='user'] - -def content_keys(repo,r):return {'content:'+digest(normalized(t)) for t in texts(repo,r)} - -def main(): - data=ROOT/'data'; work=ROOT/'work/jet-v5' - names=['train_v5.jsonl','selection_v5.jsonl','calibration_v5.jsonl','test_v5_new.jsonl'] - assert not any((data/n).exists() for n in names),'Refusing to overwrite V5' - specs=json.loads((work/'sources-complete.json').read_text()) - protected=[p for p in data.glob('*.jsonl') if any(v in p.name for v in ['val','test','selection','calibration'])] - protected+=[ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl',ROOT/'artifacts/decision-index/esci-rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl'] - blocked=set(); old_atoms=set(); counts=Counter(); native_groups=set() - for p in protected: - assert p.exists(),p - for r in read(p): - blocked.update('content:'+a for a in atoms(r.get('state',r.get('payload',{})))) - if isinstance(r.get('state'),str):blocked.add('content:'+digest(normalized(r['state']))) - parent=read(data/'train_v4_tools_focus.jsonl') - # Include all earlier training, even examples not retained in this short stage. - for p in [data/'train_v2.jsonl',data/'train_v3.jsonl',data/'train_v4.jsonl',data/'train_v4_tools_focus.jsonl']: - for r in read(p): - old_atoms.update('content:'+a for a in atoms(r['state'])) - if isinstance(r['state'],str):old_atoms.add('content:'+digest(normalized(r['state']))) - for s in specs: - if s['role']=='train':continue - for r in records(s): - blocked.update(content_keys(s['repo'],r)) - if 'hellaswag' in s['repo']:native_groups.add('hellaswag:'+r['source_id']) - candidates=[]; components=Components(); origin_seen=set() - for s in specs: - if s['role']!='train':continue - repo=s['repo']; rows=records(s) - if 'clinc' in repo: - labels=json.loads(pq.read_schema(s['path']).metadata[b'huggingface'])['info']['features']['intent']['names'] - opts={k:('outside all supported intents' if k=='oos' else k.replace('_',' ')) for k in labels} - if 'anli' in repo: - assert json.loads(pq.read_schema(s['path']).metadata[b'huggingface'])['info']['features']['label']['names']==['entailment','neutral','contradiction'] - for idx,r in enumerate(rows): - keys=content_keys(repo,r) - if keys&blocked:counts['evaluation_content']+=1;continue - origin=f'{repo}@{s["revision"]}/{s["file"]}:{r.get("uid",r.get("ind",idx))}' - if origin in origin_seen:counts['duplicate_source_id']+=1;continue - origin_seen.add(origin) - if 'clinc' in repo: - family='wide_intent'; group='clinc:'+digest(normalized(r['text'])) - row=choice(r['text'],'Which intent best describes the user request? Choose outside scope if none apply.',opts,labels[r['intent']],family,origin,group) - elif 'hellaswag' in repo: - family='commonsense_completion';group='hellaswag:'+r['source_id'] - if group in native_groups:counts['native_eval_video_or_article']+=1;continue - options={str(i):v for i,v in enumerate(r['endings'])} - row=choice(r['ctx'],'Which continuation is most plausible?',options,str(int(r['label'])),family,origin,group) - elif 'anli' in repo: - family='adversarial_entailment';group='anli:'+digest(normalized(r['premise'])) - options=dict(entailment='the premise entails the hypothesis',neutral='the premise does not determine the hypothesis',contradiction='the premise contradicts the hypothesis') - row=choice(dict(premise=r['premise'],hypothesis=r['hypothesis']),'How does the hypothesis relate to the premise?',options,list(options)[r['label']],family,origin,group) - else: - family='tool_response_preference'; tools=[json.loads(t) if isinstance(t,str) else t for t in r['tools']] - group='when2call-tools:'+digest(sorted(t['name'] for t in tools)) - options=dict(preferred=r['chosen_response']['content'],rejected=r['rejected_response']['content']) - if options['preferred']==options['rejected']:counts['identical_choices']+=1;continue - row=choice(dict(tools=tools,messages=r['messages']),'Which response should the assistant give next, given the available tools?',options,'preferred',family,origin,group) - # Anonymous keys prevent a preferred/rejected key from revealing the answer. - oldkeys=list(row['question']['criteria']); oldgold=row['gold'] - row['question']['criteria']={f'option_{i}':row['question']['criteria'][k] for i,k in enumerate(oldkeys)} - row['gold']=f'option_{oldkeys.index(oldgold)}' - row['source']='v5:'+family - row['provenance']['content_keys']=sorted(keys) - for key in keys:components.union(group,key) - candidates.append(row) - forced={components.find(r['provenance']['group']) for r in candidates if set(r['provenance']['content_keys'])&old_atoms} - tok=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8',local_files_only=True) - bins=defaultdict(list);seen=set() - # Random traversal also bounds tokenization cost after ample eligible groups. - random.Random(SEED).shuffle(candidates) - for r in candidates: - p=r['provenance'];group=components.find(p['group']);n=int(digest([SEED,group])[:8],16)/2**32 - split='train' if group in forced or n<.75 else 'val' if n<.84 else 'calibration' if n<.92 else 'test' - signature=statekey(r['state']) - if signature in seen:counts['duplicate_state']+=1;continue - seen.add(signature) - if len(bins[split,r['family']])>5000:continue - if len(tok.encode(render_state(r['state']),add_special_tokens=False))>2048:counts['long_full_state']+=1;continue - if len(tok.encode(build_prompt(tok,r['state'],Question.from_dict(r['question'])),add_special_tokens=False))>3000:counts['long_full_prompt']+=1;continue - p.update(component=group,split=split);bins[split,r['family']].append(r) - splits=defaultdict(list) - for (split,family),rs in sorted(bins.items()): - random.Random(SEED).shuffle(rs) - rs=rs[:2000 if split=='train' else 150] - assert len(rs)>=(150 if split!='train' else 1500),(split,family,len(rs)) - splits[split]+=rs - owners={};origins=set() - for split,rs in splits.items(): - for r in rs: - assert r['provenance']['origin'] not in origins;origins.add(r['provenance']['origin']) - for key in [r['provenance']['component'],statekey(r['state'])]+r['provenance']['content_keys']: - assert owners.setdefault(key,split)==split,(key,split) - assert not(set(r['provenance']['content_keys'])&blocked) - assert r['target'][list(r['question']['criteria']).index(r['gold'])]==1 - def balanced(rs,limit): - groups=defaultdict(list) - for r in rs:groups[r.get('family',r['source'])].append(r) - for k,g in sorted(groups.items()):random.Random(SEED).shuffle(g) - result=[] - while groups and len(result) logs/jet-v5/smoke.log 2>&1 -bash train_cuda.sh --init-adapter "$BASE" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train data/train_v5.jsonl --val data/selection_v5.jsonl --out "$NEXT" \ - --epochs 1 --lr 1e-5 --rank 16 --grad-checkpoint --max-batch-tokens 3072 --max-state-tokens 2048 \ - --use-example-weights --eval-every 250 --val-limit 10000 --seed 240925 > logs/jet-v5/train.log 2>&1 -.venv/bin/jet-calibrate --adapter "$NEXT" --data data/calibration_v5.jsonl --limit 10000 --batch-size 2 \ - --max-state-tokens 4096 > logs/jet-v5/calibration.log 2>&1 -for name in v4 v5; do - if [[ "$name" == v4 ]]; then RUN="$BASE"; else RUN="$NEXT"; fi - for dataset in test_v5_new test test_v4_new test_v4_tools; do - CONTEXT=2048; if [[ "$dataset" == test ]]; then CONTEXT=4096; fi - .venv/bin/jet-eval --adapter "$RUN" --data "data/$dataset.jsonl" --batch-size 2 \ - --max-state-tokens "$CONTEXT" --json "docs/training/jet-v5/$name-$dataset.json" > "logs/jet-v5/$name-$dataset.log" 2>&1 - done - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz --out "artifacts/decision-index/runs/diagnostic-v5-$name" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter="$RUN" --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$name-diagnostic.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/diagnostic-v5-$name/results.jsonl" --out "docs/training/jet-v5/$name-diagnostic.json" > "logs/jet-v5/$name-report.log" 2>&1 -done -sha256sum -c work/jet-v3/protected.sha256 > logs/jet-v5/protected.log -sha256sum -c work/jet-v3/published.sha256 > logs/jet-v5/published.log -printf 'V5 training and evaluation completed.\n' diff --git a/docs/training/jet-v5/code/src/data/decision_training.py b/docs/training/jet-v5/code/src/data/decision_training.py deleted file mode 100644 index 658ab80d47588f88d2cc0aa62496192f8e0b10b9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code/src/data/decision_training.py +++ /dev/null @@ -1,280 +0,0 @@ -"""Reproducible v3 data from pinned TRAIN files and locally executable generators. - -Run after scripts/rebuild_index_diagnostic.py and scripts/download_v3_sources.py: - uv run --no-sync python -m data.decision_training -No benchmark normalizer supplies training rows. Evaluation is exclusion-only. -""" -from __future__ import annotations -import ast -import csv -import hashlib -import json -import random -import re -from collections import Counter, defaultdict -from pathlib import Path - -import pyarrow.parquet as pq -from format import Question, render_state, build_prompt - -SEED = 230923 -ROOT = Path(__file__).resolve().parents[2] - -def digest(x): - return hashlib.sha256(json.dumps(x, sort_keys=True, ensure_ascii=False).encode()).hexdigest() - -def normalized(s): - return ' '.join(s.casefold().split()) - -def atoms(value): - if isinstance(value, str): - if len(normalized(value)) >= 32: - yield digest(normalized(value)) - elif isinstance(value, dict): - for v in value.values(): yield from atoms(v) - elif isinstance(value, list): - for v in value: yield from atoms(v) - -def statekey(value): - return digest(normalized(value) if isinstance(value, str) else value) - -def choice(state, instruction, options, gold, family, origin, group, identities=(), oracle=None): - rng = random.Random(SEED + int(digest([origin, family])[:12], 16)) - keys = list(options); rng.shuffle(keys) - if family in ('arithmetic', 'code_behavior'): - # Anonymous keys must not reveal the right answer (e.g. output_0). - gold_position = keys.index(gold) - options = {f'option_{i}': options[k] for i, k in enumerate(keys)} - keys = list(options) - gold = keys[gold_position] - result = dict(state=state, question=dict(type='choice', instructions=instruction, - criteria={k: options[k] for k in keys}), target=[float(k == gold) for k in keys], - source='v3:'+family, family=family, provenance=dict(origin=origin, original_split='train', - group=group, identities=list(identities)), gold=gold) - if oracle is not None: result['provenance']['oracle'] = oracle - return result - -def public_candidates(sources): - for spec in sources: - repo, file = spec['repo'], spec['file'] - if 'glaive' in repo: - yield from tool_candidates(spec) - continue - table = pq.read_table(spec['path']) - indices = list(range(len(table))); random.Random(SEED).shuffle(indices) - # Bounded deterministic subset, not a new partition of the public evaluation files. - for idx in indices[:40000]: - r = table.slice(idx, 1).to_pylist()[0] - origin = f'{repo}@{spec["revision"]}/{file}:{r.get("example_id", r.get("idx",idx))}' - if 'esci' in repo: - if r['product_locale'] != 'us': continue - product = {k: r[k] for k in ['product_title','product_description','product_bullet_point','product_brand','product_color'] if r[k]} - if len(json.dumps(product)) > 5000: continue - options = dict(Exact='exact match to the requested product', Substitute='a substitute that can satisfy the same need', Complement='an accessory or complementary product', Irrelevant='irrelevant to the search intent') - yield choice(dict(search_query=r['query'],product=product),'How relevant is the product to the search query?',options,r['esci_label'],'product_relevance',origin,f'esci-query:{r["query_id"]}',[f'esci-product:{r["product_id"]}', 'query:'+normalized(r['query'])]) - elif 'snli' in repo: - if r['label'] < 0: continue - options=dict(entailment='the premise implies the hypothesis',neutral='the premise does not determine whether the hypothesis is true',contradiction='the premise contradicts the hypothesis') - yield choice(dict(premise=r['premise'],hypothesis=r['hypothesis']),'What is the relation of the hypothesis to the premise?',options,list(options)[r['label']],'entailment',origin,'snli-premise:'+normalized(r['premise'])) - elif 'glue' in repo: - yield choice(dict(query=r['question'],document=r['sentence']),'Does this document sentence contain the answer to the query?',dict(yes='contains the answer',no='does not contain the answer'),'yes' if r['label']==0 else 'no','document_relevance',origin,'qnli-query:'+normalized(r['question']),['qnli-doc:'+normalized(r['sentence'])]) - elif 'irony' in file: - yield choice(r['text'],'Does the text express verbal irony?',dict(yes='ironic',no='not ironic'),'yes' if r['label'] else 'no','sarcasm_irony',origin,'tweet:'+normalized(r['text'])) - elif 'stance' in file: - topic={'stance_abortion':'legalization of abortion','stance_atheism':'atheism','stance_climate':'climate change is a real concern','stance_feminist':'the feminist movement','stance_hillary':'Hillary Clinton'}[file.split('/')[0]] - options=dict(none='no clear stance towards the target',against='opposes the target',favor='supports the target') - yield choice(dict(text=r['text'],target=topic),'What stance does the author express towards the target?',options,list(options)[r['label']],'stance',origin,'tweet:'+normalized(r['text'])) - # Actual author-labelled sarcasm, separate official TRAIN partition. Rephrases stay grouped. - file=ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/raw/repos/isarcasm/train/train.En.csv' - for idx,r in enumerate(csv.DictReader(file.open())): - if not r.get('tweet'): continue - origin=f'iSarcasmEval@dfc708b53bde1bb571abfb5692f63231c2232195/train/train.En.csv:{idx}' - group='isarcasm:'+str(idx) - yield choice(r['tweet'],'Is the author being sarcastic?',dict(yes='sarcastic',no='not sarcastic'),'yes' if int(r['sarcastic']) else 'no','sarcasm_irony',origin,group) - if r.get('rephrase'): - yield choice(r['rephrase'],'Is the author being sarcastic?',dict(yes='sarcastic',no='not sarcastic'),'no','sarcasm_irony',origin+':rephrase',group) - -def tool_candidates(spec): - records=json.load(open(spec['path'])) - parsed=[] - relevance=[] - for idx,r in enumerate(records): - # Use only the first request and its immediate labelled function call. - match=re.search(r'USER:\s*(.*?)\n+ASSISTANT:\s*(.*?)(?:<\|endoftext\|>|\n\n)',r['chat'],re.S) - if not match: continue - try: - positive=match[2].startswith('') - call=ast.literal_eval(match[2].replace('', '', 1).strip()) if positive else None - tools=[]; rem=r['system'][r['system'].index('{'):] - while rem.strip(): - item,end=json.JSONDecoder().raw_decode(rem.lstrip());tools.append(item);rem=rem.lstrip()[end:] - names=[x['name'] for x in tools] - if len(set(names)) != len(names): continue - if positive and call['name'] not in names: continue - if not positive and not re.search(r"(don't|do not) have (the )?(capability|function|ability)|only (have|allows|function)|cannot (assist|help|perform)", match[2], re.I): continue - if len(match[1]) > 1800: continue - except (ValueError,SyntaxError,KeyError,TypeError): continue - if len(tools)==1: - relevance.append((idx,match[1].strip(),tools,positive)) - if positive: parsed.append((idx,match[1].strip(),tools,call['name'])) - catalog={t['name']:t['description'] for _,_,ts,_ in parsed for t in ts} - names=sorted(catalog) - for idx,query,tools,gold in parsed: - origin=f'{spec["repo"]}@{spec["revision"]}/{spec["file"]}:{idx}' - group='tool-catalog:'+digest(sorted(t['name'] for t in tools)) - rng=random.Random(SEED+idx) - options={t['name']:t['description'] for t in tools} - # Selection uses source-labelled calls plus catalog distractors. Binary relevance - # uses the native single-tool call/refusal annotation, without added distractors. - others=[n for n in names if n not in options] - for n in rng.sample(others,min(5,len(others))): options[n]=catalog[n] - if sum(len(x) for x in options.values())>2200: continue - yield choice(query,'Select the tool that directly fulfills the user request.',options,gold,'tool_selection',origin,group) - for idx,query,tools,positive in relevance: - tool=tools[0] - origin=f'{spec["repo"]}@{spec["revision"]}/{spec["file"]}:{idx}' - group='tool-catalog:'+digest([tool['name']]) - yield choice(dict(request=query,tool=tool),'Is this tool relevant for fulfilling the user request?', - dict(yes='the tool can fulfill the request',no='the request requires a different capability'), - 'yes' if positive else 'no','tool_relevance',origin,group) - -def generators(n=10000): - rng=random.Random(SEED) - for i in range(n): - values={x:bool(rng.getrandbits(1)) for x in ['active','verified','paid','blocked','admin']} - def rule(depth): - if depth==0 or rng.random()<0.22: - name=rng.choice(list(values));return f'not {name}' if rng.random()<0.3 else name - return f'({rule(depth-1)} {rng.choice(["and","or"])} {rule(depth-1)})' - expr=rule(rng.randint(2,4)); result=bool(eval(expr,{'__builtins__':{}},values)) - origin=f'generator:boolean:{i}'; group='boolean-rule:'+expr - yield choice(dict(facts=values,allow_if=expr),'Under this exact Boolean rule, is access allowed?',dict(allow='access is allowed',deny='access is denied'),'allow' if result else 'deny','boolean_rules',origin,group,oracle=dict(expression=expr,variables=values,result=result)) - a,b,c=[rng.randint(-30,50) for _ in range(3)];d=rng.randint(1,12) - expr=rng.choice([f'({a} + {b}) * {c}',f'{a} * {b} - {c}',f'({a} - {b}) // {d}',f'{a} + {b} * {c}',f'({a} * {b}) % {d}',f'abs({a} - {b}) + {c}']) - result=eval(expr,{'__builtins__':{},'abs':abs}) - choices={result,result+1,result-1,result+rng.choice([3,5,10])} - options={f'value_{j}':str(v) for j,v in enumerate(sorted(choices))};gold=next(k for k,v in options.items() if v==str(result)) - yield choice(dict(expression=expr),'Evaluate the Python integer expression. // is floor division and % is modulo.',options,gold,'arithmetic',f'generator:arithmetic:{i}','arithmetic:'+expr,oracle=dict(expression=expr,result=result)) - k=rng.randint(-8,12);m=rng.randint(2,8);inp=[rng.randint(-10,15) for _ in range(rng.randint(3,7))] - body=rng.choice([f'return sum(v + {k} for v in x if v % {m} != 0)',f'return [v * {k} for v in x][::{m}]',f'return sorted(set(x + [{k}]))[:{m}]',f'return sum(x[-{m}:]) + {k}',f'return [v for v in x if v > {k}][::-1]',f'return len([v for v in x if v % {m} == {k%m}])']) - code='def f(x):\n '+body - ns={};exec(code,{'__builtins__':{},'sum':sum,'sorted':sorted,'set':set,'len':len},ns) - result=ns['f'](inp) - if isinstance(result,list): bad=[result[::-1],result+[k],result[1:],[]] - else: bad=[result-1,result+1,result+3,0] - outputs=list(dict.fromkeys([repr(result)]+[repr(x) for x in bad]))[:4] - if len(outputs)<2: continue - options={f'output_{j}':v for j,v in enumerate(outputs)} - yield choice(dict(code=code,input=inp),'What value does f return for the supplied input?',options,'output_0','code_behavior',f'generator:code:{i}','program:'+code,oracle=dict(result=result,executable=True)) - -class Components: - def __init__(self):self.parent={} - def find(self,x): - self.parent.setdefault(x,x) - while x!=self.parent[x]: - self.parent[x]=self.parent[self.parent[x]];x=self.parent[x] - return x - def union(self,a,b): - a,b=self.find(a),self.find(b) - if a!=b:self.parent[max(a,b)]=min(a,b) - -def read(path):return [json.loads(line) for line in path.open() if line.strip()] -def write(path,rows): - path.write_text(''.join(json.dumps(r,ensure_ascii=False)+'\n' for r in rows)) - -def main(): - sources=json.load(open(ROOT/'work/jet-v3/sources.json')) - from transformers import AutoTokenizer - tokenizer=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16') - data=ROOT/'data'; protected=[data/x for x in ['val.jsonl','test.jsonl','score_eval.jsonl']] - suite=ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl' - assert suite.exists(), 'Rebuild the evaluation corpus first, for exclusion-only use.' - held=[r for p in protected+[suite] for r in read(p)] - blocked_states={statekey(r['state']) for r in held} - blocked_atoms={a for r in held for a in atoms(r['state'])} - legacy=read(data/'train_v2.jsonl') - legacy_keys={statekey(r['state']) for r in legacy} - legacy_atoms={a for r in legacy for a in atoms(r['state'])} - stats=Counter();seen=set();candidates=[] - for r in list(public_candidates(sources))+list(generators()): - if statekey(r['state']) in blocked_states or set(atoms(r['state'])) & blocked_atoms: - stats['excluded_evaluation_content']+=1;continue - # Original identity + family permits multiple tasks from one source row, kept together. - identity=(r['provenance']['origin'],r['family']) - semantic=digest([r['state'],r['question']['instructions'],r['question']['criteria']]) - if identity in seen or semantic in seen: - stats['duplicate']+=1;continue - if len(tokenizer.encode(render_state(r['state']), add_special_tokens=False)) > 1024: - stats['over_state_capacity']+=1;continue - if len(tokenizer.encode(build_prompt(tokenizer,r['state'],Question.from_dict(r['question'])), add_special_tokens=False)) > 2048: - stats['over_prompt_capacity']+=1;continue - seen.add(identity);seen.add(semantic);candidates.append(r) - components=Components() - for r in candidates: - p=r['provenance'];key=p['group'] - links=['origin:'+p['origin'],'state:'+statekey(r['state'])]+p['identities'] - if r['family'] in ('tool_selection', 'tool_relevance'): - query=r['state'] if isinstance(r['state'],str) else r['state']['request'] - links.append('tool-query:'+normalized(query)) - # Entire same tweet/premise/query/product/program components stay in one partition. - for link in links:components.union(key,link) - forced_train=set() - for r in candidates: - if statekey(r['state']) in legacy_keys or set(atoms(r['state'])) & legacy_atoms: - forced_train.add(components.find(r['provenance']['group'])) - bins=defaultdict(list) - for r in candidates: - group=components.find(r['provenance']['group']);b=int(digest([SEED,group])[:8],16)/2**32 - split='train' if group in forced_train or b<.76 else 'val' if b<.84 else 'calibration' if b<.92 else 'test' - r['provenance']['component']=group;r['provenance']['split']=split - bins[(split,r['family'])].append(r) - out=defaultdict(list);counts={} - for (split,family),rows in sorted(bins.items()): - # Equal family budget, with round-robin classes; no duplicate oversampling. - random.Random(SEED).shuffle(rows);classes=defaultdict(list) - for r in rows:classes[r['gold'] if family not in ['tool_selection','arithmetic','code_behavior'] else 'all'].append(r) - selected=[];limit=2000 if split=='train' else 160 - while classes and len(selected) list[dict]: - rows = [json.loads(line) for line in path.open()] - return rows[:limit] if limit else rows - - -def adapter_base(adapter: str | None, fallback: str) -> str: - if adapter and (cfg := Path(adapter) / "adapter_config.json").exists(): - config = json.loads(cfg.read_text()) - base = config.get("base_model", fallback) - if config.get("base_revision"): - from huggingface_hub import snapshot_download - return snapshot_download(base, revision=config["base_revision"]) - return base - return fallback - - -def collect_logits(jet: Jet, rows: list[dict], batch_size: int) -> tuple[list[np.ndarray], float]: - """Raw label logits for every row, batched by prompt length. Returns (logits, ms per question).""" - # Batch shapes vary, so MLX's freed-buffer cache otherwise grows until CUDA runs out (as in jet-train). - mx.set_cache_limit(1_000_000_000) - items = [(r["state"], Question.from_dict(r["question"])) for r in rows] - order = sorted(range(len(items)), key=lambda i: len(str(items[i][0]))) - logits: list[np.ndarray | None] = [None] * len(items) - started = time.perf_counter() - for start in range(0, len(order), batch_size): - idx = order[start : start + batch_size] - out, _ = jet.raw_logits([items[i] for i in idx]) - for i, z in zip(idx, out): - logits[i] = z - print(f"\r {min(start + batch_size, len(order))}/{len(order)}", end="", flush=True) - print() - return logits, (time.perf_counter() - started) * 1000 / max(len(items), 1) - - -def softmax(z: np.ndarray, t: float) -> np.ndarray: - z = z / t - e = np.exp(z - z.max()) - return e / e.sum() - - -def nll(logits: list[np.ndarray], targets: list[np.ndarray], t: float) -> float: - return float(np.mean([-(tg * np.log(np.clip(softmax(z, t), 1e-12, 1))).sum() for z, tg in zip(logits, targets)])) - - -def metrics(probs: list[np.ndarray], targets: list[np.ndarray], bins: int = 10) -> dict[str, float]: - top = np.array([p.max() for p in probs]) - hit = np.array([p.argmax() == t.argmax() for p, t in zip(probs, targets)], dtype=float) - # ECE over the model's top prediction, scored against the (possibly soft) target's argmax. - ece = 0.0 - for lo in np.linspace(0, 1, bins, endpoint=False): - sel = (top > lo) & (top <= lo + 1 / bins) - if sel.any(): - ece += sel.mean() * abs(top[sel].mean() - hit[sel].mean()) - return { - "n": len(probs), - "acc": float(hit.mean()), - "nll": float(np.mean([-(t * np.log(np.clip(p, 1e-12, 1))).sum() for p, t in zip(probs, targets)])), - "brier": float(np.mean([((p - t) ** 2).sum() for p, t in zip(probs, targets)])), - "ece": float(ece), - } - - -def rank(x: np.ndarray) -> np.ndarray: - """Average ranks (ties share their mean rank), for Spearman correlation.""" - order = np.argsort(x, kind="stable") - ranks = np.empty(len(x)) - ranks[order] = np.arange(len(x)) - for v in np.unique(x): - tie = x == v - ranks[tie] = ranks[tie].mean() - return ranks - - -def ordinal_metrics(probs: list[np.ndarray], targets: list[np.ndarray]) -> dict[str, float]: - """Score questions: compare expected levels, normalized to 0-1 so scales of any size mix. - - mae: mean |E[pred] - E[target]| on the 0-1 scale; within1: argmax level within one of the - target's expected level; spearman: rank correlation of expected scores. - """ - pred = np.array([np.dot(np.arange(len(p)), p) / (len(p) - 1) for p in probs]) - true = np.array([np.dot(np.arange(len(t)), t) / (len(t) - 1) for t in targets]) - within = [abs(int(p.argmax()) - np.dot(np.arange(len(t)), t)) <= 1 for p, t in zip(probs, targets)] - spearman = float(np.corrcoef(rank(pred), rank(true))[0, 1]) if len(pred) > 2 and true.std() > 0 else float("nan") - return {"mae": float(np.abs(pred - true).mean()), "within1": float(np.mean(within)), "spearman": spearman} - - -def print_table(title: str, groups: dict[str, dict[str, float]]) -> None: - print(f"\n{title}") - print(f"{'':<42}{'n':>6}{'acc':>8}{'nll':>8}{'brier':>8}{'ece':>8}{'mae':>8}{'±1':>8}{'rho':>8}") - for name, m in sorted(groups.items()): - ordinal = "".join(f"{m[k]:>8.3f}" for k in ("mae", "within1", "spearman") if k in m) - print(f"{name[:41]:<42}{m['n']:>6}{m['acc']:>8.3f}{m['nll']:>8.3f}{m['brier']:>8.3f}{m['ece']:>8.3f}{ordinal}") - - -def main_eval() -> None: - ap = argparse.ArgumentParser(description="Evaluate a Jet model.") - ap.add_argument("--adapter", default=None, help="adapter dir; omit to evaluate the base model") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--data", type=Path, default=Path("data/test.jsonl")) - ap.add_argument("--limit", type=int, default=None) - ap.add_argument("--batch-size", type=int, default=16) - ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS) - ap.add_argument("--json", type=Path, default=None, help="also write metrics as JSON") - args = ap.parse_args() - - jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens) - rows = load_data(args.data, args.limit) - logits, ms = collect_logits(jet, rows, args.batch_size) - probs = [softmax(z, jet.temperatures[r["question"]["type"]]) for z, r in zip(logits, rows)] - targets = [np.array(r["target"]) for r in rows] - - by_source, by_type = defaultdict(list), defaultdict(list) - for i, r in enumerate(rows): - source = "distill (all)" if r["source"].startswith("distill:") else r["source"] - by_source[source].append(i) - by_type[r["question"]["type"]].append(i) - - def group(ix: list[int]) -> dict[str, float]: - m = metrics([probs[i] for i in ix], [targets[i] for i in ix]) - scored = [i for i in ix if rows[i]["question"]["type"] == "score"] - if len(scored) == len(ix): - m |= ordinal_metrics([probs[i] for i in scored], [targets[i] for i in scored]) - return m - - overall = metrics(probs, targets) - print_table("by source", {k: group(v) for k, v in by_source.items()}) - print_table("by type", {k: group(v) for k, v in by_type.items()}) - print_table("overall", {"all": overall}) - print(f"\n{ms:.1f} ms/question (batched, batch size {args.batch_size}); temperatures {jet.temperatures}") - if args.json: - args.json.write_text(json.dumps({ - "overall": overall, - "by_source": {k: group(v) for k, v in by_source.items()}, - "by_type": {k: group(v) for k, v in by_type.items()}, - "ms_per_question": ms, - "max_state_tokens": args.max_state_tokens, - }, indent=2)) - - -def main_calibrate() -> None: - ap = argparse.ArgumentParser(description="Fit per-type softmax temperatures on held-out data.") - ap.add_argument("--adapter", required=True) - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--data", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--limit", type=int, default=3000) - ap.add_argument("--batch-size", type=int, default=16) - ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS) - args = ap.parse_args() - - jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens) - rows = load_data(args.data, args.limit) - logits, _ = collect_logits(jet, rows, args.batch_size) - - temperatures = {} - for qtype in ("choice", "score", "noul"): - ix = [i for i, r in enumerate(rows) if r["question"]["type"] == qtype] - if not ix: - temperatures[qtype] = 1.0 - continue - zs, ts = [logits[i] for i in ix], [np.array(rows[i]["target"]) for i in ix] - scores = [nll(zs, ts, t) for t in TEMPERATURE_GRID] - best = float(TEMPERATURE_GRID[int(np.argmin(scores))]) - temperatures[qtype] = round(best, 4) - print(f"{qtype:>6}: n={len(ix)} T={best:.3f} nll {nll(zs, ts, 1.0):.4f} -> {min(scores):.4f}") - (Path(args.adapter) / "calibration.json").write_text(json.dumps(temperatures, indent=2)) - print(f"wrote {Path(args.adapter) / 'calibration.json'}") diff --git a/docs/training/jet-v5/code/src/format.py b/docs/training/jet-v5/code/src/format.py deleted file mode 100644 index 16ad75d3b250cde3629d60f849bd84cb61fa246a..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code/src/format.py +++ /dev/null @@ -1,143 +0,0 @@ -"""Prompt format shared by training and inference. - -Every question is reduced to "pick one label token". The model is run once over -the prompt and we read the next-token logits restricted to the label tokens: - -- choice: one label per option (A..Z, then single-token AA, AB, ...), up to 255 options -- score: one digit per level (0..9), 2-10 levels, ordered low -> high -- noul: "no" / "yes" -""" - -from __future__ import annotations - -import itertools -import json -import string -from dataclasses import dataclass -from typing import Any, Literal - -QuestionType = Literal["choice", "score", "noul"] - -MAX_CHOICE_OPTIONS = 255 -MIN_SCORE_LEVELS, MAX_SCORE_LEVELS = 2, 10 - -SYSTEM_PROMPT = ( - "You are Jet, a decision model. Read the state and the question, then answer " - "with exactly one label from the allowed labels." -) - - -@dataclass(frozen=True) -class Question: - type: QuestionType - instructions: str - # choice: {key: description}; score: [level descriptions, low -> high]; noul: None - criteria: dict[str, str] | list[str] | None = None - - @property - def keys(self) -> list[str]: - """Answer keys in label order (option keys, level descriptions, or no/yes).""" - if self.type == "choice": - return list(self.criteria) - if self.type == "score": - return list(self.criteria) - return ["no", "yes"] - - @classmethod - def from_dict(cls, d: dict[str, Any]) -> Question: - q = cls(type=d["type"], instructions=d["instructions"], criteria=d.get("criteria")) - q.validate() - return q - - def to_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"type": self.type, "instructions": self.instructions} - if self.criteria is not None: - d["criteria"] = self.criteria - return d - - def validate(self) -> None: - if self.type not in ("choice", "score", "noul"): - raise ValueError(f"unknown question type {self.type!r}") - if not isinstance(self.instructions, str) or not self.instructions.strip(): - raise ValueError("instructions must be a non-empty string") - if self.type == "choice": - if not isinstance(self.criteria, dict) or not 2 <= len(self.criteria) <= MAX_CHOICE_OPTIONS: - raise ValueError(f"choice criteria must be an object with 2-{MAX_CHOICE_OPTIONS} options") - elif self.type == "score": - if not isinstance(self.criteria, list) or not MIN_SCORE_LEVELS <= len(self.criteria) <= MAX_SCORE_LEVELS: - raise ValueError(f"score criteria must be a list of {MIN_SCORE_LEVELS}-{MAX_SCORE_LEVELS} levels") - elif self.criteria is not None: - raise ValueError("noul questions take no criteria") - - -_CHOICE_LABELS: dict[int, list[str]] = {} - - -def choice_labels(tokenizer, n: int) -> list[str]: - """A..Z, then the two-letter labels AA, AB, ... that are single tokens for this tokenizer.""" - key = id(tokenizer) - if key not in _CHOICE_LABELS: - letters = string.ascii_uppercase - pairs = ("".join(p) for p in itertools.product(letters, repeat=2)) - candidates = list(letters) + [p for p in pairs if len(tokenizer.encode(p, add_special_tokens=False)) == 1] - if len(candidates) < MAX_CHOICE_OPTIONS: - raise ValueError(f"tokenizer only has {len(candidates)} single-token choice labels") - _CHOICE_LABELS[key] = candidates[:MAX_CHOICE_OPTIONS] - return _CHOICE_LABELS[key][:n] - - -def labels_for(tokenizer, q: Question) -> list[str]: - if q.type == "choice": - return choice_labels(tokenizer, len(q.criteria)) - if q.type == "score": - return [str(i) for i in range(len(q.criteria))] - return ["no", "yes"] - - -def render_state(state: Any) -> str: - if isinstance(state, str): - return state - if isinstance(state, list) and all(isinstance(s, str) for s in state): - return "\n\n".join(state) - return json.dumps(state, ensure_ascii=False, indent=2) - - -def render_user(state: Any, q: Question, labels: list[str]) -> str: - parts = [f"\n{render_state(state)}\n", ""] - if q.type == "choice": - opts = "\n".join(f"{lab}: {key}: {desc}" for lab, (key, desc) in zip(labels, q.criteria.items())) - parts += [ - f"Question: {q.instructions}", - f"Options:\n{opts}", - f"Answer with the label of the best option ({labels[0]}-{labels[-1]}).", - ] - elif q.type == "score": - levels = "\n".join(f"{lab}: {desc}" for lab, desc in zip(labels, q.criteria)) - parts += [ - f"Question: {q.instructions}", - f"Scale (lowest to highest):\n{levels}", - f"Answer with the level number (0-{len(labels) - 1}).", - ] - else: - parts += [f"Question: {q.instructions}", "Answer yes or no."] - return "\n".join(parts) - - -def build_prompt(tokenizer, state: Any, q: Question) -> str: - return tokenizer.apply_chat_template( - [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": render_user(state, q, labels_for(tokenizer, q))}], - add_generation_prompt=True, - tokenize=False, - enable_thinking=False, - ) - - -def _label_token_id(tokenizer, label: str) -> int: - ids = tokenizer.encode(label, add_special_tokens=False) - if len(ids) != 1: - raise ValueError(f"label {label!r} is not a single token for this tokenizer") - return ids[0] - - -def label_token_ids(tokenizer, q: Question) -> list[int]: - return [_label_token_id(tokenizer, lab) for lab in labels_for(tokenizer, q)] diff --git a/docs/training/jet-v5/code/src/model.py b/docs/training/jet-v5/code/src/model.py deleted file mode 100644 index 5e28e2317876fcd76b3edafaee434bf6740ed293..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code/src/model.py +++ /dev/null @@ -1,151 +0,0 @@ -"""Jet inference: one forward pass per batch of questions, no generation.""" - -from __future__ import annotations - -import json -import time -from pathlib import Path -from typing import Any - -import mlx.core as mx -import numpy as np -from huggingface_hub import snapshot_download -from mlx_lm import load -from mlx_lm.models.cache import make_prompt_cache - -from format import Question, label_token_ids -from inference import DEFAULT_MAX_STATE_TOKENS, encode, summarize - -DEFAULT_BASE_MODEL = "mlx-community/Qwen3-0.6B-bf16" -NEG_INF = -1e9 - - -def label_logits(model, tokens: mx.array, lengths: mx.array, label_ids: mx.array, label_mask: mx.array) -> mx.array: - """Next-token logits after each prompt, restricted to its label tokens. - - tokens: (B, L) right-padded prompts; lengths: (B,) true lengths; - label_ids / label_mask: (B, K) label token ids padded to the widest question. - Returns (B, K) logits with padded labels set to -inf. - """ - hidden = model.model(tokens) - last = hidden[mx.arange(tokens.shape[0]), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - picked = mx.take_along_axis(logits, label_ids, axis=-1).astype(mx.float32) - return mx.where(label_mask, picked, NEG_INF) - - -def head_logits(model, hidden: mx.array) -> mx.array: - if getattr(model.args, "tie_word_embeddings", False): - return model.model.embed_tokens.as_linear(hidden) - return model.lm_head(hidden) - - -def shared_prefix_label_logits( - model, seqs: list[list[int]], label_lists: list[list[int]], max_cached_tokens: int = 32_768 -) -> np.ndarray: - """Like label_logits, but encodes the prompts' common prefix (system + state) once. - - The prefix KV cache is replicated across a chunk of question suffixes, so each - question only pays for its own few dozen tokens. Chunks are sized so that - replicated prefix tokens stay under max_cached_tokens. - """ - common = min(len(s) for s in seqs) - 1 # every suffix keeps at least one token - for i in range(common): - if any(s[i] != seqs[0][i] for s in seqs): - common = i - break - prefix_cache = make_prompt_cache(model) - if common: - model.model(mx.array(seqs[0][:common])[None], cache=prefix_cache) - prefix_state = [c.state for c in prefix_cache] if common else None - if prefix_state: - mx.eval(prefix_state) - - out = np.full((len(seqs), max(map(len, label_lists))), NEG_INF, dtype=np.float32) - chunk = max(1, max_cached_tokens // max(common, 1)) - for start in range(0, len(seqs), chunk): - suffixes = [s[common:] for s in seqs[start : start + chunk]] - b = len(suffixes) - cache = make_prompt_cache(model) - if prefix_state: - for c, (k, v) in zip(cache, prefix_state): - c.state = (mx.repeat(k, b, axis=0), mx.repeat(v, b, axis=0)) - tokens, lengths = pad_batch(suffixes) - hidden = model.model(tokens, cache=cache) - last = hidden[mx.arange(b), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - ids, mask = pad_labels(label_lists[start : start + chunk]) - picked = mx.where(mask, mx.take_along_axis(logits, ids, axis=-1).astype(mx.float32), NEG_INF) - out[start : start + b, : picked.shape[1]] = np.array(picked) - return out - - -def pad_batch(seqs: list[list[int]], pad: int = 0) -> tuple[mx.array, mx.array]: - width = max(len(s) for s in seqs) - arr = np.full((len(seqs), width), pad, dtype=np.int32) - for i, s in enumerate(seqs): - arr[i, : len(s)] = s - return mx.array(arr), mx.array([len(s) for s in seqs], dtype=mx.int32) - - -def pad_labels(label_lists: list[list[int]]) -> tuple[mx.array, mx.array]: - width = max(len(ls) for ls in label_lists) - ids = np.zeros((len(label_lists), width), dtype=np.int32) - mask = np.zeros((len(label_lists), width), dtype=bool) - for i, ls in enumerate(label_lists): - ids[i, : len(ls)] = ls - mask[i, : len(ls)] = True - return mx.array(ids), mx.array(mask) - - -class Jet: - def __init__( - self, - base_model: str = DEFAULT_BASE_MODEL, - adapter_path: str | None = None, - max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS, - ): - self.model, self.tokenizer = load(base_model, adapter_path=adapter_path) - self.model.eval() - self.max_state_tokens = max_state_tokens - self.temperatures = {"choice": 1.0, "score": 1.0, "noul": 1.0} - # Calibration lives next to the adapter, or inside a fused model dir. A Hub repo id has no dir - # of its own, so look in its local snapshot (fetching calibration.json if the repo has one). - model_dir = Path(base_model) if Path(base_model).exists() else Path(snapshot_download(base_model, allow_patterns=["calibration.json"])) - for directory in (adapter_path, model_dir): - if directory and (calib := Path(directory) / "calibration.json").exists(): - self.temperatures.update(json.loads(calib.read_text())) - break - - def raw_logits(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - """Uncalibrated label logits for (state, question) pairs, plus the number of - input tokens. With shared_state, the common state prefix is encoded once.""" - seqs = [encode(self.tokenizer, s, q, self.max_state_tokens) for s, q in items] - label_lists = [label_token_ids(self.tokenizer, q) for _, q in items] - if shared_state and len(items) > 1: - out = shared_prefix_label_logits(self.model, seqs, label_lists) - else: - tokens, lengths = pad_batch(seqs) - ids, mask = pad_labels(label_lists) - out = np.array(label_logits(self.model, tokens, lengths, ids, mask)) - return [out[i, : len(q.keys)] for i, (_, q) in enumerate(items)], sum(map(len, seqs)) - - def probabilities(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - logits, n_tokens = self.raw_logits(items, shared_state) - result = [] - for (_, q), z in zip(items, logits): - z = z / self.temperatures[q.type] - z = np.exp(z - z.max()) - result.append(z / z.sum()) - return result, n_tokens - - def decide(self, state: Any, questions: dict[str, Question]) -> dict[str, Any]: - """Answer every question about one state together (shared batch).""" - started = time.perf_counter() - items = [(state, q) for q in questions.values()] - probs, input_tokens = self.probabilities(items, shared_state=True) - return { - "answers": {name: summarize(q, p) for (name, q), p in zip(questions.items(), probs)}, - "usage": {"input_tokens": input_tokens}, - "latency_ms": round((time.perf_counter() - started) * 1000, 1), - } diff --git a/docs/training/jet-v5/code/src/train.py b/docs/training/jet-v5/code/src/train.py deleted file mode 100644 index 816c0765fa1a66b174e71ec7b2b95cf360664903..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/code/src/train.py +++ /dev/null @@ -1,304 +0,0 @@ -"""LoRA fine-tuning of a small LM into a Jet decision model (MLX, Apple Silicon). - -The loss is cross-entropy between the soft target and the model's next-token -distribution restricted to the question's label tokens, read at the last prompt -position only - exactly what inference computes. Score questions add a ranked -probability score term, so mass far from the target level costs more than mass -next to it (cross-entropy alone treats the levels as unordered). -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import random -import time -from functools import partial -from pathlib import Path - -import mlx.core as mx -import mlx.nn as nn -import mlx.optimizers as optim -import numpy as np -from mlx.utils import tree_flatten, tree_unflatten -from mlx_lm import load -from huggingface_hub import snapshot_download -from mlx_lm.tuner.trainer import grad_checkpoint -from mlx_lm.tuner.utils import linear_to_lora_layers - -from format import Question, label_token_ids -from model import DEFAULT_BASE_MODEL, encode, label_logits, pad_batch, pad_labels - - -class Example: - __slots__ = ("tokens", "labels", "target", "ordinal", "weight") - - def __init__(self, tokens: list[int], labels: list[int], target: list[float], ordinal: bool, weight: float = 1.0): - self.tokens, self.labels, self.target, self.ordinal = tokens, labels, target, ordinal - self.weight = weight - - -def load_examples(path: Path, tokenizer, max_state_tokens: int, smoothing: float, use_weights: bool = False) -> list[Example]: - out = [] - for line in path.open(): - ex = json.loads(line) - q = Question.from_dict(ex["question"]) - k = len(q.keys) - target = [(1 - smoothing) * t + smoothing / k for t in ex["target"]] - weight = float(ex.get("weight", 1.0)) if use_weights else 1.0 - if not math.isfinite(weight) or weight <= 0: - raise ValueError("training weights must be finite and positive") - out.append(Example(encode(tokenizer, ex["state"], q, max_state_tokens), label_token_ids(tokenizer, q), target, q.type == "score", weight)) - return out - - -def make_batches(examples: list[Example], max_batch_tokens: int, rng: random.Random | None) -> list[list[Example]]: - """Group similar lengths together; each batch's padded size stays under max_batch_tokens.""" - order = sorted(examples, key=lambda e: len(e.tokens)) - batches, cur = [], [] - for ex in order: - width = max(len(ex.tokens), max((len(e.tokens) for e in cur), default=0)) - if cur and width * (len(cur) + 1) > max_batch_tokens: - batches.append(cur) - cur = [] - cur.append(ex) - if cur: - batches.append(cur) - if rng: - rng.shuffle(batches) - return batches - - -def to_arrays(batch: list[Example]): - tokens, lengths = pad_batch([e.tokens for e in batch]) - ids, mask = pad_labels([e.labels for e in batch]) - target = np.zeros(ids.shape, dtype=np.float32) - for i, e in enumerate(batch): - target[i, : len(e.target)] = e.target - ordinal = mx.array([e.ordinal for e in batch]) - levels = mx.array([len(e.labels) for e in batch], dtype=mx.float32) - return tokens, lengths, ids, mask, mx.array(target), ordinal, levels, mx.array([e.weight for e in batch]) - - -def ranked_probability_score(logp: mx.array, target: mx.array, levels: mx.array) -> mx.array: - """Squared distance between predicted and target CDFs, divided by (levels - 1) so it stays - in 0-1 for any scale size. Padded labels have zero mass in both, so they add nothing.""" - cdf_gap = mx.cumsum(mx.exp(logp), axis=-1) - mx.cumsum(target, axis=-1) - return (cdf_gap**2).sum(axis=-1) / (levels - 1) - - -def loss_fn(model, tokens, lengths, ids, mask, target, ordinal, levels, weights, ordinal_weight=0.0): - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - loss = -(target * logp).sum(axis=-1) - if ordinal_weight: - loss = loss + ordinal_weight * mx.where(ordinal, ranked_probability_score(logp, target, levels), 0.0) - return (loss * weights).mean() - - -def evaluate(model, batches: list[list[Example]]) -> dict[str, float]: - total, nll, correct = 0, 0.0, 0 - for batch in batches: - tokens, lengths, ids, mask, target, *_ = to_arrays(batch) - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - nll += float(-(target * logp).sum()) - correct += int((mx.argmax(logits, axis=-1) == mx.argmax(target, axis=-1)).sum()) - total += len(batch) - return {"nll": nll / total, "acc": correct / total} - - -def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | None = None) -> None: - out.mkdir(parents=True, exist_ok=True) - mx.save_safetensors(str(out / "adapters.safetensors"), dict(tree_flatten(model.trainable_parameters()))) - (out / "adapter_config.json").write_text(json.dumps(config, indent=2)) - if optimizer is not None: - # Adam moments + step, so --resume continues exactly instead of restarting cold. - mx.save_safetensors(str(out / "optimizer.safetensors"), dict(tree_flatten(optimizer.state))) - - -def main() -> None: - ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit") - ap.add_argument("--train", type=Path, default=Path("data/train.jsonl")) - ap.add_argument("--val", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--out", type=Path, default=Path("adapters/jet")) - ap.add_argument("--epochs", type=float, default=2.0) - ap.add_argument("--lr", type=float, default=1e-4) - ap.add_argument("--warmup", type=int, default=100) - ap.add_argument("--rank", type=int, default=16) - ap.add_argument("--lora-scale", type=float, default=10.0) - ap.add_argument("--dropout", type=float, default=0.05) - ap.add_argument("--lora-layers", type=int, default=-1, help="-1 = all layers") - ap.add_argument("--max-batch-tokens", type=int, default=4096) - ap.add_argument("--max-state-tokens", type=int, default=1024) - ap.add_argument("--smoothing", type=float, default=0.02) - ap.add_argument("--use-example-weights", action="store_true", - help="use positive per-row weights for training only; validation remains unweighted") - ap.add_argument("--ordinal-weight", type=float, default=2.0, - help="weight of the ranked probability score term on score questions; 0 = plain cross-entropy") - ap.add_argument("--grad-checkpoint", action=argparse.BooleanOptionalAction, default=True) - ap.add_argument("--eval-every", type=int, default=250) - ap.add_argument("--val-limit", type=int, default=1000) - ap.add_argument("--cache-limit-gb", type=float, default=1.0, - help="cap on MLX's freed-buffer cache; variable batch shapes otherwise grow it toward all of RAM") - ap.add_argument("--resume", type=Path, default=None, help="adapter dir to continue from") - ap.add_argument("--init-adapter", type=Path, default=None, - help="warm-start adapter weights with a fresh optimizer and schedule") - ap.add_argument("--start-step", type=int, default=None, - help="step the resumed adapter was saved at; read from its optimizer state when present") - ap.add_argument("--seed", type=int, default=0) - args = ap.parse_args() - if args.init_adapter and (args.resume or args.start_step is not None): - ap.error("--init-adapter cannot be combined with --resume or --start-step") - if (args.out / "adapters.safetensors").exists(): - ap.error("output already contains an adapter; choose a new directory") - - args.out.mkdir(parents=True, exist_ok=True) - run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()} - run_config["data_sha256"] = { - str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val) - } - if args.init_adapter: - source_config = json.loads((args.init_adapter / "adapter_config.json").read_text()) - expected = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - if source_config.get("lora_parameters") != expected: - ap.error("initial adapter LoRA parameters do not match requested parameters") - if source_config.get("base_model") != args.base_model: - ap.error("initial adapter uses a different base model") - if source_config.get("base_revision") not in (None, args.base_revision): - ap.error("initial adapter uses a different base revision") - run_config["init_adapter_sha256"] = hashlib.sha256( - (args.init_adapter / "adapters.safetensors").read_bytes()).hexdigest() - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - mx.set_cache_limit(int(args.cache_limit_gb * 1e9)) - mx.random.seed(args.seed) - rng = random.Random(args.seed) - model_path = args.base_model - if args.base_revision: - model_path = snapshot_download(args.base_model, revision=args.base_revision) - model, tokenizer = load(model_path) - model.freeze() - num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers - lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - linear_to_lora_layers(model, num_layers, lora_parameters) - if args.init_adapter: - if source_config.get("num_layers") != num_layers: - ap.error("initial adapter layer count does not match") - weights = mx.load(str(args.init_adapter / "adapters.safetensors")) - expected_weights = dict(tree_flatten(model.trainable_parameters())) - if weights.keys() != expected_weights.keys() or any( - weights[k].shape != expected_weights[k].shape for k in expected_weights - ): - ap.error("initial adapter tensor names or shapes do not match") - model.load_weights(list(weights.items()), strict=False) - print(f"initialized weights from {args.init_adapter}; optimizer starts fresh", flush=True) - if args.resume: - model.load_weights(str(args.resume / "adapters.safetensors"), strict=False) - print(f"resumed weights from {args.resume}") - if args.grad_checkpoint: - grad_checkpoint(model.layers[0]) - n_train = sum(v.size for _, v in tree_flatten(model.trainable_parameters())) - print(f"trainable params: {n_train / 1e6:.2f}M") - - train = load_examples(args.train, tokenizer, args.max_state_tokens, args.smoothing, args.use_example_weights) - val = load_examples(args.val, tokenizer, args.max_state_tokens, 0.0) - rng.shuffle(val) - val_batches = make_batches(val[: args.val_limit], args.max_batch_tokens, None) - steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None)) - total_steps = math.ceil(steps_per_epoch * args.epochs) - print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches") - run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps) - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - warmup = max(1, min(args.warmup, total_steps // 10)) - base_schedule = optim.join_schedules( - [optim.linear_schedule(1e-7, args.lr, warmup), optim.cosine_decay(args.lr, total_steps - warmup, args.lr * 0.05)], - [warmup], - ) - optimizer_state = args.resume / "optimizer.safetensors" if args.resume else None - if optimizer_state is not None and optimizer_state.exists(): - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.init(model.trainable_parameters()) - optimizer.state = tree_unflatten(list(mx.load(str(optimizer_state)).items())) - start_step = int(optimizer.state["step"].item()) - print(f"restored optimizer state at step {start_step}") - else: - start_step = args.start_step or 0 - if args.resume: - # Cold Adam moments (MLX AdamW has no bias correction by default) make the first - # updates several times too large; ramp the LR back up to protect the checkpoint. - def schedule(step, start=start_step): - return base_schedule(step) * mx.minimum(1.0, (step - start + 1).astype(mx.float32) / warmup) - print(f"no optimizer state in {args.resume}; re-warming the LR over {warmup} steps") - else: - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.state["step"] = mx.array(start_step, mx.uint64) - value_and_grad = nn.value_and_grad(model, partial(loss_fn, ordinal_weight=args.ordinal_weight)) - - adapter_config = { - "fine_tune_type": "lora", - "num_layers": num_layers, - "lora_parameters": lora_parameters, - "base_model": args.base_model, - "base_revision": args.base_revision, - } - model.eval() - best = evaluate(model, val_batches) - # Keep the initial checkpoint eligible; a regression must not replace it merely - # because no trained checkpoint improved the validation loss. - save_adapter(model, args.out, adapter_config, optimizer) - (args.out / "metrics.jsonl").write_text(json.dumps({"step": 0, **best, - "improved": False, "initial": True}) + "\n") - print(f"step 0 val nll {best['nll']:.4f} acc {best['acc']:.3f}") - model.train() - - saved = False - step, seen_tokens, started = start_step, 0, time.perf_counter() - losses: list[float] = [] - while step < total_steps: - for batch in make_batches(train, args.max_batch_tokens, rng): - if step >= total_steps: - break - arrays = to_arrays(batch) - loss, grads = value_and_grad(model, *arrays) - optimizer.update(model, grads) - mx.eval(model.trainable_parameters(), optimizer.state, loss) - step += 1 - losses.append(loss.item()) - seen_tokens += int(arrays[1].sum()) - if step % 25 == 0: - elapsed = time.perf_counter() - started - print( - f"step {step}/{total_steps} loss {np.mean(losses):.4f} lr {optimizer.learning_rate.item():.2e} " - f"{seen_tokens / elapsed:.0f} tok/s peak mem {mx.get_peak_memory() / 1e9:.1f}GB cache {mx.get_cache_memory() / 1e9:.1f}GB", - flush=True, - ) - losses = [] - if step % args.eval_every == 0 or step == total_steps: - model.eval() - metrics = evaluate(model, val_batches) - model.train() - improved = metrics["nll"] < best["nll"] - print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{' *saved*' if improved else ''}", flush=True) - with (args.out / "metrics.jsonl").open("a") as f: - f.write(json.dumps({"step": step, **metrics, "improved": improved, - "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n") - save_adapter(model, args.out / "last", adapter_config, optimizer) - if improved: - best = metrics - save_adapter(model, args.out, adapter_config, optimizer) - saved = True - if not saved: - print("validation never improved; retaining initial weights (final weights in last/)") - print(f"best val nll {best['nll']:.4f} acc {best['acc']:.3f}; adapter in {args.out}") - - -if __name__ == "__main__": - main() diff --git a/docs/training/jet-v5/completion-audit.json b/docs/training/jet-v5/completion-audit.json deleted file mode 100644 index 22738ba898d1880f3f354b0464dc104dc2b46689..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/completion-audit.json +++ /dev/null @@ -1,187 +0,0 @@ -{ - "environment": { - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "packages": { - "mlx": "0.32.2", - "mlx-lm": "0.31.3", - "numpy": "2.5.3", - "transformers": "5.17.0", - "huggingface-hub": "1.32.0", - "decision-index": "0.1.0" - }, - "gpu": "NVIDIA GeForce RTX 4080 SUPER, 615.71.09, 16376 MiB" - }, - "selected_step": 3500, - "total_steps": 3570, - "adapter_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration": { - "choice": 1.0, - "score": 1.2968, - "noul": 1.4983 - }, - "compatibility_ok": { - "diagnostic": 26, - "v5-diagnostic-1400": 14 - }, - "whole_session_gpu_peak_mib": 9909, - "training_config": { - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v5_r2.jsonl", - "val": "data/selection_v5.jsonl", - "out": "adapters/jet-v5-panel02-r2-20260924", - "epochs": 1.0, - "lr": 1e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 3072, - "max_state_tokens": 2048, - "smoothing": 0.02, - "use_example_weights": true, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet-v4-transfer-20260924", - "start_step": null, - "seed": 240925, - "data_sha256": { - "data/train_v5_r2.jsonl": "62f33bfcad77c53d86d7097975bb52255b663d4c6f13b0e88025cb3df1d18774", - "data/selection_v5.jsonl": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3" - }, - "init_adapter_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "train_examples": 15997, - "steps_per_epoch": 3570, - "total_steps": 3570 - }, - "code_sha256": { - "src/train.py": "040ef12d706db397d6031a745faf9fc922497e6eb59f0d45b0ac18cd13b8cbfd", - "src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "src/evaluate.py": "25dcaf7bcf9ddcfdd8349eeea047ef7cc33388cf74f32a65c8a1931b4f40f72c", - "src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "src/decision_index_ensemble.py": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1", - "scripts/download_v5_sources.py": "58722a9a442f72693f83f23808247324c76f20fd0e3ba1a2a9ffc4d845142b94", - "scripts/build_v5.py": "5f915e061c536400fe79b42d5c0980445e160ab036ec5a5132ddf809cc26f1c9", - "scripts/audit_v5.py": "4596800e8b0629222ee5c780f2630e27c8218f78b0ca31caaa49cf905694fd6e", - "scripts/evaluate_v5_order.py": "c4436a1994a27473b04c9aae131ffb5bdb3ed7303a21c7507ce39dae6f5cf70c", - "scripts/rebuild_v5_diagnostic.py": "5c08c56a09a811195d1caaf916987d5a0e20c6deb4cbcb69805f4aaaeb4746cc", - "scripts/summarize_v5.py": "48834200e9c6d2874e6b12982380167cebd9faf2a4b523014f816c09ec0f85ca", - "scripts/filter_v5_retention.py": "d4ae571e0ecfd8090353139ed925071a27a4ed3d7f27f6d1c7635687e8a19b6a", - "scripts/finalize_v5.py": "20f841219ff97f5c02068a86867677f1ad36a9df12d261c13604408ebd090600", - "scripts/run_v5_cuda.sh": "c79834e70e104dd978daa8005d60d0ed5641ca44893708cd5760566e870eacc8", - "scripts/run_v5_extra.sh": "9a6f9471fdc566cce479d9ce6eb257aeefa830ab06b06c19fededc86d69f13b6", - "tests/test_decision_ensemble.py": "6bdf9dce1d41d2d6998fbac3cc2df362d57eb3e0748c00cf5023a0c9b3c16b91" - }, - "dataset": { - "rule": "Audit native-evaluation overlap in retained examples and create immutable R2 data.\n\nCatalog names and isolated short schema fields are not treated as utterances.\nV5 source content identities, complete text states and text fragments >=32\nnormalized characters are compared; inherited exposure is reported explicitly.\n", - "files": { - "train_v5": { - "sha256": "d96100fd362255482869fdfc96d02cc6d3071326155500f817fca5fdbc53d3fd", - "kept": 15997, - "removed": 3, - "removed_sources": { - "v3:sarcasm_irony": 1, - "massive": 2 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9" - ] - }, - "selection_v5": { - "sha256": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3", - "kept": 1400, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - }, - "calibration_v5": { - "sha256": "2c24ff80f07c60534f9c30adcba5fc9e36c97319e27ad262ed68befdd7dd2d1a", - "kept": 1400, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - }, - "test_v5_new": { - "sha256": "6b060a9d924a6c0711a13f8e2406ffc1b163ae80b158abd64f8288928495b9e6", - "kept": 600, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - } - }, - "inherited_exposure": { - "train_v2": { - "count": 8, - "by_source": { - "massive": 8 - }, - "state_hashes": [ - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "92dde7a604a103c21c35ecfb6ca201684031767adf42c84db55b881a5de129f8", - "b49177e05868b7af8e82a644c1ce20e521af46497adeaffe861d294d9b4bb75e", - "1efe129d040aa8d64ca7dc438f4cb43876757818864f09ae7cb53a23d1c9f08f", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "0b41b183ab200019e05ae6b8bfdf638ce2d90d39269a33ffcc94b9b035fddfb5", - "870d2ccc7ede8f14ddba0b441af3cafbc60a548e339e41d5a061445f1988b88d", - "262c9bf9846c7d15b6c044f42ca9e7d6f0e5e524b31671d54607762c57072ffb" - ] - }, - "train_v3": { - "count": 13, - "by_source": { - "v3:sarcasm_irony": 1, - "massive": 8, - "v3:tool_selection": 4 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "92dde7a604a103c21c35ecfb6ca201684031767adf42c84db55b881a5de129f8", - "b49177e05868b7af8e82a644c1ce20e521af46497adeaffe861d294d9b4bb75e", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "262c9bf9846c7d15b6c044f42ca9e7d6f0e5e524b31671d54607762c57072ffb", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "0b41b183ab200019e05ae6b8bfdf638ce2d90d39269a33ffcc94b9b035fddfb5", - "1efe129d040aa8d64ca7dc438f4cb43876757818864f09ae7cb53a23d1c9f08f", - "870d2ccc7ede8f14ddba0b441af3cafbc60a548e339e41d5a061445f1988b88d", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090" - ] - }, - "train_v4_tools_focus": { - "count": 5, - "by_source": { - "v3:sarcasm_irony": 1, - "v4:tool_intent_routing": 2, - "massive": 2 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "55769c01f1927d87fbf132c23775213854fdee7156937b41dc42ec02af6ad6d1", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "7af3840b205f11ba55c61fe5d1fc2e4bf93ff5c5f99b4543d00ea59c9408fbbe", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1" - ] - } - }, - "output": { - "path": "/home/jach/dev/jet/data/train_v5_r2.jsonl", - "sha256": "62f33bfcad77c53d86d7097975bb52255b663d4c6f13b0e88025cb3df1d18774", - "rows": 15997 - }, - "caveat": "R2 removes new-stage overlap but cannot erase inherited V4 training exposure; do not claim a fully decontaminated checkpoint or an official score." - }, - "published_model_unchanged": true, - "old_validation_test_unchanged": true, - "scope": "Partial diagnostics; inherited overlap caveat applies. No official Decision Index 0.2 score." -} diff --git a/docs/training/jet-v5/data-audit.json b/docs/training/jet-v5/data-audit.json deleted file mode 100644 index 5e48bf71b56403e0bab9b9b49cfaea0478780785..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/data-audit.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "verified_native_labels": 9800, - "counts": { - "train:wide_intent": 2000, - "train:commonsense_completion": 2000, - "train:adversarial_entailment": 2000, - "train:tool_response_preference": 2000, - "val:wide_intent": 150, - "val:commonsense_completion": 150, - "val:adversarial_entailment": 150, - "val:tool_response_preference": 150, - "calibration:wide_intent": 150, - "calibration:commonsense_completion": 150, - "calibration:adversarial_entailment": 150, - "calibration:tool_response_preference": 150, - "test:adversarial_entailment": 150, - "test:tool_response_preference": 150, - "test:commonsense_completion": 150, - "test:wide_intent": 150 - }, - "source_id_duplicates": 0, - "cross_split_related_groups": 0 -} diff --git a/docs/training/jet-v5/esci-selected.json b/docs/training/jet-v5/esci-selected.json deleted file mode 100644 index f1b59b408af7b3df38d9bb3dfbf12599c247de5f..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/esci-selected.json +++ /dev/null @@ -1,160 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "d657d1f6ad0667b6183cbe647c6be689fe4d9575f0f14a5865be5cce2e87df33", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_ensemble:TwoOrderJetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9844662850000532, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 99.04277550003826, - "p95": 134.007656999529, - "mean": 97.88671671199336 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.3860969319795632, - "reference_same_cases": null, - "median_ms": 99.04277550003826, - "detail": { - "field_accuracy": 0.526, - "case_exact_accuracy": 0.526, - "macro_f1": 0.3860969319795632, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.3860969319795632, - "skill": 0.2299859612172467, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3860969319795632, - "random": 0.20273782411694521, - "skill": 0.2299859612172467, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/esci-single.json b/docs/training/jet-v5/esci-single.json deleted file mode 100644 index d12626b9c58881e6a2bf0b57133f4eb2ba0ccecd..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/esci-single.json +++ /dev/null @@ -1,158 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea", - "results_sha256": "9eb7445408ee098e7c9de5cf9da22c6031be0abbd6d400064a929c4b17e13c73", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9938386029998583, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/esci-diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 69.47628650004845, - "p95": 87.4697710005421, - "mean": 63.29981445199337 - }, - "benchmarks": [ - { - "catalog_id": 37, - "dataset": "Amazon ESCI", - "requests": 500, - "answered": 500, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 500, - "metric": "macro-F1", - "score": 0.38382787678478314, - "reference_same_cases": null, - "median_ms": 69.47628650004845, - "detail": { - "field_accuracy": 0.544, - "case_exact_accuracy": 0.544, - "macro_f1": 0.38382787678478314, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "37": { - "catalog_id": 37, - "raw": 0.38382787678478314, - "skill": 0.22713990221254501, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.38382787678478314, - "random": 0.20273782411694521, - "skill": 0.22713990221254501, - "planned_cases": 500, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"Complement: a product that complements the requested product.\"", - "\"Exact: the product satisfies the search query.\"", - "\"Irrelevant: the product does not address the requested product need.\"", - "\"Substitute: a product that could substitute for the requested product.\"" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/evaluation-code-manifest.json b/docs/training/jet-v5/evaluation-code-manifest.json deleted file mode 100644 index c15981e56aa6333602ab45199aee705554cd9890..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code-manifest.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "recorded_utc": "2026-09-24T08:31:31.253213+00:00", - "sha256": { - "scripts/run_v5_extra.sh": "9a6f9471fdc566cce479d9ce6eb257aeefa830ab06b06c19fededc86d69f13b6", - "scripts/evaluate_v5_order.py": "c4436a1994a27473b04c9aae131ffb5bdb3ed7303a21c7507ce39dae6f5cf70c", - "src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "src/decision_index_ensemble.py": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1", - "src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "scripts/summarize_v5.py": "48834200e9c6d2874e6b12982380167cebd9faf2a4b523014f816c09ec0f85ca", - "scripts/finalize_v5.py": "20f841219ff97f5c02068a86867677f1ad36a9df12d261c13604408ebd090600" - } -} diff --git a/docs/training/jet-v5/evaluation-code/scripts/evaluate_v5_order.py b/docs/training/jet-v5/evaluation-code/scripts/evaluate_v5_order.py deleted file mode 100644 index aa349714980cc0928f85ea4e52fa29c492ed6bd2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/scripts/evaluate_v5_order.py +++ /dev/null @@ -1,43 +0,0 @@ -"""Choose single versus two-order readout using selection data only.""" -import argparse,json,random,time,hashlib -from collections import defaultdict -from pathlib import Path -import numpy as np -from decision_index_ensemble import TwoOrderJetEngine - -def main(): - ap=argparse.ArgumentParser();ap.add_argument('--adapter',required=True);args=ap.parse_args() - rows=[json.loads(x) for x in Path('data/selection_v5.jsonl').read_text().splitlines()] - groups=defaultdict(list) - for r in rows: - if r['question']['type']=='choice':groups[r.get('family',r['source'])].append(r) - for k in sorted(groups):random.Random(240925).shuffle(groups[k]) - selected=[] - while groups and len(selected)<600: - for k in list(groups): - selected.append(groups[k].pop()) - if not groups[k]:del groups[k] - if len(selected)==600:break - engine=TwoOrderJetEngine(model='mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8',adapter=args.adapter,max_tokens=8192,max_cached_tokens=8192) - results=[];started=time.monotonic() - for i,r in enumerate(selected): - response,raw=engine(r['state'],{'q':r['question']}) - keys=list(r['question']['criteria']);target=np.array(r['target']) - scores={} - for name,answer in [('single',raw['single_order_answers']['q']),('two_order',response['answers']['q'])]: - p=np.array([answer['probabilities'][k] for k in keys]) - scores[name]={'nll':float(-(target*np.log(np.clip(p,1e-12,1))).sum()),'correct':int(p.argmax()==target.argmax())} - results.append({'family':r.get('family',r['source']),'scores':scores}) - if (i+1)%25==0:print(i+1,len(selected),flush=True) - summary={} - for name in ['single','two_order']: - families=defaultdict(list) - for r in results:families[r['family']].append(r['scores'][name]) - summary[name]={'nll':float(np.mean([r['scores'][name]['nll'] for r in results])), - 'accuracy':float(np.mean([r['scores'][name]['correct'] for r in results])), - 'macro_family_accuracy':float(np.mean([np.mean([r['correct'] for r in rs]) for rs in families.values()]))} - choose=summary['two_order']['nll']=summary['single']['macro_family_accuracy'] - report={'n':len(results),'selection_sha256':hashlib.sha256(Path('data/selection_v5.jsonl').read_bytes()).hexdigest(),'selection_rule':'lower NLL and no decrease in macro family accuracy on selection data only','selected':'two_order' if choose else 'single','summary':summary,'seconds':time.monotonic()-started,'engine_provenance':engine.provenance,'rows':results} - Path('docs/training/jet-v5/inference-selection.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps({k:v for k,v in report.items() if k not in ['rows','engine_provenance']},indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/evaluation-code/scripts/finalize_v5.py b/docs/training/jet-v5/evaluation-code/scripts/finalize_v5.py deleted file mode 100644 index de05230634bb146164b5b73f3a1cea6f62d148ae..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/scripts/finalize_v5.py +++ /dev/null @@ -1,48 +0,0 @@ -"""Verify V5 completion, original-file preservation, and selected artifact hashes.""" -import csv,hashlib,json,math,shutil,subprocess,platform -from importlib.metadata import version -from pathlib import Path -from safetensors import safe_open -ROOT=Path(__file__).resolve().parents[1] -D=ROOT/'docs/training/jet-v5' -def sha(p): - with p.open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() -def main(): - adapter=ROOT/'adapters/jet-v5-panel02-r2-20260924' - config=json.loads((adapter/'training_config.json').read_text()) - metrics=[json.loads(x) for x in (adapter/'metrics.jsonl').read_text().splitlines()] - assert metrics[-1]['step']==config['total_steps'] - best=min(metrics,key=lambda r:r['nll']) - with safe_open(adapter/'optimizer.safetensors',framework='numpy') as f:step=int(f.get_tensor('step')) - assert step==best['step'],(step,best) - temperatures=json.loads((adapter/'calibration.json').read_text()) - assert all(math.isfinite(t) and t>0 for t in temperatures.values()) - data=json.loads((D/'retention-audit.json').read_text()) - assert sha(ROOT/'data/train_v5_r2.jsonl')==data['output']['sha256'] - for name,meta in data['files'].items():assert sha(ROOT/f'data/{name}.jsonl')==meta['sha256'] - original_provenance=json.loads((ROOT/'data/train_v5.provenance.json').read_text()) - for path,digest in original_provenance['protected'].items():assert sha(Path(path))==digest,path - for name in ['protected','published']: - subprocess.run(['sha256sum','-c',str(ROOT/f'work/jet-v3/{name}.sha256')],cwd=ROOT,check=True) - counts={} - for suite in ['diagnostic','v5-diagnostic-1400']: - path=ROOT/f'artifacts/decision-index/runs/v5-compatibility-{suite}/results.jsonl' - rows=[json.loads(s) for s in path.read_text().splitlines()] - statuses=[r['status'] for r in rows];assert all(s=='ok' for s in statuses),statuses - counts[suite]=len(rows) - code=D/'final-code';code.mkdir(exist_ok=True) - names=['src/train.py','src/model.py','src/evaluate.py','src/format.py','src/inference.py','src/decision_index_engine.py','src/decision_index_ensemble.py'] - names += [str(p.relative_to(ROOT)) for p in (ROOT/'scripts').glob('*v5*.py')] - names += ['scripts/run_v5_cuda.sh','scripts/run_v5_extra.sh','tests/test_decision_ensemble.py'] - code_hashes={} - for name in names: - p=ROOT/name;q=code/name;q.parent.mkdir(parents=True,exist_ok=True);shutil.copy2(p,q);code_hashes[name]=sha(p) - gpu=list(csv.DictReader((ROOT/'logs/jet-v5/gpu.csv').open())) - peak=max(int(r[' memory.used [MiB]'].strip().split()[0]) for r in gpu) - environment={'platform':platform.platform(),'packages':{p:version(p) for p in ['mlx','mlx-lm','numpy','transformers','huggingface-hub','decision-index']},'gpu':subprocess.check_output(['nvidia-smi','--query-gpu=name,driver_version,memory.total','--format=csv,noheader'],text=True).strip()} - report={'environment':environment,'selected_step':step,'total_steps':config['total_steps'],'adapter_sha256':sha(adapter/'adapters.safetensors'),'calibration':temperatures,'compatibility_ok':counts,'whole_session_gpu_peak_mib':peak,'training_config':config,'code_sha256':code_hashes,'dataset':data,'published_model_unchanged':True,'old_validation_test_unchanged':True,'scope':'Partial diagnostics; inherited overlap caveat applies. No official Decision Index 0.2 score.'} - (D/'completion-audit.json').write_text(json.dumps(report,indent=2)+'\n') - out=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs') - shutil.copy2(D/'completion-audit.json',out/'jet-v5-run-manifest.json') - print(json.dumps({k:v for k,v in report.items() if k not in ['training_config','code_sha256','dataset']},indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/evaluation-code/scripts/run_v5_extra.sh b/docs/training/jet-v5/evaluation-code/scripts/run_v5_extra.sh deleted file mode 100644 index 8235956881557027c7d5e0622f6da599594f3ec8..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/scripts/run_v5_extra.sh +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env bash -# Run only after run_v5_cuda.sh finishes; uses the GPU sequentially. -set -euo pipefail -cd "$(dirname "$0")/.." -export CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 MLX_CUDA_SDPA_CACHE_SIZE=2048 -export MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" PYTHONPATH=src PYTHONUNBUFFERED=1 -.venv/bin/python scripts/evaluate_v5_order.py --adapter adapters/jet-v5-panel02-r2-20260924 > logs/jet-v5/inference-selection.log 2>&1 -ENGINE=decision_index_engine:JetEngine -if .venv/bin/python -c 'import json,sys;sys.exit(json.load(open("docs/training/jet-v5/inference-selection.json"))["selected"]!="two_order")'; then - ENGINE=decision_index_ensemble:TwoOrderJetEngine -fi -for SUITE in diagnostic v5-diagnostic-1400; do - .venv/bin/python -m decision_index run --engine "$ENGINE" \ - --rows "artifacts/decision-index/$SUITE/compatibility.jsonl.gz" \ - --out "artifacts/decision-index/runs/v5-compatibility-$SUITE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/compatibility-$SUITE.log" 2>&1 -done - .venv/bin/jet-eval --base-model models/jet --data data/test_v5_new.jsonl --batch-size 2 \ - --max-state-tokens 2048 --json docs/training/jet-v5/baseline-test_v5_new.json > logs/jet-v5/baseline-test_v5_new.log 2>&1 -for NAME in baseline v4 v5; do - ADAPTER=adapters/jet-v4-transfer-20260924 - if [[ "$NAME" == v5 ]]; then ADAPTER=adapters/jet-v5-panel02-r2-20260924; fi - OPTIONS=(--option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$ADAPTER") - if [[ "$NAME" == baseline ]]; then - OPTIONS=(--option model=michaljach/jet --option revision=8a97cfea2df622bb03f5dc9b02567e21abd2551c) - fi - .venv/bin/python -m decision_index run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/v5-expanded-$NAME" \ - "${OPTIONS[@]}" --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$NAME-expanded.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/v5-expanded-$NAME/results.jsonl" --out "docs/training/jet-v5/$NAME-expanded.json" > "logs/jet-v5/$NAME-expanded-report.log" 2>&1 -done -if .venv/bin/python -c 'import json,sys;sys.exit(json.load(open("docs/training/jet-v5/inference-selection.json"))["selected"]!="two_order")'; then - for PAIR in 'diagnostic/diagnostic.jsonl.gz v5-two-order-original' 'v5-diagnostic-1400/diagnostic.jsonl.gz v5-two-order-expanded'; do - read -r ROWS RUN <<< "$PAIR" - .venv/bin/python -m decision_index run --engine decision_index_ensemble:TwoOrderJetEngine \ - --rows "artifacts/decision-index/$ROWS" --out "artifacts/decision-index/runs/$RUN" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$RUN.log" 2>&1 - .venv/bin/jet-bench-index report --rows "artifacts/decision-index/$ROWS" \ - --results "artifacts/decision-index/runs/$RUN/results.jsonl" --out "docs/training/jet-v5/$RUN.json" > "logs/jet-v5/$RUN-report.log" 2>&1 - done -fi -for MODE in single selected; do - TARGET=decision_index_engine:JetEngine - if [[ "$MODE" == selected ]]; then - [[ "$ENGINE" != decision_index_engine:JetEngine ]] || continue - TARGET="$ENGINE" - fi - .venv/bin/python -m decision_index run --engine "$TARGET" \ - --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/v5-code-$MODE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/code-$MODE.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/v5-code-$MODE/results.jsonl" --out "docs/training/jet-v5/code-$MODE.json" > "logs/jet-v5/code-$MODE-report.log" 2>&1 -done -for ENTRY in 'esci-diagnostic/diagnostic.jsonl.gz esci' 'isarcasm-english-diagnostic.jsonl.gz sarcasm'; do - read -r ROWS LABEL <<< "$ENTRY" - for MODE in single selected; do - TARGET=decision_index_engine:JetEngine - if [[ "$MODE" == selected ]]; then - [[ "$ENGINE" != decision_index_engine:JetEngine ]] || continue - TARGET="$ENGINE" - fi - .venv/bin/python -m decision_index run --engine "$TARGET" \ - --rows "artifacts/decision-index/$ROWS" --out "artifacts/decision-index/runs/v5-$LABEL-$MODE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$LABEL-$MODE.log" 2>&1 - .venv/bin/jet-bench-index report --rows "artifacts/decision-index/$ROWS" \ - --results "artifacts/decision-index/runs/v5-$LABEL-$MODE/results.jsonl" --out "docs/training/jet-v5/$LABEL-$MODE.json" > "logs/jet-v5/$LABEL-$MODE-report.log" 2>&1 - done -done -printf 'V5 expanded evaluation complete.\n' diff --git a/docs/training/jet-v5/evaluation-code/scripts/summarize_v5.py b/docs/training/jet-v5/evaluation-code/scripts/summarize_v5.py deleted file mode 100644 index ccb457ed8c6c81681c576e0b6d3a6bed5847ef4d..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/scripts/summarize_v5.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Summarize completed V5 evaluation without inventing a leaderboard score.""" -import hashlib,json,shutil -from pathlib import Path -ROOT=Path(__file__).resolve().parents[1] -D=ROOT/'docs/training/jet-v5' -def read(p):return json.loads(p.read_text()) -def sha(p): - with p.open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() -def main(): - adapter=ROOT/'adapters/jet-v5-panel02-r2-20260924' - config=read(adapter/'training_config.json');metrics=[json.loads(x) for x in (adapter/'metrics.jsonl').read_text().splitlines()] - assert metrics[-1]['step']==config['total_steps'],'Training incomplete' - order=read(D/'inference-selection.json') - released_new=read(D/'baseline-test_v5_new.json') - held={name:{ds:read(D/f'{name}-{ds}.json') for ds in ['test','test_v5_new','test_v4_new','test_v4_tools']} for name in ['v4','v5']} - bench={} - for name in ['v4','v5']: - bench[name]={} - for suite in ['diagnostic','expanded']: - result=read(D/f'{name}-{suite}.json');assert result['complete_for_supplied_rows'] and result['pending_requests']==0 - for b in result['summary']['benchmarks']:bench[name][b['dataset']]=b - if order['selected']=='two_order': - bench['v5_two_order']={} - for suite in ['original','expanded']: - result=read(D/f'v5-two-order-{suite}.json');assert result['complete_for_supplied_rows'] - for b in result['summary']['benchmarks']:bench['v5_two_order'][b['dataset']]=b - regressions={} - for label,old_prefix in [('code','cruxeval-full'),('esci','esci'),('sarcasm','isarcasm-en')]: - runs={'baseline':read(ROOT/f'docs/training/jet-next/{old_prefix}-baseline.json'),'v4':read(ROOT/f'docs/training/jet-next/{old_prefix}-v4.json'),'v5':read(D/f'{label}-single.json')} - if order['selected']=='two_order':runs['v5_two_order']=read(D/f'{label}-selected.json') - assert len({r['rows_sha256'] for r in runs.values()})==1, label - assert all(r['complete_for_supplied_rows'] for r in runs.values()), label - scores={} - for name,r in runs.items(): - scores[name]=(next(t['raw'] for t in r['panel_metrics_on_sample']['40']['tracks'] if t['track']=='iSarcasmEval-A-En') if label=='sarcasm' else r['summary']['benchmarks'][0]['score']) - regressions[label]={'scores':scores,'n':runs['v5']['planned_requests'],'rows_sha256':runs['v5']['rows_sha256']} - best=min(metrics,key=lambda r:r['nll']) - lines=['# Jet V5 results — Qwen3-0.6B','', - 'V5 was trained locally on 15,997 examples: 7,997 retained V4 rows plus 2,000 each for CLINC150, ANLI, HellaSwag, and When2Call response preference. Native training partitions only; new selection, calibration and test groups are separate.','', - '**These are partial diagnostics, not an official Decision Index 0.2 score or a leaderboard win.** The current board scores 40 benchmarks; the public reproduction kit still specifies the older panel. No submission was made.','', - 'R2 removes three retained short-text overlaps from continuation data. The V4 initialization still has inherited cross-dataset content overlap and is not certified decontaminated. See retention-audit.json.', '', - f'Selected training step: {best["step"]}/{config["total_steps"]}; selection NLL {best["nll"]:.4f}, accuracy {100*best["acc"]:.2f}%. Rank 16, LR 1e-5, one epoch, 3,072 batch tokens, 2,048 training-state tokens.','', - '| Held-out accuracy (%) | V4 | V5 | Change (points) |','|---|---:|---:|---:|'] - for ds in ['test','test_v5_new','test_v4_new','test_v4_tools']: - a=held['v4'][ds]['overall']['acc'];b=held['v5'][ds]['overall']['acc'];lines.append(f'| {ds} | {100*a:.2f} | {100*b:.2f} | {100*(b-a):+.2f} |') - lines+=['','| New held-out family | Released Jet | V4 | V5 |','|---|---:|---:|---:|'] - for family,a in held['v4']['test_v5_new']['by_source'].items(): - b=held['v5']['test_v5_new']['by_source'][family];lines.append(f'| {family} | {100*released_new["by_source"][family]["acc"]:.2f} | {100*a["acc"]:.2f} | {100*b["acc"]:.2f} |') - lines+=['','## Native benchmark metrics on fixed partial samples','', - '| Benchmark | Metric | Requests | Released Jet | V4 | V5 | V5 two-order |','|---|---|---:|---:|---:|---:|---:|'] - def fmt(x):return '—' if x is None else f'{100*x:.2f}' - baseline_bench={} - for path in [ROOT/'docs/training/jet-v3/diagnostic-baseline.json',D/'baseline-expanded.json']: - if path.exists(): - for b in read(path)['summary']['benchmarks']:baseline_bench[b['dataset']]=b - for name,a in bench['v4'].items(): - b=bench['v5'][name];c=bench.get('v5_two_order',{}).get(name,{}) - lines.append(f'| {name} | {a["metric"]} | {a["requests"]} | {fmt(baseline_bench.get(name,{}).get("score"))} | {fmt(a["score"])} | {fmt(b["score"])} | {fmt(c.get("score"))} |') - lines+=['','## Fixed regression checks','', '| Benchmark | Requests | Released Jet | V4 | V5 | V5 two-order |','|---|---:|---:|---:|---:|---:|'] - for label,item in regressions.items(): - values=item['scores'];lines.append(f'| {label} | {item["n"]} | {fmt(values["baseline"])} | {fmt(values["v4"])} | {fmt(values["v5"])} | {fmt(values.get("v5_two_order"))} |') - lines+=['','All reported values are native metrics multiplied by 100. A missing aggregate remains missing. Samples are small and results are descriptive, not statistical significance claims. MMLU and ARC are diagnostic-only and are not in the current 0.2 index panel.','', - f'Option-order policy chosen on {order["n"]} selection-only choice rows: **{order["selected"]}**. The two-order engine averages original/reversed probabilities using exact option keys; it keeps complete inputs and approximately doubles forward-pass work. Selection required lower NLL without lower macro family accuracy.','', - 'The first V5 attempt was stopped because three retained utterances overlapped native evaluation content. R2 removes them from continuation data. The V4 initialization still has inherited cross-dataset content overlap; this checkpoint is not certified decontaminated. See retention-audit.json.', '', - 'Single-order metrics remain separately reported. No test or benchmark score selected training steps, temperatures, or the option-order policy. Existing validation/test files and published weights were preserved.','', - 'The data protocol, source revisions and limitations are recorded in protocol.md and data/train_v5.provenance.json. Native annotations and group/source-ID separation were independently audited.'] - (D/'summary.md').write_text('\n'.join(lines)+'\n') - result={'selected_checkpoint':best,'training_config':config,'adapter_sha256':sha(adapter/'adapters.safetensors'),'calibration':read(adapter/'calibration.json'),'heldout':held,'released_new_holdout':released_new,'released_benchmarks':baseline_bench,'benchmarks':bench,'regressions':regressions,'inference_selection':order,'data_provenance_sha256':sha(ROOT/'data/train_v5.provenance.json'),'retention_audit':read(D/'retention-audit.json'),'r2_training_sha256':sha(ROOT/'data/train_v5_r2.jsonl')} - (D/'results.json').write_text(json.dumps(result,indent=2)+'\n') - out=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs');out.mkdir(parents=True,exist_ok=True) - shutil.copy2(D/'summary.md',out/'jet-v5-report.md');shutil.copy2(D/'results.json',out/'jet-v5-results.json') - print('\n'.join(lines)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/evaluation-code/src/decision_index_engine.py b/docs/training/jet-v5/evaluation-code/src/decision_index_engine.py deleted file mode 100644 index 71ce9ff6f10fb7875fb90cab6eb3bffe80861da9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/src/decision_index_engine.py +++ /dev/null @@ -1,133 +0,0 @@ -"""Jet's native MLX readout through the official Decision Index engine API. - -Install with ``uv sync --extra benchmark``. No benchmark identity or gold labels -are accepted by this adapter. The full native prompt is preserved, or rejected. -""" -from __future__ import annotations - -import hashlib -import json -import math -from pathlib import Path - -import numpy as np -from decision_index.engines import Engine, Unsupported -from huggingface_hub import snapshot_download -from transformers import AutoTokenizer - -from format import MAX_CHOICE_OPTIONS, Question, build_prompt, label_token_ids - -MODEL_ID = "michaljach/jet" -REVISION = "8a97cfea2df622bb03f5dc9b02567e21abd2551c" -DEFAULT_MAX_TOKENS = 8192 - - -def as_text(value): - """Mechanical JSON rendering for structured instructions/descriptions.""" - return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False) - - -def prepare_request(tokenizer, state, questions, max_tokens): - prepared = [] - for name, original in questions.items(): - if original["type"] != "choice": - raise Unsupported(f"Only suite choice questions supported: {original['type']}") - criteria = original["criteria"] - if not 2 <= len(criteria) <= MAX_CHOICE_OPTIONS: - raise Unsupported(f"Option count {len(criteria)} outside 2..{MAX_CHOICE_OPTIONS}") - q = Question.from_dict({ - "type": "choice", "instructions": as_text(original["instructions"]), - "criteria": {key: as_text(value) for key, value in criteria.items()}, - }) - # Do not call inference.encode: its application default truncates states. - ids = tokenizer.encode(build_prompt(tokenizer, state, q), add_special_tokens=False) - if len(ids) > max_tokens: - raise Unsupported(f"Full prompt has {len(ids)} tokens; declared limit {max_tokens}") - prepared.append((name, q, ids, label_token_ids(tokenizer, q))) - return prepared - - -def load_assets(model=MODEL_ID, revision=REVISION): - path = Path(snapshot_download(model, revision=revision, allow_patterns=[ - "config.json", "calibration.json", "tokenizer*", "special_tokens_map.json", - "added_tokens.json", "vocab.json", "merges.txt", "chat_template*", - ])) - return path, AutoTokenizer.from_pretrained(path), json.loads((path / "config.json").read_text()) - - -class JetEngine(Engine): - name = "jet" - latency = "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - - def __init__(self, model=MODEL_ID, revision=REVISION, max_tokens=DEFAULT_MAX_TOKENS, - max_cached_tokens=32768, adapter=None, **options): - super().__init__(**options) - import mlx.core as mx - from mlx_lm import load - - self.mx = mx - mx.set_cache_limit(1_000_000_000) - path, self.tokenizer, config = load_assets(model, revision) - self.max_tokens = int(max_tokens) - self.max_cached_tokens = int(max_cached_tokens) - if not 1 <= self.max_tokens <= config["max_position_embeddings"]: - raise ValueError("max_tokens must be positive and within the checkpoint context window") - if self.max_cached_tokens < 1: - raise ValueError("max_cached_tokens must be positive") - calibration = Path(adapter) / "calibration.json" if adapter else path / "calibration.json" - self.temperature = float(json.loads(calibration.read_text())["choice"]) - if not math.isfinite(self.temperature) or self.temperature <= 0: - raise ValueError("Invalid published choice temperature") - # Pin weights to the SAME resolved snapshot as the tokenizer/config. - snapshot_download(model, revision=path.name, allow_patterns=["*.safetensors", "*.safetensors.index.json"]) - self.model, _ = load(str(path), adapter_path=adapter) - self.model.eval() - self.provenance = { - "model": model, "revision": path.name, "backend": "mlx", - "max_tokens": self.max_tokens, "checkpoint_context": config["max_position_embeddings"], - "max_cached_tokens": self.max_cached_tokens, "choice_temperature": self.temperature, - "adapter_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), - "format_sha256": hashlib.sha256(Path(__file__).with_name("format.py").read_bytes()).hexdigest(), - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - } - if adapter: - self.provenance["lora_adapter"] = str(Path(adapter).resolve()) - self.provenance["lora_weights_sha256"] = hashlib.sha256( - (Path(adapter) / "adapters.safetensors").read_bytes()).hexdigest() - self.provenance["calibration_sha256"] = hashlib.sha256(calibration.read_bytes()).hexdigest() - - def __call__(self, state, questions): - from model import shared_prefix_label_logits - - prepared = prepare_request(self.tokenizer, state, questions, self.max_tokens) - if not prepared: - raise ValueError("At least one question is required") - logits = shared_prefix_label_logits( - self.model, [p[2] for p in prepared], [p[3] for p in prepared], - max_cached_tokens=self.max_cached_tokens, - ) - answers, raw = {}, {} - for i, (name, q, ids, labels) in enumerate(prepared): - z = logits[i, :len(labels)].astype(np.float64) / self.temperature - p = np.exp(z - z.max()) - p /= p.sum() - if not np.isfinite(p).all(): - raise ValueError("Non-finite model probabilities") - keys = q.keys - answers[name] = { - "type": "choice", "choice": keys[int(np.argmax(p))], - "probabilities": dict(zip(keys, map(float, p))), - "confidence": float(p.max()), - } - raw[name] = {"label_logits": logits[i, :len(labels)].tolist(), "prompt_tokens": len(ids)} - return {"model": "jet", "answers": answers, - "usage": {"input_tokens": sum(len(p[2]) for p in prepared)}}, raw - - def synchronize(self): - self.mx.synchronize() - - def runtime(self): - import platform - from importlib.metadata import version - return {"platform": platform.platform(), "mlx": version("mlx"), - "mlx_lm": version("mlx-lm"), "device": str(self.mx.default_device())} diff --git a/docs/training/jet-v5/evaluation-code/src/decision_index_ensemble.py b/docs/training/jet-v5/evaluation-code/src/decision_index_ensemble.py deleted file mode 100644 index 2d1c5954306b164274ad5dc77af04062fcb85692..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/src/decision_index_ensemble.py +++ /dev/null @@ -1,37 +0,0 @@ -"""Two-order Jet readout; averages probabilities by original option key. - -Applies uniformly to every choice question. Preserves all inputs and refuses -requests if either complete prompt exceeds capacity. No benchmark-specific data. -""" -import hashlib -from pathlib import Path -import numpy as np -from decision_index_engine import JetEngine - - -def combine(first, second): - answers={} - if set(first['answers'])!=set(second['answers']):raise ValueError('Question mismatch') - for name,a in first['answers'].items(): - b=second['answers'][name] - keys=list(a['probabilities']) - if set(keys)!=set(b['probabilities']):raise ValueError('Option mismatch') - p=np.array([(a['probabilities'][k]+b['probabilities'][k])/2 for k in keys],dtype=float) - if not np.isfinite(p).all() or (p<0).any() or p.sum()<=0:raise ValueError('Invalid probabilities') - p/=p.sum() - answers[name]={'type':'choice','choice':keys[int(p.argmax())], - 'probabilities':dict(zip(keys,map(float,p))),'confidence':float(p.max())} - return {'model':'jet-two-order','answers':answers,'usage':{'input_tokens':first['usage']['input_tokens']+second['usage']['input_tokens']}} - - -class TwoOrderJetEngine(JetEngine): - name='jet-two-order' - def __init__(self,**options): - super().__init__(**options) - self.provenance.update(choice_readout='Mean of original and reversed option-order probabilities, aligned by exact original keys.', - ensemble_code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest()) - def __call__(self,state,questions): - original,raw_original=super().__call__(state,questions) - reversed_questions={name:dict(q,criteria=dict(reversed(list(q['criteria'].items())))) for name,q in questions.items()} - reverse,raw_reverse=super().__call__(state,reversed_questions) - return combine(original,reverse),{'original':raw_original,'reversed':raw_reverse,'single_order_answers':original['answers']} diff --git a/docs/training/jet-v5/evaluation-code/src/format.py b/docs/training/jet-v5/evaluation-code/src/format.py deleted file mode 100644 index 16ad75d3b250cde3629d60f849bd84cb61fa246a..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/src/format.py +++ /dev/null @@ -1,143 +0,0 @@ -"""Prompt format shared by training and inference. - -Every question is reduced to "pick one label token". The model is run once over -the prompt and we read the next-token logits restricted to the label tokens: - -- choice: one label per option (A..Z, then single-token AA, AB, ...), up to 255 options -- score: one digit per level (0..9), 2-10 levels, ordered low -> high -- noul: "no" / "yes" -""" - -from __future__ import annotations - -import itertools -import json -import string -from dataclasses import dataclass -from typing import Any, Literal - -QuestionType = Literal["choice", "score", "noul"] - -MAX_CHOICE_OPTIONS = 255 -MIN_SCORE_LEVELS, MAX_SCORE_LEVELS = 2, 10 - -SYSTEM_PROMPT = ( - "You are Jet, a decision model. Read the state and the question, then answer " - "with exactly one label from the allowed labels." -) - - -@dataclass(frozen=True) -class Question: - type: QuestionType - instructions: str - # choice: {key: description}; score: [level descriptions, low -> high]; noul: None - criteria: dict[str, str] | list[str] | None = None - - @property - def keys(self) -> list[str]: - """Answer keys in label order (option keys, level descriptions, or no/yes).""" - if self.type == "choice": - return list(self.criteria) - if self.type == "score": - return list(self.criteria) - return ["no", "yes"] - - @classmethod - def from_dict(cls, d: dict[str, Any]) -> Question: - q = cls(type=d["type"], instructions=d["instructions"], criteria=d.get("criteria")) - q.validate() - return q - - def to_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"type": self.type, "instructions": self.instructions} - if self.criteria is not None: - d["criteria"] = self.criteria - return d - - def validate(self) -> None: - if self.type not in ("choice", "score", "noul"): - raise ValueError(f"unknown question type {self.type!r}") - if not isinstance(self.instructions, str) or not self.instructions.strip(): - raise ValueError("instructions must be a non-empty string") - if self.type == "choice": - if not isinstance(self.criteria, dict) or not 2 <= len(self.criteria) <= MAX_CHOICE_OPTIONS: - raise ValueError(f"choice criteria must be an object with 2-{MAX_CHOICE_OPTIONS} options") - elif self.type == "score": - if not isinstance(self.criteria, list) or not MIN_SCORE_LEVELS <= len(self.criteria) <= MAX_SCORE_LEVELS: - raise ValueError(f"score criteria must be a list of {MIN_SCORE_LEVELS}-{MAX_SCORE_LEVELS} levels") - elif self.criteria is not None: - raise ValueError("noul questions take no criteria") - - -_CHOICE_LABELS: dict[int, list[str]] = {} - - -def choice_labels(tokenizer, n: int) -> list[str]: - """A..Z, then the two-letter labels AA, AB, ... that are single tokens for this tokenizer.""" - key = id(tokenizer) - if key not in _CHOICE_LABELS: - letters = string.ascii_uppercase - pairs = ("".join(p) for p in itertools.product(letters, repeat=2)) - candidates = list(letters) + [p for p in pairs if len(tokenizer.encode(p, add_special_tokens=False)) == 1] - if len(candidates) < MAX_CHOICE_OPTIONS: - raise ValueError(f"tokenizer only has {len(candidates)} single-token choice labels") - _CHOICE_LABELS[key] = candidates[:MAX_CHOICE_OPTIONS] - return _CHOICE_LABELS[key][:n] - - -def labels_for(tokenizer, q: Question) -> list[str]: - if q.type == "choice": - return choice_labels(tokenizer, len(q.criteria)) - if q.type == "score": - return [str(i) for i in range(len(q.criteria))] - return ["no", "yes"] - - -def render_state(state: Any) -> str: - if isinstance(state, str): - return state - if isinstance(state, list) and all(isinstance(s, str) for s in state): - return "\n\n".join(state) - return json.dumps(state, ensure_ascii=False, indent=2) - - -def render_user(state: Any, q: Question, labels: list[str]) -> str: - parts = [f"\n{render_state(state)}\n", ""] - if q.type == "choice": - opts = "\n".join(f"{lab}: {key}: {desc}" for lab, (key, desc) in zip(labels, q.criteria.items())) - parts += [ - f"Question: {q.instructions}", - f"Options:\n{opts}", - f"Answer with the label of the best option ({labels[0]}-{labels[-1]}).", - ] - elif q.type == "score": - levels = "\n".join(f"{lab}: {desc}" for lab, desc in zip(labels, q.criteria)) - parts += [ - f"Question: {q.instructions}", - f"Scale (lowest to highest):\n{levels}", - f"Answer with the level number (0-{len(labels) - 1}).", - ] - else: - parts += [f"Question: {q.instructions}", "Answer yes or no."] - return "\n".join(parts) - - -def build_prompt(tokenizer, state: Any, q: Question) -> str: - return tokenizer.apply_chat_template( - [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": render_user(state, q, labels_for(tokenizer, q))}], - add_generation_prompt=True, - tokenize=False, - enable_thinking=False, - ) - - -def _label_token_id(tokenizer, label: str) -> int: - ids = tokenizer.encode(label, add_special_tokens=False) - if len(ids) != 1: - raise ValueError(f"label {label!r} is not a single token for this tokenizer") - return ids[0] - - -def label_token_ids(tokenizer, q: Question) -> list[int]: - return [_label_token_id(tokenizer, lab) for lab in labels_for(tokenizer, q)] diff --git a/docs/training/jet-v5/evaluation-code/src/inference.py b/docs/training/jet-v5/evaluation-code/src/inference.py deleted file mode 100644 index da8630b4dfb9168d59dd22c64e1010ab892bdfbe..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/src/inference.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Backend-independent Jet prompt encoding and typed answers.""" -from __future__ import annotations -import math -from typing import Any -import numpy as np -from format import Question, build_prompt, render_state - -DEFAULT_MAX_STATE_TOKENS = 4096 - -def encode(tokenizer, state: Any, q: Question, max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS) -> list[int]: - """Tokenize a prompt, truncating the middle of an over-long state.""" - text = render_state(state) - ids = tokenizer.encode(text, add_special_tokens=False) - if len(ids) > max_state_tokens: - half = max_state_tokens // 2 - text = tokenizer.decode(ids[:half]) + "\n[...]\n" + tokenizer.decode(ids[-half:]) - return tokenizer.encode(build_prompt(tokenizer, text, q), add_special_tokens=False) - -def summarize(q: Question, probs: np.ndarray) -> dict[str, Any]: - """Turn a label distribution into the typed answer for the question.""" - probs = probs.astype(np.float64) - k = len(probs) - entropy = -float(np.sum(probs * np.log(np.clip(probs, 1e-12, 1.0)))) - confidence = round(1.0 - entropy / math.log(k), 4) - if q.type == "choice": - keys = q.keys - return { - "type": "choice", - "choice": keys[int(np.argmax(probs))], - "probabilities": {key: round(float(p), 4) for key, p in zip(keys, probs)}, - "confidence": confidence, - } - if q.type == "score": - return { - "type": "score", - "score": round(float(np.dot(np.arange(k), probs)), 4), - "level": q.criteria[int(np.argmax(probs))], - "probabilities": [round(float(p), 4) for p in probs], - "confidence": confidence, - } - return {"type": "noul", "probability": round(float(probs[1]), 4), "confidence": confidence} diff --git a/docs/training/jet-v5/evaluation-code/src/model.py b/docs/training/jet-v5/evaluation-code/src/model.py deleted file mode 100644 index 5e28e2317876fcd76b3edafaee434bf6740ed293..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/evaluation-code/src/model.py +++ /dev/null @@ -1,151 +0,0 @@ -"""Jet inference: one forward pass per batch of questions, no generation.""" - -from __future__ import annotations - -import json -import time -from pathlib import Path -from typing import Any - -import mlx.core as mx -import numpy as np -from huggingface_hub import snapshot_download -from mlx_lm import load -from mlx_lm.models.cache import make_prompt_cache - -from format import Question, label_token_ids -from inference import DEFAULT_MAX_STATE_TOKENS, encode, summarize - -DEFAULT_BASE_MODEL = "mlx-community/Qwen3-0.6B-bf16" -NEG_INF = -1e9 - - -def label_logits(model, tokens: mx.array, lengths: mx.array, label_ids: mx.array, label_mask: mx.array) -> mx.array: - """Next-token logits after each prompt, restricted to its label tokens. - - tokens: (B, L) right-padded prompts; lengths: (B,) true lengths; - label_ids / label_mask: (B, K) label token ids padded to the widest question. - Returns (B, K) logits with padded labels set to -inf. - """ - hidden = model.model(tokens) - last = hidden[mx.arange(tokens.shape[0]), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - picked = mx.take_along_axis(logits, label_ids, axis=-1).astype(mx.float32) - return mx.where(label_mask, picked, NEG_INF) - - -def head_logits(model, hidden: mx.array) -> mx.array: - if getattr(model.args, "tie_word_embeddings", False): - return model.model.embed_tokens.as_linear(hidden) - return model.lm_head(hidden) - - -def shared_prefix_label_logits( - model, seqs: list[list[int]], label_lists: list[list[int]], max_cached_tokens: int = 32_768 -) -> np.ndarray: - """Like label_logits, but encodes the prompts' common prefix (system + state) once. - - The prefix KV cache is replicated across a chunk of question suffixes, so each - question only pays for its own few dozen tokens. Chunks are sized so that - replicated prefix tokens stay under max_cached_tokens. - """ - common = min(len(s) for s in seqs) - 1 # every suffix keeps at least one token - for i in range(common): - if any(s[i] != seqs[0][i] for s in seqs): - common = i - break - prefix_cache = make_prompt_cache(model) - if common: - model.model(mx.array(seqs[0][:common])[None], cache=prefix_cache) - prefix_state = [c.state for c in prefix_cache] if common else None - if prefix_state: - mx.eval(prefix_state) - - out = np.full((len(seqs), max(map(len, label_lists))), NEG_INF, dtype=np.float32) - chunk = max(1, max_cached_tokens // max(common, 1)) - for start in range(0, len(seqs), chunk): - suffixes = [s[common:] for s in seqs[start : start + chunk]] - b = len(suffixes) - cache = make_prompt_cache(model) - if prefix_state: - for c, (k, v) in zip(cache, prefix_state): - c.state = (mx.repeat(k, b, axis=0), mx.repeat(v, b, axis=0)) - tokens, lengths = pad_batch(suffixes) - hidden = model.model(tokens, cache=cache) - last = hidden[mx.arange(b), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - ids, mask = pad_labels(label_lists[start : start + chunk]) - picked = mx.where(mask, mx.take_along_axis(logits, ids, axis=-1).astype(mx.float32), NEG_INF) - out[start : start + b, : picked.shape[1]] = np.array(picked) - return out - - -def pad_batch(seqs: list[list[int]], pad: int = 0) -> tuple[mx.array, mx.array]: - width = max(len(s) for s in seqs) - arr = np.full((len(seqs), width), pad, dtype=np.int32) - for i, s in enumerate(seqs): - arr[i, : len(s)] = s - return mx.array(arr), mx.array([len(s) for s in seqs], dtype=mx.int32) - - -def pad_labels(label_lists: list[list[int]]) -> tuple[mx.array, mx.array]: - width = max(len(ls) for ls in label_lists) - ids = np.zeros((len(label_lists), width), dtype=np.int32) - mask = np.zeros((len(label_lists), width), dtype=bool) - for i, ls in enumerate(label_lists): - ids[i, : len(ls)] = ls - mask[i, : len(ls)] = True - return mx.array(ids), mx.array(mask) - - -class Jet: - def __init__( - self, - base_model: str = DEFAULT_BASE_MODEL, - adapter_path: str | None = None, - max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS, - ): - self.model, self.tokenizer = load(base_model, adapter_path=adapter_path) - self.model.eval() - self.max_state_tokens = max_state_tokens - self.temperatures = {"choice": 1.0, "score": 1.0, "noul": 1.0} - # Calibration lives next to the adapter, or inside a fused model dir. A Hub repo id has no dir - # of its own, so look in its local snapshot (fetching calibration.json if the repo has one). - model_dir = Path(base_model) if Path(base_model).exists() else Path(snapshot_download(base_model, allow_patterns=["calibration.json"])) - for directory in (adapter_path, model_dir): - if directory and (calib := Path(directory) / "calibration.json").exists(): - self.temperatures.update(json.loads(calib.read_text())) - break - - def raw_logits(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - """Uncalibrated label logits for (state, question) pairs, plus the number of - input tokens. With shared_state, the common state prefix is encoded once.""" - seqs = [encode(self.tokenizer, s, q, self.max_state_tokens) for s, q in items] - label_lists = [label_token_ids(self.tokenizer, q) for _, q in items] - if shared_state and len(items) > 1: - out = shared_prefix_label_logits(self.model, seqs, label_lists) - else: - tokens, lengths = pad_batch(seqs) - ids, mask = pad_labels(label_lists) - out = np.array(label_logits(self.model, tokens, lengths, ids, mask)) - return [out[i, : len(q.keys)] for i, (_, q) in enumerate(items)], sum(map(len, seqs)) - - def probabilities(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - logits, n_tokens = self.raw_logits(items, shared_state) - result = [] - for (_, q), z in zip(items, logits): - z = z / self.temperatures[q.type] - z = np.exp(z - z.max()) - result.append(z / z.sum()) - return result, n_tokens - - def decide(self, state: Any, questions: dict[str, Question]) -> dict[str, Any]: - """Answer every question about one state together (shared batch).""" - started = time.perf_counter() - items = [(state, q) for q in questions.values()] - probs, input_tokens = self.probabilities(items, shared_state=True) - return { - "answers": {name: summarize(q, p) for (name, q), p in zip(questions.items(), probs)}, - "usage": {"input_tokens": input_tokens}, - "latency_ms": round((time.perf_counter() - started) * 1000, 1), - } diff --git a/docs/training/jet-v5/expanded-capacity.json b/docs/training/jet-v5/expanded-capacity.json deleted file mode 100644 index 312830664a177f1ed74f2ae7c27f73c240a44f06..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/expanded-capacity.json +++ /dev/null @@ -1,249 +0,0 @@ -{ - "rows_sha256": "9908538b8b244efa20a2d8428104d5f56a131b37569d0e0b10867ced7beb104b", - "rows_uncompressed_sha256": "45f871b09ada003492321e32cb6865dab47cf2d2855f4667264b3b5cb02888f4", - "model_revision": "8a97cfea2df622bb03f5dc9b02567e21abd2551c", - "scope": "Only the supplied rows; token capacity, not measured accuracy or validated long-context quality.", - "limits": [ - 4096, - 6144, - 8192, - 16384, - 32768, - 40960 - ], - "benchmarks": { - "CLINC150+OOS": { - "requests": 200, - "fields": 200, - "options": { - "151": 200 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 1653.0, - "p95": 1660.05, - "max": 1669.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - }, - "ANLI": { - "requests": 200, - "fields": 200, - "options": { - "3": 200 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 196.5, - "p95": 233.0, - "max": 267.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - }, - "MMLU": { - "requests": 200, - "fields": 200, - "options": { - "4": 200 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 152.0, - "p95": 314.19999999999993, - "max": 453.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - }, - "ARC-Easy": { - "requests": 200, - "fields": 200, - "options": { - "4": 199, - "3": 1 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 126.0, - "p95": 168.04999999999998, - "max": 206.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - }, - "ARC-Challenge": { - "requests": 200, - "fields": 200, - "options": { - "4": 199, - "3": 1 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 135.0, - "p95": 177.0, - "max": 217.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - }, - "WinoGrande": { - "requests": 200, - "fields": 200, - "options": { - "2": 200 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 106.0, - "p95": 114.0, - "max": 120.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - }, - "HellaSwag": { - "requests": 200, - "fields": 200, - "options": { - "4": 200 - }, - "questions_per_request": { - "1": 200 - }, - "supported_requests": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - }, - "capacity_failures": {}, - "prompt_tokens": { - "p50": 293.0, - "p95": 347.0, - "max": 389.0 - }, - "source_groups": 200, - "supported_complete_groups": { - "4096": 200, - "6144": 200, - "8192": 200, - "16384": 200, - "32768": 200, - "40960": 200 - } - } - } -} diff --git a/docs/training/jet-v5/expanded-diagnostic-provenance.json b/docs/training/jet-v5/expanded-diagnostic-provenance.json deleted file mode 100644 index 5e017372d368a2f8fbc2249b2677a775fea823f2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/expanded-diagnostic-provenance.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "scope": "Partial diagnostic; not Decision Index 0.2", - "n": 1400, - "counts": { - "CLINC150+OOS": 200, - "ANLI": 200, - "MMLU": 200, - "ARC-Easy": 200, - "ARC-Challenge": 200, - "WinoGrande": 200, - "HellaSwag": 200 - }, - "file": "artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz", - "sha256": "9908538b8b244efa20a2d8428104d5f56a131b37569d0e0b10867ced7beb104b", - "reason": "Raised sample size before any model scoring to 200 requests per available benchmark. Original 600-row file preserved. MMLU and ARC remain non-index diagnostics." -} diff --git a/docs/training/jet-v5/final-code/scripts/audit_v5.py b/docs/training/jet-v5/final-code/scripts/audit_v5.py deleted file mode 100644 index d7497dc7c03a2139ac78b15ee136f1ba316fc886..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/audit_v5.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Independently match every new V5 label back to its pinned native train row.""" -import json,hashlib -from collections import defaultdict,Counter -from pathlib import Path -import pyarrow.parquet as pq -from data.decision_training import ROOT,read,statekey - -def main(): - specs=json.loads((ROOT/'work/jet-v5/sources-complete.json').read_text()); native=defaultdict(list);labels=None - for s in specs: - if s['role']!='train':continue - if s['file'].endswith('.parquet'): - t=pq.read_table(s['path']);rows=t.to_pylist() - if 'clinc' in s['repo']:labels=json.loads(t.schema.metadata[b'huggingface'])['info']['features']['intent']['names'] - else:rows=[json.loads(x) for x in open(s['path'])] - for i,r in enumerate(rows): - key=f'{s["repo"]}@{s["revision"]}/{s["file"]}:{r.get("uid",r.get("ind",i))}' - native[key].append(r) - owners={}; counts=Counter(); origins=set() - for split,file in [('train','train_v5'),('val','selection_v5'),('calibration','calibration_v5'),('test','test_v5_new')]: - for row in read(ROOT/f'data/{file}.jsonl'): - if not row['source'].startswith('v5:'):continue - p=row['provenance']; assert p['origin'] not in origins; origins.add(p['origin']) - answer=row['question']['criteria'][row['gold']] - candidates=native[p['origin']];family=row['family'] - def matches(r): - if family=='wide_intent': - correct=labels[r['intent']]; correct='outside all supported intents' if correct=='oos' else correct.replace('_',' ') - return row['state']==r['text'] and answer==correct and len(row['question']['criteria'])==151 - if family=='commonsense_completion':return row['state']==r['ctx'] and answer==r['endings'][int(r['label'])] - if family=='adversarial_entailment':return row['state']==dict(premise=r['premise'],hypothesis=r['hypothesis']) and answer==['the premise entails the hypothesis','the premise does not determine the hypothesis','the premise contradicts the hypothesis'][r['label']] - return row['state']['messages']==r['messages'] and answer==r['chosen_response']['content'] - assert any(matches(r) for r in candidates),(family,p['origin']) - assert row['target'][list(row['question']['criteria']).index(row['gold'])]==1 - for key in [p['component'],statekey(row['state'])]+p['content_keys']: - assert owners.setdefault(key,split)==split,(key,split) - counts[split+':'+family]+=1 - report={'verified_native_labels':sum(counts.values()),'counts':dict(counts),'source_id_duplicates':0,'cross_split_related_groups':0} - (ROOT/'docs/training/jet-v5/data-audit.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps(report,indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/final-code/scripts/build_v5.py b/docs/training/jet-v5/final-code/scripts/build_v5.py deleted file mode 100644 index 89a043bbf116e2de6eb374f6662763491b6353f2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/build_v5.py +++ /dev/null @@ -1,143 +0,0 @@ -"""V5: native TRAIN-only coverage for the expanded Decision Index panel. - -Native evaluation files are exclusion-only. All new heldouts come from source -training partitions, grouped before sampling. Never modifies earlier files. -""" -import gzip, hashlib, json, random -from collections import Counter, defaultdict -from pathlib import Path -import pyarrow.parquet as pq -from transformers import AutoTokenizer -from data.decision_training import ROOT, Components, atoms, choice, digest, normalized, read, statekey, write -from format import Question, render_state, build_prompt -SEED=240925 - -def sha(p): - with Path(p).open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() - -def records(spec): - if spec['file'].endswith('.parquet'):return pq.read_table(spec['path']).to_pylist() - return [json.loads(s) for s in open(spec['path'])] - -def texts(repo,r): - if 'clinc' in repo:return [r['text']] - if 'hellaswag' in repo:return [r['ctx']] - if 'anli' in repo:return [r['premise'],r['hypothesis']] - return [r[k] for k in ['question','orig_question'] if r.get(k)]+[m['content'] for m in r.get('messages',[]) if m['role']=='user'] - -def content_keys(repo,r):return {'content:'+digest(normalized(t)) for t in texts(repo,r)} - -def main(): - data=ROOT/'data'; work=ROOT/'work/jet-v5' - names=['train_v5.jsonl','selection_v5.jsonl','calibration_v5.jsonl','test_v5_new.jsonl'] - assert not any((data/n).exists() for n in names),'Refusing to overwrite V5' - specs=json.loads((work/'sources-complete.json').read_text()) - protected=[p for p in data.glob('*.jsonl') if any(v in p.name for v in ['val','test','selection','calibration'])] - protected+=[ROOT/'artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl',ROOT/'artifacts/decision-index/esci-rebuild/artifacts/benchmark-suite/release-v1-rebuilt/selected-rows.jsonl'] - blocked=set(); old_atoms=set(); counts=Counter(); native_groups=set() - for p in protected: - assert p.exists(),p - for r in read(p): - blocked.update('content:'+a for a in atoms(r.get('state',r.get('payload',{})))) - if isinstance(r.get('state'),str):blocked.add('content:'+digest(normalized(r['state']))) - parent=read(data/'train_v4_tools_focus.jsonl') - # Include all earlier training, even examples not retained in this short stage. - for p in [data/'train_v2.jsonl',data/'train_v3.jsonl',data/'train_v4.jsonl',data/'train_v4_tools_focus.jsonl']: - for r in read(p): - old_atoms.update('content:'+a for a in atoms(r['state'])) - if isinstance(r['state'],str):old_atoms.add('content:'+digest(normalized(r['state']))) - for s in specs: - if s['role']=='train':continue - for r in records(s): - blocked.update(content_keys(s['repo'],r)) - if 'hellaswag' in s['repo']:native_groups.add('hellaswag:'+r['source_id']) - candidates=[]; components=Components(); origin_seen=set() - for s in specs: - if s['role']!='train':continue - repo=s['repo']; rows=records(s) - if 'clinc' in repo: - labels=json.loads(pq.read_schema(s['path']).metadata[b'huggingface'])['info']['features']['intent']['names'] - opts={k:('outside all supported intents' if k=='oos' else k.replace('_',' ')) for k in labels} - if 'anli' in repo: - assert json.loads(pq.read_schema(s['path']).metadata[b'huggingface'])['info']['features']['label']['names']==['entailment','neutral','contradiction'] - for idx,r in enumerate(rows): - keys=content_keys(repo,r) - if keys&blocked:counts['evaluation_content']+=1;continue - origin=f'{repo}@{s["revision"]}/{s["file"]}:{r.get("uid",r.get("ind",idx))}' - if origin in origin_seen:counts['duplicate_source_id']+=1;continue - origin_seen.add(origin) - if 'clinc' in repo: - family='wide_intent'; group='clinc:'+digest(normalized(r['text'])) - row=choice(r['text'],'Which intent best describes the user request? Choose outside scope if none apply.',opts,labels[r['intent']],family,origin,group) - elif 'hellaswag' in repo: - family='commonsense_completion';group='hellaswag:'+r['source_id'] - if group in native_groups:counts['native_eval_video_or_article']+=1;continue - options={str(i):v for i,v in enumerate(r['endings'])} - row=choice(r['ctx'],'Which continuation is most plausible?',options,str(int(r['label'])),family,origin,group) - elif 'anli' in repo: - family='adversarial_entailment';group='anli:'+digest(normalized(r['premise'])) - options=dict(entailment='the premise entails the hypothesis',neutral='the premise does not determine the hypothesis',contradiction='the premise contradicts the hypothesis') - row=choice(dict(premise=r['premise'],hypothesis=r['hypothesis']),'How does the hypothesis relate to the premise?',options,list(options)[r['label']],family,origin,group) - else: - family='tool_response_preference'; tools=[json.loads(t) if isinstance(t,str) else t for t in r['tools']] - group='when2call-tools:'+digest(sorted(t['name'] for t in tools)) - options=dict(preferred=r['chosen_response']['content'],rejected=r['rejected_response']['content']) - if options['preferred']==options['rejected']:counts['identical_choices']+=1;continue - row=choice(dict(tools=tools,messages=r['messages']),'Which response should the assistant give next, given the available tools?',options,'preferred',family,origin,group) - # Anonymous keys prevent a preferred/rejected key from revealing the answer. - oldkeys=list(row['question']['criteria']); oldgold=row['gold'] - row['question']['criteria']={f'option_{i}':row['question']['criteria'][k] for i,k in enumerate(oldkeys)} - row['gold']=f'option_{oldkeys.index(oldgold)}' - row['source']='v5:'+family - row['provenance']['content_keys']=sorted(keys) - for key in keys:components.union(group,key) - candidates.append(row) - forced={components.find(r['provenance']['group']) for r in candidates if set(r['provenance']['content_keys'])&old_atoms} - tok=AutoTokenizer.from_pretrained('mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8',local_files_only=True) - bins=defaultdict(list);seen=set() - # Random traversal also bounds tokenization cost after ample eligible groups. - random.Random(SEED).shuffle(candidates) - for r in candidates: - p=r['provenance'];group=components.find(p['group']);n=int(digest([SEED,group])[:8],16)/2**32 - split='train' if group in forced or n<.75 else 'val' if n<.84 else 'calibration' if n<.92 else 'test' - signature=statekey(r['state']) - if signature in seen:counts['duplicate_state']+=1;continue - seen.add(signature) - if len(bins[split,r['family']])>5000:continue - if len(tok.encode(render_state(r['state']),add_special_tokens=False))>2048:counts['long_full_state']+=1;continue - if len(tok.encode(build_prompt(tok,r['state'],Question.from_dict(r['question'])),add_special_tokens=False))>3000:counts['long_full_prompt']+=1;continue - p.update(component=group,split=split);bins[split,r['family']].append(r) - splits=defaultdict(list) - for (split,family),rs in sorted(bins.items()): - random.Random(SEED).shuffle(rs) - rs=rs[:2000 if split=='train' else 150] - assert len(rs)>=(150 if split!='train' else 1500),(split,family,len(rs)) - splits[split]+=rs - owners={};origins=set() - for split,rs in splits.items(): - for r in rs: - assert r['provenance']['origin'] not in origins;origins.add(r['provenance']['origin']) - for key in [r['provenance']['component'],statekey(r['state'])]+r['provenance']['content_keys']: - assert owners.setdefault(key,split)==split,(key,split) - assert not(set(r['provenance']['content_keys'])&blocked) - assert r['target'][list(r['question']['criteria']).index(r['gold'])]==1 - def balanced(rs,limit): - groups=defaultdict(list) - for r in rs:groups[r.get('family',r['source'])].append(r) - for k,g in sorted(groups.items()):random.Random(SEED).shuffle(g) - result=[] - while groups and len(result)=summary['single']['macro_family_accuracy'] - report={'n':len(results),'selection_sha256':hashlib.sha256(Path('data/selection_v5.jsonl').read_bytes()).hexdigest(),'selection_rule':'lower NLL and no decrease in macro family accuracy on selection data only','selected':'two_order' if choose else 'single','summary':summary,'seconds':time.monotonic()-started,'engine_provenance':engine.provenance,'rows':results} - Path('docs/training/jet-v5/inference-selection.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps({k:v for k,v in report.items() if k not in ['rows','engine_provenance']},indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/final-code/scripts/filter_v5_retention.py b/docs/training/jet-v5/final-code/scripts/filter_v5_retention.py deleted file mode 100644 index e20692fcaad13ae2f8cbf555b38540242e849656..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/filter_v5_retention.py +++ /dev/null @@ -1,40 +0,0 @@ -"""Audit native-evaluation overlap in retained examples and create immutable R2 data. - -Catalog names and isolated short schema fields are not treated as utterances. -V5 source content identities, complete text states and text fragments >=32 -normalized characters are compared; inherited exposure is reported explicitly. -""" -import json -from pathlib import Path -from collections import Counter -from build_v5 import records,content_keys,sha -from data.decision_training import ROOT,read,write,atoms,digest,normalized - -def keys(row): - if row['source'].startswith('v5:'):return set(row['provenance']['content_keys']) - result={'content:'+a for a in atoms(row['state'])} - if isinstance(row['state'],str):result.add('content:'+digest(normalized(row['state']))) - return result - -def main(): - blocked=set() - for s in json.loads((ROOT/'work/jet-v5/sources-complete.json').read_text()): - if s['role']!='train': - for r in records(s):blocked.update(content_keys(s['repo'],r)) - report={'rule':__doc__,'files':{},'inherited_exposure':{}} - for name in ['train_v2','train_v3','train_v4_tools_focus']: - hits=[r for r in read(ROOT/f'data/{name}.jsonl') if keys(r)&blocked] - report['inherited_exposure'][name]={'count':len(hits),'by_source':dict(Counter(r['source'] for r in hits)),'state_hashes':[digest(r['state']) for r in hits]} - for name in ['train_v5','selection_v5','calibration_v5','test_v5_new']: - src=ROOT/f'data/{name}.jsonl';rows=read(src) - kept=[r for r in rows if not keys(r)&blocked];hits=[r for r in rows if keys(r)&blocked] - if name=='train_v5': - dst=ROOT/'data/train_v5_r2.jsonl';assert not dst.exists();write(dst,kept) - report['output']={'path':str(dst),'sha256':sha(dst),'rows':len(kept)} - else:assert not hits,(name,len(hits)) - report['files'][name]={'sha256':sha(src),'kept':len(kept),'removed':len(hits),'removed_sources':dict(Counter(r['source'] for r in hits)),'state_hashes':[digest(r['state']) for r in hits]} - smoke=read(ROOT/'work/jet-v5/smoke.jsonl');assert not any(keys(r)&blocked for r in smoke) - report['caveat']='R2 removes new-stage overlap but cannot erase inherited V4 training exposure; do not claim a fully decontaminated checkpoint or an official score.' - (ROOT/'docs/training/jet-v5/retention-audit.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps(report,indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/final-code/scripts/finalize_v5.py b/docs/training/jet-v5/final-code/scripts/finalize_v5.py deleted file mode 100644 index de05230634bb146164b5b73f3a1cea6f62d148ae..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/finalize_v5.py +++ /dev/null @@ -1,48 +0,0 @@ -"""Verify V5 completion, original-file preservation, and selected artifact hashes.""" -import csv,hashlib,json,math,shutil,subprocess,platform -from importlib.metadata import version -from pathlib import Path -from safetensors import safe_open -ROOT=Path(__file__).resolve().parents[1] -D=ROOT/'docs/training/jet-v5' -def sha(p): - with p.open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() -def main(): - adapter=ROOT/'adapters/jet-v5-panel02-r2-20260924' - config=json.loads((adapter/'training_config.json').read_text()) - metrics=[json.loads(x) for x in (adapter/'metrics.jsonl').read_text().splitlines()] - assert metrics[-1]['step']==config['total_steps'] - best=min(metrics,key=lambda r:r['nll']) - with safe_open(adapter/'optimizer.safetensors',framework='numpy') as f:step=int(f.get_tensor('step')) - assert step==best['step'],(step,best) - temperatures=json.loads((adapter/'calibration.json').read_text()) - assert all(math.isfinite(t) and t>0 for t in temperatures.values()) - data=json.loads((D/'retention-audit.json').read_text()) - assert sha(ROOT/'data/train_v5_r2.jsonl')==data['output']['sha256'] - for name,meta in data['files'].items():assert sha(ROOT/f'data/{name}.jsonl')==meta['sha256'] - original_provenance=json.loads((ROOT/'data/train_v5.provenance.json').read_text()) - for path,digest in original_provenance['protected'].items():assert sha(Path(path))==digest,path - for name in ['protected','published']: - subprocess.run(['sha256sum','-c',str(ROOT/f'work/jet-v3/{name}.sha256')],cwd=ROOT,check=True) - counts={} - for suite in ['diagnostic','v5-diagnostic-1400']: - path=ROOT/f'artifacts/decision-index/runs/v5-compatibility-{suite}/results.jsonl' - rows=[json.loads(s) for s in path.read_text().splitlines()] - statuses=[r['status'] for r in rows];assert all(s=='ok' for s in statuses),statuses - counts[suite]=len(rows) - code=D/'final-code';code.mkdir(exist_ok=True) - names=['src/train.py','src/model.py','src/evaluate.py','src/format.py','src/inference.py','src/decision_index_engine.py','src/decision_index_ensemble.py'] - names += [str(p.relative_to(ROOT)) for p in (ROOT/'scripts').glob('*v5*.py')] - names += ['scripts/run_v5_cuda.sh','scripts/run_v5_extra.sh','tests/test_decision_ensemble.py'] - code_hashes={} - for name in names: - p=ROOT/name;q=code/name;q.parent.mkdir(parents=True,exist_ok=True);shutil.copy2(p,q);code_hashes[name]=sha(p) - gpu=list(csv.DictReader((ROOT/'logs/jet-v5/gpu.csv').open())) - peak=max(int(r[' memory.used [MiB]'].strip().split()[0]) for r in gpu) - environment={'platform':platform.platform(),'packages':{p:version(p) for p in ['mlx','mlx-lm','numpy','transformers','huggingface-hub','decision-index']},'gpu':subprocess.check_output(['nvidia-smi','--query-gpu=name,driver_version,memory.total','--format=csv,noheader'],text=True).strip()} - report={'environment':environment,'selected_step':step,'total_steps':config['total_steps'],'adapter_sha256':sha(adapter/'adapters.safetensors'),'calibration':temperatures,'compatibility_ok':counts,'whole_session_gpu_peak_mib':peak,'training_config':config,'code_sha256':code_hashes,'dataset':data,'published_model_unchanged':True,'old_validation_test_unchanged':True,'scope':'Partial diagnostics; inherited overlap caveat applies. No official Decision Index 0.2 score.'} - (D/'completion-audit.json').write_text(json.dumps(report,indent=2)+'\n') - out=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs') - shutil.copy2(D/'completion-audit.json',out/'jet-v5-run-manifest.json') - print(json.dumps({k:v for k,v in report.items() if k not in ['training_config','code_sha256','dataset']},indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/final-code/scripts/rebuild_v5_diagnostic.py b/docs/training/jet-v5/final-code/scripts/rebuild_v5_diagnostic.py deleted file mode 100644 index 38bc2f6248e6f6b6decce38e743823fb4ab06f94..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/rebuild_v5_diagnostic.py +++ /dev/null @@ -1,22 +0,0 @@ -"""Rebuild additional diagnostic families with the existing official normalizers. - -This is a partial reconstruction, never a Decision Index 0.2 leaderboard score. -""" -import json,shutil -from pathlib import Path -from decision_index.suite.build.acquire import acquire -from decision_index.suite.build.layout import Layout -from decision_index.suite.build.rebuild import main as rebuild - -def main(): - work=Path('artifacts/decision-index/v5-rebuild');layout=Layout(work) - # The upstream knowledge builder needs all of these even to select HellaSwag. - acquire(layout,[5,12,24,26,27,28,29]) - # The installed acquirer stages raw/ at root; normalizers read suite/raw/. - for name in ['clinc150','anli']: - shutil.copytree(layout.root/'raw'/name,layout.raw/name,dirs_exist_ok=True) - result=rebuild(work,only=[5,12,29],skip_download=True) - out=Path(result['out']) - shutil.copy2('artifacts/decision-index/rebuild/artifacts/benchmark-suite/release-v1-rebuilt/excluded-questions.json',out/'excluded-questions.json') - print(json.dumps(result,indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/final-code/scripts/run_v5_cuda.sh b/docs/training/jet-v5/final-code/scripts/run_v5_cuda.sh deleted file mode 100644 index f2b8daf57c8de968db61e663e0f5883418eef211..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/run_v5_cuda.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail -cd "$(dirname "$0")/.." -export PYTHONUNBUFFERED=1 CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 -export MLX_CUDA_SDPA_CACHE_SIZE=2048 MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" -NEXT=adapters/jet-v5-panel02-r2-20260924 -BASE=adapters/jet-v4-transfer-20260924 -bash train_cuda.sh --init-adapter "$BASE" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train work/jet-v5/smoke.jsonl --val work/jet-v5/smoke.jsonl --out adapters/jet-v5-panel02-r2-smoke \ - --epochs 1 --lr 1e-5 --rank 16 --grad-checkpoint --max-batch-tokens 3072 --max-state-tokens 2048 \ - --eval-every 10 --val-limit 100 --seed 240925 > logs/jet-v5/smoke.log 2>&1 -bash train_cuda.sh --init-adapter "$BASE" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train data/train_v5_r2.jsonl --val data/selection_v5.jsonl --out "$NEXT" \ - --epochs 1 --lr 1e-5 --rank 16 --grad-checkpoint --max-batch-tokens 3072 --max-state-tokens 2048 \ - --use-example-weights --eval-every 250 --val-limit 10000 --seed 240925 > logs/jet-v5/train.log 2>&1 -.venv/bin/jet-calibrate --adapter "$NEXT" --data data/calibration_v5.jsonl --limit 10000 --batch-size 2 \ - --max-state-tokens 4096 > logs/jet-v5/calibration.log 2>&1 -for name in v4 v5; do - if [[ "$name" == v4 ]]; then RUN="$BASE"; else RUN="$NEXT"; fi - for dataset in test_v5_new test test_v4_new test_v4_tools; do - CONTEXT=2048; if [[ "$dataset" == test ]]; then CONTEXT=4096; fi - .venv/bin/jet-eval --adapter "$RUN" --data "data/$dataset.jsonl" --batch-size 2 \ - --max-state-tokens "$CONTEXT" --json "docs/training/jet-v5/$name-$dataset.json" > "logs/jet-v5/$name-$dataset.log" 2>&1 - done - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz --out "artifacts/decision-index/runs/diagnostic-v5-$name" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter="$RUN" --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$name-diagnostic.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/diagnostic-v5-$name/results.jsonl" --out "docs/training/jet-v5/$name-diagnostic.json" > "logs/jet-v5/$name-report.log" 2>&1 -done -sha256sum -c work/jet-v3/protected.sha256 > logs/jet-v5/protected.log -sha256sum -c work/jet-v3/published.sha256 > logs/jet-v5/published.log -printf 'V5 training and evaluation completed.\n' diff --git a/docs/training/jet-v5/final-code/scripts/run_v5_extra.sh b/docs/training/jet-v5/final-code/scripts/run_v5_extra.sh deleted file mode 100644 index 8235956881557027c7d5e0622f6da599594f3ec8..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/run_v5_extra.sh +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env bash -# Run only after run_v5_cuda.sh finishes; uses the GPU sequentially. -set -euo pipefail -cd "$(dirname "$0")/.." -export CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 MLX_CUDA_SDPA_CACHE_SIZE=2048 -export MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" PYTHONPATH=src PYTHONUNBUFFERED=1 -.venv/bin/python scripts/evaluate_v5_order.py --adapter adapters/jet-v5-panel02-r2-20260924 > logs/jet-v5/inference-selection.log 2>&1 -ENGINE=decision_index_engine:JetEngine -if .venv/bin/python -c 'import json,sys;sys.exit(json.load(open("docs/training/jet-v5/inference-selection.json"))["selected"]!="two_order")'; then - ENGINE=decision_index_ensemble:TwoOrderJetEngine -fi -for SUITE in diagnostic v5-diagnostic-1400; do - .venv/bin/python -m decision_index run --engine "$ENGINE" \ - --rows "artifacts/decision-index/$SUITE/compatibility.jsonl.gz" \ - --out "artifacts/decision-index/runs/v5-compatibility-$SUITE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/compatibility-$SUITE.log" 2>&1 -done - .venv/bin/jet-eval --base-model models/jet --data data/test_v5_new.jsonl --batch-size 2 \ - --max-state-tokens 2048 --json docs/training/jet-v5/baseline-test_v5_new.json > logs/jet-v5/baseline-test_v5_new.log 2>&1 -for NAME in baseline v4 v5; do - ADAPTER=adapters/jet-v4-transfer-20260924 - if [[ "$NAME" == v5 ]]; then ADAPTER=adapters/jet-v5-panel02-r2-20260924; fi - OPTIONS=(--option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 --option adapter="$ADAPTER") - if [[ "$NAME" == baseline ]]; then - OPTIONS=(--option model=michaljach/jet --option revision=8a97cfea2df622bb03f5dc9b02567e21abd2551c) - fi - .venv/bin/python -m decision_index run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/v5-expanded-$NAME" \ - "${OPTIONS[@]}" --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$NAME-expanded.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/v5-expanded-$NAME/results.jsonl" --out "docs/training/jet-v5/$NAME-expanded.json" > "logs/jet-v5/$NAME-expanded-report.log" 2>&1 -done -if .venv/bin/python -c 'import json,sys;sys.exit(json.load(open("docs/training/jet-v5/inference-selection.json"))["selected"]!="two_order")'; then - for PAIR in 'diagnostic/diagnostic.jsonl.gz v5-two-order-original' 'v5-diagnostic-1400/diagnostic.jsonl.gz v5-two-order-expanded'; do - read -r ROWS RUN <<< "$PAIR" - .venv/bin/python -m decision_index run --engine decision_index_ensemble:TwoOrderJetEngine \ - --rows "artifacts/decision-index/$ROWS" --out "artifacts/decision-index/runs/$RUN" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$RUN.log" 2>&1 - .venv/bin/jet-bench-index report --rows "artifacts/decision-index/$ROWS" \ - --results "artifacts/decision-index/runs/$RUN/results.jsonl" --out "docs/training/jet-v5/$RUN.json" > "logs/jet-v5/$RUN-report.log" 2>&1 - done -fi -for MODE in single selected; do - TARGET=decision_index_engine:JetEngine - if [[ "$MODE" == selected ]]; then - [[ "$ENGINE" != decision_index_engine:JetEngine ]] || continue - TARGET="$ENGINE" - fi - .venv/bin/python -m decision_index run --engine "$TARGET" \ - --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/v5-code-$MODE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/code-$MODE.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/cruxeval-full-diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/v5-code-$MODE/results.jsonl" --out "docs/training/jet-v5/code-$MODE.json" > "logs/jet-v5/code-$MODE-report.log" 2>&1 -done -for ENTRY in 'esci-diagnostic/diagnostic.jsonl.gz esci' 'isarcasm-english-diagnostic.jsonl.gz sarcasm'; do - read -r ROWS LABEL <<< "$ENTRY" - for MODE in single selected; do - TARGET=decision_index_engine:JetEngine - if [[ "$MODE" == selected ]]; then - [[ "$ENGINE" != decision_index_engine:JetEngine ]] || continue - TARGET="$ENGINE" - fi - .venv/bin/python -m decision_index run --engine "$TARGET" \ - --rows "artifacts/decision-index/$ROWS" --out "artifacts/decision-index/runs/v5-$LABEL-$MODE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$LABEL-$MODE.log" 2>&1 - .venv/bin/jet-bench-index report --rows "artifacts/decision-index/$ROWS" \ - --results "artifacts/decision-index/runs/v5-$LABEL-$MODE/results.jsonl" --out "docs/training/jet-v5/$LABEL-$MODE.json" > "logs/jet-v5/$LABEL-$MODE-report.log" 2>&1 - done -done -printf 'V5 expanded evaluation complete.\n' diff --git a/docs/training/jet-v5/final-code/scripts/summarize_v5.py b/docs/training/jet-v5/final-code/scripts/summarize_v5.py deleted file mode 100644 index ccb457ed8c6c81681c576e0b6d3a6bed5847ef4d..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/scripts/summarize_v5.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Summarize completed V5 evaluation without inventing a leaderboard score.""" -import hashlib,json,shutil -from pathlib import Path -ROOT=Path(__file__).resolve().parents[1] -D=ROOT/'docs/training/jet-v5' -def read(p):return json.loads(p.read_text()) -def sha(p): - with p.open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() -def main(): - adapter=ROOT/'adapters/jet-v5-panel02-r2-20260924' - config=read(adapter/'training_config.json');metrics=[json.loads(x) for x in (adapter/'metrics.jsonl').read_text().splitlines()] - assert metrics[-1]['step']==config['total_steps'],'Training incomplete' - order=read(D/'inference-selection.json') - released_new=read(D/'baseline-test_v5_new.json') - held={name:{ds:read(D/f'{name}-{ds}.json') for ds in ['test','test_v5_new','test_v4_new','test_v4_tools']} for name in ['v4','v5']} - bench={} - for name in ['v4','v5']: - bench[name]={} - for suite in ['diagnostic','expanded']: - result=read(D/f'{name}-{suite}.json');assert result['complete_for_supplied_rows'] and result['pending_requests']==0 - for b in result['summary']['benchmarks']:bench[name][b['dataset']]=b - if order['selected']=='two_order': - bench['v5_two_order']={} - for suite in ['original','expanded']: - result=read(D/f'v5-two-order-{suite}.json');assert result['complete_for_supplied_rows'] - for b in result['summary']['benchmarks']:bench['v5_two_order'][b['dataset']]=b - regressions={} - for label,old_prefix in [('code','cruxeval-full'),('esci','esci'),('sarcasm','isarcasm-en')]: - runs={'baseline':read(ROOT/f'docs/training/jet-next/{old_prefix}-baseline.json'),'v4':read(ROOT/f'docs/training/jet-next/{old_prefix}-v4.json'),'v5':read(D/f'{label}-single.json')} - if order['selected']=='two_order':runs['v5_two_order']=read(D/f'{label}-selected.json') - assert len({r['rows_sha256'] for r in runs.values()})==1, label - assert all(r['complete_for_supplied_rows'] for r in runs.values()), label - scores={} - for name,r in runs.items(): - scores[name]=(next(t['raw'] for t in r['panel_metrics_on_sample']['40']['tracks'] if t['track']=='iSarcasmEval-A-En') if label=='sarcasm' else r['summary']['benchmarks'][0]['score']) - regressions[label]={'scores':scores,'n':runs['v5']['planned_requests'],'rows_sha256':runs['v5']['rows_sha256']} - best=min(metrics,key=lambda r:r['nll']) - lines=['# Jet V5 results — Qwen3-0.6B','', - 'V5 was trained locally on 15,997 examples: 7,997 retained V4 rows plus 2,000 each for CLINC150, ANLI, HellaSwag, and When2Call response preference. Native training partitions only; new selection, calibration and test groups are separate.','', - '**These are partial diagnostics, not an official Decision Index 0.2 score or a leaderboard win.** The current board scores 40 benchmarks; the public reproduction kit still specifies the older panel. No submission was made.','', - 'R2 removes three retained short-text overlaps from continuation data. The V4 initialization still has inherited cross-dataset content overlap and is not certified decontaminated. See retention-audit.json.', '', - f'Selected training step: {best["step"]}/{config["total_steps"]}; selection NLL {best["nll"]:.4f}, accuracy {100*best["acc"]:.2f}%. Rank 16, LR 1e-5, one epoch, 3,072 batch tokens, 2,048 training-state tokens.','', - '| Held-out accuracy (%) | V4 | V5 | Change (points) |','|---|---:|---:|---:|'] - for ds in ['test','test_v5_new','test_v4_new','test_v4_tools']: - a=held['v4'][ds]['overall']['acc'];b=held['v5'][ds]['overall']['acc'];lines.append(f'| {ds} | {100*a:.2f} | {100*b:.2f} | {100*(b-a):+.2f} |') - lines+=['','| New held-out family | Released Jet | V4 | V5 |','|---|---:|---:|---:|'] - for family,a in held['v4']['test_v5_new']['by_source'].items(): - b=held['v5']['test_v5_new']['by_source'][family];lines.append(f'| {family} | {100*released_new["by_source"][family]["acc"]:.2f} | {100*a["acc"]:.2f} | {100*b["acc"]:.2f} |') - lines+=['','## Native benchmark metrics on fixed partial samples','', - '| Benchmark | Metric | Requests | Released Jet | V4 | V5 | V5 two-order |','|---|---|---:|---:|---:|---:|---:|'] - def fmt(x):return '—' if x is None else f'{100*x:.2f}' - baseline_bench={} - for path in [ROOT/'docs/training/jet-v3/diagnostic-baseline.json',D/'baseline-expanded.json']: - if path.exists(): - for b in read(path)['summary']['benchmarks']:baseline_bench[b['dataset']]=b - for name,a in bench['v4'].items(): - b=bench['v5'][name];c=bench.get('v5_two_order',{}).get(name,{}) - lines.append(f'| {name} | {a["metric"]} | {a["requests"]} | {fmt(baseline_bench.get(name,{}).get("score"))} | {fmt(a["score"])} | {fmt(b["score"])} | {fmt(c.get("score"))} |') - lines+=['','## Fixed regression checks','', '| Benchmark | Requests | Released Jet | V4 | V5 | V5 two-order |','|---|---:|---:|---:|---:|---:|'] - for label,item in regressions.items(): - values=item['scores'];lines.append(f'| {label} | {item["n"]} | {fmt(values["baseline"])} | {fmt(values["v4"])} | {fmt(values["v5"])} | {fmt(values.get("v5_two_order"))} |') - lines+=['','All reported values are native metrics multiplied by 100. A missing aggregate remains missing. Samples are small and results are descriptive, not statistical significance claims. MMLU and ARC are diagnostic-only and are not in the current 0.2 index panel.','', - f'Option-order policy chosen on {order["n"]} selection-only choice rows: **{order["selected"]}**. The two-order engine averages original/reversed probabilities using exact option keys; it keeps complete inputs and approximately doubles forward-pass work. Selection required lower NLL without lower macro family accuracy.','', - 'The first V5 attempt was stopped because three retained utterances overlapped native evaluation content. R2 removes them from continuation data. The V4 initialization still has inherited cross-dataset content overlap; this checkpoint is not certified decontaminated. See retention-audit.json.', '', - 'Single-order metrics remain separately reported. No test or benchmark score selected training steps, temperatures, or the option-order policy. Existing validation/test files and published weights were preserved.','', - 'The data protocol, source revisions and limitations are recorded in protocol.md and data/train_v5.provenance.json. Native annotations and group/source-ID separation were independently audited.'] - (D/'summary.md').write_text('\n'.join(lines)+'\n') - result={'selected_checkpoint':best,'training_config':config,'adapter_sha256':sha(adapter/'adapters.safetensors'),'calibration':read(adapter/'calibration.json'),'heldout':held,'released_new_holdout':released_new,'released_benchmarks':baseline_bench,'benchmarks':bench,'regressions':regressions,'inference_selection':order,'data_provenance_sha256':sha(ROOT/'data/train_v5.provenance.json'),'retention_audit':read(D/'retention-audit.json'),'r2_training_sha256':sha(ROOT/'data/train_v5_r2.jsonl')} - (D/'results.json').write_text(json.dumps(result,indent=2)+'\n') - out=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs');out.mkdir(parents=True,exist_ok=True) - shutil.copy2(D/'summary.md',out/'jet-v5-report.md');shutil.copy2(D/'results.json',out/'jet-v5-results.json') - print('\n'.join(lines)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/final-code/src/decision_index_engine.py b/docs/training/jet-v5/final-code/src/decision_index_engine.py deleted file mode 100644 index 71ce9ff6f10fb7875fb90cab6eb3bffe80861da9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/decision_index_engine.py +++ /dev/null @@ -1,133 +0,0 @@ -"""Jet's native MLX readout through the official Decision Index engine API. - -Install with ``uv sync --extra benchmark``. No benchmark identity or gold labels -are accepted by this adapter. The full native prompt is preserved, or rejected. -""" -from __future__ import annotations - -import hashlib -import json -import math -from pathlib import Path - -import numpy as np -from decision_index.engines import Engine, Unsupported -from huggingface_hub import snapshot_download -from transformers import AutoTokenizer - -from format import MAX_CHOICE_OPTIONS, Question, build_prompt, label_token_ids - -MODEL_ID = "michaljach/jet" -REVISION = "8a97cfea2df622bb03f5dc9b02567e21abd2551c" -DEFAULT_MAX_TOKENS = 8192 - - -def as_text(value): - """Mechanical JSON rendering for structured instructions/descriptions.""" - return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False) - - -def prepare_request(tokenizer, state, questions, max_tokens): - prepared = [] - for name, original in questions.items(): - if original["type"] != "choice": - raise Unsupported(f"Only suite choice questions supported: {original['type']}") - criteria = original["criteria"] - if not 2 <= len(criteria) <= MAX_CHOICE_OPTIONS: - raise Unsupported(f"Option count {len(criteria)} outside 2..{MAX_CHOICE_OPTIONS}") - q = Question.from_dict({ - "type": "choice", "instructions": as_text(original["instructions"]), - "criteria": {key: as_text(value) for key, value in criteria.items()}, - }) - # Do not call inference.encode: its application default truncates states. - ids = tokenizer.encode(build_prompt(tokenizer, state, q), add_special_tokens=False) - if len(ids) > max_tokens: - raise Unsupported(f"Full prompt has {len(ids)} tokens; declared limit {max_tokens}") - prepared.append((name, q, ids, label_token_ids(tokenizer, q))) - return prepared - - -def load_assets(model=MODEL_ID, revision=REVISION): - path = Path(snapshot_download(model, revision=revision, allow_patterns=[ - "config.json", "calibration.json", "tokenizer*", "special_tokens_map.json", - "added_tokens.json", "vocab.json", "merges.txt", "chat_template*", - ])) - return path, AutoTokenizer.from_pretrained(path), json.loads((path / "config.json").read_text()) - - -class JetEngine(Engine): - name = "jet" - latency = "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - - def __init__(self, model=MODEL_ID, revision=REVISION, max_tokens=DEFAULT_MAX_TOKENS, - max_cached_tokens=32768, adapter=None, **options): - super().__init__(**options) - import mlx.core as mx - from mlx_lm import load - - self.mx = mx - mx.set_cache_limit(1_000_000_000) - path, self.tokenizer, config = load_assets(model, revision) - self.max_tokens = int(max_tokens) - self.max_cached_tokens = int(max_cached_tokens) - if not 1 <= self.max_tokens <= config["max_position_embeddings"]: - raise ValueError("max_tokens must be positive and within the checkpoint context window") - if self.max_cached_tokens < 1: - raise ValueError("max_cached_tokens must be positive") - calibration = Path(adapter) / "calibration.json" if adapter else path / "calibration.json" - self.temperature = float(json.loads(calibration.read_text())["choice"]) - if not math.isfinite(self.temperature) or self.temperature <= 0: - raise ValueError("Invalid published choice temperature") - # Pin weights to the SAME resolved snapshot as the tokenizer/config. - snapshot_download(model, revision=path.name, allow_patterns=["*.safetensors", "*.safetensors.index.json"]) - self.model, _ = load(str(path), adapter_path=adapter) - self.model.eval() - self.provenance = { - "model": model, "revision": path.name, "backend": "mlx", - "max_tokens": self.max_tokens, "checkpoint_context": config["max_position_embeddings"], - "max_cached_tokens": self.max_cached_tokens, "choice_temperature": self.temperature, - "adapter_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), - "format_sha256": hashlib.sha256(Path(__file__).with_name("format.py").read_bytes()).hexdigest(), - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - } - if adapter: - self.provenance["lora_adapter"] = str(Path(adapter).resolve()) - self.provenance["lora_weights_sha256"] = hashlib.sha256( - (Path(adapter) / "adapters.safetensors").read_bytes()).hexdigest() - self.provenance["calibration_sha256"] = hashlib.sha256(calibration.read_bytes()).hexdigest() - - def __call__(self, state, questions): - from model import shared_prefix_label_logits - - prepared = prepare_request(self.tokenizer, state, questions, self.max_tokens) - if not prepared: - raise ValueError("At least one question is required") - logits = shared_prefix_label_logits( - self.model, [p[2] for p in prepared], [p[3] for p in prepared], - max_cached_tokens=self.max_cached_tokens, - ) - answers, raw = {}, {} - for i, (name, q, ids, labels) in enumerate(prepared): - z = logits[i, :len(labels)].astype(np.float64) / self.temperature - p = np.exp(z - z.max()) - p /= p.sum() - if not np.isfinite(p).all(): - raise ValueError("Non-finite model probabilities") - keys = q.keys - answers[name] = { - "type": "choice", "choice": keys[int(np.argmax(p))], - "probabilities": dict(zip(keys, map(float, p))), - "confidence": float(p.max()), - } - raw[name] = {"label_logits": logits[i, :len(labels)].tolist(), "prompt_tokens": len(ids)} - return {"model": "jet", "answers": answers, - "usage": {"input_tokens": sum(len(p[2]) for p in prepared)}}, raw - - def synchronize(self): - self.mx.synchronize() - - def runtime(self): - import platform - from importlib.metadata import version - return {"platform": platform.platform(), "mlx": version("mlx"), - "mlx_lm": version("mlx-lm"), "device": str(self.mx.default_device())} diff --git a/docs/training/jet-v5/final-code/src/decision_index_ensemble.py b/docs/training/jet-v5/final-code/src/decision_index_ensemble.py deleted file mode 100644 index 2d1c5954306b164274ad5dc77af04062fcb85692..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/decision_index_ensemble.py +++ /dev/null @@ -1,37 +0,0 @@ -"""Two-order Jet readout; averages probabilities by original option key. - -Applies uniformly to every choice question. Preserves all inputs and refuses -requests if either complete prompt exceeds capacity. No benchmark-specific data. -""" -import hashlib -from pathlib import Path -import numpy as np -from decision_index_engine import JetEngine - - -def combine(first, second): - answers={} - if set(first['answers'])!=set(second['answers']):raise ValueError('Question mismatch') - for name,a in first['answers'].items(): - b=second['answers'][name] - keys=list(a['probabilities']) - if set(keys)!=set(b['probabilities']):raise ValueError('Option mismatch') - p=np.array([(a['probabilities'][k]+b['probabilities'][k])/2 for k in keys],dtype=float) - if not np.isfinite(p).all() or (p<0).any() or p.sum()<=0:raise ValueError('Invalid probabilities') - p/=p.sum() - answers[name]={'type':'choice','choice':keys[int(p.argmax())], - 'probabilities':dict(zip(keys,map(float,p))),'confidence':float(p.max())} - return {'model':'jet-two-order','answers':answers,'usage':{'input_tokens':first['usage']['input_tokens']+second['usage']['input_tokens']}} - - -class TwoOrderJetEngine(JetEngine): - name='jet-two-order' - def __init__(self,**options): - super().__init__(**options) - self.provenance.update(choice_readout='Mean of original and reversed option-order probabilities, aligned by exact original keys.', - ensemble_code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest()) - def __call__(self,state,questions): - original,raw_original=super().__call__(state,questions) - reversed_questions={name:dict(q,criteria=dict(reversed(list(q['criteria'].items())))) for name,q in questions.items()} - reverse,raw_reverse=super().__call__(state,reversed_questions) - return combine(original,reverse),{'original':raw_original,'reversed':raw_reverse,'single_order_answers':original['answers']} diff --git a/docs/training/jet-v5/final-code/src/evaluate.py b/docs/training/jet-v5/final-code/src/evaluate.py deleted file mode 100644 index 0fae3a5346f00745f77e1703a3e4806ee49ba073..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/evaluate.py +++ /dev/null @@ -1,190 +0,0 @@ -"""Evaluate and calibrate a Jet model. - - jet-eval --adapter adapters/jet --data data/test.jsonl - jet-eval --data data/test.jsonl # untrained base model, as a baseline - jet-calibrate --adapter adapters/jet --data data/val.jsonl -""" - -from __future__ import annotations - -import argparse -import json -import time -from collections import defaultdict -from pathlib import Path - -import mlx.core as mx -import numpy as np - -from format import Question -from model import DEFAULT_BASE_MODEL, DEFAULT_MAX_STATE_TOKENS, Jet - -TEMPERATURE_GRID = np.exp(np.linspace(np.log(0.25), np.log(8.0), 121)) - - -def load_data(path: Path, limit: int | None) -> list[dict]: - rows = [json.loads(line) for line in path.open()] - return rows[:limit] if limit else rows - - -def adapter_base(adapter: str | None, fallback: str) -> str: - if adapter and (cfg := Path(adapter) / "adapter_config.json").exists(): - config = json.loads(cfg.read_text()) - base = config.get("base_model", fallback) - if config.get("base_revision"): - from huggingface_hub import snapshot_download - return snapshot_download(base, revision=config["base_revision"]) - return base - return fallback - - -def collect_logits(jet: Jet, rows: list[dict], batch_size: int) -> tuple[list[np.ndarray], float]: - """Raw label logits for every row, batched by prompt length. Returns (logits, ms per question).""" - # Batch shapes vary, so MLX's freed-buffer cache otherwise grows until CUDA runs out (as in jet-train). - mx.set_cache_limit(1_000_000_000) - items = [(r["state"], Question.from_dict(r["question"])) for r in rows] - order = sorted(range(len(items)), key=lambda i: len(str(items[i][0]))) - logits: list[np.ndarray | None] = [None] * len(items) - started = time.perf_counter() - for start in range(0, len(order), batch_size): - idx = order[start : start + batch_size] - out, _ = jet.raw_logits([items[i] for i in idx]) - for i, z in zip(idx, out): - logits[i] = z - print(f"\r {min(start + batch_size, len(order))}/{len(order)}", end="", flush=True) - print() - return logits, (time.perf_counter() - started) * 1000 / max(len(items), 1) - - -def softmax(z: np.ndarray, t: float) -> np.ndarray: - z = z / t - e = np.exp(z - z.max()) - return e / e.sum() - - -def nll(logits: list[np.ndarray], targets: list[np.ndarray], t: float) -> float: - return float(np.mean([-(tg * np.log(np.clip(softmax(z, t), 1e-12, 1))).sum() for z, tg in zip(logits, targets)])) - - -def metrics(probs: list[np.ndarray], targets: list[np.ndarray], bins: int = 10) -> dict[str, float]: - top = np.array([p.max() for p in probs]) - hit = np.array([p.argmax() == t.argmax() for p, t in zip(probs, targets)], dtype=float) - # ECE over the model's top prediction, scored against the (possibly soft) target's argmax. - ece = 0.0 - for lo in np.linspace(0, 1, bins, endpoint=False): - sel = (top > lo) & (top <= lo + 1 / bins) - if sel.any(): - ece += sel.mean() * abs(top[sel].mean() - hit[sel].mean()) - return { - "n": len(probs), - "acc": float(hit.mean()), - "nll": float(np.mean([-(t * np.log(np.clip(p, 1e-12, 1))).sum() for p, t in zip(probs, targets)])), - "brier": float(np.mean([((p - t) ** 2).sum() for p, t in zip(probs, targets)])), - "ece": float(ece), - } - - -def rank(x: np.ndarray) -> np.ndarray: - """Average ranks (ties share their mean rank), for Spearman correlation.""" - order = np.argsort(x, kind="stable") - ranks = np.empty(len(x)) - ranks[order] = np.arange(len(x)) - for v in np.unique(x): - tie = x == v - ranks[tie] = ranks[tie].mean() - return ranks - - -def ordinal_metrics(probs: list[np.ndarray], targets: list[np.ndarray]) -> dict[str, float]: - """Score questions: compare expected levels, normalized to 0-1 so scales of any size mix. - - mae: mean |E[pred] - E[target]| on the 0-1 scale; within1: argmax level within one of the - target's expected level; spearman: rank correlation of expected scores. - """ - pred = np.array([np.dot(np.arange(len(p)), p) / (len(p) - 1) for p in probs]) - true = np.array([np.dot(np.arange(len(t)), t) / (len(t) - 1) for t in targets]) - within = [abs(int(p.argmax()) - np.dot(np.arange(len(t)), t)) <= 1 for p, t in zip(probs, targets)] - spearman = float(np.corrcoef(rank(pred), rank(true))[0, 1]) if len(pred) > 2 and true.std() > 0 else float("nan") - return {"mae": float(np.abs(pred - true).mean()), "within1": float(np.mean(within)), "spearman": spearman} - - -def print_table(title: str, groups: dict[str, dict[str, float]]) -> None: - print(f"\n{title}") - print(f"{'':<42}{'n':>6}{'acc':>8}{'nll':>8}{'brier':>8}{'ece':>8}{'mae':>8}{'±1':>8}{'rho':>8}") - for name, m in sorted(groups.items()): - ordinal = "".join(f"{m[k]:>8.3f}" for k in ("mae", "within1", "spearman") if k in m) - print(f"{name[:41]:<42}{m['n']:>6}{m['acc']:>8.3f}{m['nll']:>8.3f}{m['brier']:>8.3f}{m['ece']:>8.3f}{ordinal}") - - -def main_eval() -> None: - ap = argparse.ArgumentParser(description="Evaluate a Jet model.") - ap.add_argument("--adapter", default=None, help="adapter dir; omit to evaluate the base model") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--data", type=Path, default=Path("data/test.jsonl")) - ap.add_argument("--limit", type=int, default=None) - ap.add_argument("--batch-size", type=int, default=16) - ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS) - ap.add_argument("--json", type=Path, default=None, help="also write metrics as JSON") - args = ap.parse_args() - - jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens) - rows = load_data(args.data, args.limit) - logits, ms = collect_logits(jet, rows, args.batch_size) - probs = [softmax(z, jet.temperatures[r["question"]["type"]]) for z, r in zip(logits, rows)] - targets = [np.array(r["target"]) for r in rows] - - by_source, by_type = defaultdict(list), defaultdict(list) - for i, r in enumerate(rows): - source = "distill (all)" if r["source"].startswith("distill:") else r["source"] - by_source[source].append(i) - by_type[r["question"]["type"]].append(i) - - def group(ix: list[int]) -> dict[str, float]: - m = metrics([probs[i] for i in ix], [targets[i] for i in ix]) - scored = [i for i in ix if rows[i]["question"]["type"] == "score"] - if len(scored) == len(ix): - m |= ordinal_metrics([probs[i] for i in scored], [targets[i] for i in scored]) - return m - - overall = metrics(probs, targets) - print_table("by source", {k: group(v) for k, v in by_source.items()}) - print_table("by type", {k: group(v) for k, v in by_type.items()}) - print_table("overall", {"all": overall}) - print(f"\n{ms:.1f} ms/question (batched, batch size {args.batch_size}); temperatures {jet.temperatures}") - if args.json: - args.json.write_text(json.dumps({ - "overall": overall, - "by_source": {k: group(v) for k, v in by_source.items()}, - "by_type": {k: group(v) for k, v in by_type.items()}, - "ms_per_question": ms, - "max_state_tokens": args.max_state_tokens, - }, indent=2)) - - -def main_calibrate() -> None: - ap = argparse.ArgumentParser(description="Fit per-type softmax temperatures on held-out data.") - ap.add_argument("--adapter", required=True) - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--data", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--limit", type=int, default=3000) - ap.add_argument("--batch-size", type=int, default=16) - ap.add_argument("--max-state-tokens", type=int, default=DEFAULT_MAX_STATE_TOKENS) - args = ap.parse_args() - - jet = Jet(adapter_base(args.adapter, args.base_model), args.adapter, args.max_state_tokens) - rows = load_data(args.data, args.limit) - logits, _ = collect_logits(jet, rows, args.batch_size) - - temperatures = {} - for qtype in ("choice", "score", "noul"): - ix = [i for i, r in enumerate(rows) if r["question"]["type"] == qtype] - if not ix: - temperatures[qtype] = 1.0 - continue - zs, ts = [logits[i] for i in ix], [np.array(rows[i]["target"]) for i in ix] - scores = [nll(zs, ts, t) for t in TEMPERATURE_GRID] - best = float(TEMPERATURE_GRID[int(np.argmin(scores))]) - temperatures[qtype] = round(best, 4) - print(f"{qtype:>6}: n={len(ix)} T={best:.3f} nll {nll(zs, ts, 1.0):.4f} -> {min(scores):.4f}") - (Path(args.adapter) / "calibration.json").write_text(json.dumps(temperatures, indent=2)) - print(f"wrote {Path(args.adapter) / 'calibration.json'}") diff --git a/docs/training/jet-v5/final-code/src/format.py b/docs/training/jet-v5/final-code/src/format.py deleted file mode 100644 index 16ad75d3b250cde3629d60f849bd84cb61fa246a..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/format.py +++ /dev/null @@ -1,143 +0,0 @@ -"""Prompt format shared by training and inference. - -Every question is reduced to "pick one label token". The model is run once over -the prompt and we read the next-token logits restricted to the label tokens: - -- choice: one label per option (A..Z, then single-token AA, AB, ...), up to 255 options -- score: one digit per level (0..9), 2-10 levels, ordered low -> high -- noul: "no" / "yes" -""" - -from __future__ import annotations - -import itertools -import json -import string -from dataclasses import dataclass -from typing import Any, Literal - -QuestionType = Literal["choice", "score", "noul"] - -MAX_CHOICE_OPTIONS = 255 -MIN_SCORE_LEVELS, MAX_SCORE_LEVELS = 2, 10 - -SYSTEM_PROMPT = ( - "You are Jet, a decision model. Read the state and the question, then answer " - "with exactly one label from the allowed labels." -) - - -@dataclass(frozen=True) -class Question: - type: QuestionType - instructions: str - # choice: {key: description}; score: [level descriptions, low -> high]; noul: None - criteria: dict[str, str] | list[str] | None = None - - @property - def keys(self) -> list[str]: - """Answer keys in label order (option keys, level descriptions, or no/yes).""" - if self.type == "choice": - return list(self.criteria) - if self.type == "score": - return list(self.criteria) - return ["no", "yes"] - - @classmethod - def from_dict(cls, d: dict[str, Any]) -> Question: - q = cls(type=d["type"], instructions=d["instructions"], criteria=d.get("criteria")) - q.validate() - return q - - def to_dict(self) -> dict[str, Any]: - d: dict[str, Any] = {"type": self.type, "instructions": self.instructions} - if self.criteria is not None: - d["criteria"] = self.criteria - return d - - def validate(self) -> None: - if self.type not in ("choice", "score", "noul"): - raise ValueError(f"unknown question type {self.type!r}") - if not isinstance(self.instructions, str) or not self.instructions.strip(): - raise ValueError("instructions must be a non-empty string") - if self.type == "choice": - if not isinstance(self.criteria, dict) or not 2 <= len(self.criteria) <= MAX_CHOICE_OPTIONS: - raise ValueError(f"choice criteria must be an object with 2-{MAX_CHOICE_OPTIONS} options") - elif self.type == "score": - if not isinstance(self.criteria, list) or not MIN_SCORE_LEVELS <= len(self.criteria) <= MAX_SCORE_LEVELS: - raise ValueError(f"score criteria must be a list of {MIN_SCORE_LEVELS}-{MAX_SCORE_LEVELS} levels") - elif self.criteria is not None: - raise ValueError("noul questions take no criteria") - - -_CHOICE_LABELS: dict[int, list[str]] = {} - - -def choice_labels(tokenizer, n: int) -> list[str]: - """A..Z, then the two-letter labels AA, AB, ... that are single tokens for this tokenizer.""" - key = id(tokenizer) - if key not in _CHOICE_LABELS: - letters = string.ascii_uppercase - pairs = ("".join(p) for p in itertools.product(letters, repeat=2)) - candidates = list(letters) + [p for p in pairs if len(tokenizer.encode(p, add_special_tokens=False)) == 1] - if len(candidates) < MAX_CHOICE_OPTIONS: - raise ValueError(f"tokenizer only has {len(candidates)} single-token choice labels") - _CHOICE_LABELS[key] = candidates[:MAX_CHOICE_OPTIONS] - return _CHOICE_LABELS[key][:n] - - -def labels_for(tokenizer, q: Question) -> list[str]: - if q.type == "choice": - return choice_labels(tokenizer, len(q.criteria)) - if q.type == "score": - return [str(i) for i in range(len(q.criteria))] - return ["no", "yes"] - - -def render_state(state: Any) -> str: - if isinstance(state, str): - return state - if isinstance(state, list) and all(isinstance(s, str) for s in state): - return "\n\n".join(state) - return json.dumps(state, ensure_ascii=False, indent=2) - - -def render_user(state: Any, q: Question, labels: list[str]) -> str: - parts = [f"\n{render_state(state)}\n", ""] - if q.type == "choice": - opts = "\n".join(f"{lab}: {key}: {desc}" for lab, (key, desc) in zip(labels, q.criteria.items())) - parts += [ - f"Question: {q.instructions}", - f"Options:\n{opts}", - f"Answer with the label of the best option ({labels[0]}-{labels[-1]}).", - ] - elif q.type == "score": - levels = "\n".join(f"{lab}: {desc}" for lab, desc in zip(labels, q.criteria)) - parts += [ - f"Question: {q.instructions}", - f"Scale (lowest to highest):\n{levels}", - f"Answer with the level number (0-{len(labels) - 1}).", - ] - else: - parts += [f"Question: {q.instructions}", "Answer yes or no."] - return "\n".join(parts) - - -def build_prompt(tokenizer, state: Any, q: Question) -> str: - return tokenizer.apply_chat_template( - [{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": render_user(state, q, labels_for(tokenizer, q))}], - add_generation_prompt=True, - tokenize=False, - enable_thinking=False, - ) - - -def _label_token_id(tokenizer, label: str) -> int: - ids = tokenizer.encode(label, add_special_tokens=False) - if len(ids) != 1: - raise ValueError(f"label {label!r} is not a single token for this tokenizer") - return ids[0] - - -def label_token_ids(tokenizer, q: Question) -> list[int]: - return [_label_token_id(tokenizer, lab) for lab in labels_for(tokenizer, q)] diff --git a/docs/training/jet-v5/final-code/src/inference.py b/docs/training/jet-v5/final-code/src/inference.py deleted file mode 100644 index da8630b4dfb9168d59dd22c64e1010ab892bdfbe..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/inference.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Backend-independent Jet prompt encoding and typed answers.""" -from __future__ import annotations -import math -from typing import Any -import numpy as np -from format import Question, build_prompt, render_state - -DEFAULT_MAX_STATE_TOKENS = 4096 - -def encode(tokenizer, state: Any, q: Question, max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS) -> list[int]: - """Tokenize a prompt, truncating the middle of an over-long state.""" - text = render_state(state) - ids = tokenizer.encode(text, add_special_tokens=False) - if len(ids) > max_state_tokens: - half = max_state_tokens // 2 - text = tokenizer.decode(ids[:half]) + "\n[...]\n" + tokenizer.decode(ids[-half:]) - return tokenizer.encode(build_prompt(tokenizer, text, q), add_special_tokens=False) - -def summarize(q: Question, probs: np.ndarray) -> dict[str, Any]: - """Turn a label distribution into the typed answer for the question.""" - probs = probs.astype(np.float64) - k = len(probs) - entropy = -float(np.sum(probs * np.log(np.clip(probs, 1e-12, 1.0)))) - confidence = round(1.0 - entropy / math.log(k), 4) - if q.type == "choice": - keys = q.keys - return { - "type": "choice", - "choice": keys[int(np.argmax(probs))], - "probabilities": {key: round(float(p), 4) for key, p in zip(keys, probs)}, - "confidence": confidence, - } - if q.type == "score": - return { - "type": "score", - "score": round(float(np.dot(np.arange(k), probs)), 4), - "level": q.criteria[int(np.argmax(probs))], - "probabilities": [round(float(p), 4) for p in probs], - "confidence": confidence, - } - return {"type": "noul", "probability": round(float(probs[1]), 4), "confidence": confidence} diff --git a/docs/training/jet-v5/final-code/src/model.py b/docs/training/jet-v5/final-code/src/model.py deleted file mode 100644 index 5e28e2317876fcd76b3edafaee434bf6740ed293..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/model.py +++ /dev/null @@ -1,151 +0,0 @@ -"""Jet inference: one forward pass per batch of questions, no generation.""" - -from __future__ import annotations - -import json -import time -from pathlib import Path -from typing import Any - -import mlx.core as mx -import numpy as np -from huggingface_hub import snapshot_download -from mlx_lm import load -from mlx_lm.models.cache import make_prompt_cache - -from format import Question, label_token_ids -from inference import DEFAULT_MAX_STATE_TOKENS, encode, summarize - -DEFAULT_BASE_MODEL = "mlx-community/Qwen3-0.6B-bf16" -NEG_INF = -1e9 - - -def label_logits(model, tokens: mx.array, lengths: mx.array, label_ids: mx.array, label_mask: mx.array) -> mx.array: - """Next-token logits after each prompt, restricted to its label tokens. - - tokens: (B, L) right-padded prompts; lengths: (B,) true lengths; - label_ids / label_mask: (B, K) label token ids padded to the widest question. - Returns (B, K) logits with padded labels set to -inf. - """ - hidden = model.model(tokens) - last = hidden[mx.arange(tokens.shape[0]), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - picked = mx.take_along_axis(logits, label_ids, axis=-1).astype(mx.float32) - return mx.where(label_mask, picked, NEG_INF) - - -def head_logits(model, hidden: mx.array) -> mx.array: - if getattr(model.args, "tie_word_embeddings", False): - return model.model.embed_tokens.as_linear(hidden) - return model.lm_head(hidden) - - -def shared_prefix_label_logits( - model, seqs: list[list[int]], label_lists: list[list[int]], max_cached_tokens: int = 32_768 -) -> np.ndarray: - """Like label_logits, but encodes the prompts' common prefix (system + state) once. - - The prefix KV cache is replicated across a chunk of question suffixes, so each - question only pays for its own few dozen tokens. Chunks are sized so that - replicated prefix tokens stay under max_cached_tokens. - """ - common = min(len(s) for s in seqs) - 1 # every suffix keeps at least one token - for i in range(common): - if any(s[i] != seqs[0][i] for s in seqs): - common = i - break - prefix_cache = make_prompt_cache(model) - if common: - model.model(mx.array(seqs[0][:common])[None], cache=prefix_cache) - prefix_state = [c.state for c in prefix_cache] if common else None - if prefix_state: - mx.eval(prefix_state) - - out = np.full((len(seqs), max(map(len, label_lists))), NEG_INF, dtype=np.float32) - chunk = max(1, max_cached_tokens // max(common, 1)) - for start in range(0, len(seqs), chunk): - suffixes = [s[common:] for s in seqs[start : start + chunk]] - b = len(suffixes) - cache = make_prompt_cache(model) - if prefix_state: - for c, (k, v) in zip(cache, prefix_state): - c.state = (mx.repeat(k, b, axis=0), mx.repeat(v, b, axis=0)) - tokens, lengths = pad_batch(suffixes) - hidden = model.model(tokens, cache=cache) - last = hidden[mx.arange(b), lengths - 1][:, None, :] - logits = head_logits(model, last)[:, 0, :] - ids, mask = pad_labels(label_lists[start : start + chunk]) - picked = mx.where(mask, mx.take_along_axis(logits, ids, axis=-1).astype(mx.float32), NEG_INF) - out[start : start + b, : picked.shape[1]] = np.array(picked) - return out - - -def pad_batch(seqs: list[list[int]], pad: int = 0) -> tuple[mx.array, mx.array]: - width = max(len(s) for s in seqs) - arr = np.full((len(seqs), width), pad, dtype=np.int32) - for i, s in enumerate(seqs): - arr[i, : len(s)] = s - return mx.array(arr), mx.array([len(s) for s in seqs], dtype=mx.int32) - - -def pad_labels(label_lists: list[list[int]]) -> tuple[mx.array, mx.array]: - width = max(len(ls) for ls in label_lists) - ids = np.zeros((len(label_lists), width), dtype=np.int32) - mask = np.zeros((len(label_lists), width), dtype=bool) - for i, ls in enumerate(label_lists): - ids[i, : len(ls)] = ls - mask[i, : len(ls)] = True - return mx.array(ids), mx.array(mask) - - -class Jet: - def __init__( - self, - base_model: str = DEFAULT_BASE_MODEL, - adapter_path: str | None = None, - max_state_tokens: int = DEFAULT_MAX_STATE_TOKENS, - ): - self.model, self.tokenizer = load(base_model, adapter_path=adapter_path) - self.model.eval() - self.max_state_tokens = max_state_tokens - self.temperatures = {"choice": 1.0, "score": 1.0, "noul": 1.0} - # Calibration lives next to the adapter, or inside a fused model dir. A Hub repo id has no dir - # of its own, so look in its local snapshot (fetching calibration.json if the repo has one). - model_dir = Path(base_model) if Path(base_model).exists() else Path(snapshot_download(base_model, allow_patterns=["calibration.json"])) - for directory in (adapter_path, model_dir): - if directory and (calib := Path(directory) / "calibration.json").exists(): - self.temperatures.update(json.loads(calib.read_text())) - break - - def raw_logits(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - """Uncalibrated label logits for (state, question) pairs, plus the number of - input tokens. With shared_state, the common state prefix is encoded once.""" - seqs = [encode(self.tokenizer, s, q, self.max_state_tokens) for s, q in items] - label_lists = [label_token_ids(self.tokenizer, q) for _, q in items] - if shared_state and len(items) > 1: - out = shared_prefix_label_logits(self.model, seqs, label_lists) - else: - tokens, lengths = pad_batch(seqs) - ids, mask = pad_labels(label_lists) - out = np.array(label_logits(self.model, tokens, lengths, ids, mask)) - return [out[i, : len(q.keys)] for i, (_, q) in enumerate(items)], sum(map(len, seqs)) - - def probabilities(self, items: list[tuple[Any, Question]], shared_state: bool = False) -> tuple[list[np.ndarray], int]: - logits, n_tokens = self.raw_logits(items, shared_state) - result = [] - for (_, q), z in zip(items, logits): - z = z / self.temperatures[q.type] - z = np.exp(z - z.max()) - result.append(z / z.sum()) - return result, n_tokens - - def decide(self, state: Any, questions: dict[str, Question]) -> dict[str, Any]: - """Answer every question about one state together (shared batch).""" - started = time.perf_counter() - items = [(state, q) for q in questions.values()] - probs, input_tokens = self.probabilities(items, shared_state=True) - return { - "answers": {name: summarize(q, p) for (name, q), p in zip(questions.items(), probs)}, - "usage": {"input_tokens": input_tokens}, - "latency_ms": round((time.perf_counter() - started) * 1000, 1), - } diff --git a/docs/training/jet-v5/final-code/src/train.py b/docs/training/jet-v5/final-code/src/train.py deleted file mode 100644 index 816c0765fa1a66b174e71ec7b2b95cf360664903..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/src/train.py +++ /dev/null @@ -1,304 +0,0 @@ -"""LoRA fine-tuning of a small LM into a Jet decision model (MLX, Apple Silicon). - -The loss is cross-entropy between the soft target and the model's next-token -distribution restricted to the question's label tokens, read at the last prompt -position only - exactly what inference computes. Score questions add a ranked -probability score term, so mass far from the target level costs more than mass -next to it (cross-entropy alone treats the levels as unordered). -""" - -from __future__ import annotations - -import argparse -import hashlib -import json -import math -import random -import time -from functools import partial -from pathlib import Path - -import mlx.core as mx -import mlx.nn as nn -import mlx.optimizers as optim -import numpy as np -from mlx.utils import tree_flatten, tree_unflatten -from mlx_lm import load -from huggingface_hub import snapshot_download -from mlx_lm.tuner.trainer import grad_checkpoint -from mlx_lm.tuner.utils import linear_to_lora_layers - -from format import Question, label_token_ids -from model import DEFAULT_BASE_MODEL, encode, label_logits, pad_batch, pad_labels - - -class Example: - __slots__ = ("tokens", "labels", "target", "ordinal", "weight") - - def __init__(self, tokens: list[int], labels: list[int], target: list[float], ordinal: bool, weight: float = 1.0): - self.tokens, self.labels, self.target, self.ordinal = tokens, labels, target, ordinal - self.weight = weight - - -def load_examples(path: Path, tokenizer, max_state_tokens: int, smoothing: float, use_weights: bool = False) -> list[Example]: - out = [] - for line in path.open(): - ex = json.loads(line) - q = Question.from_dict(ex["question"]) - k = len(q.keys) - target = [(1 - smoothing) * t + smoothing / k for t in ex["target"]] - weight = float(ex.get("weight", 1.0)) if use_weights else 1.0 - if not math.isfinite(weight) or weight <= 0: - raise ValueError("training weights must be finite and positive") - out.append(Example(encode(tokenizer, ex["state"], q, max_state_tokens), label_token_ids(tokenizer, q), target, q.type == "score", weight)) - return out - - -def make_batches(examples: list[Example], max_batch_tokens: int, rng: random.Random | None) -> list[list[Example]]: - """Group similar lengths together; each batch's padded size stays under max_batch_tokens.""" - order = sorted(examples, key=lambda e: len(e.tokens)) - batches, cur = [], [] - for ex in order: - width = max(len(ex.tokens), max((len(e.tokens) for e in cur), default=0)) - if cur and width * (len(cur) + 1) > max_batch_tokens: - batches.append(cur) - cur = [] - cur.append(ex) - if cur: - batches.append(cur) - if rng: - rng.shuffle(batches) - return batches - - -def to_arrays(batch: list[Example]): - tokens, lengths = pad_batch([e.tokens for e in batch]) - ids, mask = pad_labels([e.labels for e in batch]) - target = np.zeros(ids.shape, dtype=np.float32) - for i, e in enumerate(batch): - target[i, : len(e.target)] = e.target - ordinal = mx.array([e.ordinal for e in batch]) - levels = mx.array([len(e.labels) for e in batch], dtype=mx.float32) - return tokens, lengths, ids, mask, mx.array(target), ordinal, levels, mx.array([e.weight for e in batch]) - - -def ranked_probability_score(logp: mx.array, target: mx.array, levels: mx.array) -> mx.array: - """Squared distance between predicted and target CDFs, divided by (levels - 1) so it stays - in 0-1 for any scale size. Padded labels have zero mass in both, so they add nothing.""" - cdf_gap = mx.cumsum(mx.exp(logp), axis=-1) - mx.cumsum(target, axis=-1) - return (cdf_gap**2).sum(axis=-1) / (levels - 1) - - -def loss_fn(model, tokens, lengths, ids, mask, target, ordinal, levels, weights, ordinal_weight=0.0): - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - loss = -(target * logp).sum(axis=-1) - if ordinal_weight: - loss = loss + ordinal_weight * mx.where(ordinal, ranked_probability_score(logp, target, levels), 0.0) - return (loss * weights).mean() - - -def evaluate(model, batches: list[list[Example]]) -> dict[str, float]: - total, nll, correct = 0, 0.0, 0 - for batch in batches: - tokens, lengths, ids, mask, target, *_ = to_arrays(batch) - logits = label_logits(model, tokens, lengths, ids, mask) - logp = logits - mx.logsumexp(logits, axis=-1, keepdims=True) - nll += float(-(target * logp).sum()) - correct += int((mx.argmax(logits, axis=-1) == mx.argmax(target, axis=-1)).sum()) - total += len(batch) - return {"nll": nll / total, "acc": correct / total} - - -def save_adapter(model, out: Path, config: dict, optimizer: optim.Optimizer | None = None) -> None: - out.mkdir(parents=True, exist_ok=True) - mx.save_safetensors(str(out / "adapters.safetensors"), dict(tree_flatten(model.trainable_parameters()))) - (out / "adapter_config.json").write_text(json.dumps(config, indent=2)) - if optimizer is not None: - # Adam moments + step, so --resume continues exactly instead of restarting cold. - mx.save_safetensors(str(out / "optimizer.safetensors"), dict(tree_flatten(optimizer.state))) - - -def main() -> None: - ap = argparse.ArgumentParser(description="Fine-tune a Jet decision model with LoRA.") - ap.add_argument("--base-model", default=DEFAULT_BASE_MODEL) - ap.add_argument("--base-revision", default=None, help="pin a Hub base model to this commit") - ap.add_argument("--train", type=Path, default=Path("data/train.jsonl")) - ap.add_argument("--val", type=Path, default=Path("data/val.jsonl")) - ap.add_argument("--out", type=Path, default=Path("adapters/jet")) - ap.add_argument("--epochs", type=float, default=2.0) - ap.add_argument("--lr", type=float, default=1e-4) - ap.add_argument("--warmup", type=int, default=100) - ap.add_argument("--rank", type=int, default=16) - ap.add_argument("--lora-scale", type=float, default=10.0) - ap.add_argument("--dropout", type=float, default=0.05) - ap.add_argument("--lora-layers", type=int, default=-1, help="-1 = all layers") - ap.add_argument("--max-batch-tokens", type=int, default=4096) - ap.add_argument("--max-state-tokens", type=int, default=1024) - ap.add_argument("--smoothing", type=float, default=0.02) - ap.add_argument("--use-example-weights", action="store_true", - help="use positive per-row weights for training only; validation remains unweighted") - ap.add_argument("--ordinal-weight", type=float, default=2.0, - help="weight of the ranked probability score term on score questions; 0 = plain cross-entropy") - ap.add_argument("--grad-checkpoint", action=argparse.BooleanOptionalAction, default=True) - ap.add_argument("--eval-every", type=int, default=250) - ap.add_argument("--val-limit", type=int, default=1000) - ap.add_argument("--cache-limit-gb", type=float, default=1.0, - help="cap on MLX's freed-buffer cache; variable batch shapes otherwise grow it toward all of RAM") - ap.add_argument("--resume", type=Path, default=None, help="adapter dir to continue from") - ap.add_argument("--init-adapter", type=Path, default=None, - help="warm-start adapter weights with a fresh optimizer and schedule") - ap.add_argument("--start-step", type=int, default=None, - help="step the resumed adapter was saved at; read from its optimizer state when present") - ap.add_argument("--seed", type=int, default=0) - args = ap.parse_args() - if args.init_adapter and (args.resume or args.start_step is not None): - ap.error("--init-adapter cannot be combined with --resume or --start-step") - if (args.out / "adapters.safetensors").exists(): - ap.error("output already contains an adapter; choose a new directory") - - args.out.mkdir(parents=True, exist_ok=True) - run_config = {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()} - run_config["data_sha256"] = { - str(p): hashlib.sha256(p.read_bytes()).hexdigest() for p in (args.train, args.val) - } - if args.init_adapter: - source_config = json.loads((args.init_adapter / "adapter_config.json").read_text()) - expected = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - if source_config.get("lora_parameters") != expected: - ap.error("initial adapter LoRA parameters do not match requested parameters") - if source_config.get("base_model") != args.base_model: - ap.error("initial adapter uses a different base model") - if source_config.get("base_revision") not in (None, args.base_revision): - ap.error("initial adapter uses a different base revision") - run_config["init_adapter_sha256"] = hashlib.sha256( - (args.init_adapter / "adapters.safetensors").read_bytes()).hexdigest() - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - mx.set_cache_limit(int(args.cache_limit_gb * 1e9)) - mx.random.seed(args.seed) - rng = random.Random(args.seed) - model_path = args.base_model - if args.base_revision: - model_path = snapshot_download(args.base_model, revision=args.base_revision) - model, tokenizer = load(model_path) - model.freeze() - num_layers = len(model.layers) if args.lora_layers < 0 else args.lora_layers - lora_parameters = {"rank": args.rank, "scale": args.lora_scale, "dropout": args.dropout} - linear_to_lora_layers(model, num_layers, lora_parameters) - if args.init_adapter: - if source_config.get("num_layers") != num_layers: - ap.error("initial adapter layer count does not match") - weights = mx.load(str(args.init_adapter / "adapters.safetensors")) - expected_weights = dict(tree_flatten(model.trainable_parameters())) - if weights.keys() != expected_weights.keys() or any( - weights[k].shape != expected_weights[k].shape for k in expected_weights - ): - ap.error("initial adapter tensor names or shapes do not match") - model.load_weights(list(weights.items()), strict=False) - print(f"initialized weights from {args.init_adapter}; optimizer starts fresh", flush=True) - if args.resume: - model.load_weights(str(args.resume / "adapters.safetensors"), strict=False) - print(f"resumed weights from {args.resume}") - if args.grad_checkpoint: - grad_checkpoint(model.layers[0]) - n_train = sum(v.size for _, v in tree_flatten(model.trainable_parameters())) - print(f"trainable params: {n_train / 1e6:.2f}M") - - train = load_examples(args.train, tokenizer, args.max_state_tokens, args.smoothing, args.use_example_weights) - val = load_examples(args.val, tokenizer, args.max_state_tokens, 0.0) - rng.shuffle(val) - val_batches = make_batches(val[: args.val_limit], args.max_batch_tokens, None) - steps_per_epoch = len(make_batches(train, args.max_batch_tokens, None)) - total_steps = math.ceil(steps_per_epoch * args.epochs) - print(f"train {len(train)} examples, {steps_per_epoch} steps/epoch, {total_steps} steps; val {len(val_batches)} batches") - run_config.update(train_examples=len(train), steps_per_epoch=steps_per_epoch, total_steps=total_steps) - (args.out / "training_config.json").write_text(json.dumps(run_config, indent=2)) - - warmup = max(1, min(args.warmup, total_steps // 10)) - base_schedule = optim.join_schedules( - [optim.linear_schedule(1e-7, args.lr, warmup), optim.cosine_decay(args.lr, total_steps - warmup, args.lr * 0.05)], - [warmup], - ) - optimizer_state = args.resume / "optimizer.safetensors" if args.resume else None - if optimizer_state is not None and optimizer_state.exists(): - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.init(model.trainable_parameters()) - optimizer.state = tree_unflatten(list(mx.load(str(optimizer_state)).items())) - start_step = int(optimizer.state["step"].item()) - print(f"restored optimizer state at step {start_step}") - else: - start_step = args.start_step or 0 - if args.resume: - # Cold Adam moments (MLX AdamW has no bias correction by default) make the first - # updates several times too large; ramp the LR back up to protect the checkpoint. - def schedule(step, start=start_step): - return base_schedule(step) * mx.minimum(1.0, (step - start + 1).astype(mx.float32) / warmup) - print(f"no optimizer state in {args.resume}; re-warming the LR over {warmup} steps") - else: - schedule = base_schedule - optimizer = optim.AdamW(learning_rate=schedule, weight_decay=0.01) - optimizer.state["step"] = mx.array(start_step, mx.uint64) - value_and_grad = nn.value_and_grad(model, partial(loss_fn, ordinal_weight=args.ordinal_weight)) - - adapter_config = { - "fine_tune_type": "lora", - "num_layers": num_layers, - "lora_parameters": lora_parameters, - "base_model": args.base_model, - "base_revision": args.base_revision, - } - model.eval() - best = evaluate(model, val_batches) - # Keep the initial checkpoint eligible; a regression must not replace it merely - # because no trained checkpoint improved the validation loss. - save_adapter(model, args.out, adapter_config, optimizer) - (args.out / "metrics.jsonl").write_text(json.dumps({"step": 0, **best, - "improved": False, "initial": True}) + "\n") - print(f"step 0 val nll {best['nll']:.4f} acc {best['acc']:.3f}") - model.train() - - saved = False - step, seen_tokens, started = start_step, 0, time.perf_counter() - losses: list[float] = [] - while step < total_steps: - for batch in make_batches(train, args.max_batch_tokens, rng): - if step >= total_steps: - break - arrays = to_arrays(batch) - loss, grads = value_and_grad(model, *arrays) - optimizer.update(model, grads) - mx.eval(model.trainable_parameters(), optimizer.state, loss) - step += 1 - losses.append(loss.item()) - seen_tokens += int(arrays[1].sum()) - if step % 25 == 0: - elapsed = time.perf_counter() - started - print( - f"step {step}/{total_steps} loss {np.mean(losses):.4f} lr {optimizer.learning_rate.item():.2e} " - f"{seen_tokens / elapsed:.0f} tok/s peak mem {mx.get_peak_memory() / 1e9:.1f}GB cache {mx.get_cache_memory() / 1e9:.1f}GB", - flush=True, - ) - losses = [] - if step % args.eval_every == 0 or step == total_steps: - model.eval() - metrics = evaluate(model, val_batches) - model.train() - improved = metrics["nll"] < best["nll"] - print(f"step {step} val nll {metrics['nll']:.4f} acc {metrics['acc']:.3f}{' *saved*' if improved else ''}", flush=True) - with (args.out / "metrics.jsonl").open("a") as f: - f.write(json.dumps({"step": step, **metrics, "improved": improved, - "peak_memory_gb": mx.get_peak_memory() / 1e9}) + "\n") - save_adapter(model, args.out / "last", adapter_config, optimizer) - if improved: - best = metrics - save_adapter(model, args.out, adapter_config, optimizer) - saved = True - if not saved: - print("validation never improved; retaining initial weights (final weights in last/)") - print(f"best val nll {best['nll']:.4f} acc {best['acc']:.3f}; adapter in {args.out}") - - -if __name__ == "__main__": - main() diff --git a/docs/training/jet-v5/final-code/tests/test_decision_ensemble.py b/docs/training/jet-v5/final-code/tests/test_decision_ensemble.py deleted file mode 100644 index 3ddb0691afeec5477f7044140d22d3dccc387b28..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/final-code/tests/test_decision_ensemble.py +++ /dev/null @@ -1,31 +0,0 @@ -import unittest -from decision_index_ensemble import combine, TwoOrderJetEngine -from unittest.mock import patch -import copy -class EnsembleTests(unittest.TestCase): - def test_alignment_and_confidence(self): - a={'answers':{'q':{'probabilities':{' X':.9,'x':.1}}},'usage':{'input_tokens':5}} - b={'answers':{'q':{'probabilities':{'x':.7,' X':.3}}},'usage':{'input_tokens':6}} - r=combine(a,b) - self.assertAlmostEqual(r['answers']['q']['probabilities'][' X'],.6) - self.assertEqual(r['answers']['q']['choice'],' X') - self.assertAlmostEqual(r['answers']['q']['confidence'],.6) - self.assertEqual(r['usage']['input_tokens'],11) - self.assertEqual(a['answers']['q']['probabilities'][' X'],.9) - def test_reject_missing_option(self): - a={'answers':{'q':{'probabilities':{'a':1.}}},'usage':{'input_tokens':1}} - b={'answers':{'q':{'probabilities':{'b':1.}}},'usage':{'input_tokens':1}} - with self.assertRaises(ValueError):combine(a,b) - def test_complete_options_and_input_preserved(self): - engine=TwoOrderJetEngine.__new__(TwoOrderJetEngine) - questions={'q':{'type':'choice','instructions':'Choose','criteria':{' A':'first','a':'second','Z':'third'}}} - original=copy.deepcopy(questions);orders=[] - def forward(self,state,qs): - keys=list(qs['q']['criteria']);orders.append(keys) - return {'answers':{'q':{'probabilities':dict(zip(keys,[.5,.3,.2]))}},'usage':{'input_tokens':9}},{} - with patch('decision_index_engine.JetEngine.__call__',forward): - response,_=engine({'nested':['unchanged']},questions) - self.assertEqual(orders,[[' A','a','Z'],['Z','a',' A']]) - self.assertEqual(questions,original) - self.assertEqual(set(response['answers']['q']['probabilities']),set(original['q']['criteria'])) -if __name__=='__main__':unittest.main() diff --git a/docs/training/jet-v5/inference-selection.json b/docs/training/jet-v5/inference-selection.json deleted file mode 100644 index 378ea50d8c6622de4bb092e75d466007eb0082a2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/inference-selection.json +++ /dev/null @@ -1,7838 +0,0 @@ -{ - "n": 600, - "selection_sha256": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3", - "selection_rule": "lower NLL and no decrease in macro family accuracy on selection data only", - "selected": "two_order", - "summary": { - "single": { - "nll": 0.5060660150465175, - "accuracy": 0.8083333333333333, - "macro_family_accuracy": 0.8091601635079896 - }, - "two_order": { - "nll": 0.4959336955110757, - "accuracy": 0.82, - "macro_family_accuracy": 0.8200545026631982 - } - }, - "seconds": 52.090785968999626, - "engine_provenance": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "rows": [ - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.7962438442101873, - "correct": 0 - }, - "two_order": { - "nll": 0.9326626633977168, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.1658148583923762, - "correct": 1 - }, - "two_order": { - "nll": 0.6361975130942004, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.40458940651313996, - "correct": 1 - }, - "two_order": { - "nll": 0.4645615321752626, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.6923660044766176, - "correct": 0 - }, - "two_order": { - "nll": 2.5689158116508977, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.5105117409320202, - "correct": 1 - }, - "two_order": { - "nll": 0.5105117409320202, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.7116028977998379, - "correct": 1 - }, - "two_order": { - "nll": 0.636643120550807, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.013311021623458414, - "correct": 1 - }, - "two_order": { - "nll": 0.01900236261466538, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.11967966518987716, - "correct": 1 - }, - "two_order": { - "nll": 0.07864371912767414, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.6739367247990816, - "correct": 1 - }, - "two_order": { - "nll": 0.6116300304401876, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.9521714635828996, - "correct": 0 - }, - "two_order": { - "nll": 0.9129818841519277, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.2317984582214673, - "correct": 1 - }, - "two_order": { - "nll": 0.22241059203395605, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.822650481883524, - "correct": 0 - }, - "two_order": { - "nll": 1.8369134318728584, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.009705452324279305, - "correct": 1 - }, - "two_order": { - "nll": 0.006271733492539679, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.011008847399542228, - "correct": 1 - }, - "two_order": { - "nll": 0.015344276704632107, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.014609784756570106, - "correct": 1 - }, - "two_order": { - "nll": 0.017893040178622394, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.036413925691886916, - "correct": 1 - }, - "two_order": { - "nll": 0.043875428720715634, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0027186777265180476, - "correct": 1 - }, - "two_order": { - "nll": 0.0025971740514496547, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.0218206085452135, - "correct": 1 - }, - "two_order": { - "nll": 0.032018750419871615, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.22348104403359748, - "correct": 1 - }, - "two_order": { - "nll": 0.23081396178806582, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.029758515015450134, - "correct": 1 - }, - "two_order": { - "nll": 0.029057266507860405, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.2849594759462666, - "correct": 0 - }, - "two_order": { - "nll": 1.2179691475385028, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08362046298897138, - "correct": 1 - }, - "two_order": { - "nll": 0.08426090106475902, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.06030947699257955, - "correct": 1 - }, - "two_order": { - "nll": 0.4501004611309316, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.7402776473908522, - "correct": 1 - }, - "two_order": { - "nll": 0.6414708776135045, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.01475687196687665, - "correct": 1 - }, - "two_order": { - "nll": 0.027234500378350675, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.8137833116029765, - "correct": 0 - }, - "two_order": { - "nll": 0.7508116979925025, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.5578586042887936, - "correct": 0 - }, - "two_order": { - "nll": 1.8600090971567562, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 2.2099532797408172, - "correct": 0 - }, - "two_order": { - "nll": 2.1959700538560836, - "correct": 0 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.9179193566561494, - "correct": 0 - }, - "two_order": { - "nll": 1.2446638627286684, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.22541265160164786, - "correct": 1 - }, - "two_order": { - "nll": 0.18318840987747334, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.0218523659938747, - "correct": 1 - }, - "two_order": { - "nll": 0.03513051651277617, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.09559545419328726, - "correct": 1 - }, - "two_order": { - "nll": 0.09196700105398023, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.0630557923815305, - "correct": 1 - }, - "two_order": { - "nll": 0.07554573845155865, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.9936432896348318, - "correct": 0 - }, - "two_order": { - "nll": 0.8873544041416015, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.4336006978281376, - "correct": 1 - }, - "two_order": { - "nll": 0.44433582623488604, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.013943197179930197, - "correct": 1 - }, - "two_order": { - "nll": 0.011765148927772985, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0021851020429065234, - "correct": 1 - }, - "two_order": { - "nll": 0.0020874224800280505, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.005746727367274221, - "correct": 1 - }, - "two_order": { - "nll": 0.0061279959281322965, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.058587029414650875, - "correct": 1 - }, - "two_order": { - "nll": 0.07983733142237616, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.002254386500262537, - "correct": 1 - }, - "two_order": { - "nll": 0.0022197436715424707, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.016628744043527998, - "correct": 1 - }, - "two_order": { - "nll": 0.016662660665057585, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.1088488408659269, - "correct": 1 - }, - "two_order": { - "nll": 0.1698793756063374, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.01123490875817148, - "correct": 1 - }, - "two_order": { - "nll": 0.01603776050859953, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.12992810110229233, - "correct": 1 - }, - "two_order": { - "nll": 0.1208626577116002, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.05232202875810144, - "correct": 1 - }, - "two_order": { - "nll": 0.06024304519490543, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.024721582602869354, - "correct": 1 - }, - "two_order": { - "nll": 0.08170603988992431, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.6935087301168341, - "correct": 0 - }, - "two_order": { - "nll": 0.8669487156076334, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.0683565414930864, - "correct": 1 - }, - "two_order": { - "nll": 0.06701531279337267, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.33275424956421906, - "correct": 1 - }, - "two_order": { - "nll": 0.16208092001908112, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.1003602256656193, - "correct": 1 - }, - "two_order": { - "nll": 0.9626154417535695, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.10020655891674717, - "correct": 1 - }, - "two_order": { - "nll": 0.10989571256873003, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.6756734345843658, - "correct": 1 - }, - "two_order": { - "nll": 0.7221534900463844, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.17974463538565894, - "correct": 1 - }, - "two_order": { - "nll": 0.19337187431644245, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.08377023751297005, - "correct": 1 - }, - "two_order": { - "nll": 0.08253177122254815, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.12617239045059037, - "correct": 1 - }, - "two_order": { - "nll": 0.1295322337611133, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.11512677811328383, - "correct": 1 - }, - "two_order": { - "nll": 0.1210551299699099, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.251929081345373, - "correct": 1 - }, - "two_order": { - "nll": 0.23542259519924572, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7625300796023945, - "correct": 1 - }, - "two_order": { - "nll": 0.7788668802815122, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.011626354825328252, - "correct": 1 - }, - "two_order": { - "nll": 0.007383007073560757, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0137118622369538, - "correct": 1 - }, - "two_order": { - "nll": 0.01744432213446865, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.009124931375481345, - "correct": 1 - }, - "two_order": { - "nll": 0.02251812061780833, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.22149112171510002, - "correct": 1 - }, - "two_order": { - "nll": 0.21844063666934732, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.005746727367274221, - "correct": 1 - }, - "two_order": { - "nll": 0.0047888131640388566, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.012314703754835872, - "correct": 1 - }, - "two_order": { - "nll": 0.02192230443249389, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.2630089138799754, - "correct": 0 - }, - "two_order": { - "nll": 2.1825199432858406, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.016755766489059458, - "correct": 1 - }, - "two_order": { - "nll": 0.024094131027839846, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.8060551895596051, - "correct": 1 - }, - "two_order": { - "nll": 0.8928180216091015, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.06039324271581873, - "correct": 1 - }, - "two_order": { - "nll": 0.06254927585212194, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.08403866967962126, - "correct": 1 - }, - "two_order": { - "nll": 0.05514139286040489, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.9457165793021042, - "correct": 0 - }, - "two_order": { - "nll": 1.1577578838145093, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.11763917262786944, - "correct": 1 - }, - "two_order": { - "nll": 0.06628261406546225, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.03493413258416586, - "correct": 1 - }, - "two_order": { - "nll": 0.5644342377257953, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.388304510253787, - "correct": 0 - }, - "two_order": { - "nll": 1.4558314256247837, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.6931471805599453, - "correct": 1 - }, - "two_order": { - "nll": 0.7248539425912895, - "correct": 0 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.18167327031434768, - "correct": 1 - }, - "two_order": { - "nll": 0.13643695224587676, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.04712714527143037, - "correct": 1 - }, - "two_order": { - "nll": 0.04641823507141477, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.47407698418010663, - "correct": 1 - }, - "two_order": { - "nll": 0.3859437749078287, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.0777552140011723, - "correct": 1 - }, - "two_order": { - "nll": 0.07415342616710549, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.18876604925642768, - "correct": 1 - }, - "two_order": { - "nll": 0.1959825448070423, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.501929081345373, - "correct": 0 - }, - "two_order": { - "nll": 1.5019290813453727, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.9137959879979555, - "correct": 0 - }, - "two_order": { - "nll": 0.9135882323344903, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.03377564221712886, - "correct": 1 - }, - "two_order": { - "nll": 0.035829392090894814, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0067631237295626, - "correct": 1 - }, - "two_order": { - "nll": 0.010142029625348065, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.02065049412533139, - "correct": 1 - }, - "two_order": { - "nll": 0.02391138991880726, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.4165487032463721, - "correct": 1 - }, - "two_order": { - "nll": 0.41828086109974816, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.012509570276173327, - "correct": 1 - }, - "two_order": { - "nll": 0.010560130558004338, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.007372940020558588, - "correct": 1 - }, - "two_order": { - "nll": 0.00956211392704796, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.0906510943453984, - "correct": 0 - }, - "two_order": { - "nll": 2.287629944327559, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.10888993253392805, - "correct": 1 - }, - "two_order": { - "nll": 0.14592301201182994, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.709625368683793, - "correct": 0 - }, - "two_order": { - "nll": 1.6925888779799583, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.0772386485946437, - "correct": 1 - }, - "two_order": { - "nll": 0.07544558867480623, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.8663721559491016, - "correct": 0 - }, - "two_order": { - "nll": 0.3631840018497135, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.05973102776600069, - "correct": 1 - }, - "two_order": { - "nll": 0.5113568241551957, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 2.04694553740505, - "correct": 1 - }, - "two_order": { - "nll": 2.0445069113813306, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.19625208005058917, - "correct": 1 - }, - "two_order": { - "nll": 0.11999513252652898, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.3060357587611826, - "correct": 0 - }, - "two_order": { - "nll": 1.29925167457255, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.0742831253555125, - "correct": 1 - }, - "two_order": { - "nll": 0.07778283559504617, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.5364275039222968, - "correct": 1 - }, - "two_order": { - "nll": 0.5265510698037871, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 1.3593331378712068, - "correct": 0 - }, - "two_order": { - "nll": 1.1367127775474983, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03922555286405, - "correct": 1 - }, - "two_order": { - "nll": 0.058861000557081246, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.04044099885715609, - "correct": 1 - }, - "two_order": { - "nll": 0.03284519483058892, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.20567162015143123, - "correct": 1 - }, - "two_order": { - "nll": 0.21814558204553502, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.45093728159512503, - "correct": 1 - }, - "two_order": { - "nll": 0.379732032272534, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.32867347543143755, - "correct": 1 - }, - "two_order": { - "nll": 0.366027980242077, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.011115775485233476, - "correct": 1 - }, - "two_order": { - "nll": 0.008710733937578085, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.021123204783087764, - "correct": 1 - }, - "two_order": { - "nll": 0.019880932275047964, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.02806096048510901, - "correct": 1 - }, - "two_order": { - "nll": 0.03331146082094133, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 1.6725413182120614, - "correct": 0 - }, - "two_order": { - "nll": 1.5876377433065372, - "correct": 0 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.003692548129301451, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.11453903656316153, - "correct": 1 - }, - "two_order": { - "nll": 0.21713300606622596, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.13409467887955753, - "correct": 1 - }, - "two_order": { - "nll": 0.13200450891623475, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.0219794548477001, - "correct": 1 - }, - "two_order": { - "nll": 0.022195910803771996, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.4985666734271818, - "correct": 1 - }, - "two_order": { - "nll": 0.5112137693935074, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.048397618712257846, - "correct": 1 - }, - "two_order": { - "nll": 0.045548969679094886, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.09529282050823111, - "correct": 1 - }, - "two_order": { - "nll": 0.12431958114939884, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.3883516955984505, - "correct": 1 - }, - "two_order": { - "nll": 0.5602196006218952, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.13286455000088251, - "correct": 1 - }, - "two_order": { - "nll": 0.22871295764569366, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5601886939797442, - "correct": 1 - }, - "two_order": { - "nll": 0.27856541859234435, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.8677349199989377, - "correct": 1 - }, - "two_order": { - "nll": 0.7862052378293053, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.45093728159512503, - "correct": 1 - }, - "two_order": { - "nll": 0.5114864292446963, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.12303764004180046, - "correct": 1 - }, - "two_order": { - "nll": 0.11912769555724756, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.023974578520403952, - "correct": 1 - }, - "two_order": { - "nll": 0.03410257363611139, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.8981232641398401, - "correct": 0 - }, - "two_order": { - "nll": 0.9167030290906744, - "correct": 0 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.888986191530991, - "correct": 0 - }, - "two_order": { - "nll": 0.8899545510824985, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 1.972757165091386, - "correct": 0 - }, - "two_order": { - "nll": 1.7034488805162105, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.2811501362483148, - "correct": 1 - }, - "two_order": { - "nll": 0.2465515172722122, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.8992670153597763, - "correct": 0 - }, - "two_order": { - "nll": 0.9812028455331616, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.008279902708013648, - "correct": 1 - }, - "two_order": { - "nll": 0.015398296357624772, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.00563674397406815, - "correct": 1 - }, - "two_order": { - "nll": 0.004658725710417466, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.08459492561771735, - "correct": 1 - }, - "two_order": { - "nll": 0.047428185918928924, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.0628090799494529, - "correct": 1 - }, - "two_order": { - "nll": 0.053571713550707764, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0027186777265180476, - "correct": 1 - }, - "two_order": { - "nll": 0.0027617663864442843, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.007546534343733325, - "correct": 1 - }, - "two_order": { - "nll": 0.004286949767877182, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.3376034034838535, - "correct": 0 - }, - "two_order": { - "nll": 1.3857423147497103, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.12755677604859122, - "correct": 1 - }, - "two_order": { - "nll": 0.16901542391800325, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.3974337359418805, - "correct": 0 - }, - "two_order": { - "nll": 1.72690359381982, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.12535283400504701, - "correct": 1 - }, - "two_order": { - "nll": 0.10773297780617808, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.08866565152651158, - "correct": 1 - }, - "two_order": { - "nll": 0.32327964795693015, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.7638863383414923, - "correct": 1 - }, - "two_order": { - "nll": 1.1594010194458948, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.6043285153136739, - "correct": 1 - }, - "two_order": { - "nll": 0.27986941341797805, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.019937129401973473, - "correct": 1 - }, - "two_order": { - "nll": 0.028420067948958953, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.5500599489444167, - "correct": 1 - }, - "two_order": { - "nll": 0.6746974188626437, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.05164138927868149, - "correct": 1 - }, - "two_order": { - "nll": 0.05011320453337696, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.09797542368176994, - "correct": 1 - }, - "two_order": { - "nll": 0.09726351334210122, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.024726280462587773, - "correct": 1 - }, - "two_order": { - "nll": 0.02088559518790182, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.023974578520403952, - "correct": 1 - }, - "two_order": { - "nll": 0.019996536231008482, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.10540832970317754, - "correct": 1 - }, - "two_order": { - "nll": 0.12401222654719761, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.37719060559369166, - "correct": 1 - }, - "two_order": { - "nll": 0.4595189641775819, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.49813454751437525, - "correct": 1 - }, - "two_order": { - "nll": 0.44629111122069015, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.8111118862024624, - "correct": 1 - }, - "two_order": { - "nll": 0.8647029794270146, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.01692748629122821, - "correct": 1 - }, - "two_order": { - "nll": 0.017847491716824617, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.009938851331323587, - "correct": 1 - }, - "two_order": { - "nll": 0.022016942789862756, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.5707419917689904, - "correct": 1 - }, - "two_order": { - "nll": 0.7853162383320359, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.5646723247072885, - "correct": 1 - }, - "two_order": { - "nll": 0.5155932907498756, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003714141767506466, - "correct": 1 - }, - "two_order": { - "nll": 0.004096196540313023, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.008093860049042868, - "correct": 1 - }, - "two_order": { - "nll": 0.007450673343079281, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.12520184224761932, - "correct": 1 - }, - "two_order": { - "nll": 0.12119180088398127, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 2.9782624528562214, - "correct": 0 - }, - "two_order": { - "nll": 1.7017281466017096, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.11187865880715932, - "correct": 1 - }, - "two_order": { - "nll": 0.10217370482896676, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.01370525549903143, - "correct": 1 - }, - "two_order": { - "nll": 0.016904996168043255, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.026775467760443043, - "correct": 1 - }, - "two_order": { - "nll": 0.020074764363583807, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.6228759628047533, - "correct": 1 - }, - "two_order": { - "nll": 0.786784228273008, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.01636677528131479, - "correct": 1 - }, - "two_order": { - "nll": 0.01012739216467887, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 1.1004780245816668, - "correct": 0 - }, - "two_order": { - "nll": 0.46561460609495586, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.33943080482217397, - "correct": 1 - }, - "two_order": { - "nll": 0.3658632111873084, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.04712714527143037, - "correct": 1 - }, - "two_order": { - "nll": 0.047856981897304154, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.936453164616753, - "correct": 0 - }, - "two_order": { - "nll": 2.0144218367930353, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.06583535553267797, - "correct": 1 - }, - "two_order": { - "nll": 0.07732068700080218, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.010064043647627778, - "correct": 1 - }, - "two_order": { - "nll": 0.016633035435718455, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.27173031628021527, - "correct": 1 - }, - "two_order": { - "nll": 0.3202624993616726, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.02168530977594415, - "correct": 1 - }, - "two_order": { - "nll": 0.022490446377062392, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.501929081345373, - "correct": 0 - }, - "two_order": { - "nll": 1.3530237938956593, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.2034487591875314, - "correct": 0 - }, - "two_order": { - "nll": 1.359078480600808, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.016610422426847352, - "correct": 1 - }, - "two_order": { - "nll": 0.014013520065832527, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.026087708181620768, - "correct": 1 - }, - "two_order": { - "nll": 0.021309204962724277, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.00557041067347996, - "correct": 1 - }, - "two_order": { - "nll": 0.006471269208068428, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.3080323414874158, - "correct": 1 - }, - "two_order": { - "nll": 0.36178077629102445, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0024756851377303315, - "correct": 1 - }, - "two_order": { - "nll": 0.0025554108486462593, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 3.7580580652164497, - "correct": 0 - }, - "two_order": { - "nll": 4.2697318269228095, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.0274842422276982, - "correct": 0 - }, - "two_order": { - "nll": 1.1609150511685382, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.04797765294460553, - "correct": 1 - }, - "two_order": { - "nll": 0.04045576938398963, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.41527282596317333, - "correct": 1 - }, - "two_order": { - "nll": 0.45438910814761, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08670903145598241, - "correct": 1 - }, - "two_order": { - "nll": 0.07135205011317779, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.14939542418259533, - "correct": 1 - }, - "two_order": { - "nll": 0.42961091571025534, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.5931495561496511, - "correct": 1 - }, - "two_order": { - "nll": 0.3169669748688701, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.0927491723501417, - "correct": 0 - }, - "two_order": { - "nll": 1.0910844082666764, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5139512023903318, - "correct": 1 - }, - "two_order": { - "nll": 0.2547495528977414, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.036206787777225, - "correct": 0 - }, - "two_order": { - "nll": 2.1875535758540257, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.04170333686120593, - "correct": 1 - }, - "two_order": { - "nll": 0.03665856241778806, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.726548781548105, - "correct": 0 - }, - "two_order": { - "nll": 1.6442054639091102, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 2.6078030956397407, - "correct": 0 - }, - "two_order": { - "nll": 2.5785070221299518, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.02324546437242505, - "correct": 1 - }, - "two_order": { - "nll": 0.022215758026888015, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.029112748179552075, - "correct": 1 - }, - "two_order": { - "nll": 0.030903845799053472, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.38985199321957553, - "correct": 1 - }, - "two_order": { - "nll": 0.3796665521934768, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 2.0176750576056017, - "correct": 0 - }, - "two_order": { - "nll": 1.9055531002083894, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.6785941819171543, - "correct": 1 - }, - "two_order": { - "nll": 0.650008739499575, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.015309497179453528, - "correct": 1 - }, - "two_order": { - "nll": 0.009734264475758488, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.009681609538319353, - "correct": 1 - }, - "two_order": { - "nll": 0.009891379834551956, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.001699610342127682, - "correct": 1 - }, - "two_order": { - "nll": 0.007507398161322447, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.11512677811328383, - "correct": 1 - }, - "two_order": { - "nll": 0.10881922949921677, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.0031289163034652046, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.06836581743770373, - "correct": 1 - }, - "two_order": { - "nll": 0.3999345894424399, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.20868879817620054, - "correct": 1 - }, - "two_order": { - "nll": 0.2232492971405307, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.0638302749839662, - "correct": 1 - }, - "two_order": { - "nll": 0.06119717297997398, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.542719521688798, - "correct": 1 - }, - "two_order": { - "nll": 0.5355637930027418, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08020801114320515, - "correct": 1 - }, - "two_order": { - "nll": 0.08217725882779693, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.5537374841895398, - "correct": 1 - }, - "two_order": { - "nll": 0.2580685431096063, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.9515804765670444, - "correct": 1 - }, - "two_order": { - "nll": 1.297108821669702, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.9652599841744116, - "correct": 0 - }, - "two_order": { - "nll": 2.391673794701951, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.870067064342571, - "correct": 0 - }, - "two_order": { - "nll": 0.3677534988445831, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.8947988686080113, - "correct": 0 - }, - "two_order": { - "nll": 1.9074962155474846, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.011396447042008956, - "correct": 1 - }, - "two_order": { - "nll": 0.012149997084015268, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.4328554668787688, - "correct": 1 - }, - "two_order": { - "nll": 0.3643837867002506, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.10633723997560356, - "correct": 1 - }, - "two_order": { - "nll": 0.12232660385314509, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.058320480280248874, - "correct": 1 - }, - "two_order": { - "nll": 0.23548288803067674, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.07250595154641365, - "correct": 1 - }, - "two_order": { - "nll": 0.07781333936357654, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.42252705973354837, - "correct": 1 - }, - "two_order": { - "nll": 0.3935003839779431, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.179742032138927, - "correct": 0 - }, - "two_order": { - "nll": 1.0823698007707319, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7963601172995378, - "correct": 1 - }, - "two_order": { - "nll": 0.7987490302744598, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.006232222144049614, - "correct": 1 - }, - "two_order": { - "nll": 0.008310769655113624, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.053586321682286935, - "correct": 1 - }, - "two_order": { - "nll": 0.03971450786257627, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.005101835908736778, - "correct": 1 - }, - "two_order": { - "nll": 0.005106807063453614, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.49513275847479854, - "correct": 1 - }, - "two_order": { - "nll": 0.5598332412590689, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0024756851377303315, - "correct": 1 - }, - "two_order": { - "nll": 0.0022326892813017, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.029791545759062198, - "correct": 1 - }, - "two_order": { - "nll": 0.04944201044370984, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.4392489367905523, - "correct": 1 - }, - "two_order": { - "nll": 0.5221276052687899, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.5568674091841707, - "correct": 1 - }, - "two_order": { - "nll": 0.9148507969484643, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.1805047927172676, - "correct": 0 - }, - "two_order": { - "nll": 1.177074284066495, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.057572170432476207, - "correct": 1 - }, - "two_order": { - "nll": 0.058106572519003724, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.043633098380416346, - "correct": 1 - }, - "two_order": { - "nll": 0.07077023703911292, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.6643172956387638, - "correct": 1 - }, - "two_order": { - "nll": 0.48591311815659133, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.6931225976276034, - "correct": 1 - }, - "two_order": { - "nll": 0.9305126631688433, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.04113762101615342, - "correct": 1 - }, - "two_order": { - "nll": 0.3935628671665369, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9146110942982114, - "correct": 1 - }, - "two_order": { - "nll": 0.7316394455570759, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.013311021623458414, - "correct": 1 - }, - "two_order": { - "nll": 0.011686214777875072, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.7998352596946164, - "correct": 1 - }, - "two_order": { - "nll": 0.848315812033, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.010709655014929034, - "correct": 1 - }, - "two_order": { - "nll": 0.010232669304123834, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.012904115391931098, - "correct": 1 - }, - "two_order": { - "nll": 0.020844365846250858, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.22976731546645976, - "correct": 1 - }, - "two_order": { - "nll": 0.19680469979965134, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 2.4200785100952467, - "correct": 0 - }, - "two_order": { - "nll": 2.40708675518106, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.5490548622706681, - "correct": 1 - }, - "two_order": { - "nll": 0.6185079390251137, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.29984236548969156, - "correct": 1 - }, - "two_order": { - "nll": 0.3270907219779449, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.027001854603911128, - "correct": 1 - }, - "two_order": { - "nll": 0.02723260417049986, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.012798103711043401, - "correct": 1 - }, - "two_order": { - "nll": 0.010959286073725337, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.007709734939182858, - "correct": 1 - }, - "two_order": { - "nll": 0.006105433725127378, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.10851633836895944, - "correct": 1 - }, - "two_order": { - "nll": 0.12842262874129776, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003953210456515145, - "correct": 1 - }, - "two_order": { - "nll": 0.0035165641988045945, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.009176651152390147, - "correct": 1 - }, - "two_order": { - "nll": 0.020682140455955563, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.714784649898476, - "correct": 1 - }, - "two_order": { - "nll": 0.7333751788112584, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.056086971217846344, - "correct": 1 - }, - "two_order": { - "nll": 0.07780707366710661, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.36440912404700837, - "correct": 1 - }, - "two_order": { - "nll": 0.29150329505200273, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.09183061130498635, - "correct": 1 - }, - "two_order": { - "nll": 0.0888075782791414, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.02387846560950667, - "correct": 1 - }, - "two_order": { - "nll": 0.2163013033117818, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 2.8091744343310916, - "correct": 0 - }, - "two_order": { - "nll": 2.48290439735785, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.014541879780317887, - "correct": 1 - }, - "two_order": { - "nll": 0.03867368398904081, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.7130804188105171, - "correct": 1 - }, - "two_order": { - "nll": 0.7096499719035849, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.2085370929180823, - "correct": 0 - }, - "two_order": { - "nll": 1.9008433861132727, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.03922555286405, - "correct": 1 - }, - "two_order": { - "nll": 0.04620712389295999, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.8949734972289527, - "correct": 0 - }, - "two_order": { - "nll": 0.6738219135194214, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.10633723997560356, - "correct": 1 - }, - "two_order": { - "nll": 0.1399201315446505, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.058320480280248874, - "correct": 1 - }, - "two_order": { - "nll": 0.10368895504848531, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.42781354066031235, - "correct": 1 - }, - "two_order": { - "nll": 0.47695412577169066, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.40084106610964954, - "correct": 1 - }, - "two_order": { - "nll": 0.38290885115159834, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.3484445810050557, - "correct": 1 - }, - "two_order": { - "nll": 0.29188005601421163, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.4078167367151417, - "correct": 0 - }, - "two_order": { - "nll": 1.5001890346550013, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.018981405600261885, - "correct": 1 - }, - "two_order": { - "nll": 0.011794909763898104, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.019082145449108186, - "correct": 1 - }, - "two_order": { - "nll": 0.013902202173181895, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.0226601090024568, - "correct": 1 - }, - "two_order": { - "nll": 0.01599148762309789, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.017155391561683882, - "correct": 1 - }, - "two_order": { - "nll": 0.02516896044141924, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0023258652514146837, - "correct": 1 - }, - "two_order": { - "nll": 0.0023258652514146837, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.017508618871813857, - "correct": 1 - }, - "two_order": { - "nll": 0.012357015086827537, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.5143823943147482, - "correct": 1 - }, - "two_order": { - "nll": 0.4777986215817697, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 1.648440757626167, - "correct": 0 - }, - "two_order": { - "nll": 2.084433808812288, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.11868202455660723, - "correct": 1 - }, - "two_order": { - "nll": 0.13300351459020274, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.062250293849906005, - "correct": 1 - }, - "two_order": { - "nll": 0.0524063529678237, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.02010700002864646, - "correct": 1 - }, - "two_order": { - "nll": 0.026866624870869965, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.21290139255065663, - "correct": 1 - }, - "two_order": { - "nll": 0.5372180110946646, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.023700537030122388, - "correct": 1 - }, - "two_order": { - "nll": 0.029617243277414385, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.18644511274650602, - "correct": 1 - }, - "two_order": { - "nll": 0.11085719059543672, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9383852115014748, - "correct": 0 - }, - "two_order": { - "nll": 0.6108626105209377, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.007606094404333938, - "correct": 1 - }, - "two_order": { - "nll": 0.008386507983605993, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.1758389466112202, - "correct": 0 - }, - "two_order": { - "nll": 0.9326068537484214, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.14267505760560167, - "correct": 1 - }, - "two_order": { - "nll": 0.1561095942063894, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.024726280462587773, - "correct": 1 - }, - "two_order": { - "nll": 0.02769893658199565, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.10814781512537903, - "correct": 1 - }, - "two_order": { - "nll": 0.11288419452730226, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.8338286948065795, - "correct": 1 - }, - "two_order": { - "nll": 0.8688072341350833, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.1157534602702044, - "correct": 0 - }, - "two_order": { - "nll": 1.0737589168036015, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.4390437872861366, - "correct": 1 - }, - "two_order": { - "nll": 0.5382087126137921, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.00994647983237274, - "correct": 1 - }, - "two_order": { - "nll": 0.014026236390252049, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.009065676708340567, - "correct": 1 - }, - "two_order": { - "nll": 0.015971670791555866, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.021878827418688036, - "correct": 1 - }, - "two_order": { - "nll": 0.025395187575626216, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.09458682518587855, - "correct": 1 - }, - "two_order": { - "nll": 0.07842822317405941, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.0029912743082821974, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.013843432918108429, - "correct": 1 - }, - "two_order": { - "nll": 0.020271143002872995, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.6049920328403577, - "correct": 0 - }, - "two_order": { - "nll": 2.6523228604430176, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.02475996692142672, - "correct": 1 - }, - "two_order": { - "nll": 0.023702872756370917, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.35625562853348514, - "correct": 1 - }, - "two_order": { - "nll": 0.32624871043601655, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.1033367481555656, - "correct": 1 - }, - "two_order": { - "nll": 0.12357636073971084, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.020100442533956334, - "correct": 1 - }, - "two_order": { - "nll": 0.029786487753265575, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.04953721906085087, - "correct": 1 - }, - "two_order": { - "nll": 0.05347917164929199, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.049629326094553915, - "correct": 1 - }, - "two_order": { - "nll": 0.15955611115203602, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.4008481464288898, - "correct": 1 - }, - "two_order": { - "nll": 0.5582658116314273, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.1185720732243736, - "correct": 0 - }, - "two_order": { - "nll": 1.8961934204034514, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.5897417378836574, - "correct": 1 - }, - "two_order": { - "nll": 0.562494761437238, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.7390244093095892, - "correct": 1 - }, - "two_order": { - "nll": 0.755456169349554, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.02797290114133035, - "correct": 1 - }, - "two_order": { - "nll": 0.033583399314846316, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.04433419546256515, - "correct": 1 - }, - "two_order": { - "nll": 0.05130288605032962, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.11989995740163877, - "correct": 1 - }, - "two_order": { - "nll": 0.10587873727715114, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.1782292911492589, - "correct": 1 - }, - "two_order": { - "nll": 0.17213201571201717, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.6776442459057149, - "correct": 1 - }, - "two_order": { - "nll": 0.6776442459057149, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.1290325089493083, - "correct": 0 - }, - "two_order": { - "nll": 1.1165040460452544, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.01634726770472382, - "correct": 1 - }, - "two_order": { - "nll": 0.010121237408220313, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.028464785948947732, - "correct": 1 - }, - "two_order": { - "nll": 0.02550220473115058, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.07444129410906175, - "correct": 1 - }, - "two_order": { - "nll": 0.06739260014678862, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.03237912947055251, - "correct": 1 - }, - "two_order": { - "nll": 0.04133327850343491, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.006715348489117944, - "correct": 1 - }, - "two_order": { - "nll": 0.008546922830063827, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.1385551388238828, - "correct": 1 - }, - "two_order": { - "nll": 0.12291514152420673, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.283757919449582, - "correct": 0 - }, - "two_order": { - "nll": 1.1880164950117587, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.01790017728100203, - "correct": 1 - }, - "two_order": { - "nll": 0.015470159297741908, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.3395705678851873, - "correct": 1 - }, - "two_order": { - "nll": 0.3264796660858348, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.03767522699125972, - "correct": 1 - }, - "two_order": { - "nll": 0.03434957275365464, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.019442989217052467, - "correct": 1 - }, - "two_order": { - "nll": 0.10593630151309048, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.45144406489109, - "correct": 1 - }, - "two_order": { - "nll": 0.9592725513290883, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.05021364778922124, - "correct": 1 - }, - "two_order": { - "nll": 0.05928460200388559, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5738070605538901, - "correct": 1 - }, - "two_order": { - "nll": 0.27288456956101903, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.1313248909777884, - "correct": 0 - }, - "two_order": { - "nll": 1.1256271662319803, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.04044585599939583, - "correct": 1 - }, - "two_order": { - "nll": 0.04238813583419846, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.07590208190347222, - "correct": 1 - }, - "two_order": { - "nll": 0.05785353709678846, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.9936432896348318, - "correct": 0 - }, - "two_order": { - "nll": 0.7388492436866466, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.12692801104297263, - "correct": 1 - }, - "two_order": { - "nll": 0.10262044089564606, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 3.0741813783875895, - "correct": 0 - }, - "two_order": { - "nll": 2.888090212661871, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.14454663180212068, - "correct": 1 - }, - "two_order": { - "nll": 0.13336892375593548, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.8615548622706681, - "correct": 0 - }, - "two_order": { - "nll": 0.8082267048781631, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7413476801563812, - "correct": 1 - }, - "two_order": { - "nll": 0.7446540914834373, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.0036358301425141247, - "correct": 1 - }, - "two_order": { - "nll": 0.011363374428788531, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.011023036761974836, - "correct": 1 - }, - "two_order": { - "nll": 0.011628033542798958, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.015797051949997195, - "correct": 1 - }, - "two_order": { - "nll": 0.014041589645485604, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.028183299022021543, - "correct": 1 - }, - "two_order": { - "nll": 0.02083914386759074, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.008350547451545454, - "correct": 1 - }, - "two_order": { - "nll": 0.010051869426984677, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.059822193576600576, - "correct": 1 - }, - "two_order": { - "nll": 0.0788174690198234, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.1073584839642534, - "correct": 0 - }, - "two_order": { - "nll": 1.099543053200092, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.030784754919162004, - "correct": 1 - }, - "two_order": { - "nll": 0.028977358478290825, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 2.2641126024993437, - "correct": 0 - }, - "two_order": { - "nll": 2.315456761517189, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 3.9666817919211153, - "correct": 0 - }, - "two_order": { - "nll": 4.133289965702172, - "correct": 0 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.05018720208273875, - "correct": 1 - }, - "two_order": { - "nll": 0.12371388089628685, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.42062495394112076, - "correct": 1 - }, - "two_order": { - "nll": 0.2696129286438112, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.10145209205855144, - "correct": 1 - }, - "two_order": { - "nll": 0.05629737033283049, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.3499982821463602, - "correct": 1 - }, - "two_order": { - "nll": 0.48228929372426943, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.934343007736607, - "correct": 0 - }, - "two_order": { - "nll": 2.019950949555297, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.367242032138927, - "correct": 1 - }, - "two_order": { - "nll": 0.3531994423731511, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.0546685302138049, - "correct": 1 - }, - "two_order": { - "nll": 0.06893969822086263, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 1.179742032138927, - "correct": 0 - }, - "two_order": { - "nll": 1.3581889141358934, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.06785584399677826, - "correct": 1 - }, - "two_order": { - "nll": 0.14066019431641386, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.05490536922604806, - "correct": 1 - }, - "two_order": { - "nll": 0.05298214854478505, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 2.515823525844012, - "correct": 0 - }, - "two_order": { - "nll": 2.697471867169931, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.38687100611489994, - "correct": 1 - }, - "two_order": { - "nll": 0.41294012229819393, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.5800053530679871, - "correct": 1 - }, - "two_order": { - "nll": 0.5874527984597637, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.03374831880976058, - "correct": 1 - }, - "two_order": { - "nll": 0.030283605580750626, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.004327601621399945, - "correct": 1 - }, - "two_order": { - "nll": 0.010063633462134331, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.07728472815839725, - "correct": 1 - }, - "two_order": { - "nll": 0.07426573512343704, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.06862537882458733, - "correct": 1 - }, - "two_order": { - "nll": 0.08099962425731909, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.05009167832493603, - "correct": 1 - }, - "two_order": { - "nll": 0.07200675111863888, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.14694623361737735, - "correct": 1 - }, - "two_order": { - "nll": 0.11823618681696904, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.9040417801122773, - "correct": 0 - }, - "two_order": { - "nll": 0.6958149593318943, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.0352824330950873, - "correct": 1 - }, - "two_order": { - "nll": 0.03902629556247605, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.0890045132806183, - "correct": 0 - }, - "two_order": { - "nll": 1.1186022964899653, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.10740267510307709, - "correct": 1 - }, - "two_order": { - "nll": 0.12493311756298874, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.015103119091385607, - "correct": 1 - }, - "two_order": { - "nll": 0.013575711407450852, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.6390224632780469, - "correct": 1 - }, - "two_order": { - "nll": 0.6505474835165181, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 5.288445714499048, - "correct": 0 - }, - "two_order": { - "nll": 5.567036727904605, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.03468241747162235, - "correct": 1 - }, - "two_order": { - "nll": 0.5608219379957712, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.0320615679813738, - "correct": 1 - }, - "two_order": { - "nll": 1.146658057725097, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.05657594758178158, - "correct": 1 - }, - "two_order": { - "nll": 0.05043633907727328, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.34769599436448745, - "correct": 1 - }, - "two_order": { - "nll": 0.4973140598286106, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.10020655891674717, - "correct": 1 - }, - "two_order": { - "nll": 0.08287329204043259, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.15121395396940143, - "correct": 1 - }, - "two_order": { - "nll": 0.1557089042830205, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.05391848474544452, - "correct": 1 - }, - "two_order": { - "nll": 0.06001748242851365, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.23646800062085752, - "correct": 1 - }, - "two_order": { - "nll": 0.24315015599470585, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.9936432896348318, - "correct": 0 - }, - "two_order": { - "nll": 1.092176585001246, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.5170161081396428, - "correct": 1 - }, - "two_order": { - "nll": 0.5318799322332685, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.004032879380699412, - "correct": 1 - }, - "two_order": { - "nll": 0.008884804305699852, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.017987879489370473, - "correct": 1 - }, - "two_order": { - "nll": 0.018588663527466118, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.003714141767506466, - "correct": 1 - }, - "two_order": { - "nll": 0.0029157247345481985, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.030453252948697655, - "correct": 1 - }, - "two_order": { - "nll": 0.03045009225978931, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0024756851377303315, - "correct": 1 - }, - "two_order": { - "nll": 0.002400772388822885, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 5.005615082349949, - "correct": 0 - }, - "two_order": { - "nll": 4.454605001188563, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.6106484484823188, - "correct": 1 - }, - "two_order": { - "nll": 0.6258867334973826, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.060986818780914755, - "correct": 1 - }, - "two_order": { - "nll": 0.0685265287748769, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.5365307213042072, - "correct": 0 - }, - "two_order": { - "nll": 1.6799668286187566, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.10283628675510588, - "correct": 1 - }, - "two_order": { - "nll": 0.10015126144837756, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.02929207566709578, - "correct": 1 - }, - "two_order": { - "nll": 0.02802685823585733, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.3902169733240162, - "correct": 0 - }, - "two_order": { - "nll": 0.7819226846034211, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.031440177405090614, - "correct": 1 - }, - "two_order": { - "nll": 0.04985514538544759, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.10102056475282951, - "correct": 1 - }, - "two_order": { - "nll": 0.07821856630070322, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.4868459222186134, - "correct": 0 - }, - "two_order": { - "nll": 1.5221669859064848, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.058320480280248874, - "correct": 1 - }, - "two_order": { - "nll": 0.058320480280248874, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.5126449371013564, - "correct": 1 - }, - "two_order": { - "nll": 0.4898667223557108, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.012904115391931098, - "correct": 1 - }, - "two_order": { - "nll": 0.018424027640160028, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03262723298234382, - "correct": 1 - }, - "two_order": { - "nll": 0.03420102053845451, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.11869100754495031, - "correct": 1 - }, - "two_order": { - "nll": 0.10112323059847793, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.08695265726523116, - "correct": 1 - }, - "two_order": { - "nll": 0.08034899284481932, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.47407698418010663, - "correct": 1 - }, - "two_order": { - "nll": 0.4511314771639264, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.1356853436247023, - "correct": 0 - }, - "two_order": { - "nll": 1.2291608834901964, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.022574196981384246, - "correct": 1 - }, - "two_order": { - "nll": 0.016013866597377732, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.01162163226227119, - "correct": 1 - }, - "two_order": { - "nll": 0.0074034346384027686, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.010117129067101768, - "correct": 1 - }, - "two_order": { - "nll": 0.006357970019474579, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.10895069343827918, - "correct": 1 - }, - "two_order": { - "nll": 0.10168993851044474, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.0031289163034652046, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.029584054063654407, - "correct": 1 - }, - "two_order": { - "nll": 0.01870161006584919, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.5411654934588789, - "correct": 1 - }, - "two_order": { - "nll": 0.5207358917267862, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.025925896172401902, - "correct": 1 - }, - "two_order": { - "nll": 0.03943971527067909, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.585830940723973, - "correct": 1 - }, - "two_order": { - "nll": 0.625147537058694, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.030281427364897005, - "correct": 1 - }, - "two_order": { - "nll": 0.03379313248366333, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.04961420716846196, - "correct": 1 - }, - "two_order": { - "nll": 0.14438408669269506, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 2.200499681659196, - "correct": 0 - }, - "two_order": { - "nll": 0.8919254544855979, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 7.4726619471811, - "correct": 0 - }, - "two_order": { - "nll": 7.094663356736885, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5065779191964285, - "correct": 1 - }, - "two_order": { - "nll": 0.9854476218447148, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9939872903897226, - "correct": 0 - }, - "two_order": { - "nll": 0.899235036804645, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.2811501362483148, - "correct": 1 - }, - "two_order": { - "nll": 0.30550029196994377, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.035102546275021525, - "correct": 1 - }, - "two_order": { - "nll": 0.03446226985406008, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.061967589003198605, - "correct": 1 - }, - "two_order": { - "nll": 0.04993292406789605, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03262723298234382, - "correct": 1 - }, - "two_order": { - "nll": 0.027925346519952394, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.028869836225112227, - "correct": 1 - }, - "two_order": { - "nll": 0.0284198901850144, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.0759564345682912, - "correct": 1 - }, - "two_order": { - "nll": 0.07993386700390517, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.32176256954851534, - "correct": 1 - }, - "two_order": { - "nll": 0.34424377543326257, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.1098459347187357, - "correct": 0 - }, - "two_order": { - "nll": 1.2034983751628754, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 1.1860652852978422, - "correct": 0 - }, - "two_order": { - "nll": 0.5243423872184779, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.002181223723250433, - "correct": 1 - }, - "two_order": { - "nll": 0.005212817768302212, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.0068253385071424385, - "correct": 1 - }, - "two_order": { - "nll": 0.007268992839528124, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.03463185509145332, - "correct": 1 - }, - "two_order": { - "nll": 0.04640259560453939, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.005073178869938949, - "correct": 1 - }, - "two_order": { - "nll": 0.005409896410160542, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 5.405959232503582, - "correct": 0 - }, - "two_order": { - "nll": 5.482932515705919, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.894373358388868, - "correct": 0 - }, - "two_order": { - "nll": 1.8093069682807297, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 3.616037976382156, - "correct": 0 - }, - "two_order": { - "nll": 3.396490775776311, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.8992670153597763, - "correct": 0 - }, - "two_order": { - "nll": 1.0058348499039764, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.09835569388760058, - "correct": 1 - }, - "two_order": { - "nll": 0.08806928634538229, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.29210316504625755, - "correct": 1 - }, - "two_order": { - "nll": 0.16597219710879796, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.7589560928584228, - "correct": 1 - }, - "two_order": { - "nll": 0.6142074021861981, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.112357344587165, - "correct": 1 - }, - "two_order": { - "nll": 0.1884919965799346, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5864955770231935, - "correct": 1 - }, - "two_order": { - "nll": 1.093170074551128, - "correct": 0 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9701369836623976, - "correct": 1 - }, - "two_order": { - "nll": 1.0115430763712483, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.02884816883748237, - "correct": 1 - }, - "two_order": { - "nll": 0.026408404696559105, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.7052530894817965, - "correct": 0 - }, - "two_order": { - "nll": 1.5951101566285117, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.11620327974081737, - "correct": 1 - }, - "two_order": { - "nll": 0.12535266618315716, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.11620327974081737, - "correct": 1 - }, - "two_order": { - "nll": 0.09859674109773377, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.2643821320961903, - "correct": 1 - }, - "two_order": { - "nll": 0.25645728973281334, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.045704352100228184, - "correct": 1 - }, - "two_order": { - "nll": 0.05204749479288898, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.258973472514827, - "correct": 1 - }, - "two_order": { - "nll": 0.258973472514827, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7410562721015285, - "correct": 1 - }, - "two_order": { - "nll": 0.7311387222934478, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.009932795311007691, - "correct": 1 - }, - "two_order": { - "nll": 0.007422520584707116, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.012043778085042395, - "correct": 1 - }, - "two_order": { - "nll": 0.014386594897107266, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.09490485785276268, - "correct": 1 - }, - "two_order": { - "nll": 0.07013209555316584, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.2222705792761846, - "correct": 1 - }, - "two_order": { - "nll": 0.2302623589255876, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003714141767506466, - "correct": 1 - }, - "two_order": { - "nll": 0.003259395985427724, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.3288907709406345, - "correct": 1 - }, - "two_order": { - "nll": 0.21327015449757378, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.4856344032644102, - "correct": 1 - }, - "two_order": { - "nll": 0.5519245161164276, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.4319297921325762, - "correct": 1 - }, - "two_order": { - "nll": 0.5269974323271427, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 2.004630479455186, - "correct": 0 - }, - "two_order": { - "nll": 1.9742892021669276, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.02868822933979948, - "correct": 1 - }, - "two_order": { - "nll": 0.025874106426159123, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.03459576669508942, - "correct": 1 - }, - "two_order": { - "nll": 0.27087352330630626, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 2.264158030845119, - "correct": 0 - }, - "two_order": { - "nll": 0.8134631009908109, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.0858972069274276, - "correct": 1 - }, - "two_order": { - "nll": 0.09600524205828288, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.7166893729188244, - "correct": 0 - }, - "two_order": { - "nll": 0.3391767448608733, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.597171063911047, - "correct": 0 - }, - "two_order": { - "nll": 1.4121096383293446, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.027123830872991448, - "correct": 1 - }, - "two_order": { - "nll": 0.023827450671549705, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.08385875969189407, - "correct": 1 - }, - "two_order": { - "nll": 0.07357445339573576, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.006715348489117944, - "correct": 1 - }, - "two_order": { - "nll": 0.00794068813678362, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.04170333686120593, - "correct": 1 - }, - "two_order": { - "nll": 0.03617670838852364, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.6575638201269349, - "correct": 1 - }, - "two_order": { - "nll": 0.5425978861646539, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.11792474698156105, - "correct": 1 - }, - "two_order": { - "nll": 0.13630329063549942, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.1368710061149, - "correct": 0 - }, - "two_order": { - "nll": 1.0626950643668274, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.55456582378435, - "correct": 0 - }, - "two_order": { - "nll": 1.5998928371268295, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.010049429076506324, - "correct": 1 - }, - "two_order": { - "nll": 0.008642508002254365, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.00542118549407955, - "correct": 1 - }, - "two_order": { - "nll": 0.011177420543601808, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.008221614944632341, - "correct": 1 - }, - "two_order": { - "nll": 0.0055769531572819955, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 1.6716148124143635, - "correct": 0 - }, - "two_order": { - "nll": 1.8317965543031576, - "correct": 0 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003278433164655044, - "correct": 1 - }, - "two_order": { - "nll": 0.0038095305515816816, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.013850894981151095, - "correct": 1 - }, - "two_order": { - "nll": 0.01465057306477643, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.7163753298761524, - "correct": 0 - }, - "two_order": { - "nll": 1.5286260723601621, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.29446647793963887, - "correct": 1 - }, - "two_order": { - "nll": 0.3258874807953914, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08321519833328549, - "correct": 1 - }, - "two_order": { - "nll": 0.11671137366525368, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.12732753093959376, - "correct": 1 - }, - "two_order": { - "nll": 0.1197822916973767, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.8936444162992352, - "correct": 0 - }, - "two_order": { - "nll": 0.8162769288700414, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.020527740666052043, - "correct": 1 - }, - "two_order": { - "nll": 0.02553916616375947, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.1193859685260377, - "correct": 1 - }, - "two_order": { - "nll": 0.7500828693921648, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.7426880459891129, - "correct": 1 - }, - "two_order": { - "nll": 0.8846277737800825, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 1.3826498628379482, - "correct": 0 - }, - "two_order": { - "nll": 1.3236606636680786, - "correct": 0 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.39120167537809825, - "correct": 1 - }, - "two_order": { - "nll": 0.4159760453384539, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.03262723298234382, - "correct": 1 - }, - "two_order": { - "nll": 0.027925346519952394, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.06993619964497334, - "correct": 1 - }, - "two_order": { - "nll": 0.08209965116206885, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 1.7068637975823235, - "correct": 0 - }, - "two_order": { - "nll": 1.876101383879326, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.27517983694409986, - "correct": 1 - }, - "two_order": { - "nll": 0.26342030498614744, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.47407698418010663, - "correct": 1 - }, - "two_order": { - "nll": 0.43479786493046846, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.9195976229534382, - "correct": 1 - }, - "two_order": { - "nll": 1.059988127570009, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.018059560901135564, - "correct": 1 - }, - "two_order": { - "nll": 0.011261603430058189, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.025347851100122432, - "correct": 1 - }, - "two_order": { - "nll": 0.021338168408906694, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.10169232244838122, - "correct": 1 - }, - "two_order": { - "nll": 0.05738080300891988, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.15594032933991517, - "correct": 1 - }, - "two_order": { - "nll": 0.16295958666611504, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0025541715420446696, - "correct": 1 - }, - "two_order": { - "nll": 0.0028659004041343365, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 3.016817321575584, - "correct": 0 - }, - "two_order": { - "nll": 3.690313240118683, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.489625203354606, - "correct": 0 - }, - "two_order": { - "nll": 2.3722560685124305, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.22648999539462278, - "correct": 1 - }, - "two_order": { - "nll": 0.24668440672381586, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.019346619636800556, - "correct": 1 - }, - "two_order": { - "nll": 0.02333852171438058, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.21951917341053775, - "correct": 1 - }, - "two_order": { - "nll": 0.12799580719296932, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.23088234837527277, - "correct": 1 - }, - "two_order": { - "nll": 0.1979711571613269, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.05103326733531594, - "correct": 1 - }, - "two_order": { - "nll": 0.0528544918078841, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.3567635623461305, - "correct": 1 - }, - "two_order": { - "nll": 0.2429005971386806, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.224057001745952, - "correct": 1 - }, - "two_order": { - "nll": 0.23534559360854188, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.036892284148855904, - "correct": 1 - }, - "two_order": { - "nll": 0.03286213804108401, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.16572436944506289, - "correct": 1 - }, - "two_order": { - "nll": 0.19841111398654515, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.05164138927868149, - "correct": 1 - }, - "two_order": { - "nll": 0.05086623360133378, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 1.1157534602702044, - "correct": 0 - }, - "two_order": { - "nll": 0.9404167518122099, - "correct": 0 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 2.4746940655886234, - "correct": 0 - }, - "two_order": { - "nll": 2.5174488559636763, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.19063755653267556, - "correct": 1 - }, - "two_order": { - "nll": 0.18852724955421196, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.6473704112766769, - "correct": 1 - }, - "two_order": { - "nll": 0.6181409990129576, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.8983805958422963, - "correct": 0 - }, - "two_order": { - "nll": 0.9060585879248221, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.0028393279908568783, - "correct": 1 - }, - "two_order": { - "nll": 0.0066945368694208696, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.010614775605981513, - "correct": 1 - }, - "two_order": { - "nll": 0.010446741310680095, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 5.118827100586655, - "correct": 0 - }, - "two_order": { - "nll": 4.427865370252084, - "correct": 0 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.215359715451144, - "correct": 1 - }, - "two_order": { - "nll": 0.22606150123998225, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003278433164655044, - "correct": 1 - }, - "two_order": { - "nll": 0.003179265924662805, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.026441631237356582, - "correct": 1 - }, - "two_order": { - "nll": 0.016890827767555978, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.7499351187581018, - "correct": 1 - }, - "two_order": { - "nll": 0.7865961664324344, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.603982098643271, - "correct": 1 - }, - "two_order": { - "nll": 0.6447910157728076, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.013252984700933767, - "correct": 1 - }, - "two_order": { - "nll": 0.014992433568678087, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.08095753177773166, - "correct": 1 - }, - "two_order": { - "nll": 0.05008302517240303, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.3472470242500833, - "correct": 1 - }, - "two_order": { - "nll": 0.5680367368145695, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.2197166770896776, - "correct": 1 - }, - "two_order": { - "nll": 0.1189552662014736, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.029241988914203158, - "correct": 1 - }, - "two_order": { - "nll": 0.34363353912411093, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 3.1599409362948747, - "correct": 0 - }, - "two_order": { - "nll": 2.413279151375555, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.054882179158078234, - "correct": 1 - }, - "two_order": { - "nll": 0.05028548664508826, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.29218992336562866, - "correct": 1 - }, - "two_order": { - "nll": 0.21557788412552953, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.05657594758178158, - "correct": 1 - }, - "two_order": { - "nll": 0.051128128326165125, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03922555286405, - "correct": 1 - }, - "two_order": { - "nll": 0.03120358855434411, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 1.6041823121249388, - "correct": 0 - }, - "two_order": { - "nll": 1.640281021666949, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.07289376167527796, - "correct": 1 - }, - "two_order": { - "nll": 0.07801662357933299, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.830920142458542, - "correct": 0 - }, - "two_order": { - "nll": 1.7776290698023107, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.5731612742499131, - "correct": 1 - }, - "two_order": { - "nll": 0.6081203490928618, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.005596860029564218, - "correct": 1 - }, - "two_order": { - "nll": 0.014255789514070222, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.016264579129894582, - "correct": 1 - }, - "two_order": { - "nll": 0.014420307415330605, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.01927836957476527, - "correct": 1 - }, - "two_order": { - "nll": 0.020036507404346985, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.1873295514888485, - "correct": 1 - }, - "two_order": { - "nll": 0.15372063172369244, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0027186777265180476, - "correct": 1 - }, - "two_order": { - "nll": 0.00267690944912994, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.007146907153260153, - "correct": 1 - }, - "two_order": { - "nll": 0.007146907153260153, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.9652427732053208, - "correct": 0 - }, - "two_order": { - "nll": 1.0977541925667413, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.1463416629820826, - "correct": 0 - }, - "two_order": { - "nll": 1.1446565945938019, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.039300683739222624, - "correct": 1 - }, - "two_order": { - "nll": 0.043742145657084494, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.017729439339609295, - "correct": 1 - }, - "two_order": { - "nll": 0.017860882697377647, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.4081874448898175, - "correct": 0 - }, - "two_order": { - "nll": 1.7314774153666244, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.01066372059811311, - "correct": 1 - }, - "two_order": { - "nll": 0.10395117944004603, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.028031666900932544, - "correct": 1 - }, - "two_order": { - "nll": 0.022445844129091952, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.8066775839033432, - "correct": 1 - }, - "two_order": { - "nll": 0.7696954248199842, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.022538275020599047, - "correct": 1 - }, - "two_order": { - "nll": 0.021226350268675715, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.34224948759121215, - "correct": 1 - }, - "two_order": { - "nll": 0.28269564339869335, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.2383445081104411, - "correct": 1 - }, - "two_order": { - "nll": 0.19157051346380405, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.21918415463449092, - "correct": 1 - }, - "two_order": { - "nll": 0.24241082929058777, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.09032745452969719, - "correct": 1 - }, - "two_order": { - "nll": 0.06727277549355648, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.1283258040660719, - "correct": 1 - }, - "two_order": { - "nll": 0.11529638704960389, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.5490548622706681, - "correct": 1 - }, - "two_order": { - "nll": 0.6855335243665968, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7296720605492807, - "correct": 1 - }, - "two_order": { - "nll": 0.7913732482136114, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.021344381739088293, - "correct": 1 - }, - "two_order": { - "nll": 0.02197492218694703, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0033823260027422077, - "correct": 1 - }, - "two_order": { - "nll": 0.003548220122411682, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.12520172563460252, - "correct": 1 - }, - "two_order": { - "nll": 0.37585601518447176, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.0474104089409759, - "correct": 1 - }, - "two_order": { - "nll": 0.05728191489038566, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0026351429162579636, - "correct": 1 - }, - "two_order": { - "nll": 0.0026351429162579636, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.024222587817850274, - "correct": 1 - }, - "two_order": { - "nll": 0.02186982164798901, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.9490159780145521, - "correct": 0 - }, - "two_order": { - "nll": 0.8280041917072275, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.5930390847564373, - "correct": 0 - }, - "two_order": { - "nll": 1.8891064485744786, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.09096023884419899, - "correct": 1 - }, - "two_order": { - "nll": 0.08589516758770857, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.018609946901949177, - "correct": 1 - }, - "two_order": { - "nll": 0.016470190366733046, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.8023021344730289, - "correct": 1 - }, - "two_order": { - "nll": 1.296973050744976, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.268132830064987, - "correct": 1 - }, - "two_order": { - "nll": 0.2768716600625241, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.027505331415183104, - "correct": 1 - }, - "two_order": { - "nll": 0.17692203395410624, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.6676881777186544, - "correct": 1 - }, - "two_order": { - "nll": 0.6615013114626438, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.02324546437242505, - "correct": 1 - }, - "two_order": { - "nll": 0.023985598315494787, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.43340605072237054, - "correct": 1 - }, - "two_order": { - "nll": 0.38058980336114906, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.024726280462587773, - "correct": 1 - }, - "two_order": { - "nll": 0.020370824390199497, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.22541265160164786, - "correct": 1 - }, - "two_order": { - "nll": 0.13391489522799135, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.0645989391626965, - "correct": 1 - }, - "two_order": { - "nll": 0.05365559076948054, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.41555456462894624, - "correct": 1 - }, - "two_order": { - "nll": 0.4683187638800554, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.45093728159512503, - "correct": 1 - }, - "two_order": { - "nll": 0.4987931718783924, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.36091361875462236, - "correct": 1 - }, - "two_order": { - "nll": 0.32051824994664946, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.004225248102107815, - "correct": 1 - }, - "two_order": { - "nll": 0.006414862644828894, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.01763377729658182, - "correct": 1 - }, - "two_order": { - "nll": 0.020158339822136835, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 3.006139464882406, - "correct": 0 - }, - "two_order": { - "nll": 0.863983578966528, - "correct": 0 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.029621758008642193, - "correct": 1 - }, - "two_order": { - "nll": 0.02739086023765856, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0026351429162579636, - "correct": 1 - }, - "two_order": { - "nll": 0.0025946564096059424, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.7052189455721887, - "correct": 1 - }, - "two_order": { - "nll": 0.7232211210048433, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.11264593600646411, - "correct": 1 - }, - "two_order": { - "nll": 0.08880934788107725, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.2096421613300998, - "correct": 1 - }, - "two_order": { - "nll": 0.23750500809770175, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.11229572343004564, - "correct": 1 - }, - "two_order": { - "nll": 0.11296825616451285, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.03893702963032614, - "correct": 1 - }, - "two_order": { - "nll": 0.033706210211879466, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.8117422716906217, - "correct": 1 - }, - "two_order": { - "nll": 0.9459118127749184, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.1944627114257376, - "correct": 1 - }, - "two_order": { - "nll": 1.026712203499577, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.04696127952713844, - "correct": 1 - }, - "two_order": { - "nll": 0.3904763810754554, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.8926861221897002, - "correct": 0 - }, - "two_order": { - "nll": 1.8722083706381547, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.06583535553267797, - "correct": 1 - }, - "two_order": { - "nll": 0.0648537601404892, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.33375488321755525, - "correct": 1 - }, - "two_order": { - "nll": 0.2959266875664315, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.06785584399677826, - "correct": 1 - }, - "two_order": { - "nll": 0.05817518915315277, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 1.0334570697206436, - "correct": 0 - }, - "two_order": { - "nll": 0.9833501295116349, - "correct": 0 - } - } - } - ] -} diff --git a/docs/training/jet-v5/leaderboard-provenance.json b/docs/training/jet-v5/leaderboard-provenance.json deleted file mode 100644 index 58c52fe75ae71f3c402b2d527094672b5e855a48..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/leaderboard-provenance.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "repository": "multimodalart/jev-decision-index", - "revision": "fe5c2827ee888c88bc9a2ec6d573c47f2d41da3f", - "source": "data/index.json", - "generated_utc": "2026-09-24T06:34:59+00:00", - "sha256": "dd672173d9819d0aee7f2fcc56a2004aca5efcb05d9e7c89b1728df6426fc8dc", - "pinned_snapshot_verified": true -} diff --git a/docs/training/jet-v5/leaderboard-snapshot.json b/docs/training/jet-v5/leaderboard-snapshot.json deleted file mode 100644 index 8a7105fcc44ed6f8a03fb38275e899cbdc4958af..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/leaderboard-snapshot.json +++ /dev/null @@ -1 +0,0 @@ -{"generated_utc":"2026-09-24T06:34:59+00:00","suite":{"requests":121057,"scoreable":120615,"benchmarks":44,"jev_benchmarks":43,"panel":40,"panel_full":40,"categories":5,"dropped":["MiniWoB++","Boxoban","RTFM","ScienceWorld","Hanabi","Codenames"],"hardware":"1 x NVIDIA RTX PRO 6000","jev_version":"jev-1.13.0","panel_id":"decision-index-0.2","calibration_note":"33 benchmarks with a per-field right or wrong answer (ToolRet, BRIGHT, RouterBench and ForecastBench excluded), each weighted equally; a 1-in-6 sample of requests; confidence is the probability placed on the chosen option.","formulas":[{"id":"balanced_raw","normalization":"raw","formula":"100*sum_c(0.2*category[c])"},{"id":"balanced_skill","normalization":"skill","formula":"100*sum_c(0.2*category[c])"},{"id":"breadth_skill","normalization":"skill","shift":0.1,"formula":"100*(product_c((0.1+0.9*category[c])**0.2)-0.1)/0.9"}],"edition":"release-v2","label":"Decision Index 0.2","corpus_sha256":"b2b56d6fb636837ca469e689087bdbf373dda8de7638aa2da6793e6eda0792d5","base_edition":"release-v1","note":"Release v2 keeps every v1 row except a stratified subset cut of ToolRet (1,000 of 7,961 queries incl. excluded ones) and BRIGHT (550 of 1,384 queries); all other benchmarks, question order, options and scoring are unchanged. Kept ToolRet and BRIGHT queries retain their complete candidate lists.","added":{"edition":"v0.2","benchmarks":[56,57,58,59,61,62,64],"requests":30419,"in_panel":true,"note":"Added in 0.2 and part of the index panel; every entrant on the 0.2 board has results on all of them."},"areas":{"knowledge":[25,30,31,32,33,43,44,45,57,58],"language":[11,12,28,29,38,39,40,41,42,59],"retrieval":[4,5,10,36,37,56,61],"tools":[1,2,3,6,9,62],"arts":[20,21,22,23,48,50,64]},"not_in_index":[{"id":24,"name":"MMLU"},{"id":26,"name":"ARC-Easy"},{"id":27,"name":"ARC-Challenge"},{"id":34,"name":"SimpleBench"}],"lower_rules":[{"id":48,"name":"ForecastBench","baseline":0.25,"best":0.0,"metric":"Brier loss","baseline_label":"always predict p = 0.5","rule":"clip((0.25 - Brier) / (0.25 - 0)) x coverage"}],"index_note":"Decision Index 0.2 averages 40 benchmarks in five equal-weight areas: each area is the plain mean of its benchmarks and the index is 100 x the mean of the five areas. Every score is coverage-adjusted, so an unanswered or unsupported request counts as wrong. ForecastBench enters against its baseline: clip((0.25 - Brier) / 0.25) x coverage, so always predicting 0.5 scores zero. MMLU, ARC-Easy, ARC-Challenge, SimpleBench stay on the board as non-index benchmarks. The six interactive environments are still unrun and stay out. Every entrant on the board has results on all 40 index benchmarks. Point estimates only, no uncertainty intervals yet.","changes":["15 new models on the board, plus newer versions of 14 existing ones.","The index now counts 40 benchmarks, up from 19: 7 brand-new ones (MMLU-Pro, BBH, When2Call, RAGTruth, HoVer, New Yorker caption matching, PhishNChips), plus 15 we already ran that now count, such as HLE, ANLI, WinoGrande, HellaSwag, BANKING77 and CLINC150.","MMLU left the index: top models have nearly maxed it out, and MMLU-Pro takes its place. ARC-Easy, ARC-Challenge and SimpleBench are still shown but don't count.","ToolRet and BRIGHT are run on a subset of their data that doesn't skew results but is faster and cheaper to run."]},"categories":[{"id":"knowledge","label":"Knowledge & Reasoning","color":"#5976bd","icon":"M4 4h6a3 3 0 0 1 3 3v14a4 4 0 0 0-4-2H4z M20 4h-4a3 3 0 0 0-3 3v14a4 4 0 0 1 4-2h3z","description":"Academic knowledge, commonsense, mathematics, and code.","panel":[25,30,31,32,33,43,44,45,57,58],"panel_full":[25,30,31,32,33,43,44,45,57,58],"suite":[24,25,26,27,30,32,33,34,43,44,45,57,58,31],"group":[24,25,26,27,30,32,33,34,43,44,45,51,52,57,58],"priority":[24,25,30,45,28]},{"id":"language","label":"Language Understanding","color":"#407e9e","icon":"M4 4h16v12H9l-5 4z M8 8h8 M8 12h5","description":"Read a text and judge what it means: entailment, sentiment, sarcasm and stance.","panel":[11,12,28,29,38,39,40,41,42,59],"panel_full":[11,12,28,29,38,39,40,41,42,59],"suite":[11,12,38,39,40,41,42,28,29,59],"group":[11,12,38,39,40,41,42,28,29,59],"priority":[40,42,39,38,11]},{"id":"retrieval","label":"Retrieval & Classification","color":"#3e8e5e","icon":"M11 4a7 7 0 1 0 0 14 7 7 0 0 0 0-14z M20 20l-4.3-4.3","description":"Pick the right item from a large set: rank documents and products by relevance, route a message to one of many intents, sort out spam and phishing.","panel":[4,5,10,36,37,56,61],"panel_full":[4,5,10,36,37,56,61],"suite":[36,37,4,5,10,56,61],"group":[36,37,4,5,10,54,55,56,61],"priority":[36,37,4,5,10,55]},{"id":"tools","label":"Tools & Automation","color":"#b77a38","icon":"M5 4v11a4 4 0 0 0 4 4h10 M5 9h9a4 4 0 0 0 4-4 M2 7l3-3 3 3 m8 9 3 3-3 3","description":"Tool choice, model routing, household control, and browser or simulator actions.","panel":[1,2,3,6,9,62],"panel_full":[1,2,3,6,9,62],"suite":[1,2,3,6,9,62],"group":[1,2,3,6,9,53,62,7,16],"priority":[1,9,6]},{"id":"arts","label":"Arts & Human Taste","color":"#b6698a","icon":"M12 3a9 9 0 1 0 0 18h2a2 2 0 0 0 0-4h-1a2 2 0 0 1 0-4h4a4 4 0 0 0 4-4c0-4-5-6-9-6 M7 8h.01 M12 6h.01 M17 8h.01 M6 13h.01","description":"Poetry, music, humor, human preferences, and forecasting.","panel":[20,21,22,23,48,50,64],"panel_full":[20,21,22,23,48,50,64],"suite":[20,21,22,23,48,50,64],"group":[20,21,22,23,48,50,64],"priority":[20,22,21,23,48]},{"id":"games","label":"Games & Strategy","color":"#ad6f48","icon":"M7 8h10a4 4 0 0 1 4 4l1 5a2 2 0 0 1-3 2l-3-3H8l-3 3a2 2 0 0 1-3-2l1-5a4 4 0 0 1 4-4z M7 10v4 M5 12h4 M16 11h.01 M18 13h.01","description":"Move selection, spatial planning, rules, and decisions against other players.","panel":[],"panel_full":[31],"suite":[],"jev_only":true,"folded_into":"knowledge","group":[31,13,15,17,19],"priority":[31]}],"benchmarks":{"1":{"id":1,"dataset":"BFCL","short":"BFCL","metric":"case exact accuracy","explainer":"Choose which supplied tools are needed for a request. Tests tool selection, not arguments or execution.","baseline":{"value":0.2592,"label":"Random whole-case guesses"},"lower":false,"range":null,"cases":1694,"group":"tools","in_suite":true,"interactive":false},"2":{"id":2,"dataset":"ToolRet","short":"ToolRet","metric":"nDCG@10","explainer":"Rank candidate tools by relevance to a user request after an initial keyword search.","baseline":{"value":0.0937,"label":"Random ranking"},"lower":false,"range":null,"cases":1000,"group":"tools","in_suite":true,"interactive":false},"3":{"id":3,"dataset":"API-Bank","short":"API-Bank","metric":"accuracy","explainer":"Choose the next API to call from 53 tools using the conversation and earlier tool results.","baseline":{"value":0.0189,"label":"Uniform random choice"},"lower":false,"range":null,"cases":508,"group":"tools","in_suite":true,"interactive":false},"4":{"id":4,"dataset":"BANKING77","short":"BANKING77","metric":"macro-F1","explainer":"Identify the intent of a banking support message, such as a failed transfer or a missing card.","baseline":{"value":0.0127,"label":"Random choices · F1"},"lower":false,"range":null,"cases":3080,"group":"retrieval","in_suite":true,"interactive":false},"5":{"id":5,"dataset":"CLINC150+OOS","short":"CLINC150","metric":"macro-F1","explainer":"Classify a request into 150 assistant intents, or recognize that it is outside the supported scope.","baseline":{"value":0.006,"label":"Random choices · F1"},"lower":false,"range":null,"cases":5500,"group":"retrieval","in_suite":true,"interactive":false},"6":{"id":6,"dataset":"RouterBench","short":"RouterBench","metric":"selected quality (quality objective)","explainer":"Route a prompt to a model to maximize measured quality or balance quality with cost.","baseline":{"value":0.5735,"label":"Random model selection"},"lower":false,"range":null,"cases":5001,"group":"tools","in_suite":true,"interactive":false},"9":{"id":9,"dataset":"Home appliance simulator","short":"Home appliances","metric":"case exact accuracy","explainer":"Translate a household command into device decisions. A request passes only if every required field is correct.","baseline":{"value":0.0,"label":"Random whole-case guesses"},"lower":false,"range":null,"cases":160,"group":"tools","in_suite":true,"interactive":false},"10":{"id":10,"dataset":"SGD/SGD-X","short":"SGD","metric":"macro-F1","explainer":"Infer the user's current intent from a service dialogue. The service is supplied; this track evaluates SGD, not SGD-X or slot extraction.","baseline":{"value":0.399,"label":"Random choices · F1"},"lower":false,"range":null,"cases":2500,"group":"retrieval","in_suite":true,"interactive":false},"11":{"id":11,"dataset":"ContractNLI","short":"ContractNLI","metric":"macro-F1","explainer":"Read a contract and decide whether each of 17 legal statements is supported, contradicted, or unstated.","baseline":{"value":0.3085,"label":"Random choices · F1"},"lower":false,"range":null,"cases":123,"group":"language","in_suite":true,"interactive":false},"12":{"id":12,"dataset":"ANLI","short":"ANLI","metric":"macro-F1","explainer":"Decide whether a claim follows from, conflicts with, or is neutral to a premise designed to fool inference models.","baseline":{"value":0.3324,"label":"Random choices · F1"},"lower":false,"range":null,"cases":3200,"group":"language","in_suite":true,"interactive":false},"20":{"id":20,"dataset":"BPoMP","short":"BPoMP","metric":"accuracy","explainer":"Distinguish an original limerick from a perturbed version of a poem. This tests sensitivity to poetry structure, not aesthetic taste.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":811,"group":"arts","in_suite":true,"interactive":false},"21":{"id":21,"dataset":"Humicroedit","short":"Humicroedit","metric":"accuracy","explainer":"Choose the funnier of two edited news headlines, using human humor ratings as the answer key.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":2628,"group":"arts","in_suite":true,"interactive":false},"22":{"id":22,"dataset":"POP909-CL","short":"POP909","metric":"accuracy","explainer":"Recognize the annotated chord at a beat from symbolic musical notes, across 129 pitch-class categories.","baseline":{"value":0.0078,"label":"Uniform random choice"},"lower":false,"range":null,"cases":2000,"group":"arts","in_suite":true,"interactive":false},"23":{"id":23,"dataset":"cfcolor","short":"cfcolor","metric":"accuracy","explainer":"Predict which color palette a person preferred, using their earlier ratings and the palettes' RGB values.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":5000,"group":"arts","in_suite":true,"interactive":false},"24":{"id":24,"dataset":"MMLU","short":"MMLU","metric":"accuracy","explainer":"Answer four-choice questions across 57 academic subjects, from history and law to science and medicine.","baseline":{"value":0.25,"label":"Uniform random choice"},"lower":false,"range":null,"cases":14042,"group":"knowledge","in_suite":true,"interactive":false},"25":{"id":25,"dataset":"GPQA Diamond","short":"GPQA Diamond","metric":"accuracy","explainer":"Answer 198 graduate-level science questions in physics, chemistry, and biology, with four options each.","baseline":{"value":0.25,"label":"Uniform random choice"},"lower":false,"range":null,"cases":198,"group":"knowledge","in_suite":true,"interactive":false},"26":{"id":26,"dataset":"ARC-Easy","short":"ARC-Easy","metric":"accuracy","explainer":"Answer the easier split of school science multiple-choice questions.","baseline":{"value":0.2502,"label":"Uniform random choice"},"lower":false,"range":null,"cases":2376,"group":"knowledge","in_suite":true,"interactive":false},"27":{"id":27,"dataset":"ARC-Challenge","short":"ARC-Challenge","metric":"accuracy","explainer":"Answer the more difficult split of school science questions that simple retrieval methods struggle with.","baseline":{"value":0.2502,"label":"Uniform random choice"},"lower":false,"range":null,"cases":1172,"group":"knowledge","in_suite":true,"interactive":false},"28":{"id":28,"dataset":"WinoGrande","short":"WinoGrande","metric":"accuracy","explainer":"Use commonsense context to choose which of two words correctly completes an ambiguous sentence.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":1267,"group":"language","in_suite":true,"interactive":false},"29":{"id":29,"dataset":"HellaSwag","short":"HellaSwag","metric":"accuracy","explainer":"Choose the most plausible continuation of an everyday scene from four candidate endings.","baseline":{"value":0.25,"label":"Uniform random choice"},"lower":false,"range":null,"cases":10042,"group":"language","in_suite":true,"interactive":false},"30":{"id":30,"dataset":"GSM8K","short":"GSM8K","metric":"accuracy","explainer":"Solve grade-school arithmetic word problems by selecting a numeric answer from four or ten choices.","baseline":{"value":0.175,"label":"Uniform random choice"},"lower":false,"range":null,"cases":1319,"group":"knowledge","in_suite":true,"interactive":false},"31":{"id":31,"dataset":"ChessBench","short":"ChessBench","metric":"accuracy","explainer":"Choose the highest-valued legal chess move using published reference values. All legal moves are offered and ties count.","baseline":{"value":0.0819,"label":"Uniform random choice"},"lower":false,"range":null,"cases":5000,"group":"games","in_suite":true,"interactive":false},"32":{"id":32,"dataset":"MuSR","short":"MuSR","metric":"accuracy","explainer":"Combine clues across long narratives to solve murder mysteries, track objects, or allocate people to teams.","baseline":{"value":0.371,"label":"Uniform random choice"},"lower":false,"range":null,"cases":756,"group":"knowledge","in_suite":true,"interactive":false},"33":{"id":33,"dataset":"SATA-Bench","short":"SATA-Bench","metric":"case exact accuracy","explainer":"Select every correct answer to a question. The entire answer set must match; partial selections do not pass.","baseline":{"value":0.0131,"label":"Random whole-case guesses"},"lower":false,"range":null,"cases":1650,"group":"knowledge","in_suite":true,"interactive":false},"34":{"id":34,"dataset":"SimpleBench","short":"SimpleBench","metric":"accuracy","explainer":"Answer deceptively simple commonsense and trick questions. Only the ten public examples are evaluated here.","baseline":{"value":0.1667,"label":"Uniform random choice"},"lower":false,"range":null,"cases":10,"group":"knowledge","in_suite":true,"interactive":false},"36":{"id":36,"dataset":"BRIGHT","short":"BRIGHT","metric":"nDCG@10","explainer":"Rank documents that answer reasoning-intensive queries, within a candidate pool retrieved by keyword search.","baseline":{"value":0.0448,"label":"Random ranking"},"lower":false,"range":null,"cases":550,"group":"retrieval","in_suite":true,"interactive":false},"37":{"id":37,"dataset":"Amazon ESCI","short":"Amazon ESCI","metric":"macro-F1","explainer":"Judge whether a product is an exact match, substitute, complement, or irrelevant to a shopping query.","baseline":{"value":0.2027,"label":"Random choices · F1"},"lower":false,"range":null,"cases":5000,"group":"retrieval","in_suite":true,"interactive":false},"38":{"id":38,"dataset":"ACOS","short":"ACOS","metric":"case exact accuracy","explainer":"Identify which aspect-and-sentiment combinations appear in a review. Every combination must be right for the review to pass.","baseline":{"value":0.0,"label":"Random whole-case guesses"},"lower":false,"range":null,"cases":400,"group":"language","in_suite":true,"interactive":false},"39":{"id":39,"dataset":"FinEntity","short":"FinEntity","metric":"macro-F1","explainer":"Judge the sentiment expressed toward a specified company or financial entity in a document.","baseline":{"value":0.3201,"label":"Random choices · F1"},"lower":false,"range":null,"cases":979,"group":"language","in_suite":true,"interactive":false},"40":{"id":40,"dataset":"iSarcasmEval","short":"iSarcasmEval","metric":"Sarcasm F1 · track A, English","explainer":"Decide whether a single English text is sarcastic, scored with F1 for the sarcastic class. Headline track; the Arabic and paired-text tracks are listed under each result.","baseline":{"value":0.2227,"label":"Random choices · F1"},"lower":false,"range":null,"cases":4600,"group":"language","in_suite":true,"interactive":false},"41":{"id":41,"dataset":"VAST","short":"VAST","metric":"macro-F1","explainer":"Classify a text's stance toward a supplied topic as supportive, opposed, or neutral.","baseline":{"value":0.3333,"label":"Random choices · F1"},"lower":false,"range":null,"cases":3006,"group":"language","in_suite":true,"interactive":false},"42":{"id":42,"dataset":"NLI4CT","short":"NLI4CT","metric":"macro-F1","explainer":"Determine whether clinical-trial report sections support or contradict a statement, including deliberately altered statements.","baseline":{"value":0.486,"label":"Random choices · F1"},"lower":false,"range":null,"cases":5500,"group":"language","in_suite":true,"interactive":false},"43":{"id":43,"dataset":"CRUXEval","short":"CRUXEval","metric":"accuracy","explainer":"Predict a short Python function's output by selecting among candidate values drawn from prior model generations.","baseline":{"value":0.3697,"label":"Uniform random choice"},"lower":false,"range":null,"cases":570,"group":"knowledge","in_suite":true,"interactive":false},"44":{"id":44,"dataset":"CLadder","short":"CLadder","metric":"accuracy","explainer":"Answer yes/no questions about statistical association, intervention, and counterfactuals from supplied causal information.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":5000,"group":"knowledge","in_suite":true,"interactive":false},"45":{"id":45,"dataset":"HLE","short":"HLE","metric":"accuracy","explainer":"Answer the text-only multiple-choice subset of expert-level Humanity's Last Exam. Images and free-response questions are excluded.","baseline":{"value":0.1641,"label":"Uniform random choice"},"lower":false,"range":null,"cases":513,"group":"knowledge","in_suite":true,"interactive":false},"48":{"id":48,"dataset":"ForecastBench","short":"ForecastBench","metric":"Brier (lower is better)","explainer":"Assign probabilities to events resolved in July–September 2026 using frozen earlier context. Lower Brier loss is better.","baseline":{"value":0.25,"label":"Always predict p = 0.5"},"lower":true,"range":{"best":0.0,"worst":1.0,"name":"Brier loss"},"cases":10139,"group":"arts","in_suite":true,"interactive":false,"index_rule":{"mode":"vs baseline","baseline":0.25,"best":0.0,"rule":"clip((0.25 - Brier) / (0.25 - 0)) x coverage","baseline_label":"always predict p = 0.5"}},"50":{"id":50,"dataset":"Habermas Machine","short":"Habermas","metric":"accuracy","explainer":"Choose the consensus statement a human discussion group preferred, based on members' views and their recorded rankings.","baseline":{"value":0.311,"label":"Uniform random choice"},"lower":false,"range":null,"cases":1676,"group":"arts","in_suite":true,"interactive":false},"51":{"id":51,"dataset":"OpenBookQA","short":"OpenBookQA","metric":"accuracy","explainer":"Answer elementary-science questions from four original choices, without the supporting fact.","baseline":{"value":0.25,"label":"Uniform random choice"},"lower":false,"range":null,"cases":500,"group":"knowledge","in_suite":false,"interactive":false},"52":{"id":52,"dataset":"CommonsenseQA","short":"CommonsenseQA","metric":"accuracy","explainer":"Answer everyday commonsense questions from five original choices.","baseline":{"value":0.2,"label":"Uniform random choice"},"lower":false,"range":null,"cases":1221,"group":"knowledge","in_suite":false,"interactive":false},"53":{"id":53,"dataset":"Support-ticket calibration","short":"Support tickets","metric":"accuracy","explainer":"Judge a support ticket for queue, anger and priority, against deliberately noisy labels.","baseline":{"value":null,"label":null},"lower":false,"range":null,"cases":300,"group":"tools","in_suite":false,"interactive":false},"54":{"id":54,"dataset":"Phishing difficulty gradient","short":"Phishing gradient","metric":"accuracy","explainer":"Judge generated emails as phishing across four difficulty tiers, from obvious to misleading.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":800,"group":"retrieval","in_suite":false,"interactive":false},"55":{"id":55,"dataset":"Email spam classification","short":"Email spam","metric":"accuracy","explainer":"Detect spam in real emails, scored on the structured-focus question wording.","baseline":{"value":0.5,"label":"Uniform random choice"},"lower":false,"range":null,"cases":3000,"group":"retrieval","in_suite":false,"interactive":false},"56":{"id":56,"dataset":"PhishNChips phishing decisions","short":"PhishNChips","metric":"accuracy","explainer":"Judge whether an email and its links are malicious. Scored on the verdict question only.","baseline":{"value":0.5,"label":"uniform random choice, mean of 1/options"},"lower":false,"range":null,"cases":2000,"group":"retrieval","in_suite":true,"interactive":false,"edition_added":"v0.2"},"7":{"id":7,"dataset":"MiniWoB++","short":"MiniWoB++","metric":"episode success","explainer":"Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.","baseline":{"value":null,"label":null},"lower":false,"cases":null,"group":"tools","in_suite":false,"interactive":true},"16":{"id":16,"dataset":"ScienceWorld","short":"ScienceWorld","metric":"episode success","explainer":"Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.","baseline":{"value":null,"label":null},"lower":false,"cases":null,"group":"tools","in_suite":false,"interactive":true},"13":{"id":13,"dataset":"Boxoban","short":"Boxoban","metric":"episode success","explainer":"Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.","baseline":{"value":null,"label":null},"lower":false,"cases":null,"group":"games","in_suite":false,"interactive":true},"15":{"id":15,"dataset":"RTFM","short":"RTFM","metric":"episode success","explainer":"Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.","baseline":{"value":null,"label":null},"lower":false,"cases":null,"group":"games","in_suite":false,"interactive":true},"17":{"id":17,"dataset":"Hanabi","short":"Hanabi","metric":"episode success","explainer":"Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.","baseline":{"value":null,"label":null},"lower":false,"cases":null,"group":"games","in_suite":false,"interactive":true},"19":{"id":19,"dataset":"Codenames","short":"Codenames","metric":"episode success","explainer":"Interactive environment. No reproduction has run it, so it sits outside this edition of the index and appears only here.","baseline":{"value":null,"label":null},"lower":false,"cases":null,"group":"games","in_suite":false,"interactive":true},"57":{"id":57,"dataset":"MMLU-Pro","short":"MMLU-Pro","metric":"accuracy","explainer":"Graduate-level multiple choice across 14 subjects with up to ten options, the harder successor to MMLU.","baseline":{"value":0.1109,"label":"uniform random choice, mean of 1/options"},"lower":false,"range":null,"cases":12032,"group":"knowledge","in_suite":true,"interactive":false,"edition_added":"v0.2"},"58":{"id":58,"dataset":"BBH","short":"BBH","metric":"accuracy","explainer":"BIG-Bench Hard: 23 fixed-option reasoning task files (logic, state tracking, dates, formal fallacies and more), answered without chain of thought.","baseline":{"value":0.3101,"label":"uniform random choice, mean of 1/options"},"lower":false,"range":null,"cases":5507,"group":"knowledge","in_suite":true,"interactive":false,"edition_added":"v0.2"},"62":{"id":62,"dataset":"When2Call","short":"When2Call","metric":"accuracy","explainer":"Choose how an assistant should respond given the available tools: call a tool, ask for missing information, or say it cannot answer.","baseline":{"value":0.25,"label":"uniform random choice, mean of 1/options"},"lower":false,"range":null,"cases":3652,"group":"tools","in_suite":true,"interactive":false,"edition_added":"v0.2"},"64":{"id":64,"dataset":"New Yorker","short":"New Yorker","metric":"accuracy","explainer":"Pick the caption written for a New Yorker cartoon from five, given a text description of the cartoon.","baseline":{"value":0.2,"label":"uniform random choice, mean of 1/options"},"lower":false,"range":null,"cases":528,"group":"arts","in_suite":true,"interactive":false,"edition_added":"v0.2"},"59":{"id":59,"dataset":"RAGTruth","short":"RAGTruth","metric":"F1 on hallucinated class","explainer":"Decide whether a generated answer contains content not supported by the retrieved context it was written from. Scored with F1 on the hallucinated class.","baseline":{"value":0.4113,"label":"F1 of a fair coin; accuracy chance 0.5; always-hallucinated F1 0.518; majority-class accuracy 0.651"},"lower":false,"range":null,"cases":2700,"group":"language","in_suite":true,"interactive":false,"edition_added":"v0.2"},"61":{"id":61,"dataset":"HoVer","short":"HoVer","metric":"accuracy","explainer":"Verify a claim against the Wikipedia introductions its evidence cites (two to four documents, multi-hop). Oracle-document setting.","baseline":{"value":0.5,"label":"uniform random choice, mean of 1/options"},"lower":false,"range":null,"cases":4000,"group":"retrieval","in_suite":true,"interactive":false,"edition_added":"v0.2"}},"jev":{"engine":"jev","name":"Jev","version":"jev-1.13.0","edition":"Full release v1","results":{"7":{"score":1.0,"metric":"episode success","cases":9,"median_ms":null,"p95_ms":null,"usd":null},"16":{"score":0.067,"metric":"episode success","cases":10,"median_ms":null,"p95_ms":null,"usd":null},"13":{"score":0.0,"metric":"levels solved","cases":100,"median_ms":null,"p95_ms":null,"usd":null},"15":{"score":0.17,"metric":"episode success","cases":100,"median_ms":null,"p95_ms":null,"usd":null},"17":{"score":0.0,"metric":"episode success","cases":100,"median_ms":null,"p95_ms":null,"usd":null},"19":{"score":0.0,"metric":"episode success","cases":66,"median_ms":null,"p95_ms":null,"usd":null},"1":{"score":0.9575,"metric":"case exact accuracy","cases":1694,"median_ms":265.7,"p95_ms":382.1,"usd":0.1},"2":{"score":0.4472,"metric":"nDCG@10","cases":1000,"median_ms":322.8,"p95_ms":531.1,"usd":0.401},"3":{"score":0.8819,"metric":"accuracy","cases":508,"median_ms":351.4,"p95_ms":461.7,"usd":0.216},"4":{"score":0.7974,"metric":"macro-F1","cases":3080,"median_ms":265.6,"p95_ms":439.1,"usd":0.222},"5":{"score":0.8927,"metric":"macro-F1","cases":5500,"median_ms":247.5,"p95_ms":399.3,"usd":0.644},"6":{"score":0.7992,"metric":"selected quality (quality objective)","cases":5001,"median_ms":238.9,"p95_ms":370.9,"usd":1.3},"9":{"score":0.525,"metric":"case exact accuracy","cases":160,"median_ms":275.6,"p95_ms":360.6,"usd":0.023},"10":{"score":0.4295,"metric":"macro-F1","cases":2500,"median_ms":265.2,"p95_ms":457.0,"usd":0.19},"11":{"score":0.7169,"metric":"macro-F1","cases":123,"median_ms":306.2,"p95_ms":415.5,"usd":0.023},"12":{"score":0.7479,"metric":"macro-F1","cases":3200,"median_ms":259.2,"p95_ms":434.6,"usd":0.058},"20":{"score":0.906,"metric":"accuracy","cases":811,"median_ms":235.3,"p95_ms":392.4,"usd":0.091},"21":{"score":0.6187,"metric":"accuracy","cases":2628,"median_ms":261.9,"p95_ms":433.0,"usd":0.038},"22":{"score":0.181,"metric":"accuracy","cases":2000,"median_ms":301.8,"p95_ms":476.3,"usd":0.641},"23":{"score":0.6474,"metric":"accuracy","cases":5000,"median_ms":239.9,"p95_ms":400.8,"usd":0.192},"24":{"score":0.9173,"metric":"accuracy","cases":14042,"median_ms":232.5,"p95_ms":444.6,"usd":0.246},"25":{"score":0.7828,"metric":"accuracy","cases":198,"median_ms":253.8,"p95_ms":344.9,"usd":0.004},"26":{"score":0.9933,"metric":"accuracy","cases":2376,"median_ms":260.8,"p95_ms":411.8,"usd":0.036},"27":{"score":0.9778,"metric":"accuracy","cases":1172,"median_ms":258.1,"p95_ms":372.6,"usd":0.018},"28":{"score":0.9195,"metric":"accuracy","cases":1267,"median_ms":259.3,"p95_ms":374.6,"usd":0.018},"29":{"score":0.9452,"metric":"accuracy","cases":10042,"median_ms":231.0,"p95_ms":364.6,"usd":0.213},"30":{"score":0.7987,"metric":"accuracy","cases":1319,"median_ms":264.2,"p95_ms":412.1,"usd":0.055},"31":{"score":0.1722,"metric":"accuracy","cases":5000,"median_ms":241.0,"p95_ms":396.1,"usd":0.302},"32":{"score":0.6601,"metric":"accuracy","cases":756,"median_ms":255.7,"p95_ms":366.7,"usd":0.041},"33":{"score":0.2642,"metric":"case exact accuracy","cases":1650,"median_ms":267.3,"p95_ms":381.3,"usd":0.11},"34":{"score":0.5,"metric":"accuracy","cases":10,"median_ms":240.0,"p95_ms":403.6,"usd":0.0},"36":{"score":0.1901,"metric":"nDCG@10","cases":550,"median_ms":394.0,"p95_ms":597.6,"usd":0.314},"37":{"score":0.5521,"metric":"macro-F1","cases":5000,"median_ms":236.5,"p95_ms":396.7,"usd":0.162},"38":{"score":0.0625,"metric":"case exact accuracy","cases":400,"median_ms":343.1,"p95_ms":523.9,"usd":0.474},"39":{"score":0.8698,"metric":"macro-F1","cases":979,"median_ms":259.2,"p95_ms":371.1,"usd":0.023},"40":{"score":0.5051,"metric":"Sarcasm F1 · track A, English","cases":1400,"median_ms":243.7,"p95_ms":401.3,"usd":0.08,"tracks":[{"track":"A · Arabic","score":0.4449,"headline":false},{"track":"A · English","score":0.5051,"headline":true},{"track":"C · Arabic pairs","score":0.7,"headline":false},{"track":"C · English pairs","score":0.95,"headline":false}]},"41":{"score":0.6463,"metric":"macro-F1","cases":3006,"median_ms":262.6,"p95_ms":437.5,"usd":0.057},"42":{"score":0.8406,"metric":"macro-F1","cases":5500,"median_ms":238.5,"p95_ms":399.4,"usd":0.221},"43":{"score":0.7298,"metric":"accuracy","cases":570,"median_ms":250.9,"p95_ms":360.9,"usd":0.011},"44":{"score":0.7264,"metric":"accuracy","cases":5000,"median_ms":238.0,"p95_ms":392.3,"usd":0.09},"45":{"score":0.2008,"metric":"accuracy","cases":513,"median_ms":254.2,"p95_ms":347.5,"usd":0.016},"48":{"score":0.1736,"metric":"Brier (lower is better)","cases":10139,"median_ms":233.9,"p95_ms":368.3,"usd":0.499},"50":{"score":0.4594,"metric":"accuracy","cases":1676,"median_ms":264.0,"p95_ms":377.9,"usd":0.084},"51":{"score":0.94,"metric":"accuracy","cases":500,"median_ms":1411.9,"p95_ms":2198.7,"usd":0.0},"52":{"score":0.8747,"metric":"accuracy","cases":1221,"median_ms":1387.5,"p95_ms":2570.8,"usd":0.0},"53":{"score":0.905,"metric":"accuracy","cases":300,"median_ms":1383.9,"p95_ms":2379.8,"usd":0.0},"54":{"score":0.9463,"metric":"accuracy","cases":800,"median_ms":1840.8,"p95_ms":3083.4,"usd":0.0},"55":{"score":0.9743,"metric":"accuracy","cases":3000,"median_ms":1382.0,"p95_ms":2622.4,"usd":0.0},"56":{"score":0.6255,"metric":"accuracy","cases":2000,"median_ms":1372.4,"p95_ms":null,"usd":0.077,"answered":2000,"edition_added":"v0.2"},"57":{"score":0.827,"metric":"accuracy","cases":12032,"median_ms":1053.5,"p95_ms":null,"usd":0.28,"answered":12032,"edition_added":"v0.2"},"58":{"score":0.9292,"metric":"accuracy","cases":5507,"median_ms":1032.4,"p95_ms":null,"usd":0.108,"answered":5507,"edition_added":"v0.2"},"62":{"score":0.8097,"metric":"accuracy","cases":3652,"median_ms":1047.2,"p95_ms":null,"usd":0.208,"answered":3652,"edition_added":"v0.2"},"64":{"score":0.7008,"metric":"accuracy","cases":528,"median_ms":1043.8,"p95_ms":null,"usd":0.011,"answered":528,"edition_added":"v0.2"},"59":{"score":0.7653,"metric":"F1 on hallucinated class","cases":2700,"median_ms":1039.7,"p95_ms":null,"usd":0.13,"answered":2700,"edition_added":"v0.2"},"61":{"score":0.7285,"metric":"accuracy","cases":4000,"median_ms":945.0,"p95_ms":null,"usd":0.127,"answered":4000,"edition_added":"v0.2"}},"latency":{"median":252.8,"p95":436.6,"note":"Remote hosted API measured over HTTP. total_wall_ms includes client retries and network; http_wall_ms is the request itself. Local GPU reproductions have no network leg, so these are not a controlled speed comparison."},"interactive":[{"track":"miniwob","success_rate":1.0,"episodes":9},{"track":"minigrid_babyai","success_rate":0.0833,"episodes":12},{"track":"scienceworld","success_rate":0.0,"episodes":10},{"track":"hanabi","success_rate":null,"episodes":100},{"track":"leduc","success_rate":null,"episodes":300},{"track":"codenames","success_rate":0.0,"episodes":66},{"track":"rtfm","success_rate":0.17,"episodes":100}],"references":[{"catalog_id":25,"model":"Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)","score":0.9374,"metric":"Accuracy"},{"catalog_id":45,"model":"Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)","score":0.5913,"metric":"Accuracy"},{"catalog_id":25,"model":"GPT-6 Astra (max)","score":0.9606,"metric":"Accuracy"},{"catalog_id":45,"model":"GPT-6 Astra (max)","score":0.5468,"metric":"Accuracy"},{"catalog_id":25,"model":"Claude Opus 5 (Adaptive Reasoning, Max Effort)","score":0.9323,"metric":"Accuracy"},{"catalog_id":45,"model":"Claude Opus 5 (Adaptive Reasoning, Max Effort)","score":0.5487,"metric":"Accuracy"},{"catalog_id":25,"model":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)","score":0.9263,"metric":"Accuracy"},{"catalog_id":45,"model":"Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)","score":0.5547,"metric":"Accuracy"},{"catalog_id":25,"model":"Muse Spark 1.3 (max)","score":0.9354,"metric":"Accuracy"},{"catalog_id":45,"model":"Muse Spark 1.3 (max)","score":0.487,"metric":"Accuracy"},{"catalog_id":25,"model":"GPT-5.6 Sol (max)","score":0.9414,"metric":"Accuracy"},{"catalog_id":45,"model":"GPT-5.6 Sol (max)","score":0.4949,"metric":"Accuracy"},{"catalog_id":25,"model":"Qwen3.8 Max (0902)","score":0.9283,"metric":"Accuracy"},{"catalog_id":45,"model":"Qwen3.8 Max (0902)","score":0.431,"metric":"Accuracy"},{"catalog_id":25,"model":"GLM-5.3 (max)","score":0.9172,"metric":"Accuracy"},{"catalog_id":45,"model":"GLM-5.3 (max)","score":0.4226,"metric":"Accuracy"},{"catalog_id":25,"model":"Grok 4.6 (high)","score":0.9495,"metric":"Accuracy"},{"catalog_id":45,"model":"Grok 4.6 (high)","score":0.4291,"metric":"Accuracy"},{"catalog_id":25,"model":"Kimi K3 (max)","score":0.9354,"metric":"Accuracy"},{"catalog_id":45,"model":"Kimi K3 (max)","score":0.469,"metric":"Accuracy"},{"catalog_id":45,"model":"Step 5 Preview","score":0.4648,"metric":"Accuracy"},{"catalog_id":25,"model":"GPT-5.6 Terra (max)","score":0.9253,"metric":"Accuracy"},{"catalog_id":45,"model":"GPT-5.6 Terra (max)","score":0.4291,"metric":"Accuracy"},{"catalog_id":25,"model":"GLM 5.3 Flash","score":0.9121,"metric":"Accuracy"},{"catalog_id":45,"model":"GLM 5.3 Flash","score":0.3985,"metric":"Accuracy"},{"catalog_id":25,"model":"Gemini 3.8 Flash (high)","score":0.9525,"metric":"Accuracy"},{"catalog_id":45,"model":"Gemini 3.8 Flash (high)","score":0.4782,"metric":"Accuracy"},{"catalog_id":45,"model":"DeepSeek V4.1 Flash (Reasoning, Max Effort)","score":0.3925,"metric":"Accuracy"},{"catalog_id":34,"model":"Claude Fable 5","score":0.819,"metric":"AVG@5 accuracy"},{"catalog_id":34,"model":"Claude Opus 5","score":0.806,"metric":"AVG@5 accuracy"}],"summary_notes":["Metrics exclude failed/incomplete cases; coverage is explicit. No silent replacement of failures.","Different primary metrics are not averaged across benchmarks.","Latency includes HTTPS and concurrent service scheduling; not isolated model inference.","Full raw HTTP responses/payloads and all retry timings are in http-attempts.jsonl.","BPoMP/RouterBench/GSM8K and chunked tasks retain all linked requests for sampled cases."],"calibration":{"acc":0.7472,"conf":0.8106,"ece":0.0651,"brier":0.3476,"over95":0.0201,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0002,"conf":0.082,"acc":0.4},{"lo":0.1,"w":0.0181,"conf":0.1649,"acc":0.0998},{"lo":0.2,"w":0.0303,"conf":0.2424,"acc":0.2013},{"lo":0.3,"w":0.0266,"conf":0.3493,"acc":0.3792},{"lo":0.4,"w":0.0307,"conf":0.4512,"acc":0.3783},{"lo":0.5,"w":0.0799,"conf":0.5464,"acc":0.4796},{"lo":0.6,"w":0.0784,"conf":0.6436,"acc":0.5217},{"lo":0.7,"w":0.1128,"conf":0.7555,"acc":0.7005},{"lo":0.8,"w":0.1005,"conf":0.8488,"acc":0.708},{"lo":0.9,"w":0.5225,"conf":0.9808,"acc":0.9342}],"areas":{"knowledge":{"ece":0.0638,"acc":0.7393,"conf":0.7773},"language":{"ece":0.0617,"acc":0.8248,"conf":0.8862},"retrieval":{"ece":0.1536,"acc":0.6685,"conf":0.8206},"tools":{"ece":0.0196,"acc":0.9458,"conf":0.9534},"arts":{"ece":0.0934,"acc":0.5701,"conf":0.6605}}},"scores":{"balanced_raw":63.87,"balanced_skill":51.67,"breadth_skill":50.78},"frozen_scores":{"balanced_raw":52.51,"balanced_skill":41.76,"breadth_skill":35.57},"categories":[{"id":"knowledge","raw":0.6094,"skill":0.5054,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.7019,"skill":0.5974,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.6023,"skill":0.4335,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.7368,"skill":0.6686,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.5434,"skill":0.3786,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9575,"skill":0.9426,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4472,"skill":0.3913,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7992,"skill":0.5266,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.786,"headline":false},{"track":"RouterBench-5shot","score":0.8125,"headline":false}]},"11":{"raw":0.7169,"skill":0.5906,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9092,"skill":0.8184,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6187,"skill":0.2374,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1658,"skill":0.1593,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6441,"skill":0.2883,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.9199,"skill":0.8932,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.7857,"skill":0.7143,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.7987,"skill":0.7565,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.8218,"headline":false},{"track":"GSM8K-10choice","score":0.7756,"headline":false}]},"31":{"raw":0.1722,"skill":0.0984,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1901,"skill":0.1507,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5521,"skill":0.4382,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5051,"skill":0.3633,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4449,"headline":false},{"track":"A · English","score":0.5051,"headline":true},{"track":"C · Arabic pairs","score":0.7,"headline":false},{"track":"C · English pairs","score":0.95,"headline":false}]},"41":{"raw":0.6463,"skill":0.4695,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.7298,"skill":0.5713,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.7264,"skill":0.4528,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4594,"skill":0.2154,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":1.0,"skill":1.0,"coverage":1.0,"random":null,"tracks":[]},"16":{"raw":0.067,"skill":0.067,"coverage":1.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":null,"tracks":[]},"15":{"raw":0.17,"skill":0.17,"coverage":1.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":null,"tracks":[]},"56":{"raw":0.6255,"skill":0.251,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.827,"skill":0.8055,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.9292,"skill":0.8973,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.8097,"skill":0.7463,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.7008,"skill":0.6259,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7653,"skill":0.6014,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7285,"skill":0.457,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6601,"skill":0.4596,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2642,"skill":0.2544,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.2008,"skill":0.0439,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.7479,"skill":0.6224,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.9195,"skill":0.839,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9452,"skill":0.9269,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0625,"skill":0.0625,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8698,"skill":0.8085,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8406,"skill":0.6899,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7974,"skill":0.7948,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8927,"skill":0.8921,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4295,"skill":0.0507,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.8819,"skill":0.8796,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.525,"skill":0.525,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.3056,"skill":0.3056,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":1.0,"pending":0.0},"models":[{"engine":"akash-gemma","name":"system-one-gemma","long_name":"system-one-gemma","variant":null,"meta":{"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":268098176,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/akash-kamat/system-one-gemma","base_model":"google/gemma-3-270m","base_url":"https://huggingface.co/google/gemma-3-270m","weights_repo":null,"served_checkpoint":null},"completed":121057,"counts":{"ok":120954},"lifted":true,"calibration":{"acc":0.3421,"conf":0.5507,"ece":0.2291,"brier":0.7958,"over95":0.0157,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0416,"conf":0.0656,"acc":0.0395},{"lo":0.1,"w":0.015,"conf":0.1554,"acc":0.0865},{"lo":0.2,"w":0.0935,"conf":0.2596,"acc":0.3689},{"lo":0.3,"w":0.1021,"conf":0.3525,"acc":0.2187},{"lo":0.4,"w":0.1251,"conf":0.4511,"acc":0.2903},{"lo":0.5,"w":0.2138,"conf":0.5478,"acc":0.3956},{"lo":0.6,"w":0.1525,"conf":0.6462,"acc":0.3549},{"lo":0.7,"w":0.1092,"conf":0.7462,"acc":0.3527},{"lo":0.8,"w":0.0803,"conf":0.8484,"acc":0.4576},{"lo":0.9,"w":0.0667,"conf":0.9534,"acc":0.479}],"areas":{"knowledge":{"ece":0.2395,"acc":0.3549,"conf":0.4855},"language":{"ece":0.325,"acc":0.3207,"conf":0.6457},"retrieval":{"ece":0.2167,"acc":0.405,"conf":0.62},"tools":{"ece":0.2613,"acc":0.2167,"conf":0.478},"arts":{"ece":0.15,"acc":0.3745,"conf":0.5245}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":32.4,"p95":983.3,"mean":227.2},"results":{"1":{"score":0.3483,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":70.9},"2":{"score":0.1123,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1002.4},"3":{"score":0.0118,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":6444.1},"4":{"score":0.5347,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":47.7},"5":{"score":0.2486,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":85.5},"6":{"score":0.3575,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":552.5},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":635.3},"10":{"score":0.4519,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":43.5},"11":{"score":0.2072,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1323.8},"12":{"score":0.2389,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":32.4},"20":{"score":0.5776,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":31.9},"21":{"score":0.4726,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":29.1},"22":{"score":0.0025,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":3193.9},"23":{"score":0.524,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":29.9},"24":{"score":0.3081,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":31.6},"25":{"score":0.2449,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":32.3},"26":{"score":0.4465,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":31.5},"27":{"score":0.3097,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":31.7},"28":{"score":0.4988,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":29.1},"29":{"score":0.2841,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":32.4},"30":{"score":0.1892,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":32.0},"31":{"score":0.0912,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":64.1},"32":{"score":0.4162,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":39.5},"33":{"score":0.0012,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":241.1},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":33.4},"36":{"score":0.0413,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1029.6},"37":{"score":0.1949,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":32.9},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":1879.9},"39":{"score":0.13,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":59.3},"40":{"score":0.2506,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":28.9,"tracks":[{"track":"A · Arabic","score":0.2466,"headline":false},{"track":"A · English","score":0.2506,"headline":true},{"track":"C · Arabic pairs","score":0.335,"headline":false},{"track":"C · English pairs","score":0.345,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3489,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":30.2},"42":{"score":0.4253,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":30.1},"43":{"score":0.3404,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":32.1},"44":{"score":0.502,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":29.9},"45":{"score":0.1257,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":33.2},"48":{"score":0.3722,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":32.0},"50":{"score":0.2959,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":35.6},"56":{"score":0.5995,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":259.6,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1317,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":31.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.2853,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":30.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.1569,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":35.3,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2917,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":31.0,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.1293,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":33.1,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5052,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":30.1,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":26.48,"balanced_skill":4.85,"breadth_skill":4.14},"frozen_scores":{"balanced_raw":21.76,"balanced_skill":1.91,"breadth_skill":1.85},"categories":[{"id":"knowledge","raw":0.2328,"skill":0.0135,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.2513,"skill":0.0105,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.368,"skill":0.1529,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.1645,"skill":0.0238,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3075,"skill":0.042,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3483,"skill":0.1202,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.1123,"skill":0.0225,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.3576,"skill":0.0,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.2032,"headline":false},{"track":"RouterBench-5shot","score":0.512,"headline":false}]},"11":{"raw":0.2072,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5654,"skill":0.1309,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4726,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0027,"skill":0.0,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5242,"skill":0.0484,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3064,"skill":0.0752,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2449,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1892,"skill":0.0254,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2881,"headline":false},{"track":"GSM8K-10choice","score":0.0902,"headline":false}]},"31":{"raw":0.0912,"skill":0.0102,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0413,"skill":0.0,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.1949,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2506,"skill":0.036,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2466,"headline":false},{"track":"A · English","score":0.2506,"headline":true},{"track":"C · Arabic pairs","score":0.335,"headline":false},{"track":"C · English pairs","score":0.345,"headline":false}]},"41":{"raw":0.3489,"skill":0.0234,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3404,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.502,"skill":0.004,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.2959,"skill":0.0,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5995,"skill":0.199,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1317,"skill":0.0234,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.2853,"skill":0.0,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.1569,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2917,"skill":0.1146,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.1293,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5052,"skill":0.0104,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4162,"skill":0.0719,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0012,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1257,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.2389,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.4988,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.2841,"skill":0.0455,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.13,"skill":0.0,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4253,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.5347,"skill":0.5287,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.2486,"skill":0.2441,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4519,"skill":0.088,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0118,"skill":0.0,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"autojev-27b","name":"AutoJev-27B","long_name":"AutoJev-27B","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_estimated":false,"params_from_name":false,"trained_bytes":52173413928,"model_url":"https://huggingface.co/denis-pplx/autojev-27b","code_url":"https://github.com/denis-pplx/autojev","base_model":"Qwen/Qwen3.8-27B","base_url":"https://huggingface.co/Qwen/Qwen3.8-27B","weights_repo":"denis-pplx/autojev-27b","served_checkpoint":null},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.7377,"conf":0.7324,"ece":0.023,"brier":0.3582,"over95":0.0075,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0078,"conf":0.0763,"acc":0.2394},{"lo":0.1,"w":0.0302,"conf":0.1521,"acc":0.219},{"lo":0.2,"w":0.0323,"conf":0.2529,"acc":0.2714},{"lo":0.3,"w":0.0409,"conf":0.351,"acc":0.3724},{"lo":0.4,"w":0.0522,"conf":0.4507,"acc":0.4205},{"lo":0.5,"w":0.1407,"conf":0.554,"acc":0.6207},{"lo":0.6,"w":0.099,"conf":0.6481,"acc":0.6122},{"lo":0.7,"w":0.1053,"conf":0.7501,"acc":0.7242},{"lo":0.8,"w":0.1181,"conf":0.8513,"acc":0.8502},{"lo":0.9,"w":0.3735,"conf":0.9626,"acc":0.9602}],"areas":{"knowledge":{"ece":0.0491,"acc":0.6872,"conf":0.6676},"language":{"ece":0.0269,"acc":0.8402,"conf":0.8179},"retrieval":{"ece":0.0879,"acc":0.6703,"conf":0.7553},"tools":{"ece":0.027,"acc":0.9384,"conf":0.9404},"arts":{"ece":0.0695,"acc":0.6079,"conf":0.5912}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":104.9,"p95":1229.0,"mean":454.7},"results":{"1":{"score":0.9764,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":293.7},"2":{"score":0.4618,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":3431.3},"3":{"score":0.8406,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":1451.9},"4":{"score":0.7903,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":186.2},"5":{"score":0.878,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":295.6},"6":{"score":0.7981,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":917.7},"9":{"score":0.7063,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":4123.9},"10":{"score":0.3976,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":235.5},"11":{"score":0.7815,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":6827.6},"12":{"score":0.7065,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":91.9},"20":{"score":0.9366,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":91.9},"21":{"score":0.624,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":91.9},"22":{"score":0.388,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":1100.5},"23":{"score":0.6546,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":124.7},"24":{"score":0.8401,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":91.4},"25":{"score":0.4949,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":92.1},"26":{"score":0.992,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":91.1},"27":{"score":0.9676,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":90.8},"28":{"score":0.8516,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":90.3},"29":{"score":0.9398,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":92.2},"30":{"score":0.6736,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":92.9},"31":{"score":0.172,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":182.9},"32":{"score":0.6184,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":204.6},"33":{"score":0.2988,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":700.8},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":94.5},"36":{"score":0.1946,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":5509.8},"37":{"score":0.5509,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":102.6},"38":{"score":0.005,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":2179.9},"39":{"score":0.929,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":103.1},"40":{"score":0.6034,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":92.8,"tracks":[{"track":"A · Arabic","score":0.4112,"headline":false},{"track":"A · English","score":0.6034,"headline":true},{"track":"C · Arabic pairs","score":0.78,"headline":false},{"track":"C · English pairs","score":0.965,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.7079,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":92.2},"42":{"score":0.8481,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":124.8},"43":{"score":0.7491,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":92.2},"44":{"score":0.745,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":92.0},"45":{"score":0.1198,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":93.4},"48":{"score":0.1948,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":164.1},"50":{"score":0.4177,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":183.2},"56":{"score":0.5995,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":523.1,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.6448,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":87.3,"edition_added":"v0.2","jev":0.827},"58":{"score":0.781,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":87.3,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.8168,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":164.5,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.7027,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":86.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.8034,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":164.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.7422,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":105.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":63.37,"balanced_skill":50.94,"breadth_skill":49.74},"frozen_scores":{"balanced_raw":47.38,"balanced_skill":36.64,"breadth_skill":30.71},"categories":[{"id":"knowledge","raw":0.5297,"skill":0.4093,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.7176,"skill":0.619,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5933,"skill":0.42,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.7667,"skill":0.6998,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.5609,"skill":0.3988,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9764,"skill":0.9681,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4618,"skill":0.4074,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7981,"skill":0.5238,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7851,"headline":false},{"track":"RouterBench-5shot","score":0.8111,"headline":false}]},"11":{"raw":0.7815,"skill":0.684,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9391,"skill":0.8782,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.624,"skill":0.2481,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.3781,"skill":0.3732,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6442,"skill":0.2883,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8474,"skill":0.7965,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4949,"skill":0.3265,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.6736,"skill":0.6108,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.7612,"headline":false},{"track":"GSM8K-10choice","score":0.5861,"headline":false}]},"31":{"raw":0.172,"skill":0.0982,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1946,"skill":0.1554,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5509,"skill":0.4366,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.6034,"skill":0.4898,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4112,"headline":false},{"track":"A · English","score":0.6034,"headline":true},{"track":"C · Arabic pairs","score":0.78,"headline":false},{"track":"C · English pairs","score":0.965,"headline":false}]},"41":{"raw":0.7079,"skill":0.5619,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.7491,"skill":0.602,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.745,"skill":0.49,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4177,"skill":0.1548,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5995,"skill":0.199,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.6448,"skill":0.6005,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.781,"skill":0.6826,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.8168,"skill":0.7557,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.7027,"skill":0.6284,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.8034,"skill":0.666,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7422,"skill":0.4844,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6184,"skill":0.3933,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2988,"skill":0.2895,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1198,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.7065,"skill":0.5604,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8516,"skill":0.7032,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9398,"skill":0.9197,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.005,"skill":0.005,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.929,"skill":0.8956,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8481,"skill":0.7045,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7903,"skill":0.7876,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.878,"skill":0.8773,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.3976,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.8406,"skill":0.8375,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.7063,"skill":0.7063,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.2208,"skill":0.2208,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"bosun-v3.1-0.6b","name":"Bosun v3.1 0.6B","long_name":"Bosun v3.1 0.6B LoRA","variant":null,"meta":{"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":596049920,"params_estimated":false,"params_from_name":false,"trained_bytes":41470928,"model_url":"https://huggingface.co/Hanno-Labs/bosun-v3.1-0.6b","code_url":"https://github.com/Hanno-Labs/jev-compatible-server","base_model":"Qwen/Qwen3-0.6B-Base","base_url":"https://huggingface.co/Qwen/Qwen3-0.6B-Base","weights_repo":"Hanno-Labs/bosun-v3.1-0.6b","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.436,"conf":0.5777,"ece":0.1445,"brier":0.6711,"over95":0.008,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0544,"conf":0.0453,"acc":0.0512},{"lo":0.1,"w":0.0263,"conf":0.1506,"acc":0.1913},{"lo":0.2,"w":0.0571,"conf":0.2458,"acc":0.1226},{"lo":0.3,"w":0.0864,"conf":0.3548,"acc":0.2501},{"lo":0.4,"w":0.1014,"conf":0.4491,"acc":0.3119},{"lo":0.5,"w":0.1837,"conf":0.5505,"acc":0.4418},{"lo":0.6,"w":0.1654,"conf":0.6471,"acc":0.5073},{"lo":0.7,"w":0.128,"conf":0.7465,"acc":0.5471},{"lo":0.8,"w":0.0996,"conf":0.8495,"acc":0.598},{"lo":0.9,"w":0.0976,"conf":0.9495,"acc":0.7503}],"areas":{"knowledge":{"ece":0.1695,"acc":0.3103,"conf":0.4798},"language":{"ece":0.1595,"acc":0.5205,"conf":0.68},"retrieval":{"ece":0.101,"acc":0.6176,"conf":0.6825},"tools":{"ece":0.21,"acc":0.5258,"conf":0.6823},"arts":{"ece":0.0962,"acc":0.386,"conf":0.4816}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":107.0,"p95":577.3,"mean":282.5},"results":{"1":{"score":0.4658,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":180.8},"2":{"score":0.393,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":3195.1},"3":{"score":0.5315,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":495.8},"4":{"score":0.7613,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":140.8},"5":{"score":0.6483,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":189.4},"6":{"score":0.6409,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":422.2},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":2606.7},"10":{"score":0.5432,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":150.0},"11":{"score":0.2165,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":3371.6},"12":{"score":0.2749,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":101.0},"20":{"score":0.489,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":99.1},"21":{"score":0.5091,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":104.0},"22":{"score":0.035,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":531.5},"23":{"score":0.5094,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":114.7},"24":{"score":0.3468,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":99.4},"25":{"score":0.2755,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":96.8},"26":{"score":0.5614,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":93.9},"27":{"score":0.4334,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":93.4},"28":{"score":0.4972,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":92.5},"29":{"score":0.3277,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":97.2},"30":{"score":0.1713,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":91.9},"31":{"score":0.0834,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":121.9},"32":{"score":0.4428,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":139.2},"33":{"score":0.0145,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":936.2},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":95.2},"36":{"score":0.1268,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":3337.8},"37":{"score":0.4299,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":95.2},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":5583.6},"39":{"score":0.4799,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":164.9},"40":{"score":0.0554,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":85.8,"tracks":[{"track":"A · Arabic","score":0.022,"headline":false},{"track":"A · English","score":0.0554,"headline":true},{"track":"C · Arabic pairs","score":0.455,"headline":false},{"track":"C · English pairs","score":0.47,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.2628,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":86.0},"42":{"score":0.4968,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":100.3},"43":{"score":0.3579,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":86.2},"44":{"score":0.4948,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":87.4},"45":{"score":0.1497,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":92.4},"48":{"score":0.289,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":111.8},"50":{"score":0.3663,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":103.2},"56":{"score":0.5,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":314.3,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1774,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":88.8,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3434,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":86.7,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3001,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":115.3,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.3864,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":85.2,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.2917,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":120.5,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5055,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":97.1,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":35.23,"balanced_skill":12.72,"breadth_skill":10.42},"frozen_scores":{"balanced_raw":25.79,"balanced_skill":5.93,"breadth_skill":5.2},"categories":[{"id":"knowledge","raw":0.2511,"skill":0.0274,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.2903,"skill":0.036,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5021,"skill":0.2892,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.3886,"skill":0.2259,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3295,"skill":0.0573,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.4658,"skill":0.2788,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.393,"skill":0.3316,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.6409,"skill":0.1556,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6085,"headline":false},{"track":"RouterBench-5shot","score":0.6733,"headline":false}]},"11":{"raw":0.2165,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.4897,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5091,"skill":0.0183,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0333,"skill":0.0257,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5217,"skill":0.0434,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3579,"skill":0.1439,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2755,"skill":0.034,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1713,"skill":0.0,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2456,"headline":false},{"track":"GSM8K-10choice","score":0.097,"headline":false}]},"31":{"raw":0.0834,"skill":0.0017,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1268,"skill":0.0843,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4299,"skill":0.2849,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.0554,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.022,"headline":false},{"track":"A · English","score":0.0554,"headline":true},{"track":"C · Arabic pairs","score":0.455,"headline":false},{"track":"C · English pairs","score":0.47,"headline":false}]},"41":{"raw":0.2628,"skill":0.0,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3579,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4948,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3663,"skill":0.0804,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1774,"skill":0.0748,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3434,"skill":0.0483,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3001,"skill":0.0668,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.3864,"skill":0.233,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.2917,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5055,"skill":0.011,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4428,"skill":0.1141,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0145,"skill":0.0014,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1497,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.2749,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.4972,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3277,"skill":0.1036,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.4799,"skill":0.235,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4968,"skill":0.021,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7613,"skill":0.7582,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.6483,"skill":0.6462,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5432,"skill":0.2399,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.5315,"skill":0.5225,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"bosun-v3.1-1.7b","name":"Bosun v3.1 1.7B","long_name":"Bosun v3.1 1.7B LoRA","variant":null,"meta":{"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":1720574976,"params_estimated":false,"params_from_name":false,"trained_bytes":71879720,"model_url":"https://huggingface.co/Hanno-Labs/bosun-v3.1-1.7b","code_url":"https://github.com/Hanno-Labs/jev-compatible-server","base_model":"Qwen/Qwen3-1.7B-Base","base_url":"https://huggingface.co/Qwen/Qwen3-1.7B-Base","weights_repo":"Hanno-Labs/bosun-v3.1-1.7b","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.5078,"conf":0.6465,"ece":0.1391,"brier":0.6235,"over95":0.0263,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0505,"conf":0.0413,"acc":0.0455},{"lo":0.1,"w":0.0211,"conf":0.1502,"acc":0.1487},{"lo":0.2,"w":0.0232,"conf":0.2561,"acc":0.1817},{"lo":0.3,"w":0.0564,"conf":0.3536,"acc":0.2656},{"lo":0.4,"w":0.1163,"conf":0.4434,"acc":0.2416},{"lo":0.5,"w":0.1457,"conf":0.5499,"acc":0.4454},{"lo":0.6,"w":0.1229,"conf":0.6495,"acc":0.5456},{"lo":0.7,"w":0.1274,"conf":0.7507,"acc":0.6153},{"lo":0.8,"w":0.1357,"conf":0.8501,"acc":0.666},{"lo":0.9,"w":0.2008,"conf":0.9604,"acc":0.7687}],"areas":{"knowledge":{"ece":0.1936,"acc":0.3475,"conf":0.5412},"language":{"ece":0.1211,"acc":0.6198,"conf":0.7409},"retrieval":{"ece":0.1299,"acc":0.7441,"conf":0.8238},"tools":{"ece":0.0725,"acc":0.6822,"conf":0.7122},"arts":{"ece":0.1509,"acc":0.3973,"conf":0.5481}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":104.9,"p95":893.7,"mean":331.7},"results":{"1":{"score":0.6021,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":342.8},"2":{"score":0.3904,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":4060.6},"3":{"score":0.6083,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":983.3},"4":{"score":0.8561,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":219.1},"5":{"score":0.9088,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":334.4},"6":{"score":0.6827,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":652.8},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":2836.9},"10":{"score":0.5472,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":171.9},"11":{"score":0.4436,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":4628.4},"12":{"score":0.4048,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":101.5},"20":{"score":0.5522,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":99.9},"21":{"score":0.5384,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":100.6},"22":{"score":0.015,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":843.9},"23":{"score":0.5546,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":135.5},"24":{"score":0.4235,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":98.2},"25":{"score":0.25,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":105.5},"26":{"score":0.6654,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":93.3},"27":{"score":0.5358,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":92.5},"28":{"score":0.5178,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":91.1},"29":{"score":0.3488,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":103.0},"30":{"score":0.1494,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":101.3},"31":{"score":0.091,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":155.0},"32":{"score":0.4176,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":172.9},"33":{"score":0.0394,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":894.6},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":109.1},"36":{"score":0.1316,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":4195.5},"37":{"score":0.4784,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":116.2},"38":{"score":0.005,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":5930.9},"39":{"score":0.7229,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":102.9},"40":{"score":0.1538,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":53.0,"tracks":[{"track":"A · Arabic","score":0.0181,"headline":false},{"track":"A · English","score":0.1538,"headline":true},{"track":"C · Arabic pairs","score":0.45,"headline":false},{"track":"C · English pairs","score":0.445,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3642,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":52.4},"42":{"score":0.5324,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":67.8},"43":{"score":0.4018,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":53.9},"44":{"score":0.532,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":54.5},"45":{"score":0.1337,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":60.6},"48":{"score":0.3305,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":78.5},"50":{"score":0.361,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":62.3},"56":{"score":0.502,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":316.9,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.2463,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":56.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3684,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":53.5,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3237,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":84.1,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.3598,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":53.2,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.2212,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":89.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5363,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":68.7,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":39.45,"balanced_skill":17.91,"breadth_skill":15.36},"frozen_scores":{"balanced_raw":28.9,"balanced_skill":9.76,"breadth_skill":8.56},"categories":[{"id":"knowledge","raw":0.263,"skill":0.0462,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.3715,"skill":0.1205,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5658,"skill":0.3601,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.4345,"skill":0.2917,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3377,"skill":0.0768,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.6021,"skill":0.4629,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3904,"skill":0.3287,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.6827,"skill":0.2594,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6344,"headline":false},{"track":"RouterBench-5shot","score":0.7311,"headline":false}]},"11":{"raw":0.4436,"skill":0.1953,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5533,"skill":0.1066,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5384,"skill":0.0769,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.013,"skill":0.0053,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5383,"skill":0.0766,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.4371,"skill":0.2494,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.25,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1494,"skill":0.0,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2108,"headline":false},{"track":"GSM8K-10choice","score":0.0879,"headline":false}]},"31":{"raw":0.091,"skill":0.01,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1316,"skill":0.0893,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4784,"skill":0.3457,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.1538,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0181,"headline":false},{"track":"A · English","score":0.1538,"headline":true},{"track":"C · Arabic pairs","score":0.45,"headline":false},{"track":"C · English pairs","score":0.445,"headline":false}]},"41":{"raw":0.3642,"skill":0.0463,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4018,"skill":0.0508,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.532,"skill":0.064,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.361,"skill":0.0726,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.502,"skill":0.004,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.2463,"skill":0.1523,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3684,"skill":0.0845,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3237,"skill":0.0983,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.3598,"skill":0.1997,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.2212,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5363,"skill":0.0726,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4176,"skill":0.0741,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0394,"skill":0.0266,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1337,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.4048,"skill":0.1084,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5178,"skill":0.0356,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3488,"skill":0.1317,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.005,"skill":0.005,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7229,"skill":0.5924,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.5324,"skill":0.0903,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.8561,"skill":0.8542,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.9088,"skill":0.9082,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5472,"skill":0.2466,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.6083,"skill":0.6008,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"clm-v0.1-8b","name":"CLM-v0.1-8B","long_name":"CLM-v0.1-8B Qwen3-8B heads","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":8190735360,"params_estimated":false,"params_from_name":false,"trained_bytes":75557149,"model_url":"https://huggingface.co/Contrastive-LM/CLM-v0.1-8B","code_url":"https://github.com/Contrastive-LM/CLM","base_model":"Qwen/Qwen3-8B-Base","base_url":"https://huggingface.co/Qwen/Qwen3-8B-Base","weights_repo":"Contrastive-LM/CLM-v0.1-8B","served_checkpoint":"Contrastive-LM/CLM-v0.1-8B"},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.3776,"conf":0.6812,"ece":0.3036,"brier":0.8822,"over95":0.1158,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0011,"conf":0.0925,"acc":0.0332},{"lo":0.1,"w":0.035,"conf":0.1671,"acc":0.05},{"lo":0.2,"w":0.0497,"conf":0.242,"acc":0.0698},{"lo":0.3,"w":0.0685,"conf":0.361,"acc":0.1182},{"lo":0.4,"w":0.0659,"conf":0.4559,"acc":0.2311},{"lo":0.5,"w":0.1485,"conf":0.5496,"acc":0.39},{"lo":0.6,"w":0.1533,"conf":0.6584,"acc":0.5634},{"lo":0.7,"w":0.1071,"conf":0.7488,"acc":0.4026},{"lo":0.8,"w":0.1087,"conf":0.8508,"acc":0.4863},{"lo":0.9,"w":0.2622,"conf":0.966,"acc":0.4149}],"areas":{"knowledge":{"ece":0.2229,"acc":0.4369,"conf":0.6573},"language":{"ece":0.3593,"acc":0.4277,"conf":0.7865},"retrieval":{"ece":0.4925,"acc":0.1219,"conf":0.6144},"tools":{"ece":0.234,"acc":0.3779,"conf":0.5752},"arts":{"ece":0.3165,"acc":0.3497,"conf":0.6662}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":37.4,"p95":126.3,"mean":53.5},"results":{"1":{"score":0.389,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":51.5},"2":{"score":0.2761,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":299.3},"3":{"score":0.061,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":309.2},"4":{"score":0.0652,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":19.5},"5":{"score":0.0889,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":20.5},"6":{"score":0.3932,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":109.8},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":133.7},"10":{"score":0.1001,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":59.3},"11":{"score":0.2011,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":258.3},"12":{"score":0.3156,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":18.6},"20":{"score":0.3294,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":17.8},"21":{"score":0.5556,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":31.5},"22":{"score":0,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":111.5},"23":{"score":0.508,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":32.3},"24":{"score":0.358,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":49.4},"25":{"score":0.2041,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":50.9},"26":{"score":0.564,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":48.9},"27":{"score":0.4147,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":49.3},"28":{"score":0.5083,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":17.9},"29":{"score":0.4035,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":51.4},"30":{"score":0.2316,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":18.8},"31":{"score":0.0818,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.5},"32":{"score":0.3457,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":76.0},"33":{"score":0.0558,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":62.2},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":50.8},"36":{"score":0.1166,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":437.0},"37":{"score":0.0497,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.6},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":196.9},"39":{"score":0.3335,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":23.8},"40":{"score":0.3122,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":17.8,"tracks":[{"track":"A · Arabic","score":0.3401,"headline":false},{"track":"A · English","score":0.3122,"headline":true},{"track":"C · Arabic pairs","score":0.52,"headline":false},{"track":"C · English pairs","score":0.57,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.2271,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":19.9},"42":{"score":0.5649,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":19.8},"43":{"score":0.4281,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":48.9},"44":{"score":0.5394,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":18.2},"45":{"score":0.1577,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":52.4},"48":{"score":0.2815,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":36.4},"50":{"score":0.3711,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":72.2},"56":{"score":0.6505,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":48.1,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1466,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":52.5,"edition_added":"v0.2","jev":0.827},"58":{"score":0.308,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":20.4,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3869,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":54.7,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2538,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":35.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5177,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":28.4,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.52,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":31.2,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":27.12,"balanced_skill":6.51,"breadth_skill":6.34},"frozen_scores":{"balanced_raw":22.84,"balanced_skill":4.81,"breadth_skill":4.53},"categories":[{"id":"knowledge","raw":0.2499,"skill":0.0328,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.3384,"skill":0.069,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2273,"skill":0.0787,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.251,"skill":0.1006,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.2893,"skill":0.0442,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.389,"skill":0.1752,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.2761,"skill":0.2029,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.3932,"skill":0.0,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.2747,"headline":false},{"track":"RouterBench-5shot","score":0.5118,"headline":false}]},"11":{"raw":0.2011,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.3226,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5556,"skill":0.1111,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5219,"skill":0.0437,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3646,"skill":0.1529,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2041,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2316,"skill":0.0735,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.345,"headline":false},{"track":"GSM8K-10choice","score":0.1183,"headline":false}]},"31":{"raw":0.0818,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1166,"skill":0.0736,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.0497,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.3122,"skill":0.1152,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3401,"headline":false},{"track":"A · English","score":0.3122,"headline":true},{"track":"C · Arabic pairs","score":0.52,"headline":false},{"track":"C · English pairs","score":0.57,"headline":false}]},"41":{"raw":0.2271,"skill":0.0,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4281,"skill":0.0926,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5394,"skill":0.0788,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3711,"skill":0.0873,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6505,"skill":0.301,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1466,"skill":0.0402,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.308,"skill":0.0,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3869,"skill":0.1825,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2538,"skill":0.0673,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5177,"skill":0.1807,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.52,"skill":0.04,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3457,"skill":0.0,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0558,"skill":0.0433,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1577,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.3156,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5083,"skill":0.0166,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.4035,"skill":0.2047,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.3335,"skill":0.0197,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.5649,"skill":0.1535,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0652,"skill":0.0532,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0889,"skill":0.0834,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.1001,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.061,"skill":0.0429,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"decider-2b-fp8-http","name":"Decider 2B","long_name":"Decider 2B","variant":"FP8","meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":2274069824,"params_estimated":false,"params_from_name":false,"trained_bytes":3763692048,"model_url":"https://huggingface.co/Mapika/decider-2b","code_url":null,"base_model":"Qwen/Qwen3.5-2B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-2B-Base","weights_repo":"Mapika/decider-2b","served_checkpoint":"Mapika/decider-2b"},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.5681,"conf":0.6566,"ece":0.0885,"brier":0.5407,"over95":0.0094,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.05,"conf":0.0575,"acc":0.0288},{"lo":0.1,"w":0.0289,"conf":0.1499,"acc":0.1282},{"lo":0.2,"w":0.0265,"conf":0.2553,"acc":0.1883},{"lo":0.3,"w":0.0572,"conf":0.3525,"acc":0.2861},{"lo":0.4,"w":0.0973,"conf":0.4548,"acc":0.237},{"lo":0.5,"w":0.1327,"conf":0.5491,"acc":0.4997},{"lo":0.6,"w":0.1201,"conf":0.6498,"acc":0.5915},{"lo":0.7,"w":0.1258,"conf":0.7494,"acc":0.6396},{"lo":0.8,"w":0.1284,"conf":0.8508,"acc":0.742},{"lo":0.9,"w":0.233,"conf":0.9603,"acc":0.8816}],"areas":{"knowledge":{"ece":0.1114,"acc":0.4701,"conf":0.5815},"language":{"ece":0.0597,"acc":0.7007,"conf":0.7604},"retrieval":{"ece":0.1054,"acc":0.6542,"conf":0.7537},"tools":{"ece":0.0671,"acc":0.6702,"conf":0.7337},"arts":{"ece":0.0917,"acc":0.4347,"conf":0.5263}}},"gaps":{"share":0.0,"reasons":[{"reason":"runtime error","share":0.0}]},"latency":{"median":40.6,"p95":1003.2,"mean":282.8},"results":{"1":{"score":0.8294,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":432.2},"2":{"score":0.4324,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1429.7},"3":{"score":0.3169,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":1683.9},"4":{"score":0.7277,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":67.0},"5":{"score":0.7369,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":478.5},"6":{"score":0.7988,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":981.5},"9":{"score":0.025,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1037.4},"10":{"score":0.4885,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":460.1},"11":{"score":0.634,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1296.2},"12":{"score":0.4937,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":22.1},"20":{"score":0.6492,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":21.8},"21":{"score":0.5993,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":14.0},"22":{"score":0.0025,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":1637.8},"23":{"score":0.5696,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":40.0},"24":{"score":0.619,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":19.4},"25":{"score":0.3622,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":31.5},"26":{"score":0.9377,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":17.0},"27":{"score":0.8515,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":17.3},"28":{"score":0.6772,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":12.3},"29":{"score":0.8581,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":29.0},"30":{"score":0.2111,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":25.5},"31":{"score":0.0988,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":121.5},"32":{"score":0.5492,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":122.4},"33":{"score":0.2467,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":714.4},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":322.7},"36":{"score":0.1484,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":3120.4},"37":{"score":0.3468,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":55.3},"38":{"score":0.0025,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":692.9},"39":{"score":0.7612,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":38.0},"40":{"score":0.2243,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":15.8,"tracks":[{"track":"A · Arabic","score":0.2174,"headline":false},{"track":"A · English","score":0.2243,"headline":true},{"track":"C · Arabic pairs","score":0.375,"headline":false},{"track":"C · English pairs","score":0.55,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5104,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":24.1},"42":{"score":0.6547,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":188.0},"43":{"score":0.3298,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":24.9},"44":{"score":0.5726,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":22.5},"45":{"score":0.1198,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":49.7},"48":{"score":0.2706,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":92.4},"50":{"score":0.4218,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":105.3},"56":{"score":0.5005,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":165.0,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.3758,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":29.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5097,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":22.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5471,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":99.5,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.447,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":25.3,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.2835,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":96.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5907,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":50.1,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":44.5,"balanced_skill":26.11,"breadth_skill":24.77},"frozen_scores":{"balanced_raw":33.91,"balanced_skill":17.39,"breadth_skill":15.0},"categories":[{"id":"knowledge","raw":0.3376,"skill":0.1465,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.51,"skill":0.3125,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5056,"skill":0.2969,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.4916,"skill":0.3992,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3804,"skill":0.1506,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8294,"skill":0.7697,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4324,"skill":0.375,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7988,"skill":0.5256,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7853,"headline":false},{"track":"RouterBench-5shot","score":0.8123,"headline":false}]},"11":{"raw":0.634,"skill":0.4706,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.6481,"skill":0.2962,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5993,"skill":0.1986,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.002,"skill":0.0,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5449,"skill":0.0897,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.6439,"skill":0.5253,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3622,"skill":0.1497,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2111,"skill":0.0446,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2904,"headline":false},{"track":"GSM8K-10choice","score":0.1319,"headline":false}]},"31":{"raw":0.0988,"skill":0.0185,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1484,"skill":0.107,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3468,"skill":0.1808,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2243,"skill":0.0021,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2174,"headline":false},{"track":"A · English","score":0.2243,"headline":true},{"track":"C · Arabic pairs","score":0.375,"headline":false},{"track":"C · English pairs","score":0.55,"headline":false}]},"41":{"raw":0.5104,"skill":0.2656,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3298,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5726,"skill":0.1452,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4218,"skill":0.1609,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5005,"skill":0.001,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.3758,"skill":0.2979,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5097,"skill":0.2893,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5471,"skill":0.3961,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.447,"skill":0.3087,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.2835,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5907,"skill":0.1814,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5492,"skill":0.2833,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2467,"skill":0.2367,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1198,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.4937,"skill":0.2416,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6772,"skill":0.3544,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8581,"skill":0.8108,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0025,"skill":0.0025,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7612,"skill":0.6488,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.6547,"skill":0.3282,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7277,"skill":0.7242,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.7369,"skill":0.7353,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4885,"skill":0.1489,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.3169,"skill":0.3037,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.025,"skill":0.025,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"decider-35b-nvfp4","name":"Decider 35B-A3B","long_name":"Decider 35B-A3B","variant":"NVFP4","meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":35951822704,"params_estimated":false,"params_from_name":false,"trained_bytes":21013206328,"model_url":"https://huggingface.co/Mapika/decider-35b-a3b-nvfp4","code_url":null,"base_model":"Qwen/Qwen3.5-35B-A3B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base","weights_repo":"Mapika/decider-35b-a3b-nvfp4","served_checkpoint":"Mapika/decider-35b-a3b-nvfp4"},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.6748,"conf":0.6734,"ece":0.031,"brier":0.4153,"over95":0.0037,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0433,"conf":0.0708,"acc":0.0496},{"lo":0.1,"w":0.0304,"conf":0.1412,"acc":0.1385},{"lo":0.2,"w":0.0288,"conf":0.2567,"acc":0.3389},{"lo":0.3,"w":0.0611,"conf":0.3522,"acc":0.3491},{"lo":0.4,"w":0.0708,"conf":0.449,"acc":0.4597},{"lo":0.5,"w":0.1586,"conf":0.5586,"acc":0.4731},{"lo":0.6,"w":0.1169,"conf":0.6518,"acc":0.6765},{"lo":0.7,"w":0.1025,"conf":0.7497,"acc":0.7962},{"lo":0.8,"w":0.1033,"conf":0.8493,"acc":0.8918},{"lo":0.9,"w":0.2843,"conf":0.9704,"acc":0.9742}],"areas":{"knowledge":{"ece":0.0733,"acc":0.5585,"conf":0.622},"language":{"ece":0.0387,"acc":0.8224,"conf":0.7843},"retrieval":{"ece":0.0538,"acc":0.7811,"conf":0.7372},"tools":{"ece":0.065,"acc":0.7947,"conf":0.7368},"arts":{"ece":0.0347,"acc":0.5312,"conf":0.5078}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":99.1,"p95":210.1,"mean":116.4},"results":{"1":{"score":0.9215,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":196.7},"2":{"score":0.4542,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":363.3},"3":{"score":0.565,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":222.5},"4":{"score":0.8979,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":98.0},"5":{"score":0.9468,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":99.6},"6":{"score":0.7963,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":206.0},"9":{"score":0.2188,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":597.7},"10":{"score":0.6414,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":102.6},"11":{"score":0.7496,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":905.5},"12":{"score":0.6603,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":99.1},"20":{"score":0.8906,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":99.5},"21":{"score":0.6195,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":38.9},"22":{"score":0.0665,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":182.4},"23":{"score":0.649,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":97.6},"24":{"score":0.8261,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":39.3},"25":{"score":0.4082,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":101.4},"26":{"score":0.9899,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":39.7},"27":{"score":0.9633,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":39.7},"28":{"score":0.8863,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":39.5},"29":{"score":0.9665,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":96.5},"30":{"score":0.4606,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":99.8},"31":{"score":0.1088,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":97.9},"32":{"score":0.6184,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":99.9},"33":{"score":0.3473,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":204.5},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":100.8},"36":{"score":0.1667,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":489.7},"37":{"score":0.5065,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":98.7},"38":{"score":0.075,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":343.5},"39":{"score":0.8524,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":195.6},"40":{"score":0.5481,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":39.2,"tracks":[{"track":"A · Arabic","score":0.2937,"headline":false},{"track":"A · English","score":0.5481,"headline":true},{"track":"C · Arabic pairs","score":0.535,"headline":false},{"track":"C · English pairs","score":0.845,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5452,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":100.4},"42":{"score":0.7847,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":100.1},"43":{"score":0.5982,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":99.8},"44":{"score":0.6672,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":99.1},"45":{"score":0.0878,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":98.1},"48":{"score":0.2007,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":99.3},"50":{"score":0.4642,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":99.7},"56":{"score":0.6685,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":200.7,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.6046,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":99.6,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7133,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":97.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5972,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":99.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6402,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":99.3,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.631,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":99.9,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.635,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":98.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":57.24,"balanced_skill":43.5,"breadth_skill":42.66},"frozen_scores":{"balanced_raw":41.76,"balanced_skill":28.89,"breadth_skill":24.45},"categories":[{"id":"knowledge","raw":0.4614,"skill":0.3162,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6699,"skill":0.5406,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.6375,"skill":0.4801,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5922,"skill":0.5085,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.5009,"skill":0.3297,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9215,"skill":0.894,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4542,"skill":0.399,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7963,"skill":0.5196,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7832,"headline":false},{"track":"RouterBench-5shot","score":0.8094,"headline":false}]},"11":{"raw":0.7496,"skill":0.6379,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8918,"skill":0.7836,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6195,"skill":0.239,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0635,"skill":0.0562,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6297,"skill":0.2594,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8331,"skill":0.7775,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4082,"skill":0.2109,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4606,"skill":0.3531,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5724,"headline":false},{"track":"GSM8K-10choice","score":0.3487,"headline":false}]},"31":{"raw":0.1088,"skill":0.0293,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1667,"skill":0.1261,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5065,"skill":0.381,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5481,"skill":0.4187,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2937,"headline":false},{"track":"A · English","score":0.5481,"headline":true},{"track":"C · Arabic pairs","score":0.535,"headline":false},{"track":"C · English pairs","score":0.845,"headline":false}]},"41":{"raw":0.5452,"skill":0.3178,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.5982,"skill":0.3626,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6672,"skill":0.3344,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4642,"skill":0.2224,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6685,"skill":0.337,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.6046,"skill":0.5553,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.7133,"skill":0.5844,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5972,"skill":0.4629,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6402,"skill":0.5502,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.631,"skill":0.3732,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.635,"skill":0.27,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6184,"skill":0.3933,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3473,"skill":0.3386,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.0878,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6603,"skill":0.4912,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8863,"skill":0.7726,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9665,"skill":0.9553,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.075,"skill":0.075,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8524,"skill":0.7829,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7847,"skill":0.5811,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.8979,"skill":0.8966,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.9468,"skill":0.9465,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6414,"skill":0.4033,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.565,"skill":0.5566,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.2188,"skill":0.2188,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.1972,"skill":0.1972,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"decider-4b","name":"Decider 4B","long_name":"Decider 4B","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":8411558400,"model_url":"https://huggingface.co/Mapika/decider-4b","code_url":"https://github.com/Mapika/decider","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"Mapika/decider-4b","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.6292,"conf":0.7286,"ece":0.0994,"brier":0.4766,"over95":0.0173,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0369,"conf":0.0703,"acc":0.0666},{"lo":0.1,"w":0.0208,"conf":0.1363,"acc":0.1303},{"lo":0.2,"w":0.0158,"conf":0.2568,"acc":0.2542},{"lo":0.3,"w":0.0431,"conf":0.3544,"acc":0.2521},{"lo":0.4,"w":0.0564,"conf":0.4494,"acc":0.3416},{"lo":0.5,"w":0.111,"conf":0.5487,"acc":0.471},{"lo":0.6,"w":0.128,"conf":0.6379,"acc":0.399},{"lo":0.7,"w":0.098,"conf":0.752,"acc":0.6267},{"lo":0.8,"w":0.1164,"conf":0.8523,"acc":0.7324},{"lo":0.9,"w":0.3737,"conf":0.9716,"acc":0.9095}],"areas":{"knowledge":{"ece":0.1384,"acc":0.5283,"conf":0.6648},"language":{"ece":0.0434,"acc":0.7858,"conf":0.8292},"retrieval":{"ece":0.1502,"acc":0.6088,"conf":0.7468},"tools":{"ece":0.0634,"acc":0.8063,"conf":0.8675},"arts":{"ece":0.1048,"acc":0.4994,"conf":0.6026}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":23.4,"p95":217.2,"mean":64.7},"results":{"1":{"score":0.8796,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":90.9},"2":{"score":0.4453,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":610.7},"3":{"score":0.6043,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":419.4},"4":{"score":0.7472,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":42.1},"5":{"score":0.8655,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":80.4},"6":{"score":0.7988,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":133.1},"9":{"score":0.1625,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":181.4},"10":{"score":0.2003,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":81.8},"11":{"score":0.7234,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":157.7},"12":{"score":0.5636,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":14.8},"20":{"score":0.8344,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.8},"21":{"score":0.5967,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":11.7},"22":{"score":0.1,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":412.7},"23":{"score":0.6048,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":28.3},"24":{"score":0.7418,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":13.1},"25":{"score":0.4133,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":16.1},"26":{"score":0.9781,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":12.9},"27":{"score":0.9394,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":13.0},"28":{"score":0.8074,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":11.5},"29":{"score":0.9399,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":16.4},"30":{"score":0.5709,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":16.1},"31":{"score":0.105,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":42.5},"32":{"score":0.5372,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":43.7},"33":{"score":0.3158,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":122.1},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":15.4},"36":{"score":0.1818,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":920.0},"37":{"score":0.4238,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":22.2},"38":{"score":0.0325,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":475.5},"39":{"score":0.8604,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":21.6},"40":{"score":0.4207,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":11.6,"tracks":[{"track":"A · Arabic","score":0.0829,"headline":false},{"track":"A · English","score":0.4207,"headline":true},{"track":"C · Arabic pairs","score":0.595,"headline":false},{"track":"C · English pairs","score":0.875,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5353,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":15.0},"42":{"score":0.7369,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":35.3},"43":{"score":0.4368,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":15.0},"44":{"score":0.633,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.9},"45":{"score":0.1038,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":16.5},"48":{"score":0.294,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":35.8},"50":{"score":0.4063,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":43.3},"56":{"score":0.578,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":148.8,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5165,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":16.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6089,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":14.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7588,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":43.0,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6288,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":15.1,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5216,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":42.8,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5627,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":24.9,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":52.1,"balanced_skill":36.58,"breadth_skill":35.36},"frozen_scores":{"balanced_raw":39.68,"balanced_skill":25.67,"breadth_skill":21.91},"categories":[{"id":"knowledge","raw":0.4241,"skill":0.2559,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6142,"skill":0.4541,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5085,"skill":0.3299,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.6082,"skill":0.5316,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4499,"skill":0.2575,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8796,"skill":0.8374,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4453,"skill":0.3892,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7989,"skill":0.5257,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7854,"headline":false},{"track":"RouterBench-5shot","score":0.8123,"headline":false}]},"11":{"raw":0.7234,"skill":0.6,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.833,"skill":0.666,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5967,"skill":0.1933,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0934,"skill":0.0863,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5913,"skill":0.1825,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7598,"skill":0.6798,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4133,"skill":0.2177,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.5709,"skill":0.4832,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.6399,"headline":false},{"track":"GSM8K-10choice","score":0.5019,"headline":false}]},"31":{"raw":0.105,"skill":0.0252,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1818,"skill":0.142,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4238,"skill":0.2773,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4207,"skill":0.2548,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0829,"headline":false},{"track":"A · English","score":0.4207,"headline":true},{"track":"C · Arabic pairs","score":0.595,"headline":false},{"track":"C · English pairs","score":0.875,"headline":false}]},"41":{"raw":0.5353,"skill":0.3029,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4368,"skill":0.1065,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.633,"skill":0.266,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4063,"skill":0.1384,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.578,"skill":0.156,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5165,"skill":0.4562,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6089,"skill":0.4331,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7588,"skill":0.6784,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6288,"skill":0.536,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5216,"skill":0.1874,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5627,"skill":0.1254,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5372,"skill":0.2642,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3158,"skill":0.3067,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1038,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5636,"skill":0.3463,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8074,"skill":0.6148,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9399,"skill":0.9199,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0325,"skill":0.0325,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8604,"skill":0.7947,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7369,"skill":0.4881,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7472,"skill":0.7439,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8655,"skill":0.8647,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.2003,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.6043,"skill":0.5967,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.1625,"skill":0.1625,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"decider-chat-qwen3.6-27b","name":"Decider chat · Qwen3.6-27B","long_name":"Decider chat · Qwen3.6-27B","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":null,"base_model":"Qwen/Qwen3.6-27B","base_url":"https://huggingface.co/Qwen/Qwen3.6-27B","weights_repo":null,"served_checkpoint":"Qwen/Qwen3.6-27B"},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.6757,"conf":0.678,"ece":0.0342,"brier":0.4223,"over95":0.0009,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0172,"conf":0.0629,"acc":0.1578},{"lo":0.1,"w":0.0219,"conf":0.1531,"acc":0.1685},{"lo":0.2,"w":0.0318,"conf":0.2495,"acc":0.2453},{"lo":0.3,"w":0.0484,"conf":0.3558,"acc":0.3247},{"lo":0.4,"w":0.1034,"conf":0.453,"acc":0.3108},{"lo":0.5,"w":0.1294,"conf":0.5491,"acc":0.5344},{"lo":0.6,"w":0.129,"conf":0.6521,"acc":0.6622},{"lo":0.7,"w":0.1414,"conf":0.7518,"acc":0.7561},{"lo":0.8,"w":0.1754,"conf":0.8513,"acc":0.8858},{"lo":0.9,"w":0.202,"conf":0.9442,"acc":0.9741}],"areas":{"knowledge":{"ece":0.0685,"acc":0.5943,"conf":0.6349},"language":{"ece":0.0457,"acc":0.8102,"conf":0.7664},"retrieval":{"ece":0.0804,"acc":0.5994,"conf":0.6328},"tools":{"ece":0.0662,"acc":0.9059,"conf":0.8434},"arts":{"ece":0.0559,"acc":0.5517,"conf":0.5591}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":917.6,"p95":8775.9,"mean":2380.4},"results":{"1":{"score":0.9752,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":2184.5},"2":{"score":0.441,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":25742.0},"3":{"score":0.7972,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":16359.7},"4":{"score":0.751,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":2962.7},"5":{"score":0.8506,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":3618.0},"6":{"score":0.7991,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":3890.2},"9":{"score":0.4188,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":4307.3},"10":{"score":0.1759,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":3417.8},"11":{"score":0.74,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1807.8},"12":{"score":0.7459,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":328.7},"20":{"score":0.8764,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":403.2},"21":{"score":0.6084,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":235.1},"22":{"score":0.185,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":13042.0},"23":{"score":0.6332,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":1203.4},"24":{"score":0.8521,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":326.5},"25":{"score":0.5459,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":486.2},"26":{"score":0.9882,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":244.9},"27":{"score":0.9727,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":262.1},"28":{"score":0.8319,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":233.1},"29":{"score":0.9542,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":459.8},"30":{"score":0.5796,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":458.8},"31":{"score":0.1762,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":2052.9},"32":{"score":0.6263,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":2248.3},"33":{"score":0.3479,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":6146.3},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":24846.9},"36":{"score":0.1866,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":32122.8},"37":{"score":0.4767,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":962.1},"38":{"score":0.0075,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":16433.0},"39":{"score":0.8183,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":1595.5},"40":{"score":0.458,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":404.4,"tracks":[{"track":"A · Arabic","score":0.4615,"headline":false},{"track":"A · English","score":0.458,"headline":true},{"track":"C · Arabic pairs","score":0.8,"headline":false},{"track":"C · English pairs","score":0.975,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5839,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":580.9},"42":{"score":0.8203,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":1486.5},"43":{"score":0.6404,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":799.5},"44":{"score":0.6634,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":493.7},"45":{"score":0.1158,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":1227.1},"48":{"score":0.186,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":1654.3},"50":{"score":0.4087,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":1382.0},"56":{"score":0.6755,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":4196.8,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.6287,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":504.1,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7273,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":449.4,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.785,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":1445.0,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.7273,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":357.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7503,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":1443.1,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.68,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":917.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":58.98,"balanced_skill":46.08,"breadth_skill":45.06},"frozen_scores":{"balanced_raw":44.21,"balanced_skill":32.1,"breadth_skill":27.37},"categories":[{"id":"knowledge","raw":0.5051,"skill":0.3679,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.671,"skill":0.5491,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5423,"skill":0.3999,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.7027,"skill":0.6338,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.5277,"skill":0.3533,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9752,"skill":0.9665,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.441,"skill":0.3845,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7991,"skill":0.5262,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.786,"headline":false},{"track":"RouterBench-5shot","score":0.8122,"headline":false}]},"11":{"raw":0.74,"skill":0.624,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8784,"skill":0.7569,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6084,"skill":0.2169,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1812,"skill":0.1748,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6339,"skill":0.2678,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8592,"skill":0.8123,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.5459,"skill":0.3946,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.5796,"skill":0.4928,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.6391,"headline":false},{"track":"GSM8K-10choice","score":0.5201,"headline":false}]},"31":{"raw":0.1762,"skill":0.1028,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1866,"skill":0.147,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4767,"skill":0.3436,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.458,"skill":0.3027,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4615,"headline":false},{"track":"A · English","score":0.458,"headline":true},{"track":"C · Arabic pairs","score":0.8,"headline":false},{"track":"C · English pairs","score":0.975,"headline":false}]},"41":{"raw":0.5839,"skill":0.3759,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.6404,"skill":0.4294,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6634,"skill":0.3268,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4087,"skill":0.1418,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6755,"skill":0.351,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.6287,"skill":0.5824,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.7273,"skill":0.6047,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.785,"skill":0.7133,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.7273,"skill":0.6591,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7503,"skill":0.5758,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.68,"skill":0.36,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6263,"skill":0.4059,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3479,"skill":0.3392,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1158,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.7459,"skill":0.6194,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8319,"skill":0.6638,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9542,"skill":0.9389,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0075,"skill":0.0075,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8183,"skill":0.7328,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8203,"skill":0.6504,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.751,"skill":0.7478,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8506,"skill":0.8497,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.1759,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.7972,"skill":0.7933,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.4188,"skill":0.4188,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.256,"skill":0.256,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"decision-eos-0.8b","name":"Decision 1.0 Eos","long_name":"Decision 1.0 Eos 0.8B","variant":null,"meta":{"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_estimated":false,"params_from_name":false,"trained_bytes":1509034472,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Eos-0.8B","code_url":null,"base_model":"Qwen/Qwen3.5-0.8B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-0.8B-Base","weights_repo":"llm-semantic-router/Decision-1.0-Eos-0.8B","served_checkpoint":null},"completed":120615,"counts":{"ok":120612,"unsupported":3},"lifted":false,"calibration":{"acc":0.4653,"conf":0.5478,"ece":0.0913,"brier":0.6369,"over95":0.005,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0637,"conf":0.0421,"acc":0.0519},{"lo":0.1,"w":0.0316,"conf":0.1572,"acc":0.2009},{"lo":0.2,"w":0.0447,"conf":0.2569,"acc":0.2437},{"lo":0.3,"w":0.1353,"conf":0.3559,"acc":0.2751},{"lo":0.4,"w":0.1082,"conf":0.4492,"acc":0.4715},{"lo":0.5,"w":0.1967,"conf":0.5477,"acc":0.4937},{"lo":0.6,"w":0.1451,"conf":0.6472,"acc":0.5262},{"lo":0.7,"w":0.1051,"conf":0.7474,"acc":0.5391},{"lo":0.8,"w":0.0922,"conf":0.8494,"acc":0.6787},{"lo":0.9,"w":0.0774,"conf":0.9473,"acc":0.8244}],"areas":{"knowledge":{"ece":0.1136,"acc":0.3878,"conf":0.4958},"language":{"ece":0.1565,"acc":0.4842,"conf":0.6394},"retrieval":{"ece":0.0716,"acc":0.6163,"conf":0.561},"tools":{"ece":0.0683,"acc":0.6418,"conf":0.6053},"arts":{"ece":0.0799,"acc":0.3908,"conf":0.4625}}},"gaps":{"share":0.0,"reasons":[{"reason":"prompt longer than its 16,384-token context window","share":0.0}]},"latency":{"median":36.3,"p95":101.2,"mean":50.5},"results":{"1":{"score":0.5885,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":38.3},"2":{"score":0.3812,"jev":0.4475,"metric":"nDCG@10","requests":1000,"answered":998,"unsupported":2,"errors":0,"median_ms":211.9},"3":{"score":0.3661,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":103.1},"4":{"score":0.6479,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":39.0},"5":{"score":0.6651,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":48.9},"6":{"score":0.7255,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":58.1},"9":{"score":0.0187,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":233.6},"10":{"score":0.6296,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":37.1},"11":{"score":0.436,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":507.3},"12":{"score":0.384,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":35.9},"20":{"score":0.4934,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":35.8},"21":{"score":0.5217,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":35.0},"22":{"score":0.0215,"jev":0.1811,"metric":"accuracy","requests":2000,"answered":1999,"unsupported":1,"errors":0,"median_ms":104.0},"23":{"score":0.546,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":36.2},"24":{"score":0.4345,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":35.5},"25":{"score":0.2449,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":35.5},"26":{"score":0.8106,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":35.3},"27":{"score":0.6382,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":35.5},"28":{"score":0.5185,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":35.1},"29":{"score":0.4663,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":35.9},"30":{"score":0.2494,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":35.9},"31":{"score":0.0844,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":37.1},"32":{"score":0.5199,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":37.3},"33":{"score":0.0121,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":49.0},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":35.8},"36":{"score":0.1023,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":329.9},"37":{"score":0.3455,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":36.0},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":308.0},"39":{"score":0.6282,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":36.3},"40":{"score":0.2388,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":35.8,"tracks":[{"track":"A · Arabic","score":0.2578,"headline":false},{"track":"A · English","score":0.2388,"headline":true},{"track":"C · Arabic pairs","score":0.345,"headline":false},{"track":"C · English pairs","score":0.325,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4314,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":35.7},"42":{"score":0.6056,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":36.2},"43":{"score":0.4123,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":35.4},"44":{"score":0.5184,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":35.6},"45":{"score":0.1497,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":36.3},"48":{"score":0.2365,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":36.8},"50":{"score":0.386,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":37.2},"56":{"score":0.5,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":73.1,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1992,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":35.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.4494,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":35.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4496,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":36.1,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.3731,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":34.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.0241,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":35.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5405,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":34.9,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":38.22,"balanced_skill":17.49,"breadth_skill":15.83},"frozen_scores":{"balanced_raw":28.89,"balanced_skill":9.71,"breadth_skill":8.38},"categories":[{"id":"knowledge","raw":0.284,"skill":0.0735,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.3733,"skill":0.1441,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4901,"skill":0.287,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.4215,"skill":0.2946,"coverage":0.9997,"pending":0,"n":6},{"id":"arts","raw":0.342,"skill":0.0753,"coverage":0.9999,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.5885,"skill":0.4446,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3805,"skill":0.3178,"coverage":0.998,"random":0.0918,"tracks":[]},"6":{"raw":0.7256,"skill":0.3663,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6582,"headline":false},{"track":"RouterBench-5shot","score":0.7929,"headline":false}]},"11":{"raw":0.436,"skill":0.1844,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.4925,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5217,"skill":0.0434,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0214,"skill":0.0138,"coverage":0.9995,"random":0.0078,"tracks":[]},"23":{"raw":0.5452,"skill":0.0905,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.4437,"skill":0.2582,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2449,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2494,"skill":0.0904,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.3192,"headline":false},{"track":"GSM8K-10choice","score":0.1797,"headline":false}]},"31":{"raw":0.0844,"skill":0.0028,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1023,"skill":0.0586,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3455,"skill":0.179,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2388,"skill":0.0207,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2578,"headline":false},{"track":"A · English","score":0.2388,"headline":true},{"track":"C · Arabic pairs","score":0.345,"headline":false},{"track":"C · English pairs","score":0.325,"headline":false}]},"41":{"raw":0.4314,"skill":0.1471,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4123,"skill":0.0675,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5184,"skill":0.0368,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.386,"skill":0.1089,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1992,"skill":0.0993,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.4494,"skill":0.2019,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4496,"skill":0.2661,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.3731,"skill":0.2164,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.0241,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5405,"skill":0.081,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5199,"skill":0.2367,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0121,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1497,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.384,"skill":0.0773,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5185,"skill":0.037,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.4663,"skill":0.2884,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6282,"skill":0.4532,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.6056,"skill":0.2327,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.6479,"skill":0.6434,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.6651,"skill":0.6631,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6296,"skill":0.3837,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.3661,"skill":0.3539,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0187,"skill":0.0187,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.054,"skill":0.054,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7599,"pending":0.24},{"engine":"decision-kai-7185f514","name":"Decision 1.0 Kai","long_name":"Decision 1.0 Kai","variant":null,"meta":{"kind":"head / adapter","technique":"encoder","technique_name":"encoder","served_params":307797035,"params_estimated":true,"params_from_name":false,"trained_bytes":2287687932,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Kai","code_url":null,"base_model":"jhu-clsp/mmBERT-base","base_url":"https://huggingface.co/jhu-clsp/mmBERT-base","weights_repo":"llm-semantic-router/Decision-1.0-Kai","served_checkpoint":null},"completed":120615,"counts":{"ok":120627,"unsupported":327},"lifted":true,"calibration":{"acc":0.3773,"conf":0.5342,"ece":0.1837,"brier":0.7539,"over95":0.0036,"n":35091,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0476,"conf":0.0757,"acc":0.0629},{"lo":0.1,"w":0.0364,"conf":0.1449,"acc":0.1564},{"lo":0.2,"w":0.0922,"conf":0.2663,"acc":0.4068},{"lo":0.3,"w":0.0857,"conf":0.3503,"acc":0.2687},{"lo":0.4,"w":0.1317,"conf":0.4521,"acc":0.3118},{"lo":0.5,"w":0.221,"conf":0.5472,"acc":0.4418},{"lo":0.6,"w":0.1469,"conf":0.6454,"acc":0.4314},{"lo":0.7,"w":0.1029,"conf":0.7507,"acc":0.392},{"lo":0.8,"w":0.0954,"conf":0.8473,"acc":0.4744},{"lo":0.9,"w":0.0401,"conf":0.9317,"acc":0.5086}],"areas":{"knowledge":{"ece":0.2338,"acc":0.3687,"conf":0.4969},"language":{"ece":0.1997,"acc":0.4252,"conf":0.6249},"retrieval":{"ece":0.1161,"acc":0.4696,"conf":0.4952},"tools":{"ece":0.3039,"acc":0.2193,"conf":0.5155},"arts":{"ece":0.1702,"acc":0.333,"conf":0.5031}}},"gaps":{"share":0.0027,"reasons":[{"reason":"ran out of GPU memory on the longest prompts","share":0.0027},{"reason":"prompt longer than its 8,192-token context window","share":0.0}]},"latency":{"median":30.2,"p95":118.3,"mean":44.9},"results":{"1":{"score":0.3884,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":32.7},"2":{"score":0.2245,"jev":0.4493,"metric":"nDCG@10","requests":1000,"answered":989,"unsupported":11,"errors":0,"median_ms":168.0},"3":{"score":0.1594,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":241.9},"4":{"score":0.4073,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":31.7},"5":{"score":0.0569,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":33.8},"6":{"score":0.6674,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":78.3},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":158.6},"10":{"score":0.4846,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":33.0},"11":{"score":0.3662,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":430.8},"12":{"score":0.2428,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":29.1},"20":{"score":0.3986,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":29.2},"21":{"score":0.4574,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":28.9},"22":{"score":0.0015,"jev":0.1812,"metric":"accuracy","requests":2000,"answered":1998,"unsupported":2,"errors":0,"median_ms":159.9},"23":{"score":0.4934,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":29.2},"24":{"score":0.3106,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":30.1},"25":{"score":0.2296,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":30.0},"26":{"score":0.487,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":29.9},"27":{"score":0.3183,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":29.9},"28":{"score":0.4917,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":29.8},"29":{"score":0.3404,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":30.0},"30":{"score":0.2915,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":29.5},"31":{"score":0.0684,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":30.8},"32":{"score":0.4521,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":32.4},"33":{"score":0.0024,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":49.9},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":29.8},"36":{"score":0.0554,"jev":0.1908,"metric":"nDCG@10","requests":550,"answered":547,"unsupported":3,"errors":0,"median_ms":280.0},"37":{"score":0.2045,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":30.0},"38":{"score":0.005,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":153.9},"39":{"score":0.3695,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":30.0},"40":{"score":0.0087,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":29.3,"tracks":[{"track":"A · Arabic","score":0.0394,"headline":false},{"track":"A · English","score":0.0087,"headline":true},{"track":"C · Arabic pairs","score":0.29,"headline":false},{"track":"C · English pairs","score":0.21,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.2081,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":29.5},"42":{"score":0.2976,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":30.1},"43":{"score":0.393,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":29.5},"44":{"score":0.4978,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":29.5},"45":{"score":0.122,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":29.9},"48":{"score":0.4344,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":30.9},"50":{"score":0.3837,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":31.4},"56":{"score":0.499,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":68.3,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1385,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":29.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3497,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":29.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3757,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":30.5,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2727,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":29.3,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5177,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":31.2,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5212,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":29.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":29.0,"balanced_skill":7.03,"breadth_skill":6.59},"frozen_scores":{"balanced_raw":23.21,"balanced_skill":4.1,"breadth_skill":3.68},"categories":[{"id":"knowledge","raw":0.2545,"skill":0.0396,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.2848,"skill":0.0462,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3184,"skill":0.0924,"coverage":0.9992,"pending":0,"n":7},{"id":"tools","raw":0.3022,"skill":0.1413,"coverage":0.9982,"pending":0,"n":6},{"id":"arts","raw":0.2904,"skill":0.0317,"coverage":0.9999,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3884,"skill":0.1744,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.222,"skill":0.1433,"coverage":0.989,"random":0.0918,"tracks":[]},"6":{"raw":0.6675,"skill":0.2195,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6322,"headline":false},{"track":"RouterBench-5shot","score":0.7028,"headline":false}]},"11":{"raw":0.3662,"skill":0.0834,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.4047,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4574,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0019,"skill":0.0,"coverage":0.999,"random":0.0078,"tracks":[]},"23":{"raw":0.5127,"skill":0.0254,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.312,"skill":0.0826,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2296,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2915,"skill":0.142,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.3632,"headline":false},{"track":"GSM8K-10choice","score":0.2199,"headline":false}]},"31":{"raw":0.0684,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0551,"skill":0.0091,"coverage":0.9945,"random":0.0464,"tracks":[]},"37":{"raw":0.2045,"skill":0.0022,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.0087,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0394,"headline":false},{"track":"A · English","score":0.0087,"headline":true},{"track":"C · Arabic pairs","score":0.29,"headline":false},{"track":"C · English pairs","score":0.21,"headline":false}]},"41":{"raw":0.2081,"skill":0.0,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.393,"skill":0.0369,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4978,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3837,"skill":0.1055,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.499,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1385,"skill":0.031,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3497,"skill":0.0574,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3757,"skill":0.1676,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2727,"skill":0.0909,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5177,"skill":0.1807,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5212,"skill":0.0424,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4521,"skill":0.1289,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0024,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1218,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.2428,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.4917,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3404,"skill":0.1205,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.005,"skill":0.005,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.3695,"skill":0.0727,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.2976,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.4073,"skill":0.3997,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0569,"skill":0.0512,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4846,"skill":0.1424,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.1594,"skill":0.1432,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7593,"pending":0.24},{"engine":"decision-lex","name":"Decision 1.0 Lex","long_name":"Decision 1.0 Lex","variant":null,"meta":{"kind":"head / adapter","technique":"encoder","technique_name":"encoder","served_params":307797035,"params_estimated":true,"params_from_name":false,"trained_bytes":2287687932,"model_url":"https://huggingface.co/llm-semantic-router/decision-1.0-lex","code_url":null,"base_model":"jhu-clsp/mmBERT-base","base_url":"https://huggingface.co/jhu-clsp/mmBERT-base","weights_repo":"llm-semantic-router/decision-1.0-lex","served_checkpoint":null},"completed":120615,"counts":{"ok":120723,"unsupported":334},"lifted":true,"calibration":{"acc":0.3204,"conf":0.4657,"ece":0.1704,"brier":0.7615,"over95":0.0005,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.1347,"conf":0.0454,"acc":0.1321},{"lo":0.1,"w":0.0279,"conf":0.1415,"acc":0.1728},{"lo":0.2,"w":0.0769,"conf":0.2529,"acc":0.1184},{"lo":0.3,"w":0.1253,"conf":0.3543,"acc":0.2983},{"lo":0.4,"w":0.1199,"conf":0.4453,"acc":0.3381},{"lo":0.5,"w":0.1999,"conf":0.5478,"acc":0.4438},{"lo":0.6,"w":0.1442,"conf":0.6462,"acc":0.4167},{"lo":0.7,"w":0.1011,"conf":0.7466,"acc":0.4101},{"lo":0.8,"w":0.0554,"conf":0.8426,"acc":0.3539},{"lo":0.9,"w":0.0147,"conf":0.9203,"acc":0.0621}],"areas":{"knowledge":{"ece":0.1432,"acc":0.2717,"conf":0.4148},"language":{"ece":0.2495,"acc":0.3387,"conf":0.5875},"retrieval":{"ece":0.1707,"acc":0.39,"conf":0.3303},"tools":{"ece":0.2023,"acc":0.2976,"conf":0.4796},"arts":{"ece":0.1076,"acc":0.3625,"conf":0.4685}}},"gaps":{"share":0.0028,"reasons":[{"reason":"ran out of GPU memory on the longest prompts","share":0.0027},{"reason":"prompt longer than its 8,192-token context window","share":0.0001}]},"latency":{"median":24.6,"p95":116.3,"mean":41.1},"results":{"1":{"score":0.3872,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":28.2},"2":{"score":0.156,"jev":0.4493,"metric":"nDCG@10","requests":1000,"answered":989,"unsupported":11,"errors":0,"median_ms":160.6},"3":{"score":0.1398,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":244.8},"4":{"score":0.2356,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":26.0},"5":{"score":0.1121,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":27.5},"6":{"score":0.593,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":79.0},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":150.1},"10":{"score":0.4106,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":32.8},"11":{"score":0.2011,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":431.6},"12":{"score":0.2256,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":24.0},"20":{"score":0.4798,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.0},"21":{"score":0.4871,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":29.1},"22":{"score":0.002,"jev":0.1812,"metric":"accuracy","requests":2000,"answered":1998,"unsupported":2,"errors":0,"median_ms":160.0},"23":{"score":0.512,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.1},"24":{"score":0.3017,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":23.9},"25":{"score":0.2602,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":29.5},"26":{"score":0.476,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":23.8},"27":{"score":0.3038,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":23.8},"28":{"score":0.5067,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":23.6},"29":{"score":0.3331,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":24.3},"30":{"score":0.2343,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":24.2},"31":{"score":0.0652,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":25.3},"32":{"score":0.3896,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":27.6},"33":{"score":0.0006,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":45.4},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":24.2},"36":{"score":0.0502,"jev":0.1908,"metric":"nDCG@10","requests":550,"answered":547,"unsupported":3,"errors":0,"median_ms":278.7},"37":{"score":0.1929,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.5},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":134.7},"39":{"score":0.2178,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":24.4},"40":{"score":0.1896,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":24.1,"tracks":[{"track":"A · Arabic","score":0.2054,"headline":false},{"track":"A · English","score":0.1896,"headline":true},{"track":"C · Arabic pairs","score":0.485,"headline":false},{"track":"C · English pairs","score":0.335,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4039,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":24.2},"42":{"score":0.3026,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":24.6},"43":{"score":0.3684,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":24.0},"44":{"score":0.4838,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.1},"45":{"score":0.148,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":24.5},"48":{"score":0.4767,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":25.3},"50":{"score":0.3246,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":26.2},"56":{"score":0.4975,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":130.2,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.145,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":41.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3332,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":34.4,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3445,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":43.3,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2955,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":29.3,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.4913,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":63.9,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.501,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":59.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":27.64,"balanced_skill":4.31,"breadth_skill":4.11},"frozen_scores":{"balanced_raw":23.14,"balanced_skill":2.31,"breadth_skill":2.16},"categories":[{"id":"knowledge","raw":0.2428,"skill":0.0186,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.2872,"skill":0.0366,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2857,"skill":0.0511,"coverage":0.9992,"pending":0,"n":7},{"id":"tools","raw":0.2698,"skill":0.0895,"coverage":0.9982,"pending":0,"n":6},{"id":"arts","raw":0.2965,"skill":0.0199,"coverage":0.9999,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3872,"skill":0.1728,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.1543,"skill":0.0688,"coverage":0.989,"random":0.0918,"tracks":[]},"6":{"raw":0.593,"skill":0.0459,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.5453,"headline":false},{"track":"RouterBench-5shot","score":0.6407,"headline":false}]},"11":{"raw":0.2011,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.48,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4871,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0012,"skill":0.0,"coverage":0.999,"random":0.0078,"tracks":[]},"23":{"raw":0.4868,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3079,"skill":0.0772,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2602,"skill":0.0136,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2343,"skill":0.071,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2964,"headline":false},{"track":"GSM8K-10choice","score":0.1721,"headline":false}]},"31":{"raw":0.0652,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0499,"skill":0.0036,"coverage":0.9945,"random":0.0464,"tracks":[]},"37":{"raw":0.1929,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.1896,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2054,"headline":false},{"track":"A · English","score":0.1896,"headline":true},{"track":"C · Arabic pairs","score":0.485,"headline":false},{"track":"C · English pairs","score":0.335,"headline":false}]},"41":{"raw":0.4039,"skill":0.1059,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3684,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4838,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3246,"skill":0.0197,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.4975,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.145,"skill":0.0384,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3332,"skill":0.0335,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3445,"skill":0.126,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2955,"skill":0.1194,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.4913,"skill":0.1359,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.501,"skill":0.002,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3896,"skill":0.0296,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0006,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1477,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.2256,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5067,"skill":0.0134,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3331,"skill":0.1108,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.2178,"skill":0.0,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.3026,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.2356,"skill":0.2258,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.1121,"skill":0.1067,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4106,"skill":0.0193,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.1398,"skill":0.1232,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7593,"pending":0.24},{"engine":"decision-lux-9b","name":"Decision 1.0 Lux","long_name":"Decision 1.0 Lux 9B","variant":null,"meta":{"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_estimated":false,"params_from_name":false,"trained_bytes":15890261552,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Lux-9B","code_url":null,"base_model":"Qwen/Qwen3.5-9B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-9B-Base","weights_repo":"llm-semantic-router/Decision-1.0-Lux-9B","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.6435,"conf":0.7294,"ece":0.0988,"brier":0.4746,"over95":0.016,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.044,"conf":0.0528,"acc":0.171},{"lo":0.1,"w":0.0156,"conf":0.1402,"acc":0.1256},{"lo":0.2,"w":0.016,"conf":0.253,"acc":0.3295},{"lo":0.3,"w":0.039,"conf":0.3551,"acc":0.2816},{"lo":0.4,"w":0.0516,"conf":0.4524,"acc":0.3299},{"lo":0.5,"w":0.1143,"conf":0.5502,"acc":0.4866},{"lo":0.6,"w":0.1057,"conf":0.6513,"acc":0.5688},{"lo":0.7,"w":0.1079,"conf":0.7501,"acc":0.6423},{"lo":0.8,"w":0.1495,"conf":0.8483,"acc":0.5983},{"lo":0.9,"w":0.3564,"conf":0.9655,"acc":0.9153}],"areas":{"knowledge":{"ece":0.1434,"acc":0.5478,"conf":0.6856},"language":{"ece":0.0727,"acc":0.7606,"conf":0.8315},"retrieval":{"ece":0.0754,"acc":0.6747,"conf":0.7411},"tools":{"ece":0.0283,"acc":0.8794,"conf":0.8951},"arts":{"ece":0.1166,"acc":0.4957,"conf":0.5416}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":46.7,"p95":446.9,"mean":131.6},"results":{"1":{"score":0.9427,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":80.0},"2":{"score":0.4362,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1028.0},"3":{"score":0.7402,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":461.7},"4":{"score":0.8065,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":108.3},"5":{"score":0.8842,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":197.2},"6":{"score":0.7836,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":280.0},"9":{"score":0.25,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1086.7},"10":{"score":0.4604,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":69.6},"11":{"score":0.7166,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":2180.8},"12":{"score":0.6641,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":46.1},"20":{"score":0.686,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":46.2},"21":{"score":0.5613,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":46.2},"22":{"score":0.1945,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":462.5},"23":{"score":0.6168,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":46.8},"24":{"score":0.771,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":45.9},"25":{"score":0.4337,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":46.0},"26":{"score":0.984,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":45.8},"27":{"score":0.9488,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":45.7},"28":{"score":0.7782,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":46.0},"29":{"score":0.8867,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":46.0},"30":{"score":0.5091,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":46.0},"31":{"score":0.136,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":75.5},"32":{"score":0.5838,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":70.2},"33":{"score":0.3303,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":206.8},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":46.8},"36":{"score":0.1657,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1702.3},"37":{"score":0.4602,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":46.7},"38":{"score":0.0125,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":738.8},"39":{"score":0.8181,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":46.4},"40":{"score":0.4282,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":46.4,"tracks":[{"track":"A · Arabic","score":0.4026,"headline":false},{"track":"A · English","score":0.4282,"headline":true},{"track":"C · Arabic pairs","score":0.585,"headline":false},{"track":"C · English pairs","score":0.86,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4393,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":46.1},"42":{"score":0.8076,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":47.1},"43":{"score":0.6228,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":45.9},"44":{"score":0.645,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":45.8},"45":{"score":0.0958,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":46.5},"48":{"score":0.1907,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":54.4},"50":{"score":0.4141,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":64.7},"56":{"score":0.592,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":173.6,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5441,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":46.0,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6755,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":45.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4513,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":53.6,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6231,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":45.8,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.6599,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":56.7,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6825,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":46.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":54.7,"balanced_skill":38.98,"breadth_skill":38.1},"frozen_scores":{"balanced_raw":40.47,"balanced_skill":26.13,"breadth_skill":22.27},"categories":[{"id":"knowledge","raw":0.4576,"skill":0.3085,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6211,"skill":0.4709,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5788,"skill":0.3981,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.6007,"skill":0.5072,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4767,"skill":0.2643,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9427,"skill":0.9227,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4362,"skill":0.3792,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7836,"skill":0.4876,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.777,"headline":false},{"track":"RouterBench-5shot","score":0.7902,"headline":false}]},"11":{"raw":0.7166,"skill":0.5902,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.6953,"skill":0.3906,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5613,"skill":0.1225,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.184,"skill":0.1777,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6217,"skill":0.2434,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7806,"skill":0.7075,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4337,"skill":0.2449,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.5091,"skill":0.4134,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.63,"headline":false},{"track":"GSM8K-10choice","score":0.3882,"headline":false}]},"31":{"raw":0.136,"skill":0.059,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1657,"skill":0.125,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4602,"skill":0.3229,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4282,"skill":0.2644,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4026,"headline":false},{"track":"A · English","score":0.4282,"headline":true},{"track":"C · Arabic pairs","score":0.585,"headline":false},{"track":"C · English pairs","score":0.86,"headline":false}]},"41":{"raw":0.4393,"skill":0.1589,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.6228,"skill":0.4015,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.645,"skill":0.29,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4141,"skill":0.1496,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.592,"skill":0.184,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5441,"skill":0.4872,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6755,"skill":0.5296,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4513,"skill":0.2684,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6231,"skill":0.5289,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.6599,"skill":0.4223,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6825,"skill":0.365,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5838,"skill":0.3383,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3303,"skill":0.3214,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.0958,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6641,"skill":0.4969,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7782,"skill":0.5564,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8867,"skill":0.8489,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0125,"skill":0.0125,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8181,"skill":0.7325,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8076,"skill":0.6257,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.8065,"skill":0.804,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8842,"skill":0.8835,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4604,"skill":0.1022,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.7402,"skill":0.7352,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.25,"skill":0.25,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.2372,"skill":0.2372,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"decision-nox-0bb83350","name":"Decision 1.0 Nox","long_name":"Decision 1.0 Nox","variant":null,"meta":{"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":8422078976,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Nox","code_url":null,"base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"llm-semantic-router/Decision-1.0-Nox","served_checkpoint":null},"completed":120615,"counts":{"ok":120888},"lifted":false,"calibration":{"acc":0.5955,"conf":0.7538,"ece":0.1583,"brier":0.5539,"over95":0.0474,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0146,"conf":0.0815,"acc":0.0522},{"lo":0.1,"w":0.0328,"conf":0.1409,"acc":0.0761},{"lo":0.2,"w":0.0173,"conf":0.2536,"acc":0.1859},{"lo":0.3,"w":0.0377,"conf":0.3575,"acc":0.2254},{"lo":0.4,"w":0.0571,"conf":0.4526,"acc":0.3511},{"lo":0.5,"w":0.1007,"conf":0.5463,"acc":0.4772},{"lo":0.6,"w":0.0929,"conf":0.6505,"acc":0.534},{"lo":0.7,"w":0.1221,"conf":0.7497,"acc":0.4383},{"lo":0.8,"w":0.1121,"conf":0.8519,"acc":0.6398},{"lo":0.9,"w":0.4126,"conf":0.9737,"acc":0.8181}],"areas":{"knowledge":{"ece":0.2024,"acc":0.4917,"conf":0.6941},"language":{"ece":0.1125,"acc":0.6873,"conf":0.7963},"retrieval":{"ece":0.1394,"acc":0.7065,"conf":0.8436},"tools":{"ece":0.0827,"acc":0.872,"conf":0.932},"arts":{"ece":0.217,"acc":0.4249,"conf":0.6419}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":52.1,"p95":314.9,"mean":143.3},"results":{"1":{"score":0.8583,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":60.7},"2":{"score":0.425,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":677.5},"3":{"score":0.7756,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":319.9},"4":{"score":0.8331,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":74.5},"5":{"score":0.853,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":135.3},"6":{"score":0.7346,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":183.3},"9":{"score":0.05,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":725.6},"10":{"score":0.6422,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":53.9},"11":{"score":0.6408,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1496.4},"12":{"score":0.5978,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":51.3},"20":{"score":0.5136,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.0},"21":{"score":0.5084,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":50.8},"22":{"score":0.131,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":320.3},"23":{"score":0.592,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.7},"24":{"score":0.6054,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":50.7},"25":{"score":0.3112,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":51.0},"26":{"score":0.9541,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":50.6},"27":{"score":0.837,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":50.8},"28":{"score":0.6504,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":50.9},"29":{"score":0.7907,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":51.3},"30":{"score":0.4405,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":51.2},"31":{"score":0.074,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":52.9},"32":{"score":0.6037,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":53.2},"33":{"score":0.2788,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":139.8},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":51.9},"36":{"score":0.1384,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1123.3},"37":{"score":0.3851,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":52.4},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":483.6},"39":{"score":0.8133,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":53.0},"40":{"score":0.378,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":51.7,"tracks":[{"track":"A · Arabic","score":0.3798,"headline":false},{"track":"A · English","score":0.378,"headline":true},{"track":"C · Arabic pairs","score":0.38,"headline":false},{"track":"C · English pairs","score":0.54,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4728,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":51.5},"42":{"score":0.7517,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":52.6},"43":{"score":0.4737,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":51.2},"44":{"score":0.6338,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.2},"45":{"score":0.1018,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":52.0},"48":{"score":0.209,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":53.3},"50":{"score":0.3699,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":53.0},"56":{"score":0.5415,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":225.7,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.3669,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":50.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5784,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":53.3,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3801,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":79.5,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.4981,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":51.1,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5094,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":84.5,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6405,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":58.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":49.18,"balanced_skill":31.06,"breadth_skill":29.43},"frozen_scores":{"balanced_raw":35.01,"balanced_skill":17.96,"breadth_skill":15.05},"categories":[{"id":"knowledge","raw":0.3863,"skill":0.2161,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.5605,"skill":0.3718,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5763,"skill":0.3967,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5373,"skill":0.4251,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3986,"skill":0.1433,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8583,"skill":0.8087,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.425,"skill":0.3668,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7346,"skill":0.3805,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6939,"headline":false},{"track":"RouterBench-5shot","score":0.7753,"headline":false}]},"11":{"raw":0.6408,"skill":0.4806,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5184,"skill":0.0369,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5084,"skill":0.0167,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1292,"skill":0.1224,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6025,"skill":0.2051,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.6157,"skill":0.4876,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3112,"skill":0.0816,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4405,"skill":0.3306,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5701,"headline":false},{"track":"GSM8K-10choice","score":0.3108,"headline":false}]},"31":{"raw":0.074,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1384,"skill":0.0964,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3851,"skill":0.2287,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.378,"skill":0.1998,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3798,"headline":false},{"track":"A · English","score":0.378,"headline":true},{"track":"C · Arabic pairs","score":0.38,"headline":false},{"track":"C · English pairs","score":0.54,"headline":false}]},"41":{"raw":0.4728,"skill":0.2093,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4737,"skill":0.165,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6338,"skill":0.2676,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3699,"skill":0.0856,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5415,"skill":0.083,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.3669,"skill":0.2879,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5784,"skill":0.3889,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3801,"skill":0.1735,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.4981,"skill":0.3726,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5094,"skill":0.1666,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6405,"skill":0.281,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6037,"skill":0.37,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2788,"skill":0.2692,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1018,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5978,"skill":0.3975,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6504,"skill":0.3008,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.7907,"skill":0.7209,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8133,"skill":0.7254,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7517,"skill":0.5169,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.8331,"skill":0.831,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.853,"skill":0.8521,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6422,"skill":0.4047,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.7756,"skill":0.7713,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.05,"skill":0.05,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.164,"skill":0.164,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"decision-sol-0665a411","name":"Decision 1.0 Sol","long_name":"Decision 1.0 Sol","variant":null,"meta":{"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":2274069824,"params_estimated":false,"params_from_name":false,"trained_bytes":3772109600,"model_url":"https://huggingface.co/llm-semantic-router/Decision-1.0-Sol","code_url":null,"base_model":"Qwen/Qwen3.5-2B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-2B-Base","weights_repo":"llm-semantic-router/Decision-1.0-Sol","served_checkpoint":null},"completed":120615,"counts":{"ok":120943},"lifted":false,"calibration":{"acc":0.5275,"conf":0.6672,"ece":0.1409,"brier":0.6236,"over95":0.0521,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0497,"conf":0.0521,"acc":0.0309},{"lo":0.1,"w":0.0125,"conf":0.1533,"acc":0.1019},{"lo":0.2,"w":0.0238,"conf":0.2609,"acc":0.2862},{"lo":0.3,"w":0.0631,"conf":0.3565,"acc":0.2785},{"lo":0.4,"w":0.084,"conf":0.4484,"acc":0.3589},{"lo":0.5,"w":0.1592,"conf":0.5493,"acc":0.4576},{"lo":0.6,"w":0.1338,"conf":0.6483,"acc":0.5232},{"lo":0.7,"w":0.108,"conf":0.7505,"acc":0.6188},{"lo":0.8,"w":0.1104,"conf":0.8513,"acc":0.671},{"lo":0.9,"w":0.2556,"conf":0.9667,"acc":0.7291}],"areas":{"knowledge":{"ece":0.1848,"acc":0.4351,"conf":0.6194},"language":{"ece":0.1263,"acc":0.6165,"conf":0.7426},"retrieval":{"ece":0.1479,"acc":0.5832,"conf":0.729},"tools":{"ece":0.0632,"acc":0.7562,"conf":0.7977},"arts":{"ece":0.1113,"acc":0.4071,"conf":0.5183}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":39.6,"p95":135.7,"mean":81.1},"results":{"1":{"score":0.8855,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":40.1},"2":{"score":0.373,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":287.3},"3":{"score":0.6201,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":137.8},"4":{"score":0.7705,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":40.0},"5":{"score":0.5523,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":63.9},"6":{"score":0.6963,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":79.2},"9":{"score":0.0125,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":304.0},"10":{"score":0.6149,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":40.6},"11":{"score":0.5014,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":662.5},"12":{"score":0.4915,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":38.7},"20":{"score":0.5144,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.7},"21":{"score":0.5578,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":38.2},"22":{"score":0.0295,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":135.1},"23":{"score":0.557,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":39.1},"24":{"score":0.484,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":38.2},"25":{"score":0.2755,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":38.6},"26":{"score":0.8948,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":38.1},"27":{"score":0.6877,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":38.2},"28":{"score":0.5864,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":38.0},"29":{"score":0.5838,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":38.5},"30":{"score":0.3256,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":38.7},"31":{"score":0.0648,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":39.6},"32":{"score":0.5532,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":40.5},"33":{"score":0.1261,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":61.4},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":38.5},"36":{"score":0.1111,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":458.0},"37":{"score":0.3222,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.8},"38":{"score":0.0075,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":314.6},"39":{"score":0.6102,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":38.6},"40":{"score":0.2491,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":38.2,"tracks":[{"track":"A · Arabic","score":0.2974,"headline":false},{"track":"A · English","score":0.2491,"headline":true},{"track":"C · Arabic pairs","score":0.285,"headline":false},{"track":"C · English pairs","score":0.29,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4999,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":41.4},"42":{"score":0.6838,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":52.1},"43":{"score":0.3754,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":41.1},"44":{"score":0.5546,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":41.0},"45":{"score":0.1257,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":44.7},"48":{"score":0.3362,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":58.1},"50":{"score":0.3168,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":67.1},"56":{"score":0.5015,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":163.0,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.2472,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":41.5,"edition_added":"v0.2","jev":0.827},"58":{"score":0.4837,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":41.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4409,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":61.1,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.4432,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":41.0,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.6139,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":63.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.603,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":47.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":42.85,"balanced_skill":22.9,"breadth_skill":20.87},"frozen_scores":{"balanced_raw":30.92,"balanced_skill":12.68,"breadth_skill":10.63},"categories":[{"id":"knowledge","raw":0.3132,"skill":0.1153,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.4828,"skill":0.2582,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4965,"skill":0.3005,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5047,"skill":0.3868,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3451,"skill":0.0842,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8855,"skill":0.8454,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.373,"skill":0.3096,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.6963,"skill":0.286,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6683,"headline":false},{"track":"RouterBench-5shot","score":0.7243,"headline":false}]},"11":{"raw":0.5014,"skill":0.2789,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.518,"skill":0.0359,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5578,"skill":0.1157,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0275,"skill":0.0199,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5527,"skill":0.1053,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.4903,"skill":0.3205,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2755,"skill":0.034,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3256,"skill":0.1913,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4655,"headline":false},{"track":"GSM8K-10choice","score":0.1857,"headline":false}]},"31":{"raw":0.0648,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1111,"skill":0.0679,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3222,"skill":0.1499,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2491,"skill":0.034,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2974,"headline":false},{"track":"A · English","score":0.2491,"headline":true},{"track":"C · Arabic pairs","score":0.285,"headline":false},{"track":"C · English pairs","score":0.29,"headline":false}]},"41":{"raw":0.4999,"skill":0.2498,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3754,"skill":0.0091,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5546,"skill":0.1092,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3168,"skill":0.0085,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5015,"skill":0.003,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.2472,"skill":0.1533,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.4837,"skill":0.2516,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4409,"skill":0.2545,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.4432,"skill":0.304,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.6139,"skill":0.3441,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.603,"skill":0.206,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5532,"skill":0.2897,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.1261,"skill":0.1145,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1257,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.4915,"skill":0.2383,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5864,"skill":0.1728,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.5838,"skill":0.4451,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0075,"skill":0.0075,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6102,"skill":0.4267,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.6838,"skill":0.3848,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7705,"skill":0.7675,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.5523,"skill":0.5496,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6149,"skill":0.3592,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.6201,"skill":0.6128,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0125,"skill":0.0125,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"djev","name":"djev","long_name":"djev diffgemma","variant":null,"meta":{"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/Davipar/djev-dev","base_model":"google/diffusiongemma-26B-A4B-it","base_url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it","weights_repo":null,"served_checkpoint":"google/diffusiongemma-26B-A4B-it"},"completed":121057,"counts":{"ok":119892,"unsupported":1165},"lifted":true,"calibration":{"acc":0.6129,"conf":0.8452,"ece":0.2323,"brier":0.5974,"over95":0.0791,"n":29651,"benchmarks":33,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.003,"conf":0.1634,"acc":0.0371},{"lo":0.2,"w":0.0137,"conf":0.258,"acc":0.1395},{"lo":0.3,"w":0.025,"conf":0.3538,"acc":0.1978},{"lo":0.4,"w":0.0351,"conf":0.4563,"acc":0.2261},{"lo":0.5,"w":0.0661,"conf":0.549,"acc":0.4223},{"lo":0.6,"w":0.0734,"conf":0.6528,"acc":0.4607},{"lo":0.7,"w":0.075,"conf":0.7524,"acc":0.4869},{"lo":0.8,"w":0.1416,"conf":0.8574,"acc":0.4182},{"lo":0.9,"w":0.5671,"conf":0.9774,"acc":0.7767}],"areas":{"knowledge":{"ece":0.2598,"acc":0.5365,"conf":0.7963},"language":{"ece":0.1444,"acc":0.7699,"conf":0.9143},"retrieval":{"ece":0.4174,"acc":0.4502,"conf":0.8677},"tools":{"ece":0.1155,"acc":0.7759,"conf":0.8833},"arts":{"ece":0.2516,"acc":0.5458,"conf":0.7974}}},"gaps":{"share":0.0096,"reasons":[{"reason":"more questions per request than its answer canvas holds","share":0.0096}]},"latency":{"median":84.3,"p95":216.0,"mean":130.3},"results":{"1":{"score":0.9292,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":93.7},"2":{"score":0.3713,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":560.1},"3":{"score":0.7067,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":460.1},"4":{"score":0.6944,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":125.0},"5":{"score":0.1352,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":152.3},"6":{"score":0.7988,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":207.2},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":210.8},"10":{"score":0.1479,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":106.9},"11":{"score":0.7002,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":164.2},"12":{"score":0.5981,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":79.6},"20":{"score":0.9028,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":77.3},"21":{"score":0.6088,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":72.0},"22":{"score":0.15,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":174.6},"23":{"score":0.5942,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":86.0},"24":{"score":0.7943,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":77.6},"25":{"score":0.4745,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":80.1},"26":{"score":0.984,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":75.7},"27":{"score":0.9522,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":77.4},"28":{"score":0.6938,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":71.4},"29":{"score":0.8386,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":82.5},"30":{"score":0.3669,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":80.5},"31":{"score":0.1402,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":100.2},"32":{"score":0.633,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":101.6},"33":{"score":0.2818,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":106.9},"34":{"score":0.4,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":80.9},"36":{"score":0.1234,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":801.0},"37":{"score":0.5643,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":81.6},"38":{"score":0.006,"jev":0.1497,"metric":"case exact accuracy","requests":1565,"answered":400,"unsupported":1165,"errors":0,"median_ms":109.8},"39":{"score":0.8438,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":82.6},"40":{"score":0.5967,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":70.3,"tracks":[{"track":"A · Arabic","score":0.2825,"headline":false},{"track":"A · English","score":0.5967,"headline":true},{"track":"C · Arabic pairs","score":0.765,"headline":false},{"track":"C · English pairs","score":0.96,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6458,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":80.5},"42":{"score":0.7509,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":89.7},"43":{"score":0.4684,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":76.3},"44":{"score":0.6392,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":74.5},"45":{"score":0.1178,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":83.6},"48":{"score":0.2234,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":91.6},"50":{"score":0.4666,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":102.2},"56":{"score":0.8605,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":89.0,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4404,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":80.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6726,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":75.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5449,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":97.4,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6042,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":80.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7306,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":85.0,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6683,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":85.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":51.33,"balanced_skill":37.59,"breadth_skill":36.75},"frozen_scores":{"balanced_raw":41.53,"balanced_skill":28.52,"breadth_skill":24.45},"categories":[{"id":"knowledge","raw":0.4235,"skill":0.2612,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.64,"skill":0.4916,"coverage":0.9256,"pending":0,"n":10},{"id":"retrieval","raw":0.4563,"skill":0.3446,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5585,"skill":0.472,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4883,"skill":0.3101,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9292,"skill":0.9044,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3713,"skill":0.3077,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7989,"skill":0.5257,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7854,"headline":false},{"track":"RouterBench-5shot","score":0.8123,"headline":false}]},"11":{"raw":0.7002,"skill":0.5664,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9047,"skill":0.8095,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6088,"skill":0.2177,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1425,"skill":0.1358,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5852,"skill":0.1704,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8013,"skill":0.735,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4745,"skill":0.2993,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3669,"skill":0.2298,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.3988,"headline":false},{"track":"GSM8K-10choice","score":0.3351,"headline":false}]},"31":{"raw":0.1402,"skill":0.0635,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1234,"skill":0.0807,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5643,"skill":0.4534,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5967,"skill":0.4811,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2825,"headline":false},{"track":"A · English","score":0.5967,"headline":true},{"track":"C · Arabic pairs","score":0.765,"headline":false},{"track":"C · English pairs","score":0.96,"headline":false}]},"41":{"raw":0.6458,"skill":0.4687,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4684,"skill":0.1566,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6392,"skill":0.2784,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4666,"skill":0.2258,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.8605,"skill":0.721,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4404,"skill":0.3706,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6726,"skill":0.5254,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5449,"skill":0.3932,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6042,"skill":0.5052,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7306,"skill":0.5424,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6683,"skill":0.3366,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.633,"skill":0.4165,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2818,"skill":0.2723,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1178,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5981,"skill":0.398,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6938,"skill":0.3876,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8386,"skill":0.7848,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0015,"skill":0.0015,"coverage":0.2556,"random":0.0,"tracks":[]},"39":{"raw":0.8438,"skill":0.7703,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7509,"skill":0.5154,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.6944,"skill":0.6905,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.1352,"skill":0.13,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.1479,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.7067,"skill":0.701,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.1064,"skill":0.1064,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"gliner-base","name":"GLiNER 2.5 base","long_name":"GLiNER 2.5 base","variant":null,"meta":{"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":193581591,"params_estimated":false,"params_from_name":false,"trained_bytes":774366564,"model_url":"https://huggingface.co/fastino/gliner2.5-base-v1","code_url":null,"base_model":null,"base_url":null,"weights_repo":"fastino/gliner2.5-base-v1","served_checkpoint":null},"completed":121057,"counts":{"ok":121046,"unsupported":11},"lifted":true,"calibration":{"acc":0.3371,"conf":0.7211,"ece":0.384,"brier":0.9862,"over95":0.1618,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0262,"conf":0.0695,"acc":0.0399},{"lo":0.1,"w":0.0229,"conf":0.1434,"acc":0.0624},{"lo":0.2,"w":0.0372,"conf":0.2599,"acc":0.1158},{"lo":0.3,"w":0.0519,"conf":0.3519,"acc":0.1868},{"lo":0.4,"w":0.0729,"conf":0.4511,"acc":0.2538},{"lo":0.5,"w":0.1125,"conf":0.5481,"acc":0.3745},{"lo":0.6,"w":0.0935,"conf":0.6486,"acc":0.3657},{"lo":0.7,"w":0.0874,"conf":0.7499,"acc":0.4141},{"lo":0.8,"w":0.1061,"conf":0.8516,"acc":0.4291},{"lo":0.9,"w":0.3893,"conf":0.9682,"acc":0.3699}],"areas":{"knowledge":{"ece":0.4139,"acc":0.2842,"conf":0.6981},"language":{"ece":0.465,"acc":0.384,"conf":0.8485},"retrieval":{"ece":0.2449,"acc":0.3619,"conf":0.6068},"tools":{"ece":0.408,"acc":0.2924,"conf":0.6968},"arts":{"ece":0.2664,"acc":0.3863,"conf":0.6528}}},"gaps":{"share":0.0001,"reasons":[{"reason":"prompt text with parentheses, which the native GLiNER prompt syntax cannot carry","share":0.0001}]},"latency":{"median":14.4,"p95":220.2,"mean":277.3},"results":{"1":{"score":0.3878,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":18.2},"2":{"score":0.1559,"jev":0.4475,"metric":"nDCG@10","requests":1000,"answered":998,"unsupported":2,"errors":0,"median_ms":1743.4},"3":{"score":0.1555,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":1280.2},"4":{"score":0.2383,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":26.6},"5":{"score":0.2689,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":55.5},"6":{"score":0.6718,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":67.7},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":122.1},"10":{"score":0.459,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":38.7},"11":{"score":0.2285,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":149.7},"12":{"score":0.3048,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":13.8},"20":{"score":0.524,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":13.9},"21":{"score":0.4912,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":13.6},"22":{"score":0.036,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":530.3},"23":{"score":0.503,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.4},"24":{"score":0.3267,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":13.8},"25":{"score":0.25,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":13.5},"26":{"score":0.4821,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2373,"unsupported":3,"errors":0,"median_ms":13.8},"27":{"score":0.2858,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":13.8},"28":{"score":0.5083,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":13.4},"29":{"score":0.3479,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":14.2},"30":{"score":0.2237,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":14.2},"31":{"score":0.0742,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":25.7},"32":{"score":0.3484,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":19.5},"33":{"score":0.0012,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":20.1},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":13.8},"36":{"score":0.0507,"jev":0.1904,"metric":"nDCG@10","requests":550,"answered":549,"unsupported":1,"errors":0,"median_ms":1597.0},"37":{"score":0.2005,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.5},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":604.0},"39":{"score":0.7008,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":13.9},"40":{"score":0.2384,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":13.3,"tracks":[{"track":"A · Arabic","score":0.2591,"headline":false},{"track":"A · English","score":0.2384,"headline":true},{"track":"C · Arabic pairs","score":0.505,"headline":false},{"track":"C · English pairs","score":0.445,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3575,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":14.0},"42":{"score":0.3981,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":14.6},"43":{"score":0.4,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":13.9},"44":{"score":0.5164,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":13.9},"45":{"score":0.138,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":14.2},"48":{"score":0.4664,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":15.7},"50":{"score":0.3562,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":18.5},"56":{"score":0.5045,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":15.3,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1408,"metric":"accuracy","requests":12032,"answered":12031,"unsupported":1,"errors":0,"median_ms":13.1,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3361,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":12.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3401,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":17.8,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2652,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":12.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.4836,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":15.1,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5112,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":13.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":30.29,"balanced_skill":7.08,"breadth_skill":6.67},"frozen_scores":{"balanced_raw":24.46,"balanced_skill":3.69,"breadth_skill":3.34},"categories":[{"id":"knowledge","raw":0.2429,"skill":0.0213,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.3568,"skill":0.0886,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.319,"skill":0.0898,"coverage":0.9997,"pending":0,"n":7},{"id":"tools","raw":0.2851,"skill":0.122,"coverage":0.9997,"pending":0,"n":6},{"id":"arts","raw":0.3105,"skill":0.0322,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3878,"skill":0.1736,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.1555,"skill":0.0701,"coverage":0.998,"random":0.0918,"tracks":[]},"6":{"raw":0.6718,"skill":0.2289,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6395,"headline":false},{"track":"RouterBench-5shot","score":0.7041,"headline":false}]},"11":{"raw":0.2285,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5188,"skill":0.0377,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4912,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0362,"skill":0.0287,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5059,"skill":0.0119,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3286,"skill":0.1047,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.25,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2237,"skill":0.0608,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.3108,"headline":false},{"track":"GSM8K-10choice","score":0.1365,"headline":false}]},"31":{"raw":0.0742,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0506,"skill":0.0044,"coverage":0.9982,"random":0.0464,"tracks":[]},"37":{"raw":0.2005,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2384,"skill":0.0202,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2591,"headline":false},{"track":"A · English","score":0.2384,"headline":true},{"track":"C · Arabic pairs","score":0.505,"headline":false},{"track":"C · English pairs","score":0.445,"headline":false}]},"41":{"raw":0.3575,"skill":0.0363,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4,"skill":0.048,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5164,"skill":0.0328,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3562,"skill":0.0656,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5045,"skill":0.009,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1408,"skill":0.0336,"coverage":0.9999,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3361,"skill":0.0377,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3401,"skill":0.1201,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2652,"skill":0.0815,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.4836,"skill":0.1228,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5112,"skill":0.0224,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3484,"skill":0.0,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0012,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1377,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.3048,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5083,"skill":0.0166,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3479,"skill":0.1305,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7008,"skill":0.5599,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.3981,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.2383,"skill":0.2285,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.2689,"skill":0.2645,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.459,"skill":0.0998,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.1555,"skill":0.1392,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7598,"pending":0.24},{"engine":"gliner-multi","name":"GLiNER 2.5 multilingual","long_name":"GLiNER 2.5 multilingual","variant":null,"meta":{"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":287355159,"params_estimated":false,"params_from_name":false,"trained_bytes":1149461028,"model_url":"https://huggingface.co/fastino/gliner2.5-multi-v1","code_url":null,"base_model":null,"base_url":null,"weights_repo":"fastino/gliner2.5-multi-v1","served_checkpoint":null},"completed":121057,"counts":{"ok":121046,"unsupported":11},"lifted":true,"calibration":{"acc":0.3438,"conf":0.5676,"ece":0.2538,"brier":0.8405,"over95":0.0463,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0025,"conf":0.0847,"acc":0.087},{"lo":0.1,"w":0.0191,"conf":0.1554,"acc":0.026},{"lo":0.2,"w":0.0972,"conf":0.266,"acc":0.4205},{"lo":0.3,"w":0.1903,"conf":0.3512,"acc":0.2351},{"lo":0.4,"w":0.143,"conf":0.4483,"acc":0.236},{"lo":0.5,"w":0.1356,"conf":0.5477,"acc":0.3913},{"lo":0.6,"w":0.1094,"conf":0.6491,"acc":0.423},{"lo":0.7,"w":0.0976,"conf":0.7438,"acc":0.4335},{"lo":0.8,"w":0.0721,"conf":0.851,"acc":0.3654},{"lo":0.9,"w":0.1332,"conf":0.964,"acc":0.4188}],"areas":{"knowledge":{"ece":0.2291,"acc":0.3547,"conf":0.4796},"language":{"ece":0.2889,"acc":0.398,"conf":0.6869},"retrieval":{"ece":0.2284,"acc":0.237,"conf":0.4654},"tools":{"ece":0.4476,"acc":0.2362,"conf":0.6838},"arts":{"ece":0.2056,"acc":0.3702,"conf":0.5758}}},"gaps":{"share":0.0001,"reasons":[{"reason":"prompt text with parentheses, which the native GLiNER prompt syntax cannot carry","share":0.0001}]},"latency":{"median":15.2,"p95":294.9,"mean":211.4},"results":{"1":{"score":0.3743,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":22.8},"2":{"score":0.0902,"jev":0.4475,"metric":"nDCG@10","requests":1000,"answered":998,"unsupported":2,"errors":0,"median_ms":1570.7},"3":{"score":0.0157,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":1976.8},"4":{"score":0.0903,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":27.2},"5":{"score":0.1278,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":55.9},"6":{"score":0.5871,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":81.8},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":162.2},"10":{"score":0.4188,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":56.7},"11":{"score":0.3021,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":252.8},"12":{"score":0.3079,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":14.8},"20":{"score":0.4926,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.6},"21":{"score":0.5209,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":14.4},"22":{"score":0.034,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":438.8},"23":{"score":0.4958,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":15.1},"24":{"score":0.2811,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":14.6},"25":{"score":0.2347,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":14.0},"26":{"score":0.3527,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2373,"unsupported":3,"errors":0,"median_ms":14.4},"27":{"score":0.2568,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":14.5},"28":{"score":0.5114,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":14.3},"29":{"score":0.3007,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":14.9},"30":{"score":0.1945,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":14.8},"31":{"score":0.0738,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":21.6},"32":{"score":0.3404,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":29.2},"33":{"score":0,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":25.2},"34":{"score":0.4,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":14.4},"36":{"score":0.0475,"jev":0.1904,"metric":"nDCG@10","requests":550,"answered":549,"unsupported":1,"errors":0,"median_ms":1911.8},"37":{"score":0.1498,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":15.2},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":758.7},"39":{"score":0.6852,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":17.0},"40":{"score":0.2376,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":14.3,"tracks":[{"track":"A · Arabic","score":0.221,"headline":false},{"track":"A · English","score":0.2376,"headline":true},{"track":"C · Arabic pairs","score":0.33,"headline":false},{"track":"C · English pairs","score":0.375,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.2642,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":14.7},"42":{"score":0.4129,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":15.4},"43":{"score":0.4035,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":14.7},"44":{"score":0.4926,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.6},"45":{"score":0.132,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":15.1},"48":{"score":0.3341,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":18.1},"50":{"score":0.3598,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":23.6},"56":{"score":0.5025,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":64.7,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1199,"metric":"accuracy","requests":12032,"answered":12031,"unsupported":1,"errors":0,"median_ms":32.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3138,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":32.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3478,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":50.7,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.233,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":14.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5223,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":54.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5505,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":38.2,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":27.88,"balanced_skill":4.41,"breadth_skill":4.2},"frozen_scores":{"balanced_raw":22.74,"balanced_skill":1.83,"breadth_skill":1.74},"categories":[{"id":"knowledge","raw":0.2305,"skill":0.0094,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.3544,"skill":0.0835,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2696,"skill":0.0487,"coverage":0.9997,"pending":0,"n":7},{"id":"tools","raw":0.2358,"skill":0.0529,"coverage":0.9997,"pending":0,"n":6},{"id":"arts","raw":0.3039,"skill":0.0257,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3743,"skill":0.1553,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.09,"skill":0.0,"coverage":0.998,"random":0.0918,"tracks":[]},"6":{"raw":0.5872,"skill":0.0316,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.5411,"headline":false},{"track":"RouterBench-5shot","score":0.6332,"headline":false}]},"11":{"raw":0.3021,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.4948,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5209,"skill":0.0419,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0337,"skill":0.0262,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.4851,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.2862,"skill":0.0483,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2347,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1945,"skill":0.025,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2805,"headline":false},{"track":"GSM8K-10choice","score":0.1084,"headline":false}]},"31":{"raw":0.0738,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0474,"skill":0.001,"coverage":0.9982,"random":0.0464,"tracks":[]},"37":{"raw":0.1498,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2376,"skill":0.0192,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.221,"headline":false},{"track":"A · English","score":0.2376,"headline":true},{"track":"C · Arabic pairs","score":0.33,"headline":false},{"track":"C · English pairs","score":0.375,"headline":false}]},"41":{"raw":0.2642,"skill":0.0,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4035,"skill":0.0536,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4926,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3598,"skill":0.0708,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5025,"skill":0.005,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1198,"skill":0.01,"coverage":0.9999,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3138,"skill":0.0054,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3478,"skill":0.1304,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.233,"skill":0.0413,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5223,"skill":0.1886,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5505,"skill":0.101,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3404,"skill":0.0,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1317,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.3079,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5114,"skill":0.0228,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3007,"skill":0.0676,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6852,"skill":0.537,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4129,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0903,"skill":0.0786,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.1278,"skill":0.1225,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4188,"skill":0.0329,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0157,"skill":0.0,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7598,"pending":0.24},{"engine":"gliner-small","name":"GLiNER 2.5 small","long_name":"GLiNER 2.5 small","variant":null,"meta":{"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiNER2","served_params":73881879,"params_estimated":false,"params_from_name":false,"trained_bytes":295567700,"model_url":"https://huggingface.co/fastino/gliner2.5-small-v1","code_url":null,"base_model":null,"base_url":null,"weights_repo":"fastino/gliner2.5-small-v1","served_checkpoint":null},"completed":121057,"counts":{"ok":121046,"unsupported":11},"lifted":true,"calibration":{"acc":0.3521,"conf":0.4859,"ece":0.1755,"brier":0.7621,"over95":0.0233,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.1146,"conf":0.0403,"acc":0.0428},{"lo":0.1,"w":0.0498,"conf":0.1456,"acc":0.0889},{"lo":0.2,"w":0.0905,"conf":0.2762,"acc":0.5031},{"lo":0.3,"w":0.1281,"conf":0.3482,"acc":0.2965},{"lo":0.4,"w":0.1064,"conf":0.4512,"acc":0.3645},{"lo":0.5,"w":0.168,"conf":0.5477,"acc":0.4706},{"lo":0.6,"w":0.1314,"conf":0.6408,"acc":0.3792},{"lo":0.7,"w":0.0745,"conf":0.7485,"acc":0.4508},{"lo":0.8,"w":0.0596,"conf":0.8481,"acc":0.3855},{"lo":0.9,"w":0.0771,"conf":0.9583,"acc":0.4541}],"areas":{"knowledge":{"ece":0.1711,"acc":0.3715,"conf":0.433},"language":{"ece":0.3329,"acc":0.3208,"conf":0.6537},"retrieval":{"ece":0.0742,"acc":0.3214,"conf":0.3478},"tools":{"ece":0.0761,"acc":0.3317,"conf":0.4049},"arts":{"ece":0.0873,"acc":0.3987,"conf":0.4703}}},"gaps":{"share":0.0001,"reasons":[{"reason":"prompt text with parentheses, which the native GLiNER prompt syntax cannot carry","share":0.0001}]},"latency":{"median":14.6,"p95":122.1,"mean":158.6},"results":{"1":{"score":0.4067,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":15.9},"2":{"score":0.1113,"jev":0.4475,"metric":"nDCG@10","requests":1000,"answered":998,"unsupported":2,"errors":0,"median_ms":927.5},"3":{"score":0.0197,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":672.4},"4":{"score":0.0854,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":19.3},"5":{"score":0.136,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":31.9},"6":{"score":0.5725,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":38.0},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":71.2},"10":{"score":0.479,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":19.7},"11":{"score":0.211,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":81.8},"12":{"score":0.2537,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":13.8},"20":{"score":0.4814,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.0},"21":{"score":0.5023,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":13.4},"22":{"score":0.0465,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":286.5},"23":{"score":0.5182,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.6},"24":{"score":0.3066,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":13.6},"25":{"score":0.2653,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":14.2},"26":{"score":0.4282,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2373,"unsupported":3,"errors":0,"median_ms":13.5},"27":{"score":0.2807,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":13.5},"28":{"score":0.4988,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":13.4},"29":{"score":0.3169,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":14.4},"30":{"score":0.2346,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":14.3},"31":{"score":0.0808,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":16.4},"32":{"score":0.3777,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":16.7},"33":{"score":0.0012,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":18.3},"34":{"score":0.5,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":14.4},"36":{"score":0.0392,"jev":0.1904,"metric":"nDCG@10","requests":550,"answered":549,"unsupported":1,"errors":0,"median_ms":862.8},"37":{"score":0.2082,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.6},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":317.7},"39":{"score":0.4662,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":14.1},"40":{"score":0.2373,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":13.3,"tracks":[{"track":"A · Arabic","score":0.2475,"headline":false},{"track":"A · English","score":0.2373,"headline":true},{"track":"C · Arabic pairs","score":0.5,"headline":false},{"track":"C · English pairs","score":0.49,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3708,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":13.9},"42":{"score":0.4628,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":14.6},"43":{"score":0.4,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":13.9},"44":{"score":0.4922,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":13.8},"45":{"score":0.152,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":14.1},"48":{"score":0.5084,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":15.3},"50":{"score":0.3311,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":16.6},"56":{"score":0.5,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":14.6,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1233,"metric":"accuracy","requests":12032,"answered":12031,"unsupported":1,"errors":0,"median_ms":13.6,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3178,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":13.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3283,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":14.4,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2841,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":13.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.4099,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":13.5,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.516,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":13.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":27.9,"balanced_skill":3.88,"breadth_skill":3.81},"frozen_scores":{"balanced_raw":23.76,"balanced_skill":2.49,"breadth_skill":2.37},"categories":[{"id":"knowledge","raw":0.2445,"skill":0.018,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.3227,"skill":0.0379,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2805,"skill":0.0538,"coverage":0.9997,"pending":0,"n":7},{"id":"tools","raw":0.2397,"skill":0.0571,"coverage":0.9997,"pending":0,"n":6},{"id":"arts","raw":0.3077,"skill":0.0274,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.4067,"skill":0.1991,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.1111,"skill":0.0212,"coverage":0.998,"random":0.0918,"tracks":[]},"6":{"raw":0.5725,"skill":0.0168,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.5098,"headline":false},{"track":"RouterBench-5shot","score":0.6352,"headline":false}]},"11":{"raw":0.211,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.4838,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5023,"skill":0.0046,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0448,"skill":0.0373,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5076,"skill":0.0153,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3093,"skill":0.0791,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2653,"skill":0.0204,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2346,"skill":0.0756,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.3343,"headline":false},{"track":"GSM8K-10choice","score":0.135,"headline":false}]},"31":{"raw":0.0808,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0392,"skill":0.0,"coverage":0.9982,"random":0.0464,"tracks":[]},"37":{"raw":0.2082,"skill":0.0068,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2373,"skill":0.0188,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2475,"headline":false},{"track":"A · English","score":0.2373,"headline":true},{"track":"C · Arabic pairs","score":0.5,"headline":false},{"track":"C · English pairs","score":0.49,"headline":false}]},"41":{"raw":0.3708,"skill":0.0563,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4,"skill":0.048,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4922,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3311,"skill":0.0293,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1233,"skill":0.0139,"coverage":0.9999,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3178,"skill":0.0112,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3283,"skill":0.1044,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2841,"skill":0.1051,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.4099,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.516,"skill":0.032,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3777,"skill":0.0107,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0012,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1517,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.2537,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.4988,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3169,"skill":0.0892,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.4662,"skill":0.2149,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4628,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0854,"skill":0.0736,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.136,"skill":0.1308,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.479,"skill":0.1331,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0197,"skill":0.0008,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7598,"pending":0.24},{"engine":"harsha","name":"Qwen-2.5-1B-RLCD","long_name":"Qwen-2.5-1B-RLCD","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":1543714304,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD","base_model":"Qwen/Qwen2.5-1.5B","base_url":"https://huggingface.co/Qwen/Qwen2.5-1.5B","weights_repo":null,"served_checkpoint":null},"completed":121057,"counts":{"ok":120677,"unsupported":380},"lifted":false,"calibration":{"acc":0.3841,"conf":0.5684,"ece":0.1871,"brier":0.775,"over95":0.0341,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.1385,"conf":0.0392,"acc":0.0474},{"lo":0.1,"w":0.0097,"conf":0.1432,"acc":0.1705},{"lo":0.2,"w":0.0184,"conf":0.2576,"acc":0.121},{"lo":0.3,"w":0.0594,"conf":0.3546,"acc":0.2552},{"lo":0.4,"w":0.1002,"conf":0.4547,"acc":0.3506},{"lo":0.5,"w":0.1656,"conf":0.5484,"acc":0.4176},{"lo":0.6,"w":0.1492,"conf":0.6405,"acc":0.5332},{"lo":0.7,"w":0.1455,"conf":0.7495,"acc":0.475},{"lo":0.8,"w":0.0912,"conf":0.8516,"acc":0.4813},{"lo":0.9,"w":0.1222,"conf":0.9578,"acc":0.5039}],"areas":{"knowledge":{"ece":0.1626,"acc":0.412,"conf":0.5615},"language":{"ece":0.2699,"acc":0.4209,"conf":0.6906},"retrieval":{"ece":0.2027,"acc":0.1929,"conf":0.3787},"tools":{"ece":0.2339,"acc":0.347,"conf":0.5761},"arts":{"ece":0.1175,"acc":0.4261,"conf":0.5121}}},"gaps":{"share":0.0031,"reasons":[{"reason":"prompt longer than its context window (no truncation allowed)","share":0.0031}]},"latency":{"median":42.2,"p95":87.1,"mean":53.5},"results":{"1":{"score":0.474,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":43.5},"2":{"score":0.0584,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":250.4},"3":{"score":0.0433,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":129.0},"4":{"score":0.0259,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":46.5},"5":{"score":0.0116,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":54.4},"6":{"score":0.5886,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":63.9},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":75.3},"10":{"score":0.4644,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":45.3},"11":{"score":0.2668,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":96.9},"12":{"score":0.2819,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":41.3},"20":{"score":0.6808,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":41.3},"21":{"score":0.5068,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":40.4},"22":{"score":0.029,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":111.5},"23":{"score":0.5024,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":42.2},"24":{"score":0.3688,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":40.8},"25":{"score":0.2806,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":41.2},"26":{"score":0.5648,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":40.8},"27":{"score":0.4369,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":40.9},"28":{"score":0.5043,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":40.6},"29":{"score":0.4255,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":41.9},"30":{"score":0.1748,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":41.6},"31":{"score":0.0882,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":43.9},"32":{"score":0.3777,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":46.3},"33":{"score":0.0236,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":45.3},"34":{"score":0.5,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":40.9},"36":{"score":0.0573,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":335.4},"37":{"score":0.0892,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":42.3},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":295.7},"39":{"score":0.4332,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":41.9},"40":{"score":0.2431,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":40.6,"tracks":[{"track":"A · Arabic","score":0.2768,"headline":false},{"track":"A · English","score":0.2431,"headline":true},{"track":"C · Arabic pairs","score":0.495,"headline":false},{"track":"C · English pairs","score":0.475,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3142,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":41.2},"42":{"score":0.4373,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":42.4},"43":{"score":0.3579,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":41.3},"44":{"score":0.503,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":41.6},"45":{"score":0.1717,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":41.8},"48":{"score":0.2466,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":44.0},"50":{"score":0.3848,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":45.6},"56":{"score":0.4905,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":44.0,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1154,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":40.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3374,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":40.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.2204,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":42.6,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.1932,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":40.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.0063,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":42.4,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5012,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":40.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":26.59,"balanced_skill":4.21,"breadth_skill":4.03},"frozen_scores":{"balanced_raw":24.45,"balanced_skill":4.43,"breadth_skill":3.98},"categories":[{"id":"knowledge","raw":0.243,"skill":0.0132,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.2913,"skill":0.0435,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2343,"skill":0.0202,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.2308,"skill":0.0587,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3302,"skill":0.0748,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.474,"skill":0.29,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.0584,"skill":0.0,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.5886,"skill":0.0372,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.5345,"headline":false},{"track":"RouterBench-5shot","score":0.6427,"headline":false}]},"11":{"raw":0.2668,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.6735,"skill":0.347,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5068,"skill":0.0137,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0298,"skill":0.0222,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.51,"skill":0.02,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3832,"skill":0.1776,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2806,"skill":0.0408,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1748,"skill":0.003,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2441,"headline":false},{"track":"GSM8K-10choice","score":0.1054,"headline":false}]},"31":{"raw":0.0882,"skill":0.0069,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0573,"skill":0.0114,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.0892,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2431,"skill":0.0262,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2768,"headline":false},{"track":"A · English","score":0.2431,"headline":true},{"track":"C · Arabic pairs","score":0.495,"headline":false},{"track":"C · English pairs","score":0.475,"headline":false}]},"41":{"raw":0.3142,"skill":0.0,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3579,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.503,"skill":0.006,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3848,"skill":0.1072,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.4905,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1154,"skill":0.0051,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3374,"skill":0.0396,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.2204,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.1932,"skill":0.0,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.0063,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5012,"skill":0.0024,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3777,"skill":0.0107,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0236,"skill":0.0106,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1717,"skill":0.0091,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.2819,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5043,"skill":0.0086,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.4255,"skill":0.234,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.4332,"skill":0.1663,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4373,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0259,"skill":0.0134,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0116,"skill":0.0056,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4644,"skill":0.1088,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0433,"skill":0.0249,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0136,"skill":0.0136,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"hopper","name":"Hopper","long_name":"Hopper Qwen3.5-4B LoRA","variant":null,"meta":{"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":129927008,"model_url":"https://huggingface.co/HopitAI/hopper","code_url":"https://github.com/hopit-ai/hopper","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"HopitAI/hopper","served_checkpoint":"Qwen/Qwen3.5-4B"},"completed":121057,"counts":{"ok":106557,"unsupported":14500},"lifted":false,"calibration":{"acc":0.6333,"conf":0.7315,"ece":0.0995,"brier":0.4781,"over95":0.0019,"n":70122,"benchmarks":29,"rel":[{"lo":0.0,"w":0.0011,"conf":0.094,"acc":0.116},{"lo":0.1,"w":0.0169,"conf":0.1579,"acc":0.1233},{"lo":0.2,"w":0.0253,"conf":0.2523,"acc":0.145},{"lo":0.3,"w":0.0409,"conf":0.3534,"acc":0.2386},{"lo":0.4,"w":0.0598,"conf":0.4561,"acc":0.3065},{"lo":0.5,"w":0.1055,"conf":0.5486,"acc":0.4886},{"lo":0.6,"w":0.117,"conf":0.6521,"acc":0.5229},{"lo":0.7,"w":0.1518,"conf":0.7434,"acc":0.4808},{"lo":0.8,"w":0.2037,"conf":0.8519,"acc":0.7509},{"lo":0.9,"w":0.2779,"conf":0.9354,"acc":0.9379}],"areas":{"knowledge":{"ece":0.1616,"acc":0.5025,"conf":0.6599},"language":{"ece":0.0422,"acc":0.7407,"conf":0.7816},"retrieval":{"ece":0.1387,"acc":0.6435,"conf":0.7823},"tools":{"ece":0.0814,"acc":0.945,"conf":0.8725},"arts":{"ece":0.1181,"acc":0.6236,"conf":0.7375}}},"gaps":{"share":0.1198,"reasons":[{"reason":"hopper request.parse refuses question 'q1': 151 options, more than 26 letters","share":0.0454},{"reason":"hopper request.parse refuses question 'q1': 77 options, more than 26 letters","share":0.0254},{"reason":"hopper request.parse refuses question 'chord': 129 options, more than 26 letters","share":0.0165}]},"latency":{"median":46.0,"p95":345.5,"mean":161.7},"results":{"1":{"score":0.9126,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":127.5},"2":{"score":0.4202,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1445.3},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7946,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":191.3},"9":{"score":0.225,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1126.7},"10":{"score":0.6461,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":65.6},"11":{"score":0.6963,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1264.1},"12":{"score":0.6105,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":44.8},"20":{"score":0.8356,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":44.8},"21":{"score":0.5955,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":44.3},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.591,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":45.9},"24":{"score":0.7381,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":44.2},"25":{"score":0.3878,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":45.0},"26":{"score":0.9718,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":44.5},"27":{"score":0.9258,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":44.4},"28":{"score":0.6788,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":44.3},"29":{"score":0.8435,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":44.7},"30":{"score":0.3173,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":44.4},"31":{"score":0.2179,"jev":0.2746,"metric":"accuracy","requests":5000,"answered":1588,"unsupported":3412,"errors":0,"median_ms":45.7},"32":{"score":0.5412,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":47.2},"33":{"score":0.303,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":357.1},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":44.9},"36":{"score":0.1602,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1454.0},"37":{"score":0.3809,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":45.2},"38":{"score":0.015,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":2833.1},"39":{"score":0.8112,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":105.7},"40":{"score":0.4103,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":54.3,"tracks":[{"track":"A · Arabic","score":0.3537,"headline":false},{"track":"A · English","score":0.4103,"headline":true},{"track":"C · Arabic pairs","score":0.555,"headline":false},{"track":"C · English pairs","score":0.89,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4397,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":58.1},"42":{"score":0.7803,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":101.1},"43":{"score":0.4754,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":61.1},"44":{"score":0.6332,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":58.9},"45":{"score":0.0978,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":74.5},"48":{"score":0.2274,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":116.4},"50":{"score":0.4153,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":92.7},"56":{"score":0.6175,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":404.4,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4796,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":45.1,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6301,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":44.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.6216,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":75.7,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5701,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":45.1,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7248,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":66.0,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6432,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":53.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":45.59,"balanced_skill":30.01,"breadth_skill":28.72},"frozen_scores":{"balanced_raw":37.63,"balanced_skill":23.06,"breadth_skill":19.67},"categories":[{"id":"knowledge","raw":0.3935,"skill":0.2237,"coverage":0.9318,"pending":0,"n":10},{"id":"language","raw":0.601,"skill":0.437,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3497,"skill":0.1822,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.4957,"skill":0.4134,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.4398,"skill":0.2442,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9126,"skill":0.8821,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4202,"skill":0.3616,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7946,"skill":0.5165,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7783,"headline":false},{"track":"RouterBench-5shot","score":0.8109,"headline":false}]},"11":{"raw":0.6963,"skill":0.5609,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8384,"skill":0.6768,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5955,"skill":0.191,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.5686,"skill":0.1371,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7501,"skill":0.6668,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3878,"skill":0.1837,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3173,"skill":0.1763,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4139,"headline":false},{"track":"GSM8K-10choice","score":0.2206,"headline":false}]},"31":{"raw":0.0692,"skill":0.0,"coverage":0.3176,"random":0.0819,"tracks":[]},"36":{"raw":0.1602,"skill":0.1193,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3809,"skill":0.2234,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4103,"skill":0.2414,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3537,"headline":false},{"track":"A · English","score":0.4103,"headline":true},{"track":"C · Arabic pairs","score":0.555,"headline":false},{"track":"C · English pairs","score":0.89,"headline":false}]},"41":{"raw":0.4397,"skill":0.1595,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4754,"skill":0.1677,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6332,"skill":0.2664,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4153,"skill":0.1514,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6175,"skill":0.235,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4796,"skill":0.4147,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6301,"skill":0.4638,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.6216,"skill":0.4955,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5701,"skill":0.4626,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7248,"skill":0.5325,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6432,"skill":0.2864,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5412,"skill":0.2706,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.303,"skill":0.2937,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.0978,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6105,"skill":0.4166,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6788,"skill":0.3576,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8435,"skill":0.7913,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.015,"skill":0.015,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8112,"skill":0.7223,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7803,"skill":0.5726,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.6461,"skill":0.4111,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.225,"skill":0.225,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0904,"skill":0.0904,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6927,"pending":0.24,"new_in_edition":true},{"engine":"jeff-uncapped","name":"jeff","long_name":"jeff GLiFormer-large","variant":null,"meta":{"kind":"inference technique","technique":"GLi family","technique_name":"GLiFormer","served_params":575683963,"params_estimated":true,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/logan-markewich/jeff","base_model":"knowledgator/gliformer-large-v1","base_url":"https://huggingface.co/knowledgator/gliformer-large-v1","weights_repo":null,"served_checkpoint":"knowledgator/gliformer-large-v1"},"completed":121057,"counts":{"ok":121057},"lifted":true,"calibration":{"acc":0.4011,"conf":0.4136,"ece":0.1131,"brier":0.6841,"over95":0,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.144,"conf":0.037,"acc":0.2279},{"lo":0.1,"w":0.0297,"conf":0.146,"acc":0.1938},{"lo":0.2,"w":0.0948,"conf":0.257,"acc":0.4826},{"lo":0.3,"w":0.1957,"conf":0.3492,"acc":0.3442},{"lo":0.4,"w":0.0991,"conf":0.4394,"acc":0.4299},{"lo":0.5,"w":0.2285,"conf":0.5454,"acc":0.4335},{"lo":0.6,"w":0.1294,"conf":0.646,"acc":0.5012},{"lo":0.7,"w":0.0662,"conf":0.7403,"acc":0.5444},{"lo":0.8,"w":0.0125,"conf":0.8286,"acc":0.549},{"lo":0.9,"w":0.0002,"conf":0.9121,"acc":0.0968}],"areas":{"knowledge":{"ece":0.1143,"acc":0.3838,"conf":0.3843},"language":{"ece":0.1031,"acc":0.4133,"conf":0.5119},"retrieval":{"ece":0.3082,"acc":0.553,"conf":0.2595},"tools":{"ece":0.1054,"acc":0.3014,"conf":0.3827},"arts":{"ece":0.096,"acc":0.3592,"conf":0.4491}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":21.4,"p95":79.4,"mean":50.1},"results":{"1":{"score":0.2149,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":32.1},"2":{"score":0.1208,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":206.4},"3":{"score":0.0315,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":434.5},"4":{"score":0.6099,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":22.5},"5":{"score":0.443,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":24.4},"6":{"score":0.5384,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":46.4},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":50.1},"10":{"score":0.4479,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":48.2},"11":{"score":0.2126,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":79.4},"12":{"score":0.2934,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":18.8},"20":{"score":0.4814,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":18.9},"21":{"score":0.4665,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":18.3},"22":{"score":0.024,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":113.7},"23":{"score":0.5044,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":25.8},"24":{"score":0.316,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":18.6},"25":{"score":0.2449,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":18.9},"26":{"score":0.5059,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":18.5},"27":{"score":0.3166,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":18.5},"28":{"score":0.5036,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":18.3},"29":{"score":0.3455,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":19.0},"30":{"score":0.2002,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":20.1},"31":{"score":0.086,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":22.1},"32":{"score":0.4787,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":21.2},"33":{"score":0.0158,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":29.1},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":19.1},"36":{"score":0.0323,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":242.7},"37":{"score":0.2128,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":23.2},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":101.8},"39":{"score":0.7487,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":20.3},"40":{"score":0.2832,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":18.9,"tracks":[{"track":"A · Arabic","score":0.0679,"headline":false},{"track":"A · English","score":0.2832,"headline":true},{"track":"C · Arabic pairs","score":0.52,"headline":false},{"track":"C · English pairs","score":0.37,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4047,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":18.8},"42":{"score":0.5504,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":26.1},"43":{"score":0.3895,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":20.1},"44":{"score":0.4956,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":18.7},"45":{"score":0.1657,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":19.2},"48":{"score":0.2906,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":32.2},"50":{"score":0.3252,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":28.8},"56":{"score":0.5025,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":24.4,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1423,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":19.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.335,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":20.6,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.2333,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":34.2,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.1875,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":20.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.171,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":33.4,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5677,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":26.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":29.67,"balanced_skill":6.74,"breadth_skill":5.57},"frozen_scores":{"balanced_raw":22.33,"balanced_skill":1.57,"breadth_skill":1.51},"categories":[{"id":"knowledge","raw":0.2554,"skill":0.0315,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.3513,"skill":0.1075,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4023,"skill":0.1827,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.1898,"skill":0.0075,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.2845,"skill":0.0079,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.2149,"skill":0.0,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.1208,"skill":0.0319,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.5385,"skill":0.0,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.4941,"headline":false},{"track":"RouterBench-5shot","score":0.5828,"headline":false}]},"11":{"raw":0.2126,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.4807,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4665,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0204,"skill":0.0127,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.511,"skill":0.0221,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.316,"skill":0.088,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2449,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2002,"skill":0.0318,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2851,"headline":false},{"track":"GSM8K-10choice","score":0.1152,"headline":false}]},"31":{"raw":0.086,"skill":0.0045,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0323,"skill":0.0,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.2128,"skill":0.0127,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2832,"skill":0.0778,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0679,"headline":false},{"track":"A · English","score":0.2832,"headline":true},{"track":"C · Arabic pairs","score":0.52,"headline":false},{"track":"C · English pairs","score":0.37,"headline":false}]},"41":{"raw":0.4047,"skill":0.107,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3895,"skill":0.0313,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4956,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3252,"skill":0.0206,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5025,"skill":0.005,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1423,"skill":0.0353,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.335,"skill":0.0361,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.2333,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.1875,"skill":0.0,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.171,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5677,"skill":0.1354,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4787,"skill":0.1712,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0158,"skill":0.0027,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1657,"skill":0.0019,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.2934,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5036,"skill":0.0072,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3455,"skill":0.1273,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7487,"skill":0.6304,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.5504,"skill":0.1253,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.6099,"skill":0.6049,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.443,"skill":0.4396,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4479,"skill":0.0814,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0315,"skill":0.0128,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"jevfire-uncapped","name":"Jevfire","long_name":"Jevfire Qwen3.8-27B","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/kikoncuo/jevfire","base_model":"Qwen/Qwen3.8-27B","base_url":"https://huggingface.co/Qwen/Qwen3.8-27B","weights_repo":null,"served_checkpoint":"Qwen/Qwen3.8-27B-FP8"},"completed":121057,"counts":{"ok":121057},"lifted":true,"calibration":{"acc":0.6685,"conf":0.7316,"ece":0.0631,"brier":0.4325,"over95":0.0147,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0164,"conf":0.171,"acc":0.0521},{"lo":0.2,"w":0.0325,"conf":0.2552,"acc":0.1665},{"lo":0.3,"w":0.0437,"conf":0.3573,"acc":0.2594},{"lo":0.4,"w":0.097,"conf":0.4404,"acc":0.2962},{"lo":0.5,"w":0.1427,"conf":0.5496,"acc":0.5063},{"lo":0.6,"w":0.0974,"conf":0.6485,"acc":0.5991},{"lo":0.7,"w":0.1013,"conf":0.7494,"acc":0.7231},{"lo":0.8,"w":0.1019,"conf":0.8542,"acc":0.7528},{"lo":0.9,"w":0.3673,"conf":0.9739,"acc":0.9302}],"areas":{"knowledge":{"ece":0.0935,"acc":0.5577,"conf":0.6512},"language":{"ece":0.031,"acc":0.7762,"conf":0.8035},"retrieval":{"ece":0.0794,"acc":0.7266,"conf":0.804},"tools":{"ece":0.0466,"acc":0.9297,"conf":0.9266},"arts":{"ece":0.0845,"acc":0.5374,"conf":0.6201}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":78.1,"p95":735.5,"mean":210.5},"results":{"1":{"score":0.9557,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":191.0},"2":{"score":0.4446,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1534.5},"3":{"score":0.8406,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":746.6},"4":{"score":0.7517,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":117.3},"5":{"score":0.8128,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":267.3},"6":{"score":0.7995,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":309.9},"9":{"score":0.475,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1751.4},"10":{"score":0.6497,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":119.3},"11":{"score":0.7547,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1616.2},"12":{"score":0.6355,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":73.6},"20":{"score":0.9492,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":73.7},"21":{"score":0.5784,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":73.2},"22":{"score":0.132,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":752.3},"23":{"score":0.6046,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":82.6},"24":{"score":0.7971,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":73.5},"25":{"score":0.4694,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":73.3},"26":{"score":0.9848,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":73.6},"27":{"score":0.9599,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":73.2},"28":{"score":0.7451,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":73.8},"29":{"score":0.9046,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":74.2},"30":{"score":0.5489,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":74.8},"31":{"score":0.1732,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":116.1},"32":{"score":0.5944,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":117.6},"33":{"score":0.2752,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":404.7},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":74.8},"36":{"score":0.1969,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1920.4},"37":{"score":0.5351,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":75.5},"38":{"score":0.0125,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":1515.9},"39":{"score":0.8032,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":145.0},"40":{"score":0.5324,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":73.4,"tracks":[{"track":"A · Arabic","score":0.3969,"headline":false},{"track":"A · English","score":0.5324,"headline":true},{"track":"C · Arabic pairs","score":0.72,"headline":false},{"track":"C · English pairs","score":0.935,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6081,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":73.8},"42":{"score":0.7522,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":81.9},"43":{"score":0.5368,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":74.0},"44":{"score":0.6178,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":74.3},"45":{"score":0.1198,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":75.8},"48":{"score":0.2129,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":98.7},"50":{"score":0.395,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":107.1},"56":{"score":0.7435,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":342.7,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5727,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":74.0,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6797,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":74.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7755,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":96.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6875,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":73.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.8026,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":98.8,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.7117,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":81.2,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":59.17,"balanced_skill":45.73,"breadth_skill":44.28},"frozen_scores":{"balanced_raw":43.03,"balanced_skill":30.34,"breadth_skill":26.12},"categories":[{"id":"knowledge","raw":0.4588,"skill":0.3023,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6551,"skill":0.5179,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.6288,"skill":0.4946,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.7152,"skill":0.6449,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.5005,"skill":0.327,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9557,"skill":0.9402,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4446,"skill":0.3884,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7996,"skill":0.5273,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7866,"headline":false},{"track":"RouterBench-5shot","score":0.8125,"headline":false}]},"11":{"raw":0.7547,"skill":0.6453,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9508,"skill":0.9016,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5784,"skill":0.1568,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1287,"skill":0.1219,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6146,"skill":0.2292,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8022,"skill":0.7363,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4694,"skill":0.2925,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.5489,"skill":0.4539,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5959,"headline":false},{"track":"GSM8K-10choice","score":0.5019,"headline":false}]},"31":{"raw":0.1732,"skill":0.0995,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1969,"skill":0.1578,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5351,"skill":0.4169,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5324,"skill":0.3985,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3969,"headline":false},{"track":"A · English","score":0.5324,"headline":true},{"track":"C · Arabic pairs","score":0.72,"headline":false},{"track":"C · English pairs","score":0.935,"headline":false}]},"41":{"raw":0.6081,"skill":0.4121,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.5368,"skill":0.2652,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6178,"skill":0.2356,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.395,"skill":0.1219,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.7435,"skill":0.487,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5727,"skill":0.5194,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6797,"skill":0.5357,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7755,"skill":0.7007,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6875,"skill":0.6094,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.8026,"skill":0.6647,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7117,"skill":0.4234,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5944,"skill":0.3552,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2752,"skill":0.2656,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1198,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6355,"skill":0.454,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7451,"skill":0.4902,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9046,"skill":0.8728,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0125,"skill":0.0125,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8032,"skill":0.7105,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7522,"skill":0.5179,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7517,"skill":0.7485,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8128,"skill":0.8117,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6497,"skill":0.4171,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.8406,"skill":0.8375,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.475,"skill":0.475,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.1484,"skill":0.1484,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"jevk5","name":"JevK5","long_name":"JevK5 Qwen3.5-4B LoRA","variant":null,"meta":{"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":8411558400,"model_url":"https://huggingface.co/alibiserikbay/JevK5","code_url":"https://github.com/allebee/jevk5","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"alibiserikbay/JevK5","served_checkpoint":null},"completed":121057,"counts":{"ok":105740,"unsupported":15317},"lifted":false,"calibration":{"acc":0.6495,"conf":0.6787,"ece":0.0312,"brier":0.4516,"over95":0.0043,"n":32567,"benchmarks":29,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0141,"conf":0.1675,"acc":0.1505},{"lo":0.2,"w":0.0254,"conf":0.2471,"acc":0.2649},{"lo":0.3,"w":0.0962,"conf":0.3585,"acc":0.1928},{"lo":0.4,"w":0.0727,"conf":0.4516,"acc":0.3962},{"lo":0.5,"w":0.1561,"conf":0.5465,"acc":0.5499},{"lo":0.6,"w":0.1455,"conf":0.6466,"acc":0.6396},{"lo":0.7,"w":0.1247,"conf":0.7494,"acc":0.7088},{"lo":0.8,"w":0.1635,"conf":0.8493,"acc":0.8394},{"lo":0.9,"w":0.2018,"conf":0.9467,"acc":0.9353}],"areas":{"knowledge":{"ece":0.0688,"acc":0.5398,"conf":0.5973},"language":{"ece":0.0149,"acc":0.7447,"conf":0.7399},"retrieval":{"ece":0.0836,"acc":0.6391,"conf":0.715},"tools":{"ece":0.078,"acc":0.9085,"conf":0.8348},"arts":{"ece":0.0528,"acc":0.6401,"conf":0.6893}}},"gaps":{"share":0.1265,"reasons":[{"reason":"question 'q1' has 151 options; JevK5 answers with one of 16 letters A..P","share":0.0454},{"reason":"question 'q1' has 77 options; JevK5 answers with one of 16 letters A..P","share":0.0254},{"reason":"question 'chord' has 129 options; JevK5 answers with one of 16 letters A..P","share":0.0165}]},"latency":{"median":21.9,"p95":253.7,"mean":96.1},"results":{"1":{"score":0.8501,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":83.1},"2":{"score":0.3997,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":765.6},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7864,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":247.8},"9":{"score":0.125,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1098.8},"10":{"score":0.6298,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":61.6},"11":{"score":0.669,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1384.7},"12":{"score":0.5513,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":16.0},"20":{"score":0.8404,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":16.0},"21":{"score":0.6088,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":13.1},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.617,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":27.8},"24":{"score":0.7466,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":14.7},"25":{"score":0.4286,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":17.7},"26":{"score":0.9739,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":14.6},"27":{"score":0.9249,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":13.8},"28":{"score":0.7111,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":13.1},"29":{"score":0.8352,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":21.8},"30":{"score":0.4246,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":17.7},"31":{"score":0.3202,"jev":0.3914,"metric":"accuracy","requests":5000,"answered":787,"unsupported":4213,"errors":0,"median_ms":24.4},"32":{"score":0.5186,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":61.6},"33":{"score":0.283,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":195.8},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":17.8},"36":{"score":0.1715,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":927.9},"37":{"score":0.4081,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.6},"38":{"score":0.0075,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":952.5},"39":{"score":0.877,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":31.9},"40":{"score":0.4582,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":13.1,"tracks":[{"track":"A · Arabic","score":0.3427,"headline":false},{"track":"A · English","score":0.4582,"headline":true},{"track":"C · Arabic pairs","score":0.665,"headline":false},{"track":"C · English pairs","score":0.91,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4568,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":16.0},"42":{"score":0.7614,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":28.0},"43":{"score":0.4895,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":16.0},"44":{"score":0.5938,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":16.0},"45":{"score":0.1393,"jev":0.2049,"metric":"accuracy","requests":501,"answered":488,"unsupported":13,"errors":0,"median_ms":21.8},"48":{"score":0.1985,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":42.0},"50":{"score":0.4648,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":42.4},"56":{"score":0.7,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":155.9,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5368,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":21.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6033,"metric":"accuracy","requests":5507,"answered":5332,"unsupported":175,"errors":0,"median_ms":15.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5487,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":41.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5549,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":17.7,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.645,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":41.9,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6222,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":24.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":45.63,"balanced_skill":30.02,"breadth_skill":29.18},"frozen_scores":{"balanced_raw":38.35,"balanced_skill":24.12,"breadth_skill":20.58},"categories":[{"id":"knowledge","raw":0.4045,"skill":0.23,"coverage":0.91,"pending":0,"n":10},{"id":"language","raw":0.5972,"skill":0.4299,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3617,"skill":0.2024,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.4516,"skill":0.3596,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.4665,"skill":0.2789,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8501,"skill":0.7976,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3997,"skill":0.339,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7864,"skill":0.4975,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7677,"headline":false},{"track":"RouterBench-5shot","score":0.805,"headline":false}]},"11":{"raw":0.669,"skill":0.5214,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8436,"skill":0.6872,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6088,"skill":0.2177,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.5872,"skill":0.1745,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.761,"skill":0.6813,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4286,"skill":0.2381,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4246,"skill":0.2996,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4503,"headline":false},{"track":"GSM8K-10choice","score":0.3988,"headline":false}]},"31":{"raw":0.0504,"skill":0.0,"coverage":0.1574,"random":0.0819,"tracks":[]},"36":{"raw":0.1715,"skill":0.1311,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4081,"skill":0.2576,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4582,"skill":0.303,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3427,"headline":false},{"track":"A · English","score":0.4582,"headline":true},{"track":"C · Arabic pairs","score":0.665,"headline":false},{"track":"C · English pairs","score":0.91,"headline":false}]},"41":{"raw":0.4568,"skill":0.1852,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4895,"skill":0.19,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5938,"skill":0.1876,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4648,"skill":0.2232,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.7,"skill":0.4,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5368,"skill":0.479,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5842,"skill":0.3973,"coverage":0.9682,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5487,"skill":0.3983,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5549,"skill":0.4436,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.645,"skill":0.397,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6222,"skill":0.2444,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5186,"skill":0.2347,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.283,"skill":0.2735,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1357,"skill":0.0,"coverage":0.9741,"random":0.1641,"tracks":[]},"12":{"raw":0.5513,"skill":0.3279,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7111,"skill":0.4222,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8352,"skill":0.7803,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0075,"skill":0.0075,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.877,"skill":0.8191,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7614,"skill":0.5358,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.6298,"skill":0.384,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.125,"skill":0.125,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.206,"skill":0.206,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6863,"pending":0.24,"new_in_edition":true},{"engine":"jobe","name":"Jobe","long_name":"Jobe Qwen3.5-4B","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/MantisShrimpdev/jobe","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":null,"served_checkpoint":"Qwen/Qwen3.5-4B"},"completed":121057,"counts":{"ok":121057},"lifted":true,"calibration":{"acc":0.5881,"conf":0.6506,"ece":0.1398,"brier":0.5557,"over95":0.0219,"n":35094,"benchmarks":33,"rel":[{"lo":0.0,"w":0.096,"conf":0.0569,"acc":0.2647},{"lo":0.1,"w":0.043,"conf":0.1358,"acc":0.4986},{"lo":0.2,"w":0.0205,"conf":0.2494,"acc":0.4015},{"lo":0.3,"w":0.0393,"conf":0.3549,"acc":0.2825},{"lo":0.4,"w":0.0531,"conf":0.4491,"acc":0.3432},{"lo":0.5,"w":0.1365,"conf":0.5488,"acc":0.5012},{"lo":0.6,"w":0.1207,"conf":0.6554,"acc":0.4195},{"lo":0.7,"w":0.1011,"conf":0.7512,"acc":0.6157},{"lo":0.8,"w":0.0986,"conf":0.8525,"acc":0.6877},{"lo":0.9,"w":0.2912,"conf":0.9695,"acc":0.8742}],"areas":{"knowledge":{"ece":0.1457,"acc":0.494,"conf":0.6392},"language":{"ece":0.1308,"acc":0.6964,"conf":0.8237},"retrieval":{"ece":0.3404,"acc":0.6107,"conf":0.4449},"tools":{"ece":0.1215,"acc":0.7509,"conf":0.6344},"arts":{"ece":0.046,"acc":0.5032,"conf":0.5405}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":60.2,"p95":4163.4,"mean":561.4},"results":{"1":{"score":0.9298,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":115.7},"2":{"score":0.4067,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1565.4},"3":{"score":0.4606,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":3904.8},"4":{"score":0.6237,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":1980.5},"5":{"score":0.5441,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":4183.2},"6":{"score":0.7952,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":293.1},"9":{"score":0.0875,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1001.0},"10":{"score":0.5657,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":60.8},"11":{"score":0.5924,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1772.8},"12":{"score":0.5453,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":48.1},"20":{"score":0.8516,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":47.8},"21":{"score":0.6046,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":47.2},"22":{"score":0.0515,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":5983.0},"23":{"score":0.5688,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":48.4},"24":{"score":0.7127,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":48.0},"25":{"score":0.3724,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":48.7},"26":{"score":0.9684,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":47.4},"27":{"score":0.9172,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":47.5},"28":{"score":0.6606,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":47.0},"29":{"score":0.8351,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":49.2},"30":{"score":0.3116,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":51.0},"31":{"score":0.134,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":876.0},"32":{"score":0.5439,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":58.1},"33":{"score":0.2018,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":391.1},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":50.2},"36":{"score":0.1647,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1609.0},"37":{"score":0.3581,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":50.8},"38":{"score":0.015,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":3077.0},"39":{"score":0.6581,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":99.0},"40":{"score":0.3896,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":55.9,"tracks":[{"track":"A · Arabic","score":0.2774,"headline":false},{"track":"A · English","score":0.3896,"headline":true},{"track":"C · Arabic pairs","score":0.65,"headline":false},{"track":"C · English pairs","score":0.825,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4366,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":65.8},"42":{"score":0.7185,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":86.9},"43":{"score":0.3912,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":70.6},"44":{"score":0.5638,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":71.4},"45":{"score":0.1138,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":80.0},"48":{"score":0.2248,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":119.1},"50":{"score":0.4427,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":91.4},"56":{"score":0.6,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":435.1,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4364,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":52.8,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5733,"metric":"accuracy","requests":5507,"answered":5332,"unsupported":175,"errors":0,"median_ms":49.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.549,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":69.7,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5814,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":49.8,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5763,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":67.2,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6165,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":51.5,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":47.89,"balanced_skill":30.87,"breadth_skill":29.85},"frozen_scores":{"balanced_raw":36.88,"balanced_skill":21.6,"breadth_skill":18.88},"categories":[{"id":"knowledge","raw":0.3624,"skill":0.1738,"coverage":0.9968,"pending":0,"n":10},{"id":"language","raw":0.5427,"skill":0.3445,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4961,"skill":0.3128,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5382,"skill":0.4508,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.455,"skill":0.2618,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9298,"skill":0.9052,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4067,"skill":0.3467,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7953,"skill":0.5167,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7834,"headline":false},{"track":"RouterBench-5shot","score":0.8071,"headline":false}]},"11":{"raw":0.5924,"skill":0.4105,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8512,"skill":0.7024,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6046,"skill":0.2093,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0494,"skill":0.042,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5552,"skill":0.1104,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7242,"skill":0.6322,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3724,"skill":0.1633,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3116,"skill":0.1686,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4018,"headline":false},{"track":"GSM8K-10choice","score":0.2214,"headline":false}]},"31":{"raw":0.134,"skill":0.0568,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1647,"skill":0.1241,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3581,"skill":0.1948,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.3896,"skill":0.2147,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2774,"headline":false},{"track":"A · English","score":0.3896,"headline":true},{"track":"C · Arabic pairs","score":0.65,"headline":false},{"track":"C · English pairs","score":0.825,"headline":false}]},"41":{"raw":0.4366,"skill":0.1549,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3912,"skill":0.0341,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5638,"skill":0.1276,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4427,"skill":0.1912,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6,"skill":0.2,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4364,"skill":0.3661,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5551,"skill":0.3551,"coverage":0.9682,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.549,"skill":0.3987,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5814,"skill":0.4768,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5763,"skill":0.2803,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6165,"skill":0.233,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5439,"skill":0.2749,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2018,"skill":0.1912,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1138,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5453,"skill":0.3189,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6606,"skill":0.3212,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8351,"skill":0.7801,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.015,"skill":0.015,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6581,"skill":0.4971,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7185,"skill":0.4523,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.6237,"skill":0.6189,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.5441,"skill":0.5413,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5657,"skill":0.2774,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.4606,"skill":0.4502,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0875,"skill":0.0875,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.1008,"skill":0.1008,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"joshua-diffusion-full","name":"JoshuaSP diffusiongemma (open-jev)","long_name":"JoshuaSP diffgemma","variant":null,"meta":{"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/JoshuaSP/open-jev","base_model":"google/diffusiongemma-26B-A4B-it","base_url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it","weights_repo":null,"served_checkpoint":"google/diffusiongemma-26B-A4B-it"},"completed":121057,"counts":{"ok":120742},"lifted":false,"calibration":{"acc":0.6598,"conf":0.8986,"ece":0.2388,"brier":0.5677,"over95":0.1628,"n":72590,"benchmarks":33,"rel":[{"lo":0.0,"w":0.002,"conf":0.0894,"acc":0.0},{"lo":0.1,"w":0.0143,"conf":0.1478,"acc":0.0},{"lo":0.2,"w":0.0117,"conf":0.2486,"acc":0.0881},{"lo":0.3,"w":0.015,"conf":0.3542,"acc":0.1417},{"lo":0.4,"w":0.0185,"conf":0.4495,"acc":0.1841},{"lo":0.5,"w":0.0359,"conf":0.5496,"acc":0.4329},{"lo":0.6,"w":0.0372,"conf":0.6459,"acc":0.4039},{"lo":0.7,"w":0.0422,"conf":0.7526,"acc":0.4219},{"lo":0.8,"w":0.065,"conf":0.8528,"acc":0.5294},{"lo":0.9,"w":0.7582,"conf":0.9876,"acc":0.7524}],"areas":{"knowledge":{"ece":0.3123,"acc":0.5627,"conf":0.875},"language":{"ece":0.1913,"acc":0.7806,"conf":0.9719},"retrieval":{"ece":0.2289,"acc":0.6582,"conf":0.886},"tools":{"ece":0.0448,"acc":0.9328,"conf":0.9743},"arts":{"ece":0.2749,"acc":0.513,"conf":0.788}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":266.1,"p95":791.6,"mean":407.8},"results":{"1":{"score":0.9652,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":275.1},"2":{"score":0.4193,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1542.7},"3":{"score":0.8366,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":1438.7},"4":{"score":0.7428,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":317.1},"5":{"score":0.8349,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":466.6},"6":{"score":0.79,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":491.0},"9":{"score":0.3937,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":879.5},"10":{"score":0.4065,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":293.2},"11":{"score":0.7596,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":637.9},"12":{"score":0.6637,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":256.3},"20":{"score":0.8694,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":257.5},"21":{"score":0.6298,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":250.0},"22":{"score":0.025,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":1210.7},"23":{"score":0.5824,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":261.7},"24":{"score":0.7926,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":252.9},"25":{"score":0.449,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":253.8},"26":{"score":0.9823,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":250.1},"27":{"score":0.9445,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":251.6},"28":{"score":0.7364,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":252.2},"29":{"score":0.8328,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":260.3},"30":{"score":0.5027,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":253.8},"31":{"score":0.1416,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":279.2},"32":{"score":0.6117,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":287.1},"33":{"score":0.2745,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":287.7},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":260.8},"36":{"score":0.1718,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":2211.9},"37":{"score":0.5337,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":262.7},"38":{"score":0.015,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":1671.4},"39":{"score":0.8904,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":262.6},"40":{"score":0.66,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":250.8,"tracks":[{"track":"A · Arabic","score":0.49,"headline":false},{"track":"A · English","score":0.66,"headline":true},{"track":"C · Arabic pairs","score":0.8,"headline":false},{"track":"C · English pairs","score":0.945,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5569,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":257.9},"42":{"score":0.7843,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":267.9},"43":{"score":0.6474,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":246.1},"44":{"score":0.6778,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":256.4},"45":{"score":0.0978,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":260.7},"48":{"score":0.2959,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":278.2},"50":{"score":0.4499,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":274.3},"56":{"score":0.8535,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":282.1,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5686,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":249.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7067,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":247.5,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7544,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":270.6,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6364,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":257.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7043,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":267.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.7085,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":260.3,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":57.69,"balanced_skill":44.16,"breadth_skill":42.67},"frozen_scores":{"balanced_raw":42.75,"balanced_skill":30.31,"breadth_skill":25.64},"categories":[{"id":"knowledge","raw":0.4678,"skill":0.3263,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6603,"skill":0.5228,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.6074,"skill":0.4652,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.6932,"skill":0.6196,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4556,"skill":0.2741,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9652,"skill":0.953,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4193,"skill":0.3606,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.79,"skill":0.5041,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.779,"headline":false},{"track":"RouterBench-5shot","score":0.8011,"headline":false}]},"11":{"raw":0.7596,"skill":0.6523,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8729,"skill":0.7459,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6298,"skill":0.2595,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0263,"skill":0.0187,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5739,"skill":0.1477,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7954,"skill":0.7272,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.449,"skill":0.2653,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.5027,"skill":0.3995,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5686,"headline":false},{"track":"GSM8K-10choice","score":0.4367,"headline":false}]},"31":{"raw":0.1416,"skill":0.0651,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1718,"skill":0.1315,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5337,"skill":0.4152,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.66,"skill":0.5626,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.49,"headline":false},{"track":"A · English","score":0.66,"headline":true},{"track":"C · Arabic pairs","score":0.8,"headline":false},{"track":"C · English pairs","score":0.945,"headline":false}]},"41":{"raw":0.5569,"skill":0.3353,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.6474,"skill":0.4405,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6778,"skill":0.3556,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4499,"skill":0.2016,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.8535,"skill":0.707,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5686,"skill":0.5148,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.7067,"skill":0.5749,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7544,"skill":0.6725,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6364,"skill":0.5455,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7043,"skill":0.4977,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7085,"skill":0.417,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6117,"skill":0.3827,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2745,"skill":0.2649,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.0978,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6637,"skill":0.4963,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7364,"skill":0.4728,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8328,"skill":0.7771,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.015,"skill":0.015,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8904,"skill":0.8388,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7843,"skill":0.5804,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7428,"skill":0.7395,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8349,"skill":0.8339,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4065,"skill":0.0125,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.8366,"skill":0.8335,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.3937,"skill":0.3937,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"kev-0.8b-raised","name":"Kev 0.8B","long_name":"Kev 0.8B","variant":null,"meta":{"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_estimated":false,"params_from_name":false,"trained_bytes":45441727,"model_url":"https://huggingface.co/jaredpalmer/kev-0.8b","code_url":"https://github.com/jaredpalmer/kev","base_model":"Qwen/Qwen3.5-0.8B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-0.8B-Base","weights_repo":"jaredpalmer/kev-0.8b","served_checkpoint":"jaredpalmer/kev-0.8b"},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.4695,"conf":0.5304,"ece":0.0751,"brier":0.6447,"over95":0.0037,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0604,"conf":0.0427,"acc":0.0714},{"lo":0.1,"w":0.0546,"conf":0.1391,"acc":0.2258},{"lo":0.2,"w":0.0448,"conf":0.2587,"acc":0.2172},{"lo":0.3,"w":0.1411,"conf":0.3494,"acc":0.3021},{"lo":0.4,"w":0.0921,"conf":0.4494,"acc":0.4564},{"lo":0.5,"w":0.2022,"conf":0.5478,"acc":0.5181},{"lo":0.6,"w":0.1413,"conf":0.6468,"acc":0.5718},{"lo":0.7,"w":0.1212,"conf":0.7491,"acc":0.598},{"lo":0.8,"w":0.0902,"conf":0.8453,"acc":0.689},{"lo":0.9,"w":0.0521,"conf":0.9347,"acc":0.7352}],"areas":{"knowledge":{"ece":0.0965,"acc":0.3749,"conf":0.4666},"language":{"ece":0.1084,"acc":0.5509,"conf":0.6541},"retrieval":{"ece":0.1392,"acc":0.5533,"conf":0.4649},"tools":{"ece":0.0603,"acc":0.65,"conf":0.6243},"arts":{"ece":0.0817,"acc":0.3749,"conf":0.4566}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":41.2,"p95":108.0,"mean":99.3},"results":{"1":{"score":0.6405,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":78.4},"2":{"score":0.3588,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":188.6},"3":{"score":0.4291,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":111.1},"4":{"score":0.7622,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":43.4},"5":{"score":0.0922,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":46.6},"6":{"score":0.7036,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":88.2},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":94.5},"10":{"score":0.5354,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":84.2},"11":{"score":0.3509,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":96.9},"12":{"score":0.3288,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":39.5},"20":{"score":0.5186,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":39.5},"21":{"score":0.519,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":38.0},"22":{"score":0.0145,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":123.9},"23":{"score":0.5672,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":45.6},"24":{"score":0.4494,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":39.2},"25":{"score":0.3061,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":40.1},"26":{"score":0.8161,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":38.7},"27":{"score":0.6092,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":38.5},"28":{"score":0.5162,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":38.3},"29":{"score":0.4229,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":39.5},"30":{"score":0.1478,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":39.2},"31":{"score":0.0918,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":40.9},"32":{"score":0.4907,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":40.9},"33":{"score":0.0115,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":86.1},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":40.0},"36":{"score":0.1141,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":322.1},"37":{"score":0.209,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":41.7},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":346.1},"39":{"score":0.7012,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":41.6},"40":{"score":0.1585,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":38.7,"tracks":[{"track":"A · Arabic","score":0.0733,"headline":false},{"track":"A · English","score":0.1585,"headline":true},{"track":"C · Arabic pairs","score":0.46,"headline":false},{"track":"C · English pairs","score":0.325,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3962,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":39.1},"42":{"score":0.5482,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":82.1},"43":{"score":0.3456,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":39.9},"44":{"score":0.5218,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":40.2},"45":{"score":0.1377,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":41.5},"48":{"score":0.2366,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":84.9},"50":{"score":0.3168,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":84.2},"56":{"score":0.5005,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":78.5,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.216,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":38.0,"edition_added":"v0.2","jev":0.827},"58":{"score":0.4137,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":37.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3486,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":82.0,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2917,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":39.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.4002,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":80.4,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.505,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":43.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":35.51,"balanced_skill":13.26,"breadth_skill":11.97},"frozen_scores":{"balanced_raw":27.27,"balanced_skill":7.79,"breadth_skill":6.51},"categories":[{"id":"knowledge","raw":0.2683,"skill":0.0588,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.3823,"skill":0.11,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3883,"skill":0.1661,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.4134,"skill":0.2772,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3233,"skill":0.051,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.6405,"skill":0.5147,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3588,"skill":0.294,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7036,"skill":0.305,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6693,"headline":false},{"track":"RouterBench-5shot","score":0.7379,"headline":false}]},"11":{"raw":0.3509,"skill":0.0614,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5112,"skill":0.0223,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.519,"skill":0.0381,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0138,"skill":0.0061,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.557,"skill":0.114,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.46,"skill":0.28,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3061,"skill":0.0748,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1478,"skill":0.0,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2168,"headline":false},{"track":"GSM8K-10choice","score":0.0788,"headline":false}]},"31":{"raw":0.0918,"skill":0.0108,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1141,"skill":0.071,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.209,"skill":0.0078,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.1585,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0733,"headline":false},{"track":"A · English","score":0.1585,"headline":true},{"track":"C · Arabic pairs","score":0.46,"headline":false},{"track":"C · English pairs","score":0.325,"headline":false}]},"41":{"raw":0.3962,"skill":0.0944,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3456,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5218,"skill":0.0436,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3168,"skill":0.0085,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5005,"skill":0.001,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.216,"skill":0.1182,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.4137,"skill":0.1502,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3486,"skill":0.1315,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2917,"skill":0.1146,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.4002,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.505,"skill":0.01,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4907,"skill":0.1903,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0115,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1377,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.3288,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5162,"skill":0.0324,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.4229,"skill":0.2305,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7012,"skill":0.5605,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.5482,"skill":0.121,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7622,"skill":0.7591,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0922,"skill":0.0867,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5354,"skill":0.227,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.4291,"skill":0.4181,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0536,"skill":0.0536,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"kev-4b-raised","name":"Kev 4B","long_name":"Kev 4B","variant":null,"meta":{"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":135172799,"model_url":"https://huggingface.co/jaredpalmer/kev-4b","code_url":"https://github.com/jaredpalmer/kev","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"jaredpalmer/kev-4b","served_checkpoint":"jaredpalmer/kev-4b"},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.5975,"conf":0.7972,"ece":0.1998,"brier":0.5873,"over95":0.0998,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0119,"conf":0.0808,"acc":0.0527},{"lo":0.1,"w":0.0153,"conf":0.1418,"acc":0.1444},{"lo":0.2,"w":0.0166,"conf":0.2604,"acc":0.1604},{"lo":0.3,"w":0.0384,"conf":0.3554,"acc":0.1695},{"lo":0.4,"w":0.0512,"conf":0.4513,"acc":0.2605},{"lo":0.5,"w":0.0837,"conf":0.5466,"acc":0.4282},{"lo":0.6,"w":0.0776,"conf":0.6493,"acc":0.4725},{"lo":0.7,"w":0.0835,"conf":0.7496,"acc":0.5142},{"lo":0.8,"w":0.1119,"conf":0.8536,"acc":0.6469},{"lo":0.9,"w":0.51,"conf":0.9781,"acc":0.7536}],"areas":{"knowledge":{"ece":0.2271,"acc":0.5044,"conf":0.731},"language":{"ece":0.1737,"acc":0.7049,"conf":0.8772},"retrieval":{"ece":0.1279,"acc":0.7092,"conf":0.8362},"tools":{"ece":0.1915,"acc":0.7713,"conf":0.9571},"arts":{"ece":0.2512,"acc":0.4336,"conf":0.6849}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":52.5,"p95":196.8,"mean":254.0},"results":{"1":{"score":0.9227,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":105.4},"2":{"score":0.4259,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":731.7},"3":{"score":0.5118,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":296.1},"4":{"score":0.8224,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":55.8},"5":{"score":0.7936,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":63.6},"6":{"score":0.8007,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":127.2},"9":{"score":0.275,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":179.2},"10":{"score":0.5693,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":107.9},"11":{"score":0.6224,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":155.8},"12":{"score":0.4621,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":51.5},"20":{"score":0.59,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.2},"21":{"score":0.5293,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":50.5},"22":{"score":0.0115,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":317.0},"23":{"score":0.5884,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":57.1},"24":{"score":0.6932,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":52.0},"25":{"score":0.3878,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":52.2},"26":{"score":0.971,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":51.1},"27":{"score":0.8985,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":51.8},"28":{"score":0.6851,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":50.8},"29":{"score":0.7537,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":51.8},"30":{"score":0.4011,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":51.6},"31":{"score":0.0982,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":54.1},"32":{"score":0.5824,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":54.3},"33":{"score":0.2685,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":109.0},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":51.8},"36":{"score":0.1587,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1303.6},"37":{"score":0.3848,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.7},"38":{"score":0.0175,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":313.4},"39":{"score":0.8179,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":53.5},"40":{"score":0.4231,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":51.1,"tracks":[{"track":"A · Arabic","score":0.2552,"headline":false},{"track":"A · English","score":0.4231,"headline":true},{"track":"C · Arabic pairs","score":0.495,"headline":false},{"track":"C · English pairs","score":0.755,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5094,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":51.5},"42":{"score":0.7202,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":104.7},"43":{"score":0.4684,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":51.6},"44":{"score":0.6022,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.4},"45":{"score":0.1118,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":52.1},"48":{"score":0.1859,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":105.6},"50":{"score":0.4177,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":105.8},"56":{"score":0.5005,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":114.2,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4632,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":49.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6098,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":49.7,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4682,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":106.1,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5549,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":49.8,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.3303,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":111.5,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5797,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":52.1,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":49.27,"balanced_skill":31.31,"breadth_skill":30.02},"frozen_scores":{"balanced_raw":36.33,"balanced_skill":20.16,"breadth_skill":16.82},"categories":[{"id":"knowledge","raw":0.3993,"skill":0.2266,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.5342,"skill":0.3417,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5441,"skill":0.3432,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5674,"skill":0.4769,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4187,"skill":0.177,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9227,"skill":0.8956,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4259,"skill":0.3678,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.8007,"skill":0.53,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7879,"headline":false},{"track":"RouterBench-5shot","score":0.8135,"headline":false}]},"11":{"raw":0.6224,"skill":0.4539,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5953,"skill":0.1905,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5293,"skill":0.0586,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0114,"skill":0.0037,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5657,"skill":0.1314,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7,"skill":0.6,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3878,"skill":0.1837,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4011,"skill":0.2783,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4943,"headline":false},{"track":"GSM8K-10choice","score":0.3078,"headline":false}]},"31":{"raw":0.0982,"skill":0.0178,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1587,"skill":0.1178,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3848,"skill":0.2283,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4231,"skill":0.2578,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2552,"headline":false},{"track":"A · English","score":0.4231,"headline":true},{"track":"C · Arabic pairs","score":0.495,"headline":false},{"track":"C · English pairs","score":0.755,"headline":false}]},"41":{"raw":0.5094,"skill":0.2641,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4684,"skill":0.1566,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6022,"skill":0.2044,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4177,"skill":0.1548,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5005,"skill":0.001,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4632,"skill":0.3962,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6098,"skill":0.4344,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4682,"skill":0.2909,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5549,"skill":0.4436,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.3303,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5797,"skill":0.1594,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5824,"skill":0.3361,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2685,"skill":0.2588,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1118,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.4621,"skill":0.1943,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6851,"skill":0.3702,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.7537,"skill":0.6716,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0175,"skill":0.0175,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8179,"skill":0.7322,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7202,"skill":0.4556,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.8224,"skill":0.8201,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.7936,"skill":0.7924,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5693,"skill":0.2834,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.5118,"skill":0.5024,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.275,"skill":0.275,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.2564,"skill":0.2564,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"kev-9b-raised","name":"Kev 9B","long_name":"Kev 9B","variant":null,"meta":{"kind":"LoRA + head","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_estimated":false,"params_from_name":false,"trained_bytes":181572591,"model_url":"https://huggingface.co/jaredpalmer/kev-9b","code_url":"https://github.com/jaredpalmer/kev","base_model":"Qwen/Qwen3.5-9B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-9B-Base","weights_repo":"jaredpalmer/kev-9b","served_checkpoint":"jaredpalmer/kev-9b"},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.6323,"conf":0.7962,"ece":0.1639,"brier":0.5326,"over95":0.0835,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0129,"conf":0.0886,"acc":0.0389},{"lo":0.1,"w":0.0313,"conf":0.138,"acc":0.0733},{"lo":0.2,"w":0.0158,"conf":0.2498,"acc":0.1809},{"lo":0.3,"w":0.0294,"conf":0.3541,"acc":0.2577},{"lo":0.4,"w":0.036,"conf":0.4538,"acc":0.3623},{"lo":0.5,"w":0.0909,"conf":0.5482,"acc":0.4134},{"lo":0.6,"w":0.071,"conf":0.65,"acc":0.4996},{"lo":0.7,"w":0.0865,"conf":0.7519,"acc":0.5815},{"lo":0.8,"w":0.1103,"conf":0.8548,"acc":0.6541},{"lo":0.9,"w":0.5159,"conf":0.9783,"acc":0.7958}],"areas":{"knowledge":{"ece":0.2246,"acc":0.532,"conf":0.7566},"language":{"ece":0.1256,"acc":0.7604,"conf":0.886},"retrieval":{"ece":0.116,"acc":0.7455,"conf":0.8567},"tools":{"ece":0.1603,"acc":0.7877,"conf":0.9434},"arts":{"ece":0.1379,"acc":0.4586,"conf":0.5927}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":52.2,"p95":286.1,"mean":302.7},"results":{"1":{"score":0.9451,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":106.0},"2":{"score":0.4402,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1112.0},"3":{"score":0.563,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":445.8},"4":{"score":0.8483,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":62.6},"5":{"score":0.7903,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":101.0},"6":{"score":0.8005,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":173.0},"9":{"score":0.25,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":233.6},"10":{"score":0.6398,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":121.5},"11":{"score":0.5777,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":232.2},"12":{"score":0.5626,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":51.0},"20":{"score":0.67,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":49.4},"21":{"score":0.5575,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":48.9},"22":{"score":0.1075,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":449.8},"23":{"score":0.5632,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":54.5},"24":{"score":0.7529,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":49.6},"25":{"score":0.3878,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":50.5},"26":{"score":0.9773,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":50.0},"27":{"score":0.9369,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":49.9},"28":{"score":0.7316,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":49.4},"29":{"score":0.8192,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":50.4},"30":{"score":0.4871,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":49.9},"31":{"score":0.1116,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":58.0},"32":{"score":0.5785,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":69.6},"33":{"score":0.2673,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":126.1},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":50.4},"36":{"score":0.1541,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1983.5},"37":{"score":0.4922,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":52.3},"38":{"score":0.035,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":479.5},"39":{"score":0.8838,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":53.1},"40":{"score":0.4708,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":50.8,"tracks":[{"track":"A · Arabic","score":0.3118,"headline":false},{"track":"A · English","score":0.4708,"headline":true},{"track":"C · Arabic pairs","score":0.545,"headline":false},{"track":"C · English pairs","score":0.705,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5542,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":51.4},"42":{"score":0.7486,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":104.9},"43":{"score":0.5123,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":51.4},"44":{"score":0.62,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":51.0},"45":{"score":0.1038,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":51.7},"48":{"score":0.176,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":105.8},"50":{"score":0.3944,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":104.9},"56":{"score":0.5075,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":158.4,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5113,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":50.6,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6517,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":50.1,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4962,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":107.3,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5814,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":50.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.4618,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":110.7,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.588,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":75.5,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":52.31,"balanced_skill":35.41,"breadth_skill":34.3},"frozen_scores":{"balanced_raw":38.53,"balanced_skill":23.2,"breadth_skill":19.57},"categories":[{"id":"knowledge","raw":0.4231,"skill":0.2599,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.5845,"skill":0.4068,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5743,"skill":0.3861,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5825,"skill":0.4953,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4509,"skill":0.2222,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9451,"skill":0.9259,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4402,"skill":0.3836,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.8005,"skill":0.5296,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7873,"headline":false},{"track":"RouterBench-5shot","score":0.8137,"headline":false}]},"11":{"raw":0.5777,"skill":0.3892,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.6722,"skill":0.3444,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5575,"skill":0.1149,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1012,"skill":0.0942,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5539,"skill":0.1079,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7619,"skill":0.6825,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3878,"skill":0.1837,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4871,"skill":0.384,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5853,"headline":false},{"track":"GSM8K-10choice","score":0.3889,"headline":false}]},"31":{"raw":0.1116,"skill":0.0324,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1541,"skill":0.113,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4922,"skill":0.363,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4708,"skill":0.3192,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3118,"headline":false},{"track":"A · English","score":0.4708,"headline":true},{"track":"C · Arabic pairs","score":0.545,"headline":false},{"track":"C · English pairs","score":0.705,"headline":false}]},"41":{"raw":0.5542,"skill":0.3314,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.5123,"skill":0.2262,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.62,"skill":0.24,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3944,"skill":0.1211,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5075,"skill":0.015,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5113,"skill":0.4503,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6517,"skill":0.4951,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4962,"skill":0.3283,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5814,"skill":0.4768,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.4618,"skill":0.0858,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.588,"skill":0.176,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5785,"skill":0.3299,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2673,"skill":0.2576,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1038,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5626,"skill":0.3448,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7316,"skill":0.4632,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8192,"skill":0.7589,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.035,"skill":0.035,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8838,"skill":0.8291,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7486,"skill":0.5109,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.8483,"skill":0.8463,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.7903,"skill":0.789,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6398,"skill":0.4007,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.563,"skill":0.5546,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.25,"skill":0.25,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.296,"skill":0.296,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"laya-1e28ac20","name":"Laya","long_name":"Laya","variant":null,"meta":{"kind":"full fine-tune","technique":"encoder","technique_name":"encoder","served_params":421293830,"params_estimated":false,"params_from_name":false,"trained_bytes":2329053944,"model_url":"https://huggingface.co/convaiinnovations/laya","code_url":"https://github.com/NandhaKishorM/laya","base_model":null,"base_url":null,"weights_repo":"convaiinnovations/laya","served_checkpoint":null},"completed":120943,"counts":{"ok":120616,"unsupported":327},"lifted":false,"calibration":{"acc":0.3958,"conf":0.5114,"ece":0.1323,"brier":0.7161,"over95":0.0113,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.047,"conf":0.0771,"acc":0.0455},{"lo":0.1,"w":0.0363,"conf":0.1433,"acc":0.1199},{"lo":0.2,"w":0.0652,"conf":0.2665,"acc":0.1752},{"lo":0.3,"w":0.2268,"conf":0.345,"acc":0.382},{"lo":0.4,"w":0.1048,"conf":0.4446,"acc":0.3428},{"lo":0.5,"w":0.1966,"conf":0.5458,"acc":0.4187},{"lo":0.6,"w":0.1039,"conf":0.6457,"acc":0.4688},{"lo":0.7,"w":0.0742,"conf":0.7488,"acc":0.503},{"lo":0.8,"w":0.0732,"conf":0.8519,"acc":0.5633},{"lo":0.9,"w":0.072,"conf":0.9455,"acc":0.6364}],"areas":{"knowledge":{"ece":0.1717,"acc":0.3714,"conf":0.4598},"language":{"ece":0.1218,"acc":0.545,"conf":0.6648},"retrieval":{"ece":0.097,"acc":0.2769,"conf":0.3529},"tools":{"ece":0.2475,"acc":0.2407,"conf":0.475},"arts":{"ece":0.1334,"acc":0.3744,"conf":0.5078}}},"gaps":{"share":0.0027,"reasons":[{"reason":"instructions longer than its native budget (no truncation allowed)","share":0.0027},{"reason":"prompt longer than its context window (no truncation allowed)","share":0.0},{"reason":"answer options longer than its per-option budget (no truncation allowed)","share":0.0}]},"latency":{"median":18.4,"p95":61.9,"mean":27.6},"results":{"1":{"score":0.3813,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":20.6},"2":{"score":0.0879,"jev":0.4493,"metric":"nDCG@10","requests":1000,"answered":989,"unsupported":11,"errors":0,"median_ms":178.9},"3":{"score":0.1146,"jev":0.8834,"metric":"accuracy","requests":508,"answered":506,"unsupported":2,"errors":0,"median_ms":103.4},"4":{"score":0.1429,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":21.8},"5":{"score":0.0319,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":24.8},"6":{"score":0.5714,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":37.8},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":167.6},"10":{"score":0.4241,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":21.4},"11":{"score":0.2901,"jev":0.7175,"metric":"macro-F1","requests":123,"answered":122,"unsupported":1,"errors":0,"median_ms":349.9},"12":{"score":0.4866,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":17.4},"20":{"score":0.5158,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":17.5},"21":{"score":0.4722,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":15.7},"22":{"score":0.051,"jev":0.1811,"metric":"accuracy","requests":2000,"answered":1999,"unsupported":1,"errors":0,"median_ms":65.1},"23":{"score":0.5234,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":18.3},"24":{"score":0.3068,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":17.2},"25":{"score":0.2755,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":17.7},"26":{"score":0.4697,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":15.8},"27":{"score":0.2858,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":16.9},"28":{"score":0.5051,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":15.7},"29":{"score":0.3312,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":18.0},"30":{"score":0.2165,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":17.6},"31":{"score":0.0768,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":20.4},"32":{"score":0.4322,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":20.9},"33":{"score":0.003,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":31.7},"34":{"score":0.4,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":17.7},"36":{"score":0.0797,"jev":0.1904,"metric":"nDCG@10","requests":550,"answered":549,"unsupported":1,"errors":0,"median_ms":314.3},"37":{"score":0.244,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":18.5},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":75.7},"39":{"score":0.6098,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":18.6},"40":{"score":0.149,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":15.7,"tracks":[{"track":"A · Arabic","score":0.0184,"headline":false},{"track":"A · English","score":0.149,"headline":true},{"track":"C · Arabic pairs","score":0.535,"headline":false},{"track":"C · English pairs","score":0.41,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4053,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":17.6},"42":{"score":0.4772,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":18.7},"43":{"score":0.4018,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":17.4},"44":{"score":0.5288,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":17.4},"45":{"score":0.14,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":18.0},"48":{"score":0.411,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":19.7},"50":{"score":0.3341,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":20.2},"56":{"score":0.5015,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":22.9,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1361,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":17.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.341,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":17.3,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.1194,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":19.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2708,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":17.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.4882,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":19.7,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5575,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":18.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":28.62,"balanced_skill":5.51,"breadth_skill":5.33},"frozen_scores":{"balanced_raw":24.08,"balanced_skill":3.23,"breadth_skill":3.1},"categories":[{"id":"knowledge","raw":0.2551,"skill":0.0363,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.374,"skill":0.1014,"coverage":0.9992,"pending":0,"n":10},{"id":"retrieval","raw":0.2831,"skill":0.0577,"coverage":0.9997,"pending":0,"n":7},{"id":"tools","raw":0.2122,"skill":0.0503,"coverage":0.9975,"pending":0,"n":6},{"id":"arts","raw":0.3068,"skill":0.0297,"coverage":0.9999,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3813,"skill":0.1648,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.0869,"skill":0.0,"coverage":0.989,"random":0.0918,"tracks":[]},"6":{"raw":0.5714,"skill":0.0397,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.5623,"headline":false},{"track":"RouterBench-5shot","score":0.5805,"headline":false}]},"11":{"raw":0.2875,"skill":0.0,"coverage":0.9919,"random":0.3085,"tracks":[]},"20":{"raw":0.5136,"skill":0.0271,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4722,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0472,"skill":0.0398,"coverage":0.9995,"random":0.0078,"tracks":[]},"23":{"raw":0.5095,"skill":0.019,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.3091,"skill":0.0788,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2755,"skill":0.034,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2165,"skill":0.0506,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2911,"headline":false},{"track":"GSM8K-10choice","score":0.1418,"headline":false}]},"31":{"raw":0.0768,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0796,"skill":0.0348,"coverage":0.9982,"random":0.0464,"tracks":[]},"37":{"raw":0.244,"skill":0.0517,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.149,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0184,"headline":false},{"track":"A · English","score":0.149,"headline":true},{"track":"C · Arabic pairs","score":0.535,"headline":false},{"track":"C · English pairs","score":0.41,"headline":false}]},"41":{"raw":0.4053,"skill":0.108,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4018,"skill":0.0508,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5288,"skill":0.0576,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3341,"skill":0.0336,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5015,"skill":0.003,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1361,"skill":0.0283,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.341,"skill":0.0448,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.1194,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2708,"skill":0.0885,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.4882,"skill":0.1306,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5575,"skill":0.115,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4322,"skill":0.0973,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.003,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1397,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.4866,"skill":0.231,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5051,"skill":0.0102,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3312,"skill":0.1083,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6098,"skill":0.4261,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4772,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.1429,"skill":0.1319,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0319,"skill":0.0261,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.4241,"skill":0.0418,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.1141,"skill":0.0971,"coverage":0.9961,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.7591,"pending":0.24},{"engine":"lfm2600","name":"LFM2.5-2.6B-RLCD","long_name":"LFM2.5-2.6B-RLCD","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":2697198592,"params_estimated":false,"params_from_name":false,"trained_bytes":5394427456,"model_url":"https://huggingface.co/monotykamary/LFM2.5-2.6B-RLCD","code_url":null,"base_model":"LiquidAI/LFM2.5-2.6B-Base","base_url":"https://huggingface.co/LiquidAI/LFM2.5-2.6B-Base","weights_repo":"monotykamary/LFM2.5-2.6B-RLCD","served_checkpoint":null},"completed":121057,"counts":{"ok":120412,"unsupported":645},"lifted":true,"calibration":{"acc":0.3818,"conf":0.653,"ece":0.2712,"brier":0.8152,"over95":0.0493,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0001,"conf":0.0821,"acc":0.0},{"lo":0.1,"w":0.0099,"conf":0.1744,"acc":0.0494},{"lo":0.2,"w":0.053,"conf":0.2555,"acc":0.0885},{"lo":0.3,"w":0.0854,"conf":0.3505,"acc":0.1509},{"lo":0.4,"w":0.1003,"conf":0.4478,"acc":0.2711},{"lo":0.5,"w":0.1781,"conf":0.5506,"acc":0.4199},{"lo":0.6,"w":0.1319,"conf":0.6498,"acc":0.4539},{"lo":0.7,"w":0.135,"conf":0.7478,"acc":0.4475},{"lo":0.8,"w":0.1387,"conf":0.8398,"acc":0.357},{"lo":0.9,"w":0.1676,"conf":0.9643,"acc":0.5483}],"areas":{"knowledge":{"ece":0.2331,"acc":0.3861,"conf":0.6192},"language":{"ece":0.2881,"acc":0.4095,"conf":0.6976},"retrieval":{"ece":0.2867,"acc":0.3718,"conf":0.6584},"tools":{"ece":0.5343,"acc":0.2426,"conf":0.7769},"arts":{"ece":0.1619,"acc":0.413,"conf":0.5749}}},"gaps":{"share":0.0053,"reasons":[{"reason":"request exceeds conservative GPU memory budget; reduce prompt or branch batch size","share":0.0053}]},"latency":{"median":38.8,"p95":99.4,"mean":56.2},"results":{"1":{"score":0.3872,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":43.2},"2":{"score":0.0831,"jev":0.4494,"metric":"nDCG@10","requests":1000,"answered":937,"unsupported":63,"errors":0,"median_ms":316.5},"3":{"score":0.0177,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":159.8},"4":{"score":0.1778,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":66.4},"5":{"score":0.0732,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":94.6},"6":{"score":0.5927,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":69.7},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":107.2},"10":{"score":0.5294,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":48.7},"11":{"score":0.3112,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":114.7},"12":{"score":0.3486,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":38.2},"20":{"score":0.6016,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.1},"21":{"score":0.4981,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":37.8},"22":{"score":0,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":145.6},"23":{"score":0.512,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.3},"24":{"score":0.4916,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":38.2},"25":{"score":0.352,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":38.4},"26":{"score":0.8131,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":38.2},"27":{"score":0.6596,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":38.3},"28":{"score":0.5083,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":37.7},"29":{"score":0.4556,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":38.5},"30":{"score":0.2051,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":39.0},"31":{"score":0.1048,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":55.7},"32":{"score":0.4707,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":50.9},"33":{"score":0.0006,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":53.3},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":39.7},"36":{"score":0.0642,"jev":0.1527,"metric":"nDCG@10","requests":550,"answered":364,"unsupported":186,"errors":0,"median_ms":320.0},"37":{"score":0.2226,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":37.2},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":380.6},"39":{"score":0.3685,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":39.6},"40":{"score":0.2629,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":36.6,"tracks":[{"track":"A · Arabic","score":0.2773,"headline":false},{"track":"A · English","score":0.2629,"headline":true},{"track":"C · Arabic pairs","score":0.5,"headline":false},{"track":"C · English pairs","score":0.55,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3628,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":37.7},"42":{"score":0.3342,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":39.7},"43":{"score":0.3684,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":38.0},"44":{"score":0.5158,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.0},"45":{"score":0.1357,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":39.0},"48":{"score":0.3034,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":42.0},"50":{"score":0.4075,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":47.3},"56":{"score":0.6695,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":43.6,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.2461,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":38.9,"edition_added":"v0.2","jev":0.827},"58":{"score":0.4265,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":36.7,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.2738,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":43.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.2311,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":37.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5177,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":42.0,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5248,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":36.2,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":29.88,"balanced_skill":7.13,"breadth_skill":6.93},"frozen_scores":{"balanced_raw":25.79,"balanced_skill":5.0,"breadth_skill":4.62},"categories":[{"id":"knowledge","raw":0.2826,"skill":0.071,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.347,"skill":0.0667,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.32,"skill":0.1236,"coverage":0.9517,"pending":0,"n":7},{"id":"tools","raw":0.2249,"skill":0.0415,"coverage":0.9895,"pending":0,"n":6},{"id":"arts","raw":0.3194,"skill":0.054,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3872,"skill":0.1728,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.0779,"skill":0.0,"coverage":0.937,"random":0.0918,"tracks":[]},"6":{"raw":0.5927,"skill":0.0442,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.5491,"headline":false},{"track":"RouterBench-5shot","score":0.6363,"headline":false}]},"11":{"raw":0.3112,"skill":0.0039,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5994,"skill":0.1988,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4981,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.4999,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.5022,"skill":0.3363,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.352,"skill":0.1361,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2051,"skill":0.0375,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2866,"headline":false},{"track":"GSM8K-10choice","score":0.1236,"headline":false}]},"31":{"raw":0.1048,"skill":0.025,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0425,"skill":0.0,"coverage":0.6618,"random":0.0464,"tracks":[]},"37":{"raw":0.2226,"skill":0.0249,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2629,"skill":0.0518,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2773,"headline":false},{"track":"A · English","score":0.2629,"headline":true},{"track":"C · Arabic pairs","score":0.5,"headline":false},{"track":"C · English pairs","score":0.55,"headline":false}]},"41":{"raw":0.3628,"skill":0.0442,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3684,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5158,"skill":0.0316,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4075,"skill":0.1401,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6695,"skill":0.339,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.2461,"skill":0.1521,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.4265,"skill":0.1687,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.2738,"skill":0.0317,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.2311,"skill":0.0389,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5177,"skill":0.1807,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5248,"skill":0.0496,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4707,"skill":0.1585,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0006,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1357,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.3486,"skill":0.0243,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5083,"skill":0.0166,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.4556,"skill":0.2741,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.3685,"skill":0.0712,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.3342,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.1778,"skill":0.1672,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0732,"skill":0.0676,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5294,"skill":0.217,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0177,"skill":0.0,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.744,"pending":0.24},{"engine":"lfm350-sdpa","name":"LFM2.5-350M-RLCD","long_name":"LFM2.5-350M-RLCD","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":354483968,"params_estimated":false,"params_from_name":false,"trained_bytes":708984464,"model_url":"https://huggingface.co/notnotsamuel/LFM2.5-350M-RLCD","code_url":null,"base_model":null,"base_url":null,"weights_repo":"notnotsamuel/LFM2.5-350M-RLCD","served_checkpoint":null},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.3004,"conf":0.8497,"ece":0.5495,"brier":1.187,"over95":0.3275,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0002,"conf":0.192,"acc":0.5766},{"lo":0.2,"w":0.0073,"conf":0.262,"acc":0.0486},{"lo":0.3,"w":0.018,"conf":0.3519,"acc":0.0377},{"lo":0.4,"w":0.0306,"conf":0.4506,"acc":0.0665},{"lo":0.5,"w":0.0796,"conf":0.5496,"acc":0.2987},{"lo":0.6,"w":0.0825,"conf":0.6515,"acc":0.3012},{"lo":0.7,"w":0.0916,"conf":0.7502,"acc":0.2712},{"lo":0.8,"w":0.1154,"conf":0.8504,"acc":0.2794},{"lo":0.9,"w":0.5747,"conf":0.9801,"acc":0.3332}],"areas":{"knowledge":{"ece":0.507,"acc":0.3384,"conf":0.8449},"language":{"ece":0.5795,"acc":0.279,"conf":0.8586},"retrieval":{"ece":0.6173,"acc":0.1868,"conf":0.8041},"tools":{"ece":0.5431,"acc":0.2504,"conf":0.7935},"arts":{"ece":0.5467,"acc":0.3684,"conf":0.9152}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":26.0,"p95":175.4,"mean":58.5},"results":{"1":{"score":0.3837,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":28.8},"2":{"score":0.0844,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":300.4},"3":{"score":0.0236,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":146.8},"4":{"score":0.0018,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":91.7},"5":{"score":0.0029,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":172.7},"6":{"score":0.5035,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":49.1},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":83.6},"10":{"score":0.3421,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":28.0},"11":{"score":0.2021,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":109.8},"12":{"score":0.1962,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":25.1},"20":{"score":0.5848,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":25.0},"21":{"score":0.4958,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":24.6},"22":{"score":0.051,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":266.2},"23":{"score":0.4814,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":25.4},"24":{"score":0.2389,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":25.1},"25":{"score":0.2704,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":25.2},"26":{"score":0.2647,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":24.7},"27":{"score":0.2304,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":24.8},"28":{"score":0.5201,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":24.2},"29":{"score":0.2515,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":25.8},"30":{"score":0.1827,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":25.5},"31":{"score":0.0796,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":46.3},"32":{"score":0.3617,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":28.2},"33":{"score":0.0133,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":33.5},"34":{"score":0.4,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":25.7},"36":{"score":0.0441,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":297.5},"37":{"score":0.1238,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":26.3},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":389.1},"39":{"score":0.3315,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":26.5},"40":{"score":0.2495,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":24.4,"tracks":[{"track":"A · Arabic","score":0.2505,"headline":false},{"track":"A · English","score":0.2495,"headline":true},{"track":"C · Arabic pairs","score":0.51,"headline":false},{"track":"C · English pairs","score":0.455,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.2532,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":25.1},"42":{"score":0.4288,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":25.9},"43":{"score":0.3228,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":24.9},"44":{"score":0.4974,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":24.5},"45":{"score":0.1357,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":25.9},"48":{"score":0.6215,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":26.7},"50":{"score":0.3037,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":29.4},"56":{"score":0.5,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":45.0,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1169,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":25.9,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3018,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":24.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.1525,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":27.3,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.1875,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":25.4,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5187,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":25.7,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5008,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":25.5,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":24.59,"balanced_skill":1.83,"breadth_skill":1.76},"frozen_scores":{"balanced_raw":21.83,"balanced_skill":1.77,"breadth_skill":1.66},"categories":[{"id":"knowledge","raw":0.2282,"skill":0.0044,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.2952,"skill":0.0276,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2165,"skill":0.0002,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.1913,"skill":0.0288,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.2986,"skill":0.0306,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3837,"skill":0.168,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.0844,"skill":0.0,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.5036,"skill":0.0,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.41,"headline":false},{"track":"RouterBench-5shot","score":0.5972,"headline":false}]},"11":{"raw":0.2021,"skill":0.0,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5865,"skill":0.1731,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.4958,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0485,"skill":0.0411,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.4679,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.2408,"skill":0.0,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2704,"skill":0.0272,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.1827,"skill":0.0096,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.2593,"headline":false},{"track":"GSM8K-10choice","score":0.1061,"headline":false}]},"31":{"raw":0.0796,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0441,"skill":0.0,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.1238,"skill":0.0,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2495,"skill":0.0345,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2505,"headline":false},{"track":"A · English","score":0.2495,"headline":true},{"track":"C · Arabic pairs","score":0.51,"headline":false},{"track":"C · English pairs","score":0.455,"headline":false}]},"41":{"raw":0.2532,"skill":0.0,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3228,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.4974,"skill":0.0,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3037,"skill":0.0,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1169,"skill":0.0067,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3018,"skill":0.0,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.1525,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.1875,"skill":0.0,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5187,"skill":0.1824,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5008,"skill":0.0016,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3617,"skill":0.0,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0133,"skill":0.0002,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1357,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.1962,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5201,"skill":0.0402,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.2515,"skill":0.002,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.3315,"skill":0.0168,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.4288,"skill":0.0,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0018,"skill":0.0,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0029,"skill":0.0,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.3421,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0236,"skill":0.0048,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"metask-jev-4b","name":"Metask-Jev-4B","long_name":"Metask-Jev-4B","variant":null,"meta":{"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":9078620536,"model_url":"https://huggingface.co/wayfind/metask-jev-4b-policy-mix","code_url":"https://github.com/metask-ai/metask-jev","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"wayfind/metask-jev-4b-policy-mix","served_checkpoint":null},"completed":121057,"counts":{"ok":106557,"unsupported":14500},"lifted":false,"calibration":{"acc":0.6026,"conf":0.7176,"ece":0.118,"brier":0.5202,"over95":0.0038,"n":32699,"benchmarks":29,"rel":[{"lo":0.0,"w":0.0098,"conf":0.0831,"acc":0.093},{"lo":0.1,"w":0.0243,"conf":0.1511,"acc":0.1364},{"lo":0.2,"w":0.0283,"conf":0.2593,"acc":0.309},{"lo":0.3,"w":0.0464,"conf":0.3518,"acc":0.2295},{"lo":0.4,"w":0.0541,"conf":0.4503,"acc":0.3927},{"lo":0.5,"w":0.1252,"conf":0.545,"acc":0.3722},{"lo":0.6,"w":0.0946,"conf":0.6512,"acc":0.5359},{"lo":0.7,"w":0.1395,"conf":0.7533,"acc":0.5933},{"lo":0.8,"w":0.2201,"conf":0.8581,"acc":0.6833},{"lo":0.9,"w":0.2577,"conf":0.9362,"acc":0.8817}],"areas":{"knowledge":{"ece":0.1272,"acc":0.497,"conf":0.6173},"language":{"ece":0.1375,"acc":0.6844,"conf":0.8195},"retrieval":{"ece":0.1787,"acc":0.5892,"conf":0.7643},"tools":{"ece":0.0495,"acc":0.9139,"conf":0.9008},"arts":{"ece":0.0894,"acc":0.5862,"conf":0.6742}}},"gaps":{"share":0.1198,"reasons":[{"reason":"metask serve.py HTTP 500: decision: enum choices must be 1–26 nonempty strings.","share":0.1198}]},"latency":{"median":56.1,"p95":407.6,"mean":194.1},"results":{"1":{"score":0.8666,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":126.3},"2":{"score":0.4499,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1610.7},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7962,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":278.2},"9":{"score":0.1938,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1034.2},"10":{"score":0.5765,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":66.0},"11":{"score":0.6933,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":1676.3},"12":{"score":0.5625,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":51.6},"20":{"score":0.7848,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":50.9},"21":{"score":0.5209,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":50.0},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.5788,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":52.4},"24":{"score":0.6954,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":51.5},"25":{"score":0.4082,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":52.9},"26":{"score":0.9541,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":51.1},"27":{"score":0.8993,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":51.0},"28":{"score":0.6709,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":50.4},"29":{"score":0.7528,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":53.2},"30":{"score":0.3446,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":55.9},"31":{"score":0.2305,"jev":0.2746,"metric":"accuracy","requests":5000,"answered":1588,"unsupported":3412,"errors":0,"median_ms":70.0},"32":{"score":0.5479,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":61.7},"33":{"score":0.1709,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":419.4},"34":{"score":0.3,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":54.1},"36":{"score":0.1611,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1722.9},"37":{"score":0.3488,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":54.9},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":3242.7},"39":{"score":0.793,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":108.1},"40":{"score":0.4762,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":58.1,"tracks":[{"track":"A · Arabic","score":0.1942,"headline":false},{"track":"A · English","score":0.4762,"headline":true},{"track":"C · Arabic pairs","score":0.565,"headline":false},{"track":"C · English pairs","score":0.805,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4303,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":69.1},"42":{"score":0.7341,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":115.1},"43":{"score":0.4632,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":82.3},"44":{"score":0.6172,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":79.8},"45":{"score":0.0838,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":101.5},"48":{"score":0.2699,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":143.4},"50":{"score":0.4004,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":132.8},"56":{"score":0.516,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":438.4,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4522,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":56.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6065,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":52.4,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4285,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":79.0,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.553,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":53.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.6638,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":75.2,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.624,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":51.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":42.64,"balanced_skill":25.59,"breadth_skill":24.09},"frozen_scores":{"balanced_raw":36.55,"balanced_skill":21.16,"breadth_skill":18.0},"categories":[{"id":"knowledge","raw":0.3768,"skill":0.2059,"coverage":0.9318,"pending":0,"n":10},{"id":"language","raw":0.5777,"skill":0.3992,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3181,"skill":0.1255,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.4558,"skill":0.3609,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.4037,"skill":0.1881,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8666,"skill":0.8199,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4499,"skill":0.3943,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7962,"skill":0.5194,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7825,"headline":false},{"track":"RouterBench-5shot","score":0.8099,"headline":false}]},"11":{"raw":0.6933,"skill":0.5565,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.783,"skill":0.566,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5209,"skill":0.0419,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.5688,"skill":0.1376,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7027,"skill":0.6036,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4082,"skill":0.2109,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3446,"skill":0.211,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4534,"headline":false},{"track":"GSM8K-10choice","score":0.2358,"headline":false}]},"31":{"raw":0.0732,"skill":0.0,"coverage":0.3176,"random":0.0819,"tracks":[]},"36":{"raw":0.1611,"skill":0.1202,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3488,"skill":0.1832,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4762,"skill":0.3261,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.1942,"headline":false},{"track":"A · English","score":0.4762,"headline":true},{"track":"C · Arabic pairs","score":0.565,"headline":false},{"track":"C · English pairs","score":0.805,"headline":false}]},"41":{"raw":0.4303,"skill":0.1454,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4632,"skill":0.1482,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6172,"skill":0.2344,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4004,"skill":0.1297,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.516,"skill":0.032,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4522,"skill":0.3839,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6065,"skill":0.4296,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4285,"skill":0.238,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.553,"skill":0.4413,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.6638,"skill":0.4289,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.624,"skill":0.248,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5479,"skill":0.2812,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.1709,"skill":0.1599,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.0838,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5625,"skill":0.3447,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6709,"skill":0.3418,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.7528,"skill":0.6704,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.793,"skill":0.6955,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7341,"skill":0.4827,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.5765,"skill":0.2953,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.1938,"skill":0.1938,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6927,"pending":0.24,"new_in_edition":true},{"engine":"mini-jev","name":"mini-jev","long_name":"mini-jev Qwen3-4B","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4022468096,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/r-ms/mini-jev","base_model":"Qwen/Qwen3-4B-Instruct-2507","base_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","weights_repo":null,"served_checkpoint":"Qwen/Qwen3-4B-Instruct-2507"},"completed":121057,"counts":{"ok":106557,"unsupported":14500},"lifted":false,"calibration":{"acc":0.6176,"conf":0.9482,"ece":0.3305,"brier":0.7012,"over95":0.264,"n":70200,"benchmarks":29,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0,"conf":null,"acc":null},{"lo":0.2,"w":0.0008,"conf":0.2684,"acc":0.0},{"lo":0.3,"w":0.0018,"conf":0.3573,"acc":0.081},{"lo":0.4,"w":0.0065,"conf":0.4561,"acc":0.3247},{"lo":0.5,"w":0.0279,"conf":0.5543,"acc":0.3071},{"lo":0.6,"w":0.0292,"conf":0.6506,"acc":0.4033},{"lo":0.7,"w":0.0352,"conf":0.7533,"acc":0.3752},{"lo":0.8,"w":0.0471,"conf":0.8553,"acc":0.3883},{"lo":0.9,"w":0.8516,"conf":0.9901,"acc":0.6618}],"areas":{"knowledge":{"ece":0.3549,"acc":0.5698,"conf":0.9247},"language":{"ece":0.3249,"acc":0.6478,"conf":0.9726},"retrieval":{"ece":0.3419,"acc":0.6439,"conf":0.9857},"tools":{"ece":0.1914,"acc":0.7885,"conf":0.9798},"arts":{"ece":0.3342,"acc":0.5947,"conf":0.9289}}},"gaps":{"share":0.1198,"reasons":[{"reason":"supports only 2–26 answer options per question","share":0.1198},{"reason":"ran out of GPU memory on the longest prompts","share":0.0}]},"latency":{"median":64.1,"p95":195.8,"mean":153.3},"results":{"1":{"score":0.7739,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":70.0},"2":{"score":0.3076,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":920.5},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7938,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":147.8},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":241.2},"10":{"score":0.5875,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":78.7},"11":{"score":0.5444,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":379.9},"12":{"score":0.5194,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":63.2},"20":{"score":0.8098,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":63.2},"21":{"score":0.5822,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":62.4},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.5418,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":64.3},"24":{"score":0.6734,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":63.0},"25":{"score":0.3061,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":63.4},"26":{"score":0.9579,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":62.5},"27":{"score":0.884,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":62.8},"28":{"score":0.5951,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":62.7},"29":{"score":0.7721,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":63.8},"30":{"score":0.4814,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":63.8},"31":{"score":0.1688,"jev":0.2746,"metric":"accuracy","requests":5000,"answered":1588,"unsupported":3412,"errors":0,"median_ms":65.2},"32":{"score":0.5957,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":83.0},"33":{"score":0.2079,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":85.7},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":63.5},"36":{"score":0.1318,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1535.6},"37":{"score":0.2921,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":64.8},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":362.9},"39":{"score":0.6397,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":64.5},"40":{"score":0.3613,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":62.6,"tracks":[{"track":"A · Arabic","score":0.2807,"headline":false},{"track":"A · English","score":0.3613,"headline":true},{"track":"C · Arabic pairs","score":0.75,"headline":false},{"track":"C · English pairs","score":0.865,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4306,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":62.7},"42":{"score":0.7075,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":65.9},"43":{"score":0.3877,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":63.4},"44":{"score":0.5442,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":62.8},"45":{"score":0.1198,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":64.1},"48":{"score":0.4745,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":70.4},"50":{"score":0.4684,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":69.8},"56":{"score":0.5125,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":80.2,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4169,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":62.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5141,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":62.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5027,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":67.2,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5549,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":62.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.0,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":71.4,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5723,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":64.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":38.73,"balanced_skill":21.22,"breadth_skill":20.36},"frozen_scores":{"balanced_raw":34.82,"balanced_skill":19.0,"breadth_skill":16.54},"categories":[{"id":"knowledge","raw":0.3627,"skill":0.1761,"coverage":0.9318,"pending":0,"n":10},{"id":"language","raw":0.457,"skill":0.2733,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.2995,"skill":0.0978,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.3963,"skill":0.2972,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.4209,"skill":0.2168,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.7739,"skill":0.6948,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3076,"skill":0.2376,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7938,"skill":0.5141,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7791,"headline":false},{"track":"RouterBench-5shot","score":0.8085,"headline":false}]},"11":{"raw":0.5444,"skill":0.3411,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8024,"skill":0.6049,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5822,"skill":0.1644,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.5381,"skill":0.0761,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7019,"skill":0.6026,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3061,"skill":0.0748,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4814,"skill":0.3748,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5588,"headline":false},{"track":"GSM8K-10choice","score":0.4041,"headline":false}]},"31":{"raw":0.0536,"skill":0.0,"coverage":0.3176,"random":0.0819,"tracks":[]},"36":{"raw":0.1318,"skill":0.0896,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.2921,"skill":0.112,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.3613,"skill":0.1783,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.2807,"headline":false},{"track":"A · English","score":0.3613,"headline":true},{"track":"C · Arabic pairs","score":0.75,"headline":false},{"track":"C · English pairs","score":0.865,"headline":false}]},"41":{"raw":0.4306,"skill":0.1459,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3877,"skill":0.0286,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5442,"skill":0.0884,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4684,"skill":0.2284,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5125,"skill":0.025,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4169,"skill":0.3442,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5141,"skill":0.2957,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5027,"skill":0.3369,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5549,"skill":0.4436,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5723,"skill":0.1446,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5957,"skill":0.3572,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2079,"skill":0.1974,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1198,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5194,"skill":0.2801,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5951,"skill":0.1902,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.7721,"skill":0.6961,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6397,"skill":0.4701,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7075,"skill":0.4309,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.5875,"skill":0.3136,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6927,"pending":0.24},{"engine":"mojev","name":"MoJev","long_name":"MoJev Qwen3.5-0.8B","variant":null,"meta":{"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":873438784,"params_estimated":false,"params_from_name":false,"trained_bytes":1965507264,"model_url":"https://huggingface.co/MoLeMo-Lab/mojev","code_url":"https://github.com/MoLeMo-Lab/mojev","base_model":"Qwen/Qwen3.5-0.8B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-0.8B-Base","weights_repo":"MoLeMo-Lab/mojev","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.4117,"conf":0.4869,"ece":0.1334,"brier":0.7146,"over95":0.0104,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.1424,"conf":0.0384,"acc":0.2167},{"lo":0.1,"w":0.0215,"conf":0.1534,"acc":0.2875},{"lo":0.2,"w":0.0302,"conf":0.2622,"acc":0.2293},{"lo":0.3,"w":0.1135,"conf":0.3556,"acc":0.3627},{"lo":0.4,"w":0.1488,"conf":0.4402,"acc":0.307},{"lo":0.5,"w":0.2105,"conf":0.5477,"acc":0.5239},{"lo":0.6,"w":0.1414,"conf":0.6463,"acc":0.5669},{"lo":0.7,"w":0.089,"conf":0.7464,"acc":0.5235},{"lo":0.8,"w":0.0599,"conf":0.8488,"acc":0.462},{"lo":0.9,"w":0.043,"conf":0.943,"acc":0.3776}],"areas":{"knowledge":{"ece":0.1119,"acc":0.3653,"conf":0.4725},"language":{"ece":0.1538,"acc":0.4506,"conf":0.5896},"retrieval":{"ece":0.22,"acc":0.523,"conf":0.3338},"tools":{"ece":0.272,"acc":0.3693,"conf":0.5089},"arts":{"ece":0.0679,"acc":0.3893,"conf":0.4463}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":41.1,"p95":68.0,"mean":49.1},"results":{"1":{"score":0.5372,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":41.0},"2":{"score":0.3273,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":190.3},"3":{"score":0.2913,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":156.2},"4":{"score":0.5187,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":40.5},"5":{"score":0.173,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":41.6},"6":{"score":0.5969,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":47.1},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":55.9},"10":{"score":0.5946,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":43.4},"11":{"score":0.3317,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":72.9},"12":{"score":0.2477,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":39.6},"20":{"score":0.5394,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":39.6},"21":{"score":0.5331,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":39.2},"22":{"score":0.0235,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":115.3},"23":{"score":0.5334,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":40.7},"24":{"score":0.4035,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":39.7},"25":{"score":0.2602,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":39.8},"26":{"score":0.7513,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":39.2},"27":{"score":0.5606,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":39.3},"28":{"score":0.5162,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":39.5},"29":{"score":0.3936,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":39.9},"30":{"score":0.2968,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":39.7},"31":{"score":0.0704,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":40.8},"32":{"score":0.4827,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":44.0},"33":{"score":0.0127,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":44.1},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":40.2},"36":{"score":0.1015,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":298.4},"37":{"score":0.2541,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":41.4},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":172.8},"39":{"score":0.5949,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":40.7},"40":{"score":0.2311,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":40.0,"tracks":[{"track":"A · Arabic","score":0.1493,"headline":false},{"track":"A · English","score":0.2311,"headline":true},{"track":"C · Arabic pairs","score":0.42,"headline":false},{"track":"C · English pairs","score":0.41,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4089,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":39.9},"42":{"score":0.5531,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":41.4},"43":{"score":0.3439,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":39.6},"44":{"score":0.5046,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":39.6},"45":{"score":0.1517,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":40.6},"48":{"score":0.3457,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":42.4},"50":{"score":0.3013,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":43.8},"56":{"score":0.5,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":41.0,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1765,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":39.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3966,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":39.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.0969,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":41.5,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.4015,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":39.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.0158,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":41.9,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.4975,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":40.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":32.36,"balanced_skill":10.9,"breadth_skill":10.42},"frozen_scores":{"balanced_raw":26.74,"balanced_skill":6.52,"breadth_skill":5.89},"categories":[{"id":"knowledge","raw":0.2696,"skill":0.0559,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.3293,"skill":0.0916,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3771,"skill":0.1612,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.3083,"skill":0.1673,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3336,"skill":0.0691,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.5372,"skill":0.3752,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3273,"skill":0.2593,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.5969,"skill":0.0914,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6114,"headline":false},{"track":"RouterBench-5shot","score":0.5824,"headline":false}]},"11":{"raw":0.3317,"skill":0.0335,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5372,"skill":0.0744,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5331,"skill":0.0662,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0259,"skill":0.0183,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5364,"skill":0.0729,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.4139,"skill":0.2186,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.2602,"skill":0.0136,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2968,"skill":0.159,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4632,"headline":false},{"track":"GSM8K-10choice","score":0.1304,"headline":false}]},"31":{"raw":0.0704,"skill":0.0,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1015,"skill":0.0578,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.2541,"skill":0.0645,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.2311,"skill":0.0108,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.1493,"headline":false},{"track":"A · English","score":0.2311,"headline":true},{"track":"C · Arabic pairs","score":0.42,"headline":false},{"track":"C · English pairs","score":0.41,"headline":false}]},"41":{"raw":0.4089,"skill":0.1134,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3439,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5046,"skill":0.0092,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3013,"skill":0.0,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.1765,"skill":0.0738,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3966,"skill":0.1254,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.0969,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.4015,"skill":0.2519,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.0158,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.4975,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.4827,"skill":0.1776,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.0127,"skill":0.0,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1517,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.2477,"skill":0.0,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.5162,"skill":0.0324,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.3936,"skill":0.1915,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.5949,"skill":0.4042,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.5531,"skill":0.1305,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.5187,"skill":0.5125,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.173,"skill":0.168,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5946,"skill":0.3255,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.2913,"skill":0.2776,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"nimble-v2","name":"Bespoke Nimble 9B v2","long_name":"Bespoke Nimble 9B v2","variant":null,"meta":{"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":9653104368,"params_estimated":false,"params_from_name":false,"trained_bytes":173188512,"model_url":"https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v2","code_url":"https://github.com/bespokelabsai/nimble","base_model":"Qwen/Qwen3.5-9B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-9B-Base","weights_repo":"bespokelabs/Bespoke-Nimble-9B-v2","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.6231,"conf":0.6513,"ece":0.0463,"brier":0.4963,"over95":0.0019,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.045,"conf":0.066,"acc":0.1059},{"lo":0.1,"w":0.0265,"conf":0.1453,"acc":0.2065},{"lo":0.2,"w":0.031,"conf":0.2583,"acc":0.3046},{"lo":0.3,"w":0.0606,"conf":0.3532,"acc":0.3399},{"lo":0.4,"w":0.0677,"conf":0.4535,"acc":0.4014},{"lo":0.5,"w":0.1527,"conf":0.5492,"acc":0.5765},{"lo":0.6,"w":0.1178,"conf":0.6511,"acc":0.6422},{"lo":0.7,"w":0.1595,"conf":0.7541,"acc":0.5829},{"lo":0.8,"w":0.1708,"conf":0.8557,"acc":0.8434},{"lo":0.9,"w":0.1683,"conf":0.9348,"acc":0.9201}],"areas":{"knowledge":{"ece":0.0959,"acc":0.5407,"conf":0.6107},"language":{"ece":0.0307,"acc":0.7503,"conf":0.7524},"retrieval":{"ece":0.0992,"acc":0.5737,"conf":0.5898},"tools":{"ece":0.0441,"acc":0.8315,"conf":0.7935},"arts":{"ece":0.0532,"acc":0.506,"conf":0.5309}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":75.9,"p95":583.8,"mean":1126.4},"results":{"1":{"score":0.9156,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":207.5},"2":{"score":0.3969,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":29359.4},"3":{"score":0.8051,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":574.4},"4":{"score":0.6938,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":142.6},"5":{"score":0.3084,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":253.0},"6":{"score":0.7831,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":436.5},"9":{"score":0.0125,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":4134.3},"10":{"score":0.5992,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":96.7},"11":{"score":0.7032,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":5287.5},"12":{"score":0.6819,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":73.7},"20":{"score":0.8666,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":73.0},"21":{"score":0.5753,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":72.1},"22":{"score":0.1465,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":564.7},"23":{"score":0.543,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":74.7},"24":{"score":0.7724,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":73.9},"25":{"score":0.4388,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":74.5},"26":{"score":0.9811,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":73.6},"27":{"score":0.9411,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":73.6},"28":{"score":0.7561,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":72.2},"29":{"score":0.8499,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":75.1},"30":{"score":0.5565,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":76.3},"31":{"score":0.1362,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":115.1},"32":{"score":0.5638,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":98.1},"33":{"score":0.2352,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":708.8},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":76.5},"36":{"score":0.1126,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":34061.8},"37":{"score":0.4553,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":76.9},"38":{"score":0.0175,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":45954.8},"39":{"score":0.6547,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":147.4},"40":{"score":0.5372,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":72.8,"tracks":[{"track":"A · Arabic","score":0.1293,"headline":false},{"track":"A · English","score":0.5372,"headline":true},{"track":"C · Arabic pairs","score":0.685,"headline":false},{"track":"C · English pairs","score":0.875,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3913,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":74.1},"42":{"score":0.7633,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":76.1},"43":{"score":0.5632,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":73.8},"44":{"score":0.6692,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":73.2},"45":{"score":0.0858,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":77.1},"48":{"score":0.2239,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":81.7},"50":{"score":0.4039,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":95.1},"56":{"score":0.656,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":751.3,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5172,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":76.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6744,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":72.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7267,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":81.7,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5568,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":74.0,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.6507,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":85.5,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6295,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":74.5,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":51.96,"balanced_skill":36.68,"breadth_skill":35.66},"frozen_scores":{"balanced_raw":39.9,"balanced_skill":25.93,"breadth_skill":22.2},"categories":[{"id":"knowledge","raw":0.444,"skill":0.2941,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6006,"skill":0.4354,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4935,"skill":0.3263,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.6067,"skill":0.527,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4533,"skill":0.2511,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9156,"skill":0.886,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3969,"skill":0.336,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7832,"skill":0.4907,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7612,"headline":false},{"track":"RouterBench-5shot","score":0.8051,"headline":false}]},"11":{"raw":0.7032,"skill":0.5708,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8624,"skill":0.7248,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5753,"skill":0.1507,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1363,"skill":0.1295,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5339,"skill":0.0677,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7844,"skill":0.7125,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4388,"skill":0.2517,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.5565,"skill":0.4679,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.6467,"headline":false},{"track":"GSM8K-10choice","score":0.4663,"headline":false}]},"31":{"raw":0.1362,"skill":0.0592,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1126,"skill":0.0694,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4553,"skill":0.3168,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5372,"skill":0.4047,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.1293,"headline":false},{"track":"A · English","score":0.5372,"headline":true},{"track":"C · Arabic pairs","score":0.685,"headline":false},{"track":"C · English pairs","score":0.875,"headline":false}]},"41":{"raw":0.3913,"skill":0.087,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.5632,"skill":0.3069,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6692,"skill":0.3384,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4039,"skill":0.1349,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.656,"skill":0.312,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5172,"skill":0.457,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6744,"skill":0.528,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7267,"skill":0.6356,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5568,"skill":0.446,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.6507,"skill":0.4067,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6295,"skill":0.259,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5638,"skill":0.3065,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2352,"skill":0.225,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.0858,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6819,"skill":0.5235,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7561,"skill":0.5122,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8499,"skill":0.7999,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0175,"skill":0.0175,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6547,"skill":0.4921,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7633,"skill":0.5395,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.6938,"skill":0.6899,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.3084,"skill":0.3042,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5992,"skill":0.3331,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.8051,"skill":0.8013,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0125,"skill":0.0125,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.1044,"skill":0.1044,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"openvons","name":"openvons","long_name":"openvons Qwen3-4B","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4022468096,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/genai-craft/openvons","base_model":"Qwen/Qwen3-4B-Instruct-2507","base_url":"https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507","weights_repo":null,"served_checkpoint":"Qwen/Qwen3-4B-Instruct-2507"},"completed":121057,"counts":{"ok":121057},"lifted":false,"calibration":{"acc":0.5956,"conf":0.9527,"ece":0.3571,"brier":0.7519,"over95":0.2958,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0,"conf":null,"acc":null},{"lo":0.2,"w":0.0007,"conf":0.2424,"acc":0.0},{"lo":0.3,"w":0.0025,"conf":0.3383,"acc":0.0431},{"lo":0.4,"w":0.0112,"conf":0.4512,"acc":0.1647},{"lo":0.5,"w":0.0216,"conf":0.5416,"acc":0.3145},{"lo":0.6,"w":0.0268,"conf":0.6423,"acc":0.2856},{"lo":0.7,"w":0.0246,"conf":0.7429,"acc":0.3081},{"lo":0.8,"w":0.0374,"conf":0.8415,"acc":0.2981},{"lo":0.9,"w":0.8751,"conf":0.9917,"acc":0.6405}],"areas":{"knowledge":{"ece":0.3446,"acc":0.5861,"conf":0.9307},"language":{"ece":0.3203,"acc":0.6607,"conf":0.981},"retrieval":{"ece":0.4285,"acc":0.54,"conf":0.9685},"tools":{"ece":0.28,"acc":0.7086,"conf":0.9886},"arts":{"ece":0.4422,"acc":0.478,"conf":0.9202}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":17.3,"p95":153.2,"mean":50.1},"results":{"1":{"score":0.8406,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":37.9},"2":{"score":0.4132,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":236.0},"3":{"score":0.435,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":337.5},"4":{"score":0.3255,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":73.2},"5":{"score":0.4172,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":100.6},"6":{"score":0.798,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":81.0},"9":{"score":0.0688,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":113.6},"10":{"score":0.6232,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":37.2},"11":{"score":0.5355,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":192.0},"12":{"score":0.5379,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":16.1},"20":{"score":0.7622,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":13.6},"21":{"score":0.5906,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":12.6},"22":{"score":0.056,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":283.6},"23":{"score":0.5326,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":14.8},"24":{"score":0.6918,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":13.3},"25":{"score":0.3878,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":16.1},"26":{"score":0.963,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":13.0},"27":{"score":0.8951,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":13.2},"28":{"score":0.6259,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":12.4},"29":{"score":0.8076,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":16.2},"30":{"score":0.4587,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":15.8},"31":{"score":0.109,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":66.5},"32":{"score":0.6011,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":32.7},"33":{"score":0.2612,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":55.7},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":16.1},"36":{"score":0.1619,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":320.1},"37":{"score":0.33,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":19.1},"38":{"score":0.01,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":277.0},"39":{"score":0.6046,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":26.6},"40":{"score":0.3333,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":13.1,"tracks":[{"track":"A · Arabic","score":0.3208,"headline":false},{"track":"A · English","score":0.3333,"headline":true},{"track":"C · Arabic pairs","score":0.735,"headline":false},{"track":"C · English pairs","score":0.87,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4249,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":16.5},"42":{"score":0.7,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":15.4},"43":{"score":0.4263,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":16.3},"44":{"score":0.5578,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":13.6},"45":{"score":0.1138,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":16.8},"48":{"score":0.3874,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":26.5},"50":{"score":0.4588,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":31.6},"56":{"score":0.5135,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":48.2,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4318,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":15.7,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5328,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":13.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5298,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":19.4,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5871,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":13.7,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5512,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":20.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6095,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":21.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":45.27,"balanced_skill":27.39,"breadth_skill":26.63},"frozen_scores":{"balanced_raw":36.39,"balanced_skill":20.61,"breadth_skill":17.88},"categories":[{"id":"knowledge","raw":0.388,"skill":0.207,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.5131,"skill":0.2993,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4258,"skill":0.2329,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5142,"skill":0.4214,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4226,"skill":0.2091,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8406,"skill":0.7848,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4132,"skill":0.3539,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.798,"skill":0.5234,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7851,"headline":false},{"track":"RouterBench-5shot","score":0.8108,"headline":false}]},"11":{"raw":0.5355,"skill":0.3282,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.7561,"skill":0.5122,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5906,"skill":0.1811,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.051,"skill":0.0435,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5143,"skill":0.0287,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7191,"skill":0.6255,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3878,"skill":0.1837,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.4587,"skill":0.3503,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.5663,"headline":false},{"track":"GSM8K-10choice","score":0.351,"headline":false}]},"31":{"raw":0.109,"skill":0.0296,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1619,"skill":0.1211,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.33,"skill":0.1597,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.3333,"skill":0.1423,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3208,"headline":false},{"track":"A · English","score":0.3333,"headline":true},{"track":"C · Arabic pairs","score":0.735,"headline":false},{"track":"C · English pairs","score":0.87,"headline":false}]},"41":{"raw":0.4249,"skill":0.1374,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4263,"skill":0.0898,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5578,"skill":0.1156,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4588,"skill":0.2146,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5135,"skill":0.027,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4318,"skill":0.3609,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5328,"skill":0.3228,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5298,"skill":0.3731,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5871,"skill":0.4839,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5512,"skill":0.2376,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6095,"skill":0.219,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6011,"skill":0.3658,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2612,"skill":0.2514,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1138,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5379,"skill":0.3078,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6259,"skill":0.2518,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8076,"skill":0.7435,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.01,"skill":0.01,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6046,"skill":0.4184,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7,"skill":0.4163,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.3255,"skill":0.3168,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.4172,"skill":0.4137,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.6232,"skill":0.373,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.435,"skill":0.4241,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0688,"skill":0.0688,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"pngwn-space-uncapped-full","name":"open-jev (pngwn)","long_name":"open-jev (pngwn)","variant":null,"meta":{"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":121956232,"model_url":"https://huggingface.co/pngwn/system-one-qwen3.5-4b-scorer-v2b","code_url":null,"base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"pngwn/system-one-qwen3.5-4b-scorer-v2b","served_checkpoint":null},"completed":121057,"counts":{"ok":120856},"lifted":true,"calibration":{"acc":0.579,"conf":0.6017,"ece":0.0683,"brier":0.5553,"over95":0.0063,"n":72590,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0981,"conf":0.0513,"acc":0.2456},{"lo":0.1,"w":0.0253,"conf":0.1465,"acc":0.294},{"lo":0.2,"w":0.0274,"conf":0.2455,"acc":0.223},{"lo":0.3,"w":0.0425,"conf":0.3567,"acc":0.3158},{"lo":0.4,"w":0.0765,"conf":0.4535,"acc":0.4316},{"lo":0.5,"w":0.184,"conf":0.5494,"acc":0.5337},{"lo":0.6,"w":0.1471,"conf":0.6477,"acc":0.6172},{"lo":0.7,"w":0.1455,"conf":0.7442,"acc":0.5654},{"lo":0.8,"w":0.1138,"conf":0.8501,"acc":0.8406},{"lo":0.9,"w":0.1398,"conf":0.9663,"acc":0.9163}],"areas":{"knowledge":{"ece":0.1286,"acc":0.5153,"conf":0.6438},"language":{"ece":0.0192,"acc":0.698,"conf":0.707},"retrieval":{"ece":0.2241,"acc":0.5681,"conf":0.3573},"tools":{"ece":0.1496,"acc":0.7304,"conf":0.6205},"arts":{"ece":0.0824,"acc":0.436,"conf":0.4955}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":133.2,"p95":615.4,"mean":313.4},"results":{"1":{"score":0.8973,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":148.0},"2":{"score":0.4146,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":2655.4},"3":{"score":0.4843,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":688.0},"4":{"score":0.5048,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":296.8},"5":{"score":0.5762,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":576.4},"6":{"score":0.7988,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":305.6},"9":{"score":0.0437,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":269.9},"10":{"score":0.5904,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":176.1},"11":{"score":0.6368,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":491.0},"12":{"score":0.5879,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":123.7},"20":{"score":0.519,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":121.7},"21":{"score":0.5521,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":117.6},"22":{"score":0.056,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":1080.2},"23":{"score":0.5828,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":138.8},"24":{"score":0.8058,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":118.3},"25":{"score":0.4388,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":125.7},"26":{"score":0.9621,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":116.0},"27":{"score":0.9121,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":116.8},"28":{"score":0.708,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":116.0},"29":{"score":0.7713,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":122.8},"30":{"score":0.3669,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":121.6},"31":{"score":0.1004,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":235.6},"32":{"score":0.5811,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":210.4},"33":{"score":0.3073,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":154.7},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":168.9},"36":{"score":0.1623,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":4159.5},"37":{"score":0.3067,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":133.1},"38":{"score":0.0025,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":855.6},"39":{"score":0.7463,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":121.6},"40":{"score":0.3756,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":117.2,"tracks":[{"track":"A · Arabic","score":0.3459,"headline":false},{"track":"A · English","score":0.3756,"headline":true},{"track":"C · Arabic pairs","score":0.5,"headline":false},{"track":"C · English pairs","score":0.675,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3828,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":125.3},"42":{"score":0.7434,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":144.9},"43":{"score":0.4825,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":121.3},"44":{"score":0.6316,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":123.5},"45":{"score":0.1098,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":129.5},"48":{"score":0.2185,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":158.4},"50":{"score":0.4206,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":154.6},"56":{"score":0.5025,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":112.3,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.8635,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":110.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5929,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":108.3,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4663,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":110.7,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.4564,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":108.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.0038,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":109.8,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5533,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":112.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":46.14,"balanced_skill":28.76,"breadth_skill":27.59},"frozen_scores":{"balanced_raw":36.3,"balanced_skill":19.99,"breadth_skill":16.56},"categories":[{"id":"knowledge","raw":0.4475,"skill":0.284,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.4958,"skill":0.337,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4566,"skill":0.2506,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5175,"skill":0.4248,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3896,"skill":0.1419,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8973,"skill":0.8613,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4146,"skill":0.3553,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7989,"skill":0.5257,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7854,"headline":false},{"track":"RouterBench-5shot","score":0.8123,"headline":false}]},"11":{"raw":0.6368,"skill":0.4747,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5395,"skill":0.079,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5521,"skill":0.1043,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0535,"skill":0.0461,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.579,"skill":0.158,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8357,"skill":0.7809,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4388,"skill":0.2517,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3669,"skill":0.2372,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4655,"headline":false},{"track":"GSM8K-10choice","score":0.2684,"headline":false}]},"31":{"raw":0.1004,"skill":0.0202,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1623,"skill":0.1215,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3067,"skill":0.1304,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.3756,"skill":0.1967,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3459,"headline":false},{"track":"A · English","score":0.3756,"headline":true},{"track":"C · Arabic pairs","score":0.5,"headline":false},{"track":"C · English pairs","score":0.675,"headline":false}]},"41":{"raw":0.3828,"skill":0.0742,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4825,"skill":0.1789,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6316,"skill":0.2632,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4206,"skill":0.1592,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5025,"skill":0.005,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.8635,"skill":0.8465,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5929,"skill":0.4099,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4663,"skill":0.2884,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.4564,"skill":0.3205,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.0038,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.5533,"skill":0.1066,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5811,"skill":0.334,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3073,"skill":0.2981,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1098,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5879,"skill":0.3827,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.708,"skill":0.416,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.7713,"skill":0.6951,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0025,"skill":0.0025,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7463,"skill":0.6269,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7434,"skill":0.5008,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.5048,"skill":0.4984,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.5762,"skill":0.5736,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5904,"skill":0.3185,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.4843,"skill":0.4744,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0437,"skill":0.0437,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.126,"skill":0.126,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"razorback-one-read-3e296f08","name":"razorback16 openjev diffusiongemma (NVFP4, vLLM)","long_name":"razorback16 diffgemma","variant":"one read, NVFP4","meta":{"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/vllm-project/vllm/pull/57250","base_model":"google/diffusiongemma-26B-A4B-it","base_url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it","weights_repo":null,"served_checkpoint":"nvidia/diffusiongemma-26B-A4B-it-NVFP4"},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.6017,"conf":0.8479,"ece":0.2463,"brier":0.6272,"over95":0.1107,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0025,"conf":0.1666,"acc":0.081},{"lo":0.2,"w":0.0134,"conf":0.2539,"acc":0.1579},{"lo":0.3,"w":0.02,"conf":0.3586,"acc":0.1896},{"lo":0.4,"w":0.0326,"conf":0.4503,"acc":0.2936},{"lo":0.5,"w":0.0677,"conf":0.5486,"acc":0.3721},{"lo":0.6,"w":0.0658,"conf":0.6504,"acc":0.455},{"lo":0.7,"w":0.1063,"conf":0.7375,"acc":0.3444},{"lo":0.8,"w":0.1103,"conf":0.8561,"acc":0.5411},{"lo":0.9,"w":0.5814,"conf":0.9795,"acc":0.7474}],"areas":{"knowledge":{"ece":0.2629,"acc":0.5389,"conf":0.8018},"language":{"ece":0.1665,"acc":0.7462,"conf":0.9127},"retrieval":{"ece":0.4133,"acc":0.4638,"conf":0.8771},"tools":{"ece":0.1201,"acc":0.7218,"conf":0.8355},"arts":{"ece":0.2959,"acc":0.5303,"conf":0.8262}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":76.7,"p95":167.1,"mean":91.3},"results":{"1":{"score":0.9569,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":54.6},"2":{"score":0.319,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":289.0},"3":{"score":0.5728,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":286.3},"4":{"score":0.6903,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":69.2},"5":{"score":0.0201,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":95.9},"6":{"score":0.7988,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":100.0},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":149.8},"10":{"score":0.3397,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":58.3},"11":{"score":0.5409,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":94.0},"12":{"score":0.5693,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":32.9},"20":{"score":0.8906,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":32.1},"21":{"score":0.6024,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":30.5},"22":{"score":0.112,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":119.4},"23":{"score":0.5956,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":39.6},"24":{"score":0.7775,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":32.2},"25":{"score":0.4796,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":75.6},"26":{"score":0.9806,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":63.8},"27":{"score":0.936,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":64.7},"28":{"score":0.6993,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":57.2},"29":{"score":0.8112,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":77.8},"30":{"score":0.3904,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":76.3},"31":{"score":0.1402,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":155.0},"32":{"score":0.6303,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":123.3},"33":{"score":0.2994,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":147.7},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":343.5},"36":{"score":0.0928,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1453.7},"37":{"score":0.5629,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":86.8},"38":{"score":0.005,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":260.3},"39":{"score":0.8503,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":78.3},"40":{"score":0.5638,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":59.1,"tracks":[{"track":"A · Arabic","score":0.3514,"headline":false},{"track":"A · English","score":0.5638,"headline":true},{"track":"C · Arabic pairs","score":0.82,"headline":false},{"track":"C · English pairs","score":0.93,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.5923,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":70.3},"42":{"score":0.7406,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":108.6},"43":{"score":0.4719,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":66.3},"44":{"score":0.623,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":57.8},"45":{"score":0.1018,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":90.6},"48":{"score":0.2359,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":105.1},"50":{"score":0.4761,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":136.6},"56":{"score":0.813,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":98.1,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4176,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":80.6,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6621,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":66.2,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5181,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":136.3,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5758,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":64.6,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7081,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":86.9,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6837,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":88.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":49.73,"balanced_skill":34.99,"breadth_skill":34.35},"frozen_scores":{"balanced_raw":40.36,"balanced_skill":26.93,"breadth_skill":23.3},"categories":[{"id":"knowledge","raw":0.4216,"skill":0.2595,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6081,"skill":0.4449,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4575,"skill":0.3135,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.5276,"skill":0.4399,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.4718,"skill":0.2915,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9569,"skill":0.9418,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.319,"skill":0.2501,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7989,"skill":0.5257,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7854,"headline":false},{"track":"RouterBench-5shot","score":0.8123,"headline":false}]},"11":{"raw":0.5409,"skill":0.3361,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8915,"skill":0.783,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6024,"skill":0.2047,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1073,"skill":0.1003,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5934,"skill":0.1867,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7849,"skill":0.7132,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4796,"skill":0.3061,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3904,"skill":0.2594,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4299,"headline":false},{"track":"GSM8K-10choice","score":0.351,"headline":false}]},"31":{"raw":0.1402,"skill":0.0635,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.0928,"skill":0.0486,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5629,"skill":0.4518,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5638,"skill":0.4388,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3514,"headline":false},{"track":"A · English","score":0.5638,"headline":true},{"track":"C · Arabic pairs","score":0.82,"headline":false},{"track":"C · English pairs","score":0.93,"headline":false}]},"41":{"raw":0.5923,"skill":0.3885,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4719,"skill":0.1622,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.623,"skill":0.246,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4761,"skill":0.2397,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.813,"skill":0.626,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4176,"skill":0.345,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6621,"skill":0.5102,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5181,"skill":0.3575,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5758,"skill":0.4697,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7081,"skill":0.5042,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6837,"skill":0.3674,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6303,"skill":0.4122,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2994,"skill":0.2901,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1018,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5693,"skill":0.3549,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6993,"skill":0.3986,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8112,"skill":0.7483,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.005,"skill":0.005,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8503,"skill":0.7798,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7406,"skill":0.4953,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.6903,"skill":0.6863,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0201,"skill":0.0142,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.3397,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.5728,"skill":0.5646,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0564,"skill":0.0564,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"reflex-27b","name":"reflex 27B","long_name":"reflex Qwen3.8-27B-FP8","variant":"FP8","meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":null,"base_model":"Qwen/Qwen3.8-27B","base_url":"https://huggingface.co/Qwen/Qwen3.8-27B","weights_repo":null,"served_checkpoint":"Qwen/Qwen3.8-27B-FP8"},"completed":120615,"counts":{"ok":106115,"unsupported":14500},"lifted":false,"calibration":{"acc":0.6835,"conf":0.7228,"ece":0.0394,"brier":0.4261,"over95":0.0066,"n":32694,"benchmarks":29,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0063,"conf":0.1657,"acc":0.1372},{"lo":0.2,"w":0.0158,"conf":0.256,"acc":0.2103},{"lo":0.3,"w":0.0463,"conf":0.3551,"acc":0.3455},{"lo":0.4,"w":0.0695,"conf":0.4535,"acc":0.397},{"lo":0.5,"w":0.1844,"conf":0.5472,"acc":0.4325},{"lo":0.6,"w":0.1334,"conf":0.6491,"acc":0.6021},{"lo":0.7,"w":0.1252,"conf":0.7497,"acc":0.7429},{"lo":0.8,"w":0.128,"conf":0.8523,"acc":0.8365},{"lo":0.9,"w":0.2911,"conf":0.9596,"acc":0.9466}],"areas":{"knowledge":{"ece":0.0733,"acc":0.5913,"conf":0.6645},"language":{"ece":0.0154,"acc":0.7841,"conf":0.7785},"retrieval":{"ece":0.19,"acc":0.5203,"conf":0.7103},"tools":{"ece":0.0467,"acc":0.984,"conf":0.9402},"arts":{"ece":0.0544,"acc":0.6652,"conf":0.6701}}},"gaps":{"share":0.1202,"reasons":[{"reason":"supports at most 26 answer options per question","share":0.1202}]},"latency":{"median":88.1,"p95":396.6,"mean":208.0},"results":{"1":{"score":0.9728,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":200.7},"2":{"score":0.4383,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":2035.4},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7989,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":385.9},"9":{"score":0.5687,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":996.8},"10":{"score":0.3932,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":256.6},"11":{"score":0.7744,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":797.5},"12":{"score":0.6483,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":72.5},"20":{"score":0.9548,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":72.2},"21":{"score":0.6305,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":66.7},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.6534,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":74.6},"24":{"score":0.8213,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":72.8},"25":{"score":0.5102,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":87.4},"26":{"score":0.987,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":72.0},"27":{"score":0.9701,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":72.1},"28":{"score":0.7979,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":71.1},"29":{"score":0.9384,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":79.2},"30":{"score":0.6497,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":87.5},"31":{"score":0.2771,"jev":0.2746,"metric":"accuracy","requests":5000,"answered":1588,"unsupported":3412,"errors":0,"median_ms":174.0},"32":{"score":0.6197,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":230.3},"33":{"score":0.2752,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":286.9},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":77.4},"36":{"score":0.1898,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":2507.4},"37":{"score":0.5252,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":145.0},"38":{"score":0.01,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":2009.0},"39":{"score":0.7866,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":111.3},"40":{"score":0.4867,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":73.1,"tracks":[{"track":"A · Arabic","score":0.4062,"headline":false},{"track":"A · English","score":0.4867,"headline":true},{"track":"C · Arabic pairs","score":0.77,"headline":false},{"track":"C · English pairs","score":0.97,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6036,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":79.5},"42":{"score":0.8053,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":143.3},"43":{"score":0.6053,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":78.5},"44":{"score":0.6362,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":72.4},"45":{"score":0.1138,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":88.5},"48":{"score":0.1933,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":166.9},"50":{"score":0.435,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":219.2},"56":{"score":0.734,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":293.2,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.6173,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":87.3,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7042,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":75.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7662,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":160.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.7159,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":87.3,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7413,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":180.7,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.7093,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":150.2,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":52.26,"balanced_skill":39.1,"breadth_skill":37.52},"frozen_scores":{"balanced_raw":43.8,"balanced_skill":32.09,"breadth_skill":26.53},"categories":[{"id":"knowledge","raw":0.482,"skill":0.3379,"coverage":0.9318,"pending":0,"n":10},{"id":"language","raw":0.6593,"skill":0.5284,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3645,"skill":0.2059,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.5908,"skill":0.5213,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.5166,"skill":0.3613,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9728,"skill":0.9633,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4383,"skill":0.3815,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7989,"skill":0.5259,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7851,"headline":false},{"track":"RouterBench-5shot","score":0.8127,"headline":false}]},"11":{"raw":0.7744,"skill":0.6738,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.957,"skill":0.9139,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6305,"skill":0.261,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.6513,"skill":0.3026,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8261,"skill":0.7681,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.5102,"skill":0.3469,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.6497,"skill":0.5775,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.6998,"headline":false},{"track":"GSM8K-10choice","score":0.5997,"headline":false}]},"31":{"raw":0.088,"skill":0.0067,"coverage":0.3176,"random":0.0819,"tracks":[]},"36":{"raw":0.1898,"skill":0.1504,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5252,"skill":0.4045,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4867,"skill":0.3396,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4062,"headline":false},{"track":"A · English","score":0.4867,"headline":true},{"track":"C · Arabic pairs","score":0.77,"headline":false},{"track":"C · English pairs","score":0.97,"headline":false}]},"41":{"raw":0.6036,"skill":0.4054,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.6053,"skill":0.3737,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6362,"skill":0.2724,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.435,"skill":0.1799,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.734,"skill":0.468,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.6173,"skill":0.5696,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.7042,"skill":0.5712,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7662,"skill":0.6883,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.7159,"skill":0.6449,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7413,"skill":0.5606,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7093,"skill":0.4186,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6197,"skill":0.3954,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2752,"skill":0.2656,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1138,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6483,"skill":0.4732,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7979,"skill":0.5958,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9384,"skill":0.9179,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.01,"skill":0.01,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.7866,"skill":0.6861,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8053,"skill":0.6212,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.3932,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.5687,"skill":0.5687,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.2268,"skill":0.2268,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6927,"pending":0.24},{"engine":"rune-26b-a4b","name":"Surogate Rune 26B-A4B","long_name":"Surogate Rune 26B-A4B","variant":"JD-Q6_K","meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":25805936206,"params_estimated":false,"params_from_name":false,"trained_bytes":104290457600,"model_url":"https://huggingface.co/surogate/rune-26b-a4b-GGUF","code_url":"https://github.com/invergent-ai/surogate","base_model":"google/gemma-4-26B-A4B-it","base_url":"https://huggingface.co/google/gemma-4-26B-A4B-it","weights_repo":"surogate/rune-26b-a4b-GGUF","served_checkpoint":null},"completed":120615,"counts":{"ok":120614,"unsupported":1},"lifted":false,"calibration":{"acc":0.6735,"conf":0.8679,"ece":0.1944,"brier":0.5014,"over95":0.0645,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.004,"conf":0.0753,"acc":0.0},{"lo":0.1,"w":0.0122,"conf":0.1569,"acc":0.1337},{"lo":0.2,"w":0.0137,"conf":0.2484,"acc":0.1341},{"lo":0.3,"w":0.0178,"conf":0.3539,"acc":0.1493},{"lo":0.4,"w":0.0264,"conf":0.4594,"acc":0.2751},{"lo":0.5,"w":0.0429,"conf":0.5514,"acc":0.3915},{"lo":0.6,"w":0.0537,"conf":0.6474,"acc":0.4417},{"lo":0.7,"w":0.0684,"conf":0.7523,"acc":0.4832},{"lo":0.8,"w":0.0891,"conf":0.8578,"acc":0.5239},{"lo":0.9,"w":0.6718,"conf":0.9787,"acc":0.8036}],"areas":{"knowledge":{"ece":0.248,"acc":0.5955,"conf":0.8435},"language":{"ece":0.1348,"acc":0.7912,"conf":0.926},"retrieval":{"ece":0.2801,"acc":0.6169,"conf":0.897},"tools":{"ece":0.0616,"acc":0.9045,"conf":0.9555},"arts":{"ece":0.1903,"acc":0.5556,"conf":0.7459}}},"gaps":{"share":0.0,"reasons":[{"reason":"surogate 400 decisions_template_boundary: the chat template did not render question 'q1' a…","share":0.0}]},"latency":{"median":679.7,"p95":5789.3,"mean":1598.8},"results":{"1":{"score":0.9469,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":1469.4},"2":{"score":0.4429,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":11785.7},"3":{"score":0.8051,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":6408.7},"4":{"score":0.7625,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":699.6},"5":{"score":0.8697,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":1389.2},"6":{"score":0.7994,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":2709.2},"9":{"score":0.3875,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":6559.2},"10":{"score":0.1704,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":1079.5},"11":{"score":0.7748,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":6427.9},"12":{"score":0.7114,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":289.7},"20":{"score":0.9052,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":289.7},"21":{"score":0.6309,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":439.8},"22":{"score":0.1425,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":3208.9},"23":{"score":0.6302,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":459.8},"24":{"score":0.8289,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":329.7},"25":{"score":0.449,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":309.7},"26":{"score":0.9886,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":289.8},"27":{"score":0.965,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":289.6},"28":{"score":0.8106,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":289.7},"29":{"score":0.922,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":339.6},"30":{"score":0.7502,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":399.8},"31":{"score":0.186,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":659.4},"32":{"score":0.6543,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":818.8},"33":{"score":0.3455,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":2469.5},"34":{"score":0.2,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":2199.3},"36":{"score":0.1824,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":9578.8},"37":{"score":0.5302,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":1259.4},"38":{"score":0.005,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":11102.7},"39":{"score":0.85,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":989.7},"40":{"score":0.5053,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":859.8,"tracks":[{"track":"A · Arabic","score":0.5219,"headline":false},{"track":"A · English","score":0.5053,"headline":true},{"track":"C · Arabic pairs","score":0.82,"headline":false},{"track":"C · English pairs","score":0.95,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6119,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":569.8},"42":{"score":0.8015,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":599.5},"43":{"score":0.7018,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":429.6},"44":{"score":0.703,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":400.1},"45":{"score":0.106,"jev":0.204,"metric":"accuracy","requests":501,"answered":500,"unsupported":1,"errors":0,"median_ms":569.3},"48":{"score":0.2145,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":689.5},"50":{"score":0.435,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":629.1},"56":{"score":0.7875,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":2109.4,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.6329,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":329.8,"edition_added":"v0.2","jev":0.827},"58":{"score":0.74,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":329.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7426,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":693.2,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.661,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":499.8,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7743,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":809.8,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.715,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":489.8,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":59.39,"balanced_skill":47.23,"breadth_skill":46.35},"frozen_scores":{"balanced_raw":45.36,"balanced_skill":33.9,"breadth_skill":28.73},"categories":[{"id":"knowledge","raw":0.5268,"skill":0.401,"coverage":0.9998,"pending":0,"n":10},{"id":"language","raw":0.6767,"skill":0.5555,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.574,"skill":0.4553,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.6874,"skill":0.6146,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.5049,"skill":0.3354,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9469,"skill":0.9283,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4429,"skill":0.3866,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7994,"skill":0.5269,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.786,"headline":false},{"track":"RouterBench-5shot","score":0.8127,"headline":false}]},"11":{"raw":0.7748,"skill":0.6743,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9084,"skill":0.8168,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6309,"skill":0.2618,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.1358,"skill":0.1291,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6209,"skill":0.2419,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.831,"skill":0.7746,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.449,"skill":0.2653,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.7502,"skill":0.7009,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.8059,"headline":false},{"track":"GSM8K-10choice","score":0.6945,"headline":false}]},"31":{"raw":0.186,"skill":0.1134,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1824,"skill":0.1426,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5302,"skill":0.4108,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5053,"skill":0.3635,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.5219,"headline":false},{"track":"A · English","score":0.5053,"headline":true},{"track":"C · Arabic pairs","score":0.82,"headline":false},{"track":"C · English pairs","score":0.95,"headline":false}]},"41":{"raw":0.6119,"skill":0.4178,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.7018,"skill":0.5268,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.703,"skill":0.406,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.435,"skill":0.1799,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.7875,"skill":0.575,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.6329,"skill":0.5871,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.74,"skill":0.6231,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7426,"skill":0.6568,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.661,"skill":0.5763,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7743,"skill":0.6166,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.715,"skill":0.43,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6543,"skill":0.4504,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3455,"skill":0.3368,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1058,"skill":0.0,"coverage":0.998,"random":0.1641,"tracks":[]},"12":{"raw":0.7114,"skill":0.5677,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8106,"skill":0.6212,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.922,"skill":0.896,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.005,"skill":0.005,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.85,"skill":0.7794,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8015,"skill":0.6138,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7625,"skill":0.7594,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8697,"skill":0.8689,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.1704,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.8051,"skill":0.8013,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.3875,"skill":0.3875,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.142,"skill":0.142,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"semif","name":"SemIf","long_name":"SemIf Qwen3.5-4B Base","variant":null,"meta":{"kind":"inference technique","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/TheoLeeCJ/openjev","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":null,"served_checkpoint":null},"completed":121057,"counts":{"ok":105740,"unsupported":15317},"lifted":true,"calibration":{"acc":0.6183,"conf":0.7389,"ece":0.1206,"brier":0.5107,"over95":0.0247,"n":70068,"benchmarks":29,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0004,"conf":0.1774,"acc":0.0},{"lo":0.2,"w":0.0165,"conf":0.263,"acc":0.1845},{"lo":0.3,"w":0.055,"conf":0.3561,"acc":0.2879},{"lo":0.4,"w":0.0681,"conf":0.4498,"acc":0.324},{"lo":0.5,"w":0.1611,"conf":0.5495,"acc":0.5089},{"lo":0.6,"w":0.1087,"conf":0.6493,"acc":0.5585},{"lo":0.7,"w":0.151,"conf":0.7458,"acc":0.4753},{"lo":0.8,"w":0.1115,"conf":0.8532,"acc":0.6777},{"lo":0.9,"w":0.3276,"conf":0.9689,"acc":0.877}],"areas":{"knowledge":{"ece":0.1553,"acc":0.5132,"conf":0.6685},"language":{"ece":0.1291,"acc":0.6952,"conf":0.8215},"retrieval":{"ece":0.1814,"acc":0.6173,"conf":0.7888},"tools":{"ece":0.0383,"acc":0.919,"conf":0.8897},"arts":{"ece":0.0532,"acc":0.611,"conf":0.6642}}},"gaps":{"share":0.1265,"reasons":[{"reason":"supports only 2–16 answer options per question","share":0.1265}]},"latency":{"median":109.9,"p95":250.8,"mean":190.1},"results":{"1":{"score":0.9327,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":115.4},"2":{"score":0.4083,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1152.9},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7954,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":206.9},"9":{"score":0.1,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":354.7},"10":{"score":0.568,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":118.9},"11":{"score":0.5951,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":498.0},"12":{"score":0.5491,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":108.7},"20":{"score":0.8522,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":108.3},"21":{"score":0.5993,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":107.6},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.566,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":110.1},"24":{"score":0.7144,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":108.3},"25":{"score":0.3673,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":109.2},"26":{"score":0.9676,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":108.1},"27":{"score":0.9181,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":108.4},"28":{"score":0.6598,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":107.3},"29":{"score":0.8272,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":109.5},"30":{"score":0.3131,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":111.0},"31":{"score":0.3316,"jev":0.3914,"metric":"accuracy","requests":5000,"answered":787,"unsupported":4213,"errors":0,"median_ms":110.9},"32":{"score":0.5465,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":116.3},"33":{"score":0.2,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":134.4},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":109.8},"36":{"score":0.1657,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1541.1},"37":{"score":0.3616,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":112.1},"38":{"score":0.0125,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":470.4},"39":{"score":0.6596,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":111.5},"40":{"score":0.39,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":108.5,"tracks":[{"track":"A · Arabic","score":0.281,"headline":false},{"track":"A · English","score":0.39,"headline":true},{"track":"C · Arabic pairs","score":0.645,"headline":false},{"track":"C · English pairs","score":0.84,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4396,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":109.4},"42":{"score":0.7196,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":110.4},"43":{"score":0.3895,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":108.9},"44":{"score":0.5596,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":107.9},"45":{"score":0.1107,"jev":0.2049,"metric":"accuracy","requests":501,"answered":488,"unsupported":13,"errors":0,"median_ms":111.3},"48":{"score":0.2252,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":112.8},"50":{"score":0.4433,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":116.2},"56":{"score":0.6005,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":119.3,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4368,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":110.4,"edition_added":"v0.2","jev":0.827},"58":{"score":0.5756,"metric":"accuracy","requests":5507,"answered":5332,"unsupported":175,"errors":0,"median_ms":107.1,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5468,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":116.0,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5777,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":109.9,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5344,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":114.2,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6162,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":54.1,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":42.65,"balanced_skill":25.7,"breadth_skill":24.57},"frozen_scores":{"balanced_raw":36.38,"balanced_skill":21.22,"breadth_skill":18.17},"categories":[{"id":"knowledge","raw":0.353,"skill":0.167,"coverage":0.91,"pending":0,"n":10},{"id":"language","raw":0.5387,"skill":0.3378,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3303,"skill":0.1484,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.4639,"skill":0.3784,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.4465,"skill":0.2535,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9327,"skill":0.9092,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4083,"skill":0.3485,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7954,"skill":0.5172,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7828,"headline":false},{"track":"RouterBench-5shot","score":0.8079,"headline":false}]},"11":{"raw":0.5951,"skill":0.4144,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8517,"skill":0.7034,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5993,"skill":0.1986,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.5545,"skill":0.109,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7275,"skill":0.6367,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3673,"skill":0.1565,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3131,"skill":0.1702,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4003,"headline":false},{"track":"GSM8K-10choice","score":0.2259,"headline":false}]},"31":{"raw":0.0522,"skill":0.0,"coverage":0.1574,"random":0.0819,"tracks":[]},"36":{"raw":0.1657,"skill":0.1251,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3616,"skill":0.1992,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.39,"skill":0.2152,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.281,"headline":false},{"track":"A · English","score":0.39,"headline":true},{"track":"C · Arabic pairs","score":0.645,"headline":false},{"track":"C · English pairs","score":0.84,"headline":false}]},"41":{"raw":0.4396,"skill":0.1595,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3895,"skill":0.0313,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5596,"skill":0.1192,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4433,"skill":0.1921,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6005,"skill":0.201,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4368,"skill":0.3666,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.5573,"skill":0.3583,"coverage":0.9682,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5468,"skill":0.3957,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5777,"skill":0.4721,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5344,"skill":0.2091,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6162,"skill":0.2324,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5465,"skill":0.279,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2,"skill":0.1894,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1078,"skill":0.0,"coverage":0.9741,"random":0.1641,"tracks":[]},"12":{"raw":0.5491,"skill":0.3246,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6598,"skill":0.3196,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8272,"skill":0.7696,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0125,"skill":0.0125,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.6596,"skill":0.4993,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7196,"skill":0.4545,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.568,"skill":0.2812,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.1,"skill":0.1,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0992,"skill":0.0992,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6863,"pending":0.24},{"engine":"solomon-v11-bf16-encoding","name":"Solomon v1.1","long_name":"Solomon v1.1","variant":null,"meta":{"kind":"head / adapter","technique":"autoregressive","technique_name":"autoregressive","served_params":27781427952,"params_estimated":false,"params_from_name":false,"trained_bytes":903468342,"model_url":"https://huggingface.co/DoccyHealth/Solomon","code_url":null,"base_model":"Qwen/Qwen3.8-27B","base_url":"https://huggingface.co/Qwen/Qwen3.8-27B","weights_repo":"DoccyHealth/Solomon","served_checkpoint":null},"completed":121057,"counts":{"ok":93471,"unsupported":27574},"lifted":false,"calibration":{"acc":0.7017,"conf":0.7274,"ece":0.1012,"brier":0.4456,"over95":0.0291,"n":69333,"benchmarks":28,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0044,"conf":0.1867,"acc":0.3613},{"lo":0.2,"w":0.0123,"conf":0.2605,"acc":0.2044},{"lo":0.3,"w":0.1279,"conf":0.3483,"acc":0.5935},{"lo":0.4,"w":0.0695,"conf":0.4484,"acc":0.5093},{"lo":0.5,"w":0.127,"conf":0.5513,"acc":0.5625},{"lo":0.6,"w":0.1007,"conf":0.6462,"acc":0.621},{"lo":0.7,"w":0.0789,"conf":0.751,"acc":0.653},{"lo":0.8,"w":0.0982,"conf":0.8521,"acc":0.7032},{"lo":0.9,"w":0.3811,"conf":0.97,"acc":0.8706}],"areas":{"knowledge":{"ece":0.1461,"acc":0.6524,"conf":0.601},"language":{"ece":0.0728,"acc":0.8105,"conf":0.8641},"retrieval":{"ece":0.2617,"acc":0.5834,"conf":0.8451},"tools":{"ece":0.0072,"acc":0.991,"conf":0.9929},"arts":{"ece":0.0823,"acc":0.5919,"conf":0.6736}}},"gaps":{"share":0.2278,"reasons":[{"reason":"supports only 2–8 answer options per question","share":0.2243},{"reason":"request shape outside the model's schema","share":0.0034},{"reason":"request larger than its native request budget","share":0.0001}]},"latency":{"median":220.5,"p95":937.0,"mean":475.6},"results":{"1":{"score":0.9799,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":504.8},"2":{"score":0.4502,"jev":0.4484,"metric":"nDCG@10","requests":1000,"answered":995,"unsupported":5,"errors":0,"median_ms":4226.4},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":null,"jev":null,"metric":null,"requests":10000,"answered":0,"unsupported":10000,"errors":0,"median_ms":null},"9":{"score":null,"jev":null,"metric":null,"requests":160,"answered":0,"unsupported":160,"errors":0,"median_ms":null},"10":{"score":0.592,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":458.5},"11":{"score":0.7153,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":2779.2},"12":{"score":0.6708,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":215.2},"20":{"score":0.9414,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":217.9},"21":{"score":0.5422,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":218.3},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.502,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":257.6},"24":{"score":0.6995,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":215.6},"25":{"score":0.4286,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":221.6},"26":{"score":0.8657,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":214.6},"27":{"score":0.8456,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":212.6},"28":{"score":0.8106,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":212.4},"29":{"score":0.9099,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":216.8},"30":{"score":null,"jev":null,"metric":null,"requests":2638,"answered":1319,"unsupported":1319,"errors":0,"median_ms":218.9},"31":{"score":0.4019,"jev":0.4533,"metric":"accuracy","requests":5000,"answered":525,"unsupported":4475,"errors":0,"median_ms":218.4},"32":{"score":0.6064,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":348.2},"33":{"score":0.2055,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":1070.7},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":216.3},"36":{"score":0.1878,"jev":0.1904,"metric":"nDCG@10","requests":550,"answered":549,"unsupported":1,"errors":0,"median_ms":4628.0},"37":{"score":0.5618,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":222.6},"38":{"score":0.0075,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":7654.5},"39":{"score":0.9186,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":333.5},"40":{"score":0.4822,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":222.2,"tracks":[{"track":"A · Arabic","score":0.4583,"headline":false},{"track":"A · English","score":0.4822,"headline":true},{"track":"C · Arabic pairs","score":0.715,"headline":false},{"track":"C · English pairs","score":0.9,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.694,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":218.9},"42":{"score":0.8176,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":239.3},"43":{"score":0.6933,"jev":0.734,"metric":"accuracy","requests":570,"answered":564,"unsupported":6,"errors":0,"median_ms":220.4},"44":{"score":0.7044,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":220.5},"45":{"score":0.1474,"jev":0.2138,"metric":"accuracy","requests":501,"answered":407,"unsupported":94,"errors":0,"median_ms":223.1},"48":{"score":0.2385,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":286.7},"50":{"score":0.3729,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":268.6},"56":{"score":0.6815,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":1173.6,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.7072,"metric":"accuracy","requests":12032,"answered":1250,"unsupported":10782,"errors":0,"median_ms":214.9,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7254,"metric":"accuracy","requests":5507,"answered":5081,"unsupported":426,"errors":0,"median_ms":216.8,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.8264,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":306.9,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.517,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":218.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7552,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":295.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.711,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":233.1,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":44.32,"balanced_skill":31.96,"breadth_skill":30.29},"frozen_scores":{"balanced_raw":36.58,"balanced_skill":25.49,"breadth_skill":21.18},"categories":[{"id":"knowledge","raw":0.3536,"skill":0.2239,"coverage":0.6933,"pending":0,"n":10},{"id":"language","raw":0.6782,"skill":0.5588,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3905,"skill":0.2435,"coverage":0.714,"pending":0,"n":7},{"id":"tools","raw":0.3757,"skill":0.3556,"coverage":0.4992,"pending":0,"n":6},{"id":"arts","raw":0.4181,"skill":0.2163,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9799,"skill":0.9729,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4479,"skill":0.3921,"coverage":0.995,"random":0.0918,"tracks":[]},"6":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.0,"headline":false},{"track":"RouterBench-5shot","score":0.0,"headline":false}]},"11":{"raw":0.7153,"skill":0.5882,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9437,"skill":0.8874,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.5422,"skill":0.0845,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.505,"skill":0.01,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.6953,"skill":0.5938,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4286,"skill":0.2381,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.0,"headline":false},{"track":"GSM8K-10choice","score":0.0,"headline":false}]},"31":{"raw":0.0422,"skill":0.0,"coverage":0.105,"random":0.0819,"tracks":[]},"36":{"raw":0.1875,"skill":0.148,"coverage":0.9982,"random":0.0464,"tracks":[]},"37":{"raw":0.5618,"skill":0.4504,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.4822,"skill":0.3338,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4583,"headline":false},{"track":"A · English","score":0.4822,"headline":true},{"track":"C · Arabic pairs","score":0.715,"headline":false},{"track":"C · English pairs","score":0.9,"headline":false}]},"41":{"raw":0.694,"skill":0.541,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.686,"skill":0.5018,"coverage":0.9895,"random":0.3697,"tracks":[]},"44":{"raw":0.7044,"skill":0.4088,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.3729,"skill":0.0899,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.6815,"skill":0.363,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.0735,"skill":0.0,"coverage":0.1039,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6693,"skill":0.5207,"coverage":0.9226,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.8264,"skill":0.7685,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.517,"skill":0.3962,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7552,"skill":0.5842,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.711,"skill":0.422,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6064,"skill":0.3742,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2055,"skill":0.195,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1197,"skill":0.0,"coverage":0.8124,"random":0.1641,"tracks":[]},"12":{"raw":0.6708,"skill":0.5069,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8106,"skill":0.6212,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9099,"skill":0.8799,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0075,"skill":0.0075,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.9186,"skill":0.8803,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8176,"skill":0.6451,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.592,"skill":0.3211,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0,"tracks":[]},"48":{"raw":0.046,"skill":0.046,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6035,"pending":0.24},{"engine":"tev1-4b","name":"Tev1-4B-experimental","long_name":"Together Tev1 Qwen3.5-4B","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":4659865088,"params_estimated":false,"params_from_name":false,"trained_bytes":9319828096,"model_url":"https://huggingface.co/togethercomputer/Tev1-4B-experimental","code_url":"https://github.com/togethercomputer/tev1","base_model":"Qwen/Qwen3.5-4B-Base","base_url":"https://huggingface.co/Qwen/Qwen3.5-4B-Base","weights_repo":"togethercomputer/Tev1-4B-experimental","served_checkpoint":null},"completed":120615,"counts":{"ok":105883,"unsupported":14732},"lifted":false,"calibration":{"acc":0.6127,"conf":0.7417,"ece":0.1291,"brier":0.5139,"over95":0.0168,"n":32653,"benchmarks":29,"rel":[{"lo":0.0,"w":0.0002,"conf":0.0919,"acc":0.0},{"lo":0.1,"w":0.012,"conf":0.1624,"acc":0.11},{"lo":0.2,"w":0.0216,"conf":0.2506,"acc":0.1198},{"lo":0.3,"w":0.0489,"conf":0.352,"acc":0.1994},{"lo":0.4,"w":0.0538,"conf":0.4561,"acc":0.3715},{"lo":0.5,"w":0.1425,"conf":0.5511,"acc":0.4759},{"lo":0.6,"w":0.1156,"conf":0.6492,"acc":0.5373},{"lo":0.7,"w":0.1288,"conf":0.7497,"acc":0.606},{"lo":0.8,"w":0.155,"conf":0.8473,"acc":0.5463},{"lo":0.9,"w":0.3216,"conf":0.9674,"acc":0.8904}],"areas":{"knowledge":{"ece":0.1806,"acc":0.5111,"conf":0.6917},"language":{"ece":0.0705,"acc":0.7148,"conf":0.785},"retrieval":{"ece":0.2745,"acc":0.4965,"conf":0.7693},"tools":{"ece":0.0258,"acc":0.9313,"conf":0.9228},"arts":{"ece":0.1059,"acc":0.5864,"conf":0.6903}}},"gaps":{"share":0.1221,"reasons":[{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'q1' has 151","share":0.0456},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'q1' has 77","share":0.0255},{"reason":"Tev1 answers with one of the labels A-X (2-24 options); question 'chord' has 129","share":0.0166}]},"latency":{"median":31.4,"p95":157.9,"mean":66.0},"results":{"1":{"score":0.902,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":72.4},"2":{"score":0.4453,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":437.4},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.6781,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":150.7},"9":{"score":0.1562,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":366.8},"10":{"score":0.407,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":67.9},"11":{"score":0.6772,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":342.2},"12":{"score":0.5639,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":30.3},"20":{"score":0.7662,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":30.3},"21":{"score":0.583,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":29.5},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.5844,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":65.1},"24":{"score":0.7227,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":30.0},"25":{"score":0.4235,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":30.8},"26":{"score":0.976,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":29.9},"27":{"score":0.9215,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":29.8},"28":{"score":0.6961,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":29.5},"29":{"score":0.8193,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":30.9},"30":{"score":0.3666,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":30.7},"31":{"score":0.22,"jev":0.2929,"metric":"accuracy","requests":5000,"answered":1359,"unsupported":3641,"errors":0,"median_ms":66.1},"32":{"score":0.5798,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":68.1},"33":{"score":0.203,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":113.9},"34":{"score":0,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":31.0},"36":{"score":0.1721,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":543.0},"37":{"score":0.4421,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":32.0},"38":{"score":0.005,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":430.9},"39":{"score":0.8631,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":59.1},"40":{"score":0.3594,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":30.3,"tracks":[{"track":"A · Arabic","score":0.3776,"headline":false},{"track":"A · English","score":0.3594,"headline":true},{"track":"C · Arabic pairs","score":0.57,"headline":false},{"track":"C · English pairs","score":0.89,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4766,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":30.8},"42":{"score":0.7734,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":32.4},"43":{"score":0.5018,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":30.2},"44":{"score":0.631,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":30.2},"45":{"score":0.1165,"jev":0.2028,"metric":"accuracy","requests":501,"answered":498,"unsupported":3,"errors":0,"median_ms":31.2},"48":{"score":0.1997,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":67.0},"50":{"score":0.4159,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":67.7},"56":{"score":0.5065,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":99.8,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.4537,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":31.1,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6348,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":30.4,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.6054,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":66.0,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5549,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":30.7,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.5033,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":66.6,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.641,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":33.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":43.7,"balanced_skill":26.32,"breadth_skill":24.8},"frozen_scores":{"balanced_raw":37.54,"balanced_skill":22.22,"breadth_skill":18.94},"categories":[{"id":"knowledge","raw":0.397,"skill":0.232,"coverage":0.9266,"pending":0,"n":10},{"id":"language","raw":0.5737,"skill":0.3941,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3098,"skill":0.1058,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.4645,"skill":0.3544,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.4398,"skill":0.23,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.902,"skill":0.8677,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4453,"skill":0.3892,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.6781,"skill":0.2394,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6627,"headline":false},{"track":"RouterBench-5shot","score":0.6935,"headline":false}]},"11":{"raw":0.6772,"skill":0.5332,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.7699,"skill":0.5399,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.583,"skill":0.1659,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.5535,"skill":0.107,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7375,"skill":0.65,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4235,"skill":0.2313,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3666,"skill":0.2365,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4632,"headline":false},{"track":"GSM8K-10choice","score":0.2699,"headline":false}]},"31":{"raw":0.0598,"skill":0.0,"coverage":0.2718,"random":0.0819,"tracks":[]},"36":{"raw":0.1721,"skill":0.1318,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4421,"skill":0.3002,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.3594,"skill":0.1759,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3776,"headline":false},{"track":"A · English","score":0.3594,"headline":true},{"track":"C · Arabic pairs","score":0.57,"headline":false},{"track":"C · English pairs","score":0.89,"headline":false}]},"41":{"raw":0.4766,"skill":0.2149,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.5018,"skill":0.2095,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.631,"skill":0.262,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4159,"skill":0.1522,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5065,"skill":0.013,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.4537,"skill":0.3856,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6348,"skill":0.4706,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.6054,"skill":0.4739,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5549,"skill":0.4436,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.5033,"skill":0.1563,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.641,"skill":0.282,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5798,"skill":0.332,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.203,"skill":0.1924,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1158,"skill":0.0,"coverage":0.994,"random":0.1641,"tracks":[]},"12":{"raw":0.5639,"skill":0.3468,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.6961,"skill":0.3922,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8193,"skill":0.7591,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.005,"skill":0.005,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8631,"skill":0.7986,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7734,"skill":0.5591,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.407,"skill":0.0133,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.1562,"skill":0.1562,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.2012,"skill":0.2012,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6909,"pending":0.24,"new_in_edition":true},{"engine":"this-that-1.2","name":"this-that 1.2","long_name":"FLock this-that 1.2","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":1881825088,"params_estimated":false,"params_from_name":false,"trained_bytes":3763692048,"model_url":"https://huggingface.co/flock-io/this-that-model-1.2","code_url":"https://github.com/FLock-io/this-that-model","base_model":"Mapika/decider-2b","base_url":"https://huggingface.co/Mapika/decider-2b","weights_repo":"flock-io/this-that-model-1.2","served_checkpoint":null},"completed":120615,"counts":{"ok":120615},"lifted":false,"calibration":{"acc":0.5703,"conf":0.7846,"ece":0.2142,"brier":0.6133,"over95":0.0761,"n":35089,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0115,"conf":0.0848,"acc":0.0291},{"lo":0.1,"w":0.0323,"conf":0.1451,"acc":0.0451},{"lo":0.2,"w":0.0174,"conf":0.2502,"acc":0.1757},{"lo":0.3,"w":0.0262,"conf":0.3546,"acc":0.1461},{"lo":0.4,"w":0.0373,"conf":0.4552,"acc":0.3002},{"lo":0.5,"w":0.0922,"conf":0.5488,"acc":0.4085},{"lo":0.6,"w":0.082,"conf":0.649,"acc":0.4596},{"lo":0.7,"w":0.1191,"conf":0.744,"acc":0.396},{"lo":0.8,"w":0.1098,"conf":0.8547,"acc":0.5823},{"lo":0.9,"w":0.4721,"conf":0.9786,"acc":0.771}],"areas":{"knowledge":{"ece":0.256,"acc":0.474,"conf":0.73},"language":{"ece":0.1613,"acc":0.7028,"conf":0.8627},"retrieval":{"ece":0.2394,"acc":0.6524,"conf":0.8918},"tools":{"ece":0.1689,"acc":0.6815,"conf":0.8496},"arts":{"ece":0.2192,"acc":0.4308,"conf":0.65}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":38.6,"p95":80.0,"mean":47.3},"results":{"1":{"score":0.8235,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":39.6},"2":{"score":0.37,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":193.4},"3":{"score":0.5236,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":150.5},"4":{"score":0.7157,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":42.1},"5":{"score":0.6608,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":45.7},"6":{"score":0.6848,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":54.9},"9":{"score":0,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":53.5},"10":{"score":0.5428,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":42.0},"11":{"score":0.6468,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":82.3},"12":{"score":0.4966,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":37.3},"20":{"score":0.5526,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":37.3},"21":{"score":0.6043,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":37.1},"22":{"score":0.0135,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":119.2},"23":{"score":0.5808,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.8},"24":{"score":0.5958,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":38.0},"25":{"score":0.3316,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":38.3},"26":{"score":0.9213,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":38.0},"27":{"score":0.8319,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":37.9},"28":{"score":0.674,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":38.0},"29":{"score":0.8067,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":38.5},"30":{"score":0.2817,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":38.3},"31":{"score":0.0868,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":40.1},"32":{"score":0.5465,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":41.2},"33":{"score":0.3127,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":41.7},"34":{"score":0.1,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":37.2},"36":{"score":0.078,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":281.0},"37":{"score":0.3783,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":38.5},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":154.4},"39":{"score":0.747,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":38.1},"40":{"score":0.1942,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":37.3,"tracks":[{"track":"A · Arabic","score":0.0372,"headline":false},{"track":"A · English","score":0.1942,"headline":true},{"track":"C · Arabic pairs","score":0.33,"headline":false},{"track":"C · English pairs","score":0.51,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.4996,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":37.3},"42":{"score":0.6757,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":39.0},"43":{"score":0.3193,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":37.4},"44":{"score":0.5642,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":37.5},"45":{"score":0.1238,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":38.1},"48":{"score":0.3267,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":40.4},"50":{"score":0.4057,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":41.5},"56":{"score":0.5,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":48.8,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.3718,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":38.0,"edition_added":"v0.2","jev":0.827},"58":{"score":0.4868,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":37.7,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.5709,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":40.5,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.4356,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":37.7,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.1417,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":60.1,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.597,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":39.0,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":43.76,"balanced_skill":25.03,"breadth_skill":23.61},"frozen_scores":{"balanced_raw":32.3,"balanced_skill":14.97,"breadth_skill":13.06},"categories":[{"id":"knowledge","raw":0.3425,"skill":0.1504,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.4882,"skill":0.3072,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.4961,"skill":0.2939,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.4955,"skill":0.3785,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.3659,"skill":0.1217,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.8235,"skill":0.7617,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.37,"skill":0.3063,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.6848,"skill":0.2606,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.6501,"headline":false},{"track":"RouterBench-5shot","score":0.7196,"headline":false}]},"11":{"raw":0.6468,"skill":0.4892,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.5526,"skill":0.1052,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6043,"skill":0.2085,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0126,"skill":0.0049,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.5505,"skill":0.101,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.617,"skill":0.4893,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3316,"skill":0.1088,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.2817,"skill":0.129,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.3442,"headline":false},{"track":"GSM8K-10choice","score":0.2191,"headline":false}]},"31":{"raw":0.0868,"skill":0.0054,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.078,"skill":0.0332,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.3783,"skill":0.2201,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.1942,"skill":0.0,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0372,"headline":false},{"track":"A · English","score":0.1942,"headline":true},{"track":"C · Arabic pairs","score":0.33,"headline":false},{"track":"C · English pairs","score":0.51,"headline":false}]},"41":{"raw":0.4996,"skill":0.2494,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.3193,"skill":0.0,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.5642,"skill":0.1284,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4057,"skill":0.1375,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.5,"skill":0.0,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.3718,"skill":0.2934,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.4868,"skill":0.2561,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.5709,"skill":0.4279,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.4356,"skill":0.2945,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.1417,"skill":0.0,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.597,"skill":0.194,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.5465,"skill":0.279,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3127,"skill":0.3036,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1238,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.4966,"skill":0.246,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.674,"skill":0.348,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8067,"skill":0.7423,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.747,"skill":0.6279,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.6757,"skill":0.3691,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7157,"skill":0.712,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.6608,"skill":0.6588,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.5428,"skill":0.2393,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.5236,"skill":0.5144,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24},{"engine":"verdict-8af2496e","name":"Verdict","long_name":"Verdict","variant":null,"meta":{"kind":"full fine-tune","technique":"GLi family","technique_name":"GLiClass","served_params":151378176,"params_estimated":false,"params_from_name":false,"trained_bytes":605529340,"model_url":"https://huggingface.co/heman10x/rlcd-modernbert-151m","code_url":"https://github.com/Heman10x-NGU/Verdict-open-jev","base_model":"knowledgator/gliclass-modern-base-v2.0","base_url":"https://huggingface.co/knowledgator/gliclass-modern-base-v2.0","weights_repo":"heman10x/rlcd-modernbert-151m","served_checkpoint":null},"completed":121057,"counts":{"ok":49595,"abstained":56399,"unsupported":15063},"lifted":false,"calibration":{"acc":0.3907,"conf":0.5206,"ece":0.13,"brier":0.6737,"over95":0,"n":58190,"benchmarks":29,"rel":[{"lo":0.0,"w":0.0146,"conf":0.0763,"acc":0.0637},{"lo":0.1,"w":0.0143,"conf":0.1474,"acc":0.1513},{"lo":0.2,"w":0.0308,"conf":0.2639,"acc":0.1987},{"lo":0.3,"w":0.122,"conf":0.3563,"acc":0.2337},{"lo":0.4,"w":0.2219,"conf":0.4536,"acc":0.452},{"lo":0.5,"w":0.2882,"conf":0.5445,"acc":0.3913},{"lo":0.6,"w":0.2347,"conf":0.6472,"acc":0.4543},{"lo":0.7,"w":0.0538,"conf":0.7378,"acc":0.453},{"lo":0.8,"w":0.0184,"conf":0.8387,"acc":0.4591},{"lo":0.9,"w":0.0013,"conf":0.9127,"acc":0.332}],"areas":{"knowledge":{"ece":0.1555,"acc":0.3722,"conf":0.4933},"language":{"ece":0.0844,"acc":0.4755,"conf":0.5469},"retrieval":{"ece":0.3057,"acc":0.1527,"conf":0.4584},"tools":{"ece":0.3816,"acc":0.241,"conf":0.6226},"arts":{"ece":0.0817,"acc":0.4493,"conf":0.5238}}},"gaps":{"share":0.5903,"reasons":[{"reason":"chose its own abstain option instead of an answer","share":0.4659},{"reason":"supports only 2–24 answer options per question","share":0.1217},{"reason":"prompt longer than its 8,192-token context window","share":0.0027}]},"latency":{"median":10.6,"p95":61.4,"mean":23.0},"results":{"1":{"score":0.3878,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":17.7},"2":{"score":0.134,"jev":0.4658,"metric":"nDCG@10","requests":1000,"answered":705,"unsupported":11,"errors":0,"median_ms":157.4},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.5084,"jev":0.7566,"metric":"selected quality (quality objective)","requests":10000,"answered":7536,"unsupported":0,"errors":0,"median_ms":41.9},"9":{"score":0,"jev":1,"metric":"case exact accuracy","requests":160,"answered":1,"unsupported":0,"errors":0,"median_ms":139.9},"10":{"score":0.0792,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":17.7},"11":{"score":0.2116,"jev":0.7251,"metric":"macro-F1","requests":123,"answered":31,"unsupported":1,"errors":0,"median_ms":364.3},"12":{"score":0.1492,"jev":0.7416,"metric":"macro-F1","requests":3200,"answered":328,"unsupported":0,"errors":0,"median_ms":9.9},"20":{"score":0.4761,"jev":0.9046,"metric":"accuracy","requests":5000,"answered":2283,"unsupported":0,"errors":0,"median_ms":9.9},"21":{"score":0.386,"jev":0.6228,"metric":"accuracy","requests":2628,"answered":228,"unsupported":0,"errors":0,"median_ms":9.6},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.6562,"jev":0.6979,"metric":"accuracy","requests":5000,"answered":96,"unsupported":0,"errors":0,"median_ms":10.2},"24":{"score":0.3088,"jev":0.9094,"metric":"accuracy","requests":14033,"answered":4812,"unsupported":0,"errors":0,"median_ms":9.7},"25":{"score":0.2874,"jev":0.7816,"metric":"accuracy","requests":196,"answered":87,"unsupported":0,"errors":0,"median_ms":10.0},"26":{"score":0.5206,"jev":0.9905,"metric":"accuracy","requests":2376,"answered":632,"unsupported":0,"errors":0,"median_ms":9.5},"27":{"score":0.2738,"jev":0.9911,"metric":"accuracy","requests":1172,"answered":336,"unsupported":0,"errors":0,"median_ms":9.6},"28":{"score":0.494,"jev":0.9186,"metric":"accuracy","requests":1267,"answered":1241,"unsupported":0,"errors":0,"median_ms":8.7},"29":{"score":0.3506,"jev":0.9561,"metric":"accuracy","requests":10042,"answered":6896,"unsupported":0,"errors":0,"median_ms":10.3},"30":{"score":0.2333,"jev":0.8067,"metric":"accuracy","requests":2638,"answered":474,"unsupported":0,"errors":0,"median_ms":10.0},"31":{"score":0.1735,"jev":0.2913,"metric":"accuracy","requests":5000,"answered":1349,"unsupported":3641,"errors":0,"median_ms":11.1},"32":{"score":0.4101,"jev":0.6646,"metric":"accuracy","requests":752,"answered":656,"unsupported":0,"errors":0,"median_ms":18.0},"33":{"score":0.0119,"jev":0.2143,"metric":"case exact accuracy","requests":1650,"answered":84,"unsupported":0,"errors":0,"median_ms":73.7},"34":{"score":0.25,"jev":0.5,"metric":"accuracy","requests":10,"answered":8,"unsupported":0,"errors":0,"median_ms":10.1},"36":{"score":0.0266,"jev":0.1322,"metric":"nDCG@10","requests":550,"answered":207,"unsupported":1,"errors":0,"median_ms":227.8},"37":{"score":0.1277,"jev":0.5555,"metric":"macro-F1","requests":5000,"answered":2146,"unsupported":0,"errors":0,"median_ms":10.4},"38":{"score":0,"jev":0.0627,"metric":"case exact accuracy","requests":1565,"answered":1559,"unsupported":0,"errors":0,"median_ms":77.5},"39":{"score":0.4211,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":10.7},"40":{"score":0.1012,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":3614,"unsupported":0,"errors":0,"median_ms":8.6,"tracks":[{"track":"A · Arabic","score":0.0,"headline":false},{"track":"A · English","score":0.1012,"headline":true},{"track":"C · Arabic pairs","score":0.195,"headline":false},{"track":"C · English pairs","score":0.1,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.3337,"jev":0.6661,"metric":"macro-F1","requests":3006,"answered":1388,"unsupported":0,"errors":0,"median_ms":10.0},"42":{"score":0.5155,"jev":0.8627,"metric":"macro-F1","requests":5500,"answered":891,"unsupported":0,"errors":0,"median_ms":10.5},"43":{"score":0.3926,"jev":0.7375,"metric":"accuracy","requests":570,"answered":461,"unsupported":0,"errors":0,"median_ms":9.8},"44":{"score":0.3788,"jev":0.7879,"metric":"accuracy","requests":5000,"answered":66,"unsupported":0,"errors":0,"median_ms":9.8},"45":{"score":0.1496,"jev":0.2044,"metric":"accuracy","requests":501,"answered":274,"unsupported":4,"errors":0,"median_ms":10.3},"48":{"score":0.227,"jev":0.1651,"metric":"Brier (lower is better)","requests":10139,"answered":5067,"unsupported":0,"errors":0,"median_ms":14.8},"50":{"score":0.3769,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1661,"unsupported":0,"errors":0,"median_ms":15.5},"56":{"score":0.5532,"metric":"accuracy","requests":2000,"answered":47,"unsupported":0,"errors":1953,"median_ms":29.4,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.1345,"metric":"accuracy","requests":12032,"answered":7116,"unsupported":0,"errors":4916,"median_ms":11.2,"edition_added":"v0.2","jev":0.827},"58":{"score":0.3361,"metric":"accuracy","requests":5507,"answered":5444,"unsupported":0,"errors":63,"median_ms":11.0,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.3681,"metric":"accuracy","requests":3652,"answered":3540,"unsupported":0,"errors":112,"median_ms":13.8,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.8,"metric":"accuracy","requests":528,"answered":5,"unsupported":0,"errors":523,"median_ms":11.2,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.119,"metric":"F1 on hallucinated class","requests":2700,"answered":2666,"unsupported":0,"errors":34,"median_ms":14.8,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.5173,"metric":"accuracy","requests":4000,"answered":1562,"unsupported":0,"errors":2438,"median_ms":10.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":12.4,"balanced_skill":1.82,"breadth_skill":1.69},"frozen_scores":{"balanced_raw":8.94,"balanced_skill":1.07,"breadth_skill":0.99},"categories":[{"id":"knowledge","raw":0.1362,"skill":0.0032,"coverage":0.4643,"pending":0,"n":10},{"id":"language","raw":0.1656,"skill":0.0149,"coverage":0.6414,"pending":0,"n":10},{"id":"retrieval","raw":0.0491,"skill":0.0,"coverage":0.3171,"pending":0,"n":7},{"id":"tools","raw":0.19,"skill":0.0532,"coverage":0.5453,"pending":0,"n":6},{"id":"arts","raw":0.079,"skill":0.0195,"coverage":0.2742,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.3878,"skill":0.1736,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.0945,"skill":0.0029,"coverage":0.705,"random":0.0918,"tracks":[]},"6":{"raw":0.3006,"skill":0.0,"coverage":0.591,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.2851,"headline":false},{"track":"RouterBench-5shot","score":0.316,"headline":false}]},"11":{"raw":0.0631,"skill":0.0,"coverage":0.252,"random":0.3085,"tracks":[]},"20":{"raw":0.0858,"skill":0.0,"coverage":0.3132,"random":0.5,"tracks":[]},"21":{"raw":0.0335,"skill":0.0,"coverage":0.0868,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.0069,"skill":0.0,"coverage":0.0192,"random":0.5,"tracks":[]},"24":{"raw":0.1001,"skill":0.0,"coverage":0.3429,"random":0.25,"tracks":[]},"25":{"raw":0.1276,"skill":0.0,"coverage":0.4439,"random":0.25,"tracks":[]},"30":{"raw":0.0133,"skill":0.0,"coverage":0.0569,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.0159,"headline":false},{"track":"GSM8K-10choice","score":0.0106,"headline":false}]},"31":{"raw":0.0468,"skill":0.0,"coverage":0.2698,"random":0.0819,"tracks":[]},"36":{"raw":0.01,"skill":0.0,"coverage":0.3764,"random":0.0464,"tracks":[]},"37":{"raw":0.0393,"skill":0.0,"coverage":0.4292,"random":0.2027,"tracks":[]},"40":{"raw":0.1012,"skill":0.0,"coverage":0.7857,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.0,"headline":false},{"track":"A · English","score":0.1012,"headline":true},{"track":"C · Arabic pairs","score":0.195,"headline":false},{"track":"C · English pairs","score":0.1,"headline":false}]},"41":{"raw":0.1301,"skill":0.0,"coverage":0.4617,"random":0.3333,"tracks":[]},"43":{"raw":0.3175,"skill":0.0,"coverage":0.8088,"random":0.3697,"tracks":[]},"44":{"raw":0.005,"skill":0.0,"coverage":0.0132,"random":0.5,"tracks":[]},"50":{"raw":0.3735,"skill":0.0908,"coverage":0.9911,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.013,"skill":0.0,"coverage":0.0235,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.0795,"skill":0.0,"coverage":0.5914,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.3323,"skill":0.0322,"coverage":0.9886,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.3568,"skill":0.1424,"coverage":0.9693,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.0076,"skill":0.0,"coverage":0.0095,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.1175,"skill":0.0,"coverage":0.9874,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.202,"skill":0.0,"coverage":0.3905,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.3577,"skill":0.0,"coverage":0.8723,"random":0.371,"tracks":[]},"33":{"raw":0.0006,"skill":0.0,"coverage":0.0509,"random":0.0131,"tracks":[]},"45":{"raw":0.0818,"skill":0.0,"coverage":0.5469,"random":0.1641,"tracks":[]},"12":{"raw":0.0153,"skill":0.0,"coverage":0.1025,"random":0.3324,"tracks":[]},"28":{"raw":0.4839,"skill":0.0,"coverage":0.9795,"random":0.5,"tracks":[]},"29":{"raw":0.2408,"skill":0.0,"coverage":0.6867,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":0.9962,"random":0.0,"tracks":[]},"39":{"raw":0.4211,"skill":0.1486,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.0835,"skill":0.0,"coverage":0.162,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.0792,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":0.0063,"random":0.0,"tracks":[]},"48":{"raw":0.046,"skill":0.046,"coverage":0.4998,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.3101,"pending":0.24},{"engine":"vllm-pr57250","name":"mmastrac diffusiongemma (vLLM PR 57250)","long_name":"mmastrac diffgemma","variant":null,"meta":{"kind":"inference technique","technique":"diffusion","technique_name":"diffusion","served_params":25823778864,"params_estimated":false,"params_from_name":false,"trained_bytes":null,"model_url":null,"code_url":"https://github.com/vllm-project/vllm/pull/57250","base_model":"google/diffusiongemma-26B-A4B-it","base_url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it","weights_repo":null,"served_checkpoint":"google/diffusiongemma-26B-A4B-it"},"completed":121057,"counts":{"ok":106518,"unsupported":14539},"lifted":false,"calibration":{"acc":0.6271,"conf":0.84,"ece":0.2129,"brier":0.5676,"over95":0.0876,"n":70145,"benchmarks":29,"rel":[{"lo":0.0,"w":0,"conf":null,"acc":null},{"lo":0.1,"w":0.0006,"conf":0.1821,"acc":0.2358},{"lo":0.2,"w":0.0077,"conf":0.2643,"acc":0.0903},{"lo":0.3,"w":0.0192,"conf":0.3628,"acc":0.2245},{"lo":0.4,"w":0.0315,"conf":0.459,"acc":0.2679},{"lo":0.5,"w":0.1098,"conf":0.541,"acc":0.2528},{"lo":0.6,"w":0.0776,"conf":0.6521,"acc":0.4727},{"lo":0.7,"w":0.081,"conf":0.7504,"acc":0.5228},{"lo":0.8,"w":0.1083,"conf":0.8526,"acc":0.5651},{"lo":0.9,"w":0.5644,"conf":0.9804,"acc":0.7895}],"areas":{"knowledge":{"ece":0.2591,"acc":0.5188,"conf":0.7777},"language":{"ece":0.1587,"acc":0.7472,"conf":0.9028},"retrieval":{"ece":0.2883,"acc":0.5026,"conf":0.7909},"tools":{"ece":0.0763,"acc":0.8329,"conf":0.9092},"arts":{"ece":0.2341,"acc":0.6413,"conf":0.8755}}},"gaps":{"share":0.1201,"reasons":[{"reason":"supports at most 26 answer options per question","share":0.1198},{"reason":"structured_server: question 'q_78d37c943d': label 'C' is not a single token","share":0.0},{"reason":"structured_server: question 'q_c3e60b168d': label 'C' is not a single token","share":0.0}]},"latency":{"median":124.7,"p95":278.8,"mean":160.3},"results":{"1":{"score":0.9622,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":116.3},"2":{"score":0.3172,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":637.4},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.7988,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":247.3},"9":{"score":0,"jev":0.5372,"metric":"case exact accuracy","requests":160,"answered":121,"unsupported":39,"errors":0,"median_ms":556.6},"10":{"score":0.4299,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":108.0},"11":{"score":0.5632,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":405.4},"12":{"score":0.5574,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":116.2},"20":{"score":0.8854,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":121.3},"21":{"score":0.6084,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":118.7},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.6098,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":139.3},"24":{"score":0.7862,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":115.2},"25":{"score":0.4898,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":124.4},"26":{"score":0.9811,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":52.7},"27":{"score":0.9403,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":53.8},"28":{"score":0.708,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":116.2},"29":{"score":0.8173,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":58.2},"30":{"score":0.3639,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":126.3},"31":{"score":0.2173,"jev":0.2746,"metric":"accuracy","requests":5000,"answered":1588,"unsupported":3412,"errors":0,"median_ms":142.0},"32":{"score":0.625,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":93.6},"33":{"score":0.2564,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":174.6},"34":{"score":0.3,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":122.6},"36":{"score":0.0904,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":822.8},"37":{"score":0.5339,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":131.2},"38":{"score":0,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":572.4},"39":{"score":0.8979,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":139.1},"40":{"score":0.5895,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":112.8,"tracks":[{"track":"A · Arabic","score":0.4853,"headline":false},{"track":"A · English","score":0.5895,"headline":true},{"track":"C · Arabic pairs","score":0.815,"headline":false},{"track":"C · English pairs","score":0.955,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6086,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":121.0},"42":{"score":0.7446,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":134.9},"43":{"score":0.4526,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":119.6},"44":{"score":0.6356,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":116.9},"45":{"score":0.1038,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":127.8},"48":{"score":0.2141,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":144.9},"50":{"score":0.4666,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":157.5},"56":{"score":0.824,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":201.5,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.413,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":127.5,"edition_added":"v0.2","jev":0.827},"58":{"score":0.6644,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":117.1,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.4874,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":121.2,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.5739,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":126.3,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.6789,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":134.2,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.6913,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":131.6,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":45.77,"balanced_skill":31.1,"breadth_skill":30.33},"frozen_scores":{"balanced_raw":39.83,"balanced_skill":26.64,"breadth_skill":22.49},"categories":[{"id":"knowledge","raw":0.4073,"skill":0.2455,"coverage":0.9318,"pending":0,"n":10},{"id":"language","raw":0.6165,"skill":0.457,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3671,"skill":0.2205,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.4276,"skill":0.3399,"coverage":0.7927,"pending":0,"n":6},{"id":"arts","raw":0.4698,"skill":0.2922,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9622,"skill":0.949,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.3172,"skill":0.2482,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7988,"skill":0.5256,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7852,"headline":false},{"track":"RouterBench-5shot","score":0.8123,"headline":false}]},"11":{"raw":0.5632,"skill":0.3683,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8896,"skill":0.7791,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6084,"skill":0.2169,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.6063,"skill":0.2125,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.7911,"skill":0.7215,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4898,"skill":0.3197,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.3639,"skill":0.2284,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.4162,"headline":false},{"track":"GSM8K-10choice","score":0.3116,"headline":false}]},"31":{"raw":0.069,"skill":0.0,"coverage":0.3176,"random":0.0819,"tracks":[]},"36":{"raw":0.0904,"skill":0.0461,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5339,"skill":0.4154,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5895,"skill":0.4719,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4853,"headline":false},{"track":"A · English","score":0.5895,"headline":true},{"track":"C · Arabic pairs","score":0.815,"headline":false},{"track":"C · English pairs","score":0.955,"headline":false}]},"41":{"raw":0.6086,"skill":0.4128,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.4526,"skill":0.1315,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6356,"skill":0.2712,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4666,"skill":0.2258,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.824,"skill":0.648,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.413,"skill":0.3398,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.6644,"skill":0.5136,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.4874,"skill":0.3165,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.5739,"skill":0.4674,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.6789,"skill":0.4546,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.6913,"skill":0.3826,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.625,"skill":0.4038,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2564,"skill":0.2465,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1038,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.5574,"skill":0.337,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.708,"skill":0.416,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8173,"skill":0.7564,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0,"skill":0.0,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8979,"skill":0.8498,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7446,"skill":0.5031,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.4299,"skill":0.0514,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.0,"skill":0.0,"coverage":0.7562,"random":0.0,"tracks":[]},"48":{"raw":0.1436,"skill":0.1436,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6927,"pending":0.24,"new_in_edition":true},{"engine":"winnow-12b-q8","name":"Winnow-12B","long_name":"Winnow-12B","variant":"Q8_0","meta":{"kind":"LoRA","technique":"autoregressive","technique_name":"autoregressive","served_params":11959730224,"params_estimated":false,"params_from_name":false,"trained_bytes":12669646592,"model_url":"https://huggingface.co/EldanRing/Winnow-12B","code_url":"https://github.com/EldanRing/winnow-inference","base_model":"google/gemma-4-12B","base_url":"https://huggingface.co/google/gemma-4-12B","weights_repo":"EldanRing/Winnow-12B","served_checkpoint":"EldanRing/Winnow-12B"},"completed":121057,"counts":{"ok":121057},"lifted":true,"calibration":{"acc":0.6494,"conf":0.8305,"ece":0.1811,"brier":0.5109,"over95":0.0605,"n":72595,"benchmarks":33,"rel":[{"lo":0.0,"w":0.0034,"conf":0.0806,"acc":0.0},{"lo":0.1,"w":0.0148,"conf":0.1522,"acc":0.1179},{"lo":0.2,"w":0.022,"conf":0.2538,"acc":0.1849},{"lo":0.3,"w":0.0268,"conf":0.3543,"acc":0.2116},{"lo":0.4,"w":0.0325,"conf":0.4537,"acc":0.2994},{"lo":0.5,"w":0.0561,"conf":0.5508,"acc":0.435},{"lo":0.6,"w":0.0827,"conf":0.6343,"acc":0.2995},{"lo":0.7,"w":0.0737,"conf":0.7509,"acc":0.4814},{"lo":0.8,"w":0.1009,"conf":0.8558,"acc":0.553},{"lo":0.9,"w":0.5873,"conf":0.976,"acc":0.8306}],"areas":{"knowledge":{"ece":0.204,"acc":0.5611,"conf":0.7638},"language":{"ece":0.1134,"acc":0.7995,"conf":0.9129},"retrieval":{"ece":0.3164,"acc":0.6017,"conf":0.918},"tools":{"ece":0.1021,"acc":0.8499,"conf":0.9437},"arts":{"ece":0.1953,"acc":0.5092,"conf":0.7045}}},"gaps":{"share":0.0,"reasons":[]},"latency":{"median":64.3,"p95":558.0,"mean":156.4},"results":{"1":{"score":0.9475,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":117.4},"2":{"score":0.4466,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":1036.2},"3":{"score":0.6732,"jev":0.8819,"metric":"accuracy","requests":508,"answered":508,"unsupported":0,"errors":0,"median_ms":798.3},"4":{"score":0.7511,"jev":0.7974,"metric":"macro-F1","requests":3080,"answered":3080,"unsupported":0,"errors":0,"median_ms":149.8},"5":{"score":0.8695,"jev":0.8927,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":208.4},"6":{"score":0.7976,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":290.1},"9":{"score":0.5437,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":356.2},"10":{"score":0.1189,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":140.6},"11":{"score":0.7289,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":479.9},"12":{"score":0.6674,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":32.4},"20":{"score":0.8314,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":34.0},"21":{"score":0.6126,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":24.9},"22":{"score":0.081,"jev":0.181,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":671.2},"23":{"score":0.6018,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":30.9},"24":{"score":0.7732,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":28.3},"25":{"score":0.3878,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":38.2},"26":{"score":0.9798,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":25.2},"27":{"score":0.948,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":26.7},"28":{"score":0.7411,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":23.6},"29":{"score":0.8512,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":41.1},"30":{"score":0.539,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":62.9},"31":{"score":0.1734,"jev":0.1722,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":133.3},"32":{"score":0.6104,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":126.6},"33":{"score":0.3358,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":151.1},"34":{"score":0.4,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":38.3},"36":{"score":0.1811,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":1319.6},"37":{"score":0.5304,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":73.4},"38":{"score":0.0075,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":958.6},"39":{"score":0.9174,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":66.9},"40":{"score":0.5773,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":45.5,"tracks":[{"track":"A · Arabic","score":0.4759,"headline":false},{"track":"A · English","score":0.5773,"headline":true},{"track":"C · Arabic pairs","score":0.84,"headline":false},{"track":"C · English pairs","score":0.96,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6863,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":34.9},"42":{"score":0.8294,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":87.0},"43":{"score":0.6947,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":56.4},"44":{"score":0.6856,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":33.2},"45":{"score":0.1038,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":41.0},"48":{"score":0.1926,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":112.9},"50":{"score":0.4183,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":126.1},"56":{"score":0.659,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":147.2,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.5483,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":63.1,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7129,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":58.1,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.8324,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":111.1,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.6477,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":61.5,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7708,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":120.1,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.7205,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":81.4,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":58.0,"balanced_skill":45.05,"breadth_skill":43.63},"frozen_scores":{"balanced_raw":43.36,"balanced_skill":31.11,"breadth_skill":26.44},"categories":[{"id":"knowledge","raw":0.4792,"skill":0.3401,"coverage":1.0,"pending":0,"n":10},{"id":"language","raw":0.6777,"skill":0.5544,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.5472,"skill":0.4183,"coverage":1.0,"pending":0,"n":7},{"id":"tools","raw":0.7068,"skill":0.6382,"coverage":1.0,"pending":0,"n":6},{"id":"arts","raw":0.489,"skill":0.3017,"coverage":1.0,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9475,"skill":0.9291,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4466,"skill":0.3906,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.7976,"skill":0.5225,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7855,"headline":false},{"track":"RouterBench-5shot","score":0.8097,"headline":false}]},"11":{"raw":0.7289,"skill":0.6079,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.8342,"skill":0.6684,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6126,"skill":0.2253,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0804,"skill":0.0732,"coverage":1,"random":0.0078,"tracks":[]},"23":{"raw":0.6002,"skill":0.2004,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.782,"skill":0.7094,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.3878,"skill":0.1837,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.539,"skill":0.4471,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.6338,"headline":false},{"track":"GSM8K-10choice","score":0.4443,"headline":false}]},"31":{"raw":0.1734,"skill":0.0997,"coverage":1,"random":0.0819,"tracks":[]},"36":{"raw":0.1811,"skill":0.1413,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.5304,"skill":0.411,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5773,"skill":0.4562,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.4759,"headline":false},{"track":"A · English","score":0.5773,"headline":true},{"track":"C · Arabic pairs","score":0.84,"headline":false},{"track":"C · English pairs","score":0.96,"headline":false}]},"41":{"raw":0.6863,"skill":0.5295,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.6947,"skill":0.5157,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6856,"skill":0.3712,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4183,"skill":0.1557,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.659,"skill":0.318,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.5483,"skill":0.492,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.7129,"skill":0.5839,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.8324,"skill":0.7765,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.6477,"skill":0.5596,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7708,"skill":0.6107,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7205,"skill":0.441,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.6104,"skill":0.3806,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.3358,"skill":0.327,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1038,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6674,"skill":0.5018,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.7411,"skill":0.4822,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.8512,"skill":0.8016,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.0075,"skill":0.0075,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.9174,"skill":0.8785,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.8294,"skill":0.6681,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.7511,"skill":0.7479,"coverage":1.0,"random":0.0127,"tracks":[]},"5":{"raw":0.8695,"skill":0.8687,"coverage":1.0,"random":0.006,"tracks":[]},"10":{"raw":0.1189,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.6732,"skill":0.6669,"coverage":1.0,"random":0.0189,"tracks":[]},"9":{"raw":0.5437,"skill":0.5437,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.2296,"skill":0.2296,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.76,"pending":0.24,"new_in_edition":true},{"engine":"xor","name":"Xor","long_name":"Xor Qwen3.6-35B-A3B","variant":null,"meta":{"kind":"full fine-tune","technique":"autoregressive","technique_name":"autoregressive","served_params":35951822704,"params_estimated":false,"params_from_name":false,"trained_bytes":70214495944,"model_url":"https://huggingface.co/juspay/xor","code_url":null,"base_model":"Qwen/Qwen3.6-35B-A3B","base_url":"https://huggingface.co/Qwen/Qwen3.6-35B-A3B","weights_repo":"juspay/xor","served_checkpoint":null},"completed":120615,"counts":{"ok":106115,"unsupported":14500},"lifted":false,"calibration":{"acc":0.7193,"conf":0.7139,"ece":0.0232,"brier":0.3831,"over95":0.004,"n":32694,"benchmarks":29,"rel":[{"lo":0.0,"w":0.0001,"conf":0.0999,"acc":0.0},{"lo":0.1,"w":0.0101,"conf":0.1611,"acc":0.1752},{"lo":0.2,"w":0.0195,"conf":0.25,"acc":0.243},{"lo":0.3,"w":0.0449,"conf":0.3535,"acc":0.2913},{"lo":0.4,"w":0.0619,"conf":0.4506,"acc":0.3605},{"lo":0.5,"w":0.1841,"conf":0.555,"acc":0.618},{"lo":0.6,"w":0.1421,"conf":0.6513,"acc":0.6501},{"lo":0.7,"w":0.1376,"conf":0.7497,"acc":0.7482},{"lo":0.8,"w":0.1465,"conf":0.8494,"acc":0.8602},{"lo":0.9,"w":0.2531,"conf":0.9533,"acc":0.9572}],"areas":{"knowledge":{"ece":0.0644,"acc":0.6703,"conf":0.6673},"language":{"ece":0.0565,"acc":0.8153,"conf":0.7608},"retrieval":{"ece":0.2824,"acc":0.4149,"conf":0.6973},"tools":{"ece":0.0637,"acc":0.9715,"conf":0.909},"arts":{"ece":0.0302,"acc":0.6767,"conf":0.6588}}},"gaps":{"share":0.1202,"reasons":[{"reason":"xor server 422: q1: 2..26 options, got 151","share":0.0456},{"reason":"xor server 422: q1: 2..26 options, got 77","share":0.0255},{"reason":"xor server 422: chord: 2..26 options, got 129","share":0.0166}]},"latency":{"median":179.5,"p95":1069.0,"mean":339.9},"results":{"1":{"score":0.9575,"jev":0.9575,"metric":"case exact accuracy","requests":1694,"answered":1694,"unsupported":0,"errors":0,"median_ms":614.4},"2":{"score":0.4359,"jev":0.4472,"metric":"nDCG@10","requests":1000,"answered":1000,"unsupported":0,"errors":0,"median_ms":2374.9},"3":{"score":null,"jev":null,"metric":null,"requests":508,"answered":0,"unsupported":508,"errors":0,"median_ms":null},"4":{"score":null,"jev":null,"metric":null,"requests":3080,"answered":0,"unsupported":3080,"errors":0,"median_ms":null},"5":{"score":null,"jev":null,"metric":null,"requests":5500,"answered":0,"unsupported":5500,"errors":0,"median_ms":null},"6":{"score":0.8002,"jev":0.7992,"metric":"selected quality (quality objective)","requests":10000,"answered":10000,"unsupported":0,"errors":0,"median_ms":849.2},"9":{"score":0.5125,"jev":0.525,"metric":"case exact accuracy","requests":160,"answered":160,"unsupported":0,"errors":0,"median_ms":1019.1},"10":{"score":0.2381,"jev":0.4295,"metric":"macro-F1","requests":2500,"answered":2500,"unsupported":0,"errors":0,"median_ms":379.4},"11":{"score":0.7429,"jev":0.7169,"metric":"macro-F1","requests":123,"answered":123,"unsupported":0,"errors":0,"median_ms":987.7},"12":{"score":0.6851,"jev":0.7479,"metric":"macro-F1","requests":3200,"answered":3200,"unsupported":0,"errors":0,"median_ms":159.6},"20":{"score":0.9662,"jev":0.906,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":139.7},"21":{"score":0.6374,"jev":0.6187,"metric":"accuracy","requests":2628,"answered":2628,"unsupported":0,"errors":0,"median_ms":129.8},"22":{"score":null,"jev":null,"metric":null,"requests":2000,"answered":0,"unsupported":2000,"errors":0,"median_ms":null},"23":{"score":0.6304,"jev":0.6474,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":199.9},"24":{"score":0.8405,"jev":0.9173,"metric":"accuracy","requests":14033,"answered":14033,"unsupported":0,"errors":0,"median_ms":149.8},"25":{"score":0.4898,"jev":0.7857,"metric":"accuracy","requests":196,"answered":196,"unsupported":0,"errors":0,"median_ms":149.9},"26":{"score":0.9903,"jev":0.9933,"metric":"accuracy","requests":2376,"answered":2376,"unsupported":0,"errors":0,"median_ms":139.8},"27":{"score":0.9608,"jev":0.9778,"metric":"accuracy","requests":1172,"answered":1172,"unsupported":0,"errors":0,"median_ms":139.8},"28":{"score":0.8082,"jev":0.9195,"metric":"accuracy","requests":1267,"answered":1267,"unsupported":0,"errors":0,"median_ms":139.8},"29":{"score":0.9364,"jev":0.9452,"metric":"accuracy","requests":10042,"answered":10042,"unsupported":0,"errors":0,"median_ms":169.7},"30":{"score":0.7191,"jev":0.7987,"metric":"accuracy","requests":2638,"answered":2638,"unsupported":0,"errors":0,"median_ms":169.5},"31":{"score":0.2286,"jev":0.2746,"metric":"accuracy","requests":5000,"answered":1588,"unsupported":3412,"errors":0,"median_ms":179.6},"32":{"score":0.621,"jev":0.6609,"metric":"accuracy","requests":752,"answered":752,"unsupported":0,"errors":0,"median_ms":269.1},"33":{"score":0.2467,"jev":0.2642,"metric":"case exact accuracy","requests":1650,"answered":1650,"unsupported":0,"errors":0,"median_ms":708.9},"34":{"score":0.4,"jev":0.5,"metric":"accuracy","requests":10,"answered":10,"unsupported":0,"errors":0,"median_ms":1453.4},"36":{"score":0.1693,"jev":0.1901,"metric":"nDCG@10","requests":550,"answered":550,"unsupported":0,"errors":0,"median_ms":2301.1},"37":{"score":0.4464,"jev":0.5521,"metric":"macro-F1","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":219.3},"38":{"score":0.01,"jev":0.0625,"metric":"case exact accuracy","requests":1565,"answered":1565,"unsupported":0,"errors":0,"median_ms":2092.2},"39":{"score":0.8874,"jev":0.8698,"metric":"macro-F1","requests":979,"answered":979,"unsupported":0,"errors":0,"median_ms":219.7},"40":{"score":0.5553,"jev":0.5051,"metric":"Sarcasm F1 · track A, English","requests":4600,"answered":4600,"unsupported":0,"errors":0,"median_ms":149.7,"tracks":[{"track":"A · Arabic","score":0.3082,"headline":false},{"track":"A · English","score":0.5553,"headline":true},{"track":"C · Arabic pairs","score":0.7,"headline":false},{"track":"C · English pairs","score":0.965,"headline":false}],"jev_note":"Jev on all track cases"},"41":{"score":0.6883,"jev":0.6463,"metric":"macro-F1","requests":3006,"answered":3006,"unsupported":0,"errors":0,"median_ms":159.8},"42":{"score":0.7255,"jev":0.8406,"metric":"macro-F1","requests":5500,"answered":5500,"unsupported":0,"errors":0,"median_ms":209.5},"43":{"score":0.6,"jev":0.7298,"metric":"accuracy","requests":570,"answered":570,"unsupported":0,"errors":0,"median_ms":169.8},"44":{"score":0.6368,"jev":0.7264,"metric":"accuracy","requests":5000,"answered":5000,"unsupported":0,"errors":0,"median_ms":159.8},"45":{"score":0.1357,"jev":0.2036,"metric":"accuracy","requests":501,"answered":501,"unsupported":0,"errors":0,"median_ms":209.1},"48":{"score":0.2011,"jev":0.1736,"metric":"Brier (lower is better)","requests":10139,"answered":10139,"unsupported":0,"errors":0,"median_ms":209.6},"50":{"score":0.4576,"jev":0.4594,"metric":"accuracy","requests":1676,"answered":1676,"unsupported":0,"errors":0,"median_ms":219.0},"56":{"score":0.654,"metric":"accuracy","requests":2000,"answered":2000,"unsupported":0,"errors":0,"median_ms":479.7,"edition_added":"v0.2","jev":0.6255},"57":{"score":0.6261,"metric":"accuracy","requests":12032,"answered":12032,"unsupported":0,"errors":0,"median_ms":159.9,"edition_added":"v0.2","jev":0.827},"58":{"score":0.7096,"metric":"accuracy","requests":5507,"answered":5507,"unsupported":0,"errors":0,"median_ms":159.9,"edition_added":"v0.2","jev":0.9292},"62":{"score":0.7993,"metric":"accuracy","requests":3652,"answered":3652,"unsupported":0,"errors":0,"median_ms":249.8,"edition_added":"v0.2","jev":0.8097},"64":{"score":0.714,"metric":"accuracy","requests":528,"answered":528,"unsupported":0,"errors":0,"median_ms":189.8,"edition_added":"v0.2","jev":0.7008},"59":{"score":0.7028,"metric":"F1 on hallucinated class","requests":2700,"answered":2700,"unsupported":0,"errors":0,"median_ms":219.8,"edition_added":"v0.2","jev":0.7653},"61":{"score":0.7422,"metric":"accuracy","requests":4000,"answered":4000,"unsupported":0,"errors":0,"median_ms":209.8,"edition_added":"v0.2","jev":0.7285}},"scores":{"balanced_raw":51.61,"balanced_skill":38.77,"breadth_skill":36.77},"frozen_scores":{"balanced_raw":43.73,"balanced_skill":32.06,"breadth_skill":26.44},"categories":[{"id":"knowledge","raw":0.4857,"skill":0.3412,"coverage":0.9318,"pending":0,"n":10},{"id":"language","raw":0.6742,"skill":0.5454,"coverage":1.0,"pending":0,"n":10},{"id":"retrieval","raw":0.3214,"skill":0.1753,"coverage":0.7143,"pending":0,"n":7},{"id":"tools","raw":0.5842,"skill":0.5159,"coverage":0.8333,"pending":0,"n":6},{"id":"arts","raw":0.515,"skill":0.3609,"coverage":0.8571,"pending":0,"n":7}],"benchmarks":{"1":{"raw":0.9575,"skill":0.9426,"coverage":1,"random":0.2592,"tracks":[]},"2":{"raw":0.4359,"skill":0.3789,"coverage":1,"random":0.0918,"tracks":[]},"6":{"raw":0.8002,"skill":0.5288,"coverage":1,"random":0.5245,"tracks":[{"track":"RouterBench-0shot","score":0.7872,"headline":false},{"track":"RouterBench-5shot","score":0.8132,"headline":false}]},"11":{"raw":0.7429,"skill":0.6282,"coverage":1,"random":0.3085,"tracks":[]},"20":{"raw":0.9664,"skill":0.9327,"coverage":1,"random":0.5,"tracks":[]},"21":{"raw":0.6374,"skill":0.2747,"coverage":1,"random":0.5,"tracks":[]},"22":{"raw":0.0,"skill":0.0,"coverage":0,"random":0.0078,"tracks":[]},"23":{"raw":0.634,"skill":0.2679,"coverage":1,"random":0.5,"tracks":[]},"24":{"raw":0.8468,"skill":0.7957,"coverage":1,"random":0.25,"tracks":[]},"25":{"raw":0.4898,"skill":0.3197,"coverage":1,"random":0.25,"tracks":[]},"30":{"raw":0.7191,"skill":0.6602,"coverage":1,"random":0.25,"tracks":[{"track":"GSM8K-4choice","score":0.7506,"headline":false},{"track":"GSM8K-10choice","score":0.6876,"headline":false}]},"31":{"raw":0.0726,"skill":0.0,"coverage":0.3176,"random":0.0819,"tracks":[]},"36":{"raw":0.1693,"skill":0.1289,"coverage":1,"random":0.0464,"tracks":[]},"37":{"raw":0.4464,"skill":0.3056,"coverage":1,"random":0.2027,"tracks":[]},"40":{"raw":0.5553,"skill":0.428,"coverage":1,"random":0.2227,"tracks":[{"track":"A · Arabic","score":0.3082,"headline":false},{"track":"A · English","score":0.5553,"headline":true},{"track":"C · Arabic pairs","score":0.7,"headline":false},{"track":"C · English pairs","score":0.965,"headline":false}]},"41":{"raw":0.6883,"skill":0.5324,"coverage":1,"random":0.3333,"tracks":[]},"43":{"raw":0.6,"skill":0.3654,"coverage":1,"random":0.3697,"tracks":[]},"44":{"raw":0.6368,"skill":0.2736,"coverage":1,"random":0.5,"tracks":[]},"50":{"raw":0.4576,"skill":0.2129,"coverage":1,"random":0.311,"tracks":[]},"7":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"16":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"13":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"15":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"17":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"19":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":null,"tracks":[]},"56":{"raw":0.654,"skill":0.308,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"57":{"raw":0.6261,"skill":0.5795,"coverage":1.0,"random":0.1109,"tracks":[],"edition_added":"v0.2"},"58":{"raw":0.7096,"skill":0.5791,"coverage":1.0,"random":0.3101,"tracks":[],"edition_added":"v0.2"},"62":{"raw":0.7993,"skill":0.7324,"coverage":1.0,"random":0.25,"tracks":[],"edition_added":"v0.2"},"64":{"raw":0.714,"skill":0.6425,"coverage":1.0,"random":0.2,"tracks":[],"edition_added":"v0.2"},"59":{"raw":0.7028,"skill":0.4952,"coverage":1.0,"random":0.4113,"tracks":[],"edition_added":"v0.2"},"61":{"raw":0.7422,"skill":0.4844,"coverage":1.0,"random":0.5,"tracks":[],"edition_added":"v0.2"},"32":{"raw":0.621,"skill":0.3975,"coverage":1.0,"random":0.371,"tracks":[]},"33":{"raw":0.2467,"skill":0.2367,"coverage":1.0,"random":0.0131,"tracks":[]},"45":{"raw":0.1357,"skill":0.0,"coverage":1.0,"random":0.1641,"tracks":[]},"12":{"raw":0.6851,"skill":0.5283,"coverage":1.0,"random":0.3324,"tracks":[]},"28":{"raw":0.8082,"skill":0.6164,"coverage":1.0,"random":0.5,"tracks":[]},"29":{"raw":0.9364,"skill":0.9152,"coverage":1.0,"random":0.25,"tracks":[]},"38":{"raw":0.01,"skill":0.01,"coverage":1.0,"random":0.0,"tracks":[]},"39":{"raw":0.8874,"skill":0.8344,"coverage":1.0,"random":0.3201,"tracks":[]},"42":{"raw":0.7255,"skill":0.466,"coverage":1.0,"random":0.486,"tracks":[]},"4":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0127,"tracks":[]},"5":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.006,"tracks":[]},"10":{"raw":0.2381,"skill":0.0,"coverage":1.0,"random":0.399,"tracks":[]},"3":{"raw":0.0,"skill":0.0,"coverage":0.0,"random":0.0189,"tracks":[]},"9":{"raw":0.5125,"skill":0.5125,"coverage":1.0,"random":0.0,"tracks":[]},"48":{"raw":0.1956,"skill":0.1956,"coverage":1.0,"random":0.25,"tracks":[],"index_rule":"vs baseline"}},"coverage":0.6927,"pending":0.24,"new_in_edition":true}]} \ No newline at end of file diff --git a/docs/training/jet-v5/protocol.md b/docs/training/jet-v5/protocol.md deleted file mode 100644 index 286024a629d7e4b8bbdae6f38adc438b5555d771..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/protocol.md +++ /dev/null @@ -1,44 +0,0 @@ -# Jet V5 protocol — Decision Index 0.2 coverage - -Protocol recorded before V5 training and test scoring. Qwen3-0.6B remains the backbone. Initialize from the selected V4 adapter; keep all existing weights, datasets and evaluations unchanged. Use only separately sourced native training partitions for new data. Native evaluation rows are exclusion-only. Source revisions and every generated-file hash are recorded in data/train_v5.provenance.json. - -New families: CLINC150 plus outside-scope intent classification with all 151 options; ANLI round-3 entailment; HellaSwag commonsense completion; When2Call chosen/rejected response preference. Related video/article, premise, tool catalog and query components stay within a split. Validation, calibration and test are separate subsets of native training partitions. Original test labels do not select checkpoints or hyperparameters. Option order is deterministic and randomized; anonymous keys do not expose preferred/rejected labels. - -Retain 8,000 family-balanced V4 training rows and add up to 2,000 rows per new family. Selection and calibration each retain 800 balanced examples from their existing V4 splits and add 150 per new family. New test has 150 per family. Keep full states within 2,048 tokens and new prompts within 3,000 tokens; do not assign full-context labels to newly cropped source text. - -First run a training-only 48-example smoke test. Then one epoch, all-layer LoRA rank 16, scale 10, dropout 0.05, LR 1e-5, fresh optimizer, gradient checkpointing, 3,072 batch tokens and 2,048 state tokens. Select lowest unweighted selection NLL; the initial checkpoint remains eligible. Evaluate every 250 steps and at completion. Fit temperatures on independent calibration data. Compare V4 and V5 on fixed new holdouts, original test, prior transfer/tool holdouts and the fixed partial benchmark diagnostic. Do not select based on these test results. - -The live Space's 2026-09-24 snapshot describes Decision Index 0.2: 40 benchmarks in five equally weighted areas. The public reproduction kit at 8e12d6d7fead9af98155c3d23323a51144baca9d still specifies the older 19-benchmark scored panel. Its README now explains that the corpus is not redistributed and must be rebuilt due to source licensing. No partial or older-suite aggregate will be represented as the current leaderboard score. - -This is one bounded improvement experiment, not a guarantee of first place. No leaderboard submission, hosted model API, or paid compute is part of this run. - -## Predeclared inference ablation - -After calibration, compare single-order readout with the average of original and -reversed option-order probabilities, matched by exact original key. Use a fixed, -family-balanced sample of up to 600 choice rows from selection_v5 only. Select -the two-order engine only if its NLL is lower and macro family accuracy does not -decrease. Preserve every option, full prompt and native instruction in both -passes. No benchmark identity or gold label affects inference. This doubles -forward-pass work and must be reported separately. Test labels do not choose the -inference policy. The one-order run remains a separately recorded comparison. - -## R2 correction before restart - -The first attempt was stopped after an independent native-source audit found -three retained short utterances overlapping native evaluation content (two -MASSIVE and one TweetEval irony example). It is not eligible for release. -R2 uses a new adapter directory and 15,997 rows, removing those examples. -Existing selection, calibration and new test sets pass this check unchanged. -The starting V4 checkpoint has inherited content overlap from legitimate native -training sources; this cannot be undone by filtering continuation data. Exact -counts and state hashes are in retention-audit.json. Do not describe R2 as fully -decontaminated, and do not claim an official leaderboard score. - -Before any expanded benchmark scoring, sample size was set to 1,400 (200 per available benchmark) to improve precision. The earlier 600-row sample is preserved and is not used for the comparison. - -The full available 570-case CRUXEval diagnostic is also fixed for post-training regression checking, motivated by the already-known V4 code regression. V4 reference results are reused from the prior identical rows/adapter/engine configuration; V5 single-order and selected two-order (if applicable) are scored separately. - -Capacity audit: every one of the 1,400 expanded requests fits the native prompt within 4,096 tokens (maximum 1,669). The declared evaluation limit stays at 8,192, shared with previous diagnostics. No prompt truncation or option pruning is used. - -V4’s earlier relevance and sarcasm gains are also regression checks: reuse the existing 500-request ESCI and all-1,400-case English-A sarcasm files without resampling. Score V5 single-order and the selected two-order policy, if applicable. Reference V4/released results use the same frozen row hashes. This is fixed before inspecting any V5 test metrics. diff --git a/docs/training/jet-v5/r2-code/scripts/evaluate_v5_order.py b/docs/training/jet-v5/r2-code/scripts/evaluate_v5_order.py deleted file mode 100644 index aa349714980cc0928f85ea4e52fa29c492ed6bd2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/r2-code/scripts/evaluate_v5_order.py +++ /dev/null @@ -1,43 +0,0 @@ -"""Choose single versus two-order readout using selection data only.""" -import argparse,json,random,time,hashlib -from collections import defaultdict -from pathlib import Path -import numpy as np -from decision_index_ensemble import TwoOrderJetEngine - -def main(): - ap=argparse.ArgumentParser();ap.add_argument('--adapter',required=True);args=ap.parse_args() - rows=[json.loads(x) for x in Path('data/selection_v5.jsonl').read_text().splitlines()] - groups=defaultdict(list) - for r in rows: - if r['question']['type']=='choice':groups[r.get('family',r['source'])].append(r) - for k in sorted(groups):random.Random(240925).shuffle(groups[k]) - selected=[] - while groups and len(selected)<600: - for k in list(groups): - selected.append(groups[k].pop()) - if not groups[k]:del groups[k] - if len(selected)==600:break - engine=TwoOrderJetEngine(model='mlx-community/Qwen3-0.6B-bf16',revision='42096995f6402fde107068cf530136fe64b604f8',adapter=args.adapter,max_tokens=8192,max_cached_tokens=8192) - results=[];started=time.monotonic() - for i,r in enumerate(selected): - response,raw=engine(r['state'],{'q':r['question']}) - keys=list(r['question']['criteria']);target=np.array(r['target']) - scores={} - for name,answer in [('single',raw['single_order_answers']['q']),('two_order',response['answers']['q'])]: - p=np.array([answer['probabilities'][k] for k in keys]) - scores[name]={'nll':float(-(target*np.log(np.clip(p,1e-12,1))).sum()),'correct':int(p.argmax()==target.argmax())} - results.append({'family':r.get('family',r['source']),'scores':scores}) - if (i+1)%25==0:print(i+1,len(selected),flush=True) - summary={} - for name in ['single','two_order']: - families=defaultdict(list) - for r in results:families[r['family']].append(r['scores'][name]) - summary[name]={'nll':float(np.mean([r['scores'][name]['nll'] for r in results])), - 'accuracy':float(np.mean([r['scores'][name]['correct'] for r in results])), - 'macro_family_accuracy':float(np.mean([np.mean([r['correct'] for r in rs]) for rs in families.values()]))} - choose=summary['two_order']['nll']=summary['single']['macro_family_accuracy'] - report={'n':len(results),'selection_sha256':hashlib.sha256(Path('data/selection_v5.jsonl').read_bytes()).hexdigest(),'selection_rule':'lower NLL and no decrease in macro family accuracy on selection data only','selected':'two_order' if choose else 'single','summary':summary,'seconds':time.monotonic()-started,'engine_provenance':engine.provenance,'rows':results} - Path('docs/training/jet-v5/inference-selection.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps({k:v for k,v in report.items() if k not in ['rows','engine_provenance']},indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/r2-code/scripts/filter_v5_retention.py b/docs/training/jet-v5/r2-code/scripts/filter_v5_retention.py deleted file mode 100644 index e20692fcaad13ae2f8cbf555b38540242e849656..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/r2-code/scripts/filter_v5_retention.py +++ /dev/null @@ -1,40 +0,0 @@ -"""Audit native-evaluation overlap in retained examples and create immutable R2 data. - -Catalog names and isolated short schema fields are not treated as utterances. -V5 source content identities, complete text states and text fragments >=32 -normalized characters are compared; inherited exposure is reported explicitly. -""" -import json -from pathlib import Path -from collections import Counter -from build_v5 import records,content_keys,sha -from data.decision_training import ROOT,read,write,atoms,digest,normalized - -def keys(row): - if row['source'].startswith('v5:'):return set(row['provenance']['content_keys']) - result={'content:'+a for a in atoms(row['state'])} - if isinstance(row['state'],str):result.add('content:'+digest(normalized(row['state']))) - return result - -def main(): - blocked=set() - for s in json.loads((ROOT/'work/jet-v5/sources-complete.json').read_text()): - if s['role']!='train': - for r in records(s):blocked.update(content_keys(s['repo'],r)) - report={'rule':__doc__,'files':{},'inherited_exposure':{}} - for name in ['train_v2','train_v3','train_v4_tools_focus']: - hits=[r for r in read(ROOT/f'data/{name}.jsonl') if keys(r)&blocked] - report['inherited_exposure'][name]={'count':len(hits),'by_source':dict(Counter(r['source'] for r in hits)),'state_hashes':[digest(r['state']) for r in hits]} - for name in ['train_v5','selection_v5','calibration_v5','test_v5_new']: - src=ROOT/f'data/{name}.jsonl';rows=read(src) - kept=[r for r in rows if not keys(r)&blocked];hits=[r for r in rows if keys(r)&blocked] - if name=='train_v5': - dst=ROOT/'data/train_v5_r2.jsonl';assert not dst.exists();write(dst,kept) - report['output']={'path':str(dst),'sha256':sha(dst),'rows':len(kept)} - else:assert not hits,(name,len(hits)) - report['files'][name]={'sha256':sha(src),'kept':len(kept),'removed':len(hits),'removed_sources':dict(Counter(r['source'] for r in hits)),'state_hashes':[digest(r['state']) for r in hits]} - smoke=read(ROOT/'work/jet-v5/smoke.jsonl');assert not any(keys(r)&blocked for r in smoke) - report['caveat']='R2 removes new-stage overlap but cannot erase inherited V4 training exposure; do not claim a fully decontaminated checkpoint or an official score.' - (ROOT/'docs/training/jet-v5/retention-audit.json').write_text(json.dumps(report,indent=2)+'\n') - print(json.dumps(report,indent=2)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/r2-code/scripts/run_v5_cuda.sh b/docs/training/jet-v5/r2-code/scripts/run_v5_cuda.sh deleted file mode 100644 index f2b8daf57c8de968db61e663e0f5883418eef211..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/r2-code/scripts/run_v5_cuda.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail -cd "$(dirname "$0")/.." -export PYTHONUNBUFFERED=1 CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 -export MLX_CUDA_SDPA_CACHE_SIZE=2048 MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" -NEXT=adapters/jet-v5-panel02-r2-20260924 -BASE=adapters/jet-v4-transfer-20260924 -bash train_cuda.sh --init-adapter "$BASE" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train work/jet-v5/smoke.jsonl --val work/jet-v5/smoke.jsonl --out adapters/jet-v5-panel02-r2-smoke \ - --epochs 1 --lr 1e-5 --rank 16 --grad-checkpoint --max-batch-tokens 3072 --max-state-tokens 2048 \ - --eval-every 10 --val-limit 100 --seed 240925 > logs/jet-v5/smoke.log 2>&1 -bash train_cuda.sh --init-adapter "$BASE" --base-revision 42096995f6402fde107068cf530136fe64b604f8 \ - --train data/train_v5_r2.jsonl --val data/selection_v5.jsonl --out "$NEXT" \ - --epochs 1 --lr 1e-5 --rank 16 --grad-checkpoint --max-batch-tokens 3072 --max-state-tokens 2048 \ - --use-example-weights --eval-every 250 --val-limit 10000 --seed 240925 > logs/jet-v5/train.log 2>&1 -.venv/bin/jet-calibrate --adapter "$NEXT" --data data/calibration_v5.jsonl --limit 10000 --batch-size 2 \ - --max-state-tokens 4096 > logs/jet-v5/calibration.log 2>&1 -for name in v4 v5; do - if [[ "$name" == v4 ]]; then RUN="$BASE"; else RUN="$NEXT"; fi - for dataset in test_v5_new test test_v4_new test_v4_tools; do - CONTEXT=2048; if [[ "$dataset" == test ]]; then CONTEXT=4096; fi - .venv/bin/jet-eval --adapter "$RUN" --data "data/$dataset.jsonl" --batch-size 2 \ - --max-state-tokens "$CONTEXT" --json "docs/training/jet-v5/$name-$dataset.json" > "logs/jet-v5/$name-$dataset.log" 2>&1 - done - bash scripts/bench_index_cuda.sh run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz --out "artifacts/decision-index/runs/diagnostic-v5-$name" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter="$RUN" --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$name-diagnostic.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/diagnostic/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/diagnostic-v5-$name/results.jsonl" --out "docs/training/jet-v5/$name-diagnostic.json" > "logs/jet-v5/$name-report.log" 2>&1 -done -sha256sum -c work/jet-v3/protected.sha256 > logs/jet-v5/protected.log -sha256sum -c work/jet-v3/published.sha256 > logs/jet-v5/published.log -printf 'V5 training and evaluation completed.\n' diff --git a/docs/training/jet-v5/r2-code/scripts/run_v5_extra.sh b/docs/training/jet-v5/r2-code/scripts/run_v5_extra.sh deleted file mode 100644 index 35af15fa5e9d8b99b671245ba117df34602e6047..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/r2-code/scripts/run_v5_extra.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# Run only after run_v5_cuda.sh finishes; uses the GPU sequentially. -set -euo pipefail -cd "$(dirname "$0")/.." -export CUDA_HOME="$PWD/.venv/cuda_home" MLX_USE_CUDA_GRAPHS=0 MLX_CUDA_SDPA_CACHE_SIZE=2048 -export MLX_PTX_CACHE_DIR="$PWD/.venv/ptx_cache" PYTHONPATH=src PYTHONUNBUFFERED=1 -.venv/bin/python scripts/evaluate_v5_order.py --adapter adapters/jet-v5-panel02-r2-20260924 > logs/jet-v5/inference-selection.log 2>&1 -ENGINE=decision_index_engine:JetEngine -if .venv/bin/python -c 'import json,sys;sys.exit(json.load(open("docs/training/jet-v5/inference-selection.json"))["selected"]!="two_order")'; then - ENGINE=decision_index_ensemble:TwoOrderJetEngine -fi -for SUITE in diagnostic v5-diagnostic; do - .venv/bin/python -m decision_index run --engine "$ENGINE" \ - --rows "artifacts/decision-index/$SUITE/compatibility.jsonl.gz" \ - --out "artifacts/decision-index/runs/v5-compatibility-$SUITE" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/compatibility-$SUITE.log" 2>&1 -done -for NAME in v4 v5; do - ADAPTER=adapters/jet-v4-transfer-20260924 - if [[ "$NAME" == v5 ]]; then ADAPTER=adapters/jet-v5-panel02-r2-20260924; fi - .venv/bin/python -m decision_index run --engine decision_index_engine:JetEngine \ - --rows artifacts/decision-index/v5-diagnostic/diagnostic.jsonl.gz \ - --out "artifacts/decision-index/runs/v5-expanded-$NAME" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter="$ADAPTER" --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$NAME-expanded.log" 2>&1 - .venv/bin/jet-bench-index report --rows artifacts/decision-index/v5-diagnostic/diagnostic.jsonl.gz \ - --results "artifacts/decision-index/runs/v5-expanded-$NAME/results.jsonl" --out "docs/training/jet-v5/$NAME-expanded.json" > "logs/jet-v5/$NAME-expanded-report.log" 2>&1 -done -if .venv/bin/python -c 'import json,sys;sys.exit(json.load(open("docs/training/jet-v5/inference-selection.json"))["selected"]!="two_order")'; then - for PAIR in 'diagnostic/diagnostic.jsonl.gz v5-two-order-original' 'v5-diagnostic/diagnostic.jsonl.gz v5-two-order-expanded'; do - read -r ROWS RUN <<< "$PAIR" - .venv/bin/python -m decision_index run --engine decision_index_ensemble:TwoOrderJetEngine \ - --rows "artifacts/decision-index/$ROWS" --out "artifacts/decision-index/runs/$RUN" \ - --option model=mlx-community/Qwen3-0.6B-bf16 --option revision=42096995f6402fde107068cf530136fe64b604f8 \ - --option adapter=adapters/jet-v5-panel02-r2-20260924 --option max_tokens=8192 --option max_cached_tokens=8192 > "logs/jet-v5/$RUN.log" 2>&1 - .venv/bin/jet-bench-index report --rows "artifacts/decision-index/$ROWS" \ - --results "artifacts/decision-index/runs/$RUN/results.jsonl" --out "docs/training/jet-v5/$RUN.json" > "logs/jet-v5/$RUN-report.log" 2>&1 - done -fi -printf 'V5 expanded evaluation complete.\n' diff --git a/docs/training/jet-v5/r2-code/scripts/summarize_v5.py b/docs/training/jet-v5/r2-code/scripts/summarize_v5.py deleted file mode 100644 index 57217d010de53cefcfeee1f980407b0fc678d0cc..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/r2-code/scripts/summarize_v5.py +++ /dev/null @@ -1,54 +0,0 @@ -"""Summarize completed V5 evaluation without inventing a leaderboard score.""" -import hashlib,json,shutil -from pathlib import Path -ROOT=Path(__file__).resolve().parents[1] -D=ROOT/'docs/training/jet-v5' -def read(p):return json.loads(p.read_text()) -def sha(p): - with p.open('rb') as f:return hashlib.file_digest(f,'sha256').hexdigest() -def main(): - adapter=ROOT/'adapters/jet-v5-panel02-r2-20260924' - config=read(adapter/'training_config.json');metrics=[json.loads(x) for x in (adapter/'metrics.jsonl').read_text().splitlines()] - assert metrics[-1]['step']==config['total_steps'],'Training incomplete' - order=read(D/'inference-selection.json') - held={name:{ds:read(D/f'{name}-{ds}.json') for ds in ['test','test_v5_new','test_v4_new','test_v4_tools']} for name in ['v4','v5']} - bench={} - for name in ['v4','v5']: - bench[name]={} - for suite in ['diagnostic','expanded']: - result=read(D/f'{name}-{suite}.json');assert result['complete_for_supplied_rows'] and result['pending_requests']==0 - for b in result['summary']['benchmarks']:bench[name][b['dataset']]=b - if order['selected']=='two_order': - bench['v5_two_order']={} - for suite in ['original','expanded']: - result=read(D/f'v5-two-order-{suite}.json');assert result['complete_for_supplied_rows'] - for b in result['summary']['benchmarks']:bench['v5_two_order'][b['dataset']]=b - best=min(metrics,key=lambda r:r['nll']) - lines=['# Jet V5 results — Qwen3-0.6B','', - 'V5 was trained locally on 15,997 examples: 7,997 retained V4 rows plus 2,000 each for CLINC150, ANLI, HellaSwag, and When2Call response preference. Native training partitions only; new selection, calibration and test groups are separate.','', - '**These are partial diagnostics, not an official Decision Index 0.2 score or a leaderboard win.** The current board scores 40 benchmarks; the public reproduction kit still specifies the older panel. No submission was made.','', - f'Selected training step: {best["step"]}/{config["total_steps"]}; selection NLL {best["nll"]:.4f}, accuracy {100*best["acc"]:.2f}%. Rank 16, LR 1e-5, one epoch, 3,072 batch tokens, 2,048 training-state tokens.','', - '| Held-out accuracy (%) | V4 | V5 | Change (points) |','|---|---:|---:|---:|'] - for ds in ['test','test_v5_new','test_v4_new','test_v4_tools']: - a=held['v4'][ds]['overall']['acc'];b=held['v5'][ds]['overall']['acc'];lines.append(f'| {ds} | {100*a:.2f} | {100*b:.2f} | {100*(b-a):+.2f} |') - lines+=['','| New held-out family | V4 | V5 |','|---|---:|---:|'] - for family,a in held['v4']['test_v5_new']['by_source'].items(): - b=held['v5']['test_v5_new']['by_source'][family];lines.append(f'| {family} | {100*a["acc"]:.2f} | {100*b["acc"]:.2f} |') - lines+=['','## Native benchmark metrics on fixed partial samples','', - '| Benchmark | Metric | Requests | V4 | V5 | V5 two-order |','|---|---|---:|---:|---:|---:|'] - def fmt(x):return '—' if x is None else f'{100*x:.2f}' - for name,a in bench['v4'].items(): - b=bench['v5'][name];c=bench.get('v5_two_order',{}).get(name,{}) - lines.append(f'| {name} | {a["metric"]} | {a["requests"]} | {fmt(a["score"])} | {fmt(b["score"])} | {fmt(c.get("score"))} |') - lines+=['','All reported values are native metrics multiplied by 100. A missing aggregate remains missing. Samples are small and results are descriptive, not statistical significance claims. MMLU and ARC are diagnostic-only and are not in the current 0.2 index panel.','', - f'Option-order policy chosen on {order["n"]} selection-only choice rows: **{order["selected"]}**. The two-order engine averages original/reversed probabilities using exact option keys; it keeps complete inputs and approximately doubles forward-pass work. Selection required lower NLL without lower macro family accuracy.','', - 'The first V5 attempt was stopped because three retained utterances overlapped native evaluation content. R2 removes them from continuation data. The V4 initialization still has inherited cross-dataset content overlap; this checkpoint is not certified decontaminated. See retention-audit.json.', '', - 'Single-order metrics remain separately reported. No test or benchmark score selected training steps, temperatures, or the option-order policy. Existing validation/test files and published weights were preserved.','', - 'The data protocol, source revisions and limitations are recorded in protocol.md and data/train_v5.provenance.json. Native annotations and group/source-ID separation were independently audited.'] - (D/'summary.md').write_text('\n'.join(lines)+'\n') - result={'selected_checkpoint':best,'training_config':config,'adapter_sha256':sha(adapter/'adapters.safetensors'),'calibration':read(adapter/'calibration.json'),'heldout':held,'benchmarks':bench,'inference_selection':order,'data_provenance_sha256':sha(ROOT/'data/train_v5.provenance.json'),'retention_audit':read(D/'retention-audit.json'),'r2_training_sha256':sha(ROOT/'data/train_v5_r2.jsonl')} - (D/'results.json').write_text(json.dumps(result,indent=2)+'\n') - out=Path('/home/jach/Documents/Codex/2026-09-23/cl/outputs');out.mkdir(parents=True,exist_ok=True) - shutil.copy2(D/'summary.md',out/'jet-v5-report.md');shutil.copy2(D/'results.json',out/'jet-v5-results.json') - print('\n'.join(lines)) -if __name__=='__main__':main() diff --git a/docs/training/jet-v5/r2-code/src/decision_index_ensemble.py b/docs/training/jet-v5/r2-code/src/decision_index_ensemble.py deleted file mode 100644 index 2d1c5954306b164274ad5dc77af04062fcb85692..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/r2-code/src/decision_index_ensemble.py +++ /dev/null @@ -1,37 +0,0 @@ -"""Two-order Jet readout; averages probabilities by original option key. - -Applies uniformly to every choice question. Preserves all inputs and refuses -requests if either complete prompt exceeds capacity. No benchmark-specific data. -""" -import hashlib -from pathlib import Path -import numpy as np -from decision_index_engine import JetEngine - - -def combine(first, second): - answers={} - if set(first['answers'])!=set(second['answers']):raise ValueError('Question mismatch') - for name,a in first['answers'].items(): - b=second['answers'][name] - keys=list(a['probabilities']) - if set(keys)!=set(b['probabilities']):raise ValueError('Option mismatch') - p=np.array([(a['probabilities'][k]+b['probabilities'][k])/2 for k in keys],dtype=float) - if not np.isfinite(p).all() or (p<0).any() or p.sum()<=0:raise ValueError('Invalid probabilities') - p/=p.sum() - answers[name]={'type':'choice','choice':keys[int(p.argmax())], - 'probabilities':dict(zip(keys,map(float,p))),'confidence':float(p.max())} - return {'model':'jet-two-order','answers':answers,'usage':{'input_tokens':first['usage']['input_tokens']+second['usage']['input_tokens']}} - - -class TwoOrderJetEngine(JetEngine): - name='jet-two-order' - def __init__(self,**options): - super().__init__(**options) - self.provenance.update(choice_readout='Mean of original and reversed option-order probabilities, aligned by exact original keys.', - ensemble_code_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest()) - def __call__(self,state,questions): - original,raw_original=super().__call__(state,questions) - reversed_questions={name:dict(q,criteria=dict(reversed(list(q['criteria'].items())))) for name,q in questions.items()} - reverse,raw_reverse=super().__call__(state,reversed_questions) - return combine(original,reverse),{'original':raw_original,'reversed':raw_reverse,'single_order_answers':original['answers']} diff --git a/docs/training/jet-v5/results.json b/docs/training/jet-v5/results.json deleted file mode 100644 index b95957daf1476622a512d277c608dd615e9f29b5..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/results.json +++ /dev/null @@ -1,9986 +0,0 @@ -{ - "selected_checkpoint": { - "step": 3500, - "nll": 0.5889549388204303, - "acc": 0.7635714285714286, - "improved": true, - "peak_memory_gb": 3.4478726 - }, - "training_config": { - "base_model": "mlx-community/Qwen3-0.6B-bf16", - "base_revision": "42096995f6402fde107068cf530136fe64b604f8", - "train": "data/train_v5_r2.jsonl", - "val": "data/selection_v5.jsonl", - "out": "adapters/jet-v5-panel02-r2-20260924", - "epochs": 1.0, - "lr": 1e-05, - "warmup": 100, - "rank": 16, - "lora_scale": 10.0, - "dropout": 0.05, - "lora_layers": -1, - "max_batch_tokens": 3072, - "max_state_tokens": 2048, - "smoothing": 0.02, - "use_example_weights": true, - "ordinal_weight": 2.0, - "grad_checkpoint": true, - "eval_every": 250, - "val_limit": 10000, - "cache_limit_gb": 1.0, - "resume": null, - "init_adapter": "adapters/jet-v4-transfer-20260924", - "start_step": null, - "seed": 240925, - "data_sha256": { - "data/train_v5_r2.jsonl": "62f33bfcad77c53d86d7097975bb52255b663d4c6f13b0e88025cb3df1d18774", - "data/selection_v5.jsonl": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3" - }, - "init_adapter_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "train_examples": 15997, - "steps_per_epoch": 3570, - "total_steps": 3570 - }, - "adapter_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration": { - "choice": 1.0, - "score": 1.2968, - "noul": 1.4983 - }, - "heldout": { - "v4": { - "test": { - "overall": { - "n": 3683, - "acc": 0.6644040184632093, - "nll": 0.8783917908240401, - "brier": 0.4363018875628721, - "ece": 0.044080123370055624 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04609997218164305, - "brier": 0.00993532707109211, - "ece": 0.03980768852763706 - }, - "boolq": { - "n": 145, - "acc": 0.7862068965517242, - "nll": 0.4650379801875558, - "brier": 0.2991545819338941, - "ece": 0.056202566623687736 - }, - "emotion": { - "n": 2500, - "acc": 0.5852, - "nll": 1.0812736734218895, - "brier": 0.5494403625547004, - "ece": 0.06851770701408388 - }, - "ag_news": { - "n": 135, - "acc": 0.8888888888888888, - "nll": 0.3087087308918988, - "brier": 0.16917814412184332, - "ece": 0.03841419992623507 - }, - "stsb": { - "n": 117, - "acc": 0.5470085470085471, - "nll": 1.0753559749796122, - "brier": 0.28199056470137557, - "ece": 0.03365990544995692, - "mae": 0.09383068554079495, - "within1": 0.8717948717948718, - "spearman": 0.877984058590352 - }, - "mnli": { - "n": 142, - "acc": 0.8309859154929577, - "nll": 0.4340047932059412, - "brier": 0.24689615776249432, - "ece": 0.07052275685357377 - }, - "yelp": { - "n": 146, - "acc": 0.6986301369863014, - "nll": 0.7119902010420209, - "brier": 0.41549555641030894, - "ece": 0.11680830300670779 - }, - "banking77": { - "n": 140, - "acc": 0.9071428571428571, - "nll": 0.3257405191460358, - "brier": 0.13279354980220526, - "ece": 0.07537455537489482 - }, - "massive": { - "n": 119, - "acc": 0.9327731092436975, - "nll": 0.2788916040219742, - "brier": 0.099814120778048, - "ece": 0.05662691129856755 - }, - "civil_comments": { - "n": 149, - "acc": 0.9328859060402684, - "nll": 0.32543574588733154, - "brier": 0.0470122361134796, - "ece": 0.10539027988510645 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6109395677106307, - "nll": 1.0674977012510223, - "brier": 0.508642577204092, - "ece": 0.054254766210197196 - }, - "noul": { - "n": 1185, - "acc": 0.7805907172995781, - "nll": 0.49928341122037284, - "brier": 0.30730912987911574, - "ece": 0.034595404093778585 - }, - "score": { - "n": 231, - "acc": 0.5930735930735931, - "nll": 0.9673109721763195, - "brier": 0.38807710158281145, - "ece": 0.06311224291334937, - "mae": 0.1062169110937673, - "within1": 0.9177489177489178, - "spearman": 0.899445205722314 - } - }, - "ms_per_question": 12.008765380939352, - "max_state_tokens": 4096 - }, - "test_v5_new": { - "overall": { - "n": 600, - "acc": 0.59, - "nll": 1.299430429075534, - "brier": 0.5974421801848655, - "ece": 0.14489226055641966 - }, - "by_source": { - "v5:adversarial_entailment": { - "n": 150, - "acc": 0.7066666666666667, - "nll": 0.7201601017514865, - "brier": 0.40315653132960594, - "ece": 0.055995979706446326 - }, - "v5:tool_response_preference": { - "n": 150, - "acc": 0.58, - "nll": 1.2166564369946717, - "brier": 0.6948852528274388, - "ece": 0.3230295610427857 - }, - "v5:commonsense_completion": { - "n": 150, - "acc": 0.3466666666666667, - "nll": 1.6604171573619049, - "brier": 0.8777165602201868, - "ece": 0.29102803190549215 - }, - "v5:wide_intent": { - "n": 150, - "acc": 0.7266666666666667, - "nll": 1.6004880201940734, - "brier": 0.41401037636223037, - "ece": 0.16799896627664568 - } - }, - "by_type": { - "choice": { - "n": 600, - "acc": 0.59, - "nll": 1.299430429075534, - "brier": 0.5974421801848655, - "ece": 0.14489226055641966 - } - }, - "ms_per_question": 36.09645434166699, - "max_state_tokens": 2048 - }, - "test_v4_new": { - "overall": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6, - "nll": 0.8641331787531574, - "brier": 0.5189303984919411, - "ece": 0.08732014621297518 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.5708333333333333, - "nll": 1.0104112304126223, - "brier": 0.553216138337243, - "ece": 0.1417993993808826 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.8083333333333333, - "nll": 0.468007056050313, - "brier": 0.2578825642308249, - "ece": 0.0798518454035123 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - } - }, - "ms_per_question": 32.63169955833215, - "max_state_tokens": 2048 - }, - "test_v4_tools": { - "overall": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "ms_per_question": 27.83224332916537, - "max_state_tokens": 2048 - } - }, - "v5": { - "test": { - "overall": { - "n": 3683, - "acc": 0.6665761607385283, - "nll": 0.8739479481013483, - "brier": 0.43611422339553557, - "ece": 0.04741072242039235 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 1.0, - "nll": 0.018827906827856268, - "brier": 0.0011956740026198359, - "ece": 0.018382265832689072 - }, - "boolq": { - "n": 145, - "acc": 0.7586206896551724, - "nll": 0.477754406849372, - "brier": 0.31074431400108665, - "ece": 0.062077112033449344 - }, - "emotion": { - "n": 2500, - "acc": 0.5884, - "nll": 1.0728664409339428, - "brier": 0.5478057465702815, - "ece": 0.07044296820163728 - }, - "ag_news": { - "n": 135, - "acc": 0.8962962962962963, - "nll": 0.31467971743808854, - "brier": 0.16612003492267927, - "ece": 0.04851973674915451 - }, - "stsb": { - "n": 117, - "acc": 0.5470085470085471, - "nll": 1.0867419113577246, - "brier": 0.2919386456474603, - "ece": 0.03596794783559621, - "mae": 0.09691258877929713, - "within1": 0.8888888888888888, - "spearman": 0.8702251401658272 - }, - "mnli": { - "n": 142, - "acc": 0.823943661971831, - "nll": 0.4528743649478739, - "brier": 0.2576190302733788, - "ece": 0.05873414958027046 - }, - "yelp": { - "n": 146, - "acc": 0.7191780821917808, - "nll": 0.7171766153307095, - "brier": 0.41457848929907215, - "ece": 0.06495250455320697 - }, - "banking77": { - "n": 140, - "acc": 0.9071428571428571, - "nll": 0.29728970748172806, - "brier": 0.12084634680012522, - "ece": 0.04855451328413831 - }, - "massive": { - "n": 119, - "acc": 0.9327731092436975, - "nll": 0.2904803742519619, - "brier": 0.11068498801003222, - "ece": 0.02553347880099001 - }, - "civil_comments": { - "n": 149, - "acc": 0.9328859060402684, - "nll": 0.3408124509224143, - "brier": 0.051982496863425036, - "ece": 0.13118187733144568 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6162329069254522, - "nll": 1.062267945216914, - "brier": 0.5083710705656884, - "ece": 0.06300196647118199 - }, - "noul": { - "n": 1185, - "acc": 0.7729957805907173, - "nll": 0.49518120399083376, - "brier": 0.30745203998831483, - "ece": 0.03731607993443807 - }, - "score": { - "n": 231, - "acc": 0.6147186147186147, - "nll": 0.9688274213046904, - "brier": 0.38701645197917245, - "ece": 0.04226513019887914, - "mae": 0.1068101010833629, - "within1": 0.9307359307359307, - "spearman": 0.8982739913973843 - } - }, - "ms_per_question": 12.24837621694279, - "max_state_tokens": 4096 - }, - "test_v5_new": { - "overall": { - "n": 600, - "acc": 0.735, - "nll": 0.7329250454087741, - "brier": 0.3475341883402461, - "ece": 0.036768475671609256 - }, - "by_source": { - "v5:adversarial_entailment": { - "n": 150, - "acc": 0.7466666666666667, - "nll": 0.6646332611391942, - "brier": 0.37303307763688937, - "ece": 0.052704064846038796 - }, - "v5:tool_response_preference": { - "n": 150, - "acc": 0.98, - "nll": 0.12095037142435709, - "brier": 0.05221765650556733, - "ece": 0.06036541581153872 - }, - "v5:commonsense_completion": { - "n": 150, - "acc": 0.3466666666666667, - "nll": 1.3898383743564289, - "brier": 0.7549682163529782, - "ece": 0.11845218261082968 - }, - "v5:wide_intent": { - "n": 150, - "acc": 0.8666666666666667, - "nll": 0.7562781747151166, - "brier": 0.20991780286554929, - "ece": 0.045211266080538416 - } - }, - "by_type": { - "choice": { - "n": 600, - "acc": 0.735, - "nll": 0.7329250454087741, - "brier": 0.3475341883402461, - "ece": 0.036768475671609256 - } - }, - "ms_per_question": 37.07279578166587, - "max_state_tokens": 2048 - }, - "test_v4_new": { - "overall": { - "n": 720, - "acc": 0.675, - "nll": 0.7720444074689617, - "brier": 0.44042844510431245, - "ece": 0.0333876195881102 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6416666666666667, - "nll": 0.8378210284436742, - "brier": 0.5083150294142842, - "ece": 0.1171224950502316 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.5666666666666667, - "nll": 1.0331417464340726, - "brier": 0.5624325577067126, - "ece": 0.15080579866965613 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.8166666666666667, - "nll": 0.44517044752913837, - "brier": 0.2505377481919405, - "ece": 0.05922942943871021 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.675, - "nll": 0.7720444074689617, - "brier": 0.44042844510431245, - "ece": 0.0333876195881102 - } - }, - "ms_per_question": 32.87891285833413, - "max_state_tokens": 2048 - }, - "test_v4_tools": { - "overall": { - "n": 240, - "acc": 0.85, - "nll": 0.38071567004856965, - "brier": 0.20079053168001415, - "ece": 0.07791914530098437 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.85, - "nll": 0.38071567004856965, - "brier": 0.20079053168001415, - "ece": 0.07791914530098437 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.85, - "nll": 0.38071567004856965, - "brier": 0.20079053168001415, - "ece": 0.07791914530098437 - } - }, - "ms_per_question": 28.23875877500086, - "max_state_tokens": 2048 - } - } - }, - "released_new_holdout": { - "overall": { - "n": 600, - "acc": 0.56, - "nll": 1.3715643659847168, - "brier": 0.6246184740195994, - "ece": 0.16034251227974894 - }, - "by_source": { - "v5:adversarial_entailment": { - "n": 150, - "acc": 0.6333333333333333, - "nll": 0.8615019588172436, - "brier": 0.4993069695071258, - "ece": 0.08038273334503174 - }, - "v5:tool_response_preference": { - "n": 150, - "acc": 0.64, - "nll": 1.0372949434661616, - "brier": 0.599172491305248, - "ece": 0.25046542525291443 - }, - "v5:commonsense_completion": { - "n": 150, - "acc": 0.28, - "nll": 1.9893529394765694, - "brier": 0.984350499832082, - "ece": 0.4115457673867543 - }, - "v5:wide_intent": { - "n": 150, - "acc": 0.6866666666666666, - "nll": 1.5981076221788924, - "brier": 0.41564393543394196, - "ece": 0.10220249791940052 - } - }, - "by_type": { - "choice": { - "n": 600, - "acc": 0.56, - "nll": 1.3715643659847168, - "brier": 0.6246184740195994, - "ece": 0.16034251227974894 - } - }, - "ms_per_question": 25.214695985000617, - "max_state_tokens": 2048 - }, - "released_benchmarks": { - "ContractNLI": { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.285649774536176, - "reference_same_cases": null, - "median_ms": 353.8561925006434, - "detail": { - "field_accuracy": 0.47794117647058826, - "case_exact_accuracy": 0, - "macro_f1": 0.285649774536176, - "custom_metrics": {} - } - }, - "GSM8K": { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.20535714285714285, - "reference_same_cases": null, - "median_ms": 43.758601999797975, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.125, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.20535714285714285, - "case_exact_accuracy": 0.07142857142857142, - "custom_metrics": {} - } - }, - "iSarcasmEval": { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 47.17806650023704, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.5730994152046783, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.35085995085995086, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.0, - "sarcasm": 0.0, - "irony": 0.2222222222222222, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.031746031746031744 - } - }, - "VAST": { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.36556995679627785, - "reference_same_cases": null, - "median_ms": 15.092844499122293, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.36556995679627785, - "custom_metrics": {} - } - }, - "NLI4CT": { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.25058158856763046, - "reference_same_cases": null, - "median_ms": 57.2153459997935, - "detail": { - "field_accuracy": 0.2545454545454545, - "case_exact_accuracy": 0.2545454545454545, - "macro_f1": 0.25058158856763046, - "custom_metrics": {} - } - }, - "CRUXEval": { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 43.94272399986221, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - "CLadder": { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.38181818181818183, - "reference_same_cases": null, - "median_ms": 42.82419500123069, - "detail": { - "field_accuracy": 0.38181818181818183, - "case_exact_accuracy": 0.38181818181818183, - "custom_metrics": {} - } - }, - "Habermas Machine": { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 62.874530000044615, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.3003030303030303 - } - } - }, - "CLINC150+OOS": { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.14481551509913024, - "reference_same_cases": null, - "median_ms": 39.677124499576166, - "detail": { - "field_accuracy": 0.265, - "case_exact_accuracy": 0.265, - "macro_f1": 0.14481551509913024, - "custom_metrics": {} - } - }, - "ANLI": { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3147420023997653, - "reference_same_cases": null, - "median_ms": 14.061678999951255, - "detail": { - "field_accuracy": 0.32, - "case_exact_accuracy": 0.32, - "macro_f1": 0.3147420023997653, - "custom_metrics": {} - } - }, - "MMLU": { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.295, - "reference_same_cases": null, - "median_ms": 13.026751999859698, - "detail": { - "field_accuracy": 0.295, - "case_exact_accuracy": 0.295, - "custom_metrics": {} - } - }, - "ARC-Easy": { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.605, - "reference_same_cases": null, - "median_ms": 12.673159500081965, - "detail": { - "field_accuracy": 0.605, - "case_exact_accuracy": 0.605, - "custom_metrics": {} - } - }, - "ARC-Challenge": { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.37, - "reference_same_cases": null, - "median_ms": 12.696092999703978, - "detail": { - "field_accuracy": 0.37, - "case_exact_accuracy": 0.37, - "custom_metrics": {} - } - }, - "WinoGrande": { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.545, - "reference_same_cases": null, - "median_ms": 12.078891500095779, - "detail": { - "field_accuracy": 0.545, - "case_exact_accuracy": 0.545, - "custom_metrics": {} - } - }, - "HellaSwag": { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.36, - "reference_same_cases": null, - "median_ms": 15.943470500133117, - "detail": { - "field_accuracy": 0.36, - "case_exact_accuracy": 0.36, - "custom_metrics": {} - } - } - }, - "benchmarks": { - "v4": { - "ContractNLI": { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.6536335738829072, - "reference_same_cases": null, - "median_ms": 454.7182974997668, - "detail": { - "field_accuracy": 0.6691176470588235, - "case_exact_accuracy": 0, - "macro_f1": 0.6536335738829072, - "custom_metrics": {} - } - }, - "GSM8K": { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.375, - "reference_same_cases": null, - "median_ms": 58.657546999711485, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.4642857142857143, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.375, - "case_exact_accuracy": 0.17857142857142858, - "custom_metrics": {} - } - }, - "iSarcasmEval": { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 65.56890750016464, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.52046783625731, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.12538452073335796, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.125, - "sarcasm": 0.2222222222222222, - "irony": 0.11764705882352941, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.06640989729225023 - } - }, - "VAST": { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.4431124431124431, - "reference_same_cases": null, - "median_ms": 30.8696254996903, - "detail": { - "field_accuracy": 0.4642857142857143, - "case_exact_accuracy": 0.4642857142857143, - "macro_f1": 0.4431124431124431, - "custom_metrics": {} - } - }, - "NLI4CT": { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5203488372093024, - "reference_same_cases": null, - "median_ms": 80.07431799978804, - "detail": { - "field_accuracy": 0.6727272727272727, - "case_exact_accuracy": 0.6727272727272727, - "macro_f1": 0.5203488372093024, - "custom_metrics": {} - } - }, - "CRUXEval": { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 59.004248000746884, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": {} - } - }, - "CLadder": { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4727272727272727, - "reference_same_cases": null, - "median_ms": 55.74357699970278, - "detail": { - "field_accuracy": 0.4727272727272727, - "case_exact_accuracy": 0.4727272727272727, - "custom_metrics": {} - } - }, - "Habermas Machine": { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 87.14271100052429, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": { - "normalized_rank_regret": 0.2815151515151515 - } - } - }, - "CLINC150+OOS": { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.28180022127390547, - "reference_same_cases": null, - "median_ms": 67.66145899973708, - "detail": { - "field_accuracy": 0.42, - "case_exact_accuracy": 0.42, - "macro_f1": 0.28180022127390547, - "custom_metrics": {} - } - }, - "ANLI": { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3629270257674592, - "reference_same_cases": null, - "median_ms": 28.05418399930204, - "detail": { - "field_accuracy": 0.37, - "case_exact_accuracy": 0.37, - "macro_f1": 0.3629270257674592, - "custom_metrics": {} - } - }, - "MMLU": { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.3, - "reference_same_cases": null, - "median_ms": 27.937596500123618, - "detail": { - "field_accuracy": 0.3, - "case_exact_accuracy": 0.3, - "custom_metrics": {} - } - }, - "ARC-Easy": { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.66, - "reference_same_cases": null, - "median_ms": 27.603171999999176, - "detail": { - "field_accuracy": 0.66, - "case_exact_accuracy": 0.66, - "custom_metrics": {} - } - }, - "ARC-Challenge": { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.39, - "reference_same_cases": null, - "median_ms": 27.517445500052418, - "detail": { - "field_accuracy": 0.39, - "case_exact_accuracy": 0.39, - "custom_metrics": {} - } - }, - "WinoGrande": { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.49, - "reference_same_cases": null, - "median_ms": 27.378785499877267, - "detail": { - "field_accuracy": 0.49, - "case_exact_accuracy": 0.49, - "custom_metrics": {} - } - }, - "HellaSwag": { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.345, - "reference_same_cases": null, - "median_ms": 30.351134999818896, - "detail": { - "field_accuracy": 0.345, - "case_exact_accuracy": 0.345, - "custom_metrics": {} - } - } - }, - "v5": { - "ContractNLI": { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.6993761418550424, - "reference_same_cases": null, - "median_ms": 457.1604524999202, - "detail": { - "field_accuracy": 0.7247899159663865, - "case_exact_accuracy": 0, - "macro_f1": 0.6993761418550424, - "custom_metrics": {} - } - }, - "GSM8K": { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.36607142857142855, - "reference_same_cases": null, - "median_ms": 57.515988999966794, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.32142857142857145, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.4107142857142857, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.36607142857142855, - "case_exact_accuracy": 0.19642857142857142, - "custom_metrics": {} - } - }, - "iSarcasmEval": { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 67.71158750007089, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.6608187134502924, - "case_exact_accuracy": 0.5357142857142857, - "macro_f1": 0.14420463629096722, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.18181818181818182, - "sarcasm": 0.0, - "irony": 0.14285714285714285, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.04638218923933209 - } - }, - "VAST": { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.4020241171403962, - "reference_same_cases": null, - "median_ms": 32.054457499725686, - "detail": { - "field_accuracy": 0.42857142857142855, - "case_exact_accuracy": 0.42857142857142855, - "macro_f1": 0.4020241171403962, - "custom_metrics": {} - } - }, - "NLI4CT": { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5327336331834083, - "reference_same_cases": null, - "median_ms": 77.43316600044636, - "detail": { - "field_accuracy": 0.6909090909090909, - "case_exact_accuracy": 0.6909090909090909, - "macro_f1": 0.5327336331834083, - "custom_metrics": {} - } - }, - "CRUXEval": { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 59.778007000204525, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": {} - } - }, - "CLadder": { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.43636363636363634, - "reference_same_cases": null, - "median_ms": 56.51821299943549, - "detail": { - "field_accuracy": 0.43636363636363634, - "case_exact_accuracy": 0.43636363636363634, - "custom_metrics": {} - } - }, - "Habermas Machine": { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 87.25654099998792, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.2715151515151515 - } - } - }, - "CLINC150+OOS": { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.34160092979549445, - "reference_same_cases": null, - "median_ms": 67.74492449994796, - "detail": { - "field_accuracy": 0.435, - "case_exact_accuracy": 0.435, - "macro_f1": 0.34160092979549445, - "custom_metrics": {} - } - }, - "ANLI": { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.36173331054221564, - "reference_same_cases": null, - "median_ms": 28.096252500290575, - "detail": { - "field_accuracy": 0.365, - "case_exact_accuracy": 0.365, - "macro_f1": 0.36173331054221564, - "custom_metrics": {} - } - }, - "MMLU": { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.305, - "reference_same_cases": null, - "median_ms": 27.939554999647953, - "detail": { - "field_accuracy": 0.305, - "case_exact_accuracy": 0.305, - "custom_metrics": {} - } - }, - "ARC-Easy": { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.595, - "reference_same_cases": null, - "median_ms": 27.784917000190035, - "detail": { - "field_accuracy": 0.595, - "case_exact_accuracy": 0.595, - "custom_metrics": {} - } - }, - "ARC-Challenge": { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 27.62665399995967, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - "WinoGrande": { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.51, - "reference_same_cases": null, - "median_ms": 27.359134499874926, - "detail": { - "field_accuracy": 0.51, - "case_exact_accuracy": 0.51, - "custom_metrics": {} - } - }, - "HellaSwag": { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.35, - "reference_same_cases": null, - "median_ms": 30.461512500096433, - "detail": { - "field_accuracy": 0.35, - "case_exact_accuracy": 0.35, - "custom_metrics": {} - } - } - }, - "v5_two_order": { - "ContractNLI": { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.7073362638505957, - "reference_same_cases": null, - "median_ms": 667.4918250000701, - "detail": { - "field_accuracy": 0.7331932773109243, - "case_exact_accuracy": 0, - "macro_f1": 0.7073362638505957, - "custom_metrics": {} - } - }, - "GSM8K": { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.5, - "reference_same_cases": null, - "median_ms": 84.65050700033316, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.5178571428571429, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.48214285714285715, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.5, - "case_exact_accuracy": 0.2857142857142857, - "custom_metrics": {} - } - }, - "iSarcasmEval": { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 95.17290849998972, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.5906432748538012, - "case_exact_accuracy": 0.48214285714285715, - "macro_f1": 0.3588834324114758, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.15384615384615385, - "sarcasm": 0.0, - "irony": 0.1111111111111111, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.03785103785103785 - } - }, - "VAST": { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.3866281866281866, - "reference_same_cases": null, - "median_ms": 60.61503049977546, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.3866281866281866, - "custom_metrics": {} - } - }, - "NLI4CT": { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.45, - "reference_same_cases": null, - "median_ms": 113.20878300011827, - "detail": { - "field_accuracy": 0.6727272727272727, - "case_exact_accuracy": 0.6727272727272727, - "macro_f1": 0.45, - "custom_metrics": {} - } - }, - "CRUXEval": { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.36363636363636365, - "reference_same_cases": null, - "median_ms": 90.6655580001825, - "detail": { - "field_accuracy": 0.36363636363636365, - "case_exact_accuracy": 0.36363636363636365, - "custom_metrics": {} - } - }, - "CLadder": { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.509090909090909, - "reference_same_cases": null, - "median_ms": 81.81080399936036, - "detail": { - "field_accuracy": 0.509090909090909, - "case_exact_accuracy": 0.509090909090909, - "custom_metrics": {} - } - }, - "Habermas Machine": { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2727272727272727, - "reference_same_cases": null, - "median_ms": 133.38184700023703, - "detail": { - "field_accuracy": 0.2727272727272727, - "case_exact_accuracy": 0.2727272727272727, - "custom_metrics": { - "normalized_rank_regret": 0.30787878787878786 - } - } - }, - "CLINC150+OOS": { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3534098658786088, - "reference_same_cases": null, - "median_ms": 138.0714999995689, - "detail": { - "field_accuracy": 0.455, - "case_exact_accuracy": 0.455, - "macro_f1": 0.3534098658786088, - "custom_metrics": {} - } - }, - "ANLI": { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3371351342582988, - "reference_same_cases": null, - "median_ms": 58.64838249999593, - "detail": { - "field_accuracy": 0.34, - "case_exact_accuracy": 0.34, - "macro_f1": 0.3371351342582988, - "custom_metrics": {} - } - }, - "MMLU": { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.335, - "reference_same_cases": null, - "median_ms": 58.66689300000871, - "detail": { - "field_accuracy": 0.335, - "case_exact_accuracy": 0.335, - "custom_metrics": {} - } - }, - "ARC-Easy": { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.635, - "reference_same_cases": null, - "median_ms": 57.258969499798695, - "detail": { - "field_accuracy": 0.635, - "case_exact_accuracy": 0.635, - "custom_metrics": {} - } - }, - "ARC-Challenge": { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.415, - "reference_same_cases": null, - "median_ms": 57.43540750017928, - "detail": { - "field_accuracy": 0.415, - "case_exact_accuracy": 0.415, - "custom_metrics": {} - } - }, - "WinoGrande": { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.46, - "reference_same_cases": null, - "median_ms": 56.90452049975647, - "detail": { - "field_accuracy": 0.46, - "case_exact_accuracy": 0.46, - "custom_metrics": {} - } - }, - "HellaSwag": { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.37, - "reference_same_cases": null, - "median_ms": 63.350262500080134, - "detail": { - "field_accuracy": 0.37, - "case_exact_accuracy": 0.37, - "custom_metrics": {} - } - } - } - }, - "regressions": { - "code": { - "scores": { - "baseline": 0.387719298245614, - "v4": 0.34385964912280703, - "v5": 0.3649122807017544, - "v5_two_order": 0.3684210526315789 - }, - "n": 570, - "rows_sha256": "c8b53c17bff3c866e544788378fea6c064221d82cc113dae8d422343394b9c16" - }, - "esci": { - "scores": { - "baseline": 0.19511369134010642, - "v4": 0.4134568939832098, - "v5": 0.38382787678478314, - "v5_two_order": 0.3860969319795632 - }, - "n": 500, - "rows_sha256": "2b1b766400236412840ab4bbec472464f958ba83cf8388a78fe1863a8eb245ea" - }, - "sarcasm": { - "scores": { - "baseline": 0.1821247892074199, - "v4": 0.23819301848049282, - "v5": 0.24107142857142858, - "v5_two_order": 0.24081632653061225 - }, - "n": 1400, - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d" - } - }, - "inference_selection": { - "n": 600, - "selection_sha256": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3", - "selection_rule": "lower NLL and no decrease in macro family accuracy on selection data only", - "selected": "two_order", - "summary": { - "single": { - "nll": 0.5060660150465175, - "accuracy": 0.8083333333333333, - "macro_family_accuracy": 0.8091601635079896 - }, - "two_order": { - "nll": 0.4959336955110757, - "accuracy": 0.82, - "macro_family_accuracy": 0.8200545026631982 - } - }, - "seconds": 52.090785968999626, - "engine_provenance": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "rows": [ - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.7962438442101873, - "correct": 0 - }, - "two_order": { - "nll": 0.9326626633977168, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.1658148583923762, - "correct": 1 - }, - "two_order": { - "nll": 0.6361975130942004, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.40458940651313996, - "correct": 1 - }, - "two_order": { - "nll": 0.4645615321752626, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.6923660044766176, - "correct": 0 - }, - "two_order": { - "nll": 2.5689158116508977, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.5105117409320202, - "correct": 1 - }, - "two_order": { - "nll": 0.5105117409320202, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.7116028977998379, - "correct": 1 - }, - "two_order": { - "nll": 0.636643120550807, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.013311021623458414, - "correct": 1 - }, - "two_order": { - "nll": 0.01900236261466538, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.11967966518987716, - "correct": 1 - }, - "two_order": { - "nll": 0.07864371912767414, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.6739367247990816, - "correct": 1 - }, - "two_order": { - "nll": 0.6116300304401876, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.9521714635828996, - "correct": 0 - }, - "two_order": { - "nll": 0.9129818841519277, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.2317984582214673, - "correct": 1 - }, - "two_order": { - "nll": 0.22241059203395605, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.822650481883524, - "correct": 0 - }, - "two_order": { - "nll": 1.8369134318728584, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.009705452324279305, - "correct": 1 - }, - "two_order": { - "nll": 0.006271733492539679, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.011008847399542228, - "correct": 1 - }, - "two_order": { - "nll": 0.015344276704632107, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.014609784756570106, - "correct": 1 - }, - "two_order": { - "nll": 0.017893040178622394, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.036413925691886916, - "correct": 1 - }, - "two_order": { - "nll": 0.043875428720715634, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0027186777265180476, - "correct": 1 - }, - "two_order": { - "nll": 0.0025971740514496547, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.0218206085452135, - "correct": 1 - }, - "two_order": { - "nll": 0.032018750419871615, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.22348104403359748, - "correct": 1 - }, - "two_order": { - "nll": 0.23081396178806582, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.029758515015450134, - "correct": 1 - }, - "two_order": { - "nll": 0.029057266507860405, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.2849594759462666, - "correct": 0 - }, - "two_order": { - "nll": 1.2179691475385028, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08362046298897138, - "correct": 1 - }, - "two_order": { - "nll": 0.08426090106475902, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.06030947699257955, - "correct": 1 - }, - "two_order": { - "nll": 0.4501004611309316, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.7402776473908522, - "correct": 1 - }, - "two_order": { - "nll": 0.6414708776135045, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.01475687196687665, - "correct": 1 - }, - "two_order": { - "nll": 0.027234500378350675, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.8137833116029765, - "correct": 0 - }, - "two_order": { - "nll": 0.7508116979925025, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.5578586042887936, - "correct": 0 - }, - "two_order": { - "nll": 1.8600090971567562, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 2.2099532797408172, - "correct": 0 - }, - "two_order": { - "nll": 2.1959700538560836, - "correct": 0 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.9179193566561494, - "correct": 0 - }, - "two_order": { - "nll": 1.2446638627286684, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.22541265160164786, - "correct": 1 - }, - "two_order": { - "nll": 0.18318840987747334, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.0218523659938747, - "correct": 1 - }, - "two_order": { - "nll": 0.03513051651277617, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.09559545419328726, - "correct": 1 - }, - "two_order": { - "nll": 0.09196700105398023, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.0630557923815305, - "correct": 1 - }, - "two_order": { - "nll": 0.07554573845155865, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.9936432896348318, - "correct": 0 - }, - "two_order": { - "nll": 0.8873544041416015, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.4336006978281376, - "correct": 1 - }, - "two_order": { - "nll": 0.44433582623488604, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.013943197179930197, - "correct": 1 - }, - "two_order": { - "nll": 0.011765148927772985, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0021851020429065234, - "correct": 1 - }, - "two_order": { - "nll": 0.0020874224800280505, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.005746727367274221, - "correct": 1 - }, - "two_order": { - "nll": 0.0061279959281322965, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.058587029414650875, - "correct": 1 - }, - "two_order": { - "nll": 0.07983733142237616, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.002254386500262537, - "correct": 1 - }, - "two_order": { - "nll": 0.0022197436715424707, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.016628744043527998, - "correct": 1 - }, - "two_order": { - "nll": 0.016662660665057585, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.1088488408659269, - "correct": 1 - }, - "two_order": { - "nll": 0.1698793756063374, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.01123490875817148, - "correct": 1 - }, - "two_order": { - "nll": 0.01603776050859953, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.12992810110229233, - "correct": 1 - }, - "two_order": { - "nll": 0.1208626577116002, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.05232202875810144, - "correct": 1 - }, - "two_order": { - "nll": 0.06024304519490543, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.024721582602869354, - "correct": 1 - }, - "two_order": { - "nll": 0.08170603988992431, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.6935087301168341, - "correct": 0 - }, - "two_order": { - "nll": 0.8669487156076334, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.0683565414930864, - "correct": 1 - }, - "two_order": { - "nll": 0.06701531279337267, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.33275424956421906, - "correct": 1 - }, - "two_order": { - "nll": 0.16208092001908112, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.1003602256656193, - "correct": 1 - }, - "two_order": { - "nll": 0.9626154417535695, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.10020655891674717, - "correct": 1 - }, - "two_order": { - "nll": 0.10989571256873003, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.6756734345843658, - "correct": 1 - }, - "two_order": { - "nll": 0.7221534900463844, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.17974463538565894, - "correct": 1 - }, - "two_order": { - "nll": 0.19337187431644245, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.08377023751297005, - "correct": 1 - }, - "two_order": { - "nll": 0.08253177122254815, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.12617239045059037, - "correct": 1 - }, - "two_order": { - "nll": 0.1295322337611133, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.11512677811328383, - "correct": 1 - }, - "two_order": { - "nll": 0.1210551299699099, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.251929081345373, - "correct": 1 - }, - "two_order": { - "nll": 0.23542259519924572, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7625300796023945, - "correct": 1 - }, - "two_order": { - "nll": 0.7788668802815122, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.011626354825328252, - "correct": 1 - }, - "two_order": { - "nll": 0.007383007073560757, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0137118622369538, - "correct": 1 - }, - "two_order": { - "nll": 0.01744432213446865, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.009124931375481345, - "correct": 1 - }, - "two_order": { - "nll": 0.02251812061780833, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.22149112171510002, - "correct": 1 - }, - "two_order": { - "nll": 0.21844063666934732, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.005746727367274221, - "correct": 1 - }, - "two_order": { - "nll": 0.0047888131640388566, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.012314703754835872, - "correct": 1 - }, - "two_order": { - "nll": 0.02192230443249389, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.2630089138799754, - "correct": 0 - }, - "two_order": { - "nll": 2.1825199432858406, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.016755766489059458, - "correct": 1 - }, - "two_order": { - "nll": 0.024094131027839846, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.8060551895596051, - "correct": 1 - }, - "two_order": { - "nll": 0.8928180216091015, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.06039324271581873, - "correct": 1 - }, - "two_order": { - "nll": 0.06254927585212194, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.08403866967962126, - "correct": 1 - }, - "two_order": { - "nll": 0.05514139286040489, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.9457165793021042, - "correct": 0 - }, - "two_order": { - "nll": 1.1577578838145093, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.11763917262786944, - "correct": 1 - }, - "two_order": { - "nll": 0.06628261406546225, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.03493413258416586, - "correct": 1 - }, - "two_order": { - "nll": 0.5644342377257953, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.388304510253787, - "correct": 0 - }, - "two_order": { - "nll": 1.4558314256247837, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.6931471805599453, - "correct": 1 - }, - "two_order": { - "nll": 0.7248539425912895, - "correct": 0 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.18167327031434768, - "correct": 1 - }, - "two_order": { - "nll": 0.13643695224587676, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.04712714527143037, - "correct": 1 - }, - "two_order": { - "nll": 0.04641823507141477, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.47407698418010663, - "correct": 1 - }, - "two_order": { - "nll": 0.3859437749078287, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.0777552140011723, - "correct": 1 - }, - "two_order": { - "nll": 0.07415342616710549, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.18876604925642768, - "correct": 1 - }, - "two_order": { - "nll": 0.1959825448070423, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.501929081345373, - "correct": 0 - }, - "two_order": { - "nll": 1.5019290813453727, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.9137959879979555, - "correct": 0 - }, - "two_order": { - "nll": 0.9135882323344903, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.03377564221712886, - "correct": 1 - }, - "two_order": { - "nll": 0.035829392090894814, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0067631237295626, - "correct": 1 - }, - "two_order": { - "nll": 0.010142029625348065, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.02065049412533139, - "correct": 1 - }, - "two_order": { - "nll": 0.02391138991880726, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.4165487032463721, - "correct": 1 - }, - "two_order": { - "nll": 0.41828086109974816, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.012509570276173327, - "correct": 1 - }, - "two_order": { - "nll": 0.010560130558004338, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.007372940020558588, - "correct": 1 - }, - "two_order": { - "nll": 0.00956211392704796, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.0906510943453984, - "correct": 0 - }, - "two_order": { - "nll": 2.287629944327559, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.10888993253392805, - "correct": 1 - }, - "two_order": { - "nll": 0.14592301201182994, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.709625368683793, - "correct": 0 - }, - "two_order": { - "nll": 1.6925888779799583, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.0772386485946437, - "correct": 1 - }, - "two_order": { - "nll": 0.07544558867480623, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.8663721559491016, - "correct": 0 - }, - "two_order": { - "nll": 0.3631840018497135, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.05973102776600069, - "correct": 1 - }, - "two_order": { - "nll": 0.5113568241551957, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 2.04694553740505, - "correct": 1 - }, - "two_order": { - "nll": 2.0445069113813306, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.19625208005058917, - "correct": 1 - }, - "two_order": { - "nll": 0.11999513252652898, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.3060357587611826, - "correct": 0 - }, - "two_order": { - "nll": 1.29925167457255, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.0742831253555125, - "correct": 1 - }, - "two_order": { - "nll": 0.07778283559504617, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.5364275039222968, - "correct": 1 - }, - "two_order": { - "nll": 0.5265510698037871, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 1.3593331378712068, - "correct": 0 - }, - "two_order": { - "nll": 1.1367127775474983, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03922555286405, - "correct": 1 - }, - "two_order": { - "nll": 0.058861000557081246, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.04044099885715609, - "correct": 1 - }, - "two_order": { - "nll": 0.03284519483058892, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.20567162015143123, - "correct": 1 - }, - "two_order": { - "nll": 0.21814558204553502, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.45093728159512503, - "correct": 1 - }, - "two_order": { - "nll": 0.379732032272534, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.32867347543143755, - "correct": 1 - }, - "two_order": { - "nll": 0.366027980242077, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.011115775485233476, - "correct": 1 - }, - "two_order": { - "nll": 0.008710733937578085, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.021123204783087764, - "correct": 1 - }, - "two_order": { - "nll": 0.019880932275047964, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.02806096048510901, - "correct": 1 - }, - "two_order": { - "nll": 0.03331146082094133, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 1.6725413182120614, - "correct": 0 - }, - "two_order": { - "nll": 1.5876377433065372, - "correct": 0 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.003692548129301451, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.11453903656316153, - "correct": 1 - }, - "two_order": { - "nll": 0.21713300606622596, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.13409467887955753, - "correct": 1 - }, - "two_order": { - "nll": 0.13200450891623475, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.0219794548477001, - "correct": 1 - }, - "two_order": { - "nll": 0.022195910803771996, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.4985666734271818, - "correct": 1 - }, - "two_order": { - "nll": 0.5112137693935074, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.048397618712257846, - "correct": 1 - }, - "two_order": { - "nll": 0.045548969679094886, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.09529282050823111, - "correct": 1 - }, - "two_order": { - "nll": 0.12431958114939884, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.3883516955984505, - "correct": 1 - }, - "two_order": { - "nll": 0.5602196006218952, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.13286455000088251, - "correct": 1 - }, - "two_order": { - "nll": 0.22871295764569366, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5601886939797442, - "correct": 1 - }, - "two_order": { - "nll": 0.27856541859234435, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.8677349199989377, - "correct": 1 - }, - "two_order": { - "nll": 0.7862052378293053, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.45093728159512503, - "correct": 1 - }, - "two_order": { - "nll": 0.5114864292446963, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.12303764004180046, - "correct": 1 - }, - "two_order": { - "nll": 0.11912769555724756, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.023974578520403952, - "correct": 1 - }, - "two_order": { - "nll": 0.03410257363611139, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.8981232641398401, - "correct": 0 - }, - "two_order": { - "nll": 0.9167030290906744, - "correct": 0 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.888986191530991, - "correct": 0 - }, - "two_order": { - "nll": 0.8899545510824985, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 1.972757165091386, - "correct": 0 - }, - "two_order": { - "nll": 1.7034488805162105, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.2811501362483148, - "correct": 1 - }, - "two_order": { - "nll": 0.2465515172722122, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.8992670153597763, - "correct": 0 - }, - "two_order": { - "nll": 0.9812028455331616, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.008279902708013648, - "correct": 1 - }, - "two_order": { - "nll": 0.015398296357624772, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.00563674397406815, - "correct": 1 - }, - "two_order": { - "nll": 0.004658725710417466, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.08459492561771735, - "correct": 1 - }, - "two_order": { - "nll": 0.047428185918928924, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.0628090799494529, - "correct": 1 - }, - "two_order": { - "nll": 0.053571713550707764, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0027186777265180476, - "correct": 1 - }, - "two_order": { - "nll": 0.0027617663864442843, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.007546534343733325, - "correct": 1 - }, - "two_order": { - "nll": 0.004286949767877182, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.3376034034838535, - "correct": 0 - }, - "two_order": { - "nll": 1.3857423147497103, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.12755677604859122, - "correct": 1 - }, - "two_order": { - "nll": 0.16901542391800325, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.3974337359418805, - "correct": 0 - }, - "two_order": { - "nll": 1.72690359381982, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.12535283400504701, - "correct": 1 - }, - "two_order": { - "nll": 0.10773297780617808, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.08866565152651158, - "correct": 1 - }, - "two_order": { - "nll": 0.32327964795693015, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.7638863383414923, - "correct": 1 - }, - "two_order": { - "nll": 1.1594010194458948, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.6043285153136739, - "correct": 1 - }, - "two_order": { - "nll": 0.27986941341797805, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.019937129401973473, - "correct": 1 - }, - "two_order": { - "nll": 0.028420067948958953, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.5500599489444167, - "correct": 1 - }, - "two_order": { - "nll": 0.6746974188626437, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.05164138927868149, - "correct": 1 - }, - "two_order": { - "nll": 0.05011320453337696, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.09797542368176994, - "correct": 1 - }, - "two_order": { - "nll": 0.09726351334210122, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.024726280462587773, - "correct": 1 - }, - "two_order": { - "nll": 0.02088559518790182, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.023974578520403952, - "correct": 1 - }, - "two_order": { - "nll": 0.019996536231008482, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.10540832970317754, - "correct": 1 - }, - "two_order": { - "nll": 0.12401222654719761, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.37719060559369166, - "correct": 1 - }, - "two_order": { - "nll": 0.4595189641775819, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.49813454751437525, - "correct": 1 - }, - "two_order": { - "nll": 0.44629111122069015, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.8111118862024624, - "correct": 1 - }, - "two_order": { - "nll": 0.8647029794270146, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.01692748629122821, - "correct": 1 - }, - "two_order": { - "nll": 0.017847491716824617, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.009938851331323587, - "correct": 1 - }, - "two_order": { - "nll": 0.022016942789862756, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.5707419917689904, - "correct": 1 - }, - "two_order": { - "nll": 0.7853162383320359, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.5646723247072885, - "correct": 1 - }, - "two_order": { - "nll": 0.5155932907498756, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003714141767506466, - "correct": 1 - }, - "two_order": { - "nll": 0.004096196540313023, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.008093860049042868, - "correct": 1 - }, - "two_order": { - "nll": 0.007450673343079281, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.12520184224761932, - "correct": 1 - }, - "two_order": { - "nll": 0.12119180088398127, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 2.9782624528562214, - "correct": 0 - }, - "two_order": { - "nll": 1.7017281466017096, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.11187865880715932, - "correct": 1 - }, - "two_order": { - "nll": 0.10217370482896676, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.01370525549903143, - "correct": 1 - }, - "two_order": { - "nll": 0.016904996168043255, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.026775467760443043, - "correct": 1 - }, - "two_order": { - "nll": 0.020074764363583807, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.6228759628047533, - "correct": 1 - }, - "two_order": { - "nll": 0.786784228273008, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.01636677528131479, - "correct": 1 - }, - "two_order": { - "nll": 0.01012739216467887, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 1.1004780245816668, - "correct": 0 - }, - "two_order": { - "nll": 0.46561460609495586, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.33943080482217397, - "correct": 1 - }, - "two_order": { - "nll": 0.3658632111873084, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.04712714527143037, - "correct": 1 - }, - "two_order": { - "nll": 0.047856981897304154, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.936453164616753, - "correct": 0 - }, - "two_order": { - "nll": 2.0144218367930353, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.06583535553267797, - "correct": 1 - }, - "two_order": { - "nll": 0.07732068700080218, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.010064043647627778, - "correct": 1 - }, - "two_order": { - "nll": 0.016633035435718455, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.27173031628021527, - "correct": 1 - }, - "two_order": { - "nll": 0.3202624993616726, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.02168530977594415, - "correct": 1 - }, - "two_order": { - "nll": 0.022490446377062392, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.501929081345373, - "correct": 0 - }, - "two_order": { - "nll": 1.3530237938956593, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.2034487591875314, - "correct": 0 - }, - "two_order": { - "nll": 1.359078480600808, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.016610422426847352, - "correct": 1 - }, - "two_order": { - "nll": 0.014013520065832527, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.026087708181620768, - "correct": 1 - }, - "two_order": { - "nll": 0.021309204962724277, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.00557041067347996, - "correct": 1 - }, - "two_order": { - "nll": 0.006471269208068428, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.3080323414874158, - "correct": 1 - }, - "two_order": { - "nll": 0.36178077629102445, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0024756851377303315, - "correct": 1 - }, - "two_order": { - "nll": 0.0025554108486462593, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 3.7580580652164497, - "correct": 0 - }, - "two_order": { - "nll": 4.2697318269228095, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.0274842422276982, - "correct": 0 - }, - "two_order": { - "nll": 1.1609150511685382, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.04797765294460553, - "correct": 1 - }, - "two_order": { - "nll": 0.04045576938398963, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.41527282596317333, - "correct": 1 - }, - "two_order": { - "nll": 0.45438910814761, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08670903145598241, - "correct": 1 - }, - "two_order": { - "nll": 0.07135205011317779, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.14939542418259533, - "correct": 1 - }, - "two_order": { - "nll": 0.42961091571025534, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.5931495561496511, - "correct": 1 - }, - "two_order": { - "nll": 0.3169669748688701, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.0927491723501417, - "correct": 0 - }, - "two_order": { - "nll": 1.0910844082666764, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5139512023903318, - "correct": 1 - }, - "two_order": { - "nll": 0.2547495528977414, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.036206787777225, - "correct": 0 - }, - "two_order": { - "nll": 2.1875535758540257, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.04170333686120593, - "correct": 1 - }, - "two_order": { - "nll": 0.03665856241778806, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.726548781548105, - "correct": 0 - }, - "two_order": { - "nll": 1.6442054639091102, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 2.6078030956397407, - "correct": 0 - }, - "two_order": { - "nll": 2.5785070221299518, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.02324546437242505, - "correct": 1 - }, - "two_order": { - "nll": 0.022215758026888015, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.029112748179552075, - "correct": 1 - }, - "two_order": { - "nll": 0.030903845799053472, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.38985199321957553, - "correct": 1 - }, - "two_order": { - "nll": 0.3796665521934768, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 2.0176750576056017, - "correct": 0 - }, - "two_order": { - "nll": 1.9055531002083894, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.6785941819171543, - "correct": 1 - }, - "two_order": { - "nll": 0.650008739499575, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.015309497179453528, - "correct": 1 - }, - "two_order": { - "nll": 0.009734264475758488, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.009681609538319353, - "correct": 1 - }, - "two_order": { - "nll": 0.009891379834551956, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.001699610342127682, - "correct": 1 - }, - "two_order": { - "nll": 0.007507398161322447, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.11512677811328383, - "correct": 1 - }, - "two_order": { - "nll": 0.10881922949921677, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.0031289163034652046, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.06836581743770373, - "correct": 1 - }, - "two_order": { - "nll": 0.3999345894424399, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.20868879817620054, - "correct": 1 - }, - "two_order": { - "nll": 0.2232492971405307, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.0638302749839662, - "correct": 1 - }, - "two_order": { - "nll": 0.06119717297997398, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.542719521688798, - "correct": 1 - }, - "two_order": { - "nll": 0.5355637930027418, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08020801114320515, - "correct": 1 - }, - "two_order": { - "nll": 0.08217725882779693, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.5537374841895398, - "correct": 1 - }, - "two_order": { - "nll": 0.2580685431096063, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.9515804765670444, - "correct": 1 - }, - "two_order": { - "nll": 1.297108821669702, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.9652599841744116, - "correct": 0 - }, - "two_order": { - "nll": 2.391673794701951, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.870067064342571, - "correct": 0 - }, - "two_order": { - "nll": 0.3677534988445831, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.8947988686080113, - "correct": 0 - }, - "two_order": { - "nll": 1.9074962155474846, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.011396447042008956, - "correct": 1 - }, - "two_order": { - "nll": 0.012149997084015268, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.4328554668787688, - "correct": 1 - }, - "two_order": { - "nll": 0.3643837867002506, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.10633723997560356, - "correct": 1 - }, - "two_order": { - "nll": 0.12232660385314509, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.058320480280248874, - "correct": 1 - }, - "two_order": { - "nll": 0.23548288803067674, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.07250595154641365, - "correct": 1 - }, - "two_order": { - "nll": 0.07781333936357654, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.42252705973354837, - "correct": 1 - }, - "two_order": { - "nll": 0.3935003839779431, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.179742032138927, - "correct": 0 - }, - "two_order": { - "nll": 1.0823698007707319, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7963601172995378, - "correct": 1 - }, - "two_order": { - "nll": 0.7987490302744598, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.006232222144049614, - "correct": 1 - }, - "two_order": { - "nll": 0.008310769655113624, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.053586321682286935, - "correct": 1 - }, - "two_order": { - "nll": 0.03971450786257627, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.005101835908736778, - "correct": 1 - }, - "two_order": { - "nll": 0.005106807063453614, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.49513275847479854, - "correct": 1 - }, - "two_order": { - "nll": 0.5598332412590689, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0024756851377303315, - "correct": 1 - }, - "two_order": { - "nll": 0.0022326892813017, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.029791545759062198, - "correct": 1 - }, - "two_order": { - "nll": 0.04944201044370984, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.4392489367905523, - "correct": 1 - }, - "two_order": { - "nll": 0.5221276052687899, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.5568674091841707, - "correct": 1 - }, - "two_order": { - "nll": 0.9148507969484643, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.1805047927172676, - "correct": 0 - }, - "two_order": { - "nll": 1.177074284066495, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.057572170432476207, - "correct": 1 - }, - "two_order": { - "nll": 0.058106572519003724, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.043633098380416346, - "correct": 1 - }, - "two_order": { - "nll": 0.07077023703911292, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.6643172956387638, - "correct": 1 - }, - "two_order": { - "nll": 0.48591311815659133, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.6931225976276034, - "correct": 1 - }, - "two_order": { - "nll": 0.9305126631688433, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.04113762101615342, - "correct": 1 - }, - "two_order": { - "nll": 0.3935628671665369, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9146110942982114, - "correct": 1 - }, - "two_order": { - "nll": 0.7316394455570759, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.013311021623458414, - "correct": 1 - }, - "two_order": { - "nll": 0.011686214777875072, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.7998352596946164, - "correct": 1 - }, - "two_order": { - "nll": 0.848315812033, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.010709655014929034, - "correct": 1 - }, - "two_order": { - "nll": 0.010232669304123834, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.012904115391931098, - "correct": 1 - }, - "two_order": { - "nll": 0.020844365846250858, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.22976731546645976, - "correct": 1 - }, - "two_order": { - "nll": 0.19680469979965134, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 2.4200785100952467, - "correct": 0 - }, - "two_order": { - "nll": 2.40708675518106, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.5490548622706681, - "correct": 1 - }, - "two_order": { - "nll": 0.6185079390251137, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.29984236548969156, - "correct": 1 - }, - "two_order": { - "nll": 0.3270907219779449, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.027001854603911128, - "correct": 1 - }, - "two_order": { - "nll": 0.02723260417049986, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.012798103711043401, - "correct": 1 - }, - "two_order": { - "nll": 0.010959286073725337, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.007709734939182858, - "correct": 1 - }, - "two_order": { - "nll": 0.006105433725127378, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.10851633836895944, - "correct": 1 - }, - "two_order": { - "nll": 0.12842262874129776, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003953210456515145, - "correct": 1 - }, - "two_order": { - "nll": 0.0035165641988045945, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.009176651152390147, - "correct": 1 - }, - "two_order": { - "nll": 0.020682140455955563, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.714784649898476, - "correct": 1 - }, - "two_order": { - "nll": 0.7333751788112584, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.056086971217846344, - "correct": 1 - }, - "two_order": { - "nll": 0.07780707366710661, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.36440912404700837, - "correct": 1 - }, - "two_order": { - "nll": 0.29150329505200273, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.09183061130498635, - "correct": 1 - }, - "two_order": { - "nll": 0.0888075782791414, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.02387846560950667, - "correct": 1 - }, - "two_order": { - "nll": 0.2163013033117818, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 2.8091744343310916, - "correct": 0 - }, - "two_order": { - "nll": 2.48290439735785, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.014541879780317887, - "correct": 1 - }, - "two_order": { - "nll": 0.03867368398904081, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.7130804188105171, - "correct": 1 - }, - "two_order": { - "nll": 0.7096499719035849, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.2085370929180823, - "correct": 0 - }, - "two_order": { - "nll": 1.9008433861132727, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.03922555286405, - "correct": 1 - }, - "two_order": { - "nll": 0.04620712389295999, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.8949734972289527, - "correct": 0 - }, - "two_order": { - "nll": 0.6738219135194214, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.10633723997560356, - "correct": 1 - }, - "two_order": { - "nll": 0.1399201315446505, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.058320480280248874, - "correct": 1 - }, - "two_order": { - "nll": 0.10368895504848531, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.42781354066031235, - "correct": 1 - }, - "two_order": { - "nll": 0.47695412577169066, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.40084106610964954, - "correct": 1 - }, - "two_order": { - "nll": 0.38290885115159834, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.3484445810050557, - "correct": 1 - }, - "two_order": { - "nll": 0.29188005601421163, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.4078167367151417, - "correct": 0 - }, - "two_order": { - "nll": 1.5001890346550013, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.018981405600261885, - "correct": 1 - }, - "two_order": { - "nll": 0.011794909763898104, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.019082145449108186, - "correct": 1 - }, - "two_order": { - "nll": 0.013902202173181895, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.0226601090024568, - "correct": 1 - }, - "two_order": { - "nll": 0.01599148762309789, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.017155391561683882, - "correct": 1 - }, - "two_order": { - "nll": 0.02516896044141924, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0023258652514146837, - "correct": 1 - }, - "two_order": { - "nll": 0.0023258652514146837, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.017508618871813857, - "correct": 1 - }, - "two_order": { - "nll": 0.012357015086827537, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.5143823943147482, - "correct": 1 - }, - "two_order": { - "nll": 0.4777986215817697, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 1.648440757626167, - "correct": 0 - }, - "two_order": { - "nll": 2.084433808812288, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.11868202455660723, - "correct": 1 - }, - "two_order": { - "nll": 0.13300351459020274, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.062250293849906005, - "correct": 1 - }, - "two_order": { - "nll": 0.0524063529678237, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.02010700002864646, - "correct": 1 - }, - "two_order": { - "nll": 0.026866624870869965, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.21290139255065663, - "correct": 1 - }, - "two_order": { - "nll": 0.5372180110946646, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.023700537030122388, - "correct": 1 - }, - "two_order": { - "nll": 0.029617243277414385, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.18644511274650602, - "correct": 1 - }, - "two_order": { - "nll": 0.11085719059543672, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9383852115014748, - "correct": 0 - }, - "two_order": { - "nll": 0.6108626105209377, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.007606094404333938, - "correct": 1 - }, - "two_order": { - "nll": 0.008386507983605993, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.1758389466112202, - "correct": 0 - }, - "two_order": { - "nll": 0.9326068537484214, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.14267505760560167, - "correct": 1 - }, - "two_order": { - "nll": 0.1561095942063894, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.024726280462587773, - "correct": 1 - }, - "two_order": { - "nll": 0.02769893658199565, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.10814781512537903, - "correct": 1 - }, - "two_order": { - "nll": 0.11288419452730226, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.8338286948065795, - "correct": 1 - }, - "two_order": { - "nll": 0.8688072341350833, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.1157534602702044, - "correct": 0 - }, - "two_order": { - "nll": 1.0737589168036015, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.4390437872861366, - "correct": 1 - }, - "two_order": { - "nll": 0.5382087126137921, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.00994647983237274, - "correct": 1 - }, - "two_order": { - "nll": 0.014026236390252049, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.009065676708340567, - "correct": 1 - }, - "two_order": { - "nll": 0.015971670791555866, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.021878827418688036, - "correct": 1 - }, - "two_order": { - "nll": 0.025395187575626216, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.09458682518587855, - "correct": 1 - }, - "two_order": { - "nll": 0.07842822317405941, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.0029912743082821974, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.013843432918108429, - "correct": 1 - }, - "two_order": { - "nll": 0.020271143002872995, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.6049920328403577, - "correct": 0 - }, - "two_order": { - "nll": 2.6523228604430176, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.02475996692142672, - "correct": 1 - }, - "two_order": { - "nll": 0.023702872756370917, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.35625562853348514, - "correct": 1 - }, - "two_order": { - "nll": 0.32624871043601655, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.1033367481555656, - "correct": 1 - }, - "two_order": { - "nll": 0.12357636073971084, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.020100442533956334, - "correct": 1 - }, - "two_order": { - "nll": 0.029786487753265575, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.04953721906085087, - "correct": 1 - }, - "two_order": { - "nll": 0.05347917164929199, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.049629326094553915, - "correct": 1 - }, - "two_order": { - "nll": 0.15955611115203602, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.4008481464288898, - "correct": 1 - }, - "two_order": { - "nll": 0.5582658116314273, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 2.1185720732243736, - "correct": 0 - }, - "two_order": { - "nll": 1.8961934204034514, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.5897417378836574, - "correct": 1 - }, - "two_order": { - "nll": 0.562494761437238, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.7390244093095892, - "correct": 1 - }, - "two_order": { - "nll": 0.755456169349554, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.02797290114133035, - "correct": 1 - }, - "two_order": { - "nll": 0.033583399314846316, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.04433419546256515, - "correct": 1 - }, - "two_order": { - "nll": 0.05130288605032962, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.11989995740163877, - "correct": 1 - }, - "two_order": { - "nll": 0.10587873727715114, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.1782292911492589, - "correct": 1 - }, - "two_order": { - "nll": 0.17213201571201717, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.6776442459057149, - "correct": 1 - }, - "two_order": { - "nll": 0.6776442459057149, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.1290325089493083, - "correct": 0 - }, - "two_order": { - "nll": 1.1165040460452544, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.01634726770472382, - "correct": 1 - }, - "two_order": { - "nll": 0.010121237408220313, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.028464785948947732, - "correct": 1 - }, - "two_order": { - "nll": 0.02550220473115058, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.07444129410906175, - "correct": 1 - }, - "two_order": { - "nll": 0.06739260014678862, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.03237912947055251, - "correct": 1 - }, - "two_order": { - "nll": 0.04133327850343491, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.006715348489117944, - "correct": 1 - }, - "two_order": { - "nll": 0.008546922830063827, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.1385551388238828, - "correct": 1 - }, - "two_order": { - "nll": 0.12291514152420673, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.283757919449582, - "correct": 0 - }, - "two_order": { - "nll": 1.1880164950117587, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.01790017728100203, - "correct": 1 - }, - "two_order": { - "nll": 0.015470159297741908, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.3395705678851873, - "correct": 1 - }, - "two_order": { - "nll": 0.3264796660858348, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.03767522699125972, - "correct": 1 - }, - "two_order": { - "nll": 0.03434957275365464, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.019442989217052467, - "correct": 1 - }, - "two_order": { - "nll": 0.10593630151309048, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.45144406489109, - "correct": 1 - }, - "two_order": { - "nll": 0.9592725513290883, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.05021364778922124, - "correct": 1 - }, - "two_order": { - "nll": 0.05928460200388559, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5738070605538901, - "correct": 1 - }, - "two_order": { - "nll": 0.27288456956101903, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.1313248909777884, - "correct": 0 - }, - "two_order": { - "nll": 1.1256271662319803, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.04044585599939583, - "correct": 1 - }, - "two_order": { - "nll": 0.04238813583419846, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.07590208190347222, - "correct": 1 - }, - "two_order": { - "nll": 0.05785353709678846, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.9936432896348318, - "correct": 0 - }, - "two_order": { - "nll": 0.7388492436866466, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.12692801104297263, - "correct": 1 - }, - "two_order": { - "nll": 0.10262044089564606, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 3.0741813783875895, - "correct": 0 - }, - "two_order": { - "nll": 2.888090212661871, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.14454663180212068, - "correct": 1 - }, - "two_order": { - "nll": 0.13336892375593548, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.8615548622706681, - "correct": 0 - }, - "two_order": { - "nll": 0.8082267048781631, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7413476801563812, - "correct": 1 - }, - "two_order": { - "nll": 0.7446540914834373, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.0036358301425141247, - "correct": 1 - }, - "two_order": { - "nll": 0.011363374428788531, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.011023036761974836, - "correct": 1 - }, - "two_order": { - "nll": 0.011628033542798958, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.015797051949997195, - "correct": 1 - }, - "two_order": { - "nll": 0.014041589645485604, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.028183299022021543, - "correct": 1 - }, - "two_order": { - "nll": 0.02083914386759074, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.008350547451545454, - "correct": 1 - }, - "two_order": { - "nll": 0.010051869426984677, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.059822193576600576, - "correct": 1 - }, - "two_order": { - "nll": 0.0788174690198234, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.1073584839642534, - "correct": 0 - }, - "two_order": { - "nll": 1.099543053200092, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.030784754919162004, - "correct": 1 - }, - "two_order": { - "nll": 0.028977358478290825, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 2.2641126024993437, - "correct": 0 - }, - "two_order": { - "nll": 2.315456761517189, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 3.9666817919211153, - "correct": 0 - }, - "two_order": { - "nll": 4.133289965702172, - "correct": 0 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.05018720208273875, - "correct": 1 - }, - "two_order": { - "nll": 0.12371388089628685, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.42062495394112076, - "correct": 1 - }, - "two_order": { - "nll": 0.2696129286438112, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.10145209205855144, - "correct": 1 - }, - "two_order": { - "nll": 0.05629737033283049, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.3499982821463602, - "correct": 1 - }, - "two_order": { - "nll": 0.48228929372426943, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.934343007736607, - "correct": 0 - }, - "two_order": { - "nll": 2.019950949555297, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.367242032138927, - "correct": 1 - }, - "two_order": { - "nll": 0.3531994423731511, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.0546685302138049, - "correct": 1 - }, - "two_order": { - "nll": 0.06893969822086263, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 1.179742032138927, - "correct": 0 - }, - "two_order": { - "nll": 1.3581889141358934, - "correct": 0 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.06785584399677826, - "correct": 1 - }, - "two_order": { - "nll": 0.14066019431641386, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.05490536922604806, - "correct": 1 - }, - "two_order": { - "nll": 0.05298214854478505, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 2.515823525844012, - "correct": 0 - }, - "two_order": { - "nll": 2.697471867169931, - "correct": 0 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.38687100611489994, - "correct": 1 - }, - "two_order": { - "nll": 0.41294012229819393, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.5800053530679871, - "correct": 1 - }, - "two_order": { - "nll": 0.5874527984597637, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.03374831880976058, - "correct": 1 - }, - "two_order": { - "nll": 0.030283605580750626, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.004327601621399945, - "correct": 1 - }, - "two_order": { - "nll": 0.010063633462134331, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.07728472815839725, - "correct": 1 - }, - "two_order": { - "nll": 0.07426573512343704, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.06862537882458733, - "correct": 1 - }, - "two_order": { - "nll": 0.08099962425731909, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.05009167832493603, - "correct": 1 - }, - "two_order": { - "nll": 0.07200675111863888, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.14694623361737735, - "correct": 1 - }, - "two_order": { - "nll": 0.11823618681696904, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.9040417801122773, - "correct": 0 - }, - "two_order": { - "nll": 0.6958149593318943, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.0352824330950873, - "correct": 1 - }, - "two_order": { - "nll": 0.03902629556247605, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.0890045132806183, - "correct": 0 - }, - "two_order": { - "nll": 1.1186022964899653, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.10740267510307709, - "correct": 1 - }, - "two_order": { - "nll": 0.12493311756298874, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.015103119091385607, - "correct": 1 - }, - "two_order": { - "nll": 0.013575711407450852, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.6390224632780469, - "correct": 1 - }, - "two_order": { - "nll": 0.6505474835165181, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 5.288445714499048, - "correct": 0 - }, - "two_order": { - "nll": 5.567036727904605, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.03468241747162235, - "correct": 1 - }, - "two_order": { - "nll": 0.5608219379957712, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.0320615679813738, - "correct": 1 - }, - "two_order": { - "nll": 1.146658057725097, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.05657594758178158, - "correct": 1 - }, - "two_order": { - "nll": 0.05043633907727328, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.34769599436448745, - "correct": 1 - }, - "two_order": { - "nll": 0.4973140598286106, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.10020655891674717, - "correct": 1 - }, - "two_order": { - "nll": 0.08287329204043259, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.15121395396940143, - "correct": 1 - }, - "two_order": { - "nll": 0.1557089042830205, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.05391848474544452, - "correct": 1 - }, - "two_order": { - "nll": 0.06001748242851365, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.23646800062085752, - "correct": 1 - }, - "two_order": { - "nll": 0.24315015599470585, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.9936432896348318, - "correct": 0 - }, - "two_order": { - "nll": 1.092176585001246, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.5170161081396428, - "correct": 1 - }, - "two_order": { - "nll": 0.5318799322332685, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.004032879380699412, - "correct": 1 - }, - "two_order": { - "nll": 0.008884804305699852, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.017987879489370473, - "correct": 1 - }, - "two_order": { - "nll": 0.018588663527466118, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.003714141767506466, - "correct": 1 - }, - "two_order": { - "nll": 0.0029157247345481985, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.030453252948697655, - "correct": 1 - }, - "two_order": { - "nll": 0.03045009225978931, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0024756851377303315, - "correct": 1 - }, - "two_order": { - "nll": 0.002400772388822885, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 5.005615082349949, - "correct": 0 - }, - "two_order": { - "nll": 4.454605001188563, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.6106484484823188, - "correct": 1 - }, - "two_order": { - "nll": 0.6258867334973826, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.060986818780914755, - "correct": 1 - }, - "two_order": { - "nll": 0.0685265287748769, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.5365307213042072, - "correct": 0 - }, - "two_order": { - "nll": 1.6799668286187566, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.10283628675510588, - "correct": 1 - }, - "two_order": { - "nll": 0.10015126144837756, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.02929207566709578, - "correct": 1 - }, - "two_order": { - "nll": 0.02802685823585733, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.3902169733240162, - "correct": 0 - }, - "two_order": { - "nll": 0.7819226846034211, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.031440177405090614, - "correct": 1 - }, - "two_order": { - "nll": 0.04985514538544759, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.10102056475282951, - "correct": 1 - }, - "two_order": { - "nll": 0.07821856630070322, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.4868459222186134, - "correct": 0 - }, - "two_order": { - "nll": 1.5221669859064848, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.058320480280248874, - "correct": 1 - }, - "two_order": { - "nll": 0.058320480280248874, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.5126449371013564, - "correct": 1 - }, - "two_order": { - "nll": 0.4898667223557108, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.012904115391931098, - "correct": 1 - }, - "two_order": { - "nll": 0.018424027640160028, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03262723298234382, - "correct": 1 - }, - "two_order": { - "nll": 0.03420102053845451, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.11869100754495031, - "correct": 1 - }, - "two_order": { - "nll": 0.10112323059847793, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.08695265726523116, - "correct": 1 - }, - "two_order": { - "nll": 0.08034899284481932, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.47407698418010663, - "correct": 1 - }, - "two_order": { - "nll": 0.4511314771639264, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.1356853436247023, - "correct": 0 - }, - "two_order": { - "nll": 1.2291608834901964, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.022574196981384246, - "correct": 1 - }, - "two_order": { - "nll": 0.016013866597377732, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.01162163226227119, - "correct": 1 - }, - "two_order": { - "nll": 0.0074034346384027686, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.010117129067101768, - "correct": 1 - }, - "two_order": { - "nll": 0.006357970019474579, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.10895069343827918, - "correct": 1 - }, - "two_order": { - "nll": 0.10168993851044474, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003177726471409912, - "correct": 1 - }, - "two_order": { - "nll": 0.0031289163034652046, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.029584054063654407, - "correct": 1 - }, - "two_order": { - "nll": 0.01870161006584919, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.5411654934588789, - "correct": 1 - }, - "two_order": { - "nll": 0.5207358917267862, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.025925896172401902, - "correct": 1 - }, - "two_order": { - "nll": 0.03943971527067909, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.585830940723973, - "correct": 1 - }, - "two_order": { - "nll": 0.625147537058694, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.030281427364897005, - "correct": 1 - }, - "two_order": { - "nll": 0.03379313248366333, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.04961420716846196, - "correct": 1 - }, - "two_order": { - "nll": 0.14438408669269506, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 2.200499681659196, - "correct": 0 - }, - "two_order": { - "nll": 0.8919254544855979, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 7.4726619471811, - "correct": 0 - }, - "two_order": { - "nll": 7.094663356736885, - "correct": 0 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5065779191964285, - "correct": 1 - }, - "two_order": { - "nll": 0.9854476218447148, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9939872903897226, - "correct": 0 - }, - "two_order": { - "nll": 0.899235036804645, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.2811501362483148, - "correct": 1 - }, - "two_order": { - "nll": 0.30550029196994377, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.035102546275021525, - "correct": 1 - }, - "two_order": { - "nll": 0.03446226985406008, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.061967589003198605, - "correct": 1 - }, - "two_order": { - "nll": 0.04993292406789605, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03262723298234382, - "correct": 1 - }, - "two_order": { - "nll": 0.027925346519952394, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.028869836225112227, - "correct": 1 - }, - "two_order": { - "nll": 0.0284198901850144, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.0759564345682912, - "correct": 1 - }, - "two_order": { - "nll": 0.07993386700390517, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.32176256954851534, - "correct": 1 - }, - "two_order": { - "nll": 0.34424377543326257, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.1098459347187357, - "correct": 0 - }, - "two_order": { - "nll": 1.2034983751628754, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 1.1860652852978422, - "correct": 0 - }, - "two_order": { - "nll": 0.5243423872184779, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.002181223723250433, - "correct": 1 - }, - "two_order": { - "nll": 0.005212817768302212, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.0068253385071424385, - "correct": 1 - }, - "two_order": { - "nll": 0.007268992839528124, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.03463185509145332, - "correct": 1 - }, - "two_order": { - "nll": 0.04640259560453939, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.005073178869938949, - "correct": 1 - }, - "two_order": { - "nll": 0.005409896410160542, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 5.405959232503582, - "correct": 0 - }, - "two_order": { - "nll": 5.482932515705919, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.894373358388868, - "correct": 0 - }, - "two_order": { - "nll": 1.8093069682807297, - "correct": 0 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 3.616037976382156, - "correct": 0 - }, - "two_order": { - "nll": 3.396490775776311, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.8992670153597763, - "correct": 0 - }, - "two_order": { - "nll": 1.0058348499039764, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.09835569388760058, - "correct": 1 - }, - "two_order": { - "nll": 0.08806928634538229, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.29210316504625755, - "correct": 1 - }, - "two_order": { - "nll": 0.16597219710879796, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.7589560928584228, - "correct": 1 - }, - "two_order": { - "nll": 0.6142074021861981, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.112357344587165, - "correct": 1 - }, - "two_order": { - "nll": 0.1884919965799346, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.5864955770231935, - "correct": 1 - }, - "two_order": { - "nll": 1.093170074551128, - "correct": 0 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.9701369836623976, - "correct": 1 - }, - "two_order": { - "nll": 1.0115430763712483, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.02884816883748237, - "correct": 1 - }, - "two_order": { - "nll": 0.026408404696559105, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 1.7052530894817965, - "correct": 0 - }, - "two_order": { - "nll": 1.5951101566285117, - "correct": 0 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.11620327974081737, - "correct": 1 - }, - "two_order": { - "nll": 0.12535266618315716, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.11620327974081737, - "correct": 1 - }, - "two_order": { - "nll": 0.09859674109773377, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.2643821320961903, - "correct": 1 - }, - "two_order": { - "nll": 0.25645728973281334, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.045704352100228184, - "correct": 1 - }, - "two_order": { - "nll": 0.05204749479288898, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.258973472514827, - "correct": 1 - }, - "two_order": { - "nll": 0.258973472514827, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7410562721015285, - "correct": 1 - }, - "two_order": { - "nll": 0.7311387222934478, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.009932795311007691, - "correct": 1 - }, - "two_order": { - "nll": 0.007422520584707116, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.012043778085042395, - "correct": 1 - }, - "two_order": { - "nll": 0.014386594897107266, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.09490485785276268, - "correct": 1 - }, - "two_order": { - "nll": 0.07013209555316584, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.2222705792761846, - "correct": 1 - }, - "two_order": { - "nll": 0.2302623589255876, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003714141767506466, - "correct": 1 - }, - "two_order": { - "nll": 0.003259395985427724, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.3288907709406345, - "correct": 1 - }, - "two_order": { - "nll": 0.21327015449757378, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.4856344032644102, - "correct": 1 - }, - "two_order": { - "nll": 0.5519245161164276, - "correct": 1 - } - } - }, - { - "family": "ag_news", - "scores": { - "single": { - "nll": 0.4319297921325762, - "correct": 1 - }, - "two_order": { - "nll": 0.5269974323271427, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 2.004630479455186, - "correct": 0 - }, - "two_order": { - "nll": 1.9742892021669276, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.02868822933979948, - "correct": 1 - }, - "two_order": { - "nll": 0.025874106426159123, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.03459576669508942, - "correct": 1 - }, - "two_order": { - "nll": 0.27087352330630626, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 2.264158030845119, - "correct": 0 - }, - "two_order": { - "nll": 0.8134631009908109, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.0858972069274276, - "correct": 1 - }, - "two_order": { - "nll": 0.09600524205828288, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.7166893729188244, - "correct": 0 - }, - "two_order": { - "nll": 0.3391767448608733, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.597171063911047, - "correct": 0 - }, - "two_order": { - "nll": 1.4121096383293446, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.027123830872991448, - "correct": 1 - }, - "two_order": { - "nll": 0.023827450671549705, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.08385875969189407, - "correct": 1 - }, - "two_order": { - "nll": 0.07357445339573576, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.006715348489117944, - "correct": 1 - }, - "two_order": { - "nll": 0.00794068813678362, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.04170333686120593, - "correct": 1 - }, - "two_order": { - "nll": 0.03617670838852364, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.6575638201269349, - "correct": 1 - }, - "two_order": { - "nll": 0.5425978861646539, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.11792474698156105, - "correct": 1 - }, - "two_order": { - "nll": 0.13630329063549942, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.1368710061149, - "correct": 0 - }, - "two_order": { - "nll": 1.0626950643668274, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 1.55456582378435, - "correct": 0 - }, - "two_order": { - "nll": 1.5998928371268295, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.010049429076506324, - "correct": 1 - }, - "two_order": { - "nll": 0.008642508002254365, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.00542118549407955, - "correct": 1 - }, - "two_order": { - "nll": 0.011177420543601808, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.008221614944632341, - "correct": 1 - }, - "two_order": { - "nll": 0.0055769531572819955, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 1.6716148124143635, - "correct": 0 - }, - "two_order": { - "nll": 1.8317965543031576, - "correct": 0 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003278433164655044, - "correct": 1 - }, - "two_order": { - "nll": 0.0038095305515816816, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.013850894981151095, - "correct": 1 - }, - "two_order": { - "nll": 0.01465057306477643, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 1.7163753298761524, - "correct": 0 - }, - "two_order": { - "nll": 1.5286260723601621, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.29446647793963887, - "correct": 1 - }, - "two_order": { - "nll": 0.3258874807953914, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.08321519833328549, - "correct": 1 - }, - "two_order": { - "nll": 0.11671137366525368, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.12732753093959376, - "correct": 1 - }, - "two_order": { - "nll": 0.1197822916973767, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.8936444162992352, - "correct": 0 - }, - "two_order": { - "nll": 0.8162769288700414, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.020527740666052043, - "correct": 1 - }, - "two_order": { - "nll": 0.02553916616375947, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.1193859685260377, - "correct": 1 - }, - "two_order": { - "nll": 0.7500828693921648, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.7426880459891129, - "correct": 1 - }, - "two_order": { - "nll": 0.8846277737800825, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 1.3826498628379482, - "correct": 0 - }, - "two_order": { - "nll": 1.3236606636680786, - "correct": 0 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.39120167537809825, - "correct": 1 - }, - "two_order": { - "nll": 0.4159760453384539, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.03262723298234382, - "correct": 1 - }, - "two_order": { - "nll": 0.027925346519952394, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.06993619964497334, - "correct": 1 - }, - "two_order": { - "nll": 0.08209965116206885, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 1.7068637975823235, - "correct": 0 - }, - "two_order": { - "nll": 1.876101383879326, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.27517983694409986, - "correct": 1 - }, - "two_order": { - "nll": 0.26342030498614744, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.47407698418010663, - "correct": 1 - }, - "two_order": { - "nll": 0.43479786493046846, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.9195976229534382, - "correct": 1 - }, - "two_order": { - "nll": 1.059988127570009, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.018059560901135564, - "correct": 1 - }, - "two_order": { - "nll": 0.011261603430058189, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.025347851100122432, - "correct": 1 - }, - "two_order": { - "nll": 0.021338168408906694, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.10169232244838122, - "correct": 1 - }, - "two_order": { - "nll": 0.05738080300891988, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.15594032933991517, - "correct": 1 - }, - "two_order": { - "nll": 0.16295958666611504, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0025541715420446696, - "correct": 1 - }, - "two_order": { - "nll": 0.0028659004041343365, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 3.016817321575584, - "correct": 0 - }, - "two_order": { - "nll": 3.690313240118683, - "correct": 0 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 2.489625203354606, - "correct": 0 - }, - "two_order": { - "nll": 2.3722560685124305, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.22648999539462278, - "correct": 1 - }, - "two_order": { - "nll": 0.24668440672381586, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.019346619636800556, - "correct": 1 - }, - "two_order": { - "nll": 0.02333852171438058, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.21951917341053775, - "correct": 1 - }, - "two_order": { - "nll": 0.12799580719296932, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.23088234837527277, - "correct": 1 - }, - "two_order": { - "nll": 0.1979711571613269, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.05103326733531594, - "correct": 1 - }, - "two_order": { - "nll": 0.0528544918078841, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.3567635623461305, - "correct": 1 - }, - "two_order": { - "nll": 0.2429005971386806, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.224057001745952, - "correct": 1 - }, - "two_order": { - "nll": 0.23534559360854188, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.036892284148855904, - "correct": 1 - }, - "two_order": { - "nll": 0.03286213804108401, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.16572436944506289, - "correct": 1 - }, - "two_order": { - "nll": 0.19841111398654515, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.05164138927868149, - "correct": 1 - }, - "two_order": { - "nll": 0.05086623360133378, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 1.1157534602702044, - "correct": 0 - }, - "two_order": { - "nll": 0.9404167518122099, - "correct": 0 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 2.4746940655886234, - "correct": 0 - }, - "two_order": { - "nll": 2.5174488559636763, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.19063755653267556, - "correct": 1 - }, - "two_order": { - "nll": 0.18852724955421196, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.6473704112766769, - "correct": 1 - }, - "two_order": { - "nll": 0.6181409990129576, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.8983805958422963, - "correct": 0 - }, - "two_order": { - "nll": 0.9060585879248221, - "correct": 0 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.0028393279908568783, - "correct": 1 - }, - "two_order": { - "nll": 0.0066945368694208696, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.010614775605981513, - "correct": 1 - }, - "two_order": { - "nll": 0.010446741310680095, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 5.118827100586655, - "correct": 0 - }, - "two_order": { - "nll": 4.427865370252084, - "correct": 0 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.215359715451144, - "correct": 1 - }, - "two_order": { - "nll": 0.22606150123998225, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.003278433164655044, - "correct": 1 - }, - "two_order": { - "nll": 0.003179265924662805, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.026441631237356582, - "correct": 1 - }, - "two_order": { - "nll": 0.016890827767555978, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.7499351187581018, - "correct": 1 - }, - "two_order": { - "nll": 0.7865961664324344, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.603982098643271, - "correct": 1 - }, - "two_order": { - "nll": 0.6447910157728076, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.013252984700933767, - "correct": 1 - }, - "two_order": { - "nll": 0.014992433568678087, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.08095753177773166, - "correct": 1 - }, - "two_order": { - "nll": 0.05008302517240303, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.3472470242500833, - "correct": 1 - }, - "two_order": { - "nll": 0.5680367368145695, - "correct": 1 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.2197166770896776, - "correct": 1 - }, - "two_order": { - "nll": 0.1189552662014736, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.029241988914203158, - "correct": 1 - }, - "two_order": { - "nll": 0.34363353912411093, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 3.1599409362948747, - "correct": 0 - }, - "two_order": { - "nll": 2.413279151375555, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.054882179158078234, - "correct": 1 - }, - "two_order": { - "nll": 0.05028548664508826, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.29218992336562866, - "correct": 1 - }, - "two_order": { - "nll": 0.21557788412552953, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.05657594758178158, - "correct": 1 - }, - "two_order": { - "nll": 0.051128128326165125, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.03922555286405, - "correct": 1 - }, - "two_order": { - "nll": 0.03120358855434411, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 1.6041823121249388, - "correct": 0 - }, - "two_order": { - "nll": 1.640281021666949, - "correct": 0 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.07289376167527796, - "correct": 1 - }, - "two_order": { - "nll": 0.07801662357933299, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 1.830920142458542, - "correct": 0 - }, - "two_order": { - "nll": 1.7776290698023107, - "correct": 0 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.5731612742499131, - "correct": 1 - }, - "two_order": { - "nll": 0.6081203490928618, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.005596860029564218, - "correct": 1 - }, - "two_order": { - "nll": 0.014255789514070222, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.016264579129894582, - "correct": 1 - }, - "two_order": { - "nll": 0.014420307415330605, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.01927836957476527, - "correct": 1 - }, - "two_order": { - "nll": 0.020036507404346985, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.1873295514888485, - "correct": 1 - }, - "two_order": { - "nll": 0.15372063172369244, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0027186777265180476, - "correct": 1 - }, - "two_order": { - "nll": 0.00267690944912994, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.007146907153260153, - "correct": 1 - }, - "two_order": { - "nll": 0.007146907153260153, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.9652427732053208, - "correct": 0 - }, - "two_order": { - "nll": 1.0977541925667413, - "correct": 0 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.1463416629820826, - "correct": 0 - }, - "two_order": { - "nll": 1.1446565945938019, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.039300683739222624, - "correct": 1 - }, - "two_order": { - "nll": 0.043742145657084494, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.017729439339609295, - "correct": 1 - }, - "two_order": { - "nll": 0.017860882697377647, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 1.4081874448898175, - "correct": 0 - }, - "two_order": { - "nll": 1.7314774153666244, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.01066372059811311, - "correct": 1 - }, - "two_order": { - "nll": 0.10395117944004603, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.028031666900932544, - "correct": 1 - }, - "two_order": { - "nll": 0.022445844129091952, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.8066775839033432, - "correct": 1 - }, - "two_order": { - "nll": 0.7696954248199842, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.022538275020599047, - "correct": 1 - }, - "two_order": { - "nll": 0.021226350268675715, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.34224948759121215, - "correct": 1 - }, - "two_order": { - "nll": 0.28269564339869335, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.2383445081104411, - "correct": 1 - }, - "two_order": { - "nll": 0.19157051346380405, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.21918415463449092, - "correct": 1 - }, - "two_order": { - "nll": 0.24241082929058777, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.09032745452969719, - "correct": 1 - }, - "two_order": { - "nll": 0.06727277549355648, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.1283258040660719, - "correct": 1 - }, - "two_order": { - "nll": 0.11529638704960389, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.5490548622706681, - "correct": 1 - }, - "two_order": { - "nll": 0.6855335243665968, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.7296720605492807, - "correct": 1 - }, - "two_order": { - "nll": 0.7913732482136114, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.021344381739088293, - "correct": 1 - }, - "two_order": { - "nll": 0.02197492218694703, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.0033823260027422077, - "correct": 1 - }, - "two_order": { - "nll": 0.003548220122411682, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 0.12520172563460252, - "correct": 1 - }, - "two_order": { - "nll": 0.37585601518447176, - "correct": 1 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.0474104089409759, - "correct": 1 - }, - "two_order": { - "nll": 0.05728191489038566, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0026351429162579636, - "correct": 1 - }, - "two_order": { - "nll": 0.0026351429162579636, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.024222587817850274, - "correct": 1 - }, - "two_order": { - "nll": 0.02186982164798901, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.9490159780145521, - "correct": 0 - }, - "two_order": { - "nll": 0.8280041917072275, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 1.5930390847564373, - "correct": 0 - }, - "two_order": { - "nll": 1.8891064485744786, - "correct": 0 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.09096023884419899, - "correct": 1 - }, - "two_order": { - "nll": 0.08589516758770857, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.018609946901949177, - "correct": 1 - }, - "two_order": { - "nll": 0.016470190366733046, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.8023021344730289, - "correct": 1 - }, - "two_order": { - "nll": 1.296973050744976, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 0.268132830064987, - "correct": 1 - }, - "two_order": { - "nll": 0.2768716600625241, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.027505331415183104, - "correct": 1 - }, - "two_order": { - "nll": 0.17692203395410624, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 0.6676881777186544, - "correct": 1 - }, - "two_order": { - "nll": 0.6615013114626438, - "correct": 1 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.02324546437242505, - "correct": 1 - }, - "two_order": { - "nll": 0.023985598315494787, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.43340605072237054, - "correct": 1 - }, - "two_order": { - "nll": 0.38058980336114906, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.024726280462587773, - "correct": 1 - }, - "two_order": { - "nll": 0.020370824390199497, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 0.22541265160164786, - "correct": 1 - }, - "two_order": { - "nll": 0.13391489522799135, - "correct": 1 - } - } - }, - { - "family": "entailment", - "scores": { - "single": { - "nll": 0.0645989391626965, - "correct": 1 - }, - "two_order": { - "nll": 0.05365559076948054, - "correct": 1 - } - } - }, - { - "family": "mnli", - "scores": { - "single": { - "nll": 0.41555456462894624, - "correct": 1 - }, - "two_order": { - "nll": 0.4683187638800554, - "correct": 1 - } - } - }, - { - "family": "sarcasm_irony", - "scores": { - "single": { - "nll": 0.45093728159512503, - "correct": 1 - }, - "two_order": { - "nll": 0.4987931718783924, - "correct": 1 - } - } - }, - { - "family": "stance_transfer", - "scores": { - "single": { - "nll": 0.36091361875462236, - "correct": 1 - }, - "two_order": { - "nll": 0.32051824994664946, - "correct": 1 - } - } - }, - { - "family": "tool_selection", - "scores": { - "single": { - "nll": 0.004225248102107815, - "correct": 1 - }, - "two_order": { - "nll": 0.006414862644828894, - "correct": 1 - } - } - }, - { - "family": "dbpedia", - "scores": { - "single": { - "nll": 0.01763377729658182, - "correct": 1 - }, - "two_order": { - "nll": 0.020158339822136835, - "correct": 1 - } - } - }, - { - "family": "massive", - "scores": { - "single": { - "nll": 3.006139464882406, - "correct": 0 - }, - "two_order": { - "nll": 0.863983578966528, - "correct": 0 - } - } - }, - { - "family": "contract_entailment", - "scores": { - "single": { - "nll": 0.029621758008642193, - "correct": 1 - }, - "two_order": { - "nll": 0.02739086023765856, - "correct": 1 - } - } - }, - { - "family": "tool_relevance", - "scores": { - "single": { - "nll": 0.0026351429162579636, - "correct": 1 - }, - "two_order": { - "nll": 0.0025946564096059424, - "correct": 1 - } - } - }, - { - "family": "banking77", - "scores": { - "single": { - "nll": 0.7052189455721887, - "correct": 1 - }, - "two_order": { - "nll": 0.7232211210048433, - "correct": 1 - } - } - }, - { - "family": "product_relevance", - "scores": { - "single": { - "nll": 0.11264593600646411, - "correct": 1 - }, - "two_order": { - "nll": 0.08880934788107725, - "correct": 1 - } - } - }, - { - "family": "stance", - "scores": { - "single": { - "nll": 0.2096421613300998, - "correct": 1 - }, - "two_order": { - "nll": 0.23750500809770175, - "correct": 1 - } - } - }, - { - "family": "tool_intent_routing", - "scores": { - "single": { - "nll": 0.11229572343004564, - "correct": 1 - }, - "two_order": { - "nll": 0.11296825616451285, - "correct": 1 - } - } - }, - { - "family": "code_behavior", - "scores": { - "single": { - "nll": 0.03893702963032614, - "correct": 1 - }, - "two_order": { - "nll": 0.033706210211879466, - "correct": 1 - } - } - }, - { - "family": "code_transfer", - "scores": { - "single": { - "nll": 0.8117422716906217, - "correct": 1 - }, - "two_order": { - "nll": 0.9459118127749184, - "correct": 0 - } - } - }, - { - "family": "wide_intent", - "scores": { - "single": { - "nll": 1.1944627114257376, - "correct": 1 - }, - "two_order": { - "nll": 1.026712203499577, - "correct": 1 - } - } - }, - { - "family": "arithmetic", - "scores": { - "single": { - "nll": 0.04696127952713844, - "correct": 1 - }, - "two_order": { - "nll": 0.3904763810754554, - "correct": 1 - } - } - }, - { - "family": "commonsense_completion", - "scores": { - "single": { - "nll": 1.8926861221897002, - "correct": 0 - }, - "two_order": { - "nll": 1.8722083706381547, - "correct": 0 - } - } - }, - { - "family": "document_relevance", - "scores": { - "single": { - "nll": 0.06583535553267797, - "correct": 1 - }, - "two_order": { - "nll": 0.0648537601404892, - "correct": 1 - } - } - }, - { - "family": "adversarial_entailment", - "scores": { - "single": { - "nll": 0.33375488321755525, - "correct": 1 - }, - "two_order": { - "nll": 0.2959266875664315, - "correct": 1 - } - } - }, - { - "family": "boolean_rules", - "scores": { - "single": { - "nll": 0.06785584399677826, - "correct": 1 - }, - "two_order": { - "nll": 0.05817518915315277, - "correct": 1 - } - } - }, - { - "family": "tool_response_preference", - "scores": { - "single": { - "nll": 1.0334570697206436, - "correct": 0 - }, - "two_order": { - "nll": 0.9833501295116349, - "correct": 0 - } - } - } - ] - }, - "data_provenance_sha256": "995a334483782501c5a9bb70d3860beda8a5f813769581ecf65866a70d69f27a", - "retention_audit": { - "rule": "Audit native-evaluation overlap in retained examples and create immutable R2 data.\n\nCatalog names and isolated short schema fields are not treated as utterances.\nV5 source content identities, complete text states and text fragments >=32\nnormalized characters are compared; inherited exposure is reported explicitly.\n", - "files": { - "train_v5": { - "sha256": "d96100fd362255482869fdfc96d02cc6d3071326155500f817fca5fdbc53d3fd", - "kept": 15997, - "removed": 3, - "removed_sources": { - "v3:sarcasm_irony": 1, - "massive": 2 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9" - ] - }, - "selection_v5": { - "sha256": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3", - "kept": 1400, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - }, - "calibration_v5": { - "sha256": "2c24ff80f07c60534f9c30adcba5fc9e36c97319e27ad262ed68befdd7dd2d1a", - "kept": 1400, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - }, - "test_v5_new": { - "sha256": "6b060a9d924a6c0711a13f8e2406ffc1b163ae80b158abd64f8288928495b9e6", - "kept": 600, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - } - }, - "inherited_exposure": { - "train_v2": { - "count": 8, - "by_source": { - "massive": 8 - }, - "state_hashes": [ - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "92dde7a604a103c21c35ecfb6ca201684031767adf42c84db55b881a5de129f8", - "b49177e05868b7af8e82a644c1ce20e521af46497adeaffe861d294d9b4bb75e", - "1efe129d040aa8d64ca7dc438f4cb43876757818864f09ae7cb53a23d1c9f08f", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "0b41b183ab200019e05ae6b8bfdf638ce2d90d39269a33ffcc94b9b035fddfb5", - "870d2ccc7ede8f14ddba0b441af3cafbc60a548e339e41d5a061445f1988b88d", - "262c9bf9846c7d15b6c044f42ca9e7d6f0e5e524b31671d54607762c57072ffb" - ] - }, - "train_v3": { - "count": 13, - "by_source": { - "v3:sarcasm_irony": 1, - "massive": 8, - "v3:tool_selection": 4 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "92dde7a604a103c21c35ecfb6ca201684031767adf42c84db55b881a5de129f8", - "b49177e05868b7af8e82a644c1ce20e521af46497adeaffe861d294d9b4bb75e", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "262c9bf9846c7d15b6c044f42ca9e7d6f0e5e524b31671d54607762c57072ffb", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "0b41b183ab200019e05ae6b8bfdf638ce2d90d39269a33ffcc94b9b035fddfb5", - "1efe129d040aa8d64ca7dc438f4cb43876757818864f09ae7cb53a23d1c9f08f", - "870d2ccc7ede8f14ddba0b441af3cafbc60a548e339e41d5a061445f1988b88d", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090" - ] - }, - "train_v4_tools_focus": { - "count": 5, - "by_source": { - "v3:sarcasm_irony": 1, - "v4:tool_intent_routing": 2, - "massive": 2 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "55769c01f1927d87fbf132c23775213854fdee7156937b41dc42ec02af6ad6d1", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "7af3840b205f11ba55c61fe5d1fc2e4bf93ff5c5f99b4543d00ea59c9408fbbe", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1" - ] - } - }, - "output": { - "path": "/home/jach/dev/jet/data/train_v5_r2.jsonl", - "sha256": "62f33bfcad77c53d86d7097975bb52255b663d4c6f13b0e88025cb3df1d18774", - "rows": 15997 - }, - "caveat": "R2 removes new-stage overlap but cannot erase inherited V4 training exposure; do not claim a fully decontaminated checkpoint or an official score." - }, - "r2_training_sha256": "62f33bfcad77c53d86d7097975bb52255b663d4c6f13b0e88025cb3df1d18774" -} diff --git a/docs/training/jet-v5/retention-audit.json b/docs/training/jet-v5/retention-audit.json deleted file mode 100644 index af7207b5d7f6f07c0d66828b7c3c22352c1ac8f2..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/retention-audit.json +++ /dev/null @@ -1,102 +0,0 @@ -{ - "rule": "Audit native-evaluation overlap in retained examples and create immutable R2 data.\n\nCatalog names and isolated short schema fields are not treated as utterances.\nV5 source content identities, complete text states and text fragments >=32\nnormalized characters are compared; inherited exposure is reported explicitly.\n", - "files": { - "train_v5": { - "sha256": "d96100fd362255482869fdfc96d02cc6d3071326155500f817fca5fdbc53d3fd", - "kept": 15997, - "removed": 3, - "removed_sources": { - "v3:sarcasm_irony": 1, - "massive": 2 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9" - ] - }, - "selection_v5": { - "sha256": "6eafe77cd79b49ae3e0831796160464b9b90b62805cde3ac189fc9bd43e080f3", - "kept": 1400, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - }, - "calibration_v5": { - "sha256": "2c24ff80f07c60534f9c30adcba5fc9e36c97319e27ad262ed68befdd7dd2d1a", - "kept": 1400, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - }, - "test_v5_new": { - "sha256": "6b060a9d924a6c0711a13f8e2406ffc1b163ae80b158abd64f8288928495b9e6", - "kept": 600, - "removed": 0, - "removed_sources": {}, - "state_hashes": [] - } - }, - "inherited_exposure": { - "train_v2": { - "count": 8, - "by_source": { - "massive": 8 - }, - "state_hashes": [ - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "92dde7a604a103c21c35ecfb6ca201684031767adf42c84db55b881a5de129f8", - "b49177e05868b7af8e82a644c1ce20e521af46497adeaffe861d294d9b4bb75e", - "1efe129d040aa8d64ca7dc438f4cb43876757818864f09ae7cb53a23d1c9f08f", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "0b41b183ab200019e05ae6b8bfdf638ce2d90d39269a33ffcc94b9b035fddfb5", - "870d2ccc7ede8f14ddba0b441af3cafbc60a548e339e41d5a061445f1988b88d", - "262c9bf9846c7d15b6c044f42ca9e7d6f0e5e524b31671d54607762c57072ffb" - ] - }, - "train_v3": { - "count": 13, - "by_source": { - "v3:sarcasm_irony": 1, - "massive": 8, - "v3:tool_selection": 4 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "92dde7a604a103c21c35ecfb6ca201684031767adf42c84db55b881a5de129f8", - "b49177e05868b7af8e82a644c1ce20e521af46497adeaffe861d294d9b4bb75e", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1", - "262c9bf9846c7d15b6c044f42ca9e7d6f0e5e524b31671d54607762c57072ffb", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "0b41b183ab200019e05ae6b8bfdf638ce2d90d39269a33ffcc94b9b035fddfb5", - "1efe129d040aa8d64ca7dc438f4cb43876757818864f09ae7cb53a23d1c9f08f", - "870d2ccc7ede8f14ddba0b441af3cafbc60a548e339e41d5a061445f1988b88d", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090", - "9fdc4a5c9a902b9670ea4805a49a6d46efb86c7c6fa8896fe27ce45135a83090" - ] - }, - "train_v4_tools_focus": { - "count": 5, - "by_source": { - "v3:sarcasm_irony": 1, - "v4:tool_intent_routing": 2, - "massive": 2 - }, - "state_hashes": [ - "5aa762ae383fbb727af3c7a36d4940a5b8c40a989452d2304fc958ff3f354e7a", - "55769c01f1927d87fbf132c23775213854fdee7156937b41dc42ec02af6ad6d1", - "0ab0aa27ad28f0a6571b6e821fc1b7a4389aec339eac6516e32f33317a4b15e9", - "7af3840b205f11ba55c61fe5d1fc2e4bf93ff5c5f99b4543d00ea59c9408fbbe", - "af6be8d6a51f8115492a637a5e9a80e5022a43dff78de8248641d295a244d2a1" - ] - } - }, - "output": { - "path": "/home/jach/dev/jet/data/train_v5_r2.jsonl", - "sha256": "62f33bfcad77c53d86d7097975bb52255b663d4c6f13b0e88025cb3df1d18774", - "rows": 15997 - }, - "caveat": "R2 removes new-stage overlap but cannot erase inherited V4 training exposure; do not claim a fully decontaminated checkpoint or an official score." -} diff --git a/docs/training/jet-v5/sarcasm-selected.json b/docs/training/jet-v5/sarcasm-selected.json deleted file mode 100644 index 0e9d16cab6a374e05a0825163fb9984fde62be25..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/sarcasm-selected.json +++ /dev/null @@ -1,162 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "8f94fe8e1e5ab360547790094103043f701ce188ca5f2e9e5c9307bc10db3346", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_ensemble:TwoOrderJetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9417307829999118, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 54.65480650036625, - "p95": 80.17837899933511, - "mean": 55.64059340214629 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 54.65480650036625, - "detail": { - "field_accuracy": 0.7342857142857143, - "case_exact_accuracy": 0.7342857142857143, - "macro_f1": 0.5398886827458256, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.24081632653061225 - }, - "category_macro_f1": 0.24081632653061225 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.24081632653061225, - "skill": 0.023324033851249745, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.24081632653061225, - "random": 0.2226862339379383, - "skill": 0.023324033851249745, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/sarcasm-single.json b/docs/training/jet-v5/sarcasm-single.json deleted file mode 100644 index ff56fc4f52601142759cfa5269c10b88b8d85904..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/sarcasm-single.json +++ /dev/null @@ -1,160 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "43bf45a8b0407fe8ee64809c94c17d721892ae1e146bdc7831c33fc596e5665d", - "results_sha256": "922f74e514b57b967a07bda9d8ffcc48b820fc49a320a5916bf743f70704e0ed", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9477255129995683, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/isarcasm-english-diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 26.99026350001077, - "p95": 59.43581800056563, - "mean": 28.649077364296186 - }, - "benchmarks": [ - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 1400, - "answered": 1400, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 1400, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 26.99026350001077, - "detail": { - "field_accuracy": 0.7571428571428571, - "case_exact_accuracy": 0.7571428571428571, - "macro_f1": 0.5482568027210885, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.24107142857142858 - }, - "category_macro_f1": 0.24107142857142858 - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "40": { - "catalog_id": 40, - "raw": 0.24107142857142858, - "skill": 0.023652217979659947, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.24107142857142858, - "random": 0.2226862339379383, - "skill": 0.023652217979659947, - "planned_cases": 1400, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/summary.md b/docs/training/jet-v5/summary.md deleted file mode 100644 index 1f7fdad390f4afb201a108eaae8f722bd1abcee0..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/summary.md +++ /dev/null @@ -1,61 +0,0 @@ -# Jet V5 results — Qwen3-0.6B - -V5 was trained locally on 15,997 examples: 7,997 retained V4 rows plus 2,000 each for CLINC150, ANLI, HellaSwag, and When2Call response preference. Native training partitions only; new selection, calibration and test groups are separate. - -**These are partial diagnostics, not an official Decision Index 0.2 score or a leaderboard win.** The current board scores 40 benchmarks; the public reproduction kit still specifies the older panel. No submission was made. - -R2 removes three retained short-text overlaps from continuation data. The V4 initialization still has inherited cross-dataset content overlap and is not certified decontaminated. See retention-audit.json. - -Selected training step: 3500/3570; selection NLL 0.5890, accuracy 76.36%. Rank 16, LR 1e-5, one epoch, 3,072 batch tokens, 2,048 training-state tokens. - -| Held-out accuracy (%) | V4 | V5 | Change (points) | -|---|---:|---:|---:| -| test | 66.44 | 66.66 | +0.22 | -| test_v5_new | 59.00 | 73.50 | +14.50 | -| test_v4_new | 65.97 | 67.50 | +1.53 | -| test_v4_tools | 93.33 | 85.00 | -8.33 | - -| New held-out family | Released Jet | V4 | V5 | -|---|---:|---:|---:| -| v5:adversarial_entailment | 63.33 | 70.67 | 74.67 | -| v5:tool_response_preference | 64.00 | 58.00 | 98.00 | -| v5:commonsense_completion | 28.00 | 34.67 | 34.67 | -| v5:wide_intent | 68.67 | 72.67 | 86.67 | - -## Native benchmark metrics on fixed partial samples - -| Benchmark | Metric | Requests | Released Jet | V4 | V5 | V5 two-order | -|---|---|---:|---:|---:|---:|---:| -| ContractNLI | macro-F1 | 56 | 28.56 | 65.36 | 69.94 | 70.73 | -| GSM8K | accuracy | 112 | 20.54 | 37.50 | 36.61 | 50.00 | -| iSarcasmEval | see separate subtask tracks | 56 | — | — | — | — | -| VAST | macro-F1 | 56 | 36.56 | 44.31 | 40.20 | 38.66 | -| NLI4CT | macro-F1 | 55 | 25.06 | 52.03 | 53.27 | 45.00 | -| CRUXEval | accuracy | 55 | 40.00 | 32.73 | 29.09 | 36.36 | -| CLadder | accuracy | 55 | 38.18 | 47.27 | 43.64 | 50.91 | -| Habermas Machine | accuracy | 55 | 29.09 | 32.73 | 29.09 | 27.27 | -| CLINC150+OOS | macro-F1 | 200 | 14.48 | 28.18 | 34.16 | 35.34 | -| ANLI | macro-F1 | 200 | 31.47 | 36.29 | 36.17 | 33.71 | -| MMLU | accuracy | 200 | 29.50 | 30.00 | 30.50 | 33.50 | -| ARC-Easy | accuracy | 200 | 60.50 | 66.00 | 59.50 | 63.50 | -| ARC-Challenge | accuracy | 200 | 37.00 | 39.00 | 40.00 | 41.50 | -| WinoGrande | accuracy | 200 | 54.50 | 49.00 | 51.00 | 46.00 | -| HellaSwag | accuracy | 200 | 36.00 | 34.50 | 35.00 | 37.00 | - -## Fixed regression checks - -| Benchmark | Requests | Released Jet | V4 | V5 | V5 two-order | -|---|---:|---:|---:|---:|---:| -| code | 570 | 38.77 | 34.39 | 36.49 | 36.84 | -| esci | 500 | 19.51 | 41.35 | 38.38 | 38.61 | -| sarcasm | 1400 | 18.21 | 23.82 | 24.11 | 24.08 | - -All reported values are native metrics multiplied by 100. A missing aggregate remains missing. Samples are small and results are descriptive, not statistical significance claims. MMLU and ARC are diagnostic-only and are not in the current 0.2 index panel. - -Option-order policy chosen on 600 selection-only choice rows: **two_order**. The two-order engine averages original/reversed probabilities using exact option keys; it keeps complete inputs and approximately doubles forward-pass work. Selection required lower NLL without lower macro family accuracy. - -The first V5 attempt was stopped because three retained utterances overlapped native evaluation content. R2 removes them from continuation data. The V4 initialization still has inherited cross-dataset content overlap; this checkpoint is not certified decontaminated. See retention-audit.json. - -Single-order metrics remain separately reported. No test or benchmark score selected training steps, temperatures, or the option-order policy. Existing validation/test files and published weights were preserved. - -The data protocol, source revisions and limitations are recorded in protocol.md and data/train_v5.provenance.json. Native annotations and group/source-ID separation were independently audited. diff --git a/docs/training/jet-v5/v4-diagnostic.json b/docs/training/jet-v5/v4-diagnostic.json deleted file mode 100644 index 4f4531642047d5105b9d489d2c48f95f9274c6ef..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v4-diagnostic.json +++ /dev/null @@ -1,515 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "a0ba6255ad98dcb5736aa6df91e74db37eb3913bf2be2215b6af3eefc67fab67", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 2.0443365099999937, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 66.02175450007053, - "p95": 473.8493209997614, - "mean": 114.807444071992 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.6536335738829072, - "reference_same_cases": null, - "median_ms": 454.7182974997668, - "detail": { - "field_accuracy": 0.6691176470588235, - "case_exact_accuracy": 0, - "macro_f1": 0.6536335738829072, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.375, - "reference_same_cases": null, - "median_ms": 58.657546999711485, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.2857142857142857, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.4642857142857143, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.375, - "case_exact_accuracy": 0.17857142857142858, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 65.56890750016464, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.52046783625731, - "case_exact_accuracy": 0.35714285714285715, - "macro_f1": 0.12538452073335796, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.125, - "sarcasm": 0.2222222222222222, - "irony": 0.11764705882352941, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.06640989729225023 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.4431124431124431, - "reference_same_cases": null, - "median_ms": 30.8696254996903, - "detail": { - "field_accuracy": 0.4642857142857143, - "case_exact_accuracy": 0.4642857142857143, - "macro_f1": 0.4431124431124431, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5203488372093024, - "reference_same_cases": null, - "median_ms": 80.07431799978804, - "detail": { - "field_accuracy": 0.6727272727272727, - "case_exact_accuracy": 0.6727272727272727, - "macro_f1": 0.5203488372093024, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 59.004248000746884, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.4727272727272727, - "reference_same_cases": null, - "median_ms": 55.74357699970278, - "detail": { - "field_accuracy": 0.4727272727272727, - "case_exact_accuracy": 0.4727272727272727, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.32727272727272727, - "reference_same_cases": null, - "median_ms": 87.14271100052429, - "detail": { - "field_accuracy": 0.32727272727272727, - "case_exact_accuracy": 0.32727272727272727, - "custom_metrics": { - "normalized_rank_regret": 0.2815151515151515 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.6536335738829072, - "skill": 0.4990948536692409, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.6536335738829072, - "random": 0.30851893087083776, - "skill": 0.4990948536692409, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.375, - "skill": 0.24603174603174605, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.4642857142857143, - "random": 0.25, - "skill": 0.28571428571428575, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.2857142857142857, - "random": 0.1, - "skill": 0.2063492063492063, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.0625, - "skill": 0.00878466561850422, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.25, - "random": 0.2226862339379383, - "skill": 0.03513866247401688, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.4431124431124431, - "skill": 0.16467966294233655, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4431124431124431, - "random": 0.3333245556439576, - "skill": 0.16467966294233655, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.32727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32727272727272727, - "random": 0.34943722943722944, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.4727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4727272727272727, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.32727272727272727, - "skill": 0.02503293807641634, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.32727272727272727, - "random": 0.31, - "skill": 0.02503293807641634, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/v4-expanded.json b/docs/training/jet-v5/v4-expanded.json deleted file mode 100644 index 4c1ae1e3f39a2c298cf8ca185f6b0f3289d8cb13..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v4-expanded.json +++ /dev/null @@ -1,274 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "9908538b8b244efa20a2d8428104d5f56a131b37569d0e0b10867ced7beb104b", - "results_sha256": "e78dc3be91789fe9c6b0d30746441904b849712c7be346419dbcc6d7e5ea23b2", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v4-transfer-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.1554, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v4-transfer-20260924", - "lora_weights_sha256": "4937c96fb56d7bd510f17b6aa51d5357b8bf8cc6de0e80eb976ebf47a1ea72ad", - "calibration_sha256": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.98541203700006, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 28.162439999505295, - "p95": 69.1245580001123, - "mean": 39.7093804907042 - }, - "benchmarks": [ - { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.28180022127390547, - "reference_same_cases": null, - "median_ms": 67.66145899973708, - "detail": { - "field_accuracy": 0.42, - "case_exact_accuracy": 0.42, - "macro_f1": 0.28180022127390547, - "custom_metrics": {} - } - }, - { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3629270257674592, - "reference_same_cases": null, - "median_ms": 28.05418399930204, - "detail": { - "field_accuracy": 0.37, - "case_exact_accuracy": 0.37, - "macro_f1": 0.3629270257674592, - "custom_metrics": {} - } - }, - { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.3, - "reference_same_cases": null, - "median_ms": 27.937596500123618, - "detail": { - "field_accuracy": 0.3, - "case_exact_accuracy": 0.3, - "custom_metrics": {} - } - }, - { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.66, - "reference_same_cases": null, - "median_ms": 27.603171999999176, - "detail": { - "field_accuracy": 0.66, - "case_exact_accuracy": 0.66, - "custom_metrics": {} - } - }, - { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.39, - "reference_same_cases": null, - "median_ms": 27.517445500052418, - "detail": { - "field_accuracy": 0.39, - "case_exact_accuracy": 0.39, - "custom_metrics": {} - } - }, - { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.49, - "reference_same_cases": null, - "median_ms": 27.378785499877267, - "detail": { - "field_accuracy": 0.49, - "case_exact_accuracy": 0.49, - "custom_metrics": {} - } - }, - { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.345, - "reference_same_cases": null, - "median_ms": 30.351134999818896, - "detail": { - "field_accuracy": 0.345, - "case_exact_accuracy": 0.345, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "24": { - "catalog_id": 24, - "raw": 0.29082132246838127, - "skill": 0.054428429957841695, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.29082132246838127, - "random": 0.25, - "skill": 0.054428429957841695, - "planned_cases": 200, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/v4-test.json b/docs/training/jet-v5/v4-test.json deleted file mode 100644 index 49f5288a1a813e4c02b326664c542cde88a03ead..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v4-test.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.6644040184632093, - "nll": 0.8783917908240401, - "brier": 0.4363018875628721, - "ece": 0.044080123370055624 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 0.9888888888888889, - "nll": 0.04609997218164305, - "brier": 0.00993532707109211, - "ece": 0.03980768852763706 - }, - "boolq": { - "n": 145, - "acc": 0.7862068965517242, - "nll": 0.4650379801875558, - "brier": 0.2991545819338941, - "ece": 0.056202566623687736 - }, - "emotion": { - "n": 2500, - "acc": 0.5852, - "nll": 1.0812736734218895, - "brier": 0.5494403625547004, - "ece": 0.06851770701408388 - }, - "ag_news": { - "n": 135, - "acc": 0.8888888888888888, - "nll": 0.3087087308918988, - "brier": 0.16917814412184332, - "ece": 0.03841419992623507 - }, - "stsb": { - "n": 117, - "acc": 0.5470085470085471, - "nll": 1.0753559749796122, - "brier": 0.28199056470137557, - "ece": 0.03365990544995692, - "mae": 0.09383068554079495, - "within1": 0.8717948717948718, - "spearman": 0.877984058590352 - }, - "mnli": { - "n": 142, - "acc": 0.8309859154929577, - "nll": 0.4340047932059412, - "brier": 0.24689615776249432, - "ece": 0.07052275685357377 - }, - "yelp": { - "n": 146, - "acc": 0.6986301369863014, - "nll": 0.7119902010420209, - "brier": 0.41549555641030894, - "ece": 0.11680830300670779 - }, - "banking77": { - "n": 140, - "acc": 0.9071428571428571, - "nll": 0.3257405191460358, - "brier": 0.13279354980220526, - "ece": 0.07537455537489482 - }, - "massive": { - "n": 119, - "acc": 0.9327731092436975, - "nll": 0.2788916040219742, - "brier": 0.099814120778048, - "ece": 0.05662691129856755 - }, - "civil_comments": { - "n": 149, - "acc": 0.9328859060402684, - "nll": 0.32543574588733154, - "brier": 0.0470122361134796, - "ece": 0.10539027988510645 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6109395677106307, - "nll": 1.0674977012510223, - "brier": 0.508642577204092, - "ece": 0.054254766210197196 - }, - "noul": { - "n": 1185, - "acc": 0.7805907172995781, - "nll": 0.49928341122037284, - "brier": 0.30730912987911574, - "ece": 0.034595404093778585 - }, - "score": { - "n": 231, - "acc": 0.5930735930735931, - "nll": 0.9673109721763195, - "brier": 0.38807710158281145, - "ece": 0.06311224291334937, - "mae": 0.1062169110937673, - "within1": 0.9177489177489178, - "spearman": 0.899445205722314 - } - }, - "ms_per_question": 12.008765380939352, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v4-test_v4_new.json b/docs/training/jet-v5/v4-test_v4_new.json deleted file mode 100644 index 24e78bce26c54ba84eef322fa22e05419553f4b6..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v4-test_v4_new.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "overall": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6, - "nll": 0.8641331787531574, - "brier": 0.5189303984919411, - "ece": 0.08732014621297518 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.5708333333333333, - "nll": 1.0104112304126223, - "brier": 0.553216138337243, - "ece": 0.1417993993808826 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.8083333333333333, - "nll": 0.468007056050313, - "brier": 0.2578825642308249, - "ece": 0.0798518454035123 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.6597222222222222, - "nll": 0.7808504884053643, - "brier": 0.4433430336866696, - "ece": 0.04629867937829759 - } - }, - "ms_per_question": 32.63169955833215, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v4-test_v4_tools.json b/docs/training/jet-v5/v4-test_v4_tools.json deleted file mode 100644 index 852783dce806e520c4afd0184840a934d79b7c02..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v4-test_v4_tools.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "overall": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.9333333333333333, - "nll": 0.21368445424207796, - "brier": 0.09434197410141065, - "ece": 0.0977091632783413 - } - }, - "ms_per_question": 27.83224332916537, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v4-test_v5_new.json b/docs/training/jet-v5/v4-test_v5_new.json deleted file mode 100644 index 5946dd86640ecc105e957e563a4bcc0f76182f07..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v4-test_v5_new.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "overall": { - "n": 600, - "acc": 0.59, - "nll": 1.299430429075534, - "brier": 0.5974421801848655, - "ece": 0.14489226055641966 - }, - "by_source": { - "v5:adversarial_entailment": { - "n": 150, - "acc": 0.7066666666666667, - "nll": 0.7201601017514865, - "brier": 0.40315653132960594, - "ece": 0.055995979706446326 - }, - "v5:tool_response_preference": { - "n": 150, - "acc": 0.58, - "nll": 1.2166564369946717, - "brier": 0.6948852528274388, - "ece": 0.3230295610427857 - }, - "v5:commonsense_completion": { - "n": 150, - "acc": 0.3466666666666667, - "nll": 1.6604171573619049, - "brier": 0.8777165602201868, - "ece": 0.29102803190549215 - }, - "v5:wide_intent": { - "n": 150, - "acc": 0.7266666666666667, - "nll": 1.6004880201940734, - "brier": 0.41401037636223037, - "ece": 0.16799896627664568 - } - }, - "by_type": { - "choice": { - "n": 600, - "acc": 0.59, - "nll": 1.299430429075534, - "brier": 0.5974421801848655, - "ece": 0.14489226055641966 - } - }, - "ms_per_question": 36.09645434166699, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v5-diagnostic.json b/docs/training/jet-v5/v5-diagnostic.json deleted file mode 100644 index 3182578bc056385154b1a43459492908e1b3abca..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-diagnostic.json +++ /dev/null @@ -1,515 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "d38e5db2bb294bf3c25ca3a87a715d9b37e6ee3e07e0a9b6869de18b4a723745", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9092100469997604, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 69.52370399994834, - "p95": 480.62934599965956, - "mean": 114.77345353199962 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.6993761418550424, - "reference_same_cases": null, - "median_ms": 457.1604524999202, - "detail": { - "field_accuracy": 0.7247899159663865, - "case_exact_accuracy": 0, - "macro_f1": 0.6993761418550424, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.36607142857142855, - "reference_same_cases": null, - "median_ms": 57.515988999966794, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.32142857142857145, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.4107142857142857, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.36607142857142855, - "case_exact_accuracy": 0.19642857142857142, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 67.71158750007089, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.6608187134502924, - "case_exact_accuracy": 0.5357142857142857, - "macro_f1": 0.14420463629096722, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.18181818181818182, - "sarcasm": 0.0, - "irony": 0.14285714285714285, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.04638218923933209 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.4020241171403962, - "reference_same_cases": null, - "median_ms": 32.054457499725686, - "detail": { - "field_accuracy": 0.42857142857142855, - "case_exact_accuracy": 0.42857142857142855, - "macro_f1": 0.4020241171403962, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.5327336331834083, - "reference_same_cases": null, - "median_ms": 77.43316600044636, - "detail": { - "field_accuracy": 0.6909090909090909, - "case_exact_accuracy": 0.6909090909090909, - "macro_f1": 0.5327336331834083, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 59.778007000204525, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.43636363636363634, - "reference_same_cases": null, - "median_ms": 56.51821299943549, - "detail": { - "field_accuracy": 0.43636363636363634, - "case_exact_accuracy": 0.43636363636363634, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2909090909090909, - "reference_same_cases": null, - "median_ms": 87.25654099998792, - "detail": { - "field_accuracy": 0.2909090909090909, - "case_exact_accuracy": 0.2909090909090909, - "custom_metrics": { - "normalized_rank_regret": 0.2715151515151515 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.6993761418550424, - "skill": 0.5652464375871382, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.6993761418550424, - "random": 0.30851893087083776, - "skill": 0.5652464375871382, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.3660714285714286, - "skill": 0.23015873015873017, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.4107142857142857, - "random": 0.25, - "skill": 0.21428571428571427, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.32142857142857145, - "random": 0.1, - "skill": 0.24603174603174605, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.07142857142857142, - "skill": 0.020271110112861156, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.2857142857142857, - "random": 0.2226862339379383, - "skill": 0.08108444045144463, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.4020241171403962, - "skill": 0.1030479854598472, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.4020241171403962, - "random": 0.3333245556439576, - "skill": 0.1030479854598472, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.2909090909090909, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2909090909090909, - "random": 0.34943722943722944, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.43636363636363634, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.43636363636363634, - "random": 0.5, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2909090909090909, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2909090909090909, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/v5-expanded.json b/docs/training/jet-v5/v5-expanded.json deleted file mode 100644 index 6bb6528a0311145e2d5cc7057981fd469136f7d9..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-expanded.json +++ /dev/null @@ -1,274 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "9908538b8b244efa20a2d8428104d5f56a131b37569d0e0b10867ced7beb104b", - "results_sha256": "92601a6b2aded1ceb0747acea8f93b5a2ce855aa1a6b35cce60c77f82cb5052e", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_engine:JetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9395102169992242, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 28.298324499701266, - "p95": 68.66376899961324, - "mean": 39.40092550857116 - }, - "benchmarks": [ - { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.34160092979549445, - "reference_same_cases": null, - "median_ms": 67.74492449994796, - "detail": { - "field_accuracy": 0.435, - "case_exact_accuracy": 0.435, - "macro_f1": 0.34160092979549445, - "custom_metrics": {} - } - }, - { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.36173331054221564, - "reference_same_cases": null, - "median_ms": 28.096252500290575, - "detail": { - "field_accuracy": 0.365, - "case_exact_accuracy": 0.365, - "macro_f1": 0.36173331054221564, - "custom_metrics": {} - } - }, - { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.305, - "reference_same_cases": null, - "median_ms": 27.939554999647953, - "detail": { - "field_accuracy": 0.305, - "case_exact_accuracy": 0.305, - "custom_metrics": {} - } - }, - { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.595, - "reference_same_cases": null, - "median_ms": 27.784917000190035, - "detail": { - "field_accuracy": 0.595, - "case_exact_accuracy": 0.595, - "custom_metrics": {} - } - }, - { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.4, - "reference_same_cases": null, - "median_ms": 27.62665399995967, - "detail": { - "field_accuracy": 0.4, - "case_exact_accuracy": 0.4, - "custom_metrics": {} - } - }, - { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.51, - "reference_same_cases": null, - "median_ms": 27.359134499874926, - "detail": { - "field_accuracy": 0.51, - "case_exact_accuracy": 0.51, - "custom_metrics": {} - } - }, - { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.35, - "reference_same_cases": null, - "median_ms": 30.461512500096433, - "detail": { - "field_accuracy": 0.35, - "case_exact_accuracy": 0.35, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "24": { - "catalog_id": 24, - "raw": 0.26829636703166115, - "skill": 0.02439515604221487, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.26829636703166115, - "random": 0.25, - "skill": 0.02439515604221487, - "planned_cases": 200, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/v5-test.json b/docs/training/jet-v5/v5-test.json deleted file mode 100644 index bb8c2db14c9ed54a614f103f31b22851b1337382..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-test.json +++ /dev/null @@ -1,112 +0,0 @@ -{ - "overall": { - "n": 3683, - "acc": 0.6665761607385283, - "nll": 0.8739479481013483, - "brier": 0.43611422339553557, - "ece": 0.04741072242039235 - }, - "by_source": { - "dbpedia": { - "n": 90, - "acc": 1.0, - "nll": 0.018827906827856268, - "brier": 0.0011956740026198359, - "ece": 0.018382265832689072 - }, - "boolq": { - "n": 145, - "acc": 0.7586206896551724, - "nll": 0.477754406849372, - "brier": 0.31074431400108665, - "ece": 0.062077112033449344 - }, - "emotion": { - "n": 2500, - "acc": 0.5884, - "nll": 1.0728664409339428, - "brier": 0.5478057465702815, - "ece": 0.07044296820163728 - }, - "ag_news": { - "n": 135, - "acc": 0.8962962962962963, - "nll": 0.31467971743808854, - "brier": 0.16612003492267927, - "ece": 0.04851973674915451 - }, - "stsb": { - "n": 117, - "acc": 0.5470085470085471, - "nll": 1.0867419113577246, - "brier": 0.2919386456474603, - "ece": 0.03596794783559621, - "mae": 0.09691258877929713, - "within1": 0.8888888888888888, - "spearman": 0.8702251401658272 - }, - "mnli": { - "n": 142, - "acc": 0.823943661971831, - "nll": 0.4528743649478739, - "brier": 0.2576190302733788, - "ece": 0.05873414958027046 - }, - "yelp": { - "n": 146, - "acc": 0.7191780821917808, - "nll": 0.7171766153307095, - "brier": 0.41457848929907215, - "ece": 0.06495250455320697 - }, - "banking77": { - "n": 140, - "acc": 0.9071428571428571, - "nll": 0.29728970748172806, - "brier": 0.12084634680012522, - "ece": 0.04855451328413831 - }, - "massive": { - "n": 119, - "acc": 0.9327731092436975, - "nll": 0.2904803742519619, - "brier": 0.11068498801003222, - "ece": 0.02553347880099001 - }, - "civil_comments": { - "n": 149, - "acc": 0.9328859060402684, - "nll": 0.3408124509224143, - "brier": 0.051982496863425036, - "ece": 0.13118187733144568 - } - }, - "by_type": { - "choice": { - "n": 2267, - "acc": 0.6162329069254522, - "nll": 1.062267945216914, - "brier": 0.5083710705656884, - "ece": 0.06300196647118199 - }, - "noul": { - "n": 1185, - "acc": 0.7729957805907173, - "nll": 0.49518120399083376, - "brier": 0.30745203998831483, - "ece": 0.03731607993443807 - }, - "score": { - "n": 231, - "acc": 0.6147186147186147, - "nll": 0.9688274213046904, - "brier": 0.38701645197917245, - "ece": 0.04226513019887914, - "mae": 0.1068101010833629, - "within1": 0.9307359307359307, - "spearman": 0.8982739913973843 - } - }, - "ms_per_question": 12.24837621694279, - "max_state_tokens": 4096 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v5-test_v4_new.json b/docs/training/jet-v5/v5-test_v4_new.json deleted file mode 100644 index e509a71a7e313156275a4d41ee4bdc015585074c..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-test_v4_new.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "overall": { - "n": 720, - "acc": 0.675, - "nll": 0.7720444074689617, - "brier": 0.44042844510431245, - "ece": 0.0333876195881102 - }, - "by_source": { - "v4:stance_transfer": { - "n": 240, - "acc": 0.6416666666666667, - "nll": 0.8378210284436742, - "brier": 0.5083150294142842, - "ece": 0.1171224950502316 - }, - "v4:code_transfer": { - "n": 240, - "acc": 0.5666666666666667, - "nll": 1.0331417464340726, - "brier": 0.5624325577067126, - "ece": 0.15080579866965613 - }, - "v4:contract_entailment": { - "n": 240, - "acc": 0.8166666666666667, - "nll": 0.44517044752913837, - "brier": 0.2505377481919405, - "ece": 0.05922942943871021 - } - }, - "by_type": { - "choice": { - "n": 720, - "acc": 0.675, - "nll": 0.7720444074689617, - "brier": 0.44042844510431245, - "ece": 0.0333876195881102 - } - }, - "ms_per_question": 32.87891285833413, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v5-test_v4_tools.json b/docs/training/jet-v5/v5-test_v4_tools.json deleted file mode 100644 index 561bdec1e440716f883265beb7b4bb3834b56a67..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-test_v4_tools.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "overall": { - "n": 240, - "acc": 0.85, - "nll": 0.38071567004856965, - "brier": 0.20079053168001415, - "ece": 0.07791914530098437 - }, - "by_source": { - "v4:tool_intent_routing": { - "n": 240, - "acc": 0.85, - "nll": 0.38071567004856965, - "brier": 0.20079053168001415, - "ece": 0.07791914530098437 - } - }, - "by_type": { - "choice": { - "n": 240, - "acc": 0.85, - "nll": 0.38071567004856965, - "brier": 0.20079053168001415, - "ece": 0.07791914530098437 - } - }, - "ms_per_question": 28.23875877500086, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v5-test_v5_new.json b/docs/training/jet-v5/v5-test_v5_new.json deleted file mode 100644 index eaf6387a05a199a201a21b9966469734a7b861b7..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-test_v5_new.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "overall": { - "n": 600, - "acc": 0.735, - "nll": 0.7329250454087741, - "brier": 0.3475341883402461, - "ece": 0.036768475671609256 - }, - "by_source": { - "v5:adversarial_entailment": { - "n": 150, - "acc": 0.7466666666666667, - "nll": 0.6646332611391942, - "brier": 0.37303307763688937, - "ece": 0.052704064846038796 - }, - "v5:tool_response_preference": { - "n": 150, - "acc": 0.98, - "nll": 0.12095037142435709, - "brier": 0.05221765650556733, - "ece": 0.06036541581153872 - }, - "v5:commonsense_completion": { - "n": 150, - "acc": 0.3466666666666667, - "nll": 1.3898383743564289, - "brier": 0.7549682163529782, - "ece": 0.11845218261082968 - }, - "v5:wide_intent": { - "n": 150, - "acc": 0.8666666666666667, - "nll": 0.7562781747151166, - "brier": 0.20991780286554929, - "ece": 0.045211266080538416 - } - }, - "by_type": { - "choice": { - "n": 600, - "acc": 0.735, - "nll": 0.7329250454087741, - "brier": 0.3475341883402461, - "ece": 0.036768475671609256 - } - }, - "ms_per_question": 37.07279578166587, - "max_state_tokens": 2048 -} \ No newline at end of file diff --git a/docs/training/jet-v5/v5-two-order-expanded.json b/docs/training/jet-v5/v5-two-order-expanded.json deleted file mode 100644 index 117bbaea0a7a648fcad27beea95ae1a30f4f9688..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-two-order-expanded.json +++ /dev/null @@ -1,276 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "9908538b8b244efa20a2d8428104d5f56a131b37569d0e0b10867ced7beb104b", - "results_sha256": "7475ade0c21f1f53d5a6dc7fb56e601ecfe7bfde7f3c4afa6e823f9850300f81", - "planned_requests": 1400, - "processed_requests": 1400, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_ensemble:TwoOrderJetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9424751719998312, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/v5-diagnostic-1400/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 1400 - }, - "successful_request_latency_ms": { - "median": 59.29019349969167, - "p95": 139.69594100035465, - "mean": 76.16256648071352 - }, - "benchmarks": [ - { - "catalog_id": 5, - "dataset": "CLINC150+OOS", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3534098658786088, - "reference_same_cases": null, - "median_ms": 138.0714999995689, - "detail": { - "field_accuracy": 0.455, - "case_exact_accuracy": 0.455, - "macro_f1": 0.3534098658786088, - "custom_metrics": {} - } - }, - { - "catalog_id": 12, - "dataset": "ANLI", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "macro-F1", - "score": 0.3371351342582988, - "reference_same_cases": null, - "median_ms": 58.64838249999593, - "detail": { - "field_accuracy": 0.34, - "case_exact_accuracy": 0.34, - "macro_f1": 0.3371351342582988, - "custom_metrics": {} - } - }, - { - "catalog_id": 24, - "dataset": "MMLU", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.335, - "reference_same_cases": null, - "median_ms": 58.66689300000871, - "detail": { - "field_accuracy": 0.335, - "case_exact_accuracy": 0.335, - "custom_metrics": {} - } - }, - { - "catalog_id": 26, - "dataset": "ARC-Easy", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.635, - "reference_same_cases": null, - "median_ms": 57.258969499798695, - "detail": { - "field_accuracy": 0.635, - "case_exact_accuracy": 0.635, - "custom_metrics": {} - } - }, - { - "catalog_id": 27, - "dataset": "ARC-Challenge", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.415, - "reference_same_cases": null, - "median_ms": 57.43540750017928, - "detail": { - "field_accuracy": 0.415, - "case_exact_accuracy": 0.415, - "custom_metrics": {} - } - }, - { - "catalog_id": 28, - "dataset": "WinoGrande", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.46, - "reference_same_cases": null, - "median_ms": 56.90452049975647, - "detail": { - "field_accuracy": 0.46, - "case_exact_accuracy": 0.46, - "custom_metrics": {} - } - }, - { - "catalog_id": 29, - "dataset": "HellaSwag", - "requests": 200, - "answered": 200, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 200, - "metric": "accuracy", - "score": 0.37, - "reference_same_cases": null, - "median_ms": 63.350262500080134, - "detail": { - "field_accuracy": 0.37, - "case_exact_accuracy": 0.37, - "custom_metrics": {} - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "24": { - "catalog_id": 24, - "raw": 0.31300343773873185, - "skill": 0.08400458365164247, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.31300343773873185, - "random": 0.25, - "skill": 0.08400458365164247, - "planned_cases": 200, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/v5-two-order-original.json b/docs/training/jet-v5/v5-two-order-original.json deleted file mode 100644 index d1ef7376154a3c4f0c89c4dc9d9765bc3bb4ac30..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/v5-two-order-original.json +++ /dev/null @@ -1,517 +0,0 @@ -{ - "scope": "Diagnostic subset only. Not an official Decision Index or full-suite estimate.", - "rows_sha256": "a790c3d5dac6068ecef13de92c7700c4da0a9ed87b998f528701ac1fe219205d", - "results_sha256": "c5f6dae1edfef53dde22a36992b26e26838686df2f6ce52fe4e9813ff451b253", - "planned_requests": 500, - "processed_requests": 500, - "pending_requests": 0, - "complete_for_supplied_rows": true, - "environment": { - "engine": "decision_index_ensemble:TwoOrderJetEngine", - "engine_options": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "adapter": "adapters/jet-v5-panel02-r2-20260924", - "max_tokens": 8192, - "max_cached_tokens": 8192 - }, - "model_source": { - "model": "mlx-community/Qwen3-0.6B-bf16", - "revision": "42096995f6402fde107068cf530136fe64b604f8", - "backend": "mlx", - "max_tokens": 8192, - "checkpoint_context": 40960, - "max_cached_tokens": 8192, - "choice_temperature": 1.0, - "adapter_sha256": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "format_sha256": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "policy": "Native Jet prompt and label-token readout; complete input or unsupported; no truncation, option pruning, or benchmark-specific prompts.", - "lora_adapter": "/home/jach/dev/jet/adapters/jet-v5-panel02-r2-20260924", - "lora_weights_sha256": "6416adc20cc1517dc5d9cf3c0a2655da8a899296c5640dfc6d5551170824d0cb", - "calibration_sha256": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", - "choice_readout": "Mean of original and reversed option-order probabilities, aligned by exact original keys.", - "ensemble_code_sha256": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1" - }, - "platform": "Linux-7.2.6-arch2-1-x86_64-with-glibc2.44", - "mlx": "0.32.2", - "mlx_lm": "0.31.3", - "device": "Device(gpu, 0)", - "loaded_seconds": 1.9530501290000757, - "frozen_corpus_sha256": null, - "rows_path": "artifacts/decision-index/diagnostic/diagnostic.jsonl.gz", - "runner_sha256": "fac268fc460a2a3470f52ee9e785fd81b37864cace132ac9e6284ede806258b8", - "latency": "Synchronized local MLX request wall time, including full prompt construction and shared-prefix inference; excludes model loading." - }, - "summary": { - "engine": "jet", - "counts": { - "ok": 500 - }, - "successful_request_latency_ms": { - "median": 97.2111895002854, - "p95": 708.4078269999736, - "mean": 174.5285182359912 - }, - "benchmarks": [ - { - "catalog_id": 11, - "dataset": "ContractNLI", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.7073362638505957, - "reference_same_cases": null, - "median_ms": 667.4918250000701, - "detail": { - "field_accuracy": 0.7331932773109243, - "case_exact_accuracy": 0, - "macro_f1": 0.7073362638505957, - "custom_metrics": {} - } - }, - { - "catalog_id": 30, - "dataset": "GSM8K", - "requests": 112, - "answered": 112, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 112, - "metric": "accuracy", - "score": 0.5, - "reference_same_cases": null, - "median_ms": 84.65050700033316, - "tracks": { - "GSM8K-10choice": { - "metric": "accuracy", - "score": 0.5178571428571429, - "scored_requests": 56 - }, - "GSM8K-4choice": { - "metric": "accuracy", - "score": 0.48214285714285715, - "scored_requests": 56 - } - }, - "detail": { - "field_accuracy": 0.5, - "case_exact_accuracy": 0.2857142857142857, - "custom_metrics": {} - } - }, - { - "catalog_id": 40, - "dataset": "iSarcasmEval", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "see separate subtask tracks", - "score": null, - "reference_same_cases": null, - "median_ms": 95.17290849998972, - "tracks": { - "iSarcasmEval-A-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 12 - }, - "iSarcasmEval-A-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 19 - }, - "iSarcasmEval-B-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 23 - }, - "iSarcasmEval-C-Ar": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - }, - "iSarcasmEval-C-En": { - "metric": "see separate subtask tracks", - "score": null, - "scored_requests": 1 - } - }, - "detail": { - "field_accuracy": 0.5906432748538012, - "case_exact_accuracy": 0.48214285714285715, - "macro_f1": 0.3588834324114758, - "custom_metrics": {}, - "positive_f1_by_field": { - "sarcastic": 0.15384615384615385, - "sarcasm": 0.0, - "irony": 0.1111111111111111, - "satire": 0.0, - "understatement": 0.0, - "overstatement": 0.0, - "rhetorical_question": 0.0 - }, - "category_macro_f1": 0.03785103785103785 - } - }, - { - "catalog_id": 41, - "dataset": "VAST", - "requests": 56, - "answered": 56, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 56, - "metric": "macro-F1", - "score": 0.3866281866281866, - "reference_same_cases": null, - "median_ms": 60.61503049977546, - "detail": { - "field_accuracy": 0.44642857142857145, - "case_exact_accuracy": 0.44642857142857145, - "macro_f1": 0.3866281866281866, - "custom_metrics": {} - } - }, - { - "catalog_id": 42, - "dataset": "NLI4CT", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "macro-F1", - "score": 0.45, - "reference_same_cases": null, - "median_ms": 113.20878300011827, - "detail": { - "field_accuracy": 0.6727272727272727, - "case_exact_accuracy": 0.6727272727272727, - "macro_f1": 0.45, - "custom_metrics": {} - } - }, - { - "catalog_id": 43, - "dataset": "CRUXEval", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.36363636363636365, - "reference_same_cases": null, - "median_ms": 90.6655580001825, - "detail": { - "field_accuracy": 0.36363636363636365, - "case_exact_accuracy": 0.36363636363636365, - "custom_metrics": {} - } - }, - { - "catalog_id": 44, - "dataset": "CLadder", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.509090909090909, - "reference_same_cases": null, - "median_ms": 81.81080399936036, - "detail": { - "field_accuracy": 0.509090909090909, - "case_exact_accuracy": 0.509090909090909, - "custom_metrics": {} - } - }, - { - "catalog_id": 50, - "dataset": "Habermas Machine", - "requests": 55, - "answered": 55, - "unsupported": 0, - "errors": 0, - "abstained": 0, - "pending": 0, - "scored_requests": 55, - "metric": "accuracy", - "score": 0.2727272727272727, - "reference_same_cases": null, - "median_ms": 133.38184700023703, - "detail": { - "field_accuracy": 0.2727272727272727, - "case_exact_accuracy": 0.2727272727272727, - "custom_metrics": { - "normalized_rank_regret": 0.30787878787878786 - } - } - } - ], - "note": "Native benchmark metrics on complete supported case groups. Unsupported cases excluded from accuracy but retained in coverage." - }, - "panel_metrics_on_sample": { - "11": { - "catalog_id": 11, - "raw": 0.7073362638505957, - "skill": 0.5767581366790283, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.7073362638505957, - "random": 0.30851893087083776, - "skill": 0.5767581366790283, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"The contract contradicts the hypothesis.\"", - "\"The contract entails the hypothesis.\"", - "\"The hypothesis is neither entailed nor contradicted by the contract.\"" - ] - } - ] - }, - "30": { - "catalog_id": 30, - "raw": 0.5, - "skill": 0.386904761904762, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "GSM8K-4choice", - "raw": 0.48214285714285715, - "random": 0.25, - "skill": 0.30952380952380953, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "GSM8K-10choice", - "raw": 0.5178571428571429, - "random": 0.1, - "skill": 0.46428571428571436, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "40": { - "catalog_id": 40, - "raw": 0.3125, - "skill": 0.2587846656185042, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "iSarcasmEval-A-En", - "raw": 0.25, - "random": 0.2226862339379383, - "skill": 0.03513866247401688, - "planned_cases": 19, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-A-Ar", - "raw": 0.0, - "random": 0.22250656045660705, - "skill": 0.0, - "planned_cases": 12, - "coverage": 1, - "pending": 0, - "label_universe": [ - "no", - "yes" - ] - }, - { - "track": "iSarcasmEval-C-En", - "raw": 0.0, - "random": 0.5, - "skill": 0.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - }, - { - "track": "iSarcasmEval-C-Ar", - "raw": 1.0, - "random": 0.5, - "skill": 1.0, - "planned_cases": 1, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "41": { - "catalog_id": 41, - "raw": 0.3866281866281866, - "skill": 0.07995439375409463, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.3866281866281866, - "random": 0.3333245556439576, - "skill": 0.07995439375409463, - "planned_cases": 56, - "coverage": 1, - "pending": 0, - "label_universe": [ - "\"against\"", - "\"favor\"", - "\"neutral\"" - ] - } - ] - }, - "43": { - "catalog_id": 43, - "raw": 0.36363636363636365, - "skill": 0.02182592494011181, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.36363636363636365, - "random": 0.34943722943722944, - "skill": 0.02182592494011181, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "44": { - "catalog_id": 44, - "raw": 0.509090909090909, - "skill": 0.018181818181818077, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.509090909090909, - "random": 0.5, - "skill": 0.018181818181818077, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "50": { - "catalog_id": 50, - "raw": 0.2727272727272727, - "skill": 0.0, - "coverage": 1, - "pending": 0, - "tracks": [ - { - "track": "all", - "raw": 0.2727272727272727, - "random": 0.31, - "skill": 0.0, - "planned_cases": 55, - "coverage": 1, - "pending": 0, - "label_universe": [] - } - ] - }, - "7": { - "catalog_id": 7, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "16": { - "catalog_id": 16, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "13": { - "catalog_id": 13, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "15": { - "catalog_id": 15, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "17": { - "catalog_id": 17, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - }, - "19": { - "catalog_id": 19, - "raw": 0.0, - "skill": 0.0, - "coverage": 0.0, - "pending": 1.0, - "status": "unrun; provisional lower bound", - "tracks": [] - } - } -} diff --git a/docs/training/jet-v5/verification.json b/docs/training/jet-v5/verification.json deleted file mode 100644 index 9a11c8284f78705935c68f01a77c24183a2cfa8a..0000000000000000000000000000000000000000 --- a/docs/training/jet-v5/verification.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "adapter_contract_and_native_model_tests": { - "passed": 8, - "failed": 0 - }, - "git_diff_check": "passed", - "training": "completed", - "expanded_evaluation": "completed" -} diff --git a/release-manifest.json b/release-manifest.json index ca14010b9e31803320443959855c7b87a8320f3b..981791df15effe7cda0ab9067c6f1868a5ad22f9 100644 --- a/release-manifest.json +++ b/release-manifest.json @@ -8,198 +8,11 @@ "reference_backend": "native MLX CUDA fused bf16" }, "files": { - "README.md": "a785b2298a0e1c5b55197e54343748e743bf98b42f59ebc6ba024d3e853a0162", - "TRAINING_HISTORY.md": "d4aa6de3db5c3358bdfc2e652cff1dfeca9c1486f38c2ccd771d4c38a5d1c6a2", + "README.md": "62a746e114cc3365b2b1f00c03bda5bee234f30186a2529eb2e1e1f09262e015", + "TRAINING_HISTORY.md": "c94f2d2570078049f66ac89ec420b7bef7fba6a614d3e8add54444dfde092cde", "calibration.json": "6d9ba9d874699bf1f48f0703a814075a175ccb460d65c2c4870a3e6fb37fdb2f", "chat_template.jinja": "87a2728cb8dc9fe424d624542f6060ec05a1d285ebbec578bb078900e33396b5", "config.json": "332d56de2e5cde28245ac8b1776639bf9f7ed55fc911dfd2399ae8872acee2a9", - "docs/training/initial-results.md": "a67cdfa2b07c992dd0881690d05251d06c8a29cc5d0cc66bafc9ee6283d378cc", - "docs/training/jet-next/baseline-adapter-check.json": "9c6f8657ee922f3b7c8eae98886a2b5321a822a7e97ddf413040ead869b14c43", - "docs/training/jet-next/baseline-score_eval.json": "9b689dcf1d731659cd352b487e22b8ed16123dade4641cf1ceda0ce228a6c234", - "docs/training/jet-next/baseline-test.json": "049700466446d476f16fb4bdf59296c03e01352a7d141b6102f1c618df146a4b", - "docs/training/jet-next/baseline-test_v3_new.json": "9bbb0759b5e238224c4a5becf0d84d273b6ba8dce1bfc2b4c21caef7ada33083", - "docs/training/jet-next/baseline-test_v4_new.json": "f3e48fcf13ae6486ed753b52fe58cdccb77661cc68d5439c2407e9005fb39666", - "docs/training/jet-next/baseline-test_v4_tools.json": "2353fb66a3743787840fd2f67d57f8ce3277f1bb193920ad8911bfd3204956ca", - "docs/training/jet-next/build-v4-snapshot.py": "37262ee1a8978d19c58a031e11598d504ed2d4b981ee2f004d6c0a8ccff9af9b", - "docs/training/jet-next/check-baseline-adapter.py": "a3c88ee071fbd67d6cc5708babdedb4bfaa290fa98c036921a233fc272f76c4d", - "docs/training/jet-next/code/scripts/audit_v4_tools.py": "5e5c38a42274220bffe681133fa3f0ac6ac000bd72e1631e2a12b33b0f52d789", - "docs/training/jet-next/code/scripts/audit_v4_transfer.py": "a878e78f59fbd1a1dd2032279222073b4c0dba01e509b231a77a74d21c24d011", - "docs/training/jet-next/code/scripts/bench_index_cuda.sh": "415a46bf5bb39a4fdb9760339abff2304fb9762969da929965b6d26636ca99fc", - "docs/training/jet-next/code/scripts/build_v4_tools.py": "35b3694f83c326b3e236cf2528ebba341af0a0be34bb85b365409e6003019626", - "docs/training/jet-next/code/scripts/build_v4_transfer.py": "37262ee1a8978d19c58a031e11598d504ed2d4b981ee2f004d6c0a8ccff9af9b", - "docs/training/jet-next/code/scripts/evaluate_next_code.sh": "7755cc0e69d834db4686af24d4c1b991e8a4ae2f4593dac9ff159c9aec01236f", - "docs/training/jet-next/code/scripts/evaluate_next_extra.sh": "21b11259db33cbbf5c6e39b8cc4526770a0c2d71c5b7a5b0040c94447745a889", - "docs/training/jet-next/code/scripts/finalize_next.py": "33b9456de4fdcca0f42ce81ef8ce2d8e35b36536e759ea47191179993bfb5b73", - "docs/training/jet-next/code/scripts/prepare_v4_focus.py": "bd47eec38af66c0f27cd8973c8f4053bfaa558874b26a82b919a262d1916848c", - "docs/training/jet-next/code/scripts/run_next_cuda.sh": "8cb6f179a0467565eb0cdf9348b55e305f628694dd82be8e15ae4c0fb4faa161", - "docs/training/jet-next/code/scripts/summarize_next.py": "584fe4addd24ce9b03de0b67635009bdaba9af39932f2d35a3505652b764f1df", - "docs/training/jet-next/code/src/bench_decision_index.py": "809bbfb59ecf007cb3b38a501ebeb36ec6fa9c68649cd21c360e5c15bde5536d", - "docs/training/jet-next/code/src/data/decision_training.py": "785b8407f93c9ba952f98d29f987be73c853707f4f3211ccd6a4865c4cee3061", - "docs/training/jet-next/code/src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "docs/training/jet-next/code/src/evaluate.py": "25dcaf7bcf9ddcfdd8349eeea047ef7cc33388cf74f32a65c8a1931b4f40f72c", - "docs/training/jet-next/code/src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "docs/training/jet-next/code/src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "docs/training/jet-next/code/src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "docs/training/jet-next/code/src/train.py": "040ef12d706db397d6031a745faf9fc922497e6eb59f0d45b0ac18cd13b8cbfd", - "docs/training/jet-next/code/train_cuda.sh": "464ebe6e844f2a02983f76f66edb7fd7f9354ed407c1ff2dfa043b97cfea6dcf", - "docs/training/jet-next/comparison.json": "7db37d89597341aa6afc4e7e695e27995e297b8254d25fe6db390b64780c7698", - "docs/training/jet-next/completion-audit.json": "aea3e1899468011fc7faad7812a6d0476a70b3ccae621ceb8f8732eea597bfc4", - "docs/training/jet-next/cruxeval-full-baseline.json": "0d6542217c36e030e7feca64934f5454daeaf8ca2c9f8f97e2cebc93f3c29798", - "docs/training/jet-next/cruxeval-full-sample.json": "627847a835eec825cb6f6cf5d656304c3594e4bb50af8fae100e149669947800", - "docs/training/jet-next/cruxeval-full-v3.json": "cef0278e286656a753dacea579fae513fc313c2f29889099954dd9617d5e6722", - "docs/training/jet-next/cruxeval-full-v4.json": "bb1918cfbc0c3e26fb8fe39b725172326542ceeba3ee6f3ab980def05e7e20b3", - "docs/training/jet-next/cruxeval-full-warm.json": "16338604f4a7ebe22f6a2e406e9ff301ba800c7676c483b851c0dab39938412d", - "docs/training/jet-next/diagnostic-v4.json": "350a5e06cc6283085b4c44866979b1b53f84f1ecd6c7ae3d0ac8694deb0ad65e", - "docs/training/jet-next/diagnostic-warm.json": "20bab9e83f3a27f0a52aec06c72fb0bb1ee342ad198f8c174594e4d88141f74b", - "docs/training/jet-next/environment-freeze.txt": "f4cea0cd658cf3b785572d6a9112efd860715c5f2772fcf6ae38fa89be95c296", - "docs/training/jet-next/esci-baseline.json": "7e3c82c863e4a93e2d5c9df4c5ee8bbf4a30258d36d848a7e23805ea6fa3a2db", - "docs/training/jet-next/esci-coverage.json": "96b15761517ff7a6a6dcedbb481c8f95ddf90c326ec21d5bb675b0fc59367344", - "docs/training/jet-next/esci-training-overlap.json": "2c4b5cb6152450cfe85a023fd27645f25d41463e1e6764669959bab91e4b4bb2", - "docs/training/jet-next/esci-v3.json": "6defc438d325ecee29b6d2ae420e15ab5ec057d69b68c39084ba0c9850a1fb99", - "docs/training/jet-next/esci-v4.json": "0d1a981b2272bed61bfa8bd74e2857bffd201d50d155e387e9021ddd13fe619f", - "docs/training/jet-next/esci-warm.json": "ff598268235088aa4acd092baf6f212bdf7caca8bee3049f04065da12646563f", - "docs/training/jet-next/isarcasm-en-baseline.json": "17242cc9157c2ab70e8999339f49d6ef00493b887f4fb8b7791271175de46f6b", - "docs/training/jet-next/isarcasm-en-v3.json": "5ac677e8636e46b4ce6487c77ee1f5324fcb427458fb9ce210ab7b985b914c53", - "docs/training/jet-next/isarcasm-en-v4.json": "8cd698f547a69537782d8298688b014b25d7f0104e85eb4464bc03893e38a33f", - "docs/training/jet-next/isarcasm-en-warm.json": "ac0c44740ebfc39006eb7ace310cd9904a24340b82d1ccaba789535775c4123b", - "docs/training/jet-next/isarcasm-english-sample.json": "fd7a29ada7f60498e13a2dc3b15a869c0be883913814d781308223bfc4a70547", - "docs/training/jet-next/leaderboard-snapshot.json": "01eda20dfb51d052821de887bfe76b06e050527fa9cd70f6a92d0d1dd67b4037", - "docs/training/jet-next/run-manifest.json": "0d0938cdfade8dd2a4bc7c998deaf0b923b6e150720f040de7e66ec53eaae40f", - "docs/training/jet-next/summary.md": "6a424719bc3f880cb77baa697d85fd2f46bdb3c264b0488dd4d50d37ecd8006f", - "docs/training/jet-next/train-v4-snapshot.py": "040ef12d706db397d6031a745faf9fc922497e6eb59f0d45b0ac18cd13b8cbfd", - "docs/training/jet-next/train-warm-snapshot.py": "d1552430ccc972c13e6371ba3030c86ff8439d7a96f46729b1add5e2fa344624", - "docs/training/jet-next/train_v4.provenance.json": "be2e053a0af8cc71dd2a636dd82d7b1889370e5aae904828731c5be457905fee", - "docs/training/jet-next/train_v4_focus.provenance.json": "8e44480f60bda6e503a86964dcbc2379ec0e1995887a71b8732e9c3988548281", - "docs/training/jet-next/train_v4_tools.provenance.json": "a49bae47e8dc0aec508f98c39969e97890480bef948e20a29d78c632f66e45c3", - "docs/training/jet-next/v3-score_eval.json": "d80f9371fae0ab416ba112588181794e0e52f1e80f4db73316a7e5996c120e48", - "docs/training/jet-next/v3-test.json": "c29ff31850e36e830fc8ed446ecf0b1d8452f14dc15b1fc011468a9396c7a53a", - "docs/training/jet-next/v3-test_v3_new.json": "adf997277737068dd2228bac72f54367b466976ae07125da426cb12bac8d8150", - "docs/training/jet-next/v3-test_v4_new.json": "9d21de23ec659fdc7e2a5abce406eb099531c11d588432834be75e0416dbb53d", - "docs/training/jet-next/v3-test_v4_tools.json": "bf397bb69d8da65b8d00582316e15ccf9134b51a8f7886121fdaf582990afcaa", - "docs/training/jet-next/v4-calibration.json": "0aaf5066fcf227488a6df68f9629b3f446631f2bb25ca7594a4721e5899a7384", - "docs/training/jet-next/v4-data-audit.json": "24ac17e1847429710d3d3ac99e541dc858adfe709d94a1a8cd2227fd086bd9ae", - "docs/training/jet-next/v4-metrics.jsonl": "329d43e3ad34c399d2a4191f75e3800d4297dce7c0b5210ae2239342db07d634", - "docs/training/jet-next/v4-score_eval.json": "fcae89e13b90b9b017c55f96859d6570da39f462724f7cc9d096ff26c8a0fbbe", - "docs/training/jet-next/v4-smoke-config.json": "78664372c6f9f699a138ea75a8bec0e319a35e753f70209b92183d1b395fc9fe", - "docs/training/jet-next/v4-smoke-metrics.jsonl": "9d7083b2c4e44a7f7730d6404fa807c59876e273040648872d3e487fb846416c", - "docs/training/jet-next/v4-test.json": "933c76312df6862aa4b1b7162dc7beadbcbdac7cded20fccb4e2ec05572245e7", - "docs/training/jet-next/v4-test_v3_new.json": "6cbb4700338d4bc9d8a6df2106f19b76cd729cd3368d14ab18bd01f677a763ae", - "docs/training/jet-next/v4-test_v4_new.json": "4cbeaaf49f7b80bb1d5d8ea6316f2b44a5c5e80d8ad97e99cef5fba91e496f22", - "docs/training/jet-next/v4-test_v4_tools.json": "e3be0601c04d508e5b93ff8968a60e89a9c37ad740cafecd6d86c8d97cc8ba5d", - "docs/training/jet-next/v4-tool-audit.json": "f192ae45956efc506eb65730c9691db7c4a5241ecb54b71c9959b9ccab81f812", - "docs/training/jet-next/v4-training-config.json": "a4e5a762c118b9a055ab7f436ae1ba41a4120ea648b0f749f5465197dbe64ffd", - "docs/training/jet-next/warm-calibration.json": "6a6dda82f9dacd1839189090ae03bcca5f3b9ca4c2c15e050609f89f2f52c226", - "docs/training/jet-next/warm-metrics.jsonl": "9deef416f05f8d60d139cb8255febab039d6a49c3d7f0332f5e6941fda1e1266", - "docs/training/jet-next/warm-score_eval.json": "959f0b63ce1a5ffb5dda09d12f46ac04ed936c8f462f5caa6471cb2e767da4c9", - "docs/training/jet-next/warm-test.json": "bafc8dffc6c06b7cc51f6910cfe5d34bede237ae4e59f548fc565b7c32297dc8", - "docs/training/jet-next/warm-test_v3_new.json": "0ca0d40b0629aabbff652622a3d95f40019dd210a523c84add1059c72ab24a69", - "docs/training/jet-next/warm-test_v4_new.json": "26869006ff96880cdf3f0db68f906fc45c570024e7da3bcbd19ebee12fdbcc16", - "docs/training/jet-next/warm-test_v4_tools.json": "256efe205802768b5f4a21904442149bddae344be22490a48aa5c5d1708363c8", - "docs/training/jet-next/warm-training-config.json": "05015f019cc7b45d27267b50ffdef308fdc65b36a20e24404c0b6597447a2d9a", - "docs/training/jet-next.md": "f814f2c5852fb069264614d08b230278af38c92ddc83184dfff8deb7d97b9eba", - "docs/training/jet-v3/adapter_config.json": "b0f7e0035cb5b63fdd3bf22df23dc22bc8955cc5645e45860b8d28d33dea359f", - "docs/training/jet-v3/baseline-score_eval.json": "9fbdaaf46e02e5c4ead5190ca3028e479452f2a0253215edd759c24f66deeb3f", - "docs/training/jet-v3/baseline-test.json": "a72b8955004e0e7649e3b0b1ec33fe308143ad96ddd7f388de32d81d4e7a2bc4", - "docs/training/jet-v3/baseline-test_v3_new.json": "187c17ac2507015cb5f7daf267229999b404ef538a72a42f06d3cd37a648fb8c", - "docs/training/jet-v3/calibration.json": "e5130c08d2de1105152ded40fdb6998f4eff21e588370263c0dc51af4386f0c5", - "docs/training/jet-v3/code.patch": "65c194af76627d5e83c258858acfc72b2b2d1aeaa62f0a58827117c4deb064e9", - "docs/training/jet-v3/comparison.json": "73590f5fe80066c72e3114ea763187f07137cb7596d9e4f8e41822436de80f97", - "docs/training/jet-v3/completion-audit.json": "1574b626af8529af0ac37b7fc3c97d0aeb63dfe16f33b534bdc78e16e612fa0d", - "docs/training/jet-v3/data-audit.json": "290341d86a29d6f82636aed7f2c821c128d80927b36611ff4ea53471bd185482", - "docs/training/jet-v3/data-provenance.json": "c99e9913a37982a26146a7b358ad326bff81edb80397fa96e2dcc358777632b2", - "docs/training/jet-v3/diagnostic-baseline.json": "8a18879468d58287a94cafa708898bef06c83c60a84b0333d50e7b9f0fff43f5", - "docs/training/jet-v3/diagnostic-v3.json": "2b6077a5c9b32ff3c0751070a0328dd22612347062a20c7d87f6503c29fe8382", - "docs/training/jet-v3/environment-freeze.txt": "f4cea0cd658cf3b785572d6a9112efd860715c5f2772fcf6ae38fa89be95c296", - "docs/training/jet-v3/esci-original-split-audit.json": "501e9853055b8e542b66c61a7ec1058e97146a445956c2258933e16de72728b7", - "docs/training/jet-v3/metrics.jsonl": "a612b54a15f4dfa88440fdd4b02dc514623da0387f598111c0b4fd4fd57fd00c", - "docs/training/jet-v3/run-manifest.json": "9430c2df0b03d734030517a83ddb47d0ad449c4e336d94859c0ad9002b3fbddf", - "docs/training/jet-v3/summary.md": "3dd7e35f57fdacc68c791c164d400b36575ca117bbf148c76bf5354454e80bba", - "docs/training/jet-v3/toolret-overlap-audit.json": "c0d27f351efd30dafeebacab33bf1d5c2f600cad437ffd6347b4b3ddd5365dd0", - "docs/training/jet-v3/training_config.json": "14d02b47b984219fcbbb5e899e3d60b7ccb595c7a8fe8880bd9c784c551ef79c", - "docs/training/jet-v3/v3-score_eval.json": "7e960ce8ea6ba1474e579c35d25a8a82ebcd4b905ef00c0ffe85bfe8b3950093", - "docs/training/jet-v3/v3-test.json": "901c6de4235b0be1b6357354e49c85613f590b4366031ac135689bea5c2eb40a", - "docs/training/jet-v3/v3-test_v3_new.json": "0307ef772eb908fd9f47550afb63fce8b7a7e286ef02ac615ff3f9b894aea508", - "docs/training/jet-v3.md": "83873ef83a901470193661296c4ff89bf10d3aea02f3166f10c03fe7226ab3a4", - "docs/training/jet-v4-release.md": "f6ca023759f9770e5e922bc2e107da21f6cbaf7c15c64219dc9e6f4229617e27", - "docs/training/jet-v5/aborted-attempt.json": "7a2f6845cae97283c0305c807c92fa77d0b04ba295305f092a79eceb51bd309b", - "docs/training/jet-v5/baseline-expanded.json": "58bc900d25701e3d2e3e0cd38c88db4005e03dbb5696a1cd3899f97f98bf025d", - "docs/training/jet-v5/baseline-test_v5_new.json": "56317c5792f40de718d708a2e99385cd16f78eb3821d9e8be337f158d2324dc7", - "docs/training/jet-v5/code/scripts/audit_v5.py": "4596800e8b0629222ee5c780f2630e27c8218f78b0ca31caaa49cf905694fd6e", - "docs/training/jet-v5/code/scripts/build_v5.py": "5f915e061c536400fe79b42d5c0980445e160ab036ec5a5132ddf809cc26f1c9", - "docs/training/jet-v5/code/scripts/download_v5_sources.py": "58722a9a442f72693f83f23808247324c76f20fd0e3ba1a2a9ffc4d845142b94", - "docs/training/jet-v5/code/scripts/run_v5_cuda.sh": "a6068fe02aa1b667315ef9ed21595b39fa6193acaec678727a035d0f8f862787", - "docs/training/jet-v5/code/src/data/decision_training.py": "785b8407f93c9ba952f98d29f987be73c853707f4f3211ccd6a4865c4cee3061", - "docs/training/jet-v5/code/src/evaluate.py": "25dcaf7bcf9ddcfdd8349eeea047ef7cc33388cf74f32a65c8a1931b4f40f72c", - "docs/training/jet-v5/code/src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "docs/training/jet-v5/code/src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "docs/training/jet-v5/code/src/train.py": "040ef12d706db397d6031a745faf9fc922497e6eb59f0d45b0ac18cd13b8cbfd", - "docs/training/jet-v5/code-selected.json": "86b97361ad2574c4ab8f0e91763238c451c2a57054a7a38359c341c2192c84b5", - "docs/training/jet-v5/code-single.json": "f5247bd18c79bd345d2108859be9287a66464d0bc663073570f77fabf871b177", - "docs/training/jet-v5/completion-audit.json": "fb1c4049406e03eaca83d75af4516b277e6f64dec7dfcd8da751979080aa9775", - "docs/training/jet-v5/data-audit.json": "2723283ba752c962229a7585119f581b3581e2f35bf41f82b5ecaf09dd3416ad", - "docs/training/jet-v5/esci-selected.json": "b559a4bd93dae53375f7537075b61b4377ac616731040fb45d08f29459847ff1", - "docs/training/jet-v5/esci-single.json": "07782a9445a316e4759f4bd9d12bb3eac537ddbb7f72e8e1232ada1c2ac63e96", - "docs/training/jet-v5/evaluation-code/scripts/evaluate_v5_order.py": "c4436a1994a27473b04c9aae131ffb5bdb3ed7303a21c7507ce39dae6f5cf70c", - "docs/training/jet-v5/evaluation-code/scripts/finalize_v5.py": "20f841219ff97f5c02068a86867677f1ad36a9df12d261c13604408ebd090600", - "docs/training/jet-v5/evaluation-code/scripts/run_v5_extra.sh": "9a6f9471fdc566cce479d9ce6eb257aeefa830ab06b06c19fededc86d69f13b6", - "docs/training/jet-v5/evaluation-code/scripts/summarize_v5.py": "48834200e9c6d2874e6b12982380167cebd9faf2a4b523014f816c09ec0f85ca", - "docs/training/jet-v5/evaluation-code/src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "docs/training/jet-v5/evaluation-code/src/decision_index_ensemble.py": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1", - "docs/training/jet-v5/evaluation-code/src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "docs/training/jet-v5/evaluation-code/src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "docs/training/jet-v5/evaluation-code/src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "docs/training/jet-v5/evaluation-code-manifest.json": "7e48db994a7a1ddbd91c93e8ec921a39c609387f0bd0732e9f3f3ec3780328c1", - "docs/training/jet-v5/expanded-capacity.json": "d29b476f0c1befdc7a301cb780d03ac675a8c913300455340f91cf216de54f15", - "docs/training/jet-v5/expanded-diagnostic-provenance.json": "6bc9f08379d59984a5a23cdf782ce3b983399874af9a792a35569280cf22c9b6", - "docs/training/jet-v5/final-code/scripts/audit_v5.py": "4596800e8b0629222ee5c780f2630e27c8218f78b0ca31caaa49cf905694fd6e", - "docs/training/jet-v5/final-code/scripts/build_v5.py": "5f915e061c536400fe79b42d5c0980445e160ab036ec5a5132ddf809cc26f1c9", - "docs/training/jet-v5/final-code/scripts/download_v5_sources.py": "58722a9a442f72693f83f23808247324c76f20fd0e3ba1a2a9ffc4d845142b94", - "docs/training/jet-v5/final-code/scripts/evaluate_v5_order.py": "c4436a1994a27473b04c9aae131ffb5bdb3ed7303a21c7507ce39dae6f5cf70c", - "docs/training/jet-v5/final-code/scripts/filter_v5_retention.py": "d4ae571e0ecfd8090353139ed925071a27a4ed3d7f27f6d1c7635687e8a19b6a", - "docs/training/jet-v5/final-code/scripts/finalize_v5.py": "20f841219ff97f5c02068a86867677f1ad36a9df12d261c13604408ebd090600", - "docs/training/jet-v5/final-code/scripts/rebuild_v5_diagnostic.py": "5c08c56a09a811195d1caaf916987d5a0e20c6deb4cbcb69805f4aaaeb4746cc", - "docs/training/jet-v5/final-code/scripts/run_v5_cuda.sh": "c79834e70e104dd978daa8005d60d0ed5641ca44893708cd5760566e870eacc8", - "docs/training/jet-v5/final-code/scripts/run_v5_extra.sh": "9a6f9471fdc566cce479d9ce6eb257aeefa830ab06b06c19fededc86d69f13b6", - "docs/training/jet-v5/final-code/scripts/summarize_v5.py": "48834200e9c6d2874e6b12982380167cebd9faf2a4b523014f816c09ec0f85ca", - "docs/training/jet-v5/final-code/src/decision_index_engine.py": "f4b03bb95fc34afd0debf8ec5210a68965fb69bd510284ba8f26f8f2f75179ba", - "docs/training/jet-v5/final-code/src/decision_index_ensemble.py": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1", - "docs/training/jet-v5/final-code/src/evaluate.py": "25dcaf7bcf9ddcfdd8349eeea047ef7cc33388cf74f32a65c8a1931b4f40f72c", - "docs/training/jet-v5/final-code/src/format.py": "f9b86dd757dbebc1fa48effbf48b7ba6861dcd6def5a025cfa937866eed02828", - "docs/training/jet-v5/final-code/src/inference.py": "38443ddac02367f5857557ebae72aaf3e4d840f407573fd0dcb3ba037a5bc19d", - "docs/training/jet-v5/final-code/src/model.py": "bdc03a5a63002f236e17fb81f681738cb238a3e4a73ca575b99d580c1f3dec03", - "docs/training/jet-v5/final-code/src/train.py": "040ef12d706db397d6031a745faf9fc922497e6eb59f0d45b0ac18cd13b8cbfd", - "docs/training/jet-v5/final-code/tests/test_decision_ensemble.py": "6bdf9dce1d41d2d6998fbac3cc2df362d57eb3e0748c00cf5023a0c9b3c16b91", - "docs/training/jet-v5/inference-selection.json": "b79337dfd463890e35f3c353f33c71468b09fef3623d0d3263079f5e51532479", - "docs/training/jet-v5/leaderboard-provenance.json": "772b9d7cdae5dd70ebc14a7fe21c12b3bf491973abc88504b0dc4041bec07ac1", - "docs/training/jet-v5/leaderboard-snapshot.json": "dd672173d9819d0aee7f2fcc56a2004aca5efcb05d9e7c89b1728df6426fc8dc", - "docs/training/jet-v5/protocol.md": "d569d35136e0eb6a2092901589d54c46411c270dc9b034dc497fe5759e738373", - "docs/training/jet-v5/r2-code/scripts/evaluate_v5_order.py": "c4436a1994a27473b04c9aae131ffb5bdb3ed7303a21c7507ce39dae6f5cf70c", - "docs/training/jet-v5/r2-code/scripts/filter_v5_retention.py": "d4ae571e0ecfd8090353139ed925071a27a4ed3d7f27f6d1c7635687e8a19b6a", - "docs/training/jet-v5/r2-code/scripts/run_v5_cuda.sh": "c79834e70e104dd978daa8005d60d0ed5641ca44893708cd5760566e870eacc8", - "docs/training/jet-v5/r2-code/scripts/run_v5_extra.sh": "adc12398a30e3994d7f71d60e84ecfa04f7cee2871ab48ccf600ddefe64d5c14", - "docs/training/jet-v5/r2-code/scripts/summarize_v5.py": "281f3e722b61e03e776c0cdadce10f3864b95b06a3da160bf77c605b7d1bc9ae", - "docs/training/jet-v5/r2-code/src/decision_index_ensemble.py": "f47a2da7845684c6a7edb67de503eea7ea3096d6d0e8bf78aa3054cbadc04dc1", - "docs/training/jet-v5/results.json": "eee01d9fe51bc3ad487e88bbfd953762cd00c1173a5afe20628f405efa55a5c7", - "docs/training/jet-v5/retention-audit.json": "48ce47beead604dc496afc334915778db45938420fa18dcbf0c73d8488adf00e", - "docs/training/jet-v5/sarcasm-selected.json": "2f080836a7b8fe9841aadd800bf56eb929dccace4a5e96d71568f7d2b12b2978", - "docs/training/jet-v5/sarcasm-single.json": "90756b79c5b6e36f16982202ea02cd1b6678401b04bbdf3b7adf1624b0537083", - "docs/training/jet-v5/summary.md": "ba6b5becb373c16ce382b54d10a2154f21a8af15243e95b7a604d566cfe48ac3", - "docs/training/jet-v5/v4-diagnostic.json": "0caf0be35f6b71a3118a81417cbf159e490af0b843f97516da15d8ecbd52e597", - "docs/training/jet-v5/v4-expanded.json": "3c6bce95781d783954ff576957d23a1d48d0435fb20cdfea7fc54ba78125b003", - "docs/training/jet-v5/v4-test.json": "57c30fc33fa972931097e337cc2a30b25871b0befb70735379ec2b2aa542cacb", - "docs/training/jet-v5/v4-test_v4_new.json": "baba6050e13db92c08902437160b6e64c23522e69ed61bb3edf33e46879ba682", - "docs/training/jet-v5/v4-test_v4_tools.json": "78d09bec52d78a09ae38bae653af4e157c11143fc33bc4c79c235a07d4219a20", - "docs/training/jet-v5/v4-test_v5_new.json": "19c9db0cd09bc7649c13e756257c85b7f85e0e2c1a59e80118fa18c3c26ba15d", - "docs/training/jet-v5/v5-diagnostic.json": "904bbf3cb13187ae2465b4cbf4d8d5aab07ac3eff89110a590c1517e780915b9", - "docs/training/jet-v5/v5-expanded.json": "a631e57744d70847651900900bb47e91b076f514ef45f21f09f035487af7cbb1", - "docs/training/jet-v5/v5-test.json": "bfbd780635898c5b32389b9c2c37c53a826fe1f1a3478a4d5cd37c96e01c1e11", - "docs/training/jet-v5/v5-test_v4_new.json": "e55d24c078a8ef3aeeb2984496105b82c7914442e5a0cdfd4a3d4835b68ddfd5", - "docs/training/jet-v5/v5-test_v4_tools.json": "c204aa5ee8baf8e95ac1ef20de30d62fb587e27db1abe2de3e3203238a12aa92", - "docs/training/jet-v5/v5-test_v5_new.json": "d1d613ffad1b42df342f264948216669e98ccd21d2ce3571f7ea98a1bf219296", - "docs/training/jet-v5/v5-two-order-expanded.json": "5601c6a03197876cf9a4524c36aac01f00e88566948260a40355bbd795a30814", - "docs/training/jet-v5/v5-two-order-original.json": "92c2e1ca70e58076b14c8e96f84cf01de296fb2eeec79dd0990375e6a6a6ca9c", - "docs/training/jet-v5/verification.json": "c1258a68e10b13c0a998b1b21bcf277a60cb5043ac723a5eecf3232970efee94", - "docs/training/jet-v5-release.md": "d7d3e7beb821e90a236d63d781e759aafd93c48da57863251cccf0184d8e18b6", "export-provenance.json": "a3f2a53d03816fe3e57ebcbdf850b84f98b52c3a135ad5cd581d96276eacc0d2", "export-validation.txt": "154c0212c4d4f64500ec13ebdffd7f405180a0ab3be6d049127144b4bb162df0", "golden.json": "98e895994a3544548a09ad72f68e599af908628ac8093ecd2eed58a679b69319", @@ -208,5 +21,10 @@ "onnx/model_q8.onnx": "7ee7a5bdda762f735c6aa9257d63f758afef546afbfc1a701ba698fc89ef67f0", "tokenizer.json": "be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506", "tokenizer_config.json": "d93ac1a2c7adb9ad022f354d822f1f0f57e32e62ef2989e902782d6ab896acfe" + }, + "training_archive": { + "revision": "ee7d33ecae2b8bee30891657dd57c92d34e0ce53", + "tag": "training-archive-20260924", + "removed_from_main": 187 } }