Add benchmark, calibration and robustness charts with result-first model cards

.gitattributes

@@ -34,3 +34,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+figures/benchmark.png filter=lfs diff=lfs merge=lfs -text
+figures/calibration.png filter=lfs diff=lfs merge=lfs -text
+figures/robustness.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -14,22 +14,108 @@ tags:
 - jev-style
 - single-prefill
 ---
-# Jev-Style-Qwen3.5-2B-Decision v2 · MLX BF16
+# Jev-Style-Qwen3.5-2B-Decision v2 (MLX BF16)
 
-A compact decision model for classification, routing and typed choices. Give it a state, a question and a list of options; receive a selected option and calibrated probabilities in one prefill.
+A **Jev-style decision model** for classification, routing and typed choices. Give it a state, a question and a list of options; one prefill returns a selected option **with calibrated probabilities**.
 
-[HF BF16](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-v2) | [GGUF Q8_0](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-v2-GGUF) | [MLX BF16](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-v2-MLX-bf16)
+| Build | Weight size | Inference |
+|---|---:|---|
+| [HF BF16](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-v2) | 3.76 GB | Transformers + decision client |
+| [GGUF Q8_0](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-v2-GGUF) | 2.01 GB | llama.cpp + decision client |
+| **MLX BF16 · this repository** | 3.76 GB | Apple Silicon + native MLX client |
 
-## This release: native MLX BF16
+**Download this build:** [model.safetensors](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-v2-MLX-bf16/resolve/main/model.safetensors?download=true). The repository also includes its calibration, inference client and evaluation records.
 
-- Native mlx-lm weight names and tensor layouts for Apple Silicon; approximately **3.76 GB** of weights.
-- **99.6% choice agreement** with merged CUDA BF16 on the frozen 500-decision deployment subset.
-- Independent calibration on 3,100 records, automatically applied by the client.
-- BF16 model weights with FP32 normalization gains and FP32 declared-option projection in the decision client.
+## Results
 
-The standard `mlx_lm.load()` loading path is verified. For the evaluated calibrated decision interface, use `jev_mlx_client.py`, which preserves normalization arithmetic and applies calibration once. Packaging verification on all 500 deployment cases is supplied in `evaluation/packaging_verification.json`.
+**81.20% macro accuracy on the fixed English reference panel**, compared with 76.68% for v1 and 75.09% for English Laya. The results below use the CUDA reference structure: 11 real-label task groups, 3,277 decisions, equal task weights, and the same 3,100-record calibration split. Results for the released deployment formats appear further below.
 
-### Download and run on Apple Silicon
+| Metric | Jev-Style v1 | English Laya | **Jev-Style v2** |
+|---|---:|---:|---:|
+| Accuracy ↑ | 76.68% | 75.09% | **81.20%** |
+| Macro-F1 ↑ | 75.42% | 73.45% | **79.78%** |
+| Negative log-likelihood ↓ | 0.5752 | 0.6318 | **0.5154** |
+| Brier score ↓ | 0.3290 | 0.3482 | **0.2787** |
+
+![Accuracy, Macro-F1, NLL and Brier comparison on the frozen English panel](figures/benchmark.png)
+
+- **Higher accuracy:** +4.53 percentage points over v1 and +6.12 over English Laya; paired 95% intervals are [+3.58, +5.52] and [+4.64, +7.52] points, respectively, within this fixed panel.
+- **Broader task coverage:** accuracy point estimates ahead of English Laya in **9 of 12 task groups**, including the separately scored teacher-reference typed-decisions group.
+- **Better probability quality against English Laya:** **18.4% lower NLL**, **20.0% lower Brier score**, and **26.4% lower task-macro ECE**.
+- **Efficient adaptation:** **36.9 minutes of main training on one H100 80GB**, using rank-32 LoRA on a 2B-class text backbone.
+
+## Calibration
+
+The reliability diagram plots the v2 model's stated confidence against observed correctness. Every real-label evaluation decision is included; the histogram shows how many predictions fall in each confidence bin. Error bars show Wilson 95% intervals. The accompanying ECE comparison averages per-task calibration errors.
+
+![Reliability diagram with confidence counts and task-macro ECE comparison](figures/calibration.png)
+
+Temperature is fitted on the calibration split. HF and MLX clients apply the supplied calibration automatically; the calibrated GGUF file incorporates it in the final normalization tensor.
+
+## Robustness
+
+**Option-order flip rate is halved relative to English Laya**, with **80.00% accuracy after permutation** on the same 400 Choice/Bool decisions. Semantic options are mapped back to their original identities before scoring.
+
+| Option-permutation test | Jev-Style v1 | English Laya | **Jev-Style v2** |
+|---|---:|---:|---:|
+| Decision flip rate ↓ | 9.25% | 12.00% | **6.00%** |
+| Accuracy after permutation ↑ | 66.75% | 67.00% | **80.00%** |
+
+![Option-order stability and accuracy after permutation](figures/robustness.png)
+
+On a separate **200-pair programmatic threshold-policy test**, both decisions in a counterfactual pair are correct in **71.50%** of pairs for v2, compared with 63.00% for v1. This test measures that specific rule family.
+
+## Task-level results
+
+<details>
+<summary><strong>Per-task accuracy: all 11 real-label tasks and the separate typed-decision group</strong></summary>
+
+| Real-label task | Examples | Jev-Style v1 | English Laya | Jev-Style v2 |
+|---|---:|---:|---:|---:|
+| AG News | 300 | 87.67% | 89.00% | 88.00% |
+| ANLI | 300 | 48.00% | 49.67% | 48.67% |
+| BoolQ | 300 | 82.67% | 75.67% | 81.67% |
+| Emotion | 300 | 58.33% | 60.33% | 85.33% |
+| Enron spam | 300 | 77.33% | 96.33% | 97.67% |
+| HANS | 300 | 68.00% | 75.00% | 68.00% |
+| IMDb | 300 | 96.67% | 93.67% | 96.33% |
+| MNLI | 300 | 86.67% | 85.00% | 88.00% |
+| RTE | 277 | 84.48% | 77.98% | 85.92% |
+| SST-2 | 300 | 92.67% | 91.67% | 93.00% |
+| SST-5 | 300 | 61.00% | 31.67% | 60.67% |
+
+The separate typed-decisions group contains 2,000 teacher-reference decisions from 400 states. Teacher agreement is 53.35% for v1, 37.55% for English Laya and **73.45% for v2** under the fixed primary interface. This group is excluded from the real-label macro. The comparison here uses the English Laya checkpoint; specialist-checkpoint and rendering sensitivity results are provided in [baseline_sensitivity.json](evaluation/baseline_sensitivity.json).
+
+</details>
+
+
+## Deployment validation
+
+| Released format | Weight size | Validated result | Evaluation set |
+|---|---:|---|---|
+| HF BF16 | 3.76 GB | **81.27%** real-label macro accuracy | Full 3,277 real-label decisions |
+| Native MLX BF16 | 3.76 GB | **99.6%** choice agreement with CUDA BF16 | Frozen 500-decision deployment subset |
+| Calibrated GGUF Q8_0 | 2.01 GB | **99.2%** choice agreement with CUDA BF16 | Same 500-decision deployment subset |
+
+Each deployment format has its own validation record. Native MLX packaging reproduces the verified MLX client's logits exactly on all 500 deployment cases. The Q8_0 model is approximately **46.7% smaller** than the BF16 GGUF export.
+
+On the same 500-case deployment subset, real-label task-macro accuracy is **79.10%** for CUDA BF16, **79.04%** for MLX BF16 and **78.69%** for Q8_0. Full-panel reference results and deployment-subset results use their respective denominators.
+
+<details>
+<summary><strong>Evaluation data and downloadable vector charts</strong></summary>
+
+- [Reference metrics and paired intervals](evaluation/reference_comparison.json)
+- [Deployment validation](evaluation/deployment.json)
+- [Baseline sensitivity results](evaluation/baseline_sensitivity.json)
+- [Data sources and split manifest](evaluation/data_manifest.json)
+- [Chart data, confidence bins and sample counts](evaluation/chart_data.json)
+- Vector charts: [benchmark](figures/benchmark.svg), [calibration](figures/calibration.svg), [robustness](figures/robustness.svg)
+
+The benchmark figures describe the fixed CUDA reference comparison. Reliability pools all real-label examples into confidence bins; task-macro ECE is the mean of 11 separate task ECE values. These are distinct aggregations. The 9/12 figure counts task-level point estimates. Individual prediction probabilities, task summaries, test protocols and calibration records were retained when drawing these charts.
+
+</details>
+
+## Quick start
 
 ```bash
 python -m pip install -U huggingface_hub
@@ -55,33 +141,8 @@ print(result["choice"])
 print(result["probabilities"])
 ```
 
-Validated with `mlx==0.32.2` and `mlx-lm==0.31.3`. The companion `calibration.json` is loaded automatically. The CUDA reference benchmark and MLX deployment check are reported separately below and in the evaluation files.
-
-## Highlights
-
-- **81.27% macro accuracy** for the released merged BF16 model across 11 real-label task groups (3,277 decisions).
-- **9 of 12 task-group accuracy point estimates ahead of English Laya** in the fixed CUDA reference comparison.
-- **+4.53 percentage points over Jev-Style v1** and **+6.12 points over English Laya** in reference macro accuracy on the same evaluation panel.
-- **18.4% lower NLL and 20.0% lower Brier score** than English Laya in the reference comparison.
-- **6.0% option-permutation flip rate**, compared with 9.25% for v1 and 12.0% for English Laya, on 400 Choice/Bool decisions.
-- **One H100 80GB, 36.9 minutes of main training**, with a 2B-class text backbone and rank-32 LoRA.
-
-The comparison uses the frozen English task panel and the CUDA reference structure. Deployment variants are measured separately below. The 9/12 count describes task-level point estimates.
-
-## Reference evaluation
-
-Real-label results are macro-averaged with equal task weights. All three models use the same calibration records and global temperature-fitting objective.
-
-| Metric | Jev-Style v1 | English Laya | Jev-Style v2 reference |
-|---|---:|---:|---:|
-| Accuracy ↑ | 76.68% | 75.09% | **81.20%** |
-| Macro-F1 ↑ | 75.42% | 73.45% | **79.78%** |
-| NLL ↓ | 0.5752 | 0.6318 | **0.5154** |
-| Brier ↓ | 0.3290 | 0.3482 | **0.2787** |
-
-Accuracy improvements have paired 95% intervals of **+3.58 to +5.52 points vs v1** and **+4.64 to +7.52 points vs English Laya** within this frozen task panel.
+Validated with `mlx==0.32.2` and `mlx-lm==0.31.3`. The companion `calibration.json` is loaded automatically. The reference benchmark and MLX deployment check are reported separately above and in the evaluation files.
 
-The panel covers sentiment, news, natural-language inference, question answering, emotion and email classification. The twelfth task group contains 2,000 teacher-reference typed decisions from 400 states and is reported separately from the real-label macro. Per-task results, all probability metrics, robustness measurements and baseline sensitivity results are supplied in the evaluation files.
 
 ## Decision interface
 

SHA256SUMS.json

@@ -4,8 +4,8 @@
     "sha256": "50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32"
   },
   "README.md": {
-    "bytes": 6850,
-    "sha256": "b4ead81dafffb3eaf1dca8c99c1de71097c0524a228449abeefa45d75199fa55"
+    "bytes": 10469,
+    "sha256": "94cc76f41bf25f0e1958ce7293e1e9c22cf34e7a9195b33f080200d983ec6113"
   },
   "calibration.json": {
     "bytes": 362,
@@ -78,5 +78,33 @@
   "tokenizer_config.json": {
     "bytes": 1156,
     "sha256": "303f948bfd44621a24f1d90b6330d8280076d08b60152f5851d9e9921c0b570e"
+  },
+  "evaluation/chart_data.json": {
+    "bytes": 4588,
+    "sha256": "0db9855803eb514792a906ff04b1d9ccdcba81babeedcffdd9ae34131f8b04f1"
+  },
+  "figures/calibration.svg": {
+    "bytes": 30706,
+    "sha256": "87b305248aef4580849edd523a29c8d7aba0774a879fe737f23c4fd460a861d0"
+  },
+  "figures/robustness.png": {
+    "bytes": 144638,
+    "sha256": "9a2032db7052f934d54db70eb490bea4eb160d6fb9c3644d07f5ffa11a91b383"
+  },
+  "figures/benchmark.png": {
+    "bytes": 216104,
+    "sha256": "8bc8eecd93cc7ff0727aaef15c644be664fd64abcee7d208e79e27218cb29051"
+  },
+  "figures/benchmark.svg": {
+    "bytes": 23602,
+    "sha256": "02c8a57ff3150f88f78cabdd99ce9aac5d9822a858929761748d1d5e679c9969"
+  },
+  "figures/robustness.svg": {
+    "bytes": 13858,
+    "sha256": "bccd285dd10dd98cac069928a127d5767cb61704e18d8ac3fa656703e81d572f"
+  },
+  "figures/calibration.png": {
+    "bytes": 219405,
+    "sha256": "0555263334858774466ffde686343d6fae3b4ca4b5572e6e41c6044291883b70"
   }
 }

evaluation/chart_data.json

@@ -0,0 +1,183 @@
+{
+  "source": "evaluation/reference_comparison.json",
+  "source_comparison_sha256": "36f56a490fc67e774cbb25b38d4dc9b5e3a42e1f65fcd748af092e127a98eff8",
+  "real_label_decisions": 3277,
+  "overview": {
+    "v1": {
+      "accuracy": 0.7667968493600263,
+      "macro_f1": 0.7542251333163978,
+      "nll": 0.575152117780084,
+      "brier": 0.3289652416178414,
+      "ece": 0.08512955323369954
+    },
+    "laya": {
+      "accuracy": 0.750889399409255,
+      "macro_f1": 0.7344520495576862,
+      "nll": 0.6318461242308782,
+      "brier": 0.3481931154806123,
+      "ece": 0.12362140723612026
+    },
+    "v2": {
+      "accuracy": 0.8120490099551472,
+      "macro_f1": 0.797772564877388,
+      "nll": 0.5153573778298738,
+      "brier": 0.27870374107073076,
+      "ece": 0.09100670643474301
+    }
+  },
+  "reliability": {
+    "model": "v2 CUDA reference",
+    "temperature": 1.0423505400296817,
+    "binning": "15 equal-width confidence bins on [0,1]",
+    "empty_bins": "omitted",
+    "uncertainty": "Wilson 95% intervals within each confidence bin",
+    "aggregation": "pooled over real-label examples",
+    "pooled_ece": 0.07282541921307938,
+    "bins": [
+      {
+        "lo": 0.3333333333333333,
+        "hi": 0.4,
+        "n": 11,
+        "mean_confidence": 0.38078092576157074,
+        "accuracy": 0.45454545454545453,
+        "wilson95": [
+          0.21271271622459764,
+          0.719908462590678
+        ]
+      },
+      {
+        "lo": 0.4,
+        "hi": 0.4666666666666667,
+        "n": 31,
+        "mean_confidence": 0.44342106018763106,
+        "accuracy": 0.3870967741935484,
+        "wilson95": [
+          0.23733101420380254,
+          0.5617589138033927
+        ]
+      },
+      {
+        "lo": 0.4666666666666667,
+        "hi": 0.5333333333333333,
+        "n": 130,
+        "mean_confidence": 0.5071245091267953,
+        "accuracy": 0.4,
+        "wilson95": [
+          0.3198243104508396,
+          0.4859160017880467
+        ]
+      },
+      {
+        "lo": 0.5333333333333333,
+        "hi": 0.6,
+        "n": 152,
+        "mean_confidence": 0.5683921263707039,
+        "accuracy": 0.5197368421052632,
+        "wilson95": [
+          0.4408087537451127,
+          0.5976919125435891
+        ]
+      },
+      {
+        "lo": 0.6,
+        "hi": 0.6666666666666666,
+        "n": 158,
+        "mean_confidence": 0.6355755895198861,
+        "accuracy": 0.5379746835443038,
+        "wilson95": [
+          0.46025816470988046,
+          0.6138884729158649
+        ]
+      },
+      {
+        "lo": 0.6666666666666666,
+        "hi": 0.7333333333333333,
+        "n": 145,
+        "mean_confidence": 0.7017602376997318,
+        "accuracy": 0.6551724137931034,
+        "wilson95": [
+          0.5747027826751803,
+          0.7276323352183527
+        ]
+      },
+      {
+        "lo": 0.7333333333333333,
+        "hi": 0.8,
+        "n": 153,
+        "mean_confidence": 0.7684716912664443,
+        "accuracy": 0.7647058823529411,
+        "wilson95": [
+          0.6915216339127415,
+          0.8249234476941671
+        ]
+      },
+      {
+        "lo": 0.8,
+        "hi": 0.8666666666666667,
+        "n": 166,
+        "mean_confidence": 0.8361544180666837,
+        "accuracy": 0.7048192771084337,
+        "wilson95": [
+          0.6314328145486011,
+          0.7689405717386028
+        ]
+      },
+      {
+        "lo": 0.8666666666666667,
+        "hi": 0.9333333333333333,
+        "n": 333,
+        "mean_confidence": 0.9044363365371433,
+        "accuracy": 0.7627627627627628,
+        "wilson95": [
+          0.7142396166140905,
+          0.8052926304337862
+        ]
+      },
+      {
+        "lo": 0.9333333333333333,
+        "hi": 1.0,
+        "n": 1998,
+        "mean_confidence": 0.984503687108951,
+        "accuracy": 0.9229229229229229,
+        "wilson95": [
+          0.9103995454324445,
+          0.9338231538993982
+        ]
+      }
+    ]
+  },
+  "permutation": {
+    "v1": {
+      "n": 400,
+      "semantic_flip_rate": 0.0925,
+      "permuted_accuracy": 0.6675
+    },
+    "laya": {
+      "n": 400,
+      "semantic_flip_rate": 0.12,
+      "permuted_accuracy": 0.67
+    },
+    "v2": {
+      "n": 400,
+      "semantic_flip_rate": 0.06,
+      "permuted_accuracy": 0.8
+    }
+  },
+  "counterfactual": {
+    "v1": {
+      "pairs": 200,
+      "both_correct": 0.63,
+      "scope": "programmatic threshold-policy shift only"
+    },
+    "laya": {
+      "pairs": 200,
+      "both_correct": 0.075,
+      "scope": "programmatic threshold-policy shift only"
+    },
+    "v2": {
+      "pairs": 200,
+      "both_correct": 0.715,
+      "scope": "programmatic threshold-policy shift only"
+    }
+  }
+}