Temperatures: apply the hard fit to score questions (0.8329), keep train for choice and noul

The train fit's score temperature (1.2162) was fitted to the smoothed ordinal target and made score probabilities too soft. Both calibration-split fits stay in temperatures.json under their names; the applied set is now choice 1.1863, noul 1.0903, score 0.8329, the rule the 27B has applied since 2026-09-26. Re-tempering the stored eval logits: question-weighted ECE over the 20 sets 0.080 to 0.071, NLL 0.589 to 0.580, no pick changes; both HelpSteer2 sets get worse. Card and eval/RESULTS.md ECE and Decision Score columns updated to match.

README.md

@@ -49,7 +49,7 @@ model-index:
       type: jevals-helpsteer2
     metrics:
     - type: decision_score_jevals
-      value: 10.4
+      value: 10.5
   - task:
       type: text-classification
       name: typed decisions (choice, noul, score)
@@ -69,7 +69,7 @@ Jebadiah (Jeb for short) is Frontier Infra's open System One style decision mode
 
 <img src="jeb-banner.png" alt="They call me Jeb. He does not talk much. He just decides." width="100%">
 
-This repository holds full bf16 weights: the v2 LoRA merged into `Qwen/Qwen3.5-9B` (revision `c2022362`, the chat checkpoint, thinking off). What changed from v1 is one thing, the base: v1 was trained on `Qwen/Qwen3.5-9B-Base`. The data, LoRA, objective and temperature fit are v1's.
+This repository holds full bf16 weights: the v2 LoRA merged into `Qwen/Qwen3.5-9B` (revision `c2022362`, the chat checkpoint, thinking off). What changed from v1 is one thing, the base: v1 was trained on `Qwen/Qwen3.5-9B-Base`. The data, LoRA, objective and temperature fits are v1's; since 2026-09-29 the score temperature applied is the fit against the label (see How it decides).
 
 Made in Texas.
 
@@ -99,9 +99,9 @@ Measured by us with AINode's bench, one logit read per question, the same render
 
 The 27B is the same recipe on `Qwen/Qwen3.8-27B`; see [its card](https://huggingface.co/frontier-infra/jebadiah-27b). For scale, Bespoke's published table puts Nimble-9B at 74.8 and Jev at 76.0 on the same 13 Nimble public subsets, with their scorer.
 
-Where v2 is not better than v1: the Jevals HelpSteer2 Decision Score is 10.4 against 11.9, and its top level flips over identical repeats on 2.3% of questions against 0.3%. The Nimble public macro is flat (77.0): MultiNLI and PAWS give back what PubMedQA and HelpSteer2 gain. ECE on the Nimble 324 set rises from 0.063 to 0.085. The headline gain is 0.6 points, most of it from the Nimble 324 set. HelpSteer2 and SummEval are not zero-shot for Jeb: the pool trains on their train split and unscored articles (no evaluation item overlaps), so those rows are held-out items of a seen rubric.
+Where v2 is not better than v1: the Jevals HelpSteer2 Decision Score is 10.5 against 11.9, and its top level flips over identical repeats on 2.3% of questions against 0.3%. The Nimble public macro is flat (77.0): MultiNLI and PAWS give back what PubMedQA and HelpSteer2 gain. ECE on the Nimble 324 set rises from 0.063 to 0.070. The headline gain is 0.6 points, most of it from the Nimble 324 set. HelpSteer2 and SummEval are not zero-shot for Jeb: the pool trains on their train split and unscored articles (no evaluation item overlaps), so those rows are held-out items of a seen rubric.
 
-Every calibrated number applies the per-type temperatures in `temperatures.json`. Decision Scores, ECE, repeat flips, the in-distribution sets, every Nimble subset and the merge check are in [`eval/RESULTS.md`](eval/RESULTS.md); the per-question records are beside it in `eval/`. The nonce robustness pass is pending for this model, so no robustness number is claimed.
+Every calibrated number applies the per-type temperatures in `temperatures.json`: choice 1.19 and noul 1.09 from the fit to the training target, score 0.83 from the fit to the label, all three on the held-out calibration split. The score temperature was refit on 2026-09-29 from 1.22 (fitted to the smoothed ordinal training target) to 0.83, the rule the 27B has used since 2026-09-26, with no change to any pick. It lowers ECE on typed-decisions from 0.156 to 0.121 and on Nimble 324 from 0.085 to 0.070, and raises it on Jevals HelpSteer2 from 0.039 to 0.082 and Nimble public HelpSteer2 from 0.073 to 0.098; the comparison is in [`eval/RESULTS.md`](eval/RESULTS.md#temperature-fits). Decision Scores, ECE, repeat flips, the in-distribution sets, every Nimble subset and the merge check are in [`eval/RESULTS.md`](eval/RESULTS.md); the per-question records are beside it in `eval/`. The nonce robustness pass is pending for this model, so no robustness number is claimed.
 
 ## Run it anywhere
 
@@ -169,7 +169,7 @@ These are independent builds. The agreement numbers under Other formats are for
 
 ## How it decides
 
-The prompt is AINode's own decide rendering (source commit `e5c08938`, hash in `prompt_contract.json`), through the chat template with thinking off. The option labels are single tokens; the answer is the distribution over those label tokens at the last prompt position, read in fp32 and temperature scaled per type (`temperatures.json`: choice 1.19, noul 1.09, score 1.22). Nothing is generated. Those are the `train` fit, which minimises NLL against the soft or ordinal target the model was taught; it softens, and it costs some ECE against hard labels on the calibration split. A sharper `hard` fit (0.79 / 0.67 / 0.83) is in the same file for a consumer who gates on the argmax. Thresholds belong to the caller: act on a high probability, confirm or escalate on a middle one, hand a low one to a person or a bigger model. The model never refuses.
+The prompt is AINode's own decide rendering (source commit `e5c08938`, hash in `prompt_contract.json`), through the chat template with thinking off. The option labels are single tokens; the answer is the distribution over those label tokens at the last prompt position, read in fp32 and temperature scaled per type (`temperatures.json`: choice 1.19, noul 1.09, score 0.83). Nothing is generated. Choice and noul use the `train` fit (NLL against the soft target the model was taught; it softens), score uses the `hard` fit (NLL against the label; it sharpens), because the ordinal score target is smoothed on purpose and a temperature fitted to it made score probabilities too soft. Both complete fits, `train` (1.19 / 1.09 / 1.22) and `hard` (0.79 / 0.67 / 0.83), are in the same file. Thresholds belong to the caller: act on a high probability, confirm or escalate on a middle one, hand a low one to a person or a bigger model. The model never refuses.
 
 ## Training
 
@@ -184,7 +184,7 @@ The prompt is AINode's own decide rendering (source commit `e5c08938`, hash in `
 - Single-hop judgments only. Split a chain of inference into hops.
 - A choice question is capped at 20 options on `/v1/systemone`; Banking77's 77 options were scored with an extended single-token alphabet for the benchmark only.
 - English data. Training cut states past 2,048 prompt tokens.
-- Calibration was fitted on the training distribution. Refit before trusting a threshold.
+- Calibration was fitted on the training distribution's calibration split, where HelpSteer2's labels are close to uniform. On traffic skewed like natural HelpSteer2 (over 70% of labels at 3 or 4) score answers are too confident: ECE on Jevals HelpSteer2 is 0.082. Refit on your own labelled traffic before trusting a threshold ([how](eval/RESULTS.md#temperature-fits)).
 
 ## Versioning and license
 

eval/RESULTS.md

@@ -16,11 +16,11 @@ The v1 columns are the published [v1](https://huggingface.co/frontier-infra/jeba
 |---|---|---:|---:|---:|---:|---:|---:|---:|
 | Jevals PubMedQA (300) | noul | 90.3 | 89.7 | 62.0 | 68.6 | 66.8 | 0.025 | 0.0% |
 | Jevals Banking77 (300, 77 options) | choice | 70.7 | 70.0 | 1.3 | 58.2 | 56.9 | 0.076 | 0.7% |
-| Jevals HelpSteer2 helpfulness (300, 5 levels) | score | 40.7 | 40.3 | 41.7 | 10.4 | 11.9 | 0.039 (raw 0.066) | 2.3% |
-| Nimble held-out eval (324) | mixed | 81.2 | 78.7 | 17.6 | 65.0 | 65.0 | 0.085 | 0.3% |
-| Kev transfer-v4 test (764) | mixed | 83.8 | 84.0 | 21.5 | 67.9 | 67.0 | 0.025 | 0.3% |
-| Kev decision-v7 test (1,440) | mixed | 81.2 | 80.8 | 20.3 | 71.4 | 70.6 | 0.050 | n/a |
-| typed-decisions test (2,000) | mixed | 78.3 | 78.9 | 15.3 | 56.5 | 56.4 | 0.156 | 0.1% |
+| Jevals HelpSteer2 helpfulness (300, 5 levels) | score | 40.7 | 40.3 | 41.7 | 10.5 | 11.9 | 0.082 (raw 0.066) | 2.3% |
+| Nimble held-out eval (324) | mixed | 81.2 | 78.7 | 17.6 | 65.0 | 65.0 | 0.070 (raw 0.059) | 0.3% |
+| Kev transfer-v4 test (764) | mixed | 83.8 | 84.0 | 21.5 | 67.9 | 67.0 | 0.027 | 0.3% |
+| Kev decision-v7 test (1,440) | mixed | 81.2 | 80.8 | 20.3 | 71.4 | 70.6 | 0.048 | n/a |
+| typed-decisions test (2,000) | mixed | 78.3 | 78.9 | 15.3 | 57.1 | 56.4 | 0.121 | 0.1% |
 
 Nimble's 13 public human-labelled subsets (3,880 questions), macro accuracy: **77.0** (v1: 77.0).
 For scale, Bespoke's published table puts Nimble-9B at 74.8 and Jev at 76.0 on the same subsets with their scorer.
@@ -30,18 +30,18 @@ For scale, Bespoke's published table puts Nimble-9B at 74.8 and Jev at 76.0 on t
 | aegis2 (250) | 78.4 | 79.2 | 38.0 | 35.8 |
 | boolq (300) | 86.0 | 85.7 | 54.6 | 57.7 |
 | civil_comments (300) | 86.0 | 86.7 | -7.4 | -4.8 |
-| helpsteer2 (249) | 43.4 | 41.0 | 13.3 | 15.2 |
+| helpsteer2 (249) | 43.4 | 41.0 | 12.3 | 15.2 |
 | massive-de-DE (350) | 84.6 | 84.9 | 74.3 | 73.3 |
 | massive-en-US (350) | 85.4 | 85.7 | 77.3 | 76.0 |
 | multinli (299) | 84.9 | 87.6 | 67.2 | 70.6 |
 | paws (250) | 83.6 | 85.6 | 54.0 | 57.2 |
 | pubmedqa (250) | 72.8 | 68.8 | 38.5 | 34.5 |
 | squad2 (299) | 79.9 | 80.3 | 42.9 | 43.6 |
-| summeval-consistency (144) | 87.5 | 87.5 | 42.6 | 36.0 |
-| summeval-relevance (240) | 52.5 | 51.7 | 23.4 | 25.9 |
+| summeval-consistency (144) | 87.5 | 87.5 | 49.4 | 36.0 |
+| summeval-relevance (240) | 52.5 | 51.7 | 23.9 | 25.9 |
 | vitaminc-dev (599) | 75.6 | 76.3 | 41.8 | 39.0 |
 
-Where v2 is not better. Jevals HelpSteer2 Decision Score is 10.4 against v1's 11.9, and its top level flips over identical repeats on 2.3% of questions against 0.3%. The Nimble public macro is flat (77.0 against 77.0): MultiNLI (84.9 against 87.6) and PAWS (83.6 against 85.6) give back what PubMedQA (72.8 against 68.8) and HelpSteer2 gain. ECE on the Nimble 324 set rises from 0.063 to 0.085. The headline gain is 0.6 points, most of it from the Nimble 324 set (81.2 against 78.7).
+Where v2 is not better. Jevals HelpSteer2 Decision Score is 10.5 against v1's 11.9, and its top level flips over identical repeats on 2.3% of questions against 0.3%. The Nimble public macro is flat (77.0 against 77.0): MultiNLI (84.9 against 87.6) and PAWS (83.6 against 85.6) give back what PubMedQA (72.8 against 68.8) and HelpSteer2 gain. ECE on the Nimble 324 set rises from 0.063 to 0.070. The headline gain is 0.6 points, most of it from the Nimble 324 set (81.2 against 78.7).
 
 Two things to read carefully, unchanged from v1 because the pool is unchanged. HelpSteer2 and SummEval are not
 zero-shot for Jeb: the pool trains on the HelpSteer2 **train** split (the Jevals and Nimble items come from
@@ -50,7 +50,10 @@ are held out entirely), so those rows are held-out items of a seen rubric. And t
 (typed-decisions, Kev decision-v7) are the test splits of sources in the pool.
 
 Every eval record (per question: option keys, probabilities, pick, label, repeat, option order) is in
-`eval/`, with `eval/results.json` carrying the full metric set. The records were scored on the unmerged adapter;
+`eval/`, with `eval/results.json` carrying the full metric set. The records were scored on the unmerged adapter with the
+previous temperatures (each file's `model.temperatures`); the ECE and Decision Score columns above re-temper those
+same logits to the temperatures now in `temperatures.json` (see Temperature fits), so they differ from `results.json`
+on the sets with score questions. Accuracy is unchanged;
 the merged weights here reproduce them (see Merge verification below).
 
 ## Robustness to irrelevant content (nonce test)
@@ -59,7 +62,28 @@ The nonce pass (a fresh UUID planted in the state or appended to the instruction
 
 ## Temperature fits
 
-`temperatures.json` carries two fits on the calibration split (689 records, 921 questions: choice 225, noul 147, score 549). The `hard` fit minimises NLL against the argmax label and sharpens (T 0.79 / 0.67 / 0.83). The `train` fit minimises NLL against the training target, the soft or ordinal distribution the model was taught, and softens (T 1.19 / 1.09 / 1.22). v2 applies `train`, as v1 does, because the ordinal target is what makes rubric probabilities honest, and the `hard` fit is exactly the sharpening that made v0's HelpSteer2 calibration worse. The price is visible in the same file: on the calibration split measured against hard labels, ECE moves from 0.074 to 0.089 (choice), 0.064 to 0.064 (noul) and 0.092 to 0.128 (score) while NLL on the training target falls (0.4594 to 0.4542, 0.3776 to 0.3768, 1.1017 to 1.0929). The ECE column in the results table is measured on the evaluation sets, not on this split, with the applied temperatures. Both fits are in the file; a consumer who gates on the argmax may prefer `hard`, and either way should refit on its own data.
+`temperatures.json` carries two fits on the calibration split (689 records, 921 questions: choice 225, noul 147, score 549), both kept under their names. The `hard` fit minimises NLL against the argmax label and sharpens (T 0.79 / 0.67 / 0.83). The `train` fit minimises NLL against the training target, the soft or ordinal distribution the model was taught, and softens (T 1.19 / 1.09 / 1.22).
+
+**Applied since 2026-09-29: `mixed`, choice 1.19 and noul 1.09 from `train`, score 0.83 from `hard`**, the rule the [27B](https://huggingface.co/frontier-infra/jebadiah-27b/blob/main/eval/RESULTS.md#temperature-fits) has applied since 2026-09-26. 9B v2 first shipped with `train` for all three types. The `train` score target is the ordinal kernel, a deliberately smoothed distribution, so fitting T to it softens past what the labels support. No model was retrained and nothing was fitted on an evaluation set: both numbers are the calibration split's own fits. On the calibration split itself, fitting on one half and scoring the other (200 random splits by record family; raw logits for its 549 score questions read through AINode on our fleet, which reproduce the recorded `hard` score fit, 0.837 against 0.833), score ECE is 0.119 with `train`, 0.093 with no temperature and 0.079 with `hard`; NLL 0.953, 0.932, 0.934.
+
+Over all 20 evaluation sets (re-tempering the stored per-question logits), question-weighted ECE 0.080 with `train`, 0.079 with no temperature, 0.089 with `hard`, 0.071 applied; macro ECE 0.071, 0.074, 0.093, 0.065; NLL 0.589, 0.591, 0.634, 0.580; macro Decision Score (Jevals) 47.9, 47.8, 46.1, 48.3. Only the sets with score questions change:
+
+| Set (score questions present) | ECE, none | ECE, `train` (previous) | ECE, `hard` | ECE, applied | NLL, `train` | NLL, applied |
+|---|---:|---:|---:|---:|---:|---:|
+| Jevals HelpSteer2 (score) | 0.066 | 0.039 | 0.082 | 0.082 | 1.308 | 1.314 |
+| Nimble 324 (mixed) | 0.059 | 0.085 | 0.037 | 0.070 | 0.490 | 0.481 |
+| Kev transfer-v4 (mixed) | 0.037 | 0.025 | 0.067 | 0.027 | 0.442 | 0.440 |
+| Kev decision-v7 (mixed) | 0.059 | 0.050 | 0.087 | 0.048 | 0.530 | 0.534 |
+| typed-decisions (mixed) | 0.125 | 0.156 | 0.078 | 0.121 | 0.591 | 0.553 |
+| Nimble public HelpSteer2 (score) | 0.072 | 0.073 | 0.098 | 0.098 | 1.272 | 1.292 |
+| Nimble public SummEval consistency (score) | 0.149 | 0.195 | 0.118 | 0.118 | 0.532 | 0.462 |
+| Nimble public SummEval relevance (score) | 0.044 | 0.084 | 0.033 | 0.033 | 1.098 | 1.080 |
+
+It is not a win everywhere: both HelpSteer2 sets were better calibrated with `train`. We checked whether a separate score temperature per rubric, fitted on the calibration split, would keep HelpSteer2 where it was. It does not, because the split's own HelpSteer2 questions also fit about 0.85: their labels are close to uniform across 0 to 4, while the Jevals and Nimble HelpSteer2 items put over 70% of their labels at 3 or 4, where the model is less accurate and too confident. That is a shift in the label mix, which no fit on this split can see. Per-rubric fits also overfit a split this small (its SummEval part is four articles, and a 17-question rubric's fit ranged from 0.8 to 15 across halves), so one temperature per type stays.
+
+Calibration was fitted on the training distribution; refit the temperatures on your own labelled traffic before trusting a threshold. Log the raw distribution (`"calibration": "raw"` on AINode, `--no-temperatures` in the scripts), label a random sample of a few hundred decisions, fit one T per question type against the label on half of them and check ECE and NLL on the other half before you replace the file.
+
+ECE here uses 10 equal-width bins on the pick's probability rounded to a whole percent (bin = round(100p) // 10), repeat 0. Binning at floor(10p) gives slightly different values on the flat score sets (on the 9B's Nimble public HelpSteer2 records with `train`, 0.060 against 0.073).
 
 ## Merge verification
 

jebadiah.json

@@ -74,7 +74,7 @@
  "temperatures": {
   "choice": 1.1863,
   "noul": 1.0903,
-  "score": 1.2162
+  "score": 0.8329
  },
  "note": "same layout, config and tokenizer as the base repo; only the LoRA-targeted language_model weights differ. The served route applies no temperature."
 }

temperatures.json

@@ -2,9 +2,25 @@
  "temperatures": {
   "choice": 1.1863,
   "noul": 1.0903,
-  "score": 1.2162
+  "score": 0.8329
  },
- "applied_target": "train",
+ "applied_target": "mixed",
+ "applied_fits": {
+  "choice": "train",
+  "noul": "train",
+  "score": "hard"
+ },
+ "previous": {
+  "applied_target": "train",
+  "temperatures": {
+   "choice": 1.1863,
+   "noul": 1.0903,
+   "score": 1.2162
+  },
+  "replaced": "2026-09-29"
+ },
+ "why": "Refit 2026-09-29 without retraining, the rule the 27B has applied since 2026-09-26. Both fits are unchanged and come from the calibration split only. Score questions calibrate better at the hard fit (T 0.83) than at the train fit (T 1.22), whose ordinal target is deliberately smoothed: on held-out halves of the calibration split score ECE 0.119 -> 0.079. Choice and noul stay on the train fit. Re-tempering the stored logits of the 20 evaluation sets: question-weighted ECE 0.0796 -> 0.0709, macro 0.0709 -> 0.0653, NLL 0.5887 -> 0.5799; accuracy unchanged. Jevals HelpSteer2 gets worse (0.039 -> 0.082). See eval/RESULTS.md, Temperature fits.",
+ "top_level_stats_describe": "the train fit (nll_before/nll_after/ece_before/ece_after below are its calibration-split numbers)",
  "calib_file": "/workspace/jeb/data-v1/calib.jsonl",
  "n": {
   "choice": 225,
@@ -45,6 +61,26 @@
     "nll_before": 1.1017,
     "nll_after": 1.0929
    }
+  },
+  "mixed": {
+   "choice": {
+    "T": 1.1863,
+    "nll_before": 0.4594,
+    "nll_after": 0.4542,
+    "source": "train"
+   },
+   "noul": {
+    "T": 1.0903,
+    "nll_before": 0.3776,
+    "nll_after": 0.3768,
+    "source": "train"
+   },
+   "score": {
+    "T": 0.8329,
+    "nll_before": 0.9298,
+    "nll_after": 0.9225,
+    "source": "hard"
+   }
   }
  },
  "nll_before": {