25b3nk
/

smollm2-360m-boolq-calibration-grpo2-v1

@@ -1,199 +1,110 @@
 ---
-library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+license: apache-2.0
+base_model: HuggingFaceTB/SmolLM2-360M-Instruct
+tags:
+- calibration
+- boolq
+- rlcd
+- grpo
+datasets:
+- google/boolq
+language:
+- en
+pipeline_tag: text-generation
 ---
+# smollm2-360m-boolq-calibration-grpo2-v1
+GRPO-trained checkpoint from **slm-calibration-rlcd** — a project training small LMs to
+make **calibrated yes/no decisions**, inspired by TypeSafe AI's "Jev" model / "RLCD"
+(Reinforcement Learning for Calibrated Decisions). Full project:
+[25b3nk/slm-calibration-rlcd](https://github.com/25b3nk/slm-calibration-rlcd).
+This is the **full GRPO-trained model**, built on top of
+[sft-warmup-v2](https://huggingface.co/25b3nk/smollm2-360m-boolq-calibration-sft-v2).
+It is a **real but modest** calibration improvement, not a finished or perfectly
+calibrated model — see the eval table and discussion below before over-reading the
+result.
+## The interesting part: why the previous GRPO attempts failed, and what fixed it
+Three earlier GRPO runs on top of the original SFT checkpoint (sft-warmup-v1) all
+**collapsed**: the model's confidence output converged to just 1-2 of the two fixed
+confidence-target constants used in v1's SFT data (0.92 and/or 0.65), regardless of
+training steps or learning rate. That's not calibration — it's the model parroting the
+SFT heuristic. Root cause: v1's SFT prior had near-zero output entropy on the
+confidence dimension, so GRPO had nothing to diversify from.
+The fix was **not** a GRPO hyperparameter change — it was upstream, in SFT: v2
+(see [sft-warmup-v2](https://huggingface.co/25b3nk/smollm2-360m-boolq-calibration-sft-v2))
+samples confidence targets from ranges instead of two fixed constants, and its prompt
+explicitly asks for a confidence value. This checkpoint (grpo2-v1) reuses the **exact
+same GRPO hyperparameters** as the failed v1 attempts (lr=1e-6, confidence-alignment
+weight λ=0.5, KL penalty β=0.04, num_generations=6, 1 epoch, 56 steps) on top of
+sft-warmup-v2, specifically to isolate the SFT/prompt change as the cause of the fix.
+It worked: this checkpoint produced **11 distinct confidence values** across the
+500-example eval set (spanning roughly 0.61-0.89), versus 1-2 distinct values for every
+prior GRPO attempt. That's the key qualitative result — genuinely non-degenerate,
+varied confidence output for the first time in this project.
+A **correctness-only control** run (grpo2-control-v1: same pipeline and hyperparameters,
+but the confidence-alignment reward term zeroed) was also trained for comparison. The
+treatment (this checkpoint) beats the control on both calibration metrics at equal
+accuracy, which validates that the calibration-specific reward terms are doing real,
+measurable work rather than just adding noise.
+## What it does
+Given a BoolQ-style passage + yes/no question, outputs:
+```
+<answer>yes|no</answer><confidence>0.XX</confidence>
+```
+## Training
+- Base: [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct),
+  warm-started with
+  [sft-warmup-v2](https://huggingface.co/25b3nk/smollm2-360m-boolq-calibration-sft-v2)
+- GRPO, 56 steps, 1 epoch, lr=1e-6, num_generations=6, KL penalty β=0.04
+- Training set: 453-example solve-rate-filtered subset of BoolQ train
+- Reward: `1[answer==label] - (confidence - 1[answer==label])^2 - λ*(confidence - p_logit)^2`
+  — correctness + Brier term + a confidence-alignment term (λ=0.5) pulling the stated
+  confidence toward the model's own internal logit-derived probability
+## Eval (same 500-example BoolQ validation slice, seed 0)
+| Metric | sft-warmup-v2 (pre-GRPO) | grpo2-control-v1 (correctness-only reward) | This checkpoint (grpo2-v1) |
+|---|---|---|---|
+| Format compliance | 1.0 | 1.0 | **1.0** |
+| Accuracy | 0.804 | 0.79 | 0.792 |
+| Brier score | 0.167 | 0.176 | **0.173** |
+| ECE | 0.090 | 0.098 | **0.086** |
+| Distinct confidence values | n/a | not measured for this comparison | **11** (spanning ~0.61-0.89) |
+Two things worth being precise about, not glossing over:
+- Both GRPO runs (this checkpoint and the control) actually have a **higher** Brier
+  score than sft-warmup-v2 alone (0.173 / 0.176 vs 0.167) — GRPO on top of this SFT
+  stage is not a strict win over the SFT-only checkpoint on every metric. The "11
+  distinct confidence values" figure is compared against the *earlier, SFT-v1-based*
+  GRPO attempts (which produced only 1-2 distinct values), not against the control,
+  which shares SFT v2's higher-entropy prior and was not measured for this specific
+  statistic.
+- Calibration (ECE 0.086) is only a modest improvement over sft-warmup-v2's own ECE
+  (0.090) — this is **not** a fully calibrated model.
+The result to take away is qualitative as much as quantitative: for the first time in
+this project, GRPO produced varied, non-degenerate confidence output instead of
+collapsing to 1-2 constants (as every prior GRPO attempt on top of sft-warmup-v1 did),
+and this checkpoint beat a correctness-only control trained on the identical pipeline
+on both calibration metrics (ECE, Brier) at essentially equal accuracy (0.792 vs 0.79)
+— evidence the calibration-specific reward terms are doing real, measurable work.
+## Full design + ADRs
+- [ADR-0001: initial design + compute](https://github.com/25b3nk/slm-calibration-rlcd/blob/main/docs/adr/0001-initial-design-and-compute.md)
+(No GRPO-specific ADR exists yet for this run at push time; see the project repo's
+tracking notes / experiment logs for the full run details referenced above.)