---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
datasets:
- openai/gsm8k
library_name: transformers
pipeline_tag: text-generation
tags:
- trl
- grpo
- trl-autoresearch
- qwen-grpo
- gsm8k
- reasoning
model-index:
- name: qwen-grpo-r5
  results:
  - task:
      type: text-generation
      name: Math word problems
    dataset:
      name: GSM8K (test)
      type: openai/gsm8k
      split: test
    metrics:
    - type: accuracy
      value: 0.7089
      name: accuracy (inspect_evals/gsm8k, 10-shot, greedy)
---

# Qwen3-0.6B + GRPO on GSM8K

GRPO ([TRL](https://github.com/huggingface/trl) `GRPOTrainer`) applied to **Qwen3-0.6B**,
the smallest model in the Qwen3 family, with a verifiable correctness reward on GSM8K.

| Model | `inspect_evals/gsm8k` (full 1319, 10-shot, greedy) |
|---|---|
| `Qwen/Qwen3-0.6B` (baseline, thinking off) | 0.4754 ± 0.0138 |
| `Qwen/Qwen3-0.6B` (baseline, thinking on) | 0.0000 — never closes `<think>` within 2560 tokens |
| **`cmpatino/qwen-grpo-r5`** (this model) | **0.7089 ± 0.0125** |

**+23.4 points absolute / +49% relative** over the untrained baseline, for **$7.96** of GPU time.

## Important: this model runs in non-thinking mode

Its chat template is patched so the generation prompt **always** ends with an empty
`<think>\n\n</think>` block. Thinking mode is not available — the 0.6B model cannot finish a
reasoning block inside a usable token budget, and training/eval formats are kept identical
on purpose. Use the tokenizer that ships with this repo.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

m = AutoModelForCausalLM.from_pretrained("cmpatino/qwen-grpo-r5", dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained("cmpatino/qwen-grpo-r5")

PROMPT = """Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{q}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.

Reasoning:"""

msgs = [{"role": "user", "content": PROMPT.format(q="Natalia sold clips to 48 friends in April, and then she sold half as many clips in May. How many clips did she sell altogether?")}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(m.device)
print(tok.decode(m.generate(ids, max_new_tokens=512, do_sample=False)[0][ids.shape[-1]:]))
```

## Reward

The reward is a line-by-line reimplementation of the scorer the benchmark actually uses —
inspect_ai's `match(numeric=True, location="end")`: strip `$ , £ € * _` and trailing periods,
split the completion on whitespace, scan tokens in reverse, and compare the first parseable
number against the gold answer at 5 significant figures. It was unit-tested against the real
scorer before training. A second reward (weight 0.2) pays for ending on an
`ANSWER: <number>` line; it saturates above 0.95 within ~25 steps.

## Training

Three sequential GRPO stages on `openai/gsm8k` `main` train (zero-shot prompts, inspect's
`MATH_PROMPT_TEMPLATE` verbatim), **251 optimizer steps total** — about 0.55 of one epoch, so
no prompt is seen twice.

| Stage | From | Steps | lr | Rollouts × prompts / step | Temp | GPU |
|---|---|---|---|---|---|---|
| r2 | `Qwen/Qwen3-0.6B` | 77 | 3e-6 | 8 × 16 | 1.0 | L4 |
| r4 | `qwen-grpo-r2` | 129 | 2e-6 | 16 × 16 | 1.0 | L40S |
| r5 | `qwen-grpo-r4-s100` | 74 | 2e-6 | 16 × 16 | 1.15 | L40S |

DAPO loss, `beta=0` (no KL penalty, no reference model), rewards scaled within each rollout
group, truncated completions masked out, `max_completion_length` 768, vLLM colocated with the
trainer on a single GPU.

Intermediate checkpoints are published as `cmpatino/qwen-grpo-r4-s{25,50,75,100,125}` and
`cmpatino/qwen-grpo-r5-s60`. Full-test scores rise roughly monotonically with cumulative steps
(0.646 → 0.662 → 0.658 → 0.683 → 0.692 → 0.708) and then flatten: `qwen-grpo-r4-s125` scores
0.7081 ± 0.0125, a statistical tie with this model.

## Caveats

- The reported score is r5's **final** checkpoint, so it involves no test-set selection — but
  r5 was *started* from `r4-s100`, picked using limit-200 test scores. Read 0.708–0.709 as one
  plateau, not two results.
- No validation split was held out of GSM8K train; the eval budget went into making the
  reported numbers full-test instead.
- Standard error is ±0.013, so checkpoint differences below ~0.03 are noise.
- Greedy decoding, single sample, canonical 10-shot `inspect_evals/gsm8k`. No self-consistency.
- Optimising directly against the benchmark's own scorer is deliberate here; it means the score
  should be read as "GSM8K-shaped arithmetic reasoning", not as general math ability.