Instructions to use cmpatino/qwen-grpo-r5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cmpatino/qwen-grpo-r5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="cmpatino/qwen-grpo-r5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("cmpatino/qwen-grpo-r5")
model = AutoModelForCausalLM.from_pretrained("cmpatino/qwen-grpo-r5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cmpatino/qwen-grpo-r5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cmpatino/qwen-grpo-r5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cmpatino/qwen-grpo-r5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/cmpatino/qwen-grpo-r5

SGLang

How to use cmpatino/qwen-grpo-r5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cmpatino/qwen-grpo-r5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cmpatino/qwen-grpo-r5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cmpatino/qwen-grpo-r5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cmpatino/qwen-grpo-r5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use cmpatino/qwen-grpo-r5 with Docker Model Runner:
```
docker model run hf.co/cmpatino/qwen-grpo-r5
```

qwen-grpo-r5 / README.md

cmpatino HF Staff

Upload README.md with huggingface_hub

06c0dac verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

4.84 kB

metadata

license: apache-2.0
base_model: Qwen/Qwen3-0.6B
datasets:
  - openai/gsm8k
library_name: transformers
pipeline_tag: text-generation
tags:
  - trl
  - grpo
  - trl-autoresearch
  - qwen-grpo
  - gsm8k
  - reasoning
model-index:
  - name: qwen-grpo-r5
    results:
      - task:
          type: text-generation
          name: Math word problems
        dataset:
          name: GSM8K (test)
          type: openai/gsm8k
          split: test
        metrics:
          - type: accuracy
            value: 0.7089
            name: accuracy (inspect_evals/gsm8k, 10-shot, greedy)

Qwen3-0.6B + GRPO on GSM8K

GRPO (TRL GRPOTrainer) applied to Qwen3-0.6B, the smallest model in the Qwen3 family, with a verifiable correctness reward on GSM8K.

Model	`inspect_evals/gsm8k` (full 1319, 10-shot, greedy)
`Qwen/Qwen3-0.6B` (baseline, thinking off)	0.4754 ± 0.0138
`Qwen/Qwen3-0.6B` (baseline, thinking on)	0.0000 — never closes `<think>` within 2560 tokens
`cmpatino/qwen-grpo-r5` (this model)	0.7089 ± 0.0125

+23.4 points absolute / +49% relative over the untrained baseline, for $7.96 of GPU time.

Important: this model runs in non-thinking mode

Its chat template is patched so the generation prompt always ends with an empty <think>\n\n</think> block. Thinking mode is not available — the 0.6B model cannot finish a reasoning block inside a usable token budget, and training/eval formats are kept identical on purpose. Use the tokenizer that ships with this repo.

from transformers import AutoModelForCausalLM, AutoTokenizer

m = AutoModelForCausalLM.from_pretrained("cmpatino/qwen-grpo-r5", dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained("cmpatino/qwen-grpo-r5")

PROMPT = """Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{q}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.

Reasoning:"""

msgs = [{"role": "user", "content": PROMPT.format(q="Natalia sold clips to 48 friends in April, and then she sold half as many clips in May. How many clips did she sell altogether?")}]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(m.device)
print(tok.decode(m.generate(ids, max_new_tokens=512, do_sample=False)[0][ids.shape[-1]:]))

Reward

The reward is a line-by-line reimplementation of the scorer the benchmark actually uses — inspect_ai's match(numeric=True, location="end"): strip $ , £ € * _ and trailing periods, split the completion on whitespace, scan tokens in reverse, and compare the first parseable number against the gold answer at 5 significant figures. It was unit-tested against the real scorer before training. A second reward (weight 0.2) pays for ending on an ANSWER: <number> line; it saturates above 0.95 within ~25 steps.

Training

Three sequential GRPO stages on openai/gsm8k main train (zero-shot prompts, inspect's MATH_PROMPT_TEMPLATE verbatim), 251 optimizer steps total — about 0.55 of one epoch, so no prompt is seen twice.

Stage	From	Steps	lr	Rollouts × prompts / step	Temp	GPU
r2	`Qwen/Qwen3-0.6B`	77	3e-6	8 × 16	1.0	L4
r4	`qwen-grpo-r2`	129	2e-6	16 × 16	1.0	L40S
r5	`qwen-grpo-r4-s100`	74	2e-6	16 × 16	1.15	L40S

DAPO loss, beta=0 (no KL penalty, no reference model), rewards scaled within each rollout group, truncated completions masked out, max_completion_length 768, vLLM colocated with the trainer on a single GPU.

Intermediate checkpoints are published as cmpatino/qwen-grpo-r4-s{25,50,75,100,125} and cmpatino/qwen-grpo-r5-s60. Full-test scores rise roughly monotonically with cumulative steps (0.646 → 0.662 → 0.658 → 0.683 → 0.692 → 0.708) and then flatten: qwen-grpo-r4-s125 scores 0.7081 ± 0.0125, a statistical tie with this model.

Caveats

The reported score is r5's final checkpoint, so it involves no test-set selection — but r5 was started from r4-s100, picked using limit-200 test scores. Read 0.708–0.709 as one plateau, not two results.
No validation split was held out of GSM8K train; the eval budget went into making the reported numbers full-test instead.
Standard error is ±0.013, so checkpoint differences below ~0.03 are noise.
Greedy decoding, single sample, canonical 10-shot inspect_evals/gsm8k. No self-consistency.
Optimising directly against the benchmark's own scorer is deliberate here; it means the score should be read as "GSM8K-shaped arithmetic reasoning", not as general math ability.