Occupying-Mars/qwen3-4b-symbolic-vanilla-ppo-v1

Qwen3-4B — Vanilla PPO on Symbolic Tool-Calling (cmp-vanilla-ppo-4gpu-v1)

Qwen/Qwen3-4B-Instruct-2507 trained with vanilla PPO (Schulman et al. 2017 — clipped surrogate, learned value head, trainer-side GAE) on the deterministic symbolic tool-calling benchmark from the GRPO-vs-PPO-under-rollout-compaction study (prime-rl fork: ChinmayK0607/blog-rl, branch feat/vanilla-ppo).

This is the uncompacted real-PPO baseline that study was missing: its compacted PPO (rollout-reward critic targets, no GAE) collapsed to 0.25/0.14 pass@4; this run shows actual PPO nearly matches GRPO.

Results (val = 72 held-out mixed tasks, greedy)

step	10	60	80-100	140	150
val reward	0.556	0.764	0.000	0.903	0.972

Reference (same env, Phase-A regime, 8xH100): full GRPO 1.000, compacted GRPO 0.986, segment-normalized GRPO 0.986.

Notable dynamics: entropy collapse around step 80 made a corrupted tool-call token the greedy argmax (val 0 for ~20 steps, 2-turn episodes) — the policy self-recovered by step 110 and finished at 0.972 with more efficient solutions (turns 14.3 -> 11.8). Critic explained variance rose from a cold start (0.01 -> 0.07+) — no warm-start needed, unlike the compacted-PPO arm.

Training setup

• Algorithm: prime-rl ppo — per-token terminal-reward stream, trainer-side GAE (gamma=1.0, lambda=0.95), clipped surrogate (eps=0.2), clipped value loss (coef 0.5), entropy_coef=0 (a 0.01 arm is a follow-up), LR 1e-6
• Data: symbolic-curriculum-v2 — 588 train / 72 val pass@4-mixed tasks (frozen-model pass@4 sweep over deterministic pools, keep tasks solved 1-3 of 4 times), depth 3-5, max 24 turns, 32k-token episodes
• Batch 64 rollouts x group 8, 150 steps, temp 0.7, 4096 completion tokens
• Hardware: 4x RTX PRO 6000 Blackwell 96GB (2 inference / 2 trainer), ~35 s/step
• W&B: https://wandb.ai/krishnapg2315/blog-rl/runs/5c4f977de99343c7b44edfe8d05fb5c0

Files

• model.safetensors — step-150 policy, value head stripped (vLLM-loadable)
• value_head.safetensors — the trained PPO value head (value_head.weight, [1, hidden]) for critic warm-start experiments (trainer.model.ppo_value_head_init)

Caveats

• Step-150 weights. The greedy policy is functional (0.972 val) but the run showed a mid-training degeneracy window — prefer temp>0 sampling or verify greedy behavior on your tasks.
• Tool-call format: Qwen3/hermes (<tool_call> JSON blocks).