Occupying-Mars/qwen3-4b-symbolic-vanilla-ppo-ent001-v1

Qwen3-4B — Vanilla PPO + entropy bonus 0.01 (cmp-vanilla-ppo-ent001-v1)

Second arm of the vanilla-PPO baseline for the GRPO-vs-PPO rollout-compaction study (prime-rl fork ChinmayK0607/blog-rl, branch feat/vanilla-ppo). Identical to qwen3-4b-symbolic-vanilla-ppo-v1 (clipped surrogate + value head + trainer-side GAE, full uncompacted rollouts) plus entropy_coef=0.01 (the PPO paper's Atari setting). Requires the vanilla LM head (fused_lm_head_token_chunk_size=disabled) — the fused kernel has no backward through entropy.

The result: collapse delayed, then terminal

step	10	30	50	70	90	100-150
val reward	0.528	0.694	0.778	0.708	0.833	0.000

The no-entropy arm collapsed at step ~80 and self-recovered to 0.972 by 150. This arm survived that window (0.833 at step 90 — its peak, and better than arm 1 ever was pre-collapse), then collapsed at ~100 with entropy pinned at 1e-4 and never recovered (1-turn degenerate episodes through 150). Vanilla PPO on this task lives on a knife's edge around steps 80-120; a 0.01 entropy bonus shifts the edge, it does not remove it.

Files

• root: step_90 policy — the pre-collapse peak (val 0.833), value head stripped, vLLM-loadable; value_head.safetensors alongside
• checkpoints/step_120/, checkpoints/step_150/ — mid/post-collapse policies for collapse forensics (val 0.000, 1-turn degenerate greedy mode)

Training setup

Same as arm 1: prime-rl ppo, GAE gamma=1.0 lambda=0.95, clip 0.2, value coef 0.5, LR 1e-6, batch 64 x group 8, 150 steps, temp 0.7, symbolic-curriculum-v2 (588 train / 72 val pass@4-mixed tasks), 4x RTX PRO 6000 Blackwell. W&B: https://wandb.ai/krishnapg2315/blog-rl/runs/fc57f9265a2d4ff8aaa80fabcc9058ab