Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,63 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-4B-Instruct-2507
+tags:
+  - reinforcement-learning
+  - ppo
+  - tool-calling
+  - agent
+  - prime-rl
+library_name: transformers
+---
+
+# Qwen3-4B — Vanilla PPO on Symbolic Tool-Calling (cmp-vanilla-ppo-4gpu-v1)
+
+Qwen/Qwen3-4B-Instruct-2507 trained with **vanilla PPO** (Schulman et al. 2017 —
+clipped surrogate, learned value head, trainer-side GAE) on the deterministic
+symbolic tool-calling benchmark from the GRPO-vs-PPO-under-rollout-compaction
+study (prime-rl fork: ChinmayK0607/blog-rl, branch feat/vanilla-ppo).
+
+This is the **uncompacted real-PPO baseline** that study was missing: its
+compacted PPO (rollout-reward critic targets, no GAE) collapsed to
+0.25/0.14 pass@4; this run shows actual PPO nearly matches GRPO.
+
+## Results (val = 72 held-out mixed tasks, greedy)
+
+| step | 10 | 60 | 80-100 | 140 | 150 |
+|---|---|---|---|---|---|
+| val reward | 0.556 | 0.764 | 0.000 | 0.903 | **0.972** |
+
+Reference (same env, Phase-A regime, 8xH100): full GRPO 1.000, compacted
+GRPO 0.986, segment-normalized GRPO 0.986.
+
+Notable dynamics: entropy collapse around step 80 made a corrupted tool-call
+token the greedy argmax (val 0 for ~20 steps, 2-turn episodes) — the policy
+**self-recovered** by step 110 and finished at 0.972 with more efficient
+solutions (turns 14.3 -> 11.8). Critic explained variance rose from a cold
+start (0.01 -> 0.07+) — no warm-start needed, unlike the compacted-PPO arm.
+
+## Training setup
+
+- Algorithm: prime-rl `ppo` — per-token terminal-reward stream, trainer-side
+  GAE (gamma=1.0, lambda=0.95), clipped surrogate (eps=0.2), clipped value
+  loss (coef 0.5), entropy_coef=0 (a 0.01 arm is a follow-up), LR 1e-6
+- Data: symbolic-curriculum-v2 — 588 train / 72 val pass@4-mixed tasks
+  (frozen-model pass@4 sweep over deterministic pools, keep tasks solved
+  1-3 of 4 times), depth 3-5, max 24 turns, 32k-token episodes
+- Batch 64 rollouts x group 8, 150 steps, temp 0.7, 4096 completion tokens
+- Hardware: 4x RTX PRO 6000 Blackwell 96GB (2 inference / 2 trainer), ~35 s/step
+- W&B: https://wandb.ai/krishnapg2315/blog-rl/runs/5c4f977de99343c7b44edfe8d05fb5c0
+
+## Files
+
+- `model.safetensors` — step-150 policy, value head stripped (vLLM-loadable)
+- `value_head.safetensors` — the trained PPO value head (`value_head.weight`,
+  [1, hidden]) for critic warm-start experiments
+  (`trainer.model.ppo_value_head_init`)
+
+## Caveats
+
+- Step-150 weights. The greedy policy is functional (0.972 val) but the run
+  showed a mid-training degeneracy window — prefer temp>0 sampling or verify
+  greedy behavior on your tasks.
+- Tool-call format: Qwen3/hermes (`<tool_call>` JSON blocks).