| <!doctype html> |
| <html lang="zh-CN"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="description" content="Harbor Qwen GRPO Experiment: an evidence-backed archive of tool-using code-agent reinforcement learning."> |
| <title>Harbor Qwen GRPO Experiment</title> |
| <link rel="stylesheet" href="styles.css"> |
| </head> |
| <body> |
| <header class="hero"> |
| <nav class="nav shell" aria-label="主导航"> |
| <a class="brand" href="#top" aria-label="返回顶部"><span>H</span> Harbor Agent RL</a> |
| <div class="nav-links"> |
| <a href="#architecture">架构</a> |
| <a href="#evidence">结果</a> |
| <a href="#artifacts">产物</a> |
| <a href="#docs">文档</a> |
| </div> |
| </nav> |
|
|
| <div class="shell hero-grid" id="top"> |
| <section> |
| <p class="eyebrow">EVIDENCE-BACKED PROJECT ARCHIVE · 2026-08-30</p> |
| <h1>Tool-using code agent<br><em>RL, measured honestly.</em></h1> |
| <p class="lede"> |
| 基于 Harbor、rLLM、veRL 与 vLLM 构建的代码智能体强化学习工程，覆盖数据冻结、 |
| 容器化执行、GRPO、原生 critic/GAE PPO、checkpoint 评测与失败归因。 |
| </p> |
| <div class="actions"> |
| <a class="button primary" href="#evidence">查看最终证据</a> |
| <a class="button secondary" href="docs/PROJECT_README.md">阅读完整 README</a> |
| </div> |
| </section> |
|
|
| <aside class="status-card" aria-label="项目状态"> |
| <div class="status-line"><span class="status-dot"></span>阶段性归档</div> |
| <p class="status-title">P0 decision</p> |
| <p class="status-decision">停止追加 PPO 训练</p> |
| <dl> |
| <div><dt>正式任务</dt><dd>267</dd></div> |
| <div><dt>代码仓库</dt><dd>89</dd></div> |
| <div><dt>任务镜像</dt><dd>267 / 267</dd></div> |
| <div><dt>PPO warm-start</dt><dd>20 + 20 steps</dd></div> |
| </dl> |
| </aside> |
| </div> |
| </header> |
|
|
| <main> |
| <section class="shell section intro"> |
| <p class="section-kicker">PROJECT QUESTION</p> |
| <h2>强化学习是否真正提升了<br>工具调用代码智能体？</h2> |
| <p class="wide-copy"> |
| 项目没有用训练 reward 或 critic loss 代替能力结论，而是冻结数据与执行协议， |
| 对原始 SFT、父 PPO checkpoint 与后续 checkpoint 进行任务级配对评测。 |
| 最终统一 64K P0 未观察到稳定的总体增益。 |
| </p> |
| </section> |
|
|
| <section class="section tinted" id="architecture"> |
| <div class="shell"> |
| <p class="section-kicker">SYSTEM ARCHITECTURE</p> |
| <h2>训练与沙箱执行解耦</h2> |
| <div class="flow" role="list" aria-label="系统流程"> |
| <article role="listitem"><b>01</b><h3>Frozen data</h3><p>SWE-Smith 清洗、Oracle 验证、仓库平衡与固定切分。</p></article> |
| <span class="arrow" aria-hidden="true">→</span> |
| <article role="listitem"><b>02</b><h3>Trainer</h3><p>AutoDL 上的 rLLM + veRL，执行 GRPO 或 critic/GAE PPO。</p></article> |
| <span class="arrow" aria-hidden="true">→</span> |
| <article role="listitem"><b>03</b><h3>Rollout</h3><p>独立 vLLM 推理服务，支持权重同步与长上下文 agent 轨迹。</p></article> |
| <span class="arrow" aria-hidden="true">→</span> |
| <article role="listitem"><b>04</b><h3>Sandbox</h3><p>Harbor 调度隔离任务环境，通过测试产生 0/1 可验证奖励。</p></article> |
| </div> |
| <div class="tech-row" aria-label="技术栈"> |
| <span>Harbor</span><span>rLLM</span><span>veRL</span><span>vLLM</span><span>FSDP</span><span>LoRA</span><span>Kubernetes</span> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="shell section" id="evidence"> |
| <div class="section-heading"> |
| <div><p class="section-kicker">UNIFIED 64K P0</p><h2>最终评测没有总体差异</h2></div> |
| <p>相同 seed、上下文、Agent 配置和每题一次采样；按任务进行配对比较。</p> |
| </div> |
|
|
| <div class="score-grid"> |
| <article class="score-card"> |
| <div class="score-head"><h3>Frozen SWE-Smith test</h3><span>38 tasks</span></div> |
| <div class="comparison"><div><small>Original SFT</small><strong>6<sup>/38</sup></strong><p>15.79%</p></div><div class="equals">=</div><div><small>Parent PPO</small><strong>6<sup>/38</sup></strong><p>15.79%</p></div></div> |
| <footer>3 wins · 32 ties · 3 losses · McNemar p = 1.0</footer> |
| </article> |
| <article class="score-card"> |
| <div class="score-head"><h3>SWE-bench Verified</h3><span>27 tasks</span></div> |
| <div class="comparison"><div><small>Original SFT</small><strong>9<sup>/27</sup></strong><p>33.33%</p></div><div class="equals">=</div><div><small>Parent PPO</small><strong>9<sup>/27</sup></strong><p>33.33%</p></div></div> |
| <footer>2 wins · 23 ties · 2 losses · McNemar p = 1.0</footer> |
| </article> |
| </div> |
|
|
| <div class="finding"> |
| <div class="finding-mark">!</div> |
| <div><h3>更多计算没有转化为更多成功</h3><p>Parent 在两个面板分别增加约 47.7% / 46.8% 的平均输入 token，运行时间约为 SFT 的 2.33 / 2.24 倍，但成功数没有增加。</p></div> |
| </div> |
| </section> |
|
|
| <section class="section dark" id="artifacts"> |
| <div class="shell"> |
| <p class="section-kicker">TRAINING EVIDENCE</p> |
| <h2>工程链路完成，能力门禁未通过</h2> |
| <div class="metric-grid"> |
| <article><strong>160</strong><span>第二轮 warm-start episodes</span></article> |
| <article><strong>32</strong><span>训练期成功 episodes</span></article> |
| <article><strong>12</strong><span>timeouts</span></article> |
| <article><strong>166.6 MiB</strong><span>策略模型 LoRA 权重；完整策略检查点目录约 177.5 MiB</span></article> |
| <article><strong>14.11 GiB</strong><span>critic BF16 shards</span></article> |
| <article><strong>−1.689</strong><span>最终 explained variance</span></article> |
| </div> |
| <p class="caveat">训练成功中有 16/32 来自同一个 Pygments 任务。VF loss 降至约 0.0078，但 explained variance 仍为负，因此不能把优化器稳定运行解释为策略能力提升。</p> |
| </div> |
| </section> |
|
|
| <section class="shell section" id="docs"> |
| <p class="section-kicker">CURATED DOCUMENTATION</p> |
| <h2>从结论回溯到实现</h2> |
| <div class="doc-grid"> |
| <a href="docs/summary.md"><span>01</span><h3>项目总结</h3><p>最终状态、关键结果和停止决策。</p></a> |
| <a href="docs/PROJECT_LAYERED_ARCHITECTURE.md"><span>02</span><h3>分层架构</h3><p>本地、AutoDL、云端执行与数据层关系。</p></a> |
| <a href="docs/PROJECT_WORK_CYCLE.md"><span>03</span><h3>完整工作周期</h3><p>从环境构建到归档的端到端过程。</p></a> |
| <a href="docs/PPO_TRAINING_RUNBOOK.md"><span>04</span><h3>PPO 训练记录</h3><p>critic、GAE、显存、checkpoint 与评测门禁。</p></a> |
| <a href="docs/KLEAR_AGENTFORGE_REPRODUCIBILITY_GAP.md"><span>05</span><h3>复现差距审计</h3><p>公开信息、项目实现与论文口径的边界。</p></a> |
| <a href="docs/PROJECT_README.md"><span>06</span><h3>项目 README</h3><p>源码入口、运行命令和历史状态。</p></a> |
| </div> |
| </section> |
|
|
| <section class="section disclaimer"> |
| <div class="shell disclaimer-grid"> |
| <div><p class="section-kicker">PUBLICATION BOUNDARY</p><h2>这是实验档案，不是效果宣称</h2></div> |
| <ul> |
| <li>不包含 `.env`、密钥、证书、Registry 密码或运行时状态。</li> |
| <li>最终 267 题数据已上传至独立的 <a href="https://huggingface.co/datasets/keryszhan/harbor-swesmith-rl-artifacts">Harbor SWE-Smith 强化学习数据集</a>，当前保持私有。</li> |
| <li>PPO 策略模型 LoRA 已上传至独立的 <a href="https://huggingface.co/keryszhan/klear-agentforge-8b-ppo-step20-adapter">第 20 步适配器模型仓库</a>，当前保持私有；使用前需取得受访问门控基础模型的访问权限。</li> |
| <li>云资源已释放，文档中的部署命令属于可复现的历史运行手册。</li> |
| </ul> |
| </div> |
| </section> |
| </main> |
|
|
| <footer class="footer"> |
| <div class="shell"><strong>Harbor Qwen GRPO Experiment</strong><span>Evidence over anecdotes.</span></div> |
| </footer> |
| </body> |
| </html> |
|
|
