index.html

<!doctype html>
<html lang="zh-CN">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="description" content="Harbor Qwen GRPO Experiment: an evidence-backed archive of tool-using code-agent reinforcement learning.">
  <title>Harbor Qwen GRPO Experiment</title>
  <link rel="stylesheet" href="styles.css">
</head>
<body>
  <header class="hero">
    <nav class="nav shell" aria-label="主导航">
      <a class="brand" href="#top" aria-label="返回顶部"><span>H</span> Harbor Agent RL</a>
      <div class="nav-links">
        <a href="#architecture">架构</a>
        <a href="#evidence">结果</a>
        <a href="#artifacts">产物</a>
        <a href="#docs">文档</a>
      </div>
    </nav>

    <div class="shell hero-grid" id="top">
      <section>
        <p class="eyebrow">EVIDENCE-BACKED PROJECT ARCHIVE · 2026-08-30</p>
        <h1>Tool-using code agent<br><em>RL, measured honestly.</em></h1>
        <p class="lede">
          基于 Harbor、rLLM、veRL 与 vLLM 构建的代码智能体强化学习工程，覆盖数据冻结、
          容器化执行、GRPO、原生 critic/GAE PPO、checkpoint 评测与失败归因。
        </p>
        <div class="actions">
          <a class="button primary" href="#evidence">查看最终证据</a>
          <a class="button secondary" href="docs/PROJECT_README.md">阅读完整 README</a>
        </div>
      </section>

      <aside class="status-card" aria-label="项目状态">
        <div class="status-line"><span class="status-dot"></span>阶段性归档</div>
        <p class="status-title">P0 decision</p>
        <p class="status-decision">停止追加 PPO 训练</p>
        <dl>
          <div><dt>正式任务</dt><dd>267</dd></div>
          <div><dt>代码仓库</dt><dd>89</dd></div>
          <div><dt>任务镜像</dt><dd>267 / 267</dd></div>
          <div><dt>PPO warm-start</dt><dd>20 + 20 steps</dd></div>
        </dl>
      </aside>
    </div>
  </header>

  <main>
    <section class="shell section intro">
      <p class="section-kicker">PROJECT QUESTION</p>
      <h2>强化学习是否真正提升了<br>工具调用代码智能体？</h2>
      <p class="wide-copy">
        项目没有用训练 reward 或 critic loss 代替能力结论，而是冻结数据与执行协议，
        对原始 SFT、父 PPO checkpoint 与后续 checkpoint 进行任务级配对评测。
        最终统一 64K P0 未观察到稳定的总体增益。
      </p>
    </section>

    <section class="section tinted" id="architecture">
      <div class="shell">
        <p class="section-kicker">SYSTEM ARCHITECTURE</p>
        <h2>训练与沙箱执行解耦</h2>
        <div class="flow" role="list" aria-label="系统流程">
          <article role="listitem"><b>01</b><h3>Frozen data</h3><p>SWE-Smith 清洗、Oracle 验证、仓库平衡与固定切分。</p></article>
          <span class="arrow" aria-hidden="true">→</span>
          <article role="listitem"><b>02</b><h3>Trainer</h3><p>AutoDL 上的 rLLM + veRL，执行 GRPO 或 critic/GAE PPO。</p></article>
          <span class="arrow" aria-hidden="true">→</span>
          <article role="listitem"><b>03</b><h3>Rollout</h3><p>独立 vLLM 推理服务，支持权重同步与长上下文 agent 轨迹。</p></article>
          <span class="arrow" aria-hidden="true">→</span>
          <article role="listitem"><b>04</b><h3>Sandbox</h3><p>Harbor 调度隔离任务环境，通过测试产生 0/1 可验证奖励。</p></article>
        </div>
        <div class="tech-row" aria-label="技术栈">
          <span>Harbor</span><span>rLLM</span><span>veRL</span><span>vLLM</span><span>FSDP</span><span>LoRA</span><span>Kubernetes</span>
        </div>
      </div>
    </section>

    <section class="shell section" id="evidence">
      <div class="section-heading">
        <div><p class="section-kicker">UNIFIED 64K P0</p><h2>最终评测没有总体差异</h2></div>
        <p>相同 seed、上下文、Agent 配置和每题一次采样；按任务进行配对比较。</p>
      </div>

      <div class="score-grid">
        <article class="score-card">
          <div class="score-head"><h3>Frozen SWE-Smith test</h3><span>38 tasks</span></div>
          <div class="comparison"><div><small>Original SFT</small><strong>6<sup>/38</sup></strong><p>15.79%</p></div><div class="equals">=</div><div><small>Parent PPO</small><strong>6<sup>/38</sup></strong><p>15.79%</p></div></div>
          <footer>3 wins · 32 ties · 3 losses · McNemar p = 1.0</footer>
        </article>
        <article class="score-card">
          <div class="score-head"><h3>SWE-bench Verified</h3><span>27 tasks</span></div>
          <div class="comparison"><div><small>Original SFT</small><strong>9<sup>/27</sup></strong><p>33.33%</p></div><div class="equals">=</div><div><small>Parent PPO</small><strong>9<sup>/27</sup></strong><p>33.33%</p></div></div>
          <footer>2 wins · 23 ties · 2 losses · McNemar p = 1.0</footer>
        </article>
      </div>

      <div class="finding">
        <div class="finding-mark">!</div>
        <div><h3>更多计算没有转化为更多成功</h3><p>Parent 在两个面板分别增加约 47.7% / 46.8% 的平均输入 token，运行时间约为 SFT 的 2.33 / 2.24 倍，但成功数没有增加。</p></div>
      </div>
    </section>

    <section class="section dark" id="artifacts">
      <div class="shell">
        <p class="section-kicker">TRAINING EVIDENCE</p>
        <h2>工程链路完成，能力门禁未通过</h2>
        <div class="metric-grid">
          <article><strong>160</strong><span>第二轮 warm-start episodes</span></article>
          <article><strong>32</strong><span>训练期成功 episodes</span></article>
          <article><strong>12</strong><span>timeouts</span></article>
          <article><strong>166.6 MiB</strong><span>策略模型 LoRA 权重；完整策略检查点目录约 177.5 MiB</span></article>
          <article><strong>14.11 GiB</strong><span>critic BF16 shards</span></article>
          <article><strong>−1.689</strong><span>最终 explained variance</span></article>
        </div>
        <p class="caveat">训练成功中有 16/32 来自同一个 Pygments 任务。VF loss 降至约 0.0078，但 explained variance 仍为负，因此不能把优化器稳定运行解释为策略能力提升。</p>
      </div>
    </section>

    <section class="shell section" id="docs">
      <p class="section-kicker">CURATED DOCUMENTATION</p>
      <h2>从结论回溯到实现</h2>
      <div class="doc-grid">
        <a href="docs/summary.md"><span>01</span><h3>项目总结</h3><p>最终状态、关键结果和停止决策。</p></a>
        <a href="docs/PROJECT_LAYERED_ARCHITECTURE.md"><span>02</span><h3>分层架构</h3><p>本地、AutoDL、云端执行与数据层关系。</p></a>
        <a href="docs/PROJECT_WORK_CYCLE.md"><span>03</span><h3>完整工作周期</h3><p>从环境构建到归档的端到端过程。</p></a>
        <a href="docs/PPO_TRAINING_RUNBOOK.md"><span>04</span><h3>PPO 训练记录</h3><p>critic、GAE、显存、checkpoint 与评测门禁。</p></a>
        <a href="docs/KLEAR_AGENTFORGE_REPRODUCIBILITY_GAP.md"><span>05</span><h3>复现差距审计</h3><p>公开信息、项目实现与论文口径的边界。</p></a>
        <a href="docs/PROJECT_README.md"><span>06</span><h3>项目 README</h3><p>源码入口、运行命令和历史状态。</p></a>
      </div>
    </section>

    <section class="section disclaimer">
      <div class="shell disclaimer-grid">
        <div><p class="section-kicker">PUBLICATION BOUNDARY</p><h2>这是实验档案，不是效果宣称</h2></div>
        <ul>
          <li>不包含 `.env`、密钥、证书、Registry 密码或运行时状态。</li>
          <li>最终 267 题数据已上传至独立的 <a href="https://huggingface.co/datasets/keryszhan/harbor-swesmith-rl-artifacts">Harbor SWE-Smith 强化学习数据集</a>，当前保持私有。</li>
          <li>PPO 策略模型 LoRA 已上传至独立的 <a href="https://huggingface.co/keryszhan/klear-agentforge-8b-ppo-step20-adapter">第 20 步适配器模型仓库</a>，当前保持私有；使用前需取得受访问门控基础模型的访问权限。</li>
          <li>云资源已释放，文档中的部署命令属于可复现的历史运行手册。</li>
        </ul>
      </div>
    </section>
  </main>

  <footer class="footer">
    <div class="shell"><strong>Harbor Qwen GRPO Experiment</strong><span>Evidence over anecdotes.</span></div>
  </footer>
</body>
</html>