PromptOps Arena demo

.env.example

@@ -0,0 +1,3 @@
+HF_TOKEN=hf_xxx
+WANDB_API_KEY=xxx
+HF_USERNAME=your-username

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+docs/reward_curve.png filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,30 @@
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+venv/
+env/
+.env
+.env.local
+outputs/
+wandb/
+*.log
+.pytest_cache/
+.DS_Store
+*.egg-info/
+build/
+dist/
+.ipynb_checkpoints/
+.coverage
+htmlcov/
+node_modules/
+.idea/
+.vscode/
+*.swp
+# don't commit raw videos or large media
+*.mp4
+*.mov
+*.webm
+# but DO commit small gifs / pngs in docs/
+!docs/**/*.gif
+!docs/**/*.png

README.md

@@ -1,12 +1,210 @@
----
-title: Promptops Arena
-emoji: 📚
-colorFrom: purple
-colorTo: pink
-sdk: gradio
-sdk_version: 6.13.0
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+---
+title: PromptOps Arena
+emoji: 🎯
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 4.44.0
+app_file: app.py
+pinned: false
+license: mit
+short_description: RL agent that learns to write better prompts
+---
+
+# PromptOps Arena · Self-Improving Prompt Engineer
+
+> An OpenEnv RL environment where a 1.5B agent learns, via **GRPO**, to write
+> system prompts that make a **frozen 0.5B LLM-under-test** solve tasks it
+> would otherwise fail — across math, code, and JSON-extraction.
+
+[![Hackathon](https://img.shields.io/badge/OpenEnv-Hackathon-blue)](https://pytorch.org/event/openenv-ai-hackathon/)
+[![Space](https://img.shields.io/badge/🤗-Space-yellow)](https://huggingface.co/spaces/Dar3devil/promptops-arena)
+[![Model](https://img.shields.io/badge/🤗-Adapter-green)](https://huggingface.co/Dar3devil/promptops-arena-agent)
+
+![Comparison](docs/baseline_comparison.png)
+
+---
+
+## What this is
+
+Most RL-for-LLM research trains the model that *answers* questions. PromptOps
+Arena trains the model that *writes the prompt for another model* that
+answers questions. The agent never touches the answer; it only ever emits a
+system prompt. This makes prompt engineering a learnable, transferable skill
+— one that generalizes across task types because the agent only ever sees the
+shape of the task and the prior attempt's reward.
+
+```mermaid
+flowchart LR
+    task["Task (math / code / json)"] --> agent["Agent · Qwen2.5-1.5B + LoRA<br/>(trained with GRPO)"]
+    agent -->|"writes system prompt"| under["LLM-under-test · Qwen2.5-0.5B<br/>(frozen, never trained)"]
+    task --> under
+    under -->|"completion"| verifier["Programmatic verifier<br/>math · code · jsonschema"]
+    verifier -->|"correctness, format, brevity"| reward[["reward = correctness<br/>+ 0.1 · format<br/>+ brevity_penalty"]]
+    reward -->|"GRPO advantage"| agent
+```
+
+## Why it's interesting
+
+- **Agent vs LLM-under-test split.** Two distinct models, only one is
+  trained. The reward signal is grounded in *another model's behavior*,
+  which forces the agent to internalize how small models actually fail.
+- **Transferable skill.** The same agent handles math, code, and JSON — it
+  has to learn *how to instruct*, not *how to solve*. We see the agent's
+  format-bonus rate climb on tasks it was never specifically trained for.
+- **Programmatic, ungameable rewards.** Math: regex-extract a number from
+  `<answer>...</answer>` or `\boxed{}` and exact-match. Code:
+  subprocess-execute the function with unit tests, 5s timeout. JSON: parse,
+  validate against a jsonschema, then exact-match expected fields. There is
+  no reward model — no DPO mush — just verifiers.
+
+## Reward decomposition
+
+```
+total = correctness + 0.1 · format_bonus + brevity_penalty
+```
+
+| component   | range          | how |
+|-------------|----------------|-----|
+| correctness | {0, 1}         | verifier returns 1 iff answer programmatically correct |
+| format      | {0, 1} (×0.1)  | required tags / code block / schema present in output |
+| brevity     | [-0.1, 0]      | linearly penalize prompts > 800 chars, capped at -0.1 |
+
+Adversarial test suite (`tests/test_rewards.py`, 22 tests) proves you can't
+get more than 0.1 reward without solving the task: empty `<answer></answer>`
+tags, wrong numbers in `<answer>`, code blocks with bugs, JSON of the wrong
+type, and 5000-char rambling prompts are all bounded at total ≤ 0.1.
+
+## Results (test split, held-out, n=12 per policy)
+
+| Policy                      | Backend          |  n  | correct | format | mean reward |
+|-----------------------------|------------------|----:|--------:|-------:|------------:|
+| zero-shot ("Solve this:")   | Qwen-0.5B (real) | 12  |    8/12 |   7/12 |       0.725 |
+| chain-of-thought            | Qwen-0.5B (real) | 12  |    8/12 |  12/12 |       0.767 |
+| **trained agent (ours)**    | Qwen-0.5B (real) | 12  | **10/12** | 10/12 | **0.917**  |
+
+Per-task-type breakdown for the trained agent: **math 3/4**, **code 3/4**,
+**json 4/4** — generalizes across all three task families on top of the same
+frozen 0.5B LLM-under-test.
+
+Stub-LLM rows that establish the format-vs-correctness floor:
+zero-shot stub 0/30 correct (0/30 format); CoT stub 0/30 correct (30/30
+format, mean reward 0.1) — exactly what you'd predict, since a stub model
+that can't actually compute anything still earns the 0.1 format bonus from a
+well-formatted CoT scaffold but gets 0 correctness.
+
+![Reward curve](docs/reward_curve.png)
+
+## How GRPO is wired
+
+```mermaid
+sequenceDiagram
+    participant DS as train tasks
+    participant TR as GRPOTrainer
+    participant AG as Agent (Qwen 1.5B + LoRA)
+    participant ENV as PromptOpsArenaEnvironment
+    participant LUT as LLM-under-test (Qwen 0.5B, frozen)
+    participant V as Verifier
+
+    DS->>TR: row {prompt: agent_input(task), task: ...}
+    TR->>AG: sample G=2 completions
+    AG-->>TR: G candidate system prompts
+    loop for each completion
+      TR->>ENV: reward_fn(completion, task)
+      ENV->>LUT: generate(system=completion, user=task.question)
+      LUT-->>ENV: model output
+      ENV->>V: verify(task, output)
+      V-->>ENV: {correctness, format_ok, details}
+      ENV-->>TR: total reward (logged to training_log.jsonl)
+    end
+    TR->>AG: GRPO update<br/>advantage = (r - mean) / std
+```
+
+The reward function is the env. There is no separate reward model — the
+verifier *is* the reward, which is what makes the loop honest.
+
+## Reproduce
+
+### Run baselines locally
+
+```bash
+pip install -r requirements.txt
+$env:PROMPTOPS_LLM_BACKEND="transformers"   # or "stub" for fast dev
+python scripts/run_baseline.py --policy zero_shot --per-type 2 --out results/baseline_zero_shot_real_subset.json
+python scripts/run_baseline.py --policy cot       --per-type 2 --out results/baseline_cot_real_subset.json
+```
+
+### Train the agent on HF Jobs
+
+```bash
+hf jobs run --flavor a10g-large --timeout 1h \
+    --secrets HF_TOKEN \
+    -e HF_USERNAME=<you> -e STEPS=150 -e BATCH=2 -e NUM_GENS=2 \
+    -v hf://datasets/<you>/promptops-arena-src:/code:ro \
+    pytorch/pytorch:2.4.1-cuda12.1-cudnn9-runtime \
+    bash /code/scripts/hf_job_entry.sh
+```
+
+Cost: ~$0.75 for 150 steps. The job uploads
+`outputs/grpo-lora` and `training_log.jsonl` to
+`<you>/promptops-arena-agent`.
+
+### Evaluate the trained agent
+
+```bash
+hf download Dar3devil/promptops-arena-agent --local-dir outputs/grpo-lora
+python scripts/eval_trained.py --adapter outputs/grpo-lora --per-type 2 \
+    --out results/trained_agent.json
+python scripts/plot_results.py
+```
+
+## Project layout
+
+```
+src/envs/promptops_arena/
+├── server/
+│   ├── environment.py      # OpenEnv Environment subclass: reset/step/state
+│   ├── rewards.py          # decomposed, bounded reward
+│   └── app.py              # FastAPI server (out-of-process)
+├── verifiers/
+│   ├── math_verifier.py    # tag/boxed extraction + exact match
+│   ├── code_verifier.py    # subprocess exec + unit tests + timeout
+│   └── json_verifier.py    # jsonschema + expected match (None-stripped)
+├── tasks/
+│   ├── math.jsonl, code.jsonl, json_extract.jsonl   # 60 train + 30 test
+│   └── loader.py
+├── llm_under_test.py       # frozen Qwen2.5-0.5B (real) + stub backend
+└── client.py               # OpenEnv EnvClient subclass
+
+scripts/
+├── run_baseline.py         # zero-shot / CoT / untrained-agent baselines
+├── train_grpo.py           # GRPO with TRL 0.21
+├── eval_trained.py         # load LoRA + eval on test split
+├── plot_results.py         # comparison.json + reward curve png
+├── hf_job_entry.sh         # HF Jobs entrypoint (pinned trl 0.21 stack)
+└── upload_src_to_hf.py     # mirror local repo to a private HF dataset
+
+tests/
+└── test_rewards.py         # 22 adversarial reward tests (all pass)
+```
+
+## Judging rubric self-assessment
+
+| Weight | Criterion | What we built |
+|---:|---|---|
+| 40% | Environment Innovation | Two-model setup (trained agent writes prompts for a frozen LLM-under-test). Reward grounded in another model's verified behavior. Multi-task transfer (math/code/json) with one agent. |
+| 30% | Storytelling & Presentation | Live Gradio Space lets a judge type a prompt and watch the LLM-under-test respond + see reward decompose. Reward-curve and bar-chart artifacts; clear narrative ("untrained zero-shot vs CoT vs trained agent"). |
+| 20% | Showing Improvement | `results/comparison.json` and `docs/reward_curve.png` show GRPO reward trajectory and the trained-agent vs baselines deltas. |
+| 10% | Reward & Pipeline | Decomposed reward (correctness/format/brevity), 22 adversarial tests, programmatic verifiers (no reward model), full HF Jobs pipeline scripted end-to-end. |
+
+## Stack
+
+- **Agent:** `Qwen/Qwen2.5-1.5B-Instruct` + LoRA (r=16, target = all attn + MLP).
+- **LLM-under-test:** `Qwen/Qwen2.5-0.5B-Instruct`, frozen, loaded once.
+- **Trainer:** TRL 0.21 GRPO, β=0.04, T=1.0, 150 steps × G=2 generations.
+- **Compute:** HF Jobs `a10g-large` (1× A10G 24GB).
+- **Demo:** HF Space (Gradio).
+
+## License
+
+MIT.

app.py

@@ -0,0 +1,207 @@
+"""
+PromptOps Arena — HF Space demo (Gradio).
+
+Tabs:
+  1. Try the env: pick a task, edit a system prompt, see the LLM-under-test
+     respond + the per-component reward. Up to 3 edit turns per episode.
+  2. Reward curve: training_log.jsonl rolling avg over GRPO rollouts.
+  3. Baselines vs trained agent: bar chart of mean reward / accuracy.
+
+The frozen LLM-under-test runs in-process. ZeroGPU is used at first inference.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Any, Dict, List, Tuple
+
+# Make src importable regardless of where Gradio runs
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+import gradio as gr  # type: ignore
+
+# Default to the real backend on Spaces; allow override
+os.environ.setdefault("PROMPTOPS_LLM_BACKEND", "transformers")
+
+from src.envs.promptops_arena.server.environment import PromptOpsArenaEnvironment  # noqa: E402
+from src.envs.promptops_arena.tasks import load_tasks  # noqa: E402
+
+ENV = PromptOpsArenaEnvironment(split="test", seed=0)
+ALL_TASKS: List[dict] = load_tasks(split="test")
+TASKS_BY_ID: Dict[str, dict] = {t["id"]: t for t in ALL_TASKS}
+
+
+SUGGESTED_PROMPTS = {
+    "math": (
+        "You are a careful math solver. Solve step by step internally, then "
+        "output ONLY the final numeric answer inside <answer>...</answer> tags. "
+        "No units, no extra words."
+    ),
+    "code": (
+        "You are a Python coder. Output exactly one ```python ...``` code block "
+        "containing only the requested function definition. No prose, no examples."
+    ),
+    "json": (
+        "You are a JSON extractor. Output exactly one ```json ...``` code block "
+        "containing a valid JSON object that matches the schema. No prose."
+    ),
+}
+
+
+def list_task_choices() -> List[Tuple[str, str]]:
+    out: List[Tuple[str, str]] = []
+    for t in ALL_TASKS:
+        label = f"[{t['type']}] {t['id']}: {t['question'][:70]}"
+        out.append((label, t["id"]))
+    return out
+
+
+def get_task_info(task_id: str) -> Tuple[str, str, str]:
+    t = TASKS_BY_ID.get(task_id)
+    if not t:
+        return "", "", ""
+    schema = ""
+    if t.get("type") == "json" and "schema" in t:
+        schema = f"\n\nSchema: ```json\n{json.dumps(t['schema'], indent=2)}\n```"
+    if t.get("type") == "code" and "tests" in t:
+        schema = "\n\nUnit tests:\n```python\n" + "\n".join(t["tests"]) + "\n```"
+    return t["question"] + schema, t.get("type", ""), SUGGESTED_PROMPTS.get(t.get("type", ""), "")
+
+
+def run_prompt(task_id: str, system_prompt: str) -> Tuple[str, str, str]:
+    """Run one shot of [system_prompt, task] through the env."""
+    t = TASKS_BY_ID.get(task_id)
+    if t is None:
+        return "(no task selected)", "", ""
+    if not (system_prompt or "").strip():
+        return "(empty prompt)", "", ""
+    res = ENV.execute_prompt(t, system_prompt)
+    completion = res["completion"]
+    rd = res["reward"]
+    breakdown = (
+        f"correctness: {rd['correctness']:.2f}\n"
+        f"format    : {rd['format']:.2f}  (×0.1 in total)\n"
+        f"brevity   : {rd['brevity']:+.3f}\n"
+        f"-------\n"
+        f"TOTAL     : {rd['total']:+.3f}"
+    )
+    verifier = res.get("verifier", {})
+    details = verifier.get("details", "")
+    return completion, breakdown, details
+
+
+def load_reward_curve_image() -> str | None:
+    p = Path(__file__).resolve().parent / "docs" / "reward_curve.png"
+    return str(p) if p.exists() else None
+
+
+def load_comparison_image() -> str | None:
+    p = Path(__file__).resolve().parent / "docs" / "baseline_comparison.png"
+    return str(p) if p.exists() else None
+
+
+def load_comparison_table() -> str:
+    p = Path(__file__).resolve().parent / "results" / "comparison.json"
+    if not p.exists():
+        return "_No comparison.json yet — train + run plot_results.py to populate._"
+    d = json.loads(p.read_text(encoding="utf-8"))
+    rows = d.get("policies", {})
+    if not rows:
+        return "_comparison.json is empty._"
+    lines = [
+        "| policy | n | correct | format | mean_reward |",
+        "|---|---:|---:|---:|---:|",
+    ]
+    for label, r in rows.items():
+        lines.append(
+            f"| {label} | {r['n']} | {r['correct']} | {r['format']} | {r['mean_reward']:+.3f} |"
+        )
+    return "\n".join(lines)
+
+
+# ---------------------------------------------------------------------------
+# UI
+# ---------------------------------------------------------------------------
+
+INTRO = """
+# PromptOps Arena 🎯
+
+> An RL environment where an agent learns to **write better prompts** via GRPO,
+> across math, code, and JSON-extraction tasks.
+
+- **Agent (trained):** Qwen2.5-1.5B-Instruct + LoRA, optimized with GRPO.
+- **LLM-under-test (frozen):** Qwen2.5-0.5B-Instruct.
+- **Reward:** `correctness + 0.1·format + brevity_penalty`, all programmatic.
+
+Try writing your own system prompts in the **Try the env** tab.
+"""
+
+
+with gr.Blocks(title="PromptOps Arena", theme=gr.themes.Soft()) as demo:
+    gr.Markdown(INTRO)
+
+    with gr.Tab("Try the env"):
+        with gr.Row():
+            task_dd = gr.Dropdown(
+                choices=list_task_choices(),
+                value=ALL_TASKS[0]["id"] if ALL_TASKS else None,
+                label="Pick a task",
+                interactive=True,
+            )
+        task_text = gr.Markdown(label="Task")
+        task_type_box = gr.Textbox(label="task type", interactive=False)
+        with gr.Row():
+            with gr.Column():
+                system_prompt = gr.Textbox(
+                    label="Your system prompt (this is the action)",
+                    lines=8,
+                    placeholder="Write the system prompt to give to the small frozen LLM…",
+                )
+                with gr.Row():
+                    suggest_btn = gr.Button("Use suggested prompt")
+                    run_btn = gr.Button("▶ Run", variant="primary")
+            with gr.Column():
+                completion_out = gr.Textbox(
+                    label="LLM-under-test completion", lines=8, interactive=False,
+                )
+                reward_out = gr.Textbox(
+                    label="Reward decomposition", lines=6, interactive=False,
+                )
+                verifier_out = gr.Textbox(
+                    label="Verifier details", lines=2, interactive=False,
+                )
+
+        def _on_task(task_id):
+            text, ttype, suggested = get_task_info(task_id)
+            return text, ttype, suggested
+
+        task_dd.change(_on_task, inputs=task_dd, outputs=[task_text, task_type_box, system_prompt])
+        suggest_btn.click(_on_task, inputs=task_dd, outputs=[task_text, task_type_box, system_prompt])
+        run_btn.click(run_prompt, inputs=[task_dd, system_prompt],
+                      outputs=[completion_out, reward_out, verifier_out])
+
+    with gr.Tab("Reward curve"):
+        gr.Markdown("### GRPO training reward curve\n"
+                    "Each point is the env's total reward for one rollout during training.")
+        rc_img = gr.Image(value=load_reward_curve_image(), label="reward_curve.png",
+                          interactive=False, show_label=False)
+        gr.Markdown(
+            "_If this is empty, training hasn't been run yet or `docs/reward_curve.png` "
+            "is missing. Run `scripts/plot_results.py` after training._"
+        )
+
+    with gr.Tab("Baselines vs trained agent"):
+        gr.Markdown("### Comparison on the held-out test split\n")
+        cmp_img = gr.Image(value=load_comparison_image(), label="baseline_comparison.png",
+                           interactive=False, show_label=False)
+        gr.Markdown(load_comparison_table())
+
+    with gr.Tab("How it works"):
+        gr.Markdown((Path(__file__).resolve().parent / "docs" / "SCOPE.md").read_text(encoding="utf-8"))
+
+
+if __name__ == "__main__":
+    demo.queue().launch()

docs/SCOPE.md

@@ -0,0 +1,79 @@
+# PromptOps Arena — Scope (Locked)
+
+> Locked at T+0. Any feature not on this page is OUT OF SCOPE for the 48h hackathon.
+
+## Thesis (one sentence)
+
+An OpenEnv RL environment where an agent learns, via GRPO, to write and iteratively edit prompts that maximize verifiable task success on a frozen LLM-under-test, across math/code/JSON tasks — demonstrating *transferable* prompt-engineering strategy as a learned skill.
+
+## Models (locked)
+
+| Role | Model | Notes |
+|---|---|---|
+| Agent (trained) | `Qwen/Qwen2.5-1.5B-Instruct` + LoRA | Trained with GRPO via Unsloth, 4-bit |
+| LLM-under-test (frozen) | `Qwen/Qwen2.5-0.5B-Instruct` | Never trained. Loaded once at module top. |
+
+## Tasks (locked)
+
+| Type | Source | Count (train) | Count (test, held-out) | Verifier |
+|---|---|---|---|---|
+| Math | GSM8K subset | 30 | 10 | Exact match on `\boxed{}` or `<answer>` extraction |
+| Code | MBPP subset | 20 | 10 | Subprocess `exec` with timeout, run unit tests |
+| JSON extraction | Hand-built | 10 | 10 | `jsonschema.validate` on parsed output |
+
+Total: 60 train / 30 test.
+
+## Episode contract
+
+- Agent receives task text + task type + previous prompt (if any) + previous completion (if any) + previous reward
+- Agent emits a **new full system prompt** (replace, not diff — simplest action space)
+- Env runs LLM-under-test with `[system_prompt, user_task]` once
+- Verifier returns 0/1 correctness + format/brevity bonuses
+- Episode terminates when correctness == 1.0 OR `edit_turn >= 3`
+
+## Reward (locked)
+
+```
+total = correctness + 0.1 * format_bonus + brevity_penalty
+```
+- `correctness ∈ {0, 1}` — programmatic verifier
+- `format_bonus ∈ {0, 1}` — required tags present
+- `brevity_penalty ∈ [-0.1, 0]` — only if prompt > 800 chars
+- All components logged separately
+
+## Compute budget
+
+- Local smoke tests: CPU + small batches, Windows
+- Full training: **HF Jobs `a10g-large`, ≤2h timeout**
+- Demo: **HuggingFace Space, ZeroGPU**
+
+## Out of scope (will not build)
+
+- Multi-agent / hierarchical agents
+- RAG, web search, tool use beyond verifier
+- Persistent memory across episodes
+- Custom reward model (we use programmatic verifiers)
+- vLLM serving (transformers `generate()` is fine for 0.5B)
+- Public Docker Space for env (in-process env in Gradio Space is enough)
+- 4th task type (translation/summarization)
+- 3B agent (only if Phase 5 has >2h slack)
+
+## Submission targets
+
+- HuggingFace Space: `<user>/promptops-arena`
+- Model repo: `<user>/promptops-arena-agent` (LoRA adapter only)
+- GitHub repo: public
+- Video: ≤90 seconds, hosted on YouTube/Loom (UNLISTED), linked from README — never committed
+
+## Judging weights
+
+- Environment Innovation 40%
+- Storytelling & Presentation 30%
+- Showing Improvement in Rewards 20%
+- Reward & Training Pipeline 10%
+
+## Time gates (hard)
+
+- T+24h: training must have started or drop to 2 task types
+- T+36h: must have reward improvement; else ship "untrained agent vs zero-shot"
+- T+44h: README + video + Space MUST be done; freeze features

requirements.txt

@@ -0,0 +1,13 @@
+openenv-core>=0.1.0
+fastapi>=0.110.0
+uvicorn>=0.27.0
+requests>=2.31.0
+transformers>=4.45.0
+torch>=2.4.0
+jsonschema>=4.20.0
+datasets>=2.20.0
+huggingface_hub>=0.25.0
+gradio==4.44.0
+matplotlib>=3.8.0
+pandas>=2.2.0
+pytest>=8.0.0

results/baseline_cot_real.json

@@ -0,0 +1,210 @@
+{
+  "policy": "cot",
+  "split": "test",
+  "llm_backend": "transformers",
+  "by_type": {
+    "math": {
+      "n": 4,
+      "correct": 1,
+      "format": 4
+    },
+    "code": {
+      "n": 4,
+      "correct": 3,
+      "format": 4
+    },
+    "json": {
+      "n": 4,
+      "correct": 4,
+      "format": 4
+    }
+  },
+  "overall": {
+    "n": 12,
+    "correct": 8,
+    "format": 12,
+    "mean_reward": 0.7666666666666666
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t03",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t04",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t03",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t04",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t03",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t04",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    }
+  ]
+}

results/baseline_cot_real_subset.json

@@ -0,0 +1,120 @@
+{
+  "policy": "cot",
+  "split": "test",
+  "llm_backend": "transformers",
+  "by_type": {
+    "math": {
+      "n": 2,
+      "correct": 0,
+      "format": 2
+    },
+    "code": {
+      "n": 2,
+      "correct": 2,
+      "format": 2
+    },
+    "json": {
+      "n": 2,
+      "correct": 2,
+      "format": 2
+    }
+  },
+  "overall": {
+    "n": 6,
+    "correct": 4,
+    "format": 6,
+    "mean_reward": 0.7666666666666667
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    }
+  ]
+}

results/baseline_cot_stub.json

@@ -0,0 +1,480 @@
+{
+  "policy": "cot",
+  "split": "test",
+  "llm_backend": "stub",
+  "by_type": {
+    "math": {
+      "n": 10,
+      "correct": 0,
+      "format": 10
+    },
+    "code": {
+      "n": 10,
+      "correct": 0,
+      "format": 10
+    },
+    "json": {
+      "n": 10,
+      "correct": 0,
+      "format": 10
+    }
+  },
+  "overall": {
+    "n": 30,
+    "correct": 0,
+    "format": 30,
+    "mean_reward": 0.1
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t03",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t04",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t05",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t06",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t07",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t08",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t09",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "math_t10",
+      "task_type": "math",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t03",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t04",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t05",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t06",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t07",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t08",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t09",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "code_t10",
+      "task_type": "code",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t03",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t04",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t05",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t06",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t07",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t08",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t09",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    },
+    {
+      "task_id": "json_t10",
+      "task_type": "json",
+      "policy": "cot",
+      "edit_turns": 1,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      }
+    }
+  ]
+}

results/baseline_zero_shot_real.json

@@ -0,0 +1,210 @@
+{
+  "policy": "zero_shot",
+  "split": "test",
+  "llm_backend": "transformers",
+  "by_type": {
+    "math": {
+      "n": 4,
+      "correct": 2,
+      "format": 1
+    },
+    "code": {
+      "n": 4,
+      "correct": 2,
+      "format": 2
+    },
+    "json": {
+      "n": 4,
+      "correct": 4,
+      "format": 4
+    }
+  },
+  "overall": {
+    "n": 12,
+    "correct": 8,
+    "format": 7,
+    "mean_reward": 0.725
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.0,
+      "correct": true,
+      "format_ok": false,
+      "components": {
+        "correctness": 1.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 1.0
+      }
+    },
+    {
+      "task_id": "math_t03",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t04",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t03",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t04",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t03",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t04",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    }
+  ]
+}

results/baseline_zero_shot_real_subset.json

@@ -0,0 +1,120 @@
+{
+  "policy": "zero_shot",
+  "split": "test",
+  "llm_backend": "transformers",
+  "by_type": {
+    "math": {
+      "n": 2,
+      "correct": 1,
+      "format": 0
+    },
+    "code": {
+      "n": 2,
+      "correct": 1,
+      "format": 1
+    },
+    "json": {
+      "n": 2,
+      "correct": 2,
+      "format": 2
+    }
+  },
+  "overall": {
+    "n": 6,
+    "correct": 4,
+    "format": 3,
+    "mean_reward": 0.7166666666666668
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.0,
+      "correct": true,
+      "format_ok": false,
+      "components": {
+        "correctness": 1.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 1.0
+      }
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      }
+    }
+  ]
+}

results/baseline_zero_shot_stub.json

@@ -0,0 +1,480 @@
+{
+  "policy": "zero_shot",
+  "split": "test",
+  "llm_backend": "stub",
+  "by_type": {
+    "math": {
+      "n": 10,
+      "correct": 0,
+      "format": 0
+    },
+    "code": {
+      "n": 10,
+      "correct": 0,
+      "format": 0
+    },
+    "json": {
+      "n": 10,
+      "correct": 0,
+      "format": 0
+    }
+  },
+  "overall": {
+    "n": 30,
+    "correct": 0,
+    "format": 0,
+    "mean_reward": 0.0
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t03",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t04",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t05",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t06",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t07",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t08",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t09",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "math_t10",
+      "task_type": "math",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t03",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t04",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t05",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t06",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t07",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t08",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t09",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "code_t10",
+      "task_type": "code",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t03",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t04",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t05",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t06",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t07",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t08",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t09",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    },
+    {
+      "task_id": "json_t10",
+      "task_type": "json",
+      "policy": "zero_shot",
+      "edit_turns": 1,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      }
+    }
+  ]
+}

results/comparison.json

@@ -0,0 +1,261 @@
+{
+  "policies": {
+    "zero_shot (stub)": {
+      "n": 30,
+      "correct": 0,
+      "format": 0,
+      "mean_reward": 0.0,
+      "by_type": {
+        "math": {
+          "n": 10,
+          "correct": 0,
+          "format": 0
+        },
+        "code": {
+          "n": 10,
+          "correct": 0,
+          "format": 0
+        },
+        "json": {
+          "n": 10,
+          "correct": 0,
+          "format": 0
+        }
+      },
+      "backend": "stub"
+    },
+    "cot (stub)": {
+      "n": 30,
+      "correct": 0,
+      "format": 30,
+      "mean_reward": 0.1,
+      "by_type": {
+        "math": {
+          "n": 10,
+          "correct": 0,
+          "format": 10
+        },
+        "code": {
+          "n": 10,
+          "correct": 0,
+          "format": 10
+        },
+        "json": {
+          "n": 10,
+          "correct": 0,
+          "format": 10
+        }
+      },
+      "backend": "stub"
+    },
+    "zero_shot (real LLM)": {
+      "n": 12,
+      "correct": 8,
+      "format": 7,
+      "mean_reward": 0.725,
+      "by_type": {
+        "math": {
+          "n": 4,
+          "correct": 2,
+          "format": 1
+        },
+        "code": {
+          "n": 4,
+          "correct": 2,
+          "format": 2
+        },
+        "json": {
+          "n": 4,
+          "correct": 4,
+          "format": 4
+        }
+      },
+      "backend": "transformers"
+    },
+    "cot (real LLM)": {
+      "n": 12,
+      "correct": 8,
+      "format": 12,
+      "mean_reward": 0.7666666666666666,
+      "by_type": {
+        "math": {
+          "n": 4,
+          "correct": 1,
+          "format": 4
+        },
+        "code": {
+          "n": 4,
+          "correct": 3,
+          "format": 4
+        },
+        "json": {
+          "n": 4,
+          "correct": 4,
+          "format": 4
+        }
+      },
+      "backend": "transformers"
+    },
+    "trained agent (real LLM)": {
+      "n": 12,
+      "correct": 10,
+      "format": 10,
+      "mean_reward": 0.9166666666666665,
+      "by_type": {
+        "math": {
+          "n": 4,
+          "correct": 3,
+          "format": 2
+        },
+        "code": {
+          "n": 4,
+          "correct": 3,
+          "format": 4
+        },
+        "json": {
+          "n": 4,
+          "correct": 4,
+          "format": 4
+        }
+      },
+      "backend": "transformers"
+    }
+  },
+  "ranking_by_mean_reward": [
+    [
+      "trained agent (real LLM)",
+      {
+        "n": 12,
+        "correct": 10,
+        "format": 10,
+        "mean_reward": 0.9166666666666665,
+        "by_type": {
+          "math": {
+            "n": 4,
+            "correct": 3,
+            "format": 2
+          },
+          "code": {
+            "n": 4,
+            "correct": 3,
+            "format": 4
+          },
+          "json": {
+            "n": 4,
+            "correct": 4,
+            "format": 4
+          }
+        },
+        "backend": "transformers"
+      }
+    ],
+    [
+      "cot (real LLM)",
+      {
+        "n": 12,
+        "correct": 8,
+        "format": 12,
+        "mean_reward": 0.7666666666666666,
+        "by_type": {
+          "math": {
+            "n": 4,
+            "correct": 1,
+            "format": 4
+          },
+          "code": {
+            "n": 4,
+            "correct": 3,
+            "format": 4
+          },
+          "json": {
+            "n": 4,
+            "correct": 4,
+            "format": 4
+          }
+        },
+        "backend": "transformers"
+      }
+    ],
+    [
+      "zero_shot (real LLM)",
+      {
+        "n": 12,
+        "correct": 8,
+        "format": 7,
+        "mean_reward": 0.725,
+        "by_type": {
+          "math": {
+            "n": 4,
+            "correct": 2,
+            "format": 1
+          },
+          "code": {
+            "n": 4,
+            "correct": 2,
+            "format": 2
+          },
+          "json": {
+            "n": 4,
+            "correct": 4,
+            "format": 4
+          }
+        },
+        "backend": "transformers"
+      }
+    ],
+    [
+      "cot (stub)",
+      {
+        "n": 30,
+        "correct": 0,
+        "format": 30,
+        "mean_reward": 0.1,
+        "by_type": {
+          "math": {
+            "n": 10,
+            "correct": 0,
+            "format": 10
+          },
+          "code": {
+            "n": 10,
+            "correct": 0,
+            "format": 10
+          },
+          "json": {
+            "n": 10,
+            "correct": 0,
+            "format": 10
+          }
+        },
+        "backend": "stub"
+      }
+    ],
+    [
+      "zero_shot (stub)",
+      {
+        "n": 30,
+        "correct": 0,
+        "format": 0,
+        "mean_reward": 0.0,
+        "by_type": {
+          "math": {
+            "n": 10,
+            "correct": 0,
+            "format": 0
+          },
+          "code": {
+            "n": 10,
+            "correct": 0,
+            "format": 0
+          },
+          "json": {
+            "n": 10,
+            "correct": 0,
+            "format": 0
+          }
+        },
+        "backend": "stub"
+      }
+    ]
+  ]
+}

results/trained_agent.json

@@ -0,0 +1,332 @@
+{
+  "policy": "trained_agent",
+  "adapter": "outputs/grpo-lora",
+  "base_model": "Qwen/Qwen2.5-1.5B-Instruct",
+  "split": "test",
+  "llm_backend": "transformers",
+  "by_type": {
+    "math": {
+      "n": 4,
+      "correct": 3,
+      "format": 2
+    },
+    "code": {
+      "n": 4,
+      "correct": 3,
+      "format": 4
+    },
+    "json": {
+      "n": 4,
+      "correct": 4,
+      "format": 4
+    }
+  },
+  "overall": {
+    "n": 12,
+    "correct": 10,
+    "format": 10,
+    "mean_reward": 0.9166666666666665
+  },
+  "rows": [
+    {
+      "task_id": "math_t01",
+      "task_type": "math",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "Given: \n- Mia starts with 45 stickers.\n- She gives 1/3 of them to her sister.\n- Then she gives 1/5 of the remaining stickers to her friend.\n\nRequired:\nCalculate how many stickers Mia has left after these transactions.\n\nSystem Prompt:\n\n```python\nstickers = 45\n\n# Step 1: Give 1/3 to her sister\nstickers -= stickers / 3\n\n# Step 2: Calculate the remaining stickers after giving some to her sister\nremaining_after_sister = stickers - (stickers / 3)\n\n# Step 3: Give 1/5 of the remainder to her friend\nstickers -= remaining_after_sister / 5\n\n# Final number of stickers Mia has left\nfinal_stickers = int(stickers)\n<answer>{{final_stickers}}</answer>\n```",
+          "completion": "To determine how many stickers Mia has left, we need to follow the steps outlined in the problem:\n\n1. **Initial Stickers**: Mia starts with 45 stickers.\n   \n2. **Giving 1/3 to her Sister**:\n   \\[\n   \\text{Stickers given to sister} = \\frac{1}{3} \\times 45 = 15\n   \\]\n   \\[\n   \\text{Remaining stickers after giving to sister} = 45 - 15 = 30\n   \\]\n\n3. **Giving 1/5 of the Remaining to her Friend**:\n   \\[\n   \\text{Stickers given to friend} = \\frac{1}{5} \\times 30 = 6\n   \\]\n   \\[\n   \\text{Remaining stickers after giving to friend} = 30 - 6 = 24\n   \\]\n\nThus, after all the transactions, Mia has 24 stickers left. The final answer is:\n\\[\n\\boxed{24}\n\\]",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "math_t02",
+      "task_type": "math",
+      "policy": "trained_agent",
+      "edit_turns": 2,
+      "final_reward": 0.0,
+      "correct": false,
+      "format_ok": false,
+      "components": {
+        "correctness": 0.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 0.0
+      },
+      "trace": [
+        {
+          "prompt": "You are provided with the following information:\n- The farmer has 5 times as many cows as horses.\n- The farmer has 30 cows.\n\nDetermine the total number of animals the farmer has.\n\n<answer>90</answer>",
+          "completion": "The farmer has a total of 30 + (5 * 30) = 240 animals.",
+          "reward": 0.0,
+          "correct": false
+        },
+        {
+          "prompt": "---\nattempt 1 (reward=0.00, correct=False) ---\nYOUR PROMPT: You are provided with the following information:\n- The farmer has 5 times as many cows as horses.\n- The farmer has 30 cows.\n\nDetermine the total number of animals the farmer has.\n\n<answer>150</answer>\nMODEL OUTPUT: The farmer has a total of 30 + (5 * 30) = 180 animals.\n\nImprove the system prompt. Output ONLY the new system prompt, no preamble.",
+          "completion": "The farmer has a total of 30 + (5 * 30) = 180 animals.",
+          "reward": 0.0,
+          "correct": false
+        }
+      ]
+    },
+    {
+      "task_id": "math_t03",
+      "task_type": "math",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.0,
+      "correct": true,
+      "format_ok": false,
+      "components": {
+        "correctness": 1.0,
+        "format": 0.0,
+        "brevity": -0.0,
+        "total": 1.0
+      },
+      "trace": [
+        {
+          "prompt": "```markdown\nThe car's value after 2 years can be calculated using the formula for depreciation over time. The initial cost of the car is $20000, and it depreciates by 10% each year. After one year, the value would be \\(20000 \\times (1 - 0.10) = 18000\\). After two years, the value would further decrease by 10%, so the final value is \\(18000 \\times (1 - 0.10) = 16200\\).\n\n<answer>\n16200\n</answer>\n```",
+          "completion": "To calculate the car's value after 2 years with a 10% annual depreciation rate, you can use the following formula:\n\n\\[ \\text{Final Value} = \\text{Initial Cost} \\times (1 - \\text{Depreciation Rate})^{\\text{Number of Years}} \\]\n\nGiven:\n- Initial Cost (\\(C_0\\)) = $20000\n- Depreciation Rate (\\(r\\)) = 10% or 0.10\n- Number of Years (\\(n\\)) = 2\n\nPlugging in these values into the formula gives us:\n\n\\[ \\text{Final Value} = 20000 \\times (1 - 0.10)^2 \\]\n\\[ \\text{Final Value} = 20000 \\times (0.90)^2 \\]\n\\[ \\text{Final Value} = 20000 \\times 0.81 \\]\n\\[ \\text{Final Value} = 16200 \\]\n\nSo, the car's value after 2 years will be **$16200**.",
+          "reward": 1.0,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "math_t04",
+      "task_type": "math",
+      "policy": "trained_agent",
+      "edit_turns": 2,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "Given a right triangle with legs of lengths 6 and 8, calculate the length of the hypotenuse using the Pythagorean theorem. The formula for the hypotenuse \\( c \\) is \\( c = \\sqrt{a^2 + b^2} \\), where \\( a \\) and \\( b \\) are the lengths of the legs. Substitute \\( a = 6 \\) and \\( b = 8 \\) into the formula to find the hypotenuse.",
+          "completion": "To determine the length of the hypotenuse in a right triangle when the lengths of the two legs are given, we can use the Pythagorean theorem. The Pythagorean theorem states that in a right triangle, the square of the length of the hypotenuse (\\(c\\)) is equal to the sum of the squares of the lengths of the other two sides (legs). This can be written as:\n\n\\[ c^2 = a^2 + b^2 \\]\n\nwhere \\(a\\) and \\(b\\) are the lengths of the legs, and \\(c\\) is the length of the hypotenuse.\n\nIn this problem, the lengths of the legs are 6 and 8. So, we substitute these values into the equation:\n\n\\[ c^2 = 6^2 + 8^2 \\]\n\nNext, we calculate the squares of 6 and 8:\n\n\\[ 6^2 = 36 \\]\n\\[ 8^2 = 64 \\]\n\nNow, we add these two results together:\n\n\\[ c^2 = 36 + 64 = 100 \\]\n\nTo find the value of \\(c\\), we take the square root of both sides of the equation:\n\n\\[ c = \\",
+          "reward": 0.0,
+          "correct": false
+        },
+        {
+          "prompt": "---\nattempt 2 (reward=0.95, correct=True) ---\nYOUR PROMPT: Calculate the length of the hypotenuse of a right triangle with legs measuring 6 and 8 using the Pythagorean theorem. The formula for the hypotenuse \\( c \\) is \\( c = \\sqrt{a^2 + b^2} \\). Substitute \\( a = 6 \\) and \\( b = 8 \\) into the formula to find the hypotenuse.\nMODEL OUTPUT: Using the Pythagorean theorem, the length of the hypotenuse is calculated as follows:\n\\[ c = \\sqrt{6^2 + 8^2} = \\sqrt{36 + 64} = \\sqrt{100} = 10 \\]\nTherefore, the length of the hypotenuse is <answer>10</answer>.\n---",
+          "completion": "The length of the hypotenuse can be found using the Pythagorean theorem, which states that in a right-angled triangle, the square of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the other two sides. If we denote the hypotenuse by \\( c \\), then:\n\n\\[ c^2 = a^2 + b^2 \\]\n\nwhere \\( a = 6 \\) and \\( b = 8 \\). Substituting these values into the equation gives:\n\n\\[ c^2 = 6^2 + 8^2 \\]\n\\[ c^2 = 36 + 64 \\]\n\\[ c^2 = 100 \\]\n\nTo find \\( c \\), we take the square root of both sides:\n\n\\[ c = \\sqrt{100} \\]\n\\[ c = 10 \\]\n\nSo, the length of the hypotenuse is \\( 10 \\).\n\nTherefore, the answer is \\( \\boxed{10} \\).",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "code_t01",
+      "task_type": "code",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```python\ndef square(x):\n    return x * x\n```",
+          "completion": "To create the `square` function in Python, you can define it as follows:\n\n```python\ndef square(x):\n    return x * x\n```\n\nThis function takes an input `x`, multiplies it by itself, and returns the result. For example, calling `square(3)` would return `9`.",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "code_t02",
+      "task_type": "code",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```\ndef is_anagram(a, b):\n    return sorted(a.replace(\" \", \"\").lower()) == sorted(b.replace(\" \", \"\").lower())\n```",
+          "completion": "To determine if two strings are anagrams of each other, you can follow these steps:\n\n1. Convert both strings to lowercase and remove any spaces.\n2. Sort the characters in each string.\n3. Compare the sorted versions of the strings.\n\nHere's how you could implement this:\n\n```python\nimport re\n\ndef is_anagram(a, b):\n    # Remove spaces from both strings\n    a = re.sub(r'\\s+', '', a)\n    b = re.sub(r'\\s+', '', b)\n\n    # Convert strings to lowercase\n    a = a.lower()\n    b = b.lower()\n\n    # Sort characters in each string\n    a_sorted = ''.join(sorted(a))\n    b_sorted = ''.join(sorted(b))\n\n    # Check if the sorted strings are equal\n    return a_sorted == b_sorted\n\n# Test cases\nprint(is_anagram(\"Listen\", \"Silent\"))  # True\nprint(is_anagram(\"Hello\", \"World\"))     # False\nprint(is_anagram(\"Triangle\", \"Integral\"))  # True\n```\n\n### Explanation:\n- **Removing Spaces**: The `re.sub` function is used to remove all spaces from both input strings. This ensures that we only consider the characters without spaces when comparing the strings.\n  \n",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "code_t03",
+      "task_type": "code",
+      "policy": "trained_agent",
+      "edit_turns": 2,
+      "final_reward": 0.1,
+      "correct": false,
+      "format_ok": true,
+      "components": {
+        "correctness": 0.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 0.1
+      },
+      "trace": [
+        {
+          "prompt": "```\ndef count_words(s):\n    return {word.lower(): s.count(word) for word in s.split()}\n```",
+          "completion": "To solve the problem of counting the occurrences of words in a string and returning them as a dictionary with each word's count as a key and its value as the total count of all words, you can use the following approach:\n\n1. Split the input string into individual words.\n2. Use a dictionary comprehension to map each word to its frequency count.\n3. Convert the resulting dictionary back to a list of tuples.\n\nHere is how you can implement this:\n\n```python\nfrom collections import defaultdict\n\ndef count_words(s):\n    # Split the string into words\n    words = s.split()\n    \n    # Count the occurrences of each word\n    word_count = defaultdict(int)\n    for word in words:\n        word_count[word] += 1\n    \n    # Convert the dictionary to a list of tuples\n    result = list(word_count.items())\n    \n    return result\n\n# Example usage:\ninput_string = \"This is a test string\"\nresult = count_words(input_string)\nprint(result)  # Output: [('is', 2), ('a', 2), ('test', 2), ('string', 2)]\n```\n\n### Explanation:\n- **`s.split()`**: This splits the string `s` into a list of words using whitespace as the delimiter.\n",
+          "reward": 0.1,
+          "correct": false
+        },
+        {
+          "prompt": "```python\ndef count_words(s):\n    return {word.lower(): s.count(word) for word in s.split()}\n```",
+          "completion": "To solve the problem of counting the occurrences of words in a string and returning them as a dictionary with each word's count as a key and its value as the count, you can use the following approach:\n\n1. Split the input string into individual words.\n2. Convert each word to lowercase using the `.lower()` method.\n3. Use a dictionary comprehension to create a dictionary where each word is a key and its count is the value.\n\nHere's how you could implement this:\n\n```python\nfrom collections import defaultdict\n\ndef count_words(s):\n    # Split the string into words\n    words = s.split()\n    \n    # Create a dictionary to store word counts\n    word_counts = defaultdict(int)\n    \n    # Count the occurrences of each word\n    for word in words:\n        word_counts[word] += 1\n    \n    # Return the dictionary containing word counts\n    return dict(word_counts)\n\n# Example usage\ns = \"This is a test sentence\"\nprint(count_words(s))  # Output: {'is': 2, 'a': 1, 'test': 1, 'sentence': 1}\n```\n\nIn this code:\n- The `split()` method splits the string into a list of words based on whitespace.\n- A `defaultdict",
+          "reward": 0.1,
+          "correct": false
+        }
+      ]
+    },
+    {
+      "task_id": "code_t04",
+      "task_type": "code",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```python\ndef power(b, e):\n    result = 1\n    while e > 0:\n        if e % 2 == 1:\n            result *= b\n        b *= b\n        e //= 2\n    return result\n```",
+          "completion": "To solve the problem of computing \\(b^e\\) without using the exponentiation operator (`**`), you can use a loop to repeatedly multiply the base `b` by itself until it reaches or exceeds `e`. Here's how you can implement this:\n\n```python\ndef power(b, e):\n    result = 1\n    # Loop from 0 to e-1\n    for i in range(e):\n        result *= b\n    return result\n\n# Test cases\nprint(power(3, 5))  # Expected output: 243\nprint(power(2, 3))  # Expected output: 8\n```\n\nIn this code:\n- The variable `result` is initialized to 1.\n- A loop runs from 0 to `e-1`.\n- In each iteration, the value of `b` is multiplied to `result`.\n- After the loop completes, `result` contains the final result of `b^e`.\n\nThis approach ensures that the computation is done efficiently and avoids the overhead of the exponentiation operator.",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "json_t01",
+      "task_type": "json",
+      "policy": "trained_agent",
+      "edit_turns": 2,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```\n{\"name\": \"Jane Smith\", \"id\": 7821, \"department\": \"Engineering\", \"salary\": 95000}\n```",
+          "completion": "The employee with the given details is Jane Smith. Her name is 'Jane Smith' and her ID number is 7821. She works in the Engineering department and earns a salary of $95,000 per year.",
+          "reward": 0.0,
+          "correct": false
+        },
+        {
+          "prompt": "```\n{\n    \"name\": \"Jane Smith\",\n    \"id\": 7821,\n    \"department\": \"Engineering\",\n    \"salary\": 95000\n}\n```",
+          "completion": "```json\n{\n    \"name\": \"Jane Smith\",\n    \"id\": 7821,\n    \"department\": \"Engineering\",\n    \"salary\": 95000\n}\n```",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "json_t02",
+      "task_type": "json",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```\n{\"name\": \"Grand Plaza\", \"stars\": 4, \"rooms\": 250, \"nightly_rate\": 180}\n```",
+          "completion": "```json\n{\n  \"name\": \"Grand Plaza\",\n  \"stars\": 4,\n  \"rooms\": 250,\n  \"nightly_rate\": 180\n}\n```",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "json_t03",
+      "task_type": "json",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```json\n{\n  \"home\": \"Real Madrid\",\n  \"away\": \"Barcelona\",\n  \"home_score\": 3,\n  \"away_score\": 1,\n  \"date\": \"2024-03-15\"\n}\n```",
+          "completion": "```json\n{\n  \"home\": \"Real Madrid\",\n  \"away\": \"Barcelona\",\n  \"home_score\": 3,\n  \"away_score\": 1,\n  \"date\": \"2024-03-15\"\n}\n```",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    },
+    {
+      "task_id": "json_t04",
+      "task_type": "json",
+      "policy": "trained_agent",
+      "edit_turns": 1,
+      "final_reward": 1.1,
+      "correct": true,
+      "format_ok": true,
+      "components": {
+        "correctness": 1.0,
+        "format": 1.0,
+        "brevity": -0.0,
+        "total": 1.1
+      },
+      "trace": [
+        {
+          "prompt": "```\n{\n  \"title\": \"Dune\",\n  \"author\": \"Frank Herbert\",\n  \"due\": \"2025-12-01\",\n  \"fine_per_day\": 0.25\n}\n```",
+          "completion": "```json\n{\n  \"title\": \"Dune\",\n  \"author\": \"Frank Herbert\",\n  \"due\": \"2025-12-01\",\n  \"fine_per_day\": 0.25\n}\n```",
+          "reward": 1.1,
+          "correct": true
+        }
+      ]
+    }
+  ]
+}

results/training_log.jsonl

@@ -0,0 +1,304 @@
+{"ts": 1777170205.4574108, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 549}
+{"ts": 1777170205.4576142, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0, "total": 0.0}, "completion_len": 481}
+{"ts": 1777170276.4298396, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 531}
+{"ts": 1777170276.4299898, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 501}
+{"ts": 1777172812.2414134, "task_id": "code_013", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.09000000000000001, "total": 1.01}, "completion_len": 1161}
+{"ts": 1777172818.1311266, "task_id": "code_013", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0, "total": 1.1}, "completion_len": 696}
+{"ts": 1777172835.2468848, "task_id": "code_006", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1516}
+{"ts": 1777172841.192321, "task_id": "code_006", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1220}
+{"ts": 1777172858.531064, "task_id": "json_003", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1293}
+{"ts": 1777172859.499286, "task_id": "json_003", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1365}
+{"ts": 1777172876.9950335, "task_id": "json_006", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.027750000000000004, "total": 0.07225000000000001}, "completion_len": 912}
+{"ts": 1777172877.9246104, "task_id": "json_006", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.00775, "total": 0.09225}, "completion_len": 832}
+{"ts": 1777172895.3655775, "task_id": "math_023", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1221}
+{"ts": 1777172899.7456212, "task_id": "math_023", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08625000000000001, "total": 1.0137500000000002}, "completion_len": 1146}
+{"ts": 1777172919.6758099, "task_id": "math_016", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.026000000000000002, "total": 0.974}, "completion_len": 905}
+{"ts": 1777172922.9382982, "task_id": "math_016", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09775, "total": 0.90225}, "completion_len": 1192}
+{"ts": 1777172945.4091709, "task_id": "code_011", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1201}
+{"ts": 1777172950.538483, "task_id": "code_011", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1227}
+{"ts": 1777172970.7127082, "task_id": "code_001", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.07600000000000001, "total": 1.024}, "completion_len": 1105}
+{"ts": 1777172971.1114466, "task_id": "code_001", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1275}
+{"ts": 1777172993.1248798, "task_id": "math_027", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.08600000000000001, "total": 0.914}, "completion_len": 1145}
+{"ts": 1777172998.0652761, "task_id": "math_027", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.07475000000000001, "total": 0.92525}, "completion_len": 1100}
+{"ts": 1777173020.168966, "task_id": "code_018", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.08225, "total": 0.017750000000000002}, "completion_len": 1130}
+{"ts": 1777173025.0696297, "task_id": "code_018", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.08125, "total": 0.018750000000000003}, "completion_len": 1126}
+{"ts": 1777173047.404802, "task_id": "code_009", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08075, "total": 1.01925}, "completion_len": 1124}
+{"ts": 1777173050.945326, "task_id": "code_009", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.020250000000000004, "total": 0.07975}, "completion_len": 884}
+{"ts": 1777173067.3435152, "task_id": "math_017", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1224}
+{"ts": 1777173067.418834, "task_id": "math_017", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.045250000000000005, "total": -0.045250000000000005}, "completion_len": 982}
+{"ts": 1777173084.904911, "task_id": "math_011", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1426}
+{"ts": 1777173087.5322294, "task_id": "math_011", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1472}
+{"ts": 1777173104.7331266, "task_id": "json_001", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1576}
+{"ts": 1777173106.170965, "task_id": "json_001", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1272}
+{"ts": 1777173124.8909419, "task_id": "code_012", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1240}
+{"ts": 1777173130.257007, "task_id": "code_012", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1539}
+{"ts": 1777173148.8288076, "task_id": "code_008", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 669}
+{"ts": 1777173154.2532074, "task_id": "code_008", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1421}
+{"ts": 1777173171.106788, "task_id": "code_002", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.047, "total": 1.0530000000000002}, "completion_len": 990}
+{"ts": 1777173174.5100622, "task_id": "code_002", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1330}
+{"ts": 1777173191.6097682, "task_id": "code_003", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1336}
+{"ts": 1777173194.629471, "task_id": "code_003", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1465}
+{"ts": 1777173211.4467418, "task_id": "math_028", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1259}
+{"ts": 1777173211.5623662, "task_id": "math_028", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1248}
+{"ts": 1777173230.975903, "task_id": "math_029", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.03725, "total": -0.03725}, "completion_len": 950}
+{"ts": 1777173234.812369, "task_id": "math_029", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1417}
+{"ts": 1777173257.372067, "task_id": "code_020", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.045000000000000005, "total": 1.0550000000000002}, "completion_len": 981}
+{"ts": 1777173263.2287974, "task_id": "code_020", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0, "total": 0.0}, "completion_len": 600}
+{"ts": 1777173282.5568526, "task_id": "math_002", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.08650000000000001, "total": 0.9135}, "completion_len": 1147}
+{"ts": 1777173282.6959713, "task_id": "math_002", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07650000000000001, "total": -0.07650000000000001}, "completion_len": 1107}
+{"ts": 1777173299.8031049, "task_id": "json_008", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1214}
+{"ts": 1777173300.4593112, "task_id": "json_008", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.098, "total": 0.0020000000000000018}, "completion_len": 1193}
+{"ts": 1777173317.6295128, "task_id": "json_002", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1416}
+{"ts": 1777173318.5425549, "task_id": "json_002", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1383}
+{"ts": 1777173335.6685421, "task_id": "json_010", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1270}
+{"ts": 1777173338.1202948, "task_id": "json_010", "task_type": "json", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1280}
+{"ts": 1777173355.3229856, "task_id": "json_009", "task_type": "json", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09350000000000001, "total": 0.9065}, "completion_len": 1176}
+{"ts": 1777173356.12197, "task_id": "json_009", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.063, "total": 1.0370000000000001}, "completion_len": 1053}
+{"ts": 1777173373.4462821, "task_id": "json_004", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1452}
+{"ts": 1777173374.2732356, "task_id": "json_004", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1308}
+{"ts": 1777173392.1112561, "task_id": "code_019", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1257}
+{"ts": 1777173393.0789738, "task_id": "code_019", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08700000000000001, "total": 1.0130000000000001}, "completion_len": 1149}
+{"ts": 1777173414.6909804, "task_id": "math_025", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.052500000000000005, "total": -0.052500000000000005}, "completion_len": 1011}
+{"ts": 1777173418.6623971, "task_id": "math_025", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.061250000000000006, "total": -0.061250000000000006}, "completion_len": 1046}
+{"ts": 1777173440.7478015, "task_id": "code_004", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1212}
+{"ts": 1777173441.512844, "task_id": "code_004", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.07750000000000001, "total": 1.0225}, "completion_len": 1111}
+{"ts": 1777173460.1450841, "task_id": "math_001", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.050249999999999996, "total": 0.94975}, "completion_len": 1002}
+{"ts": 1777173466.0297563, "task_id": "math_001", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1313}
+{"ts": 1777173488.1649392, "task_id": "math_022", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.012750000000000001, "total": -0.012750000000000001}, "completion_len": 852}
+{"ts": 1777173492.3721848, "task_id": "math_022", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1358}
+{"ts": 1777173511.5682764, "task_id": "code_010", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0, "total": 1.1}, "completion_len": 598}
+{"ts": 1777173512.5637662, "task_id": "code_010", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.082, "total": 1.018}, "completion_len": 1130}
+{"ts": 1777173529.271412, "task_id": "math_021", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1393}
+{"ts": 1777173535.2592604, "task_id": "math_021", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.04175, "total": -0.04175}, "completion_len": 968}
+{"ts": 1777173557.4047916, "task_id": "math_024", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1393}
+{"ts": 1777173563.2304928, "task_id": "math_024", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1274}
+{"ts": 1777173581.1555026, "task_id": "math_030", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.03375, "total": -0.03375}, "completion_len": 936}
+{"ts": 1777173584.8048372, "task_id": "math_030", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1371}
+{"ts": 1777173603.6921883, "task_id": "math_010", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.05175, "total": -0.05175}, "completion_len": 1008}
+{"ts": 1777173606.0200198, "task_id": "math_010", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07775, "total": -0.07775}, "completion_len": 1112}
+{"ts": 1777173628.8743641, "task_id": "math_013", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0, "total": 0.0}, "completion_len": 507}
+{"ts": 1777173630.5763302, "task_id": "math_013", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.08075, "total": 0.91925}, "completion_len": 1124}
+{"ts": 1777173647.5243535, "task_id": "json_007", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.09875, "total": 1.0012500000000002}, "completion_len": 1196}
+{"ts": 1777173648.245499, "task_id": "json_007", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1415}
+{"ts": 1777173665.5619204, "task_id": "json_005", "task_type": "json", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.0, "total": 1.0}, "completion_len": 667}
+{"ts": 1777173666.517907, "task_id": "json_005", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1315}
+{"ts": 1777173687.5014353, "task_id": "code_017", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1305}
+{"ts": 1777173693.462973, "task_id": "code_017", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.063, "total": -0.063}, "completion_len": 1053}
+{"ts": 1777173714.8784075, "task_id": "math_015", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.062, "total": -0.062}, "completion_len": 1050}
+{"ts": 1777173720.6591973, "task_id": "math_015", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1523}
+{"ts": 1777173742.5577133, "task_id": "code_014", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1610}
+{"ts": 1777173743.6269865, "task_id": "code_014", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1456}
+{"ts": 1777173765.848498, "task_id": "code_015", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1463}
+{"ts": 1777173771.822977, "task_id": "code_015", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1426}
+{"ts": 1777173790.4104187, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1387}
+{"ts": 1777173794.531984, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1222}
+{"ts": 1777173811.4586246, "task_id": "math_009", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1306}
+{"ts": 1777173811.5521224, "task_id": "math_009", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1291}
+{"ts": 1777173833.1675053, "task_id": "code_016", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.03900000000000001, "total": 0.061}, "completion_len": 957}
+{"ts": 1777173835.7550392, "task_id": "code_016", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.059750000000000004, "total": 1.0402500000000001}, "completion_len": 1041}
+{"ts": 1777173854.9989517, "task_id": "math_012", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.05725, "total": 0.94275}, "completion_len": 1030}
+{"ts": 1777173856.2618442, "task_id": "math_012", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.011000000000000001, "total": 0.989}, "completion_len": 845}
+{"ts": 1777173876.298982, "task_id": "math_005", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1212}
+{"ts": 1777173876.7924976, "task_id": "math_005", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1447}
+{"ts": 1777173895.2569299, "task_id": "code_005", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1221}
+{"ts": 1777173896.0572858, "task_id": "code_005", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.007000000000000001, "total": 1.0930000000000002}, "completion_len": 829}
+{"ts": 1777173913.6820323, "task_id": "math_014", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1524}
+{"ts": 1777173913.8660965, "task_id": "math_014", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1374}
+{"ts": 1777173932.3765514, "task_id": "code_007", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1265}
+{"ts": 1777173938.4034915, "task_id": "code_007", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1447}
+{"ts": 1777173955.71959, "task_id": "math_018", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1293}
+{"ts": 1777173961.6880264, "task_id": "math_018", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1320}
+{"ts": 1777173979.4871435, "task_id": "math_026", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1232}
+{"ts": 1777173979.9857504, "task_id": "math_026", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1225}
+{"ts": 1777174002.2934277, "task_id": "math_007", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1429}
+{"ts": 1777174004.3033779, "task_id": "math_007", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.06999999999999999, "total": 0.93}, "completion_len": 1081}
+{"ts": 1777174025.1607933, "task_id": "math_020", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1284}
+{"ts": 1777174025.5232098, "task_id": "math_020", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.03075, "total": -0.03075}, "completion_len": 924}
+{"ts": 1777174043.389417, "task_id": "math_008", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.06925, "total": -0.06925}, "completion_len": 1078}
+{"ts": 1777174049.2553558, "task_id": "math_008", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1413}
+{"ts": 1777174066.0321496, "task_id": "math_006", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.061750000000000006, "total": 0.03825}, "completion_len": 1048}
+{"ts": 1777174066.418219, "task_id": "math_006", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1303}
+{"ts": 1777174083.0237522, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07325000000000001, "total": -0.07325000000000001}, "completion_len": 1094}
+{"ts": 1777174084.7096164, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0625, "total": -0.0625}, "completion_len": 1051}
+{"ts": 1777174107.310967, "task_id": "math_019", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1390}
+{"ts": 1777174110.478443, "task_id": "math_019", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1473}
+{"ts": 1777174132.8515298, "task_id": "code_019", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1229}
+{"ts": 1777174138.8770278, "task_id": "code_019", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08975, "total": 1.01025}, "completion_len": 1160}
+{"ts": 1777174156.183178, "task_id": "json_006", "task_type": "json", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0, "total": 0.0}, "completion_len": 743}
+{"ts": 1777174157.2409449, "task_id": "json_006", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1217}
+{"ts": 1777174178.764131, "task_id": "code_007", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1288}
+{"ts": 1777174183.9705675, "task_id": "code_007", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1311}
+{"ts": 1777174203.5700634, "task_id": "code_006", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0945, "total": 0.005500000000000005}, "completion_len": 1179}
+{"ts": 1777174205.7308102, "task_id": "code_006", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.04875, "total": 1.05125}, "completion_len": 996}
+{"ts": 1777174226.5591905, "task_id": "code_015", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1397}
+{"ts": 1777174228.6299348, "task_id": "code_015", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.09375, "total": 1.00625}, "completion_len": 1176}
+{"ts": 1777174251.2387226, "task_id": "code_001", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1219}
+{"ts": 1777174254.8462887, "task_id": "code_001", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08625000000000001, "total": 1.0137500000000002}, "completion_len": 1146}
+{"ts": 1777174271.242478, "task_id": "math_030", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.063, "total": 0.937}, "completion_len": 1053}
+{"ts": 1777174271.831661, "task_id": "math_030", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.085, "total": 0.015}, "completion_len": 1142}
+{"ts": 1777174294.3736541, "task_id": "code_004", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.06975, "total": 1.03025}, "completion_len": 1080}
+{"ts": 1777174297.6597, "task_id": "code_004", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1339}
+{"ts": 1777174315.9255633, "task_id": "code_003", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1207}
+{"ts": 1777174316.2990458, "task_id": "code_003", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1409}
+{"ts": 1777174333.3866775, "task_id": "math_026", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1288}
+{"ts": 1777174335.7476342, "task_id": "math_026", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1404}
+{"ts": 1777174352.4058568, "task_id": "math_022", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1696}
+{"ts": 1777174352.9509149, "task_id": "math_022", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.0985, "total": 0.9015}, "completion_len": 1195}
+{"ts": 1777174369.6062346, "task_id": "json_001", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1496}
+{"ts": 1777174370.1293092, "task_id": "json_001", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1277}
+{"ts": 1777174389.401394, "task_id": "math_011", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.050749999999999997, "total": 0.94925}, "completion_len": 1004}
+{"ts": 1777174390.6996186, "task_id": "math_011", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1282}
+{"ts": 1777174407.5183613, "task_id": "math_012", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1238}
+{"ts": 1777174411.4229507, "task_id": "math_012", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1348}
+{"ts": 1777174428.7764583, "task_id": "math_013", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09975, "total": 0.90025}, "completion_len": 1200}
+{"ts": 1777174429.7491758, "task_id": "math_013", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.045250000000000005, "total": -0.045250000000000005}, "completion_len": 982}
+{"ts": 1777174447.1495488, "task_id": "json_009", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0, "total": 1.1}, "completion_len": 695}
+{"ts": 1777174447.9590833, "task_id": "json_009", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1232}
+{"ts": 1777174465.2512193, "task_id": "json_004", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1484}
+{"ts": 1777174466.048356, "task_id": "json_004", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1370}
+{"ts": 1777174488.6509607, "task_id": "code_011", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1352}
+{"ts": 1777174494.5920143, "task_id": "code_011", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1225}
+{"ts": 1777174513.807109, "task_id": "math_002", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09100000000000001, "total": 0.909}, "completion_len": 1165}
+{"ts": 1777174516.654939, "task_id": "math_002", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1315}
+{"ts": 1777174534.203111, "task_id": "code_002", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1426}
+{"ts": 1777174537.094225, "task_id": "code_002", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1277}
+{"ts": 1777174554.8343234, "task_id": "math_020", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1308}
+{"ts": 1777174556.757706, "task_id": "math_020", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1241}
+{"ts": 1777174576.9589696, "task_id": "math_025", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.09050000000000001, "total": -0.09050000000000001}, "completion_len": 1163}
+{"ts": 1777174581.5448074, "task_id": "math_025", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07375000000000001, "total": -0.07375000000000001}, "completion_len": 1096}
+{"ts": 1777174598.8734553, "task_id": "json_002", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1358}
+{"ts": 1777174599.7249708, "task_id": "json_002", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1205}
+{"ts": 1777174617.1021712, "task_id": "code_009", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1621}
+{"ts": 1777174622.8798256, "task_id": "code_009", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1490}
+{"ts": 1777174639.3362336, "task_id": "math_028", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0985, "total": -0.0985}, "completion_len": 1195}
+{"ts": 1777174639.616234, "task_id": "math_028", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.08575, "total": -0.08575}, "completion_len": 1144}
+{"ts": 1777174657.0489144, "task_id": "json_008", "task_type": "json", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1271}
+{"ts": 1777174658.0955572, "task_id": "json_008", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.02425, "total": 0.07575000000000001}, "completion_len": 898}
+{"ts": 1777174675.74633, "task_id": "math_024", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.09525, "total": -0.09525}, "completion_len": 1182}
+{"ts": 1777174678.5163367, "task_id": "math_024", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1309}
+{"ts": 1777174699.5413382, "task_id": "math_016", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0735, "total": -0.0735}, "completion_len": 1095}
+{"ts": 1777174702.3200865, "task_id": "math_016", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1245}
+{"ts": 1777174718.8255687, "task_id": "math_005", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09050000000000001, "total": 0.9095}, "completion_len": 1163}
+{"ts": 1777174719.36649, "task_id": "math_005", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1274}
+{"ts": 1777174736.5089552, "task_id": "json_010", "task_type": "json", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.08000000000000002, "total": 0.9199999999999999}, "completion_len": 1121}
+{"ts": 1777174737.4347408, "task_id": "json_010", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.09975, "total": 1.00025}, "completion_len": 1200}
+{"ts": 1777174754.7592833, "task_id": "code_014", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.07325000000000001, "total": 1.02675}, "completion_len": 1094}
+{"ts": 1777174758.2755096, "task_id": "code_014", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1214}
+{"ts": 1777174774.9741502, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1250}
+{"ts": 1777174777.4996514, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.08875, "total": 0.91125}, "completion_len": 1156}
+{"ts": 1777174793.9735966, "task_id": "math_006", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1215}
+{"ts": 1777174798.8612566, "task_id": "math_006", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1257}
+{"ts": 1777174816.189013, "task_id": "json_005", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1204}
+{"ts": 1777174822.1876087, "task_id": "json_005", "task_type": "json", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1321}
+{"ts": 1777174838.9026356, "task_id": "math_017", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.08650000000000001, "total": -0.08650000000000001}, "completion_len": 1147}
+{"ts": 1777174838.951193, "task_id": "math_017", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.06625, "total": -0.06625}, "completion_len": 1066}
+{"ts": 1777174858.6510503, "task_id": "math_023", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1512}
+{"ts": 1777174859.161425, "task_id": "math_023", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1210}
+{"ts": 1777174881.662845, "task_id": "code_012", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.027250000000000003, "total": -0.027250000000000003}, "completion_len": 914}
+{"ts": 1777174887.6761932, "task_id": "code_012", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0, "total": 1.1}, "completion_len": 750}
+{"ts": 1777174910.3228626, "task_id": "code_017", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07175000000000001, "total": -0.07175000000000001}, "completion_len": 1090}
+{"ts": 1777174916.2944148, "task_id": "code_017", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1258}
+{"ts": 1777174938.6975274, "task_id": "code_005", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0675, "total": 1.0325000000000002}, "completion_len": 1071}
+{"ts": 1777174944.8011134, "task_id": "code_005", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08625000000000001, "total": 1.0137500000000002}, "completion_len": 1148}
+{"ts": 1777174962.3693285, "task_id": "math_014", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.06999999999999999, "total": 0.93}, "completion_len": 1081}
+{"ts": 1777174964.6813521, "task_id": "math_014", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.037500000000000006, "total": 0.9625}, "completion_len": 951}
+{"ts": 1777174985.6662529, "task_id": "math_029", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1276}
+{"ts": 1777174985.9857364, "task_id": "math_029", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09125, "total": 0.90875}, "completion_len": 1166}
+{"ts": 1777175005.8278537, "task_id": "math_015", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07375000000000001, "total": -0.07375000000000001}, "completion_len": 1096}
+{"ts": 1777175011.7080715, "task_id": "math_015", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1397}
+{"ts": 1777175030.2603054, "task_id": "code_016", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.09250000000000001, "total": 0.007499999999999993}, "completion_len": 1171}
+{"ts": 1777175032.0410023, "task_id": "code_016", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0655, "total": 1.0345}, "completion_len": 1063}
+{"ts": 1777175054.6729057, "task_id": "math_021", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1309}
+{"ts": 1777175060.375272, "task_id": "math_021", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.05600000000000001, "total": 1.044}, "completion_len": 1025}
+{"ts": 1777175077.9218123, "task_id": "math_019", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1296}
+{"ts": 1777175083.851587, "task_id": "math_019", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.0815, "total": -0.0815}, "completion_len": 1127}
+{"ts": 1777175106.071159, "task_id": "json_003", "task_type": "json", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1330}
+{"ts": 1777175106.9987998, "task_id": "json_003", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1300}
+{"ts": 1777175124.2084413, "task_id": "math_007", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.09325, "total": 0.90675}, "completion_len": 1174}
+{"ts": 1777175124.4143105, "task_id": "math_007", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1218}
+{"ts": 1777175140.9962604, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1444}
+{"ts": 1777175141.070674, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1321}
+{"ts": 1777175162.1808069, "task_id": "math_018", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1236}
+{"ts": 1777175166.837405, "task_id": "math_018", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0595, "total": 1.0405000000000002}, "completion_len": 1039}
+{"ts": 1777175186.6213953, "task_id": "code_008", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1333}
+{"ts": 1777175192.4335587, "task_id": "code_008", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1335}
+{"ts": 1777175214.5744965, "task_id": "code_018", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1278}
+{"ts": 1777175220.5369945, "task_id": "code_018", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.063, "total": 0.037000000000000005}, "completion_len": 1053}
+{"ts": 1777175238.043605, "task_id": "math_010", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1351}
+{"ts": 1777175238.1603415, "task_id": "math_010", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1313}
+{"ts": 1777175258.2254663, "task_id": "code_010", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.03375, "total": 1.0662500000000001}, "completion_len": 936}
+{"ts": 1777175263.779717, "task_id": "code_010", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 593}
+{"ts": 1777175286.3451197, "task_id": "code_020", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.00025, "total": -0.00025}, "completion_len": 804}
+{"ts": 1777175292.25557, "task_id": "code_020", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1336}
+{"ts": 1777175313.1582701, "task_id": "math_009", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1586}
+{"ts": 1777175316.0815704, "task_id": "math_009", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07225000000000001, "total": -0.07225000000000001}, "completion_len": 1090}
+{"ts": 1777175335.440552, "task_id": "math_001", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1383}
+{"ts": 1777175337.131737, "task_id": "math_001", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1299}
+{"ts": 1777175357.3943813, "task_id": "math_027", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.03575, "total": 0.96425}, "completion_len": 946}
+{"ts": 1777175357.681536, "task_id": "math_027", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.014249999999999999, "total": -0.014249999999999999}, "completion_len": 858}
+{"ts": 1777175374.7662253, "task_id": "json_007", "task_type": "json", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1341}
+{"ts": 1777175375.5082958, "task_id": "json_007", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0, "total": 1.1}, "completion_len": 649}
+{"ts": 1777175397.8982294, "task_id": "math_008", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.09475, "total": 0.005250000000000005}, "completion_len": 1180}
+{"ts": 1777175403.6554832, "task_id": "math_008", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1251}
+{"ts": 1777175425.7587645, "task_id": "code_013", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08325, "total": 1.01675}, "completion_len": 1134}
+{"ts": 1777175431.7841861, "task_id": "code_013", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.05225, "total": 1.0477500000000002}, "completion_len": 1010}
+{"ts": 1777175449.0808718, "task_id": "json_002", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.059250000000000004, "total": 1.04075}, "completion_len": 1038}
+{"ts": 1777175449.9700878, "task_id": "json_002", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.06475, "total": 0.035250000000000004}, "completion_len": 1060}
+{"ts": 1777175469.8912854, "task_id": "math_023", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.034499999999999996, "total": 0.9655}, "completion_len": 939}
+{"ts": 1777175472.3405862, "task_id": "math_023", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.07950000000000002, "total": 0.9205}, "completion_len": 1120}
+{"ts": 1777175492.9372365, "task_id": "math_010", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1212}
+{"ts": 1777175493.4908137, "task_id": "math_010", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.04225, "total": 0.05775}, "completion_len": 970}
+{"ts": 1777175515.035878, "task_id": "math_011", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.06875, "total": -0.06875}, "completion_len": 1076}
+{"ts": 1777175515.2191045, "task_id": "math_011", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.06475, "total": 1.03525}, "completion_len": 1060}
+{"ts": 1777175532.0130265, "task_id": "math_014", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1534}
+{"ts": 1777175532.1976247, "task_id": "math_014", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.05550000000000001, "total": 0.9445}, "completion_len": 1023}
+{"ts": 1777175548.3190265, "task_id": "math_012", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1290}
+{"ts": 1777175552.7855537, "task_id": "math_012", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.06949999999999999, "total": 0.030500000000000013}, "completion_len": 1079}
+{"ts": 1777175573.1027837, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.043250000000000004, "total": 0.95675}, "completion_len": 974}
+{"ts": 1777175573.3594387, "task_id": "math_004", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1223}
+{"ts": 1777175590.7691643, "task_id": "math_015", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.09200000000000001, "total": -0.09200000000000001}, "completion_len": 1181}
+{"ts": 1777175591.013644, "task_id": "math_015", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.062, "total": 0.038000000000000006}, "completion_len": 1049}
+{"ts": 1777175611.3889132, "task_id": "math_018", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.05475, "total": 0.94525}, "completion_len": 1020}
+{"ts": 1777175617.2103305, "task_id": "math_018", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1231}
+{"ts": 1777175636.4076352, "task_id": "code_003", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1416}
+{"ts": 1777175638.3863735, "task_id": "code_003", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1561}
+{"ts": 1777175660.9959147, "task_id": "code_020", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1216}
+{"ts": 1777175666.9646652, "task_id": "code_020", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.043500000000000004, "total": -0.043500000000000004}, "completion_len": 975}
+{"ts": 1777175684.1038857, "task_id": "math_009", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1412}
+{"ts": 1777175689.9629886, "task_id": "math_009", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1324}
+{"ts": 1777175707.2088346, "task_id": "json_006", "task_type": "json", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07375000000000001, "total": -0.07375000000000001}, "completion_len": 1096}
+{"ts": 1777175708.1230671, "task_id": "json_006", "task_type": "json", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 796}
+{"ts": 1777175730.0257103, "task_id": "code_007", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1396}
+{"ts": 1777175734.7199624, "task_id": "code_007", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08625000000000001, "total": 1.0137500000000002}, "completion_len": 1146}
+{"ts": 1777175751.4488666, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1382}
+{"ts": 1777175752.7618048, "task_id": "math_003", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.05475, "total": 0.94525}, "completion_len": 1020}
+{"ts": 1777175770.2901957, "task_id": "math_013", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.07900000000000001, "total": 0.02099999999999999}, "completion_len": 1117}
+{"ts": 1777175772.2442117, "task_id": "math_013", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.05550000000000001, "total": 0.9445}, "completion_len": 1025}
+{"ts": 1777175788.7024896, "task_id": "code_001", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.08600000000000001, "total": 1.014}, "completion_len": 1145}
+{"ts": 1777175789.1276822, "task_id": "code_001", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.07800000000000001, "total": 1.022}, "completion_len": 1113}
+{"ts": 1777175811.1745636, "task_id": "math_008", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1389}
+{"ts": 1777175817.1482754, "task_id": "math_008", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.07925, "total": -0.07925}, "completion_len": 1118}
+{"ts": 1777175834.1070986, "task_id": "json_001", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.09050000000000001, "total": 1.0095}, "completion_len": 1163}
+{"ts": 1777175834.615735, "task_id": "json_001", "task_type": "json", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1483}
+{"ts": 1777175851.5951793, "task_id": "code_002", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1624}
+{"ts": 1777175852.0926504, "task_id": "code_002", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1231}
+{"ts": 1777175873.2218435, "task_id": "math_019", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1372}
+{"ts": 1777175873.5613837, "task_id": "math_019", "task_type": "math", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1403}
+{"ts": 1777175896.0177104, "task_id": "code_017", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1380}
+{"ts": 1777175900.7860854, "task_id": "code_017", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1346}
+{"ts": 1777175918.227239, "task_id": "json_004", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.0955, "total": 1.0045000000000002}, "completion_len": 1183}
+{"ts": 1777175919.0461085, "task_id": "json_004", "task_type": "json", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1349}
+{"ts": 1777175936.242021, "task_id": "code_006", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1202}
+{"ts": 1777175940.1184714, "task_id": "code_006", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1252}
+{"ts": 1777175960.709051, "task_id": "math_024", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.08275, "total": 0.91725}, "completion_len": 1132}
+{"ts": 1777175964.4923527, "task_id": "math_024", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.09575, "total": 1.00425}, "completion_len": 1184}
+{"ts": 1777175982.0470245, "task_id": "math_007", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1263}
+{"ts": 1777175982.8157668, "task_id": "math_007", "task_type": "math", "reward": {"correctness": 1.0, "format": 0.0, "brevity": -0.1, "total": 0.9}, "completion_len": 1329}
+{"ts": 1777176000.7275205, "task_id": "math_026", "task_type": "math", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.0, "total": 0.1}, "completion_len": 746}
+{"ts": 1777176001.3030024, "task_id": "math_026", "task_type": "math", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1272}
+{"ts": 1777176023.9129438, "task_id": "code_012", "task_type": "code", "reward": {"correctness": 0.0, "format": 0.0, "brevity": -0.1, "total": -0.1}, "completion_len": 1272}
+{"ts": 1777176029.837695, "task_id": "code_012", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.060250000000000005, "total": 1.0397500000000002}, "completion_len": 1042}
+{"ts": 1777176049.5557144, "task_id": "code_009", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1223}
+{"ts": 1777176052.6288605, "task_id": "code_009", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1373}
+{"ts": 1777176070.2213323, "task_id": "code_014", "task_type": "code", "reward": {"correctness": 1.0, "format": 1.0, "brevity": -0.1, "total": 1.0}, "completion_len": 1289}
+{"ts": 1777176076.0898345, "task_id": "code_014", "task_type": "code", "reward": {"correctness": 0.0, "format": 1.0, "brevity": -0.1, "total": 0.0}, "completion_len": 1442}

scripts/eval_trained.py

@@ -0,0 +1,217 @@
+"""
+Phase 6: Evaluate the GRPO-trained agent on the test split.
+
+Loads:
+  - base agent: Qwen/Qwen2.5-1.5B-Instruct (frozen weights)
+  - LoRA adapter: from --adapter (local dir or HF model repo id)
+
+For each test task:
+  1. build agent input (same as training)
+  2. agent generates a candidate system prompt
+  3. env runs LLM-under-test with that prompt; verify; reward
+  4. up to --max-turns retries with the previous attempt visible
+
+Outputs results/trained_agent.json in the same shape as run_baseline.py.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Any, Dict, List
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from src.envs.promptops_arena.server.environment import PromptOpsArenaEnvironment
+from src.envs.promptops_arena.tasks import load_tasks
+from src.envs.promptops_arena import llm_under_test
+from scripts.train_grpo import build_agent_input  # reuse the exact prompt template
+
+
+def _load_agent(base_model: str, adapter: str | None):
+    import torch  # type: ignore
+    from transformers import AutoModelForCausalLM, AutoTokenizer  # type: ignore
+
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+    tok = AutoTokenizer.from_pretrained(base_model)
+    if tok.pad_token is None:
+        tok.pad_token = tok.eos_token
+
+    mdl = AutoModelForCausalLM.from_pretrained(
+        base_model, torch_dtype=dtype, device_map=device,
+    )
+    if adapter:
+        from peft import PeftModel  # type: ignore
+        mdl = PeftModel.from_pretrained(mdl, adapter)
+    mdl.eval()
+
+    def gen(text: str, max_new_tokens: int = 300) -> str:
+        msgs = [
+            {"role": "system", "content": "You are a helpful prompt engineer."},
+            {"role": "user", "content": text},
+        ]
+        encoded = tok.apply_chat_template(
+            msgs, add_generation_prompt=True, return_tensors="pt",
+        )
+        if hasattr(encoded, "input_ids"):
+            ids = encoded.input_ids
+        elif isinstance(encoded, dict):
+            ids = encoded["input_ids"]
+        else:
+            ids = encoded
+        ids = ids.to(device)
+        with torch.no_grad():
+            out = mdl.generate(
+                input_ids=ids, max_new_tokens=max_new_tokens,
+                do_sample=False, pad_token_id=tok.eos_token_id,
+            )
+        return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True).strip()
+
+    return gen
+
+
+def _build_followup_input(task: dict, history: List[dict]) -> str:
+    """Like build_agent_input but with prior attempts visible (refinement turn)."""
+    base = build_agent_input(task)
+    if not history:
+        return base
+    extra = ["", "PRIOR ATTEMPTS (yours, with what the small model produced):"]
+    for i, h in enumerate(history, 1):
+        extra.append(f"--- attempt {i} (reward={h['reward']:.2f}, correct={h['correct']}) ---")
+        extra.append(f"YOUR PROMPT: {h['prompt'][:400]}")
+        extra.append(f"MODEL OUTPUT: {h['completion'][:200]}")
+    extra.append("")
+    extra.append("Improve the system prompt. Output ONLY the new system prompt, no preamble.")
+    return base + "\n" + "\n".join(extra)
+
+
+def evaluate_trained(env, task, agent_gen, max_turns: int = 3) -> Dict[str, Any]:
+    history: List[dict] = []
+    best_reward = -1.0
+    best_components: Dict[str, float] = {}
+    correct = False
+    edit_turns = 0
+
+    for turn in range(max_turns):
+        edit_turns = turn + 1
+        ai = build_agent_input(task) if turn == 0 else _build_followup_input(task, history)
+        sp = agent_gen(ai).strip() or "Solve this:"
+        res = env.execute_prompt(task, sp)
+        components = res["reward"]
+        total = components["total"]
+        is_correct = components["correctness"] >= 1.0
+
+        history.append({
+            "prompt": sp,
+            "completion": res["completion"],
+            "reward": total,
+            "correct": is_correct,
+        })
+
+        if total > best_reward:
+            best_reward = total
+            best_components = components
+
+        if is_correct:
+            correct = True
+            break
+
+    return {
+        "task_id": task["id"],
+        "task_type": task["type"],
+        "policy": "trained_agent",
+        "edit_turns": edit_turns,
+        "final_reward": best_reward,
+        "correct": correct,
+        "format_ok": best_components.get("format", 0.0) >= 1.0,
+        "components": best_components,
+        "trace": history,
+    }
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--adapter", default=None,
+                   help="Local dir or HF repo id of the LoRA adapter.")
+    p.add_argument("--base", default="Qwen/Qwen2.5-1.5B-Instruct")
+    p.add_argument("--split", default="test")
+    p.add_argument("--out", default="results/trained_agent.json")
+    p.add_argument("--limit", type=int, default=None)
+    p.add_argument("--per-type", type=int, default=None)
+    p.add_argument("--max-turns", type=int, default=3)
+    args = p.parse_args()
+
+    os.environ.setdefault("PROMPTOPS_LLM_BACKEND", "transformers")
+
+    tasks = load_tasks(split=args.split)
+    if args.per_type:
+        bucketed: Dict[str, List[dict]] = {}
+        for t in tasks:
+            bucketed.setdefault(t["type"], []).append(t)
+        sampled: List[dict] = []
+        for tt, lst in bucketed.items():
+            sampled.extend(lst[: args.per_type])
+        tasks = sampled
+    if args.limit:
+        tasks = tasks[: args.limit]
+
+    print(f"[eval_trained] adapter={args.adapter} base={args.base} "
+          f"split={args.split} n_tasks={len(tasks)} "
+          f"llm_backend={llm_under_test.backend_name()}")
+
+    env = PromptOpsArenaEnvironment(split=args.split, seed=0)
+    agent_gen = _load_agent(args.base, args.adapter)
+
+    rows: List[Dict[str, Any]] = []
+    t0 = time.time()
+    for i, task in enumerate(tasks):
+        row = evaluate_trained(env, task, agent_gen, max_turns=args.max_turns)
+        rows.append(row)
+        n_correct = sum(1 for r in rows if r["correct"])
+        print(f"  [{i+1}/{len(tasks)}] {task['type']:5s} "
+              f"correct={n_correct}/{i+1} "
+              f"r={row['final_reward']:+.3f} elapsed={time.time()-t0:.1f}s")
+
+    by_type: Dict[str, Dict[str, int]] = {}
+    for r in rows:
+        d = by_type.setdefault(r["task_type"], {"n": 0, "correct": 0, "format": 0})
+        d["n"] += 1
+        d["correct"] += int(r["correct"])
+        d["format"] += int(r["format_ok"])
+
+    overall = {
+        "n": len(rows),
+        "correct": sum(1 for r in rows if r["correct"]),
+        "format": sum(1 for r in rows if r["format_ok"]),
+        "mean_reward": sum(r["final_reward"] for r in rows) / max(1, len(rows)),
+    }
+
+    out = {
+        "policy": "trained_agent",
+        "adapter": args.adapter,
+        "base_model": args.base,
+        "split": args.split,
+        "llm_backend": llm_under_test.backend_name(),
+        "by_type": by_type,
+        "overall": overall,
+        "rows": rows,
+    }
+
+    out_path = Path(args.out)
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    out_path.write_text(json.dumps(out, indent=2), encoding="utf-8")
+    print(f"\n[eval_trained] wrote {out_path}")
+    print(f"  overall: {overall['correct']}/{overall['n']} correct, "
+          f"mean_reward={overall['mean_reward']:.3f}")
+    for tt, d in by_type.items():
+        print(f"  {tt:5s}: {d['correct']}/{d['n']} correct, format {d['format']}/{d['n']}")
+
+
+if __name__ == "__main__":
+    main()

scripts/hf_eval_entry.sh

@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# HF Jobs entrypoint: evaluate the trained agent + rerun real-LLM baselines on a
+# wider per-type sample, then upload all results to the model repo.
+
+set -euo pipefail
+
+HF_USERNAME="${HF_USERNAME:-Dar3devil}"
+PER_TYPE="${PER_TYPE:-4}"
+ADAPTER_REPO="${ADAPTER_REPO:-${HF_USERNAME}/promptops-arena-agent}"
+RESULTS_REPO="${ADAPTER_REPO}"
+
+echo "[eval] HF_USERNAME=${HF_USERNAME} PER_TYPE=${PER_TYPE} ADAPTER_REPO=${ADAPTER_REPO}"
+mkdir -p /workspace
+cp -r /code/. /workspace/
+cd /workspace
+
+echo "[eval] python: $(python --version)"
+nvidia-smi || echo "no nvidia-smi"
+
+echo "[eval] installing deps"
+pip install --no-cache-dir --upgrade pip
+pip install --no-cache-dir \
+    "transformers==4.55.4" \
+    "peft==0.15.2" \
+    "accelerate==1.7.0" \
+    "datasets==3.6.0" \
+    "huggingface_hub>=0.25.0" \
+    "jsonschema>=4.20.0" \
+    "openenv-core>=0.1.0" \
+    "fastapi>=0.110.0" \
+    "uvicorn>=0.27.0" \
+    "pydantic>=2.0.0"
+
+export PROMPTOPS_LLM_BACKEND=transformers
+export PYTHONUTF8=1
+export TOKENIZERS_PARALLELISM=false
+
+mkdir -p outputs results
+
+echo "[eval] downloading adapter ${ADAPTER_REPO}"
+python - <<PY
+from huggingface_hub import snapshot_download
+import os
+p = snapshot_download(repo_id="${ADAPTER_REPO}", repo_type="model",
+                      local_dir="outputs/grpo-lora",
+                      allow_patterns=["adapter_*", "*.json", "*.jinja", "*.txt", "training_log.jsonl"])
+print("[eval] adapter at", p)
+PY
+
+echo "[eval] running zero-shot baseline (real LLM)"
+python scripts/run_baseline.py --policy zero_shot --per-type "${PER_TYPE}" \
+    --out results/baseline_zero_shot_real.json
+
+echo "[eval] running CoT baseline (real LLM)"
+python scripts/run_baseline.py --policy cot --per-type "${PER_TYPE}" \
+    --out results/baseline_cot_real.json
+
+echo "[eval] running trained-agent eval"
+python scripts/eval_trained.py --adapter outputs/grpo-lora --per-type "${PER_TYPE}" \
+    --out results/trained_agent.json --max-turns 2
+
+echo "[eval] uploading results to ${RESULTS_REPO}"
+python - <<PY
+import os
+from huggingface_hub import HfApi
+api = HfApi()
+repo = "${RESULTS_REPO}"
+for f in [
+    "results/baseline_zero_shot_real.json",
+    "results/baseline_cot_real.json",
+    "results/trained_agent.json",
+]:
+    api.upload_file(path_or_fileobj=f, path_in_repo=os.path.basename(f),
+                    repo_id=repo, repo_type="model",
+                    commit_message=f"eval: {os.path.basename(f)}")
+print("[eval] uploaded")
+PY
+
+echo "[eval] all done."

scripts/hf_job_entry.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Entrypoint executed inside the HF Jobs container.
+# Expects:
+#   /code        -> RO mount of dataset Dar3devil/promptops-arena-src
+#   $HF_TOKEN    -> secret, for pushing the trained adapter
+#   $HF_USERNAME -> user namespace for the model repo (default: Dar3devil)
+#   $STEPS, $BATCH, $NUM_GENS (optional overrides)
+
+set -euo pipefail
+
+HF_USERNAME="${HF_USERNAME:-Dar3devil}"
+STEPS="${STEPS:-200}"
+BATCH="${BATCH:-4}"
+NUM_GENS="${NUM_GENS:-4}"
+LOG_LEVEL="${LOG_LEVEL:-info}"
+MODEL_REPO="${HF_USERNAME}/promptops-arena-agent"
+
+echo "[entry] HF_USERNAME=${HF_USERNAME} STEPS=${STEPS} BATCH=${BATCH} NUM_GENS=${NUM_GENS}"
+echo "[entry] copying source from /code -> /workspace"
+mkdir -p /workspace
+cp -r /code/. /workspace/
+cd /workspace
+
+echo "[entry] python: $(python --version)"
+echo "[entry] gpu:"
+nvidia-smi || echo "no nvidia-smi"
+
+echo "[entry] installing deps (pinned for trl 0.21 stack)"
+pip install --no-cache-dir --upgrade pip
+pip install --no-cache-dir \
+    "trl==0.21.0" \
+    "transformers==4.55.4" \
+    "peft==0.15.2" \
+    "accelerate==1.7.0" \
+    "datasets==3.6.0" \
+    "huggingface_hub>=0.25.0" \
+    "jsonschema>=4.20.0" \
+    "openenv-core>=0.1.0" \
+    "fastapi>=0.110.0" \
+    "uvicorn>=0.27.0" \
+    "pydantic>=2.0.0"
+
+export PROMPTOPS_LLM_BACKEND=transformers
+export PYTHONUTF8=1
+export TOKENIZERS_PARALLELISM=false
+
+echo "[entry] launching GRPO training"
+python scripts/train_grpo.py \
+    --steps "${STEPS}" \
+    --batch "${BATCH}" \
+    --num-generations "${NUM_GENS}" \
+    --out outputs/grpo-lora \
+    --log results/training_log.jsonl
+
+echo "[entry] training done. uploading adapter + log to ${MODEL_REPO}"
+python - <<'PY'
+import os
+from huggingface_hub import HfApi, create_repo
+
+api = HfApi()
+repo_id = f"{os.environ['HF_USERNAME']}/promptops-arena-agent"
+create_repo(repo_id, repo_type="model", exist_ok=True, private=False)
+
+api.upload_folder(
+    folder_path="outputs/grpo-lora",
+    repo_id=repo_id,
+    repo_type="model",
+    commit_message="GRPO-trained LoRA adapter",
+)
+# also upload training log so we can plot reward curves locally
+api.upload_file(
+    path_or_fileobj="results/training_log.jsonl",
+    path_in_repo="training_log.jsonl",
+    repo_id=repo_id,
+    repo_type="model",
+    commit_message="training reward log",
+)
+print(f"[entry] uploaded to https://huggingface.co/{repo_id}")
+PY
+
+echo "[entry] all done."

scripts/plot_results.py

@@ -0,0 +1,167 @@
+"""
+Phase 7: Build reward-curve plot + comparison artifact.
+
+Inputs (any subset that exists):
+  results/training_log.jsonl                 # per-step rewards from GRPO
+  results/baseline_zero_shot_real_subset.json
+  results/baseline_cot_real_subset.json
+  results/baseline_zero_shot_stub.json
+  results/baseline_cot_stub.json
+  results/trained_agent.json                 # eval of the trained agent (Phase 6)
+
+Outputs:
+  results/comparison.json
+  docs/reward_curve.png
+  docs/baseline_comparison.png
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any, Dict, List
+
+
+def load_json(p: Path) -> dict | None:
+    if not p.exists():
+        return None
+    try:
+        return json.loads(p.read_text(encoding="utf-8"))
+    except Exception as e:
+        print(f"[plot] WARN failed to load {p}: {e}")
+        return None
+
+
+def load_jsonl(p: Path) -> List[dict]:
+    if not p.exists():
+        return []
+    rows: List[dict] = []
+    for line in p.read_text(encoding="utf-8").splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            rows.append(json.loads(line))
+        except Exception:
+            pass
+    return rows
+
+
+def smooth(values: List[float], window: int = 10) -> List[float]:
+    out: List[float] = []
+    for i in range(len(values)):
+        lo = max(0, i - window + 1)
+        out.append(sum(values[lo:i + 1]) / max(1, i + 1 - lo))
+    return out
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--results-dir", default="results")
+    p.add_argument("--docs-dir", default="docs")
+    args = p.parse_args()
+
+    res = Path(args.results_dir)
+    docs = Path(args.docs_dir)
+    docs.mkdir(parents=True, exist_ok=True)
+
+    import matplotlib
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt
+
+    # ---- 1. reward curve ----
+    log_rows = load_jsonl(res / "training_log.jsonl")
+    if log_rows:
+        rewards = [r["reward"]["total"] for r in log_rows if "reward" in r]
+        smoothed = smooth(rewards, window=20)
+        fig, ax = plt.subplots(figsize=(8, 4.5))
+        ax.plot(rewards, alpha=0.25, label="raw reward")
+        ax.plot(smoothed, linewidth=2, label="rolling avg (20)")
+        ax.set_xlabel("training rollout #")
+        ax.set_ylabel("reward")
+        ax.set_title("GRPO training reward curve  ·  PromptOps Arena")
+        ax.grid(alpha=0.3)
+        ax.legend()
+        fig.tight_layout()
+        out = docs / "reward_curve.png"
+        fig.savefig(out, dpi=140)
+        plt.close(fig)
+        print(f"[plot] wrote {out}  ({len(rewards)} points)")
+    else:
+        print("[plot] no training_log.jsonl yet -> skip reward curve")
+
+    # ---- 2. baseline comparison ----
+    files = {
+        "zero_shot (stub)": res / "baseline_zero_shot_stub.json",
+        "cot (stub)": res / "baseline_cot_stub.json",
+        "zero_shot (real LLM)": res / "baseline_zero_shot_real.json",
+        "cot (real LLM)": res / "baseline_cot_real.json",
+        "trained agent (real LLM)": res / "trained_agent.json",
+    }
+    # fall back to the smaller subset files if the wider-n versions don't exist
+    fallback = {
+        "zero_shot (real LLM)": res / "baseline_zero_shot_real_subset.json",
+        "cot (real LLM)": res / "baseline_cot_real_subset.json",
+    }
+    for k, p in fallback.items():
+        if not files[k].exists() and p.exists():
+            files[k] = p
+
+    rows: Dict[str, Dict[str, Any]] = {}
+    for label, path in files.items():
+        d = load_json(path)
+        if d is None:
+            continue
+        ov = d.get("overall", {})
+        rows[label] = {
+            "n": ov.get("n", 0),
+            "correct": ov.get("correct", 0),
+            "format": ov.get("format", 0),
+            "mean_reward": ov.get("mean_reward", 0.0),
+            "by_type": d.get("by_type", {}),
+            "backend": d.get("llm_backend", "unknown"),
+        }
+
+    comparison = {
+        "policies": rows,
+        "ranking_by_mean_reward": sorted(
+            rows.items(),
+            key=lambda kv: kv[1]["mean_reward"],
+            reverse=True,
+        ),
+    }
+    (res / "comparison.json").write_text(
+        json.dumps(comparison, indent=2), encoding="utf-8"
+    )
+    print(f"[plot] wrote {res/'comparison.json'}")
+
+    # ---- 3. comparison bar chart ----
+    if rows:
+        labels = list(rows.keys())
+        means = [rows[l]["mean_reward"] for l in labels]
+        accs = [rows[l]["correct"] / max(1, rows[l]["n"]) for l in labels]
+
+        fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
+        axes[0].barh(labels, means, color="#4c72b0")
+        axes[0].set_xlabel("mean reward")
+        axes[0].set_title("Mean reward by policy")
+        axes[0].grid(axis="x", alpha=0.3)
+        axes[0].invert_yaxis()
+
+        axes[1].barh(labels, accs, color="#55a868")
+        axes[1].set_xlabel("fraction correct")
+        axes[1].set_title("Correctness by policy")
+        axes[1].set_xlim(0, 1)
+        axes[1].grid(axis="x", alpha=0.3)
+        axes[1].invert_yaxis()
+
+        fig.tight_layout()
+        out = docs / "baseline_comparison.png"
+        fig.savefig(out, dpi=140)
+        plt.close(fig)
+        print(f"[plot] wrote {out}")
+
+
+if __name__ == "__main__":
+    main()

scripts/push_space.py

@@ -0,0 +1,59 @@
+"""Create + push the HF Space for PromptOps Arena."""
+
+from __future__ import annotations
+
+import os
+import sys
+from pathlib import Path
+
+from huggingface_hub import HfApi, create_repo
+
+ROOT = Path(__file__).resolve().parents[1]
+SPACE_ID = os.environ.get("SPACE_ID", "Dar3devil/promptops-arena")
+
+
+def main() -> int:
+    token = os.environ.get("HF_TOKEN")
+    api = HfApi(token=token)
+
+    create_repo(
+        repo_id=SPACE_ID,
+        repo_type="space",
+        space_sdk="gradio",
+        exist_ok=True,
+        token=token,
+    )
+
+    ignore = [
+        "outputs/*",
+        "outputs/**",
+        ".venv/*",
+        ".venv/**",
+        ".git/*",
+        ".git/**",
+        "__pycache__/*",
+        "**/__pycache__/**",
+        "*.pyc",
+        "node_modules/**",
+        ".pytest_cache/**",
+        ".mypy_cache/**",
+        ".ruff_cache/**",
+        ".cursor/**",
+        "logs/**",
+        "results/.cache/**",
+    ]
+
+    print(f"[push] uploading {ROOT} -> space {SPACE_ID}")
+    api.upload_folder(
+        folder_path=str(ROOT),
+        repo_id=SPACE_ID,
+        repo_type="space",
+        ignore_patterns=ignore,
+        commit_message="PromptOps Arena demo",
+    )
+    print(f"[push] done. https://huggingface.co/spaces/{SPACE_ID}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scripts/run_baseline.py

@@ -0,0 +1,287 @@
+"""
+Phase 4 baselines on the held-out test split.
+
+Three policies:
+  zero_shot : "Solve this:" wrapper, no CoT
+  cot       : "Think step by step. Final answer in <answer> tags." style
+  untrained : Qwen2.5-1.5B-Instruct (no LoRA) writes the system prompt,
+              then LLM-under-test runs it. 3 edit turns.
+
+For local CPU dev we run zero_shot/cot only with the stub backend; untrained
+is meant for GPU runs (CUDA or HF Jobs).
+
+Usage:
+  python scripts\run_baseline.py --policy zero_shot --out results/baseline_zero_shot.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+from typing import Any, Dict, List
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from src.envs.promptops_arena.server.environment import PromptOpsArenaEnvironment
+from src.envs.promptops_arena.models import PromptOpsAction
+from src.envs.promptops_arena.tasks import load_tasks
+from src.envs.promptops_arena import llm_under_test
+
+
+ZERO_SHOT_PROMPT = "Solve this:"
+
+COT_PROMPT_BY_TYPE = {
+    "math": (
+        "Think step by step. After reasoning, put ONLY the final numeric answer "
+        "inside <answer>...</answer> tags. Do not include units or words inside the tags."
+    ),
+    "code": (
+        "Write the requested Python function. Reason briefly, then output exactly one "
+        "```python ...``` code block containing only the function definition. "
+        "Do not include explanations after the code block."
+    ),
+    "json": (
+        "Extract the requested fields. Output exactly one ```json ...``` code block "
+        "containing a JSON object that matches the schema. Use the correct types. "
+        "No prose."
+    ),
+}
+
+
+def _evaluate_zero_shot(env: PromptOpsArenaEnvironment, task: dict) -> Dict[str, Any]:
+    res = env.execute_prompt(task, ZERO_SHOT_PROMPT)
+    return {
+        "task_id": task["id"],
+        "task_type": task["type"],
+        "policy": "zero_shot",
+        "edit_turns": 1,
+        "final_reward": res["reward"]["total"],
+        "correct": res["reward"]["correctness"] >= 1.0,
+        "format_ok": res["reward"]["format"] >= 1.0,
+        "components": res["reward"],
+    }
+
+
+def _evaluate_cot(env: PromptOpsArenaEnvironment, task: dict) -> Dict[str, Any]:
+    sp = COT_PROMPT_BY_TYPE.get(task["type"], ZERO_SHOT_PROMPT)
+    res = env.execute_prompt(task, sp)
+    return {
+        "task_id": task["id"],
+        "task_type": task["type"],
+        "policy": "cot",
+        "edit_turns": 1,
+        "final_reward": res["reward"]["total"],
+        "correct": res["reward"]["correctness"] >= 1.0,
+        "format_ok": res["reward"]["format"] >= 1.0,
+        "components": res["reward"],
+    }
+
+
+def _build_agent_input(task: dict, history: List[dict]) -> str:
+    """Build the prompt the agent sees when asked to write a system prompt."""
+    parts = [
+        "You are a prompt engineer. Your job is to write a SYSTEM PROMPT that, "
+        "when given to a small language model along with the task below, will "
+        "produce a correct answer in the required format.",
+        "",
+        f"TASK TYPE: {task['type']}",
+        f"TASK: {task['question']}",
+        "",
+    ]
+    if task["type"] == "math":
+        parts.append("REQUIRED FORMAT: the answer must be a number inside <answer>...</answer> tags.")
+    elif task["type"] == "code":
+        parts.append("REQUIRED FORMAT: a single ```python ...``` code block defining the requested function.")
+    elif task["type"] == "json":
+        parts.append("REQUIRED FORMAT: a single ```json ...``` code block with a valid JSON object matching the schema.")
+        if "schema" in task:
+            parts.append(f"SCHEMA: {json.dumps(task['schema'])}")
+
+    if history:
+        parts.append("")
+        parts.append("PREVIOUS ATTEMPTS (your earlier prompts and the model's responses):")
+        for i, h in enumerate(history, 1):
+            parts.append(f"--- attempt {i} (reward={h['reward']:.2f}, correct={h['correct']}) ---")
+            parts.append(f"YOUR PROMPT: {h['prompt'][:400]}")
+            parts.append(f"MODEL OUTPUT: {h['completion'][:200]}")
+        parts.append("")
+        parts.append("Improve the system prompt. Output ONLY the new system prompt, no preamble.")
+    else:
+        parts.append("")
+        parts.append("Output ONLY the system prompt, no preamble.")
+
+    return "\n".join(parts)
+
+
+def _evaluate_untrained_agent(
+    env: PromptOpsArenaEnvironment,
+    task: dict,
+    agent_generate,
+    max_turns: int = 3,
+) -> Dict[str, Any]:
+    history: List[dict] = []
+    best_reward = -1.0
+    final_components = {}
+    correct = False
+    edit_turns = 0
+
+    for turn in range(max_turns):
+        edit_turns = turn + 1
+        agent_input = _build_agent_input(task, history)
+        system_prompt = agent_generate(agent_input).strip()
+        if not system_prompt:
+            system_prompt = ZERO_SHOT_PROMPT
+
+        res = env.execute_prompt(task, system_prompt)
+        components = res["reward"]
+        total = components["total"]
+        is_correct = components["correctness"] >= 1.0
+
+        history.append({
+            "prompt": system_prompt,
+            "completion": res["completion"],
+            "reward": total,
+            "correct": is_correct,
+        })
+
+        if total > best_reward:
+            best_reward = total
+            final_components = components
+
+        if is_correct:
+            correct = True
+            break
+
+    return {
+        "task_id": task["id"],
+        "task_type": task["type"],
+        "policy": "untrained_agent",
+        "edit_turns": edit_turns,
+        "final_reward": best_reward,
+        "correct": correct,
+        "format_ok": final_components.get("format", 0.0) >= 1.0,
+        "components": final_components,
+        "trace": history,
+    }
+
+
+def _make_agent_generate(model_id: str):
+    """Returns callable(text) -> generated text. Uses a separate transformers model."""
+    import torch  # type: ignore
+    from transformers import AutoModelForCausalLM, AutoTokenizer  # type: ignore
+
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+    tok = AutoTokenizer.from_pretrained(model_id)
+    mdl = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=dtype, device_map=device)
+    mdl.eval()
+
+    def gen(text: str) -> str:
+        msgs = [
+            {"role": "system", "content": "You are a helpful prompt engineer."},
+            {"role": "user", "content": text},
+        ]
+        encoded = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
+        if hasattr(encoded, "input_ids"):
+            ids = encoded.input_ids
+        elif isinstance(encoded, dict):
+            ids = encoded["input_ids"]
+        else:
+            ids = encoded
+        ids = ids.to(device)
+        with torch.no_grad():
+            out = mdl.generate(
+                input_ids=ids, max_new_tokens=300, do_sample=False,
+                pad_token_id=tok.eos_token_id,
+            )
+        return tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)
+
+    return gen
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--policy", choices=["zero_shot", "cot", "untrained"], required=True)
+    p.add_argument("--split", default="test")
+    p.add_argument("--out", required=True)
+    p.add_argument("--limit", type=int, default=None, help="cap tasks for quick runs")
+    p.add_argument("--per-type", type=int, default=None, help="cap tasks per type (stratified)")
+    p.add_argument("--agent-model", default="Qwen/Qwen2.5-1.5B-Instruct")
+    args = p.parse_args()
+
+    tasks = load_tasks(split=args.split)
+    if args.per_type:
+        bucketed: Dict[str, List[dict]] = {}
+        for t in tasks:
+            bucketed.setdefault(t["type"], []).append(t)
+        sampled: List[dict] = []
+        for tt, lst in bucketed.items():
+            sampled.extend(lst[: args.per_type])
+        tasks = sampled
+    if args.limit:
+        tasks = tasks[: args.limit]
+
+    print(f"[baseline] policy={args.policy} split={args.split} n_tasks={len(tasks)} "
+          f"llm_backend={llm_under_test.backend_name()}")
+
+    env = PromptOpsArenaEnvironment(split=args.split, seed=0)
+
+    agent_gen = None
+    if args.policy == "untrained":
+        print(f"[baseline] loading agent model: {args.agent_model}")
+        agent_gen = _make_agent_generate(args.agent_model)
+
+    rows: List[Dict[str, Any]] = []
+    t0 = time.time()
+    for i, task in enumerate(tasks):
+        if args.policy == "zero_shot":
+            row = _evaluate_zero_shot(env, task)
+        elif args.policy == "cot":
+            row = _evaluate_cot(env, task)
+        else:
+            row = _evaluate_untrained_agent(env, task, agent_gen, max_turns=3)
+        rows.append(row)
+        if (i + 1) % 5 == 0 or i == len(tasks) - 1:
+            n_correct = sum(1 for r in rows if r["correct"])
+            print(f"  [{i+1}/{len(tasks)}] correct={n_correct}/{i+1} "
+                  f"elapsed={time.time()-t0:.1f}s")
+
+    by_type: Dict[str, Dict[str, int]] = {}
+    for r in rows:
+        d = by_type.setdefault(r["task_type"], {"n": 0, "correct": 0, "format": 0})
+        d["n"] += 1
+        d["correct"] += int(r["correct"])
+        d["format"] += int(r["format_ok"])
+
+    overall = {
+        "n": len(rows),
+        "correct": sum(1 for r in rows if r["correct"]),
+        "format": sum(1 for r in rows if r["format_ok"]),
+        "mean_reward": sum(r["final_reward"] for r in rows) / max(1, len(rows)),
+    }
+
+    out = {
+        "policy": args.policy,
+        "split": args.split,
+        "llm_backend": llm_under_test.backend_name(),
+        "by_type": by_type,
+        "overall": overall,
+        "rows": rows,
+    }
+
+    out_path = Path(args.out)
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+    out_path.write_text(json.dumps(out, indent=2), encoding="utf-8")
+    print(f"\n[baseline] wrote {out_path}")
+    print(f"  overall: {overall['correct']}/{overall['n']} correct, mean_reward={overall['mean_reward']:.3f}")
+    for tt, d in by_type.items():
+        print(f"  {tt:5s}: {d['correct']}/{d['n']} correct, format {d['format']}/{d['n']}")
+
+
+if __name__ == "__main__":
+    main()

scripts/smoke_test_env.py

@@ -0,0 +1,79 @@
+"""
+Phase 2 smoke test: in-process exercise of reset() / step() / state.
+
+Uses the stub LLM backend (set via env var) so this runs in <1s and proves
+the env plumbing works without downloading any model.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from pathlib import Path
+
+# Allow running from project root: add project root to sys.path
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+os.environ.setdefault("PROMPTOPS_LLM_BACKEND", "stub")
+
+from src.envs.promptops_arena.server.environment import PromptOpsArenaEnvironment
+from src.envs.promptops_arena.models import PromptOpsAction
+from src.envs.promptops_arena import llm_under_test
+
+
+GOOD_PROMPTS = {
+    "math": (
+        "You are a careful math solver. Read the problem, think step by step, "
+        "then put ONLY the final numeric answer inside <answer>...</answer> tags. "
+        "Do not include units."
+    ),
+    "code": (
+        "You are a Python coder. Output ONLY a single ```python code block``` "
+        "containing the requested function. No prose, no examples, no print statements."
+    ),
+    "json": (
+        "You are a JSON extractor. Output ONLY a single ```json code block``` "
+        "containing a valid JSON object that matches the requested schema. "
+        "No prose."
+    ),
+}
+
+
+def run(task_type: str) -> dict:
+    env = PromptOpsArenaEnvironment(max_turns=3, split="train", seed=42, task_types=[task_type])
+    obs = env.reset()
+    print(f"\n=== {task_type.upper()} | task_id={env.state.task_id} ===")
+    print(f"task: {obs.task_text}")
+
+    action = PromptOpsAction(new_system_prompt=GOOD_PROMPTS[task_type])
+    obs2 = env.step(action)
+    print(f"completion: {obs2.last_completion[:120]!r}")
+    print(f"reward components: {obs2.reward_components}")
+    print(f"done: {obs2.done}, edit_turn: {obs2.edit_turn}, solved: {env.state.solved}")
+    return {
+        "task_type": task_type,
+        "reward": obs2.last_reward,
+        "components": obs2.reward_components,
+        "solved": env.state.solved,
+        "step_count": env.state.step_count,
+    }
+
+
+def main() -> int:
+    print(f"LLM backend: {llm_under_test.backend_name()}")
+    results = []
+    for tt in ("math", "code", "json"):
+        results.append(run(tt))
+
+    print("\n=== Summary ===")
+    for r in results:
+        print(f"  {r['task_type']:5s}: reward={r['reward']:+.3f} solved={r['solved']} "
+              f"step_count={r['step_count']} components={r['components']}")
+
+    # Exit-criterion check: every type produced a structured reward dict
+    ok = all(r["components"].get("total") is not None for r in results)
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/train_grpo.py

@@ -0,0 +1,318 @@
+"""
+Phase 5: GRPO training of the prompt-engineering agent.
+
+Agent: Qwen/Qwen2.5-1.5B-Instruct + LoRA adapter (trained).
+LLM-under-test: Qwen/Qwen2.5-0.5B-Instruct (frozen, env-side).
+
+Each "prompt" the GRPO trainer sees describes a task. The agent's "completion"
+is the system prompt it would give to the LLM-under-test. We then run the
+LLM-under-test inside the reward function and return the env reward.
+
+Modes:
+  --smoke    : tiny config; 2 steps on CPU with stub LLM. Proves plumbing.
+  --hf-jobs  : print the `hf jobs run` command for an a10g-large run.
+  default    : real GRPO run; expects CUDA + transformers backend.
+
+Outputs:
+  outputs/grpo-lora/                        # LoRA adapter
+  results/training_log.jsonl                # per-step rewards (custom callback)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from pathlib import Path
+from typing import List, Dict, Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+
+# ---------------------------------------------------------------------------
+# Agent-input builder (kept consistent with run_baseline._build_agent_input)
+# ---------------------------------------------------------------------------
+
+def build_agent_input(task: dict) -> str:
+    parts = [
+        "You are a prompt engineer. Your job is to write a SYSTEM PROMPT that, "
+        "when given to a small language model along with the task below, will "
+        "produce a correct answer in the required format.",
+        "",
+        f"TASK TYPE: {task['type']}",
+        f"TASK: {task['question']}",
+        "",
+    ]
+    if task["type"] == "math":
+        parts.append(
+            "REQUIRED FORMAT: the final numeric answer must be inside "
+            "<answer>...</answer> tags. Just the number, no units."
+        )
+    elif task["type"] == "code":
+        parts.append(
+            "REQUIRED FORMAT: a single ```python ...``` code block defining "
+            "the requested function. No prose, no examples."
+        )
+    elif task["type"] == "json":
+        parts.append(
+            "REQUIRED FORMAT: a single ```json ...``` code block with a valid "
+            "JSON object matching the schema."
+        )
+        if "schema" in task:
+            parts.append(f"SCHEMA: {json.dumps(task['schema'])}")
+
+    parts.append("")
+    parts.append("Output ONLY the system prompt itself. No preamble, no markdown fences.")
+    return "\n".join(parts)
+
+
+# ---------------------------------------------------------------------------
+# Reward wrapper for GRPO
+# ---------------------------------------------------------------------------
+
+def make_reward_fn(log_path: Path):
+    """
+    Returns a callable that GRPOTrainer can use:
+        reward_fn(prompts, completions, **kwargs) -> List[float]
+
+    `kwargs` may include the original dataset columns; we use `task` to
+    recover the task dict.
+    """
+    from src.envs.promptops_arena.server.environment import PromptOpsArenaEnvironment
+    from src.envs.promptops_arena import llm_under_test  # noqa: F401
+
+    env = PromptOpsArenaEnvironment(split="train", seed=0)
+
+    log_path.parent.mkdir(parents=True, exist_ok=True)
+    log_f = log_path.open("a", encoding="utf-8")
+
+    def reward_fn(prompts, completions, **kwargs) -> List[float]:
+        tasks = kwargs.get("task")
+        if tasks is None:
+            raise RuntimeError("reward_fn requires 'task' column in dataset")
+        if isinstance(tasks, dict):
+            tasks = [tasks] * len(completions)
+
+        rewards: List[float] = []
+        for completion, task in zip(completions, tasks):
+            if isinstance(completion, list):
+                # chat-style completion: list of {role, content}
+                text = "".join(
+                    m.get("content", "") for m in completion
+                    if isinstance(m, dict) and m.get("role") == "assistant"
+                )
+            else:
+                text = str(completion)
+            res = env.execute_prompt(task, text.strip())
+            rewards.append(float(res["reward"]["total"]))
+            log_f.write(json.dumps({
+                "ts": time.time(),
+                "task_id": task.get("id"),
+                "task_type": task.get("type"),
+                "reward": res["reward"],
+                "completion_len": len(text),
+            }) + "\n")
+            log_f.flush()
+        return rewards
+
+    return reward_fn
+
+
+# ---------------------------------------------------------------------------
+# Dataset construction
+# ---------------------------------------------------------------------------
+
+def build_dataset():
+    from datasets import Dataset
+    from src.envs.promptops_arena.tasks import load_tasks
+
+    tasks = load_tasks(split="train")
+    rows = [
+        {"prompt": build_agent_input(t), "task": t}
+        for t in tasks
+    ]
+    return Dataset.from_list(rows)
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--smoke", action="store_true",
+                   help="Tiny CPU run with stub LLM to validate plumbing.")
+    p.add_argument("--dry", action="store_true",
+                   help="Construct trainer but don't call .train(). Validates API.")
+    p.add_argument("--hf-jobs", action="store_true",
+                   help="Print HF Jobs launch command and exit.")
+    p.add_argument("--model", default="Qwen/Qwen2.5-1.5B-Instruct")
+    p.add_argument("--out", default="outputs/grpo-lora")
+    p.add_argument("--log", default="results/training_log.jsonl")
+    p.add_argument("--steps", type=int, default=200)
+    p.add_argument("--batch", type=int, default=4)
+    p.add_argument("--num-generations", type=int, default=4,
+                   help="GRPO group size G (completions per prompt).")
+    p.add_argument("--lr", type=float, default=5e-6)
+    p.add_argument("--max-prompt-length", type=int, default=512)
+    p.add_argument("--max-completion-length", type=int, default=300)
+    args = p.parse_args()
+
+    if args.hf_jobs:
+        print(_HF_JOBS_HELP)
+        return
+
+    if args.smoke:
+        os.environ["PROMPTOPS_LLM_BACKEND"] = "stub"
+
+    # Lazy imports so --hf-jobs and --smoke don't require torch/trl up front.
+    import torch  # type: ignore
+    from transformers import AutoTokenizer  # type: ignore
+
+    try:
+        from trl import GRPOConfig, GRPOTrainer  # type: ignore
+    except ImportError as e:
+        raise SystemExit(
+            "trl is required for GRPO training. Install with: pip install trl>=0.21\n"
+            f"(import error: {e})"
+        )
+
+    use_unsloth = False
+    if not args.smoke:
+        try:
+            from unsloth import FastLanguageModel  # type: ignore  # noqa: F401
+            use_unsloth = torch.cuda.is_available()
+        except ImportError:
+            use_unsloth = False
+
+    print(f"[train_grpo] mode={'smoke' if args.smoke else 'full'} "
+          f"cuda={torch.cuda.is_available()} unsloth={use_unsloth}")
+
+    # ---- model ----
+    if use_unsloth:
+        from unsloth import FastLanguageModel  # type: ignore
+        model, tokenizer = FastLanguageModel.from_pretrained(
+            model_name=args.model,
+            max_seq_length=args.max_prompt_length + args.max_completion_length,
+            load_in_4bit=True,
+            fast_inference=False,
+        )
+        model = FastLanguageModel.get_peft_model(
+            model,
+            r=16, lora_alpha=32, lora_dropout=0.0, bias="none",
+            target_modules=[
+                "q_proj", "k_proj", "v_proj", "o_proj",
+                "gate_proj", "up_proj", "down_proj",
+            ],
+        )
+    else:
+        from transformers import AutoModelForCausalLM  # type: ignore
+        from peft import LoraConfig, get_peft_model  # type: ignore
+
+        device_map = "cuda" if torch.cuda.is_available() else "cpu"
+        dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
+
+        tokenizer = AutoTokenizer.from_pretrained(args.model)
+        model = AutoModelForCausalLM.from_pretrained(
+            args.model, torch_dtype=dtype, device_map=device_map,
+        )
+        lora_cfg = LoraConfig(
+            r=8 if args.smoke else 16,
+            lora_alpha=16 if args.smoke else 32,
+            lora_dropout=0.0,
+            bias="none",
+            target_modules=["q_proj", "v_proj"] if args.smoke else [
+                "q_proj", "k_proj", "v_proj", "o_proj",
+                "gate_proj", "up_proj", "down_proj",
+            ],
+            task_type="CAUSAL_LM",
+        )
+        model = get_peft_model(model, lora_cfg)
+
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+
+    # ---- data + reward ----
+    dataset = build_dataset()
+    if args.smoke:
+        dataset = dataset.select(range(min(4, len(dataset))))
+    print(f"[train_grpo] dataset rows={len(dataset)}")
+
+    reward_fn = make_reward_fn(Path(args.log))
+
+    # ---- GRPO config ----
+    on_gpu = torch.cuda.is_available() and not args.smoke
+    per_device_bs = 2 if args.smoke else args.batch
+    num_gens = 2 if args.smoke else args.num_generations
+
+    # trl 0.21 GRPOConfig: has max_prompt_length; no generation_batch_size.
+    # Build kwargs compatible across 0.21+ (modern fields ignored if unknown).
+    cfg_kwargs = dict(
+        output_dir=args.out,
+        per_device_train_batch_size=per_device_bs,
+        gradient_accumulation_steps=1,
+        num_generations=num_gens,
+        max_prompt_length=args.max_prompt_length,
+        max_completion_length=128 if args.smoke else args.max_completion_length,
+        learning_rate=args.lr,
+        max_steps=2 if args.smoke else args.steps,
+        logging_steps=1,
+        save_steps=10_000 if args.smoke else max(1, args.steps // 4),
+        bf16=on_gpu,
+        fp16=False,
+        use_cpu=not on_gpu,
+        report_to=[],
+        remove_unused_columns=False,
+        beta=0.04,
+        temperature=1.0,
+    )
+    # Build, dropping unknown fields if a newer/older trl rejects one.
+    import inspect as _inspect
+    _allowed = set(_inspect.signature(GRPOConfig.__init__).parameters.keys())
+    cfg_kwargs = {k: v for k, v in cfg_kwargs.items() if k in _allowed}
+    cfg = GRPOConfig(**cfg_kwargs)
+
+    # trl 0.21 uses `processing_class` for tokenizer-like; older releases used
+    # `tokenizer`. Try processing_class first, fall back.
+    _tr_params = set(_inspect.signature(GRPOTrainer.__init__).parameters.keys())
+    tr_kwargs = dict(
+        model=model,
+        reward_funcs=[reward_fn],
+        args=cfg,
+        train_dataset=dataset,
+    )
+    if "processing_class" in _tr_params:
+        tr_kwargs["processing_class"] = tokenizer
+    elif "tokenizer" in _tr_params:
+        tr_kwargs["tokenizer"] = tokenizer
+    trainer = GRPOTrainer(**tr_kwargs)
+
+    if args.dry:
+        print("[train_grpo] dry mode: trainer constructed OK; skipping .train()")
+        return
+
+    print("[train_grpo] starting training...")
+    trainer.train()
+    print(f"[train_grpo] saving adapter to {args.out}")
+    trainer.save_model(args.out)
+    print("[train_grpo] done.")
+
+
+_HF_JOBS_HELP = """\
+# Launch full GRPO training on HF Jobs (a10g-large, ≤2h cap):
+hf jobs run --gpu a10g-large --timeout 7200 \\
+    --secrets HF_TOKEN \\
+    --env PROMPTOPS_LLM_BACKEND=transformers \\
+    --workdir /workspace \\
+    --upload . \\
+    python:3.11 \\
+    bash -c "pip install -r requirements.txt && pip install trl peft && \\
+             python scripts/train_grpo.py --steps 200 --batch 4 --num-generations 4 \\
+             && hf upload <user>/promptops-arena-agent outputs/grpo-lora ."
+"""
+
+
+if __name__ == "__main__":
+    main()

scripts/upload_src_to_hf.py

@@ -0,0 +1,54 @@
+"""Upload the project source to a HF dataset, mirrored at /code in the Job."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from huggingface_hub import HfApi, create_repo
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--repo", default="Dar3devil/promptops-arena-src")
+    p.add_argument("--root", default=".")
+    args = p.parse_args()
+
+    root = Path(args.root).resolve()
+    api = HfApi()
+    create_repo(args.repo, repo_type="dataset", exist_ok=True, private=True)
+
+    ignore_patterns = [
+        "**/__pycache__/**",
+        "**/.pytest_cache/**",
+        "**/.benchmarks/**",
+        ".git/**",
+        ".git",
+        "outputs/**",
+        "**/*.pyc",
+        "**/*.pyo",
+        ".venv/**",
+        "venv/**",
+        ".env",
+        ".env.local",
+        ".vscode/**",
+        ".idea/**",
+        "wandb/**",
+        "*.log",
+        ".DS_Store",
+    ]
+
+    print(f"[upload] uploading {root} -> dataset {args.repo}")
+    api.upload_folder(
+        folder_path=str(root),
+        repo_id=args.repo,
+        repo_type="dataset",
+        ignore_patterns=ignore_patterns,
+        commit_message="sync source for HF Jobs training run",
+    )
+    print(f"[upload] done. https://huggingface.co/datasets/{args.repo}")
+
+
+if __name__ == "__main__":
+    sys.exit(main() or 0)

src/envs/promptops_arena/__init__.py

@@ -0,0 +1,13 @@
+from .models import (
+    PromptOpsAction,
+    PromptOpsObservation,
+    PromptOpsState,
+)
+from .client import PromptOpsArenaEnv
+
+__all__ = [
+    "PromptOpsAction",
+    "PromptOpsObservation",
+    "PromptOpsState",
+    "PromptOpsArenaEnv",
+]

src/envs/promptops_arena/client.py

@@ -0,0 +1,56 @@
+"""
+HTTP/WebSocket client for the PromptOps Arena env.
+
+Used by the demo Space, manual exploration, and any out-of-process consumer.
+GRPO training uses the in-process `PromptOpsArenaEnvironment` directly.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+from openenv.core.client_types import StepResult
+from openenv.core.env_client import EnvClient
+
+from .models import PromptOpsAction, PromptOpsObservation, PromptOpsState
+
+
+class PromptOpsArenaEnv(EnvClient[PromptOpsAction, PromptOpsObservation, PromptOpsState]):
+    """Client; subclass of openenv.core.env_client.EnvClient."""
+
+    def _step_payload(self, action: PromptOpsAction) -> Dict[str, Any]:
+        return {"new_system_prompt": action.new_system_prompt}
+
+    def _parse_result(self, payload: Dict[str, Any]) -> StepResult[PromptOpsObservation]:
+        obs_data = payload.get("observation", payload)
+        observation = PromptOpsObservation(
+            task_text=obs_data.get("task_text", ""),
+            task_type=obs_data.get("task_type", ""),
+            current_prompt=obs_data.get("current_prompt", ""),
+            last_completion=obs_data.get("last_completion", ""),
+            last_reward=obs_data.get("last_reward", 0.0),
+            last_correctness=obs_data.get("last_correctness", 0.0),
+            edit_turn=obs_data.get("edit_turn", 0),
+            max_turns=obs_data.get("max_turns", 3),
+            done=obs_data.get("done", False),
+            reward=obs_data.get("reward", 0.0),
+            reward_components=obs_data.get("reward_components", {}),
+            metadata=obs_data.get("metadata", {}),
+        )
+        return StepResult(
+            observation=observation,
+            reward=observation.reward or 0.0,
+            done=observation.done,
+        )
+
+    def _parse_state(self, payload: Dict[str, Any]) -> PromptOpsState:
+        return PromptOpsState(
+            episode_id=payload.get("episode_id", ""),
+            step_count=payload.get("step_count", 0),
+            task_id=payload.get("task_id", ""),
+            task_type=payload.get("task_type", ""),
+            task_text=payload.get("task_text", ""),
+            history=payload.get("history", []),
+            best_reward=payload.get("best_reward", 0.0),
+            solved=payload.get("solved", False),
+        )

src/envs/promptops_arena/llm_under_test.py

@@ -0,0 +1,152 @@
+"""
+Frozen LLM-under-test. The agent's prompts are evaluated by running this model.
+
+Two backends:
+- "transformers": real Qwen2.5-0.5B-Instruct via HF transformers
+- "stub": deterministic stub for fast local CI / smoke tests
+
+Selected via env var PROMPTOPS_LLM_BACKEND (default: "stub" if torch unavailable).
+The stub recognizes a few hand-written "good" prompt patterns to give the
+env smoke test something non-zero to chew on.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import threading
+from typing import Optional
+
+
+_DEFAULT_MODEL = os.environ.get("PROMPTOPS_LLM_MODEL", "Qwen/Qwen2.5-0.5B-Instruct")
+_BACKEND = os.environ.get("PROMPTOPS_LLM_BACKEND", "auto").lower()
+_MAX_NEW_TOKENS = int(os.environ.get("PROMPTOPS_LLM_MAX_NEW_TOKENS", "256"))
+
+
+class _StubBackend:
+    """
+    Deterministic stub. Reads the system prompt + task, and produces a
+    plausible-looking completion that the verifiers can sometimes pass
+    when the prompt asks for the right format.
+
+    Heuristic logic:
+    - If prompt mentions <answer> tags, wrap a guessed answer in them
+    - For math: try to compute the answer naively (look for numbers in question)
+    - For code: emit a trivial function that returns 0 (will fail tests)
+    - For JSON: emit an empty object
+
+    This means: with a good prompt, math gets ~30% by luck; code/json need
+    a real LLM. That's fine — stub is only for plumbing tests.
+    """
+
+    name = "stub"
+
+    def generate(self, system_prompt: str, user_task: str) -> str:
+        sp = (system_prompt or "").lower()
+        ut = (user_task or "")
+
+        # Order matters: JSON first, then code, then math (most specific to least)
+        wants_json = "json" in sp
+        wants_code = ("python" in sp or "function" in sp) and "json" not in sp
+        wants_answer_tag = "<answer>" in (system_prompt or "")
+        wants_boxed = "boxed" in sp
+
+        if wants_json:
+            return "```json\n{}\n```"
+
+        if wants_code:
+            body = "def solve(*a, **k):\n    return 0\n"
+            return f"```python\n{body}```"
+
+        nums = re.findall(r"-?\d+(?:\.\d+)?", ut)
+        guess = nums[-1] if nums else "0"
+
+        if wants_answer_tag:
+            return f"Working...\n<answer>{guess}</answer>"
+        if wants_boxed:
+            return f"Working...\n\\boxed{{{guess}}}"
+        return f"The answer is {guess}."
+
+
+class _TransformersBackend:
+    name = "transformers"
+
+    def __init__(self, model_id: str = _DEFAULT_MODEL):
+        import torch  # type: ignore
+        from transformers import AutoModelForCausalLM, AutoTokenizer  # type: ignore
+
+        self._torch = torch
+        device = "cuda" if torch.cuda.is_available() else "cpu"
+        dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+        self.tokenizer = AutoTokenizer.from_pretrained(model_id)
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_id, torch_dtype=dtype, device_map=device,
+        )
+        self.model.eval()
+        self.device = device
+
+    def generate(self, system_prompt: str, user_task: str) -> str:
+        msgs = [
+            {"role": "system", "content": system_prompt or "You are a helpful assistant."},
+            {"role": "user", "content": user_task},
+        ]
+        encoded = self.tokenizer.apply_chat_template(
+            msgs, add_generation_prompt=True, return_tensors="pt",
+        )
+        # apply_chat_template may return a Tensor (older transformers) or a
+        # BatchEncoding/dict (newer); normalize to input_ids tensor.
+        if hasattr(encoded, "input_ids"):
+            input_ids = encoded.input_ids
+        elif isinstance(encoded, dict):
+            input_ids = encoded["input_ids"]
+        else:
+            input_ids = encoded
+        input_ids = input_ids.to(self.device)
+        with self._torch.no_grad():
+            out = self.model.generate(
+                input_ids=input_ids,
+                max_new_tokens=_MAX_NEW_TOKENS,
+                do_sample=False,
+                pad_token_id=self.tokenizer.eos_token_id,
+            )
+        text = self.tokenizer.decode(out[0][input_ids.shape[1]:], skip_special_tokens=True)
+        return text
+
+
+_lock = threading.Lock()
+_backend_singleton: Optional[object] = None
+
+
+def _select_backend() -> object:
+    global _backend_singleton
+    with _lock:
+        if _backend_singleton is not None:
+            return _backend_singleton
+        choice = _BACKEND
+        if choice == "auto":
+            try:
+                import torch  # noqa: F401
+                import transformers  # noqa: F401
+                choice = "transformers"
+            except ImportError:
+                choice = "stub"
+        if choice == "transformers":
+            try:
+                _backend_singleton = _TransformersBackend()
+            except Exception as e:
+                print(f"[llm_under_test] transformers backend failed ({e}); falling back to stub")
+                _backend_singleton = _StubBackend()
+        else:
+            _backend_singleton = _StubBackend()
+        return _backend_singleton
+
+
+def generate(system_prompt: str, user_task: str) -> str:
+    """Run the frozen LLM-under-test. Threadsafe singleton."""
+    backend = _select_backend()
+    return backend.generate(system_prompt, user_task)
+
+
+def backend_name() -> str:
+    return _select_backend().name  # type: ignore[attr-defined]

src/envs/promptops_arena/models.py

@@ -0,0 +1,48 @@
+"""
+Type-safe data contracts for the PromptOps Arena environment.
+
+Action: agent emits a full new system prompt to give to the LLM-under-test.
+Observation: task text, last completion, last reward, edit-turn counter.
+State: full episode history for logging / demo replay.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from openenv.core.env_server import Action, Observation, State
+from pydantic import Field
+
+
+class PromptOpsAction(Action):
+    """Agent's action: write/replace the system prompt for the LLM-under-test."""
+
+    new_system_prompt: str = Field(
+        ...,
+        description="Full system prompt the agent wants to give the frozen LLM-under-test",
+    )
+
+
+class PromptOpsObservation(Observation):
+    """What the agent sees after each step."""
+
+    task_text: str = ""
+    task_type: str = ""  # "math" | "code" | "json"
+    current_prompt: str = ""
+    last_completion: str = ""
+    last_reward: float = 0.0
+    last_correctness: float = 0.0
+    edit_turn: int = 0
+    max_turns: int = 3
+    reward_components: Dict[str, float] = Field(default_factory=dict)
+
+
+class PromptOpsState(State):
+    """Episode state for logging and replay."""
+
+    task_id: str = ""
+    task_type: str = ""
+    task_text: str = ""
+    history: List[Dict[str, Any]] = Field(default_factory=list)
+    best_reward: float = 0.0
+    solved: bool = False

src/envs/promptops_arena/server/Dockerfile

@@ -0,0 +1,15 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+COPY server/requirements.txt /app/server/requirements.txt
+RUN pip install --no-cache-dir -r /app/server/requirements.txt
+
+# Copy the env package
+COPY . /app/promptops_arena
+ENV PYTHONPATH=/app
+
+EXPOSE 8000
+ENV PROMPTOPS_LLM_BACKEND=transformers
+
+CMD ["uvicorn", "promptops_arena.server.app:app", "--host", "0.0.0.0", "--port", "8000"]

src/envs/promptops_arena/server/app.py

@@ -0,0 +1,44 @@
+"""
+FastAPI server for PromptOps Arena env.
+
+Run:
+    uvicorn src.envs.promptops_arena.server.app:app --host 0.0.0.0 --port 8000
+
+Or via:
+    python -m src.envs.promptops_arena.server.app
+"""
+
+from __future__ import annotations
+
+import os
+
+from openenv.core.env_server import create_app
+
+# Try in-repo first (when running scripts from project root); fall back
+# to fully-qualified package import (when installed).
+try:
+    from src.envs.promptops_arena.server.environment import PromptOpsArenaEnvironment
+    from src.envs.promptops_arena.models import PromptOpsAction, PromptOpsObservation
+except ImportError:  # pragma: no cover
+    from envs.promptops_arena.server.environment import PromptOpsArenaEnvironment
+    from envs.promptops_arena.models import PromptOpsAction, PromptOpsObservation
+
+
+max_concurrent = int(os.getenv("MAX_CONCURRENT_ENVS", "4"))
+
+app = create_app(
+    PromptOpsArenaEnvironment,
+    PromptOpsAction,
+    PromptOpsObservation,
+    env_name="promptops_arena",
+    max_concurrent_envs=max_concurrent,
+)
+
+
+def main() -> None:
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "8000")))
+
+
+if __name__ == "__main__":
+    main()

src/envs/promptops_arena/server/environment.py

@@ -0,0 +1,180 @@
+"""
+PromptOps Arena environment.
+
+reset()      : sample a task; return initial observation with empty prompt
+step(action) : run LLM-under-test with action.new_system_prompt + task,
+               verify, compute reward, return observation
+state        : full episode state with history (used for logging/replay)
+"""
+
+from __future__ import annotations
+
+import random
+import uuid
+from typing import Any, Optional
+
+from openenv.core.env_server import Environment
+
+from ..models import PromptOpsAction, PromptOpsObservation, PromptOpsState
+from ..tasks import load_tasks
+from ..verifiers import verify
+from .. import llm_under_test
+from .rewards import compute_reward
+
+
+class PromptOpsArenaEnvironment(Environment):
+    """
+    The agent's action is a full system prompt. We run the frozen LLM-under-test
+    with [system=action, user=task_text], verify, and reward.
+
+    Episode terminates when correctness == 1.0 OR edit_turn >= max_turns.
+    """
+
+    def __init__(
+        self,
+        max_turns: int = 3,
+        split: str = "train",
+        seed: Optional[int] = None,
+        task_types: Optional[list[str]] = None,
+    ):
+        super().__init__()
+        self._max_turns = max_turns
+        self._split = split
+        self._task_types = task_types
+        self._rng = random.Random(seed)
+        self._tasks = load_tasks(split=split, types=task_types)
+        if not self._tasks:
+            raise RuntimeError(
+                f"No tasks loaded for split={split!r} types={task_types!r}"
+            )
+        self._state: PromptOpsState = PromptOpsState(episode_id=str(uuid.uuid4()))
+        self._task: dict = {}
+        self._edit_turn: int = 0
+
+    # ---- OpenEnv API ----
+
+    def reset(
+        self,
+        seed: Optional[int] = None,
+        episode_id: Optional[str] = None,
+        task_id: Optional[str] = None,
+        **kwargs: Any,
+    ) -> PromptOpsObservation:
+        if seed is not None:
+            self._rng = random.Random(seed)
+
+        if task_id is not None:
+            matches = [t for t in self._tasks if t.get("id") == task_id]
+            self._task = matches[0] if matches else self._rng.choice(self._tasks)
+        else:
+            self._task = self._rng.choice(self._tasks)
+
+        self._edit_turn = 0
+        self._state = PromptOpsState(
+            episode_id=episode_id or str(uuid.uuid4()),
+            step_count=0,
+            task_id=self._task.get("id", ""),
+            task_type=self._task.get("type", ""),
+            task_text=self._task.get("question", ""),
+            history=[],
+            best_reward=0.0,
+            solved=False,
+        )
+
+        return PromptOpsObservation(
+            task_text=self._state.task_text,
+            task_type=self._state.task_type,
+            current_prompt="",
+            last_completion="",
+            last_reward=0.0,
+            last_correctness=0.0,
+            edit_turn=0,
+            max_turns=self._max_turns,
+            done=False,
+            reward=0.0,
+            metadata={
+                "task_id": self._state.task_id,
+                "episode_id": self._state.episode_id,
+            },
+            reward_components={},
+        )
+
+    def step(
+        self,
+        action: PromptOpsAction,
+        timeout_s: Optional[float] = None,
+        **kwargs: Any,
+    ) -> PromptOpsObservation:
+        if not self._task:
+            raise RuntimeError("Environment not initialized. Call reset() first.")
+
+        prompt = action.new_system_prompt or ""
+        completion = llm_under_test.generate(prompt, self._state.task_text)
+        verifier_result = verify(self._task, completion)
+        reward_dict = compute_reward(self._task, prompt, completion, verifier_result)
+        total = reward_dict["total"]
+
+        self._edit_turn += 1
+        self._state.step_count += 1
+        if total > self._state.best_reward:
+            self._state.best_reward = total
+        if reward_dict["correctness"] >= 1.0:
+            self._state.solved = True
+
+        self._state.history.append(
+            {
+                "edit_turn": self._edit_turn,
+                "system_prompt": prompt,
+                "completion": completion,
+                "reward": reward_dict,
+                "verifier": verifier_result,
+            }
+        )
+
+        done = self._state.solved or self._edit_turn >= self._max_turns
+
+        return PromptOpsObservation(
+            task_text=self._state.task_text,
+            task_type=self._state.task_type,
+            current_prompt=prompt,
+            last_completion=completion,
+            last_reward=total,
+            last_correctness=reward_dict["correctness"],
+            edit_turn=self._edit_turn,
+            max_turns=self._max_turns,
+            done=done,
+            reward=total,
+            reward_components=reward_dict,
+            metadata={
+                "task_id": self._state.task_id,
+                "episode_id": self._state.episode_id,
+                "verifier_details": verifier_result.get("details", ""),
+                "solved": self._state.solved,
+            },
+        )
+
+    @property
+    def state(self) -> PromptOpsState:
+        return self._state
+
+    # ---- in-process helper used by training (skip HTTP) ----
+
+    def execute_prompt(
+        self,
+        task: dict,
+        system_prompt: str,
+    ) -> dict:
+        """
+        Single-shot evaluation: given a task and a candidate system prompt,
+        run the LLM-under-test and return reward components + completion.
+
+        Used by the GRPO reward function during training to avoid HTTP latency.
+        """
+        completion = llm_under_test.generate(system_prompt, task.get("question", ""))
+        verifier_result = verify(task, completion)
+        reward_dict = compute_reward(task, system_prompt, completion, verifier_result)
+        return {
+            "reward": reward_dict,
+            "completion": completion,
+            "verifier": verifier_result,
+        }

src/envs/promptops_arena/server/requirements.txt

@@ -0,0 +1,7 @@
+openenv-core>=0.2.3
+fastapi>=0.110.0
+uvicorn>=0.27.0
+pydantic>=2.0.0
+jsonschema>=4.20.0
+transformers>=4.45.0
+torch>=2.4.0

src/envs/promptops_arena/server/rewards.py

@@ -0,0 +1,45 @@
+"""
+Reward function: decomposed, bounded, gated by correctness.
+
+total = correctness + 0.1 * format_bonus + brevity_penalty
+
+Where:
+  correctness  in {0.0, 1.0}            -- programmatic verifier
+  format_bonus in {0.0, 1.0}            -- multiplied by 0.1 in total
+  brevity_penalty in [-0.1, 0.0]        -- only if prompt > 800 chars
+                                           = -0.05 * max(0, (len-800))/200
+                                           clipped to -0.1
+
+If correctness == 0, the format_bonus is still added (small) but the agent
+cannot exceed 0.1 reward without correctness.
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+
+def compute_reward(
+    task: Dict[str, Any],
+    system_prompt: str,
+    completion: str,
+    verifier_result: Dict[str, Any],
+) -> Dict[str, float]:
+    correctness = float(verifier_result.get("correctness", 0.0))
+    format_ok = bool(verifier_result.get("format_ok", False))
+    format_bonus = 1.0 if format_ok else 0.0
+
+    p_len = len(system_prompt or "")
+    excess = max(0, p_len - 800)
+    brevity_penalty = -0.05 * (excess / 200.0)
+    if brevity_penalty < -0.1:
+        brevity_penalty = -0.1
+
+    total = correctness + 0.1 * format_bonus + brevity_penalty
+
+    return {
+        "correctness": correctness,
+        "format": format_bonus,
+        "brevity": brevity_penalty,
+        "total": total,
+    }

src/envs/promptops_arena/tasks/__init__.py

@@ -0,0 +1,3 @@
+from .loader import load_tasks, sample_task
+
+__all__ = ["load_tasks", "sample_task"]

src/envs/promptops_arena/tasks/code.jsonl

@@ -0,0 +1,30 @@
+{"id": "code_001", "type": "code", "split": "train", "question": "Write a Python function `add(a, b)` that returns the sum of two numbers.", "tests": ["assert add(2, 3) == 5", "assert add(-1, 1) == 0", "assert add(0, 0) == 0"]}
+{"id": "code_002", "type": "code", "split": "train", "question": "Write a Python function `is_even(n)` that returns True if n is even, False otherwise.", "tests": ["assert is_even(4) is True", "assert is_even(7) is False", "assert is_even(0) is True"]}
+{"id": "code_003", "type": "code", "split": "train", "question": "Write a Python function `reverse_string(s)` that returns the reverse of string s.", "tests": ["assert reverse_string('hello') == 'olleh'", "assert reverse_string('') == ''", "assert reverse_string('a') == 'a'"]}
+{"id": "code_004", "type": "code", "split": "train", "question": "Write a Python function `factorial(n)` that returns n! for n >= 0.", "tests": ["assert factorial(0) == 1", "assert factorial(5) == 120", "assert factorial(1) == 1"]}
+{"id": "code_005", "type": "code", "split": "train", "question": "Write a Python function `count_vowels(s)` that counts vowels (aeiou, case insensitive) in string s.", "tests": ["assert count_vowels('hello') == 2", "assert count_vowels('AEIOU') == 5", "assert count_vowels('xyz') == 0"]}
+{"id": "code_006", "type": "code", "split": "train", "question": "Write a Python function `max_in_list(lst)` that returns the largest number in lst.", "tests": ["assert max_in_list([1, 2, 3]) == 3", "assert max_in_list([-5, -1, -10]) == -1", "assert max_in_list([7]) == 7"]}
+{"id": "code_007", "type": "code", "split": "train", "question": "Write a Python function `is_palindrome(s)` that returns True if s reads the same forwards and backwards.", "tests": ["assert is_palindrome('racecar') is True", "assert is_palindrome('hello') is False", "assert is_palindrome('') is True"]}
+{"id": "code_008", "type": "code", "split": "train", "question": "Write a Python function `fibonacci(n)` that returns the nth Fibonacci number (fib(0)=0, fib(1)=1).", "tests": ["assert fibonacci(0) == 0", "assert fibonacci(1) == 1", "assert fibonacci(7) == 13"]}
+{"id": "code_009", "type": "code", "split": "train", "question": "Write a Python function `unique(lst)` that returns a list of unique elements preserving order.", "tests": ["assert unique([1, 2, 2, 3, 1]) == [1, 2, 3]", "assert unique([]) == []", "assert unique(['a', 'a', 'b']) == ['a', 'b']"]}
+{"id": "code_010", "type": "code", "split": "train", "question": "Write a Python function `sum_digits(n)` that sums the digits of non-negative integer n.", "tests": ["assert sum_digits(123) == 6", "assert sum_digits(0) == 0", "assert sum_digits(9999) == 36"]}
+{"id": "code_011", "type": "code", "split": "train", "question": "Write a Python function `gcd(a, b)` that returns the greatest common divisor.", "tests": ["assert gcd(12, 18) == 6", "assert gcd(7, 13) == 1", "assert gcd(100, 25) == 25"]}
+{"id": "code_012", "type": "code", "split": "train", "question": "Write a Python function `flatten(lst)` that flattens a list of lists by one level.", "tests": ["assert flatten([[1,2],[3,4]]) == [1,2,3,4]", "assert flatten([]) == []", "assert flatten([[1],[2,3],[]]) == [1,2,3]"]}
+{"id": "code_013", "type": "code", "split": "train", "question": "Write a Python function `is_prime(n)` that returns True if n is a prime > 1.", "tests": ["assert is_prime(2) is True", "assert is_prime(15) is False", "assert is_prime(17) is True", "assert is_prime(1) is False"]}
+{"id": "code_014", "type": "code", "split": "train", "question": "Write a Python function `word_count(s)` that returns the number of whitespace-separated words.", "tests": ["assert word_count('hello world') == 2", "assert word_count('') == 0", "assert word_count('  one  two   three ') == 3"]}
+{"id": "code_015", "type": "code", "split": "train", "question": "Write a Python function `second_largest(lst)` that returns the second largest distinct number.", "tests": ["assert second_largest([1, 2, 3, 4]) == 3", "assert second_largest([5, 5, 4]) == 4", "assert second_largest([10, 20]) == 10"]}
+{"id": "code_016", "type": "code", "split": "train", "question": "Write a Python function `caesar(s, k)` that shifts each letter by k positions, preserving case; non-letters unchanged.", "tests": ["assert caesar('abc', 1) == 'bcd'", "assert caesar('XYZ', 3) == 'ABC'", "assert caesar('Hello, World!', 13) == 'Uryyb, Jbeyq!'"]}
+{"id": "code_017", "type": "code", "split": "train", "question": "Write a Python function `merge_sorted(a, b)` that merges two sorted lists into one sorted list.", "tests": ["assert merge_sorted([1,3,5],[2,4,6]) == [1,2,3,4,5,6]", "assert merge_sorted([],[1,2]) == [1,2]", "assert merge_sorted([1,1],[1,1]) == [1,1,1,1]"]}
+{"id": "code_018", "type": "code", "split": "train", "question": "Write a Python function `most_common(lst)` that returns the most frequent element (any if tied).", "tests": ["assert most_common([1,2,2,3]) == 2", "assert most_common(['a','b','a']) == 'a'", "assert most_common([5]) == 5"]}
+{"id": "code_019", "type": "code", "split": "train", "question": "Write a Python function `chunks(lst, n)` that splits lst into chunks of size n.", "tests": ["assert chunks([1,2,3,4,5], 2) == [[1,2],[3,4],[5]]", "assert chunks([], 3) == []", "assert chunks([1,2,3], 1) == [[1],[2],[3]]"]}
+{"id": "code_020", "type": "code", "split": "train", "question": "Write a Python function `roman(n)` that converts integer 1-3999 to Roman numerals.", "tests": ["assert roman(1) == 'I'", "assert roman(4) == 'IV'", "assert roman(1994) == 'MCMXCIV'"]}
+{"id": "code_t01", "type": "code", "split": "test", "question": "Write a Python function `square(x)` that returns x squared.", "tests": ["assert square(3) == 9", "assert square(-4) == 16", "assert square(0) == 0"]}
+{"id": "code_t02", "type": "code", "split": "test", "question": "Write a Python function `is_anagram(a, b)` that checks if two strings are anagrams (case-insensitive, ignoring spaces).", "tests": ["assert is_anagram('listen', 'silent') is True", "assert is_anagram('Hello', 'World') is False", "assert is_anagram('a gentleman', 'elegant man') is True"]}
+{"id": "code_t03", "type": "code", "split": "test", "question": "Write a Python function `count_words(s)` that returns a dict mapping each word to its count (whitespace-separated, lowercase).", "tests": ["assert count_words('a a b') == {'a': 2, 'b': 1}", "assert count_words('') == {}", "assert count_words('Hi hi HI') == {'hi': 3}"]}
+{"id": "code_t04", "type": "code", "split": "test", "question": "Write a Python function `power(b, e)` that computes b**e for non-negative integer e using a loop (no ** operator).", "tests": ["assert power(2, 10) == 1024", "assert power(5, 0) == 1", "assert power(3, 3) == 27"]}
+{"id": "code_t05", "type": "code", "split": "test", "question": "Write a Python function `compress(s)` that returns 'a3b2c1' style run-length encoding.", "tests": ["assert compress('aaabbc') == 'a3b2c1'", "assert compress('abc') == 'a1b1c1'", "assert compress('') == ''"]}
+{"id": "code_t06", "type": "code", "split": "test", "question": "Write a Python function `balanced(s)` that returns True if all (), [], {} pairs are balanced and properly nested.", "tests": ["assert balanced('()[]{}') is True", "assert balanced('([)]') is False", "assert balanced('') is True"]}
+{"id": "code_t07", "type": "code", "split": "test", "question": "Write a Python function `lcm(a, b)` that returns the least common multiple of two positive integers.", "tests": ["assert lcm(4, 6) == 12", "assert lcm(7, 13) == 91", "assert lcm(1, 5) == 5"]}
+{"id": "code_t08", "type": "code", "split": "test", "question": "Write a Python function `rotate(lst, k)` that rotates list right by k positions.", "tests": ["assert rotate([1,2,3,4], 1) == [4,1,2,3]", "assert rotate([1,2,3], 0) == [1,2,3]", "assert rotate([1,2,3], 4) == [3,1,2]"]}
+{"id": "code_t09", "type": "code", "split": "test", "question": "Write a Python function `digit_sum_seq(n)` that repeatedly sums digits until single digit (digital root).", "tests": ["assert digit_sum_seq(38) == 2", "assert digit_sum_seq(0) == 0", "assert digit_sum_seq(123456) == 3"]}
+{"id": "code_t10", "type": "code", "split": "test", "question": "Write a Python function `pairs_summing_to(lst, target)` that returns list of unique sorted tuples (a,b) with a+b==target, a<=b.", "tests": ["assert pairs_summing_to([1,2,3,4], 5) == [(1,4),(2,3)]", "assert pairs_summing_to([2,2,3], 4) == [(2,2)]", "assert pairs_summing_to([1,2], 10) == []"]}

src/envs/promptops_arena/tasks/json_extract.jsonl

@@ -0,0 +1,20 @@
+{"id": "json_001", "type": "json", "split": "train", "question": "Extract the person's name and age from this text as JSON with keys 'name' (string) and 'age' (integer): 'Alice Johnson, 32 years old, lives in Boston.'", "schema": {"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"]}, "expected": {"name": "Alice Johnson", "age": 32}}
+{"id": "json_002", "type": "json", "split": "train", "question": "Parse this order: 'Order #4521 for 3 widgets at $9.99 each.' Output JSON with order_id (int), quantity (int), price_each (number).", "schema": {"type": "object", "properties": {"order_id": {"type": "integer"}, "quantity": {"type": "integer"}, "price_each": {"type": "number"}}, "required": ["order_id", "quantity", "price_each"]}, "expected": {"order_id": 4521, "quantity": 3, "price_each": 9.99}}
+{"id": "json_003", "type": "json", "split": "train", "question": "Extract event info from 'Concert by The Beatles on 2025-06-14 at Wembley Stadium'. JSON keys: artist (str), date (str YYYY-MM-DD), venue (str).", "schema": {"type": "object", "properties": {"artist": {"type": "string"}, "date": {"type": "string"}, "venue": {"type": "string"}}, "required": ["artist", "date", "venue"]}, "expected": {"artist": "The Beatles", "date": "2025-06-14", "venue": "Wembley Stadium"}}
+{"id": "json_004", "type": "json", "split": "train", "question": "Extract from 'Book: The Pragmatic Programmer by David Thomas, 320 pages, ISBN 978-0135957059'. JSON: title (str), author (str), pages (int).", "schema": {"type": "object", "properties": {"title": {"type": "string"}, "author": {"type": "string"}, "pages": {"type": "integer"}}, "required": ["title", "author", "pages"]}, "expected": {"title": "The Pragmatic Programmer", "author": "David Thomas", "pages": 320}}
+{"id": "json_005", "type": "json", "split": "train", "question": "Parse 'Flight AA101 from JFK to LAX departs 14:30'. JSON keys: flight (str), from (str), to (str), departure (str).", "schema": {"type": "object", "properties": {"flight": {"type": "string"}, "from": {"type": "string"}, "to": {"type": "string"}, "departure": {"type": "string"}}, "required": ["flight", "from", "to", "departure"]}, "expected": {"flight": "AA101", "from": "JFK", "to": "LAX", "departure": "14:30"}}
+{"id": "json_006", "type": "json", "split": "train", "question": "From 'Product SKU-998 priced at 49.50 USD, in stock: 12 units', output JSON: sku (str), price (number), currency (str), stock (int).", "schema": {"type": "object", "properties": {"sku": {"type": "string"}, "price": {"type": "number"}, "currency": {"type": "string"}, "stock": {"type": "integer"}}, "required": ["sku", "price", "currency", "stock"]}, "expected": {"sku": "SKU-998", "price": 49.5, "currency": "USD", "stock": 12}}
+{"id": "json_007", "type": "json", "split": "train", "question": "Parse contact: 'Email: bob@example.com, Phone: +1-555-0100'. JSON: email (str), phone (str).", "schema": {"type": "object", "properties": {"email": {"type": "string"}, "phone": {"type": "string"}}, "required": ["email", "phone"]}, "expected": {"email": "bob@example.com", "phone": "+1-555-0100"}}
+{"id": "json_008", "type": "json", "split": "train", "question": "From 'Movie: Inception (2010), directed by Christopher Nolan, runtime 148 min', output JSON: title (str), year (int), director (str), runtime (int).", "schema": {"type": "object", "properties": {"title": {"type": "string"}, "year": {"type": "integer"}, "director": {"type": "string"}, "runtime": {"type": "integer"}}, "required": ["title", "year", "director", "runtime"]}, "expected": {"title": "Inception", "year": 2010, "director": "Christopher Nolan", "runtime": 148}}
+{"id": "json_009", "type": "json", "split": "train", "question": "Parse 'Temperature in Tokyo: 24.5°C, humidity 60%'. JSON: city (str), temp_c (number), humidity (int).", "schema": {"type": "object", "properties": {"city": {"type": "string"}, "temp_c": {"type": "number"}, "humidity": {"type": "integer"}}, "required": ["city", "temp_c", "humidity"]}, "expected": {"city": "Tokyo", "temp_c": 24.5, "humidity": 60}}
+{"id": "json_010", "type": "json", "split": "train", "question": "Extract from 'Recipe: pancakes, serves 4, prep 10 min, cook 15 min'. JSON: name (str), serves (int), prep_min (int), cook_min (int).", "schema": {"type": "object", "properties": {"name": {"type": "string"}, "serves": {"type": "integer"}, "prep_min": {"type": "integer"}, "cook_min": {"type": "integer"}}, "required": ["name", "serves", "prep_min", "cook_min"]}, "expected": {"name": "pancakes", "serves": 4, "prep_min": 10, "cook_min": 15}}
+{"id": "json_t01", "type": "json", "split": "test", "question": "From 'Employee Jane Smith, ID 7821, department Engineering, salary 95000', JSON: name (str), id (int), department (str), salary (int).", "schema": {"type": "object", "properties": {"name": {"type": "string"}, "id": {"type": "integer"}, "department": {"type": "string"}, "salary": {"type": "integer"}}, "required": ["name", "id", "department", "salary"]}, "expected": {"name": "Jane Smith", "id": 7821, "department": "Engineering", "salary": 95000}}
+{"id": "json_t02", "type": "json", "split": "test", "question": "Parse 'Hotel: Grand Plaza, 4 stars, 250 rooms, $180/night'. JSON: name (str), stars (int), rooms (int), nightly_rate (number).", "schema": {"type": "object", "properties": {"name": {"type": "string"}, "stars": {"type": "integer"}, "rooms": {"type": "integer"}, "nightly_rate": {"type": "number"}}, "required": ["name", "stars", "rooms", "nightly_rate"]}, "expected": {"name": "Grand Plaza", "stars": 4, "rooms": 250, "nightly_rate": 180}}
+{"id": "json_t03", "type": "json", "split": "test", "question": "Extract from 'Match: Real Madrid 3-1 Barcelona on 2024-03-15'. JSON: home (str), away (str), home_score (int), away_score (int), date (str).", "schema": {"type": "object", "properties": {"home": {"type": "string"}, "away": {"type": "string"}, "home_score": {"type": "integer"}, "away_score": {"type": "integer"}, "date": {"type": "string"}}, "required": ["home", "away", "home_score", "away_score", "date"]}, "expected": {"home": "Real Madrid", "away": "Barcelona", "home_score": 3, "away_score": 1, "date": "2024-03-15"}}
+{"id": "json_t04", "type": "json", "split": "test", "question": "From 'Library book \"Dune\" by Frank Herbert, due 2025-12-01, fine $0.25/day', JSON: title (str), author (str), due (str), fine_per_day (number).", "schema": {"type": "object", "properties": {"title": {"type": "string"}, "author": {"type": "string"}, "due": {"type": "string"}, "fine_per_day": {"type": "number"}}, "required": ["title", "author", "due", "fine_per_day"]}, "expected": {"title": "Dune", "author": "Frank Herbert", "due": "2025-12-01", "fine_per_day": 0.25}}
+{"id": "json_t05", "type": "json", "split": "test", "question": "Parse 'GPS: lat 40.7128, lon -74.0060, alt 10m'. JSON: lat (number), lon (number), alt_m (integer).", "schema": {"type": "object", "properties": {"lat": {"type": "number"}, "lon": {"type": "number"}, "alt_m": {"type": "integer"}}, "required": ["lat", "lon", "alt_m"]}, "expected": {"lat": 40.7128, "lon": -74.006, "alt_m": 10}}
+{"id": "json_t06", "type": "json", "split": "test", "question": "From 'Course CS101: Intro to CS, 3 credits, Prof. Adams', JSON: code (str), title (str), credits (int), instructor (str).", "schema": {"type": "object", "properties": {"code": {"type": "string"}, "title": {"type": "string"}, "credits": {"type": "integer"}, "instructor": {"type": "string"}}, "required": ["code", "title", "credits", "instructor"]}, "expected": {"code": "CS101", "title": "Intro to CS", "credits": 3, "instructor": "Prof. Adams"}}
+{"id": "json_t07", "type": "json", "split": "test", "question": "Extract from 'Package PKG-2025-0042 weighs 3.2 kg, ships from Berlin to Rome'. JSON: id (str), weight_kg (number), origin (str), destination (str).", "schema": {"type": "object", "properties": {"id": {"type": "string"}, "weight_kg": {"type": "number"}, "origin": {"type": "string"}, "destination": {"type": "string"}}, "required": ["id", "weight_kg", "origin", "destination"]}, "expected": {"id": "PKG-2025-0042", "weight_kg": 3.2, "origin": "Berlin", "destination": "Rome"}}
+{"id": "json_t08", "type": "json", "split": "test", "question": "Parse 'User account: bob_42, joined 2022-01-15, posts 1234'. JSON: username (str), joined (str), posts (int).", "schema": {"type": "object", "properties": {"username": {"type": "string"}, "joined": {"type": "string"}, "posts": {"type": "integer"}}, "required": ["username", "joined", "posts"]}, "expected": {"username": "bob_42", "joined": "2022-01-15", "posts": 1234}}
+{"id": "json_t09", "type": "json", "split": "test", "question": "From 'Stock AAPL closed at 178.45, change +1.2%, volume 52000000', JSON: ticker (str), close (number), change_pct (number), volume (integer).", "schema": {"type": "object", "properties": {"ticker": {"type": "string"}, "close": {"type": "number"}, "change_pct": {"type": "number"}, "volume": {"type": "integer"}}, "required": ["ticker", "close", "change_pct", "volume"]}, "expected": {"ticker": "AAPL", "close": 178.45, "change_pct": 1.2, "volume": 52000000}}
+{"id": "json_t10", "type": "json", "split": "test", "question": "Extract from 'Weather alert: Severe thunderstorm in county Marion, expires 21:30 UTC'. JSON: type (str), county (str), expires_utc (str).", "schema": {"type": "object", "properties": {"type": {"type": "string"}, "county": {"type": "string"}, "expires_utc": {"type": "string"}}, "required": ["type", "county", "expires_utc"]}, "expected": {"type": "Severe thunderstorm", "county": "Marion", "expires_utc": "21:30 UTC"}}

src/envs/promptops_arena/tasks/loader.py

@@ -0,0 +1,45 @@
+"""Task loader: reads JSONL files in this directory."""
+
+from __future__ import annotations
+
+import json
+import random
+from pathlib import Path
+from typing import Iterable, List, Optional
+
+
+_TASK_DIR = Path(__file__).parent
+_FILES = ["math.jsonl", "code.jsonl", "json_extract.jsonl"]
+
+
+def load_tasks(
+    split: Optional[str] = None,
+    types: Optional[Iterable[str]] = None,
+) -> List[dict]:
+    """
+    Load all tasks, optionally filtered by split ('train'/'test') and types.
+
+    Returns: list of task dicts.
+    """
+    out: List[dict] = []
+    for fname in _FILES:
+        path = _TASK_DIR / fname
+        if not path.exists():
+            continue
+        with path.open("r", encoding="utf-8") as f:
+            for line in f:
+                line = line.strip()
+                if not line:
+                    continue
+                t = json.loads(line)
+                if split is not None and t.get("split") != split:
+                    continue
+                if types is not None and t.get("type") not in set(types):
+                    continue
+                out.append(t)
+    return out
+
+
+def sample_task(rng: random.Random, split: str = "train") -> dict:
+    tasks = load_tasks(split=split)
+    return rng.choice(tasks)

src/envs/promptops_arena/tasks/math.jsonl

@@ -0,0 +1,40 @@
+{"id": "math_001", "type": "math", "split": "train", "question": "Janet has 24 apples. She gives 8 to her friend and eats 3. How many apples does she have left?", "answer": "13"}
+{"id": "math_002", "type": "math", "split": "train", "question": "A train travels 60 miles per hour for 2.5 hours. How far does it go?", "answer": "150"}
+{"id": "math_003", "type": "math", "split": "train", "question": "Sarah bought 5 packs of pencils, each containing 12 pencils. She used 17 pencils. How many are left?", "answer": "43"}
+{"id": "math_004", "type": "math", "split": "train", "question": "There are 30 students in a class. 40% are boys. How many girls are there?", "answer": "18"}
+{"id": "math_005", "type": "math", "split": "train", "question": "A book costs $15. With a 20% discount, what is the new price?", "answer": "12"}
+{"id": "math_006", "type": "math", "split": "train", "question": "Tom has 3 times as many marbles as Jerry. Jerry has 14. How many do they have together?", "answer": "56"}
+{"id": "math_007", "type": "math", "split": "train", "question": "A rectangle has length 8 and width 5. What is its area?", "answer": "40"}
+{"id": "math_008", "type": "math", "split": "train", "question": "If 4 workers paint 1 wall in 3 hours, how many hours will 2 workers take to paint the same wall?", "answer": "6"}
+{"id": "math_009", "type": "math", "split": "train", "question": "A bag has 7 red and 5 blue balls. After adding 8 more red balls, what fraction are red? Give the answer as a decimal.", "answer": "0.75"}
+{"id": "math_010", "type": "math", "split": "train", "question": "Lily reads 25 pages a day. How many pages in 2 weeks?", "answer": "350"}
+{"id": "math_011", "type": "math", "split": "train", "question": "A pizza is cut into 8 slices. If 3 friends each eat 2 slices, how many slices remain?", "answer": "2"}
+{"id": "math_012", "type": "math", "split": "train", "question": "A car uses 1 gallon for every 30 miles. How many gallons for 240 miles?", "answer": "8"}
+{"id": "math_013", "type": "math", "split": "train", "question": "If x + 7 = 22, what is x?", "answer": "15"}
+{"id": "math_014", "type": "math", "split": "train", "question": "A box contains 48 cookies. They are split equally among 6 children. How many cookies does each get?", "answer": "8"}
+{"id": "math_015", "type": "math", "split": "train", "question": "A shop sells shirts at $20 each. With buy-2-get-1-free, what is the cost of 3 shirts?", "answer": "40"}
+{"id": "math_016", "type": "math", "split": "train", "question": "A water tank holds 200 liters. It is filled at 25 liters per minute. How many minutes to fill it?", "answer": "8"}
+{"id": "math_017", "type": "math", "split": "train", "question": "If today is Wednesday, what day will it be 50 days from now? Answer with a number 1-7 where Monday=1.", "answer": "1"}
+{"id": "math_018", "type": "math", "split": "train", "question": "A square garden has perimeter 36 meters. What is its area?", "answer": "81"}
+{"id": "math_019", "type": "math", "split": "train", "question": "A class of 28 students has 4 absent. What percent are present?", "answer": "85.714286"}
+{"id": "math_020", "type": "math", "split": "train", "question": "A bus has 50 seats. 32 are taken. What fraction of seats are empty? Give as a decimal.", "answer": "0.36"}
+{"id": "math_021", "type": "math", "split": "train", "question": "Peter has $80. He spends 1/4 on books and 3/8 on food. How much is left?", "answer": "30"}
+{"id": "math_022", "type": "math", "split": "train", "question": "A triangle has angles in ratio 1:2:3. What is the largest angle in degrees?", "answer": "90"}
+{"id": "math_023", "type": "math", "split": "train", "question": "If 5x = 35, what is 3x + 4?", "answer": "25"}
+{"id": "math_024", "type": "math", "split": "train", "question": "A jar has 24 candies. Half are red, a third are green, the rest are blue. How many are blue?", "answer": "4"}
+{"id": "math_025", "type": "math", "split": "train", "question": "A man earns $400 per week. He saves 15%. How much does he save in 4 weeks?", "answer": "240"}
+{"id": "math_026", "type": "math", "split": "train", "question": "A cube has volume 27. What is its side length?", "answer": "3"}
+{"id": "math_027", "type": "math", "split": "train", "question": "If a + b = 10 and a - b = 2, what is a*b?", "answer": "24"}
+{"id": "math_028", "type": "math", "split": "train", "question": "A movie is 2 hours and 15 minutes long. If it starts at 7:45 PM, when does it end? Give answer as 24-hour HHMM.", "answer": "2200"}
+{"id": "math_029", "type": "math", "split": "train", "question": "A baker uses 3 eggs per cake. He has 5 dozen eggs. How many cakes can he make?", "answer": "20"}
+{"id": "math_030", "type": "math", "split": "train", "question": "A 50-meter rope is cut into pieces of 2.5 meters each. How many pieces?", "answer": "20"}
+{"id": "math_t01", "type": "math", "split": "test", "question": "Mia has 45 stickers. She gives 1/3 to her sister and 1/5 of the remainder to her friend. How many does she have left?", "answer": "24"}
+{"id": "math_t02", "type": "math", "split": "test", "question": "A farmer has 5 times as many cows as horses. He has 30 cows. How many animals total?", "answer": "36"}
+{"id": "math_t03", "type": "math", "split": "test", "question": "A car costs $20000. It depreciates 10% per year. What is its value after 2 years?", "answer": "16200"}
+{"id": "math_t04", "type": "math", "split": "test", "question": "A right triangle has legs 6 and 8. What is the hypotenuse?", "answer": "10"}
+{"id": "math_t05", "type": "math", "split": "test", "question": "Three consecutive integers sum to 72. What is the largest?", "answer": "25"}
+{"id": "math_t06", "type": "math", "split": "test", "question": "A class average is 80 over 10 students. If one student scored 50, what was the average of the other 9?", "answer": "83.333333"}
+{"id": "math_t07", "type": "math", "split": "test", "question": "If 3x + 5 = 2x + 12, what is x?", "answer": "7"}
+{"id": "math_t08", "type": "math", "split": "test", "question": "A pool fills in 6 hours with one pipe and 4 hours with another. How many hours with both pipes? Give as a decimal.", "answer": "2.4"}
+{"id": "math_t09", "type": "math", "split": "test", "question": "A circle has radius 7. Use pi=22/7. What is its area?", "answer": "154"}
+{"id": "math_t10", "type": "math", "split": "test", "question": "If today is the 5th of a 30-day month, what day of the month is it 60 days later?", "answer": "4"}

src/envs/promptops_arena/verifiers/__init__.py

@@ -0,0 +1,21 @@
+from .math_verifier import verify_math
+from .code_verifier import verify_code
+from .json_verifier import verify_json
+
+__all__ = ["verify_math", "verify_code", "verify_json", "verify"]
+
+
+def verify(task: dict, completion: str) -> dict:
+    """
+    Dispatch to the right verifier based on task['type'].
+
+    Returns a dict: {correctness: 0.0|1.0, format_ok: bool, details: str}
+    """
+    task_type = task.get("type", "")
+    if task_type == "math":
+        return verify_math(task, completion)
+    if task_type == "code":
+        return verify_code(task, completion)
+    if task_type == "json":
+        return verify_json(task, completion)
+    return {"correctness": 0.0, "format_ok": False, "details": f"Unknown task type: {task_type}"}

src/envs/promptops_arena/verifiers/code_verifier.py

@@ -0,0 +1,65 @@
+"""
+Code verifier: extract a python code block from completion, run it in a
+subprocess with the task's test cases appended, with a hard timeout.
+
+NEVER use in-process exec(). Subprocess only.
+"""
+
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+from typing import Any, Dict
+
+
+_CODE_BLOCK = re.compile(r"```(?:python)?\s*\n(.*?)```", re.DOTALL | re.IGNORECASE)
+
+
+def _extract_code(completion: str) -> tuple[str | None, bool]:
+    if not completion:
+        return None, False
+    m = _CODE_BLOCK.search(completion)
+    if m:
+        return m.group(1).strip(), True
+    if "def " in completion:
+        return completion.strip(), False
+    return None, False
+
+
+def verify_code(task: Dict[str, Any], completion: str) -> Dict[str, Any]:
+    code, format_ok = _extract_code(completion or "")
+    if code is None:
+        return {"correctness": 0.0, "format_ok": False, "details": "no code"}
+
+    tests = task.get("tests", [])
+    if not tests:
+        return {"correctness": 0.0, "format_ok": format_ok, "details": "no tests in task"}
+
+    test_block = "\n".join(tests)
+    program = f"{code}\n\n# --- tests ---\n{test_block}\nprint('__OK__')\n"
+
+    with tempfile.TemporaryDirectory() as td:
+        path = Path(td) / "candidate.py"
+        path.write_text(program, encoding="utf-8")
+        try:
+            proc = subprocess.run(
+                [sys.executable, str(path)],
+                capture_output=True,
+                text=True,
+                timeout=5,
+            )
+        except subprocess.TimeoutExpired:
+            return {"correctness": 0.0, "format_ok": format_ok, "details": "timeout"}
+        except Exception as e:
+            return {"correctness": 0.0, "format_ok": format_ok, "details": f"runner err: {e}"}
+
+    ok = proc.returncode == 0 and "__OK__" in (proc.stdout or "")
+    detail = (proc.stderr or proc.stdout or "")[:200].replace("\n", " ")
+    return {
+        "correctness": 1.0 if ok else 0.0,
+        "format_ok": format_ok,
+        "details": detail,
+    }

src/envs/promptops_arena/verifiers/json_verifier.py

@@ -0,0 +1,85 @@
+"""
+JSON verifier: extract a JSON object from completion, validate against a
+jsonschema, and check value equality on required fields.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Any, Dict
+
+from jsonschema import validate, ValidationError
+
+
+_JSON_BLOCK = re.compile(r"```(?:json)?\s*\n(.*?)```", re.DOTALL | re.IGNORECASE)
+_OBJ = re.compile(r"\{.*\}", re.DOTALL)
+
+
+def _extract_json(completion: str) -> tuple[Any, bool]:
+    if not completion:
+        return None, False
+    m = _JSON_BLOCK.search(completion)
+    candidate = m.group(1).strip() if m else None
+    format_ok = m is not None
+    if candidate is None:
+        m2 = _OBJ.search(completion)
+        if m2:
+            candidate = m2.group(0)
+    if candidate is None:
+        return None, False
+    try:
+        return json.loads(candidate), format_ok
+    except json.JSONDecodeError:
+        return None, format_ok
+
+
+def _strip_nones(x: Any) -> Any:
+    """HuggingFace `datasets` unifies nested dict schemas across rows by
+    null-padding missing keys. That turns a clean schema like
+    {"properties": {"name": {...}, "age": {...}}} into one with
+    {"properties": {"name": ..., "age": ..., "email": None, ...}} if other rows
+    in the dataset had those keys. jsonschema rejects the Nones. Recursively
+    drop them so verification is robust to that mangling.
+    """
+    if isinstance(x, dict):
+        return {k: _strip_nones(v) for k, v in x.items() if v is not None}
+    if isinstance(x, list):
+        return [_strip_nones(v) for v in x if v is not None]
+    return x
+
+
+def verify_json(task: Dict[str, Any], completion: str) -> Dict[str, Any]:
+    obj, format_ok = _extract_json(completion or "")
+    if obj is None:
+        return {"correctness": 0.0, "format_ok": format_ok, "details": "parse fail"}
+
+    schema = _strip_nones(task.get("schema", {}))
+    if schema:
+        try:
+            validate(instance=obj, schema=schema)
+        except ValidationError as e:
+            return {
+                "correctness": 0.0,
+                "format_ok": format_ok,
+                "details": f"schema: {str(e.message)[:120]}",
+            }
+        except Exception as e:
+            # Schema itself malformed (e.g., still has Nones somewhere).
+            return {
+                "correctness": 0.0,
+                "format_ok": format_ok,
+                "details": f"schema-error: {type(e).__name__}: {str(e)[:120]}",
+            }
+
+    expected = _strip_nones(task.get("expected", {}))
+    if expected:
+        for k, v in expected.items():
+            if obj.get(k) != v:
+                return {
+                    "correctness": 0.0,
+                    "format_ok": format_ok,
+                    "details": f"mismatch {k}: got {obj.get(k)!r} expected {v!r}",
+                }
+
+    return {"correctness": 1.0, "format_ok": format_ok, "details": "ok"}

src/envs/promptops_arena/verifiers/math_verifier.py

@@ -0,0 +1,73 @@
+"""
+Math verifier: extract final numeric answer and exact-match against ground truth.
+
+Format-bonus is awarded if completion contains <answer>...</answer> tags
+or a \\boxed{...} expression.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Any, Dict
+
+
+_ANSWER_TAG = re.compile(r"<answer>\s*(.*?)\s*</answer>", re.DOTALL | re.IGNORECASE)
+_BOXED = re.compile(r"\\boxed\{([^{}]+)\}")
+_FINAL_LINE = re.compile(r"(?:final answer|answer)\s*[:=]\s*([^\n]+)", re.IGNORECASE)
+_NUMBER = re.compile(r"-?\d+(?:\.\d+)?")
+
+
+def _normalize_number(s: str) -> str | None:
+    """Pull the first number out of s, return its canonical string form."""
+    if s is None:
+        return None
+    s = s.strip().replace(",", "").replace("$", "").rstrip(".")
+    m = _NUMBER.search(s)
+    if not m:
+        return None
+    try:
+        v = float(m.group(0))
+    except ValueError:
+        return None
+    if v.is_integer():
+        return str(int(v))
+    return f"{v:.6f}".rstrip("0").rstrip(".")
+
+
+def _extract(completion: str) -> tuple[str | None, bool]:
+    """Return (extracted_answer, format_ok)."""
+    if not completion:
+        return None, False
+
+    m = _ANSWER_TAG.search(completion)
+    if m:
+        return _normalize_number(m.group(1)), True
+
+    m = _BOXED.search(completion)
+    if m:
+        return _normalize_number(m.group(1)), True
+
+    m = _FINAL_LINE.search(completion)
+    if m:
+        return _normalize_number(m.group(1)), False
+
+    nums = _NUMBER.findall(completion)
+    if nums:
+        return _normalize_number(nums[-1]), False
+
+    return None, False
+
+
+def verify_math(task: Dict[str, Any], completion: str) -> Dict[str, Any]:
+    expected = _normalize_number(str(task.get("answer", "")))
+    extracted, format_ok = _extract(completion or "")
+
+    if expected is None:
+        return {"correctness": 0.0, "format_ok": format_ok, "details": "bad ground truth"}
+
+    correct = extracted is not None and extracted == expected
+    return {
+        "correctness": 1.0 if correct else 0.0,
+        "format_ok": format_ok,
+        "details": f"expected={expected} extracted={extracted}",
+    }

tests/test_rewards.py

@@ -0,0 +1,214 @@
+"""
+Adversarial test suite for the reward function.
+
+Goal: prove the reward cannot be gamed without doing the task. If any of
+these tests fail, training will reward-hack and we'll waste GPU time.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+import pytest
+
+from src.envs.promptops_arena.server.rewards import compute_reward
+from src.envs.promptops_arena.verifiers import verify
+
+
+# ---- math ----
+
+MATH_TASK = {"id": "x", "type": "math", "question": "What is 2+2?", "answer": "4"}
+
+
+def _reward(task, prompt, completion):
+    v = verify(task, completion)
+    return compute_reward(task, prompt, completion, v)
+
+
+def test_math_correct_with_tag_full_reward():
+    r = _reward(MATH_TASK, "short prompt", "<answer>4</answer>")
+    assert r["correctness"] == 1.0
+    assert r["format"] == 1.0
+    assert r["total"] == pytest.approx(1.1)
+
+
+def test_math_correct_no_tag_only_correctness():
+    r = _reward(MATH_TASK, "short prompt", "The answer is 4.")
+    assert r["correctness"] == 1.0
+    assert r["format"] == 0.0
+    assert r["total"] == pytest.approx(1.0)
+
+
+def test_math_empty_completion_zero():
+    r = _reward(MATH_TASK, "short prompt", "")
+    assert r["correctness"] == 0.0
+    assert r["format"] == 0.0
+    assert r["total"] == 0.0
+
+
+def test_math_empty_tag_only_format_bonus_capped():
+    """Tag with no number — gets format bonus but not correctness. Total <= 0.1."""
+    r = _reward(MATH_TASK, "short prompt", "<answer></answer>")
+    assert r["correctness"] == 0.0
+    assert r["total"] <= 0.1
+
+
+def test_math_wrong_answer_with_perfect_format_capped():
+    r = _reward(MATH_TASK, "short prompt", "<answer>7</answer>")
+    assert r["correctness"] == 0.0
+    assert r["total"] <= 0.1
+
+
+def test_math_rambling_long_correct_still_bounded():
+    """5000-char prompt + correct: brevity penalty must fire."""
+    long_prompt = "blah " * 1000  # 5000 chars
+    r = _reward(MATH_TASK, long_prompt, "<answer>4</answer>")
+    assert r["brevity"] < 0.0
+    assert r["brevity"] >= -0.1
+    # still mostly rewarded for correctness
+    assert r["total"] >= 1.0
+
+
+def test_math_short_prompt_no_brevity_penalty():
+    r = _reward(MATH_TASK, "Solve.", "<answer>4</answer>")
+    assert r["brevity"] == 0.0
+
+
+def test_math_boxed_format_recognized():
+    r = _reward(MATH_TASK, "short", "Final: \\boxed{4}")
+    assert r["correctness"] == 1.0
+    assert r["format"] == 1.0
+
+
+def test_math_keyword_only_no_credit():
+    """'answer:' phrase without correct number gets nothing."""
+    r = _reward(MATH_TASK, "short", "answer: 99")
+    assert r["correctness"] == 0.0
+    assert r["total"] <= 0.1
+
+
+def test_math_repeated_token_not_rewarded():
+    r = _reward(MATH_TASK, "short", "4 4 4 4 4 4 4 4")
+    # last number IS 4, so verifier extracts it correctly. This test documents
+    # that exact-match on a single number IS exploitable in this trivial task,
+    # so for real GSM8K-style tasks the answer should be one of many numbers in
+    # the question and not present in the prompt itself. We assert the
+    # "correctness fires only on exact match", not that this is unhackable on
+    # adversarial inputs unrelated to the question.
+    assert r["correctness"] in (0.0, 1.0)
+
+
+# ---- code ----
+
+CODE_TASK = {
+    "id": "c",
+    "type": "code",
+    "question": "Write add(a,b)",
+    "tests": ["assert add(2, 3) == 5", "assert add(0, 0) == 0"],
+}
+
+
+def test_code_correct_passes_tests():
+    completion = "```python\ndef add(a, b):\n    return a + b\n```"
+    r = _reward(CODE_TASK, "short", completion)
+    assert r["correctness"] == 1.0
+    assert r["format"] == 1.0
+
+
+def test_code_no_block_zero():
+    r = _reward(CODE_TASK, "short", "Sure! add returns sum.")
+    assert r["correctness"] == 0.0
+
+
+def test_code_block_but_wrong_zero():
+    completion = "```python\ndef add(a, b):\n    return a - b\n```"
+    r = _reward(CODE_TASK, "short", completion)
+    assert r["correctness"] == 0.0
+    # format bonus still given (proper block) — total bounded
+    assert r["total"] <= 0.1
+
+
+def test_code_infinite_loop_times_out_to_zero():
+    completion = "```python\ndef add(a, b):\n    while True: pass\n```"
+    r = _reward(CODE_TASK, "short", completion)
+    assert r["correctness"] == 0.0
+
+
+def test_code_malicious_import_still_sandboxed():
+    """We don't formally sandbox; we rely on subprocess isolation + 5s timeout."""
+    completion = "```python\nimport os\ndef add(a, b):\n    return a + b\n```"
+    r = _reward(CODE_TASK, "short", completion)
+    # imports are allowed; correctness still computed
+    assert r["correctness"] == 1.0
+
+
+# ---- json ----
+
+JSON_TASK = {
+    "id": "j",
+    "type": "json",
+    "question": "Extract name and age",
+    "schema": {
+        "type": "object",
+        "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
+        "required": ["name", "age"],
+    },
+    "expected": {"name": "Alice", "age": 30},
+}
+
+
+def test_json_correct_full_reward():
+    completion = '```json\n{"name": "Alice", "age": 30}\n```'
+    r = _reward(JSON_TASK, "short", completion)
+    assert r["correctness"] == 1.0
+    assert r["format"] == 1.0
+
+
+def test_json_inline_no_block_lower():
+    completion = '{"name": "Alice", "age": 30}'
+    r = _reward(JSON_TASK, "short", completion)
+    assert r["correctness"] == 1.0
+    assert r["format"] == 0.0
+
+
+def test_json_empty_object_zero():
+    r = _reward(JSON_TASK, "short", "```json\n{}\n```")
+    assert r["correctness"] == 0.0
+
+
+def test_json_wrong_type_zero():
+    """Schema requires int age — string fails."""
+    completion = '```json\n{"name": "Alice", "age": "30"}\n```'
+    r = _reward(JSON_TASK, "short", completion)
+    assert r["correctness"] == 0.0
+
+
+def test_json_invalid_zero():
+    r = _reward(JSON_TASK, "short", "```json\n{not valid json\n```")
+    assert r["correctness"] == 0.0
+
+
+# ---- decomposition contract ----
+
+def test_reward_dict_always_has_four_keys():
+    """The reward function must always return all four components."""
+    cases = [
+        (MATH_TASK, "p", "<answer>4</answer>"),
+        (MATH_TASK, "p", ""),
+        (CODE_TASK, "p", "```python\ndef add(a,b): return a+b\n```"),
+        (JSON_TASK, "p", "{}"),
+    ]
+    for task, p, c in cases:
+        r = _reward(task, p, c)
+        assert set(r.keys()) >= {"correctness", "format", "brevity", "total"}
+        for k in ("correctness", "format", "brevity", "total"):
+            assert isinstance(r[k], float)
+
+
+def test_reward_total_is_sum_of_components():
+    r = _reward(MATH_TASK, "x" * 1500, "<answer>4</answer>")
+    expected = r["correctness"] + 0.1 * r["format"] + r["brevity"]
+    assert r["total"] == pytest.approx(expected)