Upload folder using huggingface_hub

README.md

@@ -1,6 +1,6 @@
 ---
 title: Exec Assistant Arena Environment Server
-emoji: 🎣
+emoji: 📋
 colorFrom: gray
 colorTo: green
 sdk: docker
@@ -11,245 +11,102 @@ tags:
   - openenv
 ---
 
-# Exec Assistant Arena Environment
+# Executive Assistant Arena
 
-A simple test environment that echoes back messages. Perfect for testing the env APIs as well as demonstrating environment usage patterns.
+An [OpenEnv](https://github.com/meta-pytorch/OpenEnv) environment that simulates a personal assistant's morning inbox. The LLM agent must resolve calendar conflicts, draft email replies, infer hidden user preferences, and handle late-breaking schedule changes.
 
-## Quick Start
-
-The simplest way to use the Exec Assistant Arena environment is through the `ExecAssistantArenaEnv` class:
-
-```python
-from exec_assistant_arena import ExecAssistantArenaAction, ExecAssistantArenaEnv
-
-try:
-    # Create environment from Docker image
-    exec_assistant_arenaenv = ExecAssistantArenaEnv.from_docker_image("exec_assistant_arena-env:latest")
-
-    # Reset
-    result = exec_assistant_arenaenv.reset()
-    print(f"Reset: {result.observation.echoed_message}")
-
-    # Send multiple messages
-    messages = ["Hello, World!", "Testing echo", "Final message"]
-
-    for msg in messages:
-        result = exec_assistant_arenaenv.step(ExecAssistantArenaAction(message=msg))
-        print(f"Sent: '{msg}'")
-        print(f"  → Echoed: '{result.observation.echoed_message}'")
-        print(f"  → Length: {result.observation.message_length}")
-        print(f"  → Reward: {result.reward}")
-
-finally:
-    # Always clean up
-    exec_assistant_arenaenv.close()
-```
-
-That's it! The `ExecAssistantArenaEnv.from_docker_image()` method handles:
-- Starting the Docker container
-- Waiting for the server to be ready
-- Connecting to the environment
-- Container cleanup when you call `close()`
-
-## Building the Docker Image
-
-Before using the environment, you need to build the Docker image:
-
-```bash
-# From project root
-docker build -t exec_assistant_arena-env:latest -f server/Dockerfile .
-```
-
-## Deploying to Hugging Face Spaces
+Trained a Qwen2.5-7B model via GRPO on this environment, showing measurable improvement across 6 decomposed reward components.
 
-You can easily deploy your OpenEnv environment to Hugging Face Spaces using the `openenv push` command:
+## The Problem
 
-```bash
-# From the environment directory (where openenv.yaml is located)
-openenv push
-
-# Or specify options
-openenv push --namespace my-org --private
-```
-
-The `openenv push` command will:
-1. Validate that the directory is an OpenEnv environment (checks for `openenv.yaml`)
-2. Prepare a custom build for Hugging Face Docker space (enables web interface)
-3. Upload to Hugging Face (ensuring you're logged in)
-
-### Prerequisites
-
-- Authenticate with Hugging Face: The command will prompt for login if not already authenticated
-
-### Options
-
-- `--directory`, `-d`: Directory containing the OpenEnv environment (defaults to current directory)
-- `--repo-id`, `-r`: Repository ID in format 'username/repo-name' (defaults to 'username/env-name' from openenv.yaml)
-- `--base-image`, `-b`: Base Docker image to use (overrides Dockerfile FROM)
-- `--private`: Deploy the space as private (default: public)
-
-### Examples
-
-```bash
-# Push to your personal namespace (defaults to username/env-name from openenv.yaml)
-openenv push
-
-# Push to a specific repository
-openenv push --repo-id my-org/my-env
-
-# Push with a custom base image
-openenv push --base-image ghcr.io/meta-pytorch/openenv-base:latest
-
-# Push as a private space
-openenv push --private
-
-# Combine options
-openenv push --repo-id my-org/my-env --base-image custom-base:latest --private
-```
+Your AI assistant double-books you, ignores your "no mornings" preference, and can't handle it when your boss reschedules a meeting at the last minute. This environment trains LLMs to actually handle real-world scheduling chaos.
 
-After deployment, your space will be available at:
-`https://huggingface.co/spaces/<repo-id>`
+## Architecture
 
-The deployed space includes:
-- **Web Interface** at `/web` - Interactive UI for exploring the environment
-- **API Documentation** at `/docs` - Full OpenAPI/Swagger interface
-- **Health Check** at `/health` - Container health monitoring
-- **WebSocket** at `/ws` - Persistent session endpoint for low-latency interactions
+- **Environment**: Procedurally generated scenarios with calendar conflicts, emails, user preferences, and late-breaking changes
+- **3 difficulty tiers**: Easy (2 conflicts), Medium (4 conflicts + late changes), Hard (6 conflicts + 2 late changes)
+- **6 reward components**: conflict resolution, preference inference, email quality, deadline adherence, efficiency, late-change recovery
+- **All rewards are rule-based** — no LLM judges, fully deterministic and verifiable
 
-## Environment Details
-
-### Action
-**ExecAssistantArenaAction**: Contains a single field
-- `message` (str) - The message to echo back
-
-### Observation
-**ExecAssistantArenaObservation**: Contains the echo response and metadata
-- `echoed_message` (str) - The message echoed back
-- `message_length` (int) - Length of the message
-- `reward` (float) - Reward based on message length (length × 0.1)
-- `done` (bool) - Always False for echo environment
-- `metadata` (dict) - Additional info like step count
-
-### Reward
-The reward is calculated as: `message_length × 0.1`
-- "Hi" → reward: 0.2
-- "Hello, World!" → reward: 1.3
-- Empty message → reward: 0.0
-
-## Advanced Usage
-
-### Connecting to an Existing Server
-
-If you already have a Exec Assistant Arena environment server running, you can connect directly:
-
-```python
-from exec_assistant_arena import ExecAssistantArenaEnv
-
-# Connect to existing server
-exec_assistant_arenaenv = ExecAssistantArenaEnv(base_url="<ENV_HTTP_URL_HERE>")
-
-# Use as normal
-result = exec_assistant_arenaenv.reset()
-result = exec_assistant_arenaenv.step(ExecAssistantArenaAction(message="Hello!"))
-```
-
-Note: When connecting to an existing server, `exec_assistant_arenaenv.close()` will NOT stop the server.
-
-### Using the Context Manager
-
-The client supports context manager usage for automatic connection management:
-
-```python
-from exec_assistant_arena import ExecAssistantArenaAction, ExecAssistantArenaEnv
-
-# Connect with context manager (auto-connects and closes)
-with ExecAssistantArenaEnv(base_url="http://localhost:8000") as env:
-    result = env.reset()
-    print(f"Reset: {result.observation.echoed_message}")
-    # Multiple steps with low latency
-    for msg in ["Hello", "World", "!"]:
-        result = env.step(ExecAssistantArenaAction(message=msg))
-        print(f"Echoed: {result.observation.echoed_message}")
-```
-
-The client uses WebSocket connections for:
-- **Lower latency**: No HTTP connection overhead per request
-- **Persistent session**: Server maintains your environment state
-- **Efficient for episodes**: Better for many sequential steps
-
-### Concurrent WebSocket Sessions
-
-The server supports multiple concurrent WebSocket connections. To enable this,
-modify `server/app.py` to use factory mode:
+## Quick Start
 
 ```python
-# In server/app.py - use factory mode for concurrent sessions
-app = create_app(
-    ExecAssistantArenaEnvironment,  # Pass class, not instance
-    ExecAssistantArenaAction,
-    ExecAssistantArenaObservation,
-    max_concurrent_envs=4,  # Allow 4 concurrent sessions
-)
+from exec_assistant_arena import ExecAssistantArenaEnv, AssistantAction
+
+with ExecAssistantArenaEnv(base_url="https://SidraMiconi-exec-assistant-arena.hf.space") as env:
+    result = env.reset(seed=42, difficulty="medium")
+    print(result.observation.tool_result)  # scenario description
+    print(result.observation.conflicts)     # scheduling conflicts
+
+    # Resolve a conflict
+    result = env.step(AssistantAction(
+        tool="reschedule",
+        arguments={"event_id": "mtg_2", "new_time": "2:00pm"}
+    ))
+    print(f"Reward: {result.reward}")  # +1.0 for resolved conflict
+
+    # Draft an email reply
+    result = env.step(AssistantAction(
+        tool="draft_reply",
+        arguments={
+            "email_id": "email_1",
+            "body": "Hey! Sure thing, I'll get the budget review to you by tomorrow."
+        }
+    ))
+
+    # Finish
+    result = env.step(AssistantAction(tool="done"))
 ```
 
-Then multiple clients can connect simultaneously:
-
-```python
-from exec_assistant_arena import ExecAssistantArenaAction, ExecAssistantArenaEnv
-from concurrent.futures import ThreadPoolExecutor
-
-def run_episode(client_id: int):
-    with ExecAssistantArenaEnv(base_url="http://localhost:8000") as env:
-        result = env.reset()
-        for i in range(10):
-            result = env.step(ExecAssistantArenaAction(message=f"Client {client_id}, step {i}"))
-        return client_id, result.observation.message_length
-
-# Run 4 episodes concurrently
-with ThreadPoolExecutor(max_workers=4) as executor:
-    results = list(executor.map(run_episode, range(4)))
-```
+## Available Tools
 
-## Development & Testing
+| Tool | Arguments | Reward |
+|------|-----------|--------|
+| `check_calendar` | none | 0 (free) |
+| `check_inbox` | none | 0 (free) |
+| `reschedule` | `event_id`, `new_time` | +1.0 resolve, -0.5 new conflict |
+| `draft_reply` | `email_id`, `body` | 0.0 to +1.0 (quality scored) |
+| `delegate_task` | `task`, `to` | +0.5 if handles late change |
+| `done` | none | terminal rewards |
 
-### Direct Environment Testing
+## Training
 
-Test the environment logic directly without starting the HTTP server:
+Trained with GRPO (Group Relative Policy Optimization) using Unsloth + TRL:
 
 ```bash
-# From the server directory
-python3 server/exec_assistant_arena_environment.py
+# On H100
+cd exec_assistant_arena
+PYTHONPATH=. uvicorn server.app:app --host 0.0.0.0 --port 8000 &
+python training/train_grpo.py
 ```
 
-This verifies that:
-- Environment resets correctly
-- Step executes actions properly
-- State tracking works
-- Rewards are calculated correctly
-
-### Running Locally
-
-Run the server locally for development:
-
-```bash
-uvicorn server.app:app --reload
-```
+Colab notebook available at `training/train_colab.ipynb` for reproducing on free T4 GPU.
 
 ## Project Structure
 
 ```
 exec_assistant_arena/
-├── .dockerignore         # Docker build exclusions
-├── __init__.py            # Module exports
-├── README.md              # This file
-├── openenv.yaml           # OpenEnv manifest
-├── pyproject.toml         # Project metadata and dependencies
-├── uv.lock                # Locked dependencies (generated)
-├── client.py              # ExecAssistantArenaEnv client
-├── models.py              # Action and Observation models
-└── server/
-    ├── __init__.py        # Server module exports
-    ├── exec_assistant_arena_environment.py  # Core environment logic
-    ├── app.py             # FastAPI application (HTTP + WebSocket endpoints)
-    └── Dockerfile         # Container image definition
+├── models.py                    # Action, Observation, State
+├── client.py                    # WebSocket client
+├── server/
+│   ├── app.py                   # FastAPI server
+│   ├── exec_assistant_arena_environment.py  # Core env logic
+│   ├── scenario_generator.py    # Procedural generation
+│   └── reward.py                # 6 decomposed reward components
+└── training/
+    ├── train_grpo.py            # H100 training script
+    ├── train_colab.ipynb        # Colab version
+    ├── eval.py                  # Before/after evaluation
+    └── scenarios/
+        ├── train_scenarios.json # 80 training scenarios
+        └── eval_scenarios.json  # 20 held-out scenarios
 ```
+
+## Links
+
+- **HF Space**: https://huggingface.co/spaces/SidraMiconi/exec-assistant-arena
+- **GitHub**: https://github.com/Sidra/chief-of-staff/tree/main/exec_assistant_arena
+- **W&B**: https://wandb.ai/code-happy-sf/exec-assistant-arena
+- **Trained Model**: https://huggingface.co/SidraMiconi/exec-assistant-arena-lora
+
+Built for the OpenEnv Hackathon SF, March 7-8, 2026.

training/eval.py

@@ -0,0 +1,154 @@
+"""Evaluation script: compare base model vs trained model on held-out scenarios.
+
+Usage:
+    python training/eval.py --base-model Qwen/Qwen2.5-7B --trained-model SidraMiconi/exec-assistant-arena-lora
+"""
+
+import json
+import os
+import sys
+import argparse
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from exec_assistant_arena import ExecAssistantArenaEnv
+from exec_assistant_arena.models import AssistantAction
+from training.train_grpo import parse_tool_calls
+
+ENV_URL = "http://localhost:8000"
+
+
+def evaluate_model(model, tokenizer, scenarios, env_url, label="model"):
+    """Run model through eval scenarios and collect metrics."""
+    from unsloth import FastLanguageModel
+    FastLanguageModel.for_inference(model)
+
+    results = []
+
+    for i, scenario in enumerate(scenarios):
+        prompt = scenario["prompt"]
+        inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2048).to("cuda")
+
+        outputs = model.generate(
+            **inputs,
+            max_new_tokens=1024,
+            temperature=0.7,
+            do_sample=True,
+        )
+        completion = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
+
+        # Score through environment
+        try:
+            with ExecAssistantArenaEnv(base_url=env_url) as env:
+                seed = scenario.get("seed", i + 80)
+                difficulty = scenario.get("difficulty", "medium")
+                env.reset(seed=seed, difficulty=difficulty)
+
+                actions = parse_tool_calls(completion)
+                total_reward = 0.0
+                for action in actions:
+                    result = env.step(action)
+                    total_reward += (result.reward or 0.0)
+                    if result.done:
+                        break
+
+                if not result.done:
+                    result = env.step(AssistantAction(tool="done"))
+                    total_reward += (result.reward or 0.0)
+
+                state = env.state()
+                results.append({
+                    "scenario_idx": i,
+                    "seed": seed,
+                    "difficulty": difficulty,
+                    "total_reward": total_reward,
+                    "conflicts_resolved": state.conflicts_resolved,
+                    "total_conflicts": state.total_conflicts,
+                    "conflict_rate": state.conflicts_resolved / max(1, state.total_conflicts),
+                    "emails_drafted": state.emails_drafted,
+                    "total_emails": state.total_emails,
+                    "preferences_inferred": state.preferences_inferred,
+                    "deadlines_met": state.deadlines_met,
+                    "unnecessary_actions": state.unnecessary_actions,
+                    "n_actions": len(actions),
+                    "completion": completion[:500],
+                })
+        except Exception as e:
+            print(f"  Error on scenario {i}: {e}")
+            results.append({"scenario_idx": i, "total_reward": -1.0, "error": str(e)})
+
+        print(f"  [{label}] Scenario {i}: reward={results[-1].get('total_reward', 'err'):.2f}")
+
+    return results
+
+
+def print_comparison(base_results, trained_results):
+    """Print side-by-side comparison."""
+    print("\n" + "=" * 70)
+    print("EVALUATION RESULTS")
+    print("=" * 70)
+
+    metrics = ["total_reward", "conflict_rate", "emails_drafted", "preferences_inferred", "unnecessary_actions"]
+
+    for metric in metrics:
+        base_vals = [r.get(metric, 0) for r in base_results if "error" not in r]
+        trained_vals = [r.get(metric, 0) for r in trained_results if "error" not in r]
+
+        if base_vals and trained_vals:
+            base_avg = sum(base_vals) / len(base_vals)
+            trained_avg = sum(trained_vals) / len(trained_vals)
+            delta = trained_avg - base_avg
+            print(f"  {metric:25s}  base={base_avg:7.2f}  trained={trained_avg:7.2f}  delta={delta:+.2f}")
+
+    print("=" * 70)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base-model", default="Qwen/Qwen2.5-7B")
+    parser.add_argument("--trained-model", default="SidraMiconi/exec-assistant-arena-lora")
+    parser.add_argument("--env-url", default=ENV_URL)
+    parser.add_argument("--output", default="training/eval_results.json")
+    args = parser.parse_args()
+
+    script_dir = os.path.dirname(os.path.abspath(__file__))
+    with open(os.path.join(script_dir, "scenarios/eval_scenarios.json")) as f:
+        scenarios = json.load(f)
+
+    print(f"Evaluating on {len(scenarios)} held-out scenarios\n")
+
+    from unsloth import FastLanguageModel
+
+    # Load base model
+    print("Loading base model...")
+    base_model, base_tokenizer = FastLanguageModel.from_pretrained(
+        model_name=args.base_model, max_seq_length=2048, load_in_4bit=True,
+    )
+    print("Evaluating base model...")
+    base_results = evaluate_model(base_model, base_tokenizer, scenarios, args.env_url, "base")
+    del base_model
+
+    # Load trained model
+    print("\nLoading trained model...")
+    trained_model, trained_tokenizer = FastLanguageModel.from_pretrained(
+        model_name=args.trained_model, max_seq_length=2048, load_in_4bit=True,
+    )
+    print("Evaluating trained model...")
+    trained_results = evaluate_model(trained_model, trained_tokenizer, scenarios, args.env_url, "trained")
+
+    print_comparison(base_results, trained_results)
+
+    # Save results
+    output = {
+        "base_model": args.base_model,
+        "trained_model": args.trained_model,
+        "base_results": base_results,
+        "trained_results": trained_results,
+    }
+    with open(args.output, "w") as f:
+        json.dump(output, f, indent=2)
+    print(f"\nResults saved to {args.output}")
+
+
+if __name__ == "__main__":
+    main()

training/scenarios/eval_scenarios.json

@@ -0,0 +1,142 @@
+[
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 9:30am (60min) - board prep with Eve Johnson [priority: high]\n[mtg_2] 9:30am (60min) - interview with Grace Wang [priority: high]\n[mtg_3] 11:00am (30min) - all-hands with Irene Davis [priority: medium]\n[mtg_4] 11:30am (30min) - 1:1 with Carol Park [priority: high]\n\nINBOX:\n[email_1] From: Henry Brown | Subject: Performance Review Follow-up [DEADLINE: today]\n  Hi, I wanted to follow up on performance review follow-up. Could you get back to me by today? Thanks, Henry Brown\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\n\nUSER PREFERENCES:\n  - User avoids meetings on Fridays\n  - User always blocks 12pm-1pm for lunch\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Reply to email email_1 from Henry Brown\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 80,
+    "difficulty": "easy",
+    "n_conflicts": 1,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 9:00am (30min) - interview with Irene Davis [priority: medium]\n[mtg_2] 9:00am (60min) - lunch meeting with Henry Brown [priority: medium]\n[mtg_3] 11:00am (30min) - 1:1 with Frank Lee [priority: medium]\n[mtg_4] 1:30pm (30min) - team standup with Eve Johnson [priority: high]\n\nINBOX:\n[email_1] From: Grace Wang | Subject: Partnership Proposal\n  Hi, I wanted to follow up on partnership proposal. Could you get back to me? Thanks, Grace Wang\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\n\nUSER PREFERENCES:\n  - User avoids meetings on Fridays\n  - User prefers informal/casual tone in emails\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Reply to email email_1 from Grace Wang\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 81,
+    "difficulty": "easy",
+    "n_conflicts": 1,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 4:30pm (30min) - client call with Carol Park [priority: medium]\n[mtg_2] 4:30pm (30min) - sprint planning with Henry Brown [priority: high]\n[mtg_3] 10:00am (30min) - strategy session with Eve Johnson [priority: medium]\n[mtg_4] 4:30pm (60min) - interview with Grace Wang [priority: medium]\n\nINBOX:\n[email_1] From: Bob Martinez | Subject: Team Offsite Planning [DEADLINE: today]\n  Hi, I wanted to follow up on team offsite planning. Could you get back to me by today? Thanks, Bob Martinez\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - User prefers formal/professional tone in emails\n  - User avoids meetings on Fridays\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_4\n  - Reply to email email_1 from Bob Martinez\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 82,
+    "difficulty": "easy",
+    "n_conflicts": 3,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 10:00am (30min) - 1:1 with Henry Brown [priority: medium]\n[mtg_2] 10:00am (30min) - strategy session with Jack Wilson [priority: medium]\n[mtg_3] 12:00pm (60min) - design review with Bob Martinez [priority: medium]\n[mtg_4] 12:30pm (60min) - team standup with Irene Davis [priority: low]\n\nINBOX:\n[email_1] From: David Kim | Subject: Hiring Update [DEADLINE: today]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by today? Thanks, David Kim\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_3 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - User prefers no meetings before 10am\n  - Client meetings cannot be rescheduled\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_3 and mtg_4\n  - Reply to email email_1 from David Kim\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 83,
+    "difficulty": "easy",
+    "n_conflicts": 2,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 11:00am (60min) - all-hands with Eve Johnson [priority: low]\n[mtg_2] 11:00am (60min) - design review with Alice Chen [priority: low]\n[mtg_3] 2:00pm (60min) - lunch meeting with Henry Brown [priority: low]\n[mtg_4] 4:30pm (30min) - board prep with Irene Davis [priority: high]\n\nINBOX:\n[email_1] From: Jack Wilson | Subject: Q3 Budget Review [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me by tomorrow? Thanks, Jack Wilson\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\n\nUSER PREFERENCES:\n  - User needs 15-min buffer between meetings\n  - User prefers 30-min meetings over 60-min\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Reply to email email_1 from Jack Wilson\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 84,
+    "difficulty": "easy",
+    "n_conflicts": 1,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 12:30pm (60min) - lunch meeting with David Kim [priority: low]\n[mtg_2] 12:30pm (60min) - all-hands with Bob Martinez [priority: low]\n[mtg_3] 10:00am (30min) - design review with Frank Lee [priority: high]\n[mtg_4] 4:00pm (60min) - board prep with Irene Davis [priority: medium]\n\nINBOX:\n[email_1] From: Grace Wang | Subject: Hiring Update [DEADLINE: today]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by today? Thanks, Grace Wang\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\n\nUSER PREFERENCES:\n  - User prefers informal/casual tone in emails\n  - Meetings with the boss always take priority\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Reply to email email_1 from Grace Wang\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 85,
+    "difficulty": "easy",
+    "n_conflicts": 1,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 5:00pm (60min) - team standup with Alice Chen [priority: high]\n[mtg_2] 5:00pm (30min) - board prep with Irene Davis [priority: medium]\n[mtg_3] 12:00pm (30min) - sprint planning with Frank Lee [priority: low]\n[mtg_4] 9:30am (30min) - interview with Henry Brown [priority: medium]\n\nINBOX:\n[email_1] From: Jack Wilson | Subject: Client Demo Prep [DEADLINE: today]\n  Hi, I wanted to follow up on client demo prep. Could you get back to me by today? Thanks, Jack Wilson\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\n\nUSER PREFERENCES:\n  - Client meetings cannot be rescheduled\n  - Meetings with the boss always take priority\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Reply to email email_1 from Jack Wilson\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 86,
+    "difficulty": "easy",
+    "n_conflicts": 1,
+    "n_emails": 1
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 12:00pm (30min) - all-hands with Carol Park [priority: low]\n[mtg_2] 12:00pm (60min) - 1:1 with David Kim [priority: high]\n[mtg_3] 12:00pm (60min) - sprint planning with Bob Martinez [priority: high]\n[mtg_4] 12:00pm (60min) - design review with Jack Wilson [priority: medium]\n[mtg_5] 12:00pm (60min) - board prep with Grace Wang [priority: high]\n[mtg_6] 1:30pm (60min) - interview with Eve Johnson [priority: low]\n\nINBOX:\n[email_1] From: Frank Lee | Subject: Q3 Budget Review [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me by tomorrow? Thanks, Frank Lee\n\n[email_2] From: Irene Davis | Subject: Q3 Budget Review [DEADLINE: today]\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me by today? Thanks, Irene Davis\n\n[email_3] From: Alice Chen | Subject: Q3 Budget Review [DEADLINE: today]\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me by today? Thanks, Alice Chen\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_5\n\nUSER PREFERENCES:\n  - User prefers 30-min meetings over 60-min\n  - Client meetings cannot be rescheduled\n  - User prefers formal/professional tone in emails\n  - User avoids meetings on Fridays\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_5\n  - Reply to email email_1 from Frank Lee\n  - Reply to email email_2 from Irene Davis\n  - Reply to email email_3 from Alice Chen\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 87,
+    "difficulty": "medium",
+    "n_conflicts": 10,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 3:30pm (30min) - client call with Grace Wang [priority: medium]\n[mtg_2] 3:30pm (60min) - design review with David Kim [priority: low]\n[mtg_3] 3:30pm (30min) - team standup with Frank Lee [priority: medium]\n[mtg_4] 3:30pm (30min) - all-hands with Bob Martinez [priority: medium]\n[mtg_5] 12:30pm (30min) - lunch meeting with Henry Brown [priority: high]\n[mtg_6] 9:00am (60min) - sprint planning with Alice Chen [priority: high]\n\nINBOX:\n[email_1] From: Eve Johnson | Subject: Performance Review Follow-up [DEADLINE: today]\n  Hi, I wanted to follow up on performance review follow-up. Could you get back to me by today? Thanks, Eve Johnson\n\n[email_2] From: Jack Wilson | Subject: Hiring Update [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by tomorrow? Thanks, Jack Wilson\n\n[email_3] From: Carol Park | Subject: Customer Escalation [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on customer escalation. Could you get back to me by tomorrow? Thanks, Carol Park\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - User prefers no meetings before 10am\n  - User prefers informal/casual tone in emails\n  - User needs 15-min buffer between meetings\n  - User prefers formal/professional tone in emails\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_4\n  - Reply to email email_1 from Eve Johnson\n  - Reply to email email_2 from Jack Wilson\n  - Reply to email email_3 from Carol Park\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 88,
+    "difficulty": "medium",
+    "n_conflicts": 6,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 4:00pm (30min) - sprint planning with Bob Martinez [priority: high]\n[mtg_2] 4:00pm (30min) - all-hands with Eve Johnson [priority: low]\n[mtg_3] 4:00pm (30min) - team standup with Carol Park [priority: high]\n[mtg_4] 4:00pm (60min) - strategy session with Henry Brown [priority: medium]\n[mtg_5] 11:00am (60min) - lunch meeting with Alice Chen [priority: high]\n[mtg_6] 10:00am (30min) - design review with Jack Wilson [priority: low]\n\nINBOX:\n[email_1] From: David Kim | Subject: Client Demo Prep\n  Hi, I wanted to follow up on client demo prep. Could you get back to me? Thanks, David Kim\n\n[email_2] From: Irene Davis | Subject: Vendor Contract Renewal [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on vendor contract renewal. Could you get back to me by tomorrow? Thanks, Irene Davis\n\n[email_3] From: Frank Lee | Subject: Customer Escalation [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on customer escalation. Could you get back to me by tomorrow? Thanks, Frank Lee\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - Client meetings cannot be rescheduled\n  - User avoids meetings on Fridays\n  - User prefers 30-min meetings over 60-min\n  - User needs 15-min buffer between meetings\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_4\n  - Reply to email email_1 from David Kim\n  - Reply to email email_2 from Irene Davis\n  - Reply to email email_3 from Frank Lee\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 89,
+    "difficulty": "medium",
+    "n_conflicts": 6,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 11:00am (30min) - lunch meeting with David Kim [priority: low]\n[mtg_2] 11:00am (30min) - sprint planning with Bob Martinez [priority: low]\n[mtg_3] 11:00am (60min) - board prep with Henry Brown [priority: medium]\n[mtg_4] 11:00am (30min) - design review with Frank Lee [priority: low]\n[mtg_5] 9:00am (30min) - client call with Eve Johnson [priority: high]\n[mtg_6] 1:30pm (30min) - all-hands with Grace Wang [priority: low]\n\nINBOX:\n[email_1] From: Carol Park | Subject: Performance Review Follow-up [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on performance review follow-up. Could you get back to me by tomorrow? Thanks, Carol Park\n\n[email_2] From: Jack Wilson | Subject: Team Offsite Planning [DEADLINE: today]\n  Hi, I wanted to follow up on team offsite planning. Could you get back to me by today? Thanks, Jack Wilson\n\n[email_3] From: Irene Davis | Subject: Client Demo Prep [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on client demo prep. Could you get back to me by tomorrow? Thanks, Irene Davis\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - User needs 15-min buffer between meetings\n  - User always blocks 12pm-1pm for lunch\n  - Client meetings cannot be rescheduled\n  - User prefers informal/casual tone in emails\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_4\n  - Reply to email email_1 from Carol Park\n  - Reply to email email_2 from Jack Wilson\n  - Reply to email email_3 from Irene Davis\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 90,
+    "difficulty": "medium",
+    "n_conflicts": 6,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 10:00am (30min) - design review with Bob Martinez [priority: medium]\n[mtg_2] 10:00am (30min) - strategy session with Carol Park [priority: low]\n[mtg_3] 10:00am (60min) - all-hands with Irene Davis [priority: medium]\n[mtg_4] 10:00am (60min) - interview with David Kim [priority: low]\n[mtg_5] 3:00pm (60min) - board prep with Grace Wang [priority: high]\n[mtg_6] 12:00pm (60min) - team standup with Frank Lee [priority: low]\n\nINBOX:\n[email_1] From: Eve Johnson | Subject: Product Launch Timeline [DEADLINE: today]\n  Hi, I wanted to follow up on product launch timeline. Could you get back to me by today? Thanks, Eve Johnson\n\n[email_2] From: Jack Wilson | Subject: Board Presentation Draft [DEADLINE: today]\n  Hi, I wanted to follow up on board presentation draft. Could you get back to me by today? Thanks, Jack Wilson\n\n[email_3] From: Henry Brown | Subject: Team Offsite Planning\n  Hi, I wanted to follow up on team offsite planning. Could you get back to me? Thanks, Henry Brown\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - User prefers 30-min meetings over 60-min\n  - User needs 15-min buffer between meetings\n  - User avoids meetings on Fridays\n  - Meetings with the boss always take priority\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_4\n  - Reply to email email_1 from Eve Johnson\n  - Reply to email email_2 from Jack Wilson\n  - Reply to email email_3 from Henry Brown\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 91,
+    "difficulty": "medium",
+    "n_conflicts": 6,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 11:30am (30min) - strategy session with Grace Wang [priority: low]\n[mtg_2] 11:30am (60min) - all-hands with Irene Davis [priority: medium]\n[mtg_3] 11:30am (30min) - 1:1 with Eve Johnson [priority: low]\n[mtg_4] 11:30am (60min) - lunch meeting with Jack Wilson [priority: high]\n[mtg_5] 11:30am (30min) - interview with Frank Lee [priority: medium]\n[mtg_6] 12:30pm (30min) - team standup with Alice Chen [priority: high]\n\nINBOX:\n[email_1] From: David Kim | Subject: Partnership Proposal [DEADLINE: today]\n  Hi, I wanted to follow up on partnership proposal. Could you get back to me by today? Thanks, David Kim\n\n[email_2] From: Henry Brown | Subject: Board Presentation Draft\n  Hi, I wanted to follow up on board presentation draft. Could you get back to me? Thanks, Henry Brown\n\n[email_3] From: Carol Park | Subject: Team Offsite Planning [DEADLINE: today]\n  Hi, I wanted to follow up on team offsite planning. Could you get back to me by today? Thanks, Carol Park\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_5\n\nUSER PREFERENCES:\n  - User dislikes back-to-back meetings\n  - User prefers 30-min meetings over 60-min\n  - User avoids meetings on Fridays\n  - Client meetings cannot be rescheduled\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_5\n  - Reply to email email_1 from David Kim\n  - Reply to email email_2 from Henry Brown\n  - Reply to email email_3 from Carol Park\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 92,
+    "difficulty": "medium",
+    "n_conflicts": 10,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 12:00pm (60min) - client call with Henry Brown [priority: high]\n[mtg_2] 12:00pm (30min) - 1:1 with Frank Lee [priority: low]\n[mtg_3] 12:00pm (30min) - team standup with Bob Martinez [priority: low]\n[mtg_4] 12:00pm (60min) - all-hands with Alice Chen [priority: medium]\n[mtg_5] 11:30am (60min) - sprint planning with Carol Park [priority: high]\n[mtg_6] 1:30pm (30min) - strategy session with Jack Wilson [priority: high]\n\nINBOX:\n[email_1] From: Eve Johnson | Subject: Q3 Budget Review [DEADLINE: today]\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me by today? Thanks, Eve Johnson\n\n[email_2] From: David Kim | Subject: Partnership Proposal [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on partnership proposal. Could you get back to me by tomorrow? Thanks, David Kim\n\n[email_3] From: Grace Wang | Subject: Team Offsite Planning [DEADLINE: today]\n  Hi, I wanted to follow up on team offsite planning. Could you get back to me by today? Thanks, Grace Wang\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_5\n\nUSER PREFERENCES:\n  - User needs 15-min buffer between meetings\n  - User prefers informal/casual tone in emails\n  - User always blocks 12pm-1pm for lunch\n  - User avoids meetings on Fridays\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_5\n  - Reply to email email_1 from Eve Johnson\n  - Reply to email email_2 from David Kim\n  - Reply to email email_3 from Grace Wang\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 93,
+    "difficulty": "medium",
+    "n_conflicts": 10,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 9:00am (30min) - all-hands with Irene Davis [priority: medium]\n[mtg_2] 9:00am (60min) - sprint planning with Carol Park [priority: medium]\n[mtg_3] 9:00am (60min) - interview with Bob Martinez [priority: high]\n[mtg_4] 9:00am (60min) - board prep with Grace Wang [priority: medium]\n[mtg_5] 11:00am (30min) - client call with Jack Wilson [priority: medium]\n[mtg_6] 1:30pm (30min) - lunch meeting with Frank Lee [priority: low]\n\nINBOX:\n[email_1] From: David Kim | Subject: Hiring Update [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by tomorrow? Thanks, David Kim\n\n[email_2] From: Alice Chen | Subject: Board Presentation Draft [DEADLINE: today]\n  Hi, I wanted to follow up on board presentation draft. Could you get back to me by today? Thanks, Alice Chen\n\n[email_3] From: Henry Brown | Subject: Product Launch Timeline [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on product launch timeline. Could you get back to me by tomorrow? Thanks, Henry Brown\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_4\n\nUSER PREFERENCES:\n  - User avoids meetings on Fridays\n  - User prefers formal/professional tone in emails\n  - User prefers 30-min meetings over 60-min\n  - User always blocks 12pm-1pm for lunch\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_4\n  - Reply to email email_1 from David Kim\n  - Reply to email email_2 from Alice Chen\n  - Reply to email email_3 from Henry Brown\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 94,
+    "difficulty": "medium",
+    "n_conflicts": 6,
+    "n_emails": 3
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 11:00am (60min) - design review with Irene Davis [priority: medium]\n[mtg_2] 11:00am (30min) - 1:1 with Jack Wilson [priority: high]\n[mtg_3] 11:00am (30min) - strategy session with Carol Park [priority: low]\n[mtg_4] 11:00am (60min) - all-hands with David Kim [priority: low]\n[mtg_5] 11:00am (30min) - interview with Alice Chen [priority: high]\n[mtg_6] 11:00am (60min) - board prep with Bob Martinez [priority: medium]\n[mtg_7] 5:00pm (30min) - client call with Frank Lee [priority: medium]\n[mtg_8] 2:30pm (60min) - team standup with Henry Brown [priority: medium]\n\nINBOX:\n[email_1] From: Grace Wang | Subject: Performance Review Follow-up [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on performance review follow-up. Could you get back to me by tomorrow? Thanks, Grace Wang\n\n[email_2] From: Eve Johnson | Subject: Team Offsite Planning [DEADLINE: today]\n  Hi, I wanted to follow up on team offsite planning. Could you get back to me by today? Thanks, Eve Johnson\n\n[email_3] From: Jack Wilson | Subject: Q3 Budget Review\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me? Thanks, Jack Wilson\n\n[email_4] From: Grace Wang | Subject: Performance Review Follow-up\n  Hi, I wanted to follow up on performance review follow-up. Could you get back to me? Thanks, Grace Wang\n\n[email_5] From: Jack Wilson | Subject: Client Demo Prep\n  Hi, I wanted to follow up on client demo prep. Could you get back to me? Thanks, Jack Wilson\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_1 overlaps with mtg_6\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_6\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_6\nCONFLICT: mtg_4 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_6\nCONFLICT: mtg_5 overlaps with mtg_6\n\nUSER PREFERENCES:\n  - Client meetings cannot be rescheduled\n  - User prefers informal/casual tone in emails\n  - User always blocks 12pm-1pm for lunch\n  - User prefers formal/professional tone in emails\n  - Meetings with the boss always take priority\n  - User dislikes back-to-back meetings\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_1 and mtg_6\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_6\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_6\n  - Resolve conflict between mtg_4 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_6\n  - Resolve conflict between mtg_5 and mtg_6\n  - Reply to email email_1 from Grace Wang\n  - Reply to email email_2 from Eve Johnson\n  - Reply to email email_3 from Jack Wilson\n  - Reply to email email_4 from Grace Wang\n  - Reply to email email_5 from Jack Wilson\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 95,
+    "difficulty": "hard",
+    "n_conflicts": 15,
+    "n_emails": 5
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 4:30pm (30min) - interview with Frank Lee [priority: low]\n[mtg_2] 4:30pm (60min) - all-hands with Jack Wilson [priority: medium]\n[mtg_3] 4:30pm (30min) - team standup with Grace Wang [priority: low]\n[mtg_4] 4:30pm (60min) - client call with Bob Martinez [priority: high]\n[mtg_5] 4:30pm (60min) - strategy session with Alice Chen [priority: low]\n[mtg_6] 4:30pm (60min) - board prep with Carol Park [priority: high]\n[mtg_7] 1:30pm (30min) - sprint planning with Irene Davis [priority: low]\n[mtg_8] 12:00pm (60min) - 1:1 with Eve Johnson [priority: medium]\n\nINBOX:\n[email_1] From: David Kim | Subject: Board Presentation Draft [DEADLINE: today]\n  Hi, I wanted to follow up on board presentation draft. Could you get back to me by today? Thanks, David Kim\n\n[email_2] From: Henry Brown | Subject: Product Launch Timeline\n  Hi, I wanted to follow up on product launch timeline. Could you get back to me? Thanks, Henry Brown\n\n[email_3] From: Grace Wang | Subject: Board Presentation Draft\n  Hi, I wanted to follow up on board presentation draft. Could you get back to me? Thanks, Grace Wang\n\n[email_4] From: David Kim | Subject: Hiring Update\n  Hi, I wanted to follow up on hiring update. Could you get back to me? Thanks, David Kim\n\n[email_5] From: Eve Johnson | Subject: Client Demo Prep [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on client demo prep. Could you get back to me by tomorrow? Thanks, Eve Johnson\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_1 overlaps with mtg_6\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_6\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_6\nCONFLICT: mtg_4 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_6\nCONFLICT: mtg_5 overlaps with mtg_6\n\nUSER PREFERENCES:\n  - Meetings with the boss always take priority\n  - User prefers no meetings before 10am\n  - Client meetings cannot be rescheduled\n  - User always blocks 12pm-1pm for lunch\n  - User dislikes back-to-back meetings\n  - User needs 15-min buffer between meetings\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_1 and mtg_6\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_6\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_6\n  - Resolve conflict between mtg_4 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_6\n  - Resolve conflict between mtg_5 and mtg_6\n  - Reply to email email_1 from David Kim\n  - Reply to email email_2 from Henry Brown\n  - Reply to email email_3 from Grace Wang\n  - Reply to email email_4 from David Kim\n  - Reply to email email_5 from Eve Johnson\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 96,
+    "difficulty": "hard",
+    "n_conflicts": 15,
+    "n_emails": 5
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 12:30pm (30min) - sprint planning with David Kim [priority: medium]\n[mtg_2] 12:30pm (60min) - all-hands with Grace Wang [priority: high]\n[mtg_3] 12:30pm (30min) - 1:1 with Frank Lee [priority: high]\n[mtg_4] 12:30pm (30min) - lunch meeting with Irene Davis [priority: high]\n[mtg_5] 12:30pm (30min) - board prep with Alice Chen [priority: high]\n[mtg_6] 12:30pm (30min) - design review with Eve Johnson [priority: low]\n[mtg_7] 12:30pm (60min) - strategy session with Henry Brown [priority: high]\n[mtg_8] 11:00am (30min) - client call with Carol Park [priority: high]\n\nINBOX:\n[email_1] From: Jack Wilson | Subject: Customer Escalation\n  Hi, I wanted to follow up on customer escalation. Could you get back to me? Thanks, Jack Wilson\n\n[email_2] From: Bob Martinez | Subject: Partnership Proposal\n  Hi, I wanted to follow up on partnership proposal. Could you get back to me? Thanks, Bob Martinez\n\n[email_3] From: Alice Chen | Subject: Client Demo Prep\n  Hi, I wanted to follow up on client demo prep. Could you get back to me? Thanks, Alice Chen\n\n[email_4] From: Irene Davis | Subject: Hiring Update [DEADLINE: today]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by today? Thanks, Irene Davis\n\n[email_5] From: Irene Davis | Subject: Hiring Update [DEADLINE: today]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by today? Thanks, Irene Davis\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_1 overlaps with mtg_6\nCONFLICT: mtg_1 overlaps with mtg_7\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_6\nCONFLICT: mtg_2 overlaps with mtg_7\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_6\nCONFLICT: mtg_3 overlaps with mtg_7\nCONFLICT: mtg_4 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_6\nCONFLICT: mtg_4 overlaps with mtg_7\nCONFLICT: mtg_5 overlaps with mtg_6\nCONFLICT: mtg_5 overlaps with mtg_7\nCONFLICT: mtg_6 overlaps with mtg_7\n\nUSER PREFERENCES:\n  - User needs 15-min buffer between meetings\n  - User prefers no meetings before 10am\n  - User dislikes back-to-back meetings\n  - User prefers 30-min meetings over 60-min\n  - User avoids meetings on Fridays\n  - Meetings with the boss always take priority\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_1 and mtg_6\n  - Resolve conflict between mtg_1 and mtg_7\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_6\n  - Resolve conflict between mtg_2 and mtg_7\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_6\n  - Resolve conflict between mtg_3 and mtg_7\n  - Resolve conflict between mtg_4 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_6\n  - Resolve conflict between mtg_4 and mtg_7\n  - Resolve conflict between mtg_5 and mtg_6\n  - Resolve conflict between mtg_5 and mtg_7\n  - Resolve conflict between mtg_6 and mtg_7\n  - Reply to email email_1 from Jack Wilson\n  - Reply to email email_2 from Bob Martinez\n  - Reply to email email_3 from Alice Chen\n  - Reply to email email_4 from Irene Davis\n  - Reply to email email_5 from Irene Davis\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 97,
+    "difficulty": "hard",
+    "n_conflicts": 21,
+    "n_emails": 5
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 2:00pm (60min) - design review with Frank Lee [priority: medium]\n[mtg_2] 2:00pm (30min) - interview with Alice Chen [priority: low]\n[mtg_3] 2:00pm (30min) - 1:1 with Jack Wilson [priority: low]\n[mtg_4] 2:00pm (60min) - all-hands with David Kim [priority: high]\n[mtg_5] 2:00pm (60min) - board prep with Irene Davis [priority: low]\n[mtg_6] 2:00pm (30min) - client call with Grace Wang [priority: low]\n[mtg_7] 10:30am (60min) - lunch meeting with Carol Park [priority: low]\n[mtg_8] 11:30am (30min) - strategy session with Eve Johnson [priority: medium]\n\nINBOX:\n[email_1] From: Henry Brown | Subject: Vendor Contract Renewal\n  Hi, I wanted to follow up on vendor contract renewal. Could you get back to me? Thanks, Henry Brown\n\n[email_2] From: Bob Martinez | Subject: Product Launch Timeline [DEADLINE: today]\n  Hi, I wanted to follow up on product launch timeline. Could you get back to me by today? Thanks, Bob Martinez\n\n[email_3] From: Alice Chen | Subject: Hiring Update [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on hiring update. Could you get back to me by tomorrow? Thanks, Alice Chen\n\n[email_4] From: Irene Davis | Subject: Q3 Budget Review [DEADLINE: today]\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me by today? Thanks, Irene Davis\n\n[email_5] From: Frank Lee | Subject: Product Launch Timeline [DEADLINE: today]\n  Hi, I wanted to follow up on product launch timeline. Could you get back to me by today? Thanks, Frank Lee\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_1 overlaps with mtg_6\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_6\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_6\nCONFLICT: mtg_4 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_6\nCONFLICT: mtg_5 overlaps with mtg_6\n\nUSER PREFERENCES:\n  - User always blocks 12pm-1pm for lunch\n  - User prefers no meetings before 10am\n  - Client meetings cannot be rescheduled\n  - User prefers formal/professional tone in emails\n  - User avoids meetings on Fridays\n  - Meetings with the boss always take priority\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_1 and mtg_6\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_6\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_6\n  - Resolve conflict between mtg_4 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_6\n  - Resolve conflict between mtg_5 and mtg_6\n  - Reply to email email_1 from Henry Brown\n  - Reply to email email_2 from Bob Martinez\n  - Reply to email email_3 from Alice Chen\n  - Reply to email email_4 from Irene Davis\n  - Reply to email email_5 from Frank Lee\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 98,
+    "difficulty": "hard",
+    "n_conflicts": 15,
+    "n_emails": 5
+  },
+  {
+    "prompt": "You are an executive assistant. Your job is to manage the user's calendar and inbox efficiently.\n\nTODAY'S CALENDAR:\n[mtg_1] 3:00pm (30min) - interview with Grace Wang [priority: medium]\n[mtg_2] 3:00pm (30min) - board prep with Jack Wilson [priority: high]\n[mtg_3] 3:00pm (60min) - team standup with David Kim [priority: high]\n[mtg_4] 3:00pm (60min) - sprint planning with Eve Johnson [priority: medium]\n[mtg_5] 3:00pm (30min) - design review with Bob Martinez [priority: medium]\n[mtg_6] 3:00pm (60min) - lunch meeting with Frank Lee [priority: high]\n[mtg_7] 5:00pm (30min) - all-hands with Irene Davis [priority: medium]\n[mtg_8] 4:00pm (30min) - client call with Alice Chen [priority: low]\n\nINBOX:\n[email_1] From: Carol Park | Subject: Q3 Budget Review\n  Hi, I wanted to follow up on q3 budget review. Could you get back to me? Thanks, Carol Park\n\n[email_2] From: Henry Brown | Subject: Client Demo Prep [DEADLINE: today]\n  Hi, I wanted to follow up on client demo prep. Could you get back to me by today? Thanks, Henry Brown\n\n[email_3] From: Irene Davis | Subject: Client Demo Prep [DEADLINE: today]\n  Hi, I wanted to follow up on client demo prep. Could you get back to me by today? Thanks, Irene Davis\n\n[email_4] From: Henry Brown | Subject: Board Presentation Draft\n  Hi, I wanted to follow up on board presentation draft. Could you get back to me? Thanks, Henry Brown\n\n[email_5] From: Bob Martinez | Subject: Client Demo Prep [DEADLINE: tomorrow]\n  Hi, I wanted to follow up on client demo prep. Could you get back to me by tomorrow? Thanks, Bob Martinez\n\nSCHEDULING CONFLICTS:\nCONFLICT: mtg_1 overlaps with mtg_2\nCONFLICT: mtg_1 overlaps with mtg_3\nCONFLICT: mtg_1 overlaps with mtg_4\nCONFLICT: mtg_1 overlaps with mtg_5\nCONFLICT: mtg_1 overlaps with mtg_6\nCONFLICT: mtg_2 overlaps with mtg_3\nCONFLICT: mtg_2 overlaps with mtg_4\nCONFLICT: mtg_2 overlaps with mtg_5\nCONFLICT: mtg_2 overlaps with mtg_6\nCONFLICT: mtg_3 overlaps with mtg_4\nCONFLICT: mtg_3 overlaps with mtg_5\nCONFLICT: mtg_3 overlaps with mtg_6\nCONFLICT: mtg_4 overlaps with mtg_5\nCONFLICT: mtg_4 overlaps with mtg_6\nCONFLICT: mtg_5 overlaps with mtg_6\n\nUSER PREFERENCES:\n  - User avoids meetings on Fridays\n  - User prefers informal/casual tone in emails\n  - Meetings with the boss always take priority\n  - User always blocks 12pm-1pm for lunch\n  - User prefers no meetings before 10am\n  - Client meetings cannot be rescheduled\n\nPENDING TASKS:\n  - Resolve conflict between mtg_1 and mtg_2\n  - Resolve conflict between mtg_1 and mtg_3\n  - Resolve conflict between mtg_1 and mtg_4\n  - Resolve conflict between mtg_1 and mtg_5\n  - Resolve conflict between mtg_1 and mtg_6\n  - Resolve conflict between mtg_2 and mtg_3\n  - Resolve conflict between mtg_2 and mtg_4\n  - Resolve conflict between mtg_2 and mtg_5\n  - Resolve conflict between mtg_2 and mtg_6\n  - Resolve conflict between mtg_3 and mtg_4\n  - Resolve conflict between mtg_3 and mtg_5\n  - Resolve conflict between mtg_3 and mtg_6\n  - Resolve conflict between mtg_4 and mtg_5\n  - Resolve conflict between mtg_4 and mtg_6\n  - Resolve conflict between mtg_5 and mtg_6\n  - Reply to email email_1 from Carol Park\n  - Reply to email email_2 from Henry Brown\n  - Reply to email email_3 from Irene Davis\n  - Reply to email email_4 from Henry Brown\n  - Reply to email email_5 from Bob Martinez\n\nAvailable tools: check_calendar, check_inbox, reschedule, draft_reply, delegate_task, done\n\nResolve all conflicts, reply to emails respecting user preferences, and call \"done\" when finished.\nEach tool call should be formatted as: {\"tool\": \"tool_name\", \"arguments\": {...}}",
+    "seed": 99,
+    "difficulty": "hard",
+    "n_conflicts": 15,
+    "n_emails": 5
+  }
+]

training/train_grpo.py

@@ -0,0 +1,195 @@
+"""GRPO Training script for the Executive Assistant Arena.
+
+Run on Northflank H100:
+    cd ~/chief-of-staff/exec_assistant_arena
+    PYTHONPATH=. uvicorn server.app:app --host 0.0.0.0 --port 8000 &
+    python training/train_grpo.py
+"""
+
+import json
+import re
+import os
+import sys
+
+import wandb
+import torch
+from unsloth import FastLanguageModel
+from trl import GRPOTrainer, GRPOConfig
+from datasets import Dataset
+
+# Add parent dir to path so we can import the env
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from exec_assistant_arena import ExecAssistantArenaEnv
+from exec_assistant_arena.models import AssistantAction
+
+# --- Config ---
+MODEL_NAME = "Qwen/Qwen2.5-7B"
+MAX_SEQ_LEN = 2048
+LORA_R = 16
+LORA_ALPHA = 16
+NUM_GENERATIONS = 6
+LEARNING_RATE = 5e-6
+BATCH_SIZE = 2
+MAX_COMPLETION_LENGTH = 1024
+NUM_EPOCHS = 3
+SAVE_STEPS = 200
+ENV_URL = "http://localhost:8000"
+HF_PUSH_REPO = "SidraMiconi/exec-assistant-arena-lora"
+
+# --- Load scenarios ---
+script_dir = os.path.dirname(os.path.abspath(__file__))
+with open(os.path.join(script_dir, "scenarios/train_scenarios.json")) as f:
+    scenarios = json.load(f)
+
+dataset = Dataset.from_dict({
+    "prompt": [s["prompt"] for s in scenarios],
+})
+
+print(f"Loaded {len(dataset)} training scenarios")
+
+
+def parse_tool_calls(completion_text: str) -> list[AssistantAction]:
+    """Parse model completion into a sequence of tool calls."""
+    actions = []
+
+    # Try to find JSON tool calls in the completion
+    # Pattern: {"tool": "...", "arguments": {...}}
+    pattern = r'\{[^{}]*"tool"\s*:\s*"([^"]+)"[^{}]*(?:"arguments"\s*:\s*(\{[^{}]*\}))?[^{}]*\}'
+    matches = re.finditer(pattern, completion_text)
+
+    for match in matches:
+        tool = match.group(1)
+        args_str = match.group(2)
+        args = {}
+        if args_str:
+            try:
+                args = json.loads(args_str)
+            except json.JSONDecodeError:
+                pass
+        actions.append(AssistantAction(tool=tool, arguments=args))
+
+    # If no tool calls found, treat the whole thing as a "done"
+    if not actions:
+        actions.append(AssistantAction(tool="done"))
+
+    return actions
+
+
+def assistant_reward(completions, prompts=None, **kwargs):
+    """Reward function: run each completion through the environment."""
+    rewards = []
+
+    for i, completion in enumerate(completions):
+        # Extract text from completion
+        if isinstance(completion, list):
+            text = "".join(
+                c.get("content", "") if isinstance(c, dict) else str(c)
+                for c in completion
+            )
+        else:
+            text = str(completion)
+
+        try:
+            with ExecAssistantArenaEnv(base_url=ENV_URL) as env:
+                # Determine scenario seed from prompt index
+                prompt_idx = i // NUM_GENERATIONS if prompts else 0
+                seed = scenarios[prompt_idx % len(scenarios)].get("seed", 0)
+                difficulty = scenarios[prompt_idx % len(scenarios)].get("difficulty", "easy")
+
+                result = env.reset(seed=seed, difficulty=difficulty)
+                actions = parse_tool_calls(text)
+
+                total_reward = 0.0
+                for action in actions:
+                    result = env.step(action)
+                    total_reward += (result.reward or 0.0)
+                    if result.done:
+                        break
+
+                # If agent didn't call "done", call it for terminal rewards
+                if not result.done:
+                    result = env.step(AssistantAction(tool="done"))
+                    total_reward += (result.reward or 0.0)
+
+                rewards.append(total_reward)
+
+                # Log component breakdown to W&B
+                state = env.state()
+                wandb.log({
+                    "reward/total": total_reward,
+                    "reward/conflicts_resolved": state.conflicts_resolved,
+                    "reward/total_conflicts": state.total_conflicts,
+                    "reward/conflict_rate": state.conflicts_resolved / max(1, state.total_conflicts),
+                    "reward/preferences_inferred": state.preferences_inferred,
+                    "reward/emails_drafted": state.emails_drafted,
+                    "reward/deadlines_met": state.deadlines_met,
+                    "reward/unnecessary_actions": state.unnecessary_actions,
+                    "reward/late_changes_handled": state.late_changes_handled,
+                    "reward/cumulative": state.cumulative_reward,
+                    "meta/difficulty": difficulty,
+                    "meta/n_actions_parsed": len(actions),
+                })
+
+        except Exception as e:
+            print(f"Error scoring completion {i}: {e}")
+            rewards.append(-1.0)
+
+    return rewards
+
+
+def main():
+    wandb.init(project="exec-assistant-arena", name="grpo-qwen2.5-7b")
+
+    print(f"Loading model: {MODEL_NAME}")
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name=MODEL_NAME,
+        max_seq_length=MAX_SEQ_LEN,
+        load_in_4bit=True,
+    )
+
+    model = FastLanguageModel.get_peft_model(
+        model,
+        r=LORA_R,
+        lora_alpha=LORA_ALPHA,
+        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
+    )
+
+    print("Model loaded. Starting GRPO training...")
+
+    config = GRPOConfig(
+        output_dir="./checkpoints",
+        num_generations=NUM_GENERATIONS,
+        learning_rate=LEARNING_RATE,
+        per_device_train_batch_size=BATCH_SIZE,
+        max_completion_length=MAX_COMPLETION_LENGTH,
+        num_train_epochs=NUM_EPOCHS,
+        save_steps=SAVE_STEPS,
+        report_to="wandb",
+        logging_steps=10,
+        bf16=True,
+        gradient_accumulation_steps=4,
+        warmup_ratio=0.05,
+        max_grad_norm=1.0,
+    )
+
+    trainer = GRPOTrainer(
+        model=model,
+        reward_funcs=assistant_reward,
+        args=config,
+        train_dataset=dataset,
+        processing_class=tokenizer,
+    )
+
+    trainer.train()
+
+    print(f"Training complete. Pushing to {HF_PUSH_REPO}...")
+    model.push_to_hub(HF_PUSH_REPO)
+    tokenizer.push_to_hub(HF_PUSH_REPO)
+    print("Done!")
+
+    wandb.finish()
+
+
+if __name__ == "__main__":
+    main()