Initial commit of PolicyEvolverEnv - Meta Hackathon

.dockerignore

@@ -0,0 +1,12 @@
+__pycache__
+*.pyc
+*.pyo
+.venv/
+.env
+.git/
+*.egg-info/
+dist/
+build/
+outputs/
+baseline_results.json
+.dockerignore

.gitignore

@@ -0,0 +1,15 @@
+.venv/
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.Python
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+*.log
+*results.json
+.DS_Store
+.env

README.md

@@ -0,0 +1,95 @@
+---
+title: PolicyEvolverEnv
+emoji: 🚀
+colorFrom: blue
+colorTo: indigo
+sdk: docker
+app_port: 8000
+---
+# PolicyEvolverEnv
+
+**PolicyEvolverEnv** is an OpenEnv-compliant reinforcement learning environment designed for the Meta × PyTorch × Scaler Hackathon.
+
+## Environment Description & Motivation
+PolicyEvolverEnv is a real-world governance sandbox where an AI agent learns to **design and evolve governance policies** through meta-reasoning over real-world operational data. In modern platforms (social media, enterprise HR, e-commerce), static policies quickly become outdated or vaguely applied, leading to inconsistent enforcement, false-positive moderation, and unrecognized fraud. 
+
+This environment simulates this challenge by presenting the agent with a corpus of operational data alongside an existing policy framework. The agent's goal is to analyze the outcomes, identify systemic flaws or ambiguities, and act directly on the policies to optimize governance outcomes. This directly tackles live production problems faced by platforms like Meta.
+
+## Observation Space
+The `Observation` received by the agent at every step describes the current operational context:
+- `task_id` (str): Identifier for the active scenario.
+- `episode_id` (str): Unique session tracker.
+- `step_count` (int): Active step number (Max 5 per episode).
+- `data_corpus` (List[Dict]): Represents operational examples like social media posts, HR incidents, or seller accounts along with the action taken or outcome.
+- `current_policies` (List[Dict]): The list of current active policies the system follows.
+- `system_metrics` & `policy_outcomes`: Operational statistics reflecting precision/recall or false-positive rates.
+- `identified_issues`: Current known flaws in the governance pipeline.
+
+## Action Space
+The Action space utilizes a highly structured Discriminated Union model to represent multi-faceted policy adjustments:
+
+**1. ProposeClarificationAction (`propose_clarification`)** 
+  - Targets an `ambiguous_term` in an existing policy.
+  - Requires a specific, measurable `suggested_definition` and `justification`.
+**2. ProposeNewRuleAction (`propose_new_rule`)** 
+  - Addresses an unhandled domain (`rule_domain`).
+  - Requires `new_rule` text, application `scope`, and `integration_points` connecting to older policies.
+**3. EvolveProcessAction (`evolve_policy`)** 
+  - The hardest action; holistically modifies existing rules.
+  - Requires a list of `policy_modifications`, realistic `expected_outcomes` deltas, and multi-metric `rollback_conditions`.
+
+*(Each action also supports an optional `think` property allowing Chain-of-Thought meta-reasoning for a reward score bonus).*
+
+## Tasks
+The environment provides three procedural tasks designed to ramp up in cognitive reasoning difficulty:
+
+| Task ID | Difficulty | Expected Score | Description |
+|---|---|---|---|
+| `task_easy` | **Easy** | `~0.80` | **Ambiguity Clarification**: Identify and clarify vague policy terms (e.g., "harassment") in a social media community guideline to improve moderation consistency. |
+| `task_medium` | **Medium** | `~0.70` | **Gap Detection**: Detect uncovered HR policy scenarios involving emerging tech (AI use, gig-worker boundaries) and propose entirely new mandatory rules. |
+| `task_hard` | **Hard** | `~0.55` | **Holistic Evolution**: Analyze complex e-commerce Trust & Safety trade-offs (e.g., false-positive suspensions vs. fraud recall) to rewrite existing volume/return rate policies simultaneously. |
+
+## Setup & Usage
+
+### 1. Local Installation
+```bash
+git clone <repository_url>
+cd policy_evolver_env
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r server/requirements.txt
+```
+
+### 2. Run the Environment API
+Start the FastAPI environment server locally:
+```bash
+uvicorn server.app:app --port 8000
+```
+This boots all core endpoint paths (`/reset`, `/step`, `/state`, `/tasks`, `/grader`, `/health`).
+
+### 3. Run the Inference Baseline
+The environment includes a built-in testing script named `inference.py` ready for deployment on Hugging Face Spaces.
+
+Export your environment variables:
+```bash
+export API_BASE_URL="https://api.openai.com/v1"
+export MODEL_NAME="gpt-4o-mini"
+export HF_TOKEN="your_huggingface_or_openai_api_key_here"
+export OPENENV_BASE_URL="http://localhost:8000"
+```
+
+Execute the agent simulation against the running environment:
+```bash
+python inference.py --mode llm --output json
+```
+*(If no API key is specified, `--mode rule` will execute the deterministic rule-based fallback).*
+
+## Baseline Scores
+The bundled deterministic fallback strategy (`inference.py --mode rule`) yields the following baseline validation scores across the active grader:
+
+- **Easy (Ambiguity Clarification):** 1.000
+- **Medium (New Rule Proposal):** 1.000
+- **Hard (Policy Evolution):** 0.950
+- **Overall Average:** 0.983
+
+*(Note: Live LLM runs generally average expected heuristic bounds around ~0.80, ~0.70, and ~0.55 respectively).*

__init__.py

@@ -0,0 +1,20 @@
+# __init__.py
+from .models import (
+    ProposeClarificationAction,
+    ProposeNewRuleAction,
+    EvolveProcessAction,
+    Action,
+    Observation,
+    State,
+)
+from .client import PolicyEvolverEnv
+
+__all__ = [
+    "PolicyEvolverEnv",
+    "ProposeClarificationAction",
+    "ProposeNewRuleAction",
+    "EvolveProcessAction",
+    "Action",
+    "Observation",
+    "State",
+]

client.py

@@ -0,0 +1,15 @@
+# client.py
+from openenv.core.env_client import EnvClient
+from .models import Action, Observation, State
+
+
+class PolicyEvolverEnv(EnvClient):
+    """
+    Client for PolicyEvolverEnv.
+    Usage:
+        async with PolicyEvolverEnv(base_url="https://your-space.hf.space") as env:
+            obs = await env.reset(task_id="task_easy")
+            result = await env.step(action)
+    """
+    observation_class = Observation
+    state_class = State

inference.py

@@ -0,0 +1,530 @@
+# baseline/run_baseline.py
+"""
+LLM-powered baseline for PolicyEvolverEnv.
+
+Primary path:  Uses AsyncOpenAI client with OPENAI_API_KEY (or HF_TOKEN) to
+               run a language model against all 3 environment tasks.
+Fallback path: Rule-based hardcoded actions used when no API key is available.
+
+Run:
+    python -m policy_evolver_env.baseline.run_baseline                  # LLM baseline (needs OPENAI_API_KEY)
+    python -m policy_evolver_env.baseline.run_baseline --mode rule      # Rule-based fallback
+    python -m policy_evolver_env.baseline.run_baseline --output json    # JSON output
+
+Expected scores (LLM):  easy ~0.80, medium ~0.70, hard ~0.55
+Expected scores (rule): easy ~0.65, medium ~0.50, hard ~0.35
+
+Required env vars:
+    OPENAI_API_KEY   — OpenAI key or HF Inference API token (primary)
+    HF_TOKEN         — Hugging Face token (fallback if no OPENAI_API_KEY)
+    API_BASE_URL     — API endpoint (default: https://api.openai.com/v1)
+    MODEL_NAME       — Model to use (default: gpt-4o-mini)
+    OPENENV_BASE_URL — Environment server (default: http://localhost:8000)
+"""
+from __future__ import annotations
+import asyncio
+import json
+import logging
+import os
+import sys
+import time
+from typing import Dict, List, Optional
+
+import httpx
+from openai import OpenAI
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
+logger = logging.getLogger(__name__)
+
+# ─────────────────────────────────────────────
+# Configuration (all from env vars)
+# ─────────────────────────────────────────────
+
+BASE_URL = os.getenv("OPENENV_BASE_URL", "http://127.0.0.1:8000")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+API_KEY = os.getenv("HF_TOKEN", "") or os.getenv("OPENAI_API_KEY", "")
+MODEL_NAME = os.getenv("MODEL_NAME", "gpt-4o-mini")
+
+
+def verify_environment() -> bool:
+    """Verify required env vars. Returns True if LLM mode is possible."""
+    if not API_KEY:
+        logger.warning(
+            "No API_KEY (HF_TOKEN) found. "
+            "LLM baseline will be skipped. Set one of these env vars to enable it."
+        )
+        return False
+    logger.info(f"API key found. Model: {MODEL_NAME}  Base URL: {API_BASE_URL}")
+    return True
+
+
+# ─────────────────────────────────────────────
+# LLM Agent
+# ─────────────────────────────────────────────
+
+class PolicyEvolverAgent:
+    """LLM-powered agent that calls the OpenAI-compatible API."""
+
+    def __init__(self):
+        self.client = OpenAI(
+            api_key=API_KEY,
+            base_url=API_BASE_URL,
+        )
+        self.model = MODEL_NAME
+
+    def _call(self, prompt: str, max_tokens: int = 700, temperature: float = 0.3) -> Optional[Dict]:
+        """Call the LLM and parse JSON response. Returns None on failure."""
+        try:
+            resp = self.client.chat.completions.create(
+                model=self.model,
+                messages=[
+                    {
+                        "role": "system",
+                        "content": (
+                            "You are a senior policy analyst. "
+                            "Always respond with a single valid JSON object and nothing else. "
+                            "No markdown fences, no preamble."
+                        ),
+                    },
+                    {"role": "user", "content": prompt},
+                ],
+                temperature=temperature,
+                max_tokens=max_tokens,
+            )
+            raw = resp.choices[0].message.content.strip()
+            # Strip accidental markdown fences
+            if raw.startswith("```"):
+                raw = raw.split("```")[1]
+                if raw.startswith("json"):
+                    raw = raw[4:]
+            return json.loads(raw)
+        except Exception as e:
+            logger.warning(f"LLM call failed: {e}")
+            return None
+
+    def handle_easy(self, obs: Dict) -> Dict:
+        """Easy task: propose clarification for an ambiguous policy term."""
+        prompt = f"""
+Analyze the following social media platform policies and user-generated data.
+Identify ONE genuinely ambiguous term that causes inconsistent moderation decisions.
+Propose a specific, measurable definition.
+
+POLICIES:
+{json.dumps(obs.get("current_policies", []), indent=2)}
+
+DATA EXAMPLES (how posts were actually handled):
+{json.dumps(obs.get("data_corpus", [])[:6], indent=2)}
+
+Respond ONLY with this JSON schema:
+{{
+  "action_type": "propose_clarification",
+  "ambiguous_term": "<the exact term from policies>",
+  "suggested_definition": "<specific, ≥15 word definition with clear criteria>",
+  "affected_policy_ids": ["<policy id>"],
+  "justification": "<why inconsistent moderation results; ≥15 words>",
+  "think": "<step-by-step reasoning: which posts were handled inconsistently and why>"
+}}
+"""
+        result = self._call(prompt, max_tokens=600)
+        if result:
+            result["action_type"] = "propose_clarification"
+            return result
+        # Fallback
+        return RULE_BASED_ACTIONS["task_easy"]
+
+    def handle_medium(self, obs: Dict) -> Dict:
+        """Medium task: detect policy gap and propose new rule."""
+        prompt = f"""
+You are reviewing corporate HR policies. The data shows real incidents that occurred.
+Find ONE scenario category NOT adequately covered by existing policies.
+Propose a specific, mandatory new rule to fill the gap.
+
+EXISTING POLICIES:
+{json.dumps(obs.get("current_policies", []), indent=2)}
+
+INCIDENT DATA:
+{json.dumps(obs.get("data_corpus", []), indent=2)}
+
+Respond ONLY with this JSON schema:
+{{
+  "action_type": "propose_new_rule",
+  "rule_domain": "<e.g. AI_use | gig_worker_post_engagement | cross_border_remote>",
+  "new_rule": "<mandatory rule using 'must'/'shall'/'required'; ≥20 words; no vague language>",
+  "scope": ["<scenario 1>", "<scenario 2>", "<scenario 3>", "<scenario 4>"],
+  "integration_points": ["<existing policy id 1>", "<existing policy id 2>"],
+  "justification": "<cite specific incident IDs and why gap exists; ≥20 words>",
+  "think": "<which incident type appears most frequently uncovered and why a rule is needed>"
+}}
+"""
+        result = self._call(prompt, max_tokens=800)
+        if result:
+            result["action_type"] = "propose_new_rule"
+            return result
+        return RULE_BASED_ACTIONS["task_medium"]
+
+    def handle_hard(self, obs: Dict) -> Dict:
+        """Hard task: holistic policy evolution with trade-off reasoning."""
+        prompt = f"""
+You are a senior Trust & Safety policy architect. The current policy framework is
+underperforming. Propose specific modifications to ≥2 existing policies to improve
+both precision (reduce false positives) and recall (catch more fraud) simultaneously.
+Acknowledge the trade-offs explicitly.
+
+CURRENT POLICIES:
+{json.dumps(obs.get("current_policies", []), indent=2)}
+
+PERFORMANCE METRICS (current vs target):
+{json.dumps(obs.get("policy_outcomes", []), indent=2)}
+
+SYSTEM METRICS:
+{json.dumps(obs.get("system_metrics", {}), indent=2)}
+
+KNOWN ISSUES:
+{json.dumps(obs.get("identified_issues", []), indent=2)}
+
+Respond ONLY with this JSON schema:
+{{
+  "action_type": "evolve_policy",
+  "policy_modifications": [
+    {{
+      "policy_id": "<exact policy id from above>",
+      "change_type": "enhance",
+      "new_text": "<specific replacement text; must be context-aware, not blanket>",
+      "reason": "<cite the specific metric that proves current policy fails>"
+    }},
+    {{
+      "policy_id": "<second policy id>",
+      "change_type": "enhance",
+      "new_text": "<replacement text>",
+      "reason": "<metric-backed reason>"
+    }}
+  ],
+  "expected_outcomes": {{
+    "false_positive_rate": <realistic delta 0.01-0.40>,
+    "fraud_detection_rate": <realistic delta 0.01-0.40>,
+    "seller_trust_score": <realistic delta 0.01-0.30>,
+    "review_queue_overload": <realistic delta 0.01-0.40>
+  }},
+  "rollback_conditions": [
+    "<specific numeric threshold that triggers revert>",
+    "<second specific condition with metric name and number>"
+  ],
+  "justification": "<explain trade-offs: what improves, what worsens, and why net positive>",
+  "think": "<identify the two worst-performing metrics and trace root cause to specific policy>"
+}}
+"""
+        result = self._call(prompt, max_tokens=1200, temperature=0.2)
+        if result:
+            result["action_type"] = "evolve_policy"
+            return result
+        return RULE_BASED_ACTIONS["task_hard"]
+
+
+# ─────────────────────────────────────────────
+# Environment interaction helpers (HTTP-based)
+# ─────────────────────────────────────────────
+
+async def env_reset(client: httpx.AsyncClient, task_id: str) -> Dict:
+    resp = await client.post(f"{BASE_URL}/reset", json={"task_id": task_id})
+    resp.raise_for_status()
+    return resp.json()
+
+
+async def env_step(client: httpx.AsyncClient, action: Dict) -> Dict:
+    resp = await client.post(f"{BASE_URL}/step", json={"action": action})
+    resp.raise_for_status()
+    return resp.json()
+
+
+async def run_single_task(
+    http: httpx.AsyncClient,
+    agent: Optional[PolicyEvolverAgent],
+    task_id: str,
+) -> Dict:
+    """Run one task with LLM agent (or rule fallback) and return result."""
+    obs = await env_reset(http, task_id)
+
+    if agent is not None:
+        if task_id == "task_easy":
+            action = agent.handle_easy(obs)
+        elif task_id == "task_medium":
+            action = agent.handle_medium(obs)
+        else:
+            action = agent.handle_hard(obs)
+        mode = "llm"
+    else:
+        action = RULE_BASED_ACTIONS[task_id]
+        mode = "rule"
+
+    result = await env_step(http, action)
+    reward = result.get("reward", 0.0)
+    logger.info(f"[{task_id}] mode={mode}  score={reward:.4f}  done={result.get('done')}")
+    return {"task_id": task_id, "reward": reward, "mode": mode, "done": result.get("done", False)}
+
+
+# ─────────────────────────────────────────────
+# Direct baseline (no HTTP — used by /baseline endpoint)
+# ─────────────────────────────────────────────
+
+async def run_direct_baseline() -> Dict:
+    """
+    Run baseline directly using environment and grader imports.
+    Used by the /baseline endpoint to avoid self-HTTP calls on HF Spaces.
+    """
+    from ..server.environment import PolicyEvolverEnvironment
+    from ..server.grader import grade
+
+    env = PolicyEvolverEnvironment()
+    use_llm = verify_environment()
+    agent = PolicyEvolverAgent() if use_llm else None
+
+    start = time.time()
+    results: List[Dict] = []
+
+    for task_id in ["task_easy", "task_medium", "task_hard"]:
+        try:
+            obs = env.reset(task_id=task_id)
+            obs_dict = obs.model_dump()
+
+            if agent is not None:
+                if task_id == "task_easy":
+                    action = agent.handle_easy(obs_dict)
+                elif task_id == "task_medium":
+                    action = agent.handle_medium(obs_dict)
+                else:
+                    action = agent.handle_hard(obs_dict)
+                mode = "llm"
+            else:
+                action = RULE_BASED_ACTIONS[task_id]
+                mode = "rule"
+
+            result_obs = env.step(action)
+            reward = result_obs.reward
+            logger.info(f"[{task_id}] mode={mode}  score={reward:.4f}  done={result_obs.done}")
+            results.append({"task_id": task_id, "reward": reward, "mode": mode, "done": result_obs.done})
+        except Exception as e:
+            logger.error(f"[{task_id}] failed: {e}")
+            results.append({"task_id": task_id, "reward": 0.0, "mode": "error", "error": str(e)})
+
+    scores = {r["task_id"]: max(0.0, min(1.0, r["reward"])) for r in results}
+    overall = sum(scores.values()) / len(scores) if scores else 0.0
+
+    return {
+        "baseline_scores": {
+            "task_easy": scores.get("task_easy", 0.0),
+            "task_medium": scores.get("task_medium", 0.0),
+            "task_hard": scores.get("task_hard", 0.0),
+            "overall_avg": round(overall, 4),
+        },
+        "mode": "llm" if use_llm else "rule_fallback",
+        "model": MODEL_NAME if use_llm else "rule-based",
+        "runtime_seconds": round(time.time() - start, 2),
+        "detail": results,
+    }
+
+
+# ─────────────────────────────────────────────
+# Main HTTP-based baseline runner
+# ─────────────────────────────────────────────
+
+async def run_llm_baseline() -> Dict:
+    """Primary baseline: LLM agent against all 3 tasks via HTTP."""
+    use_llm = verify_environment()
+    agent = PolicyEvolverAgent() if use_llm else None
+
+    start = time.time()
+    results: List[Dict] = []
+
+    async with httpx.AsyncClient(timeout=120.0) as http:
+        for task_id in ["task_easy", "task_medium", "task_hard"]:
+            if time.time() - start > 1140:
+                logger.warning("Approaching 20min time limit — stopping early")
+                break
+            try:
+                r = await run_single_task(http, agent, task_id)
+                results.append(r)
+            except Exception as e:
+                logger.error(f"[{task_id}] failed: {e}")
+                results.append({"task_id": task_id, "reward": 0.0, "mode": "error", "error": str(e)})
+
+    scores = {r["task_id"]: max(0.0, min(1.0, r["reward"])) for r in results}
+    overall = sum(scores.values()) / len(scores) if scores else 0.0
+
+    summary = {
+        "baseline_scores": {
+            "task_easy": scores.get("task_easy", 0.0),
+            "task_medium": scores.get("task_medium", 0.0),
+            "task_hard": scores.get("task_hard", 0.0),
+            "overall_avg": round(overall, 4),
+        },
+        "mode": "llm" if use_llm else "rule_fallback",
+        "model": MODEL_NAME if use_llm else "rule-based",
+        "runtime_seconds": round(time.time() - start, 2),
+        "detail": results,
+    }
+
+    # Persist for analysis
+    try:
+        with open("baseline_results.json", "w") as f:
+            json.dump(summary, f, indent=2)
+    except Exception:
+        pass
+
+    return summary
+
+
+# Keep rule-based runner available for /baseline endpoint fallback
+async def run_rule_based_baseline() -> Dict:
+    """Fallback: hardcoded rule-based actions, no LLM required."""
+    results: List[Dict] = []
+    async with httpx.AsyncClient(timeout=60.0) as http:
+        for task_id, action in RULE_BASED_ACTIONS.items():
+            try:
+                await env_reset(http, task_id)
+                result = await env_step(http, action)
+                reward = max(0.0, min(1.0, result.get("reward", 0.0)))
+                results.append({"task_id": task_id, "reward": reward})
+                logger.info(f"[{task_id}] rule score={reward:.4f}")
+            except Exception as e:
+                logger.error(f"[{task_id}] rule baseline error: {e}")
+                results.append({"task_id": task_id, "reward": 0.0})
+    scores = {r["task_id"]: r["reward"] for r in results}
+    overall = sum(scores.values()) / len(scores) if scores else 0.0
+    return {**scores, "overall_avg": round(overall, 4)}
+
+
+# ─────────────────────────────────────────────
+# Rule-based fallback actions (used when OPENAI_API_KEY not set)
+# ─────────────────────────────────────────────
+
+RULE_BASED_ACTIONS = {
+    "task_easy": {
+        "action_type": "propose_clarification",
+        "ambiguous_term": "harassment",
+        "suggested_definition": (
+            "Harassment is defined as any repeated, unwanted communication or behaviour "
+            "directed at a specific individual that a reasonable person would find threatening, "
+            "intimidating, or distressing. This includes but is not limited to targeted insults, "
+            "threats, and sustained negative attention. Single interactions may qualify if "
+            "sufficiently severe."
+        ),
+        "affected_policy_ids": ["pol_002"],
+        "justification": (
+            "The term 'harassment' is subjective and moderators apply it inconsistently. "
+            "Different reviewers may interpret the same post differently without a measurable definition."
+        ),
+        "think": (
+            "Looking at the data, posts 001 and 006 were treated differently despite similar tone. "
+            "The key ambiguous term causing inconsistency is 'harassment' in pol_002."
+        ),
+    },
+    "task_medium": {
+        "action_type": "propose_new_rule",
+        "rule_domain": "AI_use",
+        "new_rule": (
+            "Employees must disclose when AI tools are used to generate, substantially edit, or "
+            "evaluate work products that are submitted under their name, including client proposals, "
+            "code submissions, and performance evaluations. AI-assisted content must be reviewed "
+            "and validated by the submitting employee before delivery."
+        ),
+        "scope": [
+            "AI-generated client proposals",
+            "AI-written code in performance reviews",
+            "AI-assisted HR decisions",
+            "Automated content in employee-attributed work",
+        ],
+        "integration_points": ["pol_hr_001", "pol_hr_005"],
+        "justification": (
+            "Incidents 001, 004, and 007 all involve AI use that current policies do not address. "
+            "There is no rule requiring disclosure or validation of AI-generated work, creating "
+            "a gap in accountability and intellectual honesty."
+        ),
+        "think": (
+            "The uncovered domain is AI use in professional work. Three of 10 incidents involve this. "
+            "The new rule must be mandatory (not advisory) and must specify disclosure + validation."
+        ),
+    },
+    "task_hard": {
+        "action_type": "evolve_policy",
+        "policy_modifications": [
+            {
+                "policy_id": "ts_pol_001",
+                "change_type": "enhance",
+                "new_text": (
+                    "New seller accounts with more than 50 transactions in the first week will be "
+                    "reviewed only if additional risk signals are present (e.g., chargeback rate > 5%, "
+                    "price variance > 30%, or fraud reports). Seasonal categories (gifts, fashion) "
+                    "have an elevated threshold of 150 transactions during peak periods."
+                ),
+                "reason": "Blanket volume threshold causes 42% false positive rate among legitimate high-volume sellers.",
+            },
+            {
+                "policy_id": "ts_pol_002",
+                "change_type": "enhance",
+                "new_text": (
+                    "Return rate thresholds are applied per category: electronics > 10%, fashion > 25%, "
+                    "general goods > 15%. Accounts exceeding category thresholds are flagged for review, "
+                    "not automatic suspension."
+                ),
+                "reason": "Return rate varies dramatically by category; a single threshold discriminates against fashion sellers.",
+            },
+        ],
+        "expected_outcomes": {
+            "false_positive_rate": 0.20,
+            "fraud_detection_rate": 0.35,
+            "seller_trust_score": 0.15,
+            "review_queue_overload": 0.30,
+        },
+        "rollback_conditions": [
+            "false_positive_rate increases above 0.50 after policy change",
+            "fraud_detection_rate drops below 0.25 within 30 days",
+            "seller trust score decreases by more than 0.10 in 14-day survey",
+        ],
+        "justification": (
+            "The current framework has a 42% false positive rate because blanket thresholds don't "
+            "account for legitimate high-volume or high-return categories. Modifying ts_pol_001 and "
+            "ts_pol_002 to be context-aware reduces wrongful suspensions while maintaining fraud "
+            "detection via multi-signal scoring. Trade-off: fraud_detection_rate may improve more "
+            "slowly since we're relaxing volume triggers, but seller trust and queue overload improve "
+            "immediately."
+        ),
+        "think": (
+            "The system_metrics show false_positive_rate=0.42 and fraud_detection_rate=0.31. "
+            "The identified issues all point to overly broad thresholds. I should modify the two "
+            "most impactful policies and provide category-specific thresholds. "
+            "The rollback conditions should be metric-specific with concrete numbers."
+        ),
+    },
+}
+
+
+if __name__ == "__main__":
+    import argparse
+    parser = argparse.ArgumentParser(description="PolicyEvolverEnv baseline runner")
+    parser.add_argument("--mode", choices=["llm", "rule"], default="llm",
+                        help="llm = LLM agent (needs OPENAI_API_KEY); rule = hardcoded fallback")
+    parser.add_argument("--output", choices=["text", "json"], default="text")
+    args = parser.parse_args()
+
+    if args.mode == "rule":
+        summary = asyncio.run(run_rule_based_baseline())
+        scores = summary
+    else:
+        summary = asyncio.run(run_llm_baseline())
+        scores = summary.get("baseline_scores", summary)
+
+    if args.output == "json":
+        print(json.dumps(summary, indent=2))
+    else:
+        print("\n" + "=" * 50)
+        print("POLICEVOLVERENV BASELINE SCORES")
+        print("=" * 50)
+        print(f"Easy   (Ambiguity Clarification): {scores.get('task_easy', 0.0):.3f}")
+        print(f"Medium (New Rule Proposal):        {scores.get('task_medium', 0.0):.3f}")
+        print(f"Hard   (Policy Evolution):         {scores.get('task_hard', 0.0):.3f}")
+        print(f"Overall Average:                   {scores.get('overall_avg', 0.0):.3f}")
+        print("=" * 50)
+
+        for k, v in scores.items():
+            if isinstance(v, float) and not (0.0 <= v <= 1.0):
+                raise ValueError(f"Score {k}={v} outside [0.0, 1.0] — submission invalid")

models.py

@@ -0,0 +1,82 @@
+# models.py
+from __future__ import annotations
+from pydantic import BaseModel, Field
+from typing import Optional, List, Dict, Literal, Union
+from enum import Enum
+import uuid
+
+
+class PolicyActionType(str, Enum):
+    PROPOSE_CLARIFICATION = "propose_clarification"
+    PROPOSE_NEW_RULE = "propose_new_rule"
+    EVOLVE_POLICY = "evolve_policy"
+
+
+class ProposeClarificationAction(BaseModel):
+    """Easy task: identify an ambiguous policy term and clarify it."""
+    action_type: Literal[PolicyActionType.PROPOSE_CLARIFICATION] = PolicyActionType.PROPOSE_CLARIFICATION
+    ambiguous_term: str = Field(description="The exact ambiguous term found in policies")
+    suggested_definition: str = Field(description="A specific, actionable definition")
+    affected_policy_ids: List[str] = Field(default_factory=list, description="Policy IDs this affects")
+    justification: str = Field(description="Why this term is ambiguous")
+    think: Optional[str] = Field(default=None, description="Chain-of-thought reasoning (earns +0.1 bonus)")
+
+
+class ProposeNewRuleAction(BaseModel):
+    """Medium task: detect a policy gap and propose a new rule."""
+    action_type: Literal[PolicyActionType.PROPOSE_NEW_RULE] = PolicyActionType.PROPOSE_NEW_RULE
+    rule_domain: str = Field(description="Domain the rule covers, e.g. 'content_moderation'")
+    new_rule: str = Field(description="The new rule text — must be clear and actionable")
+    scope: List[str] = Field(description="List of scenario types this rule covers")
+    integration_points: List[str] = Field(default_factory=list, description="How it connects to existing policies")
+    justification: str = Field(description="Why a gap exists and why this rule fills it")
+    think: Optional[str] = Field(default=None, description="Chain-of-thought reasoning (earns +0.1 bonus)")
+
+
+class PolicyModification(BaseModel):
+    policy_id: str
+    change_type: Literal["enhance", "restrict", "add", "remove"]
+    new_text: str
+    reason: str
+
+
+class EvolveProcessAction(BaseModel):
+    """Hard task: holistically evolve the policy framework."""
+    action_type: Literal[PolicyActionType.EVOLVE_POLICY] = PolicyActionType.EVOLVE_POLICY
+    policy_modifications: List[PolicyModification] = Field(description="Specific changes to make")
+    expected_outcomes: Dict[str, float] = Field(description="Metric name → expected delta (0.0–1.0)")
+    rollback_conditions: List[str] = Field(default_factory=list, description="When to revert")
+    justification: str = Field(description="Comprehensive reasoning")
+    think: Optional[str] = Field(default=None, description="Chain-of-thought reasoning (earns +0.1 bonus)")
+
+
+from pydantic import RootModel
+
+class Action(RootModel):
+    root: Union[ProposeClarificationAction, ProposeNewRuleAction, EvolveProcessAction] = Field(..., discriminator="action_type")
+
+
+class Observation(BaseModel):
+    """What the agent sees after reset() or step()."""
+    task_id: str
+    episode_id: str
+    step_count: int
+    data_corpus: List[Dict] = Field(description="Scenarios/posts/actions for the agent to analyze")
+    current_policies: List[Dict] = Field(description="The existing policy set")
+    policy_outcomes: Optional[List[Dict]] = Field(default=None, description="Historical outcome data (hard task)")
+    system_metrics: Dict[str, float] = Field(default_factory=dict)
+    identified_issues: List[Dict] = Field(default_factory=list)
+    reward: float = 0.0
+    done: bool = False
+    info: Dict = Field(default_factory=dict)
+
+
+class State(BaseModel):
+    """Episode metadata — returned by state() endpoint."""
+    episode_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
+    task_id: str = ""
+    step_count: int = 0
+    max_steps: int = 5
+    current_score: float = 0.0
+    best_score: float = 0.0
+    actions_taken: List[str] = Field(default_factory=list)

openenv.yaml

@@ -0,0 +1,77 @@
+name: "PolicyEvolverEnv"
+description: "Policy Design and Evolution Sandbox — agents learn to evolve real-world governance frameworks through meta-reasoning"
+version: "1.0.0"
+author: "PolicyEvolution Team"
+tags:
+  - "policy"
+  - "governance"
+  - "meta-reasoning"
+  - "content-moderation"
+  - "AI-safety"
+
+environment:
+  module: "policy_evolver_env.server.environment"
+  class: "PolicyEvolverEnvironment"
+
+  observation_schema:
+    type: "object"
+    description: "Policy context, data corpus, and system state"
+
+  action_schema:
+    type: "object"
+    description: "Discriminated union on action_type field"
+    discriminator: "action_type"
+    variants:
+      - action_type: "propose_clarification"
+        schema: "ProposeClarificationAction"
+      - action_type: "propose_new_rule"
+        schema: "ProposeNewRuleAction"
+      - action_type: "evolve_policy"
+        schema: "EvolveProcessAction"
+
+  reward_range: [0.0, 1.0]
+
+runtime:
+  max_steps: 5
+  timeout_seconds: 1200
+  vcpu: 2
+  memory_gb: 8
+
+tasks:
+  - id: "task_easy"
+    difficulty: "easy"
+    description: "Identify and clarify ambiguous policy terms in a social media community guidelines"
+    expected_min_score: 0.70
+
+  - id: "task_medium"
+    difficulty: "medium"
+    description: "Detect policy gaps in corporate HR policies and propose new rules for emerging scenarios"
+    expected_min_score: 0.55
+
+  - id: "task_hard"
+    difficulty: "hard"
+    description: "Holistically evolve an e-commerce Trust & Safety framework with trade-off reasoning"
+    expected_min_score: 0.40
+
+grading:
+  module: "policy_evolver_env.server.grader"
+  function: "grade"
+  return_range: [0.0, 1.0]
+
+endpoints:
+  required:
+    - path: "/reset"
+      method: "POST"
+    - path: "/step"
+      method: "POST"
+    - path: "/state"
+      method: "GET"
+    - path: "/tasks"
+      method: "GET"
+    - path: "/grader"
+      method: "GET"
+    - path: "/baseline"
+      method: "GET"
+  optional:
+    - path: "/health"
+      method: "GET"

pyproject.toml

@@ -0,0 +1,28 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "policy-evolver-env"
+version = "1.0.0"
+description = "PolicyEvolverEnv — OpenEnv RL environment for policy evolution through meta-reasoning"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "openenv-core>=0.1.0",
+    "pydantic>=2.9.2",
+    "pydantic-settings>=2.6.1",
+    "fastapi>=0.115.4",
+    "uvicorn>=0.32.0",
+    "scikit-learn>=1.5.2",
+    "numpy>=1.26.4",
+    "openai>=1.54.4",
+    "httpx>=0.27.0",
+    "python-dotenv>=1.0.1",
+]
+
+[project.optional-dependencies]
+dev = ["pytest", "pytest-asyncio", "httpx"]
+
+[tool.setuptools.packages.find]
+include = ["policy_evolver_env*"]

rule_output3.json

@@ -0,0 +1,6 @@
+{
+  "task_easy": 0.0,
+  "task_medium": 0.0,
+  "task_hard": 0.0,
+  "overall_avg": 0.0
+}

server/Dockerfile

@@ -0,0 +1,20 @@
+FROM openenv-base:latest
+
+WORKDIR /app
+
+# Copy dependency file first for layer caching
+COPY server/requirements.txt ./requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy full package
+COPY . .
+
+# Expose port
+EXPOSE 8000
+
+# Liveness probe — validator checks this
+HEALTHCHECK --interval=30s --timeout=10s --start-period=10s --retries=3 \
+  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
+
+# Start server
+CMD ["uvicorn", "policy_evolver_env.server.app:app", "--host", "0.0.0.0", "--port", "8000"]

server/__init__.py

@@ -0,0 +1 @@
+# server/__init__.py

server/app.py

@@ -0,0 +1,63 @@
+# server/app.py
+from __future__ import annotations
+from fastapi import FastAPI, HTTPException, Query
+from fastapi.responses import JSONResponse
+from openenv.core.env_server import create_fastapi_app
+from ..models import (
+    ProposeClarificationAction, ProposeNewRuleAction, EvolveProcessAction,
+    Observation, Action
+)
+from .environment import PolicyEvolverEnvironment
+from .grader import grade
+from .tasks import TASK_REGISTRY
+import json
+
+# Create app via OpenEnv helper — pass factory callable, action/obs classes
+app = create_fastapi_app(
+    env=PolicyEvolverEnvironment,
+    action_cls=Action,          # Pydantic union
+    observation_cls=Observation,
+)
+
+
+@app.get("/health")
+async def health():
+    return {"status": "healthy", "environment": "PolicyEvolverEnv", "version": "1.0.0"}
+
+
+@app.get("/tasks")
+async def list_tasks():
+    return [
+        {
+            "task_id": tid,
+            "difficulty": t["difficulty"],
+            "description": t["description"],
+            "num_policies": t["num_policies"],
+            "num_data_points": t["num_data_points"],
+        }
+        for tid, t in TASK_REGISTRY.items()
+    ]
+
+
+@app.get("/grader")
+async def grader_endpoint(
+    task_id: str = Query(..., description="task_easy | task_medium | task_hard"),
+    action_json: str = Query(..., description="JSON-encoded action dict"),
+):
+    try:
+        action_dict = json.loads(action_json)
+    except json.JSONDecodeError:
+        raise HTTPException(status_code=400, detail="action_json must be valid JSON")
+    score = grade(action_dict, task_id)
+    return {"task_id": task_id, "score": score}
+
+
+@app.get("/baseline")
+async def run_baseline_endpoint():
+    """
+    Runs the LLM baseline (or rule-based fallback) and returns scores for all tasks.
+    Uses the grader directly instead of HTTP calls to self.
+    """
+    from ..inference import run_direct_baseline
+    results = await run_direct_baseline()
+    return results

server/environment.py

@@ -0,0 +1,120 @@
+# server/environment.py
+from __future__ import annotations
+import uuid
+import random
+from typing import Optional, Any, Dict
+from openenv.core.env_server import Environment
+from ..models import (
+    Action, Observation, State,
+    ProposeClarificationAction, ProposeNewRuleAction, EvolveProcessAction,
+)
+from .grader import grade
+from .tasks import TASK_REGISTRY
+
+
+class PolicyEvolverEnvironment(Environment[Action, Observation, State]):
+    """
+    Real-world environment: AI agent learns to evolve governance policies
+    through meta-reasoning over real-world data.
+    """
+    _instance = None
+
+    def __new__(cls, *args, **kwargs):
+        if cls._instance is None:
+            cls._instance = super().__new__(cls)
+            cls._instance._initialized = False
+        return cls._instance
+
+    def __init__(self):
+        if getattr(self, "_initialized", False):
+            return
+        super().__init__()
+        self._state = State()
+        self._current_task = None
+        self._initialized = True
+
+    def reset(
+        self,
+        seed: Optional[int] = None,
+        episode_id: Optional[str] = None,
+        **kwargs: Any,
+    ) -> Observation:
+        task_id = kwargs.get("task_id")
+        if task_id is None:
+            task_id = random.choice(list(TASK_REGISTRY.keys()))
+
+        task = TASK_REGISTRY[task_id]
+        self._current_task = task
+        self._state = State(
+            episode_id=episode_id or str(uuid.uuid4()),
+            task_id=task_id,
+            step_count=0,
+            max_steps=5,
+            current_score=0.0,
+            best_score=0.0,
+            actions_taken=[],
+        )
+
+        return Observation(
+            task_id=task_id,
+            episode_id=self._state.episode_id,
+            step_count=0,
+            data_corpus=task["data_corpus"],
+            current_policies=task["current_policies"],
+            policy_outcomes=task.get("policy_outcomes"),
+            system_metrics=task.get("system_metrics", {}),
+            identified_issues=task.get("identified_issues", []),
+            reward=0.0,
+            done=False,
+            info={"task_description": task["description"], "difficulty": task["difficulty"]},
+        )
+
+    def step(
+        self,
+        action: Action,
+        timeout_s: Optional[float] = None,
+        **kwargs: Any,
+    ) -> Observation:
+        if self._current_task is None:
+            raise RuntimeError("Call reset() before step()")
+
+        self._state.step_count += 1
+
+        # action can be a dict from the API or a Pydantic model
+        if isinstance(action, dict):
+            action_dict = action
+        else:
+            action_dict = action.model_dump() if hasattr(action, "model_dump") else dict(action)
+
+        reward = grade(action_dict, self._state.task_id)
+        self._state.current_score = reward
+        self._state.best_score = max(self._state.best_score, reward)
+
+        action_type = action_dict.get("action_type", "unknown") if isinstance(action_dict, dict) else "unknown"
+        self._state.actions_taken.append(action_type)
+
+        done = (
+            reward >= 0.90 or
+            self._state.step_count >= self._state.max_steps
+        )
+
+        return Observation(
+            task_id=self._state.task_id,
+            episode_id=self._state.episode_id,
+            step_count=self._state.step_count,
+            data_corpus=self._current_task["data_corpus"],
+            current_policies=self._current_task["current_policies"],
+            policy_outcomes=self._current_task.get("policy_outcomes"),
+            system_metrics=self._current_task.get("system_metrics", {}),
+            identified_issues=self._current_task.get("identified_issues", []),
+            reward=reward,
+            done=done,
+            info={
+                "best_score": self._state.best_score,
+                "steps_remaining": self._state.max_steps - self._state.step_count,
+            },
+        )
+
+    @property
+    def state(self) -> State:
+        return self._state

server/grader.py

@@ -0,0 +1,247 @@
+# server/grader.py
+"""
+Deterministic grader for all three PolicyEvolverEnv tasks.
+All functions return float in [0.0, 1.0].
+"""
+from __future__ import annotations
+import re
+from typing import Dict, List, Any
+from ..models import (
+    ProposeClarificationAction, ProposeNewRuleAction, EvolveProcessAction,
+    Observation
+)
+from .tasks import TASK_REGISTRY
+
+
+# ─────────────────────────────────────────────
+# Easy Task: Ambiguity Clarification
+# ─────────────────────────────────────────────
+
+def grade_clarification(action: ProposeClarificationAction, task: Dict) -> float:
+    """
+    Reward breakdown:
+      0.35 — identified term is genuinely ambiguous (in known_ambiguous_terms)
+      0.35 — definition is specific (≥12 words, contains measurement/criteria language)
+      0.20 — justification addresses WHY term causes inconsistent moderation
+      0.10 — think field provided (CoT bonus)
+    """
+    score = 0.0
+
+    # 0.35: Is the identified term actually ambiguous?
+    known = [t.lower() for t in task.get("known_ambiguous_terms", [])]
+    if action.ambiguous_term.lower() in known:
+        score += 0.35
+    else:
+        # Partial credit if it's a word that plausibly causes ambiguity
+        vague_words = ["reasonable", "substantial", "appropriate", "excessive", "significant",
+                       "severe", "abusive", "hostile", "threatening", "offensive", "respectful"]
+        if any(w in action.ambiguous_term.lower() for w in vague_words):
+            score += 0.15
+
+    # 0.35: Definition quality
+    defn = action.suggested_definition
+    defn_score = 0.0
+    words = defn.split()
+    if len(words) >= 12:
+        defn_score += 0.10
+    criteria_words = ["includes", "means", "refers to", "defined as", "encompasses",
+                      "specifically", "measurable", "example", "such as", "e.g."]
+    if any(w in defn.lower() for w in criteria_words):
+        defn_score += 0.15
+    action_words = ["will", "must", "shall", "is", "are", "requires"]
+    if any(w in defn.lower() for w in action_words):
+        defn_score += 0.10
+    score += min(defn_score, 0.35)
+
+    # 0.20: Justification quality
+    just = action.justification.lower()
+    just_score = 0.0
+    if len(action.justification.split()) >= 10:
+        just_score += 0.10
+    inconsistency_words = ["inconsistent", "vary", "subjective", "unclear", "different",
+                           "interpret", "misapply", "dispute", "ambiguous"]
+    if any(w in just for w in inconsistency_words):
+        just_score += 0.10
+    score += min(just_score, 0.20)
+
+    # 0.10: CoT bonus
+    if action.think and len(action.think.strip()) > 20:
+        score += 0.10
+
+    return round(min(score, 1.0), 4)
+
+
+# ─────────────────────────────────────────────
+# Medium Task: Gap Detection + New Rule
+# ─────────────────────────────────────────────
+
+def grade_new_rule(action: ProposeNewRuleAction, task: Dict) -> float:
+    """
+    Reward breakdown:
+      0.30 — rule_domain matches a genuinely uncovered domain
+      0.30 — rule text is specific and actionable (not vague platitude)
+      0.25 — scope covers multiple relevant scenarios
+      0.05 — integration_points reference existing policies
+      0.10 — think field provided (CoT bonus)
+    """
+    score = 0.0
+
+    # 0.30: Domain is genuinely uncovered
+    uncovered = [d.lower() for d in task.get("uncovered_domains", [])]
+    domain_lower = action.rule_domain.lower().replace(" ", "_")
+    if any(u in domain_lower or domain_lower in u for u in uncovered):
+        score += 0.30
+    else:
+        # Partial credit for related but not exact domain
+        related = ["ai", "artificial intelligence", "remote", "contractor", "freelance",
+                   "gig", "machine learning", "automation", "offshore", "cross_border"]
+        if any(r in domain_lower for r in related):
+            score += 0.15
+
+    # 0.30: Rule text quality
+    rule = action.new_rule
+    rule_score = 0.0
+    if len(rule.split()) >= 15:
+        rule_score += 0.10
+    mandatory_words = ["must", "will", "shall", "required", "prohibited", "mandatory"]
+    if any(w in rule.lower() for w in mandatory_words):
+        rule_score += 0.10
+    conditional_words = ["when", "if", "unless", "in cases where", "prior to", "before"]
+    if any(w in rule.lower() for w in conditional_words):
+        rule_score += 0.10
+    # Penalise vague language
+    vague = ["may", "should consider", "might", "perhaps", "in some cases"]
+    if any(w in rule.lower() for w in vague):
+        rule_score -= 0.10
+    score += max(min(rule_score, 0.30), 0.0)
+
+    # 0.25: Scope covers multiple scenario types
+    if len(action.scope) >= 2:
+        score += 0.15
+    if len(action.scope) >= 4:
+        score += 0.10
+
+    # 0.05: Integration points reference existing policy IDs or domains
+    if action.integration_points and len(action.integration_points) >= 1:
+        score += 0.05
+
+    # 0.10: CoT bonus
+    if action.think and len(action.think.strip()) > 20:
+        score += 0.10
+
+    return round(min(score, 1.0), 4)
+
+
+# ─────────────────────────────────────────────
+# Hard Task: Holistic Policy Evolution
+# ─────────────────────────────────────────────
+
+def grade_evolution(action: EvolveProcessAction, task: Dict) -> float:
+    """
+    Reward breakdown:
+      0.30 — ≥2 policy modifications; modifications address identified_issues
+      0.25 — expected_outcomes are realistic and cover key metrics
+      0.20 — rollback_conditions are specific (not generic)
+      0.15 — justification addresses trade-offs (both sides)
+      0.10 — think field provided (CoT bonus)
+    """
+    score = 0.0
+    identified_issues = [i["issue"].lower() for i in task.get("identified_issues", [])]
+    key_metrics = {o["metric"] for o in task.get("policy_outcomes", [])}
+
+    # 0.30: Modifications address real problems
+    mods = action.policy_modifications
+    mod_score = 0.0
+    if len(mods) >= 2:
+        mod_score += 0.15
+    # Check that at least one modification references a known policy ID or known issue
+    known_policy_ids = {p["id"] for p in task.get("current_policies", [])}
+    addressed = sum(1 for m in mods if m.policy_id in known_policy_ids or
+                    any(kw in m.new_text.lower() for kw in
+                        ["seasonal", "category", "foreign", "manual", "threshold", "volume"]))
+    if addressed >= 1:
+        mod_score += 0.10
+    if addressed >= 2:
+        mod_score += 0.05
+    score += min(mod_score, 0.30)
+
+    # 0.25: Expected outcomes realistic and cover key metrics
+    outcomes = action.expected_outcomes
+    outcome_score = 0.0
+    covered_metrics = {m for m in outcomes if m in key_metrics}
+    if len(covered_metrics) >= 2:
+        outcome_score += 0.15
+    # Values should be realistic deltas (not all 1.0)
+    non_trivial = sum(1 for v in outcomes.values() if 0.01 <= v <= 0.60)
+    if non_trivial >= 2:
+        outcome_score += 0.10
+    score += min(outcome_score, 0.25)
+
+    # 0.20: Rollback conditions are specific
+    rollbacks = action.rollback_conditions
+    rollback_score = 0.0
+    if len(rollbacks) >= 1:
+        rollback_score += 0.10
+    # Specific = contains a number or metric name
+    specific = sum(1 for r in rollbacks if
+                   re.search(r'\d+', r) or
+                   any(m in r.lower() for m in ["false positive", "fraud", "trust", "revenue", "queue"]))
+    if specific >= 1:
+        rollback_score += 0.10
+    score += min(rollback_score, 0.20)
+
+    # 0.15: Justification addresses trade-offs
+    just = action.justification.lower()
+    trade_off_pairs = [
+        (["precision", "accuracy", "false positive"], ["recall", "coverage", "missed"]),
+        (["seller trust", "legitimate"], ["fraud", "detection"]),
+        (["automation", "efficiency"], ["manual", "review"]),
+    ]
+    tradeoffs_found = 0
+    for side_a, side_b in trade_off_pairs:
+        if any(w in just for w in side_a) and any(w in just for w in side_b):
+            tradeoffs_found += 1
+    if tradeoffs_found >= 1:
+        score += 0.10
+    if tradeoffs_found >= 2:
+        score += 0.05
+
+    # 0.10: CoT bonus
+    if action.think and len(action.think.strip()) > 20:
+        score += 0.10
+
+    return round(min(score, 1.0), 4)
+
+
+# ─────────────────────────────────────────────
+# Dispatcher
+# ─────────────────────────────────────────────
+
+def grade(action_dict: Dict, task_id: str) -> float:
+    """
+    Main entry point called by /grader endpoint.
+    action_dict: the raw JSON body from the agent
+    task_id: "task_easy" | "task_medium" | "task_hard"
+    Returns float in [0.0, 1.0] — always clamped.
+    """
+    task = TASK_REGISTRY.get(task_id)
+    if task is None:
+        return 0.0
+
+    try:
+        action_type = action_dict.get("action_type")
+        if action_type == "propose_clarification":
+            action = ProposeClarificationAction(**action_dict)
+            raw = grade_clarification(action, task)
+        elif action_type == "propose_new_rule":
+            action = ProposeNewRuleAction(**action_dict)
+            raw = grade_new_rule(action, task)
+        elif action_type == "evolve_policy":
+            action = EvolveProcessAction(**action_dict)
+            raw = grade_evolution(action, task)
+        else:
+            return 0.0
+    except Exception:
+        return 0.0
+
+    return max(0.0, min(1.0, raw))

server/requirements.txt

@@ -0,0 +1,10 @@
+openenv-core>=0.1.0
+pydantic>=2.9.2
+pydantic-settings>=2.6.1
+fastapi>=0.115.4
+uvicorn>=0.32.0
+scikit-learn>=1.5.2
+numpy>=1.26.4
+openai>=1.54.4
+httpx>=0.27.0
+python-dotenv>=1.0.1

server/task_generator.py

@@ -0,0 +1,32 @@
+# server/task_generator.py
+"""
+Procedural policy scenario generator.
+Generates variants of each task by swapping domains, severity, and actor types.
+Activated by calling generate_task_variants() and updating TASK_REGISTRY.
+"""
+from __future__ import annotations
+import random
+from typing import Iterator, Dict
+
+DOMAIN_VARIANTS = {
+    "social_media": ["gaming_platform", "professional_network", "dating_app", "news_forum"],
+    "corporate_hr": ["startup_culture", "remote_first_company", "government_agency", "university"],
+    "ecommerce": ["marketplace", "subscription_service", "auction_platform", "b2b_procurement"],
+}
+
+ACTOR_VARIANTS = ["new_user", "power_user", "verified_creator", "anonymous_account", "enterprise_client"]
+
+
+def generate_easy_variant(base_task: Dict, domain: str) -> Dict:
+    variant = dict(base_task)
+    variant["task_id"] = f"task_easy_{domain.replace(' ', '_')}"
+    variant["description"] = base_task["description"].replace("social media platform", domain)
+    return variant
+
+
+def generate_task_variants(base_task: Dict, n: int = 5) -> Iterator[Dict]:
+    domain_list = list(DOMAIN_VARIANTS.values())
+    all_domains = [d for sublist in domain_list for d in sublist]
+    for i in range(min(n, len(all_domains))):
+        domain = all_domains[i]
+        yield generate_easy_variant(base_task, domain)

server/tasks/__init__.py

@@ -0,0 +1,10 @@
+# server/tasks/__init__.py
+from .task_easy import EASY_TASK
+from .task_medium import MEDIUM_TASK
+from .task_hard import HARD_TASK
+
+TASK_REGISTRY = {
+    "task_easy": EASY_TASK,
+    "task_medium": MEDIUM_TASK,
+    "task_hard": HARD_TASK,
+}

server/tasks/task_easy.py

@@ -0,0 +1,29 @@
+# server/tasks/task_easy.py
+EASY_TASK = {
+    "task_id": "task_easy",
+    "difficulty": "easy",
+    "description": (
+        "A social media platform has vague community guidelines. "
+        "Identify ONE genuinely ambiguous term and propose a specific, "
+        "measurable definition that moderators can apply consistently."
+    ),
+    "data_corpus": [
+        {"id": "post_001", "text": "You're so ugly, go back to where you came from", "action_taken": "removed"},
+        {"id": "post_002", "text": "I hate Mondays so much", "action_taken": "allowed"},
+        {"id": "post_003", "text": "This politician is a clown and should resign", "action_taken": "allowed"},
+        {"id": "post_004", "text": "Women shouldn't be in leadership roles", "action_taken": "removed"},
+        {"id": "post_005", "text": "I can't stand my neighbor's dog barking", "action_taken": "allowed"},
+        {"id": "post_006", "text": "People like you ruin everything", "action_taken": "flagged"},
+        {"id": "post_007", "text": "Go to hell with your stupid opinions", "action_taken": "flagged"},
+        {"id": "post_008", "text": "This food is disgusting, whoever made it should be ashamed", "action_taken": "allowed"},
+    ],
+    "current_policies": [
+        {"id": "pol_001", "text": "Users must not post content that is offensive or inappropriate."},
+        {"id": "pol_002", "text": "Harassment of any kind is strictly prohibited."},
+        {"id": "pol_003", "text": "Content that promotes hate speech will be removed."},
+        {"id": "pol_004", "text": "Users should communicate in a respectful manner."},
+    ],
+    "known_ambiguous_terms": ["offensive", "inappropriate", "harassment", "hate speech", "respectful"],
+    "num_policies": 4,
+    "num_data_points": 8,
+}

server/tasks/task_hard.py

@@ -0,0 +1,50 @@
+# server/tasks/task_hard.py
+HARD_TASK = {
+    "task_id": "task_hard",
+    "difficulty": "hard",
+    "description": (
+        "An e-commerce platform's Trust & Safety policy framework is underperforming. "
+        "Historical simulation data shows high false positive rates (legitimate sellers "
+        "being flagged) and missed fraud cases. Evolve the policy framework to improve "
+        "both precision and recall while maintaining seller trust. You must propose "
+        "modifications to at least 2 existing policies and justify trade-offs."
+    ),
+    "data_corpus": [
+        {"id": "seller_001", "type": "legitimate", "flags": ["new_account", "high_volume"], "outcome": "wrongly_suspended"},
+        {"id": "seller_002", "type": "fraudulent", "flags": ["price_manipulation"], "outcome": "missed"},
+        {"id": "seller_003", "type": "legitimate", "flags": ["foreign_bank"], "outcome": "wrongly_suspended"},
+        {"id": "seller_004", "type": "fraudulent", "flags": ["fake_reviews", "new_account"], "outcome": "correctly_caught"},
+        {"id": "seller_005", "type": "legitimate", "flags": ["high_returns"], "outcome": "wrongly_suspended"},
+        {"id": "seller_006", "type": "fraudulent", "flags": ["stolen_card_payments"], "outcome": "missed"},
+        {"id": "seller_007", "type": "fraudulent", "flags": ["counterfeit_goods"], "outcome": "missed"},
+        {"id": "seller_008", "type": "legitimate", "flags": ["seasonal_spike"], "outcome": "wrongly_suspended"},
+    ],
+    "current_policies": [
+        {"id": "ts_pol_001", "text": "Any new seller account with more than 50 transactions in the first week will be suspended for review."},
+        {"id": "ts_pol_002", "text": "Sellers with a return rate above 15% will be flagged for investigation."},
+        {"id": "ts_pol_003", "text": "Sellers using non-domestic bank accounts will require manual approval."},
+        {"id": "ts_pol_004", "text": "Any account with 3 or more fraud reports in 30 days will be permanently banned."},
+        {"id": "ts_pol_005", "text": "Price changes of more than 20% within 24 hours will trigger an automatic hold."},
+        {"id": "ts_pol_006", "text": "Sellers receiving 5+ negative reviews in 7 days will be suspended pending review."},
+    ],
+    "policy_outcomes": [
+        {"metric": "false_positive_rate", "value": 0.42, "target": 0.10},
+        {"metric": "fraud_detection_rate", "value": 0.31, "target": 0.85},
+        {"metric": "seller_trust_score", "value": 0.54, "target": 0.80},
+        {"metric": "review_queue_overload", "value": 0.89, "target": 0.30},
+        {"metric": "legitimate_revenue_lost", "value": 0.28, "target": 0.05},
+    ],
+    "system_metrics": {
+        "false_positive_rate": 0.42,
+        "fraud_detection_rate": 0.31,
+        "seller_trust_score": 0.54,
+        "review_queue_overload": 0.89,
+    },
+    "identified_issues": [
+        {"issue": "Blanket new-account volume rule catches legitimate seasonal sellers"},
+        {"issue": "Return rate threshold doesn't distinguish category (electronics vs. fashion)"},
+        {"issue": "Manual approval bottleneck creates 14-day delays for legitimate foreign sellers"},
+    ],
+    "num_policies": 6,
+    "num_data_points": 8,
+}

server/tasks/task_medium.py

@@ -0,0 +1,32 @@
+# server/tasks/task_medium.py
+MEDIUM_TASK = {
+    "task_id": "task_medium",
+    "difficulty": "medium",
+    "description": (
+        "A corporate HR policy set is missing rules for several emerging workplace "
+        "scenarios involving AI tools, remote work, and gig workers. "
+        "Identify ONE genuine policy gap and propose a specific new rule to address it."
+    ),
+    "data_corpus": [
+        {"id": "incident_001", "type": "AI_use", "desc": "Employee used ChatGPT to write client proposal without disclosure"},
+        {"id": "incident_002", "type": "remote_work", "desc": "Employee attended video call from a coffee shop, client data visible on screen"},
+        {"id": "incident_003", "type": "gig_worker", "desc": "Contractor accessed proprietary codebase after project ended"},
+        {"id": "incident_004", "type": "AI_use", "desc": "Manager used AI to generate performance review for employee"},
+        {"id": "incident_005", "type": "remote_work", "desc": "Employee shared screen showing salary data while on public WiFi"},
+        {"id": "incident_006", "type": "gig_worker", "desc": "Freelancer posted client project on portfolio without permission"},
+        {"id": "incident_007", "type": "AI_use", "desc": "Employee submitted AI-written code as their own in performance evaluation"},
+        {"id": "incident_008", "type": "remote_work", "desc": "Employee worked from another country for 3 months without HR approval"},
+        {"id": "incident_009", "type": "gig_worker", "desc": "Contractor attended team standup but was also working for a direct competitor"},
+        {"id": "incident_010", "type": "AI_use", "desc": "HR used AI tool to screen resumes — potential bias concerns raised"},
+    ],
+    "current_policies": [
+        {"id": "pol_hr_001", "text": "Employees must maintain confidentiality of client information at all times."},
+        {"id": "pol_hr_002", "text": "All employees are expected to comply with the company code of conduct."},
+        {"id": "pol_hr_003", "text": "Contractors must sign an NDA before beginning any project."},
+        {"id": "pol_hr_004", "text": "Employees working remotely must have a secure, dedicated workspace."},
+        {"id": "pol_hr_005", "text": "Any intellectual property created during employment belongs to the company."},
+    ],
+    "uncovered_domains": ["AI_use", "gig_worker_post_engagement", "cross_border_remote"],
+    "num_policies": 5,
+    "num_data_points": 10,
+}

test_result.json

@@ -0,0 +1,31 @@
+{
+  "baseline_scores": {
+    "task_easy": 0.0,
+    "task_medium": 0.0,
+    "task_hard": 0.0,
+    "overall_avg": 0.0
+  },
+  "mode": "llm",
+  "model": "meta-llama/Llama-3.3-70B-Instruct",
+  "runtime_seconds": 0.01,
+  "detail": [
+    {
+      "task_id": "task_easy",
+      "reward": 0.0,
+      "mode": "error",
+      "error": "All connection attempts failed"
+    },
+    {
+      "task_id": "task_medium",
+      "reward": 0.0,
+      "mode": "error",
+      "error": "All connection attempts failed"
+    },
+    {
+      "task_id": "task_hard",
+      "reward": 0.0,
+      "mode": "error",
+      "error": "All connection attempts failed"
+    }
+  ]
+}