final: add pyproject.toml, uv.lock, server/app.py for openenv validate

README.md

@@ -14,11 +14,15 @@ pinned: false
 
 # SQLSherlock-Env
 
-An RL environment where an AI agent acts as a data scientist investigating a dirty dataset.
+**An RL environment where an AI agent performs complete data quality audits on real-world datasets.**
 
-The agent discovers real data quality issues through statistical investigation — exactly like a human data scientist — fixes them with documented reasoning, validates fixes against the raw baseline, and exports the cleaned output in the same format as the input.
+Data cleaning consumes ~80% of a data scientist's time. This environment trains and evaluates AI agents to do it automatically: discover issues through statistical investigation, fix them with the right imputation strategy per column type, validate fixes, and export a clean dataset.
 
- Real datasets already have data quality problems. The issue detector scans the dataset at `reset()` time and builds a ground-truth catalogue from what it finds. The agent never sees this catalogue — it must discover everything through investigation.
+**Key design principles:**
+- **Real world data scanning** — the environment scans real datasets at `reset()` and builds a ground-truth issue catalogue from what it actually finds
+- **Any dataset** — pass any HuggingFace dataset URL, local CSV/JSON/Parquet file, or raw CSV text
+- **Production data cleaning** — handles nulls, type errors, constraint violations, outliers, duplicates, whitespace issues, and inconsistent categories
+- **Dense reward signal** — every action produces a training signal, not just end-of-episode binary feedback
 
 ---
 
@@ -28,238 +32,146 @@ The agent discovers real data quality issues through statistical investigation
 
 ```
 reset(dataset, task_id)
-        │
-        ▼
-┌───────────────────────────────────────────────────────────────────┐
-│  DatabaseEngine.__init__                                          │
-│                                                                   │
-│  1. load(source)         ← CSV / JSON / JSONL / Parquet / HF     │
-│  2. records_to_sqlite()  ← In-memory SQLite, isolated per episode│
-│  3. deep_copy(originals) ← Immutable snapshot before any edits   │
-│  4. profile_table()      ← mean/std/z-scores per column          │
-│  5. detect_issues()      ← null / type / constraint / outlier    │
-│                             duplicate / fk_violation             │
-│  6. Validator(baseline)  ← 6-check baseline captured             │
-│  7. detect_trap()        ← Task 3 only: plant 2x value in DB     │
-└───────────────────────────────────────────────────────────────────┘
-        │
-        ▼
- SQLSherlockObservation returned to agent
-        │
-        ▼
-┌─────────────────────────────────────────────────────┐
-│              Agent Step Loop                        │
-│                                                     │
-│  ┌──────────────────────────────────────────────┐  │
-│  │  Agent decides action (LLM call)             │  │
-│  │                                              │  │
-│  │  investigate:  inspect / profile / run_sql   │  │
-│  │  fix:          fix_cell / delete_row         │  │
-│  │  check:        validate                      │  │
-│  │  end:          submit / export               │  │
-│  └───────────────────┬──────────────────────────┘  │
-│                      │                             │
-│                      ▼                             │
-│  ┌──────────────────────────────────────────────┐  │
-│  │  Environment.step(action)                    │  │
-│  │                                              │  │
-│  │  1. dispatch action → DatabaseEngine        │  │
-│  │  2. reward.calc() → RB breakdown            │  │
-│  │  3. build observation (feedback + results)  │  │
-│  │  4. return (obs, reward, done, info)        │  │
-│  └──────────────────────────────────────────────┘  │
-│                                                     │
-│  Repeat until submit/export or budget exhausted     │
-└────────────────────────────────────────────��────────┘
-        │
-        ▼
-  Grader.score() → final score [0.0 – 1.0]
-```
-
-### Component Diagram
-
-```
-inference.py / train.py / custom agent
-        │  HTTP + WebSocket
-        ▼
-┌─────────────────────────────────────────────────────────────┐
-│  FastAPI App  (server/app.py)                               │
-│  POST /reset   POST /step   GET /state   GET /health        │
-│  WS /ws                                                     │
-└──────────────────────┬──────────────────────────────────────┘
-                       │
-                       ▼
-┌─────────────────────────────────────────────────────────────┐
-│  SQLSherlockEnvironment  (server/environment.py)            │
-│                                                             │
-│  reset()  ─────────────────────────────────────────────►   │
-│                                              DatabaseEngine │
-│  step(action)  ─────►  dispatch  ──────────────────────►   │
-│                              │                             │
-│                              │                             │
-│                         ┌────▼────┐                        │
-│                         │ reward  │                        │
-│                         │  .calc()│                        │
-│                         └─────────┘                        │
-│                                                             │
-│  on submit/export  ─────►  Grader.score()                  │
-└─────────────────────────────────────────────────────────────┘
-                       │
-        ┌──────────────┼──────────────────────┐
-        ▼              ▼                      ▼
-┌─────────────┐ ┌─────────────────┐ ┌──────────────────┐
-│  Database   │ │  IssueDetector  │ │    Validator      │
-│  Engine     │ │                 │ │                   │
-│             │ │  detect_issues()│ │  6-check before/  │
-│  SQLite     │ │  detect_trap()  │ │  after comparison │
-│  in-memory  │ │                 │ │                   │
-│  per episode│ │  null           │ │  null_check       │
-│             │ │  type_error     │ │  type_check       │
-│  profile_   │ │  constraint     │ │  range_check      │
-│  table()    │ │  outlier        │ │  distribution_    │
-│             │ │  duplicate      │ │    check          │
-│  z_scores   │ │  fk_violation   │ │  duplicate_check  │
-│  per row    │ │                 │ │  outlier_check    │
-└─────────────┘ └─────────────────┘ └──────────────────┘
+        |
+        v
++---------------------------------------------------------------+
+|  DatabaseEngine                                                |
+|                                                                |
+|  1. load(source)     <-- CSV / JSON / JSONL / Parquet / HF    |
+|  2. records_to_sqlite()  <-- In-memory SQLite per episode      |
+|  3. deep_copy(originals) <-- Immutable snapshot before edits   |
+|  4. profile_table()      <-- median/mode/std/z-scores          |
+|  5. detect_issues()      <-- null/type/constraint/outlier/     |
+|                              duplicate/whitespace/inconsistent  |
+|  6. Validator(baseline)  <-- 6-check baseline captured         |
+|  7. detect_trap()        <-- Task 3 only: plant 2x value       |
++---------------------------------------------------------------+
+        |
+        v
+  Observation returned to agent
+        |
+        v
++-----------------------------------------------------+
+|  Agent Step Loop                                     |
+|                                                      |
+|  investigate: inspect / profile_column / run_sql     |
+|  bulk fix:    fix_column (all nulls in one step)     |
+|  single fix:  fix_cell / delete_row                  |
+|  check:       validate                               |
+|  end:         submit / export                        |
+|                                                      |
+|  Each step -> reward signal -> observation            |
+|  Repeat until submit/export or max_steps reached     |
++-----------------------------------------------------+
+        |
+        v
+  Grader.score() -> final score [0.0 - 1.0]
 ```
 
 ### Grading Pipeline (7 steps)
 
 ```
-submit / export triggered
-         │
-         ▼
-┌─────────────────────────────────────────────────────────────┐
-│  universal.py — 7-step grader                               │
-│                                                             │
-│  Step 1: Zero-change guard                                  │
-│          └── if nothing changed → score = 0.0              │
-│                                                             │
-│  Step 2: Resolution score  (0.0 – 1.0)                     │
-│          └── per issue: confidence-weighted correct/total   │
-│              null: confidence 0.20 – 1.0 (structural=0.20)  │
-│              type_error: always 1.0                         │
-│              constraint / outlier: 0.80                     │
-│              duplicate: 0.70                                │
-│                                                             │
-│  Step 3: False-positive penalty                             │
-│          └── −0.15 per clean cell touched                   │
-│                                                             │
-│  Step 4: Trap penalty (Task 3 only)                         │
-│          └── −0.40 if trap cell was modified                │
-│                                                             │
-│  Step 5: Validation score  (0.0 – 0.30)                    │
-│          └── checks_passed / total_checks × 0.30           │
-│                                                             │
-│  Step 6: Reasoning bonus  (0.0 – 0.10)                     │
-│          └── +0.02 per fix_cell/delete_row with reason str  │
-│                                                             │
-│  Step 7: Final clamp                                        │
-│          raw = res×0.60 + val×0.30 + bonus×0.10 − fp − trap│
-│          score = clamp(raw, 0.0, 1.0)                       │
-└─────────────────────────────────────────────────────────────┘
+1. Zero-change guard    -- if nothing changed -> 0.0
+2. Resolution score     -- per issue: confidence-weighted
+3. False-positive penalty -- -0.05 per clean cell touched
+4. Trap penalty (task3) -- -0.40 if trap cell modified
+5. Validation score     -- checks_passed / total * 0.30
+6. Reasoning bonus      -- +0.05 for statistical reasoning
+7. Final: res*0.60 + val*0.30 + bonus*0.10 - fp - trap
 ```
 
 ---
 
 ## Quick Start
 
-### 1. Docker (recommended)
+### Docker (recommended)
 
 ```bash
-# Build from repo root
 docker build -t sqlsherlock-env:latest .
-
-# Run
 docker run -p 7860:7860 sqlsherlock-env:latest
-
-# Verify
 curl http://localhost:7860/health
+# -> {"status":"healthy"}
 ```
 
-### 2. Local (without Docker)
+### Local Python
 
 ```bash
+python -m venv .venv && source .venv/bin/activate  # or .venv\Scripts\activate on Windows
+pip install -r sqlsherlock_env/server/requirements.txt
 cd sqlsherlock_env
-pip install -r server/requirements.txt
-uvicorn server.app:app --host 0.0.0.0 --port 7860
+PYTHONPATH=. uvicorn server.app:app --host 0.0.0.0 --port 7860
 ```
 
-### 3. Run baseline inference
+### Run Baseline Inference
 
 ```bash
 export API_BASE_URL="https://router.huggingface.co/v1"
 export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
-export HF_TOKEN="hf_..."
+export HF_TOKEN="hf_your_token"
 export SPACE_URL="http://localhost:7860"
-
 python inference.py
 ```
 
-Expected stdout (judges parse this exactly):
+Output format (judges parse this exactly):
 
 ```
 [START] task=task1_null_and_types env=sqlsherlock_env model=Qwen/Qwen2.5-72B-Instruct
 [STEP]  step=1 action=inspect reward=0.02 done=false error=null
-[STEP]  step=2 action=profile_column(age) reward=0.03 done=false error=null
+[STEP]  step=2 action=profile_column(Age) reward=0.03 done=false error=null
+[STEP]  step=7 action=fix_column reward=0.18 done=false error=null
 ...
-[END]   success=true steps=8 score=0.820 rewards=0.02,0.03,0.15,0.15,0.05,0.15,0.10
+[END]   success=true steps=20 score=0.652 rewards=0.02,0.03,...
 ```
 
 ---
 
-## Using Your Own Dataset
-
-`inference.py` uses `phihung/titanic` for hackathon validation. To use your own dataset, connect the client directly:
+## Using Any Dataset
 
-### HuggingFace dataset
+The environment works with **any** dataset, not just Titanic:
 
 ```python
 from sqlsherlock_env.client import SQLSherlockEnv
 
 env = SQLSherlockEnv(base_url="http://localhost:7860")
-obs = env.reset(
-    dataset="your_org/your_dataset",         # any public HF dataset
-    task_id="task1_null_and_types",
-    max_rows=500,
-)
-```
 
-### Local file (CSV / JSON / JSONL / Parquet)
+# HuggingFace dataset
+obs = env.reset(dataset="scikit-learn/iris", task_id="task1_null_and_types")
 
-```python
-obs = env.reset(
-    dataset="/absolute/path/to/data.csv",
-    task_id="task2_constraints_and_fk",
-)
-```
+# Local file
+obs = env.reset(dataset="/path/to/data.csv", task_id="task2_constraints_and_fk")
 
-### Raw CSV string
-
-```python
-csv_text = "id,name,age,fare\n1,Alice,,25.0\n2,Bob,FORTY,50.0\n..."
-obs = env.reset(
-    dataset=csv_text,
-    task_id="task1_null_and_types",
-)
+# Raw CSV text
+obs = env.reset(dataset="id,name,age\n1,Alice,\n2,Bob,25\n", task_id="task1_null_and_types")
 ```
 
-### Upload via API
+Upload via API: `POST /upload_dataset` with a CSV/JSON/Parquet file.
 
-```bash
-curl -X POST http://localhost:7860/upload_dataset \
-  -F "file=@data.csv" \
-  -F "task_id=task1_null_and_types"
-```
+Download cleaned output: use `export` action, then `GET /download/{file_id}`. Output format matches input (CSV in -> CSV out, Parquet in -> Parquet out).
 
-**What the environment does with your dataset:**
-1. Loads the data (any format above)
-2. Auto-detects column types (int / float / str / bool)
-3. Scans for real data quality issues — no injection
-4. Builds a ground-truth issue catalogue the agent never sees
-5. Plants a trap value in Task 3
+---
 
-The agent then investigates, fixes, validates, and exports. The exported file matches the input format (CSV in → CSV out, Parquet in → Parquet out).
+## Data Cleaning Capabilities
+
+The agent detects and fixes these real-world data quality issues:
+
+| Issue Type | Detection Method | Fix Strategy | Tasks |
+|---|---|---|---|
+| **Null values** | `IS NULL` or empty string | Numeric: column **median**. String: column **mode** | All |
+| **Type errors** | Text in predominantly numeric column (>=80% castable) | Column median | All |
+| **Constraint violations** | Negative values in must-be-positive columns | `ABS(value)` | Task 2+ |
+| **FK violations** | Orphan references across tables | `delete_row` | Task 2+ |
+| **Whitespace** | Leading/trailing/extra spaces | Trimmed string | Task 2+ |
+| **Inconsistent categories** | Case variants ("male"/"Male"/"MALE") | Dominant form (mode) | Task 2+ |
+| **Statistical outliers** | IQR-based (robust to outlier-inflated std) | Column median | Task 3 |
+| **Duplicates** | Same natural key appearing twice | `delete_row` on later row | Task 3 |
+| **Trap** (task3 only) | Planted 2x value with z < 3 | **Do NOT touch** (-0.40 penalty) | Task 3 |
+
+**Smart imputation per column type:**
+- `profile_column` returns **median**, **mode**, **mean**, **null_rate**, **dtype**, **z_scores**
+- Numeric nulls -> median (robust to outliers)
+- String nulls -> mode (most frequent value)
+- Structural nulls (>70% null) -> "Unknown" (low confidence)
+- `fix_column` bulk-fixes ALL nulls + type errors + negatives in one step
 
 ---
 
@@ -267,82 +179,81 @@ The agent then investigates, fixes, validates, and exports. The exported file ma
 
 | `action_type` | Required fields | Description |
 |---|---|---|
-| `inspect` | `table` | View all rows |
-| `profile_column` | `table`, `column` | Stats: mean/std/min/max/nulls/z-scores |
-| `run_sql` | `sql` | SELECT query (read-only, max 50 rows) |
+| `inspect` | `table` | View all rows in the table |
+| `profile_column` | `table`, `column` | Statistics: median, mode, mean, std, null_count, null_rate, z_scores, dtype |
+| `run_sql` | `sql` | Read-only SELECT query (max 50 rows) |
 | `fix_cell` | `table`, `row_id`, `column`, `value`, `reason` | Fix one cell with justification |
-| `fix_column` | `table`, `column`, `value`, `reason` | Fix ALL nulls in a column at once (bulk) |
-| `delete_row` | `table`, `row_id`, `reason` | Remove duplicate or FK row |
-| `validate` | — | Run all 6 before/after checks |
-| `submit` | — | Score and end episode |
-| `export` | — | Write cleaned file, score and end episode |
+| `fix_column` | `table`, `column`, `value`, `reason` | Fix ALL nulls + type errors + negatives in a column (bulk) |
+| `delete_row` | `table`, `row_id`, `reason` | Remove a duplicate or FK-violation row |
+| `validate` | -- | Run all 6 before/after validation checks |
+| `submit` | -- | Score and end episode |
+| `export` | -- | Write cleaned file, score, and end episode |
 
 ---
 
 ## Reward System
 
+Dense per-step rewards -- every action produces a training signal:
+
 | Action | Reward | Cap |
 |---|---|---|
 | `inspect` | +0.02 | 3 rewarded |
 | `profile_column` | +0.03 | 3 rewarded |
 | `run_sql` | +0.03 | 3 rewarded |
-| `validate` | +0.05 × (checks_passed / 6) | 2 rewarded |
-| `fix_cell` — correct | **+0.15** | — |
-| `fix_cell` — false positive | **−0.20** | — |
-| `fix_cell` — trap cell | **−0.40** | — |
-| `fix_cell` — wrong value | **−0.10** | — |
-| `delete_row` — valid | **+0.15** | — |
-| `delete_row` — false positive | **−0.20** | �� |
-| `submit` — all resolved | +0.10 | — |
-| `submit` — issues remain | −0.10 | — |
+| `validate` | +0.05 * (checks_passed / 6) | 2 rewarded |
+| `fix_cell` -- correct | **+0.15** | -- |
+| `fix_cell` -- false positive | **-0.20** | -- |
+| `fix_cell` -- trap cell | **-0.40** | -- |
+| `fix_column` -- has issues | **+0.15 to +0.30** (proportional to issues resolved) | -- |
+| `fix_column` -- no issues | **-0.10** | -- |
+| `delete_row` -- valid | **+0.15** | -- |
+| `delete_row` -- false positive | **-0.20** | -- |
+| `submit` -- all resolved | **+0.10** | -- |
+| `submit` -- issues remain | **-0.10** | -- |
 
 ---
 
 ## Three Tasks
 
-### Task 1 — `task1_null_and_types` (Easy, max 20 steps)
+### Task 1 -- `task1_null_and_types` (Easy, max 30 steps)
 
-Find and fix **null values** and **type errors**.
+Find and fix **null values** and **type errors** across all columns.
 
-- Null: `None` or empty string in any non-PK column
-- Type error: text in a numeric column (e.g. `"FORTY"` in age)
-- Score: `resolution × 0.70 + validation × 0.30`
+- Detects: nulls, empty strings, text in numeric columns
+- Scoring: `resolution * 0.70 + validation * 0.30`
+- Grader weights each issue by confidence (structural nulls = low confidence)
 
-### Task 2 — `task2_constraints_and_fk` (Medium, max 25 steps)
+### Task 2 -- `task2_constraints_and_fk` (Medium, max 40 steps)
 
-Everything in Task 1 plus:
+Everything in Task 1, plus:
 
-- **Constraint violations**: negative values in must-be-positive columns (age, fare, price)
+- **Constraint violations**: negative values in must-be-positive columns
 - **FK violations**: orphan references in related tables
+- **Whitespace issues**: leading/trailing spaces
+- **Inconsistent categories**: case variants normalized to dominant form
 
-### Task 3 — `task3_full_audit_with_trap` (Hard, max 30 steps)
-
-Full audit including:
+### Task 3 -- `task3_full_audit_with_trap` (Hard, max 50 steps)
 
-- **Statistical outliers**: z-score > 5 in any numeric column
-- **Duplicates**: natural key appearing more than once
+Full statistical audit including:
 
-**THE TRAP**: One numeric value is set to 2x original — looks suspicious but has `z < 3`. Touching it costs **−0.40**.
+- **Outliers**: IQR-based detection (z > 5)
+- **Duplicates**: natural key collision
+- **THE TRAP**: One numeric value set to 2x original. Looks suspicious but has z < 3. Touching it costs **-0.40**.
 
-> Rule: Always `profile_column` before fixing any numeric value.
-> `z > 5` → real outlier → fix it. `z < 3` → legitimate → leave it.
+Rule: Always `profile_column` before fixing numeric values. z > 5 = outlier (fix). z < 3 = normal (leave alone).
 
 ---
 
 ## Validation (6 Checks)
 
-Run with `validate` action. Compares current state against the baseline from `reset()`:
-
 | Check | Passes when |
 |---|---|
-| `null_check` | High-confidence nulls resolved |
-| `type_check` | All type errors castable to float |
+| `null_check` | Nulls resolved (weighted by confidence) |
+| `type_check` | Type errors castable to correct type |
 | `range_check` | No negatives in must-be-positive columns |
 | `distribution_check` | Column mean drift < 20% |
 | `duplicate_check` | Duplicate count reduced |
-| `outlier_check` | No previously-flagged rows still exceed z > 5 |
-
-Returns `PASS` / `PARTIAL` / `FAIL` with per-check detail and drift warnings.
+| `outlier_check` | Flagged outlier rows no longer exceed z > 5 |
 
 ---
 
@@ -354,148 +265,72 @@ Returns `PASS` / `PARTIAL` / `FAIL` with per-check detail and drift warnings.
 | `POST` | `/reset` | Reset environment, load dataset |
 | `POST` | `/step` | Execute one action |
 | `GET` | `/state` | Current episode state |
-| `GET` | `/health` | Health check (`{"status":"ok"}`) |
+| `GET` | `/health` | Health check |
 | `GET` | `/tasks` | List all 3 tasks |
-| `POST` | `/upload_dataset` | Upload file, get session |
+| `POST` | `/upload_dataset` | Upload CSV/JSON/Parquet file |
 | `GET` | `/download/{file_id}` | Download cleaned output |
-| `GET` | `/docs` | OpenAPI docs (Swagger UI) |
+| `GET` | `/docs` | OpenAPI documentation (Swagger UI) |
 
 ---
 
-## Testing
-
-### Run all tests
-
-```bash
-cd SQLSherlock-env
-pip install pytest
-pytest tests/ -v
-```
-
-### Test checklist
-
-```
-tests/test_issue_detector.py    ← null / type_error / constraint / outlier / duplicate
-tests/test_graders.py           ← task1 / task2 / task3 scoring, trap penalty, FP penalty
-tests/test_environment.py       ← reset → step → submit full episode
-```
-
-Expected: all tests pass. If any fail, check [tests/conftest.py](tests/conftest.py) — the `DIRTY_RECORDS` fixture must cover all issue types.
-
-### Manual smoke test
-
-```bash
-# 1. Start server
-docker run -p 7860:7860 sqlsherlock-env:latest
-
-# 2. Health check
-curl http://localhost:7860/health
-# → {"status":"ok"}
-
-# 3. List tasks
-curl http://localhost:7860/tasks
-# → [{id: task1_null_and_types, ...}, ...]
+## Environment Variables
 
-# 4. Run inference (requires HF_TOKEN for model access)
-export HF_TOKEN="hf_..."
-python inference.py 2>results.txt
-# → check stdout for [START]/[STEP]/[END] lines
-# → check stderr (results.txt) for score summary
-```
+| Variable | Default | Description |
+|---|---|---|
+| `API_BASE_URL` | `https://router.huggingface.co/v1` | LLM API endpoint |
+| `MODEL_NAME` | `Qwen/Qwen2.5-72B-Instruct` | Model identifier |
+| `HF_TOKEN` | -- (required) | HuggingFace token for LLM access |
+| `SPACE_URL` | `http://localhost:7860` | Environment server URL |
 
 ---
 
-## Submission Checklist
-
-```
-[ ] docker build -t sqlsherlock-env:latest .        ← must succeed from repo root
-[ ] docker run -p 7860:7860 sqlsherlock-env:latest  ← must start, port 7860
-[ ] curl http://localhost:7860/health                ← must return {"status":"ok"}
-[ ] python inference.py                             ← must emit [START]/[STEP]/[END]
-[ ] openenv validate                                 ← must pass (openenv.yaml at root)
-[ ] Dockerfile is at repo root (not inside subdir)  ← validate-submission.sh checks this
-[ ] openenv.yaml is at repo root                    ← openenv validate checks this
-[ ] No hardcoded secrets in any file                ← use env vars only
-[ ] All env vars documented (API_BASE_URL, MODEL_NAME, HF_TOKEN, SPACE_URL)
-[ ] pytest tests/ -v                               ← all tests pass
-```
-
----
 
 ## Setup on a New Device
 
-### Option A: Docker (recommended for deployment)
+### Option A: Docker
 
 ```bash
-# 1. Clone
-git clone <your-repo-url>
-cd SQLSherlock-env
-
-# 2. Build and run
+git clone <repo-url> && cd SQLSherlock-env
 docker build -t sqlsherlock-env:latest .
 docker run -p 7860:7860 sqlsherlock-env:latest
-
-# 3. Verify (in another terminal)
-curl http://localhost:7860/health
-# → {"status":"healthy"}
-
-# 4. Run inference
-export HF_TOKEN="hf_your_token_here"
+# In another terminal:
+export HF_TOKEN="hf_your_token"
 export SPACE_URL="http://localhost:7860"
 python inference.py
 ```
 
-### Option B: Local Python (for development)
+### Option B: Local Python (Linux/Mac)
 
 ```bash
-# 1. Clone
-git clone <your-repo-url>
-cd SQLSherlock-env
-
-# 2. Create virtual environment (Python 3.11+ required)
-python -m venv .venv
-
-# 3. Activate venv
-# Linux/Mac:
-source .venv/bin/activate
-# Windows PowerShell:
-.venv\Scripts\Activate.ps1
-# Windows CMD:
-.venv\Scripts\activate.bat
-
-# 4. Install dependencies
+git clone <repo-url> && cd SQLSherlock-env
+python3 -m venv .venv && source .venv/bin/activate
 pip install -r sqlsherlock_env/server/requirements.txt
-pip install pytest   # for tests
+# Terminal 1:
+cd sqlsherlock_env && PYTHONPATH=. uvicorn server.app:app --host 0.0.0.0 --port 7860
+# Terminal 2:
+export HF_TOKEN="hf_your_token" SPACE_URL="http://localhost:7860"
+python inference.py
+```
 
-# 5. Start the server (Terminal 1)
-cd sqlsherlock_env
-# Linux/Mac:
-PYTHONPATH=. uvicorn server.app:app --host 0.0.0.0 --port 7860
-# Windows PowerShell:
-$env:PYTHONPATH = (Get-Location).Path
-python -m uvicorn server.app:app --host 0.0.0.0 --port 7860
+### Option C: Local Python (Windows PowerShell)
 
-# 6. Run inference (Terminal 2)
-cd SQLSherlock-env
-# Linux/Mac:
-export HF_TOKEN="hf_your_token_here"
-export SPACE_URL="http://localhost:7860"
-python inference.py
-# Windows PowerShell:
-$env:HF_TOKEN = "hf_your_token_here"
-$env:SPACE_URL = "http://localhost:7860"
+```powershell
+git clone <repo-url>; cd SQLSherlock-env
+python -m venv .venv; .venv\Scripts\Activate.ps1
+pip install -r sqlsherlock_env\server\requirements.txt
+# Terminal 1:
+cd sqlsherlock_env; $env:PYTHONPATH=(Get-Location).Path
+python -m uvicorn server.app:app --host 0.0.0.0 --port 7860
+# Terminal 2:
+$env:HF_TOKEN="hf_your_token"; $env:SPACE_URL="http://localhost:7860"
 python inference.py
-
-# 7. Run tests (server not needed for tests)
-cd SQLSherlock-env
-# Linux/Mac:
-PYTHONPATH=sqlsherlock_env pytest tests/ -v
-# Windows PowerShell:
-$env:PYTHONPATH = "sqlsherlock_env"
-python -m pytest tests/ -v
 ```
 
-**Python version**: 3.11+ required. Dependencies: `fastapi`, `uvicorn`, `openai`, `datasets`, `pandas`, `pyarrow`.
+### Run Tests
+
+```bash
+PYTHONPATH=sqlsherlock_env pytest tests/ -v   # 98 tests, all pass
+```
 
 ---
 
@@ -503,7 +338,6 @@ python -m pytest tests/ -v
 
 ```bash
 pip install trl transformers torch
-
 export SPACE_URL="http://localhost:7860"
 export MODEL_ID="Qwen/Qwen2.5-1.5B-Instruct"
 python train.py
@@ -511,59 +345,40 @@ python train.py
 
 ---
 
-## Environment Variables
-
-| Variable | Default | Description |
-|---|---|---|
-| `API_BASE_URL` | `https://router.huggingface.co/v1` | LLM endpoint |
-| `MODEL_NAME` | `Qwen/Qwen2.5-72B-Instruct` | Model ID |
-| `HF_TOKEN` | — | HuggingFace token (dataset access + LLM) |
-| `SPACE_URL` | `http://localhost:7860` | Environment server URL |
-
----
-
-## Baseline Scores (phihung/titanic, 150 rows)
-
-| Task | Difficulty | Expected Score |
-|---|---|---|
-| `task1_null_and_types` | Easy | 0.70 – 0.88 |
-| `task2_constraints_and_fk` | Medium | 0.55 – 0.76 |
-| `task3_full_audit_with_trap` | Hard | 0.40 – 0.65 |
-
----
-
 ## Project Structure
 
 ```
 SQLSherlock-env/
-├── Dockerfile                  ← repo root (required for HF Spaces)
-├── README.md                   ← this file
-├── openenv.yaml                ← OpenEnv + HF Spaces manifest (repo root)
-├── inference.py                ← baseline agent ([START]/[STEP]/[END] format)
-├── train.py                    ← TRL GRPO training loop
-├── sqlsherlock_env/
-│   ├── __init__.py
-│   ├── client.py               ← SQLSherlockEnv WebSocket/HTTP client
-│   ├── models.py               ← Action / Observation / State (Pydantic)
-│   └── server/
-│       ├── app.py              ← FastAPI application + WebSocket handler
-│       ├── environment.py      ← RL core: reset() / step() / get_state()
-│       ├── database.py         ← In-memory SQLite engine, per-episode
-│       ├── dataset_loader.py   ← CSV / JSON / JSONL / Parquet / HF loader
-│       ├── schema_profiler.py  ← Column statistics + z-scores
-│       ├── issue_detector.py   ← Real issue detection + trap planting
-│       ├── validator.py        ← 6-check before/after validator
-│       ├── reward.py           ← Dense per-step reward with InvestCounter
-│       ├── exporter.py         ← Format-fidelity output (CSV→CSV, etc.)
-│       ├── requirements.txt
-│       └── graders/
-│           ├── universal.py    ← 7-step scoring pipeline
-│           ├── task1.py        ← Task 1 grader
-│           ├── task2.py        ← Task 2 grader
-│           └── task3.py        ← Task 3 grader (trap-aware)
-└── tests/
-    ├── conftest.py             ← DIRTY_RECORDS fixture (all issue types)
-    ├── test_issue_detector.py
-    ├── test_graders.py
-    └── test_environment.py
++-- Dockerfile                  <- repo root (HF Spaces)
++-- README.md                   <- this file
++-- openenv.yaml                <- OpenEnv manifest (3 tasks)
++-- inference.py                <- baseline inference ([START]/[STEP]/[END])
++-- train.py                    <- TRL GRPO training scaffold
++-- .gitignore
++-- .dockerignore
++-- sqlsherlock_env/
+|   +-- __init__.py
+|   +-- client.py               <- synchronous WebSocket/HTTP client
+|   +-- models.py               <- Action / Observation / State (Pydantic)
+|   +-- server/
+|       +-- app.py              <- FastAPI + WebSocket handler
+|       +-- environment.py      <- RL core: reset() / step() / state
+|       +-- database.py         <- In-memory SQLite engine per episode
+|       +-- dataset_loader.py   <- CSV / JSON / JSONL / Parquet / HF
+|       +-- schema_profiler.py  <- Column statistics + z-scores
+|       +-- issue_detector.py   <- 8 issue types + trap planting
+|       +-- validator.py        <- 6-check before/after validator
+|       +-- reward.py           <- Dense per-step reward
+|       +-- exporter.py         <- Format-preserving output writer
+|       +-- requirements.txt
+|       +-- graders/
+|           +-- universal.py    <- 7-step scoring pipeline
+|           +-- task1.py
+|           +-- task2.py
+|           +-- task3.py
++-- tests/                      <- 98 tests
+    +-- conftest.py
+    +-- test_issue_detector.py
+    +-- test_graders.py
+    +-- test_environment.py
 ```