Add recovered GPU predictions, native SQL validation and complete runtime/training provenance

README.md

@@ -63,11 +63,11 @@ Weights were frozen at step 25,715 on 2026-09-10 at 14:00:36 UTC, before reading
 | **Full task success** | **1,106/1,196 (92.47%)** | **132/160 (82.50%)** |
 | TF-IDF retrieval + context-binding baseline, full success | 881/1,196 (73.66%) | 115/160 (71.88%) |
 
-These primary numbers are the completed CUDA BF16 evaluation captured in `evaluation/gpu-final-console-metrics.json`. The rented server subsequently refused SSH connections, so its raw final prediction files were not retrieved. Independent FP32 Mac evaluation reproduced every aggregate metric exactly; complete raw predictions, language/backend breakdowns and summaries are included in `evaluation/test-mac.*` and `evaluation/manual-mac.*`. No retraining or selection follows the test results.
+These primary numbers are the completed CUDA BF16 evaluation. Complete raw GPU predictions and summaries are in `evaluation/test-gpu.*` and `evaluation/manual-gpu.*`. Independent FP32 Mac evaluation reproduced every aggregate metric exactly; complete raw predictions, language/backend breakdowns and summaries are included in `evaluation/test-mac.*` and `evaluation/manual-mac.*`. No retraining or selection follows the test results.
 
 Evaluation uses raw greedy generation with no repair, constrained decoding, retrieval fallback or teacher fallback. Full SQL-task success requires the correct tool, exact non-SQL arguments (including project scope), compilation against the supplied schema, and matching results on two generated SQLite fixtures after dialect adaptation. SQL equivalence alone does not require correct project scope, so its percentage can exceed full success. Non-SQL tool calls require exact arguments; clarification/answer text requires exact reference wording.
 
-**These are synthetic benchmark results, not measured production task success.** Native MySQL 8.0.46/PostgreSQL 16.15 checks passed all 66 reference-operation/dialect cases; these validate reference data, not the final student's test predictions. Final native student evaluation was unavailable after SSH access failed. SQL fixture equivalence is not a proof for all possible databases.
+**These are synthetic benchmark results, not measured production task success.** Native MySQL 8.0.46/PostgreSQL 16.15 checks passed all 66 reference-operation/dialect cases. Final student checks passed 859/912 SQL tasks and 91/112 manual SQL tasks, requiring correct tool and project scope as well as result equivalence. Combined with non-SQL results, these reproduce exactly the full-task totals of 1,106/1,196 and 132/160. Native reports are included in `evaluation/native-*.json`. SQL fixture equivalence is not a proof for all possible databases.
 
 Validation scored 1,174/1,200 (97.83%); development scored 188/192 (97.92%). These were used for model selection and are not final held-out claims. The 160 manual cases use handwritten phrasing templates on 40 test families, so manual and test are not independent schema-family samples.
 
@@ -106,6 +106,6 @@ Training has no scenario-group overlap with validation, development, test or man
 - Arbitrary unseen databases, tools, complex joins, long conversations and broad Hindi translation are not established capabilities. Hindi city/table lexical mappings cover a small explicit training vocabulary.
 - A correct JSON object can contain a wrong tool, literal, project ID or query. The model is not 100% reliable.
 - It emits one action, not a complete autonomous MCP agent. Database execution and authorization belong to the host application.
-- The runtime source and frozen data support new experiments; the release is not a bitwise replay of the evolving training session. The final optimizer state and some server logs were not recovered after SSH access failed.
+- The runtime source and frozen data support new experiments; the release is not a bitwise replay of the evolving training session. Runtime configs, training logs and evaluation records were recovered after a brief SSH outage. The optional final optimizer-state download was stopped to avoid extending rental cost; inference weights are complete.
 
 Weights and project code are released under Apache-2.0. Teacher provenance is documented separately. See `manifest.json` for file hashes and `checkpoint-info.json` for selected-weight counters.

docs/experiment.md

@@ -39,7 +39,7 @@ python -m tinyquery.train --data data/tinyquery --out runs/tinyquery --copy-dim
 
 ## Stream output
 
-Once the finished export is placed in `runs/tinyquery/`:
+The finished export is available in `runs/tinyquery/`:
 
 ```sh
 .venv/bin/python -m tinyquery.chat "दिल्ली के ग्राहकों के नाम दिखाओ।" \
@@ -56,4 +56,10 @@ python -m tinyquery.evaluate --checkpoint runs/tinyquery/model.safetensors \
   --out runs/tinyquery/test-predictions.jsonl
 ```
 
-Evaluation uses raw greedy output with no JSON repair, constrained decoding or teacher fallback. Report JSON validity, tool-schema validity, tool choice, exact arguments and SQL result equivalence separately. The default SQL evaluator uses dialect adaptation to SQLite; `native_sql.py` performs separate checks on local MySQL/PostgreSQL engines. Final measured student results and failure examples will be written after training.
+Evaluation uses raw greedy output with no JSON repair, constrained decoding or teacher fallback. Report JSON validity, tool-schema validity, tool choice, exact arguments and SQL result equivalence separately. The default SQL evaluator uses dialect adaptation to SQLite; `native_sql.py` performs separate checks on local MySQL/PostgreSQL engines. The frozen 139,738,113-parameter model achieved 1,106/1,196 full test successes (92.47%) and 132/160 additional manual-phrasing successes (82.50%). Independent Mac FP32 evaluation exactly matched all aggregate CUDA BF16 metrics. The context-binding retrieval baseline achieved 73.66% and 71.88%, respectively. JSON validity was 100%; this is not 100% task success. Full raw Mac predictions and breakdowns are in `releases/tinyquery-model/evaluation/`.
+
+Published model: https://huggingface.co/karmx/TinyQuery-140M
+
+Published dataset: https://huggingface.co/datasets/karmx/TinyQuery-Tools-Multilingual
+
+Local runtime, template, dataset and checkpoint inventory: `artifacts/tinyquery/README.md`. Remote runtime configs, training logs and GPU predictions were recovered after a brief SSH outage. Native MySQL/PostgreSQL evaluation confirmed the same full-task results. The optional final optimizer-state transfer was stopped to avoid extending rental cost. The selected weights and complete datasets are local.

evaluation/development-gpu.summary.json

@@ -0,0 +1,1786 @@
+{
+  "examples": 192,
+  "seconds": 6.547292470932007,
+  "decoding": "raw greedy, no repair, no teacher fallback",
+  "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
+  "sql_metric": "Compile against the supplied schema, then compare results on two generated SQLite fixtures after dialect adaptation; not native MySQL/PostgreSQL execution.",
+  "metrics": {
+    "json_valid": {
+      "correct": 192,
+      "total": 192,
+      "rate": 1.0
+    },
+    "schema_valid": {
+      "correct": 192,
+      "total": 192,
+      "rate": 1.0
+    },
+    "action_correct": {
+      "correct": 192,
+      "total": 192,
+      "rate": 1.0
+    },
+    "tool_correct": {
+      "correct": 192,
+      "total": 192,
+      "rate": 1.0
+    },
+    "arguments_exact": {
+      "correct": 188,
+      "total": 192,
+      "rate": 0.9791666666666666
+    },
+    "sql_equivalent": {
+      "correct": 125,
+      "total": 128,
+      "rate": 0.9765625
+    },
+    "success": {
+      "correct": 188,
+      "total": 192,
+      "rate": 0.9791666666666666
+    }
+  },
+  "groups": {
+    "language:noisy_en": {
+      "json_valid": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 31,
+        "total": 31,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      }
+    },
+    "backend:supabase": {
+      "json_valid": {
+        "correct": 92,
+        "total": 92,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 92,
+        "total": 92,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 92,
+        "total": 92,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 92,
+        "total": 92,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 90,
+        "total": 92,
+        "rate": 0.9782608695652174
+      },
+      "sql_equivalent": {
+        "correct": 64,
+        "total": 66,
+        "rate": 0.9696969696969697
+      },
+      "success": {
+        "correct": 90,
+        "total": 92,
+        "rate": 0.9782608695652174
+      }
+    },
+    "operation:missing_schema": {
+      "json_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      }
+    },
+    "operation:project": {
+      "json_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 6,
+        "total": 7,
+        "rate": 0.8571428571428571
+      },
+      "sql_equivalent": {
+        "correct": 6,
+        "total": 7,
+        "rate": 0.8571428571428571
+      },
+      "success": {
+        "correct": 6,
+        "total": 7,
+        "rate": 0.8571428571428571
+      }
+    },
+    "language:hinglish": {
+      "json_valid": {
+        "correct": 52,
+        "total": 52,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 52,
+        "total": 52,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 52,
+        "total": 52,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 52,
+        "total": 52,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 51,
+        "total": 52,
+        "rate": 0.9807692307692307
+      },
+      "sql_equivalent": {
+        "correct": 36,
+        "total": 37,
+        "rate": 0.972972972972973
+      },
+      "success": {
+        "correct": 51,
+        "total": 52,
+        "rate": 0.9807692307692307
+      }
+    },
+    "operation:count_eq": {
+      "json_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      }
+    },
+    "language:en": {
+      "json_valid": {
+        "correct": 44,
+        "total": 44,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 44,
+        "total": 44,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 44,
+        "total": 44,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 44,
+        "total": 44,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 42,
+        "total": 44,
+        "rate": 0.9545454545454546
+      },
+      "sql_equivalent": {
+        "correct": 30,
+        "total": 31,
+        "rate": 0.967741935483871
+      },
+      "success": {
+        "correct": 42,
+        "total": 44,
+        "rate": 0.9545454545454546
+      }
+    },
+    "operation:group_count": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:distinct": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "operation:year": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "operation:max": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:bottom": {
+      "json_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      }
+    },
+    "operation:lt": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:avg": {
+      "json_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      }
+    },
+    "operation:sum": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 4,
+        "rate": 0.75
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 4,
+        "rate": 0.75
+      },
+      "success": {
+        "correct": 3,
+        "total": 4,
+        "rate": 0.75
+      }
+    },
+    "language:hi": {
+      "json_valid": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 48,
+        "total": 48,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 47,
+        "total": 48,
+        "rate": 0.9791666666666666
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 29,
+        "rate": 0.9655172413793104
+      },
+      "success": {
+        "correct": 47,
+        "total": 48,
+        "rate": 0.9791666666666666
+      }
+    },
+    "operation:project_eq": {
+      "json_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      }
+    },
+    "operation:not_null": {
+      "json_valid": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      }
+    },
+    "operation:or": {
+      "json_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      }
+    },
+    "operation:date_after": {
+      "json_valid": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 10,
+        "total": 10,
+        "rate": 1.0
+      }
+    },
+    "operation:top": {
+      "json_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      }
+    },
+    "operation:having": {
+      "json_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      }
+    },
+    "operation:min": {
+      "json_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 5,
+        "total": 5,
+        "rate": 1.0
+      }
+    },
+    "operation:gt": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "backend:mysql": {
+      "json_valid": {
+        "correct": 100,
+        "total": 100,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 100,
+        "total": 100,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 100,
+        "total": 100,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 100,
+        "total": 100,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 98,
+        "total": 100,
+        "rate": 0.98
+      },
+      "sql_equivalent": {
+        "correct": 61,
+        "total": 62,
+        "rate": 0.9838709677419355
+      },
+      "success": {
+        "correct": 98,
+        "total": 100,
+        "rate": 0.98
+      }
+    },
+    "operation:count": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:group_sum": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:between": {
+      "json_valid": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 6,
+        "total": 6,
+        "rate": 1.0
+      }
+    },
+    "operation:sort_asc": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "operation:sort_desc": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "operation:sql_error": {
+      "json_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 9,
+        "rate": 0.8888888888888888
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 9,
+        "rate": 0.8888888888888888
+      }
+    },
+    "operation:all": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 4,
+        "rate": 0.75
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 4,
+        "rate": 0.75
+      },
+      "success": {
+        "correct": 3,
+        "total": 4,
+        "rate": 0.75
+      }
+    },
+    "operation:and": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "operation:month": {
+      "json_valid": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      }
+    },
+    "operation:join_filter": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:eq": {
+      "json_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      }
+    },
+    "operation:list_tables": {
+      "json_valid": {
+        "correct": 12,
+        "total": 12,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 12,
+        "total": 12,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 12,
+        "total": 12,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 12,
+        "total": 12,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 12,
+        "total": 12,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 12,
+        "total": 12,
+        "rate": 1.0
+      }
+    },
+    "operation:gte": {
+      "json_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      }
+    },
+    "operation:lte": {
+      "json_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      }
+    },
+    "operation:describe": {
+      "json_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 4,
+        "total": 4,
+        "rate": 1.0
+      }
+    },
+    "operation:join": {
+      "json_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 3,
+        "total": 3,
+        "rate": 1.0
+      }
+    },
+    "operation:null": {
+      "json_valid": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 2,
+        "total": 2,
+        "rate": 1.0
+      }
+    },
+    "operation:contains": {
+      "json_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 1,
+        "total": 1,
+        "rate": 1.0
+      }
+    },
+    "operation:search": {
+      "json_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      }
+    },
+    "operation:ticket": {
+      "json_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 9,
+        "total": 9,
+        "rate": 1.0
+      }
+    },
+    "operation:weather": {
+      "json_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      }
+    },
+    "operation:read_file": {
+      "json_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 7,
+        "rate": 1.0
+      }
+    }
+  },
+  "rescoring": "Same saved raw outputs, with explicit supplied-schema compilation; no generation or output repair."
+}

evaluation/gpu-final-console-metrics.json

@@ -2,7 +2,7 @@
   "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
   "frozen_at_utc": "2026-09-10T14:00:36.513800+00:00",
   "device": "CUDA BF16",
-  "source": "Captured successful remote evaluation console output; raw remote prediction files were not retrieved before SSH began refusing connections. Independent Mac evaluation is packaged separately.",
+  "source": "Completed remote CUDA evaluation; raw GPU predictions were retrieved after SSH access recovered. Independent Mac FP32 evaluation matches all aggregate metrics.",
   "decoding": "raw greedy; no repair, constrained decoding or teacher fallback",
   "metrics": {
     "test": {

evaluation/manual-gpu.jsonl

@@ -0,0 +1,160 @@
+{"id": "manual_2ad3e251c936135e2005_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "all", "question": "all specimens pls", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens;"}}, "output": "{\"action\":\"clarify\",\"question\":\"This connection is read-only. Would you like a SELECT query instead?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": false, "tool_correct": false, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_2e2ac3c5dcf95429daaa_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "null", "question": "specimens title null records", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE title IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE title IS NULL;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_0b8db53d711baed4f4ed_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "gte", "question": "specimens cost minimum 10 include same", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE cost >= 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens ORDER BY cost ASC LIMIT 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_2ad3e251c936135e2005_en", "language": "en", "backend": "supabase", "operation": "all", "question": "I need everything in specimens.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_efee65268e6dc2bb069e_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "project", "question": "specimens only customer_name show", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT customer_name FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT customer_name FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_efee65268e6dc2bb069e_hinglish", "language": "hinglish", "backend": "supabase", "operation": "project", "question": "bas specimens ke customer_name dikha do", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT customer_name FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT customer_name FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_0b8db53d711baed4f4ed_hinglish", "language": "hinglish", "backend": "supabase", "operation": "gte", "question": "specimens me cost kam se kam 10 ho", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE cost >= 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE cost >= 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2e2ac3c5dcf95429daaa_hinglish", "language": "hinglish", "backend": "supabase", "operation": "null", "question": "specimens me jinka title NULL hai wo dikhao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE title IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE price > 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_2ad3e251c936135e2005_hinglish", "language": "hinglish", "backend": "supabase", "operation": "all", "question": "specimens ka sara data chahiye", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE display_name IS NULL;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_d735c1ac1979ff1e8620_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "sum", "question": "inspections price all add", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT SUM(price) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT SUM(price) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_0b8db53d711baed4f4ed_en", "language": "en", "backend": "supabase", "operation": "gte", "question": "Include specimens records at 10 or above in cost.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE cost >= 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE cost >= 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2e2ac3c5dcf95429daaa_en", "language": "en", "backend": "supabase", "operation": "null", "question": "Find specimens records with no title value (NULL).", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE title IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE city = 'ApecULL';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_7dd163238936d96d0bd4_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "distinct", "question": "reservations city unique only", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT DISTINCT city FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT DISTINCT city FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_efee65268e6dc2bb069e_en", "language": "en", "backend": "supabase", "operation": "project", "question": "Just the customer_name values from specimens, please.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT customer_name FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT customer_name FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_eb982cf071094adaca4f_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "project_eq", "question": "reservations category Delhi only name", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT name FROM reservations WHERE category = 'Delhi';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT name FROM reservations WHERE category = 'Delhi';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_d735c1ac1979ff1e8620_en", "language": "en", "backend": "supabase", "operation": "sum", "question": "Add up price across inspections.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT SUM(price) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT display_name FROM inspections WHERE kind = 'Adddd';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_d735c1ac1979ff1e8620_hinglish", "language": "hinglish", "backend": "supabase", "operation": "sum", "question": "inspections ke price ka jod batao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT SUM(price) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT SUM(price) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c4f25575abfd24e3cd63_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "lte", "question": "reservations cost max 1 equal also", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM reservations WHERE cost <= 1;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM reservations ORDER BY cost DESC LIMIT 1;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_7dd163238936d96d0bd4_hinglish", "language": "hinglish", "backend": "supabase", "operation": "distinct", "question": "reservations me city ki unique values kya hain", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT DISTINCT city FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT DISTINCT city FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_eb982cf071094adaca4f_hinglish", "language": "hinglish", "backend": "supabase", "operation": "project_eq", "question": "reservations me category Delhi walon ka name batao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT name FROM reservations WHERE category = 'Delhi';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT name FROM reservations WHERE category = 'Delhi';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_eb982cf071094adaca4f_en", "language": "en", "backend": "supabase", "operation": "project_eq", "question": "In reservations, give me name for category Delhi.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT name FROM reservations WHERE category = 'Delhi';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT name FROM reservations WHERE category = 'Delhi';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c4f25575abfd24e3cd63_hinglish", "language": "hinglish", "backend": "supabase", "operation": "lte", "question": "reservations me cost 1 ya usse kam ho", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM reservations WHERE cost <= 1;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM reservations WHERE cost <= 1;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_7dd163238936d96d0bd4_en", "language": "en", "backend": "supabase", "operation": "distinct", "question": "Which different city values occur in reservations?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT DISTINCT city FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT DISTINCT city FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_efee65268e6dc2bb069e_hi", "language": "hi", "backend": "supabase", "operation": "project", "question": "specimens से सिर्फ customer_name दिखाना।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT customer_name FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT customer_name FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_0b8db53d711baed4f4ed_hi", "language": "hi", "backend": "supabase", "operation": "gte", "question": "specimens में cost कम से कम 10 होना चाहिए।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE cost >= 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE cost >= 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_d735c1ac1979ff1e8620_hi", "language": "hi", "backend": "supabase", "operation": "sum", "question": "inspections के price का जोड़ बताओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT SUM(price) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT SUM(price) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2ad3e251c936135e2005_hi", "language": "hi", "backend": "supabase", "operation": "all", "question": "specimens का सारा डेटा चाहिए।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c4f25575abfd24e3cd63_hi", "language": "hi", "backend": "supabase", "operation": "lte", "question": "reservations में cost 1 या उससे कम हो।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM reservations WHERE cost <= 1;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM reservations WHERE cost <= 1;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c4f25575abfd24e3cd63_en", "language": "en", "backend": "supabase", "operation": "lte", "question": "From reservations, include cost values up to and including 1.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM reservations WHERE cost <= 1;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM reservations WHERE cost BETWEEN 1 AND 1;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_eb982cf071094adaca4f_hi", "language": "hi", "backend": "supabase", "operation": "project_eq", "question": "reservations में category Delhi हो तो उनका name बताओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT name FROM reservations WHERE category = 'Delhi';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT name FROM reservations WHERE category = 'Delhi';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_7dd163238936d96d0bd4_hi", "language": "hi", "backend": "supabase", "operation": "distinct", "question": "reservations में city के अलग-अलग मान कौन से हैं?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT DISTINCT city FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT DISTINCT city FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2e2ac3c5dcf95429daaa_hi", "language": "hi", "backend": "supabase", "operation": "null", "question": "specimens में जिनका title NULL है वे रिकॉर्ड दिखाओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM specimens WHERE title IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM specimens WHERE price > 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_5a8f2e207d0efe5b018c_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "list_tables", "question": "tables here list", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"project_id": "demo_specimens", "schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specmemens\",\"query\":\"SELECT * FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": false, "arguments_exact": false, "sql_equivalent": null, "success": false}}
+{"id": "manual_5a8f2e207d0efe5b018c_hinglish", "language": "hinglish", "backend": "supabase", "operation": "list_tables", "question": "yahan kaunsi tables hain", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"project_id": "demo_specimens", "schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.list_tables\",\"arguments\":{\"project_id\":\"demo_specmemens\",\"schemas\":[\"public\"]}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": null, "success": false}}
+{"id": "manual_8ebc9e868ee558a8ddfe_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "ambiguous", "question": "specimens best ones", "expected": {"action": "clarify", "question": "What does best mean: which column and order?"}, "output": "{\"action\":\"clarify\",\"question\":\"What does best mean: which column and order?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_35406fa024e66bcb5dd7_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "lt", "question": "specimens score less 10000", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE score < 10000;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT * FROM specimens WHERE score < 10000;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_82c104614df1ce6e2762_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "gt", "question": "specimens salary more than 100", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE salary > 100;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT * FROM specimens WHERE salary > 100;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_82c104614df1ce6e2762_en", "language": "en", "backend": "supabase", "operation": "gt", "question": "Which specimens entries have salary above 100?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE salary > 100;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specmemens\",\"query\":\"SELECT * FROM specimens WHERE salary > 100;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": true, "success": false}}
+{"id": "manual_35406fa024e66bcb5dd7_en", "language": "en", "backend": "supabase", "operation": "lt", "question": "Find specimens records below 10000 in score.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE score < 10000;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specmemens\",\"query\":\"SELECT * FROM specimens WHERE score < 10000;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": true, "success": false}}
+{"id": "manual_b564e1a7b1e7fc6d2ab4_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "avg", "question": "specimens avg score tell", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT AVG(score) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT AVG(score) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_5a8f2e207d0efe5b018c_en", "language": "en", "backend": "supabase", "operation": "list_tables", "question": "What tables can I query here?", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"project_id": "demo_specimens", "schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.list_tables\",\"arguments\":{\"project_id\":\"demo_specmens\",\"schemas\":[\"public\"]}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": null, "success": false}}
+{"id": "manual_8ebc9e868ee558a8ddfe_en", "language": "en", "backend": "supabase", "operation": "ambiguous", "question": "Pick the best entries in specimens.", "expected": {"action": "clarify", "question": "What does best mean: which column and order?"}, "output": "{\"action\":\"clarify\",\"question\":\"What does best mean: which column and order?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_8ebc9e868ee558a8ddfe_hinglish", "language": "hinglish", "backend": "supabase", "operation": "ambiguous", "question": "specimens me best records chun lo", "expected": {"action": "clarify", "question": "Best ka matlab kya hai? Kaunsa column aur order?"}, "output": "{\"action\":\"clarify\",\"question\":\"Best ka matlab kya hai? Kaunsa column aur order?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_35406fa024e66bcb5dd7_hinglish", "language": "hinglish", "backend": "supabase", "operation": "lt", "question": "specimens me score 10000 se kam wale lao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE score < 10000;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT * FROM specimens WHERE score < 10000;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_82c104614df1ce6e2762_hinglish", "language": "hinglish", "backend": "supabase", "operation": "gt", "question": "specimens me salary 100 se upar wale kaun hain", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE salary > 100;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT * FROM specimens WHERE salary > 100;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_b564e1a7b1e7fc6d2ab4_en", "language": "en", "backend": "supabase", "operation": "avg", "question": "What does score average out to in specimens?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT AVG(score) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specmemens\",\"query\":\"SELECT AVG(score) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": true, "success": false}}
+{"id": "manual_b564e1a7b1e7fc6d2ab4_hinglish", "language": "hinglish", "backend": "supabase", "operation": "avg", "question": "specimens me score ka average kya hai", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT AVG(score) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT AVG(score) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_40464b00088c9159e3d0_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "count", "question": "inspections total rows how much", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_inspections", "query": "SELECT COUNT(*) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_inspections\",\"query\":\"SELECT COUNT(*) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_475287613764159ee5eb_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "group_count", "question": "reservations count per category", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_reservations", "query": "SELECT category, COUNT(*) FROM reservations GROUP BY category;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_reservations\",\"query\":\"SELECT category, COUNT(*) FROM reservations GROUP BY category;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_fd0bab1cc08d7298f131_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "top", "question": "exhibits top 20 by salary big first", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_exhibits", "query": "SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_exhibits\",\"query\":\"SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_40464b00088c9159e3d0_hinglish", "language": "hinglish", "backend": "supabase", "operation": "count", "question": "inspections me total kitni rows hain", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_inspections", "query": "SELECT COUNT(*) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_inspections\",\"query\":\"SELECT COUNT(*) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_40464b00088c9159e3d0_en", "language": "en", "backend": "supabase", "operation": "count", "question": "How big is inspections, measured in rows?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_inspections", "query": "SELECT COUNT(*) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_inspections\",\"query\":\"SELECT COUNT(*) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_fd0bab1cc08d7298f131_hinglish", "language": "hinglish", "backend": "supabase", "operation": "top", "question": "exhibits me sabse bade salary wali 20 rows descending dikhao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_exhibits", "query": "SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_exhibits\",\"query\":\"SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_475287613764159ee5eb_hinglish", "language": "hinglish", "backend": "supabase", "operation": "group_count", "question": "reservations me har category ki count alag batao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_reservations", "query": "SELECT category, COUNT(*) FROM reservations GROUP BY category;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_reservations\",\"query\":\"SELECT category, COUNT(*) FROM reservations GROUP BY category;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_5a8f2e207d0efe5b018c_hi", "language": "hi", "backend": "supabase", "operation": "list_tables", "question": "यहाँ कौन-कौन सी टेबल हैं?", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"project_id": "demo_specimens", "schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specmemens\",\"query\":\"SELECT * FROM specimens WHERE kind = 'Delhi';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": false, "arguments_exact": false, "sql_equivalent": null, "success": false}}
+{"id": "manual_b564e1a7b1e7fc6d2ab4_hi", "language": "hi", "backend": "supabase", "operation": "avg", "question": "specimens में score का औसत क्या है?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT AVG(score) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT AVG(score) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_fd0bab1cc08d7298f131_en", "language": "en", "backend": "supabase", "operation": "top", "question": "Give me 20 entries in exhibits with the largest salary, largest first.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_exhibits", "query": "SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_exhibits\",\"query\":\"SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_475287613764159ee5eb_en", "language": "en", "backend": "supabase", "operation": "group_count", "question": "Break down the number of reservations rows by category.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_reservations", "query": "SELECT category, COUNT(*) FROM reservations GROUP BY category;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_reservations\",\"query\":\"SELECT COUNT(*) FROM reservations WHERE category = 'Areak downs';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_82c104614df1ce6e2762_hi", "language": "hi", "backend": "supabase", "operation": "gt", "question": "specimens में किनका salary 100 से ज़्यादा है?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE salary > 100;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT * FROM specimens WHERE salary > 100;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_8ebc9e868ee558a8ddfe_hi", "language": "hi", "backend": "supabase", "operation": "ambiguous", "question": "specimens में सबसे अच्छे रिकॉर्ड चुनो।", "expected": {"action": "clarify", "question": "सबसे अच्छा से आपका क्या मतलब है? कौन सा कॉलम और क्रम?"}, "output": "{\"action\":\"clarify\",\"question\":\"सबसे अच्छा से आपका क्या मतलब है? कौन सा कॉलम और क्रम?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_c4210296305105ab7472_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "ambiguous", "question": "specimens best ones", "expected": {"action": "clarify", "question": "What does best mean: which column and order?"}, "output": "{\"action\":\"clarify\",\"question\":\"What does best mean: which column and order?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_9bf8948becf74e81a103_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "list_tables", "question": "tables here list", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT * FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": false, "arguments_exact": false, "sql_equivalent": null, "success": false}}
+{"id": "manual_9bf8948becf74e81a103_hinglish", "language": "hinglish", "backend": "mysql", "operation": "list_tables", "question": "yahan kaunsi tables hain", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.list_tables\",\"arguments\":{\"schemas\":[\"public\"]}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_8efcf517aee64807a8e5_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "avg", "question": "specimens avg balance tell", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT AVG(balance) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT AVG(balance) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_adff4eabe2639c0c128c_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "all", "question": "all inspections pls", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT * FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT * FROM inspections WHERE description IS NULL;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_35406fa024e66bcb5dd7_hi", "language": "hi", "backend": "supabase", "operation": "lt", "question": "specimens में score 10000 से कम वाले रिकॉर्ड लाओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_specimens", "query": "SELECT * FROM specimens WHERE score < 10000;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_specimens\",\"query\":\"SELECT * FROM specimens WHERE score < 10000;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_23c0aa3a4596f995ad2a_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "count", "question": "specimens total rows how much", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT COUNT(*) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT COUNT(*) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_475287613764159ee5eb_hi", "language": "hi", "backend": "supabase", "operation": "group_count", "question": "reservations में हर category की गिनती अलग-अलग बताओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_reservations", "query": "SELECT category, COUNT(*) FROM reservations GROUP BY category;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_reservations\",\"query\":\"SELECT category, COUNT(*) FROM reservations GROUP BY category;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_23c0aa3a4596f995ad2a_hinglish", "language": "hinglish", "backend": "mysql", "operation": "count", "question": "specimens me total kitni rows hain", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT COUNT(*) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT COUNT(*) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2164b376f732a1c371ab_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "group_count", "question": "specimens count per kind", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT kind, COUNT(*) FROM specimens GROUP BY kind;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT kind, COUNT(*) FROM specimens GROUP BY kind;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_40464b00088c9159e3d0_hi", "language": "hi", "backend": "supabase", "operation": "count", "question": "inspections में कुल कितनी पंक्तियाँ हैं?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_inspections", "query": "SELECT COUNT(*) FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_inspections\",\"query\":\"SELECT COUNT(*) FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_8efcf517aee64807a8e5_en", "language": "en", "backend": "mysql", "operation": "avg", "question": "What does balance average out to in specimens?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT AVG(balance) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT AVG(balance) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_8efcf517aee64807a8e5_hinglish", "language": "hinglish", "backend": "mysql", "operation": "avg", "question": "specimens me balance ka average kya hai", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT AVG(balance) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT AVG(balance) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c4210296305105ab7472_en", "language": "en", "backend": "mysql", "operation": "ambiguous", "question": "Pick the best entries in specimens.", "expected": {"action": "clarify", "question": "What does best mean: which column and order?"}, "output": "{\"action\":\"clarify\",\"question\":\"What does best mean: which column and order?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_c4210296305105ab7472_hinglish", "language": "hinglish", "backend": "mysql", "operation": "ambiguous", "question": "specimens me best records chun lo", "expected": {"action": "clarify", "question": "Best ka matlab kya hai? Kaunsa column aur order?"}, "output": "{\"action\":\"clarify\",\"question\":\"Best ka matlab kya hai? Kaunsa column aur order?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_23c0aa3a4596f995ad2a_en", "language": "en", "backend": "mysql", "operation": "count", "question": "How big is specimens, measured in rows?", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT COUNT(*) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT COUNT(*) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_d30264bbc3b08aedbc51_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "sum", "question": "exhibits score all add", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT SUM(score) FROM exhibits;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT SUM(score) FROM exhibits;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_9bf8948becf74e81a103_en", "language": "en", "backend": "mysql", "operation": "list_tables", "question": "What tables can I query here?", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"schemas\":[\"public\"]}}", "metrics": {"json_valid": true, "schema_valid": false, "action_correct": true, "tool_correct": false, "arguments_exact": false, "sql_equivalent": null, "success": false, "error": "'sql' is a required property\n\nFailed validating 'required' in schema:\n    {'type': 'object',\n     'properties': {'sql': {'type': 'string'}},\n     'required': ['sql'],\n     'additionalProperties': False}\n\nOn instance:\n    {'schemas': ['public']}"}}
+{"id": "manual_adff4eabe2639c0c128c_en", "language": "en", "backend": "mysql", "operation": "all", "question": "I need everything in inspections.", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT * FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT * FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_b9c176c07809f42564a0_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "top", "question": "inspections top 5 by amount big first", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c023a5b6f95173ae856d_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "gt", "question": "exhibits score more than 50", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM exhibits WHERE score > 50;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM exhibits WHERE score > 50;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_adff4eabe2639c0c128c_hinglish", "language": "hinglish", "backend": "mysql", "operation": "all", "question": "inspections ka sara data chahiye", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT * FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT * FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_cab55a7429427fbc5093_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "distinct", "question": "inspections department unique only", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT DISTINCT department FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT DISTINCT department FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_e8f90532d34c12209a83_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "gte", "question": "inspections salary minimum 1500 include same", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM inspections WHERE salary >= 1500;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM inspections WHERE salary >= 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c023a5b6f95173ae856d_en", "language": "en", "backend": "mysql", "operation": "gt", "question": "Which exhibits entries have score above 50?", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM exhibits WHERE score > 50;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM exhibits WHERE score > 50;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_affa5e5e14444cd82bb4_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "lt", "question": "reservations salary less 10", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT * FROM reservations WHERE salary < 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE salary < 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2164b376f732a1c371ab_hinglish", "language": "hinglish", "backend": "mysql", "operation": "group_count", "question": "specimens me har kind ki count alag batao", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT kind, COUNT(*) FROM specimens GROUP BY kind;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT kind, COUNT(*) FROM specimens GROUP BY kind;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_d30264bbc3b08aedbc51_en", "language": "en", "backend": "mysql", "operation": "sum", "question": "Add up score across exhibits.", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT SUM(score) FROM exhibits;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT name FROM exhibits WHERE city = 'Adddd';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false, "error": "no such column: name"}}
+{"id": "manual_d30264bbc3b08aedbc51_hinglish", "language": "hinglish", "backend": "mysql", "operation": "sum", "question": "exhibits ke score ka jod batao", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT SUM(score) FROM exhibits;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT SUM(score) FROM exhibits;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_b02f421828127e2eb96b_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "lte", "question": "exhibits price max 2 equal also", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM exhibits WHERE price <= 2;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM exhibits ORDER BY price DESC LIMIT 2;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_b9c176c07809f42564a0_hinglish", "language": "hinglish", "backend": "mysql", "operation": "top", "question": "inspections me sabse bade amount wali 5 rows descending dikhao", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_e8f90532d34c12209a83_hinglish", "language": "hinglish", "backend": "mysql", "operation": "gte", "question": "inspections me salary kam se kam 1500 ho", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM inspections WHERE salary >= 1500;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM inspections WHERE salary >= 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2c31add7d7b45843dec1_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "project_eq", "question": "inspections kind Pune only display_name", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT display_name FROM inspections WHERE kind = 'Pune';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT display_name FROM inspections WHERE kind = 'Pune';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c023a5b6f95173ae856d_hinglish", "language": "hinglish", "backend": "mysql", "operation": "gt", "question": "exhibits me score 50 se upar wale kaun hain", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM exhibits WHERE score > 50;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM exhibits WHERE score > 50;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2164b376f732a1c371ab_en", "language": "en", "backend": "mysql", "operation": "group_count", "question": "Break down the number of specimens rows by kind.", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT kind, COUNT(*) FROM specimens GROUP BY kind;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT COUNT(*) FROM specimens WHERE kind = 'ureak down';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_b02f421828127e2eb96b_hinglish", "language": "hinglish", "backend": "mysql", "operation": "lte", "question": "exhibits me price 2 ya usse kam ho", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM exhibits WHERE price <= 2;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM exhibits WHERE price <= 2;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_8efcf517aee64807a8e5_hi", "language": "hi", "backend": "mysql", "operation": "avg", "question": "specimens में balance का औसत क्या है?", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT AVG(balance) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT AVG(balance) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_cab55a7429427fbc5093_hinglish", "language": "hinglish", "backend": "mysql", "operation": "distinct", "question": "inspections me department ki unique values kya hain", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT DISTINCT department FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT DISTINCT department FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_b9c176c07809f42564a0_en", "language": "en", "backend": "mysql", "operation": "top", "question": "Give me 5 entries in inspections with the largest amount, largest first.", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_a060620c638d2dc3759e_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "project", "question": "reservations only item_name show", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT item_name FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT item_name FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_affa5e5e14444cd82bb4_en", "language": "en", "backend": "mysql", "operation": "lt", "question": "Find reservations records below 10 in salary.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT * FROM reservations WHERE salary < 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE salary < 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_affa5e5e14444cd82bb4_hinglish", "language": "hinglish", "backend": "mysql", "operation": "lt", "question": "reservations me salary 10 se kam wale lao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT * FROM reservations WHERE salary < 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE salary < 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_e8f90532d34c12209a83_en", "language": "en", "backend": "mysql", "operation": "gte", "question": "Include inspections records at 1500 or above in salary.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM inspections WHERE salary >= 1500;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM inspections WHERE salary >= 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_04b2e5886e750df09a51_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "null", "question": "reservations display_name null records", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM reservations WHERE display_name IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE display_name IS NULL;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_9bf8948becf74e81a103_hi", "language": "hi", "backend": "mysql", "operation": "list_tables", "question": "यहाँ कौन-कौन सी टेबल हैं?", "expected": {"action": "call", "name": "warehouse.list_tables", "arguments": {"schemas": ["public"]}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT * FROM specimens WHERE kind = 'Jaipur';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": false, "arguments_exact": false, "sql_equivalent": null, "success": false}}
+{"id": "manual_cab55a7429427fbc5093_en", "language": "en", "backend": "mysql", "operation": "distinct", "question": "Which different department values occur in inspections?", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT DISTINCT department FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT DISTINCT department FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2c31add7d7b45843dec1_en", "language": "en", "backend": "mysql", "operation": "project_eq", "question": "In inspections, give me display_name for kind Pune.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT display_name FROM inspections WHERE kind = 'Pune';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT display_name FROM inspections WHERE kind = 'Pune';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2c31add7d7b45843dec1_hinglish", "language": "hinglish", "backend": "mysql", "operation": "project_eq", "question": "inspections me kind Pune walon ka display_name batao", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT display_name FROM inspections WHERE kind = 'Pune';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT display_name FROM inspections WHERE kind = 'Pune';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_a060620c638d2dc3759e_hinglish", "language": "hinglish", "backend": "mysql", "operation": "project", "question": "bas reservations ke item_name dikha do", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT item_name FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT item_name FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_b02f421828127e2eb96b_en", "language": "en", "backend": "mysql", "operation": "lte", "question": "From exhibits, include price values up to and including 2.", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM exhibits WHERE price <= 2;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM exhibits WHERE price BETWEEN 2 AND 2;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_b02f421828127e2eb96b_hi", "language": "hi", "backend": "mysql", "operation": "lte", "question": "exhibits में price 2 या उससे कम हो।", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM exhibits WHERE price <= 2;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM exhibits WHERE price <= 2;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_d30264bbc3b08aedbc51_hi", "language": "hi", "backend": "mysql", "operation": "sum", "question": "exhibits के score का जोड़ बताओ।", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT SUM(score) FROM exhibits;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT SUM(score) FROM exhibits;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_04b2e5886e750df09a51_hinglish", "language": "hinglish", "backend": "mysql", "operation": "null", "question": "reservations me jinka display_name NULL hai wo dikhao", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM reservations WHERE display_name IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE quantity > 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_a060620c638d2dc3759e_en", "language": "en", "backend": "mysql", "operation": "project", "question": "Just the item_name values from reservations, please.", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT item_name FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT item_name FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_adff4eabe2639c0c128c_hi", "language": "hi", "backend": "mysql", "operation": "all", "question": "inspections का सारा डेटा चाहिए।", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT * FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT * FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c4210296305105ab7472_hi", "language": "hi", "backend": "mysql", "operation": "ambiguous", "question": "specimens में सबसे अच्छे रिकॉर्ड चुनो।", "expected": {"action": "clarify", "question": "सबसे अच्छा से आपका क्या मतलब है? कौन सा कॉलम और क्रम?"}, "output": "{\"action\":\"clarify\",\"question\":\"सबसे अच्छा से आपका क्या मतलब है? कौन सा कॉलम और क्रम?\"}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": null, "arguments_exact": null, "sql_equivalent": null, "success": true, "text_exact": true}}
+{"id": "manual_23c0aa3a4596f995ad2a_hi", "language": "hi", "backend": "mysql", "operation": "count", "question": "specimens में कुल कितनी पंक्तियाँ हैं?", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT COUNT(*) FROM specimens;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT COUNT(*) FROM specimens;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_2164b376f732a1c371ab_hi", "language": "hi", "backend": "mysql", "operation": "group_count", "question": "specimens में हर kind की गिनती अलग-अलग बताओ।", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT kind, COUNT(*) FROM specimens GROUP BY kind;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT kind, COUNT(*) FROM specimens GROUP BY kind;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_c023a5b6f95173ae856d_hi", "language": "hi", "backend": "mysql", "operation": "gt", "question": "exhibits में किनका score 50 से ज़्यादा है?", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM exhibits WHERE score > 50;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM exhibits WHERE score > 50;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_e8f90532d34c12209a83_hi", "language": "hi", "backend": "mysql", "operation": "gte", "question": "inspections में salary कम से कम 1500 होना चाहिए।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"query": "SELECT * FROM inspections WHERE salary >= 1500;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"query\":\"SELECT * FROM inspections WHERE salary >= 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_04b2e5886e750df09a51_en", "language": "en", "backend": "mysql", "operation": "null", "question": "Find reservations records with no display_name value (NULL).", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM reservations WHERE display_name IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE display_name IS NOT NULL;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_2c31add7d7b45843dec1_hi", "language": "hi", "backend": "mysql", "operation": "project_eq", "question": "inspections में kind Pune हो तो उनका display_name बताओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT display_name FROM inspections WHERE kind = 'Pune';"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT display_name FROM inspections WHERE kind = 'Pune';\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_fd0bab1cc08d7298f131_hi", "language": "hi", "backend": "supabase", "operation": "top", "question": "exhibits में सबसे ज़्यादा salary वाली 20 पंक्तियाँ घटते क्रम में दिखाओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"project_id": "demo_exhibits", "query": "SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"project_id\":\"demo_exhibits\",\"query\":\"SELECT * FROM exhibits ORDER BY salary DESC LIMIT 20;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_cab55a7429427fbc5093_hi", "language": "hi", "backend": "mysql", "operation": "distinct", "question": "inspections में department के अलग-अलग मान कौन से हैं?", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"query": "SELECT DISTINCT department FROM inspections;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"query\":\"SELECT DISTINCT department FROM inspections;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_a060620c638d2dc3759e_hi", "language": "hi", "backend": "mysql", "operation": "project", "question": "reservations से सिर्फ item_name दिखाना।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT item_name FROM reservations;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT item_name FROM reservations;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_affa5e5e14444cd82bb4_hi", "language": "hi", "backend": "mysql", "operation": "lt", "question": "reservations में salary 10 से कम वाले रिकॉर्ड लाओ।", "expected": {"action": "call", "name": "warehouse.execute_sql", "arguments": {"sql": "SELECT * FROM reservations WHERE salary < 10;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.execute_sql\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE salary < 10;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_04b2e5886e750df09a51_hi", "language": "hi", "backend": "mysql", "operation": "null", "question": "reservations में जिनका display_name NULL है वे रिकॉर्ड दिखाओ।", "expected": {"action": "call", "name": "warehouse.run_query", "arguments": {"sql": "SELECT * FROM reservations WHERE display_name IS NULL;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.run_query\",\"arguments\":{\"sql\":\"SELECT * FROM reservations WHERE quantity > 1500;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": false, "sql_equivalent": false, "success": false}}
+{"id": "manual_b9c176c07809f42564a0_hi", "language": "hi", "backend": "mysql", "operation": "top", "question": "inspections में सबसे ज़्यादा amount वाली 5 पंक्तियाँ घटते क्रम में दिखाओ।", "expected": {"action": "call", "name": "warehouse.query", "arguments": {"query": "SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;"}}, "output": "{\"action\":\"call\",\"name\":\"warehouse.query\",\"arguments\":{\"query\":\"SELECT * FROM inspections ORDER BY amount DESC LIMIT 5;\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": true, "success": true}}
+{"id": "manual_88b478ce80eed61bdfb4_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "weather", "question": "Jaipur weather celsius pls", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Jaipur", "unit": "celsius"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Jaipur\",\"unit\":\"celsius\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_88b478ce80eed61bdfb4_hinglish", "language": "hinglish", "backend": "supabase", "operation": "weather", "question": "Jaipur ka weather celsius me batao", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Jaipur", "unit": "celsius"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Jaipur\",\"unit\":\"celsius\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_88b478ce80eed61bdfb4_en", "language": "en", "backend": "supabase", "operation": "weather", "question": "What is the weather like in Jaipur? Use celsius.", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Jaipur", "unit": "celsius"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Jaipur\",\"unit\":\"celsius\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_88b478ce80eed61bdfb4_hi", "language": "hi", "backend": "supabase", "operation": "weather", "question": "Jaipur का मौसम celsius में बताओ।", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Jaipur", "unit": "celsius"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Jaipur\",\"unit\":\"celsius\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_4e42bbd40467b2b8c27f_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "search", "question": "docs find SQL joins", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "SQL joins"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"SQL joins\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_8a696e3c3522dd69d5ac_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "ticket", "question": "ticket TKT-4486 details", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-4486"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-4486\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_4e42bbd40467b2b8c27f_en", "language": "en", "backend": "supabase", "operation": "search", "question": "Find documentation about SQL joins.", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "SQL joins"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"SQL joins\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_f73e21e0eb10d6617995_noisy_en", "language": "noisy_en", "backend": "supabase", "operation": "read_file", "question": "/config/app.json read pls", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/config/app.json"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/config/app.json\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_8a696e3c3522dd69d5ac_hinglish", "language": "hinglish", "backend": "supabase", "operation": "ticket", "question": "ticket TKT-4486 ki details lao", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-4486"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-4486\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_4e42bbd40467b2b8c27f_hinglish", "language": "hinglish", "backend": "supabase", "operation": "search", "question": "SQL joins ke bare me docs dhundho", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "SQL joins"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"SQL joins\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_f73e21e0eb10d6617995_hinglish", "language": "hinglish", "backend": "supabase", "operation": "read_file", "question": "/config/app.json file padh kar dikhao", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/config/app.json"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/config/app.json\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_8a696e3c3522dd69d5ac_en", "language": "en", "backend": "supabase", "operation": "ticket", "question": "Fetch the details for ticket TKT-4486.", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-4486"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-4486\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_f73e21e0eb10d6617995_en", "language": "en", "backend": "supabase", "operation": "read_file", "question": "Open /config/app.json and read its text.", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/config/app.json"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/config/app.json\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_8a696e3c3522dd69d5ac_hi", "language": "hi", "backend": "supabase", "operation": "ticket", "question": "टिकट TKT-4486 का विवरण लाओ।", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-4486"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-4486\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_f73e21e0eb10d6617995_hi", "language": "hi", "backend": "supabase", "operation": "read_file", "question": "/config/app.json फ़ाइल पढ़कर दिखाओ।", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/config/app.json"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/config/app.json\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_252c6053295d69a93c94_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "search", "question": "docs find database backups", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "database backups"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"database backups\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_5ebc52b3d20bde2076b3_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "ticket", "question": "ticket TKT-6124 details", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-6124"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-6124\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_754691bd1eea25d41ccf_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "weather", "question": "Delhi weather fahrenheit pls", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Delhi", "unit": "fahrenheit"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Delhi\",\"unit\":\"fahrenheit\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_252c6053295d69a93c94_en", "language": "en", "backend": "mysql", "operation": "search", "question": "Find documentation about database backups.", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "database backups"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"database backups\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_4e42bbd40467b2b8c27f_hi", "language": "hi", "backend": "supabase", "operation": "search", "question": "SQL joins के बारे में दस्तावेज़ खोजो।", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "SQL joins"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"SQL joins\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_5ebc52b3d20bde2076b3_hinglish", "language": "hinglish", "backend": "mysql", "operation": "ticket", "question": "ticket TKT-6124 ki details lao", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-6124"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-6124\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_754691bd1eea25d41ccf_hinglish", "language": "hinglish", "backend": "mysql", "operation": "weather", "question": "Delhi ka weather fahrenheit me batao", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Delhi", "unit": "fahrenheit"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Delhi\",\"unit\":\"fahrenheit\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_252c6053295d69a93c94_hinglish", "language": "hinglish", "backend": "mysql", "operation": "search", "question": "database backups ke bare me docs dhundho", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "database backups"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"database backups\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_6f3055c580b33340c854_noisy_en", "language": "noisy_en", "backend": "mysql", "operation": "read_file", "question": "/notes/setup.txt read pls", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/notes/setup.txt"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/notes/setup.txt\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_5ebc52b3d20bde2076b3_en", "language": "en", "backend": "mysql", "operation": "ticket", "question": "Fetch the details for ticket TKT-6124.", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-6124"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-6124\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_6f3055c580b33340c854_hinglish", "language": "hinglish", "backend": "mysql", "operation": "read_file", "question": "/notes/setup.txt file padh kar dikhao", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/notes/setup.txt"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/notes/setup.txt\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_6f3055c580b33340c854_en", "language": "en", "backend": "mysql", "operation": "read_file", "question": "Open /notes/setup.txt and read its text.", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/notes/setup.txt"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/notes/setup.txt\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_754691bd1eea25d41ccf_en", "language": "en", "backend": "mysql", "operation": "weather", "question": "What is the weather like in Delhi? Use fahrenheit.", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Delhi", "unit": "fahrenheit"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Delhi\",\"unit\":\"fahrenheit\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_754691bd1eea25d41ccf_hi", "language": "hi", "backend": "mysql", "operation": "weather", "question": "Delhi का मौसम fahrenheit में बताओ।", "expected": {"action": "call", "name": "external.get_weather", "arguments": {"city": "Delhi", "unit": "fahrenheit"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_weather\",\"arguments\":{\"city\":\"Delhi\",\"unit\":\"fahrenheit\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_5ebc52b3d20bde2076b3_hi", "language": "hi", "backend": "mysql", "operation": "ticket", "question": "टिकट TKT-6124 का विवरण लाओ।", "expected": {"action": "call", "name": "external.get_ticket", "arguments": {"ticket_id": "TKT-6124"}}, "output": "{\"action\":\"call\",\"name\":\"external.get_ticket\",\"arguments\":{\"ticket_id\":\"TKT-6124\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_6f3055c580b33340c854_hi", "language": "hi", "backend": "mysql", "operation": "read_file", "question": "/notes/setup.txt फ़ाइल पढ़कर दिखाओ।", "expected": {"action": "call", "name": "external.read_file", "arguments": {"path": "/notes/setup.txt"}}, "output": "{\"action\":\"call\",\"name\":\"external.read_file\",\"arguments\":{\"path\":\"/notes/setup.txt\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}
+{"id": "manual_252c6053295d69a93c94_hi", "language": "hi", "backend": "mysql", "operation": "search", "question": "database backups के बारे में दस्तावेज़ खोजो।", "expected": {"action": "call", "name": "external.search_docs", "arguments": {"query": "database backups"}}, "output": "{\"action\":\"call\",\"name\":\"external.search_docs\",\"arguments\":{\"query\":\"database backups\"}}", "metrics": {"json_valid": true, "schema_valid": true, "action_correct": true, "tool_correct": true, "arguments_exact": true, "sql_equivalent": null, "success": true}}

evaluation/manual-gpu.summary.json

@@ -0,0 +1,1008 @@
+{
+  "examples": 160,
+  "seconds": 2.227508544921875,
+  "decoding": "raw greedy, no repair, no teacher fallback",
+  "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
+  "sql_metric": "Compile against the supplied schema, then compare results on two generated SQLite fixtures after dialect adaptation; not native MySQL/PostgreSQL execution.",
+  "metrics": {
+    "json_valid": {
+      "correct": 160,
+      "total": 160,
+      "rate": 1.0
+    },
+    "schema_valid": {
+      "correct": 159,
+      "total": 160,
+      "rate": 0.99375
+    },
+    "action_correct": {
+      "correct": 159,
+      "total": 160,
+      "rate": 0.99375
+    },
+    "tool_correct": {
+      "correct": 146,
+      "total": 152,
+      "rate": 0.9605263157894737
+    },
+    "arguments_exact": {
+      "correct": 124,
+      "total": 152,
+      "rate": 0.8157894736842105
+    },
+    "sql_equivalent": {
+      "correct": 94,
+      "total": 112,
+      "rate": 0.8392857142857143
+    },
+    "success": {
+      "correct": 132,
+      "total": 160,
+      "rate": 0.825
+    }
+  },
+  "groups": {
+    "language:noisy_en": {
+      "json_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 39,
+        "total": 40,
+        "rate": 0.975
+      },
+      "tool_correct": {
+        "correct": 35,
+        "total": 38,
+        "rate": 0.9210526315789473
+      },
+      "arguments_exact": {
+        "correct": 31,
+        "total": 38,
+        "rate": 0.8157894736842105
+      },
+      "sql_equivalent": {
+        "correct": 23,
+        "total": 28,
+        "rate": 0.8214285714285714
+      },
+      "success": {
+        "correct": 33,
+        "total": 40,
+        "rate": 0.825
+      }
+    },
+    "backend:supabase": {
+      "json_valid": {
+        "correct": 80,
+        "total": 80,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 80,
+        "total": 80,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 79,
+        "total": 80,
+        "rate": 0.9875
+      },
+      "tool_correct": {
+        "correct": 73,
+        "total": 76,
+        "rate": 0.9605263157894737
+      },
+      "arguments_exact": {
+        "correct": 59,
+        "total": 76,
+        "rate": 0.7763157894736842
+      },
+      "sql_equivalent": {
+        "correct": 46,
+        "total": 56,
+        "rate": 0.8214285714285714
+      },
+      "success": {
+        "correct": 63,
+        "total": 80,
+        "rate": 0.7875
+      }
+    },
+    "operation:all": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "tool_correct": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "arguments_exact": {
+        "correct": 5,
+        "total": 8,
+        "rate": 0.625
+      },
+      "sql_equivalent": {
+        "correct": 5,
+        "total": 8,
+        "rate": 0.625
+      },
+      "success": {
+        "correct": 5,
+        "total": 8,
+        "rate": 0.625
+      }
+    },
+    "operation:null": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 2,
+        "total": 8,
+        "rate": 0.25
+      },
+      "sql_equivalent": {
+        "correct": 2,
+        "total": 8,
+        "rate": 0.25
+      },
+      "success": {
+        "correct": 2,
+        "total": 8,
+        "rate": 0.25
+      }
+    },
+    "operation:gte": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "sql_equivalent": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "success": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      }
+    },
+    "language:en": {
+      "json_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 39,
+        "total": 40,
+        "rate": 0.975
+      },
+      "action_correct": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 37,
+        "total": 38,
+        "rate": 0.9736842105263158
+      },
+      "arguments_exact": {
+        "correct": 25,
+        "total": 38,
+        "rate": 0.6578947368421053
+      },
+      "sql_equivalent": {
+        "correct": 20,
+        "total": 28,
+        "rate": 0.7142857142857143
+      },
+      "success": {
+        "correct": 27,
+        "total": 40,
+        "rate": 0.675
+      }
+    },
+    "operation:project": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "language:hinglish": {
+      "json_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 38,
+        "total": 38,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 34,
+        "total": 38,
+        "rate": 0.8947368421052632
+      },
+      "sql_equivalent": {
+        "correct": 25,
+        "total": 28,
+        "rate": 0.8928571428571429
+      },
+      "success": {
+        "correct": 36,
+        "total": 40,
+        "rate": 0.9
+      }
+    },
+    "operation:sum": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 6,
+        "total": 8,
+        "rate": 0.75
+      },
+      "sql_equivalent": {
+        "correct": 6,
+        "total": 8,
+        "rate": 0.75
+      },
+      "success": {
+        "correct": 6,
+        "total": 8,
+        "rate": 0.75
+      }
+    },
+    "operation:distinct": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:project_eq": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:lte": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 4,
+        "total": 8,
+        "rate": 0.5
+      },
+      "sql_equivalent": {
+        "correct": 4,
+        "total": 8,
+        "rate": 0.5
+      },
+      "success": {
+        "correct": 4,
+        "total": 8,
+        "rate": 0.5
+      }
+    },
+    "language:hi": {
+      "json_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 40,
+        "total": 40,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 36,
+        "total": 38,
+        "rate": 0.9473684210526315
+      },
+      "arguments_exact": {
+        "correct": 34,
+        "total": 38,
+        "rate": 0.8947368421052632
+      },
+      "sql_equivalent": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "success": {
+        "correct": 36,
+        "total": 40,
+        "rate": 0.9
+      }
+    },
+    "operation:list_tables": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 3,
+        "total": 8,
+        "rate": 0.375
+      },
+      "arguments_exact": {
+        "correct": 1,
+        "total": 8,
+        "rate": 0.125
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 1,
+        "total": 8,
+        "rate": 0.125
+      }
+    },
+    "operation:ambiguous": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:lt": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      }
+    },
+    "operation:gt": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      }
+    },
+    "operation:avg": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 7,
+        "total": 8,
+        "rate": 0.875
+      }
+    },
+    "operation:count": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:group_count": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 6,
+        "total": 8,
+        "rate": 0.75
+      },
+      "sql_equivalent": {
+        "correct": 6,
+        "total": 8,
+        "rate": 0.75
+      },
+      "success": {
+        "correct": 6,
+        "total": 8,
+        "rate": 0.75
+      }
+    },
+    "operation:top": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "backend:mysql": {
+      "json_valid": {
+        "correct": 80,
+        "total": 80,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 79,
+        "total": 80,
+        "rate": 0.9875
+      },
+      "action_correct": {
+        "correct": 80,
+        "total": 80,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 73,
+        "total": 76,
+        "rate": 0.9605263157894737
+      },
+      "arguments_exact": {
+        "correct": 65,
+        "total": 76,
+        "rate": 0.8552631578947368
+      },
+      "sql_equivalent": {
+        "correct": 48,
+        "total": 56,
+        "rate": 0.8571428571428571
+      },
+      "success": {
+        "correct": 69,
+        "total": 80,
+        "rate": 0.8625
+      }
+    },
+    "operation:weather": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:search": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:ticket": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    },
+    "operation:read_file": {
+      "json_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 8,
+        "total": 8,
+        "rate": 1.0
+      }
+    }
+  }
+}

evaluation/native-manual.json

@@ -0,0 +1,802 @@
+{
+  "seconds": 1.1846249103546143,
+  "cases": 112,
+  "gold_executes": 107,
+  "prediction_equivalent": 91,
+  "engine_versions": {
+    "mysql": "8.0.46",
+    "postgresql": "16.15"
+  },
+  "results": [
+    {
+      "id": "manual_adff4eabe2639c0c128c_en",
+      "backend": "mysql",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_adff4eabe2639c0c128c_noisy_en",
+      "backend": "mysql",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_adff4eabe2639c0c128c_hi",
+      "backend": "mysql",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_adff4eabe2639c0c128c_hinglish",
+      "backend": "mysql",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_efee65268e6dc2bb069e_en",
+      "backend": "supabase",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_efee65268e6dc2bb069e_noisy_en",
+      "backend": "supabase",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_efee65268e6dc2bb069e_hi",
+      "backend": "supabase",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_efee65268e6dc2bb069e_hinglish",
+      "backend": "supabase",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2c31add7d7b45843dec1_en",
+      "backend": "mysql",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2c31add7d7b45843dec1_noisy_en",
+      "backend": "mysql",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2c31add7d7b45843dec1_hi",
+      "backend": "mysql",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2c31add7d7b45843dec1_hinglish",
+      "backend": "mysql",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_82c104614df1ce6e2762_en",
+      "backend": "supabase",
+      "operation": "gt",
+      "gold_executes": false,
+      "prediction_equivalent": false,
+      "error": "Wrong non-SQL arguments"
+    },
+    {
+      "id": "manual_82c104614df1ce6e2762_noisy_en",
+      "backend": "supabase",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_82c104614df1ce6e2762_hi",
+      "backend": "supabase",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_82c104614df1ce6e2762_hinglish",
+      "backend": "supabase",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_35406fa024e66bcb5dd7_en",
+      "backend": "supabase",
+      "operation": "lt",
+      "gold_executes": false,
+      "prediction_equivalent": false,
+      "error": "Wrong non-SQL arguments"
+    },
+    {
+      "id": "manual_35406fa024e66bcb5dd7_noisy_en",
+      "backend": "supabase",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_35406fa024e66bcb5dd7_hi",
+      "backend": "supabase",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_35406fa024e66bcb5dd7_hinglish",
+      "backend": "supabase",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_0b8db53d711baed4f4ed_en",
+      "backend": "supabase",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_0b8db53d711baed4f4ed_noisy_en",
+      "backend": "supabase",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_0b8db53d711baed4f4ed_hi",
+      "backend": "supabase",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_0b8db53d711baed4f4ed_hinglish",
+      "backend": "supabase",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b02f421828127e2eb96b_en",
+      "backend": "mysql",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_b02f421828127e2eb96b_noisy_en",
+      "backend": "mysql",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_b02f421828127e2eb96b_hi",
+      "backend": "mysql",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b02f421828127e2eb96b_hinglish",
+      "backend": "mysql",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2e2ac3c5dcf95429daaa_en",
+      "backend": "supabase",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_2e2ac3c5dcf95429daaa_noisy_en",
+      "backend": "supabase",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2e2ac3c5dcf95429daaa_hi",
+      "backend": "supabase",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_2e2ac3c5dcf95429daaa_hinglish",
+      "backend": "supabase",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_40464b00088c9159e3d0_en",
+      "backend": "supabase",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_40464b00088c9159e3d0_noisy_en",
+      "backend": "supabase",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_40464b00088c9159e3d0_hi",
+      "backend": "supabase",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_40464b00088c9159e3d0_hinglish",
+      "backend": "supabase",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_d735c1ac1979ff1e8620_en",
+      "backend": "supabase",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_d735c1ac1979ff1e8620_noisy_en",
+      "backend": "supabase",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_d735c1ac1979ff1e8620_hi",
+      "backend": "supabase",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_d735c1ac1979ff1e8620_hinglish",
+      "backend": "supabase",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_8efcf517aee64807a8e5_en",
+      "backend": "mysql",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_8efcf517aee64807a8e5_noisy_en",
+      "backend": "mysql",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_8efcf517aee64807a8e5_hi",
+      "backend": "mysql",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_8efcf517aee64807a8e5_hinglish",
+      "backend": "mysql",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_cab55a7429427fbc5093_en",
+      "backend": "mysql",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_cab55a7429427fbc5093_noisy_en",
+      "backend": "mysql",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_cab55a7429427fbc5093_hi",
+      "backend": "mysql",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_cab55a7429427fbc5093_hinglish",
+      "backend": "mysql",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b9c176c07809f42564a0_en",
+      "backend": "mysql",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b9c176c07809f42564a0_noisy_en",
+      "backend": "mysql",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b9c176c07809f42564a0_hi",
+      "backend": "mysql",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b9c176c07809f42564a0_hinglish",
+      "backend": "mysql",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_475287613764159ee5eb_en",
+      "backend": "supabase",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_475287613764159ee5eb_noisy_en",
+      "backend": "supabase",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_475287613764159ee5eb_hi",
+      "backend": "supabase",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_475287613764159ee5eb_hinglish",
+      "backend": "supabase",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2ad3e251c936135e2005_en",
+      "backend": "supabase",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2ad3e251c936135e2005_noisy_en",
+      "backend": "supabase",
+      "operation": "all",
+      "gold_executes": false,
+      "prediction_equivalent": false,
+      "error": "Wrong tool"
+    },
+    {
+      "id": "manual_2ad3e251c936135e2005_hi",
+      "backend": "supabase",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2ad3e251c936135e2005_hinglish",
+      "backend": "supabase",
+      "operation": "all",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_a060620c638d2dc3759e_en",
+      "backend": "mysql",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_a060620c638d2dc3759e_noisy_en",
+      "backend": "mysql",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_a060620c638d2dc3759e_hi",
+      "backend": "mysql",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_a060620c638d2dc3759e_hinglish",
+      "backend": "mysql",
+      "operation": "project",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_eb982cf071094adaca4f_en",
+      "backend": "supabase",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_eb982cf071094adaca4f_noisy_en",
+      "backend": "supabase",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_eb982cf071094adaca4f_hi",
+      "backend": "supabase",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_eb982cf071094adaca4f_hinglish",
+      "backend": "supabase",
+      "operation": "project_eq",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_c023a5b6f95173ae856d_en",
+      "backend": "mysql",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_c023a5b6f95173ae856d_noisy_en",
+      "backend": "mysql",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_c023a5b6f95173ae856d_hi",
+      "backend": "mysql",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_c023a5b6f95173ae856d_hinglish",
+      "backend": "mysql",
+      "operation": "gt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_affa5e5e14444cd82bb4_en",
+      "backend": "mysql",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_affa5e5e14444cd82bb4_noisy_en",
+      "backend": "mysql",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_affa5e5e14444cd82bb4_hi",
+      "backend": "mysql",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_affa5e5e14444cd82bb4_hinglish",
+      "backend": "mysql",
+      "operation": "lt",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_c4f25575abfd24e3cd63_en",
+      "backend": "supabase",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_c4f25575abfd24e3cd63_noisy_en",
+      "backend": "supabase",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_c4f25575abfd24e3cd63_hi",
+      "backend": "supabase",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_c4f25575abfd24e3cd63_hinglish",
+      "backend": "supabase",
+      "operation": "lte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_23c0aa3a4596f995ad2a_en",
+      "backend": "mysql",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_23c0aa3a4596f995ad2a_noisy_en",
+      "backend": "mysql",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_23c0aa3a4596f995ad2a_hi",
+      "backend": "mysql",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_23c0aa3a4596f995ad2a_hinglish",
+      "backend": "mysql",
+      "operation": "count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b564e1a7b1e7fc6d2ab4_en",
+      "backend": "supabase",
+      "operation": "avg",
+      "gold_executes": false,
+      "prediction_equivalent": false,
+      "error": "Wrong non-SQL arguments"
+    },
+    {
+      "id": "manual_b564e1a7b1e7fc6d2ab4_noisy_en",
+      "backend": "supabase",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b564e1a7b1e7fc6d2ab4_hi",
+      "backend": "supabase",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_b564e1a7b1e7fc6d2ab4_hinglish",
+      "backend": "supabase",
+      "operation": "avg",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2164b376f732a1c371ab_en",
+      "backend": "mysql",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_2164b376f732a1c371ab_noisy_en",
+      "backend": "mysql",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2164b376f732a1c371ab_hi",
+      "backend": "mysql",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_2164b376f732a1c371ab_hinglish",
+      "backend": "mysql",
+      "operation": "group_count",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_e8f90532d34c12209a83_en",
+      "backend": "mysql",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_e8f90532d34c12209a83_noisy_en",
+      "backend": "mysql",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_e8f90532d34c12209a83_hi",
+      "backend": "mysql",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_e8f90532d34c12209a83_hinglish",
+      "backend": "mysql",
+      "operation": "gte",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_04b2e5886e750df09a51_en",
+      "backend": "mysql",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_04b2e5886e750df09a51_noisy_en",
+      "backend": "mysql",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_04b2e5886e750df09a51_hi",
+      "backend": "mysql",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_04b2e5886e750df09a51_hinglish",
+      "backend": "mysql",
+      "operation": "null",
+      "gold_executes": true,
+      "prediction_equivalent": false
+    },
+    {
+      "id": "manual_d30264bbc3b08aedbc51_en",
+      "backend": "mysql",
+      "operation": "sum",
+      "gold_executes": false,
+      "prediction_equivalent": false,
+      "error": "(1054, \"Unknown column 'name' in 'field list'\")"
+    },
+    {
+      "id": "manual_d30264bbc3b08aedbc51_noisy_en",
+      "backend": "mysql",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_d30264bbc3b08aedbc51_hi",
+      "backend": "mysql",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_d30264bbc3b08aedbc51_hinglish",
+      "backend": "mysql",
+      "operation": "sum",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_7dd163238936d96d0bd4_en",
+      "backend": "supabase",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_7dd163238936d96d0bd4_noisy_en",
+      "backend": "supabase",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_7dd163238936d96d0bd4_hi",
+      "backend": "supabase",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_7dd163238936d96d0bd4_hinglish",
+      "backend": "supabase",
+      "operation": "distinct",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_fd0bab1cc08d7298f131_en",
+      "backend": "supabase",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_fd0bab1cc08d7298f131_noisy_en",
+      "backend": "supabase",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_fd0bab1cc08d7298f131_hi",
+      "backend": "supabase",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    },
+    {
+      "id": "manual_fd0bab1cc08d7298f131_hinglish",
+      "backend": "supabase",
+      "operation": "top",
+      "gold_executes": true,
+      "prediction_equivalent": true
+    }
+  ],
+  "metric_note": "prediction_equivalent includes correct tool/schema/project checks and two native SQL fixtures. gold_executes is a legacy label for completion of the combined validation path, not an independent gold correctness score; prediction errors can leave it false."
+}

evaluation/test-gpu.summary.json

@@ -0,0 +1,1933 @@
+{
+  "examples": 1196,
+  "seconds": 17.66167116165161,
+  "decoding": "raw greedy, no repair, no teacher fallback",
+  "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
+  "sql_metric": "Compile against the supplied schema, then compare results on two generated SQLite fixtures after dialect adaptation; not native MySQL/PostgreSQL execution.",
+  "metrics": {
+    "json_valid": {
+      "correct": 1196,
+      "total": 1196,
+      "rate": 1.0
+    },
+    "schema_valid": {
+      "correct": 1191,
+      "total": 1196,
+      "rate": 0.995819397993311
+    },
+    "action_correct": {
+      "correct": 1192,
+      "total": 1196,
+      "rate": 0.9966555183946488
+    },
+    "tool_correct": {
+      "correct": 1088,
+      "total": 1100,
+      "rate": 0.9890909090909091
+    },
+    "arguments_exact": {
+      "correct": 1014,
+      "total": 1100,
+      "rate": 0.9218181818181819
+    },
+    "sql_equivalent": {
+      "correct": 896,
+      "total": 912,
+      "rate": 0.9824561403508771
+    },
+    "success": {
+      "correct": 1106,
+      "total": 1196,
+      "rate": 0.9247491638795987
+    }
+  },
+  "groups": {
+    "language:noisy_en": {
+      "json_valid": {
+        "correct": 299,
+        "total": 299,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 298,
+        "total": 299,
+        "rate": 0.9966555183946488
+      },
+      "action_correct": {
+        "correct": 299,
+        "total": 299,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 274,
+        "total": 275,
+        "rate": 0.9963636363636363
+      },
+      "arguments_exact": {
+        "correct": 252,
+        "total": 275,
+        "rate": 0.9163636363636364
+      },
+      "sql_equivalent": {
+        "correct": 228,
+        "total": 228,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 276,
+        "total": 299,
+        "rate": 0.9230769230769231
+      }
+    },
+    "backend:supabase": {
+      "json_valid": {
+        "correct": 588,
+        "total": 588,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 588,
+        "total": 588,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 588,
+        "total": 588,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 541,
+        "total": 544,
+        "rate": 0.9944852941176471
+      },
+      "arguments_exact": {
+        "correct": 480,
+        "total": 544,
+        "rate": 0.8823529411764706
+      },
+      "sql_equivalent": {
+        "correct": 429,
+        "total": 436,
+        "rate": 0.9839449541284404
+      },
+      "success": {
+        "correct": 524,
+        "total": 588,
+        "rate": 0.891156462585034
+      }
+    },
+    "operation:missing_schema": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      }
+    },
+    "language:en": {
+      "json_valid": {
+        "correct": 299,
+        "total": 299,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 298,
+        "total": 299,
+        "rate": 0.9966555183946488
+      },
+      "action_correct": {
+        "correct": 297,
+        "total": 299,
+        "rate": 0.9933110367892977
+      },
+      "tool_correct": {
+        "correct": 274,
+        "total": 275,
+        "rate": 0.9963636363636363
+      },
+      "arguments_exact": {
+        "correct": 246,
+        "total": 275,
+        "rate": 0.8945454545454545
+      },
+      "sql_equivalent": {
+        "correct": 222,
+        "total": 228,
+        "rate": 0.9736842105263158
+      },
+      "success": {
+        "correct": 268,
+        "total": 299,
+        "rate": 0.8963210702341137
+      }
+    },
+    "language:hinglish": {
+      "json_valid": {
+        "correct": 299,
+        "total": 299,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 297,
+        "total": 299,
+        "rate": 0.9933110367892977
+      },
+      "action_correct": {
+        "correct": 297,
+        "total": 299,
+        "rate": 0.9933110367892977
+      },
+      "tool_correct": {
+        "correct": 273,
+        "total": 275,
+        "rate": 0.9927272727272727
+      },
+      "arguments_exact": {
+        "correct": 262,
+        "total": 275,
+        "rate": 0.9527272727272728
+      },
+      "sql_equivalent": {
+        "correct": 228,
+        "total": 228,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 284,
+        "total": 299,
+        "rate": 0.9498327759197325
+      }
+    },
+    "operation:sql_error": {
+      "json_valid": {
+        "correct": 20,
+        "total": 20,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 20,
+        "total": 20,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 20,
+        "total": 20,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 20,
+        "total": 20,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 12,
+        "total": 20,
+        "rate": 0.6
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 12,
+        "total": 20,
+        "rate": 0.6
+      }
+    },
+    "backend:mysql": {
+      "json_valid": {
+        "correct": 608,
+        "total": 608,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 603,
+        "total": 608,
+        "rate": 0.9917763157894737
+      },
+      "action_correct": {
+        "correct": 604,
+        "total": 608,
+        "rate": 0.993421052631579
+      },
+      "tool_correct": {
+        "correct": 547,
+        "total": 556,
+        "rate": 0.9838129496402878
+      },
+      "arguments_exact": {
+        "correct": 534,
+        "total": 556,
+        "rate": 0.960431654676259
+      },
+      "sql_equivalent": {
+        "correct": 467,
+        "total": 476,
+        "rate": 0.9810924369747899
+      },
+      "success": {
+        "correct": 582,
+        "total": 608,
+        "rate": 0.9572368421052632
+      }
+    },
+    "operation:list_tables": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 12,
+        "total": 24,
+        "rate": 0.5
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 12,
+        "total": 24,
+        "rate": 0.5
+      }
+    },
+    "operation:all": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:count": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 23,
+        "total": 28,
+        "rate": 0.8214285714285714
+      },
+      "sql_equivalent": {
+        "correct": 23,
+        "total": 28,
+        "rate": 0.8214285714285714
+      },
+      "success": {
+        "correct": 23,
+        "total": 28,
+        "rate": 0.8214285714285714
+      }
+    },
+    "operation:avg": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:describe": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      },
+      "arguments_exact": {
+        "correct": 19,
+        "total": 24,
+        "rate": 0.7916666666666666
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 19,
+        "total": 24,
+        "rate": 0.7916666666666666
+      }
+    },
+    "operation:project": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:gte": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:min": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      },
+      "success": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      }
+    },
+    "operation:null": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:write": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      }
+    },
+    "operation:or": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:count_eq": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 25,
+        "total": 28,
+        "rate": 0.8928571428571429
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 25,
+        "total": 28,
+        "rate": 0.8928571428571429
+      }
+    },
+    "operation:contains": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:sum": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:sort_desc": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:eq": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:project_eq": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "language:hi": {
+      "json_valid": {
+        "correct": 299,
+        "total": 299,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 298,
+        "total": 299,
+        "rate": 0.9966555183946488
+      },
+      "action_correct": {
+        "correct": 299,
+        "total": 299,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 267,
+        "total": 275,
+        "rate": 0.9709090909090909
+      },
+      "arguments_exact": {
+        "correct": 254,
+        "total": 275,
+        "rate": 0.9236363636363636
+      },
+      "sql_equivalent": {
+        "correct": 218,
+        "total": 228,
+        "rate": 0.956140350877193
+      },
+      "success": {
+        "correct": 278,
+        "total": 299,
+        "rate": 0.9297658862876255
+      }
+    },
+    "operation:year": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:bottom": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:distinct": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:ambiguous": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:max": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:group_count": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:sort_asc": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:not_null": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:lte": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:lt": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:group_sum": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:between": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 21,
+        "total": 28,
+        "rate": 0.75
+      },
+      "arguments_exact": {
+        "correct": 21,
+        "total": 28,
+        "rate": 0.75
+      },
+      "sql_equivalent": {
+        "correct": 21,
+        "total": 28,
+        "rate": 0.75
+      },
+      "success": {
+        "correct": 21,
+        "total": 28,
+        "rate": 0.75
+      }
+    },
+    "operation:month": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      }
+    },
+    "operation:and": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:having": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 20,
+        "total": 28,
+        "rate": 0.7142857142857143
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 20,
+        "total": 28,
+        "rate": 0.7142857142857143
+      }
+    },
+    "operation:missing_value": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:final": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:date_after": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:gt": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 26,
+        "total": 28,
+        "rate": 0.9285714285714286
+      }
+    },
+    "operation:top": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:join_filter": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:join": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:weather": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:search": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      },
+      "arguments_exact": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      }
+    },
+    "operation:ticket": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:read_file": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    }
+  }
+}

evaluation/validation-gpu.summary.json

@@ -0,0 +1,1934 @@
+{
+  "examples": 1200,
+  "seconds": 33.96494126319885,
+  "decoding": "raw greedy, no repair, no teacher fallback",
+  "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
+  "sql_metric": "Compile against the supplied schema, then compare results on two generated SQLite fixtures after dialect adaptation; not native MySQL/PostgreSQL execution.",
+  "metrics": {
+    "json_valid": {
+      "correct": 1200,
+      "total": 1200,
+      "rate": 1.0
+    },
+    "schema_valid": {
+      "correct": 1199,
+      "total": 1200,
+      "rate": 0.9991666666666666
+    },
+    "action_correct": {
+      "correct": 1193,
+      "total": 1200,
+      "rate": 0.9941666666666666
+    },
+    "tool_correct": {
+      "correct": 1103,
+      "total": 1104,
+      "rate": 0.9990942028985508
+    },
+    "arguments_exact": {
+      "correct": 1085,
+      "total": 1104,
+      "rate": 0.9827898550724637
+    },
+    "sql_equivalent": {
+      "correct": 899,
+      "total": 912,
+      "rate": 0.9857456140350878
+    },
+    "success": {
+      "correct": 1174,
+      "total": 1200,
+      "rate": 0.9783333333333334
+    }
+  },
+  "groups": {
+    "language:noisy_en": {
+      "json_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 276,
+        "total": 276,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 275,
+        "total": 276,
+        "rate": 0.9963768115942029
+      },
+      "sql_equivalent": {
+        "correct": 227,
+        "total": 228,
+        "rate": 0.9956140350877193
+      },
+      "success": {
+        "correct": 299,
+        "total": 300,
+        "rate": 0.9966666666666667
+      }
+    },
+    "backend:supabase": {
+      "json_valid": {
+        "correct": 604,
+        "total": 604,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 603,
+        "total": 604,
+        "rate": 0.9983443708609272
+      },
+      "action_correct": {
+        "correct": 601,
+        "total": 604,
+        "rate": 0.9950331125827815
+      },
+      "tool_correct": {
+        "correct": 559,
+        "total": 560,
+        "rate": 0.9982142857142857
+      },
+      "arguments_exact": {
+        "correct": 551,
+        "total": 560,
+        "rate": 0.9839285714285714
+      },
+      "sql_equivalent": {
+        "correct": 461,
+        "total": 468,
+        "rate": 0.9850427350427351
+      },
+      "success": {
+        "correct": 592,
+        "total": 604,
+        "rate": 0.9801324503311258
+      }
+    },
+    "operation:missing_schema": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "language:en": {
+      "json_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 294,
+        "total": 300,
+        "rate": 0.98
+      },
+      "tool_correct": {
+        "correct": 276,
+        "total": 276,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 268,
+        "total": 276,
+        "rate": 0.9710144927536232
+      },
+      "sql_equivalent": {
+        "correct": 222,
+        "total": 228,
+        "rate": 0.9736842105263158
+      },
+      "success": {
+        "correct": 286,
+        "total": 300,
+        "rate": 0.9533333333333334
+      }
+    },
+    "language:hinglish": {
+      "json_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 299,
+        "total": 300,
+        "rate": 0.9966666666666667
+      },
+      "action_correct": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 275,
+        "total": 276,
+        "rate": 0.9963768115942029
+      },
+      "arguments_exact": {
+        "correct": 268,
+        "total": 276,
+        "rate": 0.9710144927536232
+      },
+      "sql_equivalent": {
+        "correct": 223,
+        "total": 228,
+        "rate": 0.9780701754385965
+      },
+      "success": {
+        "correct": 292,
+        "total": 300,
+        "rate": 0.9733333333333334
+      }
+    },
+    "backend:mysql": {
+      "json_valid": {
+        "correct": 596,
+        "total": 596,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 596,
+        "total": 596,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 592,
+        "total": 596,
+        "rate": 0.9932885906040269
+      },
+      "tool_correct": {
+        "correct": 544,
+        "total": 544,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 534,
+        "total": 544,
+        "rate": 0.9816176470588235
+      },
+      "sql_equivalent": {
+        "correct": 438,
+        "total": 444,
+        "rate": 0.9864864864864865
+      },
+      "success": {
+        "correct": 582,
+        "total": 596,
+        "rate": 0.9765100671140939
+      }
+    },
+    "operation:all": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:write": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 18,
+        "total": 24,
+        "rate": 0.75
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 18,
+        "total": 24,
+        "rate": 0.75
+      }
+    },
+    "operation:distinct": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:max": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:min": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:avg": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:group_count": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 25,
+        "total": 28,
+        "rate": 0.8928571428571429
+      },
+      "sql_equivalent": {
+        "correct": 25,
+        "total": 28,
+        "rate": 0.8928571428571429
+      },
+      "success": {
+        "correct": 25,
+        "total": 28,
+        "rate": 0.8928571428571429
+      }
+    },
+    "language:hi": {
+      "json_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 300,
+        "total": 300,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 299,
+        "total": 300,
+        "rate": 0.9966666666666667
+      },
+      "tool_correct": {
+        "correct": 276,
+        "total": 276,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 274,
+        "total": 276,
+        "rate": 0.9927536231884058
+      },
+      "sql_equivalent": {
+        "correct": 227,
+        "total": 228,
+        "rate": 0.9956140350877193
+      },
+      "success": {
+        "correct": 297,
+        "total": 300,
+        "rate": 0.99
+      }
+    },
+    "operation:sum": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 27,
+        "total": 28,
+        "rate": 0.9642857142857143
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 27,
+        "total": 28,
+        "rate": 0.9642857142857143
+      }
+    },
+    "operation:count_eq": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:count": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 22,
+        "total": 28,
+        "rate": 0.7857142857142857
+      },
+      "sql_equivalent": {
+        "correct": 22,
+        "total": 28,
+        "rate": 0.7857142857142857
+      },
+      "success": {
+        "correct": 22,
+        "total": 28,
+        "rate": 0.7857142857142857
+      }
+    },
+    "operation:sql_error": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 20,
+        "total": 24,
+        "rate": 0.8333333333333334
+      }
+    },
+    "operation:project": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      },
+      "success": {
+        "correct": 24,
+        "total": 28,
+        "rate": 0.8571428571428571
+      }
+    },
+    "operation:lt": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:top": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:group_sum": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:sort_desc": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:ambiguous": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:gte": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:eq": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:or": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:month": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:project_eq": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:contains": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:not_null": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:gt": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:lte": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:bottom": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:having": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:sort_asc": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:null": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:year": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:missing_value": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:and": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:date_after": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:final": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      },
+      "tool_correct": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "arguments_exact": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      }
+    },
+    "operation:list_tables": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:describe": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:join": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:join_filter": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:between": {
+      "json_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      },
+      "success": {
+        "correct": 28,
+        "total": 28,
+        "rate": 1.0
+      }
+    },
+    "operation:search": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:weather": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:ticket": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "arguments_exact": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      }
+    },
+    "operation:read_file": {
+      "json_valid": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "schema_valid": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      },
+      "action_correct": {
+        "correct": 24,
+        "total": 24,
+        "rate": 1.0
+      },
+      "tool_correct": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      },
+      "arguments_exact": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      },
+      "sql_equivalent": {
+        "correct": 0,
+        "total": 0,
+        "rate": 0.0
+      },
+      "success": {
+        "correct": 23,
+        "total": 24,
+        "rate": 0.9583333333333334
+      }
+    }
+  },
+  "rescoring": "Same saved raw outputs, with explicit supplied-schema compilation; no generation or output repair."
+}

manifest.json

@@ -8,8 +8,8 @@
       "sha256": "bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a"
     },
     "README.md": {
-      "bytes": 9919,
-      "sha256": "34cd941a5170b840cbfa0450a741561a42b0ee8fae4b9fa26031ffe1026ae5dd"
+      "bytes": 10146,
+      "sha256": "733afbed755bc3a312e38a64d030702d5f9f9c00d6f6911aac00d027b376974d"
     },
     "checkpoint-info.json": {
       "bytes": 144,
@@ -24,8 +24,8 @@
       "sha256": "497a2b76330e203dcbcd4e38fd9b03a1adcad2197e1e6bb545210b34fc3574ad"
     },
     "docs/experiment.md": {
-      "bytes": 5988,
-      "sha256": "1bba1663c1c927ca91299dedcfc200055420677df357829fd5271bd1ba8fe953"
+      "bytes": 6910,
+      "sha256": "e10dad698c4d6c1f71f0c9d45fa952d04178ff586aa2ab5b8bb2848512e147f6"
     },
     "evaluation/baseline-manual.jsonl": {
       "bytes": 64508,
@@ -43,13 +43,29 @@
       "bytes": 38746,
       "sha256": "80bb5910474dc7103c73265d8268f4de9b40209a964b54c5a33d9345d5f0d638"
     },
+    "evaluation/development-gpu.jsonl": {
+      "bytes": 128566,
+      "sha256": "4556fde9515b9836f6df11d0c380a391f785cc377c595da926ab06cd366df240"
+    },
+    "evaluation/development-gpu.summary.json": {
+      "bytes": 34343,
+      "sha256": "d2cebb68d2ca1e3f091949679100b5ba6a0083a9a31a8f9a1de7f9c28c715b53"
+    },
     "evaluation/development.summary.json": {
       "bytes": 34207,
       "sha256": "d9b17d2ab010df858674362a4cdf5275939d8728b594769637fa5b7954a1f2cd"
     },
     "evaluation/gpu-final-console-metrics.json": {
-      "bytes": 2062,
-      "sha256": "1760d2862fe481bb65d8bfad8110541d315e740ec1546996e777b547f4291b6b"
+      "bytes": 2031,
+      "sha256": "12484a15a5595d98390aff8be81e9f87e84adba0131750e06cc4588538c552a3"
+    },
+    "evaluation/manual-gpu.jsonl": {
+      "bytes": 98789,
+      "sha256": "6c028d9dbf90bea9dcd7daf8c4f529b149deaaabb780edcd764d391783865263"
+    },
+    "evaluation/manual-gpu.summary.json": {
+      "bytes": 19603,
+      "sha256": "4b8461ebf38ae24517aca7e5df23cee14f6b910b5d21418364632fbd8c328c1a"
     },
     "evaluation/manual-mac.jsonl": {
       "bytes": 98789,
@@ -59,6 +75,22 @@
       "bytes": 19604,
       "sha256": "ecb96cfa02aa55a196e55b3bb9bbb331729d5cc8d0a79b6dd5bfbba958cef75b"
     },
+    "evaluation/native-manual.json": {
+      "bytes": 21143,
+      "sha256": "09554ea11c7e3cd50ffef194e34147796abce34dce15b83596cab395f0467bd5"
+    },
+    "evaluation/native-test.json": {
+      "bytes": 162870,
+      "sha256": "0797678ec9dd2fc06b7929a8a1a7f925117896e61e332f771aed05708c734471"
+    },
+    "evaluation/test-gpu.jsonl": {
+      "bytes": 794895,
+      "sha256": "477f00136e84f82881ac57e34d6e71e4e3f769ff4c952c94d236630eafd409b4"
+    },
+    "evaluation/test-gpu.summary.json": {
+      "bytes": 38590,
+      "sha256": "934b47f477de840cf5ceb9c009e81862896d8f28e493ce91e305831564602202"
+    },
     "evaluation/test-mac.jsonl": {
       "bytes": 794897,
       "sha256": "6006020aefb90eecd3fa282cf3fceb445e32fc27bfe11bd967ce8cd7a86702d8"
@@ -67,6 +99,14 @@
       "bytes": 38589,
       "sha256": "17b685763d7a20d3d3256a6264eacf3f569e461cf46eb68ee9160e04ecbe8817"
     },
+    "evaluation/validation-gpu.jsonl": {
+      "bytes": 783508,
+      "sha256": "e62d00f7ae8dc6bf5f8e1339d92e1be4d40cea154e8114f3fbe7e09fd807c167"
+    },
+    "evaluation/validation-gpu.summary.json": {
+      "bytes": 38217,
+      "sha256": "18947ffabb38cb5494b24f163d82dcb00c1ebd3f885dc96412add9490b5942ef"
+    },
     "evaluation/validation.summary.json": {
       "bytes": 38081,
       "sha256": "86a6a6e36522decadd3e93be9bcdc9435c9f619fd16290cfa78542001358d354"
@@ -87,18 +127,34 @@
       "bytes": 279483882,
       "sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692"
     },
+    "provenance/benchmark-mtp-3.json": {
+      "bytes": 2239,
+      "sha256": "8be554d736f09b95c08f9a601146debf73085266bb6beca16e0ff9379e1c59f4"
+    },
+    "provenance/benchmark-mtp-off.json": {
+      "bytes": 736,
+      "sha256": "279daa97891484155c69f3a8ed815cf32d9491b2c490a71e834c4575d14ceb9e"
+    },
     "provenance/experiment-provenance.json": {
-      "bytes": 5894,
-      "sha256": "490b8844864ba2d8116ff898ec7a010ca31f40e900863a317f4b452dcdf2d836"
+      "bytes": 6032,
+      "sha256": "74e6d89c5a350acc2b3707ea1eaaca54883203f138b21c99e3f576ae9f09d5bd"
     },
     "provenance/full-audit.json": {
       "bytes": 363,
       "sha256": "2616ce8641587452aafaf88f730bd4a9546c5cd5fdd18097e3d2bf9e6c42dec6"
     },
+    "provenance/metrics.jsonl": {
+      "bytes": 89425,
+      "sha256": "c20ce670673a27fa870e70c670d5a883e4f034814ee8853c987271fc8a65fe32"
+    },
     "provenance/native-reference-validation.json": {
       "bytes": 11624,
       "sha256": "41289ff92629e67bc77f5e1790cc6e638254d0ab516fd42246fe520e228bb766"
     },
+    "provenance/selection-history.json": {
+      "bytes": 4664,
+      "sha256": "3747d4ddda2c5646e1b17136fd58b6995c558eb3d001e469f7c4319678607be9"
+    },
     "provenance/selection.json": {
       "bytes": 552,
       "sha256": "87972e47a26951aed4e2ac350da3a72c2fc054e1e70649c16f06ff270d724e27"
@@ -111,13 +167,21 @@
       "bytes": 1576,
       "sha256": "f0e55ba2ce0805e04ec3369e4a4a35bc9ff006d10367753b23502e34805635b4"
     },
+    "provenance/train-copy.log": {
+      "bytes": 109106,
+      "sha256": "c9d89785613133d7f9596f53c6245493e985ae52b7b0df7e986c180b3a6cf314"
+    },
+    "provenance/training-summary.json": {
+      "bytes": 306,
+      "sha256": "d8ffdd12931cacaa63b3b7606c140439ccd3a18423ec5f465cbf1bd6e5529316"
+    },
     "requirements.txt": {
       "bytes": 188,
       "sha256": "4d2963280263d14196c338b3a20d098d6d5853a70c410504ba717c8b81decaf1"
     },
     "runtime/README.md": {
-      "bytes": 1775,
-      "sha256": "a4f8b516d2da87f9c6313ca258680e418572a1ce607ee8d7c847950159498204"
+      "bytes": 1841,
+      "sha256": "d20ce9bb4a31dbb8e6b579588197489e9b834fc994ff05f6797b7630b8d9e965"
     },
     "runtime/download-teacher.py": {
       "bytes": 193,
@@ -151,6 +215,30 @@
       "bytes": 80,
       "sha256": "83d8a6e71e625d98eef78ce40796f522c4be10aa149deab3bc2c1849b6414ebc"
     },
+    "tinyquery/__pycache__/__init__.cpython-312.pyc": {
+      "bytes": 262,
+      "sha256": "4996088dfa6f2f7378b55b528d2049bca0a581d06be8588ea69cdd7d2e563672"
+    },
+    "tinyquery/__pycache__/chat.cpython-312.pyc": {
+      "bytes": 6701,
+      "sha256": "a13c782a1784ddda2f1002bfdfc32546adf918a90248729bba385400e7a50f11"
+    },
+    "tinyquery/__pycache__/data.cpython-312.pyc": {
+      "bytes": 25002,
+      "sha256": "278ea52498f7fb54658dcc9d8789b48c682fb01fc245c22d625034a335a19a13"
+    },
+    "tinyquery/__pycache__/evaluate.cpython-312.pyc": {
+      "bytes": 16313,
+      "sha256": "9f07677ccd818161a4e96f9353ebafe6c372f6a6c63cbff82b4b7423ed751484"
+    },
+    "tinyquery/__pycache__/model.cpython-312.pyc": {
+      "bytes": 19668,
+      "sha256": "3d0e1ee937d05bbe0a26d2af473774a899e0144f179c14782d937538a0f13492"
+    },
+    "tinyquery/__pycache__/recipes.cpython-312.pyc": {
+      "bytes": 4094,
+      "sha256": "da737efd648a3aa18d7541696ff11e6edfe95c56b0b2f869c12fd61e8d62b060"
+    },
     "tinyquery/audit_data.py": {
       "bytes": 3526,
       "sha256": "b7bf574f8f1ea4c34de3b80c0fc98fd98272161c13ce82dafe70c4bea1bd9948"

provenance/benchmark-mtp-3.json

@@ -0,0 +1,50 @@
+{
+  "label": "mtp-3",
+  "workers": 32,
+  "rounds": [
+    {
+      "round": 0,
+      "warmup": true,
+      "seconds": 12.065198183059692,
+      "tokens": 7120,
+      "json_valid": 64,
+      "requests": 64,
+      "truncated": 0,
+      "output_tps": 590.1270656288874
+    },
+    {
+      "round": 1,
+      "warmup": false,
+      "seconds": 7.756687641143799,
+      "tokens": 7186,
+      "json_valid": 64,
+      "requests": 64,
+      "truncated": 0,
+      "output_tps": 926.4263732734704
+    },
+    {
+      "round": 2,
+      "warmup": false,
+      "seconds": 7.630668878555298,
+      "tokens": 7136,
+      "json_valid": 64,
+      "requests": 64,
+      "truncated": 0,
+      "output_tps": 935.173588786498
+    }
+  ],
+  "mtp_metrics": [
+    "vllm:spec_decode_num_drafts_total{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\"} 7724.0",
+    "vllm:spec_decode_num_drafts_created{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\"} 1.7890392532883267e+09",
+    "vllm:spec_decode_num_draft_tokens_total{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\"} 23013.0",
+    "vllm:spec_decode_num_draft_tokens_created{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\"} 1.7890392532883449e+09",
+    "vllm:spec_decode_num_accepted_tokens_total{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\"} 13831.0",
+    "vllm:spec_decode_num_accepted_tokens_created{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\"} 1.7890392532883563e+09",
+    "vllm:spec_decode_num_accepted_tokens_per_pos_total{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\",position=\"0\"} 6169.0",
+    "vllm:spec_decode_num_accepted_tokens_per_pos_total{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\",position=\"1\"} 4549.0",
+    "vllm:spec_decode_num_accepted_tokens_per_pos_total{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\",position=\"2\"} 3113.0",
+    "vllm:spec_decode_num_accepted_tokens_per_pos_created{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\",position=\"0\"} 1.789039253288371e+09",
+    "vllm:spec_decode_num_accepted_tokens_per_pos_created{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\",position=\"1\"} 1.789039253288375e+09",
+    "vllm:spec_decode_num_accepted_tokens_per_pos_created{engine=\"0\",model_name=\"Qwen/Qwen3.8-27B-FP8\",position=\"2\"} 1.7890392532883773e+09"
+  ]
+}

provenance/benchmark-mtp-off.json

@@ -0,0 +1,37 @@
+{
+  "label": "mtp-off",
+  "workers": 32,
+  "rounds": [
+    {
+      "round": 0,
+      "warmup": true,
+      "seconds": 11.006732702255249,
+      "tokens": 7268,
+      "json_valid": 64,
+      "requests": 64,
+      "truncated": 0,
+      "output_tps": 660.3231128262802
+    },
+    {
+      "round": 1,
+      "warmup": false,
+      "seconds": 10.474723815917969,
+      "tokens": 7215,
+      "json_valid": 64,
+      "requests": 64,
+      "truncated": 0,
+      "output_tps": 688.8009771709386
+    },
+    {
+      "round": 2,
+      "warmup": false,
+      "seconds": 10.340507745742798,
+      "tokens": 7172,
+      "json_valid": 64,
+      "requests": 64,
+      "truncated": 0,
+      "output_tps": 693.5829628823326
+    }
+  ],
+  "mtp_metrics": []
+}

provenance/experiment-provenance.json

@@ -87,7 +87,8 @@
     "364753b",
     "a8f336f",
     "1049d88",
-    "9729c63"
+    "9729c63",
+    "369909b"
   ],
   "architecture_reference": "https://aclanthology.org/P17-1099/",
   "research_claim": "Established decoder components and a learned pointer-generator adaptation. No claim of a new research architecture or controlled ablation.",
@@ -139,5 +140,5 @@
     "training_seconds": 7099.562133073807,
     "note": "Retained-lineage trainer counters include validation/checkpoint time, exclude discarded work/model loading. Last checkpoint was not selected."
   },
-  "release_archive_limitation": "Vast.ai SSH refused connections after final evaluation completed; local selected weights and datasets were already secured. Some raw remote logs and final optimizer state were not retrieved."
+  "release_archive_limitation": "SSH briefly refused connections, then recovered. Runtime configs, training logs, candidate/selection records, GPU predictions and native evaluation reports were retrieved. The optional 1.67 GB final optimizer-state transfer was stopped to avoid extending rental cost; selected inference weights are fully secured."
 }

provenance/metrics.jsonl

@@ -0,0 +1,304 @@
+{"step": 1, "elapsed_seconds": 908.2484879493713, "processed_tokens": 111921922, "response_tokens": 13480483, "tokens_per_second": 30925.48579768987, "loss": [0.43154871463775635, 0.1907770037651062, 6.21791768935509e-05], "lr": 2.9999996689331573e-06, "gradient_norm": 0.8633310198783875}
+{"step": 2, "elapsed_seconds": 908.3439321517944, "processed_tokens": 111936905, "response_tokens": 13482285, "tokens_per_second": 46757.350974246096, "loss": [0.19502803683280945, 0.12592434883117676, 7.078047303821222e-08], "lr": 5.999998170795498e-06, "gradient_norm": 0.8814497590065002}
+{"step": 3, "elapsed_seconds": 908.52170753479, "processed_tokens": 111956260, "response_tokens": 13484788, "tokens_per_second": 58533.460380063174, "loss": [0.36421141028404236, 0.15221402049064636, 1.8626450382086546e-09], "lr": 8.999996411988006e-06, "gradient_norm": 0.8392025828361511}
+{"step": 136, "elapsed_seconds": 928.6177999973297, "processed_tokens": 114357741, "response_tokens": 13783910, "tokens_per_second": 116781.91946072895, "loss": [0.08445247262716293, 0.007911769673228264, 1.396977609147143e-07], "lr": 0.0002999099337280665, "gradient_norm": 0.05469126254320145}
+{"step": 270, "elapsed_seconds": 948.6810350418091, "processed_tokens": 116805531, "response_tokens": 14080438, "tokens_per_second": 119331.17560235807, "loss": [0.22031529247760773, 0.013450083322823048, 9.68559447755979e-07], "lr": 0.0002996558317248182, "gradient_norm": 0.08413375914096832}
+{"step": 404, "elapsed_seconds": 968.7553837299347, "processed_tokens": 119222245, "response_tokens": 14380526, "tokens_per_second": 119678.04314001724, "loss": [0.2673477232456207, 0.04301092401146889, 4.7496948241132486e-07], "lr": 0.00029923574251178997, "gradient_norm": 0.08704562485218048}
+{"step": 534, "elapsed_seconds": 988.7762727737427, "processed_tokens": 121601554, "response_tokens": 14676912, "tokens_per_second": 119471.72030251368, "loss": [0.07043754309415817, 0.001097075641155243, 2.980229751869956e-08], "lr": 0.0002986502508459204, "gradient_norm": 0.05524539202451706}
+{"step": 788, "elapsed_seconds": 1028.010220527649, "processed_tokens": 126253699, "response_tokens": 15255205, "tokens_per_second": 116673.95298143622, "loss": [0.09163491427898407, 0.0024850654881447554, 1.4901155864777138e-08], "lr": 0.0002999182348101882, "gradient_norm": 0.04540788009762764}
+{"step": 915, "elapsed_seconds": 1048.0823822021484, "processed_tokens": 128638698, "response_tokens": 15554584, "tokens_per_second": 117748.29867214349, "loss": [0.0983884334564209, 0.003186871763318777, 1.3038463464454253e-07], "lr": 0.0002996708999583449, "gradient_norm": 0.04962802678346634}
+{"step": 1045, "elapsed_seconds": 1068.1423346996307, "processed_tokens": 131057909, "response_tokens": 15859911, "tokens_per_second": 118698.57633933856, "loss": [0.10281457006931305, 0.00828690268099308, 0.0], "lr": 0.0002992586253729498, "gradient_norm": 0.0603363923728466}
+{"step": 1175, "elapsed_seconds": 1088.1931228637695, "processed_tokens": 133434275, "response_tokens": 16169606, "tokens_per_second": 118653.28070646364, "loss": [0.266403466463089, 0.008265187963843346, 2.905699432176334e-07], "lr": 0.0002986842882743974, "gradient_norm": 0.0626416876912117}
+{"step": 1297, "elapsed_seconds": 1108.3101432323456, "processed_tokens": 135740155, "response_tokens": 16471921, "tokens_per_second": 117845.36914215353, "loss": [0.3157045841217041, 0.010232290253043175, 1.1175869119028903e-08], "lr": 0.0002979423147331015, "gradient_norm": 0.061665356159210205}
+{"step": 1425, "elapsed_seconds": 1129.0470764636993, "processed_tokens": 138113077, "response_tokens": 16771713, "tokens_per_second": 117260.40051373314, "loss": [0.08619260787963867, 0.00023162926663644612, 5.587930473893721e-08], "lr": 0.00029700125585310603, "gradient_norm": 0.04804946109652519}
+{"step": 1554, "elapsed_seconds": 1149.130744934082, "processed_tokens": 140495486, "response_tokens": 17074498, "tokens_per_second": 117454.42690566128, "loss": [0.09976951032876968, 0.0023476441856473684, 4.263060418452369e-06], "lr": 0.000295929120613645, "gradient_norm": 0.049311667680740356}
+{"step": 1682, "elapsed_seconds": 1169.2621245384216, "processed_tokens": 142845349, "response_tokens": 17377731, "tokens_per_second": 117363.55980433545, "loss": [0.2818354070186615, 0.015788715332746506, 0.0], "lr": 0.00029469700029196097, "gradient_norm": 0.07585964351892471}
+{"step": 1811, "elapsed_seconds": 1189.271056175232, "processed_tokens": 145195772, "response_tokens": 17679786, "tokens_per_second": 117375.16200128265, "loss": [0.09708361327648163, 0.0008734281873330474, 3.3527591369875154e-08], "lr": 0.00029330476754836697, "gradient_norm": 0.04572117701172829}
+{"step": 1930, "elapsed_seconds": 1209.2869544029236, "processed_tokens": 147386391, "response_tokens": 17955093, "tokens_per_second": 116586.62482696884, "loss": [0.27031394839286804, 0.0016993613680824637, 2.9885561161790974e-05], "lr": 0.0002917656172899569, "gradient_norm": 0.051316529512405396}
+{"step": 2061, "elapsed_seconds": 1229.3983464241028, "processed_tokens": 149782537, "response_tokens": 18262895, "tokens_per_second": 116818.86875669625, "loss": [0.3077086806297302, 0.009644619189202785, 2.2351732908987287e-08], "lr": 0.00029005806168537734, "gradient_norm": 0.08095907419919968}
+{"step": 2191, "elapsed_seconds": 1249.8892385959625, "processed_tokens": 152138341, "response_tokens": 18570256, "tokens_per_second": 116662.13062370488, "loss": [0.2984667718410492, 0.005875740200281143, 2.0116503662848118e-07], "lr": 0.0002881591808395177, "gradient_norm": 0.0533854179084301}
+{"step": 2318, "elapsed_seconds": 1269.9646394252777, "processed_tokens": 154512344, "response_tokens": 18873231, "tokens_per_second": 116784.13039539033, "loss": [0.10047124326229095, 0.0024204121436923742, 1.8085768260789337e-06], "lr": 0.00028614064398193033, "gradient_norm": 0.05580931901931763}
+{"step": 2448, "elapsed_seconds": 1290.0944080352783, "processed_tokens": 156901323, "response_tokens": 19182890, "tokens_per_second": 116919.32154965083, "loss": [0.28015828132629395, 0.0007699052221141756, 4.284056558390148e-07], "lr": 0.00028398038470290677, "gradient_norm": 0.05512872710824013}
+{"step": 2585, "elapsed_seconds": 1310.122296333313, "processed_tokens": 159320772, "response_tokens": 19494598, "tokens_per_second": 117176.80909492115, "loss": [0.2875961661338806, 0.004762003663927317, 6.51924452199637e-08], "lr": 0.00028167904474581427, "gradient_norm": 0.05465236306190491}
+{"step": 2715, "elapsed_seconds": 1330.2994005680084, "processed_tokens": 161708774, "response_tokens": 19800473, "tokens_per_second": 117250.3763802899, "loss": [0.26748335361480713, 0.008260427042841911, 3.7252898543727042e-09], "lr": 0.00027921632232328925, "gradient_norm": 0.06223592907190323}
+{"step": 2849, "elapsed_seconds": 1350.415631532669, "processed_tokens": 164108852, "response_tokens": 20111980, "tokens_per_second": 117371.3931776106, "loss": [0.26597264409065247, 0.0007941116346046329, 1.3038494728334626e-07], "lr": 0.0002766193585413358, "gradient_norm": 0.05902467668056488}
+{"step": 2978, "elapsed_seconds": 1370.661864042282, "processed_tokens": 166459377, "response_tokens": 20408854, "tokens_per_second": 117300.24981237436, "loss": [0.2706957459449768, 0.002877524821087718, 1.8626450382086546e-09], "lr": 0.0002738629475938887, "gradient_norm": 0.05402400344610214}
+{"step": 3111, "elapsed_seconds": 1390.6675860881805, "processed_tokens": 168870626, "response_tokens": 20715465, "tokens_per_second": 117468.97798919906, "loss": [0.2813107371330261, 0.0008774700108915567, 4.195980636723107e-06], "lr": 0.00027101308208978106, "gradient_norm": 0.044036637991666794}
+{"step": 3242, "elapsed_seconds": 1410.8604300022125, "processed_tokens": 171258992, "response_tokens": 21021097, "tokens_per_second": 117509.51763861638, "loss": [0.27832743525505066, 0.003987755160778761, 3.725286745748235e-08], "lr": 0.0002680055036512554, "gradient_norm": 0.05285501107573509}
+{"step": 3374, "elapsed_seconds": 1430.9613463878632, "processed_tokens": 173666022, "response_tokens": 21326311, "tokens_per_second": 117615.85650147896, "loss": [0.09205362200737, 0.00010057290637632832, 1.1659727761070826e-06], "lr": 0.0002648671157900202, "gradient_norm": 0.03703455254435539}
+{"step": 3505, "elapsed_seconds": 1451.0809042453766, "processed_tokens": 176065984, "response_tokens": 21633743, "tokens_per_second": 117691.64463363726, "loss": [0.2992277145385742, 0.005535179283469915, 1.4901154088420299e-08], "lr": 0.00026162770481203817, "gradient_norm": 0.05833858624100685}
+{"step": 3632, "elapsed_seconds": 1471.0946264266968, "processed_tokens": 178397992, "response_tokens": 21936565, "tokens_per_second": 117641.03277810087, "loss": [0.26216834783554077, 0.0004423234495334327, 1.8626450382086546e-09], "lr": 0.0002582794720171344, "gradient_norm": 0.0500967837870121}
+{"step": 3761, "elapsed_seconds": 1491.5184891223907, "processed_tokens": 180749226, "response_tokens": 22237866, "tokens_per_second": 117534.63247078515, "loss": [0.25183528661727905, 0.00048521210555918515, 9.499464681539393e-08], "lr": 0.00025473736262469996, "gradient_norm": 0.04670266434550285}
+{"step": 3895, "elapsed_seconds": 1511.6742498874664, "processed_tokens": 183170519, "response_tokens": 22543536, "tokens_per_second": 117638.44832647493, "loss": [0.24834363162517548, 0.002185449004173279, 0.0], "lr": 0.0002511375600137569, "gradient_norm": 0.050547804683446884}
+{"step": 4027, "elapsed_seconds": 1531.7221865653992, "processed_tokens": 185551590, "response_tokens": 22853137, "tokens_per_second": 117681.71801868686, "loss": [0.2794586718082428, 0.0034163668751716614, 6.70550832637673e-08], "lr": 0.0002474455353810554, "gradient_norm": 0.04923274740576744}
+{"step": 4157, "elapsed_seconds": 1551.748538017273, "processed_tokens": 187940056, "response_tokens": 23155051, "tokens_per_second": 117740.06940270188, "loss": [0.2736979126930237, 0.00017098846728913486, 7.301446203200612e-07], "lr": 0.00024366357402404165, "gradient_norm": 0.04821041226387024}
+{"step": 4289, "elapsed_seconds": 1571.8609895706177, "processed_tokens": 190328651, "response_tokens": 23462652, "tokens_per_second": 117776.51856661966, "loss": [0.26847532391548157, 0.0025758720003068447, 0.0], "lr": 0.000239761046643582, "gradient_norm": 0.055366434156894684}
+{"step": 4423, "elapsed_seconds": 1591.9300634860992, "processed_tokens": 192762274, "response_tokens": 23770192, "tokens_per_second": 117896.31558716035, "loss": [0.2792264223098755, 0.0031546554528176785, 8.009353535953778e-08], "lr": 0.00023577904716540795, "gradient_norm": 0.055576588958501816}
+{"step": 4550, "elapsed_seconds": 1612.3580453395844, "processed_tokens": 195116030, "response_tokens": 24067011, "tokens_per_second": 117805.93111960877, "loss": [0.27011430263519287, 0.0002560628636274487, 9.313223081619526e-09], "lr": 0.0002316368238369402, "gradient_norm": 0.04241878166794777}
+{"step": 4767, "elapsed_seconds": 1645.8650951385498, "processed_tokens": 199061994, "response_tokens": 24580979, "tokens_per_second": 116578.17330665492, "loss": [0.14532282948493958, 0.00041398772737011313, 0.0], "lr": 0.0002999177657254007, "gradient_norm": 0.047247763723134995}
+{"step": 4898, "elapsed_seconds": 1665.9876363277435, "processed_tokens": 201483392, "response_tokens": 24893273, "tokens_per_second": 118456.33088505811, "loss": [0.3268917500972748, 0.010998659767210484, 0.0], "lr": 0.0002996702474936799, "gradient_norm": 0.06249326094985008}
+{"step": 5028, "elapsed_seconds": 1686.046616077423, "processed_tokens": 203877141, "response_tokens": 25208371, "tokens_per_second": 118748.87717905224, "loss": [0.16663630306720734, 0.002920327475294471, 0.0], "lr": 0.0002992560045775427, "gradient_norm": 0.05671392381191254}
+{"step": 5156, "elapsed_seconds": 1706.058125257492, "processed_tokens": 206250335, "response_tokens": 25521041, "tokens_per_second": 118709.6440232123, "loss": [0.35926228761672974, 0.007246624678373337, 1.8626450382086546e-09], "lr": 0.00029868214520507375, "gradient_norm": 0.06628864258527756}
+{"step": 5283, "elapsed_seconds": 1726.1327192783356, "processed_tokens": 208617885, "response_tokens": 25831544, "tokens_per_second": 118555.23637256343, "loss": [0.4009239673614502, 0.014073017984628677, 0.0], "lr": 0.0002979409298508813, "gradient_norm": 0.07418548315763474}
+{"step": 5413, "elapsed_seconds": 1746.820888519287, "processed_tokens": 210984117, "response_tokens": 26144003, "tokens_per_second": 117841.04476365822, "loss": [0.13545377552509308, 0.00012025396426906809, 1.7470655393481138e-06], "lr": 0.00029700247483078443, "gradient_norm": 0.045806292444467545}
+{"step": 5541, "elapsed_seconds": 1766.983329296112, "processed_tokens": 213364686, "response_tokens": 26458099, "tokens_per_second": 117873.65976841848, "loss": [0.37787389755249023, 0.009619530290365219, 0.0], "lr": 0.0002959298300934946, "gradient_norm": 0.056080032140016556}
+{"step": 5670, "elapsed_seconds": 1787.0217871665955, "processed_tokens": 215742162, "response_tokens": 26770897, "tokens_per_second": 117969.59003797242, "loss": [0.38174042105674744, 0.010644424706697464, 0.0], "lr": 0.0002946995203165499, "gradient_norm": 0.12056843936443329}
+{"step": 5796, "elapsed_seconds": 1807.0651586055756, "processed_tokens": 218097991, "response_tokens": 27081122, "tokens_per_second": 117921.7180553564, "loss": [0.38244765996932983, 0.0007910053827799857, 0.0], "lr": 0.0002933097581731487, "gradient_norm": 0.053494520485401154}
+{"step": 5917, "elapsed_seconds": 1827.1786758899689, "processed_tokens": 220332581, "response_tokens": 27375106, "tokens_per_second": 117240.38945640372, "loss": [0.3549908697605133, 0.0058676619082689285, 5.4016616957142105e-08], "lr": 0.0002917598541090923, "gradient_norm": 0.05624143406748772}
+{"step": 6044, "elapsed_seconds": 1847.2689304351807, "processed_tokens": 222693848, "response_tokens": 27687821, "tokens_per_second": 117266.92474119355, "loss": [0.3396281898021698, 0.007427466567605734, 6.388772817444988e-07], "lr": 0.00029005192785356394, "gradient_norm": 0.06228146329522133}
+{"step": 6173, "elapsed_seconds": 1867.6035182476044, "processed_tokens": 225032423, "response_tokens": 27996554, "tokens_per_second": 117076.71878210567, "loss": [0.3827144205570221, 0.011151675134897232, 3.48310749131997e-07], "lr": 0.00028816681189294097, "gradient_norm": 0.08065677434206009}
+{"step": 6301, "elapsed_seconds": 1887.6378037929535, "processed_tokens": 227409456, "response_tokens": 28306613, "tokens_per_second": 117196.94623400598, "loss": [0.3533063232898712, 0.006844637915492058, 0.0], "lr": 0.00028616084163567184, "gradient_norm": 0.06662917882204056}
+{"step": 6429, "elapsed_seconds": 1907.8066427707672, "processed_tokens": 229786396, "response_tokens": 28619260, "tokens_per_second": 117243.79588151429, "loss": [0.3964789807796478, 0.007705010939389467, 1.8626450382086546e-09], "lr": 0.0002839922888212388, "gradient_norm": 0.07494580000638962}
+{"step": 6557, "elapsed_seconds": 1927.879487991333, "processed_tokens": 232155916, "response_tokens": 28929842, "tokens_per_second": 117297.0978791339, "loss": [0.35521742701530457, 0.009383480064570904, 0.0], "lr": 0.0002816852796528642, "gradient_norm": 0.055859241634607315}
+{"step": 6685, "elapsed_seconds": 1948.0063705444336, "processed_tokens": 234523050, "response_tokens": 29241969, "tokens_per_second": 117316.6764971706, "loss": [0.3620480000972748, 0.0033410429023206234, 1.8626450382086546e-09], "lr": 0.0002792284327294236, "gradient_norm": 0.06150844320654869}
+{"step": 6816, "elapsed_seconds": 1968.153749704361, "processed_tokens": 236920511, "response_tokens": 29556155, "tokens_per_second": 117415.50334926312, "loss": [0.3379306495189667, 0.003458258928731084, 0.0], "lr": 0.0002766271152302419, "gradient_norm": 0.050736475735902786}
+{"step": 6942, "elapsed_seconds": 1988.3171072006226, "processed_tokens": 239237580, "response_tokens": 29859072, "tokens_per_second": 117276.42971369765, "loss": [0.12462294101715088, 0.00011077925591962412, 1.1175867342672063e-08], "lr": 0.0002738754541031938, "gradient_norm": 0.040205758064985275}
+{"step": 7073, "elapsed_seconds": 2008.4479353427887, "processed_tokens": 241639242, "response_tokens": 30173377, "tokens_per_second": 117383.0195080555, "loss": [0.31955355405807495, 0.003559374250471592, 0.0], "lr": 0.0002710156636742634, "gradient_norm": 0.04536367580294609}
+{"step": 7197, "elapsed_seconds": 2028.619395017624, "processed_tokens": 243992128, "response_tokens": 30479243, "tokens_per_second": 117346.03156170655, "loss": [0.36253753304481506, 0.002095017349347472, 1.8626450382086546e-09], "lr": 0.00026800899441035286, "gradient_norm": 0.05346358194947243}
+{"step": 7330, "elapsed_seconds": 2048.653178691864, "processed_tokens": 246397622, "response_tokens": 30793313, "tokens_per_second": 117475.16423790339, "loss": [0.3594958782196045, 0.005901900120079517, 0.0], "lr": 0.0002648953935553451, "gradient_norm": 0.15600541234016418}
+{"step": 7459, "elapsed_seconds": 2068.7360208034515, "processed_tokens": 248805442, "response_tokens": 31105310, "tokens_per_second": 117584.84311539511, "loss": [0.3318100571632385, 0.00833502784371376, 1.8626444386882213e-08], "lr": 0.0002616513781907853, "gradient_norm": 0.061275504529476166}
+{"step": 7588, "elapsed_seconds": 2088.8151154518127, "processed_tokens": 251187722, "response_tokens": 31416510, "tokens_per_second": 117630.80508819473, "loss": [0.35557639598846436, 0.007486994378268719, 0.0], "lr": 0.0002582903463314305, "gradient_norm": 0.05559141933917999}
+{"step": 7713, "elapsed_seconds": 2109.0381541252136, "processed_tokens": 253502086, "response_tokens": 31719730, "tokens_per_second": 117497.36501751267, "loss": [0.135602205991745, 0.0015471024671569467, 9.313223081619526e-09], "lr": 0.0002547740635557811, "gradient_norm": 0.04395522549748421}
+{"step": 7841, "elapsed_seconds": 2129.092943429947, "processed_tokens": 255869062, "response_tokens": 32032566, "tokens_per_second": 117518.40706558597, "loss": [0.3554154336452484, 0.00033600645838305354, 5.587935003603661e-09], "lr": 0.00025120539853913307, "gradient_norm": 0.046769265085458755}
+{"step": 7970, "elapsed_seconds": 2149.2316386699677, "processed_tokens": 258270621, "response_tokens": 32344890, "tokens_per_second": 117585.06166068888, "loss": [0.35058891773223877, 0.004499384667724371, 1.8253815881053015e-07], "lr": 0.00024750138987410974, "gradient_norm": 0.05050305649638176}
+{"step": 8097, "elapsed_seconds": 2169.338177919388, "processed_tokens": 260630843, "response_tokens": 32656831, "tokens_per_second": 117577.69073052052, "loss": [0.31841108202934265, 0.0016580544179305434, 3.9115498395858594e-08], "lr": 0.00024370256647638793, "gradient_norm": 0.051755983382463455}
+{"step": 8227, "elapsed_seconds": 2189.3388912677765, "processed_tokens": 263011422, "response_tokens": 32969498, "tokens_per_second": 117629.04369439556, "loss": [0.1419525295495987, 0.00011493146303109825, 5.718223405892786e-07], "lr": 0.00023981123801673825, "gradient_norm": 0.04120296239852905}
+{"step": 8361, "elapsed_seconds": 2209.427053451538, "processed_tokens": 265435780, "response_tokens": 33284986, "tokens_per_second": 117734.2525952235, "loss": [0.14495034515857697, 0.000496257736813277, 1.3224742190232064e-07], "lr": 0.0002358262532955506, "gradient_norm": 0.06064887344837189}
+{"step": 8486, "elapsed_seconds": 2229.935518026352, "processed_tokens": 267773496, "response_tokens": 33586520, "tokens_per_second": 117607.08245940911, "loss": [0.35751739144325256, 0.009265732020139694, 0.0], "lr": 0.0002316790983851465, "gradient_norm": 0.05534428730607033}
+{"step": 8612, "elapsed_seconds": 2249.9517228603363, "processed_tokens": 270115879, "response_tokens": 33895680, "tokens_per_second": 117588.39508964485, "loss": [0.3401360809803009, 0.0023628128692507744, 9.313223081619526e-09], "lr": 0.00022754136596899618, "gradient_norm": 0.050316229462623596}
+{"step": 8740, "elapsed_seconds": 2269.987391471863, "processed_tokens": 272479315, "response_tokens": 34207100, "tokens_per_second": 117599.99644966118, "loss": [0.35770103335380554, 0.0006236000335775316, 0.0], "lr": 0.00022332183538755626, "gradient_norm": 0.04810153692960739}
+{"step": 8868, "elapsed_seconds": 2289.999512910843, "processed_tokens": 274842889, "response_tokens": 34519254, "tokens_per_second": 117615.2750197661, "loss": [0.3250817358493805, 0.0011209429940208793, 0.0], "lr": 0.0002190377543761448, "gradient_norm": 0.04471077397465706}
+{"step": 8993, "elapsed_seconds": 2310.0848309993744, "processed_tokens": 277189183, "response_tokens": 34827176, "tokens_per_second": 117591.8266927189, "loss": [0.327193021774292, 0.005083609372377396, 0.0], "lr": 0.0002146639000043973, "gradient_norm": 0.06383536010980606}
+{"step": 9121, "elapsed_seconds": 2330.2197782993317, "processed_tokens": 279564039, "response_tokens": 35140112, "tokens_per_second": 117601.97744017666, "loss": [0.33953702449798584, 0.003978651482611895, 0.0], "lr": 0.00021022106357210127, "gradient_norm": 0.048339251428842545}
+{"step": 9248, "elapsed_seconds": 2350.574952363968, "processed_tokens": 281907552, "response_tokens": 35447374, "tokens_per_second": 117532.58606893636, "loss": [0.16310127079486847, 0.0002783340460155159, 1.8812613689078717e-07], "lr": 0.00020565845235638472, "gradient_norm": 0.045475419610738754}
+{"step": 9376, "elapsed_seconds": 2370.6155536174774, "processed_tokens": 284269008, "response_tokens": 35756882, "tokens_per_second": 117540.68472584446, "loss": [0.14810840785503387, 0.00017743295757099986, 1.8626450382086546e-09], "lr": 0.00020113406418190808, "gradient_norm": 0.06419546157121658}
+{"step": 9507, "elapsed_seconds": 2390.636164665222, "processed_tokens": 286666503, "response_tokens": 36069223, "tokens_per_second": 117598.54876041494, "loss": [0.3360541760921478, 0.0024485026951879263, 0.0], "lr": 0.00019658675621805338, "gradient_norm": 0.05487117916345596}
+{"step": 9634, "elapsed_seconds": 2410.698823451996, "processed_tokens": 289049621, "response_tokens": 36377114, "tokens_per_second": 117628.8422320516, "loss": [0.15553933382034302, 6.0726852098014206e-05, 1.8626450382086546e-09], "lr": 0.00019195333498864384, "gradient_norm": 0.04265289008617401}
+{"step": 9764, "elapsed_seconds": 2430.8580310344696, "processed_tokens": 291468741, "response_tokens": 36686981, "tokens_per_second": 117688.23364271648, "loss": [0.37907978892326355, 0.0016924809897318482, 0.0], "lr": 0.00018729832211643877, "gradient_norm": 0.05004430189728737}
+{"step": 9981, "elapsed_seconds": 2465.887057542801, "processed_tokens": 295466740, "response_tokens": 37228757, "tokens_per_second": 111896.0129606338, "loss": [0.3532569706439972, 0.0012273044558241963, 0.0], "lr": 0.0002999637732514939, "gradient_norm": 0.052421823143959045}
+{"step": 10113, "elapsed_seconds": 2485.9128997325897, "processed_tokens": 297864368, "response_tokens": 37559526, "tokens_per_second": 115806.5816383428, "loss": [0.3271658420562744, 0.0017842546803876758, 3.296859745205438e-07], "lr": 0.00029985428626355437, "gradient_norm": 0.051775865256786346}
+{"step": 10239, "elapsed_seconds": 2505.9521973133087, "processed_tokens": 300243608, "response_tokens": 37877574, "tokens_per_second": 116780.26903945855, "loss": [0.2790696322917938, 0.004217061679810286, 1.881260658365136e-07], "lr": 0.00029967144734171846, "gradient_norm": 0.05251772329211235}
+{"step": 10364, "elapsed_seconds": 2526.0970282554626, "processed_tokens": 302610540, "response_tokens": 38197789, "tokens_per_second": 116959.79539855833, "loss": [0.28536656498908997, 0.01042899303138256, 0.0], "lr": 0.0002994139235964697, "gradient_norm": 0.07170809805393219}
+{"step": 10491, "elapsed_seconds": 2546.216892004013, "processed_tokens": 304963773, "response_tokens": 38524366, "tokens_per_second": 116959.9732044179, "loss": [0.32007527351379395, 0.0019307598704472184, 0.0], "lr": 0.00029908282918342434, "gradient_norm": 0.05117069557309151}
+{"step": 10617, "elapsed_seconds": 2567.710956096649, "processed_tokens": 307300570, "response_tokens": 38840545, "tokens_per_second": 115506.73177192775, "loss": [0.12349037826061249, 0.0027300368528813124, 5.4016645378851535e-08], "lr": 0.0002986464329761258, "gradient_norm": 0.04309625178575516}
+{"step": 10744, "elapsed_seconds": 2587.874372959137, "processed_tokens": 309653339, "response_tokens": 39167245, "tokens_per_second": 115673.97288246689, "loss": [0.3011234700679779, 0.005556588526815176, 1.1175867342672063e-08], "lr": 0.00029816434204971273, "gradient_norm": 0.05861425772309303}
+{"step": 10873, "elapsed_seconds": 2608.0518865585327, "processed_tokens": 312039770, "response_tokens": 39495080, "tokens_per_second": 115997.06198803346, "loss": [0.30714496970176697, 0.0024179769679903984, 0.0], "lr": 0.00029760699219033647, "gradient_norm": 0.04993556812405586}
+{"step": 11002, "elapsed_seconds": 2628.143845796585, "processed_tokens": 314416891, "response_tokens": 39824120, "tokens_per_second": 116252.16190303693, "loss": [0.3411191701889038, 0.0021175381261855364, 2.028717972279992e-05], "lr": 0.0002969792550305426, "gradient_norm": 0.04793043062090874}
+{"step": 11130, "elapsed_seconds": 2648.1572704315186, "processed_tokens": 316780836, "response_tokens": 40151104, "tokens_per_second": 116436.70378327067, "loss": [0.2911241948604584, 0.0017608735943213105, 9.313200166616298e-08], "lr": 0.00029628191894148564, "gradient_norm": 0.047284357249736786}
+{"step": 11260, "elapsed_seconds": 2668.2099900245667, "processed_tokens": 319201519, "response_tokens": 40472031, "tokens_per_second": 116822.54606999848, "loss": [0.13830506801605225, 0.00029989759786985815, 0.00014342859503813088], "lr": 0.0002955093520248278, "gradient_norm": 0.050923265516757965}
+{"step": 11384, "elapsed_seconds": 2689.6243538856506, "processed_tokens": 321529071, "response_tokens": 40789973, "tokens_per_second": 116108.35448497954, "loss": [0.3494488298892975, 0.0025619259104132652, 0.0], "lr": 0.00029461137770466026, "gradient_norm": 0.05940449610352516}
+{"step": 11513, "elapsed_seconds": 2709.7854237556458, "processed_tokens": 323914665, "response_tokens": 41117386, "tokens_per_second": 116277.78595632025, "loss": [0.3494163751602173, 0.004756445996463299, 0.0], "lr": 0.0002936913443910505, "gradient_norm": 0.05144161358475685}
+{"step": 11607, "elapsed_seconds": 2729.8615753650665, "processed_tokens": 325617180, "response_tokens": 41353603, "tokens_per_second": 114053.18782875882, "loss": [0.3508467376232147, 0.003322755452245474, 3.0547082019438676e-07], "lr": 0.00029270592261884415, "gradient_norm": 0.06767486035823822}
+{"step": 11704, "elapsed_seconds": 2750.018823623657, "processed_tokens": 327443448, "response_tokens": 41600951, "tokens_per_second": 112499.20918757067, "loss": [0.2876226305961609, 0.005321257747709751, 2.1366791770560667e-05], "lr": 0.00029164045023965217, "gradient_norm": 0.05245555192232132}
+{"step": 11830, "elapsed_seconds": 2770.1590259075165, "processed_tokens": 329802429, "response_tokens": 41925876, "tokens_per_second": 112786.6305368816, "loss": [0.30648913979530334, 0.001747457659803331, 0.0], "lr": 0.00029050945652930676, "gradient_norm": 0.05767581984400749}
+{"step": 11960, "elapsed_seconds": 2790.1598551273346, "processed_tokens": 332185029, "response_tokens": 42248140, "tokens_per_second": 113154.78726354336, "loss": [0.3028464615345001, 0.0034870721865445375, 0.0], "lr": 0.00028931756753756006, "gradient_norm": 0.049221236258745193}
+{"step": 12085, "elapsed_seconds": 2811.4246459007263, "processed_tokens": 334510148, "response_tokens": 42560205, "tokens_per_second": 112932.98436297705, "loss": [0.1293688863515854, 0.002224905416369438, 1.8626450382086546e-09], "lr": 0.0002879734888021334, "gradient_norm": 0.045018550008535385}
+{"step": 12214, "elapsed_seconds": 2831.5157504081726, "processed_tokens": 336898729, "response_tokens": 42883396, "tokens_per_second": 113243.15170108671, "loss": [0.14137494564056396, 8.701437036506832e-05, 0.0], "lr": 0.0002866385965156605, "gradient_norm": 0.042412836104631424}
+{"step": 12340, "elapsed_seconds": 2851.661438703537, "processed_tokens": 339257215, "response_tokens": 43206321, "tokens_per_second": 113433.1847925716, "loss": [0.33566009998321533, 0.004182623699307442, 0.0], "lr": 0.00028523754519350604, "gradient_norm": 0.053930215537548065}
+{"step": 12469, "elapsed_seconds": 2871.776302576065, "processed_tokens": 341630054, "response_tokens": 43535551, "tokens_per_second": 113647.16046014578, "loss": [0.3215128481388092, 0.0012561972253024578, 0.0], "lr": 0.0002837684676092931, "gradient_norm": 0.06633294373750687}
+{"step": 12594, "elapsed_seconds": 2891.8111941814423, "processed_tokens": 343962430, "response_tokens": 43861317, "tokens_per_second": 113771.52727164273, "loss": [0.27654188871383667, 0.002799010369926691, 0.0], "lr": 0.00028223934642827874, "gradient_norm": 0.060971636325120926}
+{"step": 12722, "elapsed_seconds": 2911.9242072105408, "processed_tokens": 346351119, "response_tokens": 44185861, "tokens_per_second": 113986.92788821674, "loss": [0.3331131637096405, 0.0005476420046761632, 0.0], "lr": 0.0002806415324025537, "gradient_norm": 0.051048990339040756}
+{"step": 12849, "elapsed_seconds": 2933.305921316147, "processed_tokens": 348686831, "response_tokens": 44505485, "tokens_per_second": 113778.6709963957, "loss": [0.30309879779815674, 0.0012497176649048924, 0.0], "lr": 0.00027887148514961203, "gradient_norm": 0.04612142965197563}
+{"step": 12975, "elapsed_seconds": 2953.4375660419464, "processed_tokens": 351059957, "response_tokens": 44829165, "tokens_per_second": 113941.33866797265, "loss": [0.3358946144580841, 0.0006249516154639423, 0.0], "lr": 0.0002771427725717586, "gradient_norm": 0.04956185817718506}
+{"step": 13108, "elapsed_seconds": 2973.570965051651, "processed_tokens": 353465927, "response_tokens": 45157993, "tokens_per_second": 114153.44995070904, "loss": [0.2775788903236389, 0.0017638042336329818, 0.0], "lr": 0.00027535157416639326, "gradient_norm": 0.0438496395945549}
+{"step": 13237, "elapsed_seconds": 2993.623607158661, "processed_tokens": 355872636, "response_tokens": 45479231, "tokens_per_second": 114368.17849660329, "loss": [0.30440524220466614, 0.003385748714208603, 0.0], "lr": 0.0002735078383020303, "gradient_norm": 0.05357814207673073}
+{"step": 13367, "elapsed_seconds": 3013.6639609336853, "processed_tokens": 358248852, "response_tokens": 45806299, "tokens_per_second": 114516.52224529973, "loss": [0.2745775580406189, 0.003944603726267815, 0.0], "lr": 0.00027160547868465746, "gradient_norm": 0.04720716550946236}
+{"step": 13483, "elapsed_seconds": 3033.7058656215668, "processed_tokens": 360447506, "response_tokens": 46103793, "tokens_per_second": 114352.41902726778, "loss": [0.31393206119537354, 0.003675902495160699, 0.0], "lr": 0.0002696460788385794, "gradient_norm": 0.06093598157167435}
+{"step": 13609, "elapsed_seconds": 3054.7841308116913, "processed_tokens": 362758373, "response_tokens": 46421832, "tokens_per_second": 114189.05578073072, "loss": [0.2995062470436096, 0.0011206113267689943, 1.8626450382086546e-09], "lr": 0.00026752105849331154, "gradient_norm": 0.049194272607564926}
+{"step": 13738, "elapsed_seconds": 3074.8518850803375, "processed_tokens": 365126897, "response_tokens": 46749938, "tokens_per_second": 114311.47429254664, "loss": [0.1342056840658188, 0.002647197339683771, 1.3038510715546181e-08], "lr": 0.0002654362602177019, "gradient_norm": 0.05364096909761429}
+{"step": 13864, "elapsed_seconds": 3094.999365091324, "processed_tokens": 367508563, "response_tokens": 47070701, "tokens_per_second": 114432.51468649285, "loss": [0.29766812920570374, 0.0008958622347563505, 0.0], "lr": 0.0002633012464149396, "gradient_norm": 0.047541581094264984}
+{"step": 13996, "elapsed_seconds": 3115.1212735176086, "processed_tokens": 369914039, "response_tokens": 47399896, "tokens_per_second": 114586.21861754195, "loss": [0.2911514341831207, 0.00465848995372653, 0.0], "lr": 0.0002611100250601476, "gradient_norm": 0.049627743661403656}
+{"step": 14129, "elapsed_seconds": 3135.1378450393677, "processed_tokens": 372316724, "response_tokens": 47728775, "tokens_per_second": 114744.43379221483, "loss": [0.32286858558654785, 0.0014564181910827756, 0.0], "lr": 0.00025887494004409414, "gradient_norm": 0.05147378891706467}
+{"step": 14262, "elapsed_seconds": 3155.2244482040405, "processed_tokens": 374751517, "response_tokens": 48052110, "tokens_per_second": 114927.63772460815, "loss": [0.3115446865558624, 0.00034807983320206404, 2.2351732908987287e-08], "lr": 0.0002565817081180606, "gradient_norm": 0.05457650497555733}
+{"step": 14390, "elapsed_seconds": 3176.3333377838135, "processed_tokens": 377081616, "response_tokens": 48375247, "tokens_per_second": 114796.36728749066, "loss": [0.29713526368141174, 0.006453809328377247, 1.8626450382086546e-09], "lr": 0.0002541156724566901, "gradient_norm": 0.05270839482545853}
+{"step": 14519, "elapsed_seconds": 3196.4093503952026, "processed_tokens": 379446179, "response_tokens": 48703115, "tokens_per_second": 114876.1833266184, "loss": [0.13288944959640503, 0.006802189629524946, 0.0], "lr": 0.00025171693753724594, "gradient_norm": 0.05771338567137718}
+{"step": 14648, "elapsed_seconds": 3216.5110080242157, "processed_tokens": 381814075, "response_tokens": 49029366, "tokens_per_second": 114952.34057631249, "loss": [0.3113184869289398, 0.001145739108324051, 0.0], "lr": 0.0002492831761969049, "gradient_norm": 0.052964285016059875}
+{"step": 14777, "elapsed_seconds": 3236.5279037952423, "processed_tokens": 384171086, "response_tokens": 49357467, "tokens_per_second": 115023.190298541, "loss": [0.10856305807828903, 0.0013963804813101888, 3.725290076417309e-09], "lr": 0.0002467944588128055, "gradient_norm": 0.04483762010931969}
+{"step": 14907, "elapsed_seconds": 3256.604527950287, "processed_tokens": 386564157, "response_tokens": 49683016, "tokens_per_second": 115126.53815709465, "loss": [0.33991220593452454, 0.0033356130588799715, 7.450562833355434e-08], "lr": 0.00024427201837615565, "gradient_norm": 0.059776876121759415}
+{"step": 15033, "elapsed_seconds": 3276.6225004196167, "processed_tokens": 388927427, "response_tokens": 50004740, "tokens_per_second": 115197.1561267487, "loss": [0.3043430745601654, 0.0014142735162749887, 0.0], "lr": 0.00024170471186350437, "gradient_norm": 0.058640312403440475}
+{"step": 15161, "elapsed_seconds": 3297.8442385196686, "processed_tokens": 391276166, "response_tokens": 50327035, "tokens_per_second": 115084.54898822056, "loss": [0.33844730257987976, 0.0014892045874148607, 0.0], "lr": 0.0002389390353195533, "gradient_norm": 0.08614109456539154}
+{"step": 15291, "elapsed_seconds": 3317.98078584671, "processed_tokens": 393676894, "response_tokens": 50657640, "tokens_per_second": 115180.0838936606, "loss": [0.3307572901248932, 0.0007629194296896458, 7.450578820566989e-09], "lr": 0.00023627241105699952, "gradient_norm": 0.054231397807598114}
+{"step": 15420, "elapsed_seconds": 3338.042640686035, "processed_tokens": 396049391, "response_tokens": 50988193, "tokens_per_second": 115249.31587844115, "loss": [0.3129560649394989, 0.002595656318590045, 0.0], "lr": 0.00023357724859080975, "gradient_norm": 0.0557674802839756}
+{"step": 15550, "elapsed_seconds": 3358.1030564308167, "processed_tokens": 398426630, "response_tokens": 51317593, "tokens_per_second": 115320.88276604508, "loss": [0.3004013001918793, 0.001291940687224269, 0.0], "lr": 0.00023084444543627162, "gradient_norm": 0.06580555438995361}
+{"step": 15678, "elapsed_seconds": 3378.1231777668, "processed_tokens": 400801978, "response_tokens": 51644391, "tokens_per_second": 115392.32882217172, "loss": [0.3077039420604706, 0.002550429431721568, 0.0], "lr": 0.00022808104463977616, "gradient_norm": 0.05904633551836014}
+{"step": 15809, "elapsed_seconds": 3398.2885789871216, "processed_tokens": 403186263, "response_tokens": 51974013, "tokens_per_second": 115452.54280706006, "loss": [0.3386188745498657, 0.0007547590648755431, 0.0], "lr": 0.00022526401083014503, "gradient_norm": 0.059149742126464844}
+{"step": 15931, "elapsed_seconds": 3419.4380691051483, "processed_tokens": 405506741, "response_tokens": 52288351, "tokens_per_second": 115327.97295842992, "loss": [0.33021724224090576, 0.0006749971071258187, 1.8626450382086546e-09], "lr": 0.0002222702196958619, "gradient_norm": 0.050267841666936874}
+{"step": 16058, "elapsed_seconds": 3439.5194568634033, "processed_tokens": 407862411, "response_tokens": 52613209, "tokens_per_second": 115367.94877610325, "loss": [0.28047817945480347, 0.00039382741670124233, 9.313223081619526e-09], "lr": 0.00021940077102648923, "gradient_norm": 0.048752762377262115}
+{"step": 16187, "elapsed_seconds": 3459.5595211982727, "processed_tokens": 410251914, "response_tokens": 52933828, "tokens_per_second": 115444.41940273777, "loss": [0.27568984031677246, 0.0023535434156656265, 1.1175867342672063e-08], "lr": 0.0002165125005542804, "gradient_norm": 0.05249473452568054}
+{"step": 16314, "elapsed_seconds": 3479.6323478221893, "processed_tokens": 412616784, "response_tokens": 53258795, "tokens_per_second": 115490.43726172337, "loss": [0.3021147847175598, 0.0006328545277938247, 0.0], "lr": 0.00021357766521634805, "gradient_norm": 0.05262129753828049}
+{"step": 16512, "elapsed_seconds": 3511.620875120163, "processed_tokens": 416249178, "response_tokens": 53761202, "tokens_per_second": 115643.34177913405, "loss": [0.13382919132709503, 0.0020514633506536484, 4.345257821114501e-06], "lr": 0.0002999706241044142, "gradient_norm": 0.0483582504093647}
+{"step": 16645, "elapsed_seconds": 3531.6528108119965, "processed_tokens": 418693272, "response_tokens": 54099679, "tokens_per_second": 118827.45513951515, "loss": [0.1416928619146347, 0.00014967452443670481, 2.4790349471004447e-06], "lr": 0.00029988186414620256, "gradient_norm": 0.049274612218141556}
+{"step": 16777, "elapsed_seconds": 3551.7217643260956, "processed_tokens": 421108636, "response_tokens": 54429766, "tokens_per_second": 119336.77343922679, "loss": [0.14237307012081146, 0.0028209590818732977, 2.0618465441657463e-06], "lr": 0.0002997334320871409, "gradient_norm": 0.06901361048221588}
+{"step": 16915, "elapsed_seconds": 3571.7356514930725, "processed_tokens": 423554153, "response_tokens": 54765099, "tokens_per_second": 120049.61452784551, "loss": [0.2816813290119171, 0.0012916011037304997, 1.2852227371240588e-07], "lr": 0.0002995269705983079, "gradient_norm": 0.0569540373980999}
+{"step": 17049, "elapsed_seconds": 3591.769100666046, "processed_tokens": 425966899, "response_tokens": 55096642, "tokens_per_second": 120126.8650666003, "loss": [0.1643388867378235, 0.012851620092988014, 1.6204948849463108e-07], "lr": 0.00029925945950754016, "gradient_norm": 0.06281836330890656}
+{"step": 17179, "elapsed_seconds": 3613.5536873340607, "processed_tokens": 428337489, "response_tokens": 55426929, "tokens_per_second": 118107.05518178816, "loss": [0.25395819544792175, 0.00028534940793178976, 2.1587170522252563e-06], "lr": 0.0002989033495007995, "gradient_norm": 0.04879450425505638}
+{"step": 17312, "elapsed_seconds": 3633.680574655533, "processed_tokens": 430744059, "response_tokens": 55759949, "tokens_per_second": 118314.2793035421, "loss": [0.223005011677742, 0.004840948153287172, 2.682200488379749e-07], "lr": 0.0002985119859794712, "gradient_norm": 0.05494227632880211}
+{"step": 17445, "elapsed_seconds": 3653.819920063019, "processed_tokens": 433153050, "response_tokens": 56094371, "tokens_per_second": 118475.90430766366, "loss": [0.25600558519363403, 0.0011226015631109476, 3.352760558072987e-08], "lr": 0.00029806095750228227, "gradient_norm": 0.0551595576107502}
+{"step": 17572, "elapsed_seconds": 3673.8491349220276, "processed_tokens": 435486262, "response_tokens": 56422368, "tokens_per_second": 118257.70161352935, "loss": [0.2600080966949463, 0.004941480699926615, 1.862644793959589e-08], "lr": 0.0002975530805968756, "gradient_norm": 0.0845252126455307}
+{"step": 17677, "elapsed_seconds": 3694.07128739357, "processed_tokens": 437399829, "response_tokens": 56688363, "tokens_per_second": 115897.57787675069, "loss": [0.25206804275512695, 0.00030042833532206714, 2.4864348233677447e-06], "lr": 0.0002969842282216995, "gradient_norm": 0.04913302883505821}
+{"step": 17763, "elapsed_seconds": 3714.2697978019714, "processed_tokens": 438985831, "response_tokens": 56909449, "tokens_per_second": 112507.10898327109, "loss": [0.2686588168144226, 0.009381731040775776, 3.7252898543727042e-09], "lr": 0.00029635488284335894, "gradient_norm": 0.06380413472652435}
+{"step": 17887, "elapsed_seconds": 3735.134660959244, "processed_tokens": 441239564, "response_tokens": 57222731, "tokens_per_second": 112122.30842654697, "loss": [0.12194923311471939, 7.218200335046276e-05, 6.574999815711635e-07], "lr": 0.00029563688011878894, "gradient_norm": 0.050802603363990784}
+{"step": 18020, "elapsed_seconds": 3755.27702832222, "processed_tokens": 443649826, "response_tokens": 57546345, "tokens_per_second": 112698.21386486394, "loss": [0.24279475212097168, 0.001109174219891429, 1.8626450382086546e-09], "lr": 0.00029489297331983824, "gradient_norm": 0.05315249785780907}
+{"step": 18145, "elapsed_seconds": 3775.3441376686096, "processed_tokens": 445965820, "response_tokens": 57870920, "tokens_per_second": 112890.17039247458, "loss": [0.23513644933700562, 0.00025876928702928126, 3.725290076417309e-09], "lr": 0.0002940913520753994, "gradient_norm": 0.04390033707022667}
+{"step": 18272, "elapsed_seconds": 3795.369818687439, "processed_tokens": 448322404, "response_tokens": 58192743, "tokens_per_second": 113205.8084854894, "loss": [0.26073628664016724, 0.00019972464360762388, 2.980204101277195e-07], "lr": 0.00029323434673536705, "gradient_norm": 0.05314616113901138}
+{"step": 18402, "elapsed_seconds": 3815.4569005966187, "processed_tokens": 450681846, "response_tokens": 58518915, "tokens_per_second": 113469.71375489111, "loss": [0.27202385663986206, 0.00014950691547710449, 0.0], "lr": 0.00029231837661411556, "gradient_norm": 0.04900723323225975}
+{"step": 18527, "elapsed_seconds": 3835.4966158866882, "processed_tokens": 453025035, "response_tokens": 58839717, "tokens_per_second": 113671.19337205106, "loss": [0.27769365906715393, 0.0010122267995029688, 8.754417990530783e-08], "lr": 0.0002913486101933282, "gradient_norm": 0.05414782837033272}
+{"step": 18653, "elapsed_seconds": 3856.6268627643585, "processed_tokens": 455314176, "response_tokens": 59157866, "tokens_per_second": 113362.28665350596, "loss": [0.12310273200273514, 0.0018978567095473409, 7.4505797087454084e-09], "lr": 0.00029026137642524047, "gradient_norm": 0.055952705442905426}
+{"step": 18787, "elapsed_seconds": 3876.7008085250854, "processed_tokens": 457711094, "response_tokens": 59486745, "tokens_per_second": 113677.24178403676, "loss": [0.23850063979625702, 0.0007289565401151776, 1.8626450382086546e-09], "lr": 0.00028917974982749933, "gradient_norm": 0.04682406410574913}
+{"step": 18917, "elapsed_seconds": 3896.741352558136, "processed_tokens": 460076577, "response_tokens": 59811327, "tokens_per_second": 113892.79406047618, "loss": [0.2088341861963272, 7.429136894643307e-05, 5.401582257036353e-07], "lr": 0.0002880405067617348, "gradient_norm": 0.04570179060101509}
+{"step": 19046, "elapsed_seconds": 3916.9025361537933, "processed_tokens": 462429285, "response_tokens": 60139827, "tokens_per_second": 114025.62753471351, "loss": [0.26830554008483887, 0.000773101404774934, 3.725287101019603e-08], "lr": 0.0002868408929388106, "gradient_norm": 0.05511775240302086}
+{"step": 19171, "elapsed_seconds": 3937.0157995224, "processed_tokens": 464803842, "response_tokens": 60459562, "tokens_per_second": 114207.77070735757, "loss": [0.21826976537704468, 0.0011977964313700795, 1.3131191280990606e-06], "lr": 0.000285589361159771, "gradient_norm": 0.04703841358423233}
+{"step": 19303, "elapsed_seconds": 3957.131397485733, "processed_tokens": 467183610, "response_tokens": 60788829, "tokens_per_second": 114384.79491449687, "loss": [0.25633421540260315, 0.0025525842793285847, 9.313223969797946e-09], "lr": 0.000284284170915517, "gradient_norm": 0.06617988646030426}
+{"step": 19430, "elapsed_seconds": 3978.1743075847626, "processed_tokens": 469499356, "response_tokens": 61110274, "tokens_per_second": 114197.27396102704, "loss": [0.2855498194694519, 0.0004424470826052129, 7.823089731573418e-08], "lr": 0.0002828619041346778, "gradient_norm": 0.055999480187892914}
+{"step": 19559, "elapsed_seconds": 3998.205878973007, "processed_tokens": 471868613, "response_tokens": 61434511, "tokens_per_second": 114358.55532514365, "loss": [0.12526853382587433, 0.0005399361252784729, 1.8626450382086546e-09], "lr": 0.0002814510847221632, "gradient_norm": 0.04401494935154915}
+{"step": 19681, "elapsed_seconds": 4018.2785086631775, "processed_tokens": 474181498, "response_tokens": 61753562, "tokens_per_second": 114391.60784774092, "loss": [0.30934062600135803, 0.0009275341290049255, 5.587935003603661e-09], "lr": 0.0002799960255834743, "gradient_norm": 0.054665323346853256}
+{"step": 19810, "elapsed_seconds": 4038.370349884033, "processed_tokens": 476536553, "response_tokens": 62081166, "tokens_per_second": 114495.33866451884, "loss": [0.2658301889896393, 0.00024472500081174076, 0.0], "lr": 0.0002784813714678339, "gradient_norm": 0.05153008550405502}
+{"step": 19938, "elapsed_seconds": 4058.435922384262, "processed_tokens": 478899236, "response_tokens": 62404428, "tokens_per_second": 114610.48386019684, "loss": [0.22937339544296265, 0.003858386306092143, 4.6937952902226243e-07], "lr": 0.00027692042178499517, "gradient_norm": 0.05623491853475571}
+{"step": 20064, "elapsed_seconds": 4078.5528683662415, "processed_tokens": 481224821, "response_tokens": 62729603, "tokens_per_second": 114644.51056683833, "loss": [0.24281562864780426, 0.000396769872168079, 4.097817196679898e-08], "lr": 0.000275305588002274, "gradient_norm": 0.05402901768684387}
+{"step": 20192, "elapsed_seconds": 4099.915342092514, "processed_tokens": 483533813, "response_tokens": 63049215, "tokens_per_second": 114414.20526403244, "loss": [0.14267143607139587, 0.00031033458071760833, 6.537747481161205e-07], "lr": 0.0002735299969041079, "gradient_norm": 0.05101660266518593}
+{"step": 20321, "elapsed_seconds": 4120.00478720665, "processed_tokens": 485896981, "response_tokens": 63375588, "tokens_per_second": 114517.08490721715, "loss": [0.2755645215511322, 0.00025529510458000004, 0.0], "lr": 0.0002718228833866346, "gradient_norm": 0.05485265329480171}
+{"step": 20448, "elapsed_seconds": 4140.011512994766, "processed_tokens": 488225487, "response_tokens": 63701733, "tokens_per_second": 114574.75516672448, "loss": [0.26466453075408936, 0.0001703817251836881, 4.842869927301763e-08], "lr": 0.00027007151018442475, "gradient_norm": 0.05190766602754593}
+{"step": 20536, "elapsed_seconds": 4160.253456830978, "processed_tokens": 489840310, "response_tokens": 63925220, "tokens_per_second": 113521.30615840809, "loss": [0.26221340894699097, 0.0006640314823016524, 5.308452273311559e-07], "lr": 0.0002682597400723663, "gradient_norm": 0.055592019110918045}
+{"step": 20635, "elapsed_seconds": 4180.312757968903, "processed_tokens": 491673612, "response_tokens": 64175731, "tokens_per_second": 112876.8345806244, "loss": [0.3167467415332794, 0.000156421257997863, 0.0], "lr": 0.0002664008809753467, "gradient_norm": 0.05849520117044449}
+{"step": 20762, "elapsed_seconds": 4200.36749792099, "processed_tokens": 494041193, "response_tokens": 64500409, "tokens_per_second": 113023.37822889883, "loss": [0.11253432184457779, 2.2989885110291652e-05, 3.7252898543727042e-09], "lr": 0.00026450083047385367, "gradient_norm": 0.05402452126145363}
+{"step": 20889, "elapsed_seconds": 4221.580595970154, "processed_tokens": 496359592, "response_tokens": 64821876, "tokens_per_second": 112914.91375800144, "loss": [0.29026421904563904, 0.00020705553470179439, 0.0], "lr": 0.00026245496330848833, "gradient_norm": 0.05454329028725624}
+{"step": 21019, "elapsed_seconds": 4241.582570075989, "processed_tokens": 498719953, "response_tokens": 65150461, "tokens_per_second": 113050.70326356961, "loss": [0.11931907385587692, 0.00020332216809038073, 9.708819561637938e-06], "lr": 0.0002604714741777976, "gradient_norm": 0.05061210319399834}
+{"step": 21147, "elapsed_seconds": 4261.5904541015625, "processed_tokens": 501079056, "response_tokens": 65471554, "tokens_per_second": 113176.93546174241, "loss": [0.2958284914493561, 0.0032621356658637524, 0.0], "lr": 0.00025845798562020527, "gradient_norm": 0.06771749258041382}
+{"step": 21275, "elapsed_seconds": 4281.629371881485, "processed_tokens": 503428228, "response_tokens": 65801192, "tokens_per_second": 113279.7527187055, "loss": [0.23033955693244934, 0.0037040021270513535, 1.8626450382086546e-09], "lr": 0.00025639295787758344, "gradient_norm": 0.05780768021941185}
+{"step": 21404, "elapsed_seconds": 4301.753333091736, "processed_tokens": 505795105, "response_tokens": 66130385, "tokens_per_second": 113387.43555111949, "loss": [0.25886431336402893, 0.00010666592424968258, 3.725290076417309e-09], "lr": 0.0002542787280076883, "gradient_norm": 0.05613191798329353}
+{"step": 21536, "elapsed_seconds": 4321.850036621094, "processed_tokens": 508170871, "response_tokens": 66458155, "tokens_per_second": 113504.33078855125, "loss": [0.23975789546966553, 0.00017806935647968203, 1.8626450382086546e-09], "lr": 0.00025212865682371126, "gradient_norm": 0.053477298468351364}
+{"step": 21663, "elapsed_seconds": 4343.2227284908295, "processed_tokens": 510504912, "response_tokens": 66784594, "tokens_per_second": 113396.4754672985, "loss": [0.2387510985136032, 0.00013567759015131742, 1.1175868230850483e-08], "lr": 0.000249796170774857, "gradient_norm": 0.054141007363796234}
+{"step": 21791, "elapsed_seconds": 4363.304984331131, "processed_tokens": 512853752, "response_tokens": 67111071, "tokens_per_second": 113478.59377040695, "loss": [0.28057989478111267, 5.635600246023387e-05, 1.303851426825986e-08], "lr": 0.000247571043970057, "gradient_norm": 0.06369452178478241}
+{"step": 21922, "elapsed_seconds": 4383.376098394394, "processed_tokens": 515227895, "response_tokens": 67439891, "tokens_per_second": 113586.80608498234, "loss": [0.22255046665668488, 0.00012303348921705037, 4.284078514160683e-08], "lr": 0.00024530865312016235, "gradient_norm": 0.05626484379172325}
+{"step": 22053, "elapsed_seconds": 4403.443808794022, "processed_tokens": 517618346, "response_tokens": 67764496, "tokens_per_second": 113708.56358178008, "loss": [0.26123306155204773, 0.000291137199383229, 1.303851338008144e-08], "lr": 0.0002430094574967498, "gradient_norm": 0.06330504268407822}
+{"step": 22184, "elapsed_seconds": 4423.482757091522, "processed_tokens": 519986431, "response_tokens": 68091535, "tokens_per_second": 113804.5895386762, "loss": [0.26326310634613037, 7.487327820854262e-05, 7.4505797087454084e-09], "lr": 0.000240680729468273, "gradient_norm": 0.056382644921541214}
+{"step": 22316, "elapsed_seconds": 4443.508001565933, "processed_tokens": 522370338, "response_tokens": 68414038, "tokens_per_second": 113914.83362335156, "loss": [0.10778768360614777, 5.09243254782632e-05, 3.7252898543727042e-09], "lr": 0.00023831216131481114, "gradient_norm": 0.04911137372255325}
+{"step": 22444, "elapsed_seconds": 4464.796276807785, "processed_tokens": 524702700, "response_tokens": 68738484, "tokens_per_second": 113819.59251433585, "loss": [0.1079617440700531, 1.5049608919071034e-05, 0.0], "lr": 0.00023576580017090046, "gradient_norm": 0.05547282472252846}
+{"step": 22574, "elapsed_seconds": 4484.827783346176, "processed_tokens": 527088477, "response_tokens": 69060533, "tokens_per_second": 113926.11293777192, "loss": [0.22342099249362946, 0.0012971145333722234, 0.04531766474246979], "lr": 0.000233348187080138, "gradient_norm": 0.11747722327709198}
+{"step": 22705, "elapsed_seconds": 4504.8326251506805, "processed_tokens": 529453994, "response_tokens": 69388330, "tokens_per_second": 114011.42710517053, "loss": [0.1361839920282364, 3.6427627492230386e-05, 1.9930178041249746e-07], "lr": 0.00023088621063733144, "gradient_norm": 0.0632045567035675}
+{"step": 22830, "elapsed_seconds": 4524.859009504318, "processed_tokens": 531768727, "response_tokens": 69709214, "tokens_per_second": 114041.9096145102, "loss": [0.2562670111656189, 0.0002956130192615092, 1.4901155864777138e-08], "lr": 0.00022840906758846963, "gradient_norm": 0.07119320333003998}
+{"step": 22959, "elapsed_seconds": 4544.998483419418, "processed_tokens": 534107992, "response_tokens": 70034548, "tokens_per_second": 114082.27507921492, "loss": [0.22715537250041962, 8.241133764386177e-05, 3.7252898543727042e-09], "lr": 0.00022588572970724455, "gradient_norm": 0.05174935609102249}
+{"step": 23085, "elapsed_seconds": 4565.00549697876, "processed_tokens": 536455005, "response_tokens": 70355461, "tokens_per_second": 114142.42695171265, "loss": [0.13876833021640778, 8.744710066821426e-05, 0.0], "lr": 0.0002233355330668741, "gradient_norm": 0.05829599127173424}
+{"step": 23208, "elapsed_seconds": 4586.340465307236, "processed_tokens": 538762482, "response_tokens": 70672617, "tokens_per_second": 114025.73443922549, "loss": [0.2633236050605774, 0.00011494787759147584, 1.8626440834168534e-08], "lr": 0.0002206090121446124, "gradient_norm": 0.06396179646253586}
+{"step": 23334, "elapsed_seconds": 4606.36194396019, "processed_tokens": 541118466, "response_tokens": 70994215, "tokens_per_second": 114091.23736134332, "loss": [0.09949246793985367, 0.00016595353372395039, 5.345700060388481e-07], "lr": 0.00021800990556208093, "gradient_norm": 0.061162661761045456}
+{"step": 23460, "elapsed_seconds": 4626.457820653915, "processed_tokens": 543485735, "response_tokens": 71316682, "tokens_per_second": 114156.8892284684, "loss": [0.11760301142930984, 2.4545945052523166e-05, 0.0], "lr": 0.00021538555873834247, "gradient_norm": 0.05731401592493057}
+{"step": 23586, "elapsed_seconds": 4646.5289986133575, "processed_tokens": 545838812, "response_tokens": 71636445, "tokens_per_second": 114210.41090712897, "loss": [0.2803330719470978, 0.0001301847951253876, 0.0], "lr": 0.00021275070050534163, "gradient_norm": 0.059926893562078476}
+{"step": 23717, "elapsed_seconds": 4666.680141925812, "processed_tokens": 548204045, "response_tokens": 71964928, "tokens_per_second": 114264.67326523851, "loss": [0.27701905369758606, 0.0005103647708892822, 0.0], "lr": 0.00021007860371786595, "gradient_norm": 0.0598137304186821}
+{"step": 23851, "elapsed_seconds": 4686.869566679001, "processed_tokens": 550623084, "response_tokens": 72293035, "tokens_per_second": 114358.4604252926, "loss": [0.21063551306724548, 0.0001129034862970002, 3.7252898543727042e-09], "lr": 0.0002073792902237545, "gradient_norm": 0.05416348576545715}
+{"step": 23978, "elapsed_seconds": 4707.861273527145, "processed_tokens": 552922451, "response_tokens": 72609910, "tokens_per_second": 114275.24455346754, "loss": [0.21247832477092743, 0.0003252210735809058, 0.0], "lr": 0.00020454795657880861, "gradient_norm": 0.05000205338001251}
+{"step": 24102, "elapsed_seconds": 4727.9679391384125, "processed_tokens": 555225727, "response_tokens": 72931031, "tokens_per_second": 114279.75928412602, "loss": [0.26691240072250366, 0.0003342813579365611, 0.0], "lr": 0.00020182129597459274, "gradient_norm": 0.05965961515903473}
+{"step": 24227, "elapsed_seconds": 4748.0344297885895, "processed_tokens": 557483632, "response_tokens": 73245960, "tokens_per_second": 114251.6728924595, "loss": [0.2283758521080017, 5.9498106566024944e-05, 0.0], "lr": 0.0001990986573981517, "gradient_norm": 0.056290674954652786}
+{"step": 24341, "elapsed_seconds": 4768.060943603516, "processed_tokens": 559664885, "response_tokens": 73538225, "tokens_per_second": 114167.99648129025, "loss": [0.24222978949546814, 0.004573670215904713, 1.8626450382086546e-09], "lr": 0.0001963393183979941, "gradient_norm": 0.05593015253543854}
+{"step": 24470, "elapsed_seconds": 4788.190872430801, "processed_tokens": 562041086, "response_tokens": 73861128, "tokens_per_second": 114228.15989375494, "loss": [0.23100808262825012, 1.951560625457205e-05, 0.0], "lr": 0.00019356645293498533, "gradient_norm": 0.05463654547929764}
+{"step": 24598, "elapsed_seconds": 4808.221255779266, "processed_tokens": 564371962, "response_tokens": 74186129, "tokens_per_second": 114260.69936128761, "loss": [0.2330685406923294, 0.0028507965616881847, 1.8626450382086546e-09], "lr": 0.0001907887668623979, "gradient_norm": 0.061108410358428955}
+{"step": 24723, "elapsed_seconds": 4829.447979688644, "processed_tokens": 566678088, "response_tokens": 74503703, "tokens_per_second": 114171.56047273494, "loss": [0.10652197897434235, 1.0551312698225956e-05, 0.0], "lr": 0.00018782667766391027, "gradient_norm": 0.0519951656460762}
+{"step": 24853, "elapsed_seconds": 4849.579904556274, "processed_tokens": 569039429, "response_tokens": 74829732, "tokens_per_second": 114217.84076236523, "loss": [0.21833330392837524, 1.3446454431687016e-05, 9.313223081619526e-09], "lr": 0.0001850261463528144, "gradient_norm": 0.05690198391675949}
+{"step": 24988, "elapsed_seconds": 4869.75093460083, "processed_tokens": 571460934, "response_tokens": 75159625, "tokens_per_second": 114303.1822615522, "loss": [0.21834546327590942, 0.00011161575093865395, 0.0], "lr": 0.00018220494036790184, "gradient_norm": 0.05223153159022331}
+{"step": 25118, "elapsed_seconds": 4889.776496887207, "processed_tokens": 573814089, "response_tokens": 75487164, "tokens_per_second": 114349.07738648485, "loss": [0.22898679971694946, 2.865836358978413e-05, 7.4505797087454084e-09], "lr": 0.00017939140825833956, "gradient_norm": 0.06071241945028305}
+{"step": 25207, "elapsed_seconds": 4916.069211244583, "processed_tokens": 576446168, "response_tokens": 75845751, "tokens_per_second": 101494.90306543738, "loss": [0.2620980739593506, 0.0024472931399941444, 1.7508668292975926e-07], "lr": 0.00019996997492152985, "gradient_norm": 0.0507517084479332}
+{"step": 25266, "elapsed_seconds": 4936.327399015427, "processed_tokens": 578616540, "response_tokens": 76136847, "tokens_per_second": 104322.80386602944, "loss": [0.24765406548976898, 0.000439922499936074, 0.0], "lr": 0.00019987652793269776, "gradient_norm": 0.0466858446598053}
+{"step": 25324, "elapsed_seconds": 4956.437044143677, "processed_tokens": 580796386, "response_tokens": 76426056, "tokens_per_second": 105676.98299868492, "loss": [0.1311899721622467, 0.000198000343516469, 1.825370929964265e-07], "lr": 0.00019971977428249384, "gradient_norm": 0.04082944989204407}
+{"step": 25380, "elapsed_seconds": 4976.465323448181, "processed_tokens": 582883991, "response_tokens": 76711728, "tokens_per_second": 105317.8933185999, "loss": [0.21721331775188446, 0.0008530977065674961, 7.450578820566989e-09], "lr": 0.00019950544423208118, "gradient_norm": 0.04652911052107811}
+{"step": 25442, "elapsed_seconds": 4996.541926622391, "processed_tokens": 585077994, "response_tokens": 77010526, "tokens_per_second": 106108.7462605649, "loss": [0.13672229647636414, 0.00017751626728568226, 9.313225191043273e-10], "lr": 0.00019922378922400977, "gradient_norm": 0.04527631774544716}
+{"step": 25502, "elapsed_seconds": 5017.891738414764, "processed_tokens": 587225271, "response_tokens": 77299321, "tokens_per_second": 105140.28919093497, "loss": [0.2123926430940628, 0.0010321525624021888, 1.8626450382086546e-09], "lr": 0.00019886297634755186, "gradient_norm": 0.04381302371621132}
+{"step": 25564, "elapsed_seconds": 5038.177809476852, "processed_tokens": 589445879, "response_tokens": 77602023, "tokens_per_second": 105756.94698162876, "loss": [0.22278551757335663, 7.712626393185928e-05, 1.8626449271863521e-09], "lr": 0.00019845267233238402, "gradient_norm": 0.04588639363646507}
+{"step": 25621, "elapsed_seconds": 5058.567676067352, "processed_tokens": 591593685, "response_tokens": 77891754, "tokens_per_second": 105704.29317141687, "loss": [0.2311723828315735, 0.0012827030150219798, 2.7939672797572257e-09], "lr": 0.0001979777023084312, "gradient_norm": 0.046684931963682175}
+{"step": 25680, "elapsed_seconds": 5078.880606174469, "processed_tokens": 593754572, "response_tokens": 78187655, "tokens_per_second": 105779.29867359245, "loss": [0.21642303466796875, 0.00087172124767676, 2.0489069996187936e-08], "lr": 0.00019744236026284428, "gradient_norm": 0.0421120747923851}
+{"step": 25784, "elapsed_seconds": 5112.527508497238, "processed_tokens": 597621790, "response_tokens": 78720448, "tokens_per_second": 126535.19736724488, "loss": [0.2524937391281128, 0.0007059996132738888, 5.308531214609502e-08], "lr": 0.00019996005924833988, "gradient_norm": 0.05005459487438202}
+{"step": 25858, "elapsed_seconds": 5132.638480186462, "processed_tokens": 600271195, "response_tokens": 79074472, "tokens_per_second": 129132.2686280846, "loss": [0.22555382549762726, 0.00011124074080726132, 1.8626450382086546e-09], "lr": 0.00019983955566548495, "gradient_norm": 0.042191147804260254}
+{"step": 25903, "elapsed_seconds": 5152.722485780716, "processed_tokens": 602093670, "response_tokens": 79303893, "tokens_per_second": 116363.4619228767, "loss": [0.22822335362434387, 0.00014289158571045846, 4.656612428988183e-09], "lr": 0.0001996379686275513, "gradient_norm": 0.04432006925344467}
+{"step": 25958, "elapsed_seconds": 5172.89496588707, "processed_tokens": 604041007, "response_tokens": 79570279, "tokens_per_second": 111397.90436061396, "loss": [0.22871896624565125, 0.00018412091594655067, 0.0], "lr": 0.00019935377626390183, "gradient_norm": 0.04673426225781441}
+{"step": 26029, "elapsed_seconds": 5193.107325077057, "processed_tokens": 606717777, "response_tokens": 79938135, "tokens_per_second": 115617.0655703528, "loss": [0.2526637315750122, 0.0003527102817315608, 9.313225191043273e-10], "lr": 0.00019898601728505583, "gradient_norm": 0.049026910215616226}
+{"step": 26103, "elapsed_seconds": 5214.103974580765, "processed_tokens": 609344242, "response_tokens": 80301933, "tokens_per_second": 117250.49813062738, "loss": [0.23149025440216064, 0.0010118808131664991, 1.1175794867313016e-07], "lr": 0.00019851723322445271, "gradient_norm": 0.05396774783730507}
+{"step": 26175, "elapsed_seconds": 5234.110981702805, "processed_tokens": 612010770, "response_tokens": 80665195, "tokens_per_second": 119512.56471784838, "loss": [0.23830246925354004, 0.002025751629844308, 0.0], "lr": 0.00019799113182354227, "gradient_norm": 0.05478023365139961}
+{"step": 26251, "elapsed_seconds": 5254.365170717239, "processed_tokens": 614740720, "response_tokens": 81035768, "tokens_per_second": 121421.64488542647, "loss": [0.2251967340707779, 0.00032063733669929206, 7.450577044210149e-09], "lr": 0.0001973778809168374, "gradient_norm": 0.044623252004384995}
+{"step": 26319, "elapsed_seconds": 5274.50902223587, "processed_tokens": 617442430, "response_tokens": 81385035, "tokens_per_second": 122825.88892487696, "loss": [0.18715132772922516, 3.847252082778141e-05, 0.0], "lr": 0.00019669254459142214, "gradient_norm": 0.06453073024749756}
+{"step": 26394, "elapsed_seconds": 5294.639350652695, "processed_tokens": 620153013, "response_tokens": 81756696, "tokens_per_second": 124002.64491472933, "loss": [0.23194871842861176, 0.00015388515021186322, 0.0], "lr": 0.00019592073278176981, "gradient_norm": 0.04670575633645058}
+{"step": 26467, "elapsed_seconds": 5314.679719924927, "processed_tokens": 622846338, "response_tokens": 82123568, "tokens_per_second": 124939.34561418694, "loss": [0.12307383120059967, 0.00011361160431988537, 2.9802276202417488e-08], "lr": 0.00019507664549240546, "gradient_norm": 0.0446750670671463}
+{"step": 26536, "elapsed_seconds": 5335.538303852081, "processed_tokens": 625446557, "response_tokens": 82479922, "tokens_per_second": 124915.33716067024, "loss": [0.1275108903646469, 5.5500400776509196e-05, 0.0], "lr": 0.00019411857905704064, "gradient_norm": 0.04312723129987717}
+{"step": 26608, "elapsed_seconds": 5355.786419868469, "processed_tokens": 628172474, "response_tokens": 82847511, "tokens_per_second": 125661.66127183489, "loss": [0.2233191579580307, 0.0002616745769046247, 3.7252894102834944e-09], "lr": 0.00019311433081137315, "gradient_norm": 0.04890083149075508}
+{"step": 26683, "elapsed_seconds": 5375.912564992905, "processed_tokens": 630874953, "response_tokens": 83221903, "tokens_per_second": 126273.123404496, "loss": [0.2251911759376526, 9.104866330744699e-05, 4.004676767976889e-08], "lr": 0.0001920373990214962, "gradient_norm": 0.04337163269519806}
+{"step": 26756, "elapsed_seconds": 5396.117376804352, "processed_tokens": 633587590, "response_tokens": 83588254, "tokens_per_second": 126804.1440500293, "loss": [0.20802441239356995, 1.712974699330516e-05, 4.656611984898973e-09], "lr": 0.00019088127416519733, "gradient_norm": 0.0399833545088768}
+{"step": 26829, "elapsed_seconds": 5416.259795665741, "processed_tokens": 636296546, "response_tokens": 83956846, "tokens_per_second": 127282.08210077939, "loss": [0.20693105459213257, 7.526679110014811e-05, 0.0], "lr": 0.0001896553557637961, "gradient_norm": 0.049167193472385406}
+{"step": 26902, "elapsed_seconds": 5436.411583185196, "processed_tokens": 639004021, "response_tokens": 84319256, "tokens_per_second": 127696.27885960757, "loss": [0.21545366942882538, 0.00041065464029088616, 9.313222193441106e-09], "lr": 0.00018835631567267884, "gradient_norm": 0.04486871138215065}
+{"step": 26970, "elapsed_seconds": 5457.221465110779, "processed_tokens": 641599890, "response_tokens": 84668620, "tokens_per_second": 127527.79837150307, "loss": [0.240667462348938, 1.3936629329691641e-05, 9.313225191043273e-10], "lr": 0.0001869391748709959, "gradient_norm": 0.04903648793697357}
+{"step": 27044, "elapsed_seconds": 5477.407452106476, "processed_tokens": 644313438, "response_tokens": 85037255, "tokens_per_second": 127889.48397266246, "loss": [0.1880003958940506, 2.105894782289397e-05, 5.401652103387278e-08], "lr": 0.00018549529637041157, "gradient_norm": 0.04191764444112778}
+{"step": 27118, "elapsed_seconds": 5497.656978607178, "processed_tokens": 647029144, "response_tokens": 85411542, "tokens_per_second": 128200.36253868538, "loss": [0.19999276101589203, 0.0002963773731607944, 0.0], "lr": 0.00018398541734489644, "gradient_norm": 0.06635136157274246}
+{"step": 27192, "elapsed_seconds": 5517.943989515305, "processed_tokens": 649752860, "response_tokens": 85782821, "tokens_per_second": 128489.16084878118, "loss": [0.2667265236377716, 0.00029757697484456003, 9.313225191043273e-10], "lr": 0.00018239208807611706, "gradient_norm": 0.05468040704727173}
+{"step": 27263, "elapsed_seconds": 5538.060927629471, "processed_tokens": 652460260, "response_tokens": 86140707, "tokens_per_second": 128764.20150276272, "loss": [0.21918535232543945, 0.00028279831167310476, 0.0], "lr": 0.0001807452084653884, "gradient_norm": 0.045111797749996185}
+{"step": 27335, "elapsed_seconds": 5558.180685997009, "processed_tokens": 655159638, "response_tokens": 86509992, "tokens_per_second": 128997.48570174696, "loss": [0.11107341945171356, 0.0001305518817389384, 0.0], "lr": 0.0001790341652688958, "gradient_norm": 0.0540677085518837}
+{"step": 27404, "elapsed_seconds": 5579.099003314972, "processed_tokens": 657748014, "response_tokens": 86849846, "tokens_per_second": 128771.43084987895, "loss": [0.10766057670116425, 0.00024654529988765717, 1.8626430176027498e-08], "lr": 0.00017720013782644646, "gradient_norm": 0.04469163343310356}
+{"step": 27456, "elapsed_seconds": 5599.107519626617, "processed_tokens": 659572254, "response_tokens": 87100675, "tokens_per_second": 127286.95412694145, "loss": [0.22591117024421692, 6.0390335420379415e-05, 1.8626449271863521e-09], "lr": 0.00017538641738108128, "gradient_norm": 0.04565886780619621}
+{"step": 27510, "elapsed_seconds": 5619.200255155563, "processed_tokens": 661520145, "response_tokens": 87364198, "tokens_per_second": 126129.81318975694, "loss": [0.1279420405626297, 2.168243190681096e-05, 1.8626449271863521e-09], "lr": 0.00017349284261328665, "gradient_norm": 0.04865342751145363}
+{"step": 27582, "elapsed_seconds": 5639.249629497528, "processed_tokens": 664190478, "response_tokens": 87728625, "tokens_per_second": 126388.55619254935, "loss": [0.2673834562301636, 6.542704795720056e-05, 3.7252898543727042e-09], "lr": 0.00017155255146335317, "gradient_norm": 0.043771784752607346}
+{"step": 27655, "elapsed_seconds": 5659.419516801834, "processed_tokens": 666907336, "response_tokens": 88090237, "tokens_per_second": 126684.13189776547, "loss": [0.2528161108493805, 2.3589700504089706e-05, 9.313225191043273e-10], "lr": 0.0001695374022366451, "gradient_norm": 0.04673686996102333}
+{"step": 27730, "elapsed_seconds": 5679.537953853607, "processed_tokens": 669611668, "response_tokens": 88465956, "tokens_per_second": 126949.19607777805, "loss": [0.11293316632509232, 0.008382326923310757, 0.0], "lr": 0.00016747377087355153, "gradient_norm": 0.04974951222538948}
+{"step": 27800, "elapsed_seconds": 5700.4919691085815, "processed_tokens": 672227911, "response_tokens": 88824141, "tokens_per_second": 126877.08851250132, "loss": [0.10672646015882492, 2.4110759113682434e-05, 3.7252898543727042e-09], "lr": 0.00016527339986286175, "gradient_norm": 0.046534985303878784}
+{"step": 27874, "elapsed_seconds": 5720.747381687164, "processed_tokens": 674951498, "response_tokens": 89193513, "tokens_per_second": 127121.57818052091, "loss": [0.2654046416282654, 0.0006154325092211366, 9.313225191043273e-10], "lr": 0.00016310992298343942, "gradient_norm": 0.04878932237625122}
+{"step": 27948, "elapsed_seconds": 5740.871675491333, "processed_tokens": 677655622, "response_tokens": 89562266, "tokens_per_second": 127346.53543878424, "loss": [0.2265034317970276, 0.0008859907975420356, 3.7252894102834944e-09], "lr": 0.00016089899948529285, "gradient_norm": 0.05025656148791313}
+{"step": 28018, "elapsed_seconds": 5760.87898182869, "processed_tokens": 680341263, "response_tokens": 89917595, "tokens_per_second": 127552.62627676292, "loss": [0.2051408588886261, 2.457363916619215e-05, 1.583246955760842e-08], "lr": 0.0001586481696704234, "gradient_norm": 0.04390602931380272}
+{"step": 28091, "elapsed_seconds": 5781.149836778641, "processed_tokens": 683060251, "response_tokens": 90295247, "tokens_per_second": 127746.27496964218, "loss": [0.20218190550804138, 0.0001657813845667988, 1.676378857951022e-08], "lr": 0.00015634293308821663, "gradient_norm": 0.05544305965304375}
+{"step": 28161, "elapsed_seconds": 5801.248344659805, "processed_tokens": 685750395, "response_tokens": 90657558, "tokens_per_second": 127919.2654648237, "loss": [0.11119592934846878, 1.2525791134976316e-05, 1.482531956753519e-06], "lr": 0.00015398279556539402, "gradient_norm": 0.04046820104122162}
+{"step": 28229, "elapsed_seconds": 5822.357288360596, "processed_tokens": 688377924, "response_tokens": 91002025, "tokens_per_second": 127819.66299245268, "loss": [0.19319677352905273, 2.5379102225997485e-05, 0.0], "lr": 0.00015150013887354314, "gradient_norm": 0.06660953164100647}
+{"step": 28300, "elapsed_seconds": 5842.585310935974, "processed_tokens": 691090797, "response_tokens": 91365324, "tokens_per_second": 127989.38600293608, "loss": [0.22157303988933563, 1.750283809087705e-05, 8.381898730647208e-09], "lr": 0.00014905289238111014, "gradient_norm": 0.0510459803044796}
+{"step": 28375, "elapsed_seconds": 5862.657488822937, "processed_tokens": 693785683, "response_tokens": 91732589, "tokens_per_second": 128152.77344075138, "loss": [0.211478129029274, 1.4654911865363829e-05, 0.0], "lr": 0.00014660086078715367, "gradient_norm": 0.04450482130050659}
+{"step": 28449, "elapsed_seconds": 5882.700231552124, "processed_tokens": 696478569, "response_tokens": 92101689, "tokens_per_second": 128310.11037906318, "loss": [0.20593388378620148, 5.498185055330396e-05, 4.656611984898973e-09], "lr": 0.0001441194507985167, "gradient_norm": 0.04384686052799225}
+{"step": 28521, "elapsed_seconds": 5902.927330255508, "processed_tokens": 699191771, "response_tokens": 92466781, "tokens_per_second": 128455.51194278368, "loss": [0.2467084527015686, 1.2367637282295618e-05, 1.8626449271863521e-09], "lr": 0.00014158410786989072, "gradient_norm": 0.0495985746383667}
+{"step": 28594, "elapsed_seconds": 5923.075324296951, "processed_tokens": 701890860, "response_tokens": 92835456, "tokens_per_second": 128589.09004600091, "loss": [0.20544487237930298, 3.807840766967274e-05, 0.0], "lr": 0.0001390299873367676, "gradient_norm": 0.04226988926529884}
+{"step": 28663, "elapsed_seconds": 5944.1514365673065, "processed_tokens": 704527682, "response_tokens": 93177454, "tokens_per_second": 128502.99580552259, "loss": [0.20489086210727692, 1.7862585082184523e-05, 7.450577044210149e-09], "lr": 0.00013632803106813476, "gradient_norm": 0.04889727383852005}
+{"step": 28733, "elapsed_seconds": 5964.23751783371, "processed_tokens": 707219674, "response_tokens": 93525104, "tokens_per_second": 128630.15561741436, "loss": [0.22143810987472534, 6.774240318918601e-05, 1.8626449271863521e-09], "lr": 0.00013373214052596813, "gradient_norm": 0.04734161123633385}
+{"step": 28805, "elapsed_seconds": 5984.245912075043, "processed_tokens": 709880068, "response_tokens": 93884271, "tokens_per_second": 128727.37561434855, "loss": [0.2631942629814148, 0.0001306216581724584, 0.0], "lr": 0.00013114725654558666, "gradient_norm": 0.04944726452231407}
+{"step": 28856, "elapsed_seconds": 6004.434819459915, "processed_tokens": 711713570, "response_tokens": 94140569, "tokens_per_second": 127888.24939736817, "loss": [0.11824528872966766, 2.5955754608730786e-05, 3.7252894102834944e-09], "lr": 0.0001284771423152795, "gradient_norm": 0.04496327415108681}
+{"step": 28911, "elapsed_seconds": 6024.550703763962, "processed_tokens": 713664840, "response_tokens": 94409399, "tokens_per_second": 127221.75316705665, "loss": [0.22915637493133545, 0.00019705483282450587, 1.8626450382086546e-09], "lr": 0.00012583184755738503, "gradient_norm": 0.05013510212302208}
+{"step": 28982, "elapsed_seconds": 6044.7052574157715, "processed_tokens": 716299023, "response_tokens": 94770212, "tokens_per_second": 127295.34396163309, "loss": [0.20284231007099152, 0.00031748146284371614, 1.0617004875257408e-07], "lr": 0.00012313516795089315, "gradient_norm": 0.045069772750139236}
+{"step": 29079, "elapsed_seconds": 6073.134215593338, "processed_tokens": 719863101, "response_tokens": 95251806, "tokens_per_second": 127303.19164083849, "loss": [0.0006699951482005417, 0.0006699951482005417, 1.1175865566315224e-08], "lr": 9.992467126169638e-05, "gradient_norm": 0.028235170990228653}
+{"step": 29153, "elapsed_seconds": 6093.400689125061, "processed_tokens": 722600939, "response_tokens": 95623373, "tokens_per_second": 131207.2177353278, "loss": [4.502661613514647e-05, 4.502661613514647e-05, 1.8626449271863521e-09], "lr": 9.969418411046458e-05, "gradient_norm": 0.0021071447990834713}
+{"step": 29229, "elapsed_seconds": 6113.51254940033, "processed_tokens": 725316305, "response_tokens": 95997535, "tokens_per_second": 132471.48444147056, "loss": [0.0002606948546599597, 0.0002606948546599597, 9.313225191043273e-10], "lr": 9.931149910391107e-05, "gradient_norm": 0.02077270857989788}
+{"step": 29304, "elapsed_seconds": 6133.813632726669, "processed_tokens": 728054046, "response_tokens": 96371453, "tokens_per_second": 133070.48041256153, "loss": [0.0008880241075530648, 0.0008880241075530648, 1.7695120035909895e-08], "lr": 9.877145817237917e-05, "gradient_norm": 0.010123246349394321}
+{"step": 29379, "elapsed_seconds": 6154.055971860886, "processed_tokens": 730778552, "response_tokens": 96749364, "tokens_per_second": 133375.64165893302, "loss": [3.873390960507095e-05, 3.873390960507095e-05, 0.0], "lr": 9.807880476297946e-05, "gradient_norm": 0.0023269683588296175}
+{"step": 29447, "elapsed_seconds": 6174.685478448868, "processed_tokens": 733359807, "response_tokens": 97090987, "tokens_per_second": 131977.1709084698, "loss": [6.367154855979607e-05, 6.367154855979607e-05, 0.0], "lr": 9.721791964748367e-05, "gradient_norm": 0.004446391481906176}
+{"step": 29522, "elapsed_seconds": 6194.865514993668, "processed_tokens": 736088919, "response_tokens": 97459617, "tokens_per_second": 132440.9405074648, "loss": [3.817456672550179e-05, 3.817456672550179e-05, 9.313225191043273e-10], "lr": 9.623501823055166e-05, "gradient_norm": 0.002111125737428665}
+{"step": 29597, "elapsed_seconds": 6215.080131530762, "processed_tokens": 738817508, "response_tokens": 97831673, "tokens_per_second": 132757.6724833005, "loss": [0.00011607155465753749, 0.00011607155465753749, 0.0], "lr": 9.510836814024812e-05, "gradient_norm": 0.004171773791313171}
+{"step": 29669, "elapsed_seconds": 6235.2005610466, "processed_tokens": 741531119, "response_tokens": 98197013, "tokens_per_second": 132990.72386775663, "loss": [0.0015528369694948196, 0.0015528369694948196, 3.259623326812289e-08], "lr": 9.38422296477012e-05, "gradient_norm": 0.0477127768099308}
+{"step": 29742, "elapsed_seconds": 6255.4500851631165, "processed_tokens": 744252456, "response_tokens": 98560992, "tokens_per_second": 133130.67746035714, "loss": [4.147428990108892e-05, 4.147428990108892e-05, 0.0], "lr": 9.243715592088872e-05, "gradient_norm": 0.0015426499303430319}
+{"step": 29814, "elapsed_seconds": 6275.503318309784, "processed_tokens": 746955807, "response_tokens": 98920477, "tokens_per_second": 133281.89142891482, "loss": [6.239989943424007e-06, 6.239989943424007e-06, 5.587934559514451e-09], "lr": 9.091334389145859e-05, "gradient_norm": 0.0001166293368441984}
+{"step": 29886, "elapsed_seconds": 6296.347902297974, "processed_tokens": 749567885, "response_tokens": 99277793, "tokens_per_second": 132599.3066710614, "loss": [2.784799289656803e-05, 2.784799289656803e-05, 0.0], "lr": 8.920319943282002e-05, "gradient_norm": 0.0012914807302877307}
+{"step": 29959, "elapsed_seconds": 6316.5202667713165, "processed_tokens": 752282500, "response_tokens": 99645656, "tokens_per_second": 132750.21950589484, "loss": [2.824791044986341e-05, 2.824791044986341e-05, 1.3038505386475663e-08], "lr": 8.743575587543916e-05, "gradient_norm": 0.0009692060411907732}
+{"step": 30031, "elapsed_seconds": 6336.744044303894, "processed_tokens": 755004969, "response_tokens": 100005900, "tokens_per_second": 132883.2753478507, "loss": [6.394382944563404e-05, 6.394382944563404e-05, 1.8626449271863521e-09], "lr": 8.55455158179848e-05, "gradient_norm": 0.005837075412273407}
+{"step": 30104, "elapsed_seconds": 6356.991518974304, "processed_tokens": 757728511, "response_tokens": 100375433, "tokens_per_second": 132991.79070753665, "loss": [4.327292845118791e-05, 4.327292845118791e-05, 1.8626449271863521e-09], "lr": 8.354765132293401e-05, "gradient_norm": 0.0021789090242236853}
+{"step": 30175, "elapsed_seconds": 6377.160702943802, "processed_tokens": 760442205, "response_tokens": 100729090, "tokens_per_second": 133088.51738475738, "loss": [4.964303661836311e-05, 4.964303661836311e-05, 3.259622260998185e-08], "lr": 8.145287094007594e-05, "gradient_norm": 0.003502823878079653}
+{"step": 30250, "elapsed_seconds": 6397.19084239006, "processed_tokens": 763136492, "response_tokens": 101099750, "tokens_per_second": 133171.32832626626, "loss": [0.00033782838727347553, 0.00033782838727347553, 0.0], "lr": 7.929431891968933e-05, "gradient_norm": 0.01214758213609457}
+{"step": 30321, "elapsed_seconds": 6418.167233943939, "processed_tokens": 765762308, "response_tokens": 101460356, "tokens_per_second": 132712.29720585785, "loss": [0.0014928276650607586, 0.0014928276650607586, 1.3038508051010922e-08], "lr": 7.69439655340436e-05, "gradient_norm": 0.035266462713479996}
+{"step": 30394, "elapsed_seconds": 6438.180299282074, "processed_tokens": 768461614, "response_tokens": 101825483, "tokens_per_second": 132824.77050224843, "loss": [5.301893543219194e-05, 5.301893543219194e-05, 0.0], "lr": 7.461615233556088e-05, "gradient_norm": 0.005762320943176746}
+{"step": 30469, "elapsed_seconds": 6458.329412460327, "processed_tokens": 771178048, "response_tokens": 102194588, "tokens_per_second": 132923.77509574834, "loss": [0.00013588322326540947, 0.00013588322326540947, 2.7939675018018306e-09], "lr": 7.221696874817945e-05, "gradient_norm": 0.008774744346737862}
+{"step": 30544, "elapsed_seconds": 6478.367642879486, "processed_tokens": 773837086, "response_tokens": 102557906, "tokens_per_second": 132913.1516008558, "loss": [2.0074056010344066e-05, 2.0074056010344066e-05, 6.5192553577730905e-09], "lr": 6.97630257129139e-05, "gradient_norm": 0.0009545228094793856}
+{"step": 30594, "elapsed_seconds": 6498.581494569778, "processed_tokens": 775684317, "response_tokens": 102808558, "tokens_per_second": 131029.3323722008, "loss": [7.211016782093793e-05, 7.211016782093793e-05, 4.749738380382951e-08], "lr": 6.723944939183453e-05, "gradient_norm": 0.004236270673573017}
+{"step": 30648, "elapsed_seconds": 6518.717577457428, "processed_tokens": 777641294, "response_tokens": 103075302, "tokens_per_second": 129566.22822878169, "loss": [6.307139119599015e-05, 6.307139119599015e-05, 7.338315754168434e-06], "lr": 6.471982357248467e-05, "gradient_norm": 0.007099492941051722}
+{"step": 30718, "elapsed_seconds": 6539.602779626846, "processed_tokens": 780196130, "response_tokens": 103421075, "tokens_per_second": 129255.56240584966, "loss": [3.738953091669828e-05, 3.738953091669828e-05, 7.450578820566989e-09], "lr": 6.19905841183857e-05, "gradient_norm": 0.002970520406961441}
+{"step": 30792, "elapsed_seconds": 6559.839084148407, "processed_tokens": 782930339, "response_tokens": 103786967, "tokens_per_second": 129489.45623818973, "loss": [7.962183008203283e-05, 7.962183008203283e-05, 1.1175867342672063e-08], "lr": 5.9373109699456586e-05, "gradient_norm": 0.00470252800732851}
+{"step": 30866, "elapsed_seconds": 6580.050744056702, "processed_tokens": 785657823, "response_tokens": 104154506, "tokens_per_second": 129698.69681464706, "loss": [0.0009725401178002357, 0.0009725401178002357, 0.0], "lr": 5.6726142167341476e-05, "gradient_norm": 0.015773070976138115}
+{"step": 30942, "elapsed_seconds": 6600.155783653259, "processed_tokens": 788376047, "response_tokens": 104523916, "tokens_per_second": 129900.86966713158, "loss": [5.539510311791673e-06, 5.539510311791673e-06, 6.51925624595151e-09], "lr": 5.408754202082361e-05, "gradient_norm": 0.00015087453357409686}
+{"step": 31018, "elapsed_seconds": 6620.282869815826, "processed_tokens": 791091394, "response_tokens": 104892688, "tokens_per_second": 130078.58572069992, "loss": [6.144347480585566e-06, 6.144347480585566e-06, 9.313225191043273e-10], "lr": 5.14565260702831e-05, "gradient_norm": 0.00031936480081640184}
+{"step": 31094, "elapsed_seconds": 6640.5326018333435, "processed_tokens": 793817739, "response_tokens": 105273294, "tokens_per_second": 130235.65518832701, "loss": [5.863247861270793e-05, 5.863247861270793e-05, 0.0], "lr": 4.882876652824354e-05, "gradient_norm": 0.003360284026712179}
+{"step": 31165, "elapsed_seconds": 6661.2567348480225, "processed_tokens": 796415643, "response_tokens": 105624042, "tokens_per_second": 130069.4232747754, "loss": [0.0002370124711887911, 0.0002370124711887911, 1.396983329016166e-08], "lr": 4.615276948052715e-05, "gradient_norm": 0.013108043931424618}
+{"step": 31239, "elapsed_seconds": 6681.357529640198, "processed_tokens": 799127304, "response_tokens": 105989311, "tokens_per_second": 130224.04420861696, "loss": [8.568815246690065e-06, 8.568815246690065e-06, 3.7252894102834944e-09], "lr": 4.360594078637149e-05, "gradient_norm": 0.00021665218810085207}
+{"step": 31312, "elapsed_seconds": 6701.369346380234, "processed_tokens": 801822116, "response_tokens": 106354164, "tokens_per_second": 130360.98448315286, "loss": [6.740175012964755e-05, 6.740175012964755e-05, 6.891766446415204e-08], "lr": 4.108555718890984e-05, "gradient_norm": 0.0023239031434059143}
+{"step": 31383, "elapsed_seconds": 6721.386372566223, "processed_tokens": 804514219, "response_tokens": 106722162, "tokens_per_second": 130484.65512547243, "loss": [1.1302076927677263e-06, 1.1302076927677263e-06, 9.313225191043273e-10], "lr": 3.8604697186789676e-05, "gradient_norm": 2.253541060781572e-05}
+{"step": 31439, "elapsed_seconds": 6741.692917585373, "processed_tokens": 806568367, "response_tokens": 107007677, "tokens_per_second": 129619.94950036726, "loss": [2.8806107366108336e-05, 2.8806107366108336e-05, 9.313225191043273e-10], "lr": 3.61815432582426e-05, "gradient_norm": 0.001966773299500346}
+{"step": 31491, "elapsed_seconds": 6761.740720510483, "processed_tokens": 808474586, "response_tokens": 107266214, "tokens_per_second": 128643.08351522843, "loss": [2.1813850253238343e-05, 2.1813850253238343e-05, 1.8626450382086546e-09], "lr": 3.380951914134261e-05, "gradient_norm": 0.0007480962085537612}
+{"step": 31550, "elapsed_seconds": 6783.465214967728, "processed_tokens": 810601075, "response_tokens": 107558409, "tokens_per_second": 127728.34117873233, "loss": [1.1157623703184072e-05, 1.1157623703184072e-05, 0.0], "lr": 3.1363247917217284e-05, "gradient_norm": 0.0004215927910991013}
+{"step": 31598, "elapsed_seconds": 6803.528786897659, "processed_tokens": 812429662, "response_tokens": 107803224, "tokens_per_second": 126750.27313632476, "loss": [3.877933977491921e-06, 3.877933977491921e-06, 1.8626449271863521e-09], "lr": 2.9172723228985422e-05, "gradient_norm": 0.00016833978588692844}
+{"step": 31661, "elapsed_seconds": 6823.733549594879, "processed_tokens": 814665495, "response_tokens": 108111057, "tokens_per_second": 126328.45084979273, "loss": [8.241543582698796e-06, 8.241543582698796e-06, 8.381900507004048e-09], "lr": 2.7050305135666466e-05, "gradient_norm": 0.0006007819320075214}
+{"step": 31733, "elapsed_seconds": 6843.8288724422455, "processed_tokens": 817373671, "response_tokens": 108475493, "tokens_per_second": 126542.85637724277, "loss": [6.7131359173799865e-06, 6.7131359173799865e-06, 6.5192553577730905e-09], "lr": 2.5026741167110433e-05, "gradient_norm": 0.00016361952293664217}
+{"step": 31810, "elapsed_seconds": 6864.165573596954, "processed_tokens": 820116602, "response_tokens": 108846709, "tokens_per_second": 126751.76564989563, "loss": [9.005384526972193e-06, 9.005384526972193e-06, 8.381900507004048e-09], "lr": 2.3105266622723523e-05, "gradient_norm": 0.00047974311746656895}
+{"step": 31884, "elapsed_seconds": 6884.379106521606, "processed_tokens": 822841324, "response_tokens": 109217172, "tokens_per_second": 126947.36198276925, "loss": [1.736591730150394e-05, 1.736591730150394e-05, 9.313170323821396e-08], "lr": 2.1284856275920633e-05, "gradient_norm": 0.0007709717028774321}
+{"step": 31952, "elapsed_seconds": 6905.1312692165375, "processed_tokens": 825439283, "response_tokens": 109568005, "tokens_per_second": 126904.56114832584, "loss": [3.90421928386786e-06, 3.90421928386786e-06, 5.587934559514451e-09], "lr": 1.9556213143986514e-05, "gradient_norm": 9.323659469373524e-05}
+{"step": 32019, "elapsed_seconds": 6925.17325758934, "processed_tokens": 828137443, "response_tokens": 109902757, "tokens_per_second": 127081.97376527014, "loss": [8.019295637495816e-05, 8.019295637495816e-05, 0.0], "lr": 1.799344566959681e-05, "gradient_norm": 0.003094858257099986}
+{"step": 32096, "elapsed_seconds": 6945.369554519653, "processed_tokens": 830859235, "response_tokens": 110282452, "tokens_per_second": 127255.89399173473, "loss": [5.6502311053918675e-05, 5.6502311053918675e-05, 4.656611984898973e-09], "lr": 1.655418257722952e-05, "gradient_norm": 0.002490660408511758}
+{"step": 32170, "elapsed_seconds": 6965.40497469902, "processed_tokens": 833564738, "response_tokens": 110648094, "tokens_per_second": 127426.73061472393, "loss": [4.433678441273514e-06, 4.433678441273514e-06, 3.632153067201216e-08], "lr": 1.525724466383012e-05, "gradient_norm": 0.0001215417287312448}
+{"step": 32241, "elapsed_seconds": 6985.468643188477, "processed_tokens": 836257521, "response_tokens": 111013893, "tokens_per_second": 127572.72009499492, "loss": [8.167473424691707e-06, 8.167473424691707e-06, 3.7252898543727042e-09], "lr": 1.4097012758387704e-05, "gradient_norm": 0.00024123852199409157}
+{"step": 32314, "elapsed_seconds": 7005.473510980606, "processed_tokens": 838947930, "response_tokens": 111384709, "tokens_per_second": 127717.9513437489, "loss": [7.182440458564088e-05, 7.182440458564088e-05, 2.7939675018018306e-09], "lr": 1.307516922845098e-05, "gradient_norm": 0.0018888970371335745}
+{"step": 32383, "elapsed_seconds": 7026.676253557205, "processed_tokens": 841599013, "response_tokens": 111734465, "tokens_per_second": 127659.52744712733, "loss": [1.1904349776159506e-05, 1.1904349776159506e-05, 5.587933671336032e-09], "lr": 1.2141966801358144e-05, "gradient_norm": 0.0003674357139971107}
+{"step": 32456, "elapsed_seconds": 7046.933282136917, "processed_tokens": 844325829, "response_tokens": 112103850, "tokens_per_second": 127801.19564382166, "loss": [3.8011842207197333e-06, 3.8011842207197333e-06, 0.0], "lr": 1.1410746758735553e-05, "gradient_norm": 6.11467330600135e-05}
+{"step": 32530, "elapsed_seconds": 7067.19274353981, "processed_tokens": 847046652, "response_tokens": 112481905, "tokens_per_second": 127930.98894500388, "loss": [3.34519336320227e-06, 3.34519336320227e-06, 5.12226279170136e-08], "lr": 1.0827972830295375e-05, "gradient_norm": 7.843935600249097e-05}
+{"step": 32602, "elapsed_seconds": 7087.248184919357, "processed_tokens": 849741466, "response_tokens": 112846457, "tokens_per_second": 128055.81152372909, "loss": [1.9342547602718696e-05, 1.9342547602718696e-05, 6.51925669004072e-09], "lr": 1.0403162001981639e-05, "gradient_norm": 0.0015587583184242249}

provenance/selection-history.json

@@ -0,0 +1,141 @@
+{
+  "frozen_at_utc": "2026-09-10T14:00:36.513800+00:00",
+  "selected": {
+    "checkpoint": "/workspace/tinyquery/candidates/step-25715/model.safetensors",
+    "step": 25715,
+    "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
+    "development_scores": {
+      "validation": {
+        "correct": 1174,
+        "total": 1200,
+        "rate": 0.9783333333333334
+      },
+      "development": {
+        "correct": 188,
+        "total": 192,
+        "rate": 0.9791666666666666
+      }
+    },
+    "selection_score": 0.97875,
+    "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+  },
+  "candidates": [
+    {
+      "checkpoint": "/workspace/tinyquery/candidates/step-25715/model.safetensors",
+      "step": 25715,
+      "checkpoint_sha256": "9489b5cde69c11f64b3e7031182e8248aa83790f1ad4aeefbf99a4e5e152e692",
+      "development_scores": {
+        "validation": {
+          "correct": 1174,
+          "total": 1200,
+          "rate": 0.9783333333333334
+        },
+        "development": {
+          "correct": 188,
+          "total": 192,
+          "rate": 0.9791666666666666
+        }
+      },
+      "selection_score": 0.97875,
+      "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+    },
+    {
+      "checkpoint": "/workspace/tinyquery/candidates/average-25715-30320-50/model.safetensors",
+      "step": 30320,
+      "method": "weighted_parameter_average",
+      "checkpoint_sha256": "f4fc8a2586a4775ce0427f4ccfedcde303eb82934736c83ca53a76721145d89e",
+      "development_scores": {
+        "validation": {
+          "correct": 1180,
+          "total": 1200,
+          "rate": 0.9833333333333333
+        },
+        "development": {
+          "correct": 187,
+          "total": 192,
+          "rate": 0.9739583333333334
+        }
+      },
+      "selection_score": 0.9786458333333333,
+      "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+    },
+    {
+      "checkpoint": "/workspace/tinyquery/candidates/average-25715-30320-75/model.safetensors",
+      "step": 30320,
+      "method": "weighted_parameter_average",
+      "checkpoint_sha256": "59e70a2370278d77b307a90ed6d2d1535cac15c3db63ede3379e94cb37054326",
+      "development_scores": {
+        "validation": {
+          "correct": 1175,
+          "total": 1200,
+          "rate": 0.9791666666666666
+        },
+        "development": {
+          "correct": 187,
+          "total": 192,
+          "rate": 0.9739583333333334
+        }
+      },
+      "selection_score": 0.9765625,
+      "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+    },
+    {
+      "checkpoint": "/workspace/tinyquery/candidates/step-26969/model.safetensors",
+      "step": 26969,
+      "checkpoint_sha256": "45032f6ffa4dc0768f72987c2722119163d04c653276f98f52b1741c53345f14",
+      "development_scores": {
+        "validation": {
+          "correct": 1174,
+          "total": 1200,
+          "rate": 0.9783333333333334
+        },
+        "development": {
+          "correct": 187,
+          "total": 192,
+          "rate": 0.9739583333333334
+        }
+      },
+      "selection_score": 0.9761458333333334,
+      "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+    },
+    {
+      "checkpoint": "/workspace/tinyquery/candidates/step-28662/model.safetensors",
+      "step": 28662,
+      "checkpoint_sha256": "52bdd44fbcaa22ce806f32128fa46456806899ace9d887c1cf76bb146d6ba236",
+      "development_scores": {
+        "validation": {
+          "correct": 1172,
+          "total": 1200,
+          "rate": 0.9766666666666667
+        },
+        "development": {
+          "correct": 186,
+          "total": 192,
+          "rate": 0.96875
+        }
+      },
+      "selection_score": 0.9727083333333333,
+      "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+    },
+    {
+      "checkpoint": "/workspace/tinyquery/candidates/step-30320/model.safetensors",
+      "step": 30320,
+      "checkpoint_sha256": "49f8b2ef703ee91fe005d1f90ec9e9ea0aed920130f87b43669de6d5365b0220",
+      "development_scores": {
+        "validation": {
+          "correct": 1178,
+          "total": 1200,
+          "rate": 0.9816666666666667
+        },
+        "development": {
+          "correct": 185,
+          "total": 192,
+          "rate": 0.9635416666666666
+        }
+      },
+      "selection_score": 0.9726041666666667,
+      "rule": "Equal mean of full validation and development task success. Test/manual excluded."
+    }
+  ],
+  "test_and_manual_not_used_for_selection": true
+}

provenance/training-summary.json

@@ -0,0 +1,14 @@
+{
+  "parameters": 139738113,
+  "step": 32642,
+  "processed_tokens": 851198143,
+  "response_tokens": 113051294,
+  "training_seconds": 7099.562133073807,
+  "validation": [
+    0.0029369813855737448,
+    0.0029369813855737448,
+    0.06618711352348328
+  ],
+  "random_initialization": true,
+  "device": "cuda"
+}

runtime/README.md

@@ -4,7 +4,7 @@ The student runs locally on Apple Silicon MPS, CUDA, or CPU. Use Python 3.12 and
 
 `serve-teacher.sh` reconstructs the recorded working launch settings. The exact downloaded teacher revision is pinned in `download-teacher.py`. CUDA training used PyTorch 2.13.0+cu130; use the appropriate official CUDA wheel index. The teacher used vLLM 0.28.0 and Transformers 5.15.1. Do not install the CUDA environment into the Mac inference environment.
 
-The published dataset freezes the final corpus; original templates, prompts, verification records and programmatic generators are included in the companion dataset and source package. Training evolved through seven curricula; a fresh run on the final corpus is a new experiment, not a bitwise replay. Earlier training encountered 6,778 rows later filtered out. The released weights are selected by validation and development success, not the last trainer checkpoint. The final optimizer state and some raw remote logs could not be retrieved after the rented SSH endpoint refused connections.
+The published dataset freezes the final corpus; original templates, prompts, verification records and programmatic generators are included in the companion dataset and source package. Training evolved through seven curricula; a fresh run on the final corpus is a new experiment, not a bitwise replay. Earlier training encountered 6,778 rows later filtered out. The released weights are selected by validation and development success, not the last trainer checkpoint. Runtime configs, training logs and evaluation records were recovered after a brief SSH outage. The optional 1.67 GB final optimizer-state transfer was stopped to avoid extending rental cost.
 
 Example fresh training from the published final corpus: