aniruddhr04/qwen2.5-3b-nl2sql-spider-grpo

Qwen2.5-3B-Instruct — NL2SQL Spider GRPO (step 100)

Qwen/Qwen2.5-3B-Instruct trained with GRPO in a tool-calling environment where the model issues real run_sql queries against live SQLite databases and is rewarded on whether its final query returns the same rows as the gold query.

Trained with verl 0.7.1 through Oumi's DatabaseExecutableEnvironment.

Results — Spider dev (1034 rows, greedy, single sample)

metric	base	this model	change
official test-suite execution accuracy	0.336	0.644	×1.92
official exact-set-match	0.195	0.397	×2.04
legacy evaluation.py execution	0.264	0.583	×2.21

Quote the test-suite number (0.644). It is the current official Spider metric (Zhong, Yu & Klein, EMNLP 2020, taoyds/test-suite-sql-eval). The legacy evaluation.py under-reports this model by 6.1 points because 242 of its 1034 predictions cannot be parsed by Spider's process_sql.py (subqueries, trailing ;, AS aliases), and an unparseable prediction scores 0 regardless of the rows it returns.

By difficulty (test-suite)

	easy (248)	medium (446)	hard (174)	extra (166)	all
this model	0.863	0.668	0.592	0.307	0.644
base	0.431	0.370	0.322	0.114	0.336

Multi-table reasoning is the bottleneck: easy 0.86 → extra-hard 0.31.

Intended use

A research artifact demonstrating that execution feedback works as an RL signal for text-to-SQL. It is not a state-of-the-art Spider model — published 3B systems reach 79–84% using constrained decoding, schema ranking, full SFT, and multi-candidate inference, none of which are used here. The result is 100 RL steps on a generic (non-code) instruct model.

How it was trained

algorithm	GRPO (adv_estimator: grpo, norm_adv_by_std_in_grpo: true), KL loss disabled
hardware	8×H100, single node, ~80 s/step, ~2h30m total
batch	64 prompts × n=8 = 512 trajectories/step
completion budget	2048 tokens shared across all turns
lr	1e-6 constant, AdamW
sampling	train temp 1.0; eval greedy (t=0, n=1)
reward	binary result-set match against the gold query
train data	6,997 of 7,000 train_spider.json rows (3 dropped — their gold SQL does not execute)
eval data	1,034 dev.json rows, none dropped

The reward is outcome-only: it looks at the final SQL and nothing else — not turn count, not whether a tool was called, not style.

Prompt format

The model expects a system message containing the full DDL of every table in the target database, a user question, and hermes-style tool calling with a single run_sql(query) tool. It answers with a fenced ```sql block.

[system] You are a SQL assistant for a SQLite database with this schema:
         CREATE TABLE singer (...)
         You may call the tool run_sql(query) to execute a SQL query and inspect
         results. When you are done, give your final answer as a single SQL query
         inside a ```sql code block.
[user]   How many singers do we have?

[assistant] <tool_call>{"name":"run_sql","arguments":{"query":"SELECT COUNT(*) FROM singer;"}}</tool_call>
[tool]      {"columns": ["COUNT(*)"], "rows": [[6]]}
[assistant] ```sql
            SELECT COUNT(*) FROM singer;
            ```

To reproduce the numbers above you must replay the training tool loop. One-shot prompting measures a different system and understates badly. Tool responses must be truncated to 256 characters, middle-elided, exactly as verl does during training:

half = max_length // 2
text = text[:half] + "...(truncated)..." + text[-half:]

Behavioral profile

	base	this model
valid executable SQL	—	94%
answers with a query it actually ran	43.9%	85.4%
run_sql calls per row	1.79	1.02
predictions unparseable by Spider's parser	62.6%	24.0%

Training made tool use precise rather than eliminating it, and made SQL markedly more standard despite the reward never mentioning style. The dominant remaining failure is 197 rows (19.1%) that run a query, see real rows, and still answer wrong — schema grounding on multi-table questions, not tool use and not syntax.

Limitations

• Single seed, no error bars, one run per configuration.
• Evaluated on Spider dev. The held-out test split was never touched.
• Greedy single-sample only; no self-consistency or multi-candidate reranking.
• A later checkpoint (step 250) scores higher on the internal metric (0.752 vs 0.727) but was never run through official eval. This checkpoint (step 100) is the one every number above refers to.

Citation

Environment and reward implementation: oumi-ai/oumi PRs #2560 (verl tool adapter), #2561 (NL2SQL env + reward), #2562 (Spider demo).