Update README.md

README.md

@@ -1,4 +1,8 @@
 ---
+title: WikiSQL Qwen2.5-Coder QLoRA
+emoji: 🧠
+colorFrom: blue
+colorTo: green
 license: apache-2.0
 base_model: Qwen/Qwen2.5-Coder-3B-Instruct
 tags:
@@ -16,19 +20,21 @@ language:
 
 ## 1. Introduction
 
-This project fine-tunes a language model for natural-language-to-SQL generation. The goal is to take a table schema and a plain-English question, then return the correct SQL query. This task is important because many users need to query structured data but may not know SQL syntax. Current LLMs can often generate SQL, but they may struggle with exact column names, formatting, capitalization, and producing only the query without extra text. I fine-tuned [`Qwen/Qwen2.5-Coder-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) using QLoRA on WikiSQL-style examples. After fine-tuning, normalized exact-match accuracy on the held-out WikiSQL test set improved from 0.27 to 0.64, showing that the adapter made the model more consistent at producing the expected SQL format.
+Natural-language-to-SQL generation is a useful task because many people need to work with structured data but do not know how to write SQL queries. The goal of this project is to take a table schema and a plain-English question and return the correct SQL query. Current LLMs can often generate SQL, but they may struggle with exact column names, capitalization, formatting, and returning only the query without extra explanation. To address this, I fine-tuned `Qwen/Qwen2.5-Coder-3B-Instruct` using QLoRA on WikiSQL-style examples. After fine-tuning, the model’s normalized exact-match accuracy on the held-out WikiSQL test set improved from 0.27 to 0.64.
 
 ## 2. Data
 
-I used the [`mlx-community/wikisql`](https://huggingface.co/datasets/mlx-community/wikisql) dataset from Hugging Face. WikiSQL is a natural-language-to-SQL dataset where each example contains a table, column names, a plain-English question, and the target SQL query. I reformatted each example into an instruction-response format. The instruction includes the table ID, the list of columns, and the user question, while the response is the correct SQL query. I used the dataset’s built-in splits rather than creating a new random split: 500 examples from the training split for fine-tuning, 100 examples from the validation split for selecting the best adapter, and 100 examples from the test split for final evaluation.
+For fine-tuning, I used the `mlx-community/wikisql` dataset from Hugging Face. Each example contains a table, column names, a natural-language question, and a target SQL query. I reformatted each example into an instruction-response format. The instruction includes the table ID, the list of columns, and the user question, while the response is the correct SQL query. I used the dataset’s built-in splits instead of creating a new random split: 500 training examples for fine-tuning, 100 validation examples for selecting the best adapter, and 100 test examples for final evaluation.
 
 ## 3. Methodology
 
-I used QLoRA fine-tuning instead of full fine-tuning because the base model has about 3 billion parameters, making full fine-tuning more expensive on available GPU resources. QLoRA keeps the base model quantized while training small low-rank adapter weights, which makes the training process more practical while still adapting the model to the SQL-generation task. The base model was loaded in 4-bit precision using bitsandbytes, and LoRA adapters were trained on the attention and feed-forward projection layers. I tested three QLoRA hyperparameter settings: rank 8 with alpha 16, rank 16 with alpha 32, and rank 16 with a lower learning rate and higher dropout. The best adapter used rank 16, alpha 32, dropout 0.05, and learning rate 0.0002. Training used 50 max steps, per-device train batch size 2, gradient accumulation 4, bf16, paged AdamW 8-bit optimization, and validation loss to select the best checkpoint.
+For the training method, I used QLoRA fine-tuning. I chose QLoRA because it is more practical than full fine-tuning for a 3-billion-parameter model on available GPU resources. QLoRA loads the base model in 4-bit precision and trains small LoRA adapter weights instead of updating all of the original model parameters. This makes training more memory-efficient while still allowing the model to adapt to the text-to-SQL task.
+
+I tested three QLoRA hyperparameter settings. The first used rank 8 and alpha 16 to keep the adapter smaller. The second used rank 16 and alpha 32 to give the adapter more capacity. The third also used rank 16 and alpha 32, but lowered the learning rate and increased dropout to reduce overfitting. The best adapter used rank 16, alpha 32, dropout 0.05, and learning rate 0.0002. Training used 50 max steps, per-device train batch size 2, gradient accumulation 4, bf16, paged AdamW 8-bit optimization, and validation loss to select the best checkpoint.
 
 ## 4. Evaluation
 
-I evaluated the model before and after fine-tuning using a held-out WikiSQL test split and three `lm_eval` benchmark tasks. The WikiSQL test split is the most important evaluation because it directly measures the project task: converting a schema and natural-language question into the correct SQL query. The `lm_eval` tasks were used as broader computer-science and reasoning checks to see whether task-specific SQL fine-tuning caused major drops in general performance.
+To evaluate the model, I compared the base model and the fine-tuned adapter on the held-out WikiSQL test set and three `lm_eval` benchmark tasks. The WikiSQL test set is the most important benchmark because it directly measures the final project task: converting a schema and natural-language question into the correct SQL query. The three `lm_eval` tasks were included to check whether SQL fine-tuning caused major changes in broader computer-science performance.
 
 | Benchmark | Base Model | Fine-Tuned Adapter | Change |
 |---|---:|---:|---:|
@@ -37,13 +43,15 @@ I evaluated the model before and after fine-tuning using a held-out WikiSQL test
 | global_mmlu_full_en_high_school_computer_science | 0.68 | 0.70 | +0.02 |
 | global_mmlu_full_en_computer_security | 0.75 | 0.73 | -0.02 |
 
-The main result is that WikiSQL normalized exact-match accuracy increased from 0.27 to 0.64 after QLoRA fine-tuning. This suggests that the adapter learned the expected SQL output format and improved the model’s ability to map natural-language questions to table columns, conditions, and SQL structure. The broader `lm_eval` scores stayed relatively close to the base model scores. College computer science decreased slightly, high school computer science increased slightly, and computer security decreased slightly. Overall, the results suggest that the QLoRA adapter improved the main text-to-SQL task without seriously damaging broader computer-science performance.
+The results show that QLoRA fine-tuning substantially improved the main text-to-SQL task. WikiSQL normalized exact-match accuracy increased from 0.27 to 0.64, which suggests that the adapter learned the expected SQL output format and became better at mapping natural-language questions to columns, conditions, and SQL structure. The broader benchmark scores stayed relatively stable. College computer science decreased slightly, high school computer science increased slightly, and computer security decreased slightly. Overall, this suggests that the adapter improved task-specific SQL generation without seriously hurting broader computer-science performance.
 
-For model selection, I originally compared multiple open Hugging Face models during prompting experiments, including TinyLlama, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-7B-Instruct. TinyLlama served as a small baseline, Qwen2.5-Coder-3B-Instruct served as the medium code-focused model, and Qwen2.5-Coder-7B-Instruct served as the larger code-focused comparison model. Qwen2.5-Coder-3B-Instruct was selected for fine-tuning because it gave the best balance of text-to-SQL performance and practicality for training on Rivanna.
+I originally compared multiple open Hugging Face models during prompting experiments, including TinyLlama, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-7B-Instruct. TinyLlama served as a small baseline, Qwen2.5-Coder-3B-Instruct served as the medium code-focused model, and Qwen2.5-Coder-7B-Instruct served as the larger code-focused comparison model. Qwen2.5-Coder-3B-Instruct was selected for fine-tuning because it gave the best balance of text-to-SQL performance and practicality for training on Rivanna.
 
 ## 5. Usage and Intended Uses
 
-This model is intended for natural-language-to-SQL generation on WikiSQL-style single-table questions. A user provides a table ID, column names, and a plain-English question, and the model returns only the SQL query. The model is best suited for educational or experimental text-to-SQL tasks where the schema is provided directly in the prompt. It is not intended for production database use without validation, because generated SQL may still contain incorrect column names, conditions, or assumptions.
+The intended use case for this model is natural-language-to-SQL generation on WikiSQL-style single-table questions. A user provides a table ID, column names, and a plain-English question, and the model returns only the SQL query. This model is best used for educational or experimental text-to-SQL tasks where the schema is provided directly in the prompt. It should not be used for production database querying without human review because the model may still generate incorrect SQL.
+
+Below is an example of how to load the QLoRA adapter on top of the original base model.
 
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
@@ -71,4 +79,33 @@ base_model = AutoModelForCausalLM.from_pretrained(
 )
 
 model = PeftModel.from_pretrained(base_model, adapter_name)
-model.eval()
+model.eval()
+
+
+## Prompt Format
+
+
+You are a text-to-SQL assistant. Return only the SQL query.
+
+Table: 1-10015132-16
+Columns: Player, No., Nationality, Position, Years in Toronto, School/Club Team
+Question: What is terrence ross' nationality
+
+SQL:
+
+## Expected Output Format
+SELECT Nationality FROM 1-10015132-16 WHERE Player = 'Terrence Ross'
+
+
+
+
+Limitations
+
+The main limitation of this model is that it was fine-tuned on WikiSQL-style single-table examples. Because of this, it may not generalize well to complex SQL tasks involving joins, nested queries, multiple tables, or unfamiliar database schemas. The model can still hallucinate column names or produce invalid SQL, especially if the prompt does not include enough schema information. The evaluation uses normalized exact-match accuracy, which is strict and may count logically similar SQL queries as incorrect if they differ in formatting, capitalization, or string values. Since this repository contains a QLoRA adapter, it should be loaded on top of the original Qwen2.5-Coder-3B-Instruct base model.
+
+
+Citations and Links
+Base model: Qwen/Qwen2.5-Coder-3B-Instruct
+Dataset: mlx-community/wikisql
+PEFT library: peft
+Evaluation library: lm-evaluation-harness