---
base_model: meta-llama/Llama-3.2-3B-Instruct
library_name: peft
pipeline_tag: text-generation
language:
- en
tags:
- peft
- lora
- text-to-sql
- nl2sql
- spider
- sqlite
---

# Llama 3.2 3B Instruct NL2SQL LoRA

This repository contains the final LoRA adapter for the Llama 3.2 3B Instruct model line used in a master's thesis project on local large language models for NL2SQL generation.

The adapter generates SQLite queries from natural-language questions and a relational database schema. It must be loaded together with the corresponding base model.

## Base model

- Model: `meta-llama/Llama-3.2-3B-Instruct`
- Revision: `0cb88a4f764b7a12671c53f0838cd831a0843b95`
- Access to the base model may require acceptance of Meta's license terms on Hugging Face.

## Adapter

- Method: LoRA supervised fine-tuning
- LoRA rank: 8
- LoRA alpha: 16
- LoRA dropout: 0.05
- Target modules: all suitable linear modules
- Quantization during training: none
- Maximum training sequence length: 2,048 tokens
- Best checkpoint: `checkpoint-509`
- The published root adapter corresponds to the selected best checkpoint.

SHA-256 of `adapter_model.safetensors`:

`fcd4241f7a2e8e0388f13f0dd9517486cbee43fc3169c983a54e7b716c0e502d`

## Training configuration

- Training examples: 25,000
- Spider Train examples: 6,960
- SQL Create Context examples: 18,040
- Validation set: MixedVal2500-v2
- Validation examples: 2,500
- Learning rate: `1e-4`
- Scheduler: constant
- Train batch size: 2
- Gradient accumulation steps: 4
- Effective batch size: 8
- Seed: 42
- Maximum epochs: 5
- Early stopping patience: 2
- Early stopping threshold: 0.001
- Precision: FP16
- Gradient checkpointing: enabled
- Attention implementation: FlashAttention 2

Spider Dev was not used for training, validation, early stopping, or checkpoint selection.

## Evaluation

The final adapter was evaluated on all 1,032 Spider Dev cases.

Zero-shot evaluation:

- Execution Match Accuracy: 61.05% (630/1,032)
- Execution Success Rate: 86.82% (896/1,032)
- Maximum input length: 2,048 tokens
- Maximum generated tokens: 256

The corresponding starting model achieved an Execution Match Accuracy of 55.04% (568/1,032) under the same zero-shot evaluation condition.

## Intended prompt behavior

The model is instructed to return only a valid SQLite query:

- no explanation
- no Markdown
- no comments
- no unnecessary tables or columns
- only `SELECT` or `WITH` queries
- output terminated with a semicolon

The native Llama chat template and the exact project-specific prompt construction are documented in the accompanying GitHub repository.

## Loading

~~~python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "meta-llama/Llama-3.2-3B-Instruct"
adapter_id = "mehmet1899/llama32-3b-instruct-nl2sql-lora"
adapter_revision = "87afdd0c565da4570ebd129a4098f50719e0f76e"

tokenizer = AutoTokenizer.from_pretrained(
    base_model_id,
    revision="0cb88a4f764b7a12671c53f0838cd831a0843b95",
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    revision="0cb88a4f764b7a12671c53f0838cd831a0843b95",
    torch_dtype=torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    model,
    adapter_id,
    revision=adapter_revision,
)
model.eval()
~~~

## Reproducibility

Code, training and evaluation configurations, environment information, run manifests, and result summaries are available at:

`https://github.com/md181099/nl2sql-masterthesis`

The files `training_metadata.json`, `training_history.csv`, and `training_history.jsonl` provide additional training provenance.

## Limitations

- The adapter was evaluated primarily on the Spider benchmark and SQLite databases.
- Performance on other database systems or unseen schema conventions is not guaranteed.
- Access to the base model is governed by Meta's model license.
- Execution Match depends on the database contents and the execution-based evaluation procedure.
- The model may still generate invalid, incomplete, or semantically incorrect SQL.
- The adapter should not be used to execute unrestricted queries against production databases without validation and access controls.