Add model card

README.md

@@ -0,0 +1,110 @@
+---
+license: cc-by-4.0
+base_model: Qwen/Qwen2.5-Coder-14B-Instruct
+library_name: peft
+tags:
+  - text-to-sql
+  - clinical
+  - fhir
+  - healthcare
+  - dora
+  - lora
+  - qlora
+  - duckdb
+datasets:
+  - adelelsayed1991/fhirsql-reasoning-sql
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# Model Card: fhirsql-reasoning-sql adapters
+
+- **Model repo:** https://huggingface.co/adelelsayed1991/fhirsql-reasoning-sql-adapters
+- **Dataset repo:** https://huggingface.co/datasets/adelelsayed1991/fhirsql-reasoning-sql
+- **Code & paper:** https://github.com/adelelsayed/fhirsql-reasoning-sql
+
+DoRA/LoRA adapters for `Qwen/Qwen2.5-Coder-14B-Instruct`, fine-tuned to translate natural-language hospital questions into a structured JSON query plan followed by DuckDB SQL, against a FHIR-derived schema. See `PAPER.md` for the full study.
+
+## Repository contents
+
+Six adapters: 3 random seeds (42, 43, 44) x 2 training stages. SFT checkpoints are selected by **validation loss**; RL checkpoints by **dev-set execution match** (each stage's own trainer criterion):
+
+```
+sft/seed_42/best/    sft/seed_43/best/    sft/seed_44/best/
+rl/seed_42/best/     rl/seed_43/best/     rl/seed_44/best/
+```
+
+- `sft/` — supervised fine-tuning only (DoRA, rank 16, alpha 32). **Use these.**
+- `rl/` — the corresponding `sft/` seed's checkpoint, continued with DAPO/GRPO reinforcement learning. Per `PAPER.md` Section 5.3, the RL stage does not improve on its SFT starting point on this task (and is marginally worse in-corpus); all three seeds early-stopped at step 40 of 200 having peaked at the first evaluation checkpoint. These adapters are published for completeness and reproducibility, not because they outperform `sft/`.
+
+Each `best/` folder contains a standard PEFT adapter (`adapter_config.json`, `adapter_model.safetensors`, ~271MB).
+
+## Intended use
+
+Research artifact for reproducing or extending `PAPER.md`'s results. Generates a `{plan JSON}` + fenced ` ```sql ` completion for a natural-language question, given a prompt containing the DDL from `schema/schema.sql`. Not intended for use outside that schema/prompt format, and not validated on real (non-synthetic) patient data or real clinical schemas.
+
+## How to load
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from peft import PeftModel
+import torch
+
+base_name = "Qwen/Qwen2.5-Coder-14B-Instruct"
+tokenizer = AutoTokenizer.from_pretrained(base_name)
+base_model = AutoModelForCausalLM.from_pretrained(
+    base_name, dtype=torch.bfloat16,
+    quantization_config=BitsAndBytesConfig(
+        load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16,
+        bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4",
+    ),
+)
+model = PeftModel.from_pretrained(
+    base_model,
+    "adelelsayed1991/fhirsql-reasoning-sql-adapters",
+    subfolder="sft/seed_42/best",   # recommended; swap to "rl/..." to reproduce the RL arm
+)
+```
+
+Prompting details (system prompt template, schema DDL extraction, plan-then-SQL output format) are in `sft_train.ipynb` and `rl_train.ipynb`'s `SYSTEM_PROMPT_TEMPLATE`/`build_messages` cells.
+
+## Training data
+
+`data/training/sft_final_plan.jsonl` (10,696 rows: 9,680 execution-verified gold SQL + 1,016 abstention examples, spanning 82 archetypes and 2,420 distinct executable gold SQL statements), generated from a synthetic (Synthea) patient corpus — no real patient data was used anywhere in this project. Trained on Google Colab (G4 GPU, 96 GB RAM), 2 epochs per seed. See `DESIGN.md` and `METHODOLOGY_LOG.md` for full corpus and training-data generation methodology.
+
+## Evaluation summary
+
+Mean across the 3 SFT seeds on a held-out benchmark drawn from a disjoint patient population. Full results in `PAPER.md` Section 5; underlying per-seed data in `results/`.
+
+| | Frozen base | SFT adapter |
+|---|---:|---:|
+| Execution correctness, familiar concepts | 34.8% | 100.0% |
+| Execution correctness, unseen concepts | 60.8% | 89.1% |
+| Abstention precision / recall (familiar arm) | 68% / 72% | 100% / 100% |
+| Terminology-hardcoding rate | 0.0% | 0.0% |
+
+The frozen column uses a complete schema description including the `valuesets` DDL (the fair
+comparison). Under the exact training-time prompt, which omitted it, the frozen model scores
+10.2% / 31.3% — see `PAPER.md` §5.6.
+
+"Unseen concepts" are 84 clinical concepts appearing nowhere in training (verified by set
+intersection; zero shared question/query pairs) — the adapters compose correct
+terminology-resolution queries for them without ever having been trained on their codes. That
+89.1% is an upper bound; text-match metrics give 81.4% as a lower bound (`PAPER.md` §6).
+
+Abstention figures measure *retention* of trained refusal categories: the unanswerable questions
+are reused verbatim across training and evaluation, so this is not held-out refusal
+generalization (`PAPER.md` §2.7).
+
+## Limitations
+
+- Trained and evaluated entirely on synthetic (Synthea) data against one specific flattened schema (`schema/schema.sql`) — not validated against real clinical data or a different schema design.
+- All training and evaluation questions come from the same template-and-persona back-translation factory (`scripts/question_templates.py`, `scripts/personas.py`); robustness to free-form clinician phrasing outside that distribution is untested.
+- Single base model and scale (`Qwen2.5-Coder-14B-Instruct`, 14B parameters, 4-bit). Behavior at other scales or with other base models is untested.
+- The frozen-baseline column above is the *fair* comparison (complete schema description). The training-time prompt omitted the `valuesets` DDL, under which the frozen model scores lower; `PAPER.md` §5.6 quantifies both.
+- The unseen-concept arm shows a real ~11-point accuracy gap relative to familiar concepts. 78% of those failures are one benign, well-characterized pattern (over-applying SNOMED CT's parenthetical qualifier convention to concepts coded in other systems) — see `PAPER.md` Section 6.
+
+## License
+
+Adapter weights are released under **CC-BY-4.0**, matching the repository's paper/data license; the repository's code is Apache-2.0. See `LICENSE` and `CITATION.cff`. The base model `Qwen/Qwen2.5-Coder-14B-Instruct` retains its own license.