| --- |
| tags: |
| - LoRA |
| - QLoRa |
| - LoRA Adapter |
| - LLaMA |
| model-index: |
| - name: lora-sql-guanaco-13b-adapter |
| results: [] |
| datasets: |
| - richardr1126/sql-create-context_guanaco_style |
| spaces: |
| - richardr1126/NL2SQL-Guanaco-Chat |
| --- |
| |
| # lora-sql-guanaco-13b-adapter |
|
|
| This is a LoRA adapter for [richardr1126/guanaco-13b-merged](https://huggingface.co/richardr1126/guanaco-13b-merged), or any other merged guanaco-13b model, fine tuned from LLaMA. |
| <br> |
| This LoRA was fine-tuned using QLoRA techniques on the [richardr1126/sql-create-context_guanaco_style](https://huggingface.co/datasets/richardr1126/sql-create-context_guanaco_style) dataset. |
|
|
| ### Training hyperparameters |
|
|
| The following hyperparameters were used during training: |
| - learning_rate: 0.0002 |
| - train_batch_size: 4 |
| - eval_batch_size: 8 |
| - seed: 42 |
| - gradient_accumulation_steps: 4 |
| - total_train_batch_size: 16 |
| - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| - lr_scheduler_type: linear |
| - lr_scheduler_warmup_ratio: 0.03 |
| - training_steps: 1875 |
| - mixed_precision_training: Native AMP |
|
|
| ### Framework versions |
|
|
| - Transformers 4.30.0.dev0 |
| - Pytorch 2.0.1+cu118 |
| - Datasets 2.13.0 |
| - Tokenizers 0.13.3 |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{dettmers2023qlora, |
| title={QLoRA: Efficient Finetuning of Quantized LLMs}, |
| author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke}, |
| journal={arXiv preprint arXiv:2305.14314}, |
| year={2023} |
| } |
| ``` |
