| --- |
| metrics: |
| - accuracy |
| pipeline_tag: text-classification |
| tags: |
| - code |
| datasets: |
| - shukdevdatta123/twitter_sentiment_preprocessed |
| language: |
| - en |
| base_model: distilbert/distilbert-base-uncased |
| library_name: transformers |
| license: cc-by-nd-4.0 |
| --- |
| # DistilBERT-base-uncased LoRA Text Classification Model |
|
|
| ## Model Description |
| This model is a fine-tuned version of `distilbert-base-uncased` on an unspecified dataset. It achieves the following results on the evaluation set: |
|
|
| - **Loss:** 0.4649 |
| - **Accuracy:** 84.16% |
|
|
| ## Intended Uses & Limitations |
| This is a text-classification based model. |
|
|
| ## Training and Evaluation Data |
| Look below for more details about the performances. |
|
|
| ## Steps to follow |
| - Installing the Libraries |
| - Loading the Dataset from HuggingFace |
| - Train_test Split the Dataset |
| - Model |
| - Preprocess Data |
| - Evaluation |
| - Apply untrained base model("distilbert-base-uncased") to text |
| - Train Model using LoRA |
| - Generate Prediction |
| - Save the Model and the Tokenizer |
| - Load the Model and the Tokenizer to test |
| - Push Model to HuggingFaceHub |
| |
| |
| |
| |
| ### Training Hyperparameters |
| The following hyperparameters were used during training: |
| |
| - **Learning Rate:** 0.001 |
| - **Train Batch Size:** 4 |
| - **Eval Batch Size:** 4 |
| - **Seed:** 42 |
| - **Optimizer:** Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| - **LR Scheduler Type:** Linear |
| - **Number of Epochs:** 10 |
| |
| ### Training Results |
| | Epoch | Training Loss | Validation Loss | Validation Accuracy | |
| |-------|---------------|-----------------|---------------------| |
| | 1.0 | 0.5924 | 0.5523 | 78.45% | |
| | 2.0 | 0.5983 | 0.5236 | 80.29% | |
| | 3.0 | 0.5703 | 0.4498 | 79.56% | |
| | 4.0 | 0.5526 | 0.4976 | 80.66% | |
| | 5.0 | 0.5326 | 0.4317 | 80.85% | |
| | 6.0 | 0.5851 | 0.4562 | 82.87% | |
| | 7.0 | 0.5466 | 0.4713 | 81.95% | |
| | 8.0 | 0.5494 | 0.5072 | 82.50% | |
| | 9.0 | 0.5748 | 0.4802 | 82.87% | |
| | 10.0 | 0.5001 | 0.4649 | 84.16% | |
| |
| ## Framework Versions |
| - **PEFT:** 0.12.0 |
| - **Transformers:** 4.42.4 |
| - **PyTorch:** 2.4.0+cu121 |
| - **Datasets:** 2.21.0 |
| - **Tokenizers:** 0.19.1 |
| |
| # Dataset Viewer |
| |
| You can view the dataset using the following link: |
| |
| [View Twitter Sentiment Preprocessed Dataset](https://huggingface.co/datasets/shukdevdatta123/twitter_sentiment_preprocessed/) |
| |
| Simply click the link to open the dataset viewer in your browser. |
| |
| # Model Viewer |
| |
| You can view the model using the following link: |
| |
| [View Model in HuggingFace](https://huggingface.co/shukdevdatta123/distilbert-base-uncased-lora-text-classification/) |
| |
| Simply click the link to open the model file in your browser. |
| |
| Check out the "Fine-tune LLM.pptx" file for the theory behind this code. |
| |
| # Github Repository |
| |
| You can view the github using the following link: |
| |
| [View GitHub Repository](https://github.com/shukdevtroy/Fine-Tune-LLM-using-LoRA-on-custom-dataset/) |
| |
| Simply click the link to open the github repo in your browser. |
| |
| Check out the "Fine-tune LLM.pptx" file in the GitHub repo for the theory behind this code. |
