README.md

---
library_name: transformers
license: mit
tags:
  - sentiment-analysis
  - bert
  - lora
  - peft
  - huggingface
  - transformers
  - text-classification
  - low-resource
model-index:
  - name: LoRA-BERT for Sentiment Analysis (SST-2)
    results:
      - task:
          type: text-classification
          name: Sentiment Analysis
        dataset:
          type: glue
          name: SST2
        metrics:
          - type: accuracy
            value: 0.9117
            name: Accuracy
datasets:
- stanfordnlp/sst2
language:
- en
metrics:
- accuracy
base_model:
- google-bert/bert-base-uncased
pipeline_tag: text-classification
---

# 🤖 LoRA-BERT for Sentiment Analysis (SST-2)

This is a lightweight, parameter-efficient BERT model fine-tuned with [LoRA (Low-Rank Adaptation)](https://arxiv.org/abs/2106.09685) for binary sentiment classification on the SST-2 dataset.

---

## 💡 Model Highlights

- ✅ Fine-tuned using **LoRA** (r=8, α=16) on top of `bert-base-uncased`
- ✅ Trained on [SST2](https://huggingface.co/datasets/stanfordnlp/sst2)
- ✅ Achieves ~91.17% validation accuracy
- ✅ Lightweight: only LoRA adapter weights are updated

---

## 📊 Results

| Epoch | Training Loss | Validation Loss | Accuracy |
|-------|---------------|-----------------|----------|
| 1     | 0.3030        | 0.2467          | 89.91%   |
| 2     | 0.1972        | 0.2424          | 90.94%   |
| 3     | 0.2083        | 0.2395          | 91.17%   |
| 4     | 0.1936        | 0.2464          | 90.94%   |
| 5     | 0.1914        | 0.2491          | 90.83%   |

Early stopping could be applied from Epoch 3 based on validation metrics.

---

## 🛠️ Usage

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel, PeftConfig

model_id = "Harsh-Gupta/bert-lora-sentiment"

# Load PEFT config + model
config = PeftConfig.from_pretrained(model_id)
base_model = AutoModelForSequenceClassification.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(base_model, model_id)

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Predict
text = "This movie was absolutely amazing!"
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
with torch.no_grad():
    outputs = model(**inputs)
    probs = outputs.logits.softmax(dim=-1)
    pred = probs.argmax().item()
```

---
## LoRA Configuration
```python
LoraConfig(
    r=32,
    lora_alpha=4,
    target_modules=["query", "value"],
    lora_dropout=0.1,
    bias="none",
    task_type="SEQ_CLS"
)
```
---
## 🔍 Intended Use
- Sentiment classification for binary text (positive/negative)

- Can be adapted to other domains: movie reviews, product reviews, tweets

---
## 🧠 Author
- Harsh Gupta
- MCA, Jawaharlal Nehru University (JNU)
- GitHub: [2003Harsh](https://github.com/2003HARSH)