---
library_name: peft
base_model: meta-llama/Llama-3.2-1B
tags:
- lora
- peft
- quantization
- glue
- sst2
---

# llama32-1B-sst2-int8-lora-seed42

LoRA adapter trained on GLUE **SST2** on top of a
**int8** backbone of `meta-llama/Llama-3.2-1B`.

Part of a controlled study of whether the backbone bit-width changes what a LoRA
adapter learns. For a given (model size, seed) the adapter initialisation is
**identical** across the bf16 / int8 / nf4 arms, and the data order, optimiser,
schedule and LoRA hyperparameters are held fixed — so any difference in the
learned update is attributable to the backbone.

## Result

| metric | validation | test |
|---|---|---|
| accuracy | 0.9640 | 0.9541 |
| macro-F1 | 0.9635 | 0.9541 |
| loss | 0.1229 | 0.1597 |

Test-set majority-class baseline: 0.5092

- peak GPU memory: 2.97 GiB
- training time: 47.9 min (936 steps)
- GPU: NVIDIA GeForce RTX 4090

## Setup

- seed: `42` · adapter init: `shared:lora_init_1B_seed42.pt:128tensors`
- LoRA: r=16, alpha=32, dropout=0.0, bias=none,
  target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj']
- trainable params: 3,407,872
- epochs 3, lr 0.0002,
  max_len 256, batch 4
  x grad_accum 16,
  cosine schedule, warmup 0.03

## Prompt format

Trained as causal LM with the loss on the answer letter only (prompt tokens
masked to -100):

```
Sentence: ...

Is the sentiment of this sentence positive or negative?

A. Negative
B. Positive

Answer:
```

Evaluated by conditional likelihood over the answer letters
(Negative, Positive).

> GLUE `test` is unlabeled, so the official `validation` split is used as TEST
> and the validation set is carved from `train` (disjoint).