Mistral 7B – LoRA r=4 (MedQuAD)
Fine-tuned with LoRA (r=4) using dataset lavita/MedQuAD
Base model: mistralai/Mistral-7B-Instruct-v0.3
Quantization: 4-bit NF4
GPU: A100 80GB
📊 Final Results
- Validation Loss: 0.8431
- Perplexity: 2.324
🧾 Training Metrics
| Epoch | Step | Training Loss | Validation Loss |
|---|---|---|---|
| 0.33 | 200 | 0.855 | 0.852 |
| 0.66 | 400 | 0.821 | 0.826 |
| 1.0 | 600 | 0.807 | 0.812 |
💻 Example usage
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
model = AutoPeftModelForCausalLM.from_pretrained("REPO_ID", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
text = "[INST] What is anemia? [/INST]"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=80)
print(tokenizer.decode(out[0], skip_special_tokens=True)
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for OSCARcr/mistral-medquad-lora-r4
Base model
mistralai/Mistral-7B-v0.3 Finetuned
mistralai/Mistral-7B-Instruct-v0.3