Spectral Surgery — Math
Collection
Math reasoning adapters and Spectral Surgery ablations on MetaMathQA-50K. • 12 items • Updated
Llama-3.1-8B-Instruct + MetaMathQA + Spectral Surgery
This repository contains a Spectral Surgery adapter derived from the Llama-3.1-8B-Instruct MetaMathQA-50K LoRA checkpoint.
Base Model
meta-llama/Llama-3.1-8B-Instruct
Training
- Dataset: MetaMathQA
- Samples: 50K
- LoRA rank: 16
Spectral Surgery
- Target modules:
o_proj+down_proj - Fast HNS steps: 8
- Stable HNS steps: 2
Evaluation
Evaluation on GSM8K.
| Model | GSM8K |
|---|---|
| Base | 65.20% (860/1319) |
| LoRA | 77.18% (1018/1319) |
| HNS 8+2, o_proj + down_proj | 78.39% (1034/1319) |
| HNS 8+2, all modules | 79.38% (1047/1319) |
| HNS 4+1, o_proj + down_proj | 78.17% (1031/1319) |
| HNS 4+1, all modules | 79.38% (1047/1319) |
Relative to the vanilla LoRA checkpoint, this configuration improves GSM8K accuracy by 1.21 percentage points (+16 correct answers).
- Downloads last month
- 9
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for tianzl66/Llama-3.1-8B-Instruct-MetaMathQA-50K-SpectralSurgery-HNS8p2-OD
Base model
meta-llama/Llama-3.1-8B Finetuned
meta-llama/Llama-3.1-8B-Instruct