tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-SpectralSurgery-HNS8p2-AllMods

Qwen2.5-7B-Instruct + Commonsense170K — Spectral Surgery HNS 8+2

This repository contains a post-hoc Spectral Surgery adapter derived from the Qwen2.5-7B-Instruct Commonsense170K LoRA checkpoint. No additional gradient-based training is performed during Spectral Surgery.

Source LoRA

• Source: tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-LoRA
• Dataset: Commonsense170K, 170,420 examples
• Epochs: 2 (10,652 optimizer steps)
• Sequence length: 2,048
• Global batch size: 32
• Learning rate: 2e-4
• LoRA rank/alpha/dropout: 16 / 32 / 0.05
• Target modules: all seven LoRA projection types
• Seed: 42

Spectral Surgery

• Method: Hybrid Newton-Schulz (HNS)
• Scope: all 196 LoRA modules
• Fast/stable steps: 8 + 2
• Output rank: 16
• Nuclear norm: preserved
• Mean effective rank: 11.1021 → 16.0000

Exact per-module statistics and coefficients are recorded in spectral_edit_meta.json.

Evaluation

Greedy evaluation on the eight-task commonsense suite, using the tokenizer chat template in non-thinking mode and at most 8 generated tokens.

Model	Macro accuracy	Micro accuracy	Correct
Base	83.4194%	84.1786%	18,872 / 22,419
LoRA	89.7406%	91.2976%	20,468 / 22,419
Spectral Surgery HNS 8+2	89.9206%	91.1147%	20,427 / 22,419

HNS changes macro accuracy by +0.1801 percentage points relative to the source LoRA. Its micro accuracy changes by -0.1829 points (-41 correct answers), so the effect is a redistribution across tasks rather than a uniform improvement.

Task	Base	LoRA	HNS 8+2	HNS − LoRA
BoolQ	85.9327%	87.7982%	87.5841%	-0.2141 pp
PIQA	85.9086%	90.0979%	89.9891%	-0.1088 pp
SocialIQA	75.0256%	82.1392%	81.6274%	-0.5118 pp
HellaSwag	83.9574%	94.1346%	93.6766%	-0.4581 pp
WinoGrande	64.7987%	89.0292%	88.3189%	-0.7103 pp
ARC-Easy	96.0859%	95.2441%	95.7492%	+0.5051 pp
ARC-Challenge	89.8464%	88.4812%	89.4198%	+0.9386 pp
OpenBookQA	85.8000%	91.0000%	93.0000%	+2.0000 pp

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(
    base,
    "tianzl66/Qwen2.5-7B-Instruct-CommonSense170K-SpectralSurgery-HNS8p2-AllMods",
)

Files

• adapter_model.safetensors / adapter_config.json: edited PEFT adapter
• spectral_edit_meta.json: exact HNS configuration and per-module statistics
• run_args.json, run_config.json, training_args.json: source training configuration
• eval-commonsense8/: aggregate metrics and per-example HNS predictions
• comparison-summary.json / .tsv: Base, LoRA, and HNS comparison