Qwen3-8B + Instruction Following + Spectral Surgery

This repository contains a Spectral Surgery adapter derived from the all-linear instruction-following LoRA checkpoint of Qwen3-8B.

Post-hoc Spectral Surgery applies Hybrid Newton-Schulz (HNS) editing directly to the LoRA adapter. No additional gradient-based training is performed during Spectral Surgery.

Base Model

Qwen/Qwen3-8B

Source LoRA

Dataset: Tulu-3 SFT instruction-following split
Samples: 29,980
Epochs: 2
Sequence length: 4096
Global batch size: 128
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.05
Learning rate: 4e-4
LR schedule: cosine, warmup ratio 0.03
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Chat template: non-thinking (enable_thinking=False)
Seed: 42

Spectral Surgery

Target: all LoRA modules
Modules edited: 252
Fast HNS steps: 4
Stable HNS steps: 1
Output rank: 16
Nuclear norm: preserved
Mean effective rank: 11.7908 → 15.9982 (4+1) / 16.0000 (8+2)
See spectral_edit_meta.json for exact edit metadata.

Evaluation

Evaluation on IFEval (541 prompts, 834 instructions).

Model	Prompt Strict	Prompt Loose	Instruction Strict	Instruction Loose
Base	75.23%	81.70%	82.97%	87.53%
LoRA SFT	74.31%	78.37%	81.41%	85.13%
Spectral Surgery HNS 4+1	75.79%	80.78%	83.21%	86.69%
Spectral Surgery HNS 8+2	75.60%	80.78%	83.09%	86.57%

Relative to the vanilla LoRA checkpoint, Spectral Surgery HNS 4+1 improves:

Prompt Strict by 1.48 percentage points
Prompt Loose by 2.40 percentage points
Instruction Strict by 1.80 percentage points
Instruction Loose by 1.56 percentage points

Compared with the base model, this HNS edit improves both strict metrics; the two loose metrics remain slightly below base.

Settings: Qwen3 non-thinking chat template (enable_thinking=False), greedy decoding, max_new_tokens=2048, vLLM backend, FLASH_ATTENTION, max model length 4096, seed 42.