Spectral Surgery
Collection
16 items • Updated
Qwen3-8B + metamath — LoRA, training seed 43
This repository contains the unedited final LoRA adapter, not a full 8B base model and not an HNS-edited adapter. It is one of the 12 B300 replication runs (two bases × three training tasks × seeds43/44), completed September 13, 2026. HNS is a post-hoc transformation of this saved LoRA, not additional training. All HNS scores below use this source checkpoint; derived HNS weights are not uploaded here. Reconstruction code and metadata are included.
Base Model
- Model: Qwen/Qwen3-8B.
- Download revision:
b968826d9c46dd6066d109eabc6255188de91218. Per-file local download revisions and weight/tokenizer SHA256 hashes are inpublication.json. - Base weights are not redistributed. Users must obtain access to the base and comply with its license, acceptable-use policy and dataset terms. No independent license grant is implied by this adapter release.
Training — requested versus effective configuration
| Field | Actual saved run / provenance |
|---|---|
| Dataset | meta-math/MetaMathQA, local train parquet snapshot |
| Source rows | 395000 |
| Selected rows before truncation filtering | 50000 |
| Actual response-supervised training rows | 50000 |
| Training epochs | 3.0 |
| Actual final optimizer updates | 4689 |
| Maximum sequence length | 4096 |
| Per-device micro-batch / accumulation | 16 / 2 |
| GPU count / effective global batch | 1 NVIDIA B300 / 32 |
| LR / scheduler | 0.0001 / cosine |
| Actual warmup | 0 steps (zero warmup) |
| CLI-requested warmup ratio | 0.05 — did NOT take effect |
| Scheduler kwargs | null |
| Optimizer | adamw_torch_fused, Adam betas (0.9, 0.999), epsilon 1e-08 |
| Weight decay / max gradient norm | 0.0 / 1.0 |
| LoRA | r=16, alpha=32, dropout=0.05, bias=none |
| Target modules | down_proj, v_proj, k_proj, up_proj, o_proj, q_proj, gate_proj |
| Precision / checkpointing | bf16 / gradient checkpointing enabled; not QLoRA |
| SFT | Chat template non_thinking; response-only loss, prompt labels masked -100 |
| Truncation / padding | Right truncation; drop examples with no supervised completion tokens; dynamic right padding to multiple8; no packing |
| Training seed / Trainer data_seed | 43 / 43 |
| Dataset subset seed | 42, unchanged between seeds43/44 |
| Determinism | full_determinism=False; no claim of bitwise reproducibility |
Warmup audit correction: this code passes CLI warmup_ratio, then filters TrainingArguments kwargs against the installed signature. Transformers5.16.1 does not expose that argument, so the requested ratio was dropped and warmup_steps=0 remained. training_args.json and the archived source are authoritative for the effective run, not the requested CLI alone. The reproduction command deliberately requests ratio0. Do not describe these runs as having 5% or 10% warmup.
Data selection: valid-format filtering, then datasets.Dataset.shuffle(seed=42).select(range(50000)) for Magicoder/MetaMath; Tulu uses the full valid split without downsampling. The row-index list in data/selected_source_indices.json.gz references the exact local parquet row ordering before tokenization/truncation filtering. Source file SHA256: 5bcc40288ba3927eb859a10e07e118c09543789c1e7373a645174345f526b76e. Upstream dataset revision was not recorded by the original asset export; do not claim that downloading current main recreates the exact bytes/order. Verify the file hash or resolve snapshot provenance before claiming exact data reproduction. Training text is not redistributed here.
Saved evidence: run_args.json (requested), run_config.json (pre-tokenization estimates), training_args.json (effective), trainer_state.json (actual final steps and logged training metrics), requirements-freeze.txt, and publication.json. Local paths and credential fields are sanitized. Pickled optimizer states / training_args.bin are intentionally omitted.
Evaluation
Benchmark: GSM8K, primary metric strict_accuracy, 1319 items. Each trained checkpoint was evaluated once on the complete available in-domain split. This is not three repetitions of inference on one checkpoint. Seeds43/44 are separate training runs. No training-seed CI or significance claim is made from a single row.
| Setting | Value |
|---|---|
| Backend / attention | vLLM / FLASH_ATTN, tensor parallel1 |
| Sampling | Greedy: temperature0, top_p1, inference seed42 |
| Chat | non_thinking render mode; Qwen enable_thinking=False |
| Maximum model length / new tokens | 4096 / 512 |
| GPU memory / max concurrent sequences | 0.94 / 1024 |
| Token budget / adapter block / prompt chunk | 65536 / 11 / 512 |
| Scheduling / prefix cache | async_scheduling=False / enable_prefix_caching=False |
| Numerics | VLLM_BATCH_INVARIANT=1, CUBLAS_WORKSPACE_CONFIG=:4096:8 |
| Compilation cache | Disabled, per-job cache root, short IPC temp path |
HumanEval uses chat strict-continuation prompts, max_new_tokens512, pass@1, code-execution timeout3s and 32 CPU workers. GSM8K uses max_new_tokens512 and the strict answer extractor in the archived scorer (not a 2048-token model-card evaluation). IFEval uses its 541-item train-named evaluation split, max_new_tokens2048, prompt-level strict accuracy. The GSM8K/IFEval local benchmark inputs were reconstructed from earlier scored outputs (gold and instruction metadata), not newly sampled; the exact input file and hash are included under evaluation/benchmark_input/. HumanEval input is the local test parquet. Do not mix these results with earlier model-card scores from other prompts/token budgets.
| Method | Score (%) | Correct / samples | Change vs LoRA (pp) |
|---|---|---|---|
| Base (not a training replicate) | 85.75 | 1131/1319 | — |
| LoRA (weights in this repository) | 84.00 | 1108/1319 | +0.00 |
| 0+0 SVD reconstruction control | 83.93 | 1107/1319 | -0.08 |
| HNS 2+0, all modules | 87.41 | 1153/1319 | +3.41 |
| HNS 2+1, all modules | 87.72 | 1157/1319 | +3.71 |
| HNS 2+2, all modules | 86.96 | 1147/1319 | +2.96 |
| HNS 4+0, all modules | 87.19 | 1150/1319 | +3.18 |
| HNS 4+1, all modules | 86.81 | 1145/1319 | +2.81 |
| HNS 4+2, all modules | 87.19 | 1150/1319 | +3.18 |
| HNS 8+0, all modules | 87.11 | 1149/1319 | +3.11 |
| HNS 8+1, all modules | 87.19 | 1150/1319 | +3.18 |
| HNS 8+2, all modules | 87.11 | 1149/1319 | +3.11 |
HNS grid: all seven LoRA module types, output rank16, strength1, preserve original module nuclear norm, fast steps2/4/8 × stable steps0/1/2. 0+0 is an SVD-factorization reconstruction control, not spectral editing. Maxima on this test set are descriptive, not validated parameter selection. In IFEval, reconstruction itself can change scores materially; all gains over LoRA cannot automatically be attributed to spectral editing.
Machine-readable full metrics are in evaluation/results.json; paired per-item evidence and generated token IDs/text are in evaluation/items/<variant>/scored.jsonl.gz and predictions.jsonl.gz. Off-task forgetting evaluation was still incomplete at publication preparation; no incomplete forgetting scores are included or implied.
Reproduction and loading
Use an isolated environment matching the recorded package versions. requirements-freeze.txt is the full training environment inventory, not a guarantee that all platform-specific packages install on arbitrary systems. The source archive is a publication-time snapshot, with per-file hashes; a clean training-time Git commit was not saved. Python version and evaluation-time package versions are recorded in publication.json (current evaluation environment observation is distinguished from training inventory).
# In the downloaded repository directory:
tar -xzf code/source_snapshot.tar.gz
pip install --no-deps -e source_snapshot
# Obtain the exact training parquet under /path/to/data/; its SHA256 is checked.
python code/reproduce_hns_seed_checkpoint.py train --data-dir /path/to/data --output-dir /path/to/new-run
# Rebuild all nine HNS variants plus the 0+0 control from the root LoRA:
python code/reproduce_hns_seed_checkpoint.py build-hns --output-dir /path/to/rebuilt-grid
# In-domain benchmark inputs are included; record any inference-budget override:
python code/reproduce_hns_seed_checkpoint.py evaluate --grid-dir /path/to/rebuilt-grid --output-dir /path/to/new-eval
The archived training entrypoint and the explicit command implement the actual zero-warmup run. With other Transformers versions, defaults/Trainer behavior may differ; recorded data hashes, model revisions, tokenizer files, preprocessing and effective settings are necessary checks, not a promise of bitwise identical training. For the successful Llama43 evaluation the recorded batched-token budget was131072, whereas the other new groups used65536. This stack has also failed at131072 in other initializations; evaluate --token-budget 65536 is a safer alternative but a changed evaluation configuration and must be reported.
build-hns also fetches the other two source LoRAs for this same base/training seed, verifies their published weight hashes, and reconstructs the original three-task 33-adapter manifest ordering. This preserves adapter registration IDs for the target-task evaluation rather than renumbering an isolated 11-adapter grid. --peer-root can point to an offline directory containing those named repository folders. For stability the portable helper disables the compilation cache; this differs from the earlier successful Llama43 run, whose scheduler log shows default compile-cache use. No original compiled cache is redistributed, and exact-token numerical identity is not promised.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
repo_id = "tianzl66/Qwen3-8B-MetaMathQA-50K-LoRA-Seed43"
base_id = "Qwen/Qwen3-8B"
base_revision = 'b968826d9c46dd6066d109eabc6255188de91218'
tokenizer = AutoTokenizer.from_pretrained(repo_id)
base = AutoModelForCausalLM.from_pretrained(
base_id, revision=base_revision, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, repo_id)
model.eval()
adapter_config.json uses the public base-model ID rather than a private filesystem path; this metadata normalization does not change adapter_model.safetensors. Original adapter-config hash is recorded separately. Root tokenizer and chat-template files are the saved training artifacts. Loading example is not itself a benchmark reproduction protocol.
Comparison with the historical seed42 checkpoint
The older checkpoint is separately listed in comparison/three_run_scores.json and comparison/configuration_audit.md. It is a historical reference, not verified to be an identical-recipe third seed. New runs use larger micro-batches, padding8 and actual zero warmup; original dataset identity and some Llama settings are not fully verified. Both new seeds share the same saved non-seed recipe. Old Llama Magicoder/MetaMath training seed labels lack complete original Trainer evidence in this audit. Do not pool 42/43/44 into a strict identical-configuration three-seed mean±SD. Standard deviation is not a confidence interval.
Files and integrity
adapter_model.safetensors,adapter_config.json, saved tokenizer/chat template: loadable PEFT source LoRA.- Training JSON evidence, final
trainer_state.json, requirements inventory. publication.json,data/*,hns/*,evaluation/*,comparison/*: provenance, HNS metadata, input/output evidence and historical comparison.code/*: source archive and portable train/build/evaluate helper.MANIFEST.sha256: hashes of all prepared payload files except itself; remote commit ID is tracked in the publisher's upload receipt.
The pre-normalization trained weight SHA256 is 2357aede2caec8a7861b935294eb7f32b35a07bc30694154d8d7a20bc85df10b. No intermediate checkpoint, optimizer state, full base weight or scheduler log is uploaded. This release documents reproducibility boundaries rather than claiming random variation was eliminated.
- Downloads last month
- 17