README.md

---
base_model: Qwen/Qwen3-8B
library_name: peft
pipeline_tag: text-generation
datasets:
- ise-uiuc/Magicoder-Evol-Instruct-110K
tags:
- lora
- spectral-surgery
- training-seed-replication
- b300
- code
- seed-44
---

# Qwen3-8B + magicoder — LoRA, training seed 44

This repository contains the **unedited final LoRA adapter**, not a full 8B base model and not an HNS-edited adapter. It is one of the 12 B300 replication runs (two bases × three training tasks × seeds43/44), completed September 13, 2026. HNS is a post-hoc transformation of this saved LoRA, not additional training. All HNS scores below use this source checkpoint; derived HNS weights are not uploaded here. Reconstruction code and metadata are included.

## Base Model

- Model: [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B).
- Download revision: `b968826d9c46dd6066d109eabc6255188de91218`. Per-file local download revisions and weight/tokenizer SHA256 hashes are in `publication.json`.
- Base weights are not redistributed. Users must obtain access to the base and comply with its license, acceptable-use policy and dataset terms. No independent license grant is implied by this adapter release.

## Training — requested versus effective configuration

| Field | Actual saved run / provenance |
|---|---|
| Dataset | `ise-uiuc/Magicoder-Evol-Instruct-110K`, local train parquet snapshot |
| Source rows | 111183 |
| Selected rows before truncation filtering | 50000 |
| Actual response-supervised training rows | 49936 |
| Training epochs | 1.0 |
| Actual final optimizer updates | 1561 |
| Maximum sequence length | 4096 |
| Per-device micro-batch / accumulation | 16 / 2 |
| GPU count / effective global batch | 1 NVIDIA B300 / 32 |
| LR / scheduler | 2e-05 / `cosine` |
| **Actual warmup** | **0 steps (zero warmup)** |
| CLI-requested warmup ratio | 0.05 — did NOT take effect |
| Scheduler kwargs | `null` |
| Optimizer | `adamw_torch_fused`, Adam betas (0.9, 0.999), epsilon 1e-08 |
| Weight decay / max gradient norm | 0.0 / 1.0 |
| LoRA | r=16, alpha=32, dropout=0.05, bias=none |
| Target modules | up_proj, q_proj, k_proj, gate_proj, o_proj, v_proj, down_proj |
| Precision / checkpointing | bf16 / gradient checkpointing enabled; not QLoRA |
| SFT | Chat template `non_thinking`; response-only loss, prompt labels masked -100 |
| Truncation / padding | Right truncation; drop examples with no supervised completion tokens; dynamic right padding to multiple8; no packing |
| Training seed / Trainer data_seed | 44 / 44 |
| Dataset subset seed | 42, unchanged between seeds43/44 |
| Determinism | full_determinism=False; no claim of bitwise reproducibility |

**Warmup audit correction:** this code passes CLI `warmup_ratio`, then filters TrainingArguments kwargs against the installed signature. Transformers5.16.1 does not expose that argument, so the requested ratio was dropped and `warmup_steps=0` remained. `training_args.json` and the archived source are authoritative for the effective run, not the requested CLI alone. The reproduction command deliberately requests ratio0. Do not describe these runs as having 5% or 10% warmup.

Data selection: valid-format filtering, then `datasets.Dataset.shuffle(seed=42).select(range(50000))` for Magicoder/MetaMath; Tulu uses the full valid split without downsampling. The row-index list in `data/selected_source_indices.json.gz` references the exact local parquet row ordering **before** tokenization/truncation filtering. Source file SHA256: `ff1dda789a725a9b8840f63aa9956bcff397b8dc4311cfc98aaa2c11ba0ed0ca`. Upstream dataset revision was not recorded by the original asset export; do not claim that downloading current main recreates the exact bytes/order. Verify the file hash or resolve snapshot provenance before claiming exact data reproduction. Training text is not redistributed here.

Saved evidence: `run_args.json` (requested), `run_config.json` (pre-tokenization estimates), `training_args.json` (effective), `trainer_state.json` (actual final steps and logged training metrics), `requirements-freeze.txt`, and `publication.json`. Local paths and credential fields are sanitized. Pickled optimizer states / training_args.bin are intentionally omitted.

## Evaluation

Benchmark: **HumanEval**, primary metric `pass_at_1`, 164 items. Each trained checkpoint was evaluated once on the complete available in-domain split. This is not three repetitions of inference on one checkpoint. Seeds43/44 are separate training runs. No training-seed CI or significance claim is made from a single row.

| Setting | Value |
|---|---|
| Backend / attention | vLLM / FLASH_ATTN, tensor parallel1 |
| Sampling | Greedy: temperature0, top_p1, inference seed42 |
| Chat | non_thinking render mode; Qwen enable_thinking=False |
| Maximum model length / new tokens | 4096 / 512 |
| GPU memory / max concurrent sequences | 0.94 / 1024 |
| Token budget / adapter block / prompt chunk | 65536 / 11 / 512 |
| Scheduling / prefix cache | async_scheduling=False / enable_prefix_caching=False |
| Numerics | VLLM_BATCH_INVARIANT=1, CUBLAS_WORKSPACE_CONFIG=:4096:8 |
| Compilation cache | Disabled, per-job cache root, short IPC temp path |

HumanEval uses chat strict-continuation prompts, max_new_tokens512, pass@1, code-execution timeout3s and 32 CPU workers. GSM8K uses max_new_tokens512 and the strict answer extractor in the archived scorer (not a 2048-token model-card evaluation). IFEval uses its 541-item `train`-named evaluation split, max_new_tokens2048, prompt-level strict accuracy. The GSM8K/IFEval local benchmark inputs were reconstructed from earlier scored outputs (gold and instruction metadata), not newly sampled; the exact input file and hash are included under `evaluation/benchmark_input/`. HumanEval input is the local test parquet. Do not mix these results with earlier model-card scores from other prompts/token budgets.

| Method | Score (%) | Correct / samples | Change vs LoRA (pp) |
|---|---:|---|---:|
| Base (not a training replicate) | 66.46 | 109/164 | — |
| LoRA (weights in this repository) | 64.63 | 106/164 | +0.00 |
| 0+0 SVD reconstruction control | 64.63 | 106/164 | +0.00 |
| HNS 2+0, all modules | 72.56 | 119/164 | +7.93 |
| HNS 2+1, all modules | 72.56 | 119/164 | +7.93 |
| HNS 2+2, all modules | 73.17 | 120/164 | +8.54 |
| HNS 4+0, all modules | 72.56 | 119/164 | +7.93 |
| HNS 4+1, all modules | 74.39 | 122/164 | +9.76 |
| HNS 4+2, all modules | 73.78 | 121/164 | +9.15 |
| HNS 8+0, all modules | 76.22 | 125/164 | +11.59 |
| HNS 8+1, all modules | 73.78 | 121/164 | +9.15 |
| HNS 8+2, all modules | 75.00 | 123/164 | +10.37 |

HNS grid: all seven LoRA module types, output rank16, strength1, preserve original module nuclear norm, fast steps2/4/8 × stable steps0/1/2. `0+0` is an SVD-factorization reconstruction control, not spectral editing. Maxima on this test set are descriptive, not validated parameter selection. In IFEval, reconstruction itself can change scores materially; all gains over LoRA cannot automatically be attributed to spectral editing.

Machine-readable full metrics are in `evaluation/results.json`; paired per-item evidence and generated token IDs/text are in `evaluation/items/<variant>/scored.jsonl.gz` and `predictions.jsonl.gz`. Off-task forgetting evaluation was still incomplete at publication preparation; no incomplete forgetting scores are included or implied.

## Reproduction and loading

Use an isolated environment matching the recorded package versions. `requirements-freeze.txt` is the full training environment inventory, not a guarantee that all platform-specific packages install on arbitrary systems. The source archive is a publication-time snapshot, with per-file hashes; a clean training-time Git commit was not saved. Python version and evaluation-time package versions are recorded in `publication.json` (current evaluation environment observation is distinguished from training inventory).

```bash
# In the downloaded repository directory:
tar -xzf code/source_snapshot.tar.gz
pip install --no-deps -e source_snapshot
# Obtain the exact training parquet under /path/to/data/; its SHA256 is checked.
python code/reproduce_hns_seed_checkpoint.py train --data-dir /path/to/data --output-dir /path/to/new-run
# Rebuild all nine HNS variants plus the 0+0 control from the root LoRA:
python code/reproduce_hns_seed_checkpoint.py build-hns --output-dir /path/to/rebuilt-grid
# In-domain benchmark inputs are included; record any inference-budget override:
python code/reproduce_hns_seed_checkpoint.py evaluate --grid-dir /path/to/rebuilt-grid --output-dir /path/to/new-eval
```

The archived training entrypoint and the explicit command implement the actual zero-warmup run. With other Transformers versions, defaults/Trainer behavior may differ; recorded data hashes, model revisions, tokenizer files, preprocessing and effective settings are necessary checks, not a promise of bitwise identical training. For the successful Llama43 evaluation the recorded batched-token budget was131072, whereas the other new groups used65536. This stack has also failed at131072 in other initializations; `evaluate --token-budget 65536` is a safer alternative but a changed evaluation configuration and must be reported.

`build-hns` also fetches the other two source LoRAs for this same base/training seed, verifies their published weight hashes, and reconstructs the original three-task 33-adapter manifest ordering. This preserves adapter registration IDs for the target-task evaluation rather than renumbering an isolated 11-adapter grid. `--peer-root` can point to an offline directory containing those named repository folders. For stability the portable helper disables the compilation cache; this differs from the earlier successful Llama43 run, whose scheduler log shows default compile-cache use. No original compiled cache is redistributed, and exact-token numerical identity is not promised.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

repo_id = "tianzl66/Qwen3-8B-Magicoder-50K-LoRA-E1-Seed44"
base_id = "Qwen/Qwen3-8B"
base_revision = 'b968826d9c46dd6066d109eabc6255188de91218'
tokenizer = AutoTokenizer.from_pretrained(repo_id)
base = AutoModelForCausalLM.from_pretrained(
    base_id, revision=base_revision, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, repo_id)
model.eval()
```

`adapter_config.json` uses the public base-model ID rather than a private filesystem path; this metadata normalization does not change `adapter_model.safetensors`. Original adapter-config hash is recorded separately. Root tokenizer and chat-template files are the saved training artifacts. Loading example is not itself a benchmark reproduction protocol.

## Comparison with the historical seed42 checkpoint

The older checkpoint is separately listed in `comparison/three_run_scores.json` and `comparison/configuration_audit.md`. It is a historical reference, **not verified to be an identical-recipe third seed**. New runs use larger micro-batches, padding8 and actual zero warmup; original dataset identity and some Llama settings are not fully verified. Both new seeds share the same saved non-seed recipe. Old Llama Magicoder/MetaMath training seed labels lack complete original Trainer evidence in this audit. Do not pool 42/43/44 into a strict identical-configuration three-seed mean±SD. Standard deviation is not a confidence interval.

## Files and integrity

- `adapter_model.safetensors`, `adapter_config.json`, saved tokenizer/chat template: loadable PEFT source LoRA.
- Training JSON evidence, final `trainer_state.json`, requirements inventory.
- `publication.json`, `data/*`, `hns/*`, `evaluation/*`, `comparison/*`: provenance, HNS metadata, input/output evidence and historical comparison.
- `code/*`: source archive and portable train/build/evaluate helper.
- `MANIFEST.sha256`: hashes of all prepared payload files except itself; remote commit ID is tracked in the publisher's upload receipt.

The pre-normalization trained weight SHA256 is `eb3899888d043eb204ac10f9557344a1fabc0bb4d4e82965ae6a615047243932`. No intermediate checkpoint, optimizer state, full base weight or scheduler log is uploaded. This release documents reproducibility boundaries rather than claiming random variation was eliminated.