DiogenesChen122/Qwen3.6-27B-Lora-20260826

Qwen3.6-27B — DrSparse Phase 0 SFT LoRA (2026-08-26)

Sequence-level distillation of GPT-5.6 teacher trajectories into Qwen/Qwen3.6-27B for CUDA sparse kernel generation (SpMV / SpMM / SpGEMM). Part of Dr.Sparse — Phase 0 checkpoint used as the initialization for Phase 1 (DPO) and Phase 2 (multi-turn GRPO).

Supersedes the 2026-08-17 checkpoint, which was trained on the smaller 172-sample 1.5× dataset. This one uses the full 4046-sample 1.05× corpus.

Adapter

• Format: PEFT LoRA (~934 MB)
• Rank / α: 64 / 128
• Target modules (12): q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (dense + MLP) and in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj (linear-attention layers of Qwen3.6's hybrid stack). Vision tower is excluded — the adapter lands only on model.language_model.*.
• base_model_name_or_path: Qwen/Qwen3.6-27B.

Data

• Teacher: GPT-5.6, tree-search runs across the entire Dr.Sparse evaluation corpus (level 1–4, multiple GPU generations).
• Selection: rejection-sampling SFT — keep a turn iff the kernel compiled, verified numerically, and beat cuSPARSE by ≥ 1.05×. 4046 kept (3841 train / 205 val, held out by run_id) out of 5420 mined turns.
• Op mix (4046 samples): 1023 spmm_k8 · 810 spmm_k32 · 435 spmm_k128 · 329 spmm_k256 · 753 spgemm · 696 spmv. Much better balance than the earlier 1.5× threshold — SpMV and SpGEMM are meaningfully represented now.
• Corpus size: ~57 M tokens/epoch, med sequence 13.5 k tokens, max 22 k.

Training

framework	verl 0.8.0, SFT trainer
precision	bfloat16 (frozen base + bf16 compute; LoRA in fp32)
optimizer	AdamW, LR 1e-5, cosine, 3 % warmup, weight decay 0
epochs / steps	3 / 2880 total optimizer steps
global batch	4 (dynamic sequence packing)
max_length	24 576 (samples above this dropped at build time)
loss mask	assistant-turn only (pre-flight asserted)
hardware	2 × NVIDIA B200 (single node, FSDP2)
wall clock	≈ 42 h total (18.0 h on resume from step 960)

Val loss (best) = 0.03070 at the end of epoch 3. Training curves in the Dr.Sparse W&B project drsparse_distill.

Loading

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen3.6-27B"
lora_id = "DiogenesChen122/Qwen3.6-27B-Lora-20260826"

tok = AutoTokenizer.from_pretrained(lora_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    base_id, torch_dtype=torch.bfloat16,
    device_map="auto", trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, lora_id)

Merge and serve via vLLM:

CUDA_VISIBLE_DEVICES=0,1 vllm serve <merged-dir> \
  --served-model-name Qwen/Qwen3.6-27B \
  --tensor-parallel-size 2 --dtype bfloat16 \
  --language-model-only --trust-remote-code \
  --max-model-len 32768 --port 8027

Coder system prompt

Trained with the exact system turn the Dr.Sparse coding agent uses at inference:

You are a CUDA sparse kernel specialist.

Chat template is applied with enable_thinking=False.