SoFarSoGoodya/DeepMath-R1-Distill-Qwen-7B

DeepMath (LoRA SFT + DPO on DeepSeek-R1-Distill-Qwen-7B)

🇨🇳 中文简介 below.

DeepMath is a math-reasoning fine-tune of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, produced by a two-stage LoRA pipeline with LLaMA-Factory (template deepseekr1):

DeepSeek-R1-Distill-Qwen-7B ──SFT (LoRA, NuminaMath-CoT)──► merged ──DPO (LoRA, math prefs)──► DeepMath

This repo contains LoRA adapters, not a merged model. It was built in a 5-day sprint on rented consumer GPUs (8×5090 for SFT, 1×5090 for DPO) — the full engineering story is in the GitHub repo.

⚠️ What's in here & how to use it

There are two adapters, applied in sequence:

Path	Role	Size
sft_adapter/	Stage 1 — SFT LoRA (NuminaMath-CoT)	78 MB
./ (top level)	Stage 2 — DPO LoRA (math preference pairs), the final adapter	77 MB

The DPO adapter was trained on top of the SFT-merged model, so to reconstruct the final model you must apply SFT first, then DPO. Use the included merge.py:

pip install transformers peft torch accelerate
python merge.py \
  --base deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --sft  ./sft_adapter \
  --dpo  ./ \
  --out  ./DeepMath-merged

Or load adapters on the fly (see merge.py docstring / the GitHub examples/inference.py).

Training summary (honest)

Stage	Config	Outcome
SFT	8×5090, native DDP, LoRA r=8/α=16 (target=all), lr 5e-5, 1 epoch, cutoff 2048, 1,410 steps	loss 0.73 → 0.39
DPO	1×5090, QLoRA 4-bit NF4 + double-quant, lr 1e-6, 3 epochs, cutoff 1024, 456 steps	loss 0.6947 (≈ ln 2 baseline), reward acc ~0.5–0.55

Limitations & intended use

• This is a learning/pipeline-demonstration artifact, not a SOTA model. SFT converged cleanly; DPO completed without mode collapse but its gain is modest (reward accuracy only slightly above the 50% baseline) and no held-out benchmark was evaluated.
• Best understood as a reproducible reference for a resource-constrained SFT→DPO pipeline. Don't deploy it expecting strong math-reasoning gains.
• Base model DeepSeek-R1-Distill-Qwen-7B inherits its biases/limits; tokenizer derives from Qwen.

Provenance & license

• Base: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B (MIT).
• SFT data: AI-MO/NuminaMath-CoT (Apache-2.0), cleaned 100k→90,217.
• DPO data: 2,418 math preference pairs (distilabel-math-preference style).
• Adapters + code: MIT. Please retain upstream attributions.

Author: @SoFarSoGoodya — full attribution & contact in the GitHub repo.

---

中文

DeepMath 是基于 DeepSeek-R1-Distill-Qwen-7B 的数学推理微调,用 LLaMA-Factory 走 LoRA SFT(NuminaMath)→ 合并 → LoRA DPO(数学偏好对) 两段式流水线。

本仓库是 LoRA adapter(非合并模型)。含两个 adapter,需按顺序使用:先 sft_adapter/(SFT),再顶层目录(DPO,最终)。用自带的 merge.py 一条命令重建完整模型(见上方命令)。

如实说明:这是一个学习/流水线演示作品,不是 SOTA 模型。SFT 收敛良好;DPO 完整跑通但提升有限(奖励准确率仅略高于随机),且未做独立评测。权重与代码采用 MIT;请保留对上游(DeepSeek / Qwen / NuminaMath)的署名。完整工程复盘见 GitHub 仓库。