khayyam-math/khayyam-math-qwen2.5-7b-v4

Khayyam Math — Qwen2.5-7B v4 (LoRA adapter)

A LoRA fine-tune of Qwen2.5-7B-Instruct specialised for generating deterministic SVG figures and learner-facing narrations for math education. This is the v4 production adapter that powers khayyammath.com.

The model emits structured figure specs (positioned primitives + relations

• phrase-timed narration) that downstream deterministic templates render into SVG. It is a domain specialist — not a general-purpose chat model.

• 🌐 Live demo: khayyammath.com
• 📦 Source code & package: github.com/khayyam-math/khayyam-math
• 📊 Benchmark: khayyam-math/lean-math-1000 (coming)

---

Quick start (with the khayyam-math package)

The easiest path — the package wraps Qwen, OpenAI, and a remote vLLM endpoint behind one unified interface.

pip install "khayyam-math[qwen]"          # ~5 GB, includes torch + transformers

from khayyam_math import KhayyamMath

client = KhayyamMath(provider="qwen",
                     model="khayyam-math/khayyam-math-qwen2.5-7b-v4")

svg = client.generate_figure("Solve x^2 - 5x + 6 = 0")
print(svg)

The first call downloads this adapter (162 MB) plus the Qwen2.5-7B base (14 GB) into the local Hugging Face cache and runs inference on GPU if available, otherwise CPU (slow).

Switching to OpenAI (GPT-4o) in one line

client = KhayyamMath(provider="openai", model="gpt-4o")
svg = client.generate_figure("Solve x^2 - 5x + 6 = 0")

No other code changes — same generate_figure signature, same return type. See the package README for the full provider table (openai, qwen, qwen-vllm).

---

Quick start (raw transformers + peft, no package)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base    = "Qwen/Qwen2.5-7B-Instruct"
adapter = "khayyam-math/khayyam-math-qwen2.5-7b-v4"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)
model.eval()

messages = [
    {"role": "system", "content": "You generate structured math figure specs."},
    {"role": "user",   "content": "Show the unit circle with 30° marked."},
]
inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True,
).to(model.device)

out = model.generate(inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

---

Serving with vLLM

The adapter is served behind vLLM in production. To replicate locally:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-lora \
  --lora-modules khayyam-v4=khayyam-math/khayyam-math-qwen2.5-7b-v4 \
  --max-lora-rank 16 \
  --dtype bfloat16

Then query with the OpenAI-compatible client using model="khayyam-v4".

---

Training

Field	Value
Base model	Qwen/Qwen2.5-7B-Instruct
Method	LoRA (rank 16, alpha 32, dropout 0.05)
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable params	~40 M (0.5 % of base)
Optimiser	AdamW, lr 2e-4
Schedule	3 epochs, batch 1 × grad-accum 4
Precision	bf16
Max sequence length	6144 tokens
Training corpus	5,528 (prompt → structured-spec) pairs distilled with gpt-4o-mini teacher + automated inspector filter
Final training loss	0.057
Final token accuracy	0.987
Hardware	Single RTX 5090 (~5 h wall-clock)
Framework	peft 0.19, trl 1.4, transformers 5.8
Trained on	2026-05-11

The training corpus is derived from the production figure pipeline of khayyammath.com: prompts crawled from the live system, teacher responses generated with gpt-4o-mini and filtered by an automatic inspector that rejects malformed SVG, layout-overlap violations, and math-correctness failures (SymPy + Z3 + Lean checks). A subset comes from telemetry-mined production "winners" — turns where the user did not click the "Not quite right?" feedback button.

---

Evaluation

20-problem held-out benchmark, judged by gpt-4o against a fixed rubric:

Metric	Score
Valid-figure rate	18 / 20 (90 %)
Mean rubric score (all attempts)	19.5 / 30
Mean rubric score (valid only)	21.6 / 30
Head-to-head vs. v3	10 wins · 8 losses · 2 ties

v4 outperforms v3 on the held-out set despite a smaller LoRA rank (r=16 with α=32 here vs r=8 with α=16 in v3) thanks to the inspector-filtered training corpus. The two failures were both empty-SVG outputs on eigendecomposition (2×2) and a three-set Venn diagram — class-specific gaps tracked for v5.

A larger production stress test — 1000 questions drawn from a Lean-style math bench across difficulties 6, 7, 9, and 10 — is documented in the accompanying paper and will be released as the lean-math-1000 dataset.

---

Intended use

• Generating structured math figure specs (positioned primitives, relations, phrase-timed narrations) from natural-language math prompts
• Powering production tutoring systems alongside the deterministic rendering and verification layers of the khayyam-math package
• Research on autoformalization, math-reasoning + figure generation, and learner-facing AI evaluation

Not intended for: general-purpose chat, free-form code generation, or safety-critical decision making.

---

Limitations

• Empty-SVG failures on eigendecomposition and complex set diagrams (~10 % of held-out tasks). The downstream khayyam-math pipeline recovers from these by falling back to a deterministic per-domain template when present, but the model itself does not yet emit a valid spec for these cases.
• English-only training data. Other languages are untested.
• Adapter only — the base model Qwen/Qwen2.5-7B-Instruct (~14 GB) is downloaded separately on first use.
• The adapter expects the chat template bundled in this repo. Loading with a different chat template degrades output sharply.

---

License

MIT. Base model (Qwen/Qwen2.5-7B-Instruct) is released under its own license; please consult upstream for terms.

---

Citation

A peer-reviewed paper describing this work has not yet been published. If you'd like to cite the model in the meantime, please use the model's Hugging Face URL together with the release version in your bibliography:

khayyam-math/khayyam-math-qwen2.5-7b-v4
https://huggingface.co/khayyam-math/khayyam-math-qwen2.5-7b-v4
Released 2026-05-11, MIT licence.

Acknowledgements

Built on Qwen/Qwen2.5-7B-Instruct (Alibaba), trained with peft and trl (Hugging Face), and served with vLLM. The training corpus was distilled with OpenAI's gpt-4o-mini as the teacher and verified by an automated inspector built on SymPy + Z3 + Lean 4.