lattice-research/lattice-quark-1.5b

Lattice Quark

A 1.5B-parameter decoder-only language model trained from scratch by Lattice — no pretrained weights used. Pretrained on consumer hardware (RTX 5090), then instruction-tuned.

Specs

Params	1.5B
Architecture	GPT-style, 26 layers, 1536 embd, 12 heads
Context	2048 tokens
Vocab	32,768 (tiktoken-style BPE)
Pretraining	~2B tokens of SmolLM corpus, 1,500 iters
Instruction tune	465 iters, chat format (`<
Window pattern	"L" (attention sink + local window)

Training

• Phase 1 — base: 1,500 iters, val bits-per-byte 0.542 → 0.542 at completion.
• Phase 2 — SFT: instruction tuning, val loss 0.302 at step 465.

Checkpoints were streamed to this repo live during training (checkpoints/base/…, checkpoints/sft/…) — the raw .pt files are still here for reproducibility. The training pipeline is open source: github.com/olii-dev/nano-gpt.

Honest expectations

This is a from-scratch hobby-scale model. It holds a conversation, follows instructions, writes simple code, and has a stable identity — but it makes arithmetic errors, can be verbose, and will confidently say wrong things. Not for production. That's the point: it's a small model trained in the open, warts and all.

Observed behaviour (tested 2026-08-12)

Prompt	Result
"Who are you?"	"I'm Lattice Quark, a small language model built by Lattice Systems." ✓
"3 apples, take 2 away"	"you have 1 apple" ✓
"Explain gravity"	coherent multi-paragraph explanation ✓
"Fibonacci in Python"	clean implementation with error handling ✓
"17 × 23"	✗ arithmetic loop (typical at this scale)

Run it

The checkpoint uses the nanochat runner (custom architecture — no transformers config yet):

git clone https://github.com/olii-dev/nano-gpt
uv sync --extra gpu
python -m scripts.chat_cli -i sft -g quark-1.5b -s 465

The Lattice lineup

• Mini — 42M, from scratch (weights)
• Spark — 1.5B, Qwen fine-tune, the flagship (weights)
• Quark (this) — 1.5B, from scratch

More at lattice-research on Hugging Face and lattice-site-lime.vercel.app.