---
library_name: transformers
language:
  - en
tags:
  - spark-gpt
  - qwen3-moe
  - from-scratch
  - stories
  - checkpoints
---

# Lil Bard historical checkpoints

This repository contains 25 loadable Transformers checkpoints from the
pretraining run for [`N8Programs/lil-bard`](https://huggingface.co/N8Programs/lil-bard).
The final step-25,485 model is kept in that separate repository so ordinary
users do not download the full checkpoint history.

Checkpoints span step 1,000 through step 25,000 in increments of 1,000. Each
subfolder contains a complete `Qwen3MoeForCausalLM` export and tokenizer.

## Loading a checkpoint

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "N8Programs/lil-bard-checkpts"
subfolder = "step-0001000"

tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder=subfolder)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    subfolder=subfolder,
    dtype=torch.bfloat16,
    device_map="auto",
)
```

Available subfolders are `step-0001000`, `step-0002000`, ..., through
`step-0025000`.

These historical exports retain the fused expert-tensor layout used during the
run. They are tested with Transformers 5.11.0. Transformers 4.57.1 users should
use the cross-version final checkpoint in `N8Programs/lil-bard`.

## Run summary

Lil Bard is a 172,052,992-parameter English story MoE with 58,806,784 active
parameters per token, 8 experts with top-2 routing, and an 8,192-entry
whole-document byte-level BPE tokenizer. It was pretrained from scratch for
2,492,032,616 real loss tokens on two NVIDIA GB10 systems.

The final model, architecture and data details, evaluations, usage example,
and limitations are documented in the
[`N8Programs/lil-bard` model card](https://huggingface.co/N8Programs/lil-bard).
The complete training trace is available in the
[`lil_bard_moe_8x2_full` W&B run](https://wandb.ai/n8programs/sparkgpt/runs/mxk8gln1).

Only model exports are included here. The larger optimizer/scheduler resume
checkpoints are not uploaded.