---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- from-scratch
- pretraining
- bilingual
- piece-tokenizer
language:
- en
- zh
---

# Summer-0.5B-S0

Summer-0.5B-S0 is a **from-scratch** bilingual (Chinese/English) base model:
524,336,128 parameters, trained from random initialization on
**14.6B tokens** with a self-trained 81,903-piece tokenizer.

It is *not* a fine-tune or a tokenizer-swap of any existing model. The
architecture follows `Qwen/Qwen3-0.6B-Base` (28 layers / hidden 1024 /
GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight
starts from `N(0, 0.02)`.

## Stage

**S0** — monolingual only, from-scratch pretraining. 14.6B tokens, English
70% / Chinese 30%, 55,694 steps.

This is a retrain of the original `Summer-0.5B-S0`, redesigned to align with
nanochat's actual training recipe:

- **English is a single source** (FineWebEdu, `HuggingFaceFW/fineweb-edu`),
  matching nanochat's `karpathy/fineweb-edu-100b-shuffle` — not the five-source
  blend the original release used.
- **Chinese is still multiple sources**, but screened first: two sources
  found to be 51.8% / 55.7% Traditional-Chinese-dominant by sampled character
  frequency (against a downstream instruction mix that is 100% Simplified)
  were dropped rather than mixed in.
- **Packing is BOS-aligned best-fit** (`bos_bestfit`): every training row
  starts with `<bos>` and packs whole documents greedily by best fit, cropping
  only the one document (if any) that doesn't fit the remainder — the same
  algorithm as nanochat's `tokenizing_distributed_data_loader_with_state_bos_bestfit`,
  verified line-by-line against nanochat's source. The original release used a
  continuous stream with no BOS token at all.
- **Sequence length is 2048**, matching nanochat (`max_seq_len=2048` since its
  first commit, used unchanged at every depth from d4 to d26+).

**Every input to this model must start with `<bos>`.** It has never seen a
sequence that doesn't. `example_load.py` / `example_vllm.py` in this repo do
this for you — if you tokenize text yourself without prepending `<bos>`, the
model will produce degenerate repetitive output regardless of prompt.

No parallel or instruction data at any point.

## What to expect

**This is a 14.6B-token model.** For scale: `Qwen3-0.6B-Base` saw 36T tokens —
about 2,700x more. Treat the numbers below as what that budget buys, not as a
competitive result.

### Letter multiple-choice (nanochat's primary format — render the question, model answers a single letter)

| | previous release | this release |
|---|---|---|
| ARC-Easy | 0.2563 | 0.2668 |
| ARC-Challenge | 0.2338 | 0.2654 |
| MMLU | 0.2306 | 0.2551 |
| C-Eval | 0.2166 | 0.2527 |

This release scores above the random baseline (0.25) on all four tasks; the
previous release was below it on three of four.

### Likelihood-scoring protocol (lm-evaluation-harness convention)

| | previous release | this release |
|---|---|---|
| ARC-Easy (acc_norm) | 0.4949 | 0.5391 |
| ARC-Challenge (acc_norm) | 0.2671 | 0.3003 |
| MMLU (acc) | 0.2520 | 0.2465 |
| C-Eval (acc_norm) | 0.2363 | 0.2348 |
| GSM8K 8-shot (flexible-extract) | 0.0121 | 0.0167 |

### WMT22 5-shot translation

| | BLEU | COMET |
|---|---|---|
| zh->en | 0.29 | 0.4354 |
| en->zh | 2.78 | 0.5505 |

**Few-shot translation is still essentially zero — expected, this checkpoint
has never seen parallel text.** The model produces fluent but off-topic
continuations rather than translations; it does not yet follow the in-context
examples. See `Summer-0.5B-S1` for the version where in-context translation
appears. Its value is as (a) a from-scratch bilingual base model in its own
right, and (b) a starting point for annealing / SFT.

## Tokenizer

The tokenizer is a compiled C++ extension, **not** loadable by
`AutoTokenizer`. The release ships `tokenizer.py` and `example_load.py`:

```bash
pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py
```

The model code (`model.py`, `checkpoint.py`) is bundled too — the package
depends only on `torch` plus the tokenizer extension, not on `transformers`.

## Training

Full pipeline, data mixes and every design decision (including the mistakes)
are documented in <https://github.com/Ismantic/Summer>. Notably
`docs/WHY.md` records why fp32 master weights are mandatory, why the learning
rate schedule is WSD rather than cosine, and what the vocabulary swap cost.

## License

Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia,
CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see `data/source.py` upstream).
Please observe their respective licenses.