---
license: other
language: en
library_name: transformers
pipeline_tag: text-generation
tags:
- nanochat
- nemotron
- from-scratch
- perlmutter
- gpt2-tokenizer
---
# d24-sft-v1base-olmo3-2.3B

v1-base SFT chat model, OLMo-3 Dolmino-style midtrain.

nanochat-style **depth-24** decoder — 24 layers × 1536 hidden × 12 heads, SwiGLU / RoPE / RMSNorm, tied embeddings, GPT-2 BPE vocab (50304), **0.757B params**, 2048-token context.

**Lineage.** v1 pretrain (5.84B ClimbMix) → OLMo-3 Dolmino-style midtrain (2.3B corpus, 20 components incl. instruction/QA) → SFT (nanochat mix).

**Metrics.** GSM8K (greedy, full 1319): **4.93%** · SFT val lm-loss 0.222 (overfits SFT train via format familiarity).

## Use (chat)

This is a **chat model** (ChatML). The turn terminator it emits is the literal string **`<|im_end|>`** — which is **not** the `eos_token_id` (`50256` = `<|endoftext|>`) and is not even a single token. You **must** stop on the `<|im_end|>` string or generation will not stop:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "sfanm/d24-sft-v1base-olmo3-2.3B"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, torch_dtype="bfloat16", device_map="auto")

msgs = [{"role": "user", "content": "Natalia sold clips to 48 friends in April and half as many in May. How many total?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, do_sample=False, stop_strings=["<|im_end|>"], tokenizer=tok)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=False))
```

> Without `stop_strings=["<|im_end|>"]` the model rambles to `max_new_tokens`: the configured `eos_token_id` (50256) is the GPT-2 *document* EOS, which a chat turn does not end with. For vLLM, pass `stop=["<|im_end|>"]`.

*Research checkpoint from a from-scratch nanochat-d24 replication (pretrain → midtrain → SFT → RL) on NERSC Perlmutter. Trained on third-party corpora (ClimbMix, FineMath, OpenMath, MetaMath, OpenThoughts, OLMo-3 Dolmino, SmolTalk, …) — see those datasets' licenses; provided as-is for research.*