---
license: apache-2.0
language:
- en
tags:
- text-generation
- conversational
- gpt
- nanogpt
- pytorch
- from-scratch
- small-language-model
- slm
- sft
- ultrachat
- muon-optimizer
- single-gpu
- research
pipeline_tag: text-generation
---

![image](https://cdn-uploads.huggingface.co/production/uploads/6a487897487a4a16ab1124f2/IBlgQMPLpclofJGpBQWCl.png)

# GTM-v2-chat by OpenGCM

A ~120M parameter, decoder-only GPT-style model, base-pretrained from
scratch and then fine-tuned (SFT) on UltraChat to behave like a chat
assistant. Single RTX Pro 6000, ~4.16B pretraining tokens + ~1B SFT
tokens.

This is the SFT/chat variant of [GTM-v2-base](https://hf.co/OPENGCM/GTM-v2-base) — same
architecture and pretrained weights, continued with supervised
fine-tuning on conversational data. **This is still a small, from-scratch,
single-GPU hobby/research model, not a production assistant.** It is far
less capable and less reliable than commercial chat models, and can
confidently produce fluent but incorrect answers.

## What SFT changed

The base model (GTM-v2-base) does not answer questions — given
`"What is the capital of France?"` it continues the text as if it were
part of a document, without ever reliably stating the answer, no matter
how it's prompted. After SFT on UltraChat, the same underlying model
answers directly:

```
<|user|>
What is the capital of France?
<|assistant|>
The capital of France is Paris. The city has seven major cities including
London, Paris, and Paris itself.<|endofturn|>
```

That's a real, qualitative capability shift — the model went from never
producing a correct, direct answer across an entire base-pretraining run
to answering directly, with the correct fact, after SFT. **This is one
example, not a rigorous evaluation** — it is not yet known how often this
succeeds versus fails on other questions. Base-model benchmark scores
(HellaSwag, ARC) were not meaningfully different pre- vs. post-SFT, since
those are likelihood-scored on fixed text and don't test chat-style
question answering.

## Chat template

The tokenizer has no special chat tokens — conversations are plain text
with these literal markers:

```
<|user|>
{user turn}
<|assistant|>
{assistant turn}<|endofturn|>
```

Prompt the model by ending your input with `<|assistant|>\n` so it
continues as the assistant turn.

## Model details

- **Architecture**: nanoGPT-style decoder-only transformer (same as
  GTM-v2-base) — 14 layers, 8 attention heads, 704 embedding dim, ~119.4M
  parameters, 1024 token context, PyTorch fused
  `scaled_dot_product_attention`.
- **Tokenizer**: `tiktoken` GPT-2 BPE encoding (`tiktoken.get_encoding("gpt2")`),
  vocab size 50,257.
- **Optimizer**: Muon (2D weight matrices) + AdamW (embeddings, layernorms,
  biases), fresh optimizer state for the SFT stage (not carried over from
  pretraining).
- **Precision**: trained with bf16 autocast; released weights are fp32.
- **Base pretraining**: ~4.16B tokens (see GTM-v2-base for the full data
  mix: FineWeb-Edu, Cosmopedia-v2, FineMath, FineWeb).
- **SFT data**: [UltraChat](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)
  (`train_sft` split), ~266M actual tokens after tokenization — smaller
  than UltraChat's advertised size once encoded. Reaching the ~1B token
  SFT target meant roughly 3.8 epochs over this data.
- **SFT loss masking**: loss is computed only on assistant-turn tokens;
  user turns and template markers are masked out of the loss (standard
  SFT practice), so the model is trained to generate good responses, not
  to predict user text.
- **SFT learning rate**: notably lower peak LR than base pretraining
  (5e-5 vs. 3e-4), standard practice for fine-tuning on top of a
  pretrained model.

## Known limitations

- **Does not reliably stop at its own `<|endofturn|>` marker.** The
  generation loop's early-stopping is keyed to the tokenizer's built-in
  end-of-text token, not the custom `<|endofturn|>` text marker learned
  during SFT. In practice this means the model often continues generating
  past a complete, correct answer into unrelated rambling (as in the
  example above, where a correct answer is followed by unrelated,
  factually shakier text about Paris as a "port city"). **If integrating
  this model, truncate generated text at the first `<|endofturn|>`
  occurrence yourself** rather than relying on the model to stop cleanly.
- **Correctness is not guaranteed even when it answers directly.** SFT
  taught response *format* (answer the question, use the assistant turn
  structure) more reliably than it taught factual accuracy. Expect
  confidently wrong answers on many prompts.
- **Only lightly evaluated.** The example above is a single, deliberately
  chosen (cherry-picked) success case shared to illustrate the format
  shift SFT produced, not a claim about typical response quality. No
  systematic chat-quality evaluation (multiple prompts, failure-rate
  measurement, comparison against other small chat models) has been done
  yet.
- **~3.8 epochs over a relatively small SFT set (~266M tokens)** raises
  some risk of overfitting to UltraChat's specific style/topics rather
  than generalizing broadly across conversation types.
- Inherits GTM-v2-base's limitations where SFT didn't specifically
  address them: no code capability (no code data in either pretraining or
  SFT), limited context (1024 tokens), small overall knowledge base
  relative to models trained on much larger corpora.

## Usage

Requires `model.py` (included in this repo) alongside the checkpoint —
this is a plain PyTorch model, not a `transformers` `AutoModel`.

```bash
pip install torch safetensors tiktoken
```

```python
import json
import torch
from safetensors.torch import load_file
from model import GPT, GPTConfig

with open("config.json") as f:
    cfg_dict = json.load(f)
config = GPTConfig(
    vocab_size=cfg_dict["vocab_size"], block_size=cfg_dict["block_size"],
    n_layer=cfg_dict["n_layer"], n_head=cfg_dict["n_head"],
    n_embd=cfg_dict["n_embd"], dropout=cfg_dict["dropout"], bias=cfg_dict["bias"],
)
model = GPT(config)
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

import tiktoken
enc = tiktoken.get_encoding("gpt2")

prompt = "<|user|>\nWhat is the capital of France?\n<|assistant|>\n"
ids = enc.encode_ordinary(prompt)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    out = model.generate(
        x, max_new_tokens=128, temperature=0.8, top_k=50,
        eot_token=enc.eot_token, repetition_penalty=1.3,
    )

response = enc.decode(out[0].tolist())
# manually truncate at the end-of-turn marker -- the model does not
# reliably stop generating here on its own (see Known limitations)
response = response.split("<|endofturn|>")[0] + "<|endofturn|>"
print(response)
```

## License

Apache 2.0 for this repo's contents (model weights, `model.py`, and this
README). Base-pretraining data retains its own license (FineWeb,
FineWeb-Edu, Cosmopedia-v2, FineMath: ODC-BY-1.0). UltraChat
(`HuggingFaceH4/ultrachat_200k`) is MIT licensed. This repo distributes
model weights, not the training data itself.