---
license: apache-2.0
tags:
  - text-generation
  - causal-lm
  - transformer
  - decoder-only
  - research
  - layer-wise-training
language:
  - en
---

# Growing Transformers — Model unfrozen 1_9 (≈247M)

This repository contains **`growing-transformers-model-unfrozen-1-9-247m`**, a **constructively grown** (layer-wise expanded) model from the paper:

[📚 Paper (Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate)](https://huggingface.co/papers/2507.07129) - 

[📚 Paper (Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations)](https://huggingface.co/papers/2507.04886) - 

It is part of the comparative-study collection:  
https://huggingface.co/collections/Bochkov/growing-transformers-layer-wise-expansion-comparative-study

Code:  
https://github.com/AVBochkov/PGT

---

## What this model is (in one paragraph)

This is a **9-layer decoder-only Transformer** trained with a **constructive, layer-wise growth** procedure (layers are added and trained progressively while previously trained parts are frozen). Unlike the “frozen-substrate” variants, this model starts with **standard trainable token embeddings** and is trained **monolithically for the first stage (layers 1–3 + embeddings all trainable)**. After that first stage converges, the **embedding layer and the first 3 layers are frozen**, then layers **4–6** are trained, frozen, and finally layers **7–9** are trained. This model exists to isolate the effect of the **growth procedure without a frozen embedding substrate at initialization**.

---

## Main comparison (why this repo exists)

This model is intended to be compared primarily against:

- **Bochkov/growing-transformers-model-16-bit-1-9-181m** (constructive growth, frozen 16-bit embedding)
- **Bochkov/growing-transformers-model-unicode-1-9-247m** (constructive growth, frozen “visual UNICODE” embedding)

All these models share the **same Transformer stack design** in the controlled study (same depth and same `d_model / n_head`), but differ in the **embedding substrate and what is frozen when**.

---

## Important: parameter count difference (why this model is larger than 16-bit)

This checkpoint is **larger (≈247M parameters)** than the 16-bit constructive model (≈181M) **primarily because it uses a full-size embedding matrix** (standard learned embeddings at `d_model`, with `vocab_size = 65,536`), whereas the 16-bit model uses a tiny `n_embed = 16` embedding that is deterministically expanded to `d_model`.

Even with the **same Transformer blocks**, the embedding matrix dominates a large chunk of parameters when `vocab_size` is large and `n_embed = d_model`.

> Note: exact parameter totals can differ slightly across implementations (e.g., weight tying vs separate LM head). The key reason for the size gap is the **embedding dimensionality** (full `d_model` vs `16`).

---

## Training method (constructive growth schedule)

This model was trained in **three stages** (controlled study setup):

1. **Stage A (classic / unfrozen start):** train **token embeddings + layers 1–3** (all trainable)
2. **Stage B:** freeze **embeddings + layers 1–3**, add layers **4–6**, train **only layers 4–6**
3. **Stage C:** freeze **embeddings + layers 1–6**, add layers **7–9**, train **only layers 7–9**

So: it is *constructive / iterative*, but **not** “frozen embeddings Stage A”.

---

## Model architecture (controlled study)

- Type: decoder-only Transformer (GPT-like)
- Layers: **9**
- Hidden size: **d_model = 1024**
- Heads: **n_head = 32**
- Context length used in training: **1024**
- Embedding: **standard learned embedding matrix** (trainable in Stage A, frozen afterwards)

---

## Tokenizer

This model uses the **BVV tokenizer**. Canonical tokenizer repo:

- https://huggingface.co/Bochkov/bvv241-2-3  
  (collection: https://huggingface.co/collections/Bochkov/tokenizers)

**Reproducibility note:** even with the same tokenizer, this repo also includes **embedding artifacts specific to this model**, so for exact reproduction it is recommended to load the tokenizer from *this* model repo.

---

## How to use (Transformers)

```python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Bochkov/growing-transformers-model-unfrozen-1-9-247m")
model = AutoModelForCausalLM.from_pretrained("Bochkov/growing-transformers-model-unfrozen-1-9-247m", trust_remote_code=True).to('cuda')

inputs = torch.tensor([tokenizer.encode("Write a short poem about the ocean. ")], dtype=torch.long, device='cuda')

outputs = model.generate(
    inputs, 
    max_new_tokens=50,
    do_sample=False
)
print(tokenizer.decode(outputs[0].tolist()))
#Write a short poem about the ocean. The poem was written by the author of the poem The Lord of the Rings, and was published in 

inputs = torch.tensor([tokenizer.encode("Question: What is the capital of India?\nAnswer:")], dtype=torch.long, device='cuda')

outputs = model.generate(
    inputs, 
    max_new_tokens=10,
    do_sample=False
)
print(tokenizer.decode(outputs[0].tolist()))
#Question: What is the capital of India?
#Answer:San Francisco de Pa

```

---

## 🧑‍🔬 Citation & Concept
If you use this model or the underlying concepts in your research, please cite our work:
```
@article{
      bochkov2025emergent,
      title={Emergent Semantics Beyond Token Embeddings: Transformer {LM}s with Frozen Visual Unicode Representations},
      author={Andrey Bochkov},
      journal={Transactions on Machine Learning Research},
      issn={2835-8856},
      year={2025},
      url={https://openreview.net/forum?id=Odh8IynO1o},
      note={}
}

@misc{bochkov2025growingtransformersmodularcomposition,
      title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate}, 
      author={A. Bochkov},
      year={2025},
      eprint={2507.07129},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2507.07129}, 
}
```