---
license: openmdw-1.1
datasets:
- roneneldan/TinyStories
language:
- en
metrics:
- accuracy
---
# TinyTransformer 5M - TinyStories

A 5M parameter TinyTransformer model trained on the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset.

This model is part of a research series investigating the effect of **normalization strategies in small Transformer models**. All models in the series use the same dataset and are trained under the same experimental setup, with normalization being the primary architectural variable.

## Research Focus

The goal of this series is to compare different normalization strategies in **small-scale Transformer architectures** and evaluate their impact on training and model performance.

The series includes:

* **Baseline** - reference architecture
* **Pre-LayerNorm** - LayerNorm applied before the Transformer sublayers
* **Post-LayerNorm** - LayerNorm applied after the Transformer sublayers
* **Pre-RMSNorm** - RMSNorm applied before the Transformer sublayers
* **Post-RMSNorm** - RMSNorm applied after the Transformer sublayers

## Models in This Series

| Model                                                                                                                  | Normalization  | Description                            |
| ---------------------------------------------------------------------------------------------------------------------- | -------------- | -------------------------------------- |
| [TinyTransformer Baseline 5M](https://huggingface.co/aplominski/TinyTransformer-Baseline-5M-TinyStories)             | Baseline       | Reference model                        |
| [TinyTransformer Pre-LayerNorm 5M](https://huggingface.co/aplominski/TinyTransformer-Pre-LayerNorm-5M-TinyStories)   | Pre-LayerNorm  | LayerNorm before Transformer sublayers |
| [TinyTransformer Post-LayerNorm 5M](https://huggingface.co/aplominski/TinyTransformer-Post-LayerNorm-5M-TinyStories) | Post-LayerNorm | LayerNorm after Transformer sublayers  |
| [TinyTransformer Pre-RMSNorm 5M](https://huggingface.co/aplominski/TinyTransformer-Pre-RMSNorm-5M-TinyStories)       | Pre-RMSNorm    | RMSNorm before Transformer sublayers   |
| [TinyTransformer Post-RMSNorm 5M](https://huggingface.co/aplominski/TinyTransformer-Post-RMSNorm-5M-TinyStories)     | Post-RMSNorm   | RMSNorm after Transformer sublayers    |

## Dataset

All models in this series were trained on:

**TinyStories** by Ronen Eldan and Yuanzhi Li

* Dataset: [roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)
* Language: English
* Task: Masked language modeling

## Normalization Methods

The experiments compare two commonly used normalization techniques:

### Layer Normalization

Layer Normalization normalizes activations across the feature dimension and was introduced by Ba et al.

### RMS Normalization

RMSNorm simplifies LayerNorm by removing the mean-centering operation and normalizing using the root mean square of the activations.

The experiments evaluate both methods in **pre-normalization** and **post-normalization** configurations.

### Baseline

Just without any normalization

## Citations

I'm used following papers in my reaserch:

> Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). *Attention Is All You Need.* https://arxiv.org/abs/1706.03762

> Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). *Layer Normalization.* https://arxiv.org/abs/1607.06450

> Zhang, B., & Sennrich, R. (2019). *Root Mean Square Layer Normalization.* https://arxiv.org/abs/1910.07467

## License

All models in this series are released under the **OpenMDW-1.1** license.

For the full license text, see the [OpenMDW-1.1 license](https://openmdw.ai/license/1-1/).