Update README.md
Browse files# Tiny-MoE 200M
Tiny-MoE is a lightweight Mixture-of-Experts (MoE) language model trained entirely from scratch in native PyTorch. The project was built as an educational effort to explore and implement modern large language model architectures without relying on high-level model frameworks.
Unlike most educational LLM projects that build on existing implementations, Tiny-MoE includes a custom training pipeline, model architecture, and inference engine written from first principles.
> **Note**
>
> This repository contains only the trained model weights. The complete training pipeline, source code, and documentation are available in the GitHub repository.
**GitHub Repository:**
https://github.com/AbdelrhmanEbied/Tiny-MoE
---
# Model Overview
## Architecture
- Decoder-only Transformer
- Mixture-of-Experts (MoE)
- Multi-head Latent Attention (MLA)
- Rotary Position Embeddings (RoPE)
- YaRN context extension
- Shared Expert architecture
- Weight absorption for efficient inference
## Model Specifications
| Property | Value |
|----------|------:|
| Parameters | ~200M |
| Transformer Layers | 14 |
| Hidden Size | 512 |
| Attention Heads | 8 |
| Routed Experts | 8 |
| Shared Experts | 1 |
| Active Experts | Top-2 |
| Vocabulary Size | 32,000 |
| Base Context Length | 512 |
| Extended Context Length | 2048 (YaRN) |
---
# Training
Tiny-MoE was pretrained from scratch on a mixture of publicly available datasets:
- FineWeb-Edu
- Cosmopedia v2
- Open-Web-Math
Training uses a streaming data pipeline with dynamic sequence packing to maximize GPU utilization while minimizing storage requirements.
The model was trained entirely on Kaggle using two NVIDIA T4 GPUs.
---
# Capabilities
Tiny-MoE can perform tasks such as:
- Text completion
As a relatively small language model, its capabilities remain limited compared to modern multi-billion-parameter models.
---
# Limitations
This model is intended primarily for research and educational purposes.
Known limitations include:
- Limited factual knowledge
- May hallucinate incorrect information
- Weaker instruction following than larger LLMs
- Not suitable for production or safety-critical applications
---
# Intended Use
Tiny-MoE is designed for:
- Learning how modern LLMs work
- Research on Mixture-of-Experts architectures
- Experimentation with MLA and YaRN
- Educational projects
- Small-scale inference experiments
---
# Citation
If you use Tiny-MoE in your work, please cite the project:
```bibtex
@misc {tinymoe2026,
title={Tiny-MoE},
author={Abdelrhman Ebied},
year={2026},
url={https://github.com/AbdelrhmanEbied/Tiny-MoE}
}
```
---
# AI Assistance
AI was used solely as a writing and productivity assistant for documentation, grammar improvements, formatting, naming suggestions, and technical explanations.
AI was **not** used to generate the implementation of the model architecture, training pipeline, or inference system.
---
# License
This model is released under the Apache License 2.0.