RuzzGPT 116M
A 115.94M-parameter decoder-only language model pretrained from scratch using free Google Colab compute.
The model was developed as an independent experiment in language-model architecture, tokenization, dataset preparation, and resource-efficient pretraining.
116M parameters · Tesla T4 · 16 GB VRAM · ~8h 30m total workflow · $0 compute cost
Model
- Parameters: 115,944,960 (~115.94M / 0.116B)
- Architecture: Decoder-only Transformer
- Vocabulary: 20,000 tokens
- Tokenizer: SentencePiece Unigram
- Training: From scratch
- Training steps: 2,500
- Training hardware: NVIDIA Tesla T4, 16 GB VRAM
- Compute platform: Google Colab Free
- Total workflow time: ~8 hours 30 minutes
- Compute cost: $0
Why I Built It
The goal was to build a language model from the ground up rather than fine-tuning an existing pretrained model.
The project involved implementing and connecting the major components of a language-model training pipeline:
- Transformer model
- Tokenizer
- Dataset pipeline
- Training loop
- Checkpointing
- Inference
- Autoregressive generation
The experiment also became an exercise in dealing with real-world computational constraints.
Resource-Aware Compute Strategy
The experiment deliberately used different compute resources for different stages of the pipeline.
Tokenization and preprocessing — CPU
Tokenizer construction and dataset preprocessing were performed using a Google Colab CPU-only runtime.
GPU acceleration was not necessary for this stage, so the experiment did not spend GPU runtime on preprocessing work.
Transformer pretraining — GPU
After preprocessing, the runtime was switched to an NVIDIA Tesla T4 with 16 GB VRAM for the computationally intensive Transformer training stage.
The forward and backward passes of the neural network were therefore performed on the GPU.
FineWeb-Edu
│
▼
Tokenization / preprocessing
│
│ Google Colab CPU
▼
Encoded dataset
│
▼
Transformer pretraining
│
│ NVIDIA Tesla T4
▼
Model checkpoint
│
▼
CPU inference
This separation was intentional. The compute resource was selected according to the workload rather than running every stage on a GPU.
This demonstrated a practical understanding of which parts of the language-model pipeline benefit from GPU acceleration and which are primarily preprocessing workloads.
Tokenization Experiment
The initial pipeline used BPE tokenization.
During the experiment, BPE became a significant bottleneck. Approximately 2.5 hours were spent dealing with the inefficient tokenization pipeline.
Rather than continuing with an inefficient approach, I changed the tokenizer to SentencePiece Unigram.
The revised tokenizer completed the intended tokenization process in approximately 8 minutes.
This became one of the main engineering lessons of the experiment:
The choice of tokenizer can have a significant effect on the practical efficiency of an entire training pipeline.
Dataset Preparation
The intended training corpus was based on FineWeb-Edu data.
Encoding the five selected shards took approximately 3 hours and produced an approximately 8.48 GB encoded .bin dataset.
However, this introduced another practical problem.
The encoded dataset was stored temporarily in the Google Colab runtime. Because of the storage and persistence limitations of the environment, only approximately 808 MB of the encoded data was ultimately preserved through Google Drive.
As a result, the final training run did not use the complete 8.48 GB dataset originally prepared.
Instead, the final experiment was performed using the available ~808 MB portion.
Final Training Run
Because the available training data was substantially smaller than originally intended, I decided not to continue training indefinitely on the reduced dataset.
The final run used:
- 115.94M parameters
- 20,000-token vocabulary
- 2,500 training steps
- Tesla T4 16 GB VRAM
- Google Colab Free
- ~8h 30m total workflow
- $0 paid compute
The resulting model is therefore best understood as an experimental pretrained language model, rather than a fully converged production LLM.
Inference
The trained model can also be executed on a CPU-only machine.
The model was successfully loaded and run on a laptop with:
- 4 GB RAM
- CPU inference
- No dedicated GPU
This demonstrates that the resulting checkpoint can be executed even on relatively constrained consumer hardware, although generation quality and performance depend on the runtime configuration.
Current Generation Quality
The current model is not intended to compete with modern instruction-tuned models.
Because the final training run was limited to 2,500 steps and used only a fraction of the originally prepared dataset, generated text is currently highly experimental and can be incoherent.
Example:
Prompt: Hacker
Output:
Hacker LCD elongatedfirstname Testament29 Speaker placing ...
This is expected behavior for an incompletely trained model.
The primary result of this experiment was the successful implementation and execution of the complete language-model pipeline, rather than achieving state-of-the-art generation quality.
Potential Use Cases
The current checkpoint is primarily useful as an experimental and educational language-model artifact.
Possible uses include:
- Language-model research: experimenting with decoder-only Transformer architectures and training behavior.
- Tokenizer research: comparing BPE and SentencePiece Unigram pipelines.
- Training experiments: testing learning rates, batch sizes, context lengths, initialization, and other training configurations.
- Inference experiments: studying autoregressive generation on CPU-constrained hardware.
- Model architecture experiments: modifying the Transformer and observing changes in training behavior.
- Educational purposes: studying how a language model is implemented and pretrained from scratch.
- Further pretraining: using the checkpoint as a starting point for additional training with a larger and better-preserved dataset.
- Reproducibility experiments: reproducing the training pipeline under different hardware and dataset constraints.
The model is not currently recommended for production text generation, factual question answering, or deployment as a general-purpose assistant.
Repository Structure
.
├── config.py
├── data.py
├── inference.py
├── main.py
├── model.py
├── tokenizer.py
└── train.py
Main Components
model.py
Contains the Transformer language-model implementation.
tokenizer.py
Handles tokenizer construction and tokenization.
data.py
Handles dataset preparation and loading.
train.py
Contains the training procedure.
inference.py
Handles model loading and text generation.
config.py
Contains model and training configuration.
main.py
Provides the main entry point for training and inference.
Training Your Own RuzzGPT Model
The repository is also designed so that the training pipeline can be run again with a new dataset/configuration.
Using the Existing Checkpoint
When main.py detects an existing trained checkpoint, it skips dataset preparation and training:
FOUND EXISTING WEIGHTS AT data\checkpoints\latest_fineweb_edu.pt.
Skipping data prep/training.
(pass --force-train to redo data prep and retrain anyway)
This makes it possible to clone/download the project and immediately run inference without accidentally starting another expensive training process.
Force a New Training Run
To explicitly redo dataset preparation and training, use:
python main.py --force-train
The --force-train option tells the program to ignore the existing checkpoint and run the training pipeline again.
This is useful when experimenting with:
- a different dataset
- different model hyperparameters
- a different tokenizer
- a modified Transformer architecture
- a new training configuration
Training From Scratch
To train your own model from scratch:
- Remove or move the existing checkpoint:
data/checkpoints/latest_fineweb_edu.pt
Configure the desired model and training settings in the project configuration.
Prepare the training dataset using the project's data pipeline.
Run:
python main.py --force-train
The pipeline will then perform the necessary data preparation and start a new training run instead of loading the existing weights.
Alternatively, if the existing checkpoint is intentionally kept, --force-train can be used to explicitly request a fresh training run.
What Happens During Execution
The high-level workflow is:
Configuration
│
▼
Dataset preparation
│
▼
Tokenizer
│
▼
Encoded training data
│
▼
Transformer initialization
│
▼
Training
│
▼
Checkpoint
│
▼
Inference
The existing checkpoint is therefore treated as a ready-to-use artifact, while --force-train provides an explicit way to reproduce the training pipeline.
Note: Training a model of this size requires substantially more compute and memory than simply running inference. The original RuzzGPT experiment used an NVIDIA Tesla T4 with 16 GB VRAM for pretraining.
Reproducibility
The source code in this repository contains the implementation used for the experiment.
The exact final results depend on:
- Dataset availability
- Dataset preprocessing
- Random initialization
- Training configuration
- Hardware
- Runtime environment
The large raw/encoded training dataset is not included in this repository.
What I Learned
This project was less about producing another chatbot and more about understanding what is actually required to build a language model.
The experiment taught me about:
- Transformer architecture
- autoregressive language modeling
- tokenization
- BPE vs. Unigram tokenization
- large-scale text preprocessing
- dataset storage constraints
- GPU memory constraints
- checkpointing
- training infrastructure
- CPU inference
- resource-aware compute allocation
- experimental tradeoffs
Most importantly, the project demonstrated that a reasonably large language model can be implemented, pretrained, and executed using extremely limited financial and computational resources.
Future Work
Potential future experiments include:
- longer training
- larger usable datasets
- improved data pipelines
- tokenizer comparisons
- architecture scaling
- learning-rate experiments
- evaluation benchmarks
- improved checkpoint management
- continued pretraining
- inference optimization
Code
The implementation for the model and training pipeline is available in this repository.
Project Status
Experimental / Research
The model is a completed experimental pretraining run, but it is not considered a production-quality language model.
Built independently by Bekhruz Suleyman.
Training compute cost: $0.