zyberg2091/fineweb_pretrained_model_37M

FineWeb Pretrained Model - 37.8M Parameters

A 37.8M-parameter decoder-only language model implemented and pretrained from scratch in PyTorch.

Model summary

• Parameters: 37.8M
• Layers: 6
• Hidden dimension: 512
• Attention heads: 8
• KV heads: 4
• Context length: 256
• Vocabulary: 20,000
• Architecture: grouped-query attention, RoPE, pre-norm transformer
• Training data: 250M-token FineWeb slice
• Training: approximately 61,000 steps and 500M token passes
• Precision: float32

Intended purpose

This model is a research and educational artifact for studying small-language-model pretraining, architecture implementation, and parameter allocation. It is not intended as a production assistant.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "zyberg2091/fineweb_pretrained_model_37M"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)