---
language:
- en
license: mit
tags:
- learnllm
- fineweb
- pytorch
- llama
- bpe
- from-scratch
library_name: transformers
pipeline_tag: text-generation
base_model: jaweed123/TinyJLLM
---

# TinyJLLM — 100M-parameter small language model built from scratch

A decoder-only Transformer (~102.5M parameters) pretrained **from random
initialization** on ~5 GB of FineWeb (`sample-10BT`), 3 epochs / 108,000
optimizer steps. Built as a fully educational pipeline (LearnLLM Run #2):
custom 32K byte-level BPE tokenizer, from-scratch Transformer, sharded
uint16 data pipeline, BF16 training, and verified exports.

**Final metrics:** validation loss 3.50 (perplexity 33.1); the best
checkpoint (step 89K) reached 3.48 / 32.6.

## Model details

| Property | Value |
|---|---|
| Parameters | 102,450,432 (~102.5M) |
| Architecture | Llama-style decoder-only: RMSNorm, RoPE (half-split), SwiGLU, tied embeddings, no biases |
| Layers / heads / head_dim | 11 / 12 / 64 |
| Context length | 512 |
| Vocabulary | 32,000 (custom byte-level BPE, `<pad> <unk> <bos> <eos>` = 0-3) |
| Pretraining data | FineWeb sample-10BT, ~5.37 GB raw, 1.75M documents |
| Tokens seen | 3.54B (3 epochs) |
| Hardware | RTX 4060 8 GB, ~30K tok/s (torch.compile) |
| Precision | BF16 mixed precision, FP32 master weights |

## Intended use

- Educational reference: inspect a small, complete, honest pretraining run.
- Qualitative experimentation: prompt it (it follows prompts *as text*; it
  is a **base model** — no instruction tuning yet).
- A base for further stages (SFT, DPO, domain fine-tuning).

Known limitations: small scale ⇒ repetition in long generations, weak
instruction following, limited world knowledge.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("jaweed123/TinyJLLM")
tokenizer = AutoTokenizer.from_pretrained("jaweed123/TinyJLLM")

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50, temperature=0.8, top_k=50, top_p=0.95)
print(tokenizer.decode(out[0]))
```

### llama.cpp / GGUF

The repo also ships GGUF files under `gguf/` (F16, Q8_0, Q4_K_M) — load
directly with llama.cpp or `llama-cpp-python`.

## Training details

- Custom 32K byte-level BPE (trained on a 512 MB FineWeb sample).
- Tokens stored once as uint16 shards (591 train + 6 validation).
- AdamW (lr 3e-4, wd 0.1, decay/no-decay groups), warmup 1,000 + cosine
  to 1e-5, effective batch 64 (32,768 tokens/step), gradient clipping 1.0.
- Full run: ~35 h on an RTX 4060.

## Files

- `config.json` — Llama-compatible config (`LlamaForCausalLM`)
- `model.safetensors` — FP32 weights
- `tokenizer.json` / `tokenizer_config.json` — custom BPE
- `generation_config.json` — decoding defaults
- `gguf/` — llama.cpp formats

## Acknowledgments

FineWeb (HuggingFaceFW), Hugging Face `tokenizers` / `datasets`,
PyTorch, llama.cpp. Built with the LearnLLM educational pipeline
(`src/learnllm` at the project repository).