Upload TinyJLLM: 100M SLM pretrained from scratch (3 epochs FineWeb) + GGUF

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+gguf/learnllm-100m-f16.gguf filter=lfs diff=lfs merge=lfs -text
+gguf/learnllm-100m-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+gguf/learnllm-100m-q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,91 @@
 ---
+language:
+- en
 license: mit
+tags:
+- learnllm
+- fineweb
+- pytorch
+- llama
+- bpe
+- from-scratch
+library_name: transformers
+pipeline_tag: text-generation
+base_model: jaweed123/TinyJLLM
 ---
+
+# TinyJLLM — 100M-parameter small language model built from scratch
+
+A decoder-only Transformer (~102.5M parameters) pretrained **from random
+initialization** on ~5 GB of FineWeb (`sample-10BT`), 3 epochs / 108,000
+optimizer steps. Built as a fully educational pipeline (LearnLLM Run #2):
+custom 32K byte-level BPE tokenizer, from-scratch Transformer, sharded
+uint16 data pipeline, BF16 training, and verified exports.
+
+**Final metrics:** validation loss 3.50 (perplexity 33.1); the best
+checkpoint (step 89K) reached 3.48 / 32.6.
+
+## Model details
+
+| Property | Value |
+|---|---|
+| Parameters | 102,450,432 (~102.5M) |
+| Architecture | Llama-style decoder-only: RMSNorm, RoPE (half-split), SwiGLU, tied embeddings, no biases |
+| Layers / heads / head_dim | 11 / 12 / 64 |
+| Context length | 512 |
+| Vocabulary | 32,000 (custom byte-level BPE, `<pad> <unk> <bos> <eos>` = 0-3) |
+| Pretraining data | FineWeb sample-10BT, ~5.37 GB raw, 1.75M documents |
+| Tokens seen | 3.54B (3 epochs) |
+| Hardware | RTX 4060 8 GB, ~30K tok/s (torch.compile) |
+| Precision | BF16 mixed precision, FP32 master weights |
+
+## Intended use
+
+- Educational reference: inspect a small, complete, honest pretraining run.
+- Qualitative experimentation: prompt it (it follows prompts *as text*; it
+  is a **base model** — no instruction tuning yet).
+- A base for further stages (SFT, DPO, domain fine-tuning).
+
+Known limitations: small scale ⇒ repetition in long generations, weak
+instruction following, limited world knowledge.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("jaweed123/TinyJLLM")
+tokenizer = AutoTokenizer.from_pretrained("jaweed123/TinyJLLM")
+
+prompt = "The future of AI is"
+inputs = tokenizer(prompt, return_tensors="pt")
+out = model.generate(**inputs, max_new_tokens=50, temperature=0.8, top_k=50, top_p=0.95)
+print(tokenizer.decode(out[0]))
+```
+
+### llama.cpp / GGUF
+
+The repo also ships GGUF files under `gguf/` (F16, Q8_0, Q4_K_M) — load
+directly with llama.cpp or `llama-cpp-python`.
+
+## Training details
+
+- Custom 32K byte-level BPE (trained on a 512 MB FineWeb sample).
+- Tokens stored once as uint16 shards (591 train + 6 validation).
+- AdamW (lr 3e-4, wd 0.1, decay/no-decay groups), warmup 1,000 + cosine
+  to 1e-5, effective batch 64 (32,768 tokens/step), gradient clipping 1.0.
+- Full run: ~35 h on an RTX 4060.
+
+## Files
+
+- `config.json` — Llama-compatible config (`LlamaForCausalLM`)
+- `model.safetensors` — FP32 weights
+- `tokenizer.json` / `tokenizer_config.json` — custom BPE
+- `generation_config.json` — decoding defaults
+- `gguf/` — llama.cpp formats
+
+## Acknowledgments
+
+FineWeb (HuggingFaceFW), Hugging Face `tokenizers` / `datasets`,
+PyTorch, llama.cpp. Built with the LearnLLM educational pipeline
+(`src/learnllm` at the project repository).

config.json

@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "model_type": "llama",
+  "vocab_size": 32000,
+  "max_position_embeddings": 512,
+  "hidden_size": 768,
+  "intermediate_size": 2048,
+  "num_hidden_layers": 11,
+  "num_attention_heads": 12,
+  "head_dim": 64,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": true,
+  "hidden_act": "silu",
+  "initializer_range": 0.02,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "pad_token_id": 0,
+  "torch_dtype": "bfloat16",
+  "learnllm": {
+    "vocab_size": 32000,
+    "context_length": 512,
+    "d_model": 768,
+    "n_layers": 11,
+    "n_heads": 12,
+    "head_dim": null,
+    "ffn_dim": 2048,
+    "dropout": 0.0,
+    "bias": false,
+    "tie_embeddings": true,
+    "norm_type": "rmsnorm",
+    "norm_eps": 1e-05,
+    "activation": "swiglu",
+    "position_encoding": "rope",
+    "rope_theta": 10000.0,
+    "rope_scaling": null,
+    "initializer_range": 0.02,
+    "model_type": "llama",
+    "architecture_note": "Learned from scratch; parameter names follow the Llama convention so transformers/PEFT/GGUF tools can consume it."
+  }
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "max_new_tokens": 100,
+  "temperature": 0.8,
+  "top_k": 50,
+  "top_p": 0.95,
+  "bos_token_id": 2,
+  "eos_token_id": 3,
+  "pad_token_id": 0,
+  "model_type": "llama",
+  "_learnllm_note": "decoding defaults used by learnllm inference"
+}

gguf/learnllm-100m-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2851c42dcbf5f0277316d5a30cb8aad74021dd19aa479f8ed747c01c33dd525a
+size 206064416

gguf/learnllm-100m-q4_k_m.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44987b6d4a83900c5e88cbd255d886fb00730c956934d9d411e86a9a6b5a895f
+size 67940672

gguf/learnllm-100m-q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a3706ef81d80fdb73f5979425b01edbecff9bcac2afcf71d759aafac82cee92
+size 110033696

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ee23a4a3c58d0733af9296fbe0a53b59e1adf92749a8c4d476f157715bd164b
+size 409812912

tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "unk_token": "<unk>",
+  "pad_token": "<pad>",
+  "model_max_length": 512,
+  "vocab_size": 32000,
+  "clean_up_tokenization_spaces": true,
+  "_learnllm_meta": {
+    "vocab_size": 32000,
+    "min_frequency": 2,
+    "special_tokens": [
+      "<pad>",
+      "<unk>",
+      "<bos>",
+      "<eos>"
+    ],
+    "sample_size_mb": 250.0,
+    "seed": 42,
+    "save_dir": "J:\\LearnLLM\\models\\tokenizer"
+  }
+}