Upload tiny-lm LLaMA 3 Swallow Code model

README.md

@@ -1,23 +1,52 @@
 ---
-language: en
+language:
+  - en
+  - ja
 tags:
   - llama3
   - swallow-code
   - tiny-lm
+  - code
 datasets:
-  - nlp-waseda/EvalPlus-Merged-Filtered-Slim
+  - tokyotech-llm/swallow-code
 ---
 
 # Swallow Code Ibis-16 (LLaMA 3, 16 layers, 8k vocab)
 
 This model was trained with the
 [tiny-lm](https://github.com/ferjorosa/tiny-lm) repository on the
-[EvalPlus-Merged-Filtered-Slim dataset](https://huggingface.co/datasets/nlp-waseda/EvalPlus-Merged-Filtered-Slim)
-(see [Swallow Project](https://tokyotech-llm.github.io/swallow-llama/)).
+[SwallowCode dataset](https://huggingface.co/datasets/tokyotech-llm/swallow-code).
+
+The goal is educational: a compact pretraining run for studying tokenization,
+data pipelines, and transformer training end to end.
+
+## Data
+
+The tokenizer is a custom 8k-token BPE trained on SwallowCode with Karpathy's
+[`rustbpe`](https://github.com/karpathy/rustbpe) approach, then exported as a
+`tiktoken` encoding for inference.
+
+The dataset was split into 99% train and 1% validation before tokenization.
+The resulting tokenized files contain 25.95B train tokens and 262M validation
+tokens. Training used 1,024-token contiguous windows over the token stream.
+
+## Training
+
+The model was trained with PyTorch Lightning using bf16 mixed precision.
+
+- Context window: 1,024 tokens
+- Batch size: 64 sequences
+- Gradient accumulation: 4
+- Effective batch size: 262,144 tokens per optimizer step
+- Training budget: about 25B tokens over 95,350 optimizer steps
+- Optimizer: AdamW with cosine LR decay and 1% warmup
+- Peak LR: 6e-4
+- Weight decay: 0.1
 
 ## Architecture
 
 - LLaMA 3 style decoder-only transformer
+- Parameters: 18,886,912 total; 16,789,760 non-embedding
 - Layers: 16
 - Vocab size: 8192
 - Context length: 1024