| --- |
| language: |
| - en |
| - id |
| - es |
| - fr |
| - de |
| license: mit |
| tags: |
| - text-generation |
| - llama |
| - pytorch |
| - terminal |
| - small-language-model |
| pipeline_tag: text-generation |
| widget: |
| - text: "User: How do I navigate up one directory?\nAssistant:" |
| - text: "$ cd ..\n$" |
| --- |
| |
| # 5M Terminal & Multilingual Language Model (Chinchilla Optimal - 100M Tokens) |
|
|
| This repository contains a **5.0 Million Parameter Causal Language Model** trained from scratch on a compute-optimal token budget of **100 Million Tokens** ($20\times$ parameter count) following Chinchilla scaling laws. |
|
|
| The model is specialized in **Linux Terminal Commands, Shell Automation, and Multilingual Text Generation** (English, Indonesian, Spanish, French, German). |
|
|
| ## Model Architecture |
|
|
| * **Total Trainable Parameters**: 4,984,064 (~4.98M / 5.0M) |
| * **Architecture**: Decoder-only Transformer (LLaMA style) |
| * **Vocabulary Size**: 4,096 (Byte-Pair Encoding Tokenizer) |
| * **Hidden Dimension (`d_model`)**: 256 |
| * **Number of Layers (`n_layer`)**: 6 |
| * **Number of Attention Heads (`n_head`)**: 8 (Head dimension = 32) |
| * **MLP Hidden Dimension (`inter_dim`)**: 512 (SwiGLU activation) |
| * **Context Window (`max_seq_len`)**: 256 tokens |
| * **Weight Tying**: Tied Embedding and LM Head weights |
|
|
| ## Training Details |
|
|
| * **Token Budget**: 100,000,000 Tokens (Chinchilla Optimal: 20 tokens per parameter) |
| * **Dataset Composition**: |
| * **Terminal CLI & Commands**: 20% (Bash commands, `cd ..`, `ls -la`, `mkdir`, `grep`, `git`, `chmod`, `curl`, Q&A pairs) |
| * **English Text**: 24% (`wikitext-103-v1` + technical corpus) |
| * **Multilingual Text**: 56% (Indonesian, Spanish, French, German Wikipedia & general text) |
| * **Hardware**: Kaggle NVIDIA GPU (Dual Tesla T4 / P100) with FP16 AMP Mixed Precision. |
|
|
| ## Usage & Inference Example (PyTorch) |
|
|
| ```python |
| import torch |
| from tokenizers import Tokenizer |
| from huggingface_hub import hf_hub_download |
| |
| # Download model assets |
| repo_id = "kipasyangin5/5m-terminal-lm-chinchilla" |
| tok_path = hf_hub_download(repo_id=repo_id, filename="tokenizer.json") |
| weights_path = hf_hub_download(repo_id=repo_id, filename="model.pt") |
| |
| # Load Tokenizer |
| tokenizer = Tokenizer.from_file(tok_path) |
| |
| # Prompt execution |
| prompt = "User: How do I list all files including hidden ones?\nAssistant:" |
| tokens = tokenizer.encode(prompt).ids |
| input_ids = torch.tensor([tokens], dtype=torch.long) |
| |
| print("Input prompt:", prompt) |
| ``` |
|
|
| ## Citation & License |
|
|
| MIT License. Developed by `kipasyangin5`. |
|
|
