Update README.md

README.md

@@ -1,3 +1,142 @@
 ---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - llama
+  - causal-lm
+  - text-generation
+  - experimental
+  - pytorch
 license: apache-2.0
 ---
+
+# Tinizong-50M
+
+Tinizong-50M is a small experimental decoder-only language model developed from scratch as part of the Tinizong LLM project.
+
+The model was created primarily as a research and engineering platform for exploring language-model architecture, tokenizer design, dataset construction, pretraining, scaling, and instruction tuning.
+
+This release contains a Hugging Face-compatible conversion of the native Tinizong checkpoint.
+
+It is a **base language model**, not an instruction-tuned or chat model.
+
+## Model details
+
+| Property | Value |
+|---|---|
+| Architecture | Decoder-only Transformer, Llama-compatible |
+| Parameters | ~46M |
+| Vocabulary | 16,000 tokens |
+| Tokenizer | SentencePiece BPE |
+| Context length | 1,024 tokens |
+| Hidden size | 512 |
+| Transformer layers | 12 |
+| Attention heads | 8 |
+| Attention head dimension | 64 |
+| MLP intermediate size | 1,365 |
+| Positional encoding | RoPE |
+| RoPE theta | 10,000 |
+| Normalization | RMSNorm |
+| MLP | SwiGLU |
+| Weight tying | Input embeddings / LM head |
+| Attention | Multi-head causal self-attention |
+
+The native implementation was written directly in PyTorch and later converted to the Hugging Face `LlamaForCausalLM` architecture.
+
+This checkpoint corresponds to the **v15 Wikipedia + synthetic-data training experiment**.
+
+## Training
+
+Tinizong-50M was trained as a causal language model using next-token prediction.
+
+Training data included a mixture of:
+
+- Wikipedia-derived text
+- synthetic factual / educational text
+- other experimental pretraining material used during development
+
+The training corpus and methodology evolved during the project, so this release should be considered an experimental research checkpoint rather than a fully documented production model.
+
+The model uses a custom 16K SentencePiece tokenizer with byte fallback.
+
+## Hugging Face conversion
+
+The original Tinizong model uses a compact PyTorch implementation with:
+
+- combined Q/K/V projection
+- RMSNorm
+- rotary position embeddings
+- SwiGLU feed-forward layers
+- tied input/output embeddings
+
+The trained weights were mapped into Hugging Face's `LlamaForCausalLM` representation.
+
+The combined native QKV projection was split into Hugging Face `q_proj`, `k_proj`, and `v_proj` tensors. Other layers were mapped directly to their corresponding Llama components.
+
+### Numerical validation
+
+The native model and the converted Hugging Face model were evaluated with identical input token IDs.
+
+For the released checkpoint:
+
+- Maximum absolute logit difference: approximately `9.54e-6`
+- Mean absolute logit difference: approximately `1.14e-6`
+- Logit cosine similarity: `1.0000000000`
+- Next-token argmax: identical
+- Top-token ordering: identical in the validation test
+
+Autoregressive generation was also tested using the same random seed and sampling configuration.
+
+The native implementation and Hugging Face implementation produced an **exact token-for-token match** using:
+
+- temperature: `0.5`
+- top-k: `30`
+- top-p: `1.0`
+
+Hugging Face generation with KV caching was also verified against full-context recomputation.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "YOUR-HF-USERNAME/Tinizong-50M"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id,
+    use_fast=False
+)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.float32
+)
+
+prompt = "The film was released in"
+
+inputs = tokenizer(
+    prompt,
+    return_tensors="pt",
+    add_special_tokens=False
+)
+
+with torch.no_grad():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=60,
+        do_sample=True,
+        temperature=0.5,
+        top_k=30,
+        top_p=1.0,
+        use_cache=True
+    )
+
+print(
+    tokenizer.decode(
+        output[0],
+        skip_special_tokens=False
+    )
+)