---
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- llama
- causal-lm
- text-generation
- experimental
- pytorch
license: apache-2.0
datasets:
- superAVTR/wikipedia_en_512_for_pretraining
---

# Tinizong-50M

Tinizong-50M is a small experimental decoder-only language model. 

The model was created primarily as a research and engineering platform for exploring language-model architecture, tokenizer design, dataset construction, pretraining, scaling, and instruction tuning. 

This release contains a Hugging Face-compatible conversion of the native Tinizong checkpoint.

It is a **base language model**, not an instruction-tuned or chat model.

## Model details

| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer, Llama-compatible |
| Parameters | ~46M |
| Vocabulary | 16,000 tokens |
| Tokenizer | SentencePiece BPE |
| Context length | 1,024 tokens |
| Hidden size | 512 |
| Transformer layers | 12 |
| Attention heads | 8 |
| Attention head dimension | 64 |
| MLP intermediate size | 1,365 |
| Positional encoding | RoPE |
| RoPE theta | 10,000 |
| Normalization | RMSNorm |
| MLP | SwiGLU |
| Weight tying | Input embeddings / LM head |
| Attention | Multi-head causal self-attention |

The native implementation was written directly in PyTorch and later converted to the Hugging Face `LlamaForCausalLM` architecture.

## Training

Tinizong-50M was trained as a causal language model using next-token prediction.

Training data included a mixture of:

- Wikipedia-derived text
- synthetic factual / educational text
- other experimental pretraining material used during development

The training corpus and methodology evolved during the project, so this release should be considered an experimental research checkpoint rather than a fully documented production model.
The model uses a custom 16K SentencePiece tokenizer with byte fallback.

## Evaluation

Tinizong-46M was evaluated with the EleutherAI LM Evaluation Harness using the public Hugging Face checkpoint:

`superAVTR/tinizong-46M`

Evaluation settings:

- zero-shot
- Hugging Face backend
- batch size: 8
- model dtype: float32
- no custom evaluation code

| Benchmark | Metric | Score |
|---|---|---:|
| ARC-Easy | acc | 36.78% |
| ARC-Easy | acc_norm | 36.11% |
| HellaSwag | acc | 27.03% |
| HellaSwag | acc_norm | 27.36% |
| PIQA | acc | 53.16% |
| PIQA | acc_norm | 52.83% |
| LAMBADA OpenAI | accuracy | 5.38% |
| LAMBADA OpenAI | perplexity | 2539.74 |

For the multiple-choice benchmarks, `acc_norm` is the more useful comparison metric because it normalizes likelihood by completion length.

These results should be interpreted in the context of the model's small size (~46M parameters). The model performs above random-choice baseline on ARC-Easy, while HellaSwag and PIQA remain close to their respective random baselines. LAMBADA remains particularly challenging for this checkpoint.

The evaluation was run with:

```bash
#!pip install lm-eval

!lm_eval \
  --model hf \
  --model_args pretrained=superAVTR/tinizong-46M,dtype=float \
  --tasks lambada_openai,hellaswag,piqa,arc_easy \
  --device cuda:0 \
  --batch_size 8
```

## Hugging Face conversion

The original Tinizong model uses a compact PyTorch implementation with:

- combined Q/K/V projection
- RMSNorm
- rotary position embeddings
- SwiGLU feed-forward layers
- tied input/output embeddings

The trained weights were mapped into Hugging Face's `LlamaForCausalLM` representation. 
The combined native QKV projection was split into Hugging Face `q_proj`, `k_proj`, and `v_proj` tensors. Other layers were mapped directly to their corresponding Llama components.

### Numerical validation

The native model and the converted Hugging Face model were evaluated with identical input token IDs.

For the released checkpoint:

- Maximum absolute logit difference: approximately `9.54e-6`
- Mean absolute logit difference: approximately `1.14e-6`
- Logit cosine similarity: `1.0000000000`
- Next-token argmax: identical
- Top-token ordering: identical in the validation test

Autoregressive generation was also tested using the same random seed and sampling configuration.

The native implementation and Hugging Face implementation produced an **exact token-for-token match** using:

- temperature: `0.5`
- top-k: `30`
- top-p: `1.0`

Hugging Face generation with KV caching was also verified against full-context recomputation.

## Sample Generation

```text
PROMPT: Photosynthesis is a process
OUTPUT: Photosynthesis is a process that can release energy from nutrients and improve the ability of plants to digest food. <|endoftext|>
```

## Usage

```python
my_prompt = "Photosynthesis is a process"

inputs = tokenizer(
    my_prompt,
    return_tensors="pt",
    add_special_tokens=False
)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.5,
        top_k=30,
        top_p=1.0,
        use_cache=True,
        pad_token_id=1,
        eos_token_id=3,
    )

generated_text = tokenizer.decode(
    output[0],
    skip_special_tokens=False
)

print("PROMPT:", my_prompt)
print("OUTPUT:", generated_text)
```