superAVTR/tinizong-46M_v1.1

Tinizong-50M

Tinizong-50M is a small experimental decoder-only language model.

The model was created primarily as a research and engineering platform for exploring language-model architecture, tokenizer design, dataset construction, pretraining, scaling, and instruction tuning.

This release contains a Hugging Face-compatible conversion of the native Tinizong checkpoint.

It is a base language model, not an instruction-tuned or chat model.

Model details

Property	Value
Architecture	Decoder-only Transformer, Llama-compatible
Parameters	~46M
Vocabulary	16,000 tokens
Tokenizer	SentencePiece BPE
Context length	1,024 tokens
Hidden size	512
Transformer layers	12
Attention heads	8
Attention head dimension	64
MLP intermediate size	1,365
Positional encoding	RoPE
RoPE theta	10,000
Normalization	RMSNorm
MLP	SwiGLU
Weight tying	Input embeddings / LM head
Attention	Multi-head causal self-attention

The native implementation was written directly in PyTorch and later converted to the Hugging Face LlamaForCausalLM architecture.

Training

Tinizong-50M was trained as a causal language model using next-token prediction.

Training data included a mixture of:

• Wikipedia-derived text
• synthetic factual / educational text
• other experimental pretraining material used during development

The training corpus and methodology evolved during the project, so this release should be considered an experimental research checkpoint rather than a fully documented production model. The model uses a custom 16K SentencePiece tokenizer with byte fallback.

Evaluation

Tinizong-46M was evaluated with the EleutherAI LM Evaluation Harness using the public Hugging Face checkpoint:

superAVTR/tinizong-46M

Evaluation settings:

• zero-shot
• Hugging Face backend
• batch size: 8
• model dtype: float32
• no custom evaluation code

Benchmark	Metric	Score
ARC-Easy	acc	36.78%
ARC-Easy	acc_norm	36.11%
HellaSwag	acc	27.03%
HellaSwag	acc_norm	27.36%
PIQA	acc	53.16%
PIQA	acc_norm	52.83%
LAMBADA OpenAI	accuracy	5.38%
LAMBADA OpenAI	perplexity	2539.74

For the multiple-choice benchmarks, acc_norm is the more useful comparison metric because it normalizes likelihood by completion length.

These results should be interpreted in the context of the model's small size (~46M parameters). The model performs above random-choice baseline on ARC-Easy, while HellaSwag and PIQA remain close to their respective random baselines. LAMBADA remains particularly challenging for this checkpoint.

The evaluation was run with:

#!pip install lm-eval

!lm_eval \
  --model hf \
  --model_args pretrained=superAVTR/tinizong-46M,dtype=float \
  --tasks lambada_openai,hellaswag,piqa,arc_easy \
  --device cuda:0 \
  --batch_size 8

Hugging Face conversion

The original Tinizong model uses a compact PyTorch implementation with:

• combined Q/K/V projection
• RMSNorm
• rotary position embeddings
• SwiGLU feed-forward layers
• tied input/output embeddings

The trained weights were mapped into Hugging Face's LlamaForCausalLM representation. The combined native QKV projection was split into Hugging Face q_proj, k_proj, and v_proj tensors. Other layers were mapped directly to their corresponding Llama components.

Numerical validation

The native model and the converted Hugging Face model were evaluated with identical input token IDs.

For the released checkpoint:

• Maximum absolute logit difference: approximately 9.54e-6
• Mean absolute logit difference: approximately 1.14e-6
• Logit cosine similarity: 1.0000000000
• Next-token argmax: identical
• Top-token ordering: identical in the validation test

Autoregressive generation was also tested using the same random seed and sampling configuration.

The native implementation and Hugging Face implementation produced an exact token-for-token match using:

• temperature: 0.5
• top-k: 30
• top-p: 1.0

Hugging Face generation with KV caching was also verified against full-context recomputation.

Sample Generation

PROMPT: Photosynthesis is a process
OUTPUT: Photosynthesis is a process that can release energy from nutrients and improve the ability of plants to digest food. <|endoftext|>

Usage

my_prompt = "Photosynthesis is a process"

inputs = tokenizer(
    my_prompt,
    return_tensors="pt",
    add_special_tokens=False
)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.5,
        top_k=30,
        top_p=1.0,
        use_cache=True,
        pad_token_id=1,
        eos_token_id=3,
    )

generated_text = tokenizer.decode(
    output[0],
    skip_special_tokens=False
)

print("PROMPT:", my_prompt)
print("OUTPUT:", generated_text)