dheeyantra/dhee-pravega-og-atom

Dhee-Pravega-OG Atom (645M)

A 645M-parameter decoder-only language model, trained from scratch (no pretrained weights of any kind) on 30B tokens across 18 languages, with a strong focus on Indic scripts. It produces fluent, coherent, correct-script text across all 12 Indic languages it was trained on, plus English, Arabic, Spanish, French, German, and Russian.

This is a base language model: strong at multilingual text generation, not yet trained for tool-calling, instructions, or chat. Use it as-is for text generation, or as a foundation to fine-tune for those downstream tasks.

Instruction-tuned version: for chat and tool calling, see Dhee-Pravega-Atom-IT (research preview), which is fine-tuned from this checkpoint on multilingual function-calling dialogues.

Architecture

DheeForCausalLM — a decoder-only transformer assembled from three published, permissively-licensed components, combined in a way no single upstream project ships:

• Reordered sublayer norms (OLMo-2): norm applied to the output of each sublayer inside the residual, targeting activation-norm growth through depth.
• Per-head QK-norm (Qwen3): each attention head's queries/keys are normalized independently, rather than one norm shared across the whole projection — keeps attention logits stable through training.
• Local:global attention ratio (Gemma-3's published 5:1 interleave pattern): built into the architecture for efficient long-context attention.

What's original here is the combination, the tokenizer, the corpus, and the weights — the implementation derives only from Apache-2.0 reference code (transformers' OLMo-2 and Qwen3 implementations), and is verified bit-identical to stock Olmo2ForCausalLM under the degenerate configuration, confirming every inherited line behaves exactly as upstream.

Parameters	645,617,408 (embeddings: 163,847,680 / 25%)
Layers	28
Hidden size	1280
Attention heads	10 query / 2 KV (GQA, 5:1)
Head dim	128
FFN size	3456
Vocab size	128,006 (128,000 SentencePiece pieces + 6 control tokens)
Context length	32,768
Tied embeddings	yes

Training

• Data: 30.006B tokens, 18 languages at ~1.667B tokens each: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Sanskrit, Tamil, Telugu, English, Arabic, Spanish, French, German, Russian.
• Tokenizer: custom 128,000-piece SentencePiece model built for this project, with 6 reserved control tokens (<|sys|>, <|user|>, <|assistant|>, <|tool_call|>, <|tool_result|>, <|end|>) for downstream chat/tool-calling formats.
• Optimizer: Muon for 2-D hidden-layer matrices (196 tensors / 481.7M params), AdamW for embeddings, LM head, and norms (114 tensors / 163.9M params).
• Schedule: Warmup-Stable-Decay (WSD), peak LR 6e-4, 57,231 steps, 524,288 tokens/step, block size 8,192, gradient checkpointing, z-loss for logit-scale stability.
• Hardware: single NVIDIA RTX Pro 6000 Blackwell (24GB).

Evaluation

Held-out perplexity, computed on a clean dev split never seen during training (110 blocks per language):

Language	Perplexity	Language	Perplexity
Hindi	11.8	Punjabi	13.0
Malayalam	11.8	Odia	23.3
Tamil	12.8	Assamese	23.1
Marathi	13.9	Gujarati	18.2
Kannada	15.5	Bengali	18.6
Telugu	16.2	Sanskrit	36.8
English	13.7

Generated text stays reliably in-script across all 12 Indic languages. For best generation quality, use sampling (e.g. temperature ~0.7) rather than greedy decoding, which is prone to repetition loops on longer continuations — a common pattern at this model size, not specific to any one language.

Sanskrit trails the rest of the set on both perplexity and generation stability; treat it as the one language here that would benefit most from additional data in a future iteration.

This checkpoint has not been trained on tool-calling or dialogue data, so it is not suited for agentic or function-calling use without further fine-tuning on that kind of data — the reserved control tokens above are there for exactly that purpose. Dhee-Pravega-Atom-IT is our instruction-tuned release built on them.

Loading the model

Requires trust_remote_code=True (the custom architecture ships as modeling_dhee.py alongside this checkpoint):

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "dheeyantra/dhee-pravega-og-atom",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

Loading the tokenizer

Load the SentencePiece model directly and use the reserved IDs for the control tokens:

import json
import sentencepiece as spm

sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
specials = json.load(open("specials.json"))["specials"]
# specials = {"<|sys|>": 128000, "<|user|>": 128001, "<|assistant|>": 128002,
#             "<|tool_call|>": 128003, "<|tool_result|>": 128004, "<|end|>": 128005}

ids = sp.encode("your text here")
eos_id = specials["<|end|>"]  # == 128005, matches config.json's eos_token_id

Licence

Research use only. The model weights, configuration and tokenizer are released under the Dhee Research-Only Licence v1.0. Academic research, evaluation, benchmarking, red-teaming, teaching and personal experimentation are permitted. Commercial use is not. Commercial licences are available from Dheeyantra Research Labs on request (contact@dheeyantra.com).

No pretrained weights of any kind were loaded at any point: all weights in this repository were trained from random initialisation on the data described above, so no third-party model terms apply. The source file modeling_dhee.py (derived from the Apache-2.0 OLMo-2 and Qwen3 implementations in transformers) remains under Apache-2.0; see NOTICE.

If you redistribute the model or anything derived from it, include the LICENSE and NOTICE files.

Earlier revisions. Revisions published before this licence change were distributed under Apache-2.0 and remain available under the terms in force when they were distributed. The Dhee Research-Only Licence v1.0 applies from this revision onward.