atrost
/

climbmix-llama-288m-2p8b-h100-save-test

Model card Files Files and versions

atrost/climbmix-llama-288m-2p8b-h100-save-test

Small Llama-style causal language model pretrained from scratch on nvidia/Nemotron-ClimbMix.

Training details

Architecture: LlamaForCausalLM
Parameters: 288,554,240
Context length: 2048
Target tokens: 524,288
Actual tokens: 524,288
Tokens per optimizer step: 262,144
Optimizer: AdamW
LR schedule: cosine decay with warmup
Precision: bfloat16
Intended hardware: 8x H100

Model config

The model was initialized from the config embedded in train_climbmix_llama_ddp.py.

Downloads last month: 9

Safetensors

Model size

0.3B params

Tensor type

F32

·

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train atrost/climbmix-llama-288m-2p8b-h100-save-test