metadata
license: cc-by-nc-4.0
language:
- en
datasets:
- nvidia/Nemotron-ClimbMix
tags:
- llama
- causal-lm
- pretraining
- climbmix
ClimbMix StairFormer 353M
StairFormer trained on 1.2B tokens from nvidia/Nemotron-ClimbMix with nested auxiliary loss.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "atrost/climbmix-stairformer-353m-1p2b-h100"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
trust_remote_code=True,
torch_dtype="auto",
)
trust_remote_code=True is required for the custom StairFormer/asymmetric checkpoints and harmless for the dense Llama baseline.
Source revision
This clean repo was split out from atrost/climbmix-llama-288m-2p8b-h100 at commit a5b6981.