| license: cc-by-nc-4.0 | |
| language: | |
| - en | |
| datasets: | |
| - nvidia/Nemotron-ClimbMix | |
| tags: | |
| - llama | |
| - causal-lm | |
| - pretraining | |
| - climbmix | |
| # ClimbMix StairFormer 353M | |
| StairFormer trained on 1.2B tokens from nvidia/Nemotron-ClimbMix with nested auxiliary loss. | |
| ## Loading | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| repo_id = "atrost/climbmix-stairformer-353m-1p2b-h100" | |
| tokenizer = AutoTokenizer.from_pretrained(repo_id) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| repo_id, | |
| trust_remote_code=True, | |
| torch_dtype="auto", | |
| ) | |
| ``` | |
| `trust_remote_code=True` is required for the custom StairFormer/asymmetric checkpoints and harmless for the dense Llama baseline. | |
| ## Source revision | |
| This clean repo was split out from `atrost/climbmix-llama-288m-2p8b-h100` at commit `a5b6981`. | |