File size: 2,009 Bytes
6096ac4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | ---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- pretrained
- base-model
- llama
- climbmix
datasets:
- karpathy/climbmix-400b-shuffle
---
# d24-climbmix-100b
A **0.75B-parameter** dense decoder-only language model ("d24", nanochat depth-24 shape),
**pretrained from scratch on ~100B tokens** of [ClimbMix](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle).
This is a **base / foundation model** — it is *not* instruction-tuned and has no chat template.
Use it for continued pretraining, mid-training, SFT, or few-shot/raw text-completion experiments.
## Architecture
| | |
|---|---|
| Class | `LlamaForCausalLM` (SwiGLU / RoPE / RMSNorm) |
| Parameters | 756,819,456 (~0.75B) |
| Layers | 24 |
| Hidden size | 1536 |
| Attention heads | 12 (head_dim 128, no GQA: 12 KV heads) |
| FFN hidden | 4096 (gated SwiGLU) |
| Context length | 2048 |
| Vocab | 50304 (GPT-2 BPE, 50257 padded to a multiple of 128) |
| Tied embeddings | yes |
| dtype | bf16 |
| Tokenizer | GPT-2 (`<\|endoftext\|>` = id 50256 as bos/eos) |
## Training
- **Data:** ClimbMix (`karpathy/climbmix-400b-shuffle`), tokenized to GPT-2 bin/idx.
- **Tokens:** 95,368 iters × global batch 512 × seq 2048 ≈ **100B tokens**.
- **Optimizer:** cosine LR 3e-4 → 3e-5, warmup 100, AdamW.
- **Hardware:** ALCF Polaris, 256× A100 (64 nodes × 4 GPUs), pure data-parallel (TP=PP=1).
- **Framework:** NVIDIA NeMo / Megatron-Bridge (`nemo:26.04`); exported Megatron → HF with `convert/megatron_to_hf`.
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ftajwar/d24-climbmix-100b")
model = AutoModelForCausalLM.from_pretrained("ftajwar/d24-climbmix-100b", torch_dtype=torch.bfloat16)
prompt = "The capital of France is"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```
|