Base model training
timestamp: 2026-07-01 09:43:00
- run: dummy
- device_type:
- fp8: False
- fp8_recipe: tensorwise
- depth: 12
- aspect_ratio: 64
- head_dim: 128
- max_seq_len: 1024
- window_pattern: L
- num_iterations: 50
- target_flops: -1.0000
- target_param_data_ratio: 12
- device_batch_size: 4
- total_batch_size: 65,536
- embedding_lr: 0.3000
- unembedding_lr: 0.0080
- weight_decay: 0.2800
- matrix_lr: 0.0200
- scalar_lr: 0.5000
- warmup_steps: 40
- warmdown_ratio: 0.6500
- final_lr_frac: 0.0500
- resume_from_step: -1
- eval_every: 25
- eval_tokens: 131,072
- core_metric_every: -1
- core_metric_max_per_task: 500
- sample_every: -1
- save_every: -1
- model_tag: ar-d12-eager
- Number of parameters: 286,261,730
- Number of FLOPs per token: 7.738517e+08
- Calculated number of iterations: 50
- Number of training tokens: 3,276,800
- Tokens : Scaling params ratio: 0.0298
- DDP world size: 1
- warmup_steps: 40
- warmdown_ratio: 0.6500
- final_lr_frac: 0.0500
- Minimum validation bpb: 1.2144
- Final validation bpb: 1.2144
- CORE metric estimate: None
- MFU %: 11.43%
- Total training flops: 2.535757e+15
- Total training time: 0.97m
- Peak memory usage: 6444.42MiB