nanochat-ar / reports /tokenizer-training.md
Babajaan's picture
Upload nanochat-ar Aziz HPC proof-run checkpoints
b6839ad verified
|
Raw History Blame Contribute Delete
265 Bytes

Tokenizer training

timestamp: 2026-07-01 07:54:22

  • max_chars: 2,000,000,000
  • doc_cap: 10,000
  • vocab_size: 32,768
  • train_time: 143.4839
  • num_special_tokens: 9
  • token_bytes_min: 1
  • token_bytes_max: 31
  • token_bytes_mean: 10.1118
  • token_bytes_std: 4.1123