Instructions to use HectorHe/OLMoE-1B-7B-0125-sft-s1K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HectorHe/OLMoE-1B-7B-0125-sft-s1K with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="HectorHe/OLMoE-1B-7B-0125-sft-s1K")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("HectorHe/OLMoE-1B-7B-0125-sft-s1K")
model = AutoModelForCausalLM.from_pretrained("HectorHe/OLMoE-1B-7B-0125-sft-s1K", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use HectorHe/OLMoE-1B-7B-0125-sft-s1K with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HectorHe/OLMoE-1B-7B-0125-sft-s1K"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HectorHe/OLMoE-1B-7B-0125-sft-s1K",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/HectorHe/OLMoE-1B-7B-0125-sft-s1K

SGLang

How to use HectorHe/OLMoE-1B-7B-0125-sft-s1K with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HectorHe/OLMoE-1B-7B-0125-sft-s1K" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HectorHe/OLMoE-1B-7B-0125-sft-s1K",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HectorHe/OLMoE-1B-7B-0125-sft-s1K" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HectorHe/OLMoE-1B-7B-0125-sft-s1K",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use HectorHe/OLMoE-1B-7B-0125-sft-s1K with Docker Model Runner:
```
docker model run hf.co/HectorHe/OLMoE-1B-7B-0125-sft-s1K
```

HectorHe commited on Aug 14, 2025

Commit

7786777

verified ·

1 Parent(s): 9061a08

Model save

Browse files

Files changed (6) hide show

README.md +58 -0
all_results.json +8 -0
generation_config.json +6 -0
train_results.json +8 -0
trainer_state.json +1753 -0
training.log +1 -0

README.md ADDED Viewed

	@@ -0,0 +1,58 @@

+---
+base_model: allenai/OLMoE-1B-7B-0125
+library_name: transformers
+model_name: OLMoE-1B-7B-0125-sft-s1K
+tags:
+- generated_from_trainer
+- trl
+- sft
+licence: license
+---
+# Model Card for OLMoE-1B-7B-0125-sft-s1K
+This model is a fine-tuned version of [allenai/OLMoE-1B-7B-0125](https://huggingface.co/allenai/OLMoE-1B-7B-0125).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+## Quick start
+```python
+from transformers import pipeline
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="HectorHe/OLMoE-1B-7B-0125-sft-s1K", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+## Training procedure
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/hector_-carnegie-mellon-university/huggingface/runs/peejphwz)
+This model was trained with SFT.
+### Framework versions
+- TRL: 0.18.0
+- Transformers: 4.55.2
+- Pytorch: 2.6.0
+- Datasets: 4.0.0
+- Tokenizers: 0.21.4
+## Citations
+Cite TRL as:
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

all_results.json ADDED Viewed

	@@ -0,0 +1,8 @@

+{
+    "total_flos": 39782134579200.0,
+    "train_loss": 1.4404595375061036,
+    "train_runtime": 2942.5505,
+    "train_samples": 1000,
+    "train_samples_per_second": 2.048,
+    "train_steps_per_second": 0.065
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,6 @@

+{
+  "_from_model_config": true,
+  "eos_token_id": 50279,
+  "pad_token_id": 1,
+  "transformers_version": "4.55.2"
+}

train_results.json ADDED Viewed

	@@ -0,0 +1,8 @@

+{
+    "total_flos": 39782134579200.0,
+    "train_loss": 1.4404595375061036,
+    "train_runtime": 2942.5505,
+    "train_samples": 1000,
+    "train_samples_per_second": 2.048,
+    "train_steps_per_second": 0.065
+}

trainer_state.json ADDED Viewed

	@@ -0,0 +1,1753 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 5.0,
+  "eval_steps": 500,
+  "global_step": 190,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.02631578947368421,
+      "grad_norm": 9.144888032555306,
+      "learning_rate": 0.0,
+      "loss": 3.0197,
+      "mean_token_accuracy": 0.435130774974823,
+      "num_tokens": 262144.0,
+      "step": 1
+    },
+    {
+      "epoch": 0.05263157894736842,
+      "grad_norm": 9.089893864449401,
+      "learning_rate": 5.263157894736843e-07,
+      "loss": 3.0448,
+      "mean_token_accuracy": 0.43275395035743713,
+      "num_tokens": 524288.0,
+      "step": 2
+    },
+    {
+      "epoch": 0.07894736842105263,
+      "grad_norm": 8.764265040006972,
+      "learning_rate": 1.0526315789473685e-06,
+      "loss": 3.1011,
+      "mean_token_accuracy": 0.4325822591781616,
+      "num_tokens": 786432.0,
+      "step": 3
+    },
+    {
+      "epoch": 0.10526315789473684,
+      "grad_norm": 9.548287094326655,
+      "learning_rate": 1.5789473684210526e-06,
+      "loss": 3.0779,
+      "mean_token_accuracy": 0.43555426597595215,
+      "num_tokens": 1048576.0,
+      "step": 4
+    },
+    {
+      "epoch": 0.13157894736842105,
+      "grad_norm": 9.17331273114906,
+      "learning_rate": 2.105263157894737e-06,
+      "loss": 2.9723,
+      "mean_token_accuracy": 0.4433600902557373,
+      "num_tokens": 1310720.0,
+      "step": 5
+    },
+    {
+      "epoch": 0.15789473684210525,
+      "grad_norm": 9.240746915878553,
+      "learning_rate": 2.631578947368421e-06,
+      "loss": 2.9841,
+      "mean_token_accuracy": 0.44807180762290955,
+      "num_tokens": 1572864.0,
+      "step": 6
+    },
+    {
+      "epoch": 0.18421052631578946,
+      "grad_norm": 7.341867990328414,
+      "learning_rate": 3.157894736842105e-06,
+      "loss": 2.9011,
+      "mean_token_accuracy": 0.4490027129650116,
+      "num_tokens": 1835008.0,
+      "step": 7
+    },
+    {
+      "epoch": 0.21052631578947367,
+      "grad_norm": 6.348735555429503,
+      "learning_rate": 3.6842105263157896e-06,
+      "loss": 2.8076,
+      "mean_token_accuracy": 0.4570488929748535,
+      "num_tokens": 2097152.0,
+      "step": 8
+    },
+    {
+      "epoch": 0.23684210526315788,
+      "grad_norm": 6.306969416042576,
+      "learning_rate": 4.210526315789474e-06,
+      "loss": 2.8711,
+      "mean_token_accuracy": 0.44867080450057983,
+      "num_tokens": 2359296.0,
+      "step": 9
+    },
+    {
+      "epoch": 0.2631578947368421,
+      "grad_norm": 4.435909523033183,
+      "learning_rate": 4.736842105263158e-06,
+      "loss": 2.721,
+      "mean_token_accuracy": 0.4588572680950165,
+      "num_tokens": 2621440.0,
+      "step": 10
+    },
+    {
+      "epoch": 0.2894736842105263,
+      "grad_norm": 3.9993913214616366,
+      "learning_rate": 5.263157894736842e-06,
+      "loss": 2.6792,
+      "mean_token_accuracy": 0.4573693573474884,
+      "num_tokens": 2883584.0,
+      "step": 11
+    },
+    {
+      "epoch": 0.3157894736842105,
+      "grad_norm": 3.923561561790813,
+      "learning_rate": 5.789473684210527e-06,
+      "loss": 2.6165,
+      "mean_token_accuracy": 0.4673383831977844,
+      "num_tokens": 3145728.0,
+      "step": 12
+    },
+    {
+      "epoch": 0.34210526315789475,
+      "grad_norm": 3.527199323221065,
+      "learning_rate": 6.31578947368421e-06,
+      "loss": 2.5569,
+      "mean_token_accuracy": 0.46943292021751404,
+      "num_tokens": 3407872.0,
+      "step": 13
+    },
+    {
+      "epoch": 0.3684210526315789,
+      "grad_norm": 3.540386269836106,
+      "learning_rate": 6.842105263157896e-06,
+      "loss": 2.4068,
+      "mean_token_accuracy": 0.48950067162513733,
+      "num_tokens": 3670016.0,
+      "step": 14
+    },
+    {
+      "epoch": 0.39473684210526316,
+      "grad_norm": 2.8367351336803397,
+      "learning_rate": 7.368421052631579e-06,
+      "loss": 2.38,
+      "mean_token_accuracy": 0.4991988241672516,
+      "num_tokens": 3932160.0,
+      "step": 15
+    },
+    {
+      "epoch": 0.42105263157894735,
+      "grad_norm": 2.7891206971825206,
+      "learning_rate": 7.894736842105265e-06,
+      "loss": 2.4209,
+      "mean_token_accuracy": 0.4855557978153229,
+      "num_tokens": 4194304.0,
+      "step": 16
+    },
+    {
+      "epoch": 0.4473684210526316,
+      "grad_norm": 2.215561269771985,
+      "learning_rate": 8.421052631578948e-06,
+      "loss": 2.3677,
+      "mean_token_accuracy": 0.4911602735519409,
+      "num_tokens": 4456448.0,
+      "step": 17
+    },
+    {
+      "epoch": 0.47368421052631576,
+      "grad_norm": 2.2769577851472915,
+      "learning_rate": 8.947368421052632e-06,
+      "loss": 2.238,
+      "mean_token_accuracy": 0.5115599632263184,
+      "num_tokens": 4718592.0,
+      "step": 18
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 2.2246857453539426,
+      "learning_rate": 9.473684210526315e-06,
+      "loss": 2.1407,
+      "mean_token_accuracy": 0.5227155089378357,
+      "num_tokens": 4980736.0,
+      "step": 19
+    },
+    {
+      "epoch": 0.5263157894736842,
+      "grad_norm": 2.0661055710278697,
+      "learning_rate": 1e-05,
+      "loss": 2.207,
+      "mean_token_accuracy": 0.5132614970207214,
+      "num_tokens": 5242880.0,
+      "step": 20
+    },
+    {
+      "epoch": 0.5526315789473685,
+      "grad_norm": 2.1978917959916364,
+      "learning_rate": 9.999240587349952e-06,
+      "loss": 2.0393,
+      "mean_token_accuracy": 0.5380829572677612,
+      "num_tokens": 5505024.0,
+      "step": 21
+    },
+    {
+      "epoch": 0.5789473684210527,
+      "grad_norm": 1.9187572184546404,
+      "learning_rate": 9.996962605714288e-06,
+      "loss": 2.0578,
+      "mean_token_accuracy": 0.5336993336677551,
+      "num_tokens": 5767168.0,
+      "step": 22
+    },
+    {
+      "epoch": 0.6052631578947368,
+      "grad_norm": 1.751193492821015,
+      "learning_rate": 9.993166823949924e-06,
+      "loss": 2.007,
+      "mean_token_accuracy": 0.5419515371322632,
+      "num_tokens": 6029312.0,
+      "step": 23
+    },
+    {
+      "epoch": 0.631578947368421,
+      "grad_norm": 1.7120403735686438,
+      "learning_rate": 9.987854523196724e-06,
+      "loss": 1.9728,
+      "mean_token_accuracy": 0.5452325940132141,
+      "num_tokens": 6291456.0,
+      "step": 24
+    },
+    {
+      "epoch": 0.6578947368421053,
+      "grad_norm": 1.6167100976334166,
+      "learning_rate": 9.981027496445084e-06,
+      "loss": 1.97,
+      "mean_token_accuracy": 0.5456637144088745,
+      "num_tokens": 6553600.0,
+      "step": 25
+    },
+    {
+      "epoch": 0.6842105263157895,
+      "grad_norm": 1.6494993317134798,
+      "learning_rate": 9.972688047930773e-06,
+      "loss": 1.9057,
+      "mean_token_accuracy": 0.5569413304328918,
+      "num_tokens": 6815744.0,
+      "step": 26
+    },
+    {
+      "epoch": 0.7105263157894737,
+      "grad_norm": 1.4116871328104919,
+      "learning_rate": 9.962838992357212e-06,
+      "loss": 1.8411,
+      "mean_token_accuracy": 0.5720035433769226,
+      "num_tokens": 7077888.0,
+      "step": 27
+    },
+    {
+      "epoch": 0.7368421052631579,
+      "grad_norm": 1.3548679701598223,
+      "learning_rate": 9.951483653945461e-06,
+      "loss": 1.8332,
+      "mean_token_accuracy": 0.5701990127563477,
+      "num_tokens": 7340032.0,
+      "step": 28
+    },
+    {
+      "epoch": 0.7631578947368421,
+      "grad_norm": 1.5289726872863745,
+      "learning_rate": 9.938625865312252e-06,
+      "loss": 1.8215,
+      "mean_token_accuracy": 0.5682075023651123,
+      "num_tokens": 7602176.0,
+      "step": 29
+    },
+    {
+      "epoch": 0.7894736842105263,
+      "grad_norm": 1.4048047900834215,
+      "learning_rate": 9.9242699661764e-06,
+      "loss": 1.764,
+      "mean_token_accuracy": 0.5825715661048889,
+      "num_tokens": 7864320.0,
+      "step": 30
+    },
+    {
+      "epoch": 0.8157894736842105,
+      "grad_norm": 1.3199591889964046,
+      "learning_rate": 9.908420801894083e-06,
+      "loss": 1.698,
+      "mean_token_accuracy": 0.5920604467391968,
+      "num_tokens": 8118799.0,
+      "step": 31
+    },
+    {
+      "epoch": 0.8421052631578947,
+      "grad_norm": 1.2695821423340337,
+      "learning_rate": 9.89108372182346e-06,
+      "loss": 1.7723,
+      "mean_token_accuracy": 0.581972599029541,
+      "num_tokens": 8380943.0,
+      "step": 32
+    },
+    {
+      "epoch": 0.868421052631579,
+      "grad_norm": 1.2217610450324474,
+      "learning_rate": 9.872264577519163e-06,
+      "loss": 1.6117,
+      "mean_token_accuracy": 0.6087397933006287,
+      "num_tokens": 8643087.0,
+      "step": 33
+    },
+    {
+      "epoch": 0.8947368421052632,
+      "grad_norm": 1.5890070323202978,
+      "learning_rate": 9.8519697207573e-06,
+      "loss": 1.6977,
+      "mean_token_accuracy": 0.5938796997070312,
+      "num_tokens": 8905231.0,
+      "step": 34
+    },
+    {
+      "epoch": 0.9210526315789473,
+      "grad_norm": 18.086842934362615,
+      "learning_rate": 9.830206001391627e-06,
+      "loss": 1.6338,
+      "mean_token_accuracy": 0.610822856426239,
+      "num_tokens": 9167375.0,
+      "step": 35
+    },
+    {
+      "epoch": 0.9473684210526315,
+      "grad_norm": 1.47252943858889,
+      "learning_rate": 9.806980765041615e-06,
+      "loss": 1.6437,
+      "mean_token_accuracy": 0.6038067936897278,
+      "num_tokens": 9429519.0,
+      "step": 36
+    },
+    {
+      "epoch": 0.9736842105263158,
+      "grad_norm": 1.1229751841198494,
+      "learning_rate": 9.782301850613165e-06,
+      "loss": 1.591,
+      "mean_token_accuracy": 0.6120895147323608,
+      "num_tokens": 9691663.0,
+      "step": 37
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 1.0329182091569846,
+      "learning_rate": 9.756177587652857e-06,
+      "loss": 1.6121,
+      "mean_token_accuracy": 0.6088923811912537,
+      "num_tokens": 9953807.0,
+      "step": 38
+    },
+    {
+      "epoch": 1.0263157894736843,
+      "grad_norm": 1.037629173374562,
+      "learning_rate": 9.728616793536588e-06,
+      "loss": 1.5069,
+      "mean_token_accuracy": 0.6265947222709656,
+      "num_tokens": 10215951.0,
+      "step": 39
+    },
+    {
+      "epoch": 1.0526315789473684,
+      "grad_norm": 1.1247130552032012,
+      "learning_rate": 9.699628770493557e-06,
+      "loss": 1.6151,
+      "mean_token_accuracy": 0.6127304434776306,
+      "num_tokens": 10478095.0,
+      "step": 40
+    },
+    {
+      "epoch": 1.0789473684210527,
+      "grad_norm": 0.9326470270807049,
+      "learning_rate": 9.669223302466609e-06,
+      "loss": 1.5202,
+      "mean_token_accuracy": 0.6245765089988708,
+      "num_tokens": 10740239.0,
+      "step": 41
+    },
+    {
+      "epoch": 1.1052631578947367,
+      "grad_norm": 1.081738987272881,
+      "learning_rate": 9.637410651809985e-06,
+      "loss": 1.4707,
+      "mean_token_accuracy": 0.639268696308136,
+      "num_tokens": 11002383.0,
+      "step": 42
+    },
+    {
+      "epoch": 1.131578947368421,
+      "grad_norm": 1.0128599536731275,
+      "learning_rate": 9.604201555825618e-06,
+      "loss": 1.5681,
+      "mean_token_accuracy": 0.6182166337966919,
+      "num_tokens": 11264527.0,
+      "step": 43
+    },
+    {
+      "epoch": 1.1578947368421053,
+      "grad_norm": 0.9486897801531551,
+      "learning_rate": 9.5696072231391e-06,
+      "loss": 1.4973,
+      "mean_token_accuracy": 0.6312530636787415,
+      "num_tokens": 11526671.0,
+      "step": 44
+    },
+    {
+      "epoch": 1.1842105263157894,
+      "grad_norm": 0.9528661635292972,
+      "learning_rate": 9.533639329916592e-06,
+      "loss": 1.3808,
+      "mean_token_accuracy": 0.6557158827781677,
+      "num_tokens": 11788815.0,
+      "step": 45
+    },
+    {
+      "epoch": 1.2105263157894737,
+      "grad_norm": 15.454966425556524,
+      "learning_rate": 9.496310015923921e-06,
+      "loss": 1.5053,
+      "mean_token_accuracy": 0.6316651105880737,
+      "num_tokens": 12050959.0,
+      "step": 46
+    },
+    {
+      "epoch": 1.236842105263158,
+      "grad_norm": 0.9538351658576188,
+      "learning_rate": 9.4576318804292e-06,
+      "loss": 1.4802,
+      "mean_token_accuracy": 0.6355336904525757,
+      "num_tokens": 12313103.0,
+      "step": 47
+    },
+    {
+      "epoch": 1.263157894736842,
+      "grad_norm": 0.8318020814980096,
+      "learning_rate": 9.417617977950383e-06,
+      "loss": 1.4011,
+      "mean_token_accuracy": 0.6495734453201294,
+      "num_tokens": 12575247.0,
+      "step": 48
+    },
+    {
+      "epoch": 1.2894736842105263,
+      "grad_norm": 0.87491229088957,
+      "learning_rate": 9.376281813849114e-06,
+      "loss": 1.4425,
+      "mean_token_accuracy": 0.6401004195213318,
+      "num_tokens": 12837391.0,
+      "step": 49
+    },
+    {
+      "epoch": 1.3157894736842106,
+      "grad_norm": 13.352523689341348,
+      "learning_rate": 9.333637339772472e-06,
+      "loss": 1.4498,
+      "mean_token_accuracy": 0.6377197504043579,
+      "num_tokens": 13099535.0,
+      "step": 50
+    },
+    {
+      "epoch": 1.3421052631578947,
+      "grad_norm": 2.6904758771681445,
+      "learning_rate": 9.289698948944046e-06,
+      "loss": 1.3891,
+      "mean_token_accuracy": 0.6509392857551575,
+      "num_tokens": 13361679.0,
+      "step": 51
+    },
+    {
+      "epoch": 1.368421052631579,
+      "grad_norm": 1.035669421403492,
+      "learning_rate": 9.244481471305977e-06,
+      "loss": 1.3858,
+      "mean_token_accuracy": 0.6511223912239075,
+      "num_tokens": 13623823.0,
+      "step": 52
+    },
+    {
+      "epoch": 1.3947368421052633,
+      "grad_norm": 88.70855789773888,
+      "learning_rate": 9.198000168513604e-06,
+      "loss": 1.4689,
+      "mean_token_accuracy": 0.6373191475868225,
+      "num_tokens": 13885967.0,
+      "step": 53
+    },
+    {
+      "epoch": 1.4210526315789473,
+      "grad_norm": 0.8796467760984398,
+      "learning_rate": 9.150270728784411e-06,
+      "loss": 1.441,
+      "mean_token_accuracy": 0.6398676633834839,
+      "num_tokens": 14148111.0,
+      "step": 54
+    },
+    {
+      "epoch": 1.4473684210526316,
+      "grad_norm": 0.8537471648823969,
+      "learning_rate": 9.101309261602978e-06,
+      "loss": 1.4113,
+      "mean_token_accuracy": 0.6479100584983826,
+      "num_tokens": 14410255.0,
+      "step": 55
+    },
+    {
+      "epoch": 1.4736842105263157,
+      "grad_norm": 0.8164741757465972,
+      "learning_rate": 9.051132292283772e-06,
+      "loss": 1.4088,
+      "mean_token_accuracy": 0.6451835036277771,
+      "num_tokens": 14664734.0,
+      "step": 56
+    },
+    {
+      "epoch": 1.5,
+      "grad_norm": 0.8363413113484846,
+      "learning_rate": 8.999756756393562e-06,
+      "loss": 1.4238,
+      "mean_token_accuracy": 0.6428511738777161,
+      "num_tokens": 14926878.0,
+      "step": 57
+    },
+    {
+      "epoch": 1.526315789473684,
+      "grad_norm": 0.7766725489164594,
+      "learning_rate": 8.947199994035402e-06,
+      "loss": 1.3153,
+      "mean_token_accuracy": 0.6633500456809998,
+      "num_tokens": 15189022.0,
+      "step": 58
+    },
+    {
+      "epoch": 1.5526315789473686,
+      "grad_norm": 0.8059596602743602,
+      "learning_rate": 8.893479743996034e-06,
+      "loss": 1.3448,
+      "mean_token_accuracy": 0.65765780210495,
+      "num_tokens": 15451166.0,
+      "step": 59
+    },
+    {
+      "epoch": 1.5789473684210527,
+      "grad_norm": 0.7324537445066931,
+      "learning_rate": 8.838614137758767e-06,
+      "loss": 1.3363,
+      "mean_token_accuracy": 0.6614806056022644,
+      "num_tokens": 15713310.0,
+      "step": 60
+    },
+    {
+      "epoch": 1.6052631578947367,
+      "grad_norm": 0.77781086842083,
+      "learning_rate": 8.782621693383793e-06,
+      "loss": 1.4157,
+      "mean_token_accuracy": 0.6485319137573242,
+      "num_tokens": 15975454.0,
+      "step": 61
+    },
+    {
+      "epoch": 1.631578947368421,
+      "grad_norm": 0.7061723115441305,
+      "learning_rate": 8.725521309258031e-06,
+      "loss": 1.3202,
+      "mean_token_accuracy": 0.6643381714820862,
+      "num_tokens": 16237598.0,
+      "step": 62
+    },
+    {
+      "epoch": 1.6578947368421053,
+      "grad_norm": 0.7141045939197578,
+      "learning_rate": 8.667332257716603e-06,
+      "loss": 1.3325,
+      "mean_token_accuracy": 0.6641473770141602,
+      "num_tokens": 16499742.0,
+      "step": 63
+    },
+    {
+      "epoch": 1.6842105263157894,
+      "grad_norm": 0.740270282620103,
+      "learning_rate": 8.608074178538101e-06,
+      "loss": 1.428,
+      "mean_token_accuracy": 0.6470096707344055,
+      "num_tokens": 16761886.0,
+      "step": 64
+    },
+    {
+      "epoch": 1.7105263157894737,
+      "grad_norm": 0.6639377114211265,
+      "learning_rate": 8.547767072315835e-06,
+      "loss": 1.4018,
+      "mean_token_accuracy": 0.6485319137573242,
+      "num_tokens": 17024030.0,
+      "step": 65
+    },
+    {
+      "epoch": 1.736842105263158,
+      "grad_norm": 0.6739541270775268,
+      "learning_rate": 8.486431293707298e-06,
+      "loss": 1.3142,
+      "mean_token_accuracy": 0.6693703532218933,
+      "num_tokens": 17286174.0,
+      "step": 66
+    },
+    {
+      "epoch": 1.763157894736842,
+      "grad_norm": 1.5441747838886493,
+      "learning_rate": 8.42408754456412e-06,
+      "loss": 1.3586,
+      "mean_token_accuracy": 0.6568832993507385,
+      "num_tokens": 17548318.0,
+      "step": 67
+    },
+    {
+      "epoch": 1.7894736842105263,
+      "grad_norm": 0.7004893065739863,
+      "learning_rate": 8.360756866944858e-06,
+      "loss": 1.3285,
+      "mean_token_accuracy": 0.6602864265441895,
+      "num_tokens": 17810462.0,
+      "step": 68
+    },
+    {
+      "epoch": 1.8157894736842106,
+      "grad_norm": 0.6598266465075178,
+      "learning_rate": 8.296460636012942e-06,
+      "loss": 1.3476,
+      "mean_token_accuracy": 0.6597141623497009,
+      "num_tokens": 18072606.0,
+      "step": 69
+    },
+    {
+      "epoch": 1.8421052631578947,
+      "grad_norm": 3.373108959827941,
+      "learning_rate": 8.231220552822206e-06,
+      "loss": 1.3502,
+      "mean_token_accuracy": 0.6596264243125916,
+      "num_tokens": 18334750.0,
+      "step": 70
+    },
+    {
+      "epoch": 1.868421052631579,
+      "grad_norm": 0.7335650187416184,
+      "learning_rate": 8.165058636992411e-06,
+      "loss": 1.3047,
+      "mean_token_accuracy": 0.670862078666687,
+      "num_tokens": 18596894.0,
+      "step": 71
+    },
+    {
+      "epoch": 1.8947368421052633,
+      "grad_norm": 0.6500785746933061,
+      "learning_rate": 8.09799721927726e-06,
+      "loss": 1.3302,
+      "mean_token_accuracy": 0.6620299816131592,
+      "num_tokens": 18859038.0,
+      "step": 72
+    },
+    {
+      "epoch": 1.9210526315789473,
+      "grad_norm": 0.6274604158335885,
+      "learning_rate": 8.030058934027395e-06,
+      "loss": 1.2792,
+      "mean_token_accuracy": 0.6728650331497192,
+      "num_tokens": 19121182.0,
+      "step": 73
+    },
+    {
+      "epoch": 1.9473684210526314,
+      "grad_norm": 0.6291519836042733,
+      "learning_rate": 7.961266711550922e-06,
+      "loss": 1.3297,
+      "mean_token_accuracy": 0.6599164009094238,
+      "num_tokens": 19383326.0,
+      "step": 74
+    },
+    {
+      "epoch": 1.973684210526316,
+      "grad_norm": 0.6444596016200881,
+      "learning_rate": 7.891643770374052e-06,
+      "loss": 1.3164,
+      "mean_token_accuracy": 0.6649523973464966,
+      "num_tokens": 19645470.0,
+      "step": 75
+    },
+    {
+      "epoch": 2.0,
+      "grad_norm": 28.209700826902587,
+      "learning_rate": 7.821213609404459e-06,
+      "loss": 1.3146,
+      "mean_token_accuracy": 0.6623695492744446,
+      "num_tokens": 19907614.0,
+      "step": 76
+    },
+    {
+      "epoch": 2.026315789473684,
+      "grad_norm": 0.708893212650595,
+      "learning_rate": 7.75e-06,
+      "loss": 1.2725,
+      "mean_token_accuracy": 0.6715068221092224,
+      "num_tokens": 20169758.0,
+      "step": 77
+    },
+    {
+      "epoch": 2.0526315789473686,
+      "grad_norm": 0.9273454935271076,
+      "learning_rate": 7.678026977945497e-06,
+      "loss": 1.2723,
+      "mean_token_accuracy": 0.6746886968612671,
+      "num_tokens": 20431902.0,
+      "step": 78
+    },
+    {
+      "epoch": 2.0789473684210527,
+      "grad_norm": 0.6077828732079009,
+      "learning_rate": 7.605318835340237e-06,
+      "loss": 1.1928,
+      "mean_token_accuracy": 0.6885720491409302,
+      "num_tokens": 20694046.0,
+      "step": 79
+    },
+    {
+      "epoch": 2.1052631578947367,
+      "grad_norm": 0.6625126850722808,
+      "learning_rate": 7.531900112399004e-06,
+      "loss": 1.2694,
+      "mean_token_accuracy": 0.6737463474273682,
+      "num_tokens": 20956190.0,
+      "step": 80
+    },
+    {
+      "epoch": 2.1315789473684212,
+      "grad_norm": 0.6763431442698962,
+      "learning_rate": 7.4577955891693295e-06,
+      "loss": 1.2715,
+      "mean_token_accuracy": 0.6732046008110046,
+      "num_tokens": 21218334.0,
+      "step": 81
+    },
+    {
+      "epoch": 2.1578947368421053,
+      "grad_norm": 0.5832389341094574,
+      "learning_rate": 7.383030277167825e-06,
+      "loss": 1.2205,
+      "mean_token_accuracy": 0.683753490447998,
+      "num_tokens": 21480478.0,
+      "step": 82
+    },
+    {
+      "epoch": 2.1842105263157894,
+      "grad_norm": 2.2405291609094293,
+      "learning_rate": 7.307629410938364e-06,
+      "loss": 1.278,
+      "mean_token_accuracy": 0.6732503771781921,
+      "num_tokens": 21742622.0,
+      "step": 83
+    },
+    {
+      "epoch": 2.2105263157894735,
+      "grad_norm": 2.138457688507137,
+      "learning_rate": 7.231618439535006e-06,
+      "loss": 1.2778,
+      "mean_token_accuracy": 0.668744683265686,
+      "num_tokens": 22004766.0,
+      "step": 84
+    },
+    {
+      "epoch": 2.236842105263158,
+      "grad_norm": 0.6375029873512914,
+      "learning_rate": 7.155023017932521e-06,
+      "loss": 1.2534,
+      "mean_token_accuracy": 0.6742499470710754,
+      "num_tokens": 22266910.0,
+      "step": 85
+    },
+    {
+      "epoch": 2.263157894736842,
+      "grad_norm": 0.6682751703287256,
+      "learning_rate": 7.0778689983673955e-06,
+      "loss": 1.2666,
+      "mean_token_accuracy": 0.6781108975410461,
+      "num_tokens": 22529054.0,
+      "step": 86
+    },
+    {
+      "epoch": 2.2894736842105265,
+      "grad_norm": 0.5844308031288254,
+      "learning_rate": 7.000182421612281e-06,
+      "loss": 1.2878,
+      "mean_token_accuracy": 0.6716785430908203,
+      "num_tokens": 22791198.0,
+      "step": 87
+    },
+    {
+      "epoch": 2.3157894736842106,
+      "grad_norm": 0.6304946926699229,
+      "learning_rate": 6.921989508186787e-06,
+      "loss": 1.2245,
+      "mean_token_accuracy": 0.6867827773094177,
+      "num_tokens": 23053342.0,
+      "step": 88
+    },
+    {
+      "epoch": 2.3421052631578947,
+      "grad_norm": 0.588447827602647,
+      "learning_rate": 6.843316649507627e-06,
+      "loss": 1.2488,
+      "mean_token_accuracy": 0.6804381608963013,
+      "num_tokens": 23315486.0,
+      "step": 89
+    },
+    {
+      "epoch": 2.3684210526315788,
+      "grad_norm": 0.5752511341385269,
+      "learning_rate": 6.764190398981063e-06,
+      "loss": 1.2183,
+      "mean_token_accuracy": 0.6857373714447021,
+      "num_tokens": 23577630.0,
+      "step": 90
+    },
+    {
+      "epoch": 2.3947368421052633,
+      "grad_norm": 0.601133537795538,
+      "learning_rate": 6.684637463040698e-06,
+      "loss": 1.1987,
+      "mean_token_accuracy": 0.6845928430557251,
+      "num_tokens": 23839774.0,
+      "step": 91
+    },
+    {
+      "epoch": 2.4210526315789473,
+      "grad_norm": 1.599039072518535,
+      "learning_rate": 6.604684692133597e-06,
+      "loss": 1.2244,
+      "mean_token_accuracy": 0.6837954521179199,
+      "num_tokens": 24101918.0,
+      "step": 92
+    },
+    {
+      "epoch": 2.4473684210526314,
+      "grad_norm": 17.31027379733536,
+      "learning_rate": 6.524359071657822e-06,
+      "loss": 1.2287,
+      "mean_token_accuracy": 0.6827691793441772,
+      "num_tokens": 24364062.0,
+      "step": 93
+    },
+    {
+      "epoch": 2.473684210526316,
+      "grad_norm": 0.6809165019277617,
+      "learning_rate": 6.443687712854397e-06,
+      "loss": 1.2708,
+      "mean_token_accuracy": 0.676279604434967,
+      "num_tokens": 24626206.0,
+      "step": 94
+    },
+    {
+      "epoch": 2.5,
+      "grad_norm": 0.6442727822424891,
+      "learning_rate": 6.362697843656823e-06,
+      "loss": 1.2795,
+      "mean_token_accuracy": 0.6701677441596985,
+      "num_tokens": 24888350.0,
+      "step": 95
+    },
+    {
+      "epoch": 2.526315789473684,
+      "grad_norm": 0.5769629627925874,
+      "learning_rate": 6.281416799501188e-06,
+      "loss": 1.2311,
+      "mean_token_accuracy": 0.6823189854621887,
+      "num_tokens": 25150494.0,
+      "step": 96
+    },
+    {
+      "epoch": 2.5526315789473686,
+      "grad_norm": 1.233425967241537,
+      "learning_rate": 6.199872014099995e-06,
+      "loss": 1.2665,
+      "mean_token_accuracy": 0.6751083731651306,
+      "num_tokens": 25412638.0,
+      "step": 97
+    },
+    {
+      "epoch": 2.5789473684210527,
+      "grad_norm": 0.7476662613111043,
+      "learning_rate": 6.118091010182837e-06,
+      "loss": 1.2243,
+      "mean_token_accuracy": 0.6817924976348877,
+      "num_tokens": 25674782.0,
+      "step": 98
+    },
+    {
+      "epoch": 2.6052631578947367,
+      "grad_norm": 1.58371089124971,
+      "learning_rate": 6.036101390207006e-06,
+      "loss": 1.1538,
+      "mean_token_accuracy": 0.7012002468109131,
+      "num_tokens": 25936926.0,
+      "step": 99
+    },
+    {
+      "epoch": 2.6315789473684212,
+      "grad_norm": 1.4370365700790926,
+      "learning_rate": 5.953930827041209e-06,
+      "loss": 1.2446,
+      "mean_token_accuracy": 0.6795377731323242,
+      "num_tokens": 26199070.0,
+      "step": 100
+    },
+    {
+      "epoch": 2.6578947368421053,
+      "grad_norm": 0.7891205485960927,
+      "learning_rate": 5.871607054625497e-06,
+      "loss": 1.2842,
+      "mean_token_accuracy": 0.6718769073486328,
+      "num_tokens": 26461214.0,
+      "step": 101
+    },
+    {
+      "epoch": 2.6842105263157894,
+      "grad_norm": 7.060494358042542,
+      "learning_rate": 5.7891578586106165e-06,
+      "loss": 1.1096,
+      "mean_token_accuracy": 0.7096241116523743,
+      "num_tokens": 26723358.0,
+      "step": 102
+    },
+    {
+      "epoch": 2.7105263157894735,
+      "grad_norm": 2.0263515651724164,
+      "learning_rate": 5.706611066979874e-06,
+      "loss": 1.269,
+      "mean_token_accuracy": 0.6746886968612671,
+      "num_tokens": 26985502.0,
+      "step": 103
+    },
+    {
+      "epoch": 2.736842105263158,
+      "grad_norm": 1.570123901253544,
+      "learning_rate": 5.62399454065673e-06,
+      "loss": 1.2304,
+      "mean_token_accuracy": 0.681018054485321,
+      "num_tokens": 27247646.0,
+      "step": 104
+    },
+    {
+      "epoch": 2.763157894736842,
+      "grad_norm": 0.7111522615328816,
+      "learning_rate": 5.54133616410127e-06,
+      "loss": 1.2478,
+      "mean_token_accuracy": 0.6804953813552856,
+      "num_tokens": 27509790.0,
+      "step": 105
+    },
+    {
+      "epoch": 2.7894736842105265,
+      "grad_norm": 0.9119519373287289,
+      "learning_rate": 5.458663835898732e-06,
+      "loss": 1.2073,
+      "mean_token_accuracy": 0.6871718764305115,
+      "num_tokens": 27771934.0,
+      "step": 106
+    },
+    {
+      "epoch": 2.8157894736842106,
+      "grad_norm": 0.9240457779811717,
+      "learning_rate": 5.376005459343272e-06,
+      "loss": 1.2718,
+      "mean_token_accuracy": 0.6782024502754211,
+      "num_tokens": 28034078.0,
+      "step": 107
+    },
+    {
+      "epoch": 2.8421052631578947,
+      "grad_norm": 0.5926290558432593,
+      "learning_rate": 5.293388933020127e-06,
+      "loss": 1.2398,
+      "mean_token_accuracy": 0.6791562438011169,
+      "num_tokens": 28296222.0,
+      "step": 108
+    },
+    {
+      "epoch": 2.8684210526315788,
+      "grad_norm": 0.5937415300824982,
+      "learning_rate": 5.210842141389385e-06,
+      "loss": 1.2589,
+      "mean_token_accuracy": 0.6773592829704285,
+      "num_tokens": 28558366.0,
+      "step": 109
+    },
+    {
+      "epoch": 2.8947368421052633,
+      "grad_norm": 0.8205931665334268,
+      "learning_rate": 5.1283929453745055e-06,
+      "loss": 1.2793,
+      "mean_token_accuracy": 0.6735174059867859,
+      "num_tokens": 28820510.0,
+      "step": 110
+    },
+    {
+      "epoch": 2.9210526315789473,
+      "grad_norm": 0.5510081067492951,
+      "learning_rate": 5.046069172958794e-06,
+      "loss": 1.2383,
+      "mean_token_accuracy": 0.6795263290405273,
+      "num_tokens": 29082654.0,
+      "step": 111
+    },
+    {
+      "epoch": 2.9473684210526314,
+      "grad_norm": 0.5717504770508175,
+      "learning_rate": 4.963898609792994e-06,
+      "loss": 1.3184,
+      "mean_token_accuracy": 0.6652461290359497,
+      "num_tokens": 29344798.0,
+      "step": 112
+    },
+    {
+      "epoch": 2.973684210526316,
+      "grad_norm": 0.6116330301476128,
+      "learning_rate": 4.881908989817163e-06,
+      "loss": 1.1489,
+      "mean_token_accuracy": 0.6971180438995361,
+      "num_tokens": 29606942.0,
+      "step": 113
+    },
+    {
+      "epoch": 3.0,
+      "grad_norm": 0.5899271048276149,
+      "learning_rate": 4.8001279859000074e-06,
+      "loss": 1.1643,
+      "mean_token_accuracy": 0.6946829557418823,
+      "num_tokens": 29861421.0,
+      "step": 114
+    },
+    {
+      "epoch": 3.026315789473684,
+      "grad_norm": 0.6052609643580157,
+      "learning_rate": 4.718583200498814e-06,
+      "loss": 1.2267,
+      "mean_token_accuracy": 0.6843944787979126,
+      "num_tokens": 30123565.0,
+      "step": 115
+    },
+    {
+      "epoch": 3.0526315789473686,
+      "grad_norm": 0.5571230863291556,
+      "learning_rate": 4.637302156343178e-06,
+      "loss": 1.137,
+      "mean_token_accuracy": 0.7024783492088318,
+      "num_tokens": 30385709.0,
+      "step": 116
+    },
+    {
+      "epoch": 3.0789473684210527,
+      "grad_norm": 1.9895691560023896,
+      "learning_rate": 4.556312287145604e-06,
+      "loss": 1.151,
+      "mean_token_accuracy": 0.6993651390075684,
+      "num_tokens": 30647853.0,
+      "step": 117
+    },
+    {
+      "epoch": 3.1052631578947367,
+      "grad_norm": 1.091284264386462,
+      "learning_rate": 4.475640928342182e-06,
+      "loss": 1.132,
+      "mean_token_accuracy": 0.7014902234077454,
+      "num_tokens": 30909997.0,
+      "step": 118
+    },
+    {
+      "epoch": 3.1315789473684212,
+      "grad_norm": 0.6626943880051851,
+      "learning_rate": 4.395315307866404e-06,
+      "loss": 1.2015,
+      "mean_token_accuracy": 0.6918988823890686,
+      "num_tokens": 31172141.0,
+      "step": 119
+    },
+    {
+      "epoch": 3.1578947368421053,
+      "grad_norm": 0.6523541542451436,
+      "learning_rate": 4.315362536959305e-06,
+      "loss": 1.2112,
+      "mean_token_accuracy": 0.6867483854293823,
+      "num_tokens": 31434285.0,
+      "step": 120
+    },
+    {
+      "epoch": 3.1842105263157894,
+      "grad_norm": 0.8841056815148688,
+      "learning_rate": 4.2358096010189385e-06,
+      "loss": 1.2752,
+      "mean_token_accuracy": 0.6752685904502869,
+      "num_tokens": 31696429.0,
+      "step": 121
+    },
+    {
+      "epoch": 3.2105263157894735,
+      "grad_norm": 0.6012402010833562,
+      "learning_rate": 4.156683350492376e-06,
+      "loss": 1.1933,
+      "mean_token_accuracy": 0.6922765970230103,
+      "num_tokens": 31958573.0,
+      "step": 122
+    },
+    {
+      "epoch": 3.236842105263158,
+      "grad_norm": 0.6163214035759135,
+      "learning_rate": 4.078010491813215e-06,
+      "loss": 1.2009,
+      "mean_token_accuracy": 0.6913189888000488,
+      "num_tokens": 32220717.0,
+      "step": 123
+    },
+    {
+      "epoch": 3.263157894736842,
+      "grad_norm": 0.5669681844522956,
+      "learning_rate": 3.999817578387723e-06,
+      "loss": 1.1801,
+      "mean_token_accuracy": 0.6925512552261353,
+      "num_tokens": 32482861.0,
+      "step": 124
+    },
+    {
+      "epoch": 3.2894736842105265,
+      "grad_norm": 0.5503406805003092,
+      "learning_rate": 3.922131001632607e-06,
+      "loss": 1.0835,
+      "mean_token_accuracy": 0.7118712663650513,
+      "num_tokens": 32745005.0,
+      "step": 125
+    },
+    {
+      "epoch": 3.3157894736842106,
+      "grad_norm": 0.7031631763052849,
+      "learning_rate": 3.8449769820674824e-06,
+      "loss": 1.1845,
+      "mean_token_accuracy": 0.6908764243125916,
+      "num_tokens": 33007149.0,
+      "step": 126
+    },
+    {
+      "epoch": 3.3421052631578947,
+      "grad_norm": 0.5523930786032378,
+      "learning_rate": 3.7683815604649955e-06,
+      "loss": 1.1769,
+      "mean_token_accuracy": 0.694538950920105,
+      "num_tokens": 33269293.0,
+      "step": 127
+    },
+    {
+      "epoch": 3.3684210526315788,
+      "grad_norm": 0.5731302432477412,
+      "learning_rate": 3.692370589061639e-06,
+      "loss": 1.1629,
+      "mean_token_accuracy": 0.6966297030448914,
+      "num_tokens": 33531437.0,
+      "step": 128
+    },
+    {
+      "epoch": 3.3947368421052633,
+      "grad_norm": 0.5520020236878893,
+      "learning_rate": 3.616969722832174e-06,
+      "loss": 1.1374,
+      "mean_token_accuracy": 0.7009102702140808,
+      "num_tokens": 33793581.0,
+      "step": 129
+    },
+    {
+      "epoch": 3.4210526315789473,
+      "grad_norm": 2.5073145307120246,
+      "learning_rate": 3.542204410830672e-06,
+      "loss": 1.1954,
+      "mean_token_accuracy": 0.6902927160263062,
+      "num_tokens": 34055725.0,
+      "step": 130
+    },
+    {
+      "epoch": 3.4473684210526314,
+      "grad_norm": 0.7447715458686521,
+      "learning_rate": 3.468099887600999e-06,
+      "loss": 1.1337,
+      "mean_token_accuracy": 0.6988958716392517,
+      "num_tokens": 34317869.0,
+      "step": 131
+    },
+    {
+      "epoch": 3.473684210526316,
+      "grad_norm": 0.55360781640575,
+      "learning_rate": 3.394681164659766e-06,
+      "loss": 1.1612,
+      "mean_token_accuracy": 0.693768322467804,
+      "num_tokens": 34580013.0,
+      "step": 132
+    },
+    {
+      "epoch": 3.5,
+      "grad_norm": 0.6209441329712366,
+      "learning_rate": 3.3219730220545065e-06,
+      "loss": 1.1428,
+      "mean_token_accuracy": 0.6991400718688965,
+      "num_tokens": 34842157.0,
+      "step": 133
+    },
+    {
+      "epoch": 3.526315789473684,
+      "grad_norm": 0.5778728779594863,
+      "learning_rate": 3.2500000000000015e-06,
+      "loss": 1.1229,
+      "mean_token_accuracy": 0.7043516039848328,
+      "num_tokens": 35104301.0,
+      "step": 134
+    },
+    {
+      "epoch": 3.5526315789473686,
+      "grad_norm": 1.0328742077694102,
+      "learning_rate": 3.178786390595544e-06,
+      "loss": 1.0963,
+      "mean_token_accuracy": 0.709181547164917,
+      "num_tokens": 35366445.0,
+      "step": 135
+    },
+    {
+      "epoch": 3.5789473684210527,
+      "grad_norm": 0.7214124791993152,
+      "learning_rate": 3.1083562296259494e-06,
+      "loss": 1.1063,
+      "mean_token_accuracy": 0.7070298194885254,
+      "num_tokens": 35628589.0,
+      "step": 136
+    },
+    {
+      "epoch": 3.6052631578947367,
+      "grad_norm": 0.5288818064176912,
+      "learning_rate": 3.0387332884490806e-06,
+      "loss": 1.095,
+      "mean_token_accuracy": 0.7074266076087952,
+      "num_tokens": 35890733.0,
+      "step": 137
+    },
+    {
+      "epoch": 3.6315789473684212,
+      "grad_norm": 0.5389789995349765,
+      "learning_rate": 2.9699410659726076e-06,
+      "loss": 1.1101,
+      "mean_token_accuracy": 0.7031764984130859,
+      "num_tokens": 36152877.0,
+      "step": 138
+    },
+    {
+      "epoch": 3.6578947368421053,
+      "grad_norm": 0.5522512649782974,
+      "learning_rate": 2.9020027807227426e-06,
+      "loss": 1.2377,
+      "mean_token_accuracy": 0.6814453601837158,
+      "num_tokens": 36415021.0,
+      "step": 139
+    },
+    {
+      "epoch": 3.6842105263157894,
+      "grad_norm": 0.5402428472844846,
+      "learning_rate": 2.8349413630075907e-06,
+      "loss": 1.1458,
+      "mean_token_accuracy": 0.6977704167366028,
+      "num_tokens": 36677165.0,
+      "step": 140
+    },
+    {
+      "epoch": 3.7105263157894735,
+      "grad_norm": 0.5940564905132736,
+      "learning_rate": 2.7687794471777958e-06,
+      "loss": 1.1514,
+      "mean_token_accuracy": 0.6953821182250977,
+      "num_tokens": 36939309.0,
+      "step": 141
+    },
+    {
+      "epoch": 3.736842105263158,
+      "grad_norm": 0.5734295274738652,
+      "learning_rate": 2.70353936398706e-06,
+      "loss": 1.1403,
+      "mean_token_accuracy": 0.6989117860794067,
+      "num_tokens": 37193788.0,
+      "step": 142
+    },
+    {
+      "epoch": 3.763157894736842,
+      "grad_norm": 0.532059870308926,
+      "learning_rate": 2.639243133055145e-06,
+      "loss": 1.2474,
+      "mean_token_accuracy": 0.680209219455719,
+      "num_tokens": 37455932.0,
+      "step": 143
+    },
+    {
+      "epoch": 3.7894736842105265,
+      "grad_norm": 0.5510330430152282,
+      "learning_rate": 2.575912455435883e-06,
+      "loss": 1.2086,
+      "mean_token_accuracy": 0.688400387763977,
+      "num_tokens": 37718076.0,
+      "step": 144
+    },
+    {
+      "epoch": 3.8157894736842106,
+      "grad_norm": 0.5658132485612524,
+      "learning_rate": 2.513568706292705e-06,
+      "loss": 1.2624,
+      "mean_token_accuracy": 0.6760048866271973,
+      "num_tokens": 37980220.0,
+      "step": 145
+    },
+    {
+      "epoch": 3.8421052631578947,
+      "grad_norm": 0.5251146314121506,
+      "learning_rate": 2.4522329276841664e-06,
+      "loss": 1.2014,
+      "mean_token_accuracy": 0.6878548264503479,
+      "num_tokens": 38242364.0,
+      "step": 146
+    },
+    {
+      "epoch": 3.8684210526315788,
+      "grad_norm": 0.5357295896942552,
+      "learning_rate": 2.391925821461901e-06,
+      "loss": 1.1545,
+      "mean_token_accuracy": 0.6958895325660706,
+      "num_tokens": 38504508.0,
+      "step": 147
+    },
+    {
+      "epoch": 3.8947368421052633,
+      "grad_norm": 0.5705951115408219,
+      "learning_rate": 2.3326677422833992e-06,
+      "loss": 1.1662,
+      "mean_token_accuracy": 0.6927115321159363,
+      "num_tokens": 38766652.0,
+      "step": 148
+    },
+    {
+      "epoch": 3.9210526315789473,
+      "grad_norm": 0.5394182117597536,
+      "learning_rate": 2.2744786907419704e-06,
+      "loss": 1.1902,
+      "mean_token_accuracy": 0.6899531483650208,
+      "num_tokens": 39028796.0,
+      "step": 149
+    },
+    {
+      "epoch": 3.9473684210526314,
+      "grad_norm": 0.547923373437002,
+      "learning_rate": 2.2173783066162076e-06,
+      "loss": 1.1707,
+      "mean_token_accuracy": 0.6980451345443726,
+      "num_tokens": 39290940.0,
+      "step": 150
+    },
+    {
+      "epoch": 3.973684210526316,
+      "grad_norm": 0.5518777404038198,
+      "learning_rate": 2.1613858622412336e-06,
+      "loss": 1.1786,
+      "mean_token_accuracy": 0.6942414045333862,
+      "num_tokens": 39553084.0,
+      "step": 151
+    },
+    {
+      "epoch": 4.0,
+      "grad_norm": 0.5521786822039609,
+      "learning_rate": 2.1065202560039678e-06,
+      "loss": 1.1497,
+      "mean_token_accuracy": 0.699094295501709,
+      "num_tokens": 39815228.0,
+      "step": 152
+    },
+    {
+      "epoch": 4.026315789473684,
+      "grad_norm": 0.5822084229656587,
+      "learning_rate": 2.0528000059646e-06,
+      "loss": 1.1643,
+      "mean_token_accuracy": 0.6988691687583923,
+      "num_tokens": 40077372.0,
+      "step": 153
+    },
+    {
+      "epoch": 4.052631578947368,
+      "grad_norm": 0.5350433596377399,
+      "learning_rate": 2.0002432436064385e-06,
+      "loss": 1.1162,
+      "mean_token_accuracy": 0.7054846882820129,
+      "num_tokens": 40339516.0,
+      "step": 154
+    },
+    {
+      "epoch": 4.078947368421052,
+      "grad_norm": 0.5518812481108508,
+      "learning_rate": 1.94886770771623e-06,
+      "loss": 1.1493,
+      "mean_token_accuracy": 0.6981443166732788,
+      "num_tokens": 40601660.0,
+      "step": 155
+    },
+    {
+      "epoch": 4.105263157894737,
+      "grad_norm": 0.5480415544046374,
+      "learning_rate": 1.8986907383970228e-06,
+      "loss": 1.1315,
+      "mean_token_accuracy": 0.7005326151847839,
+      "num_tokens": 40863804.0,
+      "step": 156
+    },
+    {
+      "epoch": 4.131578947368421,
+      "grad_norm": 0.7656041762600009,
+      "learning_rate": 1.8497292712155907e-06,
+      "loss": 1.1118,
+      "mean_token_accuracy": 0.7078768014907837,
+      "num_tokens": 41125948.0,
+      "step": 157
+    },
+    {
+      "epoch": 4.157894736842105,
+      "grad_norm": 0.5497688927341485,
+      "learning_rate": 1.8019998314863974e-06,
+      "loss": 1.2162,
+      "mean_token_accuracy": 0.6840701699256897,
+      "num_tokens": 41388092.0,
+      "step": 158
+    },
+    {
+      "epoch": 4.184210526315789,
+      "grad_norm": 0.5634861823726028,
+      "learning_rate": 1.7555185286940252e-06,
+      "loss": 1.1001,
+      "mean_token_accuracy": 0.7082964777946472,
+      "num_tokens": 41650236.0,
+      "step": 159
+    },
+    {
+      "epoch": 4.2105263157894735,
+      "grad_norm": 0.5562593279549117,
+      "learning_rate": 1.710301051055955e-06,
+      "loss": 1.1233,
+      "mean_token_accuracy": 0.706102728843689,
+      "num_tokens": 41912380.0,
+      "step": 160
+    },
+    {
+      "epoch": 4.2368421052631575,
+      "grad_norm": 0.5649313028818561,
+      "learning_rate": 1.666362660227529e-06,
+      "loss": 1.1423,
+      "mean_token_accuracy": 0.6994147300720215,
+      "num_tokens": 42174524.0,
+      "step": 161
+    },
+    {
+      "epoch": 4.2631578947368425,
+      "grad_norm": 0.8616736787357053,
+      "learning_rate": 1.6237181861508889e-06,
+      "loss": 1.1505,
+      "mean_token_accuracy": 0.6955958008766174,
+      "num_tokens": 42436668.0,
+      "step": 162
+    },
+    {
+      "epoch": 4.2894736842105265,
+      "grad_norm": 0.8986161306924315,
+      "learning_rate": 1.5823820220496197e-06,
+      "loss": 1.1581,
+      "mean_token_accuracy": 0.6982167959213257,
+      "num_tokens": 42698812.0,
+      "step": 163
+    },
+    {
+      "epoch": 4.315789473684211,
+      "grad_norm": 0.5467975548359034,
+      "learning_rate": 1.5423681195707997e-06,
+      "loss": 1.1762,
+      "mean_token_accuracy": 0.6948594450950623,
+      "num_tokens": 42960956.0,
+      "step": 164
+    },
+    {
+      "epoch": 4.342105263157895,
+      "grad_norm": 0.5361701527040977,
+      "learning_rate": 1.5036899840760815e-06,
+      "loss": 1.0706,
+      "mean_token_accuracy": 0.7150683403015137,
+      "num_tokens": 43223100.0,
+      "step": 165
+    },
+    {
+      "epoch": 4.368421052631579,
+      "grad_norm": 0.6682056700117512,
+      "learning_rate": 1.4663606700834102e-06,
+      "loss": 1.1245,
+      "mean_token_accuracy": 0.7043439149856567,
+      "num_tokens": 43485244.0,
+      "step": 166
+    },
+    {
+      "epoch": 4.394736842105263,
+      "grad_norm": 0.5358865574721129,
+      "learning_rate": 1.4303927768609016e-06,
+      "loss": 1.1313,
+      "mean_token_accuracy": 0.7052977085113525,
+      "num_tokens": 43747388.0,
+      "step": 167
+    },
+    {
+      "epoch": 4.421052631578947,
+      "grad_norm": 0.525962123793606,
+      "learning_rate": 1.3957984441743832e-06,
+      "loss": 1.1384,
+      "mean_token_accuracy": 0.7020624876022339,
+      "num_tokens": 44009532.0,
+      "step": 168
+    },
+    {
+      "epoch": 4.447368421052632,
+      "grad_norm": 0.5796984218177971,
+      "learning_rate": 1.3625893481900155e-06,
+      "loss": 1.0882,
+      "mean_token_accuracy": 0.7103375792503357,
+      "num_tokens": 44271676.0,
+      "step": 169
+    },
+    {
+      "epoch": 4.473684210526316,
+      "grad_norm": 0.5393374011777778,
+      "learning_rate": 1.3307766975333922e-06,
+      "loss": 1.0466,
+      "mean_token_accuracy": 0.7210162281990051,
+      "num_tokens": 44533820.0,
+      "step": 170
+    },
+    {
+      "epoch": 4.5,
+      "grad_norm": 0.5349090159384385,
+      "learning_rate": 1.3003712295064433e-06,
+      "loss": 1.1418,
+      "mean_token_accuracy": 0.6994032859802246,
+      "num_tokens": 44795964.0,
+      "step": 171
+    },
+    {
+      "epoch": 4.526315789473684,
+      "grad_norm": 0.5057581166787403,
+      "learning_rate": 1.2713832064634127e-06,
+      "loss": 1.0858,
+      "mean_token_accuracy": 0.7137597799301147,
+      "num_tokens": 45058108.0,
+      "step": 172
+    },
+    {
+      "epoch": 4.552631578947368,
+      "grad_norm": 0.7631542771888906,
+      "learning_rate": 1.2438224123471442e-06,
+      "loss": 1.1672,
+      "mean_token_accuracy": 0.6955614686012268,
+      "num_tokens": 45320252.0,
+      "step": 173
+    },
+    {
+      "epoch": 4.578947368421053,
+      "grad_norm": 0.5440066367064722,
+      "learning_rate": 1.2176981493868374e-06,
+      "loss": 1.139,
+      "mean_token_accuracy": 0.7027553915977478,
+      "num_tokens": 45574731.0,
+      "step": 174
+    },
+    {
+      "epoch": 4.605263157894737,
+      "grad_norm": 0.518246804782426,
+      "learning_rate": 1.1930192349583866e-06,
+      "loss": 1.1575,
+      "mean_token_accuracy": 0.6979191899299622,
+      "num_tokens": 45836875.0,
+      "step": 175
+    },
+    {
+      "epoch": 4.631578947368421,
+      "grad_norm": 0.5364733989201356,
+      "learning_rate": 1.1697939986083732e-06,
+      "loss": 1.0587,
+      "mean_token_accuracy": 0.7159839868545532,
+      "num_tokens": 46099019.0,
+      "step": 176
+    },
+    {
+      "epoch": 4.657894736842105,
+      "grad_norm": 0.5170966910701931,
+      "learning_rate": 1.1480302792427021e-06,
+      "loss": 1.0986,
+      "mean_token_accuracy": 0.709250271320343,
+      "num_tokens": 46361163.0,
+      "step": 177
+    },
+    {
+      "epoch": 4.684210526315789,
+      "grad_norm": 0.5157074509948011,
+      "learning_rate": 1.127735422480838e-06,
+      "loss": 1.0903,
+      "mean_token_accuracy": 0.7117949724197388,
+      "num_tokens": 46623307.0,
+      "step": 178
+    },
+    {
+      "epoch": 4.7105263157894735,
+      "grad_norm": 0.5766154191021017,
+      "learning_rate": 1.10891627817654e-06,
+      "loss": 1.0934,
+      "mean_token_accuracy": 0.709181547164917,
+      "num_tokens": 46885451.0,
+      "step": 179
+    },
+    {
+      "epoch": 4.7368421052631575,
+      "grad_norm": 0.5135486472483117,
+      "learning_rate": 1.0915791981059174e-06,
+      "loss": 1.1199,
+      "mean_token_accuracy": 0.7034855484962463,
+      "num_tokens": 47147595.0,
+      "step": 180
+    },
+    {
+      "epoch": 4.7631578947368425,
+      "grad_norm": 0.7547407302261836,
+      "learning_rate": 1.0757300338236027e-06,
+      "loss": 1.1165,
+      "mean_token_accuracy": 0.7076631188392639,
+      "num_tokens": 47409739.0,
+      "step": 181
+    },
+    {
+      "epoch": 4.7894736842105265,
+      "grad_norm": 0.5690959613109913,
+      "learning_rate": 1.0613741346877498e-06,
+      "loss": 1.1585,
+      "mean_token_accuracy": 0.6951303482055664,
+      "num_tokens": 47671883.0,
+      "step": 182
+    },
+    {
+      "epoch": 4.815789473684211,
+      "grad_norm": 0.5277151826124948,
+      "learning_rate": 1.0485163460545396e-06,
+      "loss": 1.1409,
+      "mean_token_accuracy": 0.6986250281333923,
+      "num_tokens": 47934027.0,
+      "step": 183
+    },
+    {
+      "epoch": 4.842105263157895,
+      "grad_norm": 0.5192139515319969,
+      "learning_rate": 1.03716100764279e-06,
+      "loss": 1.165,
+      "mean_token_accuracy": 0.6956835389137268,
+      "num_tokens": 48196171.0,
+      "step": 184
+    },
+    {
+      "epoch": 4.868421052631579,
+      "grad_norm": 0.539511614407237,
+      "learning_rate": 1.0273119520692274e-06,
+      "loss": 1.1842,
+      "mean_token_accuracy": 0.6949090361595154,
+      "num_tokens": 48458315.0,
+      "step": 185
+    },
+    {
+      "epoch": 4.894736842105263,
+      "grad_norm": 0.5375538776614278,
+      "learning_rate": 1.0189725035549165e-06,
+      "loss": 1.1754,
+      "mean_token_accuracy": 0.6964083909988403,
+      "num_tokens": 48720459.0,
+      "step": 186
+    },
+    {
+      "epoch": 4.921052631578947,
+      "grad_norm": 0.5066546944417655,
+      "learning_rate": 1.0121454768032772e-06,
+      "loss": 1.1248,
+      "mean_token_accuracy": 0.7042332887649536,
+      "num_tokens": 48982603.0,
+      "step": 187
+    },
+    {
+      "epoch": 4.947368421052632,
+      "grad_norm": 0.516984174228912,
+      "learning_rate": 1.0068331760500773e-06,
+      "loss": 1.1127,
+      "mean_token_accuracy": 0.705129861831665,
+      "num_tokens": 49244747.0,
+      "step": 188
+    },
+    {
+      "epoch": 4.973684210526316,
+      "grad_norm": 0.5078294598491031,
+      "learning_rate": 1.0030373942857136e-06,
+      "loss": 1.0498,
+      "mean_token_accuracy": 0.7175978422164917,
+      "num_tokens": 49506891.0,
+      "step": 189
+    },
+    {
+      "epoch": 5.0,
+      "grad_norm": 0.5042595032615071,
+      "learning_rate": 1.000759412650048e-06,
+      "loss": 1.1777,
+      "mean_token_accuracy": 0.694607675075531,
+      "num_tokens": 49769035.0,
+      "step": 190
+    },
+    {
+      "epoch": 5.0,
+      "step": 190,
+      "total_flos": 39782134579200.0,
+      "train_loss": 1.4404595375061036,
+      "train_runtime": 2942.5505,
+      "train_samples_per_second": 2.048,
+      "train_steps_per_second": 0.065
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 190,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 5,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 39782134579200.0,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training.log CHANGED Viewed

@@ -1296,3 +1296,4 @@ weight_decay=0.0,
   )
   (lm_head): Linear(in_features=2048, out_features=50304, bias=False)
 )

   )
   (lm_head): Linear(in_features=2048, out_features=50304, bias=False)
 )
+2025-08-14 08:33:33 - INFO - __main__ - *** Save model ***