Instructions to use razor5050/codex-qwen2-5-0-5b-unsloth-codex1m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use razor5050/codex-qwen2-5-0-5b-unsloth-codex1m with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="razor5050/codex-qwen2-5-0-5b-unsloth-codex1m")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("razor5050/codex-qwen2-5-0-5b-unsloth-codex1m", device_map="auto")

PEFT
How to use razor5050/codex-qwen2-5-0-5b-unsloth-codex1m with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use razor5050/codex-qwen2-5-0-5b-unsloth-codex1m with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "razor5050/codex-qwen2-5-0-5b-unsloth-codex1m"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "razor5050/codex-qwen2-5-0-5b-unsloth-codex1m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/razor5050/codex-qwen2-5-0-5b-unsloth-codex1m

SGLang

How to use razor5050/codex-qwen2-5-0-5b-unsloth-codex1m with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "razor5050/codex-qwen2-5-0-5b-unsloth-codex1m" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "razor5050/codex-qwen2-5-0-5b-unsloth-codex1m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "razor5050/codex-qwen2-5-0-5b-unsloth-codex1m" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "razor5050/codex-qwen2-5-0-5b-unsloth-codex1m",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Unsloth Desktop
Docker Model Runner
How to use razor5050/codex-qwen2-5-0-5b-unsloth-codex1m with Docker Model Runner:
```
docker model run hf.co/razor5050/codex-qwen2-5-0-5b-unsloth-codex1m
```

razor5050 commited on May 9

Commit

ec8c8da

verified ·

1 Parent(s): c33bca6

Upload under8000 checkpoint step 4202

Browse files

Files changed (12) hide show

.gitattributes +1 -0
checkpoints_under8000/checkpoint-4202/README.md +210 -0
checkpoints_under8000/checkpoint-4202/adapter_config.json +52 -0
checkpoints_under8000/checkpoint-4202/adapter_model.safetensors +3 -0
checkpoints_under8000/checkpoint-4202/chat_template.jinja +54 -0
checkpoints_under8000/checkpoint-4202/optimizer.pt +3 -0
checkpoints_under8000/checkpoint-4202/rng_state.pth +3 -0
checkpoints_under8000/checkpoint-4202/scheduler.pt +3 -0
checkpoints_under8000/checkpoint-4202/tokenizer.json +3 -0
checkpoints_under8000/checkpoint-4202/tokenizer_config.json +201 -0
checkpoints_under8000/checkpoint-4202/trainer_state.json +3006 -0
checkpoints_under8000/checkpoint-4202/training_args.bin +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoints_under8000/checkpoint-1457/tokenizer.json filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 checkpoints_under8000/checkpoint-1457/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+checkpoints_under8000/checkpoint-4202/tokenizer.json filter=lfs diff=lfs merge=lfs -text

checkpoints_under8000/checkpoint-4202/README.md ADDED Viewed

	@@ -0,0 +1,210 @@

+---
+base_model: unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.19.1

checkpoints_under8000/checkpoint-4202/adapter_config.json ADDED Viewed

	@@ -0,0 +1,52 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "Qwen2ForCausalLM",
+    "parent_library": "transformers.models.qwen2.modeling_qwen2",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/qwen2.5-0.5b-instruct-unsloth-bnb-4bit",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 32,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "v_proj",
+    "k_proj",
+    "q_proj",
+    "gate_proj",
+    "down_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

checkpoints_under8000/checkpoint-4202/adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4cb522e394135e711619aa0a110ad472042c72f5d40dd90e85e9f24742f39756
+size 70430032

checkpoints_under8000/checkpoint-4202/chat_template.jinja ADDED Viewed

	@@ -0,0 +1,54 @@

+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

checkpoints_under8000/checkpoint-4202/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:01aa25b6b26786e7be36e80cf1e43ad2a72a0cc0fd9e579fdc998aff3a849b1c
+size 36140069

checkpoints_under8000/checkpoint-4202/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:89645f9f6a2a8823b334e0748e4f5ea683a27b3d02632f0a241397db55d82fad
+size 14645

checkpoints_under8000/checkpoint-4202/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5a58b5693fe031b359f2dd4e314add47663d6441ebc64d82727bccabf8963960
+size 1465

checkpoints_under8000/checkpoint-4202/tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bd5948af71b4f56cf697f7580814c7ce8b80595ef985544efcacf716126a2e31
+size 11422356

checkpoints_under8000/checkpoint-4202/tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,201 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 32768,
+  "pad_token": "<|PAD_TOKEN|>",
+  "padding_side": "right",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<|PAD_TOKEN|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  }
+}

checkpoints_under8000/checkpoint-4202/trainer_state.json ADDED Viewed

	@@ -0,0 +1,3006 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.05628727675805647,
+  "eval_steps": 1000,
+  "global_step": 4202,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.00013395353821527004,
+      "grad_norm": 4.978700160980225,
+      "learning_rate": 8.035714285714286e-07,
+      "loss": 1.4198334693908692,
+      "step": 10
+    },
+    {
+      "epoch": 0.0002679070764305401,
+      "grad_norm": 3.940929412841797,
+      "learning_rate": 1.6964285714285713e-06,
+      "loss": 1.3955652236938476,
+      "step": 20
+    },
+    {
+      "epoch": 0.0004018606146458101,
+      "grad_norm": 2.915323495864868,
+      "learning_rate": 2.5892857142857148e-06,
+      "loss": 1.3398881912231446,
+      "step": 30
+    },
+    {
+      "epoch": 0.0005358141528610801,
+      "grad_norm": 2.512028932571411,
+      "learning_rate": 3.4821428571428576e-06,
+      "loss": 1.2956685066223144,
+      "step": 40
+    },
+    {
+      "epoch": 0.0006697676910763502,
+      "grad_norm": 2.561004161834717,
+      "learning_rate": 4.375e-06,
+      "loss": 1.2670017242431642,
+      "step": 50
+    },
+    {
+      "epoch": 0.0008037212292916202,
+      "grad_norm": 2.296826124191284,
+      "learning_rate": 5.267857142857143e-06,
+      "loss": 1.2354182243347167,
+      "step": 60
+    },
+    {
+      "epoch": 0.0009376747675068903,
+      "grad_norm": 2.3733971118927,
+      "learning_rate": 6.160714285714286e-06,
+      "loss": 1.1916924476623536,
+      "step": 70
+    },
+    {
+      "epoch": 0.0010716283057221603,
+      "grad_norm": 2.520475387573242,
+      "learning_rate": 7.0535714285714286e-06,
+      "loss": 1.1873969078063964,
+      "step": 80
+    },
+    {
+      "epoch": 0.0012055818439374302,
+      "grad_norm": 2.4990739822387695,
+      "learning_rate": 7.946428571428571e-06,
+      "loss": 1.1627039909362793,
+      "step": 90
+    },
+    {
+      "epoch": 0.0013395353821527003,
+      "grad_norm": 2.3932080268859863,
+      "learning_rate": 8.839285714285716e-06,
+      "loss": 1.1169233322143555,
+      "step": 100
+    },
+    {
+      "epoch": 0.0014734889203679704,
+      "grad_norm": 2.6543712615966797,
+      "learning_rate": 9.732142857142858e-06,
+      "loss": 1.1316115379333496,
+      "step": 110
+    },
+    {
+      "epoch": 0.0016074424585832403,
+      "grad_norm": 2.6352458000183105,
+      "learning_rate": 1.0625e-05,
+      "loss": 1.0918511390686034,
+      "step": 120
+    },
+    {
+      "epoch": 0.0017413959967985105,
+      "grad_norm": 2.4628708362579346,
+      "learning_rate": 1.1517857142857142e-05,
+      "loss": 1.1151651382446288,
+      "step": 130
+    },
+    {
+      "epoch": 0.0018753495350137806,
+      "grad_norm": 2.482074737548828,
+      "learning_rate": 1.2410714285714287e-05,
+      "loss": 1.094371223449707,
+      "step": 140
+    },
+    {
+      "epoch": 0.0020093030732290505,
+      "grad_norm": 2.431954860687256,
+      "learning_rate": 1.3303571428571429e-05,
+      "loss": 1.0961938858032227,
+      "step": 150
+    },
+    {
+      "epoch": 0.0021432566114443206,
+      "grad_norm": 2.8230552673339844,
+      "learning_rate": 1.4196428571428572e-05,
+      "loss": 1.083104705810547,
+      "step": 160
+    },
+    {
+      "epoch": 0.0022772101496595907,
+      "grad_norm": 2.721494436264038,
+      "learning_rate": 1.5089285714285714e-05,
+      "loss": 1.0713379859924317,
+      "step": 170
+    },
+    {
+      "epoch": 0.0024111636878748604,
+      "grad_norm": 2.3565499782562256,
+      "learning_rate": 1.5982142857142857e-05,
+      "loss": 1.099677276611328,
+      "step": 180
+    },
+    {
+      "epoch": 0.0025451172260901305,
+      "grad_norm": 2.345555543899536,
+      "learning_rate": 1.6875000000000004e-05,
+      "loss": 1.0562498092651367,
+      "step": 190
+    },
+    {
+      "epoch": 0.0026790707643054006,
+      "grad_norm": 2.381127119064331,
+      "learning_rate": 1.7767857142857143e-05,
+      "loss": 1.0502027511596679,
+      "step": 200
+    },
+    {
+      "epoch": 0.0028130243025206707,
+      "grad_norm": 2.4133875370025635,
+      "learning_rate": 1.8660714285714287e-05,
+      "loss": 1.0681496620178224,
+      "step": 210
+    },
+    {
+      "epoch": 0.002946977840735941,
+      "grad_norm": 2.5238466262817383,
+      "learning_rate": 1.955357142857143e-05,
+      "loss": 1.0440272331237792,
+      "step": 220
+    },
+    {
+      "epoch": 0.003080931378951211,
+      "grad_norm": 2.7593183517456055,
+      "learning_rate": 2.0446428571428573e-05,
+      "loss": 1.0714456558227539,
+      "step": 230
+    },
+    {
+      "epoch": 0.0032148849171664807,
+      "grad_norm": 2.5562100410461426,
+      "learning_rate": 2.1339285714285713e-05,
+      "loss": 1.0298255920410155,
+      "step": 240
+    },
+    {
+      "epoch": 0.003348838455381751,
+      "grad_norm": 2.6205103397369385,
+      "learning_rate": 2.2232142857142856e-05,
+      "loss": 1.050998592376709,
+      "step": 250
+    },
+    {
+      "epoch": 0.003482791993597021,
+      "grad_norm": 2.5965018272399902,
+      "learning_rate": 2.3125000000000003e-05,
+      "loss": 1.0283641815185547,
+      "step": 260
+    },
+    {
+      "epoch": 0.003616745531812291,
+      "grad_norm": 2.3729209899902344,
+      "learning_rate": 2.4017857142857146e-05,
+      "loss": 1.0502397537231445,
+      "step": 270
+    },
+    {
+      "epoch": 0.003750699070027561,
+      "grad_norm": 2.378077983856201,
+      "learning_rate": 2.4910714285714286e-05,
+      "loss": 1.0391980171203614,
+      "step": 280
+    },
+    {
+      "epoch": 0.003884652608242831,
+      "grad_norm": 2.4107515811920166,
+      "learning_rate": 2.5803571428571433e-05,
+      "loss": 1.0300152778625489,
+      "step": 290
+    },
+    {
+      "epoch": 0.004018606146458101,
+      "grad_norm": 2.4866559505462646,
+      "learning_rate": 2.6696428571428573e-05,
+      "loss": 1.0498000144958497,
+      "step": 300
+    },
+    {
+      "epoch": 0.004152559684673371,
+      "grad_norm": 2.3568379878997803,
+      "learning_rate": 2.7589285714285716e-05,
+      "loss": 1.0401558876037598,
+      "step": 310
+    },
+    {
+      "epoch": 0.004286513222888641,
+      "grad_norm": 2.456519603729248,
+      "learning_rate": 2.8482142857142856e-05,
+      "loss": 1.0105724334716797,
+      "step": 320
+    },
+    {
+      "epoch": 0.004420466761103911,
+      "grad_norm": 2.8484723567962646,
+      "learning_rate": 2.9375000000000003e-05,
+      "loss": 1.0271486282348632,
+      "step": 330
+    },
+    {
+      "epoch": 0.004554420299319181,
+      "grad_norm": 2.829651117324829,
+      "learning_rate": 3.0267857142857142e-05,
+      "loss": 1.0125882148742675,
+      "step": 340
+    },
+    {
+      "epoch": 0.0046883738375344515,
+      "grad_norm": 1.9819118976593018,
+      "learning_rate": 3.116071428571429e-05,
+      "loss": 1.0134092330932618,
+      "step": 350
+    },
+    {
+      "epoch": 0.004822327375749721,
+      "grad_norm": 2.5545337200164795,
+      "learning_rate": 3.205357142857143e-05,
+      "loss": 1.006214714050293,
+      "step": 360
+    },
+    {
+      "epoch": 0.004956280913964991,
+      "grad_norm": 2.3971850872039795,
+      "learning_rate": 3.2946428571428576e-05,
+      "loss": 1.014259147644043,
+      "step": 370
+    },
+    {
+      "epoch": 0.005090234452180261,
+      "grad_norm": 2.188464641571045,
+      "learning_rate": 3.383928571428572e-05,
+      "loss": 1.0164517402648925,
+      "step": 380
+    },
+    {
+      "epoch": 0.005224187990395531,
+      "grad_norm": 2.607635021209717,
+      "learning_rate": 3.4732142857142855e-05,
+      "loss": 1.0006816864013672,
+      "step": 390
+    },
+    {
+      "epoch": 0.005358141528610801,
+      "grad_norm": 1.9828215837478638,
+      "learning_rate": 3.5625000000000005e-05,
+      "loss": 0.994498348236084,
+      "step": 400
+    },
+    {
+      "epoch": 0.005492095066826071,
+      "grad_norm": 2.0254552364349365,
+      "learning_rate": 3.651785714285714e-05,
+      "loss": 1.0143715858459472,
+      "step": 410
+    },
+    {
+      "epoch": 0.0056260486050413415,
+      "grad_norm": 2.088826894760132,
+      "learning_rate": 3.7410714285714285e-05,
+      "loss": 1.003225040435791,
+      "step": 420
+    },
+    {
+      "epoch": 0.005760002143256612,
+      "grad_norm": 2.4125075340270996,
+      "learning_rate": 3.8303571428571435e-05,
+      "loss": 0.9795364379882813,
+      "step": 430
+    },
+    {
+      "epoch": 0.005893955681471882,
+      "grad_norm": 2.0874435901641846,
+      "learning_rate": 3.919642857142857e-05,
+      "loss": 0.9827790260314941,
+      "step": 440
+    },
+    {
+      "epoch": 0.006027909219687152,
+      "grad_norm": 1.7993626594543457,
+      "learning_rate": 4.0089285714285715e-05,
+      "loss": 0.992791748046875,
+      "step": 450
+    },
+    {
+      "epoch": 0.006161862757902422,
+      "grad_norm": 2.1616885662078857,
+      "learning_rate": 4.098214285714286e-05,
+      "loss": 0.9879550933837891,
+      "step": 460
+    },
+    {
+      "epoch": 0.006295816296117691,
+      "grad_norm": 2.0187010765075684,
+      "learning_rate": 4.1875e-05,
+      "loss": 0.9812018394470214,
+      "step": 470
+    },
+    {
+      "epoch": 0.006429769834332961,
+      "grad_norm": 2.0268852710723877,
+      "learning_rate": 4.2767857142857145e-05,
+      "loss": 1.017441749572754,
+      "step": 480
+    },
+    {
+      "epoch": 0.0065637233725482315,
+      "grad_norm": 2.1466526985168457,
+      "learning_rate": 4.366071428571429e-05,
+      "loss": 0.9769362449645996,
+      "step": 490
+    },
+    {
+      "epoch": 0.006697676910763502,
+      "grad_norm": 2.0106499195098877,
+      "learning_rate": 4.455357142857143e-05,
+      "loss": 0.9902260780334473,
+      "step": 500
+    },
+    {
+      "epoch": 0.006831630448978772,
+      "grad_norm": 1.843783974647522,
+      "learning_rate": 4.5446428571428574e-05,
+      "loss": 1.0007816314697267,
+      "step": 510
+    },
+    {
+      "epoch": 0.006965583987194042,
+      "grad_norm": 1.8174176216125488,
+      "learning_rate": 4.633928571428572e-05,
+      "loss": 0.9792203903198242,
+      "step": 520
+    },
+    {
+      "epoch": 0.007099537525409312,
+      "grad_norm": 1.8151051998138428,
+      "learning_rate": 4.723214285714286e-05,
+      "loss": 0.992054557800293,
+      "step": 530
+    },
+    {
+      "epoch": 0.007233491063624582,
+      "grad_norm": 1.7471234798431396,
+      "learning_rate": 4.8125000000000004e-05,
+      "loss": 0.9979231834411622,
+      "step": 540
+    },
+    {
+      "epoch": 0.007367444601839852,
+      "grad_norm": 1.7692842483520508,
+      "learning_rate": 4.901785714285714e-05,
+      "loss": 0.9751591682434082,
+      "step": 550
+    },
+    {
+      "epoch": 0.007501398140055122,
+      "grad_norm": 1.6787338256835938,
+      "learning_rate": 4.991071428571429e-05,
+      "loss": 0.9884813308715821,
+      "step": 560
+    },
+    {
+      "epoch": 0.0076353516782703915,
+      "grad_norm": 1.7012407779693604,
+      "learning_rate": 5.080357142857143e-05,
+      "loss": 0.9855763435363769,
+      "step": 570
+    },
+    {
+      "epoch": 0.007769305216485662,
+      "grad_norm": 1.915123701095581,
+      "learning_rate": 5.169642857142858e-05,
+      "loss": 0.9511891365051269,
+      "step": 580
+    },
+    {
+      "epoch": 0.007903258754700932,
+      "grad_norm": 1.833243727684021,
+      "learning_rate": 5.258928571428572e-05,
+      "loss": 0.9984745025634766,
+      "step": 590
+    },
+    {
+      "epoch": 0.008037212292916202,
+      "grad_norm": 1.450955867767334,
+      "learning_rate": 5.348214285714286e-05,
+      "loss": 0.9740289688110352,
+      "step": 600
+    },
+    {
+      "epoch": 0.008171165831131472,
+      "grad_norm": 1.7209330797195435,
+      "learning_rate": 5.4375e-05,
+      "loss": 0.9547987937927246,
+      "step": 610
+    },
+    {
+      "epoch": 0.008305119369346742,
+      "grad_norm": 1.7298052310943604,
+      "learning_rate": 5.526785714285715e-05,
+      "loss": 0.9855975151062012,
+      "step": 620
+    },
+    {
+      "epoch": 0.008439072907562012,
+      "grad_norm": 1.6286202669143677,
+      "learning_rate": 5.616071428571429e-05,
+      "loss": 0.9657292366027832,
+      "step": 630
+    },
+    {
+      "epoch": 0.008573026445777282,
+      "grad_norm": 2.2121224403381348,
+      "learning_rate": 5.705357142857143e-05,
+      "loss": 0.9826187133789063,
+      "step": 640
+    },
+    {
+      "epoch": 0.008706979983992552,
+      "grad_norm": 1.4359034299850464,
+      "learning_rate": 5.794642857142858e-05,
+      "loss": 0.9880683898925782,
+      "step": 650
+    },
+    {
+      "epoch": 0.008840933522207823,
+      "grad_norm": 1.8250757455825806,
+      "learning_rate": 5.883928571428572e-05,
+      "loss": 0.9728729248046875,
+      "step": 660
+    },
+    {
+      "epoch": 0.008974887060423093,
+      "grad_norm": 1.6160730123519897,
+      "learning_rate": 5.973214285714286e-05,
+      "loss": 0.9673951148986817,
+      "step": 670
+    },
+    {
+      "epoch": 0.009108840598638363,
+      "grad_norm": 1.623353362083435,
+      "learning_rate": 6.0624999999999996e-05,
+      "loss": 0.9910868644714356,
+      "step": 680
+    },
+    {
+      "epoch": 0.009242794136853633,
+      "grad_norm": 1.7099947929382324,
+      "learning_rate": 6.151785714285715e-05,
+      "loss": 0.9758204460144043,
+      "step": 690
+    },
+    {
+      "epoch": 0.009376747675068903,
+      "grad_norm": 1.6103285551071167,
+      "learning_rate": 6.241071428571428e-05,
+      "loss": 0.9683603286743164,
+      "step": 700
+    },
+    {
+      "epoch": 0.009510701213284173,
+      "grad_norm": 1.703170895576477,
+      "learning_rate": 6.330357142857143e-05,
+      "loss": 0.959897232055664,
+      "step": 710
+    },
+    {
+      "epoch": 0.009644654751499442,
+      "grad_norm": 1.662630319595337,
+      "learning_rate": 6.419642857142858e-05,
+      "loss": 0.9767607688903809,
+      "step": 720
+    },
+    {
+      "epoch": 0.009778608289714712,
+      "grad_norm": 1.5833293199539185,
+      "learning_rate": 6.508928571428571e-05,
+      "loss": 0.9608363151550293,
+      "step": 730
+    },
+    {
+      "epoch": 0.009912561827929982,
+      "grad_norm": 1.7040482759475708,
+      "learning_rate": 6.598214285714286e-05,
+      "loss": 0.9693010330200196,
+      "step": 740
+    },
+    {
+      "epoch": 0.010046515366145252,
+      "grad_norm": 1.505820393562317,
+      "learning_rate": 6.6875e-05,
+      "loss": 0.9575628280639649,
+      "step": 750
+    },
+    {
+      "epoch": 0.010180468904360522,
+      "grad_norm": 1.812294602394104,
+      "learning_rate": 6.776785714285716e-05,
+      "loss": 0.9544397354125976,
+      "step": 760
+    },
+    {
+      "epoch": 0.010314422442575792,
+      "grad_norm": 1.6276874542236328,
+      "learning_rate": 6.866071428571429e-05,
+      "loss": 0.9317039489746094,
+      "step": 770
+    },
+    {
+      "epoch": 0.010448375980791062,
+      "grad_norm": 1.6243042945861816,
+      "learning_rate": 6.955357142857143e-05,
+      "loss": 0.9851680755615234,
+      "step": 780
+    },
+    {
+      "epoch": 0.010582329519006332,
+      "grad_norm": 1.5983399152755737,
+      "learning_rate": 7.044642857142859e-05,
+      "loss": 0.9748747825622559,
+      "step": 790
+    },
+    {
+      "epoch": 0.010716283057221603,
+      "grad_norm": 1.7678436040878296,
+      "learning_rate": 7.133928571428572e-05,
+      "loss": 0.9590046882629395,
+      "step": 800
+    },
+    {
+      "epoch": 0.010850236595436873,
+      "grad_norm": 1.5059410333633423,
+      "learning_rate": 7.223214285714286e-05,
+      "loss": 0.9823746681213379,
+      "step": 810
+    },
+    {
+      "epoch": 0.010984190133652143,
+      "grad_norm": 1.4575616121292114,
+      "learning_rate": 7.3125e-05,
+      "loss": 0.9733425140380859,
+      "step": 820
+    },
+    {
+      "epoch": 0.011118143671867413,
+      "grad_norm": 1.6164171695709229,
+      "learning_rate": 7.401785714285715e-05,
+      "loss": 0.927185344696045,
+      "step": 830
+    },
+    {
+      "epoch": 0.011252097210082683,
+      "grad_norm": 1.512903094291687,
+      "learning_rate": 7.491071428571429e-05,
+      "loss": 0.9673010826110839,
+      "step": 840
+    },
+    {
+      "epoch": 0.011386050748297953,
+      "grad_norm": 1.4509457349777222,
+      "learning_rate": 7.580357142857143e-05,
+      "loss": 0.9444644927978516,
+      "step": 850
+    },
+    {
+      "epoch": 0.011520004286513223,
+      "grad_norm": 1.3291261196136475,
+      "learning_rate": 7.669642857142858e-05,
+      "loss": 0.9484622001647949,
+      "step": 860
+    },
+    {
+      "epoch": 0.011653957824728493,
+      "grad_norm": 1.6370415687561035,
+      "learning_rate": 7.758928571428572e-05,
+      "loss": 0.9409455299377442,
+      "step": 870
+    },
+    {
+      "epoch": 0.011787911362943763,
+      "grad_norm": 1.5580850839614868,
+      "learning_rate": 7.848214285714286e-05,
+      "loss": 0.9556178092956543,
+      "step": 880
+    },
+    {
+      "epoch": 0.011921864901159034,
+      "grad_norm": 1.4624580144882202,
+      "learning_rate": 7.9375e-05,
+      "loss": 0.9679806709289551,
+      "step": 890
+    },
+    {
+      "epoch": 0.012055818439374304,
+      "grad_norm": 1.5962079763412476,
+      "learning_rate": 8.026785714285715e-05,
+      "loss": 0.9726933479309082,
+      "step": 900
+    },
+    {
+      "epoch": 0.012189771977589574,
+      "grad_norm": 1.3573912382125854,
+      "learning_rate": 8.116071428571429e-05,
+      "loss": 0.9682340621948242,
+      "step": 910
+    },
+    {
+      "epoch": 0.012323725515804844,
+      "grad_norm": 1.5596022605895996,
+      "learning_rate": 8.205357142857143e-05,
+      "loss": 0.9629892349243164,
+      "step": 920
+    },
+    {
+      "epoch": 0.012457679054020112,
+      "grad_norm": 1.730648398399353,
+      "learning_rate": 8.294642857142858e-05,
+      "loss": 0.9566517829895019,
+      "step": 930
+    },
+    {
+      "epoch": 0.012591632592235382,
+      "grad_norm": 1.5507956743240356,
+      "learning_rate": 8.383928571428572e-05,
+      "loss": 0.9654373168945313,
+      "step": 940
+    },
+    {
+      "epoch": 0.012725586130450653,
+      "grad_norm": 1.4160267114639282,
+      "learning_rate": 8.473214285714286e-05,
+      "loss": 0.9531091690063477,
+      "step": 950
+    },
+    {
+      "epoch": 0.012859539668665923,
+      "grad_norm": 1.617100477218628,
+      "learning_rate": 8.5625e-05,
+      "loss": 0.9591686248779296,
+      "step": 960
+    },
+    {
+      "epoch": 0.012993493206881193,
+      "grad_norm": 1.296432375907898,
+      "learning_rate": 8.651785714285715e-05,
+      "loss": 0.971707534790039,
+      "step": 970
+    },
+    {
+      "epoch": 0.013127446745096463,
+      "grad_norm": 1.402993083000183,
+      "learning_rate": 8.74107142857143e-05,
+      "loss": 0.9584191322326661,
+      "step": 980
+    },
+    {
+      "epoch": 0.013261400283311733,
+      "grad_norm": 1.4196568727493286,
+      "learning_rate": 8.830357142857142e-05,
+      "loss": 0.9672185897827148,
+      "step": 990
+    },
+    {
+      "epoch": 0.013395353821527003,
+      "grad_norm": 1.3073877096176147,
+      "learning_rate": 8.919642857142858e-05,
+      "loss": 0.9563102722167969,
+      "step": 1000
+    },
+    {
+      "epoch": 0.013395353821527003,
+      "eval_loss": 0.9335520267486572,
+      "eval_runtime": 282.864,
+      "eval_samples_per_second": 10.652,
+      "eval_steps_per_second": 10.652,
+      "step": 1000
+    },
+    {
+      "epoch": 0.013529307359742273,
+      "grad_norm": 1.2876602411270142,
+      "learning_rate": 9.008928571428572e-05,
+      "loss": 0.9841785430908203,
+      "step": 1010
+    },
+    {
+      "epoch": 0.013663260897957543,
+      "grad_norm": 1.302293062210083,
+      "learning_rate": 9.098214285714285e-05,
+      "loss": 0.9624536514282227,
+      "step": 1020
+    },
+    {
+      "epoch": 0.013797214436172813,
+      "grad_norm": 1.4643065929412842,
+      "learning_rate": 9.1875e-05,
+      "loss": 0.9604171752929688,
+      "step": 1030
+    },
+    {
+      "epoch": 0.013931167974388084,
+      "grad_norm": 1.5588184595108032,
+      "learning_rate": 9.276785714285715e-05,
+      "loss": 0.9482678413391114,
+      "step": 1040
+    },
+    {
+      "epoch": 0.014065121512603354,
+      "grad_norm": 1.3061965703964233,
+      "learning_rate": 9.36607142857143e-05,
+      "loss": 0.9968178749084473,
+      "step": 1050
+    },
+    {
+      "epoch": 0.014199075050818624,
+      "grad_norm": 1.49628746509552,
+      "learning_rate": 9.455357142857143e-05,
+      "loss": 0.965756893157959,
+      "step": 1060
+    },
+    {
+      "epoch": 0.014333028589033894,
+      "grad_norm": 1.5427523851394653,
+      "learning_rate": 9.544642857142858e-05,
+      "loss": 0.9401347160339355,
+      "step": 1070
+    },
+    {
+      "epoch": 0.014466982127249164,
+      "grad_norm": 1.44222891330719,
+      "learning_rate": 9.633928571428573e-05,
+      "loss": 0.9700592994689942,
+      "step": 1080
+    },
+    {
+      "epoch": 0.014600935665464434,
+      "grad_norm": 1.2180535793304443,
+      "learning_rate": 9.723214285714286e-05,
+      "loss": 0.9597712516784668,
+      "step": 1090
+    },
+    {
+      "epoch": 0.014734889203679704,
+      "grad_norm": 1.4644688367843628,
+      "learning_rate": 9.8125e-05,
+      "loss": 0.9620055198669434,
+      "step": 1100
+    },
+    {
+      "epoch": 0.014868842741894974,
+      "grad_norm": 1.4942972660064697,
+      "learning_rate": 9.901785714285716e-05,
+      "loss": 0.9615335464477539,
+      "step": 1110
+    },
+    {
+      "epoch": 0.015002796280110245,
+      "grad_norm": 1.1068652868270874,
+      "learning_rate": 9.991071428571429e-05,
+      "loss": 0.9622197151184082,
+      "step": 1120
+    },
+    {
+      "epoch": 0.015136749818325513,
+      "grad_norm": 1.283275842666626,
+      "learning_rate": 0.00010080357142857144,
+      "loss": 0.9707681655883789,
+      "step": 1130
+    },
+    {
+      "epoch": 0.015270703356540783,
+      "grad_norm": 1.197508454322815,
+      "learning_rate": 0.00010169642857142859,
+      "loss": 0.9689389228820801,
+      "step": 1140
+    },
+    {
+      "epoch": 0.015404656894756053,
+      "grad_norm": 1.2500603199005127,
+      "learning_rate": 0.00010258928571428572,
+      "loss": 0.9651185035705566,
+      "step": 1150
+    },
+    {
+      "epoch": 0.015538610432971323,
+      "grad_norm": 1.6802270412445068,
+      "learning_rate": 0.00010348214285714286,
+      "loss": 0.9665432929992676,
+      "step": 1160
+    },
+    {
+      "epoch": 0.015672563971186593,
+      "grad_norm": 1.4018360376358032,
+      "learning_rate": 0.000104375,
+      "loss": 0.9392754554748535,
+      "step": 1170
+    },
+    {
+      "epoch": 0.015806517509401864,
+      "grad_norm": 1.5453063249588013,
+      "learning_rate": 0.00010526785714285715,
+      "loss": 0.9618069648742675,
+      "step": 1180
+    },
+    {
+      "epoch": 0.015940471047617134,
+      "grad_norm": 1.2306983470916748,
+      "learning_rate": 0.00010616071428571428,
+      "loss": 0.940494441986084,
+      "step": 1190
+    },
+    {
+      "epoch": 0.016074424585832404,
+      "grad_norm": 1.1923718452453613,
+      "learning_rate": 0.00010705357142857145,
+      "loss": 0.9696602821350098,
+      "step": 1200
+    },
+    {
+      "epoch": 0.016208378124047674,
+      "grad_norm": 1.3687347173690796,
+      "learning_rate": 0.00010794642857142858,
+      "loss": 0.9562197685241699,
+      "step": 1210
+    },
+    {
+      "epoch": 0.016342331662262944,
+      "grad_norm": 1.3242262601852417,
+      "learning_rate": 0.00010883928571428572,
+      "loss": 0.941429328918457,
+      "step": 1220
+    },
+    {
+      "epoch": 0.016476285200478214,
+      "grad_norm": 1.2945473194122314,
+      "learning_rate": 0.00010973214285714286,
+      "loss": 0.9632730484008789,
+      "step": 1230
+    },
+    {
+      "epoch": 0.016610238738693484,
+      "grad_norm": 1.1794791221618652,
+      "learning_rate": 0.000110625,
+      "loss": 0.9528386116027832,
+      "step": 1240
+    },
+    {
+      "epoch": 0.016744192276908754,
+      "grad_norm": 1.261305570602417,
+      "learning_rate": 0.00011151785714285714,
+      "loss": 0.945485782623291,
+      "step": 1250
+    },
+    {
+      "epoch": 0.016878145815124024,
+      "grad_norm": 1.310412883758545,
+      "learning_rate": 0.00011241071428571428,
+      "loss": 0.9500171661376953,
+      "step": 1260
+    },
+    {
+      "epoch": 0.017012099353339295,
+      "grad_norm": 1.4417651891708374,
+      "learning_rate": 0.00011330357142857145,
+      "loss": 0.965580940246582,
+      "step": 1270
+    },
+    {
+      "epoch": 0.017146052891554565,
+      "grad_norm": 1.2227723598480225,
+      "learning_rate": 0.00011419642857142858,
+      "loss": 0.9576460838317871,
+      "step": 1280
+    },
+    {
+      "epoch": 0.017280006429769835,
+      "grad_norm": 1.278008222579956,
+      "learning_rate": 0.00011508928571428572,
+      "loss": 0.9216486930847168,
+      "step": 1290
+    },
+    {
+      "epoch": 0.017413959967985105,
+      "grad_norm": 1.2406392097473145,
+      "learning_rate": 0.00011598214285714287,
+      "loss": 0.9882441520690918,
+      "step": 1300
+    },
+    {
+      "epoch": 0.017547913506200375,
+      "grad_norm": 1.2370669841766357,
+      "learning_rate": 0.000116875,
+      "loss": 0.9419057846069336,
+      "step": 1310
+    },
+    {
+      "epoch": 0.017681867044415645,
+      "grad_norm": 1.1850550174713135,
+      "learning_rate": 0.00011776785714285714,
+      "loss": 0.976494026184082,
+      "step": 1320
+    },
+    {
+      "epoch": 0.017815820582630915,
+      "grad_norm": 1.1672782897949219,
+      "learning_rate": 0.00011866071428571428,
+      "loss": 0.9599658012390136,
+      "step": 1330
+    },
+    {
+      "epoch": 0.017949774120846185,
+      "grad_norm": 1.2031351327896118,
+      "learning_rate": 0.00011955357142857144,
+      "loss": 0.9466784477233887,
+      "step": 1340
+    },
+    {
+      "epoch": 0.018083727659061456,
+      "grad_norm": 1.423079013824463,
+      "learning_rate": 0.00012044642857142858,
+      "loss": 0.9467257499694824,
+      "step": 1350
+    },
+    {
+      "epoch": 0.018217681197276726,
+      "grad_norm": 1.3132128715515137,
+      "learning_rate": 0.00012133928571428572,
+      "loss": 0.95529203414917,
+      "step": 1360
+    },
+    {
+      "epoch": 0.018351634735491996,
+      "grad_norm": 1.3859386444091797,
+      "learning_rate": 0.00012223214285714287,
+      "loss": 0.9389223098754883,
+      "step": 1370
+    },
+    {
+      "epoch": 0.018485588273707266,
+      "grad_norm": 1.1685659885406494,
+      "learning_rate": 0.000123125,
+      "loss": 0.94387845993042,
+      "step": 1380
+    },
+    {
+      "epoch": 0.018619541811922536,
+      "grad_norm": 1.2127257585525513,
+      "learning_rate": 0.00012401785714285715,
+      "loss": 0.918062400817871,
+      "step": 1390
+    },
+    {
+      "epoch": 0.018753495350137806,
+      "grad_norm": 1.1995147466659546,
+      "learning_rate": 0.00012491071428571428,
+      "loss": 0.9396192550659179,
+      "step": 1400
+    },
+    {
+      "epoch": 0.018887448888353076,
+      "grad_norm": 1.2880109548568726,
+      "learning_rate": 0.00012580357142857144,
+      "loss": 0.9649562835693359,
+      "step": 1410
+    },
+    {
+      "epoch": 0.019021402426568346,
+      "grad_norm": 1.2551807165145874,
+      "learning_rate": 0.00012669642857142857,
+      "loss": 0.9511116027832032,
+      "step": 1420
+    },
+    {
+      "epoch": 0.019155355964783616,
+      "grad_norm": 1.184615135192871,
+      "learning_rate": 0.00012758928571428573,
+      "loss": 0.9791507720947266,
+      "step": 1430
+    },
+    {
+      "epoch": 0.019289309502998883,
+      "grad_norm": 1.1787488460540771,
+      "learning_rate": 0.00012848214285714286,
+      "loss": 0.9635199546813965,
+      "step": 1440
+    },
+    {
+      "epoch": 0.019423263041214153,
+      "grad_norm": 1.0999921560287476,
+      "learning_rate": 0.00012937500000000001,
+      "loss": 0.9462879180908204,
+      "step": 1450
+    },
+    {
+      "epoch": 0.019557216579429423,
+      "grad_norm": 1.3516278266906738,
+      "learning_rate": 0.00013026785714285714,
+      "loss": 0.9511301040649414,
+      "step": 1460
+    },
+    {
+      "epoch": 0.019691170117644693,
+      "grad_norm": 1.2714283466339111,
+      "learning_rate": 0.00013116071428571427,
+      "loss": 0.934202003479004,
+      "step": 1470
+    },
+    {
+      "epoch": 0.019825123655859964,
+      "grad_norm": 1.1116617918014526,
+      "learning_rate": 0.00013205357142857143,
+      "loss": 0.9517221450805664,
+      "step": 1480
+    },
+    {
+      "epoch": 0.019959077194075234,
+      "grad_norm": 1.1716386079788208,
+      "learning_rate": 0.0001329464285714286,
+      "loss": 0.9329569816589356,
+      "step": 1490
+    },
+    {
+      "epoch": 0.020093030732290504,
+      "grad_norm": 1.1489286422729492,
+      "learning_rate": 0.00013383928571428572,
+      "loss": 0.9706562995910645,
+      "step": 1500
+    },
+    {
+      "epoch": 0.020226984270505774,
+      "grad_norm": 1.3109049797058105,
+      "learning_rate": 0.00013473214285714287,
+      "loss": 0.9514205932617188,
+      "step": 1510
+    },
+    {
+      "epoch": 0.020360937808721044,
+      "grad_norm": 1.218209147453308,
+      "learning_rate": 0.000135625,
+      "loss": 0.9647247314453125,
+      "step": 1520
+    },
+    {
+      "epoch": 0.020494891346936314,
+      "grad_norm": 1.1620960235595703,
+      "learning_rate": 0.00013651785714285713,
+      "loss": 0.9633931159973145,
+      "step": 1530
+    },
+    {
+      "epoch": 0.020628844885151584,
+      "grad_norm": 1.1143990755081177,
+      "learning_rate": 0.0001374107142857143,
+      "loss": 0.9660541534423828,
+      "step": 1540
+    },
+    {
+      "epoch": 0.020762798423366854,
+      "grad_norm": 1.12709641456604,
+      "learning_rate": 0.00013830357142857145,
+      "loss": 0.9445898056030273,
+      "step": 1550
+    },
+    {
+      "epoch": 0.020896751961582125,
+      "grad_norm": 1.2789790630340576,
+      "learning_rate": 0.00013919642857142858,
+      "loss": 0.9396877288818359,
+      "step": 1560
+    },
+    {
+      "epoch": 0.021030705499797395,
+      "grad_norm": 1.3755370378494263,
+      "learning_rate": 0.00014008928571428573,
+      "loss": 0.9562576293945313,
+      "step": 1570
+    },
+    {
+      "epoch": 0.021164659038012665,
+      "grad_norm": 1.106921911239624,
+      "learning_rate": 0.00014098214285714286,
+      "loss": 0.9208305358886719,
+      "step": 1580
+    },
+    {
+      "epoch": 0.021298612576227935,
+      "grad_norm": 1.2218865156173706,
+      "learning_rate": 0.000141875,
+      "loss": 0.923252010345459,
+      "step": 1590
+    },
+    {
+      "epoch": 0.021432566114443205,
+      "grad_norm": 1.1265087127685547,
+      "learning_rate": 0.00014276785714285715,
+      "loss": 0.9264353752136231,
+      "step": 1600
+    },
+    {
+      "epoch": 0.021566519652658475,
+      "grad_norm": 1.3815466165542603,
+      "learning_rate": 0.00014366071428571428,
+      "loss": 0.9643337249755859,
+      "step": 1610
+    },
+    {
+      "epoch": 0.021700473190873745,
+      "grad_norm": 1.1408675909042358,
+      "learning_rate": 0.00014455357142857144,
+      "loss": 0.9644880294799805,
+      "step": 1620
+    },
+    {
+      "epoch": 0.021834426729089015,
+      "grad_norm": 1.202513575553894,
+      "learning_rate": 0.0001454464285714286,
+      "loss": 0.9403377532958984,
+      "step": 1630
+    },
+    {
+      "epoch": 0.021968380267304286,
+      "grad_norm": 1.024915337562561,
+      "learning_rate": 0.00014633928571428572,
+      "loss": 0.9280579566955567,
+      "step": 1640
+    },
+    {
+      "epoch": 0.022102333805519556,
+      "grad_norm": 1.126615047454834,
+      "learning_rate": 0.00014723214285714285,
+      "loss": 0.9409577369689941,
+      "step": 1650
+    },
+    {
+      "epoch": 0.022236287343734826,
+      "grad_norm": 1.0569106340408325,
+      "learning_rate": 0.000148125,
+      "loss": 0.9590131759643554,
+      "step": 1660
+    },
+    {
+      "epoch": 0.022370240881950096,
+      "grad_norm": 1.1735954284667969,
+      "learning_rate": 0.00014901785714285714,
+      "loss": 0.9538652420043945,
+      "step": 1670
+    },
+    {
+      "epoch": 0.022504194420165366,
+      "grad_norm": 1.0429877042770386,
+      "learning_rate": 0.0001499107142857143,
+      "loss": 0.9786016464233398,
+      "step": 1680
+    },
+    {
+      "epoch": 0.022638147958380636,
+      "grad_norm": 1.189475417137146,
+      "learning_rate": 0.00015080357142857145,
+      "loss": 0.955595874786377,
+      "step": 1690
+    },
+    {
+      "epoch": 0.022772101496595906,
+      "grad_norm": 1.0221614837646484,
+      "learning_rate": 0.00015169642857142858,
+      "loss": 0.9822449684143066,
+      "step": 1700
+    },
+    {
+      "epoch": 0.022906055034811176,
+      "grad_norm": 1.1084915399551392,
+      "learning_rate": 0.0001525892857142857,
+      "loss": 0.9466286659240722,
+      "step": 1710
+    },
+    {
+      "epoch": 0.023040008573026446,
+      "grad_norm": 1.1933523416519165,
+      "learning_rate": 0.00015348214285714287,
+      "loss": 0.9203065872192383,
+      "step": 1720
+    },
+    {
+      "epoch": 0.023173962111241717,
+      "grad_norm": 1.0246378183364868,
+      "learning_rate": 0.000154375,
+      "loss": 0.947907829284668,
+      "step": 1730
+    },
+    {
+      "epoch": 0.023307915649456987,
+      "grad_norm": 1.2799917459487915,
+      "learning_rate": 0.00015526785714285716,
+      "loss": 0.9746535301208497,
+      "step": 1740
+    },
+    {
+      "epoch": 0.023441869187672257,
+      "grad_norm": 0.9834460616111755,
+      "learning_rate": 0.00015616071428571428,
+      "loss": 0.9352348327636719,
+      "step": 1750
+    },
+    {
+      "epoch": 0.023575822725887527,
+      "grad_norm": 1.1289196014404297,
+      "learning_rate": 0.00015705357142857144,
+      "loss": 0.9422959327697754,
+      "step": 1760
+    },
+    {
+      "epoch": 0.023709776264102797,
+      "grad_norm": 1.1601067781448364,
+      "learning_rate": 0.00015794642857142857,
+      "loss": 0.9540572166442871,
+      "step": 1770
+    },
+    {
+      "epoch": 0.023843729802318067,
+      "grad_norm": 1.0564992427825928,
+      "learning_rate": 0.00015883928571428573,
+      "loss": 0.9810628890991211,
+      "step": 1780
+    },
+    {
+      "epoch": 0.023977683340533337,
+      "grad_norm": 1.4195152521133423,
+      "learning_rate": 0.00015973214285714286,
+      "loss": 0.9447280883789062,
+      "step": 1790
+    },
+    {
+      "epoch": 0.024111636878748607,
+      "grad_norm": 1.1684353351593018,
+      "learning_rate": 0.00016062500000000001,
+      "loss": 0.9330779075622558,
+      "step": 1800
+    },
+    {
+      "epoch": 0.024245590416963878,
+      "grad_norm": 1.0490233898162842,
+      "learning_rate": 0.00016151785714285714,
+      "loss": 0.9627155303955078,
+      "step": 1810
+    },
+    {
+      "epoch": 0.024379543955179148,
+      "grad_norm": 1.1879408359527588,
+      "learning_rate": 0.00016241071428571427,
+      "loss": 0.9369135856628418,
+      "step": 1820
+    },
+    {
+      "epoch": 0.024513497493394418,
+      "grad_norm": 1.038156509399414,
+      "learning_rate": 0.00016330357142857146,
+      "loss": 0.9641859054565429,
+      "step": 1830
+    },
+    {
+      "epoch": 0.024647451031609688,
+      "grad_norm": 1.281324028968811,
+      "learning_rate": 0.0001641964285714286,
+      "loss": 0.9745452880859375,
+      "step": 1840
+    },
+    {
+      "epoch": 0.024781404569824955,
+      "grad_norm": 1.0497291088104248,
+      "learning_rate": 0.00016508928571428572,
+      "loss": 0.9583146095275878,
+      "step": 1850
+    },
+    {
+      "epoch": 0.024915358108040225,
+      "grad_norm": 1.2138155698776245,
+      "learning_rate": 0.00016598214285714287,
+      "loss": 0.94415283203125,
+      "step": 1860
+    },
+    {
+      "epoch": 0.025049311646255495,
+      "grad_norm": 1.0328693389892578,
+      "learning_rate": 0.000166875,
+      "loss": 0.9546913146972656,
+      "step": 1870
+    },
+    {
+      "epoch": 0.025183265184470765,
+      "grad_norm": 0.9754835367202759,
+      "learning_rate": 0.00016776785714285713,
+      "loss": 0.941204833984375,
+      "step": 1880
+    },
+    {
+      "epoch": 0.025317218722686035,
+      "grad_norm": 1.1597093343734741,
+      "learning_rate": 0.0001686607142857143,
+      "loss": 0.9581302642822266,
+      "step": 1890
+    },
+    {
+      "epoch": 0.025451172260901305,
+      "grad_norm": 0.9526762366294861,
+      "learning_rate": 0.00016955357142857145,
+      "loss": 0.9658097267150879,
+      "step": 1900
+    },
+    {
+      "epoch": 0.025585125799116575,
+      "grad_norm": 1.0967228412628174,
+      "learning_rate": 0.00017044642857142858,
+      "loss": 0.9606739997863769,
+      "step": 1910
+    },
+    {
+      "epoch": 0.025719079337331845,
+      "grad_norm": 1.2501522302627563,
+      "learning_rate": 0.00017133928571428573,
+      "loss": 0.9462210655212402,
+      "step": 1920
+    },
+    {
+      "epoch": 0.025853032875547115,
+      "grad_norm": 1.2555738687515259,
+      "learning_rate": 0.00017223214285714286,
+      "loss": 0.978199577331543,
+      "step": 1930
+    },
+    {
+      "epoch": 0.025986986413762386,
+      "grad_norm": 1.2749037742614746,
+      "learning_rate": 0.000173125,
+      "loss": 0.9561247825622559,
+      "step": 1940
+    },
+    {
+      "epoch": 0.026120939951977656,
+      "grad_norm": 1.1155575513839722,
+      "learning_rate": 0.00017401785714285715,
+      "loss": 0.9505109786987305,
+      "step": 1950
+    },
+    {
+      "epoch": 0.026254893490192926,
+      "grad_norm": 1.1074986457824707,
+      "learning_rate": 0.00017491071428571428,
+      "loss": 0.964531135559082,
+      "step": 1960
+    },
+    {
+      "epoch": 0.026388847028408196,
+      "grad_norm": 1.0668226480484009,
+      "learning_rate": 0.00017580357142857144,
+      "loss": 0.9583681106567383,
+      "step": 1970
+    },
+    {
+      "epoch": 0.026522800566623466,
+      "grad_norm": 1.0635207891464233,
+      "learning_rate": 0.0001766964285714286,
+      "loss": 0.9607739448547363,
+      "step": 1980
+    },
+    {
+      "epoch": 0.026656754104838736,
+      "grad_norm": 1.1774014234542847,
+      "learning_rate": 0.00017758928571428572,
+      "loss": 0.938353443145752,
+      "step": 1990
+    },
+    {
+      "epoch": 0.026790707643054006,
+      "grad_norm": 0.9912426471710205,
+      "learning_rate": 0.00017848214285714285,
+      "loss": 0.9563817977905273,
+      "step": 2000
+    },
+    {
+      "epoch": 0.026790707643054006,
+      "eval_loss": 0.9294622540473938,
+      "eval_runtime": 282.9209,
+      "eval_samples_per_second": 10.65,
+      "eval_steps_per_second": 10.65,
+      "step": 2000
+    },
+    {
+      "epoch": 0.026924661181269276,
+      "grad_norm": 1.1386301517486572,
+      "learning_rate": 0.000179375,
+      "loss": 0.9594663619995117,
+      "step": 2010
+    },
+    {
+      "epoch": 0.027058614719484547,
+      "grad_norm": 1.0435603857040405,
+      "learning_rate": 0.00018026785714285714,
+      "loss": 0.9453421592712402,
+      "step": 2020
+    },
+    {
+      "epoch": 0.027192568257699817,
+      "grad_norm": 1.1574785709381104,
+      "learning_rate": 0.0001811607142857143,
+      "loss": 0.9297013282775879,
+      "step": 2030
+    },
+    {
+      "epoch": 0.027326521795915087,
+      "grad_norm": 1.1167242527008057,
+      "learning_rate": 0.00018205357142857145,
+      "loss": 0.9530548095703125,
+      "step": 2040
+    },
+    {
+      "epoch": 0.027460475334130357,
+      "grad_norm": 1.0436396598815918,
+      "learning_rate": 0.00018294642857142858,
+      "loss": 0.9412404060363769,
+      "step": 2050
+    },
+    {
+      "epoch": 0.027594428872345627,
+      "grad_norm": 1.184274435043335,
+      "learning_rate": 0.0001838392857142857,
+      "loss": 0.9605161666870117,
+      "step": 2060
+    },
+    {
+      "epoch": 0.027728382410560897,
+      "grad_norm": 0.9393510222434998,
+      "learning_rate": 0.00018473214285714287,
+      "loss": 0.9509821891784668,
+      "step": 2070
+    },
+    {
+      "epoch": 0.027862335948776167,
+      "grad_norm": 1.0484318733215332,
+      "learning_rate": 0.000185625,
+      "loss": 0.9221634864807129,
+      "step": 2080
+    },
+    {
+      "epoch": 0.027996289486991437,
+      "grad_norm": 0.9934367537498474,
+      "learning_rate": 0.00018651785714285716,
+      "loss": 0.9437132835388183,
+      "step": 2090
+    },
+    {
+      "epoch": 0.028130243025206707,
+      "grad_norm": 0.9708594679832458,
+      "learning_rate": 0.00018741071428571429,
+      "loss": 0.9631563186645508,
+      "step": 2100
+    },
+    {
+      "epoch": 0.028264196563421978,
+      "grad_norm": 1.0960266590118408,
+      "learning_rate": 0.00018830357142857144,
+      "loss": 0.9347660064697265,
+      "step": 2110
+    },
+    {
+      "epoch": 0.028398150101637248,
+      "grad_norm": 1.4339450597763062,
+      "learning_rate": 0.00018919642857142857,
+      "loss": 0.9685755729675293,
+      "step": 2120
+    },
+    {
+      "epoch": 0.028532103639852518,
+      "grad_norm": 0.9974057078361511,
+      "learning_rate": 0.00019008928571428573,
+      "loss": 0.9545693397521973,
+      "step": 2130
+    },
+    {
+      "epoch": 0.028666057178067788,
+      "grad_norm": 1.0973010063171387,
+      "learning_rate": 0.00019098214285714286,
+      "loss": 0.9272083282470703,
+      "step": 2140
+    },
+    {
+      "epoch": 0.028800010716283058,
+      "grad_norm": 0.9838769435882568,
+      "learning_rate": 0.00019187500000000002,
+      "loss": 0.9372439384460449,
+      "step": 2150
+    },
+    {
+      "epoch": 0.028933964254498328,
+      "grad_norm": 1.0080406665802002,
+      "learning_rate": 0.00019276785714285715,
+      "loss": 0.9752328872680665,
+      "step": 2160
+    },
+    {
+      "epoch": 0.0290679177927136,
+      "grad_norm": 0.9965051412582397,
+      "learning_rate": 0.00019366071428571427,
+      "loss": 0.9454973220825196,
+      "step": 2170
+    },
+    {
+      "epoch": 0.02920187133092887,
+      "grad_norm": 0.9947363138198853,
+      "learning_rate": 0.00019455357142857146,
+      "loss": 0.9513129234313965,
+      "step": 2180
+    },
+    {
+      "epoch": 0.02933582486914414,
+      "grad_norm": 1.1437948942184448,
+      "learning_rate": 0.0001954464285714286,
+      "loss": 0.9673953056335449,
+      "step": 2190
+    },
+    {
+      "epoch": 0.02946977840735941,
+      "grad_norm": 1.0892202854156494,
+      "learning_rate": 0.00019633928571428572,
+      "loss": 0.9653889656066894,
+      "step": 2200
+    },
+    {
+      "epoch": 0.02960373194557468,
+      "grad_norm": 1.149424433708191,
+      "learning_rate": 0.00019723214285714287,
+      "loss": 0.9513349533081055,
+      "step": 2210
+    },
+    {
+      "epoch": 0.02973768548378995,
+      "grad_norm": 1.014336347579956,
+      "learning_rate": 0.000198125,
+      "loss": 0.9285200119018555,
+      "step": 2220
+    },
+    {
+      "epoch": 0.02987163902200522,
+      "grad_norm": 1.0589832067489624,
+      "learning_rate": 0.00019901785714285713,
+      "loss": 0.9304631233215332,
+      "step": 2230
+    },
+    {
+      "epoch": 0.03000559256022049,
+      "grad_norm": 0.9858236312866211,
+      "learning_rate": 0.0001999107142857143,
+      "loss": 0.9667643547058106,
+      "step": 2240
+    },
+    {
+      "epoch": 0.03013954609843576,
+      "grad_norm": 0.9708908796310425,
+      "learning_rate": 0.0001999999923770744,
+      "loss": 0.9756393432617188,
+      "step": 2250
+    },
+    {
+      "epoch": 0.030273499636651026,
+      "grad_norm": 1.029149055480957,
+      "learning_rate": 0.00019999996602622187,
+      "loss": 0.9660782814025879,
+      "step": 2260
+    },
+    {
+      "epoch": 0.030407453174866296,
+      "grad_norm": 0.9579015374183655,
+      "learning_rate": 0.00019999992085333718,
+      "loss": 0.9369187355041504,
+      "step": 2270
+    },
+    {
+      "epoch": 0.030541406713081566,
+      "grad_norm": 1.0455657243728638,
+      "learning_rate": 0.0001999998568584289,
+      "loss": 0.957420539855957,
+      "step": 2280
+    },
+    {
+      "epoch": 0.030675360251296836,
+      "grad_norm": 1.1843011379241943,
+      "learning_rate": 0.00019999977404150902,
+      "loss": 0.9495012283325195,
+      "step": 2290
+    },
+    {
+      "epoch": 0.030809313789512106,
+      "grad_norm": 1.0620977878570557,
+      "learning_rate": 0.00019999967240259315,
+      "loss": 0.9708341598510742,
+      "step": 2300
+    },
+    {
+      "epoch": 0.030943267327727376,
+      "grad_norm": 1.0837901830673218,
+      "learning_rate": 0.0001999995519417004,
+      "loss": 0.9840412139892578,
+      "step": 2310
+    },
+    {
+      "epoch": 0.031077220865942647,
+      "grad_norm": 0.9072780013084412,
+      "learning_rate": 0.00019999941265885348,
+      "loss": 0.9503616333007813,
+      "step": 2320
+    },
+    {
+      "epoch": 0.031211174404157917,
+      "grad_norm": 0.8991392850875854,
+      "learning_rate": 0.00019999925455407855,
+      "loss": 0.960198974609375,
+      "step": 2330
+    },
+    {
+      "epoch": 0.03134512794237319,
+      "grad_norm": 0.8700608015060425,
+      "learning_rate": 0.00019999907762740542,
+      "loss": 0.9340170860290528,
+      "step": 2340
+    },
+    {
+      "epoch": 0.03147908148058846,
+      "grad_norm": 1.0618027448654175,
+      "learning_rate": 0.00019999888187886735,
+      "loss": 0.9670058250427246,
+      "step": 2350
+    },
+    {
+      "epoch": 0.03161303501880373,
+      "grad_norm": 1.0053410530090332,
+      "learning_rate": 0.0001999986673085012,
+      "loss": 0.9664807319641113,
+      "step": 2360
+    },
+    {
+      "epoch": 0.031746988557019,
+      "grad_norm": 1.0876250267028809,
+      "learning_rate": 0.0001999984339163474,
+      "loss": 0.9429721832275391,
+      "step": 2370
+    },
+    {
+      "epoch": 0.03188094209523427,
+      "grad_norm": 1.1047455072402954,
+      "learning_rate": 0.00019999818170244983,
+      "loss": 0.9404646873474121,
+      "step": 2380
+    },
+    {
+      "epoch": 0.03201489563344954,
+      "grad_norm": 1.0188040733337402,
+      "learning_rate": 0.00019999791066685596,
+      "loss": 0.9436681747436524,
+      "step": 2390
+    },
+    {
+      "epoch": 0.03214884917166481,
+      "grad_norm": 0.9918760657310486,
+      "learning_rate": 0.0001999976208096168,
+      "loss": 0.9435869216918945,
+      "step": 2400
+    },
+    {
+      "epoch": 0.03228280270988008,
+      "grad_norm": 1.0088706016540527,
+      "learning_rate": 0.00019999731213078697,
+      "loss": 0.9398737907409668,
+      "step": 2410
+    },
+    {
+      "epoch": 0.03241675624809535,
+      "grad_norm": 0.9979799389839172,
+      "learning_rate": 0.00019999698463042448,
+      "loss": 0.9468537330627441,
+      "step": 2420
+    },
+    {
+      "epoch": 0.03255070978631062,
+      "grad_norm": 0.9645936489105225,
+      "learning_rate": 0.00019999663830859104,
+      "loss": 0.9193555831909179,
+      "step": 2430
+    },
+    {
+      "epoch": 0.03268466332452589,
+      "grad_norm": 0.9175757169723511,
+      "learning_rate": 0.00019999627316535178,
+      "loss": 0.9474472999572754,
+      "step": 2440
+    },
+    {
+      "epoch": 0.03281861686274116,
+      "grad_norm": 0.9668617844581604,
+      "learning_rate": 0.0001999958892007755,
+      "loss": 0.9571184158325196,
+      "step": 2450
+    },
+    {
+      "epoch": 0.03295257040095643,
+      "grad_norm": 1.0439015626907349,
+      "learning_rate": 0.00019999548641493438,
+      "loss": 0.9383399009704589,
+      "step": 2460
+    },
+    {
+      "epoch": 0.0330865239391717,
+      "grad_norm": 1.014618158340454,
+      "learning_rate": 0.00019999506480790434,
+      "loss": 0.8999407768249512,
+      "step": 2470
+    },
+    {
+      "epoch": 0.03322047747738697,
+      "grad_norm": 0.9947917461395264,
+      "learning_rate": 0.00019999462437976463,
+      "loss": 0.9517890930175781,
+      "step": 2480
+    },
+    {
+      "epoch": 0.03335443101560224,
+      "grad_norm": 0.9013609290122986,
+      "learning_rate": 0.0001999941651305982,
+      "loss": 0.9355831146240234,
+      "step": 2490
+    },
+    {
+      "epoch": 0.03348838455381751,
+      "grad_norm": 1.045623540878296,
+      "learning_rate": 0.0001999936870604915,
+      "loss": 0.9567943572998047,
+      "step": 2500
+    },
+    {
+      "epoch": 0.03362233809203278,
+      "grad_norm": 1.0059949159622192,
+      "learning_rate": 0.00019999319016953448,
+      "loss": 0.9816353797912598,
+      "step": 2510
+    },
+    {
+      "epoch": 0.03375629163024805,
+      "grad_norm": 0.9956102967262268,
+      "learning_rate": 0.0001999926744578207,
+      "loss": 0.9603646278381348,
+      "step": 2520
+    },
+    {
+      "epoch": 0.03389024516846332,
+      "grad_norm": 0.9674333333969116,
+      "learning_rate": 0.0001999921399254472,
+      "loss": 0.9202863693237304,
+      "step": 2530
+    },
+    {
+      "epoch": 0.03402419870667859,
+      "grad_norm": 1.149797797203064,
+      "learning_rate": 0.00019999158657251458,
+      "loss": 0.9565104484558106,
+      "step": 2540
+    },
+    {
+      "epoch": 0.03415815224489386,
+      "grad_norm": 0.8696104288101196,
+      "learning_rate": 0.00019999101439912702,
+      "loss": 0.9499526977539062,
+      "step": 2550
+    },
+    {
+      "epoch": 0.03429210578310913,
+      "grad_norm": 0.9146940112113953,
+      "learning_rate": 0.0001999904234053922,
+      "loss": 0.9595541000366211,
+      "step": 2560
+    },
+    {
+      "epoch": 0.0344260593213244,
+      "grad_norm": 1.1251157522201538,
+      "learning_rate": 0.00019998981359142136,
+      "loss": 0.9481052398681641,
+      "step": 2570
+    },
+    {
+      "epoch": 0.03456001285953967,
+      "grad_norm": 0.951270341873169,
+      "learning_rate": 0.0001999891849573293,
+      "loss": 0.9557564735412598,
+      "step": 2580
+    },
+    {
+      "epoch": 0.03469396639775494,
+      "grad_norm": 0.931270956993103,
+      "learning_rate": 0.0001999885375032343,
+      "loss": 0.9400008201599122,
+      "step": 2590
+    },
+    {
+      "epoch": 0.03482791993597021,
+      "grad_norm": 0.9405684471130371,
+      "learning_rate": 0.00019998787122925827,
+      "loss": 0.9368779182434082,
+      "step": 2600
+    },
+    {
+      "epoch": 0.03496187347418548,
+      "grad_norm": 1.1437263488769531,
+      "learning_rate": 0.00019998718613552658,
+      "loss": 0.9466694831848145,
+      "step": 2610
+    },
+    {
+      "epoch": 0.03509582701240075,
+      "grad_norm": 1.1577832698822021,
+      "learning_rate": 0.0001999864822221682,
+      "loss": 0.9129865646362305,
+      "step": 2620
+    },
+    {
+      "epoch": 0.03522978055061602,
+      "grad_norm": 1.0464088916778564,
+      "learning_rate": 0.0001999857594893156,
+      "loss": 0.9279396057128906,
+      "step": 2630
+    },
+    {
+      "epoch": 0.03536373408883129,
+      "grad_norm": 1.053524136543274,
+      "learning_rate": 0.00019998501793710484,
+      "loss": 0.9761667251586914,
+      "step": 2640
+    },
+    {
+      "epoch": 0.03549768762704656,
+      "grad_norm": 0.902043342590332,
+      "learning_rate": 0.0001999842575656755,
+      "loss": 0.9456755638122558,
+      "step": 2650
+    },
+    {
+      "epoch": 0.03563164116526183,
+      "grad_norm": 1.0060131549835205,
+      "learning_rate": 0.00019998347837517063,
+      "loss": 0.9625540733337402,
+      "step": 2660
+    },
+    {
+      "epoch": 0.0357655947034771,
+      "grad_norm": 1.1090943813323975,
+      "learning_rate": 0.00019998268036573697,
+      "loss": 0.9310230255126953,
+      "step": 2670
+    },
+    {
+      "epoch": 0.03589954824169237,
+      "grad_norm": 1.0450677871704102,
+      "learning_rate": 0.00019998186353752467,
+      "loss": 0.9483952522277832,
+      "step": 2680
+    },
+    {
+      "epoch": 0.03603350177990764,
+      "grad_norm": 1.1651952266693115,
+      "learning_rate": 0.00019998102789068752,
+      "loss": 0.9506073951721191,
+      "step": 2690
+    },
+    {
+      "epoch": 0.03616745531812291,
+      "grad_norm": 1.033385157585144,
+      "learning_rate": 0.00019998017342538274,
+      "loss": 0.9435412406921386,
+      "step": 2700
+    },
+    {
+      "epoch": 0.03630140885633818,
+      "grad_norm": 0.9711186289787292,
+      "learning_rate": 0.00019997930014177125,
+      "loss": 0.9563213348388672,
+      "step": 2710
+    },
+    {
+      "epoch": 0.03643536239455345,
+      "grad_norm": 1.0140986442565918,
+      "learning_rate": 0.00019997840804001732,
+      "loss": 0.9228851318359375,
+      "step": 2720
+    },
+    {
+      "epoch": 0.03656931593276872,
+      "grad_norm": 1.0165528059005737,
+      "learning_rate": 0.00019997749712028892,
+      "loss": 0.9404541969299316,
+      "step": 2730
+    },
+    {
+      "epoch": 0.03670326947098399,
+      "grad_norm": 0.9036928415298462,
+      "learning_rate": 0.00019997656738275751,
+      "loss": 0.9369661331176757,
+      "step": 2740
+    },
+    {
+      "epoch": 0.03683722300919926,
+      "grad_norm": 0.9310727119445801,
+      "learning_rate": 0.00019997561882759807,
+      "loss": 0.9242416381835937,
+      "step": 2750
+    },
+    {
+      "epoch": 0.03697117654741453,
+      "grad_norm": 0.9902796745300293,
+      "learning_rate": 0.00019997465145498914,
+      "loss": 0.9326069831848145,
+      "step": 2760
+    },
+    {
+      "epoch": 0.0371051300856298,
+      "grad_norm": 1.0533158779144287,
+      "learning_rate": 0.00019997366526511274,
+      "loss": 0.9423623085021973,
+      "step": 2770
+    },
+    {
+      "epoch": 0.03723908362384507,
+      "grad_norm": 1.045711874961853,
+      "learning_rate": 0.0001999726602581546,
+      "loss": 0.9074527740478515,
+      "step": 2780
+    },
+    {
+      "epoch": 0.03737303716206034,
+      "grad_norm": 0.9685200452804565,
+      "learning_rate": 0.00019997163643430382,
+      "loss": 0.9323711395263672,
+      "step": 2790
+    },
+    {
+      "epoch": 0.03750699070027561,
+      "grad_norm": 1.0000747442245483,
+      "learning_rate": 0.0001999705937937531,
+      "loss": 0.9754042625427246,
+      "step": 2800
+    },
+    {
+      "epoch": 0.03764094423849088,
+      "grad_norm": 0.9732158184051514,
+      "learning_rate": 0.0001999695323366987,
+      "loss": 0.9792140007019043,
+      "step": 2810
+    },
+    {
+      "epoch": 0.03777489777670615,
+      "grad_norm": 0.990386426448822,
+      "learning_rate": 0.0001999684520633404,
+      "loss": 0.9250985145568847,
+      "step": 2820
+    },
+    {
+      "epoch": 0.03790885131492142,
+      "grad_norm": 1.0131051540374756,
+      "learning_rate": 0.00019996735297388157,
+      "loss": 0.9338252067565918,
+      "step": 2830
+    },
+    {
+      "epoch": 0.03804280485313669,
+      "grad_norm": 0.9239967465400696,
+      "learning_rate": 0.000199966235068529,
+      "loss": 0.9674043655395508,
+      "step": 2840
+    },
+    {
+      "epoch": 0.03817675839135196,
+      "grad_norm": 0.9224802255630493,
+      "learning_rate": 0.00019996509834749318,
+      "loss": 0.9093979835510254,
+      "step": 2850
+    },
+    {
+      "epoch": 0.03831071192956723,
+      "grad_norm": 0.9065786004066467,
+      "learning_rate": 0.00019996394281098802,
+      "loss": 0.9495321273803711,
+      "step": 2860
+    },
+    {
+      "epoch": 0.0384446654677825,
+      "grad_norm": 1.0348045825958252,
+      "learning_rate": 0.00019996276845923104,
+      "loss": 0.9240209579467773,
+      "step": 2870
+    },
+    {
+      "epoch": 0.038578619005997766,
+      "grad_norm": 0.9366620182991028,
+      "learning_rate": 0.0001999615752924433,
+      "loss": 0.9238608360290528,
+      "step": 2880
+    },
+    {
+      "epoch": 0.038712572544213036,
+      "grad_norm": 1.122186303138733,
+      "learning_rate": 0.0001999603633108493,
+      "loss": 0.9154872894287109,
+      "step": 2890
+    },
+    {
+      "epoch": 0.038846526082428307,
+      "grad_norm": 0.8368102312088013,
+      "learning_rate": 0.0001999591325146772,
+      "loss": 0.9037948608398437,
+      "step": 2900
+    },
+    {
+      "epoch": 0.03898047962064358,
+      "grad_norm": 0.8084287047386169,
+      "learning_rate": 0.00019995788290415867,
+      "loss": 0.9526560783386231,
+      "step": 2910
+    },
+    {
+      "epoch": 0.03911443315885885,
+      "grad_norm": 0.9385167360305786,
+      "learning_rate": 0.00019995661447952894,
+      "loss": 0.9274347305297852,
+      "step": 2920
+    },
+    {
+      "epoch": 0.03924838669707412,
+      "grad_norm": 1.0126752853393555,
+      "learning_rate": 0.00019995532724102668,
+      "loss": 0.9336477279663086,
+      "step": 2930
+    },
+    {
+      "epoch": 0.03938234023528939,
+      "grad_norm": 1.0768028497695923,
+      "learning_rate": 0.00019995402118889424,
+      "loss": 0.9312108039855957,
+      "step": 2940
+    },
+    {
+      "epoch": 0.03951629377350466,
+      "grad_norm": 0.9114157557487488,
+      "learning_rate": 0.00019995269632337742,
+      "loss": 0.9413155555725098,
+      "step": 2950
+    },
+    {
+      "epoch": 0.03965024731171993,
+      "grad_norm": 0.8750312924385071,
+      "learning_rate": 0.00019995135264472557,
+      "loss": 0.9509553909301758,
+      "step": 2960
+    },
+    {
+      "epoch": 0.0397842008499352,
+      "grad_norm": 1.0956413745880127,
+      "learning_rate": 0.00019994999015319164,
+      "loss": 0.9265983581542969,
+      "step": 2970
+    },
+    {
+      "epoch": 0.03991815438815047,
+      "grad_norm": 0.9434391856193542,
+      "learning_rate": 0.00019994860884903204,
+      "loss": 0.9004026412963867,
+      "step": 2980
+    },
+    {
+      "epoch": 0.04005210792636574,
+      "grad_norm": 0.8206298351287842,
+      "learning_rate": 0.00019994720873250677,
+      "loss": 0.9307710647583007,
+      "step": 2990
+    },
+    {
+      "epoch": 0.04018606146458101,
+      "grad_norm": 1.0000940561294556,
+      "learning_rate": 0.00019994578980387937,
+      "loss": 0.9217443466186523,
+      "step": 3000
+    },
+    {
+      "epoch": 0.04018606146458101,
+      "eval_loss": 0.9095186591148376,
+      "eval_runtime": 282.8579,
+      "eval_samples_per_second": 10.652,
+      "eval_steps_per_second": 10.652,
+      "step": 3000
+    },
+    {
+      "epoch": 0.04032001500279628,
+      "grad_norm": 0.8734673857688904,
+      "learning_rate": 0.0001999443520634169,
+      "loss": 0.941364860534668,
+      "step": 3010
+    },
+    {
+      "epoch": 0.04045396854101155,
+      "grad_norm": 0.8950884938240051,
+      "learning_rate": 0.00019994289551139,
+      "loss": 0.9462560653686524,
+      "step": 3020
+    },
+    {
+      "epoch": 0.04058792207922682,
+      "grad_norm": 1.0811048746109009,
+      "learning_rate": 0.00019994142014807277,
+      "loss": 0.9378127098083496,
+      "step": 3030
+    },
+    {
+      "epoch": 0.04072187561744209,
+      "grad_norm": 1.3110917806625366,
+      "learning_rate": 0.00019993992597374297,
+      "loss": 0.9490732192993164,
+      "step": 3040
+    },
+    {
+      "epoch": 0.04085582915565736,
+      "grad_norm": 0.892918050289154,
+      "learning_rate": 0.00019993841298868175,
+      "loss": 0.9350913047790528,
+      "step": 3050
+    },
+    {
+      "epoch": 0.04098978269387263,
+      "grad_norm": 0.8810139298439026,
+      "learning_rate": 0.00019993688119317396,
+      "loss": 0.9095968246459961,
+      "step": 3060
+    },
+    {
+      "epoch": 0.0411237362320879,
+      "grad_norm": 0.9049996137619019,
+      "learning_rate": 0.00019993533058750788,
+      "loss": 0.9268231391906738,
+      "step": 3070
+    },
+    {
+      "epoch": 0.04125768977030317,
+      "grad_norm": 0.8555030226707458,
+      "learning_rate": 0.0001999337611719754,
+      "loss": 0.9202340126037598,
+      "step": 3080
+    },
+    {
+      "epoch": 0.04139164330851844,
+      "grad_norm": 0.8748414516448975,
+      "learning_rate": 0.00019993217294687185,
+      "loss": 0.9081048965454102,
+      "step": 3090
+    },
+    {
+      "epoch": 0.04152559684673371,
+      "grad_norm": 0.9201762080192566,
+      "learning_rate": 0.00019993056591249624,
+      "loss": 0.9255125045776367,
+      "step": 3100
+    },
+    {
+      "epoch": 0.04165955038494898,
+      "grad_norm": 0.8035100102424622,
+      "learning_rate": 0.00019992894006915098,
+      "loss": 0.9364505767822265,
+      "step": 3110
+    },
+    {
+      "epoch": 0.04179350392316425,
+      "grad_norm": 0.8489510416984558,
+      "learning_rate": 0.00019992729541714214,
+      "loss": 0.9114150047302246,
+      "step": 3120
+    },
+    {
+      "epoch": 0.04192745746137952,
+      "grad_norm": 0.8885366916656494,
+      "learning_rate": 0.00019992563195677927,
+      "loss": 0.9378640174865722,
+      "step": 3130
+    },
+    {
+      "epoch": 0.04206141099959479,
+      "grad_norm": 0.916833221912384,
+      "learning_rate": 0.00019992394968837546,
+      "loss": 0.935064697265625,
+      "step": 3140
+    },
+    {
+      "epoch": 0.04219536453781006,
+      "grad_norm": 0.8948741555213928,
+      "learning_rate": 0.0001999222486122473,
+      "loss": 0.9183822631835937,
+      "step": 3150
+    },
+    {
+      "epoch": 0.04232931807602533,
+      "grad_norm": 1.0157934427261353,
+      "learning_rate": 0.000199920528728715,
+      "loss": 0.946108055114746,
+      "step": 3160
+    },
+    {
+      "epoch": 0.0424632716142406,
+      "grad_norm": 0.8575133085250854,
+      "learning_rate": 0.00019991879003810231,
+      "loss": 0.9633334159851075,
+      "step": 3170
+    },
+    {
+      "epoch": 0.04259722515245587,
+      "grad_norm": 0.9529309272766113,
+      "learning_rate": 0.00019991703254073647,
+      "loss": 0.9019985198974609,
+      "step": 3180
+    },
+    {
+      "epoch": 0.04273117869067114,
+      "grad_norm": 0.9916787147521973,
+      "learning_rate": 0.00019991525623694827,
+      "loss": 0.9476448059082031,
+      "step": 3190
+    },
+    {
+      "epoch": 0.04286513222888641,
+      "grad_norm": 0.8963183164596558,
+      "learning_rate": 0.00019991346112707204,
+      "loss": 0.9350907325744628,
+      "step": 3200
+    },
+    {
+      "epoch": 0.04299908576710168,
+      "grad_norm": 0.9970526695251465,
+      "learning_rate": 0.00019991164721144564,
+      "loss": 0.9250227928161621,
+      "step": 3210
+    },
+    {
+      "epoch": 0.04313303930531695,
+      "grad_norm": 0.8881255388259888,
+      "learning_rate": 0.00019990981449041054,
+      "loss": 0.9250835418701172,
+      "step": 3220
+    },
+    {
+      "epoch": 0.04326699284353222,
+      "grad_norm": 1.0573492050170898,
+      "learning_rate": 0.00019990796296431165,
+      "loss": 0.9150334358215332,
+      "step": 3230
+    },
+    {
+      "epoch": 0.04340094638174749,
+      "grad_norm": 1.0080435276031494,
+      "learning_rate": 0.00019990609263349746,
+      "loss": 0.9671030044555664,
+      "step": 3240
+    },
+    {
+      "epoch": 0.04353489991996276,
+      "grad_norm": 0.9293102025985718,
+      "learning_rate": 0.00019990420349832004,
+      "loss": 0.9286737442016602,
+      "step": 3250
+    },
+    {
+      "epoch": 0.04366885345817803,
+      "grad_norm": 0.9268156886100769,
+      "learning_rate": 0.00019990229555913493,
+      "loss": 0.9326079368591309,
+      "step": 3260
+    },
+    {
+      "epoch": 0.0438028069963933,
+      "grad_norm": 0.9221058487892151,
+      "learning_rate": 0.0001999003688163013,
+      "loss": 0.9419919013977051,
+      "step": 3270
+    },
+    {
+      "epoch": 0.04393676053460857,
+      "grad_norm": 0.8882941007614136,
+      "learning_rate": 0.0001998984232701817,
+      "loss": 0.9261360168457031,
+      "step": 3280
+    },
+    {
+      "epoch": 0.04407071407282384,
+      "grad_norm": 0.844829797744751,
+      "learning_rate": 0.00019989645892114243,
+      "loss": 0.927768325805664,
+      "step": 3290
+    },
+    {
+      "epoch": 0.04420466761103911,
+      "grad_norm": 1.0086514949798584,
+      "learning_rate": 0.00019989447576955317,
+      "loss": 0.9342447280883789,
+      "step": 3300
+    },
+    {
+      "epoch": 0.04433862114925438,
+      "grad_norm": 0.9930247068405151,
+      "learning_rate": 0.00019989247381578719,
+      "loss": 0.9354818344116211,
+      "step": 3310
+    },
+    {
+      "epoch": 0.04447257468746965,
+      "grad_norm": 0.9238676428794861,
+      "learning_rate": 0.0001998904530602213,
+      "loss": 0.95006103515625,
+      "step": 3320
+    },
+    {
+      "epoch": 0.04460652822568492,
+      "grad_norm": 0.8584195971488953,
+      "learning_rate": 0.0001998884135032358,
+      "loss": 0.9177757263183594,
+      "step": 3330
+    },
+    {
+      "epoch": 0.04474048176390019,
+      "grad_norm": 0.9681214094161987,
+      "learning_rate": 0.0001998863551452147,
+      "loss": 0.9331873893737793,
+      "step": 3340
+    },
+    {
+      "epoch": 0.04487443530211546,
+      "grad_norm": 0.9767482876777649,
+      "learning_rate": 0.0001998842779865453,
+      "loss": 0.8993824005126954,
+      "step": 3350
+    },
+    {
+      "epoch": 0.04500838884033073,
+      "grad_norm": 0.9796115159988403,
+      "learning_rate": 0.00019988218202761867,
+      "loss": 0.9324639320373536,
+      "step": 3360
+    },
+    {
+      "epoch": 0.045142342378546,
+      "grad_norm": 1.0159355401992798,
+      "learning_rate": 0.0001998800672688292,
+      "loss": 0.9196332931518555,
+      "step": 3370
+    },
+    {
+      "epoch": 0.04527629591676127,
+      "grad_norm": 0.9687798619270325,
+      "learning_rate": 0.000199877933710575,
+      "loss": 0.9572076797485352,
+      "step": 3380
+    },
+    {
+      "epoch": 0.04541024945497654,
+      "grad_norm": 0.9174926280975342,
+      "learning_rate": 0.00019987578135325764,
+      "loss": 0.9035839080810547,
+      "step": 3390
+    },
+    {
+      "epoch": 0.04554420299319181,
+      "grad_norm": 0.8543787002563477,
+      "learning_rate": 0.00019987361019728223,
+      "loss": 0.9235164642333984,
+      "step": 3400
+    },
+    {
+      "epoch": 0.04567815653140708,
+      "grad_norm": 0.870416522026062,
+      "learning_rate": 0.00019987142024305743,
+      "loss": 0.9195279121398926,
+      "step": 3410
+    },
+    {
+      "epoch": 0.04581211006962235,
+      "grad_norm": 0.8485437631607056,
+      "learning_rate": 0.00019986921149099543,
+      "loss": 0.9079272270202636,
+      "step": 3420
+    },
+    {
+      "epoch": 0.04594606360783762,
+      "grad_norm": 0.8823762536048889,
+      "learning_rate": 0.00019986698394151197,
+      "loss": 0.9315704345703125,
+      "step": 3430
+    },
+    {
+      "epoch": 0.04608001714605289,
+      "grad_norm": 0.8347775340080261,
+      "learning_rate": 0.0001998647375950263,
+      "loss": 0.8943398475646973,
+      "step": 3440
+    },
+    {
+      "epoch": 0.04621397068426816,
+      "grad_norm": 0.8320932388305664,
+      "learning_rate": 0.0001998624724519613,
+      "loss": 0.9377827644348145,
+      "step": 3450
+    },
+    {
+      "epoch": 0.04634792422248343,
+      "grad_norm": 1.092563271522522,
+      "learning_rate": 0.0001998601885127432,
+      "loss": 0.9222759246826172,
+      "step": 3460
+    },
+    {
+      "epoch": 0.0464818777606987,
+      "grad_norm": 0.9765579104423523,
+      "learning_rate": 0.00019985788577780194,
+      "loss": 0.9315988540649414,
+      "step": 3470
+    },
+    {
+      "epoch": 0.04661583129891397,
+      "grad_norm": 0.8835811614990234,
+      "learning_rate": 0.00019985556424757098,
+      "loss": 0.9130311012268066,
+      "step": 3480
+    },
+    {
+      "epoch": 0.046749784837129243,
+      "grad_norm": 0.8476258516311646,
+      "learning_rate": 0.00019985322392248724,
+      "loss": 0.9215794563293457,
+      "step": 3490
+    },
+    {
+      "epoch": 0.046883738375344514,
+      "grad_norm": 0.9369233846664429,
+      "learning_rate": 0.00019985086480299121,
+      "loss": 0.9167479515075684,
+      "step": 3500
+    },
+    {
+      "epoch": 0.047017691913559784,
+      "grad_norm": 0.9186804890632629,
+      "learning_rate": 0.00019984848688952696,
+      "loss": 0.9383148193359375,
+      "step": 3510
+    },
+    {
+      "epoch": 0.047151645451775054,
+      "grad_norm": 0.9544302821159363,
+      "learning_rate": 0.000199846090182542,
+      "loss": 0.9238338470458984,
+      "step": 3520
+    },
+    {
+      "epoch": 0.047285598989990324,
+      "grad_norm": 0.9397124648094177,
+      "learning_rate": 0.00019984367468248748,
+      "loss": 0.9094552993774414,
+      "step": 3530
+    },
+    {
+      "epoch": 0.047419552528205594,
+      "grad_norm": 0.8298595547676086,
+      "learning_rate": 0.00019984124038981808,
+      "loss": 0.9345894813537597,
+      "step": 3540
+    },
+    {
+      "epoch": 0.047553506066420864,
+      "grad_norm": 0.9456910490989685,
+      "learning_rate": 0.00019983878730499193,
+      "loss": 0.9422292709350586,
+      "step": 3550
+    },
+    {
+      "epoch": 0.047687459604636134,
+      "grad_norm": 0.8198268413543701,
+      "learning_rate": 0.00019983631542847075,
+      "loss": 0.9278390884399415,
+      "step": 3560
+    },
+    {
+      "epoch": 0.047821413142851404,
+      "grad_norm": 0.8617146015167236,
+      "learning_rate": 0.00019983382476071983,
+      "loss": 0.9171154975891114,
+      "step": 3570
+    },
+    {
+      "epoch": 0.047955366681066675,
+      "grad_norm": 0.8447020649909973,
+      "learning_rate": 0.00019983131530220793,
+      "loss": 0.9147285461425781,
+      "step": 3580
+    },
+    {
+      "epoch": 0.048089320219281945,
+      "grad_norm": 0.8757874965667725,
+      "learning_rate": 0.00019982878705340743,
+      "loss": 0.936093807220459,
+      "step": 3590
+    },
+    {
+      "epoch": 0.048223273757497215,
+      "grad_norm": 0.8899102807044983,
+      "learning_rate": 0.00019982624001479415,
+      "loss": 0.9166989326477051,
+      "step": 3600
+    },
+    {
+      "epoch": 0.048357227295712485,
+      "grad_norm": 0.8696548342704773,
+      "learning_rate": 0.0001998236741868475,
+      "loss": 0.9303563117980957,
+      "step": 3610
+    },
+    {
+      "epoch": 0.048491180833927755,
+      "grad_norm": 0.9574766159057617,
+      "learning_rate": 0.00019982108957005047,
+      "loss": 0.9008687019348145,
+      "step": 3620
+    },
+    {
+      "epoch": 0.048625134372143025,
+      "grad_norm": 0.988602876663208,
+      "learning_rate": 0.00019981848616488948,
+      "loss": 0.926761245727539,
+      "step": 3630
+    },
+    {
+      "epoch": 0.048759087910358295,
+      "grad_norm": 0.9067688584327698,
+      "learning_rate": 0.00019981586397185456,
+      "loss": 0.9246554374694824,
+      "step": 3640
+    },
+    {
+      "epoch": 0.048893041448573565,
+      "grad_norm": 0.8702062368392944,
+      "learning_rate": 0.00019981322299143927,
+      "loss": 0.9095652580261231,
+      "step": 3650
+    },
+    {
+      "epoch": 0.049026994986788835,
+      "grad_norm": 0.9878699779510498,
+      "learning_rate": 0.00019981056322414072,
+      "loss": 0.8977669715881348,
+      "step": 3660
+    },
+    {
+      "epoch": 0.049160948525004106,
+      "grad_norm": 0.8746298551559448,
+      "learning_rate": 0.00019980788467045945,
+      "loss": 0.9227336883544922,
+      "step": 3670
+    },
+    {
+      "epoch": 0.049294902063219376,
+      "grad_norm": 0.9934686422348022,
+      "learning_rate": 0.0001998051873308997,
+      "loss": 0.9178746223449707,
+      "step": 3680
+    },
+    {
+      "epoch": 0.049428855601434646,
+      "grad_norm": 0.9328760504722595,
+      "learning_rate": 0.00019980247120596916,
+      "loss": 0.9450766563415527,
+      "step": 3690
+    },
+    {
+      "epoch": 0.04956280913964991,
+      "grad_norm": 0.8085207343101501,
+      "learning_rate": 0.00019979973629617902,
+      "loss": 0.9002262115478515,
+      "step": 3700
+    },
+    {
+      "epoch": 0.04969676267786518,
+      "grad_norm": 0.9033879041671753,
+      "learning_rate": 0.00019979698260204405,
+      "loss": 0.9274055480957031,
+      "step": 3710
+    },
+    {
+      "epoch": 0.04983071621608045,
+      "grad_norm": 0.9597737193107605,
+      "learning_rate": 0.00019979421012408257,
+      "loss": 0.8976191520690918,
+      "step": 3720
+    },
+    {
+      "epoch": 0.04996466975429572,
+      "grad_norm": 0.8724941611289978,
+      "learning_rate": 0.00019979141886281642,
+      "loss": 0.9168087005615234,
+      "step": 3730
+    },
+    {
+      "epoch": 0.05009862329251099,
+      "grad_norm": 0.8592389225959778,
+      "learning_rate": 0.00019978860881877096,
+      "loss": 0.9181900024414062,
+      "step": 3740
+    },
+    {
+      "epoch": 0.05023257683072626,
+      "grad_norm": 0.9238866567611694,
+      "learning_rate": 0.0001997857799924751,
+      "loss": 0.9294312477111817,
+      "step": 3750
+    },
+    {
+      "epoch": 0.05036653036894153,
+      "grad_norm": 0.9541118144989014,
+      "learning_rate": 0.0001997829323844613,
+      "loss": 0.9044769287109375,
+      "step": 3760
+    },
+    {
+      "epoch": 0.0505004839071568,
+      "grad_norm": 0.8983111381530762,
+      "learning_rate": 0.0001997800659952655,
+      "loss": 0.9011771202087402,
+      "step": 3770
+    },
+    {
+      "epoch": 0.05063443744537207,
+      "grad_norm": 0.9186615943908691,
+      "learning_rate": 0.00019977718082542722,
+      "loss": 0.9071616172790528,
+      "step": 3780
+    },
+    {
+      "epoch": 0.05076839098358734,
+      "grad_norm": 0.8877631425857544,
+      "learning_rate": 0.00019977427687548955,
+      "loss": 0.9309605598449707,
+      "step": 3790
+    },
+    {
+      "epoch": 0.05090234452180261,
+      "grad_norm": 0.8731006979942322,
+      "learning_rate": 0.00019977135414599905,
+      "loss": 0.9176346778869628,
+      "step": 3800
+    },
+    {
+      "epoch": 0.05103629806001788,
+      "grad_norm": 0.9903236627578735,
+      "learning_rate": 0.00019976841263750583,
+      "loss": 0.9287304878234863,
+      "step": 3810
+    },
+    {
+      "epoch": 0.05117025159823315,
+      "grad_norm": 1.0615185499191284,
+      "learning_rate": 0.00019976545235056352,
+      "loss": 0.924833106994629,
+      "step": 3820
+    },
+    {
+      "epoch": 0.05130420513644842,
+      "grad_norm": 0.9164450764656067,
+      "learning_rate": 0.00019976247328572938,
+      "loss": 0.9383651733398437,
+      "step": 3830
+    },
+    {
+      "epoch": 0.05143815867466369,
+      "grad_norm": 1.0046485662460327,
+      "learning_rate": 0.000199759475443564,
+      "loss": 0.9078041076660156,
+      "step": 3840
+    },
+    {
+      "epoch": 0.05157211221287896,
+      "grad_norm": 0.886355996131897,
+      "learning_rate": 0.0001997564588246318,
+      "loss": 0.8979540824890136,
+      "step": 3850
+    },
+    {
+      "epoch": 0.05170606575109423,
+      "grad_norm": 0.8408886194229126,
+      "learning_rate": 0.00019975342342950044,
+      "loss": 0.900606346130371,
+      "step": 3860
+    },
+    {
+      "epoch": 0.0518400192893095,
+      "grad_norm": 0.881829023361206,
+      "learning_rate": 0.0001997503692587413,
+      "loss": 0.8997422218322754,
+      "step": 3870
+    },
+    {
+      "epoch": 0.05197397282752477,
+      "grad_norm": 1.0530098676681519,
+      "learning_rate": 0.0001997472963129292,
+      "loss": 0.9246989250183105,
+      "step": 3880
+    },
+    {
+      "epoch": 0.05210792636574004,
+      "grad_norm": 0.9508712291717529,
+      "learning_rate": 0.0001997442045926426,
+      "loss": 0.9115647315979004,
+      "step": 3890
+    },
+    {
+      "epoch": 0.05224187990395531,
+      "grad_norm": 0.9167556166648865,
+      "learning_rate": 0.00019974109409846337,
+      "loss": 0.9261734962463379,
+      "step": 3900
+    },
+    {
+      "epoch": 0.05237583344217058,
+      "grad_norm": 0.9922856092453003,
+      "learning_rate": 0.00019973796483097696,
+      "loss": 0.9105630874633789,
+      "step": 3910
+    },
+    {
+      "epoch": 0.05250978698038585,
+      "grad_norm": 0.9163697957992554,
+      "learning_rate": 0.00019973481679077238,
+      "loss": 0.9143681526184082,
+      "step": 3920
+    },
+    {
+      "epoch": 0.05264374051860112,
+      "grad_norm": 0.8309245109558105,
+      "learning_rate": 0.0001997316499784422,
+      "loss": 0.9261678695678711,
+      "step": 3930
+    },
+    {
+      "epoch": 0.05277769405681639,
+      "grad_norm": 1.0028313398361206,
+      "learning_rate": 0.0001997284643945824,
+      "loss": 0.8874808311462402,
+      "step": 3940
+    },
+    {
+      "epoch": 0.05291164759503166,
+      "grad_norm": 0.9336344003677368,
+      "learning_rate": 0.0001997252600397926,
+      "loss": 0.9186717987060546,
+      "step": 3950
+    },
+    {
+      "epoch": 0.05304560113324693,
+      "grad_norm": 0.9435511231422424,
+      "learning_rate": 0.00019972203691467596,
+      "loss": 0.9015359878540039,
+      "step": 3960
+    },
+    {
+      "epoch": 0.0531795546714622,
+      "grad_norm": 0.8353652954101562,
+      "learning_rate": 0.0001997187950198391,
+      "loss": 0.9050702095031739,
+      "step": 3970
+    },
+    {
+      "epoch": 0.05331350820967747,
+      "grad_norm": 0.9390276074409485,
+      "learning_rate": 0.00019971553435589224,
+      "loss": 0.9628662109375,
+      "step": 3980
+    },
+    {
+      "epoch": 0.05344746174789274,
+      "grad_norm": 0.9069141149520874,
+      "learning_rate": 0.00019971225492344905,
+      "loss": 0.8978482246398926,
+      "step": 3990
+    },
+    {
+      "epoch": 0.05358141528610801,
+      "grad_norm": 0.9027342796325684,
+      "learning_rate": 0.00019970895672312685,
+      "loss": 0.9324676513671875,
+      "step": 4000
+    },
+    {
+      "epoch": 0.05358141528610801,
+      "eval_loss": 0.8909962773323059,
+      "eval_runtime": 282.9255,
+      "eval_samples_per_second": 10.649,
+      "eval_steps_per_second": 10.649,
+      "step": 4000
+    },
+    {
+      "epoch": 0.05371536882432328,
+      "grad_norm": 0.8347414135932922,
+      "learning_rate": 0.0001997056397555464,
+      "loss": 0.9355817794799804,
+      "step": 4010
+    },
+    {
+      "epoch": 0.05384932236253855,
+      "grad_norm": 0.9647659063339233,
+      "learning_rate": 0.000199702304021332,
+      "loss": 0.9352194786071777,
+      "step": 4020
+    },
+    {
+      "epoch": 0.05398327590075382,
+      "grad_norm": 0.9430063962936401,
+      "learning_rate": 0.0001996989495211115,
+      "loss": 0.9100798606872559,
+      "step": 4030
+    },
+    {
+      "epoch": 0.05411722943896909,
+      "grad_norm": 0.8570693731307983,
+      "learning_rate": 0.00019969557625551635,
+      "loss": 0.8997575759887695,
+      "step": 4040
+    },
+    {
+      "epoch": 0.05425118297718436,
+      "grad_norm": 0.8614152073860168,
+      "learning_rate": 0.0001996921842251814,
+      "loss": 0.919250202178955,
+      "step": 4050
+    },
+    {
+      "epoch": 0.05438513651539963,
+      "grad_norm": 1.0438923835754395,
+      "learning_rate": 0.00019968877343074515,
+      "loss": 0.9120957374572753,
+      "step": 4060
+    },
+    {
+      "epoch": 0.0545190900536149,
+      "grad_norm": 0.8420084118843079,
+      "learning_rate": 0.00019968534387284956,
+      "loss": 0.9383444786071777,
+      "step": 4070
+    },
+    {
+      "epoch": 0.054653043591830174,
+      "grad_norm": 0.9416694641113281,
+      "learning_rate": 0.00019968189555214012,
+      "loss": 0.9132110595703125,
+      "step": 4080
+    },
+    {
+      "epoch": 0.054786997130045444,
+      "grad_norm": 0.9215363264083862,
+      "learning_rate": 0.0001996784284692659,
+      "loss": 0.9173206329345703,
+      "step": 4090
+    },
+    {
+      "epoch": 0.054920950668260714,
+      "grad_norm": 1.1527929306030273,
+      "learning_rate": 0.00019967494262487946,
+      "loss": 0.9360475540161133,
+      "step": 4100
+    },
+    {
+      "epoch": 0.055054904206475984,
+      "grad_norm": 0.9070743322372437,
+      "learning_rate": 0.00019967143801963693,
+      "loss": 0.9068495750427246,
+      "step": 4110
+    },
+    {
+      "epoch": 0.055188857744691254,
+      "grad_norm": 0.8676186800003052,
+      "learning_rate": 0.00019966791465419795,
+      "loss": 0.9177586555480957,
+      "step": 4120
+    },
+    {
+      "epoch": 0.055322811282906524,
+      "grad_norm": 0.9942190051078796,
+      "learning_rate": 0.00019966437252922563,
+      "loss": 0.937378215789795,
+      "step": 4130
+    },
+    {
+      "epoch": 0.055456764821121794,
+      "grad_norm": 1.0533899068832397,
+      "learning_rate": 0.00019966081164538672,
+      "loss": 0.9167658805847168,
+      "step": 4140
+    },
+    {
+      "epoch": 0.055590718359337064,
+      "grad_norm": 0.9439006447792053,
+      "learning_rate": 0.00019965723200335146,
+      "loss": 0.9186976432800293,
+      "step": 4150
+    },
+    {
+      "epoch": 0.055724671897552334,
+      "grad_norm": 0.8586645126342773,
+      "learning_rate": 0.0001996536336037936,
+      "loss": 0.9189139366149902,
+      "step": 4160
+    },
+    {
+      "epoch": 0.055858625435767605,
+      "grad_norm": 0.9884462952613831,
+      "learning_rate": 0.00019965001644739043,
+      "loss": 0.9102540969848633,
+      "step": 4170
+    },
+    {
+      "epoch": 0.055992578973982875,
+      "grad_norm": 0.8929790258407593,
+      "learning_rate": 0.00019964638053482277,
+      "loss": 0.9131571769714355,
+      "step": 4180
+    },
+    {
+      "epoch": 0.056126532512198145,
+      "grad_norm": 0.8665063977241516,
+      "learning_rate": 0.00019964272586677498,
+      "loss": 0.9067962646484375,
+      "step": 4190
+    },
+    {
+      "epoch": 0.056260486050413415,
+      "grad_norm": 0.8934453129768372,
+      "learning_rate": 0.0001996390524439349,
+      "loss": 0.9221485137939454,
+      "step": 4200
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 74653,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 10000000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.234043755324503e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoints_under8000/checkpoint-4202/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:43d5e17e7d1b7abbd7c31f1efd5000edcb46956afe635e0e7493688330884f71
+size 5777