Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,202 @@
+---
+base_model: togethercomputer/Qwen3.6-35B-A3B_bnb_4bit_t
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.1

adapter_config.json

@@ -0,0 +1,36 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "togethercomputer/Qwen3.6-35B-A3B_bnb_4bit_t",
+  "bias": "none",
+  "corda_config": null,
+  "eva_config": null,
+  "exclude_modules": [],
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_bias": false,
+  "lora_dropout": 0.05000000074505806,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "o_proj",
+    "v_proj",
+    "k_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f9501ef7627d21969fcda912b64f1cdeff8b1e4d6f3e2842d2f68a9ad53849a
+size 55061008

chat_template.jinja

@@ -0,0 +1,151 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,111 @@
+{
+  "architectures": [
+    "Qwen3_5MoeForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_output_gate": true,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 248046,
+  "full_attention_interval": 4,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 16,
+  "linear_num_value_heads": 32,
+  "linear_value_head_dim": 128,
+  "mamba_ssm_dtype": "float32",
+  "max_position_embeddings": 262144,
+  "model_type": "qwen3_5_moe_text",
+  "moe_intermediate_size": 512,
+  "mtp_num_hidden_layers": 1,
+  "mtp_use_dedicated_embeddings": false,
+  "num_attention_heads": 16,
+  "num_experts": 256,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 2,
+  "output_router_logits": false,
+  "pad_token_id": 248044,
+  "partial_rotary_factor": 0.25,
+  "quantization_config": {
+    "_load_in_4bit": true,
+    "_load_in_8bit": false,
+    "bnb_4bit_compute_dtype": "bfloat16",
+    "bnb_4bit_quant_storage": "bfloat16",
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": true,
+    "load_in_8bit": false,
+    "quant_method": "bitsandbytes"
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      11,
+      11,
+      10
+    ],
+    "partial_rotary_factor": 0.25,
+    "rope_theta": 10000000,
+    "rope_type": "default"
+  },
+  "router_aux_loss_coef": 0.001,
+  "shared_expert_intermediate_size": 512,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.10.1",
+  "use_cache": false,
+  "vocab_size": 248320,
+  "torch_dtype": "bfloat16"
+}

special_tokens_map.json

@@ -0,0 +1,11 @@
+{
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "eos_token": "<|im_end|>",
+  "image_token": "<|image_pad|>",
+  "pad_token": "<|endoftext|>",
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": false,
+  "local_files_only": true,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

trainer_state.json

@@ -0,0 +1,2268 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 318,
+  "global_step": 318,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0031446540880503146,
+      "grad_norm": 1.4494149684906006,
+      "learning_rate": 0.0,
+      "loss": 2.1591796875,
+      "step": 1
+    },
+    {
+      "epoch": 0.006289308176100629,
+      "grad_norm": 1.472015619277954,
+      "learning_rate": 1.0000000000000002e-06,
+      "loss": 1.9853515625,
+      "step": 2
+    },
+    {
+      "epoch": 0.009433962264150943,
+      "grad_norm": 1.2991633415222168,
+      "learning_rate": 2.0000000000000003e-06,
+      "loss": 1.94287109375,
+      "step": 3
+    },
+    {
+      "epoch": 0.012578616352201259,
+      "grad_norm": 1.3701603412628174,
+      "learning_rate": 3e-06,
+      "loss": 1.96337890625,
+      "step": 4
+    },
+    {
+      "epoch": 0.015723270440251572,
+      "grad_norm": 1.668572187423706,
+      "learning_rate": 4.000000000000001e-06,
+      "loss": 2.1435546875,
+      "step": 5
+    },
+    {
+      "epoch": 0.018867924528301886,
+      "grad_norm": 1.2014765739440918,
+      "learning_rate": 5e-06,
+      "loss": 1.95751953125,
+      "step": 6
+    },
+    {
+      "epoch": 0.0220125786163522,
+      "grad_norm": 1.3996855020523071,
+      "learning_rate": 6e-06,
+      "loss": 2.08203125,
+      "step": 7
+    },
+    {
+      "epoch": 0.025157232704402517,
+      "grad_norm": 1.3427141904830933,
+      "learning_rate": 7e-06,
+      "loss": 2.046142578125,
+      "step": 8
+    },
+    {
+      "epoch": 0.02830188679245283,
+      "grad_norm": 1.6375218629837036,
+      "learning_rate": 8.000000000000001e-06,
+      "loss": 2.1396484375,
+      "step": 9
+    },
+    {
+      "epoch": 0.031446540880503145,
+      "grad_norm": 0.7162802815437317,
+      "learning_rate": 9e-06,
+      "loss": 1.611328125,
+      "step": 10
+    },
+    {
+      "epoch": 0.03459119496855346,
+      "grad_norm": 1.6569421291351318,
+      "learning_rate": 1e-05,
+      "loss": 2.0888671875,
+      "step": 11
+    },
+    {
+      "epoch": 0.03773584905660377,
+      "grad_norm": 1.2393434047698975,
+      "learning_rate": 9.999739903471488e-06,
+      "loss": 1.94384765625,
+      "step": 12
+    },
+    {
+      "epoch": 0.040880503144654086,
+      "grad_norm": 1.2549704313278198,
+      "learning_rate": 9.998959640946033e-06,
+      "loss": 1.9033203125,
+      "step": 13
+    },
+    {
+      "epoch": 0.0440251572327044,
+      "grad_norm": 1.196319580078125,
+      "learning_rate": 9.997659293601066e-06,
+      "loss": 1.92578125,
+      "step": 14
+    },
+    {
+      "epoch": 0.04716981132075472,
+      "grad_norm": 0.961357831954956,
+      "learning_rate": 9.995838996722916e-06,
+      "loss": 1.71826171875,
+      "step": 15
+    },
+    {
+      "epoch": 0.050314465408805034,
+      "grad_norm": 0.7349967956542969,
+      "learning_rate": 9.993498939692744e-06,
+      "loss": 1.650146484375,
+      "step": 16
+    },
+    {
+      "epoch": 0.05345911949685535,
+      "grad_norm": 0.8328500986099243,
+      "learning_rate": 9.990639365966835e-06,
+      "loss": 1.73095703125,
+      "step": 17
+    },
+    {
+      "epoch": 0.05660377358490566,
+      "grad_norm": 0.7642322778701782,
+      "learning_rate": 9.987260573051268e-06,
+      "loss": 1.638671875,
+      "step": 18
+    },
+    {
+      "epoch": 0.059748427672955975,
+      "grad_norm": 0.6029979586601257,
+      "learning_rate": 9.983362912470967e-06,
+      "loss": 1.62841796875,
+      "step": 19
+    },
+    {
+      "epoch": 0.06289308176100629,
+      "grad_norm": 0.81263667345047,
+      "learning_rate": 9.978946789733126e-06,
+      "loss": 1.64306640625,
+      "step": 20
+    },
+    {
+      "epoch": 0.0660377358490566,
+      "grad_norm": 0.6555175185203552,
+      "learning_rate": 9.97401266428502e-06,
+      "loss": 1.5634765625,
+      "step": 21
+    },
+    {
+      "epoch": 0.06918238993710692,
+      "grad_norm": 0.5500423908233643,
+      "learning_rate": 9.968561049466214e-06,
+      "loss": 1.6044921875,
+      "step": 22
+    },
+    {
+      "epoch": 0.07232704402515723,
+      "grad_norm": 0.4210081100463867,
+      "learning_rate": 9.96259251245514e-06,
+      "loss": 1.506103515625,
+      "step": 23
+    },
+    {
+      "epoch": 0.07547169811320754,
+      "grad_norm": 0.4597223401069641,
+      "learning_rate": 9.9561076742101e-06,
+      "loss": 1.5380859375,
+      "step": 24
+    },
+    {
+      "epoch": 0.07861635220125786,
+      "grad_norm": 0.5838366150856018,
+      "learning_rate": 9.949107209404664e-06,
+      "loss": 1.58203125,
+      "step": 25
+    },
+    {
+      "epoch": 0.08176100628930817,
+      "grad_norm": 0.5811012387275696,
+      "learning_rate": 9.941591846357467e-06,
+      "loss": 1.4892578125,
+      "step": 26
+    },
+    {
+      "epoch": 0.08490566037735849,
+      "grad_norm": 0.5185905694961548,
+      "learning_rate": 9.933562366956445e-06,
+      "loss": 1.463134765625,
+      "step": 27
+    },
+    {
+      "epoch": 0.0880503144654088,
+      "grad_norm": 0.4009871184825897,
+      "learning_rate": 9.925019606577486e-06,
+      "loss": 1.4609375,
+      "step": 28
+    },
+    {
+      "epoch": 0.09119496855345911,
+      "grad_norm": 0.4346908926963806,
+      "learning_rate": 9.915964453997516e-06,
+      "loss": 1.47900390625,
+      "step": 29
+    },
+    {
+      "epoch": 0.09433962264150944,
+      "grad_norm": 0.4531169831752777,
+      "learning_rate": 9.906397851302036e-06,
+      "loss": 1.435546875,
+      "step": 30
+    },
+    {
+      "epoch": 0.09748427672955975,
+      "grad_norm": 0.32488882541656494,
+      "learning_rate": 9.896320793787106e-06,
+      "loss": 1.35546875,
+      "step": 31
+    },
+    {
+      "epoch": 0.10062893081761007,
+      "grad_norm": 0.37860193848609924,
+      "learning_rate": 9.885734329855798e-06,
+      "loss": 1.43505859375,
+      "step": 32
+    },
+    {
+      "epoch": 0.10377358490566038,
+      "grad_norm": 0.4280271530151367,
+      "learning_rate": 9.874639560909118e-06,
+      "loss": 1.4111328125,
+      "step": 33
+    },
+    {
+      "epoch": 0.1069182389937107,
+      "grad_norm": 0.3741612732410431,
+      "learning_rate": 9.863037641231424e-06,
+      "loss": 1.3740234375,
+      "step": 34
+    },
+    {
+      "epoch": 0.11006289308176101,
+      "grad_norm": 0.2793862521648407,
+      "learning_rate": 9.850929777870324e-06,
+      "loss": 1.33642578125,
+      "step": 35
+    },
+    {
+      "epoch": 0.11320754716981132,
+      "grad_norm": 0.3218320310115814,
+      "learning_rate": 9.838317230511111e-06,
+      "loss": 1.393310546875,
+      "step": 36
+    },
+    {
+      "epoch": 0.11635220125786164,
+      "grad_norm": 0.3182618319988251,
+      "learning_rate": 9.8252013113457e-06,
+      "loss": 1.422607421875,
+      "step": 37
+    },
+    {
+      "epoch": 0.11949685534591195,
+      "grad_norm": 0.3342706561088562,
+      "learning_rate": 9.811583384936108e-06,
+      "loss": 1.34912109375,
+      "step": 38
+    },
+    {
+      "epoch": 0.12264150943396226,
+      "grad_norm": 0.23300714790821075,
+      "learning_rate": 9.797464868072489e-06,
+      "loss": 1.3857421875,
+      "step": 39
+    },
+    {
+      "epoch": 0.12578616352201258,
+      "grad_norm": 0.26877281069755554,
+      "learning_rate": 9.782847229625729e-06,
+      "loss": 1.34716796875,
+      "step": 40
+    },
+    {
+      "epoch": 0.1289308176100629,
+      "grad_norm": 0.20799706876277924,
+      "learning_rate": 9.767731990394638e-06,
+      "loss": 1.3388671875,
+      "step": 41
+    },
+    {
+      "epoch": 0.1320754716981132,
+      "grad_norm": 0.20554538071155548,
+      "learning_rate": 9.752120722947717e-06,
+      "loss": 1.3515625,
+      "step": 42
+    },
+    {
+      "epoch": 0.13522012578616352,
+      "grad_norm": 0.3131389915943146,
+      "learning_rate": 9.736015051459551e-06,
+      "loss": 1.32373046875,
+      "step": 43
+    },
+    {
+      "epoch": 0.13836477987421383,
+      "grad_norm": 0.15417590737342834,
+      "learning_rate": 9.719416651541839e-06,
+      "loss": 1.302734375,
+      "step": 44
+    },
+    {
+      "epoch": 0.14150943396226415,
+      "grad_norm": 0.22120970487594604,
+      "learning_rate": 9.702327250069058e-06,
+      "loss": 1.27978515625,
+      "step": 45
+    },
+    {
+      "epoch": 0.14465408805031446,
+      "grad_norm": 0.16401228308677673,
+      "learning_rate": 9.68474862499881e-06,
+      "loss": 1.24658203125,
+      "step": 46
+    },
+    {
+      "epoch": 0.14779874213836477,
+      "grad_norm": 0.17903250455856323,
+      "learning_rate": 9.666682605186834e-06,
+      "loss": 1.299560546875,
+      "step": 47
+    },
+    {
+      "epoch": 0.1509433962264151,
+      "grad_norm": 0.18163374066352844,
+      "learning_rate": 9.648131070196749e-06,
+      "loss": 1.261474609375,
+      "step": 48
+    },
+    {
+      "epoch": 0.1540880503144654,
+      "grad_norm": 0.14828112721443176,
+      "learning_rate": 9.62909595010449e-06,
+      "loss": 1.33056640625,
+      "step": 49
+    },
+    {
+      "epoch": 0.15723270440251572,
+      "grad_norm": 0.1566939502954483,
+      "learning_rate": 9.609579225297524e-06,
+      "loss": 1.2666015625,
+      "step": 50
+    },
+    {
+      "epoch": 0.16037735849056603,
+      "grad_norm": 0.17874041199684143,
+      "learning_rate": 9.589582926268798e-06,
+      "loss": 1.36767578125,
+      "step": 51
+    },
+    {
+      "epoch": 0.16352201257861634,
+      "grad_norm": 0.16513372957706451,
+      "learning_rate": 9.569109133405495e-06,
+      "loss": 1.3505859375,
+      "step": 52
+    },
+    {
+      "epoch": 0.16666666666666666,
+      "grad_norm": 0.14548315107822418,
+      "learning_rate": 9.548159976772593e-06,
+      "loss": 1.29736328125,
+      "step": 53
+    },
+    {
+      "epoch": 0.16981132075471697,
+      "grad_norm": 0.15943686664104462,
+      "learning_rate": 9.526737635891262e-06,
+      "loss": 1.25634765625,
+      "step": 54
+    },
+    {
+      "epoch": 0.17295597484276728,
+      "grad_norm": 0.13931553065776825,
+      "learning_rate": 9.504844339512096e-06,
+      "loss": 1.2578125,
+      "step": 55
+    },
+    {
+      "epoch": 0.1761006289308176,
+      "grad_norm": 0.14830654859542847,
+      "learning_rate": 9.482482365383254e-06,
+      "loss": 1.315673828125,
+      "step": 56
+    },
+    {
+      "epoch": 0.1792452830188679,
+      "grad_norm": 0.12663020193576813,
+      "learning_rate": 9.45965404001347e-06,
+      "loss": 1.24853515625,
+      "step": 57
+    },
+    {
+      "epoch": 0.18238993710691823,
+      "grad_norm": 0.1341799944639206,
+      "learning_rate": 9.436361738430016e-06,
+      "loss": 1.26416015625,
+      "step": 58
+    },
+    {
+      "epoch": 0.18553459119496854,
+      "grad_norm": 0.12654203176498413,
+      "learning_rate": 9.412607883931608e-06,
+      "loss": 1.26416015625,
+      "step": 59
+    },
+    {
+      "epoch": 0.18867924528301888,
+      "grad_norm": 0.12026432156562805,
+      "learning_rate": 9.388394947836278e-06,
+      "loss": 1.2392578125,
+      "step": 60
+    },
+    {
+      "epoch": 0.1918238993710692,
+      "grad_norm": 0.11050467193126678,
+      "learning_rate": 9.363725449224281e-06,
+      "loss": 1.21728515625,
+      "step": 61
+    },
+    {
+      "epoch": 0.1949685534591195,
+      "grad_norm": 0.11030342429876328,
+      "learning_rate": 9.338601954675995e-06,
+      "loss": 1.35595703125,
+      "step": 62
+    },
+    {
+      "epoch": 0.19811320754716982,
+      "grad_norm": 0.11015181988477707,
+      "learning_rate": 9.313027078004903e-06,
+      "loss": 1.197509765625,
+      "step": 63
+    },
+    {
+      "epoch": 0.20125786163522014,
+      "grad_norm": 0.11742514371871948,
+      "learning_rate": 9.287003479985667e-06,
+      "loss": 1.2265625,
+      "step": 64
+    },
+    {
+      "epoch": 0.20440251572327045,
+      "grad_norm": 0.12025771290063858,
+      "learning_rate": 9.260533868077283e-06,
+      "loss": 1.24658203125,
+      "step": 65
+    },
+    {
+      "epoch": 0.20754716981132076,
+      "grad_norm": 0.10835342109203339,
+      "learning_rate": 9.233620996141421e-06,
+      "loss": 1.20751953125,
+      "step": 66
+    },
+    {
+      "epoch": 0.21069182389937108,
+      "grad_norm": 0.10320407897233963,
+      "learning_rate": 9.206267664155906e-06,
+      "loss": 1.16796875,
+      "step": 67
+    },
+    {
+      "epoch": 0.2138364779874214,
+      "grad_norm": 0.10821900516748428,
+      "learning_rate": 9.178476717923415e-06,
+      "loss": 1.2431640625,
+      "step": 68
+    },
+    {
+      "epoch": 0.2169811320754717,
+      "grad_norm": 0.10739477723836899,
+      "learning_rate": 9.150251048775403e-06,
+      "loss": 1.251220703125,
+      "step": 69
+    },
+    {
+      "epoch": 0.22012578616352202,
+      "grad_norm": 0.108885757625103,
+      "learning_rate": 9.121593593271297e-06,
+      "loss": 1.35107421875,
+      "step": 70
+    },
+    {
+      "epoch": 0.22327044025157233,
+      "grad_norm": 0.10561460256576538,
+      "learning_rate": 9.092507332892968e-06,
+      "loss": 1.27880859375,
+      "step": 71
+    },
+    {
+      "epoch": 0.22641509433962265,
+      "grad_norm": 0.09498227387666702,
+      "learning_rate": 9.062995293734562e-06,
+      "loss": 1.2900390625,
+      "step": 72
+    },
+    {
+      "epoch": 0.22955974842767296,
+      "grad_norm": 0.10343296080827713,
+      "learning_rate": 9.033060546187651e-06,
+      "loss": 1.164794921875,
+      "step": 73
+    },
+    {
+      "epoch": 0.23270440251572327,
+      "grad_norm": 0.10015164315700531,
+      "learning_rate": 9.002706204621802e-06,
+      "loss": 1.28173828125,
+      "step": 74
+    },
+    {
+      "epoch": 0.2358490566037736,
+      "grad_norm": 0.10706444084644318,
+      "learning_rate": 8.971935427060563e-06,
+      "loss": 1.15380859375,
+      "step": 75
+    },
+    {
+      "epoch": 0.2389937106918239,
+      "grad_norm": 0.10046739131212234,
+      "learning_rate": 8.940751414852904e-06,
+      "loss": 1.25927734375,
+      "step": 76
+    },
+    {
+      "epoch": 0.24213836477987422,
+      "grad_norm": 0.09688299149274826,
+      "learning_rate": 8.90915741234015e-06,
+      "loss": 1.32666015625,
+      "step": 77
+    },
+    {
+      "epoch": 0.24528301886792453,
+      "grad_norm": 0.1397327482700348,
+      "learning_rate": 8.877156706518453e-06,
+      "loss": 1.2939453125,
+      "step": 78
+    },
+    {
+      "epoch": 0.24842767295597484,
+      "grad_norm": 0.10606326162815094,
+      "learning_rate": 8.84475262669681e-06,
+      "loss": 1.298828125,
+      "step": 79
+    },
+    {
+      "epoch": 0.25157232704402516,
+      "grad_norm": 0.10689855366945267,
+      "learning_rate": 8.811948544150693e-06,
+      "loss": 1.2236328125,
+      "step": 80
+    },
+    {
+      "epoch": 0.25471698113207547,
+      "grad_norm": 0.10143591463565826,
+      "learning_rate": 8.778747871771293e-06,
+      "loss": 1.173828125,
+      "step": 81
+    },
+    {
+      "epoch": 0.2578616352201258,
+      "grad_norm": 0.09580762684345245,
+      "learning_rate": 8.745154063710464e-06,
+      "loss": 1.23779296875,
+      "step": 82
+    },
+    {
+      "epoch": 0.2610062893081761,
+      "grad_norm": 0.09986191987991333,
+      "learning_rate": 8.71117061502135e-06,
+      "loss": 1.2890625,
+      "step": 83
+    },
+    {
+      "epoch": 0.2641509433962264,
+      "grad_norm": 0.09059023857116699,
+      "learning_rate": 8.676801061294764e-06,
+      "loss": 1.255859375,
+      "step": 84
+    },
+    {
+      "epoch": 0.2672955974842767,
+      "grad_norm": 0.09870278090238571,
+      "learning_rate": 8.642048978291347e-06,
+      "loss": 1.247802734375,
+      "step": 85
+    },
+    {
+      "epoch": 0.27044025157232704,
+      "grad_norm": 0.09271863102912903,
+      "learning_rate": 8.60691798156956e-06,
+      "loss": 1.2841796875,
+      "step": 86
+    },
+    {
+      "epoch": 0.27358490566037735,
+      "grad_norm": 0.10226648300886154,
+      "learning_rate": 8.571411726109518e-06,
+      "loss": 1.173583984375,
+      "step": 87
+    },
+    {
+      "epoch": 0.27672955974842767,
+      "grad_norm": 0.08855801820755005,
+      "learning_rate": 8.535533905932739e-06,
+      "loss": 1.27197265625,
+      "step": 88
+    },
+    {
+      "epoch": 0.279874213836478,
+      "grad_norm": 0.09819382429122925,
+      "learning_rate": 8.49928825371781e-06,
+      "loss": 1.2265625,
+      "step": 89
+    },
+    {
+      "epoch": 0.2830188679245283,
+      "grad_norm": 0.0931096225976944,
+      "learning_rate": 8.46267854041206e-06,
+      "loss": 1.2666015625,
+      "step": 90
+    },
+    {
+      "epoch": 0.2861635220125786,
+      "grad_norm": 0.09582605212926865,
+      "learning_rate": 8.425708574839221e-06,
+      "loss": 1.2294921875,
+      "step": 91
+    },
+    {
+      "epoch": 0.2893081761006289,
+      "grad_norm": 0.09605991095304489,
+      "learning_rate": 8.388382203303181e-06,
+      "loss": 1.26953125,
+      "step": 92
+    },
+    {
+      "epoch": 0.29245283018867924,
+      "grad_norm": 0.08945699781179428,
+      "learning_rate": 8.3507033091878e-06,
+      "loss": 1.26611328125,
+      "step": 93
+    },
+    {
+      "epoch": 0.29559748427672955,
+      "grad_norm": 0.09489528834819794,
+      "learning_rate": 8.312675812552898e-06,
+      "loss": 1.22314453125,
+      "step": 94
+    },
+    {
+      "epoch": 0.29874213836477986,
+      "grad_norm": 0.0868961364030838,
+      "learning_rate": 8.274303669726427e-06,
+      "loss": 1.24072265625,
+      "step": 95
+    },
+    {
+      "epoch": 0.3018867924528302,
+      "grad_norm": 0.09084431827068329,
+      "learning_rate": 8.235590872892837e-06,
+      "loss": 1.2958984375,
+      "step": 96
+    },
+    {
+      "epoch": 0.3050314465408805,
+      "grad_norm": 0.09020113945007324,
+      "learning_rate": 8.196541449677758e-06,
+      "loss": 1.23828125,
+      "step": 97
+    },
+    {
+      "epoch": 0.3081761006289308,
+      "grad_norm": 0.08751130104064941,
+      "learning_rate": 8.157159462728956e-06,
+      "loss": 1.2392578125,
+      "step": 98
+    },
+    {
+      "epoch": 0.3113207547169811,
+      "grad_norm": 0.09124515950679779,
+      "learning_rate": 8.117449009293668e-06,
+      "loss": 1.230712890625,
+      "step": 99
+    },
+    {
+      "epoch": 0.31446540880503143,
+      "grad_norm": 0.08627773076295853,
+      "learning_rate": 8.077414220792328e-06,
+      "loss": 1.222900390625,
+      "step": 100
+    },
+    {
+      "epoch": 0.31761006289308175,
+      "grad_norm": 0.10408709198236465,
+      "learning_rate": 8.03705926238874e-06,
+      "loss": 1.2490234375,
+      "step": 101
+    },
+    {
+      "epoch": 0.32075471698113206,
+      "grad_norm": 0.0939742848277092,
+      "learning_rate": 7.996388332556735e-06,
+      "loss": 1.19482421875,
+      "step": 102
+    },
+    {
+      "epoch": 0.3238993710691824,
+      "grad_norm": 0.19658835232257843,
+      "learning_rate": 7.955405662643384e-06,
+      "loss": 1.249267578125,
+      "step": 103
+    },
+    {
+      "epoch": 0.3270440251572327,
+      "grad_norm": 0.08333238214254379,
+      "learning_rate": 7.914115516428751e-06,
+      "loss": 1.2451171875,
+      "step": 104
+    },
+    {
+      "epoch": 0.330188679245283,
+      "grad_norm": 0.08837057650089264,
+      "learning_rate": 7.872522189682318e-06,
+      "loss": 1.27587890625,
+      "step": 105
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 0.09254736453294754,
+      "learning_rate": 7.830630009716038e-06,
+      "loss": 1.1962890625,
+      "step": 106
+    },
+    {
+      "epoch": 0.33647798742138363,
+      "grad_norm": 0.10213299095630646,
+      "learning_rate": 7.788443334934148e-06,
+      "loss": 1.24658203125,
+      "step": 107
+    },
+    {
+      "epoch": 0.33962264150943394,
+      "grad_norm": 0.08779237419366837,
+      "learning_rate": 7.745966554379708e-06,
+      "loss": 1.24365234375,
+      "step": 108
+    },
+    {
+      "epoch": 0.34276729559748426,
+      "grad_norm": 0.09156819432973862,
+      "learning_rate": 7.703204087277989e-06,
+      "loss": 1.26806640625,
+      "step": 109
+    },
+    {
+      "epoch": 0.34591194968553457,
+      "grad_norm": 0.0824931263923645,
+      "learning_rate": 7.660160382576683e-06,
+      "loss": 1.237060546875,
+      "step": 110
+    },
+    {
+      "epoch": 0.3490566037735849,
+      "grad_norm": 0.0844307392835617,
+      "learning_rate": 7.616839918483061e-06,
+      "loss": 1.21240234375,
+      "step": 111
+    },
+    {
+      "epoch": 0.3522012578616352,
+      "grad_norm": 0.0942101925611496,
+      "learning_rate": 7.573247201998051e-06,
+      "loss": 1.19580078125,
+      "step": 112
+    },
+    {
+      "epoch": 0.3553459119496855,
+      "grad_norm": 0.09428957849740982,
+      "learning_rate": 7.529386768447342e-06,
+      "loss": 1.212890625,
+      "step": 113
+    },
+    {
+      "epoch": 0.3584905660377358,
+      "grad_norm": 0.10116898268461227,
+      "learning_rate": 7.485263181009539e-06,
+      "loss": 1.2587890625,
+      "step": 114
+    },
+    {
+      "epoch": 0.36163522012578614,
+      "grad_norm": 0.08661499619483948,
+      "learning_rate": 7.440881030241407e-06,
+      "loss": 1.26025390625,
+      "step": 115
+    },
+    {
+      "epoch": 0.36477987421383645,
+      "grad_norm": 0.0860162153840065,
+      "learning_rate": 7.396244933600285e-06,
+      "loss": 1.236328125,
+      "step": 116
+    },
+    {
+      "epoch": 0.36792452830188677,
+      "grad_norm": 0.0877000018954277,
+      "learning_rate": 7.351359534963684e-06,
+      "loss": 1.26806640625,
+      "step": 117
+    },
+    {
+      "epoch": 0.3710691823899371,
+      "grad_norm": 0.08836546540260315,
+      "learning_rate": 7.306229504146154e-06,
+      "loss": 1.240234375,
+      "step": 118
+    },
+    {
+      "epoch": 0.3742138364779874,
+      "grad_norm": 0.0874360054731369,
+      "learning_rate": 7.260859536413429e-06,
+      "loss": 1.25146484375,
+      "step": 119
+    },
+    {
+      "epoch": 0.37735849056603776,
+      "grad_norm": 0.09354016184806824,
+      "learning_rate": 7.215254351993957e-06,
+      "loss": 1.267578125,
+      "step": 120
+    },
+    {
+      "epoch": 0.3805031446540881,
+      "grad_norm": 0.0862109363079071,
+      "learning_rate": 7.169418695587791e-06,
+      "loss": 1.191650390625,
+      "step": 121
+    },
+    {
+      "epoch": 0.3836477987421384,
+      "grad_norm": 0.08533702045679092,
+      "learning_rate": 7.123357335872982e-06,
+      "loss": 1.214111328125,
+      "step": 122
+    },
+    {
+      "epoch": 0.3867924528301887,
+      "grad_norm": 0.08776259422302246,
+      "learning_rate": 7.0770750650094335e-06,
+      "loss": 1.22802734375,
+      "step": 123
+    },
+    {
+      "epoch": 0.389937106918239,
+      "grad_norm": 0.09998597204685211,
+      "learning_rate": 7.0305766981403365e-06,
+      "loss": 1.2080078125,
+      "step": 124
+    },
+    {
+      "epoch": 0.39308176100628933,
+      "grad_norm": 0.0824577659368515,
+      "learning_rate": 6.983867072891213e-06,
+      "loss": 1.23583984375,
+      "step": 125
+    },
+    {
+      "epoch": 0.39622641509433965,
+      "grad_norm": 0.08219949156045914,
+      "learning_rate": 6.936951048866616e-06,
+      "loss": 1.22607421875,
+      "step": 126
+    },
+    {
+      "epoch": 0.39937106918238996,
+      "grad_norm": 0.08929724246263504,
+      "learning_rate": 6.889833507144534e-06,
+      "loss": 1.220458984375,
+      "step": 127
+    },
+    {
+      "epoch": 0.4025157232704403,
+      "grad_norm": 0.0950295552611351,
+      "learning_rate": 6.842519349768582e-06,
+      "loss": 1.1494140625,
+      "step": 128
+    },
+    {
+      "epoch": 0.4056603773584906,
+      "grad_norm": 0.09173066914081573,
+      "learning_rate": 6.7950134992379935e-06,
+      "loss": 1.1640625,
+      "step": 129
+    },
+    {
+      "epoch": 0.4088050314465409,
+      "grad_norm": 0.08214280754327774,
+      "learning_rate": 6.747320897995493e-06,
+      "loss": 1.228515625,
+      "step": 130
+    },
+    {
+      "epoch": 0.4119496855345912,
+      "grad_norm": 0.08603708446025848,
+      "learning_rate": 6.699446507913083e-06,
+      "loss": 1.25390625,
+      "step": 131
+    },
+    {
+      "epoch": 0.41509433962264153,
+      "grad_norm": 0.0935707688331604,
+      "learning_rate": 6.651395309775837e-06,
+      "loss": 1.18408203125,
+      "step": 132
+    },
+    {
+      "epoch": 0.41823899371069184,
+      "grad_norm": 0.08640452474355698,
+      "learning_rate": 6.6031723027636775e-06,
+      "loss": 1.2451171875,
+      "step": 133
+    },
+    {
+      "epoch": 0.42138364779874216,
+      "grad_norm": 0.08143847435712814,
+      "learning_rate": 6.554782503931298e-06,
+      "loss": 1.24267578125,
+      "step": 134
+    },
+    {
+      "epoch": 0.42452830188679247,
+      "grad_norm": 0.08352352678775787,
+      "learning_rate": 6.5062309476861714e-06,
+      "loss": 1.22216796875,
+      "step": 135
+    },
+    {
+      "epoch": 0.4276729559748428,
+      "grad_norm": 0.08868604898452759,
+      "learning_rate": 6.457522685264793e-06,
+      "loss": 1.2099609375,
+      "step": 136
+    },
+    {
+      "epoch": 0.4308176100628931,
+      "grad_norm": 0.08718673884868622,
+      "learning_rate": 6.408662784207149e-06,
+      "loss": 1.2470703125,
+      "step": 137
+    },
+    {
+      "epoch": 0.4339622641509434,
+      "grad_norm": 0.08005645871162415,
+      "learning_rate": 6.359656327829498e-06,
+      "loss": 1.241455078125,
+      "step": 138
+    },
+    {
+      "epoch": 0.4371069182389937,
+      "grad_norm": 0.09011710435152054,
+      "learning_rate": 6.310508414695511e-06,
+      "loss": 1.21142578125,
+      "step": 139
+    },
+    {
+      "epoch": 0.44025157232704404,
+      "grad_norm": 0.10179577022790909,
+      "learning_rate": 6.261224158085826e-06,
+      "loss": 1.26611328125,
+      "step": 140
+    },
+    {
+      "epoch": 0.44339622641509435,
+      "grad_norm": 0.08692909777164459,
+      "learning_rate": 6.211808685466063e-06,
+      "loss": 1.23388671875,
+      "step": 141
+    },
+    {
+      "epoch": 0.44654088050314467,
+      "grad_norm": 0.09201560169458389,
+      "learning_rate": 6.162267137953374e-06,
+      "loss": 1.24853515625,
+      "step": 142
+    },
+    {
+      "epoch": 0.449685534591195,
+      "grad_norm": 0.08598781377077103,
+      "learning_rate": 6.112604669781572e-06,
+      "loss": 1.2255859375,
+      "step": 143
+    },
+    {
+      "epoch": 0.4528301886792453,
+      "grad_norm": 0.08363979309797287,
+      "learning_rate": 6.062826447764883e-06,
+      "loss": 1.217041015625,
+      "step": 144
+    },
+    {
+      "epoch": 0.4559748427672956,
+      "grad_norm": 0.0804029256105423,
+      "learning_rate": 6.012937650760406e-06,
+      "loss": 1.2509765625,
+      "step": 145
+    },
+    {
+      "epoch": 0.4591194968553459,
+      "grad_norm": 0.08158931881189346,
+      "learning_rate": 5.962943469129303e-06,
+      "loss": 1.234375,
+      "step": 146
+    },
+    {
+      "epoch": 0.46226415094339623,
+      "grad_norm": 0.08835593611001968,
+      "learning_rate": 5.91284910419681e-06,
+      "loss": 1.27587890625,
+      "step": 147
+    },
+    {
+      "epoch": 0.46540880503144655,
+      "grad_norm": 0.10376080870628357,
+      "learning_rate": 5.862659767711094e-06,
+      "loss": 1.1826171875,
+      "step": 148
+    },
+    {
+      "epoch": 0.46855345911949686,
+      "grad_norm": 0.09770874679088593,
+      "learning_rate": 5.812380681301031e-06,
+      "loss": 1.1572265625,
+      "step": 149
+    },
+    {
+      "epoch": 0.4716981132075472,
+      "grad_norm": 0.10530182719230652,
+      "learning_rate": 5.762017075932952e-06,
+      "loss": 1.2587890625,
+      "step": 150
+    },
+    {
+      "epoch": 0.4748427672955975,
+      "grad_norm": 0.08281303942203522,
+      "learning_rate": 5.711574191366427e-06,
+      "loss": 1.1708984375,
+      "step": 151
+    },
+    {
+      "epoch": 0.4779874213836478,
+      "grad_norm": 0.08968009054660797,
+      "learning_rate": 5.66105727560912e-06,
+      "loss": 1.269775390625,
+      "step": 152
+    },
+    {
+      "epoch": 0.4811320754716981,
+      "grad_norm": 0.07928328961133957,
+      "learning_rate": 5.6104715843708e-06,
+      "loss": 1.21484375,
+      "step": 153
+    },
+    {
+      "epoch": 0.48427672955974843,
+      "grad_norm": 0.08762849122285843,
+      "learning_rate": 5.559822380516539e-06,
+      "loss": 1.220458984375,
+      "step": 154
+    },
+    {
+      "epoch": 0.48742138364779874,
+      "grad_norm": 0.08722134679555893,
+      "learning_rate": 5.509114933519179e-06,
+      "loss": 1.25634765625,
+      "step": 155
+    },
+    {
+      "epoch": 0.49056603773584906,
+      "grad_norm": 0.0900566354393959,
+      "learning_rate": 5.458354518911086e-06,
+      "loss": 1.236328125,
+      "step": 156
+    },
+    {
+      "epoch": 0.4937106918238994,
+      "grad_norm": 0.08496430516242981,
+      "learning_rate": 5.4075464177353165e-06,
+      "loss": 1.21630859375,
+      "step": 157
+    },
+    {
+      "epoch": 0.4968553459119497,
+      "grad_norm": 0.08240823447704315,
+      "learning_rate": 5.356695915996162e-06,
+      "loss": 1.22607421875,
+      "step": 158
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.08470958471298218,
+      "learning_rate": 5.3058083041092145e-06,
+      "loss": 1.22705078125,
+      "step": 159
+    },
+    {
+      "epoch": 0.5031446540880503,
+      "grad_norm": 0.08934212476015091,
+      "learning_rate": 5.25488887635095e-06,
+      "loss": 1.285400390625,
+      "step": 160
+    },
+    {
+      "epoch": 0.5062893081761006,
+      "grad_norm": 0.09710770845413208,
+      "learning_rate": 5.2039429303079294e-06,
+      "loss": 1.232421875,
+      "step": 161
+    },
+    {
+      "epoch": 0.5094339622641509,
+      "grad_norm": 0.08722967654466629,
+      "learning_rate": 5.152975766325631e-06,
+      "loss": 1.263427734375,
+      "step": 162
+    },
+    {
+      "epoch": 0.5125786163522013,
+      "grad_norm": 0.07902610301971436,
+      "learning_rate": 5.101992686957028e-06,
+      "loss": 1.25732421875,
+      "step": 163
+    },
+    {
+      "epoch": 0.5157232704402516,
+      "grad_norm": 0.08694876730442047,
+      "learning_rate": 5.050998996410899e-06,
+      "loss": 1.298095703125,
+      "step": 164
+    },
+    {
+      "epoch": 0.5188679245283019,
+      "grad_norm": 0.08544755727052689,
+      "learning_rate": 5e-06,
+      "loss": 1.208984375,
+      "step": 165
+    },
+    {
+      "epoch": 0.5220125786163522,
+      "grad_norm": 0.0834759846329689,
+      "learning_rate": 4.949001003589102e-06,
+      "loss": 1.23779296875,
+      "step": 166
+    },
+    {
+      "epoch": 0.5251572327044025,
+      "grad_norm": 0.08612226694822311,
+      "learning_rate": 4.898007313042975e-06,
+      "loss": 1.283935546875,
+      "step": 167
+    },
+    {
+      "epoch": 0.5283018867924528,
+      "grad_norm": 0.09189831465482712,
+      "learning_rate": 4.84702423367437e-06,
+      "loss": 1.17431640625,
+      "step": 168
+    },
+    {
+      "epoch": 0.5314465408805031,
+      "grad_norm": 0.09417322278022766,
+      "learning_rate": 4.796057069692073e-06,
+      "loss": 1.21875,
+      "step": 169
+    },
+    {
+      "epoch": 0.5345911949685535,
+      "grad_norm": 0.0859440341591835,
+      "learning_rate": 4.74511112364905e-06,
+      "loss": 1.237548828125,
+      "step": 170
+    },
+    {
+      "epoch": 0.5377358490566038,
+      "grad_norm": 0.09315012395381927,
+      "learning_rate": 4.694191695890788e-06,
+      "loss": 1.21923828125,
+      "step": 171
+    },
+    {
+      "epoch": 0.5408805031446541,
+      "grad_norm": 0.08770447224378586,
+      "learning_rate": 4.643304084003839e-06,
+      "loss": 1.22705078125,
+      "step": 172
+    },
+    {
+      "epoch": 0.5440251572327044,
+      "grad_norm": 0.08003748953342438,
+      "learning_rate": 4.592453582264684e-06,
+      "loss": 1.28173828125,
+      "step": 173
+    },
+    {
+      "epoch": 0.5471698113207547,
+      "grad_norm": 0.09152880311012268,
+      "learning_rate": 4.541645481088914e-06,
+      "loss": 1.24169921875,
+      "step": 174
+    },
+    {
+      "epoch": 0.550314465408805,
+      "grad_norm": 0.07619733363389969,
+      "learning_rate": 4.4908850664808245e-06,
+      "loss": 1.2568359375,
+      "step": 175
+    },
+    {
+      "epoch": 0.5534591194968553,
+      "grad_norm": 0.08338797092437744,
+      "learning_rate": 4.4401776194834615e-06,
+      "loss": 1.16650390625,
+      "step": 176
+    },
+    {
+      "epoch": 0.5566037735849056,
+      "grad_norm": 0.07812988758087158,
+      "learning_rate": 4.389528415629201e-06,
+      "loss": 1.24755859375,
+      "step": 177
+    },
+    {
+      "epoch": 0.559748427672956,
+      "grad_norm": 0.0926586166024208,
+      "learning_rate": 4.33894272439088e-06,
+      "loss": 1.214599609375,
+      "step": 178
+    },
+    {
+      "epoch": 0.5628930817610063,
+      "grad_norm": 0.10608647018671036,
+      "learning_rate": 4.2884258086335755e-06,
+      "loss": 1.16162109375,
+      "step": 179
+    },
+    {
+      "epoch": 0.5660377358490566,
+      "grad_norm": 0.0896821841597557,
+      "learning_rate": 4.23798292406705e-06,
+      "loss": 1.21923828125,
+      "step": 180
+    },
+    {
+      "epoch": 0.5691823899371069,
+      "grad_norm": 0.10245795547962189,
+      "learning_rate": 4.187619318698971e-06,
+      "loss": 1.207763671875,
+      "step": 181
+    },
+    {
+      "epoch": 0.5723270440251572,
+      "grad_norm": 0.0842718854546547,
+      "learning_rate": 4.137340232288908e-06,
+      "loss": 1.30224609375,
+      "step": 182
+    },
+    {
+      "epoch": 0.5754716981132075,
+      "grad_norm": 0.09193960577249527,
+      "learning_rate": 4.087150895803192e-06,
+      "loss": 1.218994140625,
+      "step": 183
+    },
+    {
+      "epoch": 0.5786163522012578,
+      "grad_norm": 0.08665217459201813,
+      "learning_rate": 4.0370565308706986e-06,
+      "loss": 1.1982421875,
+      "step": 184
+    },
+    {
+      "epoch": 0.5817610062893082,
+      "grad_norm": 0.0908685177564621,
+      "learning_rate": 3.987062349239596e-06,
+      "loss": 1.21240234375,
+      "step": 185
+    },
+    {
+      "epoch": 0.5849056603773585,
+      "grad_norm": 0.08689988404512405,
+      "learning_rate": 3.937173552235117e-06,
+      "loss": 1.296875,
+      "step": 186
+    },
+    {
+      "epoch": 0.5880503144654088,
+      "grad_norm": 0.11394862085580826,
+      "learning_rate": 3.887395330218429e-06,
+      "loss": 1.20361328125,
+      "step": 187
+    },
+    {
+      "epoch": 0.5911949685534591,
+      "grad_norm": 0.0949663296341896,
+      "learning_rate": 3.837732862046627e-06,
+      "loss": 1.263671875,
+      "step": 188
+    },
+    {
+      "epoch": 0.5943396226415094,
+      "grad_norm": 0.08149112015962601,
+      "learning_rate": 3.7881913145339387e-06,
+      "loss": 1.25390625,
+      "step": 189
+    },
+    {
+      "epoch": 0.5974842767295597,
+      "grad_norm": 0.09439687430858612,
+      "learning_rate": 3.738775841914175e-06,
+      "loss": 1.2255859375,
+      "step": 190
+    },
+    {
+      "epoch": 0.60062893081761,
+      "grad_norm": 0.0807260051369667,
+      "learning_rate": 3.689491585304491e-06,
+      "loss": 1.22998046875,
+      "step": 191
+    },
+    {
+      "epoch": 0.6037735849056604,
+      "grad_norm": 0.08422300219535828,
+      "learning_rate": 3.640343672170503e-06,
+      "loss": 1.23828125,
+      "step": 192
+    },
+    {
+      "epoch": 0.6069182389937107,
+      "grad_norm": 0.08983633667230606,
+      "learning_rate": 3.5913372157928515e-06,
+      "loss": 1.21337890625,
+      "step": 193
+    },
+    {
+      "epoch": 0.610062893081761,
+      "grad_norm": 0.09498164057731628,
+      "learning_rate": 3.5424773147352085e-06,
+      "loss": 1.17724609375,
+      "step": 194
+    },
+    {
+      "epoch": 0.6132075471698113,
+      "grad_norm": 0.0914149209856987,
+      "learning_rate": 3.4937690523138302e-06,
+      "loss": 1.232421875,
+      "step": 195
+    },
+    {
+      "epoch": 0.6163522012578616,
+      "grad_norm": 0.0758446678519249,
+      "learning_rate": 3.4452174960687033e-06,
+      "loss": 1.1982421875,
+      "step": 196
+    },
+    {
+      "epoch": 0.6194968553459119,
+      "grad_norm": 0.10985689610242844,
+      "learning_rate": 3.3968276972363224e-06,
+      "loss": 1.2041015625,
+      "step": 197
+    },
+    {
+      "epoch": 0.6226415094339622,
+      "grad_norm": 0.09818878769874573,
+      "learning_rate": 3.3486046902241663e-06,
+      "loss": 1.19287109375,
+      "step": 198
+    },
+    {
+      "epoch": 0.6257861635220126,
+      "grad_norm": 0.09397387504577637,
+      "learning_rate": 3.3005534920869175e-06,
+      "loss": 1.267578125,
+      "step": 199
+    },
+    {
+      "epoch": 0.6289308176100629,
+      "grad_norm": 0.09300831705331802,
+      "learning_rate": 3.252679102004509e-06,
+      "loss": 1.24462890625,
+      "step": 200
+    },
+    {
+      "epoch": 0.6320754716981132,
+      "grad_norm": 0.08829600363969803,
+      "learning_rate": 3.204986500762006e-06,
+      "loss": 1.25390625,
+      "step": 201
+    },
+    {
+      "epoch": 0.6352201257861635,
+      "grad_norm": 0.0840512067079544,
+      "learning_rate": 3.1574806502314206e-06,
+      "loss": 1.11474609375,
+      "step": 202
+    },
+    {
+      "epoch": 0.6383647798742138,
+      "grad_norm": 0.08670662343502045,
+      "learning_rate": 3.110166492855468e-06,
+      "loss": 1.203857421875,
+      "step": 203
+    },
+    {
+      "epoch": 0.6415094339622641,
+      "grad_norm": 0.08876918256282806,
+      "learning_rate": 3.063048951133386e-06,
+      "loss": 1.237060546875,
+      "step": 204
+    },
+    {
+      "epoch": 0.6446540880503144,
+      "grad_norm": 0.0912042111158371,
+      "learning_rate": 3.016132927108787e-06,
+      "loss": 1.20458984375,
+      "step": 205
+    },
+    {
+      "epoch": 0.6477987421383647,
+      "grad_norm": 0.09555654227733612,
+      "learning_rate": 2.9694233018596665e-06,
+      "loss": 1.240478515625,
+      "step": 206
+    },
+    {
+      "epoch": 0.6509433962264151,
+      "grad_norm": 0.10727142542600632,
+      "learning_rate": 2.9229249349905686e-06,
+      "loss": 1.22119140625,
+      "step": 207
+    },
+    {
+      "epoch": 0.6540880503144654,
+      "grad_norm": 0.08861692994832993,
+      "learning_rate": 2.8766426641270197e-06,
+      "loss": 1.1953125,
+      "step": 208
+    },
+    {
+      "epoch": 0.6572327044025157,
+      "grad_norm": 0.09431105107069016,
+      "learning_rate": 2.83058130441221e-06,
+      "loss": 1.26220703125,
+      "step": 209
+    },
+    {
+      "epoch": 0.660377358490566,
+      "grad_norm": 0.08846162259578705,
+      "learning_rate": 2.7847456480060476e-06,
+      "loss": 1.22021484375,
+      "step": 210
+    },
+    {
+      "epoch": 0.6635220125786163,
+      "grad_norm": 0.07520295679569244,
+      "learning_rate": 2.7391404635865725e-06,
+      "loss": 1.20068359375,
+      "step": 211
+    },
+    {
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.08306454122066498,
+      "learning_rate": 2.6937704958538483e-06,
+      "loss": 1.154296875,
+      "step": 212
+    },
+    {
+      "epoch": 0.6698113207547169,
+      "grad_norm": 0.08136342465877533,
+      "learning_rate": 2.648640465036316e-06,
+      "loss": 1.1875,
+      "step": 213
+    },
+    {
+      "epoch": 0.6729559748427673,
+      "grad_norm": 0.0894201323390007,
+      "learning_rate": 2.603755066399718e-06,
+      "loss": 1.272705078125,
+      "step": 214
+    },
+    {
+      "epoch": 0.6761006289308176,
+      "grad_norm": 0.10022783279418945,
+      "learning_rate": 2.559118969758595e-06,
+      "loss": 1.2265625,
+      "step": 215
+    },
+    {
+      "epoch": 0.6792452830188679,
+      "grad_norm": 0.0807405486702919,
+      "learning_rate": 2.514736818990463e-06,
+      "loss": 1.18994140625,
+      "step": 216
+    },
+    {
+      "epoch": 0.6823899371069182,
+      "grad_norm": 0.08904239535331726,
+      "learning_rate": 2.470613231552661e-06,
+      "loss": 1.11181640625,
+      "step": 217
+    },
+    {
+      "epoch": 0.6855345911949685,
+      "grad_norm": 0.08609310537576675,
+      "learning_rate": 2.4267527980019523e-06,
+      "loss": 1.216796875,
+      "step": 218
+    },
+    {
+      "epoch": 0.6886792452830188,
+      "grad_norm": 0.08292370289564133,
+      "learning_rate": 2.383160081516941e-06,
+      "loss": 1.2783203125,
+      "step": 219
+    },
+    {
+      "epoch": 0.6918238993710691,
+      "grad_norm": 0.10215222090482712,
+      "learning_rate": 2.339839617423318e-06,
+      "loss": 1.248046875,
+      "step": 220
+    },
+    {
+      "epoch": 0.6949685534591195,
+      "grad_norm": 0.08919369429349899,
+      "learning_rate": 2.296795912722014e-06,
+      "loss": 1.26953125,
+      "step": 221
+    },
+    {
+      "epoch": 0.6981132075471698,
+      "grad_norm": 0.09696005284786224,
+      "learning_rate": 2.254033445620293e-06,
+      "loss": 1.19677734375,
+      "step": 222
+    },
+    {
+      "epoch": 0.7012578616352201,
+      "grad_norm": 0.09767638891935349,
+      "learning_rate": 2.211556665065854e-06,
+      "loss": 1.23193359375,
+      "step": 223
+    },
+    {
+      "epoch": 0.7044025157232704,
+      "grad_norm": 0.08746252954006195,
+      "learning_rate": 2.169369990283963e-06,
+      "loss": 1.228515625,
+      "step": 224
+    },
+    {
+      "epoch": 0.7075471698113207,
+      "grad_norm": 0.0967244878411293,
+      "learning_rate": 2.1274778103176854e-06,
+      "loss": 1.240234375,
+      "step": 225
+    },
+    {
+      "epoch": 0.710691823899371,
+      "grad_norm": 0.08571658283472061,
+      "learning_rate": 2.08588448357125e-06,
+      "loss": 1.265625,
+      "step": 226
+    },
+    {
+      "epoch": 0.7138364779874213,
+      "grad_norm": 0.10917211323976517,
+      "learning_rate": 2.0445943373566178e-06,
+      "loss": 1.21875,
+      "step": 227
+    },
+    {
+      "epoch": 0.7169811320754716,
+      "grad_norm": 0.08573350310325623,
+      "learning_rate": 2.0036116674432653e-06,
+      "loss": 1.181884765625,
+      "step": 228
+    },
+    {
+      "epoch": 0.720125786163522,
+      "grad_norm": 0.09475314617156982,
+      "learning_rate": 1.962940737611264e-06,
+      "loss": 1.20849609375,
+      "step": 229
+    },
+    {
+      "epoch": 0.7232704402515723,
+      "grad_norm": 0.08965165168046951,
+      "learning_rate": 1.922585779207674e-06,
+      "loss": 1.17822265625,
+      "step": 230
+    },
+    {
+      "epoch": 0.7264150943396226,
+      "grad_norm": 0.08729615807533264,
+      "learning_rate": 1.8825509907063328e-06,
+      "loss": 1.27294921875,
+      "step": 231
+    },
+    {
+      "epoch": 0.7295597484276729,
+      "grad_norm": 0.09284593164920807,
+      "learning_rate": 1.8428405372710446e-06,
+      "loss": 1.2294921875,
+      "step": 232
+    },
+    {
+      "epoch": 0.7327044025157232,
+      "grad_norm": 0.0874868854880333,
+      "learning_rate": 1.8034585503222441e-06,
+      "loss": 1.11474609375,
+      "step": 233
+    },
+    {
+      "epoch": 0.7358490566037735,
+      "grad_norm": 0.08022027462720871,
+      "learning_rate": 1.7644091271071645e-06,
+      "loss": 1.214111328125,
+      "step": 234
+    },
+    {
+      "epoch": 0.7389937106918238,
+      "grad_norm": 0.0947025790810585,
+      "learning_rate": 1.7256963302735752e-06,
+      "loss": 1.19580078125,
+      "step": 235
+    },
+    {
+      "epoch": 0.7421383647798742,
+      "grad_norm": 0.07570719718933105,
+      "learning_rate": 1.687324187447102e-06,
+      "loss": 1.265625,
+      "step": 236
+    },
+    {
+      "epoch": 0.7452830188679245,
+      "grad_norm": 0.08627166599035263,
+      "learning_rate": 1.6492966908122033e-06,
+      "loss": 1.240234375,
+      "step": 237
+    },
+    {
+      "epoch": 0.7484276729559748,
+      "grad_norm": 0.09424252808094025,
+      "learning_rate": 1.611617796696821e-06,
+      "loss": 1.22412109375,
+      "step": 238
+    },
+    {
+      "epoch": 0.7515723270440252,
+      "grad_norm": 0.08505000174045563,
+      "learning_rate": 1.5742914251607794e-06,
+      "loss": 1.22509765625,
+      "step": 239
+    },
+    {
+      "epoch": 0.7547169811320755,
+      "grad_norm": 0.0981978178024292,
+      "learning_rate": 1.5373214595879416e-06,
+      "loss": 1.228515625,
+      "step": 240
+    },
+    {
+      "epoch": 0.7578616352201258,
+      "grad_norm": 0.08118585497140884,
+      "learning_rate": 1.500711746282192e-06,
+      "loss": 1.25537109375,
+      "step": 241
+    },
+    {
+      "epoch": 0.7610062893081762,
+      "grad_norm": 0.08585264533758163,
+      "learning_rate": 1.4644660940672628e-06,
+      "loss": 1.310791015625,
+      "step": 242
+    },
+    {
+      "epoch": 0.7641509433962265,
+      "grad_norm": 0.08260893076658249,
+      "learning_rate": 1.4285882738904822e-06,
+      "loss": 1.175048828125,
+      "step": 243
+    },
+    {
+      "epoch": 0.7672955974842768,
+      "grad_norm": 0.09124623984098434,
+      "learning_rate": 1.3930820184304423e-06,
+      "loss": 1.22314453125,
+      "step": 244
+    },
+    {
+      "epoch": 0.7704402515723271,
+      "grad_norm": 0.0838477611541748,
+      "learning_rate": 1.357951021708655e-06,
+      "loss": 1.238525390625,
+      "step": 245
+    },
+    {
+      "epoch": 0.7735849056603774,
+      "grad_norm": 0.10469858348369598,
+      "learning_rate": 1.323198938705238e-06,
+      "loss": 1.23486328125,
+      "step": 246
+    },
+    {
+      "epoch": 0.7767295597484277,
+      "grad_norm": 0.09047719091176987,
+      "learning_rate": 1.2888293849786503e-06,
+      "loss": 1.189697265625,
+      "step": 247
+    },
+    {
+      "epoch": 0.779874213836478,
+      "grad_norm": 0.09436509758234024,
+      "learning_rate": 1.2548459362895377e-06,
+      "loss": 1.19091796875,
+      "step": 248
+    },
+    {
+      "epoch": 0.7830188679245284,
+      "grad_norm": 0.09370195865631104,
+      "learning_rate": 1.2212521282287093e-06,
+      "loss": 1.20654296875,
+      "step": 249
+    },
+    {
+      "epoch": 0.7861635220125787,
+      "grad_norm": 0.09401333332061768,
+      "learning_rate": 1.188051455849309e-06,
+      "loss": 1.1904296875,
+      "step": 250
+    },
+    {
+      "epoch": 0.789308176100629,
+      "grad_norm": 0.08302223682403564,
+      "learning_rate": 1.1552473733031893e-06,
+      "loss": 1.224853515625,
+      "step": 251
+    },
+    {
+      "epoch": 0.7924528301886793,
+      "grad_norm": 0.08327268809080124,
+      "learning_rate": 1.1228432934815487e-06,
+      "loss": 1.2490234375,
+      "step": 252
+    },
+    {
+      "epoch": 0.7955974842767296,
+      "grad_norm": 0.09186363220214844,
+      "learning_rate": 1.0908425876598512e-06,
+      "loss": 1.2255859375,
+      "step": 253
+    },
+    {
+      "epoch": 0.7987421383647799,
+      "grad_norm": 0.10745052248239517,
+      "learning_rate": 1.0592485851470973e-06,
+      "loss": 1.23876953125,
+      "step": 254
+    },
+    {
+      "epoch": 0.8018867924528302,
+      "grad_norm": 0.09070909023284912,
+      "learning_rate": 1.0280645729394368e-06,
+      "loss": 1.21923828125,
+      "step": 255
+    },
+    {
+      "epoch": 0.8050314465408805,
+      "grad_norm": 0.09789802879095078,
+      "learning_rate": 9.972937953781985e-07,
+      "loss": 1.210693359375,
+      "step": 256
+    },
+    {
+      "epoch": 0.8081761006289309,
+      "grad_norm": 0.09813636541366577,
+      "learning_rate": 9.6693945381235e-07,
+      "loss": 1.248046875,
+      "step": 257
+    },
+    {
+      "epoch": 0.8113207547169812,
+      "grad_norm": 0.0862487405538559,
+      "learning_rate": 9.370047062654386e-07,
+      "loss": 1.1884765625,
+      "step": 258
+    },
+    {
+      "epoch": 0.8144654088050315,
+      "grad_norm": 0.08022520691156387,
+      "learning_rate": 9.074926671070322e-07,
+      "loss": 1.191650390625,
+      "step": 259
+    },
+    {
+      "epoch": 0.8176100628930818,
+      "grad_norm": 0.08337897062301636,
+      "learning_rate": 8.784064067287057e-07,
+      "loss": 1.1689453125,
+      "step": 260
+    },
+    {
+      "epoch": 0.8207547169811321,
+      "grad_norm": 0.09539865702390671,
+      "learning_rate": 8.497489512245971e-07,
+      "loss": 1.1767578125,
+      "step": 261
+    },
+    {
+      "epoch": 0.8238993710691824,
+      "grad_norm": 0.09431402385234833,
+      "learning_rate": 8.215232820765851e-07,
+      "loss": 1.2138671875,
+      "step": 262
+    },
+    {
+      "epoch": 0.8270440251572327,
+      "grad_norm": 0.10074929147958755,
+      "learning_rate": 7.937323358440935e-07,
+      "loss": 1.26904296875,
+      "step": 263
+    },
+    {
+      "epoch": 0.8301886792452831,
+      "grad_norm": 0.08571036159992218,
+      "learning_rate": 7.663790038585794e-07,
+      "loss": 1.23046875,
+      "step": 264
+    },
+    {
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.08719199895858765,
+      "learning_rate": 7.394661319227175e-07,
+      "loss": 1.23876953125,
+      "step": 265
+    },
+    {
+      "epoch": 0.8364779874213837,
+      "grad_norm": 0.08014900237321854,
+      "learning_rate": 7.129965200143335e-07,
+      "loss": 1.227294921875,
+      "step": 266
+    },
+    {
+      "epoch": 0.839622641509434,
+      "grad_norm": 0.1006547287106514,
+      "learning_rate": 6.86972921995096e-07,
+      "loss": 1.232421875,
+      "step": 267
+    },
+    {
+      "epoch": 0.8427672955974843,
+      "grad_norm": 0.09184549003839493,
+      "learning_rate": 6.613980453240065e-07,
+      "loss": 1.17724609375,
+      "step": 268
+    },
+    {
+      "epoch": 0.8459119496855346,
+      "grad_norm": 0.08853388577699661,
+      "learning_rate": 6.36274550775719e-07,
+      "loss": 1.205322265625,
+      "step": 269
+    },
+    {
+      "epoch": 0.8490566037735849,
+      "grad_norm": 0.08986873179674149,
+      "learning_rate": 6.116050521637218e-07,
+      "loss": 1.197265625,
+      "step": 270
+    },
+    {
+      "epoch": 0.8522012578616353,
+      "grad_norm": 0.09413354843854904,
+      "learning_rate": 5.873921160683943e-07,
+      "loss": 1.21044921875,
+      "step": 271
+    },
+    {
+      "epoch": 0.8553459119496856,
+      "grad_norm": 0.10209821909666061,
+      "learning_rate": 5.636382615699842e-07,
+      "loss": 1.228759765625,
+      "step": 272
+    },
+    {
+      "epoch": 0.8584905660377359,
+      "grad_norm": 0.09260933846235275,
+      "learning_rate": 5.403459599865307e-07,
+      "loss": 1.19091796875,
+      "step": 273
+    },
+    {
+      "epoch": 0.8616352201257862,
+      "grad_norm": 0.0983046144247055,
+      "learning_rate": 5.175176346167465e-07,
+      "loss": 1.22705078125,
+      "step": 274
+    },
+    {
+      "epoch": 0.8647798742138365,
+      "grad_norm": 0.08732131868600845,
+      "learning_rate": 4.951556604879049e-07,
+      "loss": 1.2109375,
+      "step": 275
+    },
+    {
+      "epoch": 0.8679245283018868,
+      "grad_norm": 0.09008695185184479,
+      "learning_rate": 4.732623641087403e-07,
+      "loss": 1.22607421875,
+      "step": 276
+    },
+    {
+      "epoch": 0.8710691823899371,
+      "grad_norm": 0.09183256328105927,
+      "learning_rate": 4.5184002322740784e-07,
+      "loss": 1.27099609375,
+      "step": 277
+    },
+    {
+      "epoch": 0.8742138364779874,
+      "grad_norm": 0.10036151111125946,
+      "learning_rate": 4.3089086659450774e-07,
+      "loss": 1.24609375,
+      "step": 278
+    },
+    {
+      "epoch": 0.8773584905660378,
+      "grad_norm": 0.08253729343414307,
+      "learning_rate": 4.1041707373120354e-07,
+      "loss": 1.240234375,
+      "step": 279
+    },
+    {
+      "epoch": 0.8805031446540881,
+      "grad_norm": 0.08661764860153198,
+      "learning_rate": 3.9042077470247574e-07,
+      "loss": 1.2265625,
+      "step": 280
+    },
+    {
+      "epoch": 0.8836477987421384,
+      "grad_norm": 0.11636807769536972,
+      "learning_rate": 3.709040498955102e-07,
+      "loss": 1.21630859375,
+      "step": 281
+    },
+    {
+      "epoch": 0.8867924528301887,
+      "grad_norm": 0.09158287197351456,
+      "learning_rate": 3.518689298032524e-07,
+      "loss": 1.1962890625,
+      "step": 282
+    },
+    {
+      "epoch": 0.889937106918239,
+      "grad_norm": 0.08876530826091766,
+      "learning_rate": 3.333173948131663e-07,
+      "loss": 1.27099609375,
+      "step": 283
+    },
+    {
+      "epoch": 0.8930817610062893,
+      "grad_norm": 0.08798261731863022,
+      "learning_rate": 3.1525137500119207e-07,
+      "loss": 1.24951171875,
+      "step": 284
+    },
+    {
+      "epoch": 0.8962264150943396,
+      "grad_norm": 0.08027750253677368,
+      "learning_rate": 2.9767274993094285e-07,
+      "loss": 1.24560546875,
+      "step": 285
+    },
+    {
+      "epoch": 0.89937106918239,
+      "grad_norm": 0.09272100776433945,
+      "learning_rate": 2.8058334845816214e-07,
+      "loss": 1.199462890625,
+      "step": 286
+    },
+    {
+      "epoch": 0.9025157232704403,
+      "grad_norm": 0.10138490796089172,
+      "learning_rate": 2.6398494854045055e-07,
+      "loss": 1.2041015625,
+      "step": 287
+    },
+    {
+      "epoch": 0.9056603773584906,
+      "grad_norm": 0.10046400874853134,
+      "learning_rate": 2.478792770522842e-07,
+      "loss": 1.17041015625,
+      "step": 288
+    },
+    {
+      "epoch": 0.9088050314465409,
+      "grad_norm": 0.08769627660512924,
+      "learning_rate": 2.32268009605362e-07,
+      "loss": 1.277587890625,
+      "step": 289
+    },
+    {
+      "epoch": 0.9119496855345912,
+      "grad_norm": 0.08884772658348083,
+      "learning_rate": 2.171527703742715e-07,
+      "loss": 1.254638671875,
+      "step": 290
+    },
+    {
+      "epoch": 0.9150943396226415,
+      "grad_norm": 0.07322319597005844,
+      "learning_rate": 2.0253513192751374e-07,
+      "loss": 1.20361328125,
+      "step": 291
+    },
+    {
+      "epoch": 0.9182389937106918,
+      "grad_norm": 0.08685608953237534,
+      "learning_rate": 1.884166150638933e-07,
+      "loss": 1.252685546875,
+      "step": 292
+    },
+    {
+      "epoch": 0.9213836477987422,
+      "grad_norm": 0.08980607241392136,
+      "learning_rate": 1.7479868865430072e-07,
+      "loss": 1.177734375,
+      "step": 293
+    },
+    {
+      "epoch": 0.9245283018867925,
+      "grad_norm": 0.0856226310133934,
+      "learning_rate": 1.6168276948889007e-07,
+      "loss": 1.2509765625,
+      "step": 294
+    },
+    {
+      "epoch": 0.9276729559748428,
+      "grad_norm": 0.0801997110247612,
+      "learning_rate": 1.4907022212967803e-07,
+      "loss": 1.2158203125,
+      "step": 295
+    },
+    {
+      "epoch": 0.9308176100628931,
+      "grad_norm": 0.08406123518943787,
+      "learning_rate": 1.3696235876857812e-07,
+      "loss": 1.2265625,
+      "step": 296
+    },
+    {
+      "epoch": 0.9339622641509434,
+      "grad_norm": 0.08864335715770721,
+      "learning_rate": 1.253604390908819e-07,
+      "loss": 1.2197265625,
+      "step": 297
+    },
+    {
+      "epoch": 0.9371069182389937,
+      "grad_norm": 0.10364779084920883,
+      "learning_rate": 1.1426567014420297e-07,
+      "loss": 1.198486328125,
+      "step": 298
+    },
+    {
+      "epoch": 0.940251572327044,
+      "grad_norm": 0.0869453102350235,
+      "learning_rate": 1.0367920621289496e-07,
+      "loss": 1.20751953125,
+      "step": 299
+    },
+    {
+      "epoch": 0.9433962264150944,
+      "grad_norm": 0.08727333694696426,
+      "learning_rate": 9.360214869796492e-08,
+      "loss": 1.207275390625,
+      "step": 300
+    },
+    {
+      "epoch": 0.9465408805031447,
+      "grad_norm": 0.09867426753044128,
+      "learning_rate": 8.403554600248498e-08,
+      "loss": 1.198974609375,
+      "step": 301
+    },
+    {
+      "epoch": 0.949685534591195,
+      "grad_norm": 0.10300202667713165,
+      "learning_rate": 7.498039342251573e-08,
+      "loss": 1.2158203125,
+      "step": 302
+    },
+    {
+      "epoch": 0.9528301886792453,
+      "grad_norm": 0.09961962699890137,
+      "learning_rate": 6.643763304355566e-08,
+      "loss": 1.24267578125,
+      "step": 303
+    },
+    {
+      "epoch": 0.9559748427672956,
+      "grad_norm": 0.08277103304862976,
+      "learning_rate": 5.8408153642533493e-08,
+      "loss": 1.27392578125,
+      "step": 304
+    },
+    {
+      "epoch": 0.9591194968553459,
+      "grad_norm": 0.09768108278512955,
+      "learning_rate": 5.089279059533658e-08,
+      "loss": 1.2109375,
+      "step": 305
+    },
+    {
+      "epoch": 0.9622641509433962,
+      "grad_norm": 0.08892697095870972,
+      "learning_rate": 4.389232578989988e-08,
+      "loss": 1.17431640625,
+      "step": 306
+    },
+    {
+      "epoch": 0.9654088050314465,
+      "grad_norm": 0.08239562809467316,
+      "learning_rate": 3.7407487544861565e-08,
+      "loss": 1.228759765625,
+      "step": 307
+    },
+    {
+      "epoch": 0.9685534591194969,
+      "grad_norm": 0.09559322148561478,
+      "learning_rate": 3.143895053378698e-08,
+      "loss": 1.22509765625,
+      "step": 308
+    },
+    {
+      "epoch": 0.9716981132075472,
+      "grad_norm": 0.09237507730722427,
+      "learning_rate": 2.59873357149798e-08,
+      "loss": 1.183837890625,
+      "step": 309
+    },
+    {
+      "epoch": 0.9748427672955975,
+      "grad_norm": 0.08496708422899246,
+      "learning_rate": 2.1053210266875346e-08,
+      "loss": 1.21435546875,
+      "step": 310
+    },
+    {
+      "epoch": 0.9779874213836478,
+      "grad_norm": 0.0905926525592804,
+      "learning_rate": 1.6637087529033925e-08,
+      "loss": 1.20458984375,
+      "step": 311
+    },
+    {
+      "epoch": 0.9811320754716981,
+      "grad_norm": 0.09607579559087753,
+      "learning_rate": 1.2739426948732426e-08,
+      "loss": 1.19677734375,
+      "step": 312
+    },
+    {
+      "epoch": 0.9842767295597484,
+      "grad_norm": 0.08372277021408081,
+      "learning_rate": 9.36063403316534e-09,
+      "loss": 1.118896484375,
+      "step": 313
+    },
+    {
+      "epoch": 0.9874213836477987,
+      "grad_norm": 0.08898620307445526,
+      "learning_rate": 6.501060307256835e-09,
+      "loss": 1.208984375,
+      "step": 314
+    },
+    {
+      "epoch": 0.9905660377358491,
+      "grad_norm": 0.0899147167801857,
+      "learning_rate": 4.161003277085574e-09,
+      "loss": 1.23486328125,
+      "step": 315
+    },
+    {
+      "epoch": 0.9937106918238994,
+      "grad_norm": 0.09119053184986115,
+      "learning_rate": 2.3407063989361324e-09,
+      "loss": 1.259521484375,
+      "step": 316
+    },
+    {
+      "epoch": 0.9968553459119497,
+      "grad_norm": 0.08343734592199326,
+      "learning_rate": 1.040359053967599e-09,
+      "loss": 1.231689453125,
+      "step": 317
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 0.0955134704709053,
+      "learning_rate": 2.6009652851211044e-10,
+      "loss": 1.25927734375,
+      "step": 318
+    },
+    {
+      "epoch": 1.0,
+      "eval_loss": 1.35986328125,
+      "eval_runtime": 10.9098,
+      "eval_samples_per_second": 0.55,
+      "eval_steps_per_second": 0.092,
+      "step": 318
+    }
+  ],
+  "logging_steps": 1.0,
+  "max_steps": 318,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 0,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 8.935569965579239e+18,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}