Instructions to use jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200")
model = AutoModelForCausalLM.from_pretrained("jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200

SGLang

How to use jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200 with Docker Model Runner:
```
docker model run hf.co/jackf857/llama-3-8b-base-r-dpo-ultrafeedback-4xh200
```

jackf857 commited on Apr 20

Commit

cebf074

verified ·

1 Parent(s): d20a7e9

End of training

Browse files

Files changed (12) hide show

README.md +8 -8
all_results.json +12 -12
eval_results.json +9 -9
model-00001-of-00007.safetensors +1 -1
model-00002-of-00007.safetensors +1 -1
model-00003-of-00007.safetensors +1 -1
model-00004-of-00007.safetensors +1 -1
model-00005-of-00007.safetensors +1 -1
model-00006-of-00007.safetensors +1 -1
model-00007-of-00007.safetensors +1 -1
train_results.json +3 -3
trainer_state.json +351 -351

README.md CHANGED Viewed

@@ -19,17 +19,17 @@ should probably proofread and complete it, then remove this comment. -->
 This model is a fine-tuned version of [W-61/llama-3-8b-base-sft-ultrachat-8xh200](https://huggingface.co/W-61/llama-3-8b-base-sft-ultrachat-8xh200) on the HuggingFaceH4/ultrafeedback_binarized dataset.
 It achieves the following results on the evaluation set:
-- Loss: 0.7402
 - R Dpo/chosen Len: 291.2620
 - R Dpo/rejected Len: 248.3960
 - R Dpo/length Delta: 42.8660
-- R Dpo/regularization Term: 2.1433
-- Logps/chosen: -396.0221
-- Logps/rejected: -365.0935
 - Logps/ref Chosen: -289.1346
 - Logps/ref Rejected: -264.7782
-- Logits/chosen: -0.4675
-- Logits/rejected: -0.4732
 ## Model description
@@ -66,8 +66,8 @@ The following hyperparameters were used during training:
 | Training Loss | Epoch  | Step | Validation Loss | R Dpo/chosen Len | R Dpo/rejected Len | R Dpo/length Delta | R Dpo/regularization Term | Logps/chosen | Logps/rejected | Logps/ref Chosen | Logps/ref Rejected | Logits/chosen | Logits/rejected |
 |:-------------:|:------:|:----:|:---------------:|:----------------:|:------------------:|:------------------:|:-------------------------:|:------------:|:--------------:|:----------------:|:------------------:|:-------------:|:---------------:|
-| 6.5508        | 0.4188 | 200  | 0.8642          | 291.2620         | 248.3960           | 42.8660            | 2.1433                    | -383.5491    | -354.5439      | -289.1346        | -264.7782          | -0.4487       | -0.4526         |
-| 5.5495        | 0.8377 | 400  | 0.7402          | 291.2620         | 248.3960           | 42.8660            | 2.1433                    | -396.0221    | -365.0935      | -289.1346        | -264.7782          | -0.4675       | -0.4732         |
 ### Framework versions

 This model is a fine-tuned version of [W-61/llama-3-8b-base-sft-ultrachat-8xh200](https://huggingface.co/W-61/llama-3-8b-base-sft-ultrachat-8xh200) on the HuggingFaceH4/ultrafeedback_binarized dataset.
 It achieves the following results on the evaluation set:
+- Loss: 0.5080
 - R Dpo/chosen Len: 291.2620
 - R Dpo/rejected Len: 248.3960
 - R Dpo/length Delta: 42.8660
+- R Dpo/regularization Term: 0.0
+- Logps/chosen: -288.0679
+- Logps/rejected: -272.9751
 - Logps/ref Chosen: -289.1346
 - Logps/ref Rejected: -264.7782
+- Logits/chosen: -0.7442
+- Logits/rejected: -0.7460
 ## Model description
 | Training Loss | Epoch  | Step | Validation Loss | R Dpo/chosen Len | R Dpo/rejected Len | R Dpo/length Delta | R Dpo/regularization Term | Logps/chosen | Logps/rejected | Logps/ref Chosen | Logps/ref Rejected | Logits/chosen | Logits/rejected |
 |:-------------:|:------:|:----:|:---------------:|:----------------:|:------------------:|:------------------:|:-------------------------:|:------------:|:--------------:|:----------------:|:------------------:|:-------------:|:---------------:|
+| 4.178         | 0.4188 | 200  | 0.5266          | 291.2620         | 248.3960           | 42.8660            | 0.0                       | -287.2393    | -271.3152      | -289.1346        | -264.7782          | -0.7479       | -0.7490         |
+| 4.0423        | 0.8377 | 400  | 0.5080          | 291.2620         | 248.3960           | 42.8660            | 0.0                       | -288.0679    | -272.9751      | -289.1346        | -264.7782          | -0.7442       | -0.7460         |
 ### Framework versions

all_results.json CHANGED Viewed

@@ -1,24 +1,24 @@
 {
     "epoch": 0.9989528795811519,
-    "eval_logits/chosen": -0.4671335220336914,
-    "eval_logits/rejected": -0.4723200798034668,
-    "eval_logps/chosen": -392.7605895996094,
     "eval_logps/ref_chosen": -289.1346435546875,
     "eval_logps/ref_rejected": -264.7782287597656,
-    "eval_logps/rejected": -363.01165771484375,
-    "eval_loss": 0.7231313586235046,
     "eval_r_dpo/chosen_len": 291.2619934082031,
     "eval_r_dpo/length_delta": 42.86600112915039,
-    "eval_r_dpo/regularization_term": 2.1433000564575195,
     "eval_r_dpo/rejected_len": 248.39599609375,
-    "eval_runtime": 81.8341,
     "eval_samples": 2000,
-    "eval_samples_per_second": 24.44,
-    "eval_steps_per_second": 1.527,
     "total_flos": 0.0,
-    "train_loss": 9.076586509400693,
-    "train_runtime": 6089.4941,
     "train_samples": 61135,
-    "train_samples_per_second": 10.039,
     "train_steps_per_second": 0.078
 }

 {
     "epoch": 0.9989528795811519,
+    "eval_logits/chosen": -0.7394673824310303,
+    "eval_logits/rejected": -0.7407819628715515,
+    "eval_logps/chosen": -288.1478271484375,
     "eval_logps/ref_chosen": -289.1346435546875,
     "eval_logps/ref_rejected": -264.7782287597656,
+    "eval_logps/rejected": -273.09014892578125,
+    "eval_loss": 0.5070626139640808,
     "eval_r_dpo/chosen_len": 291.2619934082031,
     "eval_r_dpo/length_delta": 42.86600112915039,
+    "eval_r_dpo/regularization_term": 0.0,
     "eval_r_dpo/rejected_len": 248.39599609375,
+    "eval_runtime": 81.9848,
     "eval_samples": 2000,
+    "eval_samples_per_second": 24.395,
+    "eval_steps_per_second": 1.525,
     "total_flos": 0.0,
+    "train_loss": 4.32008807264284,
+    "train_runtime": 6099.6278,
     "train_samples": 61135,
+    "train_samples_per_second": 10.023,
     "train_steps_per_second": 0.078
 }

eval_results.json CHANGED Viewed

@@ -1,18 +1,18 @@
 {
     "epoch": 0.9989528795811519,
-    "eval_logits/chosen": -0.4671335220336914,
-    "eval_logits/rejected": -0.4723200798034668,
-    "eval_logps/chosen": -392.7605895996094,
     "eval_logps/ref_chosen": -289.1346435546875,
     "eval_logps/ref_rejected": -264.7782287597656,
-    "eval_logps/rejected": -363.01165771484375,
-    "eval_loss": 0.7231313586235046,
     "eval_r_dpo/chosen_len": 291.2619934082031,
     "eval_r_dpo/length_delta": 42.86600112915039,
-    "eval_r_dpo/regularization_term": 2.1433000564575195,
     "eval_r_dpo/rejected_len": 248.39599609375,
-    "eval_runtime": 81.8341,
     "eval_samples": 2000,
-    "eval_samples_per_second": 24.44,
-    "eval_steps_per_second": 1.527
 }

 {
     "epoch": 0.9989528795811519,
+    "eval_logits/chosen": -0.7394673824310303,
+    "eval_logits/rejected": -0.7407819628715515,
+    "eval_logps/chosen": -288.1478271484375,
     "eval_logps/ref_chosen": -289.1346435546875,
     "eval_logps/ref_rejected": -264.7782287597656,
+    "eval_logps/rejected": -273.09014892578125,
+    "eval_loss": 0.5070626139640808,
     "eval_r_dpo/chosen_len": 291.2619934082031,
     "eval_r_dpo/length_delta": 42.86600112915039,
+    "eval_r_dpo/regularization_term": 0.0,
     "eval_r_dpo/rejected_len": 248.39599609375,
+    "eval_runtime": 81.9848,
     "eval_samples": 2000,
+    "eval_samples_per_second": 24.395,
+    "eval_steps_per_second": 1.525
 }

model-00001-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3d6523f5f80607bf2607784a75990a261b374b2a12171c88805ed3928343dd2c
 size 4886466168

 version https://git-lfs.github.com/spec/v1
+oid sha256:e4af25c24cbbbec408efbf3243a7870e030d3c563899066bef1ab333eca8cde2
 size 4886466168

model-00002-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:09b6a9eed227e62bc63ad02e2f7c82ec2dc3371c142db22d7ca807b47460881e
 size 4832007448

 version https://git-lfs.github.com/spec/v1
+oid sha256:2fb1c3ca3e1629e1f2be0b945ae20667861b4d53fb6cc1b9cadcef7fff48fab8
 size 4832007448

model-00003-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:127bc79a18e3cf6285815e21cc3c84fe4391da423cf100dd9c47db09a5d41988
 size 4999813112

 version https://git-lfs.github.com/spec/v1
+oid sha256:ef1bb58a5a53c29f1b245acaf023b18f3078b096c6d0a0f8eadd6573e9629ab3
 size 4999813112

model-00004-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f98834df155a4e0b347c4b7988ffdbcddf78fad607b1be66cae6b23f0bacdc22
 size 4999813128

 version https://git-lfs.github.com/spec/v1
+oid sha256:e084916b38d2156579312b165da8c8ec42a4036b78e292d9a1f29c702fe6c8b1
 size 4999813128

model-00005-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e3c21b4bd4668d77d733c6297392551acdf3129edae066de160c37904031f09
 size 4832007496

 version https://git-lfs.github.com/spec/v1
+oid sha256:7a809250d30eab837e1d0dce893711087df73ac9e3511fe2319af1991948ad29
 size 4832007496

model-00006-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:db9c7930d11f00d98b096cf16ef38932f0695592f699cf02b09fa5c3362a4bdb
 size 4999813120

 version https://git-lfs.github.com/spec/v1
+oid sha256:6aee84b0d0695de5e61d419e9497cb4452ccf94fb837d7942a3f163103e85c0b
 size 4999813120

model-00007-of-00007.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f6e64fddba6538729c890324361f4954d18860b1dc80e4fb86d419d5419ecb47
 size 2571158184

 version https://git-lfs.github.com/spec/v1
+oid sha256:7c91d53c4580564ae5626f15f8416938532482cd8493bc7996c92074068f8793
 size 2571158184

train_results.json CHANGED Viewed

@@ -1,9 +1,9 @@
 {
     "epoch": 0.9989528795811519,
     "total_flos": 0.0,
-    "train_loss": 9.076586509400693,
-    "train_runtime": 6089.4941,
     "train_samples": 61135,
-    "train_samples_per_second": 10.039,
     "train_steps_per_second": 0.078
 }

 {
     "epoch": 0.9989528795811519,
     "total_flos": 0.0,
+    "train_loss": 4.32008807264284,
+    "train_runtime": 6099.6278,
     "train_samples": 61135,
+    "train_samples_per_second": 10.023,
     "train_steps_per_second": 0.078
 }

trainer_state.json CHANGED Viewed

@@ -11,7 +11,7 @@
   "log_history": [
     {
       "epoch": 0.0020942408376963353,
-      "grad_norm": 368.09649658203125,
       "learning_rate": 0.0,
       "logits/chosen": -0.5995081663131714,
       "logits/rejected": -0.6144353747367859,
@@ -19,845 +19,845 @@
       "logps/ref_chosen": -267.5935363769531,
       "logps/ref_rejected": -204.2306671142578,
       "logps/rejected": -204.23907470703125,
-      "loss": 22.5561,
       "r_dpo/chosen_len": 257.75,
       "r_dpo/length_delta": 47.875,
-      "r_dpo/regularization_term": 2.3937501907348633,
       "r_dpo/rejected_len": 209.875,
       "step": 1
     },
     {
       "epoch": 0.020942408376963352,
-      "grad_norm": 340.15557861328125,
       "learning_rate": 9.375e-08,
-      "logits/chosen": -0.6315425634384155,
-      "logits/rejected": -0.6362923979759216,
-      "logps/chosen": -296.7152404785156,
       "logps/ref_chosen": -296.72576904296875,
       "logps/ref_rejected": -258.9681396484375,
-      "logps/rejected": -259.0311279296875,
-      "loss": 21.6099,
       "r_dpo/chosen_len": 291.8680419921875,
       "r_dpo/length_delta": 49.76388931274414,
-      "r_dpo/regularization_term": 2.488194465637207,
       "r_dpo/rejected_len": 242.1041717529297,
       "step": 10
     },
     {
       "epoch": 0.041884816753926704,
-      "grad_norm": 340.13922119140625,
       "learning_rate": 1.9791666666666664e-07,
-      "logits/chosen": -0.5944095253944397,
-      "logits/rejected": -0.6244563460350037,
-      "logps/chosen": -298.0061340332031,
       "logps/ref_chosen": -297.9349365234375,
       "logps/ref_rejected": -256.9902648925781,
-      "logps/rejected": -257.0635681152344,
-      "loss": 21.2111,
       "r_dpo/chosen_len": 291.29998779296875,
       "r_dpo/length_delta": 52.89374923706055,
-      "r_dpo/regularization_term": 2.6446876525878906,
       "r_dpo/rejected_len": 238.40625,
       "step": 20
     },
     {
       "epoch": 0.06282722513089005,
-      "grad_norm": 393.5495300292969,
       "learning_rate": 3.020833333333333e-07,
-      "logits/chosen": -0.6105726957321167,
-      "logits/rejected": -0.6022344827651978,
-      "logps/chosen": -278.83709716796875,
       "logps/ref_chosen": -278.64752197265625,
       "logps/ref_rejected": -249.309814453125,
-      "logps/rejected": -249.5636749267578,
-      "loss": 22.3384,
       "r_dpo/chosen_len": 270.8812561035156,
       "r_dpo/length_delta": 25.228124618530273,
-      "r_dpo/regularization_term": 1.2614065408706665,
       "r_dpo/rejected_len": 245.6531219482422,
       "step": 30
     },
     {
       "epoch": 0.08376963350785341,
-      "grad_norm": 313.8265686035156,
       "learning_rate": 4.0625e-07,
-      "logits/chosen": -0.6067706346511841,
-      "logits/rejected": -0.6315016746520996,
-      "logps/chosen": -284.3418273925781,
       "logps/ref_chosen": -283.49981689453125,
       "logps/ref_rejected": -265.32733154296875,
-      "logps/rejected": -266.49444580078125,
-      "loss": 21.1039,
       "r_dpo/chosen_len": 281.43438720703125,
       "r_dpo/length_delta": 33.34375,
-      "r_dpo/regularization_term": 1.6671874523162842,
       "r_dpo/rejected_len": 248.0906219482422,
       "step": 40
     },
     {
       "epoch": 0.10471204188481675,
-      "grad_norm": 358.73443603515625,
       "learning_rate": 4.999932966293553e-07,
-      "logits/chosen": -0.6113789081573486,
-      "logits/rejected": -0.652940571308136,
-      "logps/chosen": -283.0608825683594,
       "logps/ref_chosen": -280.224365234375,
       "logps/ref_rejected": -274.3541259765625,
-      "logps/rejected": -277.8731384277344,
-      "loss": 21.4782,
       "r_dpo/chosen_len": 290.32501220703125,
       "r_dpo/length_delta": 35.11249923706055,
-      "r_dpo/regularization_term": 1.7556250095367432,
       "r_dpo/rejected_len": 255.21249389648438,
       "step": 50
     },
     {
       "epoch": 0.1256544502617801,
-      "grad_norm": 423.79644775390625,
       "learning_rate": 4.991893270335525e-07,
-      "logits/chosen": -0.6010288596153259,
-      "logits/rejected": -0.61609947681427,
-      "logps/chosen": -289.21649169921875,
       "logps/ref_chosen": -281.12664794921875,
       "logps/ref_rejected": -259.86456298828125,
-      "logps/rejected": -268.3977355957031,
-      "loss": 21.018,
       "r_dpo/chosen_len": 273.953125,
       "r_dpo/length_delta": 29.084375381469727,
-      "r_dpo/regularization_term": 1.454218864440918,
       "r_dpo/rejected_len": 244.86874389648438,
       "step": 60
     },
     {
       "epoch": 0.14659685863874344,
-      "grad_norm": 370.2671813964844,
       "learning_rate": 4.970496218214204e-07,
-      "logits/chosen": -0.6249256730079651,
-      "logits/rejected": -0.6328210234642029,
-      "logps/chosen": -305.2099609375,
       "logps/ref_chosen": -287.71063232421875,
       "logps/ref_rejected": -276.839599609375,
-      "logps/rejected": -294.93865966796875,
-      "loss": 17.2466,
       "r_dpo/chosen_len": 267.4937438964844,
       "r_dpo/length_delta": 14.484375,
-      "r_dpo/regularization_term": 0.7242187857627869,
       "r_dpo/rejected_len": 253.00936889648438,
       "step": 70
     },
     {
       "epoch": 0.16753926701570682,
-      "grad_norm": 417.7720947265625,
       "learning_rate": 4.935856505068998e-07,
-      "logits/chosen": -0.5453386306762695,
-      "logits/rejected": -0.5457024574279785,
-      "logps/chosen": -310.01947021484375,
       "logps/ref_chosen": -280.123046875,
       "logps/ref_rejected": -258.8989562988281,
-      "logps/rejected": -286.91168212890625,
-      "loss": 15.9909,
       "r_dpo/chosen_len": 267.4781188964844,
       "r_dpo/length_delta": 32.46562576293945,
-      "r_dpo/regularization_term": 1.6232811212539673,
       "r_dpo/rejected_len": 235.0124969482422,
       "step": 80
     },
     {
       "epoch": 0.18848167539267016,
-      "grad_norm": 488.6341857910156,
       "learning_rate": 4.8881598109976e-07,
-      "logits/chosen": -0.500390887260437,
-      "logits/rejected": -0.5129397511482239,
-      "logps/chosen": -321.3196716308594,
       "logps/ref_chosen": -278.02545166015625,
       "logps/ref_rejected": -251.0922393798828,
-      "logps/rejected": -291.7796325683594,
-      "loss": 13.228,
       "r_dpo/chosen_len": 274.20623779296875,
       "r_dpo/length_delta": 44.97187423706055,
-      "r_dpo/regularization_term": 2.248593807220459,
       "r_dpo/rejected_len": 229.234375,
       "step": 90
     },
     {
       "epoch": 0.2094240837696335,
-      "grad_norm": 601.153076171875,
       "learning_rate": 4.827661805750437e-07,
-      "logits/chosen": -0.45114779472351074,
-      "logits/rejected": -0.46944254636764526,
-      "logps/chosen": -328.85711669921875,
       "logps/ref_chosen": -274.0089416503906,
       "logps/ref_rejected": -274.14447021484375,
-      "logps/rejected": -332.6839599609375,
-      "loss": 11.6202,
       "r_dpo/chosen_len": 275.3343811035156,
       "r_dpo/length_delta": 21.912500381469727,
-      "r_dpo/regularization_term": 1.0956249237060547,
       "r_dpo/rejected_len": 253.421875,
       "step": 100
     },
     {
       "epoch": 0.23036649214659685,
-      "grad_norm": 591.3831176757812,
       "learning_rate": 4.75468677825789e-07,
-      "logits/chosen": -0.4335498809814453,
-      "logits/rejected": -0.4618416428565979,
-      "logps/chosen": -345.44000244140625,
       "logps/ref_chosen": -273.23333740234375,
       "logps/ref_rejected": -263.88787841796875,
-      "logps/rejected": -331.32720947265625,
-      "loss": 9.6059,
       "r_dpo/chosen_len": 283.43438720703125,
       "r_dpo/length_delta": 50.34375,
-      "r_dpo/regularization_term": 2.5171875953674316,
       "r_dpo/rejected_len": 233.0906219482422,
       "step": 110
     },
     {
       "epoch": 0.2513089005235602,
-      "grad_norm": 456.3512878417969,
       "learning_rate": 4.669625898336438e-07,
-      "logits/chosen": -0.4128504693508148,
-      "logits/rejected": -0.4164280891418457,
-      "logps/chosen": -344.1505432128906,
       "logps/ref_chosen": -269.54656982421875,
       "logps/ref_rejected": -272.7981872558594,
-      "logps/rejected": -348.3555908203125,
-      "loss": 9.6405,
       "r_dpo/chosen_len": 264.7593688964844,
       "r_dpo/length_delta": 13.840624809265137,
-      "r_dpo/regularization_term": 0.6920312643051147,
       "r_dpo/rejected_len": 250.9187469482422,
       "step": 120
     },
     {
       "epoch": 0.27225130890052357,
-      "grad_norm": 656.2766723632812,
       "learning_rate": 4.5729351198915705e-07,
-      "logits/chosen": -0.4158397316932678,
-      "logits/rejected": -0.40780720114707947,
-      "logps/chosen": -346.33355712890625,
       "logps/ref_chosen": -275.03448486328125,
       "logps/ref_rejected": -276.39862060546875,
-      "logps/rejected": -350.0035705566406,
-      "loss": 9.8093,
       "r_dpo/chosen_len": 266.625,
       "r_dpo/length_delta": 18.668750762939453,
-      "r_dpo/regularization_term": 0.9334374666213989,
       "r_dpo/rejected_len": 247.9562530517578,
       "step": 130
     },
     {
       "epoch": 0.2931937172774869,
-      "grad_norm": 693.1464233398438,
       "learning_rate": 4.4651327368569684e-07,
-      "logits/chosen": -0.4354820251464844,
-      "logits/rejected": -0.44006985425949097,
-      "logps/chosen": -348.00384521484375,
       "logps/ref_chosen": -276.0029602050781,
       "logps/ref_rejected": -255.9320526123047,
-      "logps/rejected": -328.1044921875,
-      "loss": 8.7584,
       "r_dpo/chosen_len": 261.46875,
       "r_dpo/length_delta": 22.375,
-      "r_dpo/regularization_term": 1.118749976158142,
       "r_dpo/rejected_len": 239.09375,
       "step": 140
     },
     {
       "epoch": 0.31413612565445026,
-      "grad_norm": 516.0886840820312,
       "learning_rate": 4.346796604970912e-07,
-      "logits/chosen": -0.42371028661727905,
-      "logits/rejected": -0.4311944544315338,
-      "logps/chosen": -380.05743408203125,
       "logps/ref_chosen": -298.2093505859375,
       "logps/ref_rejected": -254.8907012939453,
-      "logps/rejected": -329.69818115234375,
-      "loss": 7.9472,
       "r_dpo/chosen_len": 283.84375,
       "r_dpo/length_delta": 48.359375,
-      "r_dpo/regularization_term": 2.41796875,
       "r_dpo/rejected_len": 235.484375,
       "step": 150
     },
     {
       "epoch": 0.33507853403141363,
-      "grad_norm": 553.9716796875,
       "learning_rate": 4.218561044282098e-07,
-      "logits/chosen": -0.41702669858932495,
-      "logits/rejected": -0.4278343617916107,
-      "logps/chosen": -358.8205871582031,
       "logps/ref_chosen": -281.94189453125,
       "logps/ref_rejected": -255.5653533935547,
-      "logps/rejected": -329.0650329589844,
-      "loss": 6.9722,
       "r_dpo/chosen_len": 267.828125,
       "r_dpo/length_delta": 41.368751525878906,
-      "r_dpo/regularization_term": 2.0684375762939453,
       "r_dpo/rejected_len": 226.45938110351562,
       "step": 160
     },
     {
       "epoch": 0.35602094240837695,
-      "grad_norm": 515.0377807617188,
       "learning_rate": 4.081113438988443e-07,
-      "logits/chosen": -0.3972104489803314,
-      "logits/rejected": -0.40150484442710876,
-      "logps/chosen": -367.43701171875,
       "logps/ref_chosen": -288.32232666015625,
       "logps/ref_rejected": -239.85415649414062,
-      "logps/rejected": -313.2834777832031,
-      "loss": 6.8528,
       "r_dpo/chosen_len": 285.203125,
       "r_dpo/length_delta": 46.396873474121094,
-      "r_dpo/regularization_term": 2.3198437690734863,
       "r_dpo/rejected_len": 238.80624389648438,
       "step": 170
     },
     {
       "epoch": 0.3769633507853403,
-      "grad_norm": 458.9293518066406,
       "learning_rate": 3.935190552834828e-07,
-      "logits/chosen": -0.3960086405277252,
-      "logits/rejected": -0.41429123282432556,
-      "logps/chosen": -367.7419738769531,
       "logps/ref_chosen": -286.17889404296875,
       "logps/ref_rejected": -249.9820098876953,
-      "logps/rejected": -329.16925048828125,
-      "loss": 6.6833,
       "r_dpo/chosen_len": 266.09063720703125,
       "r_dpo/length_delta": 40.12812423706055,
-      "r_dpo/regularization_term": 2.006406307220459,
       "r_dpo/rejected_len": 225.96249389648438,
       "step": 180
     },
     {
       "epoch": 0.39790575916230364,
-      "grad_norm": 509.046875,
       "learning_rate": 3.781574579820464e-07,
-      "logits/chosen": -0.4154319167137146,
-      "logits/rejected": -0.436189740896225,
-      "logps/chosen": -360.9049377441406,
       "logps/ref_chosen": -280.9278259277344,
       "logps/ref_rejected": -254.3533477783203,
-      "logps/rejected": -332.7271423339844,
-      "loss": 6.5164,
       "r_dpo/chosen_len": 276.33123779296875,
       "r_dpo/length_delta": 41.993751525878906,
-      "r_dpo/regularization_term": 2.099687337875366,
       "r_dpo/rejected_len": 234.33749389648438,
       "step": 190
     },
     {
       "epoch": 0.418848167539267,
-      "grad_norm": 442.76031494140625,
       "learning_rate": 3.621088951385353e-07,
-      "logits/chosen": -0.4301896095275879,
-      "logits/rejected": -0.44756612181663513,
-      "logps/chosen": -328.45867919921875,
       "logps/ref_chosen": -253.1712188720703,
       "logps/ref_rejected": -241.90478515625,
-      "logps/rejected": -316.90570068359375,
-      "loss": 6.5508,
       "r_dpo/chosen_len": 248.0749969482422,
       "r_dpo/length_delta": 28.131250381469727,
-      "r_dpo/regularization_term": 1.4065625667572021,
       "r_dpo/rejected_len": 219.94375610351562,
       "step": 200
     },
     {
       "epoch": 0.418848167539267,
-      "eval_logits/chosen": -0.4486958086490631,
-      "eval_logits/rejected": -0.4525853395462036,
-      "eval_logps/chosen": -383.5491027832031,
       "eval_logps/ref_chosen": -289.1346435546875,
       "eval_logps/ref_rejected": -264.7782287597656,
-      "eval_logps/rejected": -354.54388427734375,
-      "eval_loss": 0.8642193675041199,
       "eval_r_dpo/chosen_len": 291.2619934082031,
       "eval_r_dpo/length_delta": 42.86600112915039,
-      "eval_r_dpo/regularization_term": 2.1433000564575195,
       "eval_r_dpo/rejected_len": 248.39599609375,
-      "eval_runtime": 81.9238,
-      "eval_samples_per_second": 24.413,
       "eval_steps_per_second": 1.526,
       "step": 200
     },
     {
       "epoch": 0.4397905759162304,
-      "grad_norm": 602.9417724609375,
       "learning_rate": 3.454593922550693e-07,
-      "logits/chosen": -0.4147619605064392,
-      "logits/rejected": -0.4369305968284607,
-      "logps/chosen": -374.42071533203125,
       "logps/ref_chosen": -287.9228210449219,
       "logps/ref_rejected": -263.35595703125,
-      "logps/rejected": -345.69012451171875,
-      "loss": 7.3297,
       "r_dpo/chosen_len": 280.3125,
       "r_dpo/length_delta": 36.68437576293945,
-      "r_dpo/regularization_term": 1.8342187404632568,
       "r_dpo/rejected_len": 243.6281280517578,
       "step": 210
     },
     {
       "epoch": 0.4607329842931937,
-      "grad_norm": 499.6620178222656,
       "learning_rate": 3.2829819606729477e-07,
-      "logits/chosen": -0.458057701587677,
-      "logits/rejected": -0.45478391647338867,
-      "logps/chosen": -358.32745361328125,
       "logps/ref_chosen": -282.3331604003906,
       "logps/ref_rejected": -272.5645446777344,
-      "logps/rejected": -353.17620849609375,
-      "loss": 7.2761,
       "r_dpo/chosen_len": 261.359375,
       "r_dpo/length_delta": 17.865625381469727,
-      "r_dpo/regularization_term": 0.8932813405990601,
       "r_dpo/rejected_len": 243.49374389648438,
       "step": 220
     },
     {
       "epoch": 0.4816753926701571,
-      "grad_norm": 656.3023071289062,
       "learning_rate": 3.1071729615293424e-07,
-      "logits/chosen": -0.42080315947532654,
-      "logits/rejected": -0.4201149344444275,
-      "logps/chosen": -355.51019287109375,
       "logps/ref_chosen": -276.1485595703125,
       "logps/ref_rejected": -252.81198120117188,
-      "logps/rejected": -333.1983642578125,
-      "loss": 6.0592,
       "r_dpo/chosen_len": 264.43438720703125,
       "r_dpo/length_delta": 31.256250381469727,
-      "r_dpo/regularization_term": 1.5628125667572021,
       "r_dpo/rejected_len": 233.17813110351562,
       "step": 230
     },
     {
       "epoch": 0.5026178010471204,
-      "grad_norm": 420.3479919433594,
       "learning_rate": 2.9281093183781403e-07,
-      "logits/chosen": -0.40378203988075256,
-      "logits/rejected": -0.4264686107635498,
-      "logps/chosen": -352.75689697265625,
       "logps/ref_chosen": -270.52520751953125,
       "logps/ref_rejected": -254.83334350585938,
-      "logps/rejected": -336.2455749511719,
-      "loss": 6.313,
       "r_dpo/chosen_len": 271.81561279296875,
       "r_dpo/length_delta": 37.099998474121094,
-      "r_dpo/regularization_term": 1.8550001382827759,
       "r_dpo/rejected_len": 234.7156219482422,
       "step": 240
     },
     {
       "epoch": 0.5235602094240838,
-      "grad_norm": 652.2021484375,
       "learning_rate": 2.7467508704251135e-07,
-      "logits/chosen": -0.4280974864959717,
-      "logits/rejected": -0.4387652277946472,
-      "logps/chosen": -376.8214111328125,
       "logps/ref_chosen": -288.9503173828125,
       "logps/ref_rejected": -265.0694580078125,
-      "logps/rejected": -350.45782470703125,
-      "loss": 6.6243,
       "r_dpo/chosen_len": 277.50311279296875,
       "r_dpo/length_delta": 41.103126525878906,
-      "r_dpo/regularization_term": 2.0551562309265137,
       "r_dpo/rejected_len": 236.39999389648438,
       "step": 250
     },
     {
       "epoch": 0.5445026178010471,
-      "grad_norm": 615.6605224609375,
       "learning_rate": 2.5640697577740815e-07,
-      "logits/chosen": -0.4276336133480072,
-      "logits/rejected": -0.4253779947757721,
-      "logps/chosen": -371.8492126464844,
       "logps/ref_chosen": -288.6393737792969,
       "logps/ref_rejected": -265.315673828125,
-      "logps/rejected": -350.7149658203125,
-      "loss": 6.6966,
       "r_dpo/chosen_len": 271.48126220703125,
       "r_dpo/length_delta": 24.390625,
-      "r_dpo/regularization_term": 1.2195312976837158,
       "r_dpo/rejected_len": 247.0906219482422,
       "step": 260
     },
     {
       "epoch": 0.5654450261780105,
-      "grad_norm": 631.5802612304688,
       "learning_rate": 2.381045210440644e-07,
-      "logits/chosen": -0.4200795292854309,
-      "logits/rejected": -0.4278905391693115,
-      "logps/chosen": -367.51129150390625,
       "logps/ref_chosen": -280.1373596191406,
       "logps/ref_rejected": -264.84295654296875,
-      "logps/rejected": -358.1659851074219,
-      "loss": 6.3009,
       "r_dpo/chosen_len": 272.2875061035156,
       "r_dpo/length_delta": 19.956249237060547,
-      "r_dpo/regularization_term": 0.9978125691413879,
       "r_dpo/rejected_len": 252.3312530517578,
       "step": 270
     },
     {
       "epoch": 0.5863874345549738,
-      "grad_norm": 454.7221374511719,
       "learning_rate": 2.1986582993616925e-07,
-      "logits/chosen": -0.45643919706344604,
-      "logits/rejected": -0.4634791314601898,
-      "logps/chosen": -397.8352966308594,
       "logps/ref_chosen": -301.8018798828125,
       "logps/ref_rejected": -254.75112915039062,
-      "logps/rejected": -343.8644104003906,
-      "loss": 6.1331,
       "r_dpo/chosen_len": 285.44061279296875,
       "r_dpo/length_delta": 52.962501525878906,
-      "r_dpo/regularization_term": 2.648125171661377,
       "r_dpo/rejected_len": 232.47811889648438,
       "step": 280
     },
     {
       "epoch": 0.6073298429319371,
-      "grad_norm": 716.6207275390625,
       "learning_rate": 2.0178866775369774e-07,
-      "logits/chosen": -0.4405458867549896,
-      "logits/rejected": -0.42741164565086365,
-      "logps/chosen": -403.048583984375,
       "logps/ref_chosen": -302.79217529296875,
       "logps/ref_rejected": -292.9220275878906,
-      "logps/rejected": -396.4620056152344,
-      "loss": 6.6895,
       "r_dpo/chosen_len": 294.90625,
       "r_dpo/length_delta": 20.774999618530273,
-      "r_dpo/regularization_term": 1.0387500524520874,
       "r_dpo/rejected_len": 274.1312561035156,
       "step": 290
     },
     {
       "epoch": 0.6282722513089005,
-      "grad_norm": 534.1116333007812,
       "learning_rate": 1.839699339491937e-07,
-      "logits/chosen": -0.45565739274024963,
-      "logits/rejected": -0.44568490982055664,
-      "logps/chosen": -362.76483154296875,
       "logps/ref_chosen": -275.8238220214844,
       "logps/ref_rejected": -264.05743408203125,
-      "logps/rejected": -354.30914306640625,
-      "loss": 6.3761,
       "r_dpo/chosen_len": 266.859375,
       "r_dpo/length_delta": 20.734375,
-      "r_dpo/regularization_term": 1.036718726158142,
       "r_dpo/rejected_len": 246.125,
       "step": 300
     },
     {
       "epoch": 0.6492146596858639,
-      "grad_norm": 511.6585998535156,
       "learning_rate": 1.6650514271527465e-07,
-      "logits/chosen": -0.42221489548683167,
-      "logits/rejected": -0.43842315673828125,
-      "logps/chosen": -399.4841613769531,
       "logps/ref_chosen": -296.6716003417969,
       "logps/ref_rejected": -278.68426513671875,
-      "logps/rejected": -379.6441955566406,
-      "loss": 5.939,
       "r_dpo/chosen_len": 292.91876220703125,
       "r_dpo/length_delta": 32.55937576293945,
-      "r_dpo/regularization_term": 1.6279687881469727,
       "r_dpo/rejected_len": 260.359375,
       "step": 310
     },
     {
       "epoch": 0.6701570680628273,
-      "grad_norm": 508.40338134765625,
       "learning_rate": 1.4948791099758052e-07,
-      "logits/chosen": -0.44548073410987854,
-      "logits/rejected": -0.46428006887435913,
-      "logps/chosen": -382.5770263671875,
       "logps/ref_chosen": -284.1717529296875,
       "logps/ref_rejected": -261.2606506347656,
-      "logps/rejected": -357.56134033203125,
-      "loss": 5.7545,
       "r_dpo/chosen_len": 279.90313720703125,
       "r_dpo/length_delta": 44.537498474121094,
-      "r_dpo/regularization_term": 2.226875066757202,
       "r_dpo/rejected_len": 235.36563110351562,
       "step": 320
     },
     {
       "epoch": 0.6910994764397905,
-      "grad_norm": 529.1087036132812,
       "learning_rate": 1.3300945667758012e-07,
-      "logits/chosen": -0.4670342803001404,
-      "logits/rejected": -0.4756082594394684,
-      "logps/chosen": -378.3393249511719,
       "logps/ref_chosen": -283.40338134765625,
       "logps/ref_rejected": -271.27569580078125,
-      "logps/rejected": -373.77020263671875,
-      "loss": 6.1088,
       "r_dpo/chosen_len": 267.67498779296875,
       "r_dpo/length_delta": 13.015625,
-      "r_dpo/regularization_term": 0.6507812738418579,
       "r_dpo/rejected_len": 254.6593780517578,
       "step": 330
     },
     {
       "epoch": 0.7120418848167539,
-      "grad_norm": 536.99365234375,
       "learning_rate": 1.1715810961514072e-07,
-      "logits/chosen": -0.47216472029685974,
-      "logits/rejected": -0.47674980759620667,
-      "logps/chosen": -354.98577880859375,
       "logps/ref_chosen": -259.82537841796875,
       "logps/ref_rejected": -243.50222778320312,
-      "logps/rejected": -337.8619079589844,
-      "loss": 5.8753,
       "r_dpo/chosen_len": 256.11248779296875,
       "r_dpo/length_delta": 32.546875,
-      "r_dpo/regularization_term": 1.627343773841858,
       "r_dpo/rejected_len": 223.5656280517578,
       "step": 340
     },
     {
       "epoch": 0.7329842931937173,
-      "grad_norm": 597.7227783203125,
       "learning_rate": 1.0201883817182949e-07,
-      "logits/chosen": -0.4812525808811188,
-      "logits/rejected": -0.49402889609336853,
-      "logps/chosen": -398.0356140136719,
       "logps/ref_chosen": -298.24725341796875,
       "logps/ref_rejected": -272.657958984375,
-      "logps/rejected": -365.20068359375,
-      "loss": 5.8123,
       "r_dpo/chosen_len": 281.4624938964844,
       "r_dpo/length_delta": 45.275001525878906,
-      "r_dpo/regularization_term": 2.2637500762939453,
       "r_dpo/rejected_len": 236.1875,
       "step": 350
     },
     {
       "epoch": 0.7539267015706806,
-      "grad_norm": 569.5155029296875,
       "learning_rate": 8.76727937529367e-08,
-      "logits/chosen": -0.44678616523742676,
-      "logits/rejected": -0.4613564610481262,
-      "logps/chosen": -376.3524475097656,
       "logps/ref_chosen": -281.881103515625,
       "logps/ref_rejected": -265.4746398925781,
-      "logps/rejected": -360.0444030761719,
-      "loss": 6.1064,
       "r_dpo/chosen_len": 272.64373779296875,
       "r_dpo/length_delta": 30.071874618530273,
-      "r_dpo/regularization_term": 1.5035936832427979,
       "r_dpo/rejected_len": 242.57186889648438,
       "step": 360
     },
     {
       "epoch": 0.774869109947644,
-      "grad_norm": 511.41351318359375,
       "learning_rate": 7.419687580962222e-08,
-      "logits/chosen": -0.47953400015830994,
-      "logits/rejected": -0.482210248708725,
-      "logps/chosen": -394.7923583984375,
       "logps/ref_chosen": -302.17822265625,
       "logps/ref_rejected": -265.92877197265625,
-      "logps/rejected": -357.67889404296875,
-      "loss": 5.755,
       "r_dpo/chosen_len": 273.88751220703125,
       "r_dpo/length_delta": 33.759376525878906,
-      "r_dpo/regularization_term": 1.6879686117172241,
       "r_dpo/rejected_len": 240.1281280517578,
       "step": 370
     },
     {
       "epoch": 0.7958115183246073,
-      "grad_norm": 606.5574340820312,
       "learning_rate": 6.166331963291519e-08,
-      "logits/chosen": -0.44388914108276367,
-      "logits/rejected": -0.4509609341621399,
-      "logps/chosen": -399.1026916503906,
       "logps/ref_chosen": -301.07965087890625,
       "logps/ref_rejected": -266.7089538574219,
-      "logps/rejected": -364.1946716308594,
-      "loss": 5.794,
       "r_dpo/chosen_len": 286.75311279296875,
       "r_dpo/length_delta": 33.53125,
-      "r_dpo/regularization_term": 1.6765625476837158,
       "r_dpo/rejected_len": 253.2218780517578,
       "step": 380
     },
     {
       "epoch": 0.8167539267015707,
-      "grad_norm": 557.215087890625,
       "learning_rate": 5.013930914912476e-08,
-      "logits/chosen": -0.44885340332984924,
-      "logits/rejected": -0.45168790221214294,
-      "logps/chosen": -397.15771484375,
       "logps/ref_chosen": -296.6472473144531,
       "logps/ref_rejected": -278.953857421875,
-      "logps/rejected": -381.54345703125,
-      "loss": 5.489,
       "r_dpo/chosen_len": 287.91876220703125,
       "r_dpo/length_delta": 30.081249237060547,
-      "r_dpo/regularization_term": 1.5040624141693115,
       "r_dpo/rejected_len": 257.8374938964844,
       "step": 390
     },
     {
       "epoch": 0.837696335078534,
-      "grad_norm": 459.5953063964844,
       "learning_rate": 3.968661679220467e-08,
-      "logits/chosen": -0.4598144590854645,
-      "logits/rejected": -0.46983155608177185,
-      "logps/chosen": -399.8034362792969,
       "logps/ref_chosen": -296.6556091308594,
       "logps/ref_rejected": -256.9266662597656,
-      "logps/rejected": -354.4169921875,
-      "loss": 5.5495,
       "r_dpo/chosen_len": 278.96875,
       "r_dpo/length_delta": 39.60625076293945,
-      "r_dpo/regularization_term": 1.9803125858306885,
       "r_dpo/rejected_len": 239.3625030517578,
       "step": 400
     },
     {
       "epoch": 0.837696335078534,
-      "eval_logits/chosen": -0.4675108790397644,
-      "eval_logits/rejected": -0.47323355078697205,
-      "eval_logps/chosen": -396.0220642089844,
       "eval_logps/ref_chosen": -289.1346435546875,
       "eval_logps/ref_rejected": -264.7782287597656,
-      "eval_logps/rejected": -365.0935363769531,
-      "eval_loss": 0.7402477860450745,
       "eval_r_dpo/chosen_len": 291.2619934082031,
       "eval_r_dpo/length_delta": 42.86600112915039,
-      "eval_r_dpo/regularization_term": 2.1433000564575195,
       "eval_r_dpo/rejected_len": 248.39599609375,
-      "eval_runtime": 82.0229,
-      "eval_samples_per_second": 24.383,
       "eval_steps_per_second": 1.524,
       "step": 400
     },
     {
       "epoch": 0.8586387434554974,
-      "grad_norm": 499.20574951171875,
       "learning_rate": 3.036127238347164e-08,
-      "logits/chosen": -0.4523307681083679,
-      "logits/rejected": -0.4475272595882416,
-      "logps/chosen": -396.6153259277344,
       "logps/ref_chosen": -290.08154296875,
       "logps/ref_rejected": -272.56988525390625,
-      "logps/rejected": -375.4905700683594,
-      "loss": 6.0301,
       "r_dpo/chosen_len": 282.40625,
       "r_dpo/length_delta": 26.265625,
-      "r_dpo/regularization_term": 1.3132812976837158,
       "r_dpo/rejected_len": 256.140625,
       "step": 410
     },
     {
       "epoch": 0.8795811518324608,
-      "grad_norm": 514.4651489257812,
       "learning_rate": 2.2213262793589482e-08,
-      "logits/chosen": -0.45558518171310425,
-      "logits/rejected": -0.4580914080142975,
-      "logps/chosen": -412.6549377441406,
       "logps/ref_chosen": -307.40240478515625,
       "logps/ref_rejected": -279.85760498046875,
-      "logps/rejected": -385.1617431640625,
-      "loss": 5.3657,
       "r_dpo/chosen_len": 296.8343811035156,
       "r_dpo/length_delta": 37.165626525878906,
-      "r_dpo/regularization_term": 1.8582813739776611,
       "r_dpo/rejected_len": 259.66876220703125,
       "step": 420
     },
     {
       "epoch": 0.900523560209424,
-      "grad_norm": 964.3095703125,
       "learning_rate": 1.5286263996730026e-08,
-      "logits/chosen": -0.458394855260849,
-      "logits/rejected": -0.4722791314125061,
-      "logps/chosen": -402.4847106933594,
       "logps/ref_chosen": -297.7133483886719,
       "logps/ref_rejected": -266.862060546875,
-      "logps/rejected": -365.95977783203125,
-      "loss": 5.3139,
       "r_dpo/chosen_len": 290.703125,
       "r_dpo/length_delta": 48.868751525878906,
-      "r_dpo/regularization_term": 2.4434375762939453,
       "r_dpo/rejected_len": 241.83438110351562,
       "step": 430
     },
     {
       "epoch": 0.9214659685863874,
-      "grad_norm": 587.9862670898438,
       "learning_rate": 9.617406953185136e-09,
-      "logits/chosen": -0.44800668954849243,
-      "logits/rejected": -0.4568398892879486,
-      "logps/chosen": -396.49908447265625,
       "logps/ref_chosen": -293.84521484375,
       "logps/ref_rejected": -289.6698303222656,
-      "logps/rejected": -398.7309875488281,
-      "loss": 6.16,
       "r_dpo/chosen_len": 285.3656311035156,
       "r_dpo/length_delta": 12.868749618530273,
-      "r_dpo/regularization_term": 0.643437385559082,
       "r_dpo/rejected_len": 272.49688720703125,
       "step": 440
     },
     {
       "epoch": 0.9424083769633508,
-      "grad_norm": 353.2596435546875,
       "learning_rate": 5.2370785753763356e-09,
-      "logits/chosen": -0.4659084379673004,
-      "logits/rejected": -0.47555309534072876,
-      "logps/chosen": -395.29638671875,
       "logps/ref_chosen": -297.1932373046875,
       "logps/ref_rejected": -262.9091796875,
-      "logps/rejected": -358.9178466796875,
-      "loss": 5.0475,
       "r_dpo/chosen_len": 282.81561279296875,
       "r_dpo/length_delta": 40.631248474121094,
-      "r_dpo/regularization_term": 2.031562566757202,
       "r_dpo/rejected_len": 242.1843719482422,
       "step": 450
     },
     {
       "epoch": 0.9633507853403142,
-      "grad_norm": 345.8104553222656,
       "learning_rate": 2.168758844148272e-09,
-      "logits/chosen": -0.4650910496711731,
-      "logits/rejected": -0.4829515516757965,
-      "logps/chosen": -416.01031494140625,
       "logps/ref_chosen": -312.42291259765625,
       "logps/ref_rejected": -278.7356262207031,
-      "logps/rejected": -377.20782470703125,
-      "loss": 5.8124,
       "r_dpo/chosen_len": 288.9125061035156,
       "r_dpo/length_delta": 43.23125076293945,
-      "r_dpo/regularization_term": 2.161562442779541,
       "r_dpo/rejected_len": 245.68124389648438,
       "step": 460
     },
     {
       "epoch": 0.9842931937172775,
-      "grad_norm": 570.2479248046875,
       "learning_rate": 4.288949484559934e-10,
-      "logits/chosen": -0.44196611642837524,
-      "logits/rejected": -0.44385018944740295,
-      "logps/chosen": -373.54534912109375,
       "logps/ref_chosen": -278.13385009765625,
       "logps/ref_rejected": -256.70648193359375,
-      "logps/rejected": -356.46002197265625,
-      "loss": 5.1791,
       "r_dpo/chosen_len": 268.8687438964844,
       "r_dpo/length_delta": 26.262500762939453,
-      "r_dpo/regularization_term": 1.3131250143051147,
       "r_dpo/rejected_len": 242.6062469482422,
       "step": 470
     },
@@ -865,9 +865,9 @@
       "epoch": 0.9989528795811519,
       "step": 477,
       "total_flos": 0.0,
-      "train_loss": 9.076586509400693,
-      "train_runtime": 6089.4941,
-      "train_samples_per_second": 10.039,
       "train_steps_per_second": 0.078
     }
   ],

   "log_history": [
     {
       "epoch": 0.0020942408376963353,
+      "grad_norm": 286.114501953125,
       "learning_rate": 0.0,
       "logits/chosen": -0.5995081663131714,
       "logits/rejected": -0.6144353747367859,
       "logps/ref_chosen": -267.5935363769531,
       "logps/ref_rejected": -204.2306671142578,
       "logps/rejected": -204.23907470703125,
+      "loss": 5.5602,
       "r_dpo/chosen_len": 257.75,
       "r_dpo/length_delta": 47.875,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 209.875,
       "step": 1
     },
     {
       "epoch": 0.020942408376963352,
+      "grad_norm": 265.7362365722656,
       "learning_rate": 9.375e-08,
+      "logits/chosen": -0.6309213042259216,
+      "logits/rejected": -0.63545161485672,
+      "logps/chosen": -296.74920654296875,
       "logps/ref_chosen": -296.72576904296875,
       "logps/ref_rejected": -258.9681396484375,
+      "logps/rejected": -259.0067443847656,
+      "loss": 5.5477,
       "r_dpo/chosen_len": 291.8680419921875,
       "r_dpo/length_delta": 49.76388931274414,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 242.1041717529297,
       "step": 10
     },
     {
       "epoch": 0.041884816753926704,
+      "grad_norm": 296.5692138671875,
       "learning_rate": 1.9791666666666664e-07,
+      "logits/chosen": -0.597561240196228,
+      "logits/rejected": -0.6285912990570068,
+      "logps/chosen": -297.9436340332031,
       "logps/ref_chosen": -297.9349365234375,
       "logps/ref_rejected": -256.9902648925781,
+      "logps/rejected": -256.98876953125,
+      "loss": 5.5369,
       "r_dpo/chosen_len": 291.29998779296875,
       "r_dpo/length_delta": 52.89374923706055,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 238.40625,
       "step": 20
     },
     {
       "epoch": 0.06282722513089005,
+      "grad_norm": 279.9869384765625,
       "learning_rate": 3.020833333333333e-07,
+      "logits/chosen": -0.6136946678161621,
+      "logits/rejected": -0.605636477470398,
+      "logps/chosen": -278.4833679199219,
       "logps/ref_chosen": -278.64752197265625,
       "logps/ref_rejected": -249.309814453125,
+      "logps/rejected": -249.22158813476562,
+      "loss": 5.495,
       "r_dpo/chosen_len": 270.8812561035156,
       "r_dpo/length_delta": 25.228124618530273,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 245.6531219482422,
       "step": 30
     },
     {
       "epoch": 0.08376963350785341,
+      "grad_norm": 258.0906677246094,
       "learning_rate": 4.0625e-07,
+      "logits/chosen": -0.6181729435920715,
+      "logits/rejected": -0.6429071426391602,
+      "logps/chosen": -282.7382507324219,
       "logps/ref_chosen": -283.49981689453125,
       "logps/ref_rejected": -265.32733154296875,
+      "logps/rejected": -265.03875732421875,
+      "loss": 5.3372,
       "r_dpo/chosen_len": 281.43438720703125,
       "r_dpo/length_delta": 33.34375,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 248.0906219482422,
       "step": 40
     },
     {
       "epoch": 0.10471204188481675,
+      "grad_norm": 267.5690002441406,
       "learning_rate": 4.999932966293553e-07,
+      "logits/chosen": -0.6292977333068848,
+      "logits/rejected": -0.6698824763298035,
+      "logps/chosen": -278.5817565917969,
       "logps/ref_chosen": -280.224365234375,
       "logps/ref_rejected": -274.3541259765625,
+      "logps/rejected": -273.8267517089844,
+      "loss": 5.1553,
       "r_dpo/chosen_len": 290.32501220703125,
       "r_dpo/length_delta": 35.11249923706055,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 255.21249389648438,
       "step": 50
     },
     {
       "epoch": 0.1256544502617801,
+      "grad_norm": 258.2756652832031,
       "learning_rate": 4.991893270335525e-07,
+      "logits/chosen": -0.6408439874649048,
+      "logits/rejected": -0.6542555093765259,
+      "logps/chosen": -278.864501953125,
       "logps/ref_chosen": -281.12664794921875,
       "logps/ref_rejected": -259.86456298828125,
+      "logps/rejected": -259.77728271484375,
+      "loss": 4.9283,
       "r_dpo/chosen_len": 273.953125,
       "r_dpo/length_delta": 29.084375381469727,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 244.86874389648438,
       "step": 60
     },
     {
       "epoch": 0.14659685863874344,
+      "grad_norm": 233.6326446533203,
       "learning_rate": 4.970496218214204e-07,
+      "logits/chosen": -0.7015193700790405,
+      "logits/rejected": -0.7072200179100037,
+      "logps/chosen": -284.4013977050781,
       "logps/ref_chosen": -287.71063232421875,
       "logps/ref_rejected": -276.839599609375,
+      "logps/rejected": -276.4761047363281,
+      "loss": 4.76,
       "r_dpo/chosen_len": 267.4937438964844,
       "r_dpo/length_delta": 14.484375,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 253.00936889648438,
       "step": 70
     },
     {
       "epoch": 0.16753926701570682,
+      "grad_norm": 234.1904296875,
       "learning_rate": 4.935856505068998e-07,
+      "logits/chosen": -0.6775287389755249,
+      "logits/rejected": -0.6742324829101562,
+      "logps/chosen": -275.77288818359375,
       "logps/ref_chosen": -280.123046875,
       "logps/ref_rejected": -258.8989562988281,
+      "logps/rejected": -258.0046691894531,
+      "loss": 4.6236,
       "r_dpo/chosen_len": 267.4781188964844,
       "r_dpo/length_delta": 32.46562576293945,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 235.0124969482422,
       "step": 80
     },
     {
       "epoch": 0.18848167539267016,
+      "grad_norm": 249.3949737548828,
       "learning_rate": 4.8881598109976e-07,
+      "logits/chosen": -0.6876357197761536,
+      "logits/rejected": -0.694031834602356,
+      "logps/chosen": -273.43939208984375,
       "logps/ref_chosen": -278.02545166015625,
       "logps/ref_rejected": -251.0922393798828,
+      "logps/rejected": -251.4302978515625,
+      "loss": 4.5727,
       "r_dpo/chosen_len": 274.20623779296875,
       "r_dpo/length_delta": 44.97187423706055,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 229.234375,
       "step": 90
     },
     {
       "epoch": 0.2094240837696335,
+      "grad_norm": 247.84217834472656,
       "learning_rate": 4.827661805750437e-07,
+      "logits/chosen": -0.6776241660118103,
+      "logits/rejected": -0.6956905126571655,
+      "logps/chosen": -269.8210754394531,
       "logps/ref_chosen": -274.0089416503906,
       "logps/ref_rejected": -274.14447021484375,
+      "logps/rejected": -275.0433349609375,
+      "loss": 4.6252,
       "r_dpo/chosen_len": 275.3343811035156,
       "r_dpo/length_delta": 21.912500381469727,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 253.421875,
       "step": 100
     },
     {
       "epoch": 0.23036649214659685,
+      "grad_norm": 227.6590576171875,
       "learning_rate": 4.75468677825789e-07,
+      "logits/chosen": -0.701612114906311,
+      "logits/rejected": -0.7261066436767578,
+      "logps/chosen": -269.59173583984375,
       "logps/ref_chosen": -273.23333740234375,
       "logps/ref_rejected": -263.88787841796875,
+      "logps/rejected": -266.48272705078125,
+      "loss": 4.2996,
       "r_dpo/chosen_len": 283.43438720703125,
       "r_dpo/length_delta": 50.34375,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 233.0906219482422,
       "step": 110
     },
     {
       "epoch": 0.2513089005235602,
+      "grad_norm": 243.6549835205078,
       "learning_rate": 4.669625898336438e-07,
+      "logits/chosen": -0.7124736905097961,
+      "logits/rejected": -0.7152305841445923,
+      "logps/chosen": -267.23974609375,
       "logps/ref_chosen": -269.54656982421875,
       "logps/ref_rejected": -272.7981872558594,
+      "logps/rejected": -275.8104553222656,
+      "loss": 4.5034,
       "r_dpo/chosen_len": 264.7593688964844,
       "r_dpo/length_delta": 13.840624809265137,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 250.9187469482422,
       "step": 120
     },
     {
       "epoch": 0.27225130890052357,
+      "grad_norm": 274.58111572265625,
       "learning_rate": 4.5729351198915705e-07,
+      "logits/chosen": -0.7257631421089172,
+      "logits/rejected": -0.7154949903488159,
+      "logps/chosen": -272.1502990722656,
       "logps/ref_chosen": -275.03448486328125,
       "logps/ref_rejected": -276.39862060546875,
+      "logps/rejected": -279.466796875,
+      "loss": 4.445,
       "r_dpo/chosen_len": 266.625,
       "r_dpo/length_delta": 18.668750762939453,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 247.9562530517578,
       "step": 130
     },
     {
       "epoch": 0.2931937172774869,
+      "grad_norm": 247.52589416503906,
       "learning_rate": 4.4651327368569684e-07,
+      "logits/chosen": -0.7425049543380737,
+      "logits/rejected": -0.7479813098907471,
+      "logps/chosen": -273.00714111328125,
       "logps/ref_chosen": -276.0029602050781,
       "logps/ref_rejected": -255.9320526123047,
+      "logps/rejected": -259.60784912109375,
+      "loss": 4.4958,
       "r_dpo/chosen_len": 261.46875,
       "r_dpo/length_delta": 22.375,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 239.09375,
       "step": 140
     },
     {
       "epoch": 0.31413612565445026,
+      "grad_norm": 264.3229675292969,
       "learning_rate": 4.346796604970912e-07,
+      "logits/chosen": -0.7595945596694946,
+      "logits/rejected": -0.7660830616950989,
+      "logps/chosen": -294.779541015625,
       "logps/ref_chosen": -298.2093505859375,
       "logps/ref_rejected": -254.8907012939453,
+      "logps/rejected": -258.84454345703125,
+      "loss": 4.2489,
       "r_dpo/chosen_len": 283.84375,
       "r_dpo/length_delta": 48.359375,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 235.484375,
       "step": 150
     },
     {
       "epoch": 0.33507853403141363,
+      "grad_norm": 212.1317138671875,
       "learning_rate": 4.218561044282098e-07,
+      "logits/chosen": -0.7484012246131897,
+      "logits/rejected": -0.7548493146896362,
+      "logps/chosen": -280.19281005859375,
       "logps/ref_chosen": -281.94189453125,
       "logps/ref_rejected": -255.5653533935547,
+      "logps/rejected": -261.9532470703125,
+      "loss": 4.0494,
       "r_dpo/chosen_len": 267.828125,
       "r_dpo/length_delta": 41.368751525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 226.45938110351562,
       "step": 160
     },
     {
       "epoch": 0.35602094240837695,
+      "grad_norm": 263.6576843261719,
       "learning_rate": 4.081113438988443e-07,
+      "logits/chosen": -0.7331063151359558,
+      "logits/rejected": -0.7357569932937622,
+      "logps/chosen": -287.6741638183594,
       "logps/ref_chosen": -288.32232666015625,
       "logps/ref_rejected": -239.85415649414062,
+      "logps/rejected": -246.4853973388672,
+      "loss": 4.1229,
       "r_dpo/chosen_len": 285.203125,
       "r_dpo/length_delta": 46.396873474121094,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 238.80624389648438,
       "step": 170
     },
     {
       "epoch": 0.3769633507853403,
+      "grad_norm": 204.8552703857422,
       "learning_rate": 3.935190552834828e-07,
+      "logits/chosen": -0.7205232977867126,
+      "logits/rejected": -0.7394840121269226,
+      "logps/chosen": -285.03363037109375,
       "logps/ref_chosen": -286.17889404296875,
       "logps/ref_rejected": -249.9820098876953,
+      "logps/rejected": -256.336669921875,
+      "loss": 4.2693,
       "r_dpo/chosen_len": 266.09063720703125,
       "r_dpo/length_delta": 40.12812423706055,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 225.96249389648438,
       "step": 180
     },
     {
       "epoch": 0.39790575916230364,
+      "grad_norm": 227.1675567626953,
       "learning_rate": 3.781574579820464e-07,
+      "logits/chosen": -0.7226096987724304,
+      "logits/rejected": -0.7422462701797485,
+      "logps/chosen": -279.1224365234375,
       "logps/ref_chosen": -280.9278259277344,
       "logps/ref_rejected": -254.3533477783203,
+      "logps/rejected": -262.07611083984375,
+      "loss": 4.0622,
       "r_dpo/chosen_len": 276.33123779296875,
       "r_dpo/length_delta": 41.993751525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 234.33749389648438,
       "step": 190
     },
     {
       "epoch": 0.418848167539267,
+      "grad_norm": 257.41168212890625,
       "learning_rate": 3.621088951385353e-07,
+      "logits/chosen": -0.741692841053009,
+      "logits/rejected": -0.7598770260810852,
+      "logps/chosen": -250.05776977539062,
       "logps/ref_chosen": -253.1712188720703,
       "logps/ref_rejected": -241.90478515625,
+      "logps/rejected": -246.9326629638672,
+      "loss": 4.178,
       "r_dpo/chosen_len": 248.0749969482422,
       "r_dpo/length_delta": 28.131250381469727,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 219.94375610351562,
       "step": 200
     },
     {
       "epoch": 0.418848167539267,
+      "eval_logits/chosen": -0.7478832006454468,
+      "eval_logits/rejected": -0.7490243911743164,
+      "eval_logps/chosen": -287.2392883300781,
       "eval_logps/ref_chosen": -289.1346435546875,
       "eval_logps/ref_rejected": -264.7782287597656,
+      "eval_logps/rejected": -271.315185546875,
+      "eval_loss": 0.5266098380088806,
       "eval_r_dpo/chosen_len": 291.2619934082031,
       "eval_r_dpo/length_delta": 42.86600112915039,
+      "eval_r_dpo/regularization_term": 0.0,
       "eval_r_dpo/rejected_len": 248.39599609375,
+      "eval_runtime": 81.9324,
+      "eval_samples_per_second": 24.41,
       "eval_steps_per_second": 1.526,
       "step": 200
     },
     {
       "epoch": 0.4397905759162304,
+      "grad_norm": 272.2992858886719,
       "learning_rate": 3.454593922550693e-07,
+      "logits/chosen": -0.7123077511787415,
+      "logits/rejected": -0.7341128587722778,
+      "logps/chosen": -285.9898376464844,
       "logps/ref_chosen": -287.9228210449219,
       "logps/ref_rejected": -263.35595703125,
+      "logps/rejected": -270.154541015625,
+      "loss": 4.2219,
       "r_dpo/chosen_len": 280.3125,
       "r_dpo/length_delta": 36.68437576293945,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 243.6281280517578,
       "step": 210
     },
     {
       "epoch": 0.4607329842931937,
+      "grad_norm": 244.5552520751953,
       "learning_rate": 3.2829819606729477e-07,
+      "logits/chosen": -0.7547723054885864,
+      "logits/rejected": -0.7528045773506165,
+      "logps/chosen": -280.8878479003906,
       "logps/ref_chosen": -282.3331604003906,
       "logps/ref_rejected": -272.5645446777344,
+      "logps/rejected": -279.7627868652344,
+      "loss": 4.0545,
       "r_dpo/chosen_len": 261.359375,
       "r_dpo/length_delta": 17.865625381469727,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 243.49374389648438,
       "step": 220
     },
     {
       "epoch": 0.4816753926701571,
+      "grad_norm": 290.22955322265625,
       "learning_rate": 3.1071729615293424e-07,
+      "logits/chosen": -0.7329878211021423,
+      "logits/rejected": -0.7301102876663208,
+      "logps/chosen": -276.18194580078125,
       "logps/ref_chosen": -276.1485595703125,
       "logps/ref_rejected": -252.81198120117188,
+      "logps/rejected": -262.63323974609375,
+      "loss": 3.8953,
       "r_dpo/chosen_len": 264.43438720703125,
       "r_dpo/length_delta": 31.256250381469727,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 233.17813110351562,
       "step": 230
     },
     {
       "epoch": 0.5026178010471204,
+      "grad_norm": 242.89169311523438,
       "learning_rate": 2.9281093183781403e-07,
+      "logits/chosen": -0.7076805233955383,
+      "logits/rejected": -0.729147732257843,
+      "logps/chosen": -270.7227783203125,
       "logps/ref_chosen": -270.52520751953125,
       "logps/ref_rejected": -254.83334350585938,
+      "logps/rejected": -263.5503234863281,
+      "loss": 4.1412,
       "r_dpo/chosen_len": 271.81561279296875,
       "r_dpo/length_delta": 37.099998474121094,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 234.7156219482422,
       "step": 240
     },
     {
       "epoch": 0.5235602094240838,
+      "grad_norm": 281.21990966796875,
       "learning_rate": 2.7467508704251135e-07,
+      "logits/chosen": -0.7120658159255981,
+      "logits/rejected": -0.7247544527053833,
+      "logps/chosen": -288.3058166503906,
       "logps/ref_chosen": -288.9503173828125,
       "logps/ref_rejected": -265.0694580078125,
+      "logps/rejected": -273.70220947265625,
+      "loss": 4.2125,
       "r_dpo/chosen_len": 277.50311279296875,
       "r_dpo/length_delta": 41.103126525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 236.39999389648438,
       "step": 250
     },
     {
       "epoch": 0.5445026178010471,
+      "grad_norm": 305.6956787109375,
       "learning_rate": 2.5640697577740815e-07,
+      "logits/chosen": -0.7282508611679077,
+      "logits/rejected": -0.7263582944869995,
+      "logps/chosen": -287.1412353515625,
       "logps/ref_chosen": -288.6393737792969,
       "logps/ref_rejected": -265.315673828125,
+      "logps/rejected": -273.0039367675781,
+      "loss": 4.111,
       "r_dpo/chosen_len": 271.48126220703125,
       "r_dpo/length_delta": 24.390625,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 247.0906219482422,
       "step": 260
     },
     {
       "epoch": 0.5654450261780105,
+      "grad_norm": 248.96470642089844,
       "learning_rate": 2.381045210440644e-07,
+      "logits/chosen": -0.7059388756752014,
+      "logits/rejected": -0.7142220735549927,
+      "logps/chosen": -279.7881774902344,
       "logps/ref_chosen": -280.1373596191406,
       "logps/ref_rejected": -264.84295654296875,
+      "logps/rejected": -273.27484130859375,
+      "loss": 4.011,
       "r_dpo/chosen_len": 272.2875061035156,
       "r_dpo/length_delta": 19.956249237060547,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 252.3312530517578,
       "step": 270
     },
     {
       "epoch": 0.5863874345549738,
+      "grad_norm": 216.84811401367188,
       "learning_rate": 2.1986582993616925e-07,
+      "logits/chosen": -0.7371554374694824,
+      "logits/rejected": -0.7457467317581177,
+      "logps/chosen": -300.7189636230469,
       "logps/ref_chosen": -301.8018798828125,
       "logps/ref_rejected": -254.75112915039062,
+      "logps/rejected": -263.86175537109375,
+      "loss": 3.9179,
       "r_dpo/chosen_len": 285.44061279296875,
       "r_dpo/length_delta": 52.962501525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 232.47811889648438,
       "step": 280
     },
     {
       "epoch": 0.6073298429319371,
+      "grad_norm": 258.0394592285156,
       "learning_rate": 2.0178866775369774e-07,
+      "logits/chosen": -0.7233304977416992,
+      "logits/rejected": -0.7041932344436646,
+      "logps/chosen": -303.56158447265625,
       "logps/ref_chosen": -302.79217529296875,
       "logps/ref_rejected": -292.9220275878906,
+      "logps/rejected": -301.3799743652344,
+      "loss": 4.2767,
       "r_dpo/chosen_len": 294.90625,
       "r_dpo/length_delta": 20.774999618530273,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 274.1312561035156,
       "step": 290
     },
     {
       "epoch": 0.6282722513089005,
+      "grad_norm": 259.0774230957031,
       "learning_rate": 1.839699339491937e-07,
+      "logits/chosen": -0.7432774305343628,
+      "logits/rejected": -0.7294503450393677,
+      "logps/chosen": -273.68316650390625,
       "logps/ref_chosen": -275.8238220214844,
       "logps/ref_rejected": -264.05743408203125,
+      "logps/rejected": -269.7857666015625,
+      "loss": 4.1741,
       "r_dpo/chosen_len": 266.859375,
       "r_dpo/length_delta": 20.734375,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 246.125,
       "step": 300
     },
     {
       "epoch": 0.6492146596858639,
+      "grad_norm": 330.3496398925781,
       "learning_rate": 1.6650514271527465e-07,
+      "logits/chosen": -0.7020508646965027,
+      "logits/rejected": -0.7177096605300903,
+      "logps/chosen": -294.5872802734375,
       "logps/ref_chosen": -296.6716003417969,
       "logps/ref_rejected": -278.68426513671875,
+      "logps/rejected": -285.25396728515625,
+      "loss": 4.015,
       "r_dpo/chosen_len": 292.91876220703125,
       "r_dpo/length_delta": 32.55937576293945,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 260.359375,
       "step": 310
     },
     {
       "epoch": 0.6701570680628273,
+      "grad_norm": 258.3315734863281,
       "learning_rate": 1.4948791099758052e-07,
+      "logits/chosen": -0.7137759923934937,
+      "logits/rejected": -0.7293487191200256,
+      "logps/chosen": -283.85919189453125,
       "logps/ref_chosen": -284.1717529296875,
       "logps/ref_rejected": -261.2606506347656,
+      "logps/rejected": -270.0918884277344,
+      "loss": 4.014,
       "r_dpo/chosen_len": 279.90313720703125,
       "r_dpo/length_delta": 44.537498474121094,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 235.36563110351562,
       "step": 320
     },
     {
       "epoch": 0.6910994764397905,
+      "grad_norm": 268.5577087402344,
       "learning_rate": 1.3300945667758012e-07,
+      "logits/chosen": -0.7268847227096558,
+      "logits/rejected": -0.7360986471176147,
+      "logps/chosen": -284.22406005859375,
       "logps/ref_chosen": -283.40338134765625,
       "logps/ref_rejected": -271.27569580078125,
+      "logps/rejected": -280.9209899902344,
+      "loss": 4.0598,
       "r_dpo/chosen_len": 267.67498779296875,
       "r_dpo/length_delta": 13.015625,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 254.6593780517578,
       "step": 330
     },
     {
       "epoch": 0.7120418848167539,
+      "grad_norm": 247.73794555664062,
       "learning_rate": 1.1715810961514072e-07,
+      "logits/chosen": -0.7288721203804016,
+      "logits/rejected": -0.7343642115592957,
+      "logps/chosen": -263.1542053222656,
       "logps/ref_chosen": -259.82537841796875,
       "logps/ref_rejected": -243.50222778320312,
+      "logps/rejected": -255.48574829101562,
+      "loss": 4.0065,
       "r_dpo/chosen_len": 256.11248779296875,
       "r_dpo/length_delta": 32.546875,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 223.5656280517578,
       "step": 340
     },
     {
       "epoch": 0.7329842931937173,
+      "grad_norm": 270.27557373046875,
       "learning_rate": 1.0201883817182949e-07,
+      "logits/chosen": -0.7445515990257263,
+      "logits/rejected": -0.7549839615821838,
+      "logps/chosen": -299.8614501953125,
       "logps/ref_chosen": -298.24725341796875,
       "logps/ref_rejected": -272.657958984375,
+      "logps/rejected": -283.16290283203125,
+      "loss": 4.0568,
       "r_dpo/chosen_len": 281.4624938964844,
       "r_dpo/length_delta": 45.275001525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 236.1875,
       "step": 350
     },
     {
       "epoch": 0.7539267015706806,
+      "grad_norm": 239.69332885742188,
       "learning_rate": 8.76727937529367e-08,
+      "logits/chosen": -0.7207680344581604,
+      "logits/rejected": -0.7342051863670349,
+      "logps/chosen": -282.36883544921875,
       "logps/ref_chosen": -281.881103515625,
       "logps/ref_rejected": -265.4746398925781,
+      "logps/rejected": -275.1003112792969,
+      "loss": 4.1996,
       "r_dpo/chosen_len": 272.64373779296875,
       "r_dpo/length_delta": 30.071874618530273,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 242.57186889648438,
       "step": 360
     },
     {
       "epoch": 0.774869109947644,
+      "grad_norm": 198.85562133789062,
       "learning_rate": 7.419687580962222e-08,
+      "logits/chosen": -0.7548901438713074,
+      "logits/rejected": -0.7558341026306152,
+      "logps/chosen": -302.01898193359375,
       "logps/ref_chosen": -302.17822265625,
       "logps/ref_rejected": -265.92877197265625,
+      "logps/rejected": -275.6526794433594,
+      "loss": 3.8979,
       "r_dpo/chosen_len": 273.88751220703125,
       "r_dpo/length_delta": 33.759376525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 240.1281280517578,
       "step": 370
     },
     {
       "epoch": 0.7958115183246073,
+      "grad_norm": 286.9093933105469,
       "learning_rate": 6.166331963291519e-08,
+      "logits/chosen": -0.7173658013343811,
+      "logits/rejected": -0.7228809595108032,
+      "logps/chosen": -300.5129699707031,
       "logps/ref_chosen": -301.07965087890625,
       "logps/ref_rejected": -266.7089538574219,
+      "logps/rejected": -274.7568359375,
+      "loss": 4.1149,
       "r_dpo/chosen_len": 286.75311279296875,
       "r_dpo/length_delta": 33.53125,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 253.2218780517578,
       "step": 380
     },
     {
       "epoch": 0.8167539267015707,
+      "grad_norm": 256.02996826171875,
       "learning_rate": 5.013930914912476e-08,
+      "logits/chosen": -0.7370638251304626,
+      "logits/rejected": -0.7369734048843384,
+      "logps/chosen": -295.3925476074219,
       "logps/ref_chosen": -296.6472473144531,
       "logps/ref_rejected": -278.953857421875,
+      "logps/rejected": -287.25274658203125,
+      "loss": 4.0032,
       "r_dpo/chosen_len": 287.91876220703125,
       "r_dpo/length_delta": 30.081249237060547,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 257.8374938964844,
       "step": 390
     },
     {
       "epoch": 0.837696335078534,
+      "grad_norm": 229.88453674316406,
       "learning_rate": 3.968661679220467e-08,
+      "logits/chosen": -0.726976752281189,
+      "logits/rejected": -0.7373378872871399,
+      "logps/chosen": -295.95574951171875,
       "logps/ref_chosen": -296.6556091308594,
       "logps/ref_rejected": -256.9266662597656,
+      "logps/rejected": -264.715576171875,
+      "loss": 4.0423,
       "r_dpo/chosen_len": 278.96875,
       "r_dpo/length_delta": 39.60625076293945,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 239.3625030517578,
       "step": 400
     },
     {
       "epoch": 0.837696335078534,
+      "eval_logits/chosen": -0.7442260980606079,
+      "eval_logits/rejected": -0.7460020780563354,
+      "eval_logps/chosen": -288.06787109375,
       "eval_logps/ref_chosen": -289.1346435546875,
       "eval_logps/ref_rejected": -264.7782287597656,
+      "eval_logps/rejected": -272.9751281738281,
+      "eval_loss": 0.5080092549324036,
       "eval_r_dpo/chosen_len": 291.2619934082031,
       "eval_r_dpo/length_delta": 42.86600112915039,
+      "eval_r_dpo/regularization_term": 0.0,
       "eval_r_dpo/rejected_len": 248.39599609375,
+      "eval_runtime": 82.0059,
+      "eval_samples_per_second": 24.388,
       "eval_steps_per_second": 1.524,
       "step": 400
     },
     {
       "epoch": 0.8586387434554974,
+      "grad_norm": 257.00933837890625,
       "learning_rate": 3.036127238347164e-08,
+      "logits/chosen": -0.7183164358139038,
+      "logits/rejected": -0.7075697183609009,
+      "logps/chosen": -290.19342041015625,
       "logps/ref_chosen": -290.08154296875,
       "logps/ref_rejected": -272.56988525390625,
+      "logps/rejected": -280.45965576171875,
+      "loss": 4.0132,
       "r_dpo/chosen_len": 282.40625,
       "r_dpo/length_delta": 26.265625,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 256.140625,
       "step": 410
     },
     {
       "epoch": 0.8795811518324608,
+      "grad_norm": 258.52166748046875,
       "learning_rate": 2.2213262793589482e-08,
+      "logits/chosen": -0.7189906239509583,
+      "logits/rejected": -0.7180750966072083,
+      "logps/chosen": -306.213623046875,
       "logps/ref_chosen": -307.40240478515625,
       "logps/ref_rejected": -279.85760498046875,
+      "logps/rejected": -287.7927551269531,
+      "loss": 3.9865,
       "r_dpo/chosen_len": 296.8343811035156,
       "r_dpo/length_delta": 37.165626525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 259.66876220703125,
       "step": 420
     },
     {
       "epoch": 0.900523560209424,
+      "grad_norm": 225.66854858398438,
       "learning_rate": 1.5286263996730026e-08,
+      "logits/chosen": -0.7201143503189087,
+      "logits/rejected": -0.7314451932907104,
+      "logps/chosen": -297.2125244140625,
       "logps/ref_chosen": -297.7133483886719,
       "logps/ref_rejected": -266.862060546875,
+      "logps/rejected": -276.1941223144531,
+      "loss": 4.0823,
       "r_dpo/chosen_len": 290.703125,
       "r_dpo/length_delta": 48.868751525878906,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 241.83438110351562,
       "step": 430
     },
     {
       "epoch": 0.9214659685863874,
+      "grad_norm": 296.7327880859375,
       "learning_rate": 9.617406953185136e-09,
+      "logits/chosen": -0.7148677706718445,
+      "logits/rejected": -0.7244837880134583,
+      "logps/chosen": -293.4369201660156,
       "logps/ref_chosen": -293.84521484375,
       "logps/ref_rejected": -289.6698303222656,
+      "logps/rejected": -297.15863037109375,
+      "loss": 4.2486,
       "r_dpo/chosen_len": 285.3656311035156,
       "r_dpo/length_delta": 12.868749618530273,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 272.49688720703125,
       "step": 440
     },
     {
       "epoch": 0.9424083769633508,
+      "grad_norm": 255.14761352539062,
       "learning_rate": 5.2370785753763356e-09,
+      "logits/chosen": -0.735411524772644,
+      "logits/rejected": -0.742519199848175,
+      "logps/chosen": -295.8509826660156,
       "logps/ref_chosen": -297.1932373046875,
       "logps/ref_rejected": -262.9091796875,
+      "logps/rejected": -271.23907470703125,
+      "loss": 3.8827,
       "r_dpo/chosen_len": 282.81561279296875,
       "r_dpo/length_delta": 40.631248474121094,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 242.1843719482422,
       "step": 450
     },
     {
       "epoch": 0.9633507853403142,
+      "grad_norm": 254.89694213867188,
       "learning_rate": 2.168758844148272e-09,
+      "logits/chosen": -0.7343727946281433,
+      "logits/rejected": -0.7490428686141968,
+      "logps/chosen": -311.859619140625,
       "logps/ref_chosen": -312.42291259765625,
       "logps/ref_rejected": -278.7356262207031,
+      "logps/rejected": -286.7757263183594,
+      "loss": 4.1604,
       "r_dpo/chosen_len": 288.9125061035156,
       "r_dpo/length_delta": 43.23125076293945,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 245.68124389648438,
       "step": 460
     },
     {
       "epoch": 0.9842931937172775,
+      "grad_norm": 231.78506469726562,
       "learning_rate": 4.288949484559934e-10,
+      "logits/chosen": -0.7186932563781738,
+      "logits/rejected": -0.7156038284301758,
+      "logps/chosen": -276.20965576171875,
       "logps/ref_chosen": -278.13385009765625,
       "logps/ref_rejected": -256.70648193359375,
+      "logps/rejected": -264.2013244628906,
+      "loss": 3.9555,
       "r_dpo/chosen_len": 268.8687438964844,
       "r_dpo/length_delta": 26.262500762939453,
+      "r_dpo/regularization_term": 0.0,
       "r_dpo/rejected_len": 242.6062469482422,
       "step": 470
     },
       "epoch": 0.9989528795811519,
       "step": 477,
       "total_flos": 0.0,
+      "train_loss": 4.32008807264284,
+      "train_runtime": 6099.6278,
+      "train_samples_per_second": 10.023,
       "train_steps_per_second": 0.078
     }
   ],