Upload iter_0079

.gitattributes

@@ -48,3 +48,4 @@ iter_0029/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 iter_0034/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 iter_0039/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 iter_0074/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+iter_0079/tokenizer.json filter=lfs diff=lfs merge=lfs -text

iter_0079/README.md

@@ -0,0 +1,207 @@
+---
+base_model: Qwen/Qwen2.5-Coder-7B-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:Qwen/Qwen2.5-Coder-7B-Instruct
+- lora
+- transformers
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

iter_0079/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-Coder-7B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "k_proj",
+    "v_proj",
+    "o_proj",
+    "gate_proj",
+    "down_proj",
+    "q_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

iter_0079/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e56fd6c8deeca863b2ef8a964d8a279d32803fae7d0baeafbf610a0bc834c123
+size 161533192

iter_0079/chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

iter_0079/dual_variables.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2e3774d94a62cd1b686b652a8b022a78337730903e2d54e6e0d0baaaf61bcbc
+size 1020

iter_0079/log.json

@@ -0,0 +1,1442 @@
+[
+  {
+    "iteration": 0,
+    "elapsed_s": 283.72707653045654,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6677320075757578,
+    "avg_questions": 0.71875,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.0012642779620364308,
+    "value_loss": 0.9520214434180941,
+    "approx_kl": 0.009626129126575376,
+    "clip_frac": 0.18303571481789863,
+    "entropy": 0.25815688552600996,
+    "kl_penalty": 0.011675153459821428,
+    "kl_per_seq": 0.011675153459821428,
+    "kl_seq_max": 0.18973214285714285
+  },
+  {
+    "iteration": 1,
+    "elapsed_s": 265.6153476238251,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6511522331834831,
+    "avg_questions": 0.75,
+    "avg_turns": 1.71875,
+    "ppo_loss": -0.02305267165814127,
+    "value_loss": 0.3821369180721896,
+    "approx_kl": 0.006722210083223347,
+    "clip_frac": 0.09438775533011981,
+    "entropy": 0.22343750271414006,
+    "kl_penalty": 0.010663713727678572,
+    "kl_per_seq": 0.010663713727678572,
+    "kl_seq_max": 0.14174107142857142
+  },
+  {
+    "iteration": 2,
+    "elapsed_s": 283.4707496166229,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6145833333333333,
+    "avg_questions": 0.71875,
+    "avg_turns": 1.6875,
+    "ppo_loss": 0.0003246570039274437,
+    "value_loss": 0.5281469119446618,
+    "approx_kl": 0.005086135942422386,
+    "clip_frac": 0.0595238097012043,
+    "entropy": 0.20671099424362183,
+    "kl_penalty": 0.022153581891741072,
+    "kl_per_seq": 0.022153581891741072,
+    "kl_seq_max": 0.15959821428571427
+  },
+  {
+    "iteration": 3,
+    "elapsed_s": 277.0273263454437,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8185267857142857,
+    "avg_questions": 0.875,
+    "avg_turns": 1.84375,
+    "ppo_loss": 0.017307979986071587,
+    "value_loss": 0.2391173227224499,
+    "approx_kl": 0.006241402057639789,
+    "clip_frac": 0.0859375,
+    "entropy": 0.24670861288905144,
+    "kl_penalty": 0.0052337646484375,
+    "kl_per_seq": 0.0052337646484375,
+    "kl_seq_max": 0.130126953125
+  },
+  {
+    "iteration": 4,
+    "elapsed_s": 286.0855031013489,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6283482142857142,
+    "avg_questions": 0.875,
+    "avg_turns": 1.875,
+    "ppo_loss": -0.023693807190284133,
+    "value_loss": 0.23578157555311918,
+    "approx_kl": 0.006290083052590489,
+    "clip_frac": 0.08984375,
+    "entropy": 0.2442768611945212,
+    "kl_penalty": 0.013458251953125,
+    "kl_per_seq": 0.013458251953125,
+    "kl_seq_max": 0.13818359375
+  },
+  {
+    "iteration": 5,
+    "elapsed_s": 242.1764998435974,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7465401785714285,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.78125,
+    "ppo_loss": -0.0412354568640391,
+    "value_loss": 0.36194167503466207,
+    "approx_kl": 0.02360554877668619,
+    "clip_frac": 0.21354166666666666,
+    "entropy": 0.28217028516034287,
+    "kl_penalty": 0.044108072916666664,
+    "kl_per_seq": 0.044108072916666664,
+    "kl_seq_max": 0.22916666666666666
+  },
+  {
+    "iteration": 6,
+    "elapsed_s": 297.9436032772064,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6431051587301587,
+    "avg_questions": 0.84375,
+    "avg_turns": 1.84375,
+    "ppo_loss": -0.01692520675715059,
+    "value_loss": 0.28041012212634087,
+    "approx_kl": 0.006847011674835812,
+    "clip_frac": 0.11197916697710752,
+    "entropy": 0.25802480173297226,
+    "kl_penalty": 0.01638031005859375,
+    "kl_per_seq": 0.01638031005859375,
+    "kl_seq_max": 0.14501953125
+  },
+  {
+    "iteration": 7,
+    "elapsed_s": 330.66188168525696,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6246899801587302,
+    "avg_questions": 0.8125,
+    "avg_turns": 1.78125,
+    "ppo_loss": 0.004835611267480999,
+    "value_loss": 0.42471681741881184,
+    "approx_kl": 0.006292660720646381,
+    "clip_frac": 0.078125,
+    "entropy": 0.22681239678058773,
+    "kl_penalty": 0.0085601806640625,
+    "kl_per_seq": 0.0085601806640625,
+    "kl_seq_max": 0.14404296875
+  },
+  {
+    "iteration": 8,
+    "elapsed_s": 300.30717039108276,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6534733495670996,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.017197133707148687,
+    "value_loss": 0.3457076509616205,
+    "approx_kl": 0.006416547777397292,
+    "clip_frac": 0.07589285714285714,
+    "entropy": 0.22316500225237437,
+    "kl_penalty": 0.004743303571428571,
+    "kl_per_seq": 0.004743303571428571,
+    "kl_seq_max": 0.197265625
+  },
+  {
+    "iteration": 9,
+    "elapsed_s": 286.7236270904541,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7079376352813853,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.011897004608597075,
+    "value_loss": 0.411878263843911,
+    "approx_kl": 0.007929061938609396,
+    "clip_frac": 0.11607142857142858,
+    "entropy": 0.2569970648203577,
+    "kl_penalty": 0.00020926339285714285,
+    "kl_per_seq": 0.00020926339285714285,
+    "kl_seq_max": 0.16936383928571427
+  },
+  {
+    "iteration": 10,
+    "elapsed_s": 276.631822347641,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8028939810189811,
+    "avg_questions": 0.84375,
+    "avg_turns": 1.78125,
+    "ppo_loss": -0.033312247483991086,
+    "value_loss": 0.2869396444875747,
+    "approx_kl": 0.008976982207968831,
+    "clip_frac": 0.12109375,
+    "entropy": 0.25112444773549214,
+    "kl_penalty": 0.01202392578125,
+    "kl_per_seq": 0.01202392578125,
+    "kl_seq_max": 0.203857421875
+  },
+  {
+    "iteration": 11,
+    "elapsed_s": 255.2351098060608,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.5704326923076923,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.005331723640362422,
+    "value_loss": 0.42935604602098465,
+    "approx_kl": 0.016193445358011458,
+    "clip_frac": 0.2222222222222222,
+    "entropy": 0.23103458185990652,
+    "kl_penalty": 0.0,
+    "kl_per_seq": 0.0,
+    "kl_seq_max": 0.2294921875
+  },
+  {
+    "iteration": 12,
+    "elapsed_s": 254.100581407547,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.5869724025974025,
+    "avg_questions": 0.71875,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.03959607765344637,
+    "value_loss": 0.34272722474166323,
+    "approx_kl": 0.012407979046526765,
+    "clip_frac": 0.20684523933700152,
+    "entropy": 0.24096926195280893,
+    "kl_penalty": 0.007428850446428571,
+    "kl_per_seq": 0.007428850446428571,
+    "kl_seq_max": 0.2572544642857143
+  },
+  {
+    "iteration": 13,
+    "elapsed_s": 335.7170238494873,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6823145604395605,
+    "avg_questions": 0.84375,
+    "avg_turns": 1.84375,
+    "ppo_loss": -0.03688290709008773,
+    "value_loss": 0.21781166903674604,
+    "approx_kl": 0.013189310727951428,
+    "clip_frac": 0.19583333432674407,
+    "entropy": 0.2323363353808721,
+    "kl_penalty": 0.024055989583333333,
+    "kl_per_seq": 0.024055989583333333,
+    "kl_seq_max": 0.3768229166666667
+  },
+  {
+    "iteration": 14,
+    "elapsed_s": 264.29171442985535,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7472098214285713,
+    "avg_questions": 0.875,
+    "avg_turns": 1.875,
+    "ppo_loss": -0.027799080746869247,
+    "value_loss": 0.4135706809659799,
+    "approx_kl": 0.012586234758297602,
+    "clip_frac": 0.15625,
+    "entropy": 0.2517068100472291,
+    "kl_penalty": 0.0135498046875,
+    "kl_per_seq": 0.0135498046875,
+    "kl_seq_max": 0.3229166666666667
+  },
+  {
+    "iteration": 15,
+    "elapsed_s": 241.14811420440674,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6659226190476191,
+    "avg_questions": 0.65625,
+    "avg_turns": 1.65625,
+    "ppo_loss": -0.03369090032009851,
+    "value_loss": 0.34638335804144543,
+    "approx_kl": 0.013905501381183663,
+    "clip_frac": 0.19047619189534867,
+    "entropy": 0.22407633740277516,
+    "kl_penalty": 0.036342075892857144,
+    "kl_per_seq": 0.036342075892857144,
+    "kl_seq_max": 0.45982142857142855
+  },
+  {
+    "iteration": 16,
+    "elapsed_s": 238.80150389671326,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6585475288600288,
+    "avg_questions": 0.90625,
+    "avg_turns": 1.84375,
+    "ppo_loss": -0.056268499715736615,
+    "value_loss": 0.3362144205581259,
+    "approx_kl": 0.019216162325055512,
+    "clip_frac": 0.3024691376421187,
+    "entropy": 0.2751925124062432,
+    "kl_penalty": 0.022162543402777776,
+    "kl_per_seq": 0.022162543402777776,
+    "kl_seq_max": 0.5263310185185185
+  },
+  {
+    "iteration": 17,
+    "elapsed_s": 232.22505855560303,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7668154761904762,
+    "avg_questions": 0.90625,
+    "avg_turns": 1.875,
+    "ppo_loss": -0.0331775728199217,
+    "value_loss": 0.3092526584449742,
+    "approx_kl": 0.01231207988328404,
+    "clip_frac": 0.20833333333333334,
+    "entropy": 0.25865906063053345,
+    "kl_penalty": 0.07600911458333333,
+    "kl_per_seq": 0.07600911458333333,
+    "kl_seq_max": 0.65625
+  },
+  {
+    "iteration": 18,
+    "elapsed_s": 311.01340675354004,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6972830988455989,
+    "avg_questions": 0.8125,
+    "avg_turns": 1.78125,
+    "ppo_loss": 0.009420987218618393,
+    "value_loss": 0.4465955500304699,
+    "approx_kl": 0.015329486131668091,
+    "clip_frac": 0.26,
+    "entropy": 0.21777834206819535,
+    "kl_penalty": 0.04673828125,
+    "kl_per_seq": 0.04673828125,
+    "kl_seq_max": 0.56625
+  },
+  {
+    "iteration": 19,
+    "elapsed_s": 249.58963227272034,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7551739926739928,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.037938538430766625,
+    "value_loss": 0.36645138856362214,
+    "approx_kl": 0.018135146047411996,
+    "clip_frac": 0.2746212130243128,
+    "entropy": 0.24184788357127796,
+    "kl_penalty": 0.002840909090909091,
+    "kl_per_seq": 0.002840909090909091,
+    "kl_seq_max": 0.6051136363636364
+  },
+  {
+    "iteration": 20,
+    "elapsed_s": 292.7258870601654,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7158272283272281,
+    "avg_questions": 0.8125,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.04037410455445448,
+    "value_loss": 0.35967613384127617,
+    "approx_kl": 0.016659200191497803,
+    "clip_frac": 0.23958333333333334,
+    "entropy": 0.24879397824406624,
+    "kl_penalty": 0.111328125,
+    "kl_per_seq": 0.111328125,
+    "kl_seq_max": 1.0013020833333333
+  },
+  {
+    "iteration": 21,
+    "elapsed_s": 184.79159307479858,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.743264678030303,
+    "avg_questions": 0.75,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.019979689947583458,
+    "value_loss": 0.42602486231110315,
+    "approx_kl": 0.009891370269046589,
+    "clip_frac": 0.11363636363636363,
+    "entropy": 0.2355451394211162,
+    "kl_penalty": 0.09295099431818182,
+    "kl_per_seq": 0.09295099431818182,
+    "kl_seq_max": 0.8309659090909091
+  },
+  {
+    "iteration": 22,
+    "elapsed_s": 269.67527770996094,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.731497061965812,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.06142320060594515,
+    "value_loss": 0.32878022911873733,
+    "approx_kl": 0.01706851857968352,
+    "clip_frac": 0.28598484938794916,
+    "entropy": 0.22095515849915418,
+    "kl_penalty": 0.03235973011363636,
+    "kl_per_seq": 0.03235973011363636,
+    "kl_seq_max": 0.8068181818181818
+  },
+  {
+    "iteration": 23,
+    "elapsed_s": 287.49955129623413,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7941430097680098,
+    "avg_questions": 0.90625,
+    "avg_turns": 1.90625,
+    "ppo_loss": -0.02508886717259884,
+    "value_loss": 0.23930740232268968,
+    "approx_kl": 0.01800820985550268,
+    "clip_frac": 0.20972222255335915,
+    "entropy": 0.26301920496755177,
+    "kl_penalty": 0.2238498263888889,
+    "kl_per_seq": 0.2238498263888889,
+    "kl_seq_max": 1.0772569444444444
+  },
+  {
+    "iteration": 24,
+    "elapsed_s": 234.66244316101074,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7230979957542457,
+    "avg_questions": 0.75,
+    "avg_turns": 1.71875,
+    "ppo_loss": -0.007767030461267991,
+    "value_loss": 0.5475986464457079,
+    "approx_kl": 0.014794111082499678,
+    "clip_frac": 0.27435064993121405,
+    "entropy": 0.24171581187031485,
+    "kl_penalty": 0.06498579545454546,
+    "kl_per_seq": 0.06498579545454546,
+    "kl_seq_max": 0.9232954545454546
+  },
+  {
+    "iteration": 25,
+    "elapsed_s": 319.03157591819763,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6251251352813854,
+    "avg_questions": 0.75,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.04657436514066325,
+    "value_loss": 0.41042184829711914,
+    "approx_kl": 0.023139544607450564,
+    "clip_frac": 0.2708333333333333,
+    "entropy": 0.24344126342071426,
+    "kl_penalty": 0.04966905381944445,
+    "kl_per_seq": 0.04966905381944445,
+    "kl_seq_max": 0.7352430555555556
+  },
+  {
+    "iteration": 26,
+    "elapsed_s": 278.54370284080505,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7391369047619047,
+    "avg_questions": 0.90625,
+    "avg_turns": 1.84375,
+    "ppo_loss": -0.011602610815316439,
+    "value_loss": 0.37313461070880294,
+    "approx_kl": 0.0172485489747487,
+    "clip_frac": 0.25781250186264515,
+    "entropy": 0.2445761663839221,
+    "kl_penalty": 0.2266845703125,
+    "kl_per_seq": 0.2266845703125,
+    "kl_seq_max": 1.1806640625
+  },
+  {
+    "iteration": 27,
+    "elapsed_s": 246.0258011817932,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.654265873015873,
+    "avg_questions": 0.84375,
+    "avg_turns": 1.78125,
+    "ppo_loss": -0.00885420270702418,
+    "value_loss": 0.40379785308066535,
+    "approx_kl": 0.017032040392651278,
+    "clip_frac": 0.20588235294117646,
+    "entropy": 0.22635302780305638,
+    "kl_penalty": 0.062270220588235295,
+    "kl_per_seq": 0.062270220588235295,
+    "kl_seq_max": 0.6709558823529411
+  },
+  {
+    "iteration": 28,
+    "elapsed_s": 328.36840200424194,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.715736607142857,
+    "avg_questions": 0.90625,
+    "avg_turns": 1.78125,
+    "ppo_loss": -0.03573833931894863,
+    "value_loss": 0.44778402368812,
+    "approx_kl": 0.016466796398162842,
+    "clip_frac": 0.25735294117647056,
+    "entropy": 0.27092264066724214,
+    "kl_penalty": 0.3297334558823529,
+    "kl_per_seq": 0.3297334558823529,
+    "kl_seq_max": 1.3786764705882353
+  },
+  {
+    "iteration": 29,
+    "elapsed_s": 263.61545872688293,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7022214764402265,
+    "avg_questions": 0.875,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.042088701079289116,
+    "value_loss": 0.374366395175457,
+    "approx_kl": 0.0180306788533926,
+    "clip_frac": 0.15625,
+    "entropy": 0.2805991160372893,
+    "kl_penalty": 0.06697591145833333,
+    "kl_per_seq": 0.06697591145833333,
+    "kl_seq_max": 1.2955729166666667
+  },
+  {
+    "iteration": 30,
+    "elapsed_s": 303.52203917503357,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6189224837662338,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.009862170120080312,
+    "value_loss": 0.588172604640325,
+    "approx_kl": 0.018847733736038208,
+    "clip_frac": 0.21666666666666667,
+    "entropy": 0.2482275903224945,
+    "kl_penalty": 0.0869140625,
+    "kl_per_seq": 0.0869140625,
+    "kl_seq_max": 0.8333333333333334
+  },
+  {
+    "iteration": 31,
+    "elapsed_s": 249.05430436134338,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7663931988150738,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.055953570586793566,
+    "value_loss": 0.4904465508811614,
+    "approx_kl": 0.018582439159645754,
+    "clip_frac": 0.2867647058823529,
+    "entropy": 0.21470373255365036,
+    "kl_penalty": 0.09777113970588236,
+    "kl_per_seq": 0.09777113970588236,
+    "kl_seq_max": 1.2426470588235294
+  },
+  {
+    "iteration": 32,
+    "elapsed_s": 212.35868120193481,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.75,
+    "avg_questions": 0.65625,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.03906370233744383,
+    "value_loss": 0.45569973066449165,
+    "approx_kl": 0.01507616974413395,
+    "clip_frac": 0.234375,
+    "entropy": 0.21894092485308647,
+    "kl_penalty": 0.0474853515625,
+    "kl_per_seq": 0.0474853515625,
+    "kl_seq_max": 0.763671875
+  },
+  {
+    "iteration": 33,
+    "elapsed_s": 326.989426612854,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.5256744123931625,
+    "avg_questions": 0.90625,
+    "avg_turns": 1.8125,
+    "ppo_loss": -0.03463981207460165,
+    "value_loss": 0.4421174842864275,
+    "approx_kl": 0.02097482793033123,
+    "clip_frac": 0.328125,
+    "entropy": 0.24500909820199013,
+    "kl_penalty": 0.127197265625,
+    "kl_per_seq": 0.127197265625,
+    "kl_seq_max": 1.5625
+  },
+  {
+    "iteration": 34,
+    "elapsed_s": 274.3689548969269,
+    "lambda1": 0.009375000000000001,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7299278846153846,
+    "avg_questions": 1.09375,
+    "avg_turns": 1.9375,
+    "ppo_loss": -0.03937706899117021,
+    "value_loss": 1.6494529334937824,
+    "approx_kl": 0.01805254674571402,
+    "clip_frac": 0.29411764793536244,
+    "entropy": 0.22951026655295315,
+    "kl_penalty": 0.19114774816176472,
+    "kl_per_seq": 0.19114774816176472,
+    "kl_seq_max": 1.5441176470588236
+  },
+  {
+    "iteration": 35,
+    "elapsed_s": 330.2895474433899,
+    "lambda1": 0.003125000000000001,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7578525641025642,
+    "avg_questions": 0.9375,
+    "avg_turns": 1.84375,
+    "ppo_loss": -0.03926877165213227,
+    "value_loss": 0.3948600087314844,
+    "approx_kl": 0.01893954398110509,
+    "clip_frac": 0.24218750186264515,
+    "entropy": 0.25698490906506777,
+    "kl_penalty": 0.0867919921875,
+    "kl_per_seq": 0.0867919921875,
+    "kl_seq_max": 1.2705078125
+  },
+  {
+    "iteration": 36,
+    "elapsed_s": 237.82562851905823,
+    "lambda1": 8.673617379884035e-19,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7183847402597403,
+    "avg_questions": 0.96875,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.021690011769533158,
+    "value_loss": 0.4621114403009415,
+    "approx_kl": 0.01778200566768646,
+    "clip_frac": 0.1625,
+    "entropy": 0.23396530598402024,
+    "kl_penalty": 0.23720703125,
+    "kl_per_seq": 0.23720703125,
+    "kl_seq_max": 2.1328125
+  },
+  {
+    "iteration": 37,
+    "elapsed_s": 306.2567505836487,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7560436958874459,
+    "avg_questions": 0.8125,
+    "avg_turns": 1.71875,
+    "ppo_loss": -0.046103228349238634,
+    "value_loss": 0.4866797626018524,
+    "approx_kl": 0.02149446721887216,
+    "clip_frac": 0.26785714365541935,
+    "entropy": 0.21997725404798985,
+    "kl_penalty": 0.22509765625,
+    "kl_per_seq": 0.22509765625,
+    "kl_seq_max": 2.1708984375
+  },
+  {
+    "iteration": 38,
+    "elapsed_s": 281.4996678829193,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8712568681318682,
+    "avg_questions": 0.84375,
+    "avg_turns": 1.71875,
+    "ppo_loss": -0.033675035120298465,
+    "value_loss": 0.32771471751232945,
+    "approx_kl": 0.02097547585144639,
+    "clip_frac": 0.2523809532324473,
+    "entropy": 0.2418164610862732,
+    "kl_penalty": 0.29797261555989585,
+    "kl_per_seq": 0.29797261555989585,
+    "kl_seq_max": 1.7291666666666667
+  },
+  {
+    "iteration": 39,
+    "elapsed_s": 244.16778016090393,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.858400974025974,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.65625,
+    "ppo_loss": 0.02497277222573757,
+    "value_loss": 0.3133168024942279,
+    "approx_kl": 0.02143717440776527,
+    "clip_frac": 0.19687500037252903,
+    "entropy": 0.22355329897254705,
+    "kl_penalty": 0.476318359375,
+    "kl_per_seq": 0.476318359375,
+    "kl_seq_max": 2.849609375
+  },
+  {
+    "iteration": 40,
+    "elapsed_s": 270.5617165565491,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7012885551948053,
+    "avg_questions": 0.71875,
+    "avg_turns": 1.65625,
+    "ppo_loss": -0.02896590158343315,
+    "value_loss": 0.5044428666432699,
+    "approx_kl": 0.018548924351731937,
+    "clip_frac": 0.23166666726271312,
+    "entropy": 0.2537897606690725,
+    "kl_penalty": 0.33916015625,
+    "kl_per_seq": 0.33916015625,
+    "kl_seq_max": 2.1421875
+  },
+  {
+    "iteration": 41,
+    "elapsed_s": 227.65656161308289,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7698569225912976,
+    "avg_questions": 0.71875,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.015438761096447706,
+    "value_loss": 0.3268631212413311,
+    "approx_kl": 0.011113337241113186,
+    "clip_frac": 0.125,
+    "entropy": 0.2236450556665659,
+    "kl_penalty": 0.373291015625,
+    "kl_per_seq": 0.373291015625,
+    "kl_seq_max": 2.173828125
+  },
+  {
+    "iteration": 42,
+    "elapsed_s": 252.90317630767822,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6684659090909091,
+    "avg_questions": 0.8125,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.053143983972924094,
+    "value_loss": 0.4450294630868094,
+    "approx_kl": 0.01749115596924509,
+    "clip_frac": 0.22321428571428573,
+    "entropy": 0.22862583505255835,
+    "kl_penalty": 0.15562220982142858,
+    "kl_per_seq": 0.15562220982142858,
+    "kl_seq_max": 1.6573660714285714
+  },
+  {
+    "iteration": 43,
+    "elapsed_s": 258.8288621902466,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6711185515873015,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.71875,
+    "ppo_loss": -0.05501775095860163,
+    "value_loss": 0.4743456025918325,
+    "approx_kl": 0.017434147807459038,
+    "clip_frac": 0.24523809651533762,
+    "entropy": 0.23283665478229523,
+    "kl_penalty": 0.2046875,
+    "kl_per_seq": 0.2046875,
+    "kl_seq_max": 1.6729166666666666
+  },
+  {
+    "iteration": 44,
+    "elapsed_s": 250.52874207496643,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7868607954545455,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.6875,
+    "ppo_loss": -0.029178939903011687,
+    "value_loss": 0.34881506172510296,
+    "approx_kl": 0.02014158207636613,
+    "clip_frac": 0.23397435935644004,
+    "entropy": 0.23651472994914421,
+    "kl_penalty": 0.35682091346153844,
+    "kl_per_seq": 0.35682091346153844,
+    "kl_seq_max": 1.8653846153846154
+  },
+  {
+    "iteration": 45,
+    "elapsed_s": 285.62387681007385,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7336920891608392,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.048381321132183075,
+    "value_loss": 0.4691288539996514,
+    "approx_kl": 0.01970805571629451,
+    "clip_frac": 0.25961538461538464,
+    "entropy": 0.25246551403632533,
+    "kl_penalty": 0.455078125,
+    "kl_per_seq": 0.455078125,
+    "kl_seq_max": 2.2836538461538463
+  },
+  {
+    "iteration": 46,
+    "elapsed_s": 278.89680767059326,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7764756944444444,
+    "avg_questions": 0.65625,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.011009233919056978,
+    "value_loss": 0.3233566351912238,
+    "approx_kl": 0.025823744183236904,
+    "clip_frac": 0.26136363636363635,
+    "entropy": 0.2608498538082296,
+    "kl_penalty": 0.5964133522727273,
+    "kl_per_seq": 0.5964133522727273,
+    "kl_seq_max": 2.2457386363636362
+  },
+  {
+    "iteration": 47,
+    "elapsed_s": 276.2091202735901,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7869791666666667,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.78125,
+    "ppo_loss": -0.03388428608221667,
+    "value_loss": 0.3851106393017939,
+    "approx_kl": 0.014619423342602593,
+    "clip_frac": 0.1875,
+    "entropy": 0.23170028374131238,
+    "kl_penalty": 0.4081333705357143,
+    "kl_per_seq": 0.4081333705357143,
+    "kl_seq_max": 1.9955357142857142
+  },
+  {
+    "iteration": 48,
+    "elapsed_s": 236.10829496383667,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.760744724025974,
+    "avg_questions": 0.65625,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.029560362750833683,
+    "value_loss": 0.30854430252855475,
+    "approx_kl": 0.012752801179885864,
+    "clip_frac": 0.125,
+    "entropy": 0.2456689097664573,
+    "kl_penalty": 0.12127130681818182,
+    "kl_per_seq": 0.12127130681818182,
+    "kl_seq_max": 1.0795454545454546
+  },
+  {
+    "iteration": 49,
+    "elapsed_s": 244.12733364105225,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7892992424242425,
+    "avg_questions": 0.53125,
+    "avg_turns": 1.5,
+    "ppo_loss": -0.02731230203062296,
+    "value_loss": 0.34456007927656174,
+    "approx_kl": 0.016848372295498848,
+    "clip_frac": 0.21875,
+    "entropy": 0.20181632600724697,
+    "kl_penalty": 0.60888671875,
+    "kl_per_seq": 0.60888671875,
+    "kl_seq_max": 2.890625
+  },
+  {
+    "iteration": 50,
+    "elapsed_s": 268.71904134750366,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.71171875,
+    "avg_questions": 0.625,
+    "avg_turns": 1.53125,
+    "ppo_loss": -0.044096055626869204,
+    "value_loss": 0.4949160411953926,
+    "approx_kl": 0.0211080864071846,
+    "clip_frac": 0.275,
+    "entropy": 0.19351754635572432,
+    "kl_penalty": 0.5908203125,
+    "kl_per_seq": 0.5908203125,
+    "kl_seq_max": 2.521875
+  },
+  {
+    "iteration": 51,
+    "elapsed_s": 271.2546772956848,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8065273268398269,
+    "avg_questions": 0.9375,
+    "avg_turns": 1.84375,
+    "ppo_loss": -0.03524641847858826,
+    "value_loss": 0.3588395019372304,
+    "approx_kl": 0.015938463727555547,
+    "clip_frac": 0.14583333333333334,
+    "entropy": 0.22262138252456984,
+    "kl_penalty": 0.59375,
+    "kl_per_seq": 0.59375,
+    "kl_seq_max": 2.57421875
+  },
+  {
+    "iteration": 52,
+    "elapsed_s": 256.3747491836548,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.710558712121212,
+    "avg_questions": 0.71875,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.05150786861777305,
+    "value_loss": 0.4820778578519821,
+    "approx_kl": 0.014533989131450653,
+    "clip_frac": 0.2,
+    "entropy": 0.22074273526668547,
+    "kl_penalty": 0.88818359375,
+    "kl_per_seq": 0.88818359375,
+    "kl_seq_max": 2.5921875
+  },
+  {
+    "iteration": 53,
+    "elapsed_s": 239.81769323349,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8397693452380953,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.028743519680574536,
+    "value_loss": 0.3174218134954572,
+    "approx_kl": 0.02339409850537777,
+    "clip_frac": 0.234375,
+    "entropy": 0.21596916718408465,
+    "kl_penalty": 0.4505615234375,
+    "kl_per_seq": 0.4505615234375,
+    "kl_seq_max": 2.5791015625
+  },
+  {
+    "iteration": 54,
+    "elapsed_s": 245.02525782585144,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7170491834554334,
+    "avg_questions": 0.5625,
+    "avg_turns": 1.46875,
+    "ppo_loss": -0.02395312935113907,
+    "value_loss": 0.438307486474514,
+    "approx_kl": 0.01764700987841934,
+    "clip_frac": 0.1875,
+    "entropy": 0.20973262637853624,
+    "kl_penalty": 0.541796875,
+    "kl_per_seq": 0.541796875,
+    "kl_seq_max": 2.7859375
+  },
+  {
+    "iteration": 55,
+    "elapsed_s": 248.9802281856537,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7436992694805196,
+    "avg_questions": 0.53125,
+    "avg_turns": 1.5,
+    "ppo_loss": -0.027877020090818404,
+    "value_loss": 0.32779628187417986,
+    "approx_kl": 0.018842114508152007,
+    "clip_frac": 0.2375,
+    "entropy": 0.21401429921388626,
+    "kl_penalty": 0.9197265625,
+    "kl_per_seq": 0.9197265625,
+    "kl_seq_max": 2.659375
+  },
+  {
+    "iteration": 56,
+    "elapsed_s": 279.31820583343506,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8969358766233766,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.75,
+    "ppo_loss": -0.025293706357479094,
+    "value_loss": 0.18105527286728224,
+    "approx_kl": 0.020793992777665457,
+    "clip_frac": 0.23333333333333334,
+    "entropy": 0.23770597924788792,
+    "kl_penalty": 0.7494791666666667,
+    "kl_per_seq": 0.7494791666666667,
+    "kl_seq_max": 3.176041666666667
+  },
+  {
+    "iteration": 57,
+    "elapsed_s": 221.75046491622925,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7461241883116883,
+    "avg_questions": 0.5,
+    "avg_turns": 1.5,
+    "ppo_loss": -0.03699349205602299,
+    "value_loss": 0.37915432724085724,
+    "approx_kl": 0.01956542039459402,
+    "clip_frac": 0.17045454545454544,
+    "entropy": 0.21263412928039377,
+    "kl_penalty": 0.75390625,
+    "kl_per_seq": 0.75390625,
+    "kl_seq_max": 3.0255681818181817
+  },
+  {
+    "iteration": 58,
+    "elapsed_s": 294.8354675769806,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8488417832167832,
+    "avg_questions": 0.78125,
+    "avg_turns": 1.65625,
+    "ppo_loss": -0.019974307715892793,
+    "value_loss": 0.4829483419656754,
+    "approx_kl": 0.020990538224577904,
+    "clip_frac": 0.27750000059604646,
+    "entropy": 0.24574429392814637,
+    "kl_penalty": 0.7201171875,
+    "kl_per_seq": 0.7201171875,
+    "kl_seq_max": 2.884375
+  },
+  {
+    "iteration": 59,
+    "elapsed_s": 263.8665945529938,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.792870670995671,
+    "avg_questions": 0.59375,
+    "avg_turns": 1.5625,
+    "ppo_loss": -0.0022541616450656543,
+    "value_loss": 0.5600416118448431,
+    "approx_kl": 0.021971105174584823,
+    "clip_frac": 0.23863636363636365,
+    "entropy": 0.28346856886690314,
+    "kl_penalty": 0.8048650568181818,
+    "kl_per_seq": 0.8048650568181818,
+    "kl_seq_max": 4.284090909090909
+  },
+  {
+    "iteration": 60,
+    "elapsed_s": 301.4362123012543,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6950892857142857,
+    "avg_questions": 0.5,
+    "avg_turns": 1.4375,
+    "ppo_loss": -0.021061294194724824,
+    "value_loss": 0.4493260218037499,
+    "approx_kl": 0.018223573350244097,
+    "clip_frac": 0.2314814825852712,
+    "entropy": 0.23080702126026154,
+    "kl_penalty": 0.7815755208333334,
+    "kl_per_seq": 0.7815755208333334,
+    "kl_seq_max": 3.6006944444444446
+  },
+  {
+    "iteration": 61,
+    "elapsed_s": 300.4338631629944,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7377480158730159,
+    "avg_questions": 0.46875,
+    "avg_turns": 1.46875,
+    "ppo_loss": -0.01074105961015448,
+    "value_loss": 0.3422271218150854,
+    "approx_kl": 0.016436716279713437,
+    "clip_frac": 0.09375,
+    "entropy": 0.19642564747482538,
+    "kl_penalty": 0.74609375,
+    "kl_per_seq": 0.74609375,
+    "kl_seq_max": 2.431640625
+  },
+  {
+    "iteration": 62,
+    "elapsed_s": 250.01636481285095,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6838068181818182,
+    "avg_questions": 0.53125,
+    "avg_turns": 1.5,
+    "ppo_loss": -0.030568745732307435,
+    "value_loss": 0.4051216900348663,
+    "approx_kl": 0.015840204805135726,
+    "clip_frac": 0.225,
+    "entropy": 0.23866871148347854,
+    "kl_penalty": 0.4041015625,
+    "kl_per_seq": 0.4041015625,
+    "kl_seq_max": 3.096875
+  },
+  {
+    "iteration": 63,
+    "elapsed_s": 253.27067923545837,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.8834034455128205,
+    "avg_questions": 0.53125,
+    "avg_turns": 1.53125,
+    "ppo_loss": -0.0541976864139239,
+    "value_loss": 0.2538473131135106,
+    "approx_kl": 0.024151877189675968,
+    "clip_frac": 0.21875,
+    "entropy": 0.23326275746027628,
+    "kl_penalty": 0.8837890625,
+    "kl_per_seq": 0.8837890625,
+    "kl_seq_max": 3.3515625
+  },
+  {
+    "iteration": 64,
+    "elapsed_s": 300.5772490501404,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.780899916056166,
+    "avg_questions": 0.59375,
+    "avg_turns": 1.5625,
+    "ppo_loss": -0.0275198295712471,
+    "value_loss": 0.32388446231683093,
+    "approx_kl": 0.021999106638961367,
+    "clip_frac": 0.1388888888888889,
+    "entropy": 0.2057228833436966,
+    "kl_penalty": 0.7454427083333334,
+    "kl_per_seq": 0.7454427083333334,
+    "kl_seq_max": 3.0885416666666665
+  },
+  {
+    "iteration": 65,
+    "elapsed_s": 306.58283853530884,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7140117694805196,
+    "avg_questions": 0.5625,
+    "avg_turns": 1.4375,
+    "ppo_loss": -0.017171917110681535,
+    "value_loss": 0.5959388718008996,
+    "approx_kl": 0.019240123592317103,
+    "clip_frac": 0.20416666716337203,
+    "entropy": 0.25609137415885924,
+    "kl_penalty": 0.85302734375,
+    "kl_per_seq": 0.85302734375,
+    "kl_seq_max": 3.390625
+  },
+  {
+    "iteration": 66,
+    "elapsed_s": 318.9189774990082,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6512896825396824,
+    "avg_questions": 0.46875,
+    "avg_turns": 1.4375,
+    "ppo_loss": -7.790114198412214e-05,
+    "value_loss": 0.41655169427394867,
+    "approx_kl": 0.01800111441739968,
+    "clip_frac": 0.24404762046677725,
+    "entropy": 0.2527682568345751,
+    "kl_penalty": 1.2622767857142858,
+    "kl_per_seq": 1.2622767857142858,
+    "kl_seq_max": 3.919642857142857
+  },
+  {
+    "iteration": 67,
+    "elapsed_s": 244.33280968666077,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7559513403263405,
+    "avg_questions": 0.4375,
+    "avg_turns": 1.34375,
+    "ppo_loss": -0.014708490188543996,
+    "value_loss": 0.3386020374794801,
+    "approx_kl": 0.025815573520958424,
+    "clip_frac": 0.2743055572112401,
+    "entropy": 0.21764958649873734,
+    "kl_penalty": 0.61328125,
+    "kl_per_seq": 0.61328125,
+    "kl_seq_max": 3.5286458333333335
+  },
+  {
+    "iteration": 68,
+    "elapsed_s": 244.10136604309082,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7436631944444444,
+    "avg_questions": 0.46875,
+    "avg_turns": 1.46875,
+    "ppo_loss": -0.038494867139628956,
+    "value_loss": 0.5338093712925911,
+    "approx_kl": 0.018255861808678935,
+    "clip_frac": 0.2270408176950046,
+    "entropy": 0.18586944522602217,
+    "kl_penalty": 0.6918247767857143,
+    "kl_per_seq": 0.6918247767857143,
+    "kl_seq_max": 3.2232142857142856
+  },
+  {
+    "iteration": 69,
+    "elapsed_s": 363.0174329280853,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.5770449168886668,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.625,
+    "ppo_loss": -0.013602728645006815,
+    "value_loss": 0.4071305493513743,
+    "approx_kl": 0.020264718267652724,
+    "clip_frac": 0.25,
+    "entropy": 0.2366808040274514,
+    "kl_penalty": 0.5333116319444444,
+    "kl_per_seq": 0.5333116319444444,
+    "kl_seq_max": 3.5
+  },
+  {
+    "iteration": 70,
+    "elapsed_s": 287.4284553527832,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6315848214285714,
+    "avg_questions": 0.53125,
+    "avg_turns": 1.4375,
+    "ppo_loss": -0.007373106932001454,
+    "value_loss": 0.5669373742171696,
+    "approx_kl": 0.02011945703998208,
+    "clip_frac": 0.17857142857142858,
+    "entropy": 0.23269647572721755,
+    "kl_penalty": 0.705078125,
+    "kl_per_seq": 0.705078125,
+    "kl_seq_max": 4.09375
+  },
+  {
+    "iteration": 71,
+    "elapsed_s": 308.7073607444763,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7120084776334776,
+    "avg_questions": 0.4375,
+    "avg_turns": 1.40625,
+    "ppo_loss": -0.0013772226870059968,
+    "value_loss": 0.3877395071089268,
+    "approx_kl": 0.02122223093174398,
+    "clip_frac": 0.2950000002980232,
+    "entropy": 0.22844351828098297,
+    "kl_penalty": 0.48984375,
+    "kl_per_seq": 0.48984375,
+    "kl_seq_max": 3.303125
+  },
+  {
+    "iteration": 72,
+    "elapsed_s": 305.704142332077,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7791684877622378,
+    "avg_questions": 0.53125,
+    "avg_turns": 1.46875,
+    "ppo_loss": -0.01751005243171345,
+    "value_loss": 0.3979918211698532,
+    "approx_kl": 0.023104452528059483,
+    "clip_frac": 0.28571428629485046,
+    "entropy": 0.22728038443760437,
+    "kl_penalty": 0.69921875,
+    "kl_per_seq": 0.69921875,
+    "kl_seq_max": 2.9886363636363638
+  },
+  {
+    "iteration": 73,
+    "elapsed_s": 255.09643387794495,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6977220695970695,
+    "avg_questions": 0.375,
+    "avg_turns": 1.375,
+    "ppo_loss": -0.004332103663020664,
+    "value_loss": 0.3307652622461319,
+    "approx_kl": 0.016643876002894506,
+    "clip_frac": 0.20833333333333334,
+    "entropy": 0.20503475103113386,
+    "kl_penalty": 0.6547309027777778,
+    "kl_per_seq": 0.6547309027777778,
+    "kl_seq_max": 4.559027777777778
+  },
+  {
+    "iteration": 74,
+    "elapsed_s": 272.90518045425415,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7313487293956045,
+    "avg_questions": 0.3125,
+    "avg_turns": 1.3125,
+    "ppo_loss": -0.051379142018655934,
+    "value_loss": 0.34960736831029254,
+    "approx_kl": 0.02406678597132365,
+    "clip_frac": 0.22916666666666666,
+    "entropy": 0.19534940707186857,
+    "kl_penalty": 0.5764973958333334,
+    "kl_per_seq": 0.5764973958333334,
+    "kl_seq_max": 4.106770833333333
+  },
+  {
+    "iteration": 75,
+    "elapsed_s": 317.8711678981781,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.6857142857142857,
+    "avg_questions": 0.375,
+    "avg_turns": 1.3125,
+    "ppo_loss": -0.002149124443531036,
+    "value_loss": 0.42152651995420454,
+    "approx_kl": 0.013256344199180602,
+    "clip_frac": 0.1875,
+    "entropy": 0.20173095315694808,
+    "kl_penalty": 1.421875,
+    "kl_per_seq": 1.421875,
+    "kl_seq_max": 3.61875
+  },
+  {
+    "iteration": 76,
+    "elapsed_s": 235.38484358787537,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.760865003052503,
+    "avg_questions": 0.5,
+    "avg_turns": 1.4375,
+    "ppo_loss": -0.037924519181251524,
+    "value_loss": 0.5137223064899444,
+    "approx_kl": 0.018000446259975433,
+    "clip_frac": 0.2583333343267441,
+    "entropy": 0.22564681768417358,
+    "kl_penalty": 1.0486328125,
+    "kl_per_seq": 1.0486328125,
+    "kl_seq_max": 5.35
+  },
+  {
+    "iteration": 77,
+    "elapsed_s": 275.4559555053711,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7680517399267399,
+    "avg_questions": 0.6875,
+    "avg_turns": 1.5625,
+    "ppo_loss": -0.03971121230950722,
+    "value_loss": 0.45966682869654435,
+    "approx_kl": 0.021722187789586875,
+    "clip_frac": 0.23076923076923078,
+    "entropy": 0.24846581312326285,
+    "kl_penalty": 0.6517427884615384,
+    "kl_per_seq": 0.6517427884615384,
+    "kl_seq_max": 2.816105769230769
+  },
+  {
+    "iteration": 78,
+    "elapsed_s": 269.4535267353058,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.7603665865384616,
+    "avg_questions": 0.4375,
+    "avg_turns": 1.40625,
+    "ppo_loss": -0.03098818761381236,
+    "value_loss": 0.4363382512872869,
+    "approx_kl": 0.018023115159435707,
+    "clip_frac": 0.18863636390729385,
+    "entropy": 0.18647770380431955,
+    "kl_penalty": 0.9266690340909091,
+    "kl_per_seq": 0.9266690340909091,
+    "kl_seq_max": 3.9630681818181817
+  },
+  {
+    "iteration": 79,
+    "elapsed_s": 313.7352375984192,
+    "lambda1": 0.0,
+    "lambda2": 0.0,
+    "n_episodes": 32,
+    "avg_reward": 0.5877386502386502,
+    "avg_questions": 0.5,
+    "avg_turns": 1.375,
+    "ppo_loss": -0.028404296460476788,
+    "value_loss": 0.62438211386854,
+    "approx_kl": 0.026743617924776943,
+    "clip_frac": 0.29545454545454547,
+    "entropy": 0.24313818595626138,
+    "kl_penalty": 1.1049360795454546,
+    "kl_per_seq": 1.1049360795454546,
+    "kl_seq_max": 4.875
+  }
+]

iter_0079/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbb9ccc29b4517e9dccf26d1b9f1bb48113a6eb969a61c50ba9c8a2b0089d586
+size 11421990

iter_0079/tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "truncation_side": "left",
+  "unk_token": null
+}

iter_0079/train_state.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6edcea3c839e65b3657db8ac08ce4f47538b0c28960d6d7e0e28dce47a33d50b
+size 104946244

iter_0079/value_heads.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4064ba08fdca3781b563972f866638e8a858f94cec38cf5f78b7334cc2afaaee
+size 44096056