/opt/conda/lib/python3.11/site-packages/trl/generation/__init__.py:22: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.13.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():
/opt/conda/lib/python3.11/site-packages/trl/generation/vllm_client.py:39: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.13.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():
/opt/conda/lib/python3.11/site-packages/trl/generation/vllm_generation.py:37: UserWarning: TRL currently supports vLLM versions: 0.10.2, 0.11.0, 0.11.1, 0.11.2, 0.12.0. You have version 0.13.0 installed. We recommend installing a supported version to avoid compatibility issues.
  if is_vllm_available():
[2026-03-26 12:09:24] INFO train_trl_v2.py:627: ============================================================
[2026-03-26 12:09:24] INFO train_trl_v2.py:628: GRPO Gin Rummy Training V2 (simplified rollout, no tools)
[2026-03-26 12:09:24] INFO train_trl_v2.py:629: ============================================================
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
wandb: Currently logged in as: alxai (good-start-labs) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run mvhlef9f
wandb: Tracking run with wandb version 0.25.1
wandb: Run data is saved locally in /workspace/GinRummy_RLenv/wandb/run-20260326_120924-mvhlef9f
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run grpo-v2-20260326_120924
wandb: ⭐️ View project at https://wandb.ai/good-start-labs/ginrummy-grpo
wandb: 🚀 View run at https://wandb.ai/good-start-labs/ginrummy-grpo/runs/mvhlef9f
wandb: Detected [anthropic, mcp, openai, agents, verifiers] in use.
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
[2026-03-26 12:09:26] INFO train_trl_v2.py:656: Gin Rummy server: {'status': 'ok', 'active_episodes': 104}
[2026-03-26 12:09:26] INFO train_trl_v2.py:664: vLLM server models: {'object': 'list', 'data': [{'id': '/workspace/models/Qwen3-8B', 'object': 'model', 'created': 1774526966, 'owned_by': 'vllm', 'root': '/workspace/models/Qwen3-8B', 'parent': None, 'max_model_len': 4096, 'permission': [{'id': 'modelperm-bd770e649d5268fb', 'object': 'model_permission', 'created': 1774526966, 'allow_create_engine': False, 'allow_sampling': True, 'allow_logprobs': True, 'allow_search_indices': False, 'allow_view': True, 'allow_fine_tuning': False, 'organization': '*', 'group': None, 'is_blocking': False}]}]}
[2026-03-26 12:09:26] INFO train_trl_v2.py:674: Loading tokenizer from /workspace/models/Qwen3-8B
[2026-03-26 12:09:26] INFO train_trl_v2.py:681: Dataset: 2000 samples
[2026-03-26 12:09:28] INFO train_trl_v2.py:736: Training config:
[2026-03-26 12:09:28] INFO train_trl_v2.py:737:   Model:           /workspace/models/Qwen3-8B
[2026-03-26 12:09:28] INFO train_trl_v2.py:738:   Opponent:        random
[2026-03-26 12:09:28] INFO train_trl_v2.py:739:   LoRA:            r=16, alpha=32
[2026-03-26 12:09:28] INFO train_trl_v2.py:740:   LR:              1.00e-05
[2026-03-26 12:09:28] INFO train_trl_v2.py:741:   Batch size:      1 x 4 accum = 4 effective
[2026-03-26 12:09:28] INFO train_trl_v2.py:745:   Num generations: 8
[2026-03-26 12:09:28] INFO train_trl_v2.py:746:   Max steps:       500
[2026-03-26 12:09:28] INFO train_trl_v2.py:747:   Beta (KL):       0.1000
[2026-03-26 12:09:28] INFO train_trl_v2.py:748:   vLLM server:     http://localhost:8000 (external)
[2026-03-26 12:09:28] INFO train_trl_v2.py:749:   Game server:     http://ginrummy-server:8005
[2026-03-26 12:09:28] INFO train_trl_v2.py:750:   W&B:             good-start-labs/ginrummy-grpo
[2026-03-26 12:09:28] INFO train_trl_v2.py:754: Initializing GRPOTrainer with LoRA...

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:01<00:05,  1.46s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:02<00:04,  1.45s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [00:04<00:02,  1.43s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [00:05<00:01,  1.31s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:05<00:00,  1.01it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:05<00:00,  1.17s/it]
[2026-03-26 12:09:35] INFO train_trl_v2.py:764: Trainer initialized successfully.
[2026-03-26 12:09:35] INFO train_trl_v2.py:767: Running smoke test: 2 games (opponent=random)...
[2026-03-26 12:09:56] INFO train_trl_v2.py:773:   Game 0: 32 turns, reward=0.0, deadwood=52, 2048 completion tokens
[2026-03-26 12:10:01] INFO train_trl_v2.py:773:   Game 1: 14 turns, reward=-1.0, deadwood=65, 896 completion tokens
[2026-03-26 12:10:01] INFO train_trl_v2.py:782: Smoke test passed: 2/2 games completed.
[2026-03-26 12:10:01] INFO train_trl_v2.py:785: ============================================================
[2026-03-26 12:10:01] INFO train_trl_v2.py:786: Starting GRPO training: 500 steps
[2026-03-26 12:10:01] INFO train_trl_v2.py:787: ============================================================
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.

  0%|          | 0/500 [00:00<?, ?it/s][2026-03-26 12:10:01] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:10:15] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.5
Could not estimate the number of tokens of the input, floating-point operations will not be computed

  0%|          | 1/500 [00:18<2:37:13, 18.91s/it]
                                                 

  0%|          | 1/500 [00:18<2:37:13, 18.91s/it]
  0%|          | 2/500 [00:20<1:13:57,  8.91s/it]
                                                 

  0%|          | 2/500 [00:20<1:13:57,  8.91s/it][2026-03-26 12:10:22] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:10:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.8

  1%|          | 3/500 [00:38<1:48:28, 13.10s/it]
                                                 

  1%|          | 3/500 [00:38<1:48:28, 13.10s/it]
  1%|          | 4/500 [00:40<1:12:07,  8.72s/it]
                                                 

  1%|          | 4/500 [00:40<1:12:07,  8.72s/it][2026-03-26 12:10:42] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:10:56] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.1

  1%|          | 5/500 [00:59<1:40:12, 12.15s/it]
                                                 

  1%|          | 5/500 [00:59<1:40:12, 12.15s/it]
  1%|          | 6/500 [01:01<1:11:56,  8.74s/it]
                                                 

  1%|          | 6/500 [01:01<1:11:56,  8.74s/it][2026-03-26 12:11:02] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:11:16] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.6

  1%|▏         | 7/500 [01:19<1:38:19, 11.97s/it]
                                                 

  1%|▏         | 7/500 [01:19<1:38:19, 11.97s/it]
  2%|▏         | 8/500 [01:21<1:12:27,  8.84s/it]
                                                 

  2%|▏         | 8/500 [01:22<1:12:27,  8.84s/it][2026-03-26 12:11:23] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:11:37] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.2

  2%|▏         | 9/500 [01:40<1:35:47, 11.71s/it]
                                                 

  2%|▏         | 9/500 [01:40<1:35:47, 11.71s/it]
  2%|▏         | 10/500 [01:42<1:11:27,  8.75s/it]
                                                  

  2%|▏         | 10/500 [01:42<1:11:27,  8.75s/it][2026-03-26 12:11:43] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:11:57] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=24.1

  2%|▏         | 11/500 [02:00<1:35:13, 11.68s/it]
                                                  

  2%|▏         | 11/500 [02:00<1:35:13, 11.68s/it]
  2%|▏         | 12/500 [02:02<1:11:25,  8.78s/it]
                                                  

  2%|▏         | 12/500 [02:02<1:11:25,  8.78s/it][2026-03-26 12:12:04] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:12:17] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.6

  3%|▎         | 13/500 [02:20<1:34:04, 11.59s/it]
                                                  
{'loss': 0.7311, 'grad_norm': 1.2703578472137451, 'learning_rate': 0.0, 'num_tokens': 11416.0, 'completions/mean_length': 1248.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.30000001192092896, 'rewards/game_reward_func/std': 0.12300987541675568, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.30000001192092896, 'reward_std': 0.12300986796617508, 'frac_reward_zero_std': 0.0, 'kl': 0.0, 'entropy': 0.26630012691020966, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.0, 'step_time': 18.661106028004724, 'epoch': 0.0}
{'loss': -0.7311, 'grad_norm': 0.9118824601173401, 'learning_rate': 1.0000000000000002e-06, 'kl': 0.0, 'entropy': 0.23953953012824059, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.0, 'step_time': 1.7744415939960163, 'epoch': 0.0}
{'loss': 0.6542, 'grad_norm': 1.378021001815796, 'learning_rate': 2.0000000000000003e-06, 'num_tokens': 22960.0, 'completions/mean_length': 1264.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.36800000071525574, 'rewards/game_reward_func/std': 0.17468993365764618, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.36800000071525574, 'reward_std': 0.17468993365764618, 'frac_reward_zero_std': 0.0, 'kl': 0.0016002704505808651, 'entropy': 0.2787575460970402, 'clip_ratio/low_mean': 0.012261284864507616, 'clip_ratio/low_min': 0.012261284864507616, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.012261284864507616, 'step_time': 17.887763895996613, 'epoch': 0.0}
{'loss': -0.653, 'grad_norm': 0.6448683738708496, 'learning_rate': 3e-06, 'kl': 0.0010757488198578358, 'entropy': 0.23558492213487625, 'clip_ratio/low_mean': 0.0016447368543595076, 'clip_ratio/low_min': 0.0016447368543595076, 'clip_ratio/high_mean': 0.006541752954944968, 'clip_ratio/high_max': 0.006541752954944968, 'clip_ratio/region_mean': 0.008186489809304476, 'step_time': 1.8421715260046767, 'epoch': 0.0}
{'loss': -0.3422, 'grad_norm': 0.3476867973804474, 'learning_rate': 4.000000000000001e-06, 'num_tokens': 35720.0, 'completions/mean_length': 1416.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3465000092983246, 'rewards/game_reward_func/std': 0.11096847057342529, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3465000092983246, 'reward_std': 0.11096847057342529, 'frac_reward_zero_std': 0.0, 'kl': 0.0010565579868853092, 'entropy': 0.23397449404001236, 'clip_ratio/low_mean': 0.002583947964012623, 'clip_ratio/low_min': 0.002583947964012623, 'clip_ratio/high_mean': 0.002964564715512097, 'clip_ratio/high_max': 0.002964564715512097, 'clip_ratio/region_mean': 0.00554851267952472, 'step_time': 18.020510948001174, 'epoch': 0.0}
{'loss': 0.346, 'grad_norm': 1.2632008790969849, 'learning_rate': 5e-06, 'kl': 0.0014800139470025897, 'entropy': 0.2601131200790405, 'clip_ratio/low_mean': 0.0027901786379516125, 'clip_ratio/low_min': 0.0027901786379516125, 'clip_ratio/high_mean': 0.005856749718077481, 'clip_ratio/high_max': 0.005856749718077481, 'clip_ratio/region_mean': 0.008646928356029093, 'step_time': 1.938839957001619, 'epoch': 0.0}
{'loss': -0.4675, 'grad_norm': 0.4939422607421875, 'learning_rate': 6e-06, 'num_tokens': 49248.0, 'completions/mean_length': 1512.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.39675000309944153, 'rewards/game_reward_func/std': 0.1945013850927353, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.39675000309944153, 'reward_std': 0.1945013850927353, 'frac_reward_zero_std': 0.0, 'kl': 0.0012919246219098568, 'entropy': 0.22095352038741112, 'clip_ratio/low_mean': 0.0055593649740330875, 'clip_ratio/low_min': 0.0055593649740330875, 'clip_ratio/high_mean': 0.003952552564442158, 'clip_ratio/high_max': 0.003952552564442158, 'clip_ratio/region_mean': 0.009511917538475245, 'step_time': 18.43235268398712, 'epoch': 0.0}
{'loss': 0.4703, 'grad_norm': 0.5931963920593262, 'learning_rate': 7e-06, 'kl': 0.0011750049598049372, 'entropy': 0.24466417729854584, 'clip_ratio/low_mean': 0.005679086490999907, 'clip_ratio/low_min': 0.005679086490999907, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.005679086490999907, 'step_time': 1.9397033009954612, 'epoch': 0.0}
{'loss': 0.3418, 'grad_norm': 0.6700596213340759, 'learning_rate': 8.000000000000001e-06, 'num_tokens': 61048.0, 'completions/mean_length': 1296.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3400000035762787, 'rewards/game_reward_func/std': 0.16841445863246918, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3400000035762787, 'reward_std': 0.16841444373130798, 'frac_reward_zero_std': 0.0, 'kl': 0.0016367266362067312, 'entropy': 0.2796311117708683, 'clip_ratio/low_mean': 0.010387731366790831, 'clip_ratio/low_min': 0.010387731366790831, 'clip_ratio/high_mean': 0.001220703125, 'clip_ratio/high_max': 0.001220703125, 'clip_ratio/region_mean': 0.011608434491790831, 'step_time': 17.836813251997228, 'epoch': 0.0}
{'loss': -0.3383, 'grad_norm': 0.5039687752723694, 'learning_rate': 9e-06, 'kl': 0.0017149746126960963, 'entropy': 0.2602635324001312, 'clip_ratio/low_mean': 0.00146484375, 'clip_ratio/low_min': 0.00146484375, 'clip_ratio/high_mean': 0.0065910411067306995, 'clip_ratio/high_max': 0.0065910411067306995, 'clip_ratio/region_mean': 0.0080558848567307, 'step_time': 1.9359904849989107, 'epoch': 0.0}
{'loss': -0.6473, 'grad_norm': 0.6766774654388428, 'learning_rate': 1e-05, 'num_tokens': 74832.0, 'completions/mean_length': 1544.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3490000069141388, 'rewards/game_reward_func/std': 0.22395408153533936, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3490000069141388, 'reward_std': 0.22395408153533936, 'frac_reward_zero_std': 0.0, 'kl': 0.0011824098328361288, 'entropy': 0.2513961158692837, 'clip_ratio/low_mean': 0.0015120967291295528, 'clip_ratio/low_min': 0.0015120967291295528, 'clip_ratio/high_mean': 0.009591239038854837, 'clip_ratio/high_max': 0.009591239038854837, 'clip_ratio/region_mean': 0.01110333576798439, 'step_time': 18.169056972998078, 'epoch': 0.0}
{'loss': 0.6487, 'grad_norm': 0.8085058331489563, 'learning_rate': 9.999897234791831e-06, 'kl': 0.0012790967157343403, 'entropy': 0.2670045495033264, 'clip_ratio/low_mean': 0.006935413461178541, 'clip_ratio/low_min': 0.006935413461178541, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.006935413461178541, 'step_time': 1.9337993489971268, 'epoch': 0.0}

  3%|▎         | 13/500 [02:20<1:34:04, 11.59s/it]
  3%|▎         | 14/500 [02:22<1:11:02,  8.77s/it]
                                                  

  3%|▎         | 14/500 [02:22<1:11:02,  8.77s/it][2026-03-26 12:12:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:12:38] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=18.9

  3%|▎         | 15/500 [02:40<1:33:13, 11.53s/it]
                                                  

  3%|▎         | 15/500 [02:40<1:33:13, 11.53s/it]
  3%|▎         | 16/500 [02:42<1:09:55,  8.67s/it]
                                                  

  3%|▎         | 16/500 [02:42<1:09:55,  8.67s/it][2026-03-26 12:12:44] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:12:57] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=23.6

  3%|▎         | 17/500 [03:00<1:31:20, 11.35s/it]
                                                  

  3%|▎         | 17/500 [03:00<1:31:20, 11.35s/it]
  4%|▎         | 18/500 [03:02<1:08:30,  8.53s/it]
                                                  

  4%|▎         | 18/500 [03:02<1:08:30,  8.53s/it][2026-03-26 12:13:04] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:13:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.0

  4%|▍         | 19/500 [03:20<1:32:02, 11.48s/it]
                                                  

  4%|▍         | 19/500 [03:20<1:32:02, 11.48s/it]
  4%|▍         | 20/500 [03:22<1:09:07,  8.64s/it]
                                                  

  4%|▍         | 20/500 [03:22<1:09:07,  8.64s/it][2026-03-26 12:13:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:13:32] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=14.5

  4%|▍         | 21/500 [03:34<1:15:28,  9.45s/it]
                                                  

  4%|▍         | 21/500 [03:34<1:15:28,  9.45s/it]
  4%|▍         | 22/500 [03:35<55:53,  7.02s/it]  
                                                

  4%|▍         | 22/500 [03:35<55:53,  7.02s/it][2026-03-26 12:13:37] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:13:48] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.1

  5%|▍         | 23/500 [03:51<1:16:05,  9.57s/it]
                                                  

  5%|▍         | 23/500 [03:51<1:16:05,  9.57s/it]
  5%|▍         | 24/500 [03:52<57:11,  7.21s/it]  
                                                
{'loss': 0.0524, 'grad_norm': 0.37725985050201416, 'learning_rate': 9.999588943391597e-06, 'num_tokens': 85800.0, 'completions/mean_length': 1192.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3737500011920929, 'rewards/game_reward_func/std': 0.22843676805496216, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3737500011920929, 'reward_std': 0.22843676805496216, 'frac_reward_zero_std': 0.0, 'kl': 0.0022907064703758806, 'entropy': 0.25113483518362045, 'clip_ratio/low_mean': 0.011035156436264515, 'clip_ratio/low_min': 0.011035156436264515, 'clip_ratio/high_mean': 0.007626487989909947, 'clip_ratio/high_max': 0.007626487989909947, 'clip_ratio/region_mean': 0.018661644426174462, 'step_time': 17.858797487002448, 'epoch': 0.0}
{'loss': -0.051, 'grad_norm': 1.4613996744155884, 'learning_rate': 9.99907513847195e-06, 'kl': 0.001683577458607033, 'entropy': 0.23984654620289803, 'clip_ratio/low_mean': 0.007755265804007649, 'clip_ratio/low_min': 0.007755265804007649, 'clip_ratio/high_mean': 0.0023193359375, 'clip_ratio/high_max': 0.0023193359375, 'clip_ratio/region_mean': 0.01007460174150765, 'step_time': 2.0770349300000817, 'epoch': 0.0}
{'loss': 0.1176, 'grad_norm': 0.6078194379806519, 'learning_rate': 9.9983558411534e-06, 'num_tokens': 96896.0, 'completions/mean_length': 1208.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2797499895095825, 'rewards/game_reward_func/std': 0.22377906739711761, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2797499895095825, 'reward_std': 0.22377905249595642, 'frac_reward_zero_std': 0.0, 'kl': 0.0017886888526845723, 'entropy': 0.2685876861214638, 'clip_ratio/low_mean': 0.008533296291716397, 'clip_ratio/low_min': 0.008533296291716397, 'clip_ratio/high_mean': 0.0018310546875, 'clip_ratio/high_max': 0.0018310546875, 'clip_ratio/region_mean': 0.010364350979216397, 'step_time': 17.748768789002497, 'epoch': 0.0}
{'loss': -0.1167, 'grad_norm': 0.18955405056476593, 'learning_rate': 9.99743108100344e-06, 'kl': 0.001609333325177431, 'entropy': 0.2570161633193493, 'clip_ratio/low_mean': 0.00867638224735856, 'clip_ratio/low_min': 0.00867638224735856, 'clip_ratio/high_mean': 0.005536247743293643, 'clip_ratio/high_max': 0.005536247743293643, 'clip_ratio/region_mean': 0.014212629990652204, 'step_time': 1.8361948429883341, 'epoch': 0.0}
{'loss': 0.7102, 'grad_norm': 0.9210221171379089, 'learning_rate': 9.99630089603534e-06, 'num_tokens': 110424.0, 'completions/mean_length': 1512.0, 'completions/min_length': 1024.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3709999918937683, 'rewards/game_reward_func/std': 0.11952286213636398, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3709999918937683, 'reward_std': 0.11952286213636398, 'frac_reward_zero_std': 0.0, 'kl': 0.0013527728442568332, 'entropy': 0.2535262033343315, 'clip_ratio/low_mean': 0.005859375, 'clip_ratio/low_min': 0.005859375, 'clip_ratio/high_mean': 0.0013586956774815917, 'clip_ratio/high_max': 0.0013586956774815917, 'clip_ratio/region_mean': 0.007218070677481592, 'step_time': 17.429810998000903, 'epoch': 0.0}
{'loss': -0.7056, 'grad_norm': 0.7034807801246643, 'learning_rate': 9.994965332706574e-06, 'kl': 0.0014684856869280338, 'entropy': 0.24203230068087578, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.0061949765076860785, 'clip_ratio/high_max': 0.0061949765076860785, 'clip_ratio/region_mean': 0.0061949765076860785, 'step_time': 1.7887957229977474, 'epoch': 0.0}
{'loss': -0.3529, 'grad_norm': 0.3996693789958954, 'learning_rate': 9.993424445916923e-06, 'num_tokens': 124144.0, 'completions/mean_length': 1536.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.41200000047683716, 'rewards/game_reward_func/std': 0.1692335605621338, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.41200000047683716, 'reward_std': 0.1692335605621338, 'frac_reward_zero_std': 0.0, 'kl': 0.0020204139582347125, 'entropy': 0.26582732051610947, 'clip_ratio/low_mean': 0.0008680555620230734, 'clip_ratio/low_min': 0.0008680555620230734, 'clip_ratio/high_mean': 0.009033203125, 'clip_ratio/high_max': 0.009033203125, 'clip_ratio/region_mean': 0.009901258687023073, 'step_time': 18.18854262199602, 'epoch': 0.0}
{'loss': 0.3545, 'grad_norm': 0.4561241865158081, 'learning_rate': 9.991678299006206e-06, 'kl': 0.001845063961809501, 'entropy': 0.2244689092040062, 'clip_ratio/low_mean': 0.004021895583719015, 'clip_ratio/low_min': 0.004021895583719015, 'clip_ratio/high_mean': 0.004160679993219674, 'clip_ratio/high_max': 0.004160679993219674, 'clip_ratio/region_mean': 0.008182575576938689, 'step_time': 1.8352309159963625, 'epoch': 0.01}
{'loss': -0.2714, 'grad_norm': 0.5801978707313538, 'learning_rate': 9.989726963751683e-06, 'num_tokens': 133000.0, 'completions/mean_length': 928.0, 'completions/min_length': 320.0, 'completions/max_length': 1216.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4179999828338623, 'rewards/game_reward_func/std': 0.1260838806629181, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4179999828338623, 'reward_std': 0.1260838657617569, 'frac_reward_zero_std': 0.0, 'kl': 0.0036801371606998146, 'entropy': 0.2656554728746414, 'clip_ratio/low_mean': 0.0053385416977107525, 'clip_ratio/low_min': 0.0053385416977107525, 'clip_ratio/high_mean': 0.010156250093132257, 'clip_ratio/high_max': 0.010156250093132257, 'clip_ratio/region_mean': 0.01549479179084301, 'step_time': 11.270478925995121, 'epoch': 0.01}
{'loss': 0.2747, 'grad_norm': 0.3618091642856598, 'learning_rate': 9.987570520365105e-06, 'kl': 0.002476754831150174, 'entropy': 0.2638486512005329, 'clip_ratio/low_mean': 0.003960503498092294, 'clip_ratio/low_min': 0.003960503498092294, 'clip_ratio/high_mean': 0.008229834493249655, 'clip_ratio/high_max': 0.008229834493249655, 'clip_ratio/region_mean': 0.012190337991341949, 'step_time': 1.2165334679957596, 'epoch': 0.01}
{'loss': -0.4079, 'grad_norm': 0.5920016169548035, 'learning_rate': 9.98520905748941e-06, 'num_tokens': 145248.0, 'completions/mean_length': 1352.0, 'completions/min_length': 576.0, 'completions/max_length': 1728.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4482499957084656, 'rewards/game_reward_func/std': 0.10583242028951645, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4482499957084656, 'reward_std': 0.10583242774009705, 'frac_reward_zero_std': 0.0, 'kl': 0.0019835778803098947, 'entropy': 0.22366196662187576, 'clip_ratio/low_mean': 0.003842561040073633, 'clip_ratio/low_min': 0.003842561040073633, 'clip_ratio/high_mean': 0.0053125000558793545, 'clip_ratio/high_max': 0.0053125000558793545, 'clip_ratio/region_mean': 0.009155061095952988, 'step_time': 15.398506085002737, 'epoch': 0.01}

  5%|▍         | 24/500 [03:52<57:11,  7.21s/it][2026-03-26 12:13:54] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:14:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.5

  5%|▌         | 25/500 [04:10<1:23:11, 10.51s/it]
                                                  

  5%|▌         | 25/500 [04:10<1:23:11, 10.51s/it]
  5%|▌         | 26/500 [04:12<1:02:54,  7.96s/it]
                                                  

  5%|▌         | 26/500 [04:12<1:02:54,  7.96s/it][2026-03-26 12:14:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:14:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=15.0

  5%|▌         | 27/500 [04:27<1:18:46,  9.99s/it]
                                                  

  5%|▌         | 27/500 [04:27<1:18:46,  9.99s/it]
  6%|▌         | 28/500 [04:29<59:51,  7.61s/it]  
                                                

  6%|▌         | 28/500 [04:29<59:51,  7.61s/it][2026-03-26 12:14:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:14:45] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:4 D:4 (0% win) | avg_turns=23.9

  6%|▌         | 29/500 [04:47<1:24:37, 10.78s/it]
                                                  

  6%|▌         | 29/500 [04:47<1:24:37, 10.78s/it]
  6%|▌         | 30/500 [04:49<1:03:21,  8.09s/it]
                                                  

  6%|▌         | 30/500 [04:49<1:03:21,  8.09s/it][2026-03-26 12:14:51] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:15:05] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=24.5

  6%|▌         | 31/500 [05:07<1:26:32, 11.07s/it]
                                                  

  6%|▌         | 31/500 [05:07<1:26:32, 11.07s/it]
  6%|▋         | 32/500 [05:09<1:05:25,  8.39s/it]
                                                  

  6%|▋         | 32/500 [05:09<1:05:25,  8.39s/it][2026-03-26 12:15:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:15:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.2

  7%|▋         | 33/500 [05:28<1:28:20, 11.35s/it]
                                                  

  7%|▋         | 33/500 [05:28<1:28:20, 11.35s/it]
  7%|▋         | 34/500 [05:30<1:06:25,  8.55s/it]
                                                  

  7%|▋         | 34/500 [05:30<1:06:25,  8.55s/it][2026-03-26 12:15:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:15:45] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.2

  7%|▋         | 35/500 [05:48<1:29:59, 11.61s/it]
                                                  
{'loss': 0.411, 'grad_norm': 0.8924457430839539, 'learning_rate': 9.982642672195093e-06, 'kl': 0.002659397287061438, 'entropy': 0.26087865233421326, 'clip_ratio/low_mean': 0.003719022322911769, 'clip_ratio/low_min': 0.003719022322911769, 'clip_ratio/high_mean': 0.0032552082557231188, 'clip_ratio/high_max': 0.0032552082557231188, 'clip_ratio/region_mean': 0.006974230578634888, 'step_time': 1.5502357759978622, 'epoch': 0.01}
{'loss': 0.282, 'grad_norm': 0.27934467792510986, 'learning_rate': 9.979871469976197e-06, 'num_tokens': 157176.0, 'completions/mean_length': 1312.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3774999976158142, 'rewards/game_reward_func/std': 0.19165073335170746, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3774999976158142, 'reward_std': 0.19165071845054626, 'frac_reward_zero_std': 0.0, 'kl': 0.0029709297814406455, 'entropy': 0.26703882962465286, 'clip_ratio/low_mean': 0.0037164322566241026, 'clip_ratio/low_min': 0.0037164322566241026, 'clip_ratio/high_mean': 0.005667021963745356, 'clip_ratio/high_max': 0.005667021963745356, 'clip_ratio/region_mean': 0.009383454220369458, 'step_time': 18.01735942599771, 'epoch': 0.01}
{'loss': -0.2793, 'grad_norm': 0.4846305549144745, 'learning_rate': 9.976895564745993e-06, 'kl': 0.0028077951283194125, 'entropy': 0.2519722580909729, 'clip_ratio/low_mean': 0.0042296432657167315, 'clip_ratio/low_min': 0.0042296432657167315, 'clip_ratio/high_mean': 0.0026010366855189204, 'clip_ratio/high_max': 0.0026010366855189204, 'clip_ratio/region_mean': 0.006830679951235652, 'step_time': 1.840723302004335, 'epoch': 0.01}
{'loss': -0.8879, 'grad_norm': 1.507447361946106, 'learning_rate': 9.973715078832288e-06, 'num_tokens': 166288.0, 'completions/mean_length': 960.0, 'completions/min_length': 384.0, 'completions/max_length': 1728.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.36800000071525574, 'rewards/game_reward_func/std': 0.12101949006319046, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.36800000071525574, 'reward_std': 0.12101946026086807, 'frac_reward_zero_std': 0.0, 'kl': 0.004187333019217476, 'entropy': 0.30112677812576294, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.016753472038544714, 'clip_ratio/high_max': 0.016753472038544714, 'clip_ratio/region_mean': 0.016753472038544714, 'step_time': 14.590596582005674, 'epoch': 0.01}
{'loss': 0.8882, 'grad_norm': 1.2979530096054077, 'learning_rate': 9.970330142972403e-06, 'kl': 0.0035498038632795215, 'entropy': 0.26076918095350266, 'clip_ratio/low_mean': 0.011601435486227274, 'clip_ratio/low_min': 0.011601435486227274, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.011601435486227274, 'step_time': 1.8892537510037073, 'epoch': 0.01}
{'loss': 0.6481, 'grad_norm': 0.8032415509223938, 'learning_rate': 9.966740896307791e-06, 'num_tokens': 179944.0, 'completions/mean_length': 1528.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.26875001192092896, 'rewards/game_reward_func/std': 0.16706179082393646, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.26875001192092896, 'reward_std': 0.16706180572509766, 'frac_reward_zero_std': 0.0, 'kl': 0.003559837641660124, 'entropy': 0.22043316811323166, 'clip_ratio/low_mean': 0.012141752988100052, 'clip_ratio/low_min': 0.012141752988100052, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.015071440488100052, 'step_time': 18.0001499929931, 'epoch': 0.01}
{'loss': -0.6465, 'grad_norm': 0.7329373359680176, 'learning_rate': 9.962947486378325e-06, 'kl': 0.0029886664706282318, 'entropy': 0.24545591324567795, 'clip_ratio/low_mean': 0.0033854166977107525, 'clip_ratio/low_min': 0.0033854166977107525, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.0072916666977107525, 'step_time': 1.6468631110037677, 'epoch': 0.01}
{'loss': -0.0024, 'grad_norm': 0.39514848589897156, 'learning_rate': 9.95895006911623e-06, 'num_tokens': 193920.0, 'completions/mean_length': 1568.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.47724997997283936, 'rewards/game_reward_func/std': 0.10305025428533554, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.47724997997283936, 'reward_std': 0.10305025428533554, 'frac_reward_zero_std': 0.0, 'kl': 0.0019525277893990278, 'entropy': 0.2386438474059105, 'clip_ratio/low_mean': 0.0033587832003831863, 'clip_ratio/low_min': 0.0033587832003831863, 'clip_ratio/high_mean': 0.004532877588644624, 'clip_ratio/high_max': 0.004532877588644624, 'clip_ratio/region_mean': 0.00789166078902781, 'step_time': 17.850198957996326, 'epoch': 0.01}
{'loss': 0.0022, 'grad_norm': 0.11765027046203613, 'learning_rate': 9.954748808839675e-06, 'kl': 0.0041847508400678635, 'entropy': 0.22028210759162903, 'clip_ratio/low_mean': 0.010102370753884315, 'clip_ratio/low_min': 0.010102370753884315, 'clip_ratio/high_mean': 0.001638104789890349, 'clip_ratio/high_max': 0.001638104789890349, 'clip_ratio/region_mean': 0.011740475543774664, 'step_time': 1.937003928993363, 'epoch': 0.01}
{'loss': 0.0413, 'grad_norm': 0.40993887186050415, 'learning_rate': 9.950343878246011e-06, 'num_tokens': 207256.0, 'completions/mean_length': 1488.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4479999840259552, 'rewards/game_reward_func/std': 0.12842784821987152, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4479999840259552, 'reward_std': 0.12842784821987152, 'frac_reward_zero_std': 0.0, 'kl': 0.0024580511962994933, 'entropy': 0.23049171641469002, 'clip_ratio/low_mean': 0.003742423257790506, 'clip_ratio/low_min': 0.003742423257790506, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.005695548257790506, 'step_time': 18.10352541400789, 'epoch': 0.01}
{'loss': -0.0382, 'grad_norm': 0.09478321671485901, 'learning_rate': 9.945735458404681e-06, 'kl': 0.004223223135340959, 'entropy': 0.2603817693889141, 'clip_ratio/low_mean': 0.00146484375, 'clip_ratio/low_min': 0.00146484375, 'clip_ratio/high_mean': 0.005431190016679466, 'clip_ratio/high_max': 0.005431190016679466, 'clip_ratio/region_mean': 0.006896033766679466, 'step_time': 1.8424778470071033, 'epoch': 0.01}

  7%|▋         | 35/500 [05:48<1:29:59, 11.61s/it]
  7%|▋         | 36/500 [05:50<1:07:17,  8.70s/it]
                                                  

  7%|▋         | 36/500 [05:50<1:07:17,  8.70s/it][2026-03-26 12:15:52] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:16:06] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=25.0

  7%|▋         | 37/500 [06:09<1:29:27, 11.59s/it]
                                                  

  7%|▋         | 37/500 [06:09<1:29:27, 11.59s/it]
  8%|▊         | 38/500 [06:11<1:07:24,  8.75s/it]
                                                  

  8%|▊         | 38/500 [06:11<1:07:24,  8.75s/it][2026-03-26 12:16:12] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:16:26] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=18.9

  8%|▊         | 39/500 [06:29<1:28:01, 11.46s/it]
                                                  

  8%|▊         | 39/500 [06:29<1:28:01, 11.46s/it]
  8%|▊         | 40/500 [06:31<1:06:23,  8.66s/it]
                                                  

  8%|▊         | 40/500 [06:31<1:06:23,  8.66s/it][2026-03-26 12:16:32] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:16:47] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=23.4

  8%|▊         | 41/500 [06:49<1:28:47, 11.61s/it]
                                                  

  8%|▊         | 41/500 [06:49<1:28:47, 11.61s/it]
  8%|▊         | 42/500 [06:51<1:07:04,  8.79s/it]
                                                  

  8%|▊         | 42/500 [06:51<1:07:04,  8.79s/it][2026-03-26 12:16:53] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:17:07] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.0

  9%|▊         | 43/500 [07:10<1:28:33, 11.63s/it]
                                                  

  9%|▊         | 43/500 [07:10<1:28:33, 11.63s/it]
  9%|▉         | 44/500 [07:12<1:06:27,  8.74s/it]
                                                  

  9%|▉         | 44/500 [07:12<1:06:27,  8.74s/it][2026-03-26 12:17:13] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:17:27] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.5

  9%|▉         | 45/500 [07:30<1:27:36, 11.55s/it]
                                                  

  9%|▉         | 45/500 [07:30<1:27:36, 11.55s/it]
  9%|▉         | 46/500 [07:32<1:05:45,  8.69s/it]
                                                  
{'loss': 0.5023, 'grad_norm': 0.7311879396438599, 'learning_rate': 9.94092373874978e-06, 'num_tokens': 220592.0, 'completions/mean_length': 1488.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.31150001287460327, 'rewards/game_reward_func/std': 0.22233501076698303, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.31150001287460327, 'reward_std': 0.22233499586582184, 'frac_reward_zero_std': 0.0, 'kl': 0.003378644469194114, 'entropy': 0.2507108487188816, 'clip_ratio/low_mean': 0.003656684042653069, 'clip_ratio/low_min': 0.003656684042653069, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.007562934042653069, 'step_time': 18.56313062400295, 'epoch': 0.01}
{'loss': -0.4997, 'grad_norm': 0.4542434513568878, 'learning_rate': 9.935908917072253e-06, 'kl': 0.003385032119695097, 'entropy': 0.23588594049215317, 'clip_ratio/low_mean': 0.004093477735295892, 'clip_ratio/low_min': 0.004093477735295892, 'clip_ratio/high_mean': 0.002685546875, 'clip_ratio/high_max': 0.002685546875, 'clip_ratio/region_mean': 0.006779024610295892, 'step_time': 1.7430000550011755, 'epoch': 0.01}
{'loss': -0.3411, 'grad_norm': 2.324951648712158, 'learning_rate': 9.930691199511775e-06, 'num_tokens': 234824.0, 'completions/mean_length': 1600.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37174999713897705, 'rewards/game_reward_func/std': 0.17969954013824463, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37174999713897705, 'reward_std': 0.17969955503940582, 'frac_reward_zero_std': 0.0, 'kl': 0.08595234923996031, 'entropy': 0.24261534214019775, 'clip_ratio/low_mean': 0.004830729216337204, 'clip_ratio/low_min': 0.004830729216337204, 'clip_ratio/high_mean': 0.002197265625, 'clip_ratio/high_max': 0.002197265625, 'clip_ratio/region_mean': 0.007027994841337204, 'step_time': 18.17433087700192, 'epoch': 0.01}
{'loss': 0.3537, 'grad_norm': 0.47868114709854126, 'learning_rate': 9.925270800548285e-06, 'kl': 0.003437404171563685, 'entropy': 0.2581177204847336, 'clip_ratio/low_mean': 0.006016188650391996, 'clip_ratio/low_min': 0.006016188650391996, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.006016188650391996, 'step_time': 1.9384033300084411, 'epoch': 0.01}
{'loss': 0.6968, 'grad_norm': 1.1288676261901855, 'learning_rate': 9.91964794299315e-06, 'num_tokens': 245920.0, 'completions/mean_length': 1208.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40525001287460327, 'rewards/game_reward_func/std': 0.10795732587575912, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40525001287460327, 'reward_std': 0.10795733332633972, 'frac_reward_zero_std': 0.0, 'kl': 0.004214003623928875, 'entropy': 0.25121646001935005, 'clip_ratio/low_mean': 0.005688864155672491, 'clip_ratio/low_min': 0.005688864155672491, 'clip_ratio/high_mean': 0.002197265625, 'clip_ratio/high_max': 0.002197265625, 'clip_ratio/region_mean': 0.00788612978067249, 'step_time': 17.58538366099674, 'epoch': 0.01}
{'loss': -0.6904, 'grad_norm': 1.232204794883728, 'learning_rate': 9.91382285798002e-06, 'kl': 0.0060158078558743, 'entropy': 0.2769062779843807, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.009165095107164234, 'clip_ratio/high_max': 0.009165095107164234, 'clip_ratio/region_mean': 0.009165095107164234, 'step_time': 1.9417751119981403, 'epoch': 0.01}
{'loss': 0.4521, 'grad_norm': 1.0299487113952637, 'learning_rate': 9.907795784955327e-06, 'num_tokens': 259319.0, 'completions/mean_length': 1495.875, 'completions/min_length': 767.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2952499985694885, 'rewards/game_reward_func/std': 0.21173013746738434, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2952499985694885, 'reward_std': 0.21173012256622314, 'frac_reward_zero_std': 0.0, 'kl': 0.005752986820880324, 'entropy': 0.261820912361145, 'clip_ratio/low_mean': 0.005823436309583485, 'clip_ratio/low_min': 0.005823436309583485, 'clip_ratio/high_mean': 0.002197265625, 'clip_ratio/high_max': 0.002197265625, 'clip_ratio/region_mean': 0.008020701934583485, 'step_time': 18.31070703900332, 'epoch': 0.01}
{'loss': -0.448, 'grad_norm': 0.6066391468048096, 'learning_rate': 9.901566971668437e-06, 'kl': 0.006584499031305313, 'entropy': 0.22241296991705894, 'clip_ratio/low_mean': 0.0026041667442768812, 'clip_ratio/low_min': 0.0026041667442768812, 'clip_ratio/high_mean': 0.004733615554869175, 'clip_ratio/high_max': 0.004733615554869175, 'clip_ratio/region_mean': 0.007337782299146056, 'step_time': 2.041916042995581, 'epoch': 0.01}
{'loss': 0.1367, 'grad_norm': 0.5847355723381042, 'learning_rate': 9.895136674161466e-06, 'num_tokens': 270991.0, 'completions/mean_length': 1280.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3310000002384186, 'rewards/game_reward_func/std': 0.18307219445705414, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3310000002384186, 'reward_std': 0.18307219445705414, 'frac_reward_zero_std': 0.0, 'kl': 0.00945055519696325, 'entropy': 0.26019302010536194, 'clip_ratio/low_mean': 0.011625743703916669, 'clip_ratio/low_min': 0.011625743703916669, 'clip_ratio/high_mean': 0.0062255859375, 'clip_ratio/high_max': 0.0062255859375, 'clip_ratio/region_mean': 0.01785132964141667, 'step_time': 18.06641900199611, 'epoch': 0.01}
{'loss': -0.1374, 'grad_norm': 0.33526256680488586, 'learning_rate': 9.888505156758758e-06, 'kl': 0.0078803151845932, 'entropy': 0.25524886325001717, 'clip_ratio/low_mean': 0.005161830224096775, 'clip_ratio/low_min': 0.005161830224096775, 'clip_ratio/high_mean': 0.0035294241970404983, 'clip_ratio/high_max': 0.0035294241970404983, 'clip_ratio/region_mean': 0.008691254421137273, 'step_time': 1.8476177639968228, 'epoch': 0.01}
{'loss': 0.339, 'grad_norm': 0.48425182700157166, 'learning_rate': 9.881672692056022e-06, 'num_tokens': 281895.0, 'completions/mean_length': 1184.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3024999797344208, 'rewards/game_reward_func/std': 0.21212327480316162, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3024999797344208, 'reward_std': 0.21212328970432281, 'frac_reward_zero_std': 0.0, 'kl': 0.01138731581158936, 'entropy': 0.27973926812410355, 'clip_ratio/low_mean': 0.0035960477543994784, 'clip_ratio/low_min': 0.0035960477543994784, 'clip_ratio/high_mean': 0.009508070070296526, 'clip_ratio/high_max': 0.009508070070296526, 'clip_ratio/region_mean': 0.013104117824696004, 'step_time': 17.916723713002284, 'epoch': 0.01}

  9%|▉         | 46/500 [07:32<1:05:45,  8.69s/it][2026-03-26 12:17:33] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:17:48] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=24.4

  9%|▉         | 47/500 [07:50<1:27:32, 11.59s/it]
                                                  

  9%|▉         | 47/500 [07:50<1:27:32, 11.59s/it]
 10%|▉         | 48/500 [07:52<1:05:42,  8.72s/it]
                                                  

 10%|▉         | 48/500 [07:52<1:05:42,  8.72s/it][2026-03-26 12:17:54] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:18:07] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.4

 10%|▉         | 49/500 [08:10<1:27:04, 11.58s/it]
                                                  

 10%|▉         | 49/500 [08:10<1:27:04, 11.58s/it]
 10%|█         | 50/500 [08:13<1:05:38,  8.75s/it]
                                                  

 10%|█         | 50/500 [08:13<1:05:38,  8.75s/it][2026-03-26 12:18:15] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:18:29] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.8

 10%|█         | 51/500 [08:31<1:28:09, 11.78s/it]
                                                  

 10%|█         | 51/500 [08:31<1:28:09, 11.78s/it]
 10%|█         | 52/500 [08:34<1:06:20,  8.88s/it]
                                                  

 10%|█         | 52/500 [08:34<1:06:20,  8.88s/it][2026-03-26 12:18:35] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:18:49] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.8

 11%|█         | 53/500 [08:52<1:26:52, 11.66s/it]
                                                  

 11%|█         | 53/500 [08:52<1:26:52, 11.66s/it]
 11%|█         | 54/500 [08:54<1:05:24,  8.80s/it]
                                                  

 11%|█         | 54/500 [08:54<1:05:24,  8.80s/it][2026-03-26 12:18:55] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:19:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.0

 11%|█         | 55/500 [09:11<1:23:43, 11.29s/it]
                                                  

 11%|█         | 55/500 [09:11<1:23:43, 11.29s/it]
 11%|█         | 56/500 [09:13<1:02:39,  8.47s/it]
                                                  

 11%|█         | 56/500 [09:13<1:02:39,  8.47s/it][2026-03-26 12:19:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:19:28] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=23.9

 11%|█▏        | 57/500 [09:30<1:23:00, 11.24s/it]
                                                  
{'loss': -0.3337, 'grad_norm': 0.7402136921882629, 'learning_rate': 9.874639560909118e-06, 'kl': 0.02336124365683645, 'entropy': 0.25737059488892555, 'clip_ratio/low_mean': 0.007068452541716397, 'clip_ratio/low_min': 0.007068452541716397, 'clip_ratio/high_mean': 0.004677220480516553, 'clip_ratio/high_max': 0.004677220480516553, 'clip_ratio/region_mean': 0.01174567302223295, 'step_time': 1.8346707320015412, 'epoch': 0.01}
{'loss': -0.5756, 'grad_norm': 0.6919435262680054, 'learning_rate': 9.867406052422525e-06, 'num_tokens': 295807.0, 'completions/mean_length': 1560.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3112500011920929, 'rewards/game_reward_func/std': 0.1595061719417572, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3112500011920929, 'reward_std': 0.1595061719417572, 'frac_reward_zero_std': 0.0, 'kl': 0.010332581121474504, 'entropy': 0.24211493879556656, 'clip_ratio/low_mean': 0.0034304887522011995, 'clip_ratio/low_min': 0.0034304887522011995, 'clip_ratio/high_mean': 0.00341796875, 'clip_ratio/high_max': 0.00341796875, 'clip_ratio/region_mean': 0.0068484575022011995, 'step_time': 18.200671311991755, 'epoch': 0.01}
{'loss': 0.5807, 'grad_norm': 0.8814780116081238, 'learning_rate': 9.85997246393744e-06, 'kl': 0.009504524408839643, 'entropy': 0.2582596093416214, 'clip_ratio/low_mean': 0.008399600046686828, 'clip_ratio/low_min': 0.008399600046686828, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.008399600046686828, 'step_time': 1.8498824839989538, 'epoch': 0.01}
{'loss': -0.0038, 'grad_norm': 0.605004072189331, 'learning_rate': 9.852339101019574e-06, 'num_tokens': 306647.0, 'completions/mean_length': 1176.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3762499988079071, 'rewards/game_reward_func/std': 0.14493915438652039, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3762499988079071, 'reward_std': 0.1449391394853592, 'frac_reward_zero_std': 0.0, 'kl': 0.010156315634958446, 'entropy': 0.26986611634492874, 'clip_ratio/low_mean': 0.003993055608589202, 'clip_ratio/low_min': 0.003993055608589202, 'clip_ratio/high_mean': 0.0024937221023719758, 'clip_ratio/high_max': 0.0024937221023719758, 'clip_ratio/region_mean': 0.006486777710961178, 'step_time': 18.08193530900462, 'epoch': 0.01}
{'loss': 0.0082, 'grad_norm': 0.20254813134670258, 'learning_rate': 9.844506277446577e-06, 'kl': 0.011706970748491585, 'entropy': 0.2735404148697853, 'clip_ratio/low_mean': 0.007949561229906976, 'clip_ratio/low_min': 0.007949561229906976, 'clip_ratio/high_mean': 0.00523337337654084, 'clip_ratio/high_max': 0.00523337337654084, 'clip_ratio/region_mean': 0.013182934606447816, 'step_time': 1.9481514919971232, 'epoch': 0.01}
{'loss': -0.3046, 'grad_norm': 0.5297548174858093, 'learning_rate': 9.836474315195148e-06, 'num_tokens': 320751.0, 'completions/mean_length': 1584.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44350001215934753, 'rewards/game_reward_func/std': 0.17707706987857819, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44350001215934753, 'reward_std': 0.17707706987857819, 'frac_reward_zero_std': 0.0, 'kl': 0.01393700612243265, 'entropy': 0.2453332617878914, 'clip_ratio/low_mean': 0.006101930397562683, 'clip_ratio/low_min': 0.006101930397562683, 'clip_ratio/high_mean': 0.0025634765625, 'clip_ratio/high_max': 0.0025634765625, 'clip_ratio/region_mean': 0.008665406960062683, 'step_time': 18.272207233007066, 'epoch': 0.01}
{'loss': 0.3057, 'grad_norm': 0.3152812719345093, 'learning_rate': 9.828243544427795e-06, 'kl': 0.009449550649151206, 'entropy': 0.26430220901966095, 'clip_ratio/low_mean': 0.00697544653667137, 'clip_ratio/low_min': 0.00697544653667137, 'clip_ratio/high_mean': 0.0027343749534338713, 'clip_ratio/high_max': 0.0027343749534338713, 'clip_ratio/region_mean': 0.009709821490105242, 'step_time': 1.9366614570026286, 'epoch': 0.01}
{'loss': -0.1297, 'grad_norm': 0.34348005056381226, 'learning_rate': 9.819814303479268e-06, 'num_tokens': 332295.0, 'completions/mean_length': 1264.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.421750009059906, 'rewards/game_reward_func/std': 0.16340811550617218, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.421750009059906, 'reward_std': 0.16340811550617218, 'frac_reward_zero_std': 0.0, 'kl': 0.02464227844029665, 'entropy': 0.23445511981844902, 'clip_ratio/low_mean': 0.006532117957249284, 'clip_ratio/low_min': 0.006532117957249284, 'clip_ratio/high_mean': 0.0048828125, 'clip_ratio/high_max': 0.0048828125, 'clip_ratio/region_mean': 0.011414930457249284, 'step_time': 17.95580527700804, 'epoch': 0.01}
{'loss': 0.1357, 'grad_norm': 0.16837072372436523, 'learning_rate': 9.811186938842645e-06, 'kl': 0.013365620281547308, 'entropy': 0.2500630356371403, 'clip_ratio/low_mean': 0.005056550784502178, 'clip_ratio/low_min': 0.005056550784502178, 'clip_ratio/high_mean': 0.0018503289902582765, 'clip_ratio/high_max': 0.0018503289902582765, 'clip_ratio/region_mean': 0.006906879774760455, 'step_time': 1.9345195530040655, 'epoch': 0.01}
{'loss': -0.46, 'grad_norm': 1.3193824291229248, 'learning_rate': 9.802361805155097e-06, 'num_tokens': 343455.0, 'completions/mean_length': 1216.0, 'completions/min_length': 384.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4169999957084656, 'rewards/game_reward_func/std': 0.13701720535755157, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4169999957084656, 'reward_std': 0.13701719045639038, 'frac_reward_zero_std': 0.0, 'kl': 0.02491458156146109, 'entropy': 0.23001604154706, 'clip_ratio/low_mean': 0.002025462919846177, 'clip_ratio/low_min': 0.002025462919846177, 'clip_ratio/high_mean': 0.014499616110697389, 'clip_ratio/high_max': 0.014499616110697389, 'clip_ratio/region_mean': 0.016525079030543566, 'step_time': 16.952559712000948, 'epoch': 0.01}
{'loss': 0.466, 'grad_norm': 0.63550865650177, 'learning_rate': 9.793339265183303e-06, 'kl': 0.018233364331535995, 'entropy': 0.27315307036042213, 'clip_ratio/low_mean': 0.004166666592936963, 'clip_ratio/low_min': 0.004166666592936963, 'clip_ratio/high_mean': 0.006673177005723119, 'clip_ratio/high_max': 0.006673177005723119, 'clip_ratio/region_mean': 0.010839843598660082, 'step_time': 1.7163508660014486, 'epoch': 0.01}

 11%|█▏        | 57/500 [09:31<1:23:00, 11.24s/it]
 12%|█▏        | 58/500 [09:32<1:02:18,  8.46s/it]
                                                  

 12%|█▏        | 58/500 [09:32<1:02:18,  8.46s/it][2026-03-26 12:19:34] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:19:47] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.9

 12%|█▏        | 59/500 [09:50<1:21:48, 11.13s/it]
                                                  

 12%|█▏        | 59/500 [09:50<1:21:48, 11.13s/it]
 12%|█▏        | 60/500 [09:52<1:01:25,  8.38s/it]
                                                  

 12%|█▏        | 60/500 [09:52<1:01:25,  8.38s/it][2026-03-26 12:19:53] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:20:07] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.4

 12%|█▏        | 61/500 [10:10<1:22:28, 11.27s/it]
                                                  

 12%|█▏        | 61/500 [10:10<1:22:28, 11.27s/it]
 12%|█▏        | 62/500 [10:12<1:02:52,  8.61s/it]
                                                  

 12%|█▏        | 62/500 [10:12<1:02:52,  8.61s/it][2026-03-26 12:20:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:20:28] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.0

 13%|█▎        | 63/500 [10:30<1:23:41, 11.49s/it]
                                                  

 13%|█▎        | 63/500 [10:30<1:23:41, 11.49s/it]
 13%|█▎        | 64/500 [10:32<1:02:50,  8.65s/it]
                                                  

 13%|█▎        | 64/500 [10:32<1:02:50,  8.65s/it][2026-03-26 12:20:34] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:20:48] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 13%|█▎        | 65/500 [10:51<1:23:45, 11.55s/it]
                                                  

 13%|█▎        | 65/500 [10:51<1:23:45, 11.55s/it]
 13%|█▎        | 66/500 [10:53<1:02:53,  8.69s/it]
                                                  

 13%|█▎        | 66/500 [10:53<1:02:53,  8.69s/it][2026-03-26 12:20:54] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:21:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=15.2

 13%|█▎        | 67/500 [11:11<1:22:18, 11.40s/it]
                                                  

 13%|█▎        | 67/500 [11:11<1:22:18, 11.40s/it]
 14%|█▎        | 68/500 [11:13<1:02:04,  8.62s/it]
                                                  
{'loss': -0.1986, 'grad_norm': 0.4041138291358948, 'learning_rate': 9.784119689808545e-06, 'num_tokens': 357111.0, 'completions/mean_length': 1528.0, 'completions/min_length': 960.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.5005000233650208, 'rewards/game_reward_func/std': 0.0472772978246212, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.5005000233650208, 'reward_std': 0.0472773015499115, 'frac_reward_zero_std': 0.0, 'kl': 0.009759790147654712, 'entropy': 0.26815033331513405, 'clip_ratio/low_mean': 0.0020874669426120818, 'clip_ratio/low_min': 0.0020874669426120818, 'clip_ratio/high_mean': 0.008158052805811167, 'clip_ratio/high_max': 0.008158052805811167, 'clip_ratio/region_mean': 0.010245519748423249, 'step_time': 17.532979248004267, 'epoch': 0.01}
{'loss': 0.2012, 'grad_norm': 0.3587263226509094, 'learning_rate': 9.774703458011453e-06, 'kl': 0.009741984657011926, 'entropy': 0.23178835213184357, 'clip_ratio/low_mean': 0.006148811662569642, 'clip_ratio/low_min': 0.006148811662569642, 'clip_ratio/high_mean': 0.004579741507768631, 'clip_ratio/high_max': 0.004579741507768631, 'clip_ratio/region_mean': 0.010728553170338273, 'step_time': 1.7826927360001719, 'epoch': 0.01}
{'loss': -0.1342, 'grad_norm': 0.4646485447883606, 'learning_rate': 9.765090956856437e-06, 'num_tokens': 368719.0, 'completions/mean_length': 1272.0, 'completions/min_length': 960.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4505000114440918, 'rewards/game_reward_func/std': 0.10197898745536804, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4505000114440918, 'reward_std': 0.10197898745536804, 'frac_reward_zero_std': 0.0, 'kl': 0.019867150112986565, 'entropy': 0.23169637471437454, 'clip_ratio/low_mean': 0.00461090705357492, 'clip_ratio/low_min': 0.00461090705357492, 'clip_ratio/high_mean': 0.007903874153271317, 'clip_ratio/high_max': 0.007903874153271317, 'clip_ratio/region_mean': 0.012514781206846237, 'step_time': 17.180410724999092, 'epoch': 0.01}
{'loss': 0.14, 'grad_norm': 0.2922990024089813, 'learning_rate': 9.755282581475769e-06, 'kl': 0.010048139840364456, 'entropy': 0.2505747973918915, 'clip_ratio/low_mean': 0.0035644532181322575, 'clip_ratio/low_min': 0.0035644532181322575, 'clip_ratio/high_mean': 0.0037037369329482317, 'clip_ratio/high_max': 0.0037037369329482317, 'clip_ratio/region_mean': 0.007268190151080489, 'step_time': 1.7769609199967817, 'epoch': 0.01}
{'loss': -0.4429, 'grad_norm': 0.835355818271637, 'learning_rate': 9.745278735053345e-06, 'num_tokens': 379047.0, 'completions/mean_length': 1112.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4009999930858612, 'rewards/game_reward_func/std': 0.17774459719657898, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4009999930858612, 'reward_std': 0.17774459719657898, 'frac_reward_zero_std': 0.0, 'kl': 0.020866676466539502, 'entropy': 0.24227824807167053, 'clip_ratio/low_mean': 0.0148737981216982, 'clip_ratio/low_min': 0.0148737981216982, 'clip_ratio/high_mean': 0.00584756163880229, 'clip_ratio/high_max': 0.00584756163880229, 'clip_ratio/region_mean': 0.02072135976050049, 'step_time': 17.850780615000986, 'epoch': 0.02}
{'loss': 0.4495, 'grad_norm': 0.7259355783462524, 'learning_rate': 9.735079828808107e-06, 'kl': 0.016639498295262456, 'entropy': 0.26700256019830704, 'clip_ratio/low_mean': 0.007405598880723119, 'clip_ratio/low_min': 0.007405598880723119, 'clip_ratio/high_mean': 0.004616477061063051, 'clip_ratio/high_max': 0.004616477061063051, 'clip_ratio/region_mean': 0.01202207594178617, 'step_time': 2.228147371999512, 'epoch': 0.02}
{'loss': 0.2956, 'grad_norm': 0.39565184712409973, 'learning_rate': 9.724686281977146e-06, 'num_tokens': 389695.0, 'completions/mean_length': 1152.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4232500195503235, 'rewards/game_reward_func/std': 0.16430611908435822, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4232500195503235, 'reward_std': 0.16430610418319702, 'frac_reward_zero_std': 0.0, 'kl': 0.018536285730078816, 'entropy': 0.2736159414052963, 'clip_ratio/low_mean': 0.012145777232944965, 'clip_ratio/low_min': 0.012145777232944965, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.012145777232944965, 'step_time': 18.012939628999447, 'epoch': 0.02}
{'loss': -0.2909, 'grad_norm': 1.07630455493927, 'learning_rate': 9.714098521798466e-06, 'kl': 0.024602580699138343, 'entropy': 0.2830547094345093, 'clip_ratio/low_mean': 0.0060058594681322575, 'clip_ratio/low_min': 0.0060058594681322575, 'clip_ratio/high_mean': 0.0054931640625, 'clip_ratio/high_max': 0.0054931640625, 'clip_ratio/region_mean': 0.011499023530632257, 'step_time': 1.833961829994223, 'epoch': 0.02}
{'loss': -0.0371, 'grad_norm': 0.19897222518920898, 'learning_rate': 9.703316983493414e-06, 'num_tokens': 401687.0, 'completions/mean_length': 1320.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37575000524520874, 'rewards/game_reward_func/std': 0.18036292493343353, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37575000524520874, 'reward_std': 0.18036293983459473, 'frac_reward_zero_std': 0.0, 'kl': 0.016148767434060574, 'entropy': 0.25516459345817566, 'clip_ratio/low_mean': 0.0018650141428224742, 'clip_ratio/low_min': 0.0018650141428224742, 'clip_ratio/high_mean': 0.002708333428017795, 'clip_ratio/high_max': 0.002708333428017795, 'clip_ratio/region_mean': 0.004573347570840269, 'step_time': 18.14622504699946, 'epoch': 0.02}
{'loss': 0.0434, 'grad_norm': 0.47934389114379883, 'learning_rate': 9.692342110248802e-06, 'kl': 0.032788171200081706, 'entropy': 0.23678312078118324, 'clip_ratio/low_mean': 0.011503232875838876, 'clip_ratio/low_min': 0.011503232875838876, 'clip_ratio/high_mean': 0.0030517578125, 'clip_ratio/high_max': 0.0030517578125, 'clip_ratio/region_mean': 0.014554990688338876, 'step_time': 1.8449398709999514, 'epoch': 0.02}
{'loss': 0.3704, 'grad_norm': 0.5599517226219177, 'learning_rate': 9.681174353198687e-06, 'num_tokens': 410991.0, 'completions/mean_length': 984.0, 'completions/min_length': 64.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.30300000309944153, 'rewards/game_reward_func/std': 0.2509433627128601, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.30300000309944153, 'reward_std': 0.2509433627128601, 'frac_reward_zero_std': 0.0, 'kl': 0.015080954879522324, 'entropy': 0.22787098959088326, 'clip_ratio/low_mean': 0.009762457804754376, 'clip_ratio/low_min': 0.009762457804754376, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.011471442179754376, 'step_time': 17.547503040994343, 'epoch': 0.02}

 14%|█▎        | 68/500 [11:13<1:02:04,  8.62s/it][2026-03-26 12:21:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:21:28] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.0

 14%|█▍        | 69/500 [11:31<1:23:22, 11.61s/it]
                                                  

 14%|█▍        | 69/500 [11:31<1:23:22, 11.61s/it]
 14%|█▍        | 70/500 [11:33<1:02:47,  8.76s/it]
                                                  

 14%|█▍        | 70/500 [11:33<1:02:47,  8.76s/it][2026-03-26 12:21:35] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:21:49] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.0

 14%|█▍        | 71/500 [11:51<1:22:26, 11.53s/it]
                                                  

 14%|█▍        | 71/500 [11:51<1:22:26, 11.53s/it]
 14%|█▍        | 72/500 [11:53<1:02:07,  8.71s/it]
                                                  

 14%|█▍        | 72/500 [11:53<1:02:07,  8.71s/it][2026-03-26 12:21:55] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:22:09] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=25.0

 15%|█▍        | 73/500 [12:12<1:22:27, 11.59s/it]
                                                  

 15%|█▍        | 73/500 [12:12<1:22:27, 11.59s/it]
 15%|█▍        | 74/500 [12:14<1:02:07,  8.75s/it]
                                                  

 15%|█▍        | 74/500 [12:14<1:02:07,  8.75s/it][2026-03-26 12:22:16] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:22:29] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=20.2

 15%|█▌        | 75/500 [12:31<1:20:06, 11.31s/it]
                                                  

 15%|█▌        | 75/500 [12:31<1:20:06, 11.31s/it]
 15%|█▌        | 76/500 [12:33<1:00:46,  8.60s/it]
                                                  

 15%|█▌        | 76/500 [12:33<1:00:46,  8.60s/it][2026-03-26 12:22:35] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:22:49] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 15%|█▌        | 77/500 [12:52<1:20:47, 11.46s/it]
                                                  

 15%|█▌        | 77/500 [12:52<1:20:47, 11.46s/it]
 16%|█▌        | 78/500 [12:54<1:00:54,  8.66s/it]
                                                  

 16%|█▌        | 78/500 [12:54<1:00:54,  8.66s/it][2026-03-26 12:22:55] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:23:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.1

 16%|█▌        | 79/500 [13:11<1:18:09, 11.14s/it]
                                                  
{'loss': -0.37, 'grad_norm': 1.5910128355026245, 'learning_rate': 9.669814171405818e-06, 'kl': 0.041634669061750174, 'entropy': 0.25338973477482796, 'clip_ratio/low_mean': 0.0033482143189758062, 'clip_ratio/low_min': 0.0033482143189758062, 'clip_ratio/high_mean': 0.023166232742369175, 'clip_ratio/high_max': 0.023166232742369175, 'clip_ratio/region_mean': 0.02651444706134498, 'step_time': 1.9310360399977071, 'epoch': 0.02}
{'loss': -0.3535, 'grad_norm': 0.4202077090740204, 'learning_rate': 9.658262031842772e-06, 'num_tokens': 423687.0, 'completions/mean_length': 1408.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4455000162124634, 'rewards/game_reward_func/std': 0.13819138705730438, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4455000162124634, 'reward_std': 0.13819138705730438, 'frac_reward_zero_std': 0.0, 'kl': 0.013328683795407414, 'entropy': 0.24881090596318245, 'clip_ratio/low_mean': 0.0027901786379516125, 'clip_ratio/low_min': 0.0027901786379516125, 'clip_ratio/high_mean': 0.009406594559550285, 'clip_ratio/high_max': 0.009406594559550285, 'clip_ratio/region_mean': 0.012196773197501898, 'step_time': 18.405056678995606, 'epoch': 0.02}
{'loss': 0.3592, 'grad_norm': 0.5237486958503723, 'learning_rate': 9.64651840937276e-06, 'kl': 0.016587212216109037, 'entropy': 0.24620841816067696, 'clip_ratio/low_mean': 0.00440548884216696, 'clip_ratio/low_min': 0.00440548884216696, 'clip_ratio/high_mean': 0.00241815485060215, 'clip_ratio/high_max': 0.00241815485060215, 'clip_ratio/region_mean': 0.00682364369276911, 'step_time': 1.934532287006732, 'epoch': 0.02}
{'loss': -0.3262, 'grad_norm': 0.4535679817199707, 'learning_rate': 9.63458378673011e-06, 'num_tokens': 434847.0, 'completions/mean_length': 1216.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4012500047683716, 'rewards/game_reward_func/std': 0.12086326628923416, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4012500047683716, 'reward_std': 0.12086325883865356, 'frac_reward_zero_std': 0.0, 'kl': 0.01723103306721896, 'entropy': 0.25628236681222916, 'clip_ratio/low_mean': 0.009114583488553762, 'clip_ratio/low_min': 0.009114583488553762, 'clip_ratio/high_mean': 0.003988942829892039, 'clip_ratio/high_max': 0.003988942829892039, 'clip_ratio/region_mean': 0.013103526318445802, 'step_time': 17.809366556997702, 'epoch': 0.02}
{'loss': 0.3337, 'grad_norm': 0.4472566545009613, 'learning_rate': 9.622458654500408e-06, 'kl': 0.020181585336104035, 'entropy': 0.27352556213736534, 'clip_ratio/low_mean': 0.0075250009540468454, 'clip_ratio/low_min': 0.0075250009540468454, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.008989844704046845, 'step_time': 1.9321167220041389, 'epoch': 0.02}
{'loss': -0.5524, 'grad_norm': 0.6507447957992554, 'learning_rate': 9.610143511100354e-06, 'num_tokens': 449079.0, 'completions/mean_length': 1600.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.32749998569488525, 'rewards/game_reward_func/std': 0.15072302520275116, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.32749998569488525, 'reward_std': 0.15072302520275116, 'frac_reward_zero_std': 0.0, 'kl': 0.011596128810197115, 'entropy': 0.23924082145094872, 'clip_ratio/low_mean': 0.0015190972480922937, 'clip_ratio/low_min': 0.0015190972480922937, 'clip_ratio/high_mean': 0.0054931640625, 'clip_ratio/high_max': 0.0054931640625, 'clip_ratio/region_mean': 0.007012261310592294, 'step_time': 18.145686085001216, 'epoch': 0.02}
{'loss': 0.5586, 'grad_norm': 0.7676028609275818, 'learning_rate': 9.597638862757255e-06, 'kl': 0.015030737733468413, 'entropy': 0.23856254667043686, 'clip_ratio/low_mean': 0.002847831405233592, 'clip_ratio/low_min': 0.002847831405233592, 'clip_ratio/high_mean': 0.005408653989434242, 'clip_ratio/high_max': 0.005408653989434242, 'clip_ratio/region_mean': 0.008256485394667834, 'step_time': 1.9344448830015608, 'epoch': 0.02}
{'loss': -0.0967, 'grad_norm': 0.8117038607597351, 'learning_rate': 9.584945223488227e-06, 'num_tokens': 460879.0, 'completions/mean_length': 1296.0, 'completions/min_length': 768.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4452499747276306, 'rewards/game_reward_func/std': 0.09991960972547531, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4452499747276306, 'reward_std': 0.09991960972547531, 'frac_reward_zero_std': 0.0, 'kl': 0.02077258354984224, 'entropy': 0.22222533449530602, 'clip_ratio/low_mean': 0.0032552084885537624, 'clip_ratio/low_min': 0.0032552084885537624, 'clip_ratio/high_mean': 0.005389267927967012, 'clip_ratio/high_max': 0.005389267927967012, 'clip_ratio/region_mean': 0.008644476416520774, 'step_time': 17.129639016995498, 'epoch': 0.02}
{'loss': 0.1037, 'grad_norm': 0.1223176047205925, 'learning_rate': 9.572063115079063e-06, 'kl': 0.010637322906404734, 'entropy': 0.27497586235404015, 'clip_ratio/low_mean': 0.004715401912108064, 'clip_ratio/low_min': 0.004715401912108064, 'clip_ratio/high_mean': 0.004101562313735485, 'clip_ratio/high_max': 0.004101562313735485, 'clip_ratio/region_mean': 0.008816964225843549, 'step_time': 2.0947255530045368, 'epoch': 0.02}
{'loss': -0.2546, 'grad_norm': 0.28048112988471985, 'learning_rate': 9.558993067062785e-06, 'num_tokens': 472871.0, 'completions/mean_length': 1320.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4622499942779541, 'rewards/game_reward_func/std': 0.1327185481786728, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4622499942779541, 'reward_std': 0.1327185481786728, 'frac_reward_zero_std': 0.0, 'kl': 0.008632111013866961, 'entropy': 0.23352985829114914, 'clip_ratio/low_mean': 0.005812871968373656, 'clip_ratio/low_min': 0.005812871968373656, 'clip_ratio/high_mean': 0.005126953125, 'clip_ratio/high_max': 0.005126953125, 'clip_ratio/region_mean': 0.010939825093373656, 'step_time': 17.96006135299831, 'epoch': 0.02}
{'loss': 0.2607, 'grad_norm': 0.532623291015625, 'learning_rate': 9.545735616697875e-06, 'kl': 0.02239810861647129, 'entropy': 0.25814249366521835, 'clip_ratio/low_mean': 0.007508680660976097, 'clip_ratio/low_min': 0.007508680660976097, 'clip_ratio/high_mean': 0.0011718750465661287, 'clip_ratio/high_max': 0.0011718750465661287, 'clip_ratio/region_mean': 0.008680555707542226, 'step_time': 1.9435167420015205, 'epoch': 0.02}

 16%|█▌        | 79/500 [13:11<1:18:09, 11.14s/it]
 16%|█▌        | 80/500 [13:13<58:46,  8.40s/it]  
                                                

 16%|█▌        | 80/500 [13:13<58:46,  8.40s/it][2026-03-26 12:23:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:23:28] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.8

 16%|█▌        | 81/500 [13:31<1:19:26, 11.38s/it]
                                                  

 16%|█▌        | 81/500 [13:31<1:19:26, 11.38s/it]
 16%|█▋        | 82/500 [13:33<59:42,  8.57s/it]  
                                                

 16%|█▋        | 82/500 [13:33<59:42,  8.57s/it][2026-03-26 12:23:35] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:23:48] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.9

 17%|█▋        | 83/500 [13:51<1:18:18, 11.27s/it]
                                                  

 17%|█▋        | 83/500 [13:51<1:18:18, 11.27s/it]
 17%|█▋        | 84/500 [13:52<58:44,  8.47s/it]  
                                                

 17%|█▋        | 84/500 [13:52<58:44,  8.47s/it][2026-03-26 12:23:54] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:24:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.0

 17%|█▋        | 85/500 [14:11<1:18:39, 11.37s/it]
                                                  

 17%|█▋        | 85/500 [14:11<1:18:39, 11.37s/it]
 17%|█▋        | 86/500 [14:13<59:19,  8.60s/it]  
                                                

 17%|█▋        | 86/500 [14:13<59:19,  8.60s/it][2026-03-26 12:24:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:24:28] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.6

 17%|█▋        | 87/500 [14:31<1:19:13, 11.51s/it]
                                                  

 17%|█▋        | 87/500 [14:31<1:19:13, 11.51s/it]
 18%|█▊        | 88/500 [14:33<59:42,  8.70s/it]  
                                                

 18%|█▊        | 88/500 [14:33<59:42,  8.70s/it][2026-03-26 12:24:35] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:24:49] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.5

 18%|█▊        | 89/500 [14:52<1:19:27, 11.60s/it]
                                                  

 18%|█▊        | 89/500 [14:52<1:19:27, 11.60s/it]
 18%|█▊        | 90/500 [14:54<1:00:03,  8.79s/it]
                                                  
{'loss': 0.2774, 'grad_norm': 0.2770790159702301, 'learning_rate': 9.532291308946191e-06, 'num_tokens': 485119.0, 'completions/mean_length': 1352.0, 'completions/min_length': 960.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4892500042915344, 'rewards/game_reward_func/std': 0.1184130311012268, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4892500042915344, 'reward_std': 0.11841302365064621, 'frac_reward_zero_std': 0.0, 'kl': 0.008896102313883603, 'entropy': 0.2482627034187317, 'clip_ratio/low_mean': 0.0030975878471508622, 'clip_ratio/low_min': 0.0030975878471508622, 'clip_ratio/high_mean': 0.007606907980516553, 'clip_ratio/high_max': 0.007606907980516553, 'clip_ratio/region_mean': 0.010704495827667415, 'step_time': 16.76706083001045, 'epoch': 0.02}
{'loss': -0.2738, 'grad_norm': 0.8436809182167053, 'learning_rate': 9.518660696450567e-06, 'kl': 0.013509609503671527, 'entropy': 0.2446403093636036, 'clip_ratio/low_mean': 0.007626488106325269, 'clip_ratio/low_min': 0.007626488106325269, 'clip_ratio/high_mean': 0.004017857252620161, 'clip_ratio/high_max': 0.004017857252620161, 'clip_ratio/region_mean': 0.01164434535894543, 'step_time': 1.806377782006166, 'epoch': 0.02}
{'loss': 0.2053, 'grad_norm': 0.5018308758735657, 'learning_rate': 9.504844339512096e-06, 'num_tokens': 497175.0, 'completions/mean_length': 1328.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4477499723434448, 'rewards/game_reward_func/std': 0.17552675306797028, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4477499723434448, 'reward_std': 0.1755267232656479, 'frac_reward_zero_std': 0.0, 'kl': 0.009575106319971383, 'entropy': 0.2838059291243553, 'clip_ratio/low_mean': 0.00533805217128247, 'clip_ratio/low_min': 0.00533805217128247, 'clip_ratio/high_mean': 0.0025201612152159214, 'clip_ratio/high_max': 0.0025201612152159214, 'clip_ratio/region_mean': 0.007858213386498392, 'step_time': 18.141151331998117, 'epoch': 0.02}
{'loss': -0.2009, 'grad_norm': 0.26939791440963745, 'learning_rate': 9.490842806067095e-06, 'kl': 0.012707547517493367, 'entropy': 0.2749124690890312, 'clip_ratio/low_mean': 0.0035156250232830644, 'clip_ratio/low_min': 0.0035156250232830644, 'clip_ratio/high_mean': 0.005289713619276881, 'clip_ratio/high_max': 0.005289713619276881, 'clip_ratio/region_mean': 0.008805338642559946, 'step_time': 1.8440483090089401, 'epoch': 0.02}
{'loss': -0.2085, 'grad_norm': 0.6680570840835571, 'learning_rate': 9.476656671663766e-06, 'num_tokens': 508782.0, 'completions/mean_length': 1271.875, 'completions/min_length': 768.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40149998664855957, 'rewards/game_reward_func/std': 0.08124566078186035, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40149998664855957, 'reward_std': 0.08124566078186035, 'frac_reward_zero_std': 0.0, 'kl': 0.013618084136396646, 'entropy': 0.258772648870945, 'clip_ratio/low_mean': 0.008391203824430704, 'clip_ratio/low_min': 0.008391203824430704, 'clip_ratio/high_mean': 0.008593750186264515, 'clip_ratio/high_max': 0.008593750186264515, 'clip_ratio/region_mean': 0.01698495401069522, 'step_time': 17.37781554099638, 'epoch': 0.02}
{'loss': 0.2111, 'grad_norm': 0.4711031913757324, 'learning_rate': 9.462286519438531e-06, 'kl': 0.010778617113828659, 'entropy': 0.24979310482740402, 'clip_ratio/low_mean': 0.0047819833271205425, 'clip_ratio/low_min': 0.0047819833271205425, 'clip_ratio/high_mean': 0.003440989414229989, 'clip_ratio/high_max': 0.003440989414229989, 'clip_ratio/region_mean': 0.008222972741350532, 'step_time': 1.7788199030037504, 'epoch': 0.02}
{'loss': -0.2662, 'grad_norm': 0.4164794385433197, 'learning_rate': 9.44773294009206e-06, 'num_tokens': 520454.0, 'completions/mean_length': 1280.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.35850000381469727, 'rewards/game_reward_func/std': 0.13842275738716125, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.35850000381469727, 'reward_std': 0.13842274248600006, 'frac_reward_zero_std': 0.0, 'kl': 0.008711347123607993, 'entropy': 0.24212338402867317, 'clip_ratio/low_mean': 0.0015980113530531526, 'clip_ratio/low_min': 0.0015980113530531526, 'clip_ratio/high_mean': 0.008766456972807646, 'clip_ratio/high_max': 0.008766456972807646, 'clip_ratio/region_mean': 0.010364468325860798, 'step_time': 17.97237525098899, 'epoch': 0.02}
{'loss': 0.2677, 'grad_norm': 0.4135317802429199, 'learning_rate': 9.432996531865001e-06, 'kl': 0.013873829739168286, 'entropy': 0.26194532215595245, 'clip_ratio/low_mean': 0.012204072205349803, 'clip_ratio/low_min': 0.012204072205349803, 'clip_ratio/high_mean': 0.0013586956774815917, 'clip_ratio/high_max': 0.0013586956774815917, 'clip_ratio/region_mean': 0.013562767882831395, 'step_time': 1.93639817499934, 'epoch': 0.02}
{'loss': -0.0653, 'grad_norm': 0.2705158293247223, 'learning_rate': 9.418077900513377e-06, 'num_tokens': 533470.0, 'completions/mean_length': 1448.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40974998474121094, 'rewards/game_reward_func/std': 0.14530141651630402, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40974998474121094, 'reward_std': 0.14530141651630402, 'frac_reward_zero_std': 0.0, 'kl': 0.008368166512809694, 'entropy': 0.24835006147623062, 'clip_ratio/low_mean': 0.005340786650776863, 'clip_ratio/low_min': 0.005340786650776863, 'clip_ratio/high_mean': 0.00400506891310215, 'clip_ratio/high_max': 0.00400506891310215, 'clip_ratio/region_mean': 0.009345855563879013, 'step_time': 18.122395150996454, 'epoch': 0.02}
{'loss': 0.0679, 'grad_norm': 0.19436584413051605, 'learning_rate': 9.40297765928369e-06, 'kl': 0.011528968811035156, 'entropy': 0.24082398042082787, 'clip_ratio/low_mean': 0.004855769220739603, 'clip_ratio/low_min': 0.004855769220739603, 'clip_ratio/high_mean': 0.005859375, 'clip_ratio/high_max': 0.005859375, 'clip_ratio/region_mean': 0.010715144220739603, 'step_time': 1.941151335995528, 'epoch': 0.02}
{'loss': 0.5412, 'grad_norm': 1.307245135307312, 'learning_rate': 9.387696428887715e-06, 'num_tokens': 545910.0, 'completions/mean_length': 1376.0, 'completions/min_length': 192.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3935000002384186, 'rewards/game_reward_func/std': 0.18239283561706543, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3935000002384186, 'reward_std': 0.18239283561706543, 'frac_reward_zero_std': 0.0, 'kl': 0.019680773839354515, 'entropy': 0.251349076628685, 'clip_ratio/low_mean': 0.01023995561990887, 'clip_ratio/low_min': 0.01023995561990887, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.01023995561990887, 'step_time': 18.18792653601122, 'epoch': 0.02}

 18%|█▊        | 90/500 [14:54<1:00:03,  8.79s/it][2026-03-26 12:24:55] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:25:10] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.8

 18%|█▊        | 91/500 [15:12<1:19:31, 11.67s/it]
                                                  

 18%|█▊        | 91/500 [15:12<1:19:31, 11.67s/it]
 18%|█▊        | 92/500 [15:14<59:52,  8.80s/it]  
                                                

 18%|█▊        | 92/500 [15:14<59:52,  8.80s/it][2026-03-26 12:25:16] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:25:30] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.1

 19%|█▊        | 93/500 [15:32<1:18:45, 11.61s/it]
                                                  

 19%|█▊        | 93/500 [15:32<1:18:45, 11.61s/it]
 19%|█▉        | 94/500 [15:34<59:06,  8.73s/it]  
                                                

 19%|█▉        | 94/500 [15:34<59:06,  8.73s/it][2026-03-26 12:25:36] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:25:50] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.5

 19%|█▉        | 95/500 [15:52<1:17:43, 11.51s/it]
                                                  

 19%|█▉        | 95/500 [15:52<1:17:43, 11.51s/it]
 19%|█▉        | 96/500 [15:54<58:08,  8.63s/it]  
                                                

 19%|█▉        | 96/500 [15:54<58:08,  8.63s/it][2026-03-26 12:25:56] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:26:10] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.6

 19%|█▉        | 97/500 [16:13<1:18:04, 11.62s/it]
                                                  

 19%|█▉        | 97/500 [16:13<1:18:04, 11.62s/it]
 20%|█▉        | 98/500 [16:15<58:34,  8.74s/it]  
                                                

 20%|█▉        | 98/500 [16:15<58:34,  8.74s/it][2026-03-26 12:26:17] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:26:31] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=24.4

 20%|█▉        | 99/500 [16:34<1:17:57, 11.67s/it]
                                                  

 20%|█▉        | 99/500 [16:34<1:17:57, 11.67s/it]
 20%|██        | 100/500 [16:35<58:17,  8.74s/it] 
                                                 

 20%|██        | 100/500 [16:35<58:17,  8.74s/it][2026-03-26 12:26:37] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:26:52] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:6 D:1 (12% win) | avg_turns=24.1

 20%|██        | 101/500 [16:54<1:17:59, 11.73s/it]
                                                   
{'loss': -0.5335, 'grad_norm': 0.5538692474365234, 'learning_rate': 9.372234837476979e-06, 'kl': 0.009756987332366407, 'entropy': 0.2562131732702255, 'clip_ratio/low_mean': 0.0029296875, 'clip_ratio/low_min': 0.0029296875, 'clip_ratio/high_mean': 0.006232561427168548, 'clip_ratio/high_max': 0.006232561427168548, 'clip_ratio/region_mean': 0.009162248927168548, 'step_time': 2.04855148599745, 'epoch': 0.02}
{'loss': -0.4668, 'grad_norm': 0.6208488345146179, 'learning_rate': 9.356593520616948e-06, 'num_tokens': 559502.0, 'completions/mean_length': 1520.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.351500004529953, 'rewards/game_reward_func/std': 0.18540996313095093, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.351500004529953, 'reward_std': 0.18540997803211212, 'frac_reward_zero_std': 0.0, 'kl': 0.008412036928348243, 'entropy': 0.24747803807258606, 'clip_ratio/low_mean': 0.0014880952658131719, 'clip_ratio/low_min': 0.0014880952658131719, 'clip_ratio/high_mean': 0.0086669921875, 'clip_ratio/high_max': 0.0086669921875, 'clip_ratio/region_mean': 0.010155087453313172, 'step_time': 18.21035574000416, 'epoch': 0.02}
{'loss': 0.4727, 'grad_norm': 0.6527822017669678, 'learning_rate': 9.340773121260893e-06, 'kl': 0.01239015581086278, 'entropy': 0.2345513366162777, 'clip_ratio/low_mean': 0.0031770834466442466, 'clip_ratio/low_min': 0.0031770834466442466, 'clip_ratio/high_mean': 0.009097222005948424, 'clip_ratio/high_max': 0.009097222005948424, 'clip_ratio/region_mean': 0.012274305452592671, 'step_time': 1.9339004240027862, 'epoch': 0.02}
{'loss': 0.3119, 'grad_norm': 0.5921084880828857, 'learning_rate': 9.324774289723469e-06, 'num_tokens': 570726.0, 'completions/mean_length': 1224.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4307499825954437, 'rewards/game_reward_func/std': 0.15414997935295105, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4307499825954437, 'reward_std': 0.15414997935295105, 'frac_reward_zero_std': 0.0, 'kl': 0.015545016969554126, 'entropy': 0.27744556963443756, 'clip_ratio/low_mean': 0.00802534050308168, 'clip_ratio/low_min': 0.00802534050308168, 'clip_ratio/high_mean': 0.0028935184236615896, 'clip_ratio/high_max': 0.0028935184236615896, 'clip_ratio/region_mean': 0.010918858926743269, 'step_time': 17.968344063003315, 'epoch': 0.02}
{'loss': -0.3087, 'grad_norm': 0.5090638995170593, 'learning_rate': 9.308597683653976e-06, 'kl': 0.011252274853177369, 'entropy': 0.2297956384718418, 'clip_ratio/low_mean': 0.00642456510104239, 'clip_ratio/low_min': 0.00642456510104239, 'clip_ratio/high_mean': 0.004690987756475806, 'clip_ratio/high_max': 0.004690987756475806, 'clip_ratio/region_mean': 0.011115552857518196, 'step_time': 1.8422318889861344, 'epoch': 0.02}
{'loss': 0.5628, 'grad_norm': 1.176437258720398, 'learning_rate': 9.292243968009332e-06, 'num_tokens': 581630.0, 'completions/mean_length': 1184.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3479999899864197, 'rewards/game_reward_func/std': 0.1694900095462799, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3479999899864197, 'reward_std': 0.1694899946451187, 'frac_reward_zero_std': 0.0, 'kl': 0.02618809835985303, 'entropy': 0.2768627181649208, 'clip_ratio/low_mean': 0.01196169841568917, 'clip_ratio/low_min': 0.01196169841568917, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.01196169841568917, 'step_time': 17.809624538000207, 'epoch': 0.02}
{'loss': -0.5564, 'grad_norm': 0.6894444823265076, 'learning_rate': 9.275713815026732e-06, 'kl': 0.013439389294944704, 'entropy': 0.23642109334468842, 'clip_ratio/low_mean': 0.008976862765848637, 'clip_ratio/low_min': 0.008976862765848637, 'clip_ratio/high_mean': 0.0029109074966982007, 'clip_ratio/high_max': 0.0029109074966982007, 'clip_ratio/region_mean': 0.011887770262546837, 'step_time': 1.7464606290013762, 'epoch': 0.02}
{'loss': 0.5904, 'grad_norm': 0.7918547987937927, 'learning_rate': 9.259007904196023e-06, 'num_tokens': 595158.0, 'completions/mean_length': 1512.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3739999830722809, 'rewards/game_reward_func/std': 0.1843940019607544, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3739999830722809, 'reward_std': 0.184393972158432, 'frac_reward_zero_std': 0.0, 'kl': 0.013827139977365732, 'entropy': 0.24959608912467957, 'clip_ratio/low_mean': 0.007419431582093239, 'clip_ratio/low_min': 0.007419431582093239, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.007419431582093239, 'step_time': 18.41702655300469, 'epoch': 0.02}
{'loss': -0.5853, 'grad_norm': 0.6921424865722656, 'learning_rate': 9.242126922231763e-06, 'kl': 0.014194503659382463, 'entropy': 0.24797579646110535, 'clip_ratio/low_mean': 0.004807692486792803, 'clip_ratio/low_min': 0.004807692486792803, 'clip_ratio/high_mean': 0.004604241868946701, 'clip_ratio/high_max': 0.004604241868946701, 'clip_ratio/region_mean': 0.009411934355739504, 'step_time': 1.838694526013569, 'epoch': 0.02}
{'loss': 0.3594, 'grad_norm': 0.8678798675537109, 'learning_rate': 9.225071563045007e-06, 'num_tokens': 609070.0, 'completions/mean_length': 1560.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3502500057220459, 'rewards/game_reward_func/std': 0.21275456249713898, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3502500057220459, 'reward_std': 0.21275456249713898, 'frac_reward_zero_std': 0.0, 'kl': 0.023772327229380608, 'entropy': 0.25318100303411484, 'clip_ratio/low_mean': 0.006861772562842816, 'clip_ratio/low_min': 0.006861772562842816, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.008326616312842816, 'step_time': 18.304131276010594, 'epoch': 0.02}
{'loss': -0.3534, 'grad_norm': 0.7321760654449463, 'learning_rate': 9.207842527714767e-06, 'kl': 0.01743428665213287, 'entropy': 0.22174935415387154, 'clip_ratio/low_mean': 0.0039992560050450265, 'clip_ratio/low_min': 0.0039992560050450265, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.007905506005045027, 'step_time': 1.7516652759950375, 'epoch': 0.03}

 20%|██        | 101/500 [16:54<1:17:59, 11.73s/it]
 20%|██        | 102/500 [16:56<58:41,  8.85s/it]  
                                                 

 20%|██        | 102/500 [16:56<58:41,  8.85s/it][2026-03-26 12:26:58] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:27:12] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.0

 21%|██        | 103/500 [17:14<1:16:49, 11.61s/it]
                                                   

 21%|██        | 103/500 [17:14<1:16:49, 11.61s/it]
 21%|██        | 104/500 [17:16<57:56,  8.78s/it]  
                                                 

 21%|██        | 104/500 [17:16<57:56,  8.78s/it][2026-03-26 12:27:18] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:27:31] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.0

 21%|██        | 105/500 [17:33<1:13:40, 11.19s/it]
                                                   

 21%|██        | 105/500 [17:33<1:13:40, 11.19s/it]
 21%|██        | 106/500 [17:35<55:09,  8.40s/it]  
                                                 

 21%|██        | 106/500 [17:35<55:09,  8.40s/it][2026-03-26 12:27:37] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:27:51] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.2

 21%|██▏       | 107/500 [17:53<1:14:07, 11.32s/it]
                                                   

 21%|██▏       | 107/500 [17:53<1:14:07, 11.32s/it]
 22%|██▏       | 108/500 [17:55<55:42,  8.53s/it]  
                                                 

 22%|██▏       | 108/500 [17:55<55:42,  8.53s/it][2026-03-26 12:27:57] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:28:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.9

 22%|██▏       | 109/500 [18:10<1:08:22, 10.49s/it]
                                                   

 22%|██▏       | 109/500 [18:10<1:08:22, 10.49s/it]
 22%|██▏       | 110/500 [18:12<51:31,  7.93s/it]  
                                                 

 22%|██▏       | 110/500 [18:12<51:31,  7.93s/it][2026-03-26 12:28:14] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:28:28] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.9

 22%|██▏       | 111/500 [18:30<1:10:54, 10.94s/it]
                                                   

 22%|██▏       | 111/500 [18:30<1:10:54, 10.94s/it]
 22%|██▏       | 112/500 [18:32<53:25,  8.26s/it]  
                                                 
{'loss': 0.2249, 'grad_norm': 0.31513455510139465, 'learning_rate': 9.190440524459203e-06, 'num_tokens': 622854.0, 'completions/mean_length': 1544.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3270000219345093, 'rewards/game_reward_func/std': 0.4917897582054138, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3270000219345093, 'reward_std': 0.4917897582054138, 'frac_reward_zero_std': 0.0, 'kl': 0.0517479432746768, 'entropy': 0.24607734754681587, 'clip_ratio/low_mean': 0.00533818791154772, 'clip_ratio/low_min': 0.00533818791154772, 'clip_ratio/high_mean': 0.0025634765625, 'clip_ratio/high_max': 0.0025634765625, 'clip_ratio/region_mean': 0.00790166447404772, 'step_time': 18.12354909300484, 'epoch': 0.03}
{'loss': -0.2107, 'grad_norm': 1.5904510021209717, 'learning_rate': 9.172866268606514e-06, 'kl': 0.03411788237281144, 'entropy': 0.23334073647856712, 'clip_ratio/low_mean': 0.00502867775503546, 'clip_ratio/low_min': 0.00502867775503546, 'clip_ratio/high_mean': 0.0032552082557231188, 'clip_ratio/high_max': 0.0032552082557231188, 'clip_ratio/region_mean': 0.008283886010758579, 'step_time': 1.938077571001486, 'epoch': 0.03}
{'loss': 0.545, 'grad_norm': 0.7099527716636658, 'learning_rate': 9.15512048256552e-06, 'num_tokens': 634525.0, 'completions/mean_length': 1279.875, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3199999928474426, 'rewards/game_reward_func/std': 0.18397514522075653, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3199999928474426, 'reward_std': 0.18397516012191772, 'frac_reward_zero_std': 0.0, 'kl': 0.026215307880192995, 'entropy': 0.26509760320186615, 'clip_ratio/low_mean': 0.0074738990515470505, 'clip_ratio/low_min': 0.0074738990515470505, 'clip_ratio/high_mean': 0.004616477061063051, 'clip_ratio/high_max': 0.004616477061063051, 'clip_ratio/region_mean': 0.012090376112610102, 'step_time': 17.868418955004017, 'epoch': 0.03}
{'loss': -0.5345, 'grad_norm': 0.9589053988456726, 'learning_rate': 9.137203895795983e-06, 'kl': 0.02827680017799139, 'entropy': 0.22707270085811615, 'clip_ratio/low_mean': 0.0015980113530531526, 'clip_ratio/low_min': 0.0015980113530531526, 'clip_ratio/high_mean': 0.008701869286596775, 'clip_ratio/high_max': 0.008701869286596775, 'clip_ratio/region_mean': 0.010299880639649928, 'step_time': 1.9965392540107132, 'epoch': 0.03}
{'loss': 0.009, 'grad_norm': 0.33801960945129395, 'learning_rate': 9.119117244778609e-06, 'num_tokens': 646709.0, 'completions/mean_length': 1344.0, 'completions/min_length': 320.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4359999895095825, 'rewards/game_reward_func/std': 0.11741013824939728, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4359999895095825, 'reward_std': 0.11741013079881668, 'frac_reward_zero_std': 0.0, 'kl': 0.017973181791603565, 'entropy': 0.24204719439148903, 'clip_ratio/low_mean': 0.003512731404043734, 'clip_ratio/low_min': 0.003512731404043734, 'clip_ratio/high_mean': 0.0072195869870483875, 'clip_ratio/high_max': 0.0072195869870483875, 'clip_ratio/region_mean': 0.010732318391092122, 'step_time': 16.63926879299106, 'epoch': 0.03}
{'loss': -0.0024, 'grad_norm': 0.5857541561126709, 'learning_rate': 9.10086127298478e-06, 'kl': 0.02820323989726603, 'entropy': 0.23740052059292793, 'clip_ratio/low_mean': 0.0035099637461826205, 'clip_ratio/low_min': 0.0035099637461826205, 'clip_ratio/high_mean': 0.009988839505240321, 'clip_ratio/high_max': 0.009988839505240321, 'clip_ratio/region_mean': 0.013498803251422942, 'step_time': 1.7210496080006124, 'epoch': 0.03}
{'loss': 0.1267, 'grad_norm': 0.31531473994255066, 'learning_rate': 9.082436730845993e-06, 'num_tokens': 658509.0, 'completions/mean_length': 1296.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4490000009536743, 'rewards/game_reward_func/std': 0.11362092941999435, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4490000009536743, 'reward_std': 0.11362090706825256, 'frac_reward_zero_std': 0.0, 'kl': 0.01982481451705098, 'entropy': 0.25011543929576874, 'clip_ratio/low_mean': 0.0035099262604489923, 'clip_ratio/low_min': 0.0035099262604489923, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.007416176260448992, 'step_time': 17.931809545996657, 'epoch': 0.03}
{'loss': -0.1225, 'grad_norm': 0.24294602870941162, 'learning_rate': 9.063844375723014e-06, 'kl': 0.017330227419734, 'entropy': 0.25771986693143845, 'clip_ratio/low_mean': 0.005316840251907706, 'clip_ratio/low_min': 0.005316840251907706, 'clip_ratio/high_mean': 0.0032958984375, 'clip_ratio/high_max': 0.0032958984375, 'clip_ratio/region_mean': 0.008612738689407706, 'step_time': 1.8336029300044174, 'epoch': 0.03}
{'loss': -0.5125, 'grad_norm': 1.283988356590271, 'learning_rate': 9.045084971874738e-06, 'num_tokens': 669093.0, 'completions/mean_length': 1144.0, 'completions/min_length': 512.0, 'completions/max_length': 1728.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.36274999380111694, 'rewards/game_reward_func/std': 0.13560841977596283, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.36274999380111694, 'reward_std': 0.13560841977596283, 'frac_reward_zero_std': 0.0, 'kl': 0.02267158147878945, 'entropy': 0.28035804629325867, 'clip_ratio/low_mean': 0.001953125, 'clip_ratio/low_min': 0.001953125, 'clip_ratio/high_mean': 0.01050298917107284, 'clip_ratio/high_max': 0.01050298917107284, 'clip_ratio/region_mean': 0.01245611417107284, 'step_time': 14.917600227003277, 'epoch': 0.03}
{'loss': 0.5191, 'grad_norm': 1.045464038848877, 'learning_rate': 9.026159290426782e-06, 'kl': 0.013380892807617784, 'entropy': 0.2457539178431034, 'clip_ratio/low_mean': 0.0036490115453489125, 'clip_ratio/low_min': 0.0036490115453489125, 'clip_ratio/high_mean': 0.005859375, 'clip_ratio/high_max': 0.005859375, 'clip_ratio/region_mean': 0.009508386545348912, 'step_time': 1.7872885960023268, 'epoch': 0.03}
{'loss': -0.0951, 'grad_norm': 0.7608670592308044, 'learning_rate': 9.007068109339783e-06, 'num_tokens': 679677.0, 'completions/mean_length': 1144.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3372499942779541, 'rewards/game_reward_func/std': 0.15640950202941895, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3372499942779541, 'reward_std': 0.15640950202941895, 'frac_reward_zero_std': 0.0, 'kl': 0.01693384675309062, 'entropy': 0.2910619340837002, 'clip_ratio/low_mean': 0.002946671098470688, 'clip_ratio/low_min': 0.002946671098470688, 'clip_ratio/high_mean': 0.00785006012301892, 'clip_ratio/high_max': 0.00785006012301892, 'clip_ratio/region_mean': 0.010796731221489608, 'step_time': 17.774019993004913, 'epoch': 0.03}

 22%|██▏       | 112/500 [18:32<53:25,  8.26s/it][2026-03-26 12:28:34] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:28:48] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=24.0

 23%|██▎       | 113/500 [18:51<1:12:55, 11.31s/it]
                                                   

 23%|██▎       | 113/500 [18:51<1:12:55, 11.31s/it]
 23%|██▎       | 114/500 [18:53<54:36,  8.49s/it]  
                                                 

 23%|██▎       | 114/500 [18:53<54:36,  8.49s/it][2026-03-26 12:28:54] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:29:08] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.9

 23%|██▎       | 115/500 [19:11<1:13:25, 11.44s/it]
                                                   

 23%|██▎       | 115/500 [19:11<1:13:25, 11.44s/it]
 23%|██▎       | 116/500 [19:13<54:55,  8.58s/it]  
                                                 

 23%|██▎       | 116/500 [19:13<54:55,  8.58s/it][2026-03-26 12:29:15] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:29:29] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=27.1

 23%|██▎       | 117/500 [19:32<1:14:15, 11.63s/it]
                                                   

 23%|██▎       | 117/500 [19:32<1:14:15, 11.63s/it]
 24%|██▎       | 118/500 [19:34<55:44,  8.76s/it]  
                                                 

 24%|██▎       | 118/500 [19:34<55:44,  8.76s/it][2026-03-26 12:29:35] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:29:49] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:6 D:1 (12% win) | avg_turns=20.6

 24%|██▍       | 119/500 [19:52<1:13:07, 11.51s/it]
                                                   

 24%|██▍       | 119/500 [19:52<1:13:07, 11.51s/it]
 24%|██▍       | 120/500 [19:54<55:05,  8.70s/it]  
                                                 

 24%|██▍       | 120/500 [19:54<55:05,  8.70s/it][2026-03-26 12:29:55] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:30:09] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=22.9

 24%|██▍       | 121/500 [20:12<1:12:37, 11.50s/it]
                                                   

 24%|██▍       | 121/500 [20:12<1:12:37, 11.50s/it]
 24%|██▍       | 122/500 [20:14<54:43,  8.69s/it]  
                                                 

 24%|██▍       | 122/500 [20:14<54:43,  8.69s/it][2026-03-26 12:30:16] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:30:29] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=20.1

 25%|██▍       | 123/500 [20:32<1:12:19, 11.51s/it]
                                                   
{'loss': 0.1005, 'grad_norm': 0.3730720281600952, 'learning_rate': 8.987812213377423e-06, 'kl': 0.021604516310617328, 'entropy': 0.2528129890561104, 'clip_ratio/low_mean': 0.0038629540940746665, 'clip_ratio/low_min': 0.0038629540940746665, 'clip_ratio/high_mean': 0.0030517578125, 'clip_ratio/high_max': 0.0030517578125, 'clip_ratio/region_mean': 0.0069147119065746665, 'step_time': 1.8348184569986188, 'epoch': 0.03}
{'loss': 0.3009, 'grad_norm': 0.4029550552368164, 'learning_rate': 8.968392394074164e-06, 'num_tokens': 693397.0, 'completions/mean_length': 1536.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33399999141693115, 'rewards/game_reward_func/std': 0.11695176362991333, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33399999141693115, 'reward_std': 0.11695176362991333, 'frac_reward_zero_std': 0.0, 'kl': 0.010866183671168983, 'entropy': 0.2479185163974762, 'clip_ratio/low_mean': 0.006365184439346194, 'clip_ratio/low_min': 0.006365184439346194, 'clip_ratio/high_mean': 0.0014816810144111514, 'clip_ratio/high_max': 0.0014816810144111514, 'clip_ratio/region_mean': 0.007846865453757346, 'step_time': 18.224278342997422, 'epoch': 0.03}
{'loss': -0.2982, 'grad_norm': 0.2715843915939331, 'learning_rate': 8.948809449702712e-06, 'kl': 0.007904741796664894, 'entropy': 0.23955756053328514, 'clip_ratio/low_mean': 0.0035486984997987747, 'clip_ratio/low_min': 0.0035486984997987747, 'clip_ratio/high_mean': 0.005615234375, 'clip_ratio/high_max': 0.005615234375, 'clip_ratio/region_mean': 0.009163932874798775, 'step_time': 1.7552590829945984, 'epoch': 0.03}
{'loss': 0.7011, 'grad_norm': 1.0780951976776123, 'learning_rate': 8.929064185241214e-06, 'num_tokens': 706029.0, 'completions/mean_length': 1400.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.30674999952316284, 'rewards/game_reward_func/std': 0.19571098685264587, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.30674999952316284, 'reward_std': 0.19571097195148468, 'frac_reward_zero_std': 0.0, 'kl': 0.013662695419043303, 'entropy': 0.2939666584134102, 'clip_ratio/low_mean': 0.009556654375046492, 'clip_ratio/low_min': 0.009556654375046492, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.009556654375046492, 'step_time': 18.14913904599962, 'epoch': 0.03}
{'loss': -0.6936, 'grad_norm': 0.7805951833724976, 'learning_rate': 8.90915741234015e-06, 'kl': 0.009292536182329059, 'entropy': 0.22472086176276207, 'clip_ratio/low_mean': 0.004317434038966894, 'clip_ratio/low_min': 0.004317434038966894, 'clip_ratio/high_mean': 0.00424429087433964, 'clip_ratio/high_max': 0.00424429087433964, 'clip_ratio/region_mean': 0.008561724913306534, 'step_time': 1.7389709049894009, 'epoch': 0.03}
{'loss': -0.2999, 'grad_norm': 0.3553387224674225, 'learning_rate': 8.889089949288986e-06, 'num_tokens': 721349.0, 'completions/mean_length': 1736.0, 'completions/min_length': 1088.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2737500071525574, 'rewards/game_reward_func/std': 0.17715348303318024, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2737500071525574, 'reward_std': 0.17715348303318024, 'frac_reward_zero_std': 0.0, 'kl': 0.01040347630623728, 'entropy': 0.2428945191204548, 'clip_ratio/low_mean': 0.0013786765048280358, 'clip_ratio/low_min': 0.0013786765048280358, 'clip_ratio/high_mean': 0.004824431031011045, 'clip_ratio/high_max': 0.004824431031011045, 'clip_ratio/region_mean': 0.006203107535839081, 'step_time': 18.573224940002547, 'epoch': 0.03}
{'loss': 0.3044, 'grad_norm': 0.3893299400806427, 'learning_rate': 8.868862620982534e-06, 'kl': 0.006898009101860225, 'entropy': 0.23510432243347168, 'clip_ratio/low_mean': 0.0030996816931292415, 'clip_ratio/low_min': 0.0030996816931292415, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.0045645254431292415, 'step_time': 1.8645651080005337, 'epoch': 0.03}
{'loss': -0.3755, 'grad_norm': 0.6858262419700623, 'learning_rate': 8.84847625888703e-06, 'num_tokens': 733341.0, 'completions/mean_length': 1320.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.18974998593330383, 'rewards/game_reward_func/std': 0.39937442541122437, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.18974998593330383, 'reward_std': 0.399374395608902, 'frac_reward_zero_std': 0.0, 'kl': 0.010353296296671033, 'entropy': 0.2437976635992527, 'clip_ratio/low_mean': 0.003248054184950888, 'clip_ratio/low_min': 0.003248054184950888, 'clip_ratio/high_mean': 0.0027587891090661287, 'clip_ratio/high_max': 0.0027587891090661287, 'clip_ratio/region_mean': 0.006006843294017017, 'step_time': 17.77039009199507, 'epoch': 0.03}
{'loss': 0.3818, 'grad_norm': 0.5822904109954834, 'learning_rate': 8.827931701005974e-06, 'kl': 0.010310318088158965, 'entropy': 0.23503846675157547, 'clip_ratio/low_mean': 0.007482329034246504, 'clip_ratio/low_min': 0.007482329034246504, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.007482329034246504, 'step_time': 1.9320831459917827, 'epoch': 0.03}
{'loss': -0.6398, 'grad_norm': 0.6592799425125122, 'learning_rate': 8.807229791845673e-06, 'num_tokens': 746485.0, 'completions/mean_length': 1464.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.242249995470047, 'rewards/game_reward_func/std': 0.2621830105781555, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.242249995470047, 'reward_std': 0.2621830105781555, 'frac_reward_zero_std': 0.0, 'kl': 0.006247999961487949, 'entropy': 0.22328509390354156, 'clip_ratio/low_mean': 0.0020380434580147266, 'clip_ratio/low_min': 0.0020380434580147266, 'clip_ratio/high_mean': 0.0068359375, 'clip_ratio/high_max': 0.0068359375, 'clip_ratio/region_mean': 0.008873980958014727, 'step_time': 17.867190609998943, 'epoch': 0.03}
{'loss': 0.6418, 'grad_norm': 0.9540932178497314, 'learning_rate': 8.786371382380527e-06, 'kl': 0.013963257195428014, 'entropy': 0.2729039452970028, 'clip_ratio/low_mean': 0.009858631063252687, 'clip_ratio/low_min': 0.009858631063252687, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.009858631063252687, 'step_time': 1.933188852002786, 'epoch': 0.03}

 25%|██▍       | 123/500 [20:32<1:12:19, 11.51s/it]
 25%|██▍       | 124/500 [20:34<54:30,  8.70s/it]  
                                                 

 25%|██▍       | 124/500 [20:34<54:30,  8.70s/it][2026-03-26 12:30:36] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:30:50] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=25.9

 25%|██▌       | 125/500 [20:53<1:12:40, 11.63s/it]
                                                   

 25%|██▌       | 125/500 [20:53<1:12:40, 11.63s/it]
 25%|██▌       | 126/500 [20:55<54:19,  8.71s/it]  
                                                 

 25%|██▌       | 126/500 [20:55<54:19,  8.71s/it][2026-03-26 12:30:56] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:31:07] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=18.0

 25%|██▌       | 127/500 [21:09<1:05:25, 10.52s/it]
                                                   

 25%|██▌       | 127/500 [21:09<1:05:25, 10.52s/it]
 26%|██▌       | 128/500 [21:11<48:46,  7.87s/it]  
                                                 

 26%|██▌       | 128/500 [21:11<48:46,  7.87s/it][2026-03-26 12:31:13] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:31:26] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=18.8

 26%|██▌       | 129/500 [21:29<1:07:08, 10.86s/it]
                                                   

 26%|██▌       | 129/500 [21:29<1:07:08, 10.86s/it]
 26%|██▌       | 130/500 [21:31<50:36,  8.21s/it]  
                                                 

 26%|██▌       | 130/500 [21:31<50:36,  8.21s/it][2026-03-26 12:31:32] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:31:46] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.9

 26%|██▌       | 131/500 [21:49<1:09:18, 11.27s/it]
                                                   

 26%|██▌       | 131/500 [21:49<1:09:18, 11.27s/it]
 26%|██▋       | 132/500 [21:51<52:05,  8.49s/it]  
                                                 

 26%|██▋       | 132/500 [21:51<52:05,  8.49s/it][2026-03-26 12:31:53] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:32:07] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=26.6

 27%|██▋       | 133/500 [22:10<1:10:14, 11.48s/it]
                                                   

 27%|██▋       | 133/500 [22:10<1:10:14, 11.48s/it]
 27%|██▋       | 134/500 [22:12<52:32,  8.61s/it]  
                                                 
{'loss': 0.161, 'grad_norm': 0.5502048134803772, 'learning_rate': 8.765357330018056e-06, 'num_tokens': 758221.0, 'completions/mean_length': 1288.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2057499885559082, 'rewards/game_reward_func/std': 0.2749035060405731, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2057499885559082, 'reward_std': 0.2749035060405731, 'frac_reward_zero_std': 0.0, 'kl': 0.012727877707220614, 'entropy': 0.26961446925997734, 'clip_ratio/low_mean': 0.010486421408131719, 'clip_ratio/low_min': 0.010486421408131719, 'clip_ratio/high_mean': 0.003662109375, 'clip_ratio/high_max': 0.003662109375, 'clip_ratio/region_mean': 0.014148530783131719, 'step_time': 17.923229395993985, 'epoch': 0.03}
{'loss': -0.1574, 'grad_norm': 0.26975181698799133, 'learning_rate': 8.74418849856364e-06, 'kl': 0.017956147203221917, 'entropy': 0.24346638098359108, 'clip_ratio/low_mean': 0.009097450645640492, 'clip_ratio/low_min': 0.009097450645640492, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.012027138145640492, 'step_time': 1.9686110869952245, 'epoch': 0.03}
{'loss': 0.4118, 'grad_norm': 0.6816949844360352, 'learning_rate': 8.722865758185036e-06, 'num_tokens': 772901.0, 'completions/mean_length': 1656.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.34724998474121094, 'rewards/game_reward_func/std': 0.1750524491071701, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.34724998474121094, 'reward_std': 0.1750524342060089, 'frac_reward_zero_std': 0.0, 'kl': 0.00973020144738257, 'entropy': 0.2248334363102913, 'clip_ratio/low_mean': 0.0055843659793026745, 'clip_ratio/low_min': 0.0055843659793026745, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.0072933503543026745, 'step_time': 18.283243077006773, 'epoch': 0.03}
{'loss': -0.4083, 'grad_norm': 0.7254436612129211, 'learning_rate': 8.701389985376578e-06, 'kl': 0.009300228906795382, 'entropy': 0.2307119034230709, 'clip_ratio/low_mean': 0.0018028846243396401, 'clip_ratio/low_min': 0.0018028846243396401, 'clip_ratio/high_mean': 0.008745979052037, 'clip_ratio/high_max': 0.008745979052037, 'clip_ratio/region_mean': 0.01054886367637664, 'step_time': 1.7465104729999439, 'epoch': 0.03}
{'loss': 0.2852, 'grad_norm': 0.579273521900177, 'learning_rate': 8.679762062923176e-06, 'num_tokens': 783549.0, 'completions/mean_length': 1152.0, 'completions/min_length': 640.0, 'completions/max_length': 1664.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.43299999833106995, 'rewards/game_reward_func/std': 0.12977783381938934, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.43299999833106995, 'reward_std': 0.12977783381938934, 'frac_reward_zero_std': 0.0, 'kl': 0.018729238538071513, 'entropy': 0.29446572810411453, 'clip_ratio/low_mean': 0.009842218132689595, 'clip_ratio/low_min': 0.009842218132689595, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.013748468132689595, 'step_time': 14.624060366993945, 'epoch': 0.03}
{'loss': -0.2847, 'grad_norm': 0.47373929619789124, 'learning_rate': 8.657982879864007e-06, 'kl': 0.010724836261942983, 'entropy': 0.24631743133068085, 'clip_ratio/low_mean': 0.003281249897554517, 'clip_ratio/low_min': 0.003281249897554517, 'clip_ratio/high_mean': 0.00565953366458416, 'clip_ratio/high_max': 0.00565953366458416, 'clip_ratio/region_mean': 0.008940783562138677, 'step_time': 1.5211523469988606, 'epoch': 0.03}
{'loss': -0.1195, 'grad_norm': 0.513495147228241, 'learning_rate': 8.636053331455986e-06, 'num_tokens': 794581.0, 'completions/mean_length': 1200.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37849998474121094, 'rewards/game_reward_func/std': 0.17318446934223175, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37849998474121094, 'reward_std': 0.17318446934223175, 'frac_reward_zero_std': 0.0, 'kl': 0.02387714560609311, 'entropy': 0.24244952574372292, 'clip_ratio/low_mean': 0.0069754463620483875, 'clip_ratio/low_min': 0.0069754463620483875, 'clip_ratio/high_mean': 0.0049990699626505375, 'clip_ratio/high_max': 0.0049990699626505375, 'clip_ratio/region_mean': 0.011974516324698925, 'step_time': 17.65651998899557, 'epoch': 0.03}
{'loss': 0.1275, 'grad_norm': 0.9646378755569458, 'learning_rate': 8.613974319136959e-06, 'kl': 0.01778136577922851, 'entropy': 0.2227303721010685, 'clip_ratio/low_mean': 0.0050426137750037014, 'clip_ratio/low_min': 0.0050426137750037014, 'clip_ratio/high_mean': 0.0030761719681322575, 'clip_ratio/high_max': 0.0030761719681322575, 'clip_ratio/region_mean': 0.008118785743135959, 'step_time': 1.838784879008017, 'epoch': 0.03}
{'loss': -0.3985, 'grad_norm': 0.6886336803436279, 'learning_rate': 8.591746750488639e-06, 'num_tokens': 807213.0, 'completions/mean_length': 1400.0, 'completions/min_length': 192.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3697499930858612, 'rewards/game_reward_func/std': 0.20761899650096893, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3697499930858612, 'reward_std': 0.20761898159980774, 'frac_reward_zero_std': 0.0, 'kl': 0.016167270950973034, 'entropy': 0.2467125467956066, 'clip_ratio/low_mean': 0.004271859652362764, 'clip_ratio/low_min': 0.004271859652362764, 'clip_ratio/high_mean': 0.00302358774933964, 'clip_ratio/high_max': 0.00302358774933964, 'clip_ratio/region_mean': 0.007295447401702404, 'step_time': 18.232511702008196, 'epoch': 0.03}
{'loss': 0.4103, 'grad_norm': 0.8582061529159546, 'learning_rate': 8.569371539199316e-06, 'kl': 0.04871207382529974, 'entropy': 0.25146709010005, 'clip_ratio/low_mean': 0.010996586293913424, 'clip_ratio/low_min': 0.010996586293913424, 'clip_ratio/high_mean': 0.0020751953125, 'clip_ratio/high_max': 0.0020751953125, 'clip_ratio/region_mean': 0.013071781606413424, 'step_time': 1.8347265560078085, 'epoch': 0.03}
{'loss': 0.5244, 'grad_norm': 0.6337992548942566, 'learning_rate': 8.54684960502629e-06, 'num_tokens': 822277.0, 'completions/mean_length': 1704.0, 'completions/min_length': 1216.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.27674999833106995, 'rewards/game_reward_func/std': 0.21747365593910217, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.27674999833106995, 'reward_std': 0.21747365593910217, 'frac_reward_zero_std': 0.0, 'kl': 0.01632353477180004, 'entropy': 0.24017898365855217, 'clip_ratio/low_mean': 0.006710401736199856, 'clip_ratio/low_min': 0.006710401736199856, 'clip_ratio/high_mean': 0.0013427734375, 'clip_ratio/high_max': 0.0013427734375, 'clip_ratio/region_mean': 0.008053175173699856, 'step_time': 18.290430741995806, 'epoch': 0.03}

 27%|██▋       | 134/500 [22:12<52:32,  8.61s/it][2026-03-26 12:32:13] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:32:26] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=18.0

 27%|██▋       | 135/500 [22:29<1:07:42, 11.13s/it]
                                                   

 27%|██▋       | 135/500 [22:29<1:07:42, 11.13s/it]
 27%|██▋       | 136/500 [22:31<50:48,  8.38s/it]  
                                                 

 27%|██▋       | 136/500 [22:31<50:48,  8.38s/it][2026-03-26 12:32:32] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:32:46] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.9

 27%|██▋       | 137/500 [22:49<1:08:29, 11.32s/it]
                                                   

 27%|██▋       | 137/500 [22:49<1:08:29, 11.32s/it]
 28%|██▊       | 138/500 [22:51<51:53,  8.60s/it]  
                                                 

 28%|██▊       | 138/500 [22:51<51:53,  8.60s/it][2026-03-26 12:32:53] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:33:06] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.0

 28%|██▊       | 139/500 [23:09<1:08:39, 11.41s/it]
                                                   

 28%|██▊       | 139/500 [23:09<1:08:39, 11.41s/it]
 28%|██▊       | 140/500 [23:11<51:21,  8.56s/it]  
                                                 

 28%|██▊       | 140/500 [23:11<51:21,  8.56s/it][2026-03-26 12:33:13] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:33:26] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.1

 28%|██▊       | 141/500 [23:29<1:08:35, 11.46s/it]
                                                   

 28%|██▊       | 141/500 [23:29<1:08:35, 11.46s/it]
 28%|██▊       | 142/500 [23:31<51:30,  8.63s/it]  
                                                 

 28%|██▊       | 142/500 [23:31<51:30,  8.63s/it][2026-03-26 12:33:33] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:33:46] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.6

 29%|██▊       | 143/500 [23:49<1:08:06, 11.45s/it]
                                                   

 29%|██▊       | 143/500 [23:49<1:08:06, 11.45s/it]
 29%|██▉       | 144/500 [23:51<51:07,  8.62s/it]  
                                                 

 29%|██▉       | 144/500 [23:51<51:07,  8.62s/it][2026-03-26 12:33:53] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:34:06] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.2

 29%|██▉       | 145/500 [24:09<1:07:54, 11.48s/it]
                                                   
{'loss': -0.5192, 'grad_norm': 0.7234227657318115, 'learning_rate': 8.52418187375806e-06, 'kl': 0.015193892642855644, 'entropy': 0.2393970973789692, 'clip_ratio/low_mean': 0.0016741071594879031, 'clip_ratio/low_min': 0.0016741071594879031, 'clip_ratio/high_mean': 0.006801757728680968, 'clip_ratio/high_max': 0.006801757728680968, 'clip_ratio/region_mean': 0.008475864888168871, 'step_time': 1.756109733003541, 'epoch': 0.03}
{'loss': 0.1322, 'grad_norm': 0.24678459763526917, 'learning_rate': 8.501369277176275e-06, 'num_tokens': 832925.0, 'completions/mean_length': 1152.0, 'completions/min_length': 704.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.42674997448921204, 'rewards/game_reward_func/std': 0.12515105307102203, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.42674997448921204, 'reward_std': 0.12515103816986084, 'frac_reward_zero_std': 0.0, 'kl': 0.02310286625288427, 'entropy': 0.26554569229483604, 'clip_ratio/low_mean': 0.006417410681024194, 'clip_ratio/low_min': 0.006417410681024194, 'clip_ratio/high_mean': 0.009482563473284245, 'clip_ratio/high_max': 0.009482563473284245, 'clip_ratio/region_mean': 0.01589997415430844, 'step_time': 16.803992986002413, 'epoch': 0.03}
{'loss': -0.1253, 'grad_norm': 0.7826751470565796, 'learning_rate': 8.478412753017433e-06, 'kl': 0.024643357377499342, 'entropy': 0.21662675216794014, 'clip_ratio/low_mean': 0.0026881720405071974, 'clip_ratio/low_min': 0.0026881720405071974, 'clip_ratio/high_mean': 0.007504735025577247, 'clip_ratio/high_max': 0.007504735025577247, 'clip_ratio/region_mean': 0.010192907066084445, 'step_time': 1.8004611200012732, 'epoch': 0.03}
{'loss': 0.4745, 'grad_norm': 0.774519681930542, 'learning_rate': 8.455313244934324e-06, 'num_tokens': 845045.0, 'completions/mean_length': 1336.0, 'completions/min_length': 64.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.43174999952316284, 'rewards/game_reward_func/std': 0.19921399652957916, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.43174999952316284, 'reward_std': 0.19921399652957916, 'frac_reward_zero_std': 0.0, 'kl': 0.01972282282076776, 'entropy': 0.25413502380251884, 'clip_ratio/low_mean': 0.008768442668952048, 'clip_ratio/low_min': 0.008768442668952048, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.008768442668952048, 'step_time': 17.998860387990135, 'epoch': 0.03}
{'loss': -0.4665, 'grad_norm': 0.9366447925567627, 'learning_rate': 8.432071702457253e-06, 'kl': 0.045985091011971235, 'entropy': 0.32948510348796844, 'clip_ratio/low_mean': 0.002269345277454704, 'clip_ratio/low_min': 0.002269345277454704, 'clip_ratio/high_mean': 0.0177001953125, 'clip_ratio/high_max': 0.0177001953125, 'clip_ratio/region_mean': 0.019969540589954704, 'step_time': 2.0786396769981366, 'epoch': 0.03}
{'loss': 0.6209, 'grad_norm': 1.0906918048858643, 'learning_rate': 8.408689080954997e-06, 'num_tokens': 856205.0, 'completions/mean_length': 1216.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.35600000619888306, 'rewards/game_reward_func/std': 0.2175079733133316, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.35600000619888306, 'reward_std': 0.2175079733133316, 'frac_reward_zero_std': 0.0, 'kl': 0.03232350619509816, 'entropy': 0.26345430314540863, 'clip_ratio/low_mean': 0.01133628177922219, 'clip_ratio/low_min': 0.01133628177922219, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.01133628177922219, 'step_time': 17.77782362799917, 'epoch': 0.03}
{'loss': -0.6092, 'grad_norm': 0.8163111805915833, 'learning_rate': 8.38516634159555e-06, 'kl': 0.01914301561191678, 'entropy': 0.24661560729146004, 'clip_ratio/low_mean': 0.005020679964218289, 'clip_ratio/low_min': 0.005020679964218289, 'clip_ratio/high_mean': 0.002685546875, 'clip_ratio/high_max': 0.002685546875, 'clip_ratio/region_mean': 0.007706226839218289, 'step_time': 1.7395189360104268, 'epoch': 0.04}
{'loss': 0.4507, 'grad_norm': 0.9030769467353821, 'learning_rate': 8.361504451306585e-06, 'num_tokens': 869989.0, 'completions/mean_length': 1544.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44325000047683716, 'rewards/game_reward_func/std': 0.0925816223025322, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44325000047683716, 'reward_std': 0.0925816223025322, 'frac_reward_zero_std': 0.0, 'kl': 0.026089325780048966, 'entropy': 0.23946155607700348, 'clip_ratio/low_mean': 0.007153320359066129, 'clip_ratio/low_min': 0.007153320359066129, 'clip_ratio/high_mean': 0.0022135416511446238, 'clip_ratio/high_max': 0.0022135416511446238, 'clip_ratio/region_mean': 0.009366862010210752, 'step_time': 18.054101473004266, 'epoch': 0.04}
{'loss': -0.4414, 'grad_norm': 0.66694176197052, 'learning_rate': 8.337704382735741e-06, 'kl': 0.019994647474959493, 'entropy': 0.2361157163977623, 'clip_ratio/low_mean': 0.001953125, 'clip_ratio/low_min': 0.001953125, 'clip_ratio/high_mean': 0.0034912109840661287, 'clip_ratio/high_max': 0.0034912109840661287, 'clip_ratio/region_mean': 0.005444335984066129, 'step_time': 1.8559434129929286, 'epoch': 0.04}
{'loss': 0.2595, 'grad_norm': 0.30412086844444275, 'learning_rate': 8.313767114210615e-06, 'num_tokens': 881469.0, 'completions/mean_length': 1256.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3567500114440918, 'rewards/game_reward_func/std': 0.2273925393819809, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3567500114440918, 'reward_std': 0.2273925244808197, 'frac_reward_zero_std': 0.0, 'kl': 0.021042161155492067, 'entropy': 0.2541835717856884, 'clip_ratio/low_mean': 0.005254836403764784, 'clip_ratio/low_min': 0.005254836403764784, 'clip_ratio/high_mean': 0.0036272320430725813, 'clip_ratio/high_max': 0.0036272320430725813, 'clip_ratio/region_mean': 0.008882068446837366, 'step_time': 17.822891699004686, 'epoch': 0.04}
{'loss': -0.247, 'grad_norm': 0.7204186320304871, 'learning_rate': 8.289693629698564e-06, 'kl': 0.03055092808790505, 'entropy': 0.25377408415079117, 'clip_ratio/low_mean': 0.003627232159487903, 'clip_ratio/low_min': 0.003627232159487903, 'clip_ratio/high_mean': 0.005684989038854837, 'clip_ratio/high_max': 0.005684989038854837, 'clip_ratio/region_mean': 0.00931222119834274, 'step_time': 1.8447701100012637, 'epoch': 0.04}

 29%|██▉       | 145/500 [24:09<1:07:54, 11.48s/it]
 29%|██▉       | 146/500 [24:11<51:10,  8.67s/it]  
                                                 

 29%|██▉       | 146/500 [24:11<51:10,  8.67s/it][2026-03-26 12:34:13] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:34:27] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.2

 29%|██▉       | 147/500 [24:30<1:08:01, 11.56s/it]
                                                   

 29%|██▉       | 147/500 [24:30<1:08:01, 11.56s/it]
 30%|██▉       | 148/500 [24:32<51:02,  8.70s/it]  
                                                 

 30%|██▉       | 148/500 [24:32<51:02,  8.70s/it][2026-03-26 12:34:33] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:34:45] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.5

 30%|██▉       | 149/500 [24:48<1:03:32, 10.86s/it]
                                                   

 30%|██▉       | 149/500 [24:48<1:03:32, 10.86s/it]
 30%|███       | 150/500 [24:50<47:32,  8.15s/it]  
                                                 

 30%|███       | 150/500 [24:50<47:32,  8.15s/it][2026-03-26 12:34:52] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:35:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=22.0

 30%|███       | 151/500 [25:06<1:02:23, 10.73s/it]
                                                   

 30%|███       | 151/500 [25:06<1:02:23, 10.73s/it]
 30%|███       | 152/500 [25:08<47:14,  8.15s/it]  
                                                 

 30%|███       | 152/500 [25:08<47:14,  8.15s/it][2026-03-26 12:35:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:35:24] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.9

 31%|███       | 153/500 [25:27<1:05:11, 11.27s/it]
                                                   

 31%|███       | 153/500 [25:27<1:05:11, 11.27s/it]
 31%|███       | 154/500 [25:29<49:01,  8.50s/it]  
                                                 

 31%|███       | 154/500 [25:29<49:01,  8.50s/it][2026-03-26 12:35:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:35:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 31%|███       | 155/500 [25:47<1:05:30, 11.39s/it]
                                                   

 31%|███       | 155/500 [25:47<1:05:30, 11.39s/it]
 31%|███       | 156/500 [25:49<49:11,  8.58s/it]  
                                                 
{'loss': -0.6344, 'grad_norm': 1.0566189289093018, 'learning_rate': 8.265484918766243e-06, 'num_tokens': 891733.0, 'completions/mean_length': 1104.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4139999747276306, 'rewards/game_reward_func/std': 0.14992569386959076, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4139999747276306, 'reward_std': 0.14992569386959076, 'frac_reward_zero_std': 0.0, 'kl': 0.02606410370208323, 'entropy': 0.2647419944405556, 'clip_ratio/low_mean': 0.004616477293893695, 'clip_ratio/low_min': 0.004616477293893695, 'clip_ratio/high_mean': 0.00872124545276165, 'clip_ratio/high_max': 0.00872124545276165, 'clip_ratio/region_mean': 0.013337722746655345, 'step_time': 17.974753557995427, 'epoch': 0.04}
{'loss': 0.6426, 'grad_norm': 1.9593377113342285, 'learning_rate': 8.241141976538944e-06, 'kl': 0.027273591374978423, 'entropy': 0.26224450021982193, 'clip_ratio/low_mean': 0.006431308254832402, 'clip_ratio/low_min': 0.006431308254832402, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.006431308254832402, 'step_time': 1.937276322998514, 'epoch': 0.04}
{'loss': -0.0875, 'grad_norm': 0.43709975481033325, 'learning_rate': 8.216665803659671e-06, 'num_tokens': 905069.0, 'completions/mean_length': 1488.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3084999918937683, 'rewards/game_reward_func/std': 0.247348815202713, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3084999918937683, 'reward_std': 0.24734880030155182, 'frac_reward_zero_std': 0.0, 'kl': 0.024348976323381066, 'entropy': 0.25067778304219246, 'clip_ratio/low_mean': 0.004142342135310173, 'clip_ratio/low_min': 0.004142342135310173, 'clip_ratio/high_mean': 0.0038869758136570454, 'clip_ratio/high_max': 0.0038869758136570454, 'clip_ratio/region_mean': 0.008029317948967218, 'step_time': 18.125559483996767, 'epoch': 0.04}
{'loss': 0.0954, 'grad_norm': 0.22344818711280823, 'learning_rate': 8.192057406248028e-06, 'kl': 0.01570237218402326, 'entropy': 0.23357190191745758, 'clip_ratio/low_mean': 0.009948073420673609, 'clip_ratio/low_min': 0.009948073420673609, 'clip_ratio/high_mean': 0.001220703125, 'clip_ratio/high_max': 0.001220703125, 'clip_ratio/region_mean': 0.011168776545673609, 'step_time': 1.8415443069970934, 'epoch': 0.04}
{'loss': 0.4959, 'grad_norm': 1.0008870363235474, 'learning_rate': 8.16731779585885e-06, 'num_tokens': 915461.0, 'completions/mean_length': 1120.0, 'completions/min_length': 384.0, 'completions/max_length': 1856.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4729999899864197, 'rewards/game_reward_func/std': 0.11101607978343964, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4729999899864197, 'reward_std': 0.11101608723402023, 'frac_reward_zero_std': 0.0, 'kl': 0.028356908820569515, 'entropy': 0.26802515611052513, 'clip_ratio/low_mean': 0.007170024677179754, 'clip_ratio/low_min': 0.007170024677179754, 'clip_ratio/high_mean': 0.0036272320430725813, 'clip_ratio/high_max': 0.0036272320430725813, 'clip_ratio/region_mean': 0.010797256720252335, 'step_time': 15.772305691993097, 'epoch': 0.04}
{'loss': -0.4884, 'grad_norm': 0.7636714577674866, 'learning_rate': 8.142447989440618e-06, 'kl': 0.016368623822927475, 'entropy': 0.2647194564342499, 'clip_ratio/low_mean': 0.001220703125, 'clip_ratio/low_min': 0.001220703125, 'clip_ratio/high_mean': 0.005920601426623762, 'clip_ratio/high_max': 0.005920601426623762, 'clip_ratio/region_mean': 0.007141304551623762, 'step_time': 1.6480487329972675, 'epoch': 0.04}
{'loss': -0.3515, 'grad_norm': 0.6523174047470093, 'learning_rate': 8.117449009293668e-06, 'num_tokens': 928157.0, 'completions/mean_length': 1408.0, 'completions/min_length': 832.0, 'completions/max_length': 1856.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.42500001192092896, 'rewards/game_reward_func/std': 0.1411665678024292, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.42500001192092896, 'reward_std': 0.141166552901268, 'frac_reward_zero_std': 0.0, 'kl': 0.02003554650582373, 'entropy': 0.2530813552439213, 'clip_ratio/low_mean': 0.0010937500046566129, 'clip_ratio/low_min': 0.0010937500046566129, 'clip_ratio/high_mean': 0.010193452471867204, 'clip_ratio/high_max': 0.010193452471867204, 'clip_ratio/region_mean': 0.011287202476523817, 'step_time': 16.145576311995683, 'epoch': 0.04}
{'loss': 0.356, 'grad_norm': 0.38203758001327515, 'learning_rate': 8.092321883028157e-06, 'kl': 0.021110249683260918, 'entropy': 0.2407982125878334, 'clip_ratio/low_mean': 0.005682289076503366, 'clip_ratio/low_min': 0.005682289076503366, 'clip_ratio/high_mean': 0.0033052884973585606, 'clip_ratio/high_max': 0.0033052884973585606, 'clip_ratio/region_mean': 0.008987577573861927, 'step_time': 1.965320891002193, 'epoch': 0.04}
{'loss': 0.4984, 'grad_norm': 0.6875277757644653, 'learning_rate': 8.067067643521834e-06, 'num_tokens': 940213.0, 'completions/mean_length': 1328.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4229999780654907, 'rewards/game_reward_func/std': 0.1691795438528061, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4229999780654907, 'reward_std': 0.1691795438528061, 'frac_reward_zero_std': 0.0, 'kl': 0.02553822146728635, 'entropy': 0.2559574581682682, 'clip_ratio/low_mean': 0.007890402688644826, 'clip_ratio/low_min': 0.007890402688644826, 'clip_ratio/high_mean': 0.002685546875, 'clip_ratio/high_max': 0.002685546875, 'clip_ratio/region_mean': 0.010575949563644826, 'step_time': 18.37750306000089, 'epoch': 0.04}
{'loss': -0.4918, 'grad_norm': 0.8890807032585144, 'learning_rate': 8.041687328877566e-06, 'kl': 0.029950854368507862, 'entropy': 0.2223314754664898, 'clip_ratio/low_mean': 0.0068359375, 'clip_ratio/low_min': 0.0068359375, 'clip_ratio/high_mean': 0.005875651026144624, 'clip_ratio/high_max': 0.005875651026144624, 'clip_ratio/region_mean': 0.012711588526144624, 'step_time': 1.8491635499958647, 'epoch': 0.04}
{'loss': 0.1023, 'grad_norm': 0.2516108453273773, 'learning_rate': 8.016181982380682e-06, 'num_tokens': 952205.0, 'completions/mean_length': 1320.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3217499852180481, 'rewards/game_reward_func/std': 0.22442927956581116, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3217499852180481, 'reward_std': 0.22442927956581116, 'frac_reward_zero_std': 0.0, 'kl': 0.026909329928457737, 'entropy': 0.23996460810303688, 'clip_ratio/low_mean': 0.0034910929680336267, 'clip_ratio/low_min': 0.0034910929680336267, 'clip_ratio/high_mean': 0.009579613106325269, 'clip_ratio/high_max': 0.009579613106325269, 'clip_ratio/region_mean': 0.013070706074358895, 'step_time': 17.944149845010543, 'epoch': 0.04}

 31%|███       | 156/500 [25:49<49:11,  8.58s/it][2026-03-26 12:35:51] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:36:05] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.8

 31%|███▏      | 157/500 [26:07<1:05:19, 11.43s/it]
                                                   

 31%|███▏      | 157/500 [26:07<1:05:19, 11.43s/it]
 32%|███▏      | 158/500 [26:09<49:02,  8.60s/it]  
                                                 

 32%|███▏      | 158/500 [26:09<49:02,  8.60s/it][2026-03-26 12:36:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:36:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.1

 32%|███▏      | 159/500 [26:28<1:05:36, 11.55s/it]
                                                   

 32%|███▏      | 159/500 [26:28<1:05:36, 11.55s/it]
 32%|███▏      | 160/500 [26:30<49:24,  8.72s/it]  
                                                 

 32%|███▏      | 160/500 [26:30<49:24,  8.72s/it][2026-03-26 12:36:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:36:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.8

 32%|███▏      | 161/500 [26:47<1:03:54, 11.31s/it]
                                                   

 32%|███▏      | 161/500 [26:47<1:03:54, 11.31s/it]
 32%|███▏      | 162/500 [26:49<48:11,  8.55s/it]  
                                                 

 32%|███▏      | 162/500 [26:49<48:11,  8.55s/it][2026-03-26 12:36:51] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:37:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.5

 33%|███▎      | 163/500 [27:07<1:03:45, 11.35s/it]
                                                   

 33%|███▎      | 163/500 [27:07<1:03:45, 11.35s/it]
 33%|███▎      | 164/500 [27:09<48:03,  8.58s/it]  
                                                 

 33%|███▎      | 164/500 [27:09<48:03,  8.58s/it][2026-03-26 12:37:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:37:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.5

 33%|███▎      | 165/500 [27:27<1:03:56, 11.45s/it]
                                                   

 33%|███▎      | 165/500 [27:27<1:03:56, 11.45s/it]
 33%|███▎      | 166/500 [27:30<48:28,  8.71s/it]  
                                                 

 33%|███▎      | 166/500 [27:30<48:28,  8.71s/it][2026-03-26 12:37:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:37:43] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=18.6

 33%|███▎      | 167/500 [27:46<1:00:32, 10.91s/it]
                                                   
{'loss': -0.0969, 'grad_norm': 0.2479717880487442, 'learning_rate': 7.99055265245608e-06, 'kl': 0.01870459388010204, 'entropy': 0.26002977415919304, 'clip_ratio/low_mean': 0.0034505208022892475, 'clip_ratio/low_min': 0.0034505208022892475, 'clip_ratio/high_mean': 0.010120738297700882, 'clip_ratio/high_max': 0.010120738297700882, 'clip_ratio/region_mean': 0.01357125909999013, 'step_time': 1.8495571660096175, 'epoch': 0.04}
{'loss': 0.0163, 'grad_norm': 0.21290144324302673, 'learning_rate': 7.96480039262513e-06, 'num_tokens': 965797.0, 'completions/mean_length': 1520.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.36000001430511475, 'rewards/game_reward_func/std': 0.1589321494102478, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.36000001430511475, 'reward_std': 0.158932164311409, 'frac_reward_zero_std': 0.0, 'kl': 0.02067579352296889, 'entropy': 0.2289961464703083, 'clip_ratio/low_mean': 0.007585656829178333, 'clip_ratio/low_min': 0.007585656829178333, 'clip_ratio/high_mean': 0.0010986328125, 'clip_ratio/high_max': 0.0010986328125, 'clip_ratio/region_mean': 0.008684289641678333, 'step_time': 17.900619063002523, 'epoch': 0.04}
{'loss': -0.0118, 'grad_norm': 0.17760813236236572, 'learning_rate': 7.938926261462366e-06, 'kl': 0.02006071526557207, 'entropy': 0.22811203077435493, 'clip_ratio/low_mean': 0.00921702478080988, 'clip_ratio/low_min': 0.00921702478080988, 'clip_ratio/high_mean': 0.005126953125, 'clip_ratio/high_max': 0.005126953125, 'clip_ratio/region_mean': 0.01434397790580988, 'step_time': 1.848071443986555, 'epoch': 0.04}
{'loss': -0.3865, 'grad_norm': 0.6323093771934509, 'learning_rate': 7.912931322551981e-06, 'num_tokens': 978045.0, 'completions/mean_length': 1352.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3267499804496765, 'rewards/game_reward_func/std': 0.16174031794071198, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3267499804496765, 'reward_std': 0.16174031794071198, 'frac_reward_zero_std': 0.0, 'kl': 0.021611787611618638, 'entropy': 0.24305499345064163, 'clip_ratio/low_mean': 0.002170138992369175, 'clip_ratio/low_min': 0.002170138992369175, 'clip_ratio/high_mean': 0.007258488796651363, 'clip_ratio/high_max': 0.007258488796651363, 'clip_ratio/region_mean': 0.009428627789020538, 'step_time': 18.230322329007322, 'epoch': 0.04}
{'loss': 0.3935, 'grad_norm': 0.38729429244995117, 'learning_rate': 7.886816644444099e-06, 'kl': 0.025408490328118205, 'entropy': 0.2600284703075886, 'clip_ratio/low_mean': 0.004378255223855376, 'clip_ratio/low_min': 0.004378255223855376, 'clip_ratio/high_mean': 0.006944444729015231, 'clip_ratio/high_max': 0.006944444729015231, 'clip_ratio/region_mean': 0.011322699952870607, 'step_time': 1.9333899169942015, 'epoch': 0.04}
{'loss': -0.2194, 'grad_norm': 0.7579354047775269, 'learning_rate': 7.860583300610849e-06, 'num_tokens': 988565.0, 'completions/mean_length': 1136.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44699999690055847, 'rewards/game_reward_func/std': 0.08255214989185333, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44699999690055847, 'reward_std': 0.08255214244127274, 'frac_reward_zero_std': 0.0, 'kl': 0.024407658958807588, 'entropy': 0.2777163088321686, 'clip_ratio/low_mean': 0.005779763916507363, 'clip_ratio/low_min': 0.005779763916507363, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.007732888916507363, 'step_time': 17.187364218996663, 'epoch': 0.04}
{'loss': 0.2281, 'grad_norm': 0.4051305055618286, 'learning_rate': 7.83423236940225e-06, 'kl': 0.031068320851773024, 'entropy': 0.2635137774050236, 'clip_ratio/low_mean': 0.008027343777939677, 'clip_ratio/low_min': 0.008027343777939677, 'clip_ratio/high_mean': 0.0052083334885537624, 'clip_ratio/high_max': 0.0052083334885537624, 'clip_ratio/region_mean': 0.01323567726649344, 'step_time': 1.9341864320012974, 'epoch': 0.04}
{'loss': -0.5845, 'grad_norm': 0.980498731136322, 'learning_rate': 7.807764934001875e-06, 'num_tokens': 999469.0, 'completions/mean_length': 1184.0, 'completions/min_length': 192.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33149999380111694, 'rewards/game_reward_func/std': 0.18541306257247925, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33149999380111694, 'reward_std': 0.18541304767131805, 'frac_reward_zero_std': 0.0, 'kl': 0.01748046465218067, 'entropy': 0.25619230419397354, 'clip_ratio/low_mean': 0.0034374999813735485, 'clip_ratio/low_min': 0.0034374999813735485, 'clip_ratio/high_mean': 0.0051765862153843045, 'clip_ratio/high_max': 0.0051765862153843045, 'clip_ratio/region_mean': 0.008614086196757853, 'step_time': 17.705067172988493, 'epoch': 0.04}
{'loss': 0.6001, 'grad_norm': 1.8071682453155518, 'learning_rate': 7.781182082382325e-06, 'kl': 0.03149851318448782, 'entropy': 0.25597981736063957, 'clip_ratio/low_mean': 0.012690525152720511, 'clip_ratio/low_min': 0.012690525152720511, 'clip_ratio/high_mean': 0.0033482143189758062, 'clip_ratio/high_max': 0.0033482143189758062, 'clip_ratio/region_mean': 0.016038739471696317, 'step_time': 1.9353046420073952, 'epoch': 0.04}
{'loss': 0.0644, 'grad_norm': 0.1517593413591385, 'learning_rate': 7.754484907260513e-06, 'num_tokens': 1011397.0, 'completions/mean_length': 1312.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3852500021457672, 'rewards/game_reward_func/std': 0.20271286368370056, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3852500021457672, 'reward_std': 0.20271284878253937, 'frac_reward_zero_std': 0.0, 'kl': 0.018153085839003325, 'entropy': 0.24150046706199646, 'clip_ratio/low_mean': 0.0053487541154026985, 'clip_ratio/low_min': 0.0053487541154026985, 'clip_ratio/high_mean': 0.009410511469468474, 'clip_ratio/high_max': 0.009410511469468474, 'clip_ratio/region_mean': 0.014759265584871173, 'step_time': 17.963183703999675, 'epoch': 0.04}
{'loss': -0.0597, 'grad_norm': 0.3215278685092926, 'learning_rate': 7.727674506052744e-06, 'kl': 0.018686247640289366, 'entropy': 0.23824882879853249, 'clip_ratio/low_mean': 0.007199642364867032, 'clip_ratio/low_min': 0.007199642364867032, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.010129329864867032, 'step_time': 2.1345725700011826, 'epoch': 0.04}

 33%|███▎      | 167/500 [27:46<1:00:32, 10.91s/it]
 34%|███▎      | 168/500 [27:48<45:13,  8.17s/it]  
                                                 

 34%|███▎      | 168/500 [27:48<45:13,  8.17s/it][2026-03-26 12:37:49] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:38:03] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.1

 34%|███▍      | 169/500 [28:06<1:01:41, 11.18s/it]
                                                   

 34%|███▍      | 169/500 [28:06<1:01:41, 11.18s/it]
 34%|███▍      | 170/500 [28:08<46:12,  8.40s/it]  
                                                 

 34%|███▍      | 170/500 [28:08<46:12,  8.40s/it][2026-03-26 12:38:09] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:38:23] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=22.6

 34%|███▍      | 171/500 [28:26<1:02:19, 11.37s/it]
                                                   

 34%|███▍      | 171/500 [28:26<1:02:19, 11.37s/it]
 34%|███▍      | 172/500 [28:28<46:48,  8.56s/it]  
                                                 

 34%|███▍      | 172/500 [28:28<46:48,  8.56s/it][2026-03-26 12:38:30] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:38:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.5

 35%|███▍      | 173/500 [28:46<1:02:54, 11.54s/it]
                                                   

 35%|███▍      | 173/500 [28:46<1:02:54, 11.54s/it]
 35%|███▍      | 174/500 [28:49<47:22,  8.72s/it]  
                                                 

 35%|███▍      | 174/500 [28:49<47:22,  8.72s/it][2026-03-26 12:38:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:39:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.8

 35%|███▌      | 175/500 [29:06<1:02:11, 11.48s/it]
                                                   

 35%|███▌      | 175/500 [29:06<1:02:11, 11.48s/it]
 35%|███▌      | 176/500 [29:09<46:51,  8.68s/it]  
                                                 

 35%|███▌      | 176/500 [29:09<46:51,  8.68s/it][2026-03-26 12:39:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:39:24] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=25.0

 35%|███▌      | 177/500 [29:27<1:02:18, 11.57s/it]
                                                   

 35%|███▌      | 177/500 [29:27<1:02:18, 11.57s/it]
 36%|███▌      | 178/500 [29:29<46:33,  8.68s/it]  
                                                 
{'loss': 0.3048, 'grad_norm': 0.44258031249046326, 'learning_rate': 7.700751980829601e-06, 'num_tokens': 1022365.0, 'completions/mean_length': 1192.0, 'completions/min_length': 640.0, 'completions/max_length': 1856.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.437250018119812, 'rewards/game_reward_func/std': 0.08933365345001221, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.437250018119812, 'reward_std': 0.08933365345001221, 'frac_reward_zero_std': 0.0, 'kl': 0.014074761886149645, 'entropy': 0.2514192424714565, 'clip_ratio/low_mean': 0.00450095150154084, 'clip_ratio/low_min': 0.00450095150154084, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.00645407650154084, 'step_time': 15.879298537001887, 'epoch': 0.04}
{'loss': -0.2965, 'grad_norm': 0.875178337097168, 'learning_rate': 7.673718438270649e-06, 'kl': 0.018200083635747433, 'entropy': 0.24272442981600761, 'clip_ratio/low_mean': 0.0026518722879700363, 'clip_ratio/low_min': 0.0026518722879700363, 'clip_ratio/high_mean': 0.007924107136204839, 'clip_ratio/high_max': 0.007924107136204839, 'clip_ratio/region_mean': 0.010575979424174875, 'step_time': 1.6451688499873853, 'epoch': 0.04}
{'loss': 0.1509, 'grad_norm': 0.22043107450008392, 'learning_rate': 7.646574989618938e-06, 'num_tokens': 1034613.0, 'completions/mean_length': 1352.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.34424999356269836, 'rewards/game_reward_func/std': 0.21899624168872833, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.34424999356269836, 'reward_std': 0.21899622678756714, 'frac_reward_zero_std': 0.0, 'kl': 0.014811874832957983, 'entropy': 0.2860284820199013, 'clip_ratio/low_mean': 0.007492897799238563, 'clip_ratio/low_min': 0.007492897799238563, 'clip_ratio/high_mean': 0.0027901786379516125, 'clip_ratio/high_max': 0.0027901786379516125, 'clip_ratio/region_mean': 0.010283076437190175, 'step_time': 18.01544014398678, 'epoch': 0.04}
{'loss': -0.1494, 'grad_norm': 0.23988106846809387, 'learning_rate': 7.619322750635327e-06, 'kl': 0.010766998282633722, 'entropy': 0.23382820188999176, 'clip_ratio/low_mean': 0.005328634520992637, 'clip_ratio/low_min': 0.005328634520992637, 'clip_ratio/high_mean': 0.0032958984375, 'clip_ratio/high_max': 0.0032958984375, 'clip_ratio/region_mean': 0.008624532958492637, 'step_time': 1.7444170110029518, 'epoch': 0.04}
{'loss': -0.0, 'grad_norm': 0.17429932951927185, 'learning_rate': 7.591962841552627e-06, 'num_tokens': 1047629.0, 'completions/mean_length': 1448.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2772499918937683, 'rewards/game_reward_func/std': 0.17337633669376373, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2772499918937683, 'reward_std': 0.17337635159492493, 'frac_reward_zero_std': 0.0, 'kl': 0.01003746734932065, 'entropy': 0.23320601880550385, 'clip_ratio/low_mean': 0.0038283475441858172, 'clip_ratio/low_min': 0.0038283475441858172, 'clip_ratio/high_mean': 0.003056278917938471, 'clip_ratio/high_max': 0.003056278917938471, 'clip_ratio/region_mean': 0.006884626462124288, 'step_time': 18.11217355199915, 'epoch': 0.04}
{'loss': 0.0075, 'grad_norm': 0.6890637278556824, 'learning_rate': 7.564496387029532e-06, 'kl': 0.03386564273387194, 'entropy': 0.2597682327032089, 'clip_ratio/low_mean': 0.004712084773927927, 'clip_ratio/low_min': 0.004712084773927927, 'clip_ratio/high_mean': 0.009765625, 'clip_ratio/high_max': 0.009765625, 'clip_ratio/region_mean': 0.014477709773927927, 'step_time': 1.8478225080034463, 'epoch': 0.04}
{'loss': -0.0549, 'grad_norm': 0.4053575396537781, 'learning_rate': 7.536924516104411e-06, 'num_tokens': 1061605.0, 'completions/mean_length': 1568.0, 'completions/min_length': 1088.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.35600000619888306, 'rewards/game_reward_func/std': 0.12455177307128906, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.35600000619888306, 'reward_std': 0.12455176562070847, 'frac_reward_zero_std': 0.0, 'kl': 0.01026323961559683, 'entropy': 0.2515435852110386, 'clip_ratio/low_mean': 0.00048828125, 'clip_ratio/low_min': 0.00048828125, 'clip_ratio/high_mean': 0.008537741261534393, 'clip_ratio/high_max': 0.008537741261534393, 'clip_ratio/region_mean': 0.009026022511534393, 'step_time': 18.32019732099434, 'epoch': 0.04}
{'loss': 0.0574, 'grad_norm': 0.16655004024505615, 'learning_rate': 7.509248362148889e-06, 'kl': 0.00814858905505389, 'entropy': 0.24083318561315536, 'clip_ratio/low_mean': 0.007021405501291156, 'clip_ratio/low_min': 0.007021405501291156, 'clip_ratio/high_mean': 0.0017903646221384406, 'clip_ratio/high_max': 0.0017903646221384406, 'clip_ratio/region_mean': 0.008811770123429596, 'step_time': 1.9463874840002973, 'epoch': 0.04}
{'loss': -0.1315, 'grad_norm': 0.7454023957252502, 'learning_rate': 7.481469062821252e-06, 'num_tokens': 1073149.0, 'completions/mean_length': 1264.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37400001287460327, 'rewards/game_reward_func/std': 0.16858741641044617, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37400001287460327, 'reward_std': 0.16858741641044617, 'frac_reward_zero_std': 0.0, 'kl': 0.020015535526908934, 'entropy': 0.2694781422615051, 'clip_ratio/low_mean': 0.009068080340512097, 'clip_ratio/low_min': 0.009068080340512097, 'clip_ratio/high_mean': 0.003484552609734237, 'clip_ratio/high_max': 0.003484552609734237, 'clip_ratio/region_mean': 0.012552632950246334, 'step_time': 17.7431104089992, 'epoch': 0.04}
{'loss': 0.1367, 'grad_norm': 0.1514463573694229, 'learning_rate': 7.453587760019691e-06, 'kl': 0.012305698823183775, 'entropy': 0.22731315717101097, 'clip_ratio/low_mean': 0.0053906249813735485, 'clip_ratio/low_min': 0.0053906249813735485, 'clip_ratio/high_mean': 0.005365953897126019, 'clip_ratio/high_max': 0.005365953897126019, 'clip_ratio/region_mean': 0.010756578878499568, 'step_time': 1.945360460995289, 'epoch': 0.04}
{'loss': 0.333, 'grad_norm': 0.6387293934822083, 'learning_rate': 7.42560559983536e-06, 'num_tokens': 1087381.0, 'completions/mean_length': 1600.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.26749998331069946, 'rewards/game_reward_func/std': 0.26752355694770813, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.26749998331069946, 'reward_std': 0.26752355694770813, 'frac_reward_zero_std': 0.0, 'kl': 0.00905159895773977, 'entropy': 0.25141823291778564, 'clip_ratio/low_mean': 0.005229037255048752, 'clip_ratio/low_min': 0.005229037255048752, 'clip_ratio/high_mean': 0.00244140625, 'clip_ratio/high_max': 0.00244140625, 'clip_ratio/region_mean': 0.007670443505048752, 'step_time': 18.153919037009473, 'epoch': 0.04}

 36%|███▌      | 178/500 [29:29<46:33,  8.68s/it][2026-03-26 12:39:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:39:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.9

 36%|███▌      | 179/500 [29:47<1:01:44, 11.54s/it]
                                                   

 36%|███▌      | 179/500 [29:47<1:01:44, 11.54s/it]
 36%|███▌      | 180/500 [29:49<46:43,  8.76s/it]  
                                                 

 36%|███▌      | 180/500 [29:49<46:43,  8.76s/it][2026-03-26 12:39:51] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:40:05] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.8

 36%|███▌      | 181/500 [30:08<1:01:43, 11.61s/it]
                                                   

 36%|███▌      | 181/500 [30:08<1:01:43, 11.61s/it]
 36%|███▋      | 182/500 [30:10<46:16,  8.73s/it]  
                                                 

 36%|███▋      | 182/500 [30:10<46:16,  8.73s/it][2026-03-26 12:40:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:40:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.5

 37%|███▋      | 183/500 [30:28<1:01:00, 11.55s/it]
                                                   

 37%|███▋      | 183/500 [30:28<1:01:00, 11.55s/it]
 37%|███▋      | 184/500 [30:30<45:55,  8.72s/it]  
                                                 

 37%|███▋      | 184/500 [30:30<45:55,  8.72s/it][2026-03-26 12:40:32] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:40:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=14.6

 37%|███▋      | 185/500 [30:46<57:35, 10.97s/it]
                                                 

 37%|███▋      | 185/500 [30:46<57:35, 10.97s/it]
 37%|███▋      | 186/500 [30:48<43:10,  8.25s/it]
                                                 

 37%|███▋      | 186/500 [30:48<43:10,  8.25s/it][2026-03-26 12:40:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:41:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=22.6

 37%|███▋      | 187/500 [31:06<58:46, 11.27s/it]
                                                 

 37%|███▋      | 187/500 [31:06<58:46, 11.27s/it]
 38%|███▊      | 188/500 [31:08<44:09,  8.49s/it]
                                                 

 38%|███▊      | 188/500 [31:08<44:09,  8.49s/it][2026-03-26 12:41:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:41:24] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.8

 38%|███▊      | 189/500 [31:27<59:03, 11.39s/it]
                                                 
{'loss': -0.329, 'grad_norm': 0.29323580861091614, 'learning_rate': 7.39752373250527e-06, 'kl': 0.008358746767044067, 'entropy': 0.23673546314239502, 'clip_ratio/low_mean': 0.0035333806881681085, 'clip_ratio/low_min': 0.0035333806881681085, 'clip_ratio/high_mean': 0.0048828125, 'clip_ratio/high_max': 0.0048828125, 'clip_ratio/region_mean': 0.008416193188168108, 'step_time': 1.7425838810086134, 'epoch': 0.04}
{'loss': 0.4998, 'grad_norm': 0.5952917337417603, 'learning_rate': 7.369343312364994e-06, 'num_tokens': 1100013.0, 'completions/mean_length': 1400.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.48649999499320984, 'rewards/game_reward_func/std': 0.15546336770057678, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.48649999499320984, 'reward_std': 0.15546336770057678, 'frac_reward_zero_std': 0.0, 'kl': 0.010142113897018135, 'entropy': 0.24634059518575668, 'clip_ratio/low_mean': 0.0053635211661458015, 'clip_ratio/low_min': 0.0053635211661458015, 'clip_ratio/high_mean': 0.001880787080153823, 'clip_ratio/high_max': 0.001880787080153823, 'clip_ratio/region_mean': 0.0072443082462996244, 'step_time': 18.050893530002213, 'epoch': 0.04}
{'loss': -0.4969, 'grad_norm': 0.7900440692901611, 'learning_rate': 7.34106549780123e-06, 'kl': 0.016452719923108816, 'entropy': 0.2811375744640827, 'clip_ratio/low_mean': 0.002864583395421505, 'clip_ratio/low_min': 0.002864583395421505, 'clip_ratio/high_mean': 0.009872704278677702, 'clip_ratio/high_max': 0.009872704278677702, 'clip_ratio/region_mean': 0.012737287674099207, 'step_time': 2.0960522850073176, 'epoch': 0.04}
{'loss': -0.0356, 'grad_norm': 0.2370855212211609, 'learning_rate': 7.312691451204178e-06, 'num_tokens': 1113605.0, 'completions/mean_length': 1520.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3034999966621399, 'rewards/game_reward_func/std': 0.22357293963432312, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3034999966621399, 'reward_std': 0.22357292473316193, 'frac_reward_zero_std': 0.0, 'kl': 0.01166569977067411, 'entropy': 0.25188159570097923, 'clip_ratio/low_mean': 0.0031951122218742967, 'clip_ratio/low_min': 0.0031951122218742967, 'clip_ratio/high_mean': 0.002314453129656613, 'clip_ratio/high_max': 0.002314453129656613, 'clip_ratio/region_mean': 0.0055095653515309095, 'step_time': 18.08891215600306, 'epoch': 0.05}
{'loss': 0.0402, 'grad_norm': 0.2547043561935425, 'learning_rate': 7.284222338919758e-06, 'kl': 0.010943132569082081, 'entropy': 0.26541613042354584, 'clip_ratio/low_mean': 0.0033854168141260743, 'clip_ratio/low_min': 0.0033854168141260743, 'clip_ratio/high_mean': 0.0142822265625, 'clip_ratio/high_max': 0.0142822265625, 'clip_ratio/region_mean': 0.017667643376626074, 'step_time': 1.8525253999905544, 'epoch': 0.05}
{'loss': -0.0622, 'grad_norm': 0.3374200165271759, 'learning_rate': 7.255659331201673e-06, 'num_tokens': 1126045.0, 'completions/mean_length': 1376.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.34549999237060547, 'rewards/game_reward_func/std': 0.1943920999765396, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.34549999237060547, 'reward_std': 0.19439208507537842, 'frac_reward_zero_std': 0.0, 'kl': 0.013737852335907519, 'entropy': 0.29726412519812584, 'clip_ratio/low_mean': 0.0035511363530531526, 'clip_ratio/low_min': 0.0035511363530531526, 'clip_ratio/high_mean': 0.005089393118396401, 'clip_ratio/high_max': 0.005089393118396401, 'clip_ratio/region_mean': 0.008640529471449554, 'step_time': 17.94003063400305, 'epoch': 0.05}
{'loss': 0.0653, 'grad_norm': 0.19634690880775452, 'learning_rate': 7.227003602163296e-06, 'kl': 0.008138354984112084, 'entropy': 0.2477806657552719, 'clip_ratio/low_mean': 0.002652037364896387, 'clip_ratio/low_min': 0.002652037364896387, 'clip_ratio/high_mean': 0.002490942017175257, 'clip_ratio/high_max': 0.002490942017175257, 'clip_ratio/region_mean': 0.005142979382071644, 'step_time': 1.9438043029877008, 'epoch': 0.05}
{'loss': -0.1727, 'grad_norm': 0.870709240436554, 'learning_rate': 7.198256329729412e-06, 'num_tokens': 1134965.0, 'completions/mean_length': 936.0, 'completions/min_length': 256.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4442499876022339, 'rewards/game_reward_func/std': 0.08754713088274002, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4442499876022339, 'reward_std': 0.08754712343215942, 'frac_reward_zero_std': 0.0, 'kl': 0.023665013257414103, 'entropy': 0.2565850354731083, 'clip_ratio/low_mean': 0.0006167762912809849, 'clip_ratio/low_min': 0.0006167762912809849, 'clip_ratio/high_mean': 0.015011161100119352, 'clip_ratio/high_max': 0.015011161100119352, 'clip_ratio/region_mean': 0.015627937391400337, 'step_time': 16.05293608300417, 'epoch': 0.05}
{'loss': 0.1819, 'grad_norm': 0.4685305655002594, 'learning_rate': 7.169418695587791e-06, 'kl': 0.018009875784628093, 'entropy': 0.2549636848270893, 'clip_ratio/low_mean': 0.00834334921091795, 'clip_ratio/low_min': 0.00834334921091795, 'clip_ratio/high_mean': 0.005859375, 'clip_ratio/high_max': 0.005859375, 'clip_ratio/region_mean': 0.01420272421091795, 'step_time': 1.7199227569944924, 'epoch': 0.05}
{'loss': -0.4925, 'grad_norm': 0.7916879057884216, 'learning_rate': 7.140491885140629e-06, 'num_tokens': 1147981.0, 'completions/mean_length': 1448.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2917500138282776, 'rewards/game_reward_func/std': 0.20514926314353943, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2917500138282776, 'reward_std': 0.20514924824237823, 'frac_reward_zero_std': 0.0, 'kl': 0.013305310509167612, 'entropy': 0.2166127972304821, 'clip_ratio/low_mean': 0.0014322916977107525, 'clip_ratio/low_min': 0.0014322916977107525, 'clip_ratio/high_mean': 0.01300920732319355, 'clip_ratio/high_max': 0.01300920732319355, 'clip_ratio/region_mean': 0.014441499020904303, 'step_time': 18.155060776996834, 'epoch': 0.05}
{'loss': 0.5435, 'grad_norm': 24.14201545715332, 'learning_rate': 7.1114770874558e-06, 'kl': 0.010860578855499625, 'entropy': 0.24741873517632484, 'clip_ratio/low_mean': 0.004821538022952154, 'clip_ratio/low_min': 0.004821538022952154, 'clip_ratio/high_mean': 0.0009765625, 'clip_ratio/high_max': 0.0009765625, 'clip_ratio/region_mean': 0.005798100522952154, 'step_time': 1.8391862339994987, 'epoch': 0.05}

 38%|███▊      | 189/500 [31:27<59:03, 11.39s/it]
 38%|███▊      | 190/500 [31:28<44:10,  8.55s/it]
                                                 

 38%|███▊      | 190/500 [31:28<44:10,  8.55s/it][2026-03-26 12:41:30] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:41:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.1

 38%|███▊      | 191/500 [31:47<58:59, 11.46s/it]
                                                 

 38%|███▊      | 191/500 [31:47<58:59, 11.46s/it]
 38%|███▊      | 192/500 [31:49<44:16,  8.62s/it]
                                                 

 38%|███▊      | 192/500 [31:49<44:16,  8.62s/it][2026-03-26 12:41:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:41:59] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=13.5

 39%|███▊      | 193/500 [32:00<48:33,  9.49s/it]
                                                 

 39%|███▊      | 193/500 [32:00<48:33,  9.49s/it]
 39%|███▉      | 194/500 [32:02<36:25,  7.14s/it]
                                                 

 39%|███▉      | 194/500 [32:02<36:25,  7.14s/it][2026-03-26 12:42:03] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:42:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=14.4

 39%|███▉      | 195/500 [32:14<44:35,  8.77s/it]
                                                 

 39%|███▉      | 195/500 [32:14<44:35,  8.77s/it]
 39%|███▉      | 196/500 [32:16<33:25,  6.60s/it]
                                                 

 39%|███▉      | 196/500 [32:16<33:25,  6.60s/it][2026-03-26 12:42:18] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:42:31] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=18.5

 39%|███▉      | 197/500 [32:34<50:32, 10.01s/it]
                                                 

 39%|███▉      | 197/500 [32:34<50:32, 10.01s/it]
 40%|███▉      | 198/500 [32:36<38:28,  7.64s/it]
                                                 

 40%|███▉      | 198/500 [32:36<38:28,  7.64s/it][2026-03-26 12:42:38] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:42:51] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=18.8

 40%|███▉      | 199/500 [32:54<54:05, 10.78s/it]
                                                 

 40%|███▉      | 199/500 [32:54<54:05, 10.78s/it]
 40%|████      | 200/500 [32:56<40:55,  8.18s/it]
                                                 
{'loss': 0.279, 'grad_norm': 0.4225047528743744, 'learning_rate': 7.082375495217996e-06, 'num_tokens': 1160037.0, 'completions/mean_length': 1328.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4114999771118164, 'rewards/game_reward_func/std': 0.17285749316215515, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4114999771118164, 'reward_std': 0.17285749316215515, 'frac_reward_zero_std': 0.0, 'kl': 0.01114012545440346, 'entropy': 0.24496406689286232, 'clip_ratio/low_mean': 0.005810424452647567, 'clip_ratio/low_min': 0.005810424452647567, 'clip_ratio/high_mean': 0.0024038462433964014, 'clip_ratio/high_max': 0.0024038462433964014, 'clip_ratio/region_mean': 0.008214270696043968, 'step_time': 17.968759364994185, 'epoch': 0.05}
{'loss': -0.2748, 'grad_norm': 0.4174695909023285, 'learning_rate': 7.053188304679691e-06, 'kl': 0.013615078758448362, 'entropy': 0.2699027583003044, 'clip_ratio/low_mean': 0.006474247667938471, 'clip_ratio/low_min': 0.006474247667938471, 'clip_ratio/high_mean': 0.004150390625, 'clip_ratio/high_max': 0.004150390625, 'clip_ratio/region_mean': 0.01062463829293847, 'step_time': 1.7416140659843222, 'epoch': 0.05}
{'loss': -0.2396, 'grad_norm': 0.2839125096797943, 'learning_rate': 7.023916715611969e-06, 'num_tokens': 1173309.0, 'completions/mean_length': 1480.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33399999141693115, 'rewards/game_reward_func/std': 0.2793236970901489, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33399999141693115, 'reward_std': 0.2793236970901489, 'frac_reward_zero_std': 0.0, 'kl': 0.00743022991809994, 'entropy': 0.26245446503162384, 'clip_ratio/low_mean': 0.007784816436469555, 'clip_ratio/low_min': 0.007784816436469555, 'clip_ratio/high_mean': 0.0020751953125, 'clip_ratio/high_max': 0.0020751953125, 'clip_ratio/region_mean': 0.009860011748969555, 'step_time': 18.05359436300205, 'epoch': 0.05}
{'loss': 0.2406, 'grad_norm': 0.46282699704170227, 'learning_rate': 6.994561931255209e-06, 'kl': 0.009219875559210777, 'entropy': 0.2589598186314106, 'clip_ratio/low_mean': 0.007923702709376812, 'clip_ratio/low_min': 0.007923702709376812, 'clip_ratio/high_mean': 0.0006103515625, 'clip_ratio/high_max': 0.0006103515625, 'clip_ratio/region_mean': 0.008534054271876812, 'step_time': 1.835897427998134, 'epoch': 0.05}
{'loss': 0.3108, 'grad_norm': 0.5637630820274353, 'learning_rate': 6.965125158269619e-06, 'num_tokens': 1181653.0, 'completions/mean_length': 864.0, 'completions/min_length': 256.0, 'completions/max_length': 1280.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3889999985694885, 'rewards/game_reward_func/std': 0.1904400885105133, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3889999985694885, 'reward_std': 0.1904401034116745, 'frac_reward_zero_std': 0.0, 'kl': 0.0157995386980474, 'entropy': 0.2640823423862457, 'clip_ratio/low_mean': 0.0077167588751763105, 'clip_ratio/low_min': 0.0077167588751763105, 'clip_ratio/high_mean': 0.0074218749068677425, 'clip_ratio/high_max': 0.0074218749068677425, 'clip_ratio/region_mean': 0.015138633782044053, 'step_time': 11.41529949800315, 'epoch': 0.05}
{'loss': -0.3096, 'grad_norm': 0.9787462949752808, 'learning_rate': 6.935607606685642e-06, 'kl': 0.018559006974101067, 'entropy': 0.27080951631069183, 'clip_ratio/low_mean': 0.0015190972480922937, 'clip_ratio/low_min': 0.0015190972480922937, 'clip_ratio/high_mean': 0.02131696417927742, 'clip_ratio/high_max': 0.02131696417927742, 'clip_ratio/region_mean': 0.022836061427369714, 'step_time': 1.5490651359868934, 'epoch': 0.05}
{'loss': 0.4518, 'grad_norm': 0.5931235551834106, 'learning_rate': 6.906010489854209e-06, 'num_tokens': 1190445.0, 'completions/mean_length': 920.0, 'completions/min_length': 256.0, 'completions/max_length': 1408.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4987500011920929, 'rewards/game_reward_func/std': 0.09066067636013031, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4987500011920929, 'reward_std': 0.09066066890954971, 'frac_reward_zero_std': 0.0, 'kl': 0.013020561542361975, 'entropy': 0.29459473490715027, 'clip_ratio/low_mean': 0.004730620887130499, 'clip_ratio/low_min': 0.004730620887130499, 'clip_ratio/high_mean': 0.0027901786379516125, 'clip_ratio/high_max': 0.0027901786379516125, 'clip_ratio/region_mean': 0.007520799525082111, 'step_time': 12.467804281994177, 'epoch': 0.05}
{'loss': -0.4435, 'grad_norm': 0.7475272417068481, 'learning_rate': 6.876335024396872e-06, 'kl': 0.013369933469220996, 'entropy': 0.274789534509182, 'clip_ratio/low_mean': 0.0035978618543595076, 'clip_ratio/low_min': 0.0035978618543595076, 'clip_ratio/high_mean': 0.005063100950792432, 'clip_ratio/high_max': 0.005063100950792432, 'clip_ratio/region_mean': 0.00866096280515194, 'step_time': 1.390628591994755, 'epoch': 0.05}
{'loss': -0.5927, 'grad_norm': 0.6586681008338928, 'learning_rate': 6.846582430155783e-06, 'num_tokens': 1201349.0, 'completions/mean_length': 1184.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.34700000286102295, 'rewards/game_reward_func/std': 0.20381224155426025, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.34700000286102295, 'reward_std': 0.20381224155426025, 'frac_reward_zero_std': 0.0, 'kl': 0.008334161015227437, 'entropy': 0.23959610611200333, 'clip_ratio/low_mean': 0.0029296875, 'clip_ratio/low_min': 0.0029296875, 'clip_ratio/high_mean': 0.006175853544846177, 'clip_ratio/high_max': 0.006175853544846177, 'clip_ratio/region_mean': 0.009105541044846177, 'step_time': 17.79823397799919, 'epoch': 0.05}
{'loss': 0.6004, 'grad_norm': 1.2887471914291382, 'learning_rate': 6.816753930143558e-06, 'kl': 0.014046989497728646, 'entropy': 0.2887718677520752, 'clip_ratio/low_mean': 0.014489850495010614, 'clip_ratio/low_min': 0.014489850495010614, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.014489850495010614, 'step_time': 1.938543356001901, 'epoch': 0.05}
{'loss': -0.5293, 'grad_norm': 0.6048192381858826, 'learning_rate': 6.786850750493006e-06, 'num_tokens': 1212381.0, 'completions/mean_length': 1200.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4009999930858612, 'rewards/game_reward_func/std': 0.17906104028224945, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4009999930858612, 'reward_std': 0.17906104028224945, 'frac_reward_zero_std': 0.0, 'kl': 0.008305221679620445, 'entropy': 0.23132331296801567, 'clip_ratio/low_mean': 0.009638410061597824, 'clip_ratio/low_min': 0.009638410061597824, 'clip_ratio/high_mean': 0.0037434896221384406, 'clip_ratio/high_max': 0.0037434896221384406, 'clip_ratio/region_mean': 0.013381899683736265, 'step_time': 17.934640273997502, 'epoch': 0.05}

 40%|████      | 200/500 [32:56<40:55,  8.18s/it][2026-03-26 12:42:58] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:43:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=27.4

 40%|████      | 201/500 [33:16<57:40, 11.57s/it]
                                                 

 40%|████      | 201/500 [33:16<57:40, 11.57s/it]
 40%|████      | 202/500 [33:18<43:15,  8.71s/it]
                                                 

 40%|████      | 202/500 [33:18<43:15,  8.71s/it][2026-03-26 12:43:19] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:43:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.2

 41%|████      | 203/500 [33:36<57:01, 11.52s/it]
                                                 

 41%|████      | 203/500 [33:36<57:01, 11.52s/it]
 41%|████      | 204/500 [33:38<42:55,  8.70s/it]
                                                 

 41%|████      | 204/500 [33:38<42:55,  8.70s/it][2026-03-26 12:43:40] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:43:54] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.8

 41%|████      | 205/500 [33:56<57:02, 11.60s/it]
                                                 

 41%|████      | 205/500 [33:56<57:02, 11.60s/it]
 41%|████      | 206/500 [33:58<42:45,  8.73s/it]
                                                 

 41%|████      | 206/500 [33:58<42:45,  8.73s/it][2026-03-26 12:44:00] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:44:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=16.2

 41%|████▏     | 207/500 [34:16<55:49, 11.43s/it]
                                                 

 41%|████▏     | 207/500 [34:16<55:49, 11.43s/it]
 42%|████▏     | 208/500 [34:19<42:29,  8.73s/it]
                                                 

 42%|████▏     | 208/500 [34:19<42:29,  8.73s/it][2026-03-26 12:44:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:44:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=15.1

 42%|████▏     | 209/500 [34:36<55:21, 11.41s/it]
                                                 

 42%|████▏     | 209/500 [34:36<55:21, 11.41s/it]
 42%|████▏     | 210/500 [34:38<41:32,  8.59s/it]
                                                 

 42%|████▏     | 210/500 [34:38<41:32,  8.59s/it][2026-03-26 12:44:40] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:44:54] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.5

 42%|████▏     | 211/500 [34:57<55:23, 11.50s/it]
                                                 
{'loss': 0.5314, 'grad_norm': 0.8553312420845032, 'learning_rate': 6.7568741204067145e-06, 'kl': 0.013861644780263305, 'entropy': 0.29508380964398384, 'clip_ratio/low_mean': 0.015564966015517712, 'clip_ratio/low_min': 0.015564966015517712, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.015564966015517712, 'step_time': 1.9380701699992642, 'epoch': 0.05}
{'loss': -0.2277, 'grad_norm': 0.3055247664451599, 'learning_rate': 6.726825272106539e-06, 'num_tokens': 1227829.0, 'completions/mean_length': 1752.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3725000023841858, 'rewards/game_reward_func/std': 0.19275669753551483, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3725000023841858, 'reward_std': 0.19275669753551483, 'frac_reward_zero_std': 0.0, 'kl': 0.006506737205199897, 'entropy': 0.24550427496433258, 'clip_ratio/low_mean': 0.0016526442486792803, 'clip_ratio/low_min': 0.0016526442486792803, 'clip_ratio/high_mean': 0.00790204736404121, 'clip_ratio/high_max': 0.00790204736404121, 'clip_ratio/region_mean': 0.00955469161272049, 'step_time': 18.903245559995412, 'epoch': 0.05}
{'loss': 0.2286, 'grad_norm': 0.553497850894928, 'learning_rate': 6.696705440782939e-06, 'kl': 0.0072099840035662055, 'entropy': 0.27081198990345, 'clip_ratio/low_mean': 0.008169450622517616, 'clip_ratio/low_min': 0.008169450622517616, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.009878434997517616, 'step_time': 1.8395173550088657, 'epoch': 0.05}
{'loss': -0.2805, 'grad_norm': 0.252554714679718, 'learning_rate': 6.66651586454421e-06, 'num_tokens': 1239629.0, 'completions/mean_length': 1296.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3465000092983246, 'rewards/game_reward_func/std': 0.2262861728668213, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3465000092983246, 'reward_std': 0.2262861728668213, 'frac_reward_zero_std': 0.0, 'kl': 0.008505140664055943, 'entropy': 0.26714538037776947, 'clip_ratio/low_mean': 0.0034505209187045693, 'clip_ratio/low_min': 0.0034505209187045693, 'clip_ratio/high_mean': 0.00341796875, 'clip_ratio/high_max': 0.00341796875, 'clip_ratio/region_mean': 0.006868489668704569, 'step_time': 17.91484584200225, 'epoch': 0.05}
{'loss': 0.284, 'grad_norm': 0.3805002272129059, 'learning_rate': 6.636257784365585e-06, 'kl': 0.009978676913306117, 'entropy': 0.2823578454554081, 'clip_ratio/low_mean': 0.009211260708980262, 'clip_ratio/low_min': 0.009211260708980262, 'clip_ratio/high_mean': 0.005580357275903225, 'clip_ratio/high_max': 0.005580357275903225, 'clip_ratio/region_mean': 0.014791617984883487, 'step_time': 1.9355351329941186, 'epoch': 0.05}
{'loss': 0.0519, 'grad_norm': 0.3224562704563141, 'learning_rate': 6.605932444038229e-06, 'num_tokens': 1252197.0, 'completions/mean_length': 1392.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3295000195503235, 'rewards/game_reward_func/std': 0.2163192182779312, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3295000195503235, 'reward_std': 0.21631920337677002, 'frac_reward_zero_std': 0.0, 'kl': 0.009730355581268668, 'entropy': 0.246091827750206, 'clip_ratio/low_mean': 0.010788928717374802, 'clip_ratio/low_min': 0.010788928717374802, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.012253772467374802, 'step_time': 18.17992235800193, 'epoch': 0.05}
{'loss': -0.0492, 'grad_norm': 0.2636805474758148, 'learning_rate': 6.575541090118105e-06, 'kl': 0.007734466111287475, 'entropy': 0.2651773691177368, 'clip_ratio/low_mean': 0.006698119803331792, 'clip_ratio/low_min': 0.006698119803331792, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.008407104178331792, 'step_time': 1.8379793119966052, 'epoch': 0.05}
{'loss': -0.6203, 'grad_norm': 0.6015227437019348, 'learning_rate': 6.545084971874738e-06, 'num_tokens': 1261949.0, 'completions/mean_length': 1040.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3920000195503235, 'rewards/game_reward_func/std': 0.12468475103378296, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3920000195503235, 'reward_std': 0.12468474358320236, 'frac_reward_zero_std': 0.0, 'kl': 0.01023938984144479, 'entropy': 0.2642246372997761, 'clip_ratio/low_mean': 0.0042317709885537624, 'clip_ratio/low_min': 0.0042317709885537624, 'clip_ratio/high_mean': 0.010329629061743617, 'clip_ratio/high_max': 0.010329629061743617, 'clip_ratio/region_mean': 0.01456140005029738, 'step_time': 17.55969851199916, 'epoch': 0.05}
{'loss': 0.6239, 'grad_norm': 1.264682412147522, 'learning_rate': 6.514565341239861e-06, 'kl': 0.01889632362872362, 'entropy': 0.2821279987692833, 'clip_ratio/low_mean': 0.011735025967936963, 'clip_ratio/low_min': 0.011735025967936963, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.011735025967936963, 'step_time': 2.251932130995556, 'epoch': 0.05}
{'loss': -0.0639, 'grad_norm': 0.41820448637008667, 'learning_rate': 6.483983452755953e-06, 'num_tokens': 1271125.0, 'completions/mean_length': 968.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3567500114440918, 'rewards/game_reward_func/std': 0.10017663240432739, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3567500114440918, 'reward_std': 0.1001766249537468, 'frac_reward_zero_std': 0.0, 'kl': 0.015159380156546831, 'entropy': 0.27040310949087143, 'clip_ratio/low_mean': 0.010788690764456987, 'clip_ratio/low_min': 0.010788690764456987, 'clip_ratio/high_mean': 0.004524739691987634, 'clip_ratio/high_max': 0.004524739691987634, 'clip_ratio/region_mean': 0.015313430456444621, 'step_time': 17.475808082992444, 'epoch': 0.05}
{'loss': 0.072, 'grad_norm': 0.2873750925064087, 'learning_rate': 6.4533405635246696e-06, 'kl': 0.010642370907589793, 'entropy': 0.2588064782321453, 'clip_ratio/low_mean': 0.004371279734186828, 'clip_ratio/low_min': 0.004371279734186828, 'clip_ratio/high_mean': 0.006762695498764515, 'clip_ratio/high_max': 0.006762695498764515, 'clip_ratio/region_mean': 0.011133975232951343, 'step_time': 1.8369934469956206, 'epoch': 0.05}

 42%|████▏     | 211/500 [34:57<55:23, 11.50s/it]
 42%|████▏     | 212/500 [34:58<41:15,  8.59s/it]
                                                 

 42%|████▏     | 212/500 [34:58<41:15,  8.59s/it][2026-03-26 12:45:00] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:45:14] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.1

 43%|████▎     | 213/500 [35:17<55:17, 11.56s/it]
                                                 

 43%|████▎     | 213/500 [35:17<55:17, 11.56s/it]
 43%|████▎     | 214/500 [35:19<41:27,  8.70s/it]
                                                 

 43%|████▎     | 214/500 [35:19<41:27,  8.70s/it][2026-03-26 12:45:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:45:34] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:6 D:1 (12% win) | avg_turns=20.1

 43%|████▎     | 215/500 [35:37<55:24, 11.67s/it]
                                                 

 43%|████▎     | 215/500 [35:37<55:24, 11.67s/it]
 43%|████▎     | 216/500 [35:39<41:31,  8.77s/it]
                                                 

 43%|████▎     | 216/500 [35:39<41:31,  8.77s/it][2026-03-26 12:45:41] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:45:55] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.0

 43%|████▎     | 217/500 [35:58<54:36, 11.58s/it]
                                                 

 43%|████▎     | 217/500 [35:58<54:36, 11.58s/it]
 44%|████▎     | 218/500 [36:00<40:57,  8.71s/it]
                                                 

 44%|████▎     | 218/500 [36:00<40:57,  8.71s/it][2026-03-26 12:46:01] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:46:16] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=23.8

 44%|████▍     | 219/500 [36:18<54:27, 11.63s/it]
                                                 

 44%|████▍     | 219/500 [36:18<54:27, 11.63s/it]
 44%|████▍     | 220/500 [36:20<40:49,  8.75s/it]
                                                 

 44%|████▍     | 220/500 [36:20<40:49,  8.75s/it][2026-03-26 12:46:22] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:46:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=25.4

 44%|████▍     | 221/500 [36:39<54:33, 11.73s/it]
                                                 

 44%|████▍     | 221/500 [36:39<54:33, 11.73s/it]
 44%|████▍     | 222/500 [36:41<41:03,  8.86s/it]
                                                 
{'loss': 0.3961, 'grad_norm': 0.4191979467868805, 'learning_rate': 6.4226379331551625e-06, 'num_tokens': 1283053.0, 'completions/mean_length': 1312.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3465000092983246, 'rewards/game_reward_func/std': 0.18203531205654144, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3465000092983246, 'reward_std': 0.18203531205654144, 'frac_reward_zero_std': 0.0, 'kl': 0.013940507778897882, 'entropy': 0.2722763940691948, 'clip_ratio/low_mean': 0.013605057552922517, 'clip_ratio/low_min': 0.013605057552922517, 'clip_ratio/high_mean': 0.0078125, 'clip_ratio/high_max': 0.0078125, 'clip_ratio/region_mean': 0.021417557552922517, 'step_time': 18.09675444799359, 'epoch': 0.05}
{'loss': -0.3928, 'grad_norm': 0.3949280083179474, 'learning_rate': 6.3918768237123175e-06, 'kl': 0.00967147364281118, 'entropy': 0.26150748133659363, 'clip_ratio/low_mean': 0.002874201047234237, 'clip_ratio/low_min': 0.002874201047234237, 'clip_ratio/high_mean': 0.0020751953125, 'clip_ratio/high_max': 0.0020751953125, 'clip_ratio/region_mean': 0.004949396359734237, 'step_time': 1.6533192730130395, 'epoch': 0.05}
{'loss': 0.3116, 'grad_norm': 0.3324141800403595, 'learning_rate': 6.361058499664856e-06, 'num_tokens': 1295813.0, 'completions/mean_length': 1416.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40450000762939453, 'rewards/game_reward_func/std': 0.14698785543441772, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40450000762939453, 'reward_std': 0.14698785543441772, 'frac_reward_zero_std': 0.0, 'kl': 0.01196898939087987, 'entropy': 0.2504596374928951, 'clip_ratio/low_mean': 0.008394883363507688, 'clip_ratio/low_min': 0.008394883363507688, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.008394883363507688, 'step_time': 18.28659947799315, 'epoch': 0.05}
{'loss': -0.3069, 'grad_norm': 0.6142439246177673, 'learning_rate': 6.330184227833376e-06, 'kl': 0.013713051099330187, 'entropy': 0.25049013644456863, 'clip_ratio/low_mean': 0.003700657980516553, 'clip_ratio/low_min': 0.003700657980516553, 'clip_ratio/high_mean': 0.00638895807787776, 'clip_ratio/high_max': 0.00638895807787776, 'clip_ratio/region_mean': 0.010089616058394313, 'step_time': 1.8351749270077562, 'epoch': 0.05}
{'loss': 0.4577, 'grad_norm': 0.6973555088043213, 'learning_rate': 6.299255277338265e-06, 'num_tokens': 1307549.0, 'completions/mean_length': 1288.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.27674999833106995, 'rewards/game_reward_func/std': 0.4521516263484955, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.27674999833106995, 'reward_std': 0.4521515965461731, 'frac_reward_zero_std': 0.0, 'kl': 0.015530714299529791, 'entropy': 0.28302566707134247, 'clip_ratio/low_mean': 0.013270518509671092, 'clip_ratio/low_min': 0.013270518509671092, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.013270518509671092, 'step_time': 18.398509445985837, 'epoch': 0.05}
{'loss': -0.4527, 'grad_norm': 0.525338888168335, 'learning_rate': 6.268272919547537e-06, 'kl': 0.010823466582223773, 'entropy': 0.23336509615182877, 'clip_ratio/low_mean': 0.0035425646346993744, 'clip_ratio/low_min': 0.0035425646346993744, 'clip_ratio/high_mean': 0.003452845965512097, 'clip_ratio/high_max': 0.003452845965512097, 'clip_ratio/region_mean': 0.006995410600211471, 'step_time': 1.84977948100277, 'epoch': 0.05}
{'loss': -0.0251, 'grad_norm': 0.21777763962745667, 'learning_rate': 6.237238428024573e-06, 'num_tokens': 1318709.0, 'completions/mean_length': 1216.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4610000252723694, 'rewards/game_reward_func/std': 0.2261301875114441, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4610000252723694, 'reward_std': 0.2261301726102829, 'frac_reward_zero_std': 0.0, 'kl': 0.01845973450690508, 'entropy': 0.23271263763308525, 'clip_ratio/low_mean': 0.006727647501975298, 'clip_ratio/low_min': 0.006727647501975298, 'clip_ratio/high_mean': 0.0018310546875, 'clip_ratio/high_max': 0.0018310546875, 'clip_ratio/region_mean': 0.008558702189475298, 'step_time': 17.937958021000668, 'epoch': 0.05}
{'loss': 0.0305, 'grad_norm': 0.4057733714580536, 'learning_rate': 6.2061530784757625e-06, 'kl': 0.0258250767365098, 'entropy': 0.27720657363533974, 'clip_ratio/low_mean': 0.017161458730697632, 'clip_ratio/low_min': 0.017161458730697632, 'clip_ratio/high_mean': 0.00048828125, 'clip_ratio/high_max': 0.00048828125, 'clip_ratio/region_mean': 0.017649739980697632, 'step_time': 1.8549263030072325, 'epoch': 0.05}
{'loss': -0.2398, 'grad_norm': 0.33841878175735474, 'learning_rate': 6.175018148698077e-06, 'num_tokens': 1332301.0, 'completions/mean_length': 1520.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3100000023841858, 'rewards/game_reward_func/std': 0.19436930119991302, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3100000023841858, 'reward_std': 0.19436930119991302, 'frac_reward_zero_std': 0.0, 'kl': 0.013670429470948875, 'entropy': 0.23932534456253052, 'clip_ratio/low_mean': 0.007309422362595797, 'clip_ratio/low_min': 0.007309422362595797, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.011215672362595797, 'step_time': 18.262068319003447, 'epoch': 0.05}
{'loss': 0.2434, 'grad_norm': 0.3990619480609894, 'learning_rate': 6.143834918526528e-06, 'kl': 0.013204775983467698, 'entropy': 0.2466520369052887, 'clip_ratio/low_mean': 0.005336520494893193, 'clip_ratio/low_min': 0.005336520494893193, 'clip_ratio/high_mean': 0.0030517578125, 'clip_ratio/high_max': 0.0030517578125, 'clip_ratio/region_mean': 0.008388278307393193, 'step_time': 1.844983537994267, 'epoch': 0.06}
{'loss': 0.5199, 'grad_norm': 0.6013551354408264, 'learning_rate': 6.112604669781572e-06, 'num_tokens': 1346725.0, 'completions/mean_length': 1624.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2927500009536743, 'rewards/game_reward_func/std': 0.24473412334918976, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2927500009536743, 'reward_std': 0.24473412334918976, 'frac_reward_zero_std': 0.0, 'kl': 0.01013857510406524, 'entropy': 0.2319561243057251, 'clip_ratio/low_mean': 0.005239784601144493, 'clip_ratio/low_min': 0.005239784601144493, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.005239784601144493, 'step_time': 18.513036515993008, 'epoch': 0.06}

 44%|████▍     | 222/500 [36:41<41:03,  8.86s/it][2026-03-26 12:46:43] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:46:56] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 45%|████▍     | 223/500 [36:59<53:53, 11.67s/it]
                                                 

 45%|████▍     | 223/500 [36:59<53:53, 11.67s/it]
 45%|████▍     | 224/500 [37:01<40:31,  8.81s/it]
                                                 

 45%|████▍     | 224/500 [37:01<40:31,  8.81s/it][2026-03-26 12:47:03] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:47:17] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=22.8

 45%|████▌     | 225/500 [37:20<53:32, 11.68s/it]
                                                 

 45%|████▌     | 225/500 [37:20<53:32, 11.68s/it]
 45%|████▌     | 226/500 [37:22<40:06,  8.78s/it]
                                                 

 45%|████▌     | 226/500 [37:22<40:06,  8.78s/it][2026-03-26 12:47:23] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:47:37] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.8

 45%|████▌     | 227/500 [37:40<52:48, 11.61s/it]
                                                 

 45%|████▌     | 227/500 [37:40<52:48, 11.61s/it]
 46%|████▌     | 228/500 [37:42<39:34,  8.73s/it]
                                                 

 46%|████▌     | 228/500 [37:42<39:34,  8.73s/it][2026-03-26 12:47:44] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:47:58] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:4 D:4 (0% win) | avg_turns=25.4

 46%|████▌     | 229/500 [38:01<53:01, 11.74s/it]
                                                 

 46%|████▌     | 229/500 [38:01<53:01, 11.74s/it]
 46%|████▌     | 230/500 [38:03<39:33,  8.79s/it]
                                                 

 46%|████▌     | 230/500 [38:03<39:33,  8.79s/it][2026-03-26 12:48:04] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:48:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.4

 46%|████▌     | 231/500 [38:21<52:15, 11.66s/it]
                                                 

 46%|████▌     | 231/500 [38:21<52:15, 11.66s/it]
 46%|████▋     | 232/500 [38:23<39:00,  8.73s/it]
                                                 

 46%|████▋     | 232/500 [38:23<39:00,  8.73s/it][2026-03-26 12:48:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:48:39] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:4 D:4 (0% win) | avg_turns=26.9

 47%|████▋     | 233/500 [38:41<51:52, 11.66s/it]
                                                 
{'loss': -0.5142, 'grad_norm': 0.703680694103241, 'learning_rate': 6.0813286862164175e-06, 'kl': 0.016784558538347483, 'entropy': 0.23108558729290962, 'clip_ratio/low_mean': 0.005661690374836326, 'clip_ratio/low_min': 0.005661690374836326, 'clip_ratio/high_mean': 0.0028076171875, 'clip_ratio/high_max': 0.0028076171875, 'clip_ratio/region_mean': 0.008469307562336326, 'step_time': 1.9885316400032025, 'epoch': 0.06}
{'loss': -0.329, 'grad_norm': 0.3983440399169922, 'learning_rate': 6.050008253464247e-06, 'num_tokens': 1358717.0, 'completions/mean_length': 1320.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40849998593330383, 'rewards/game_reward_func/std': 0.18792931735515594, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40849998593330383, 'reward_std': 0.18792931735515594, 'frac_reward_zero_std': 0.0, 'kl': 0.012540544848889112, 'entropy': 0.23493219166994095, 'clip_ratio/low_mean': 0.00410348002333194, 'clip_ratio/low_min': 0.00410348002333194, 'clip_ratio/high_mean': 0.005696614505723119, 'clip_ratio/high_max': 0.005696614505723119, 'clip_ratio/region_mean': 0.009800094529055059, 'step_time': 18.055555033999553, 'epoch': 0.06}
{'loss': 0.3324, 'grad_norm': 0.45625635981559753, 'learning_rate': 6.018644658985378e-06, 'kl': 0.023395690135657787, 'entropy': 0.27820154279470444, 'clip_ratio/low_mean': 0.008370535681024194, 'clip_ratio/low_min': 0.008370535681024194, 'clip_ratio/high_mean': 0.0021875000093132257, 'clip_ratio/high_max': 0.0021875000093132257, 'clip_ratio/region_mean': 0.01055803569033742, 'step_time': 1.9302449009992415, 'epoch': 0.06}
{'loss': -0.0903, 'grad_norm': 0.21289511024951935, 'learning_rate': 5.987239192014336e-06, 'num_tokens': 1371797.0, 'completions/mean_length': 1456.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3295000195503235, 'rewards/game_reward_func/std': 0.1538413017988205, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3295000195503235, 'reward_std': 0.1538412868976593, 'frac_reward_zero_std': 0.0, 'kl': 0.015329011715948582, 'entropy': 0.2426268346607685, 'clip_ratio/low_mean': 0.004603794659487903, 'clip_ratio/low_min': 0.004603794659487903, 'clip_ratio/high_mean': 0.005411783931776881, 'clip_ratio/high_max': 0.005411783931776881, 'clip_ratio/region_mean': 0.010015578591264784, 'step_time': 18.20456473300146, 'epoch': 0.06}
{'loss': 0.0964, 'grad_norm': 0.3102976083755493, 'learning_rate': 5.955793143506863e-06, 'kl': 0.014718587510287762, 'entropy': 0.26802463456988335, 'clip_ratio/low_mean': 0.006883223773911595, 'clip_ratio/low_min': 0.006883223773911595, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.008348067523911595, 'step_time': 1.837624229992798, 'epoch': 0.06}
{'loss': 0.2799, 'grad_norm': 0.2779141068458557, 'learning_rate': 5.9243078060868445e-06, 'num_tokens': 1383341.0, 'completions/mean_length': 1264.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.27399998903274536, 'rewards/game_reward_func/std': 0.18274417519569397, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.27399998903274536, 'reward_std': 0.18274417519569397, 'frac_reward_zero_std': 0.0, 'kl': 0.03253060788847506, 'entropy': 0.2628173865377903, 'clip_ratio/low_mean': 0.005523989908397198, 'clip_ratio/low_min': 0.005523989908397198, 'clip_ratio/high_mean': 0.0064561632461845875, 'clip_ratio/high_max': 0.0064561632461845875, 'clip_ratio/region_mean': 0.011980153154581785, 'step_time': 18.007081961011863, 'epoch': 0.06}
{'loss': -0.2735, 'grad_norm': 0.49212440848350525, 'learning_rate': 5.892784473993184e-06, 'kl': 0.019429692765697837, 'entropy': 0.26247867196798325, 'clip_ratio/low_mean': 0.007988741621375084, 'clip_ratio/low_min': 0.007988741621375084, 'clip_ratio/high_mean': 0.0018310546875, 'clip_ratio/high_max': 0.0018310546875, 'clip_ratio/region_mean': 0.009819796308875084, 'step_time': 1.839533300990297, 'epoch': 0.06}
{'loss': 0.2082, 'grad_norm': 0.39328107237815857, 'learning_rate': 5.861224443026595e-06, 'num_tokens': 1397765.0, 'completions/mean_length': 1624.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3034999966621399, 'rewards/game_reward_func/std': 0.1917818784713745, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3034999966621399, 'reward_std': 0.1917818784713745, 'frac_reward_zero_std': 0.0, 'kl': 0.013757947366684675, 'entropy': 0.24471933767199516, 'clip_ratio/low_mean': 0.002444556448608637, 'clip_ratio/low_min': 0.002444556448608637, 'clip_ratio/high_mean': 0.0032958984375, 'clip_ratio/high_max': 0.0032958984375, 'clip_ratio/region_mean': 0.005740454886108637, 'step_time': 18.587576380006794, 'epoch': 0.06}
{'loss': -0.2009, 'grad_norm': 0.2617088854312897, 'learning_rate': 5.82962901049634e-06, 'kl': 0.020070819184184074, 'entropy': 0.26721708476543427, 'clip_ratio/low_mean': 0.004889641655609012, 'clip_ratio/low_min': 0.004889641655609012, 'clip_ratio/high_mean': 0.003662109375, 'clip_ratio/high_max': 0.003662109375, 'clip_ratio/region_mean': 0.008551751030609012, 'step_time': 1.740702221999527, 'epoch': 0.06}
{'loss': 0.6724, 'grad_norm': 0.825606107711792, 'learning_rate': 5.797999475166897e-06, 'num_tokens': 1410141.0, 'completions/mean_length': 1368.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3227500021457672, 'rewards/game_reward_func/std': 0.19100916385650635, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3227500021457672, 'reward_std': 0.19100916385650635, 'frac_reward_zero_std': 0.0, 'kl': 0.01979410694912076, 'entropy': 0.24218152090907097, 'clip_ratio/low_mean': 0.008502702228724957, 'clip_ratio/low_min': 0.008502702228724957, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.008502702228724957, 'step_time': 18.149521940998966, 'epoch': 0.06}
{'loss': -0.6643, 'grad_norm': 1.0645573139190674, 'learning_rate': 5.766337137204579e-06, 'kl': 0.03237156383693218, 'entropy': 0.2506673038005829, 'clip_ratio/low_mean': 0.00537109375, 'clip_ratio/low_min': 0.00537109375, 'clip_ratio/high_mean': 0.00398393115028739, 'clip_ratio/high_max': 0.00398393115028739, 'clip_ratio/region_mean': 0.00935502490028739, 'step_time': 1.7388444329990307, 'epoch': 0.06}

 47%|████▋     | 233/500 [38:41<51:52, 11.66s/it]
 47%|████▋     | 234/500 [38:43<38:51,  8.77s/it]
                                                 

 47%|████▋     | 234/500 [38:43<38:51,  8.77s/it][2026-03-26 12:48:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:48:56] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=16.4

 47%|████▋     | 235/500 [38:58<46:36, 10.55s/it]
                                                 

 47%|████▋     | 235/500 [38:58<46:36, 10.55s/it]
 47%|████▋     | 236/500 [39:00<35:02,  7.96s/it]
                                                 

 47%|████▋     | 236/500 [39:00<35:02,  7.96s/it][2026-03-26 12:49:02] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:49:16] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=25.1

 47%|████▋     | 237/500 [39:18<48:37, 11.09s/it]
                                                 

 47%|████▋     | 237/500 [39:18<48:37, 11.09s/it]
 48%|████▊     | 238/500 [39:20<36:24,  8.34s/it]
                                                 

 48%|████▊     | 238/500 [39:20<36:24,  8.34s/it][2026-03-26 12:49:22] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:49:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=22.8

 48%|████▊     | 239/500 [39:38<49:11, 11.31s/it]
                                                 

 48%|████▊     | 239/500 [39:38<49:11, 11.31s/it]
 48%|████▊     | 240/500 [39:41<37:05,  8.56s/it]
                                                 

 48%|████▊     | 240/500 [39:41<37:05,  8.56s/it][2026-03-26 12:49:42] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:49:53] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=16.5

 48%|████▊     | 241/500 [39:55<44:07, 10.22s/it]
                                                 

 48%|████▊     | 241/500 [39:55<44:07, 10.22s/it]
 48%|████▊     | 242/500 [39:56<32:49,  7.63s/it]
                                                 

 48%|████▊     | 242/500 [39:56<32:49,  7.63s/it][2026-03-26 12:49:58] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:50:10] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=14.5

 49%|████▊     | 243/500 [40:13<44:21, 10.35s/it]
                                                 

 49%|████▊     | 243/500 [40:13<44:21, 10.35s/it]
 49%|████▉     | 244/500 [40:15<33:21,  7.82s/it]
                                                 
{'loss': -0.5131, 'grad_norm': 0.6835972666740417, 'learning_rate': 5.734643298124091e-06, 'num_tokens': 1425333.0, 'completions/mean_length': 1720.0, 'completions/min_length': 1024.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2997499704360962, 'rewards/game_reward_func/std': 0.2774710953235626, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2997499704360962, 'reward_std': 0.2774710953235626, 'frac_reward_zero_std': 0.0, 'kl': 0.015875425888225436, 'entropy': 0.23331518843770027, 'clip_ratio/low_mean': 0.0025111606810241938, 'clip_ratio/low_min': 0.0025111606810241938, 'clip_ratio/high_mean': 0.00537109375, 'clip_ratio/high_max': 0.00537109375, 'clip_ratio/region_mean': 0.007882254431024194, 'step_time': 18.32226269599778, 'epoch': 0.06}
{'loss': 0.516, 'grad_norm': 0.8062074184417725, 'learning_rate': 5.702919260735015e-06, 'kl': 0.020901104900985956, 'entropy': 0.23934780806303024, 'clip_ratio/low_mean': 0.007725835312157869, 'clip_ratio/low_min': 0.007725835312157869, 'clip_ratio/high_mean': 0.002685546875, 'clip_ratio/high_max': 0.002685546875, 'clip_ratio/region_mean': 0.01041138218715787, 'step_time': 1.838790706002328, 'epoch': 0.06}
{'loss': -0.402, 'grad_norm': 0.9630575180053711, 'learning_rate': 5.671166329088278e-06, 'num_tokens': 1435149.0, 'completions/mean_length': 1048.0, 'completions/min_length': 512.0, 'completions/max_length': 1664.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44075000286102295, 'rewards/game_reward_func/std': 0.11713697761297226, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44075000286102295, 'reward_std': 0.11713698506355286, 'frac_reward_zero_std': 0.0, 'kl': 0.03065951121971011, 'entropy': 0.27280624955892563, 'clip_ratio/low_mean': 0.002527573611587286, 'clip_ratio/low_min': 0.002527573611587286, 'clip_ratio/high_mean': 0.00941941887140274, 'clip_ratio/high_max': 0.00941941887140274, 'clip_ratio/region_mean': 0.011946992482990026, 'step_time': 14.562416471002507, 'epoch': 0.06}
{'loss': 0.4107, 'grad_norm': 0.4977514147758484, 'learning_rate': 5.6393858084225305e-06, 'kl': 0.03179439529776573, 'entropy': 0.2719993218779564, 'clip_ratio/low_mean': 0.010564403142780066, 'clip_ratio/low_min': 0.010564403142780066, 'clip_ratio/high_mean': 0.0031250000465661287, 'clip_ratio/high_max': 0.0031250000465661287, 'clip_ratio/region_mean': 0.013689403189346194, 'step_time': 1.7747654850027175, 'epoch': 0.06}
{'loss': 0.1943, 'grad_norm': 0.3754398226737976, 'learning_rate': 5.6075790051105025e-06, 'num_tokens': 1449445.0, 'completions/mean_length': 1608.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.24724999070167542, 'rewards/game_reward_func/std': 0.1866668164730072, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.24724999070167542, 'reward_std': 0.1866668164730072, 'frac_reward_zero_std': 0.0, 'kl': 0.020410085562616587, 'entropy': 0.25063085556030273, 'clip_ratio/low_mean': 0.0049342341953888535, 'clip_ratio/low_min': 0.0049342341953888535, 'clip_ratio/high_mean': 0.0009765625, 'clip_ratio/high_max': 0.0009765625, 'clip_ratio/region_mean': 0.0059107966953888535, 'step_time': 18.22863069799496, 'epoch': 0.06}
{'loss': -0.1886, 'grad_norm': 0.2603602707386017, 'learning_rate': 5.575747226605298e-06, 'kl': 0.02099286811426282, 'entropy': 0.24772514775395393, 'clip_ratio/low_mean': 0.004637366393581033, 'clip_ratio/low_min': 0.004637366393581033, 'clip_ratio/high_mean': 0.0047607421875, 'clip_ratio/high_max': 0.0047607421875, 'clip_ratio/region_mean': 0.009398108581081033, 'step_time': 1.740224817993294, 'epoch': 0.06}
{'loss': -0.101, 'grad_norm': 0.2580835223197937, 'learning_rate': 5.543891781386655e-06, 'num_tokens': 1462525.0, 'completions/mean_length': 1456.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3642500042915344, 'rewards/game_reward_func/std': 0.19851860404014587, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3642500042915344, 'reward_std': 0.19851861894130707, 'frac_reward_zero_std': 0.0, 'kl': 0.01992031536065042, 'entropy': 0.21121863275766373, 'clip_ratio/low_mean': 0.004012872232124209, 'clip_ratio/low_min': 0.004012872232124209, 'clip_ratio/high_mean': 0.0040283203125, 'clip_ratio/high_max': 0.0040283203125, 'clip_ratio/region_mean': 0.00804119254462421, 'step_time': 18.081244332992355, 'epoch': 0.06}
{'loss': 0.108, 'grad_norm': 0.3886973261833191, 'learning_rate': 5.512013978907157e-06, 'kl': 0.030579879647120833, 'entropy': 0.24758990108966827, 'clip_ratio/low_mean': 0.005614224122837186, 'clip_ratio/low_min': 0.005614224122837186, 'clip_ratio/high_mean': 0.006603422574698925, 'clip_ratio/high_max': 0.006603422574698925, 'clip_ratio/region_mean': 0.012217646697536111, 'step_time': 1.9520304900070187, 'epoch': 0.06}
{'loss': 0.5301, 'grad_norm': 0.7449299693107605, 'learning_rate': 5.480115129538409e-06, 'num_tokens': 1472405.0, 'completions/mean_length': 1056.0, 'completions/min_length': 512.0, 'completions/max_length': 1600.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4829999804496765, 'rewards/game_reward_func/std': 0.12163646519184113, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4829999804496765, 'reward_std': 0.12163645774126053, 'frac_reward_zero_std': 0.0, 'kl': 0.02648977469652891, 'entropy': 0.2670651972293854, 'clip_ratio/low_mean': 0.005478050734382123, 'clip_ratio/low_min': 0.005478050734382123, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.005478050734382123, 'step_time': 13.984897773007106, 'epoch': 0.06}
{'loss': -0.5218, 'grad_norm': 0.6447721719741821, 'learning_rate': 5.448196544517168e-06, 'kl': 0.03270642692223191, 'entropy': 0.2408519685268402, 'clip_ratio/low_mean': 0.005287247593514621, 'clip_ratio/low_min': 0.005287247593514621, 'clip_ratio/high_mean': 0.006130514666438103, 'clip_ratio/high_max': 0.006130514666438103, 'clip_ratio/region_mean': 0.011417762259952724, 'step_time': 1.4539154099984444, 'epoch': 0.06}
{'loss': -0.5997, 'grad_norm': 2.7930314540863037, 'learning_rate': 5.4162595358914475e-06, 'num_tokens': 1481325.0, 'completions/mean_length': 936.0, 'completions/min_length': 64.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.343999981880188, 'rewards/game_reward_func/std': 0.17752262949943542, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.343999981880188, 'reward_std': 0.17752262949943542, 'frac_reward_zero_std': 0.0, 'kl': 0.041131940204650164, 'entropy': 0.36148417741060257, 'clip_ratio/low_mean': 0.0012428977061063051, 'clip_ratio/low_min': 0.0012428977061063051, 'clip_ratio/high_mean': 0.022306743543595076, 'clip_ratio/high_max': 0.022306743543595076, 'clip_ratio/region_mean': 0.02354964124970138, 'step_time': 16.533598347006773, 'epoch': 0.06}

 49%|████▉     | 244/500 [40:15<33:21,  7.82s/it][2026-03-26 12:50:17] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:50:30] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.4

 49%|████▉     | 245/500 [40:33<46:18, 10.90s/it]
                                                 

 49%|████▉     | 245/500 [40:33<46:18, 10.90s/it]
 49%|████▉     | 246/500 [40:35<35:00,  8.27s/it]
                                                 

 49%|████▉     | 246/500 [40:35<35:00,  8.27s/it][2026-03-26 12:50:37] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:50:51] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.8

 49%|████▉     | 247/500 [40:53<47:36, 11.29s/it]
                                                 

 49%|████▉     | 247/500 [40:53<47:36, 11.29s/it]
 50%|████▉     | 248/500 [40:56<35:52,  8.54s/it]
                                                 

 50%|████▉     | 248/500 [40:56<35:52,  8.54s/it][2026-03-26 12:50:57] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:51:11] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.8

 50%|████▉     | 249/500 [41:14<47:56, 11.46s/it]
                                                 

 50%|████▉     | 249/500 [41:14<47:56, 11.46s/it]
 50%|█████     | 250/500 [41:16<36:05,  8.66s/it]
                                                 

 50%|█████     | 250/500 [41:16<36:05,  8.66s/it][2026-03-26 12:51:18] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:51:32] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.5

 50%|█████     | 251/500 [41:35<49:02, 11.82s/it]
                                                 

 50%|█████     | 251/500 [41:35<49:02, 11.82s/it]
 50%|█████     | 252/500 [41:37<36:41,  8.88s/it]
                                                 

 50%|█████     | 252/500 [41:37<36:41,  8.88s/it][2026-03-26 12:51:39] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:51:53] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.0

 51%|█████     | 253/500 [41:56<48:15, 11.72s/it]
                                                 

 51%|█████     | 253/500 [41:56<48:15, 11.72s/it]
 51%|█████     | 254/500 [41:58<36:07,  8.81s/it]
                                                 

 51%|█████     | 254/500 [41:58<36:07,  8.81s/it][2026-03-26 12:51:59] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:52:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.4

 51%|█████     | 255/500 [42:16<47:42, 11.69s/it]
                                                 
{'loss': 0.6055, 'grad_norm': 0.7741627097129822, 'learning_rate': 5.384305416466584e-06, 'kl': 0.05229442357085645, 'entropy': 0.2790217809379101, 'clip_ratio/low_mean': 0.014713541662786156, 'clip_ratio/low_min': 0.014713541662786156, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.014713541662786156, 'step_time': 1.7321054240019294, 'epoch': 0.06}
{'loss': -0.1471, 'grad_norm': 0.7694888114929199, 'learning_rate': 5.35233549975127e-06, 'num_tokens': 1493701.0, 'completions/mean_length': 1368.0, 'completions/min_length': 192.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3947499990463257, 'rewards/game_reward_func/std': 0.14774471521377563, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3947499990463257, 'reward_std': 0.14774471521377563, 'frac_reward_zero_std': 0.0, 'kl': 0.035948242992162704, 'entropy': 0.21670282632112503, 'clip_ratio/low_mean': 0.0043612392619252205, 'clip_ratio/low_min': 0.0043612392619252205, 'clip_ratio/high_mean': 0.012858073227107525, 'clip_ratio/high_max': 0.012858073227107525, 'clip_ratio/region_mean': 0.017219312489032745, 'step_time': 17.916737331004697, 'epoch': 0.06}
{'loss': 0.1543, 'grad_norm': 0.178475484251976, 'learning_rate': 5.320351099903565e-06, 'kl': 0.018726404756307602, 'entropy': 0.2525177672505379, 'clip_ratio/low_mean': 0.0012865823518950492, 'clip_ratio/low_min': 0.0012865823518950492, 'clip_ratio/high_mean': 0.004464285797439516, 'clip_ratio/high_max': 0.004464285797439516, 'clip_ratio/region_mean': 0.005750868149334565, 'step_time': 1.9585235040067346, 'epoch': 0.06}
{'loss': -0.4261, 'grad_norm': 0.7530806660652161, 'learning_rate': 5.288353531676873e-06, 'num_tokens': 1506781.0, 'completions/mean_length': 1456.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37700000405311584, 'rewards/game_reward_func/std': 0.17473244667053223, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37700000405311584, 'reward_std': 0.17473246157169342, 'frac_reward_zero_std': 0.0, 'kl': 0.01646543387323618, 'entropy': 0.24503140151500702, 'clip_ratio/low_mean': 0.002812500111758709, 'clip_ratio/low_min': 0.002812500111758709, 'clip_ratio/high_mean': 0.007658585207536817, 'clip_ratio/high_max': 0.007658585207536817, 'clip_ratio/region_mean': 0.010471085319295526, 'step_time': 18.15297327600274, 'epoch': 0.06}
{'loss': 0.4296, 'grad_norm': 0.6305484771728516, 'learning_rate': 5.256344110365896e-06, 'kl': 0.017888212809339166, 'entropy': 0.25470296293497086, 'clip_ratio/low_mean': 0.004007523195468821, 'clip_ratio/low_min': 0.004007523195468821, 'clip_ratio/high_mean': 0.005580357275903225, 'clip_ratio/high_max': 0.005580357275903225, 'clip_ratio/region_mean': 0.009587880471372046, 'step_time': 1.9415261690010084, 'epoch': 0.06}
{'loss': -0.5753, 'grad_norm': 0.8840705752372742, 'learning_rate': 5.224324151752575e-06, 'num_tokens': 1518837.0, 'completions/mean_length': 1328.0, 'completions/min_length': 192.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4287499785423279, 'rewards/game_reward_func/std': 0.21832071244716644, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4287499785423279, 'reward_std': 0.21832069754600525, 'frac_reward_zero_std': 0.0, 'kl': 0.018400981789454818, 'entropy': 0.23667491227388382, 'clip_ratio/low_mean': 0.004403409198857844, 'clip_ratio/low_min': 0.004403409198857844, 'clip_ratio/high_mean': 0.005147999618202448, 'clip_ratio/high_max': 0.005147999618202448, 'clip_ratio/region_mean': 0.009551408817060292, 'step_time': 18.088076539999747, 'epoch': 0.06}
{'loss': 0.5838, 'grad_norm': 0.8449382781982422, 'learning_rate': 5.192294972051992e-06, 'kl': 0.030270833522081375, 'entropy': 0.2543853297829628, 'clip_ratio/low_mean': 0.011867841356433928, 'clip_ratio/low_min': 0.011867841356433928, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.011867841356433928, 'step_time': 1.9359135850027087, 'epoch': 0.06}
{'loss': 0.5406, 'grad_norm': 0.8369089961051941, 'learning_rate': 5.160257887858278e-06, 'num_tokens': 1531789.0, 'completions/mean_length': 1440.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.42250001430511475, 'rewards/game_reward_func/std': 0.1676245778799057, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.42250001430511475, 'reward_std': 0.1676245927810669, 'frac_reward_zero_std': 0.0, 'kl': 0.017186618177220225, 'entropy': 0.22060364484786987, 'clip_ratio/low_mean': 0.010642841923981905, 'clip_ratio/low_min': 0.010642841923981905, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.010642841923981905, 'step_time': 18.30540810700768, 'epoch': 0.06}
{'loss': -0.5357, 'grad_norm': 0.657913863658905, 'learning_rate': 5.128214216090478e-06, 'kl': 0.01735060173086822, 'entropy': 0.23576858639717102, 'clip_ratio/low_mean': 0.007155987434089184, 'clip_ratio/low_min': 0.007155987434089184, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.010085674934089184, 'step_time': 1.8355355149979005, 'epoch': 0.06}
{'loss': -0.5238, 'grad_norm': 0.7204838395118713, 'learning_rate': 5.0961652739384356e-06, 'num_tokens': 1544485.0, 'completions/mean_length': 1408.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4347499907016754, 'rewards/game_reward_func/std': 0.1423875391483307, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4347499907016754, 'reward_std': 0.1423875391483307, 'frac_reward_zero_std': 0.0, 'kl': 0.014192736940458417, 'entropy': 0.23348695412278175, 'clip_ratio/low_mean': 0.0027343749534338713, 'clip_ratio/low_min': 0.0027343749534338713, 'clip_ratio/high_mean': 0.006863064249046147, 'clip_ratio/high_max': 0.006863064249046147, 'clip_ratio/region_mean': 0.009597439202480018, 'step_time': 18.18956487700052, 'epoch': 0.06}
{'loss': 0.5269, 'grad_norm': 0.7422488927841187, 'learning_rate': 5.064112378808636e-06, 'kl': 0.020533819682896137, 'entropy': 0.2732246182858944, 'clip_ratio/low_mean': 0.01041342900134623, 'clip_ratio/low_min': 0.01041342900134623, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.01041342900134623, 'step_time': 1.8445250169970677, 'epoch': 0.06}

 51%|█████     | 255/500 [42:16<47:42, 11.69s/it]
 51%|█████     | 256/500 [42:18<35:52,  8.82s/it]
                                                 

 51%|█████     | 256/500 [42:18<35:52,  8.82s/it][2026-03-26 12:52:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:52:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=16.5

 51%|█████▏    | 257/500 [42:36<46:57, 11.59s/it]
                                                 

 51%|█████▏    | 257/500 [42:36<46:57, 11.59s/it]
 52%|█████▏    | 258/500 [42:38<35:10,  8.72s/it]
                                                 

 52%|█████▏    | 258/500 [42:38<35:10,  8.72s/it][2026-03-26 12:52:40] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:52:54] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.6

 52%|█████▏    | 259/500 [42:57<46:34, 11.60s/it]
                                                 

 52%|█████▏    | 259/500 [42:57<46:34, 11.60s/it]
 52%|█████▏    | 260/500 [42:59<34:53,  8.72s/it]
                                                 

 52%|█████▏    | 260/500 [42:59<34:53,  8.72s/it][2026-03-26 12:53:00] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:53:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.8

 52%|█████▏    | 261/500 [43:15<43:46, 10.99s/it]
                                                 

 52%|█████▏    | 261/500 [43:15<43:46, 10.99s/it]
 52%|█████▏    | 262/500 [43:17<32:45,  8.26s/it]
                                                 

 52%|█████▏    | 262/500 [43:17<32:45,  8.26s/it][2026-03-26 12:53:18] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:53:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.8

 53%|█████▎    | 263/500 [43:35<44:48, 11.34s/it]
                                                 

 53%|█████▎    | 263/500 [43:35<44:48, 11.34s/it]
 53%|█████▎    | 264/500 [43:38<34:06,  8.67s/it]
                                                 

 53%|█████▎    | 264/500 [43:38<34:06,  8.67s/it][2026-03-26 12:53:39] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:53:53] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.0

 53%|█████▎    | 265/500 [43:56<45:06, 11.52s/it]
                                                 

 53%|█████▎    | 265/500 [43:56<45:06, 11.52s/it]
 53%|█████▎    | 266/500 [43:58<33:40,  8.63s/it]
                                                 
{'loss': -0.1417, 'grad_norm': 0.610390305519104, 'learning_rate': 5.032056848270056e-06, 'num_tokens': 1557373.0, 'completions/mean_length': 1432.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3867499828338623, 'rewards/game_reward_func/std': 0.13594931364059448, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3867499828338623, 'reward_std': 0.1359492987394333, 'frac_reward_zero_std': 0.0, 'kl': 0.025966285495087504, 'entropy': 0.2616888955235481, 'clip_ratio/low_mean': 0.0046701389364898205, 'clip_ratio/low_min': 0.0046701389364898205, 'clip_ratio/high_mean': 0.008496093563735485, 'clip_ratio/high_max': 0.008496093563735485, 'clip_ratio/region_mean': 0.013166232500225306, 'step_time': 18.209013600004255, 'epoch': 0.06}
{'loss': 0.1474, 'grad_norm': 0.19266168773174286, 'learning_rate': 5e-06, 'kl': 0.0140003461856395, 'entropy': 0.223244559019804, 'clip_ratio/low_mean': 0.004647473921068013, 'clip_ratio/low_min': 0.004647473921068013, 'clip_ratio/high_mean': 0.0023437500931322575, 'clip_ratio/high_max': 0.0023437500931322575, 'clip_ratio/region_mean': 0.00699122401420027, 'step_time': 1.9503147909999825, 'epoch': 0.06}
{'loss': -0.2576, 'grad_norm': 0.23986101150512695, 'learning_rate': 4.967943151729945e-06, 'num_tokens': 1567253.0, 'completions/mean_length': 1056.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33100003004074097, 'rewards/game_reward_func/std': 0.25297996401786804, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33100003004074097, 'reward_std': 0.25297996401786804, 'frac_reward_zero_std': 0.0, 'kl': 0.02289751300122589, 'entropy': 0.24584965035319328, 'clip_ratio/low_mean': 0.011671401560306549, 'clip_ratio/low_min': 0.011671401560306549, 'clip_ratio/high_mean': 0.0069247158244252205, 'clip_ratio/high_max': 0.0069247158244252205, 'clip_ratio/region_mean': 0.01859611738473177, 'step_time': 17.887436245997378, 'epoch': 0.06}
{'loss': 0.2637, 'grad_norm': 0.940700352191925, 'learning_rate': 4.935887621191364e-06, 'kl': 0.027521030511707067, 'entropy': 0.2852812856435776, 'clip_ratio/low_mean': 0.009722699178382754, 'clip_ratio/low_min': 0.009722699178382754, 'clip_ratio/high_mean': 0.0023193359375, 'clip_ratio/high_max': 0.0023193359375, 'clip_ratio/region_mean': 0.012042035115882754, 'step_time': 1.8447213330073282, 'epoch': 0.06}
{'loss': 0.4419, 'grad_norm': 0.48655036091804504, 'learning_rate': 4.903834726061565e-06, 'num_tokens': 1578733.0, 'completions/mean_length': 1256.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4305000305175781, 'rewards/game_reward_func/std': 0.1649441421031952, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4305000305175781, 'reward_std': 0.1649441421031952, 'frac_reward_zero_std': 0.0, 'kl': 0.016556331887841225, 'entropy': 0.24118458107113838, 'clip_ratio/low_mean': 0.0061061407322995365, 'clip_ratio/low_min': 0.0061061407322995365, 'clip_ratio/high_mean': 0.00560461962595582, 'clip_ratio/high_max': 0.00560461962595582, 'clip_ratio/region_mean': 0.011710760358255357, 'step_time': 18.11230182300642, 'epoch': 0.06}
{'loss': -0.4352, 'grad_norm': 0.8369249701499939, 'learning_rate': 4.871785783909523e-06, 'kl': 0.02281445381231606, 'entropy': 0.27866310626268387, 'clip_ratio/low_mean': 0.0034620099468156695, 'clip_ratio/low_min': 0.0034620099468156695, 'clip_ratio/high_mean': 0.007080078125, 'clip_ratio/high_max': 0.007080078125, 'clip_ratio/region_mean': 0.01054208807181567, 'step_time': 1.835859752012766, 'epoch': 0.07}
{'loss': -0.2956, 'grad_norm': 0.6435661911964417, 'learning_rate': 4.839742112141725e-06, 'num_tokens': 1590277.0, 'completions/mean_length': 1264.0, 'completions/min_length': 256.0, 'completions/max_length': 1856.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4635000228881836, 'rewards/game_reward_func/std': 0.08769916743040085, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4635000228881836, 'reward_std': 0.08769916743040085, 'frac_reward_zero_std': 0.0, 'kl': 0.012808160157874227, 'entropy': 0.27121880277991295, 'clip_ratio/low_mean': 0.0048069392796605825, 'clip_ratio/low_min': 0.0048069392796605825, 'clip_ratio/high_mean': 0.0046777037205174565, 'clip_ratio/high_max': 0.0046777037205174565, 'clip_ratio/region_mean': 0.009484643000178039, 'step_time': 16.12712614798511, 'epoch': 0.07}
{'loss': 0.2994, 'grad_norm': 0.32287493348121643, 'learning_rate': 4.807705027948008e-06, 'kl': 0.0282395645044744, 'entropy': 0.27052558586001396, 'clip_ratio/low_mean': 0.005859375, 'clip_ratio/low_min': 0.005859375, 'clip_ratio/high_mean': 0.014555431553162634, 'clip_ratio/high_max': 0.014555431553162634, 'clip_ratio/region_mean': 0.020414806553162634, 'step_time': 1.7136157549975906, 'epoch': 0.07}
{'loss': -0.167, 'grad_norm': 0.2732711434364319, 'learning_rate': 4.775675848247427e-06, 'num_tokens': 1603869.0, 'completions/mean_length': 1520.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4307499825954437, 'rewards/game_reward_func/std': 0.15098130702972412, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4307499825954437, 'reward_std': 0.15098130702972412, 'frac_reward_zero_std': 0.0, 'kl': 0.014388922834768891, 'entropy': 0.23028169944882393, 'clip_ratio/low_mean': 0.004075129167176783, 'clip_ratio/low_min': 0.004075129167176783, 'clip_ratio/high_mean': 0.0031152344308793545, 'clip_ratio/high_max': 0.0031152344308793545, 'clip_ratio/region_mean': 0.0071903635980561376, 'step_time': 18.360726077997242, 'epoch': 0.07}
{'loss': 0.1729, 'grad_norm': 0.2113896608352661, 'learning_rate': 4.743655889634105e-06, 'kl': 0.01587767736054957, 'entropy': 0.23632622882723808, 'clip_ratio/low_mean': 0.004025208647362888, 'clip_ratio/low_min': 0.004025208647362888, 'clip_ratio/high_mean': 0.0017361111240461469, 'clip_ratio/high_max': 0.0017361111240461469, 'clip_ratio/region_mean': 0.005761319771409035, 'step_time': 2.2362708120053867, 'epoch': 0.07}
{'loss': 0.5131, 'grad_norm': 0.9121636152267456, 'learning_rate': 4.711646468323129e-06, 'num_tokens': 1615029.0, 'completions/mean_length': 1216.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.35899999737739563, 'rewards/game_reward_func/std': 0.16943266987800598, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.35899999737739563, 'reward_std': 0.16943268477916718, 'frac_reward_zero_std': 0.0, 'kl': 0.021516271168366075, 'entropy': 0.26568198576569557, 'clip_ratio/low_mean': 0.012053789338096976, 'clip_ratio/low_min': 0.012053789338096976, 'clip_ratio/high_mean': 0.0015625000232830644, 'clip_ratio/high_max': 0.0015625000232830644, 'clip_ratio/region_mean': 0.01361628936138004, 'step_time': 17.967107480995764, 'epoch': 0.07}

 53%|█████▎    | 266/500 [43:58<33:40,  8.63s/it][2026-03-26 12:53:59] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:54:14] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=25.2

 53%|█████▎    | 267/500 [44:16<45:07, 11.62s/it]
                                                 

 53%|█████▎    | 267/500 [44:16<45:07, 11.62s/it]
 54%|█████▎    | 268/500 [44:18<33:48,  8.74s/it]
                                                 

 54%|█████▎    | 268/500 [44:18<33:48,  8.74s/it][2026-03-26 12:54:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:54:34] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=22.1

 54%|█████▍    | 269/500 [44:37<44:37, 11.59s/it]
                                                 

 54%|█████▍    | 269/500 [44:37<44:37, 11.59s/it]
 54%|█████▍    | 270/500 [44:39<33:43,  8.80s/it]
                                                 

 54%|█████▍    | 270/500 [44:39<33:43,  8.80s/it][2026-03-26 12:54:40] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:54:54] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=21.1

 54%|█████▍    | 271/500 [44:57<44:19, 11.61s/it]
                                                 

 54%|█████▍    | 271/500 [44:57<44:19, 11.61s/it]
 54%|█████▍    | 272/500 [44:59<33:03,  8.70s/it]
                                                 

 54%|█████▍    | 272/500 [44:59<33:03,  8.70s/it][2026-03-26 12:55:01] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:55:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.8

 55%|█████▍    | 273/500 [45:15<41:14, 10.90s/it]
                                                 

 55%|█████▍    | 273/500 [45:15<41:14, 10.90s/it]
 55%|█████▍    | 274/500 [45:17<30:38,  8.14s/it]
                                                 

 55%|█████▍    | 274/500 [45:17<30:38,  8.14s/it][2026-03-26 12:55:18] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:55:32] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.4

 55%|█████▌    | 275/500 [45:35<41:41, 11.12s/it]
                                                 

 55%|█████▌    | 275/500 [45:35<41:41, 11.12s/it]
 55%|█████▌    | 276/500 [45:37<31:26,  8.42s/it]
                                                 

 55%|█████▌    | 276/500 [45:37<31:26,  8.42s/it][2026-03-26 12:55:38] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:55:51] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=20.1

 55%|█████▌    | 277/500 [45:54<40:38, 10.93s/it]
                                                 
{'loss': -0.5092, 'grad_norm': 0.712942361831665, 'learning_rate': 4.679648900096436e-06, 'kl': 0.01797645748592913, 'entropy': 0.25688786059617996, 'clip_ratio/low_mean': 0.002197265625, 'clip_ratio/low_min': 0.002197265625, 'clip_ratio/high_mean': 0.01381835900247097, 'clip_ratio/high_max': 0.01381835900247097, 'clip_ratio/region_mean': 0.01601562462747097, 'step_time': 1.7394656449978356, 'epoch': 0.07}
{'loss': -0.1337, 'grad_norm': 0.2750883400440216, 'learning_rate': 4.64766450024873e-06, 'num_tokens': 1629389.0, 'completions/mean_length': 1616.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3512499928474426, 'rewards/game_reward_func/std': 0.2094098925590515, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3512499928474426, 'reward_std': 0.2094098925590515, 'frac_reward_zero_std': 0.0, 'kl': 0.017213734099641442, 'entropy': 0.2600885257124901, 'clip_ratio/low_mean': 0.003995196893811226, 'clip_ratio/low_min': 0.003995196893811226, 'clip_ratio/high_mean': 0.0074055991135537624, 'clip_ratio/high_max': 0.0074055991135537624, 'clip_ratio/region_mean': 0.011400796007364988, 'step_time': 18.412844526006666, 'epoch': 0.07}
{'loss': 0.1384, 'grad_norm': 0.3075898289680481, 'learning_rate': 4.615694583533418e-06, 'kl': 0.011693647829815745, 'entropy': 0.2239084504544735, 'clip_ratio/low_mean': 0.004033564822748303, 'clip_ratio/low_min': 0.004033564822748303, 'clip_ratio/high_mean': 0.003662109375, 'clip_ratio/high_max': 0.003662109375, 'clip_ratio/region_mean': 0.007695674197748303, 'step_time': 1.8402623150032014, 'epoch': 0.07}
{'loss': -0.4659, 'grad_norm': 0.6215663552284241, 'learning_rate': 4.583740464108554e-06, 'num_tokens': 1642149.0, 'completions/mean_length': 1416.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.22474999725818634, 'rewards/game_reward_func/std': 0.2529662847518921, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.22474999725818634, 'reward_std': 0.2529662549495697, 'frac_reward_zero_std': 0.0, 'kl': 0.024164324859157205, 'entropy': 0.23390688747167587, 'clip_ratio/low_mean': 0.009259259211830795, 'clip_ratio/low_min': 0.009259259211830795, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.012188946711830795, 'step_time': 18.072011927994026, 'epoch': 0.07}
{'loss': 0.4726, 'grad_norm': 0.9439697265625, 'learning_rate': 4.551803455482833e-06, 'kl': 0.01512395334430039, 'entropy': 0.2776440568268299, 'clip_ratio/low_mean': 0.0060614224057644606, 'clip_ratio/low_min': 0.0060614224057644606, 'clip_ratio/high_mean': 0.0008544921875, 'clip_ratio/high_max': 0.0008544921875, 'clip_ratio/region_mean': 0.0069159145932644606, 'step_time': 2.1017759980022674, 'epoch': 0.07}
{'loss': 0.1809, 'grad_norm': 0.9411332607269287, 'learning_rate': 4.5198848704615915e-06, 'num_tokens': 1654397.0, 'completions/mean_length': 1352.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3477500081062317, 'rewards/game_reward_func/std': 0.23208974301815033, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3477500081062317, 'reward_std': 0.23208972811698914, 'frac_reward_zero_std': 0.0, 'kl': 0.021425790153443813, 'entropy': 0.2597842514514923, 'clip_ratio/low_mean': 0.008510044543072581, 'clip_ratio/low_min': 0.008510044543072581, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.010219028918072581, 'step_time': 17.995372010998835, 'epoch': 0.07}
{'loss': -0.1734, 'grad_norm': 0.23560510575771332, 'learning_rate': 4.487986021092844e-06, 'kl': 0.015200563007965684, 'entropy': 0.26563502848148346, 'clip_ratio/low_mean': 0.00416100537404418, 'clip_ratio/low_min': 0.00416100537404418, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.00709069287404418, 'step_time': 1.7488993180013495, 'epoch': 0.07}
{'loss': 0.2756, 'grad_norm': 0.6997728943824768, 'learning_rate': 4.456108218613346e-06, 'num_tokens': 1666965.0, 'completions/mean_length': 1392.0, 'completions/min_length': 384.0, 'completions/max_length': 1792.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4074999988079071, 'rewards/game_reward_func/std': 0.1194116473197937, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4074999988079071, 'reward_std': 0.1194116473197937, 'frac_reward_zero_std': 0.0, 'kl': 0.02421433525159955, 'entropy': 0.2596450634300709, 'clip_ratio/low_mean': 0.007595486124046147, 'clip_ratio/low_min': 0.007595486124046147, 'clip_ratio/high_mean': 0.00491898157633841, 'clip_ratio/high_max': 0.00491898157633841, 'clip_ratio/region_mean': 0.012514467700384557, 'step_time': 15.898648001006222, 'epoch': 0.07}
{'loss': -0.2697, 'grad_norm': 0.5049347281455994, 'learning_rate': 4.424252773394704e-06, 'kl': 0.0172907083760947, 'entropy': 0.24858644604682922, 'clip_ratio/low_mean': 0.0009765625, 'clip_ratio/low_min': 0.0009765625, 'clip_ratio/high_mean': 0.010435268166474998, 'clip_ratio/high_max': 0.010435268166474998, 'clip_ratio/region_mean': 0.011411830666474998, 'step_time': 1.5353530580032384, 'epoch': 0.07}
{'loss': -0.326, 'grad_norm': 0.7309715747833252, 'learning_rate': 4.392420994889498e-06, 'num_tokens': 1679341.0, 'completions/mean_length': 1368.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4507499933242798, 'rewards/game_reward_func/std': 0.14483070373535156, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4507499933242798, 'reward_std': 0.14483067393302917, 'frac_reward_zero_std': 0.0, 'kl': 0.014409537892788649, 'entropy': 0.2479974366724491, 'clip_ratio/low_mean': 0.00146484375, 'clip_ratio/low_min': 0.00146484375, 'clip_ratio/high_mean': 0.0037407769123092294, 'clip_ratio/high_max': 0.0037407769123092294, 'clip_ratio/region_mean': 0.005205620662309229, 'step_time': 17.90263989999221, 'epoch': 0.07}
{'loss': 0.333, 'grad_norm': 0.7071624994277954, 'learning_rate': 4.3606141915774695e-06, 'kl': 0.020995844155550003, 'entropy': 0.24377666413784027, 'clip_ratio/low_mean': 0.008830670500174165, 'clip_ratio/low_min': 0.008830670500174165, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.011760358000174165, 'step_time': 1.9337389719948987, 'epoch': 0.07}

 55%|█████▌    | 277/500 [45:54<40:38, 10.93s/it]
 56%|█████▌    | 278/500 [45:56<30:53,  8.35s/it]
                                                 

 56%|█████▌    | 278/500 [45:56<30:53,  8.35s/it][2026-03-26 12:55:58] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:56:12] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.2

 56%|█████▌    | 279/500 [46:14<41:37, 11.30s/it]
                                                 

 56%|█████▌    | 279/500 [46:14<41:37, 11.30s/it]
 56%|█████▌    | 280/500 [46:16<31:13,  8.52s/it]
                                                 

 56%|█████▌    | 280/500 [46:16<31:13,  8.52s/it][2026-03-26 12:56:18] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:56:32] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.9

 56%|█████▌    | 281/500 [46:35<41:53, 11.48s/it]
                                                 

 56%|█████▌    | 281/500 [46:35<41:53, 11.48s/it]
 56%|█████▋    | 282/500 [46:36<31:16,  8.61s/it]
                                                 

 56%|█████▋    | 282/500 [46:36<31:16,  8.61s/it][2026-03-26 12:56:38] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:56:52] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.6

 57%|█████▋    | 283/500 [46:55<41:29, 11.47s/it]
                                                 

 57%|█████▋    | 283/500 [46:55<41:29, 11.47s/it]
 57%|█████▋    | 284/500 [46:57<31:05,  8.64s/it]
                                                 

 57%|█████▋    | 284/500 [46:57<31:05,  8.64s/it][2026-03-26 12:56:58] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:57:07] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:7 D:0 (12% win) | avg_turns=14.1

 57%|█████▋    | 285/500 [47:08<34:17,  9.57s/it]
                                                 

 57%|█████▋    | 285/500 [47:08<34:17,  9.57s/it]
 57%|█████▋    | 286/500 [47:10<25:19,  7.10s/it]
                                                 

 57%|█████▋    | 286/500 [47:10<25:19,  7.10s/it][2026-03-26 12:57:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:57:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.8

 57%|█████▋    | 287/500 [47:28<36:48, 10.37s/it]
                                                 

 57%|█████▋    | 287/500 [47:28<36:48, 10.37s/it]
 58%|█████▊    | 288/500 [47:30<27:47,  7.86s/it]
                                                 
{'loss': -0.3199, 'grad_norm': 0.572421133518219, 'learning_rate': 4.3288336709117246e-06, 'num_tokens': 1691077.0, 'completions/mean_length': 1288.0, 'completions/min_length': 256.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4242500066757202, 'rewards/game_reward_func/std': 0.10916533321142197, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4242500066757202, 'reward_std': 0.10916533321142197, 'frac_reward_zero_std': 0.0, 'kl': 0.021113675320520997, 'entropy': 0.2868380844593048, 'clip_ratio/low_mean': 0.004009046126157045, 'clip_ratio/low_min': 0.004009046126157045, 'clip_ratio/high_mean': 0.005269607878290117, 'clip_ratio/high_max': 0.005269607878290117, 'clip_ratio/region_mean': 0.009278654004447162, 'step_time': 16.632864867002354, 'epoch': 0.07}
{'loss': 0.3269, 'grad_norm': 0.49147939682006836, 'learning_rate': 4.297080739264987e-06, 'kl': 0.028822183376178145, 'entropy': 0.23623725771903992, 'clip_ratio/low_mean': 0.013257211481686682, 'clip_ratio/low_min': 0.013257211481686682, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.014722055231686682, 'step_time': 2.1284800239882316, 'epoch': 0.07}
{'loss': -0.2924, 'grad_norm': 0.4864688813686371, 'learning_rate': 4.265356701875911e-06, 'num_tokens': 1703389.0, 'completions/mean_length': 1360.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.27950000762939453, 'rewards/game_reward_func/std': 0.2782368063926697, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.27950000762939453, 'reward_std': 0.2782368063926697, 'frac_reward_zero_std': 0.0, 'kl': 0.02292561810463667, 'entropy': 0.2254074364900589, 'clip_ratio/low_mean': 0.003004807746037841, 'clip_ratio/low_min': 0.003004807746037841, 'clip_ratio/high_mean': 0.009257311699911952, 'clip_ratio/high_max': 0.009257311699911952, 'clip_ratio/region_mean': 0.012262119445949793, 'step_time': 18.012781867008016, 'epoch': 0.07}
{'loss': 0.2988, 'grad_norm': 0.6805055737495422, 'learning_rate': 4.23366286279542e-06, 'kl': 0.019264145521447062, 'entropy': 0.20477037131786346, 'clip_ratio/low_mean': 0.006038203573552892, 'clip_ratio/low_min': 0.006038203573552892, 'clip_ratio/high_mean': 0.0020751953125, 'clip_ratio/high_max': 0.0020751953125, 'clip_ratio/region_mean': 0.008113398886052892, 'step_time': 1.8363729780030553, 'epoch': 0.07}
{'loss': 0.3587, 'grad_norm': 0.6775110960006714, 'learning_rate': 4.2020005248331056e-06, 'num_tokens': 1715509.0, 'completions/mean_length': 1336.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3657499849796295, 'rewards/game_reward_func/std': 0.20603033900260925, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3657499849796295, 'reward_std': 0.20603033900260925, 'frac_reward_zero_std': 0.0, 'kl': 0.026255127042531967, 'entropy': 0.25265662744641304, 'clip_ratio/low_mean': 0.009847861947491765, 'clip_ratio/low_min': 0.009847861947491765, 'clip_ratio/high_mean': 0.0030691963620483875, 'clip_ratio/high_max': 0.0030691963620483875, 'clip_ratio/region_mean': 0.012917058309540153, 'step_time': 18.17636139300157, 'epoch': 0.07}
{'loss': -0.3517, 'grad_norm': 0.4983607530593872, 'learning_rate': 4.170370989503662e-06, 'kl': 0.01941736601293087, 'entropy': 0.24185915663838387, 'clip_ratio/low_mean': 0.003046875004656613, 'clip_ratio/low_min': 0.003046875004656613, 'clip_ratio/high_mean': 0.003173828125, 'clip_ratio/high_max': 0.003173828125, 'clip_ratio/region_mean': 0.006220703129656613, 'step_time': 1.7449861139975837, 'epoch': 0.07}
{'loss': -0.2277, 'grad_norm': 0.3008696436882019, 'learning_rate': 4.138775556973406e-06, 'num_tokens': 1727501.0, 'completions/mean_length': 1320.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33824998140335083, 'rewards/game_reward_func/std': 0.21422934532165527, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33824998140335083, 'reward_std': 0.21422934532165527, 'frac_reward_zero_std': 0.0, 'kl': 0.03043250273913145, 'entropy': 0.2596267759799957, 'clip_ratio/low_mean': 0.010580357280559838, 'clip_ratio/low_min': 0.010580357280559838, 'clip_ratio/high_mean': 0.0025634765625, 'clip_ratio/high_max': 0.0025634765625, 'clip_ratio/region_mean': 0.013143833843059838, 'step_time': 17.99211230200308, 'epoch': 0.07}
{'loss': 0.2369, 'grad_norm': 0.7672620415687561, 'learning_rate': 4.107215526006818e-06, 'kl': 0.02019678126089275, 'entropy': 0.2396726757287979, 'clip_ratio/low_mean': 0.0043454694095999, 'clip_ratio/low_min': 0.0043454694095999, 'clip_ratio/high_mean': 0.004301525303162634, 'clip_ratio/high_max': 0.004301525303162634, 'clip_ratio/region_mean': 0.008646994712762535, 'step_time': 1.8486657290050061, 'epoch': 0.07}
{'loss': -0.3586, 'grad_norm': 0.790709912776947, 'learning_rate': 4.075692193913156e-06, 'num_tokens': 1736165.0, 'completions/mean_length': 904.0, 'completions/min_length': 448.0, 'completions/max_length': 1280.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.18324998021125793, 'rewards/game_reward_func/std': 0.39876946806907654, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.18324998021125793, 'reward_std': 0.39876946806907654, 'frac_reward_zero_std': 0.0, 'kl': 0.027118190191686153, 'entropy': 0.27663252502679825, 'clip_ratio/low_mean': 0.009393600979819894, 'clip_ratio/low_min': 0.009393600979819894, 'clip_ratio/high_mean': 0.006103515625, 'clip_ratio/high_max': 0.006103515625, 'clip_ratio/region_mean': 0.015497116604819894, 'step_time': 11.621698134003964, 'epoch': 0.07}
{'loss': 0.3641, 'grad_norm': 0.6248423457145691, 'learning_rate': 4.04420685649314e-06, 'kl': 0.030488838907331228, 'entropy': 0.275240708142519, 'clip_ratio/low_mean': 0.011728050652891397, 'clip_ratio/low_min': 0.011728050652891397, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.011728050652891397, 'step_time': 1.2157868060021428, 'epoch': 0.07}
{'loss': 0.4504, 'grad_norm': 0.5286465883255005, 'learning_rate': 4.012760807985665e-06, 'num_tokens': 1746685.0, 'completions/mean_length': 1136.0, 'completions/min_length': 64.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.29500001668930054, 'rewards/game_reward_func/std': 0.19537802040576935, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.29500001668930054, 'reward_std': 0.19537802040576935, 'frac_reward_zero_std': 0.0, 'kl': 0.0215478939935565, 'entropy': 0.277647215873003, 'clip_ratio/low_mean': 0.004937066056299955, 'clip_ratio/low_min': 0.004937066056299955, 'clip_ratio/high_mean': 0.004150390625, 'clip_ratio/high_max': 0.004150390625, 'clip_ratio/region_mean': 0.009087456681299955, 'step_time': 17.80910668399156, 'epoch': 0.07}

 58%|█████▊    | 288/500 [47:30<27:47,  7.86s/it][2026-03-26 12:57:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:57:45] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.5

 58%|█████▊    | 289/500 [47:48<38:32, 10.96s/it]
                                                 

 58%|█████▊    | 289/500 [47:48<38:32, 10.96s/it]
 58%|█████▊    | 290/500 [47:50<29:05,  8.31s/it]
                                                 

 58%|█████▊    | 290/500 [47:50<29:05,  8.31s/it][2026-03-26 12:57:52] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:58:06] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=22.4

 58%|█████▊    | 291/500 [48:08<39:15, 11.27s/it]
                                                 

 58%|█████▊    | 291/500 [48:08<39:15, 11.27s/it]
 58%|█████▊    | 292/500 [48:11<29:45,  8.58s/it]
                                                 

 58%|█████▊    | 292/500 [48:11<29:45,  8.58s/it][2026-03-26 12:58:12] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:58:26] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=20.0

 59%|█████▊    | 293/500 [48:29<39:34, 11.47s/it]
                                                 

 59%|█████▊    | 293/500 [48:29<39:34, 11.47s/it]
 59%|█████▉    | 294/500 [48:31<29:38,  8.63s/it]
                                                 

 59%|█████▉    | 294/500 [48:31<29:38,  8.63s/it][2026-03-26 12:58:32] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:58:46] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.0

 59%|█████▉    | 295/500 [48:49<39:08, 11.46s/it]
                                                 

 59%|█████▉    | 295/500 [48:49<39:08, 11.46s/it]
 59%|█████▉    | 296/500 [48:51<29:19,  8.62s/it]
                                                 

 59%|█████▉    | 296/500 [48:51<29:19,  8.62s/it][2026-03-26 12:58:52] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:59:05] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=22.6

 59%|█████▉    | 297/500 [49:07<36:42, 10.85s/it]
                                                 

 59%|█████▉    | 297/500 [49:07<36:42, 10.85s/it]
 60%|█████▉    | 298/500 [49:09<27:16,  8.10s/it]
                                                 

 60%|█████▉    | 298/500 [49:09<27:16,  8.10s/it][2026-03-26 12:59:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:59:23] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.2

 60%|█████▉    | 299/500 [49:26<36:46, 10.98s/it]
                                                 
{'loss': -0.4447, 'grad_norm': 1.0069416761398315, 'learning_rate': 3.9813553410146225e-06, 'kl': 0.04201185726560652, 'entropy': 0.26215359941124916, 'clip_ratio/low_mean': 0.015625, 'clip_ratio/low_min': 0.015625, 'clip_ratio/high_mean': 0.012325274874456227, 'clip_ratio/high_max': 0.012325274874456227, 'clip_ratio/region_mean': 0.027950274874456227, 'step_time': 1.8369800690124976, 'epoch': 0.07}
{'loss': -0.4082, 'grad_norm': 0.7734007239341736, 'learning_rate': 3.949991746535753e-06, 'num_tokens': 1759125.0, 'completions/mean_length': 1376.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4300000071525574, 'rewards/game_reward_func/std': 0.16980662941932678, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4300000071525574, 'reward_std': 0.16980662941932678, 'frac_reward_zero_std': 0.0, 'kl': 0.02809220738708973, 'entropy': 0.24567141756415367, 'clip_ratio/low_mean': 0.005533854244276881, 'clip_ratio/low_min': 0.005533854244276881, 'clip_ratio/high_mean': 0.0062255859375, 'clip_ratio/high_max': 0.0062255859375, 'clip_ratio/region_mean': 0.011759440181776881, 'step_time': 18.017005047004204, 'epoch': 0.07}
{'loss': 0.4143, 'grad_norm': 0.7142171859741211, 'learning_rate': 3.918671313783583e-06, 'kl': 0.02350993873551488, 'entropy': 0.2389484904706478, 'clip_ratio/low_mean': 0.005577651667408645, 'clip_ratio/low_min': 0.005577651667408645, 'clip_ratio/high_mean': 0.0023437500931322575, 'clip_ratio/high_max': 0.0023437500931322575, 'clip_ratio/region_mean': 0.007921401760540903, 'step_time': 1.9403347519910312, 'epoch': 0.07}
{'loss': -0.4026, 'grad_norm': 0.5170515775680542, 'learning_rate': 3.887395330218429e-06, 'num_tokens': 1772013.0, 'completions/mean_length': 1432.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.31825000047683716, 'rewards/game_reward_func/std': 0.2090206742286682, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.31825000047683716, 'reward_std': 0.2090206742286682, 'frac_reward_zero_std': 0.0, 'kl': 0.03079533064737916, 'entropy': 0.245974313467741, 'clip_ratio/low_mean': 0.007048872299492359, 'clip_ratio/low_min': 0.007048872299492359, 'clip_ratio/high_mean': 0.0032958984375, 'clip_ratio/high_max': 0.0032958984375, 'clip_ratio/region_mean': 0.01034477073699236, 'step_time': 18.00297353599308, 'epoch': 0.07}
{'loss': 0.4108, 'grad_norm': 0.6960827708244324, 'learning_rate': 3.856165081473474e-06, 'kl': 0.023423217702656984, 'entropy': 0.23657916858792305, 'clip_ratio/low_mean': 0.008246527751907706, 'clip_ratio/low_min': 0.008246527751907706, 'clip_ratio/high_mean': 0.0018310546875, 'clip_ratio/high_max': 0.0018310546875, 'clip_ratio/region_mean': 0.010077582439407706, 'step_time': 2.108810399993672, 'epoch': 0.07}
{'loss': 0.2508, 'grad_norm': 0.5233822464942932, 'learning_rate': 3.824981851301924e-06, 'num_tokens': 1783749.0, 'completions/mean_length': 1288.0, 'completions/min_length': 64.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2590000033378601, 'rewards/game_reward_func/std': 0.2812380790710449, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2590000033378601, 'reward_std': 0.2812380790710449, 'frac_reward_zero_std': 0.0, 'kl': 0.02300026034936309, 'entropy': 0.2513539530336857, 'clip_ratio/low_mean': 0.005013020941987634, 'clip_ratio/low_min': 0.005013020941987634, 'clip_ratio/high_mean': 0.0025634765625, 'clip_ratio/high_max': 0.0025634765625, 'clip_ratio/region_mean': 0.007576497504487634, 'step_time': 18.026298874996428, 'epoch': 0.07}
{'loss': -0.2431, 'grad_norm': 1.1879854202270508, 'learning_rate': 3.7938469215242374e-06, 'kl': 0.028581555001437664, 'entropy': 0.2676904983818531, 'clip_ratio/low_mean': 0.005629595601931214, 'clip_ratio/low_min': 0.005629595601931214, 'clip_ratio/high_mean': 0.0089111328125, 'clip_ratio/high_max': 0.0089111328125, 'clip_ratio/region_mean': 0.014540728414431214, 'step_time': 1.8351755700059584, 'epoch': 0.07}
{'loss': 0.5116, 'grad_norm': 0.6566150188446045, 'learning_rate': 3.7627615719754294e-06, 'num_tokens': 1794909.0, 'completions/mean_length': 1216.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3474999964237213, 'rewards/game_reward_func/std': 0.18169283866882324, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3474999964237213, 'reward_std': 0.18169282376766205, 'frac_reward_zero_std': 0.0, 'kl': 0.02620744612067938, 'entropy': 0.23189912736415863, 'clip_ratio/low_mean': 0.0037179789505898952, 'clip_ratio/low_min': 0.0037179789505898952, 'clip_ratio/high_mean': 0.0037202381063252687, 'clip_ratio/high_max': 0.0037202381063252687, 'clip_ratio/region_mean': 0.007438217056915164, 'step_time': 17.85518525601219, 'epoch': 0.07}
{'loss': -0.4998, 'grad_norm': 0.9670708775520325, 'learning_rate': 3.731727080452464e-06, 'kl': 0.041278940392658114, 'entropy': 0.26472189649939537, 'clip_ratio/low_mean': 0.002704326994717121, 'clip_ratio/low_min': 0.002704326994717121, 'clip_ratio/high_mean': 0.0108642578125, 'clip_ratio/high_max': 0.0108642578125, 'clip_ratio/region_mean': 0.013568584807217121, 'step_time': 1.8305479049959104, 'epoch': 0.07}
{'loss': -0.6998, 'grad_norm': 1.4206494092941284, 'learning_rate': 3.7007447226617367e-06, 'num_tokens': 1807925.0, 'completions/mean_length': 1448.0, 'completions/min_length': 896.0, 'completions/max_length': 1792.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.49424999952316284, 'rewards/game_reward_func/std': 0.1020500659942627, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.49424999952316284, 'reward_std': 0.1020500585436821, 'frac_reward_zero_std': 0.0, 'kl': 0.032221375964581966, 'entropy': 0.23942124471068382, 'clip_ratio/low_mean': 0.0026041667442768812, 'clip_ratio/low_min': 0.0026041667442768812, 'clip_ratio/high_mean': 0.01040762453339994, 'clip_ratio/high_max': 0.01040762453339994, 'clip_ratio/region_mean': 0.01301179127767682, 'step_time': 15.880832435002958, 'epoch': 0.07}
{'loss': 0.7038, 'grad_norm': 0.8522257208824158, 'learning_rate': 3.669815772166625e-06, 'kl': 0.024793454445898533, 'entropy': 0.24582336097955704, 'clip_ratio/low_mean': 0.009397828893270344, 'clip_ratio/low_min': 0.009397828893270344, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.009397828893270344, 'step_time': 1.5448317029949976, 'epoch': 0.07}

 60%|█████▉    | 299/500 [49:26<36:46, 10.98s/it]
 60%|██████    | 300/500 [49:28<27:34,  8.27s/it]
                                                 

 60%|██████    | 300/500 [49:28<27:34,  8.27s/it][2026-03-26 12:59:30] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 12:59:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=24.2

 60%|██████    | 301/500 [49:47<37:47, 11.40s/it]
                                                 

 60%|██████    | 301/500 [49:47<37:47, 11.40s/it]
 60%|██████    | 302/500 [49:49<28:19,  8.58s/it]
                                                 

 60%|██████    | 302/500 [49:49<28:19,  8.58s/it][2026-03-26 12:59:51] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:00:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=20.0

 61%|██████    | 303/500 [50:07<37:07, 11.31s/it]
                                                 

 61%|██████    | 303/500 [50:07<37:07, 11.31s/it]
 61%|██████    | 304/500 [50:09<27:56,  8.55s/it]
                                                 

 61%|██████    | 304/500 [50:09<27:56,  8.55s/it][2026-03-26 13:00:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:00:24] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.5

 61%|██████    | 305/500 [50:27<37:04, 11.41s/it]
                                                 

 61%|██████    | 305/500 [50:27<37:04, 11.41s/it]
 61%|██████    | 306/500 [50:29<28:03,  8.68s/it]
                                                 

 61%|██████    | 306/500 [50:29<28:03,  8.68s/it][2026-03-26 13:00:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:00:42] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.1

 61%|██████▏   | 307/500 [50:44<34:10, 10.63s/it]
                                                 

 61%|██████▏   | 307/500 [50:44<34:10, 10.63s/it]
 62%|██████▏   | 308/500 [50:46<25:30,  7.97s/it]
                                                 

 62%|██████▏   | 308/500 [50:46<25:30,  7.97s/it][2026-03-26 13:00:48] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:01:02] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:4 D:4 (0% win) | avg_turns=20.1

 62%|██████▏   | 309/500 [51:04<35:17, 11.09s/it]
                                                 

 62%|██████▏   | 309/500 [51:04<35:17, 11.09s/it]
 62%|██████▏   | 310/500 [51:06<26:23,  8.33s/it]
                                                 
{'loss': -0.0761, 'grad_norm': 0.44970351457595825, 'learning_rate': 3.638941500335145e-06, 'num_tokens': 1820237.0, 'completions/mean_length': 1360.0, 'completions/min_length': 320.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40024998784065247, 'rewards/game_reward_func/std': 0.13092827796936035, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40024998784065247, 'reward_std': 0.13092827796936035, 'frac_reward_zero_std': 0.0, 'kl': 0.03872614912688732, 'entropy': 0.22572849690914154, 'clip_ratio/low_mean': 0.00632440485060215, 'clip_ratio/low_min': 0.00632440485060215, 'clip_ratio/high_mean': 0.01132812537252903, 'clip_ratio/high_max': 0.01132812537252903, 'clip_ratio/region_mean': 0.01765253022313118, 'step_time': 17.50465673099825, 'epoch': 0.07}
{'loss': 0.0848, 'grad_norm': 0.18154384195804596, 'learning_rate': 3.608123176287685e-06, 'kl': 0.02371317334473133, 'entropy': 0.24094361066818237, 'clip_ratio/low_mean': 0.005642361124046147, 'clip_ratio/low_min': 0.005642361124046147, 'clip_ratio/high_mean': 0.0031922042835503817, 'clip_ratio/high_max': 0.0031922042835503817, 'clip_ratio/region_mean': 0.008834565407596529, 'step_time': 1.7812111339953844, 'epoch': 0.07}
{'loss': -0.0903, 'grad_norm': 0.2498335987329483, 'learning_rate': 3.5773620668448384e-06, 'num_tokens': 1834085.0, 'completions/mean_length': 1552.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3440000116825104, 'rewards/game_reward_func/std': 0.1129298210144043, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3440000116825104, 'reward_std': 0.1129298210144043, 'frac_reward_zero_std': 0.0, 'kl': 0.02444121870212257, 'entropy': 0.2412313111126423, 'clip_ratio/low_mean': 0.0033052884973585606, 'clip_ratio/low_min': 0.0033052884973585606, 'clip_ratio/high_mean': 0.006159002194181085, 'clip_ratio/high_max': 0.006159002194181085, 'clip_ratio/region_mean': 0.009464290691539645, 'step_time': 18.121426009005518, 'epoch': 0.08}
{'loss': 0.0952, 'grad_norm': 0.34557202458381653, 'learning_rate': 3.5466594364753325e-06, 'kl': 0.022106583463028073, 'entropy': 0.24932316318154335, 'clip_ratio/low_mean': 0.008383218431845307, 'clip_ratio/low_min': 0.008383218431845307, 'clip_ratio/high_mean': 0.0010986328125, 'clip_ratio/high_max': 0.0010986328125, 'clip_ratio/region_mean': 0.009481851244345307, 'step_time': 1.8399694639956579, 'epoch': 0.08}
{'loss': -0.4044, 'grad_norm': 1.341059684753418, 'learning_rate': 3.516016547244047e-06, 'num_tokens': 1845757.0, 'completions/mean_length': 1280.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4035000205039978, 'rewards/game_reward_func/std': 0.1269814372062683, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4035000205039978, 'reward_std': 0.1269814372062683, 'frac_reward_zero_std': 0.0, 'kl': 0.03460659924894571, 'entropy': 0.2292524129152298, 'clip_ratio/low_mean': 0.004427083418704569, 'clip_ratio/low_min': 0.004427083418704569, 'clip_ratio/high_mean': 0.0060941255651414394, 'clip_ratio/high_max': 0.0060941255651414394, 'clip_ratio/region_mean': 0.010521208983846009, 'step_time': 17.477353144007793, 'epoch': 0.08}
{'loss': 0.4171, 'grad_norm': 0.5843909382820129, 'learning_rate': 3.48543465876014e-06, 'kl': 0.039746048860251904, 'entropy': 0.23487048968672752, 'clip_ratio/low_mean': 0.003000452881678939, 'clip_ratio/low_min': 0.003000452881678939, 'clip_ratio/high_mean': 0.008854166837409139, 'clip_ratio/high_max': 0.008854166837409139, 'clip_ratio/region_mean': 0.011854619719088078, 'step_time': 1.9364629550036625, 'epoch': 0.08}
{'loss': 0.0075, 'grad_norm': 0.25725603103637695, 'learning_rate': 3.4549150281252635e-06, 'num_tokens': 1857173.0, 'completions/mean_length': 1248.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3569999933242798, 'rewards/game_reward_func/std': 0.1430004984140396, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3569999933242798, 'reward_std': 0.1430004984140396, 'frac_reward_zero_std': 0.0, 'kl': 0.03767681587487459, 'entropy': 0.2379019632935524, 'clip_ratio/low_mean': 0.006215277942828834, 'clip_ratio/low_min': 0.006215277942828834, 'clip_ratio/high_mean': 0.004723590333014727, 'clip_ratio/high_max': 0.004723590333014727, 'clip_ratio/region_mean': 0.01093886827584356, 'step_time': 17.89533356398897, 'epoch': 0.08}
{'loss': 0.0069, 'grad_norm': 0.8577322959899902, 'learning_rate': 3.424458909881897e-06, 'kl': 0.061169853899627924, 'entropy': 0.276141993701458, 'clip_ratio/low_mean': 0.010475852293893695, 'clip_ratio/low_min': 0.010475852293893695, 'clip_ratio/high_mean': 0.001220703125, 'clip_ratio/high_max': 0.001220703125, 'clip_ratio/region_mean': 0.011696555418893695, 'step_time': 2.1348168940021424, 'epoch': 0.08}
{'loss': 0.4246, 'grad_norm': 0.44700396060943604, 'learning_rate': 3.3940675559617724e-06, 'num_tokens': 1868397.0, 'completions/mean_length': 1224.0, 'completions/min_length': 704.0, 'completions/max_length': 1728.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.45225000381469727, 'rewards/game_reward_func/std': 0.10957808792591095, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.45225000381469727, 'reward_std': 0.10957808792591095, 'frac_reward_zero_std': 0.0, 'kl': 0.0366992331109941, 'entropy': 0.25634415820240974, 'clip_ratio/low_mean': 0.007701210910454392, 'clip_ratio/low_min': 0.007701210910454392, 'clip_ratio/high_mean': 0.0032051283051259816, 'clip_ratio/high_max': 0.0032051283051259816, 'clip_ratio/region_mean': 0.010906339215580374, 'step_time': 15.014557049005816, 'epoch': 0.08}
{'loss': -0.4177, 'grad_norm': 1.019350528717041, 'learning_rate': 3.363742215634416e-06, 'kl': 0.031583297066390514, 'entropy': 0.22586475685238838, 'clip_ratio/low_mean': 0.004817708395421505, 'clip_ratio/low_min': 0.004817708395421505, 'clip_ratio/high_mean': 0.007634943118318915, 'clip_ratio/high_max': 0.007634943118318915, 'clip_ratio/region_mean': 0.01245265151374042, 'step_time': 1.6176407230013865, 'epoch': 0.08}
{'loss': -0.075, 'grad_norm': 0.8350553512573242, 'learning_rate': 3.3334841354557923e-06, 'num_tokens': 1880197.0, 'completions/mean_length': 1296.0, 'completions/min_length': 64.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2097499966621399, 'rewards/game_reward_func/std': 0.25100356340408325, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2097499966621399, 'reward_std': 0.25100356340408325, 'frac_reward_zero_std': 0.0, 'kl': 0.037145393434911966, 'entropy': 0.2812579721212387, 'clip_ratio/low_mean': 0.01149338943650946, 'clip_ratio/low_min': 0.01149338943650946, 'clip_ratio/high_mean': 0.0135498046875, 'clip_ratio/high_max': 0.0135498046875, 'clip_ratio/region_mean': 0.02504319412400946, 'step_time': 18.165557332999015, 'epoch': 0.08}

 62%|██████▏   | 310/500 [51:06<26:23,  8.33s/it][2026-03-26 13:01:08] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:01:22] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:6 D:1 (12% win) | avg_turns=17.6

 62%|██████▏   | 311/500 [51:24<35:11, 11.17s/it]
                                                 

 62%|██████▏   | 311/500 [51:24<35:11, 11.17s/it]
 62%|██████▏   | 312/500 [51:26<26:29,  8.46s/it]
                                                 

 62%|██████▏   | 312/500 [51:26<26:29,  8.46s/it][2026-03-26 13:01:28] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:01:42] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.5

 63%|██████▎   | 313/500 [51:45<35:44, 11.47s/it]
                                                 

 63%|██████▎   | 313/500 [51:45<35:44, 11.47s/it]
 63%|██████▎   | 314/500 [51:47<26:45,  8.63s/it]
                                                 

 63%|██████▎   | 314/500 [51:47<26:45,  8.63s/it][2026-03-26 13:01:48] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:02:02] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.4

 63%|██████▎   | 315/500 [52:05<35:25, 11.49s/it]
                                                 

 63%|██████▎   | 315/500 [52:05<35:25, 11.49s/it]
 63%|██████▎   | 316/500 [52:07<26:31,  8.65s/it]
                                                 

 63%|██████▎   | 316/500 [52:07<26:31,  8.65s/it][2026-03-26 13:02:09] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:02:23] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.9

 63%|██████▎   | 317/500 [52:25<35:12, 11.54s/it]
                                                 

 63%|██████▎   | 317/500 [52:25<35:12, 11.54s/it]
 64%|██████▎   | 318/500 [52:27<26:26,  8.72s/it]
                                                 

 64%|██████▎   | 318/500 [52:27<26:26,  8.72s/it][2026-03-26 13:02:29] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:02:43] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:5 D:2 (12% win) | avg_turns=22.9

 64%|██████▍   | 319/500 [52:46<35:08, 11.65s/it]
                                                 

 64%|██████▍   | 319/500 [52:46<35:08, 11.65s/it]
 64%|██████▍   | 320/500 [52:48<26:23,  8.80s/it]
                                                 

 64%|██████▍   | 320/500 [52:48<26:23,  8.80s/it][2026-03-26 13:02:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:03:03] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.4

 64%|██████▍   | 321/500 [53:06<34:35, 11.59s/it]
                                                 
{'loss': 0.0825, 'grad_norm': 0.41089868545532227, 'learning_rate': 3.303294559217063e-06, 'kl': 0.030982867116108537, 'entropy': 0.2285434491932392, 'clip_ratio/low_mean': 0.006835937616415322, 'clip_ratio/low_min': 0.006835937616415322, 'clip_ratio/high_mean': 0.0052490234375, 'clip_ratio/high_max': 0.0052490234375, 'clip_ratio/region_mean': 0.012084961053915322, 'step_time': 1.7440119229941047, 'epoch': 0.08}
{'loss': 0.3528, 'grad_norm': 0.44013160467147827, 'learning_rate': 3.273174727893463e-06, 'num_tokens': 1890653.0, 'completions/mean_length': 1128.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.24125000834465027, 'rewards/game_reward_func/std': 0.4480688273906708, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.24125000834465027, 'reward_std': 0.4480687975883484, 'frac_reward_zero_std': 0.0, 'kl': 0.044754324946552515, 'entropy': 0.24735482782125473, 'clip_ratio/low_mean': 0.011930338572710752, 'clip_ratio/low_min': 0.011930338572710752, 'clip_ratio/high_mean': 0.000732421875, 'clip_ratio/high_max': 0.000732421875, 'clip_ratio/region_mean': 0.012662760447710752, 'step_time': 17.62529167899629, 'epoch': 0.08}
{'loss': -0.3465, 'grad_norm': 0.7408607006072998, 'learning_rate': 3.2431258795932863e-06, 'kl': 0.044542118441313505, 'entropy': 0.26834816485643387, 'clip_ratio/low_mean': 0.006513531552627683, 'clip_ratio/low_min': 0.006513531552627683, 'clip_ratio/high_mean': 0.030338541604578495, 'clip_ratio/high_max': 0.030338541604578495, 'clip_ratio/region_mean': 0.03685207315720618, 'step_time': 1.9302691029952257, 'epoch': 0.08}
{'loss': 0.653, 'grad_norm': 0.9342052936553955, 'learning_rate': 3.213149249506997e-06, 'num_tokens': 1902581.0, 'completions/mean_length': 1312.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3734999895095825, 'rewards/game_reward_func/std': 0.13194696605205536, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3734999895095825, 'reward_std': 0.13194696605205536, 'frac_reward_zero_std': 0.0, 'kl': 0.0362149141728878, 'entropy': 0.2642764486372471, 'clip_ratio/low_mean': 0.011461194837465882, 'clip_ratio/low_min': 0.011461194837465882, 'clip_ratio/high_mean': 0.003700657980516553, 'clip_ratio/high_max': 0.003700657980516553, 'clip_ratio/region_mean': 0.015161852817982435, 'step_time': 18.30033660698973, 'epoch': 0.08}
{'loss': -0.6454, 'grad_norm': 1.073986530303955, 'learning_rate': 3.183246069856443e-06, 'kl': 0.02387647097930312, 'entropy': 0.24025368317961693, 'clip_ratio/low_mean': 0.00390625, 'clip_ratio/low_min': 0.00390625, 'clip_ratio/high_mean': 0.009400698821991682, 'clip_ratio/high_max': 0.009400698821991682, 'clip_ratio/region_mean': 0.013306948821991682, 'step_time': 1.836587606005196, 'epoch': 0.08}
{'loss': 0.1922, 'grad_norm': 0.49076706171035767, 'learning_rate': 3.1534175698442194e-06, 'num_tokens': 1914445.0, 'completions/mean_length': 1304.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3607500195503235, 'rewards/game_reward_func/std': 0.16571210324764252, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3607500195503235, 'reward_std': 0.16571210324764252, 'frac_reward_zero_std': 0.0, 'kl': 0.02890775678679347, 'entropy': 0.27247408777475357, 'clip_ratio/low_mean': 0.006520850176457316, 'clip_ratio/low_min': 0.006520850176457316, 'clip_ratio/high_mean': 0.002197265625, 'clip_ratio/high_max': 0.002197265625, 'clip_ratio/region_mean': 0.008718115801457316, 'step_time': 17.9811902149886, 'epoch': 0.08}
{'loss': -0.1863, 'grad_norm': 0.29517579078674316, 'learning_rate': 3.12366497560313e-06, 'kl': 0.02298547839745879, 'entropy': 0.2340693660080433, 'clip_ratio/low_mean': 0.004329527961090207, 'clip_ratio/low_min': 0.004329527961090207, 'clip_ratio/high_mean': 0.0037597656482830644, 'clip_ratio/high_max': 0.0037597656482830644, 'clip_ratio/region_mean': 0.008089293609373271, 'step_time': 1.8372055510117207, 'epoch': 0.08}
{'loss': 0.064, 'grad_norm': 0.2692500650882721, 'learning_rate': 3.093989510145792e-06, 'num_tokens': 1927077.0, 'completions/mean_length': 1400.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33924999833106995, 'rewards/game_reward_func/std': 0.16724298894405365, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33924999833106995, 'reward_std': 0.16724297404289246, 'frac_reward_zero_std': 0.0, 'kl': 0.024909410160034895, 'entropy': 0.24390112608671188, 'clip_ratio/low_mean': 0.0034016927238553762, 'clip_ratio/low_min': 0.0034016927238553762, 'clip_ratio/high_mean': 0.006591796875, 'clip_ratio/high_max': 0.006591796875, 'clip_ratio/region_mean': 0.009993489598855376, 'step_time': 18.118967247995897, 'epoch': 0.08}
{'loss': -0.0584, 'grad_norm': 0.34936532378196716, 'learning_rate': 3.0643923933143603e-06, 'kl': 0.028192368103191257, 'entropy': 0.26899679750204086, 'clip_ratio/low_mean': 0.004882812383584678, 'clip_ratio/low_min': 0.004882812383584678, 'clip_ratio/high_mean': 0.008886718656867743, 'clip_ratio/high_max': 0.008886718656867743, 'clip_ratio/region_mean': 0.01376953104045242, 'step_time': 1.9305988580017583, 'epoch': 0.08}
{'loss': -0.015, 'grad_norm': 0.65650475025177, 'learning_rate': 3.0348748417303826e-06, 'num_tokens': 1940221.0, 'completions/mean_length': 1464.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.20274999737739563, 'rewards/game_reward_func/std': 0.45323970913887024, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.20274999737739563, 'reward_std': 0.45323967933654785, 'frac_reward_zero_std': 0.0, 'kl': 0.026200105901807547, 'entropy': 0.2248445451259613, 'clip_ratio/low_mean': 0.005026608821935952, 'clip_ratio/low_min': 0.005026608821935952, 'clip_ratio/high_mean': 0.002197265625, 'clip_ratio/high_max': 0.002197265625, 'clip_ratio/region_mean': 0.007223874446935952, 'step_time': 18.292472574990825, 'epoch': 0.08}
{'loss': 0.0243, 'grad_norm': 0.19402426481246948, 'learning_rate': 3.005438068744792e-06, 'kl': 0.026843019761145115, 'entropy': 0.2675686329603195, 'clip_ratio/low_mean': 0.007127637742087245, 'clip_ratio/low_min': 0.007127637742087245, 'clip_ratio/high_mean': 0.0032958984375, 'clip_ratio/high_max': 0.0032958984375, 'clip_ratio/region_mean': 0.010423536179587245, 'step_time': 1.9786610339942854, 'epoch': 0.08}

 64%|██████▍   | 321/500 [53:06<34:35, 11.59s/it]
 64%|██████▍   | 322/500 [53:08<25:52,  8.72s/it]
                                                 

 64%|██████▍   | 322/500 [53:08<25:52,  8.72s/it][2026-03-26 13:03:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:03:24] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=23.0

 65%|██████▍   | 323/500 [53:26<34:09, 11.58s/it]
                                                 

 65%|██████▍   | 323/500 [53:26<34:09, 11.58s/it]
 65%|██████▍   | 324/500 [53:28<25:38,  8.74s/it]
                                                 

 65%|██████▍   | 324/500 [53:28<25:38,  8.74s/it][2026-03-26 13:03:30] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:03:45] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=28.0

 65%|██████▌   | 325/500 [53:47<34:10, 11.72s/it]
                                                 

 65%|██████▌   | 325/500 [53:47<34:10, 11.72s/it]
 65%|██████▌   | 326/500 [53:49<25:33,  8.81s/it]
                                                 

 65%|██████▌   | 326/500 [53:49<25:33,  8.81s/it][2026-03-26 13:03:51] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:04:05] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.0

 65%|██████▌   | 327/500 [54:08<33:51, 11.74s/it]
                                                 

 65%|██████▌   | 327/500 [54:08<33:51, 11.74s/it]
 66%|██████▌   | 328/500 [54:10<25:18,  8.83s/it]
                                                 

 66%|██████▌   | 328/500 [54:10<25:18,  8.83s/it][2026-03-26 13:04:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:04:24] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=14.9

 66%|██████▌   | 329/500 [54:26<31:33, 11.07s/it]
                                                 

 66%|██████▌   | 329/500 [54:26<31:33, 11.07s/it]
 66%|██████▌   | 330/500 [54:28<23:39,  8.35s/it]
                                                 

 66%|██████▌   | 330/500 [54:28<23:39,  8.35s/it][2026-03-26 13:04:30] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:04:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.5

 66%|██████▌   | 331/500 [54:46<32:00, 11.36s/it]
                                                 

 66%|██████▌   | 331/500 [54:46<32:00, 11.36s/it]
 66%|██████▋   | 332/500 [54:48<23:52,  8.53s/it]
                                                 
{'loss': -0.0489, 'grad_norm': 0.255706787109375, 'learning_rate': 2.976083284388031e-06, 'num_tokens': 1951573.0, 'completions/mean_length': 1240.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37574997544288635, 'rewards/game_reward_func/std': 0.23297441005706787, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37574997544288635, 'reward_std': 0.23297439515590668, 'frac_reward_zero_std': 0.0, 'kl': 0.03395518800243735, 'entropy': 0.27863485738635063, 'clip_ratio/low_mean': 0.015136718575377017, 'clip_ratio/low_min': 0.015136718575377017, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.016845702950377017, 'step_time': 17.940046682990214, 'epoch': 0.08}
{'loss': 0.0522, 'grad_norm': 0.2717525064945221, 'learning_rate': 2.9468116953203107e-06, 'kl': 0.026228982023894787, 'entropy': 0.25914060696959496, 'clip_ratio/low_mean': 0.009281517064664513, 'clip_ratio/low_min': 0.009281517064664513, 'clip_ratio/high_mean': 0.00244140625, 'clip_ratio/high_max': 0.00244140625, 'clip_ratio/region_mean': 0.011722923314664513, 'step_time': 1.8452745999966282, 'epoch': 0.08}
{'loss': -0.655, 'grad_norm': 0.8525506258010864, 'learning_rate': 2.9176245047820064e-06, 'num_tokens': 1964781.0, 'completions/mean_length': 1472.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2874999940395355, 'rewards/game_reward_func/std': 0.16493721306324005, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2874999940395355, 'reward_std': 0.16493722796440125, 'frac_reward_zero_std': 0.0, 'kl': 0.017816907726228237, 'entropy': 0.2364048883318901, 'clip_ratio/low_mean': 0.001953125, 'clip_ratio/low_min': 0.001953125, 'clip_ratio/high_mean': 0.0047607421875, 'clip_ratio/high_max': 0.0047607421875, 'clip_ratio/region_mean': 0.0067138671875, 'step_time': 18.08168513799319, 'epoch': 0.08}
{'loss': 0.6612, 'grad_norm': 1.1427733898162842, 'learning_rate': 2.8885229125442022e-06, 'kl': 0.028900342993438244, 'entropy': 0.23402726277709007, 'clip_ratio/low_mean': 0.005939327471423894, 'clip_ratio/low_min': 0.005939327471423894, 'clip_ratio/high_mean': 0.00305706518702209, 'clip_ratio/high_max': 0.00305706518702209, 'clip_ratio/region_mean': 0.008996392658445984, 'step_time': 1.9342646829973091, 'epoch': 0.08}
{'loss': 0.0153, 'grad_norm': 0.18731831014156342, 'learning_rate': 2.859508114859374e-06, 'num_tokens': 1980549.0, 'completions/mean_length': 1792.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.34299999475479126, 'rewards/game_reward_func/std': 0.18480569124221802, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.34299999475479126, 'reward_std': 0.18480569124221802, 'frac_reward_zero_std': 0.0, 'kl': 0.01966826803982258, 'entropy': 0.2200573980808258, 'clip_ratio/low_mean': 0.004329637158662081, 'clip_ratio/low_min': 0.004329637158662081, 'clip_ratio/high_mean': 0.005126953125, 'clip_ratio/high_max': 0.005126953125, 'clip_ratio/region_mean': 0.00945659028366208, 'step_time': 18.494804696005303, 'epoch': 0.08}
{'loss': -0.0114, 'grad_norm': 0.20699413120746613, 'learning_rate': 2.83058130441221e-06, 'kl': 0.01859803544357419, 'entropy': 0.25647516176104546, 'clip_ratio/low_mean': 0.001638104789890349, 'clip_ratio/low_min': 0.001638104789890349, 'clip_ratio/high_mean': 0.004315734549891204, 'clip_ratio/high_max': 0.004315734549891204, 'clip_ratio/region_mean': 0.0059538393397815526, 'step_time': 1.8624855800007936, 'epoch': 0.08}
{'loss': 0.3576, 'grad_norm': 0.3870702087879181, 'learning_rate': 2.80174367027059e-06, 'num_tokens': 1993245.0, 'completions/mean_length': 1408.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3895000219345093, 'rewards/game_reward_func/std': 0.25456684827804565, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3895000219345093, 'reward_std': 0.25456684827804565, 'frac_reward_zero_std': 0.0, 'kl': 0.02207787986844778, 'entropy': 0.23298294842243195, 'clip_ratio/low_mean': 0.009409978985786438, 'clip_ratio/low_min': 0.009409978985786438, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.009409978985786438, 'step_time': 18.395229415997164, 'epoch': 0.08}
{'loss': -0.3506, 'grad_norm': 0.8169320225715637, 'learning_rate': 2.772996397836704e-06, 'kl': 0.027258400805294514, 'entropy': 0.24761397391557693, 'clip_ratio/low_mean': 0.007848011679016054, 'clip_ratio/low_min': 0.007848011679016054, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.009801136679016054, 'step_time': 1.8352505520088016, 'epoch': 0.08}
{'loss': -0.2452, 'grad_norm': 0.3484136164188385, 'learning_rate': 2.7443406687983267e-06, 'num_tokens': 2002293.0, 'completions/mean_length': 952.0, 'completions/min_length': 64.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.437749981880188, 'rewards/game_reward_func/std': 0.08272113651037216, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.437749981880188, 'reward_std': 0.08272112905979156, 'frac_reward_zero_std': 0.0, 'kl': 0.04098095232620835, 'entropy': 0.21891586109995842, 'clip_ratio/low_mean': 0.009765625, 'clip_ratio/low_min': 0.009765625, 'clip_ratio/high_mean': 0.004947916720993817, 'clip_ratio/high_max': 0.004947916720993817, 'clip_ratio/region_mean': 0.014713541720993817, 'step_time': 16.162291510001523, 'epoch': 0.08}
{'loss': 0.2571, 'grad_norm': 0.8805883526802063, 'learning_rate': 2.7157776610802416e-06, 'kl': 0.06294362433254719, 'entropy': 0.3486936539411545, 'clip_ratio/low_mean': 0.005859375, 'clip_ratio/low_min': 0.005859375, 'clip_ratio/high_mean': 0.01999080879613757, 'clip_ratio/high_max': 0.01999080879613757, 'clip_ratio/region_mean': 0.02585018379613757, 'step_time': 1.8266277839793474, 'epoch': 0.08}
{'loss': 0.613, 'grad_norm': 0.9991276264190674, 'learning_rate': 2.687308548795825e-06, 'num_tokens': 2014733.0, 'completions/mean_length': 1376.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3960000276565552, 'rewards/game_reward_func/std': 0.14082208275794983, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3960000276565552, 'reward_std': 0.14082208275794983, 'frac_reward_zero_std': 0.0, 'kl': 0.026754408376291394, 'entropy': 0.24336044117808342, 'clip_ratio/low_mean': 0.010656696977093816, 'clip_ratio/low_min': 0.010656696977093816, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.010656696977093816, 'step_time': 18.20815905999916, 'epoch': 0.08}

 66%|██████▋   | 332/500 [54:48<23:52,  8.53s/it][2026-03-26 13:04:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:05:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.8

 67%|██████▋   | 333/500 [55:07<31:56, 11.48s/it]
                                                 

 67%|██████▋   | 333/500 [55:07<31:56, 11.48s/it]
 67%|██████▋   | 334/500 [55:09<24:09,  8.73s/it]
                                                 

 67%|██████▋   | 334/500 [55:09<24:09,  8.73s/it][2026-03-26 13:05:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:05:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 67%|██████▋   | 335/500 [55:27<31:50, 11.58s/it]
                                                 

 67%|██████▋   | 335/500 [55:27<31:50, 11.58s/it]
 67%|██████▋   | 336/500 [55:29<23:48,  8.71s/it]
                                                 

 67%|██████▋   | 336/500 [55:29<23:48,  8.71s/it][2026-03-26 13:05:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:05:44] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=16.6

 67%|██████▋   | 337/500 [55:47<30:36, 11.27s/it]
                                                 

 67%|██████▋   | 337/500 [55:47<30:36, 11.27s/it]
 68%|██████▊   | 338/500 [55:48<22:47,  8.44s/it]
                                                 

 68%|██████▊   | 338/500 [55:48<22:47,  8.44s/it][2026-03-26 13:05:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:06:04] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.6

 68%|██████▊   | 339/500 [56:07<30:29, 11.37s/it]
                                                 

 68%|██████▊   | 339/500 [56:07<30:29, 11.37s/it]
 68%|██████▊   | 340/500 [56:09<23:07,  8.67s/it]
                                                 

 68%|██████▊   | 340/500 [56:09<23:07,  8.67s/it][2026-03-26 13:06:11] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:06:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=24.2

 68%|██████▊   | 341/500 [56:27<30:34, 11.54s/it]
                                                 

 68%|██████▊   | 341/500 [56:27<30:34, 11.54s/it]
 68%|██████▊   | 342/500 [56:29<22:57,  8.72s/it]
                                                 

 68%|██████▊   | 342/500 [56:29<22:57,  8.72s/it][2026-03-26 13:06:31] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:06:45] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=24.0

 69%|██████▊   | 343/500 [56:48<30:30, 11.66s/it]
                                                 
{'loss': -0.6072, 'grad_norm': 0.81817626953125, 'learning_rate': 2.6589345021987725e-06, 'kl': 0.02577045140787959, 'entropy': 0.23196546733379364, 'clip_ratio/low_mean': 0.0010127314599230886, 'clip_ratio/low_min': 0.0010127314599230886, 'clip_ratio/high_mean': 0.00714789517223835, 'clip_ratio/high_max': 0.00714789517223835, 'clip_ratio/region_mean': 0.008160626632161438, 'step_time': 1.7450225669963402, 'epoch': 0.08}
{'loss': -0.3007, 'grad_norm': 0.755142331123352, 'learning_rate': 2.6306566876350072e-06, 'num_tokens': 2027301.0, 'completions/mean_length': 1392.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.29749998450279236, 'rewards/game_reward_func/std': 0.22629879415035248, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.29749998450279236, 'reward_std': 0.22629879415035248, 'frac_reward_zero_std': 0.0, 'kl': 0.02205155324190855, 'entropy': 0.24596551060676575, 'clip_ratio/low_mean': 0.0016526442486792803, 'clip_ratio/low_min': 0.0016526442486792803, 'clip_ratio/high_mean': 0.008169543696567416, 'clip_ratio/high_max': 0.008169543696567416, 'clip_ratio/region_mean': 0.009822187945246696, 'step_time': 18.170510022006056, 'epoch': 0.08}
{'loss': 0.3061, 'grad_norm': 0.5652704834938049, 'learning_rate': 2.6024762674947313e-06, 'kl': 0.026241149753332138, 'entropy': 0.21577683463692665, 'clip_ratio/low_mean': 0.007915295893326402, 'clip_ratio/low_min': 0.007915295893326402, 'clip_ratio/high_mean': 0.002685546875, 'clip_ratio/high_max': 0.002685546875, 'clip_ratio/region_mean': 0.010600842768326402, 'step_time': 2.1445055729927844, 'epoch': 0.08}
{'loss': -0.1267, 'grad_norm': 0.29954561591148376, 'learning_rate': 2.5743944001646394e-06, 'num_tokens': 2039293.0, 'completions/mean_length': 1320.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.46824997663497925, 'rewards/game_reward_func/std': 0.14497463405132294, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.46824997663497925, 'reward_std': 0.14497463405132294, 'frac_reward_zero_std': 0.0, 'kl': 0.041945003904402256, 'entropy': 0.2631969526410103, 'clip_ratio/low_mean': 0.015787760261446238, 'clip_ratio/low_min': 0.015787760261446238, 'clip_ratio/high_mean': 0.0009765625, 'clip_ratio/high_max': 0.0009765625, 'clip_ratio/region_mean': 0.016764322761446238, 'step_time': 18.048894155006565, 'epoch': 0.08}
{'loss': 0.1354, 'grad_norm': 0.17866206169128418, 'learning_rate': 2.5464122399803126e-06, 'kl': 0.01679236232303083, 'entropy': 0.23532576486468315, 'clip_ratio/low_mean': 0.0048928981414064765, 'clip_ratio/low_min': 0.0048928981414064765, 'clip_ratio/high_mean': 0.00341796875, 'clip_ratio/high_max': 0.00341796875, 'clip_ratio/region_mean': 0.008310866891406476, 'step_time': 1.840365701005794, 'epoch': 0.08}
{'loss': -0.4155, 'grad_norm': 2.791098117828369, 'learning_rate': 2.5185309371787515e-06, 'num_tokens': 2049300.0, 'completions/mean_length': 1071.875, 'completions/min_length': 64.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3607499897480011, 'rewards/game_reward_func/std': 0.19223478436470032, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3607499897480011, 'reward_std': 0.19223476946353912, 'frac_reward_zero_std': 0.0, 'kl': 0.03030424378812313, 'entropy': 0.3451201021671295, 'clip_ratio/low_mean': 0.007240372360683978, 'clip_ratio/low_min': 0.007240372360683978, 'clip_ratio/high_mean': 0.011462135007604957, 'clip_ratio/high_max': 0.011462135007604957, 'clip_ratio/region_mean': 0.018702507368288934, 'step_time': 17.029437934004818, 'epoch': 0.08}
{'loss': 0.4079, 'grad_norm': 0.38732418417930603, 'learning_rate': 2.4907516378511137e-06, 'kl': 0.04050979437306523, 'entropy': 0.24577968567609787, 'clip_ratio/low_mean': 0.00719391496386379, 'clip_ratio/low_min': 0.00719391496386379, 'clip_ratio/high_mean': 0.0065104165114462376, 'clip_ratio/high_max': 0.0065104165114462376, 'clip_ratio/region_mean': 0.013704331475310028, 'step_time': 1.6892060199970729, 'epoch': 0.08}
{'loss': -0.0156, 'grad_norm': 0.5853883624076843, 'learning_rate': 2.46307548389559e-06, 'num_tokens': 2060780.0, 'completions/mean_length': 1256.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.33375000953674316, 'rewards/game_reward_func/std': 0.13987112045288086, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.33375000953674316, 'reward_std': 0.13987110555171967, 'frac_reward_zero_std': 0.0, 'kl': 0.023319246247410774, 'entropy': 0.22884277999401093, 'clip_ratio/low_mean': 0.005888967891223729, 'clip_ratio/low_min': 0.005888967891223729, 'clip_ratio/high_mean': 0.003374023362994194, 'clip_ratio/high_max': 0.003374023362994194, 'clip_ratio/region_mean': 0.009262991254217923, 'step_time': 17.985719739008346, 'epoch': 0.08}
{'loss': 0.0236, 'grad_norm': 0.23999543488025665, 'learning_rate': 2.43550361297047e-06, 'kl': 0.03694783849641681, 'entropy': 0.27553990483283997, 'clip_ratio/low_mean': 0.010579427005723119, 'clip_ratio/low_min': 0.010579427005723119, 'clip_ratio/high_mean': 0.00592113402672112, 'clip_ratio/high_max': 0.00592113402672112, 'clip_ratio/region_mean': 0.01650056103244424, 'step_time': 1.9356583539993153, 'epoch': 0.09}
{'loss': -0.2518, 'grad_norm': 0.3931756615638733, 'learning_rate': 2.408037158447375e-06, 'num_tokens': 2074628.0, 'completions/mean_length': 1552.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.343999981880188, 'rewards/game_reward_func/std': 0.19990569353103638, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.343999981880188, 'reward_std': 0.19990569353103638, 'frac_reward_zero_std': 0.0, 'kl': 0.01804250292479992, 'entropy': 0.24256493151187897, 'clip_ratio/low_mean': 0.003173828125, 'clip_ratio/low_min': 0.003173828125, 'clip_ratio/high_mean': 0.0032958984375, 'clip_ratio/high_max': 0.0032958984375, 'clip_ratio/region_mean': 0.0064697265625, 'step_time': 18.083175084997492, 'epoch': 0.09}
{'loss': 0.2576, 'grad_norm': 0.5479761362075806, 'learning_rate': 2.3806772493646725e-06, 'kl': 0.024431786965578794, 'entropy': 0.23681015148758888, 'clip_ratio/low_mean': 0.00555889424867928, 'clip_ratio/low_min': 0.00555889424867928, 'clip_ratio/high_mean': 0.005446300376206636, 'clip_ratio/high_max': 0.005446300376206636, 'clip_ratio/region_mean': 0.011005194624885917, 'step_time': 1.933583361998899, 'epoch': 0.09}

 69%|██████▊   | 343/500 [56:48<30:30, 11.66s/it]
 69%|██████▉   | 344/500 [56:50<22:47,  8.77s/it]
                                                 

 69%|██████▉   | 344/500 [56:50<22:47,  8.77s/it][2026-03-26 13:06:52] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:07:05] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.4

 69%|██████▉   | 345/500 [57:08<29:58, 11.60s/it]
                                                 

 69%|██████▉   | 345/500 [57:08<29:58, 11.60s/it]
 69%|██████▉   | 346/500 [57:10<22:29,  8.76s/it]
                                                 

 69%|██████▉   | 346/500 [57:10<22:29,  8.76s/it][2026-03-26 13:07:12] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:07:25] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.0

 69%|██████▉   | 347/500 [57:28<29:27, 11.55s/it]
                                                 

 69%|██████▉   | 347/500 [57:28<29:27, 11.55s/it]
 70%|██████▉   | 348/500 [57:31<22:12,  8.76s/it]
                                                 

 70%|██████▉   | 348/500 [57:31<22:12,  8.76s/it][2026-03-26 13:07:32] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:07:42] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=14.5

 70%|██████▉   | 349/500 [57:44<25:33, 10.15s/it]
                                                 

 70%|██████▉   | 349/500 [57:44<25:33, 10.15s/it]
 70%|███████   | 350/500 [57:46<18:59,  7.60s/it]
                                                 

 70%|███████   | 350/500 [57:46<18:59,  7.60s/it][2026-03-26 13:07:48] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:08:01] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=18.9

 70%|███████   | 351/500 [58:04<26:59, 10.87s/it]
                                                 

 70%|███████   | 351/500 [58:04<26:59, 10.87s/it]
 70%|███████   | 352/500 [58:06<20:15,  8.21s/it]
                                                 

 70%|███████   | 352/500 [58:06<20:15,  8.21s/it][2026-03-26 13:08:08] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:08:22] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=21.4

 71%|███████   | 353/500 [58:24<27:31, 11.23s/it]
                                                 

 71%|███████   | 353/500 [58:24<27:31, 11.23s/it]
 71%|███████   | 354/500 [58:27<20:44,  8.53s/it]
                                                 
{'loss': 0.0614, 'grad_norm': 0.14701591432094574, 'learning_rate': 2.353425010381063e-06, 'num_tokens': 2088348.0, 'completions/mean_length': 1536.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.30699998140335083, 'rewards/game_reward_func/std': 0.23858632147312164, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.30699998140335083, 'reward_std': 0.23858630657196045, 'frac_reward_zero_std': 0.0, 'kl': 0.021332051837816834, 'entropy': 0.2612199932336807, 'clip_ratio/low_mean': 0.001461429230403155, 'clip_ratio/low_min': 0.001461429230403155, 'clip_ratio/high_mean': 0.0060628256760537624, 'clip_ratio/high_max': 0.0060628256760537624, 'clip_ratio/region_mean': 0.0075242549064569175, 'step_time': 18.337008494992915, 'epoch': 0.09}
{'loss': -0.0557, 'grad_norm': 0.24759796261787415, 'learning_rate': 2.3262815617293517e-06, 'kl': 0.01379805919714272, 'entropy': 0.24848361685872078, 'clip_ratio/low_mean': 0.004076363518834114, 'clip_ratio/low_min': 0.004076363518834114, 'clip_ratio/high_mean': 0.0008544921875, 'clip_ratio/high_max': 0.0008544921875, 'clip_ratio/region_mean': 0.004930855706334114, 'step_time': 1.8397782080064644, 'epoch': 0.09}
{'loss': -0.3757, 'grad_norm': 0.5741221308708191, 'learning_rate': 2.2992480191704003e-06, 'num_tokens': 2100212.0, 'completions/mean_length': 1304.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3162500262260437, 'rewards/game_reward_func/std': 0.18346954882144928, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3162500262260437, 'reward_std': 0.18346954882144928, 'frac_reward_zero_std': 0.0, 'kl': 0.024853709619492292, 'entropy': 0.2557961866259575, 'clip_ratio/low_mean': 0.005221485625952482, 'clip_ratio/low_min': 0.005221485625952482, 'clip_ratio/high_mean': 0.006016322411596775, 'clip_ratio/high_max': 0.006016322411596775, 'clip_ratio/region_mean': 0.011237808037549257, 'step_time': 18.058670597994933, 'epoch': 0.09}
{'loss': 0.3793, 'grad_norm': 1.4466511011123657, 'learning_rate': 2.272325493947257e-06, 'kl': 0.024952402338385582, 'entropy': 0.23270966112613678, 'clip_ratio/low_mean': 0.00847353576682508, 'clip_ratio/low_min': 0.00847353576682508, 'clip_ratio/high_mean': 0.0021033654920756817, 'clip_ratio/high_max': 0.0021033654920756817, 'clip_ratio/region_mean': 0.010576901258900762, 'step_time': 1.934688000997994, 'epoch': 0.09}
{'loss': -0.0787, 'grad_norm': 0.35457438230514526, 'learning_rate': 2.245515092739488e-06, 'num_tokens': 2112396.0, 'completions/mean_length': 1344.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.39149999618530273, 'rewards/game_reward_func/std': 0.18876364827156067, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.39149999618530273, 'reward_std': 0.18876364827156067, 'frac_reward_zero_std': 0.0, 'kl': 0.023530170088633895, 'entropy': 0.25681497529149055, 'clip_ratio/low_mean': 0.0033919475972652435, 'clip_ratio/low_min': 0.0033919475972652435, 'clip_ratio/high_mean': 0.0044531249441206455, 'clip_ratio/high_max': 0.0044531249441206455, 'clip_ratio/region_mean': 0.007845072541385889, 'step_time': 17.86836194100033, 'epoch': 0.09}
{'loss': 0.0877, 'grad_norm': 0.1866183578968048, 'learning_rate': 2.2188179176176767e-06, 'kl': 0.024016158189624548, 'entropy': 0.26513347029685974, 'clip_ratio/low_mean': 0.0036718749906867743, 'clip_ratio/low_min': 0.0036718749906867743, 'clip_ratio/high_mean': 0.010825893143191934, 'clip_ratio/high_max': 0.010825893143191934, 'clip_ratio/region_mean': 0.014497768133878708, 'step_time': 2.072044853004627, 'epoch': 0.09}
{'loss': -0.0665, 'grad_norm': 1.1937735080718994, 'learning_rate': 2.1922350659981262e-06, 'num_tokens': 2121252.0, 'completions/mean_length': 928.0, 'completions/min_length': 384.0, 'completions/max_length': 1536.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4390000104904175, 'rewards/game_reward_func/std': 0.1484087109565735, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4390000104904175, 'reward_std': 0.1484086960554123, 'frac_reward_zero_std': 0.0, 'kl': 0.038655143696814775, 'entropy': 0.2805517949163914, 'clip_ratio/low_mean': 0.0058283731341362, 'clip_ratio/low_min': 0.0058283731341362, 'clip_ratio/high_mean': 0.012044270988553762, 'clip_ratio/high_max': 0.012044270988553762, 'clip_ratio/region_mean': 0.017872644122689962, 'step_time': 13.283508120002807, 'epoch': 0.09}
{'loss': 0.0723, 'grad_norm': 0.22383837401866913, 'learning_rate': 2.165767630597752e-06, 'kl': 0.025114331860095263, 'entropy': 0.2852625921368599, 'clip_ratio/low_mean': 0.008593750186264515, 'clip_ratio/low_min': 0.008593750186264515, 'clip_ratio/high_mean': 0.0031250000465661287, 'clip_ratio/high_max': 0.0031250000465661287, 'clip_ratio/region_mean': 0.011718750232830644, 'step_time': 1.4784397000039462, 'epoch': 0.09}
{'loss': 0.3794, 'grad_norm': 0.9806253910064697, 'learning_rate': 2.139416699389153e-06, 'num_tokens': 2132348.0, 'completions/mean_length': 1208.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.35074999928474426, 'rewards/game_reward_func/std': 0.16368241608142853, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.35074999928474426, 'reward_std': 0.16368240118026733, 'frac_reward_zero_std': 0.0, 'kl': 0.03198112128302455, 'entropy': 0.24329522252082825, 'clip_ratio/low_mean': 0.00546499399933964, 'clip_ratio/low_min': 0.00546499399933964, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.00937124399933964, 'step_time': 17.946443551998527, 'epoch': 0.09}
{'loss': -0.3673, 'grad_norm': 0.6559845209121704, 'learning_rate': 2.1131833555559037e-06, 'kl': 0.039682835107669234, 'entropy': 0.27439989522099495, 'clip_ratio/low_mean': 0.008984375046566129, 'clip_ratio/low_min': 0.008984375046566129, 'clip_ratio/high_mean': 0.009765625, 'clip_ratio/high_max': 0.009765625, 'clip_ratio/region_mean': 0.01875000004656613, 'step_time': 1.8387398980048602, 'epoch': 0.09}
{'loss': 0.2546, 'grad_norm': 0.5094444155693054, 'learning_rate': 2.08706867744802e-06, 'num_tokens': 2144724.0, 'completions/mean_length': 1368.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2822500169277191, 'rewards/game_reward_func/std': 0.25169697403907776, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2822500169277191, 'reward_std': 0.25169697403907776, 'frac_reward_zero_std': 0.0, 'kl': 0.027503154007717967, 'entropy': 0.2751697450876236, 'clip_ratio/low_mean': 0.00427592039341107, 'clip_ratio/low_min': 0.00427592039341107, 'clip_ratio/high_mean': 0.0010986328125, 'clip_ratio/high_max': 0.0010986328125, 'clip_ratio/region_mean': 0.00537455320591107, 'step_time': 18.100148155004717, 'epoch': 0.09}

 71%|███████   | 354/500 [58:27<20:44,  8.53s/it][2026-03-26 13:08:28] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:08:42] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=15.8

 71%|███████   | 355/500 [58:44<27:20, 11.32s/it]
                                                 

 71%|███████   | 355/500 [58:44<27:20, 11.32s/it]
 71%|███████   | 356/500 [58:46<20:27,  8.53s/it]
                                                 

 71%|███████   | 356/500 [58:46<20:27,  8.53s/it][2026-03-26 13:08:48] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:09:02] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.2

 71%|███████▏  | 357/500 [59:05<27:14, 11.43s/it]
                                                 

 71%|███████▏  | 357/500 [59:05<27:14, 11.43s/it]
 72%|███████▏  | 358/500 [59:07<20:26,  8.64s/it]
                                                 

 72%|███████▏  | 358/500 [59:07<20:26,  8.64s/it][2026-03-26 13:09:08] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:09:23] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.4

 72%|███████▏  | 359/500 [59:25<27:20, 11.63s/it]
                                                 

 72%|███████▏  | 359/500 [59:25<27:20, 11.63s/it]
 72%|███████▏  | 360/500 [59:27<20:24,  8.75s/it]
                                                 

 72%|███████▏  | 360/500 [59:27<20:24,  8.75s/it][2026-03-26 13:09:29] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:09:43] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.8

 72%|███████▏  | 361/500 [59:46<27:10, 11.73s/it]
                                                 

 72%|███████▏  | 361/500 [59:46<27:10, 11.73s/it]
 72%|███████▏  | 362/500 [59:48<20:16,  8.82s/it]
                                                 

 72%|███████▏  | 362/500 [59:48<20:16,  8.82s/it][2026-03-26 13:09:50] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:10:03] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=15.5

 73%|███████▎  | 363/500 [1:00:06<26:12, 11.47s/it]
                                                   

 73%|███████▎  | 363/500 [1:00:06<26:12, 11.47s/it]
 73%|███████▎  | 364/500 [1:00:08<19:38,  8.67s/it]
                                                   

 73%|███████▎  | 364/500 [1:00:08<19:38,  8.67s/it][2026-03-26 13:10:10] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:10:20] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:7 D:0 (12% win) | avg_turns=13.6

 73%|███████▎  | 365/500 [1:00:22<23:17, 10.35s/it]
                                                   
{'loss': -0.2476, 'grad_norm': 0.359628826379776, 'learning_rate': 2.061073738537635e-06, 'kl': 0.017832791432738304, 'entropy': 0.2592063657939434, 'clip_ratio/low_mean': 0.004956794669851661, 'clip_ratio/low_min': 0.004956794669851661, 'clip_ratio/high_mean': 0.00634765625, 'clip_ratio/high_max': 0.00634765625, 'clip_ratio/region_mean': 0.01130445091985166, 'step_time': 2.0301397739967797, 'epoch': 0.09}
{'loss': 0.2006, 'grad_norm': 0.13613079488277435, 'learning_rate': 2.0351996073748713e-06, 'num_tokens': 2154220.0, 'completions/mean_length': 1008.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4059999883174896, 'rewards/game_reward_func/std': 0.21387581527233124, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4059999883174896, 'reward_std': 0.21387580037117004, 'frac_reward_zero_std': 0.0, 'kl': 0.0395058230496943, 'entropy': 0.2886006496846676, 'clip_ratio/low_mean': 0.008426339365541935, 'clip_ratio/low_min': 0.008426339365541935, 'clip_ratio/high_mean': 0.0032552082557231188, 'clip_ratio/high_max': 0.0032552082557231188, 'clip_ratio/region_mean': 0.011681547621265054, 'step_time': 17.63156503999926, 'epoch': 0.09}
{'loss': -0.1921, 'grad_norm': 0.4407423734664917, 'learning_rate': 2.00944734754392e-06, 'kl': 0.024475280195474625, 'entropy': 0.2676215283572674, 'clip_ratio/low_mean': 0.003988486947491765, 'clip_ratio/low_min': 0.003988486947491765, 'clip_ratio/high_mean': 0.006527171470224857, 'clip_ratio/high_max': 0.006527171470224857, 'clip_ratio/region_mean': 0.010515658417716622, 'step_time': 1.8367616760006058, 'epoch': 0.09}
{'loss': -0.6381, 'grad_norm': 0.7936199903488159, 'learning_rate': 1.983818017619318e-06, 'num_tokens': 2166532.0, 'completions/mean_length': 1360.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37150001525878906, 'rewards/game_reward_func/std': 0.1875031441450119, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37150001525878906, 'reward_std': 0.1875031292438507, 'frac_reward_zero_std': 0.0, 'kl': 0.01986829936504364, 'entropy': 0.26405827701091766, 'clip_ratio/low_mean': 0.002697172691114247, 'clip_ratio/low_min': 0.002697172691114247, 'clip_ratio/high_mean': 0.0048828125, 'clip_ratio/high_max': 0.0048828125, 'clip_ratio/region_mean': 0.007579985191114247, 'step_time': 18.043352288994356, 'epoch': 0.09}
{'loss': 0.6445, 'grad_norm': 1.026626467704773, 'learning_rate': 1.9583126711224342e-06, 'kl': 0.027183874510228634, 'entropy': 0.24544226378202438, 'clip_ratio/low_mean': 0.008638041792437434, 'clip_ratio/low_min': 0.008638041792437434, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.008638041792437434, 'step_time': 1.9337880760067492, 'epoch': 0.09}
{'loss': 0.5186, 'grad_norm': 0.6629675030708313, 'learning_rate': 1.932932356478168e-06, 'num_tokens': 2178908.0, 'completions/mean_length': 1368.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.42000001668930054, 'rewards/game_reward_func/std': 0.12716694176197052, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.42000001668930054, 'reward_std': 0.12716694176197052, 'frac_reward_zero_std': 0.0, 'kl': 0.02225555945187807, 'entropy': 0.27246955409646034, 'clip_ratio/low_mean': 0.002706863160710782, 'clip_ratio/low_min': 0.002706863160710782, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.006613113160710782, 'step_time': 18.426971573993796, 'epoch': 0.09}
{'loss': -0.5112, 'grad_norm': 0.9699128270149231, 'learning_rate': 1.9076781169718426e-06, 'kl': 0.029990104725584388, 'entropy': 0.25973282381892204, 'clip_ratio/low_mean': 0.0009428879129700363, 'clip_ratio/low_min': 0.0009428879129700363, 'clip_ratio/high_mean': 0.01321698073297739, 'clip_ratio/high_max': 0.01321698073297739, 'clip_ratio/region_mean': 0.014159868645947427, 'step_time': 1.8342517030032468, 'epoch': 0.09}
{'loss': 0.364, 'grad_norm': 0.7196502089500427, 'learning_rate': 1.8825509907063328e-06, 'num_tokens': 2191476.0, 'completions/mean_length': 1392.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4240000247955322, 'rewards/game_reward_func/std': 0.16352196037769318, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4240000247955322, 'reward_std': 0.16352196037769318, 'frac_reward_zero_std': 0.0, 'kl': 0.022789191687479615, 'entropy': 0.2563805542886257, 'clip_ratio/low_mean': 0.007911571534350514, 'clip_ratio/low_min': 0.007911571534350514, 'clip_ratio/high_mean': 0.002828663680702448, 'clip_ratio/high_max': 0.002828663680702448, 'clip_ratio/region_mean': 0.010740235215052962, 'step_time': 18.506376546989486, 'epoch': 0.09}
{'loss': -0.3588, 'grad_norm': 0.6463605165481567, 'learning_rate': 1.857552010559382e-06, 'kl': 0.02516472525894642, 'entropy': 0.27737167477607727, 'clip_ratio/low_mean': 0.0028409091755747795, 'clip_ratio/low_min': 0.0028409091755747795, 'clip_ratio/high_mean': 0.0079345703125, 'clip_ratio/high_max': 0.0079345703125, 'clip_ratio/region_mean': 0.01077547948807478, 'step_time': 1.842618886999844, 'epoch': 0.09}
{'loss': -0.69, 'grad_norm': 1.1513458490371704, 'learning_rate': 1.8326822041411524e-06, 'num_tokens': 2200844.0, 'completions/mean_length': 992.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3154999911785126, 'rewards/game_reward_func/std': 0.2067621350288391, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3154999911785126, 'reward_std': 0.20676209032535553, 'frac_reward_zero_std': 0.0, 'kl': 0.03181964484974742, 'entropy': 0.25372743606567383, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.011811523465439677, 'clip_ratio/high_max': 0.011811523465439677, 'clip_ratio/region_mean': 0.011811523465439677, 'step_time': 17.49621318300342, 'epoch': 0.09}
{'loss': 0.6969, 'grad_norm': 1.231480360031128, 'learning_rate': 1.8079425937519729e-06, 'kl': 0.049401501193642616, 'entropy': 0.27565161511301994, 'clip_ratio/low_mean': 0.01806640625, 'clip_ratio/low_min': 0.01806640625, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.01806640625, 'step_time': 1.9266430780044175, 'epoch': 0.09}

 73%|███████▎  | 365/500 [1:00:22<23:17, 10.35s/it]
 73%|███████▎  | 366/500 [1:00:24<17:20,  7.77s/it]
                                                   

 73%|███████▎  | 366/500 [1:00:24<17:20,  7.77s/it][2026-03-26 13:10:26] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:10:38] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.9

 73%|███████▎  | 367/500 [1:00:41<23:22, 10.54s/it]
                                                   

 73%|███████▎  | 367/500 [1:00:41<23:22, 10.54s/it]
 74%|███████▎  | 368/500 [1:00:43<17:35,  8.00s/it]
                                                   

 74%|███████▎  | 368/500 [1:00:43<17:35,  8.00s/it][2026-03-26 13:10:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:10:59] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.4

 74%|███████▍  | 369/500 [1:01:01<24:18, 11.13s/it]
                                                   

 74%|███████▍  | 369/500 [1:01:01<24:18, 11.13s/it]
 74%|███████▍  | 370/500 [1:01:03<18:12,  8.40s/it]
                                                   

 74%|███████▍  | 370/500 [1:01:03<18:12,  8.40s/it][2026-03-26 13:11:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:11:19] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.4

 74%|███████▍  | 371/500 [1:01:22<24:30, 11.40s/it]
                                                   

 74%|███████▍  | 371/500 [1:01:22<24:30, 11.40s/it]
 74%|███████▍  | 372/500 [1:01:24<18:18,  8.58s/it]
                                                   

 74%|███████▍  | 372/500 [1:01:24<18:18,  8.58s/it][2026-03-26 13:11:25] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:11:40] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=24.1

 75%|███████▍  | 373/500 [1:01:42<24:24, 11.53s/it]
                                                   

 75%|███████▍  | 373/500 [1:01:42<24:24, 11.53s/it]
 75%|███████▍  | 374/500 [1:01:44<18:09,  8.65s/it]
                                                   

 75%|███████▍  | 374/500 [1:01:44<18:09,  8.65s/it][2026-03-26 13:11:46] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:12:00] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=26.6

 75%|███████▌  | 375/500 [1:02:03<24:16, 11.65s/it]
                                                   

 75%|███████▌  | 375/500 [1:02:03<24:16, 11.65s/it]
 75%|███████▌  | 376/500 [1:02:05<18:15,  8.83s/it]
                                                   
{'loss': 0.2706, 'grad_norm': 0.28328993916511536, 'learning_rate': 1.7833341963403312e-06, 'num_tokens': 2209252.0, 'completions/mean_length': 872.0, 'completions/min_length': 384.0, 'completions/max_length': 1664.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2672500014305115, 'rewards/game_reward_func/std': 0.4341651201248169, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2672500014305115, 'reward_std': 0.4341651201248169, 'frac_reward_zero_std': 0.0, 'kl': 0.03415539534762502, 'entropy': 0.30375317484140396, 'clip_ratio/low_mean': 0.007723141461610794, 'clip_ratio/low_min': 0.007723141461610794, 'clip_ratio/high_mean': 0.005078124813735485, 'clip_ratio/high_max': 0.005078124813735485, 'clip_ratio/region_mean': 0.01280126627534628, 'step_time': 14.137246211001184, 'epoch': 0.09}
{'loss': -0.2616, 'grad_norm': 1.6517057418823242, 'learning_rate': 1.7588580234610592e-06, 'kl': 0.06532103382050991, 'entropy': 0.2610364519059658, 'clip_ratio/low_mean': 0.015676398063078523, 'clip_ratio/low_min': 0.015676398063078523, 'clip_ratio/high_mean': 0.0042968750931322575, 'clip_ratio/high_max': 0.0042968750931322575, 'clip_ratio/region_mean': 0.01997327315621078, 'step_time': 1.5765180730013526, 'epoch': 0.09}
{'loss': 0.1065, 'grad_norm': 0.5590525269508362, 'learning_rate': 1.7345150812337564e-06, 'num_tokens': 2219836.0, 'completions/mean_length': 1144.0, 'completions/min_length': 320.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.45750001072883606, 'rewards/game_reward_func/std': 0.08745775371789932, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.45750001072883606, 'reward_std': 0.08745776116847992, 'frac_reward_zero_std': 0.0, 'kl': 0.025580907939001918, 'entropy': 0.21648409590125084, 'clip_ratio/low_mean': 0.0033482143189758062, 'clip_ratio/low_min': 0.0033482143189758062, 'clip_ratio/high_mean': 0.009954637149348855, 'clip_ratio/high_max': 0.009954637149348855, 'clip_ratio/region_mean': 0.013302851468324661, 'step_time': 16.851544295997883, 'epoch': 0.09}
{'loss': -0.0993, 'grad_norm': 0.34814658761024475, 'learning_rate': 1.7103063703014372e-06, 'kl': 0.027512364089488983, 'entropy': 0.26991357654333115, 'clip_ratio/low_mean': 0.0016447368543595076, 'clip_ratio/low_min': 0.0016447368543595076, 'clip_ratio/high_mean': 0.015516493236646056, 'clip_ratio/high_max': 0.015516493236646056, 'clip_ratio/region_mean': 0.017161230091005564, 'step_time': 1.868655932994443, 'epoch': 0.09}
{'loss': -0.1481, 'grad_norm': 14.72995662689209, 'learning_rate': 1.6862328857893856e-06, 'num_tokens': 2231700.0, 'completions/mean_length': 1304.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.31975001096725464, 'rewards/game_reward_func/std': 0.19148871302604675, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.31975001096725464, 'reward_std': 0.19148872792720795, 'frac_reward_zero_std': 0.0, 'kl': 0.1971161118708551, 'entropy': 0.257837675511837, 'clip_ratio/low_mean': 0.006051060918252915, 'clip_ratio/low_min': 0.006051060918252915, 'clip_ratio/high_mean': 0.0038285688497126102, 'clip_ratio/high_max': 0.0038285688497126102, 'clip_ratio/region_mean': 0.009879629767965525, 'step_time': 18.27543229600269, 'epoch': 0.09}
{'loss': 0.1739, 'grad_norm': 0.7903316020965576, 'learning_rate': 1.6622956172642601e-06, 'kl': 0.03440267965197563, 'entropy': 0.27369947731494904, 'clip_ratio/low_mean': 0.00846726621966809, 'clip_ratio/low_min': 0.00846726621966809, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.01042039121966809, 'step_time': 1.8451627230097074, 'epoch': 0.09}
{'loss': 0.4295, 'grad_norm': 0.405428946018219, 'learning_rate': 1.6384955486934157e-06, 'num_tokens': 2244076.0, 'completions/mean_length': 1368.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.42799997329711914, 'rewards/game_reward_func/std': 0.15812110900878906, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.42799997329711914, 'reward_std': 0.15812109410762787, 'frac_reward_zero_std': 0.0, 'kl': 0.024640012299641967, 'entropy': 0.24715323001146317, 'clip_ratio/low_mean': 0.004608294926583767, 'clip_ratio/low_min': 0.004608294926583767, 'clip_ratio/high_mean': 0.005709134507924318, 'clip_ratio/high_max': 0.005709134507924318, 'clip_ratio/region_mean': 0.010317429434508085, 'step_time': 18.188269126992964, 'epoch': 0.09}
{'loss': -0.4259, 'grad_norm': 0.42609572410583496, 'learning_rate': 1.6148336584044539e-06, 'kl': 0.023468466941267252, 'entropy': 0.2444527931511402, 'clip_ratio/low_mean': 0.009033203125, 'clip_ratio/low_min': 0.009033203125, 'clip_ratio/high_mean': 0.0057373046875, 'clip_ratio/high_max': 0.0057373046875, 'clip_ratio/region_mean': 0.0147705078125, 'step_time': 1.8354281630090554, 'epoch': 0.09}
{'loss': 0.228, 'grad_norm': 0.5393027663230896, 'learning_rate': 1.5913109190450033e-06, 'num_tokens': 2257860.0, 'completions/mean_length': 1544.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2887499928474426, 'rewards/game_reward_func/std': 0.2625870108604431, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2887499928474426, 'reward_std': 0.2625869810581207, 'frac_reward_zero_std': 0.0, 'kl': 0.024541332153603435, 'entropy': 0.23956499993801117, 'clip_ratio/low_mean': 0.010125113185495138, 'clip_ratio/low_min': 0.010125113185495138, 'clip_ratio/high_mean': 0.0020751953125, 'clip_ratio/high_max': 0.0020751953125, 'clip_ratio/region_mean': 0.012200308497995138, 'step_time': 18.244751907004684, 'epoch': 0.09}
{'loss': -0.2236, 'grad_norm': 0.32525697350502014, 'learning_rate': 1.567928297542749e-06, 'kl': 0.022837891709059477, 'entropy': 0.2155812606215477, 'clip_ratio/low_mean': 0.005273437476716936, 'clip_ratio/low_min': 0.005273437476716936, 'clip_ratio/high_mean': 0.0037841796875, 'clip_ratio/high_max': 0.0037841796875, 'clip_ratio/region_mean': 0.009057617164216936, 'step_time': 1.73951062298147, 'epoch': 0.09}
{'loss': -0.1092, 'grad_norm': 0.3894057273864746, 'learning_rate': 1.544686755065677e-06, 'num_tokens': 2272924.0, 'completions/mean_length': 1704.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3657500147819519, 'rewards/game_reward_func/std': 0.18953007459640503, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3657500147819519, 'reward_std': 0.18953005969524384, 'frac_reward_zero_std': 0.0, 'kl': 0.020377319306135178, 'entropy': 0.2313036099076271, 'clip_ratio/low_mean': 0.003128476091660559, 'clip_ratio/low_min': 0.003128476091660559, 'clip_ratio/high_mean': 0.0055745444260537624, 'clip_ratio/high_max': 0.0055745444260537624, 'clip_ratio/region_mean': 0.008703020517714322, 'step_time': 18.476587493008992, 'epoch': 0.09}

 75%|███████▌  | 376/500 [1:02:05<18:15,  8.83s/it][2026-03-26 13:12:07] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:12:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.1

 75%|███████▌  | 377/500 [1:02:20<22:07, 10.79s/it]
                                                   

 75%|███████▌  | 377/500 [1:02:20<22:07, 10.79s/it]
 76%|███████▌  | 378/500 [1:02:22<16:26,  8.09s/it]
                                                   

 76%|███████▌  | 378/500 [1:02:22<16:26,  8.09s/it][2026-03-26 13:12:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:12:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.1

 76%|███████▌  | 379/500 [1:02:38<20:47, 10.31s/it]
                                                   

 76%|███████▌  | 379/500 [1:02:38<20:47, 10.31s/it]
 76%|███████▌  | 380/500 [1:02:40<15:32,  7.77s/it]
                                                   

 76%|███████▌  | 380/500 [1:02:40<15:32,  7.77s/it][2026-03-26 13:12:41] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:12:55] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.2

 76%|███████▌  | 381/500 [1:02:58<21:39, 10.92s/it]
                                                   

 76%|███████▌  | 381/500 [1:02:58<21:39, 10.92s/it]
 76%|███████▋  | 382/500 [1:03:00<16:13,  8.25s/it]
                                                   

 76%|███████▋  | 382/500 [1:03:00<16:13,  8.25s/it][2026-03-26 13:13:02] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:13:15] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.9

 77%|███████▋  | 383/500 [1:03:18<21:53, 11.23s/it]
                                                   

 77%|███████▋  | 383/500 [1:03:18<21:53, 11.23s/it]
 77%|███████▋  | 384/500 [1:03:20<16:25,  8.50s/it]
                                                   

 77%|███████▋  | 384/500 [1:03:20<16:25,  8.50s/it][2026-03-26 13:13:22] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:13:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.2

 77%|███████▋  | 385/500 [1:03:38<21:53, 11.42s/it]
                                                   

 77%|███████▋  | 385/500 [1:03:38<21:53, 11.42s/it]
 77%|███████▋  | 386/500 [1:03:40<16:16,  8.57s/it]
                                                   

 77%|███████▋  | 386/500 [1:03:40<16:16,  8.57s/it][2026-03-26 13:13:42] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:13:56] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.8

 77%|███████▋  | 387/500 [1:03:59<21:34, 11.46s/it]
                                                   
{'loss': 0.1154, 'grad_norm': 0.12251555174589157, 'learning_rate': 1.5215872469825682e-06, 'kl': 0.01585030253045261, 'entropy': 0.23908702284097672, 'clip_ratio/low_mean': 0.00349934896803461, 'clip_ratio/low_min': 0.00349934896803461, 'clip_ratio/high_mean': 0.0045420179958455265, 'clip_ratio/high_max': 0.0045420179958455265, 'clip_ratio/region_mean': 0.008041366963880137, 'step_time': 2.0788795389962615, 'epoch': 0.09}
{'loss': -0.4305, 'grad_norm': 1.0199294090270996, 'learning_rate': 1.4986307228237268e-06, 'num_tokens': 2285172.0, 'completions/mean_length': 1352.0, 'completions/min_length': 896.0, 'completions/max_length': 1728.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.39424997568130493, 'rewards/game_reward_func/std': 0.09659599512815475, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.39424997568130493, 'reward_std': 0.09659599512815475, 'frac_reward_zero_std': 0.0, 'kl': 0.022996187210083008, 'entropy': 0.25633611530065536, 'clip_ratio/low_mean': 0.0017361111240461469, 'clip_ratio/low_min': 0.0017361111240461469, 'clip_ratio/high_mean': 0.005876736075151712, 'clip_ratio/high_max': 0.005876736075151712, 'clip_ratio/region_mean': 0.007612847199197859, 'step_time': 15.217957134009339, 'epoch': 0.09}
{'loss': 0.4397, 'grad_norm': 0.6552366614341736, 'learning_rate': 1.4758181262419425e-06, 'kl': 0.023449684493243694, 'entropy': 0.26180189847946167, 'clip_ratio/low_mean': 0.005789896356873214, 'clip_ratio/low_min': 0.005789896356873214, 'clip_ratio/high_mean': 0.005580357275903225, 'clip_ratio/high_max': 0.005580357275903225, 'clip_ratio/region_mean': 0.01137025363277644, 'step_time': 1.6239525450000656, 'epoch': 0.09}
{'loss': 0.0565, 'grad_norm': 0.9160023331642151, 'learning_rate': 1.4531503949737107e-06, 'num_tokens': 2295372.0, 'completions/mean_length': 1096.0, 'completions/min_length': 320.0, 'completions/max_length': 1792.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.5322500467300415, 'rewards/game_reward_func/std': 0.10273648053407669, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.5322500467300415, 'reward_std': 0.10273648053407669, 'frac_reward_zero_std': 0.0, 'kl': 0.030500881373882294, 'entropy': 0.24624697864055634, 'clip_ratio/low_mean': 0.0063159497221931815, 'clip_ratio/low_min': 0.0063159497221931815, 'clip_ratio/high_mean': 0.0054687499068677425, 'clip_ratio/high_max': 0.0054687499068677425, 'clip_ratio/region_mean': 0.011784699629060924, 'step_time': 15.328537758003222, 'epoch': 0.09}
{'loss': -0.0471, 'grad_norm': 0.5836339592933655, 'learning_rate': 1.4306284608006837e-06, 'kl': 0.042324821231886744, 'entropy': 0.2458001971244812, 'clip_ratio/low_mean': 0.0078125, 'clip_ratio/low_min': 0.0078125, 'clip_ratio/high_mean': 0.0067323625553399324, 'clip_ratio/high_max': 0.0067323625553399324, 'clip_ratio/region_mean': 0.014544862555339932, 'step_time': 1.6805169560029753, 'epoch': 0.1}
{'loss': 0.2944, 'grad_norm': 0.4822467863559723, 'learning_rate': 1.4082532495113627e-06, 'num_tokens': 2307684.0, 'completions/mean_length': 1360.0, 'completions/min_length': 832.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3499999940395355, 'rewards/game_reward_func/std': 0.17273597419261932, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3499999940395355, 'reward_std': 0.17273594439029694, 'frac_reward_zero_std': 0.0, 'kl': 0.02541993698105216, 'entropy': 0.25197651982307434, 'clip_ratio/low_mean': 0.005352492677047849, 'clip_ratio/low_min': 0.005352492677047849, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.009258742677047849, 'step_time': 18.081314676994225, 'epoch': 0.1}
{'loss': -0.2868, 'grad_norm': 0.6589658260345459, 'learning_rate': 1.3860256808630429e-06, 'kl': 0.023873369907960296, 'entropy': 0.2582089193165302, 'clip_ratio/low_mean': 0.0035511364694684744, 'clip_ratio/low_min': 0.0035511364694684744, 'clip_ratio/high_mean': 0.00623984495177865, 'clip_ratio/high_max': 0.00623984495177865, 'clip_ratio/region_mean': 0.009790981421247125, 'step_time': 1.8442915529958555, 'epoch': 0.1}
{'loss': 0.1348, 'grad_norm': 0.7518264651298523, 'learning_rate': 1.3639466685440133e-06, 'num_tokens': 2318268.0, 'completions/mean_length': 1144.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3682500123977661, 'rewards/game_reward_func/std': 0.16191864013671875, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3682500123977661, 'reward_std': 0.16191864013671875, 'frac_reward_zero_std': 0.0, 'kl': 0.02257026848383248, 'entropy': 0.24731066823005676, 'clip_ratio/low_mean': 0.005409595905803144, 'clip_ratio/low_min': 0.005409595905803144, 'clip_ratio/high_mean': 0.00439453125, 'clip_ratio/high_max': 0.00439453125, 'clip_ratio/region_mean': 0.009804127155803144, 'step_time': 17.997235774004366, 'epoch': 0.1}
{'loss': -0.1269, 'grad_norm': 0.4826048016548157, 'learning_rate': 1.3420171201359933e-06, 'kl': 0.03154592774808407, 'entropy': 0.266424510627985, 'clip_ratio/low_mean': 0.007685673423111439, 'clip_ratio/low_min': 0.007685673423111439, 'clip_ratio/high_mean': 0.0035328585654497147, 'clip_ratio/high_max': 0.0035328585654497147, 'clip_ratio/region_mean': 0.011218531988561153, 'step_time': 1.9288087259919848, 'epoch': 0.1}
{'loss': 0.5805, 'grad_norm': 1.1161065101623535, 'learning_rate': 1.3202379370768254e-06, 'num_tokens': 2329556.0, 'completions/mean_length': 1232.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37150001525878906, 'rewards/game_reward_func/std': 0.18921415507793427, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37150001525878906, 'reward_std': 0.18921416997909546, 'frac_reward_zero_std': 0.0, 'kl': 0.037325466284528375, 'entropy': 0.2732667811214924, 'clip_ratio/low_mean': 0.016214037546887994, 'clip_ratio/low_min': 0.016214037546887994, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.016214037546887994, 'step_time': 18.050565757002914, 'epoch': 0.1}
{'loss': -0.5737, 'grad_norm': 0.6306779384613037, 'learning_rate': 1.298610014623423e-06, 'kl': 0.029482323676347733, 'entropy': 0.2629450187087059, 'clip_ratio/low_mean': 0.006138392724096775, 'clip_ratio/low_min': 0.006138392724096775, 'clip_ratio/high_mean': 0.011791992001235485, 'clip_ratio/high_max': 0.011791992001235485, 'clip_ratio/region_mean': 0.01793038472533226, 'step_time': 1.738549513007456, 'epoch': 0.1}

 77%|███████▋  | 387/500 [1:03:59<21:34, 11.46s/it]
 78%|███████▊  | 388/500 [1:04:01<16:09,  8.66s/it]
                                                   

 78%|███████▊  | 388/500 [1:04:01<16:09,  8.66s/it][2026-03-26 13:14:02] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:14:15] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=22.5

 78%|███████▊  | 389/500 [1:04:18<20:34, 11.12s/it]
                                                   

 78%|███████▊  | 389/500 [1:04:18<20:34, 11.12s/it]
 78%|███████▊  | 390/500 [1:04:20<15:30,  8.46s/it]
                                                   

 78%|███████▊  | 390/500 [1:04:20<15:30,  8.46s/it][2026-03-26 13:14:21] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:14:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.8

 78%|███████▊  | 391/500 [1:04:38<20:47, 11.45s/it]
                                                   

 78%|███████▊  | 391/500 [1:04:38<20:47, 11.45s/it]
 78%|███████▊  | 392/500 [1:04:40<15:30,  8.62s/it]
                                                   

 78%|███████▊  | 392/500 [1:04:40<15:30,  8.62s/it][2026-03-26 13:14:42] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:14:54] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.8

 79%|███████▊  | 393/500 [1:04:56<19:15, 10.80s/it]
                                                   

 79%|███████▊  | 393/500 [1:04:56<19:15, 10.80s/it]
 79%|███████▉  | 394/500 [1:04:58<14:17,  8.09s/it]
                                                   

 79%|███████▉  | 394/500 [1:04:58<14:17,  8.09s/it][2026-03-26 13:14:59] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:15:14] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=25.0

 79%|███████▉  | 395/500 [1:05:16<19:38, 11.22s/it]
                                                   

 79%|███████▉  | 395/500 [1:05:16<19:38, 11.22s/it]
 79%|███████▉  | 396/500 [1:05:19<14:50,  8.57s/it]
                                                   

 79%|███████▉  | 396/500 [1:05:19<14:50,  8.57s/it][2026-03-26 13:15:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:15:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:7 D:0 (12% win) | avg_turns=18.1

 79%|███████▉  | 397/500 [1:05:35<18:33, 10.81s/it]
                                                   

 79%|███████▉  | 397/500 [1:05:35<18:33, 10.81s/it]
 80%|███████▉  | 398/500 [1:05:37<13:49,  8.13s/it]
                                                   
{'loss': -0.1313, 'grad_norm': 0.2653455436229706, 'learning_rate': 1.2771342418149658e-06, 'num_tokens': 2341612.0, 'completions/mean_length': 1328.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4389999806880951, 'rewards/game_reward_func/std': 0.19230927526950836, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4389999806880951, 'reward_std': 0.19230926036834717, 'frac_reward_zero_std': 0.0, 'kl': 0.02226278418675065, 'entropy': 0.24574825540184975, 'clip_ratio/low_mean': 0.00793400127440691, 'clip_ratio/low_min': 0.00793400127440691, 'clip_ratio/high_mean': 0.00244140625, 'clip_ratio/high_max': 0.00244140625, 'clip_ratio/region_mean': 0.01037540752440691, 'step_time': 18.0237765379934, 'epoch': 0.1}
{'loss': 0.1359, 'grad_norm': 0.15512649714946747, 'learning_rate': 1.2558115014363592e-06, 'kl': 0.03686125134117901, 'entropy': 0.25274327397346497, 'clip_ratio/low_mean': 0.01436988910427317, 'clip_ratio/low_min': 0.01436988910427317, 'clip_ratio/high_mean': 0.0037109374534338713, 'clip_ratio/high_max': 0.0037109374534338713, 'clip_ratio/region_mean': 0.01808082655770704, 'step_time': 1.9348710589983966, 'epoch': 0.1}
{'loss': 0.0376, 'grad_norm': 0.22211098670959473, 'learning_rate': 1.234642669981946e-06, 'num_tokens': 2354564.0, 'completions/mean_length': 1440.0, 'completions/min_length': 960.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4659999907016754, 'rewards/game_reward_func/std': 0.056890878826379776, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4659999907016754, 'reward_std': 0.05689087137579918, 'frac_reward_zero_std': 0.0, 'kl': 0.022737034130841494, 'entropy': 0.2341471202671528, 'clip_ratio/low_mean': 0.00736979185603559, 'clip_ratio/low_min': 0.00736979185603559, 'clip_ratio/high_mean': 0.0023437500931322575, 'clip_ratio/high_max': 0.0023437500931322575, 'clip_ratio/region_mean': 0.009713541949167848, 'step_time': 16.714289217998157, 'epoch': 0.1}
{'loss': -0.0332, 'grad_norm': 0.2636924982070923, 'learning_rate': 1.2136286176194744e-06, 'kl': 0.020272455643862486, 'entropy': 0.28242240846157074, 'clip_ratio/low_mean': 0.0036043176660314202, 'clip_ratio/low_min': 0.0036043176660314202, 'clip_ratio/high_mean': 0.005816915770992637, 'clip_ratio/high_max': 0.005816915770992637, 'clip_ratio/region_mean': 0.009421233437024057, 'step_time': 2.0589016180019826, 'epoch': 0.1}
{'loss': 0.1362, 'grad_norm': 0.1484292894601822, 'learning_rate': 1.1927702081543279e-06, 'num_tokens': 2368156.0, 'completions/mean_length': 1520.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3645000159740448, 'rewards/game_reward_func/std': 0.11190175265073776, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3645000159740448, 'reward_std': 0.11190175265073776, 'frac_reward_zero_std': 0.0, 'kl': 0.017420669086277485, 'entropy': 0.24209515005350113, 'clip_ratio/low_mean': 0.0038004156085662544, 'clip_ratio/low_min': 0.0038004156085662544, 'clip_ratio/high_mean': 0.0032552082557231188, 'clip_ratio/high_max': 0.0032552082557231188, 'clip_ratio/region_mean': 0.007055623864289373, 'step_time': 18.238828437009943, 'epoch': 0.1}
{'loss': -0.1315, 'grad_norm': 0.42643827199935913, 'learning_rate': 1.1720682989940264e-06, 'kl': 0.03516293317079544, 'entropy': 0.2632221654057503, 'clip_ratio/low_mean': 0.0026041667442768812, 'clip_ratio/low_min': 0.0026041667442768812, 'clip_ratio/high_mean': 0.013888888759538531, 'clip_ratio/high_max': 0.013888888759538531, 'clip_ratio/region_mean': 0.016493055503815413, 'step_time': 1.8336524080077652, 'epoch': 0.1}
{'loss': 0.2004, 'grad_norm': 0.267752468585968, 'learning_rate': 1.1515237411129698e-06, 'num_tokens': 2379700.0, 'completions/mean_length': 1264.0, 'completions/min_length': 256.0, 'completions/max_length': 1792.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.47350001335144043, 'rewards/game_reward_func/std': 0.12637698650360107, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.47350001335144043, 'reward_std': 0.12637698650360107, 'frac_reward_zero_std': 0.0, 'kl': 0.017968227388337255, 'entropy': 0.2651645578444004, 'clip_ratio/low_mean': 0.004230484191793948, 'clip_ratio/low_min': 0.004230484191793948, 'clip_ratio/high_mean': 0.0037286931183189154, 'clip_ratio/high_max': 0.0037286931183189154, 'clip_ratio/region_mean': 0.007959177310112864, 'step_time': 15.726442784995015, 'epoch': 0.1}
{'loss': -0.1935, 'grad_norm': 0.7980525493621826, 'learning_rate': 1.1311373790174656e-06, 'kl': 0.03006077604368329, 'entropy': 0.24260163307189941, 'clip_ratio/low_mean': 0.0026126218144781888, 'clip_ratio/low_min': 0.0026126218144781888, 'clip_ratio/high_mean': 0.007770040770992637, 'clip_ratio/high_max': 0.007770040770992637, 'clip_ratio/region_mean': 0.010382662585470825, 'step_time': 1.6093456109956605, 'epoch': 0.1}
{'loss': -0.2863, 'grad_norm': 0.5052506327629089, 'learning_rate': 1.1109100507110133e-06, 'num_tokens': 2393932.0, 'completions/mean_length': 1600.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.38749998807907104, 'rewards/game_reward_func/std': 0.15470433235168457, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.38749998807907104, 'reward_std': 0.15470433235168457, 'frac_reward_zero_std': 0.0, 'kl': 0.01725923386402428, 'entropy': 0.23063506186008453, 'clip_ratio/low_mean': 0.0044769683154299855, 'clip_ratio/low_min': 0.0044769683154299855, 'clip_ratio/high_mean': 0.0042724609375, 'clip_ratio/high_max': 0.0042724609375, 'clip_ratio/region_mean': 0.008749429252929986, 'step_time': 18.362821030001214, 'epoch': 0.1}
{'loss': 0.2902, 'grad_norm': 0.18945755064487457, 'learning_rate': 1.0908425876598512e-06, 'kl': 0.025313499616459012, 'entropy': 0.23379123955965042, 'clip_ratio/low_mean': 0.003402217640541494, 'clip_ratio/low_min': 0.003402217640541494, 'clip_ratio/high_mean': 0.006417410681024194, 'clip_ratio/high_max': 0.006417410681024194, 'clip_ratio/region_mean': 0.009819628321565688, 'step_time': 2.1791143310038024, 'epoch': 0.1}
{'loss': 0.4474, 'grad_norm': 0.6559186577796936, 'learning_rate': 1.0709358147587883e-06, 'num_tokens': 2404643.0, 'completions/mean_length': 1159.875, 'completions/min_length': 384.0, 'completions/max_length': 1856.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2952499985694885, 'rewards/game_reward_func/std': 0.46088480949401855, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2952499985694885, 'reward_std': 0.46088477969169617, 'frac_reward_zero_std': 0.0, 'kl': 0.018103618873283267, 'entropy': 0.24089476466178894, 'clip_ratio/low_mean': 0.005981023306958377, 'clip_ratio/low_min': 0.005981023306958377, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.005981023306958377, 'step_time': 15.901698084999225, 'epoch': 0.1}

 80%|███████▉  | 398/500 [1:05:37<13:49,  8.13s/it][2026-03-26 13:15:38] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:15:52] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.2

 80%|███████▉  | 399/500 [1:05:55<18:40, 11.09s/it]
                                                   

 80%|███████▉  | 399/500 [1:05:55<18:40, 11.09s/it]
 80%|████████  | 400/500 [1:05:57<14:00,  8.40s/it]
                                                   

 80%|████████  | 400/500 [1:05:57<14:00,  8.40s/it][2026-03-26 13:15:59] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:16:12] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.4

 80%|████████  | 401/500 [1:06:15<18:41, 11.33s/it]
                                                   

 80%|████████  | 401/500 [1:06:15<18:41, 11.33s/it]
 80%|████████  | 402/500 [1:06:17<14:00,  8.57s/it]
                                                   

 80%|████████  | 402/500 [1:06:17<14:00,  8.57s/it][2026-03-26 13:16:19] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:16:32] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.9

 81%|████████  | 403/500 [1:06:35<18:27, 11.42s/it]
                                                   

 81%|████████  | 403/500 [1:06:35<18:27, 11.42s/it]
 81%|████████  | 404/500 [1:06:37<13:52,  8.67s/it]
                                                   

 81%|████████  | 404/500 [1:06:37<13:52,  8.67s/it][2026-03-26 13:16:39] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:16:53] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:4 D:4 (0% win) | avg_turns=21.5

 81%|████████  | 405/500 [1:06:56<18:13, 11.51s/it]
                                                   

 81%|████████  | 405/500 [1:06:56<18:13, 11.51s/it]
 81%|████████  | 406/500 [1:06:57<13:30,  8.63s/it]
                                                   

 81%|████████  | 406/500 [1:06:57<13:30,  8.63s/it][2026-03-26 13:16:59] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:17:13] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.5

 81%|████████▏ | 407/500 [1:07:16<17:49, 11.50s/it]
                                                   

 81%|████████▏ | 407/500 [1:07:16<17:49, 11.50s/it]
 82%|████████▏ | 408/500 [1:07:18<13:16,  8.65s/it]
                                                   

 82%|████████▏ | 408/500 [1:07:18<13:16,  8.65s/it][2026-03-26 13:17:19] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:17:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=18.2

 82%|████████▏ | 409/500 [1:07:35<17:16, 11.39s/it]
                                                   
{'loss': -0.4414, 'grad_norm': 1.1582328081130981, 'learning_rate': 1.0511905502972885e-06, 'kl': 0.04163521574810147, 'entropy': 0.2746505178511143, 'clip_ratio/low_mean': 0.008585164905525744, 'clip_ratio/low_min': 0.008585164905525744, 'clip_ratio/high_mean': 0.005642361007630825, 'clip_ratio/high_max': 0.005642361007630825, 'clip_ratio/region_mean': 0.014227525913156569, 'step_time': 1.7074462869932177, 'epoch': 0.1}
{'loss': -0.3006, 'grad_norm': 0.5671205520629883, 'learning_rate': 1.031607605925839e-06, 'num_tokens': 2415931.0, 'completions/mean_length': 1232.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4490000009536743, 'rewards/game_reward_func/std': 0.09721258282661438, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4490000009536743, 'reward_std': 0.09721257537603378, 'frac_reward_zero_std': 0.0, 'kl': 0.025434512179344893, 'entropy': 0.24732518941164017, 'clip_ratio/low_mean': 0.005923063843511045, 'clip_ratio/low_min': 0.005923063843511045, 'clip_ratio/high_mean': 0.005419921828433871, 'clip_ratio/high_max': 0.005419921828433871, 'clip_ratio/region_mean': 0.011342985671944916, 'step_time': 17.813792075001402, 'epoch': 0.1}
{'loss': 0.3057, 'grad_norm': 0.38255003094673157, 'learning_rate': 1.0121877866225783e-06, 'kl': 0.01973992050625384, 'entropy': 0.2514846809208393, 'clip_ratio/low_mean': 0.00613030546810478, 'clip_ratio/low_min': 0.00613030546810478, 'clip_ratio/high_mean': 0.00439453125, 'clip_ratio/high_max': 0.00439453125, 'clip_ratio/region_mean': 0.01052483671810478, 'step_time': 1.9317141129940865, 'epoch': 0.1}
{'loss': 0.1365, 'grad_norm': 0.2106090784072876, 'learning_rate': 9.929318906602176e-07, 'num_tokens': 2428307.0, 'completions/mean_length': 1368.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4307500123977661, 'rewards/game_reward_func/std': 0.0766061544418335, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4307500123977661, 'reward_std': 0.0766061544418335, 'frac_reward_zero_std': 0.0, 'kl': 0.023753622779622674, 'entropy': 0.2371758297085762, 'clip_ratio/low_mean': 0.0063883461989462376, 'clip_ratio/low_min': 0.0063883461989462376, 'clip_ratio/high_mean': 0.004551478894427419, 'clip_ratio/high_max': 0.004551478894427419, 'clip_ratio/region_mean': 0.010939825093373656, 'step_time': 17.5830822039934, 'epoch': 0.1}
{'loss': -0.1311, 'grad_norm': 0.5000684261322021, 'learning_rate': 9.738407095732195e-07, 'kl': 0.021920393221080303, 'entropy': 0.23405113816261292, 'clip_ratio/low_mean': 0.0023437500931322575, 'clip_ratio/low_min': 0.0023437500931322575, 'clip_ratio/high_mean': 0.0064576538279652596, 'clip_ratio/high_max': 0.0064576538279652596, 'clip_ratio/region_mean': 0.008801403921097517, 'step_time': 1.9554624410084216, 'epoch': 0.1}
{'loss': 0.5038, 'grad_norm': 0.9017499089241028, 'learning_rate': 9.549150281252633e-07, 'num_tokens': 2438891.0, 'completions/mean_length': 1144.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3502500057220459, 'rewards/game_reward_func/std': 0.21659822762012482, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3502500057220459, 'reward_std': 0.21659821271896362, 'frac_reward_zero_std': 0.0, 'kl': 0.041714358143508434, 'entropy': 0.27511710673570633, 'clip_ratio/low_mean': 0.014462425606325269, 'clip_ratio/low_min': 0.014462425606325269, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.014462425606325269, 'step_time': 17.8725395789952, 'epoch': 0.1}
{'loss': -0.4961, 'grad_norm': 0.5782556533813477, 'learning_rate': 9.361556242769871e-07, 'kl': 0.012693296186625957, 'entropy': 0.2636196203529835, 'clip_ratio/low_mean': 0.002814797800965607, 'clip_ratio/low_min': 0.002814797800965607, 'clip_ratio/high_mean': 0.0028889974346384406, 'clip_ratio/high_max': 0.0028889974346384406, 'clip_ratio/region_mean': 0.005703795235604048, 'step_time': 2.087824535992695, 'epoch': 0.1}
{'loss': -0.173, 'grad_norm': 0.6322418451309204, 'learning_rate': 9.175632691540065e-07, 'num_tokens': 2451331.0, 'completions/mean_length': 1376.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.24800001084804535, 'rewards/game_reward_func/std': 0.1970728635787964, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.24800001084804535, 'reward_std': 0.19707287847995758, 'frac_reward_zero_std': 0.0, 'kl': 0.027939027408137918, 'entropy': 0.27572380378842354, 'clip_ratio/low_mean': 0.008954327087849379, 'clip_ratio/low_min': 0.008954327087849379, 'clip_ratio/high_mean': 0.0028076171875, 'clip_ratio/high_max': 0.0028076171875, 'clip_ratio/region_mean': 0.011761944275349379, 'step_time': 17.951248125005804, 'epoch': 0.1}
{'loss': 0.1834, 'grad_norm': 0.7961837649345398, 'learning_rate': 8.991387270152202e-07, 'kl': 0.021625864785164595, 'entropy': 0.23819223046302795, 'clip_ratio/low_mean': 0.005154079874046147, 'clip_ratio/low_min': 0.005154079874046147, 'clip_ratio/high_mean': 0.00244140625, 'clip_ratio/high_max': 0.00244140625, 'clip_ratio/region_mean': 0.007595486124046147, 'step_time': 1.7395675569932791, 'epoch': 0.1}
{'loss': 0.1479, 'grad_norm': 0.48646536469459534, 'learning_rate': 8.808827552213917e-07, 'num_tokens': 2463259.0, 'completions/mean_length': 1312.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.36149999499320984, 'rewards/game_reward_func/std': 0.1435578465461731, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.36149999499320984, 'reward_std': 0.1435578316450119, 'frac_reward_zero_std': 0.0, 'kl': 0.026532822754234076, 'entropy': 0.23795316368341446, 'clip_ratio/low_mean': 0.007423647679388523, 'clip_ratio/low_min': 0.007423647679388523, 'clip_ratio/high_mean': 0.005709134507924318, 'clip_ratio/high_max': 0.005709134507924318, 'clip_ratio/region_mean': 0.013132782187312841, 'step_time': 18.012850726008764, 'epoch': 0.1}
{'loss': -0.1423, 'grad_norm': 0.5839344263076782, 'learning_rate': 8.627961042040183e-07, 'kl': 0.02389738103374839, 'entropy': 0.2738267406821251, 'clip_ratio/low_mean': 0.004106058273464441, 'clip_ratio/low_min': 0.004106058273464441, 'clip_ratio/high_mean': 0.008719307836145163, 'clip_ratio/high_max': 0.008719307836145163, 'clip_ratio/region_mean': 0.012825366109609604, 'step_time': 1.834699721992365, 'epoch': 0.1}

 82%|████████▏ | 409/500 [1:07:35<17:16, 11.39s/it]
 82%|████████▏ | 410/500 [1:07:38<12:54,  8.61s/it]
                                                   

 82%|████████▏ | 410/500 [1:07:38<12:54,  8.61s/it][2026-03-26 13:17:39] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:17:53] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 82%|████████▏ | 411/500 [1:07:56<17:10, 11.58s/it]
                                                   

 82%|████████▏ | 411/500 [1:07:56<17:10, 11.58s/it]
 82%|████████▏ | 412/500 [1:07:58<12:46,  8.71s/it]
                                                   

 82%|████████▏ | 412/500 [1:07:58<12:46,  8.71s/it][2026-03-26 13:18:00] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:18:14] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.2

 83%|████████▎ | 413/500 [1:08:17<16:53, 11.65s/it]
                                                   

 83%|████████▎ | 413/500 [1:08:17<16:53, 11.65s/it]
 83%|████████▎ | 414/500 [1:08:19<12:33,  8.76s/it]
                                                   

 83%|████████▎ | 414/500 [1:08:19<12:33,  8.76s/it][2026-03-26 13:18:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:18:33] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.4

 83%|████████▎ | 415/500 [1:08:36<15:51, 11.19s/it]
                                                   

 83%|████████▎ | 415/500 [1:08:36<15:51, 11.19s/it]
 83%|████████▎ | 416/500 [1:08:38<11:48,  8.44s/it]
                                                   

 83%|████████▎ | 416/500 [1:08:38<11:48,  8.44s/it][2026-03-26 13:18:39] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:18:53] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.9

 83%|████████▎ | 417/500 [1:08:56<15:42, 11.36s/it]
                                                   

 83%|████████▎ | 417/500 [1:08:56<15:42, 11.36s/it]
 84%|████████▎ | 418/500 [1:08:58<11:51,  8.67s/it]
                                                   

 84%|████████▎ | 418/500 [1:08:58<11:51,  8.67s/it][2026-03-26 13:19:00] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:19:14] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=24.1

 84%|████████▍ | 419/500 [1:09:17<15:42, 11.63s/it]
                                                   

 84%|████████▍ | 419/500 [1:09:17<15:42, 11.63s/it]
 84%|████████▍ | 420/500 [1:09:19<11:39,  8.75s/it]
                                                   
{'loss': -0.4019, 'grad_norm': 0.8534587025642395, 'learning_rate': 8.448795174344803e-07, 'num_tokens': 2474035.0, 'completions/mean_length': 1168.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4857499897480011, 'rewards/game_reward_func/std': 0.0954534262418747, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4857499897480011, 'reward_std': 0.0954534262418747, 'frac_reward_zero_std': 0.0, 'kl': 0.024148769676685333, 'entropy': 0.23632168024778366, 'clip_ratio/low_mean': 0.004191080806776881, 'clip_ratio/low_min': 0.004191080806776881, 'clip_ratio/high_mean': 0.006293402751907706, 'clip_ratio/high_max': 0.006293402751907706, 'clip_ratio/region_mean': 0.010484483558684587, 'step_time': 17.587602626997977, 'epoch': 0.1}
{'loss': 0.4098, 'grad_norm': 0.7941663861274719, 'learning_rate': 8.271337313934869e-07, 'kl': 0.029183259001001716, 'entropy': 0.2852955311536789, 'clip_ratio/low_mean': 0.013950892956927419, 'clip_ratio/low_min': 0.013950892956927419, 'clip_ratio/high_mean': 0.0026461693923920393, 'clip_ratio/high_max': 0.0026461693923920393, 'clip_ratio/region_mean': 0.016597062349319458, 'step_time': 1.9505240959988441, 'epoch': 0.1}
{'loss': 0.1628, 'grad_norm': 0.4946267306804657, 'learning_rate': 8.095594755407971e-07, 'num_tokens': 2486027.0, 'completions/mean_length': 1320.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.390749990940094, 'rewards/game_reward_func/std': 0.20591798424720764, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.390749990940094, 'reward_std': 0.20591798424720764, 'frac_reward_zero_std': 0.0, 'kl': 0.0379240782931447, 'entropy': 0.2817593142390251, 'clip_ratio/low_mean': 0.007968750083819032, 'clip_ratio/low_min': 0.007968750083819032, 'clip_ratio/high_mean': 0.013164062489522621, 'clip_ratio/high_max': 0.013164062489522621, 'clip_ratio/region_mean': 0.021132812573341653, 'step_time': 18.32378753899684, 'epoch': 0.1}
{'loss': -0.1582, 'grad_norm': 0.5005304217338562, 'learning_rate': 7.921574722852343e-07, 'kl': 0.01646346878260374, 'entropy': 0.22042611613869667, 'clip_ratio/low_mean': 0.007334787573199719, 'clip_ratio/low_min': 0.007334787573199719, 'clip_ratio/high_mean': 0.0029296875, 'clip_ratio/high_max': 0.0029296875, 'clip_ratio/region_mean': 0.01026447507319972, 'step_time': 1.856903642998077, 'epoch': 0.1}
{'loss': 0.0634, 'grad_norm': 0.11899735778570175, 'learning_rate': 7.749284369549954e-07, 'num_tokens': 2499363.0, 'completions/mean_length': 1488.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.47800004482269287, 'rewards/game_reward_func/std': 0.13935771584510803, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.47800004482269287, 'reward_std': 0.13935771584510803, 'frac_reward_zero_std': 0.0, 'kl': 0.0192628912627697, 'entropy': 0.2249463014304638, 'clip_ratio/low_mean': 0.006063179345801473, 'clip_ratio/low_min': 0.006063179345801473, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.009969429345801473, 'step_time': 18.302427141003136, 'epoch': 0.1}
{'loss': -0.0566, 'grad_norm': 0.2876420319080353, 'learning_rate': 7.578730777682386e-07, 'kl': 0.022076581604778767, 'entropy': 0.23842975497245789, 'clip_ratio/low_mean': 0.005952380946837366, 'clip_ratio/low_min': 0.005952380946837366, 'clip_ratio/high_mean': 0.0023193359375, 'clip_ratio/high_max': 0.0023193359375, 'clip_ratio/region_mean': 0.008271716884337366, 'step_time': 1.837689574000251, 'epoch': 0.1}
{'loss': -0.3431, 'grad_norm': 0.7235519289970398, 'learning_rate': 7.409920958039795e-07, 'num_tokens': 2511739.0, 'completions/mean_length': 1368.0, 'completions/min_length': 640.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40074998140335083, 'rewards/game_reward_func/std': 0.15339933335781097, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40074998140335083, 'reward_std': 0.15339931845664978, 'frac_reward_zero_std': 0.0, 'kl': 0.023018692852929235, 'entropy': 0.23650463670492172, 'clip_ratio/low_mean': 0.00513980258256197, 'clip_ratio/low_min': 0.00513980258256197, 'clip_ratio/high_mean': 0.0037486294750124216, 'clip_ratio/high_max': 0.0037486294750124216, 'clip_ratio/region_mean': 0.008888432057574391, 'step_time': 16.692041192000033, 'epoch': 0.1}
{'loss': 0.3506, 'grad_norm': 0.4964349865913391, 'learning_rate': 7.242861849732696e-07, 'kl': 0.017378215678036213, 'entropy': 0.2686605267226696, 'clip_ratio/low_mean': 0.00205110787646845, 'clip_ratio/low_min': 0.00205110787646845, 'clip_ratio/high_mean': 0.008749999979045242, 'clip_ratio/high_max': 0.008749999979045242, 'clip_ratio/region_mean': 0.010801107855513692, 'step_time': 1.8332267239966313, 'epoch': 0.1}
{'loss': -0.5464, 'grad_norm': 0.8531047701835632, 'learning_rate': 7.077560319906696e-07, 'num_tokens': 2523347.0, 'completions/mean_length': 1272.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.37299999594688416, 'rewards/game_reward_func/std': 0.2852728068828583, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.37299999594688416, 'reward_std': 0.2852727770805359, 'frac_reward_zero_std': 0.0, 'kl': 0.03286154894158244, 'entropy': 0.2488662227988243, 'clip_ratio/low_mean': 0.00732421875, 'clip_ratio/low_min': 0.00732421875, 'clip_ratio/high_mean': 0.002732799854129553, 'clip_ratio/high_max': 0.002732799854129553, 'clip_ratio/region_mean': 0.010057018604129553, 'step_time': 17.98781309399783, 'epoch': 0.1}
{'loss': 0.5537, 'grad_norm': 0.9447053670883179, 'learning_rate': 6.914023163460248e-07, 'kl': 0.026666713878512383, 'entropy': 0.25744036585092545, 'clip_ratio/low_mean': 0.012199377873912454, 'clip_ratio/low_min': 0.012199377873912454, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.012199377873912454, 'step_time': 2.204474391990516, 'epoch': 0.1}
{'loss': 0.1645, 'grad_norm': 0.37258797883987427, 'learning_rate': 6.752257102765325e-07, 'num_tokens': 2537131.0, 'completions/mean_length': 1544.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.38600000739097595, 'rewards/game_reward_func/std': 0.16599141061306, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.38600000739097595, 'reward_std': 0.1659913957118988, 'frac_reward_zero_std': 0.0, 'kl': 0.02052541309967637, 'entropy': 0.24147875234484673, 'clip_ratio/low_mean': 0.009161086287349463, 'clip_ratio/low_min': 0.009161086287349463, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.010625930037349463, 'step_time': 18.35078954599885, 'epoch': 0.1}

 84%|████████▍ | 420/500 [1:09:19<11:39,  8.75s/it][2026-03-26 13:19:20] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:19:35] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=27.2

 84%|████████▍ | 421/500 [1:09:37<15:29, 11.77s/it]
                                                   

 84%|████████▍ | 421/500 [1:09:37<15:29, 11.77s/it]
 84%|████████▍ | 422/500 [1:09:39<11:27,  8.81s/it]
                                                   

 84%|████████▍ | 422/500 [1:09:39<11:27,  8.81s/it][2026-03-26 13:19:41] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:19:55] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.9

 85%|████████▍ | 423/500 [1:09:58<14:57, 11.66s/it]
                                                   

 85%|████████▍ | 423/500 [1:09:58<14:57, 11.66s/it]
 85%|████████▍ | 424/500 [1:10:00<11:06,  8.77s/it]
                                                   

 85%|████████▍ | 424/500 [1:10:00<11:06,  8.77s/it][2026-03-26 13:20:01] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:20:16] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.4

 85%|████████▌ | 425/500 [1:10:18<14:41, 11.75s/it]
                                                   

 85%|████████▌ | 425/500 [1:10:18<14:41, 11.75s/it]
 85%|████████▌ | 426/500 [1:10:20<10:53,  8.83s/it]
                                                   

 85%|████████▌ | 426/500 [1:10:20<10:53,  8.83s/it][2026-03-26 13:20:22] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:20:36] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:6 D:1 (12% win) | avg_turns=22.5

 85%|████████▌ | 427/500 [1:10:39<14:14, 11.71s/it]
                                                   

 85%|████████▌ | 427/500 [1:10:39<14:14, 11.71s/it]
 86%|████████▌ | 428/500 [1:10:41<10:36,  8.84s/it]
                                                   

 86%|████████▌ | 428/500 [1:10:41<10:36,  8.84s/it][2026-03-26 13:20:43] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:20:55] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=19.2

 86%|████████▌ | 429/500 [1:10:58<13:15, 11.21s/it]
                                                   

 86%|████████▌ | 429/500 [1:10:58<13:15, 11.21s/it]
 86%|████████▌ | 430/500 [1:11:00<09:50,  8.44s/it]
                                                   

 86%|████████▌ | 430/500 [1:11:00<09:50,  8.44s/it][2026-03-26 13:21:01] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:21:15] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.8

 86%|████████▌ | 431/500 [1:11:18<13:11, 11.47s/it]
                                                   
{'loss': -0.1612, 'grad_norm': 0.3772587776184082, 'learning_rate': 6.592268787391077e-07, 'kl': 0.01771045522764325, 'entropy': 0.23651697114109993, 'clip_ratio/low_mean': 0.002273395424708724, 'clip_ratio/low_min': 0.002273395424708724, 'clip_ratio/high_mean': 0.006626673974096775, 'clip_ratio/high_max': 0.006626673974096775, 'clip_ratio/region_mean': 0.008900069398805499, 'step_time': 1.838646662014071, 'epoch': 0.1}
{'loss': 0.6291, 'grad_norm': 0.7329128384590149, 'learning_rate': 6.43406479383053e-07, 'num_tokens': 2552515.0, 'completions/mean_length': 1744.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3242499828338623, 'rewards/game_reward_func/std': 0.24032939970493317, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3242499828338623, 'reward_std': 0.24032938480377197, 'frac_reward_zero_std': 0.0, 'kl': 0.01318417303264141, 'entropy': 0.23070888966321945, 'clip_ratio/low_mean': 0.005233835079707205, 'clip_ratio/low_min': 0.005233835079707205, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.005233835079707205, 'step_time': 18.624511004985834, 'epoch': 0.11}
{'loss': -0.6268, 'grad_norm': 0.7614300847053528, 'learning_rate': 6.277651625230219e-07, 'kl': 0.017645573941990733, 'entropy': 0.23983237147331238, 'clip_ratio/low_mean': 0.004769587889313698, 'clip_ratio/low_min': 0.004769587889313698, 'clip_ratio/high_mean': 0.0048828125, 'clip_ratio/high_max': 0.0048828125, 'clip_ratio/region_mean': 0.009652400389313698, 'step_time': 1.7440905199982808, 'epoch': 0.11}
{'loss': -0.1653, 'grad_norm': 0.4356380105018616, 'learning_rate': 6.12303571112286e-07, 'num_tokens': 2564635.0, 'completions/mean_length': 1336.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3647500276565552, 'rewards/game_reward_func/std': 0.1992010921239853, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3647500276565552, 'reward_std': 0.1992010921239853, 'frac_reward_zero_std': 0.0, 'kl': 0.020558797288686037, 'entropy': 0.25156764686107635, 'clip_ratio/low_mean': 0.005056423717178404, 'clip_ratio/low_min': 0.005056423717178404, 'clip_ratio/high_mean': 0.007983398623764515, 'clip_ratio/high_max': 0.007983398623764515, 'clip_ratio/region_mean': 0.01303982234094292, 'step_time': 18.12171134899836, 'epoch': 0.11}
{'loss': 0.1712, 'grad_norm': 0.5712350010871887, 'learning_rate': 5.9702234071631e-07, 'kl': 0.028067943640053272, 'entropy': 0.24978860840201378, 'clip_ratio/low_mean': 0.012497774208895862, 'clip_ratio/low_min': 0.012497774208895862, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.014206758583895862, 'step_time': 1.835538772000291, 'epoch': 0.11}
{'loss': 0.2492, 'grad_norm': 0.31143248081207275, 'learning_rate': 5.819220994866237e-07, 'num_tokens': 2578035.0, 'completions/mean_length': 1496.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.28675001859664917, 'rewards/game_reward_func/std': 0.22822029888629913, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.28675001859664917, 'reward_std': 0.22822028398513794, 'frac_reward_zero_std': 0.0, 'kl': 0.02161014312878251, 'entropy': 0.23553497344255447, 'clip_ratio/low_mean': 0.002898507285863161, 'clip_ratio/low_min': 0.002898507285863161, 'clip_ratio/high_mean': 0.0077718100510537624, 'clip_ratio/high_max': 0.0077718100510537624, 'clip_ratio/region_mean': 0.010670317336916924, 'step_time': 18.541566322994186, 'epoch': 0.11}
{'loss': -0.2407, 'grad_norm': 0.5170923471450806, 'learning_rate': 5.670034681349995e-07, 'kl': 0.0174321373924613, 'entropy': 0.26809917390346527, 'clip_ratio/low_mean': 0.00439453125, 'clip_ratio/low_min': 0.00439453125, 'clip_ratio/high_mean': 0.0034051192924380302, 'clip_ratio/high_max': 0.0034051192924380302, 'clip_ratio/region_mean': 0.00779965054243803, 'step_time': 1.8390744200078188, 'epoch': 0.11}
{'loss': -0.3497, 'grad_norm': 0.7297412157058716, 'learning_rate': 5.522670599079416e-07, 'num_tokens': 2590987.0, 'completions/mean_length': 1440.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.21950000524520874, 'rewards/game_reward_func/std': 0.4302089214324951, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.21950000524520874, 'reward_std': 0.4302089214324951, 'frac_reward_zero_std': 0.0, 'kl': 0.020177583675831556, 'entropy': 0.24980150163173676, 'clip_ratio/low_mean': 0.0031953424331732094, 'clip_ratio/low_min': 0.0031953424331732094, 'clip_ratio/high_mean': 0.0077311196364462376, 'clip_ratio/high_max': 0.0077311196364462376, 'clip_ratio/region_mean': 0.010926462069619447, 'step_time': 18.234811274989625, 'epoch': 0.11}
{'loss': 0.3551, 'grad_norm': 0.6922006011009216, 'learning_rate': 5.377134805614714e-07, 'kl': 0.026948010316118598, 'entropy': 0.2551211267709732, 'clip_ratio/low_mean': 0.008178430143743753, 'clip_ratio/low_min': 0.008178430143743753, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.008178430143743753, 'step_time': 1.9388053189977654, 'epoch': 0.11}
{'loss': 0.4138, 'grad_norm': 0.48611289262771606, 'learning_rate': 5.233433283362349e-07, 'num_tokens': 2602275.0, 'completions/mean_length': 1232.0, 'completions/min_length': 384.0, 'completions/max_length': 1920.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4645000100135803, 'rewards/game_reward_func/std': 0.08978546410799026, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4645000100135803, 'reward_std': 0.08978547155857086, 'frac_reward_zero_std': 0.0, 'kl': 0.02168187079951167, 'entropy': 0.26357534527778625, 'clip_ratio/low_mean': 0.006555316271260381, 'clip_ratio/low_min': 0.006555316271260381, 'clip_ratio/high_mean': 0.006069711525924504, 'clip_ratio/high_max': 0.006069711525924504, 'clip_ratio/region_mean': 0.012625027797184885, 'step_time': 16.574229850994016, 'epoch': 0.11}
{'loss': -0.4091, 'grad_norm': 1.4892901182174683, 'learning_rate': 5.091571939329049e-07, 'kl': 0.04066102975048125, 'entropy': 0.28890983387827873, 'clip_ratio/low_mean': 0.008854166604578495, 'clip_ratio/low_min': 0.008854166604578495, 'clip_ratio/high_mean': 0.011359554890077561, 'clip_ratio/high_max': 0.011359554890077561, 'clip_ratio/region_mean': 0.020213721494656056, 'step_time': 1.8030485580093227, 'epoch': 0.11}

 86%|████████▌ | 431/500 [1:11:18<13:11, 11.47s/it]
 86%|████████▋ | 432/500 [1:11:20<09:47,  8.64s/it]
                                                   

 86%|████████▋ | 432/500 [1:11:20<09:47,  8.64s/it][2026-03-26 13:21:22] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:21:35] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.5

 87%|████████▋ | 433/500 [1:11:38<12:47, 11.46s/it]
                                                   

 87%|████████▋ | 433/500 [1:11:38<12:47, 11.46s/it]
 87%|████████▋ | 434/500 [1:11:40<09:31,  8.66s/it]
                                                   

 87%|████████▋ | 434/500 [1:11:40<09:31,  8.66s/it][2026-03-26 13:21:42] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:21:56] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.9

 87%|████████▋ | 435/500 [1:11:59<12:33, 11.60s/it]
                                                   

 87%|████████▋ | 435/500 [1:11:59<12:33, 11.60s/it]
 87%|████████▋ | 436/500 [1:12:01<09:18,  8.73s/it]
                                                   

 87%|████████▋ | 436/500 [1:12:01<09:18,  8.73s/it][2026-03-26 13:22:03] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:22:16] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:1 L:6 D:1 (12% win) | avg_turns=21.1

 87%|████████▋ | 437/500 [1:12:19<12:11, 11.61s/it]
                                                   

 87%|████████▋ | 437/500 [1:12:19<12:11, 11.61s/it]
 88%|████████▊ | 438/500 [1:12:21<09:01,  8.73s/it]
                                                   

 88%|████████▊ | 438/500 [1:12:21<09:01,  8.73s/it][2026-03-26 13:22:23] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:22:37] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.1

 88%|████████▊ | 439/500 [1:12:40<11:48, 11.62s/it]
                                                   

 88%|████████▊ | 439/500 [1:12:40<11:48, 11.62s/it]
 88%|████████▊ | 440/500 [1:12:42<08:48,  8.82s/it]
                                                   

 88%|████████▊ | 440/500 [1:12:42<08:48,  8.82s/it][2026-03-26 13:22:44] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:22:58] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.0

 88%|████████▊ | 441/500 [1:13:00<11:28, 11.67s/it]
                                                   

 88%|████████▊ | 441/500 [1:13:00<11:28, 11.67s/it]
 88%|████████▊ | 442/500 [1:13:02<08:30,  8.81s/it]
                                                   
{'loss': 0.2603, 'grad_norm': 0.40104448795318604, 'learning_rate': 4.951556604879049e-07, 'num_tokens': 2615867.0, 'completions/mean_length': 1520.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.40275001525878906, 'rewards/game_reward_func/std': 0.15268993377685547, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.40275001525878906, 'reward_std': 0.15268991887569427, 'frac_reward_zero_std': 0.0, 'kl': 0.01837842701934278, 'entropy': 0.26202475279569626, 'clip_ratio/low_mean': 0.0030691964784637094, 'clip_ratio/low_min': 0.0030691964784637094, 'clip_ratio/high_mean': 0.005180027103051543, 'clip_ratio/high_max': 0.005180027103051543, 'clip_ratio/region_mean': 0.008249223581515253, 'step_time': 18.358479331996932, 'epoch': 0.11}
{'loss': -0.2537, 'grad_norm': 0.3868066668510437, 'learning_rate': 4.813393035494329e-07, 'kl': 0.01732565346173942, 'entropy': 0.22887229174375534, 'clip_ratio/low_mean': 0.00451412214897573, 'clip_ratio/low_min': 0.00451412214897573, 'clip_ratio/high_mean': 0.0010986328125, 'clip_ratio/high_max': 0.0010986328125, 'clip_ratio/region_mean': 0.00561275496147573, 'step_time': 1.8401955910012475, 'epoch': 0.11}
{'loss': -0.3122, 'grad_norm': 0.5159977078437805, 'learning_rate': 4.677086910538092e-07, 'num_tokens': 2628307.0, 'completions/mean_length': 1376.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4165000021457672, 'rewards/game_reward_func/std': 0.09190522879362106, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4165000021457672, 'reward_std': 0.09190522134304047, 'frac_reward_zero_std': 0.0, 'kl': 0.022038984345272183, 'entropy': 0.24463222175836563, 'clip_ratio/low_mean': 0.004583333386108279, 'clip_ratio/low_min': 0.004583333386108279, 'clip_ratio/high_mean': 0.005586511921137571, 'clip_ratio/high_max': 0.005586511921137571, 'clip_ratio/region_mean': 0.01016984530724585, 'step_time': 17.863235089003865, 'epoch': 0.11}
{'loss': 0.3181, 'grad_norm': 0.5176377892494202, 'learning_rate': 4.542643833021254e-07, 'kl': 0.018008822109550238, 'entropy': 0.25773125141859055, 'clip_ratio/low_mean': 0.0035665761679410934, 'clip_ratio/low_min': 0.0035665761679410934, 'clip_ratio/high_mean': 0.004731172346509993, 'clip_ratio/high_max': 0.004731172346509993, 'clip_ratio/region_mean': 0.008297748514451087, 'step_time': 1.9422309220099123, 'epoch': 0.11}
{'loss': 0.1148, 'grad_norm': 0.15970012545585632, 'learning_rate': 4.410069329372152e-07, 'num_tokens': 2641451.0, 'completions/mean_length': 1464.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4132499694824219, 'rewards/game_reward_func/std': 0.14970804750919342, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4132499694824219, 'reward_std': 0.14970804750919342, 'frac_reward_zero_std': 0.0, 'kl': 0.019152209162712097, 'entropy': 0.23537284135818481, 'clip_ratio/low_mean': 0.0038900863146409392, 'clip_ratio/low_min': 0.0038900863146409392, 'clip_ratio/high_mean': 0.007035626797005534, 'clip_ratio/high_max': 0.007035626797005534, 'clip_ratio/region_mean': 0.010925713111646473, 'step_time': 18.25842839300458, 'epoch': 0.11}
{'loss': -0.1114, 'grad_norm': 0.24259723722934723, 'learning_rate': 4.279368849209381e-07, 'kl': 0.02041076822206378, 'entropy': 0.254366897046566, 'clip_ratio/low_mean': 0.005604619509540498, 'clip_ratio/low_min': 0.005604619509540498, 'clip_ratio/high_mean': 0.0028076171875, 'clip_ratio/high_max': 0.0028076171875, 'clip_ratio/region_mean': 0.008412236697040498, 'step_time': 1.8397148229923914, 'epoch': 0.11}
{'loss': -0.2695, 'grad_norm': 0.3532894551753998, 'learning_rate': 4.150547765117746e-07, 'num_tokens': 2653699.0, 'completions/mean_length': 1352.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.23275001347064972, 'rewards/game_reward_func/std': 0.41876545548439026, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.23275001347064972, 'reward_std': 0.41876545548439026, 'frac_reward_zero_std': 0.0, 'kl': 0.026121661998331547, 'entropy': 0.25272637605667114, 'clip_ratio/low_mean': 0.010832104017026722, 'clip_ratio/low_min': 0.010832104017026722, 'clip_ratio/high_mean': 0.0018857758259400725, 'clip_ratio/high_max': 0.0018857758259400725, 'clip_ratio/region_mean': 0.012717879842966795, 'step_time': 18.137923245005368, 'epoch': 0.11}
{'loss': 0.2754, 'grad_norm': 0.4949052035808563, 'learning_rate': 4.0236113724274716e-07, 'kl': 0.019219663459807634, 'entropy': 0.2616669498383999, 'clip_ratio/low_mean': 0.005010683787986636, 'clip_ratio/low_min': 0.005010683787986636, 'clip_ratio/high_mean': 0.0010986328125, 'clip_ratio/high_max': 0.0010986328125, 'clip_ratio/region_mean': 0.006109316600486636, 'step_time': 1.8402125099964906, 'epoch': 0.11}
{'loss': -0.137, 'grad_norm': 0.38423049449920654, 'learning_rate': 3.8985648889964755e-07, 'num_tokens': 2666971.0, 'completions/mean_length': 1480.0, 'completions/min_length': 384.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44725000858306885, 'rewards/game_reward_func/std': 0.21278007328510284, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44725000858306885, 'reward_std': 0.21278007328510284, 'frac_reward_zero_std': 0.0, 'kl': 0.02508135768584907, 'entropy': 0.22263111919164658, 'clip_ratio/low_mean': 0.009285094332881272, 'clip_ratio/low_min': 0.009285094332881272, 'clip_ratio/high_mean': 0.002197265625, 'clip_ratio/high_max': 0.002197265625, 'clip_ratio/region_mean': 0.011482359957881272, 'step_time': 18.18084599499707, 'epoch': 0.11}
{'loss': 0.1407, 'grad_norm': 0.12573042511940002, 'learning_rate': 3.77541345499593e-07, 'kl': 0.014975633937865496, 'entropy': 0.24247493967413902, 'clip_ratio/low_mean': 0.006030273390933871, 'clip_ratio/low_min': 0.006030273390933871, 'clip_ratio/high_mean': 0.0035807292442768812, 'clip_ratio/high_max': 0.0035807292442768812, 'clip_ratio/region_mean': 0.009611002635210752, 'step_time': 2.093893182995089, 'epoch': 0.11}
{'loss': -0.2033, 'grad_norm': 0.3992139399051666, 'learning_rate': 3.6541621326989183e-07, 'num_tokens': 2679667.0, 'completions/mean_length': 1408.0, 'completions/min_length': 576.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.43050000071525574, 'rewards/game_reward_func/std': 0.1534545123577118, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.43050000071525574, 'reward_std': 0.1534545123577118, 'frac_reward_zero_std': 0.0, 'kl': 0.015888667665421963, 'entropy': 0.24944307282567024, 'clip_ratio/low_mean': 0.005620659678243101, 'clip_ratio/low_min': 0.005620659678243101, 'clip_ratio/high_mean': 0.000732421875, 'clip_ratio/high_max': 0.000732421875, 'clip_ratio/region_mean': 0.006353081553243101, 'step_time': 18.154841946001397, 'epoch': 0.11}

 88%|████████▊ | 442/500 [1:13:02<08:30,  8.81s/it][2026-03-26 13:23:04] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:23:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=22.5

 89%|████████▊ | 443/500 [1:13:21<11:05, 11.68s/it]
                                                   

 89%|████████▊ | 443/500 [1:13:21<11:05, 11.68s/it]
 89%|████████▉ | 444/500 [1:13:23<08:11,  8.78s/it]
                                                   

 89%|████████▉ | 444/500 [1:13:23<08:11,  8.78s/it][2026-03-26 13:23:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:23:39] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=23.8

 89%|████████▉ | 445/500 [1:13:41<10:40, 11.65s/it]
                                                   

 89%|████████▉ | 445/500 [1:13:41<10:40, 11.65s/it]
 89%|████████▉ | 446/500 [1:13:43<07:53,  8.76s/it]
                                                   

 89%|████████▉ | 446/500 [1:13:43<07:53,  8.76s/it][2026-03-26 13:23:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:23:59] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=20.4

 89%|████████▉ | 447/500 [1:14:01<10:14, 11.60s/it]
                                                   

 89%|████████▉ | 447/500 [1:14:01<10:14, 11.60s/it]
 90%|████████▉ | 448/500 [1:14:03<07:33,  8.73s/it]
                                                   

 90%|████████▉ | 448/500 [1:14:03<07:33,  8.73s/it][2026-03-26 13:24:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:24:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=18.9

 90%|████████▉ | 449/500 [1:14:21<09:33, 11.25s/it]
                                                   

 90%|████████▉ | 449/500 [1:14:21<09:33, 11.25s/it]
 90%|█████████ | 450/500 [1:14:23<07:04,  8.50s/it]
                                                   

 90%|█████████ | 450/500 [1:14:23<07:04,  8.50s/it][2026-03-26 13:24:25] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:24:39] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=23.9

 90%|█████████ | 451/500 [1:14:41<09:28, 11.60s/it]
                                                   

 90%|█████████ | 451/500 [1:14:41<09:28, 11.60s/it]
 90%|█████████ | 452/500 [1:14:43<06:57,  8.70s/it]
                                                   

 90%|█████████ | 452/500 [1:14:43<06:57,  8.70s/it][2026-03-26 13:24:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:24:59] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=17.1

 91%|█████████ | 453/500 [1:15:01<09:01, 11.51s/it]
                                                   
{'loss': 0.2077, 'grad_norm': 0.17413115501403809, 'learning_rate': 3.534815906272404e-07, 'kl': 0.024427148047834635, 'entropy': 0.26615485921502113, 'clip_ratio/low_mean': 0.005176464095711708, 'clip_ratio/low_min': 0.005176464095711708, 'clip_ratio/high_mean': 0.007264254614710808, 'clip_ratio/high_max': 0.007264254614710808, 'clip_ratio/region_mean': 0.012440718710422516, 'step_time': 1.9377123590020346, 'epoch': 0.11}
{'loss': 0.4044, 'grad_norm': 0.5449001789093018, 'learning_rate': 3.417379681572297e-07, 'num_tokens': 2692619.0, 'completions/mean_length': 1440.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.41850000619888306, 'rewards/game_reward_func/std': 0.18425990641117096, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.41850000619888306, 'reward_std': 0.18425990641117096, 'frac_reward_zero_std': 0.0, 'kl': 0.027640082873404026, 'entropy': 0.23091036453843117, 'clip_ratio/low_mean': 0.007853260729461908, 'clip_ratio/low_min': 0.007853260729461908, 'clip_ratio/high_mean': 0.0030381944961845875, 'clip_ratio/high_max': 0.0030381944961845875, 'clip_ratio/region_mean': 0.010891455225646496, 'step_time': 18.190129606999108, 'epoch': 0.11}
{'loss': -0.3972, 'grad_norm': 0.5098146200180054, 'learning_rate': 3.301858285941845e-07, 'kl': 0.0188440615311265, 'entropy': 0.2609354257583618, 'clip_ratio/low_mean': 0.0014322916977107525, 'clip_ratio/low_min': 0.0014322916977107525, 'clip_ratio/high_mean': 0.00448843149933964, 'clip_ratio/high_max': 0.00448843149933964, 'clip_ratio/region_mean': 0.005920723197050393, 'step_time': 1.836915362997388, 'epoch': 0.11}
{'loss': -0.2591, 'grad_norm': 0.37110471725463867, 'learning_rate': 3.18825646801314e-07, 'num_tokens': 2706211.0, 'completions/mean_length': 1520.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.35899999737739563, 'rewards/game_reward_func/std': 0.18717752397060394, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.35899999737739563, 'reward_std': 0.18717753887176514, 'frac_reward_zero_std': 0.0, 'kl': 0.016486212145537138, 'entropy': 0.231939148157835, 'clip_ratio/low_mean': 0.004565972252748907, 'clip_ratio/low_min': 0.004565972252748907, 'clip_ratio/high_mean': 0.0035400390625, 'clip_ratio/high_max': 0.0035400390625, 'clip_ratio/region_mean': 0.008106011315248907, 'step_time': 18.17897853399336, 'epoch': 0.11}
{'loss': 0.2652, 'grad_norm': 0.7674442529678345, 'learning_rate': 3.076578897511978e-07, 'kl': 0.02742980234324932, 'entropy': 0.27791011705994606, 'clip_ratio/low_mean': 0.009011008543893695, 'clip_ratio/low_min': 0.009011008543893695, 'clip_ratio/high_mean': 0.00439453125, 'clip_ratio/high_max': 0.00439453125, 'clip_ratio/region_mean': 0.013405539793893695, 'step_time': 1.8409142880045692, 'epoch': 0.11}
{'loss': -0.3118, 'grad_norm': 0.8517618775367737, 'learning_rate': 2.966830165065876e-07, 'num_tokens': 2718075.0, 'completions/mean_length': 1304.0, 'completions/min_length': 64.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2682499885559082, 'rewards/game_reward_func/std': 0.25883129239082336, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2682499885559082, 'reward_std': 0.25883132219314575, 'frac_reward_zero_std': 0.0, 'kl': 0.030277129262685776, 'entropy': 0.2800039239227772, 'clip_ratio/low_mean': 0.012760416720993817, 'clip_ratio/low_min': 0.012760416720993817, 'clip_ratio/high_mean': 0.00244140625, 'clip_ratio/high_max': 0.00244140625, 'clip_ratio/region_mean': 0.015201822970993817, 'step_time': 18.054268542007776, 'epoch': 0.11}
{'loss': 0.3212, 'grad_norm': 0.7301962971687317, 'learning_rate': 2.8590147820153513e-07, 'kl': 0.02299443306401372, 'entropy': 0.24937628954648972, 'clip_ratio/low_mean': 0.008636025479063392, 'clip_ratio/low_min': 0.008636025479063392, 'clip_ratio/high_mean': 0.000732421875, 'clip_ratio/high_max': 0.000732421875, 'clip_ratio/region_mean': 0.009368447354063392, 'step_time': 1.8349063430068782, 'epoch': 0.11}
{'loss': -0.1255, 'grad_norm': 0.5258030891418457, 'learning_rate': 2.7531371802285436e-07, 'num_tokens': 2729171.0, 'completions/mean_length': 1208.0, 'completions/min_length': 512.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.45749998092651367, 'rewards/game_reward_func/std': 0.07001836597919464, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.45749998092651367, 'reward_std': 0.07001835852861404, 'frac_reward_zero_std': 0.0, 'kl': 0.023336397018283606, 'entropy': 0.2555895857512951, 'clip_ratio/low_mean': 0.0007560483645647764, 'clip_ratio/low_min': 0.0007560483645647764, 'clip_ratio/high_mean': 0.010968137299641967, 'clip_ratio/high_max': 0.010968137299641967, 'clip_ratio/region_mean': 0.011724185664206743, 'step_time': 16.97767008998926, 'epoch': 0.11}
{'loss': 0.13, 'grad_norm': 0.39718589186668396, 'learning_rate': 2.6492017119189415e-07, 'kl': 0.024682591669261456, 'entropy': 0.28209157660603523, 'clip_ratio/low_mean': 0.007486979244276881, 'clip_ratio/low_min': 0.007486979244276881, 'clip_ratio/high_mean': 0.00513392873108387, 'clip_ratio/high_max': 0.00513392873108387, 'clip_ratio/region_mean': 0.012620907975360751, 'step_time': 1.8801640560195665, 'epoch': 0.11}
{'loss': 0.2068, 'grad_norm': 0.4933701455593109, 'learning_rate': 2.547212649466568e-07, 'num_tokens': 2742827.0, 'completions/mean_length': 1528.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.30425000190734863, 'rewards/game_reward_func/std': 0.26502707600593567, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.30425000190734863, 'reward_std': 0.26502707600593567, 'frac_reward_zero_std': 0.0, 'kl': 0.016187379136681557, 'entropy': 0.2614542096853256, 'clip_ratio/low_mean': 0.005738467443734407, 'clip_ratio/low_min': 0.005738467443734407, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.007691592443734407, 'step_time': 18.25486712399288, 'epoch': 0.11}
{'loss': -0.2018, 'grad_norm': 0.3667513430118561, 'learning_rate': 2.447174185242324e-07, 'kl': 0.018128524534404278, 'entropy': 0.2449686862528324, 'clip_ratio/low_mean': 0.00632440485060215, 'clip_ratio/low_min': 0.00632440485060215, 'clip_ratio/high_mean': 0.00341796875, 'clip_ratio/high_max': 0.00341796875, 'clip_ratio/region_mean': 0.00974237360060215, 'step_time': 1.7579803280023043, 'epoch': 0.11}

 91%|█████████ | 453/500 [1:15:01<09:01, 11.51s/it]
 91%|█████████ | 454/500 [1:15:04<06:40,  8.70s/it]
                                                   

 91%|█████████ | 454/500 [1:15:04<06:40,  8.70s/it][2026-03-26 13:25:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:25:19] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.6

 91%|█████████ | 455/500 [1:15:22<08:40, 11.56s/it]
                                                   

 91%|█████████ | 455/500 [1:15:22<08:40, 11.56s/it]
 91%|█████████ | 456/500 [1:15:24<06:24,  8.73s/it]
                                                   

 91%|█████████ | 456/500 [1:15:24<06:24,  8.73s/it][2026-03-26 13:25:26] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:25:40] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.9

 91%|█████████▏| 457/500 [1:15:43<08:25, 11.75s/it]
                                                   

 91%|█████████▏| 457/500 [1:15:43<08:25, 11.75s/it]
 92%|█████████▏| 458/500 [1:15:45<06:11,  8.84s/it]
                                                   

 92%|█████████▏| 458/500 [1:15:45<06:11,  8.84s/it][2026-03-26 13:25:46] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:25:58] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.2

 92%|█████████▏| 459/500 [1:15:59<07:14, 10.59s/it]
                                                   

 92%|█████████▏| 459/500 [1:15:59<07:14, 10.59s/it]
 92%|█████████▏| 460/500 [1:16:01<05:17,  7.94s/it]
                                                   

 92%|█████████▏| 460/500 [1:16:01<05:17,  7.94s/it][2026-03-26 13:26:03] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:26:17] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=23.2

 92%|█████████▏| 461/500 [1:16:20<07:15, 11.18s/it]
                                                   

 92%|█████████▏| 461/500 [1:16:20<07:15, 11.18s/it]
 92%|█████████▏| 462/500 [1:16:22<05:20,  8.43s/it]
                                                   

 92%|█████████▏| 462/500 [1:16:22<05:20,  8.43s/it][2026-03-26 13:26:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:26:38] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=23.4

 93%|█████████▎| 463/500 [1:16:40<07:03, 11.43s/it]
                                                   

 93%|█████████▎| 463/500 [1:16:40<07:03, 11.43s/it]
 93%|█████████▎| 464/500 [1:16:43<05:11,  8.64s/it]
                                                   
{'loss': -0.1587, 'grad_norm': 0.6071258187294006, 'learning_rate': 2.3490904314356412e-07, 'num_tokens': 2753027.0, 'completions/mean_length': 1096.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.382750004529953, 'rewards/game_reward_func/std': 0.17548605799674988, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.382750004529953, 'reward_std': 0.17548607289791107, 'frac_reward_zero_std': 0.0, 'kl': 0.030169783625751734, 'entropy': 0.2828335165977478, 'clip_ratio/low_mean': 0.0038519965310115367, 'clip_ratio/low_min': 0.0038519965310115367, 'clip_ratio/high_mean': 0.005428538657724857, 'clip_ratio/high_max': 0.005428538657724857, 'clip_ratio/region_mean': 0.009280535188736394, 'step_time': 17.898198096998385, 'epoch': 0.11}
{'loss': 0.1661, 'grad_norm': 0.12209390103816986, 'learning_rate': 2.2529654198854834e-07, 'kl': 0.02325057378038764, 'entropy': 0.265954103320837, 'clip_ratio/low_mean': 0.00741210940759629, 'clip_ratio/low_min': 0.00741210940759629, 'clip_ratio/high_mean': 0.0042968750931322575, 'clip_ratio/high_max': 0.0042968750931322575, 'clip_ratio/region_mean': 0.011708984500728548, 'step_time': 1.9451160759999766, 'epoch': 0.11}
{'loss': -0.0576, 'grad_norm': 0.22096499800682068, 'learning_rate': 2.1588031019145638e-07, 'num_tokens': 2765019.0, 'completions/mean_length': 1320.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.43674999475479126, 'rewards/game_reward_func/std': 0.18869762122631073, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.43674999475479126, 'reward_std': 0.18869760632514954, 'frac_reward_zero_std': 0.0, 'kl': 0.02088292222470045, 'entropy': 0.2414906993508339, 'clip_ratio/low_mean': 0.0048828125, 'clip_ratio/low_min': 0.0048828125, 'clip_ratio/high_mean': 0.0047923901583999395, 'clip_ratio/high_max': 0.0047923901583999395, 'clip_ratio/region_mean': 0.00967520265839994, 'step_time': 18.062884927989217, 'epoch': 0.11}
{'loss': 0.0621, 'grad_norm': 0.29317355155944824, 'learning_rate': 2.0666073481669714e-07, 'kl': 0.02291659382171929, 'entropy': 0.2659543938934803, 'clip_ratio/low_mean': 0.00922619050834328, 'clip_ratio/low_min': 0.00922619050834328, 'clip_ratio/high_mean': 0.0031250000465661287, 'clip_ratio/high_max': 0.0031250000465661287, 'clip_ratio/region_mean': 0.012351190554909408, 'step_time': 1.947251371006132, 'epoch': 0.11}
{'loss': 0.1874, 'grad_norm': 0.3279959559440613, 'learning_rate': 1.9763819484490353e-07, 'num_tokens': 2779187.0, 'completions/mean_length': 1592.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.41100001335144043, 'rewards/game_reward_func/std': 0.16602924466133118, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.41100001335144043, 'reward_std': 0.16602924466133118, 'frac_reward_zero_std': 0.0, 'kl': 0.0158659799490124, 'entropy': 0.26367295160889626, 'clip_ratio/low_mean': 0.004589843680150807, 'clip_ratio/low_min': 0.004589843680150807, 'clip_ratio/high_mean': 0.0022977942135185003, 'clip_ratio/high_max': 0.0022977942135185003, 'clip_ratio/region_mean': 0.006887637893669307, 'step_time': 18.609176239995577, 'epoch': 0.11}
{'loss': -0.1822, 'grad_norm': 0.37999477982521057, 'learning_rate': 1.8881306115735632e-07, 'kl': 0.016163229709491134, 'entropy': 0.253302987664938, 'clip_ratio/low_mean': 0.004920314997434616, 'clip_ratio/low_min': 0.004920314997434616, 'clip_ratio/high_mean': 0.0015869140625, 'clip_ratio/high_max': 0.0015869140625, 'clip_ratio/region_mean': 0.006507229059934616, 'step_time': 1.8452423400012776, 'epoch': 0.11}
{'loss': -0.0007, 'grad_norm': 0.1402353048324585, 'learning_rate': 1.801856965207338e-07, 'num_tokens': 2789451.0, 'completions/mean_length': 1104.0, 'completions/min_length': 576.0, 'completions/max_length': 1664.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44749999046325684, 'rewards/game_reward_func/std': 0.13483428955078125, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44749999046325684, 'reward_std': 0.13483427464962006, 'frac_reward_zero_std': 0.0, 'kl': 0.02259639324620366, 'entropy': 0.25251807272434235, 'clip_ratio/low_mean': 0.007724389201030135, 'clip_ratio/low_min': 0.007724389201030135, 'clip_ratio/high_mean': 0.001953125, 'clip_ratio/high_max': 0.001953125, 'clip_ratio/region_mean': 0.009677514201030135, 'step_time': 14.547509878997516, 'epoch': 0.11}
{'loss': 0.0064, 'grad_norm': 0.5651174783706665, 'learning_rate': 1.7175645557220567e-07, 'kl': 0.03153511695563793, 'entropy': 0.25697942823171616, 'clip_ratio/low_mean': 0.004720052005723119, 'clip_ratio/low_min': 0.004720052005723119, 'clip_ratio/high_mean': 0.010750534012913704, 'clip_ratio/high_max': 0.010750534012913704, 'clip_ratio/region_mean': 0.015470586018636823, 'step_time': 1.5807381819977309, 'epoch': 0.12}
{'loss': 0.0013, 'grad_norm': 0.23468714952468872, 'learning_rate': 1.6352568480485277e-07, 'num_tokens': 2802787.0, 'completions/mean_length': 1488.0, 'completions/min_length': 512.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.31425002217292786, 'rewards/game_reward_func/std': 0.17085479199886322, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.31425002217292786, 'reward_std': 0.17085479199886322, 'frac_reward_zero_std': 0.0, 'kl': 0.022354161366820335, 'entropy': 0.22257861867547035, 'clip_ratio/low_mean': 0.003766384324990213, 'clip_ratio/low_min': 0.003766384324990213, 'clip_ratio/high_mean': 0.007498604711145163, 'clip_ratio/high_max': 0.007498604711145163, 'clip_ratio/region_mean': 0.011264989036135375, 'step_time': 18.558050715997524, 'epoch': 0.12}
{'loss': 0.0045, 'grad_norm': 0.20901696383953094, 'learning_rate': 1.5549372255342367e-07, 'kl': 0.022035055328160524, 'entropy': 0.23602649942040443, 'clip_ratio/low_mean': 0.0030111582018435, 'clip_ratio/low_min': 0.0030111582018435, 'clip_ratio/high_mean': 0.0081787109375, 'clip_ratio/high_max': 0.0081787109375, 'clip_ratio/region_mean': 0.0111898691393435, 'step_time': 1.841771826002514, 'epoch': 0.12}
{'loss': -0.3835, 'grad_norm': 0.7765830159187317, 'learning_rate': 1.4766089898042678e-07, 'num_tokens': 2816187.0, 'completions/mean_length': 1496.0, 'completions/min_length': 704.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.42900002002716064, 'rewards/game_reward_func/std': 0.11669252067804337, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.42900002002716064, 'reward_std': 0.11669251322746277, 'frac_reward_zero_std': 0.0, 'kl': 0.021464956924319267, 'entropy': 0.24247705563902855, 'clip_ratio/low_mean': 0.006081258645281196, 'clip_ratio/low_min': 0.006081258645281196, 'clip_ratio/high_mean': 0.008807842619717121, 'clip_ratio/high_max': 0.008807842619717121, 'clip_ratio/region_mean': 0.014889101264998317, 'step_time': 18.26033660599205, 'epoch': 0.12}

 93%|█████████▎| 464/500 [1:16:43<05:11,  8.64s/it][2026-03-26 13:26:44] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:26:59] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=25.9

 93%|█████████▎| 465/500 [1:17:01<06:48, 11.66s/it]
                                                   

 93%|█████████▎| 465/500 [1:17:01<06:48, 11.66s/it]
 93%|█████████▎| 466/500 [1:17:03<04:57,  8.74s/it]
                                                   

 93%|█████████▎| 466/500 [1:17:03<04:57,  8.74s/it][2026-03-26 13:27:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:27:19] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=21.1

 93%|█████████▎| 467/500 [1:17:22<06:24, 11.64s/it]
                                                   

 93%|█████████▎| 467/500 [1:17:22<06:24, 11.64s/it]
 94%|█████████▎| 468/500 [1:17:24<04:40,  8.75s/it]
                                                   

 94%|█████████▎| 468/500 [1:17:24<04:40,  8.75s/it][2026-03-26 13:27:25] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:27:39] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:5 D:3 (0% win) | avg_turns=23.0

 94%|█████████▍| 469/500 [1:17:42<05:58, 11.58s/it]
                                                   

 94%|█████████▍| 469/500 [1:17:42<05:58, 11.58s/it]
 94%|█████████▍| 470/500 [1:17:44<04:21,  8.71s/it]
                                                   

 94%|█████████▍| 470/500 [1:17:44<04:21,  8.71s/it][2026-03-26 13:27:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:28:00] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=25.0

 94%|█████████▍| 471/500 [1:18:02<05:37, 11.63s/it]
                                                   

 94%|█████████▍| 471/500 [1:18:02<05:37, 11.63s/it]
 94%|█████████▍| 472/500 [1:18:04<04:04,  8.75s/it]
                                                   

 94%|█████████▍| 472/500 [1:18:04<04:04,  8.75s/it][2026-03-26 13:28:06] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:28:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=21.6

 95%|█████████▍| 473/500 [1:18:21<04:57, 11.00s/it]
                                                   

 95%|█████████▍| 473/500 [1:18:21<04:57, 11.00s/it]
 95%|█████████▍| 474/500 [1:18:23<03:36,  8.33s/it]
                                                   

 95%|█████████▍| 474/500 [1:18:23<03:36,  8.33s/it][2026-03-26 13:28:24] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:28:38] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=21.8

 95%|█████████▌| 475/500 [1:18:41<04:42, 11.30s/it]
                                                   
{'loss': 0.3889, 'grad_norm': 0.6783075332641602, 'learning_rate': 1.4002753606256082e-07, 'kl': 0.02113367640413344, 'entropy': 0.23178082332015038, 'clip_ratio/low_mean': 0.006001420435495675, 'clip_ratio/low_min': 0.006001420435495675, 'clip_ratio/high_mean': 0.0022615131456404924, 'clip_ratio/high_max': 0.0022615131456404924, 'clip_ratio/region_mean': 0.008262933581136167, 'step_time': 1.9474247809921508, 'epoch': 0.12}
{'loss': 0.0342, 'grad_norm': 0.06918016821146011, 'learning_rate': 1.3259394757747678e-07, 'num_tokens': 2830867.0, 'completions/mean_length': 1656.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.31675001978874207, 'rewards/game_reward_func/std': 0.18005377054214478, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.31675001978874207, 'reward_std': 0.18005375564098358, 'frac_reward_zero_std': 0.0, 'kl': 0.016677700681611896, 'entropy': 0.26418495178222656, 'clip_ratio/low_mean': 0.0022786458139307797, 'clip_ratio/low_min': 0.0022786458139307797, 'clip_ratio/high_mean': 0.0038698563585057855, 'clip_ratio/high_max': 0.0038698563585057855, 'clip_ratio/region_mean': 0.006148502172436565, 'step_time': 18.51941584299493, 'epoch': 0.12}
{'loss': -0.0282, 'grad_norm': 0.3768172562122345, 'learning_rate': 1.253604390908819e-07, 'kl': 0.014420732855796814, 'entropy': 0.21987676993012428, 'clip_ratio/low_mean': 0.0011825684196082875, 'clip_ratio/low_min': 0.0011825684196082875, 'clip_ratio/high_mean': 0.0069580078125, 'clip_ratio/high_max': 0.0069580078125, 'clip_ratio/region_mean': 0.008140576232108288, 'step_time': 1.742476211009489, 'epoch': 0.12}
{'loss': -0.041, 'grad_norm': 0.3246464133262634, 'learning_rate': 1.1832730794397951e-07, 'num_tokens': 2843115.0, 'completions/mean_length': 1352.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3362500071525574, 'rewards/game_reward_func/std': 0.22953727841377258, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3362500071525574, 'reward_std': 0.2295372635126114, 'frac_reward_zero_std': 0.0, 'kl': 0.020619317423552275, 'entropy': 0.2532557174563408, 'clip_ratio/low_mean': 0.0031617648201063275, 'clip_ratio/low_min': 0.0031617648201063275, 'clip_ratio/high_mean': 0.005551292793825269, 'clip_ratio/high_max': 0.005551292793825269, 'clip_ratio/region_mean': 0.008713057613931596, 'step_time': 18.23314853100601, 'epoch': 0.12}
{'loss': 0.0463, 'grad_norm': 0.4207240641117096, 'learning_rate': 1.1149484324124326e-07, 'kl': 0.024212128948420286, 'entropy': 0.24751721695065498, 'clip_ratio/low_mean': 0.005626860074698925, 'clip_ratio/low_min': 0.005626860074698925, 'clip_ratio/high_mean': 0.001708984375, 'clip_ratio/high_max': 0.001708984375, 'clip_ratio/region_mean': 0.007335844449698925, 'step_time': 1.8452927819962497, 'epoch': 0.12}
{'loss': -0.4417, 'grad_norm': 0.6254711747169495, 'learning_rate': 1.0486332583853565e-07, 'num_tokens': 2856323.0, 'completions/mean_length': 1472.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.2762500047683716, 'rewards/game_reward_func/std': 0.2551782429218292, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.2762500047683716, 'reward_std': 0.2551782429218292, 'frac_reward_zero_std': 0.0, 'kl': 0.02105532377026975, 'entropy': 0.2513997405767441, 'clip_ratio/low_mean': 0.0030691963620483875, 'clip_ratio/low_min': 0.0030691963620483875, 'clip_ratio/high_mean': 0.0089111328125, 'clip_ratio/high_max': 0.0089111328125, 'clip_ratio/region_mean': 0.011980329174548388, 'step_time': 17.993676468009653, 'epoch': 0.12}
{'loss': 0.4458, 'grad_norm': 0.7364402413368225, 'learning_rate': 9.843302833156377e-08, 'kl': 0.02229757234454155, 'entropy': 0.2582704722881317, 'clip_ratio/low_mean': 0.009489254211075604, 'clip_ratio/low_min': 0.009489254211075604, 'clip_ratio/high_mean': 0.0013427734375, 'clip_ratio/high_max': 0.0013427734375, 'clip_ratio/region_mean': 0.010832027648575604, 'step_time': 1.8365496729966253, 'epoch': 0.12}
{'loss': 0.0721, 'grad_norm': 0.5480031371116638, 'learning_rate': 9.22042150446728e-08, 'num_tokens': 2870555.0, 'completions/mean_length': 1600.0, 'completions/min_length': 1024.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.32074999809265137, 'rewards/game_reward_func/std': 0.18745799362659454, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.32074999809265137, 'reward_std': 0.18745799362659454, 'frac_reward_zero_std': 0.0, 'kl': 0.016179392347112298, 'entropy': 0.24834495782852173, 'clip_ratio/low_mean': 0.005625734920613468, 'clip_ratio/low_min': 0.005625734920613468, 'clip_ratio/high_mean': 0.00146484375, 'clip_ratio/high_max': 0.00146484375, 'clip_ratio/region_mean': 0.007090578670613468, 'step_time': 18.263453866995405, 'epoch': 0.12}
{'loss': -0.0662, 'grad_norm': 0.2637869417667389, 'learning_rate': 8.617714201998084e-08, 'kl': 0.020139947067946196, 'entropy': 0.23213163763284683, 'clip_ratio/low_mean': 0.003818696248345077, 'clip_ratio/low_min': 0.003818696248345077, 'clip_ratio/high_mean': 0.0037841796875, 'clip_ratio/high_max': 0.0037841796875, 'clip_ratio/region_mean': 0.007602875935845077, 'step_time': 1.8404756029995042, 'epoch': 0.12}
{'loss': 0.7779, 'grad_norm': 1.3810184001922607, 'learning_rate': 8.035205700685167e-08, 'num_tokens': 2883059.0, 'completions/mean_length': 1384.0, 'completions/min_length': 640.0, 'completions/max_length': 1856.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4572499990463257, 'rewards/game_reward_func/std': 0.09728273749351501, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4572499990463257, 'reward_std': 0.09728273004293442, 'frac_reward_zero_std': 0.0, 'kl': 0.02068252582103014, 'entropy': 0.252724252641201, 'clip_ratio/low_mean': 0.0077972341096028686, 'clip_ratio/low_min': 0.0077972341096028686, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.0077972341096028686, 'step_time': 16.071293048007647, 'epoch': 0.12}
{'loss': -0.7725, 'grad_norm': 1.2629133462905884, 'learning_rate': 7.47291994517163e-08, 'kl': 0.017404370941221714, 'entropy': 0.2545333094894886, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.011779009597375989, 'clip_ratio/high_max': 0.011779009597375989, 'clip_ratio/region_mean': 0.011779009597375989, 'step_time': 1.9461919440072961, 'epoch': 0.12}

 95%|█████████▌| 475/500 [1:18:41<04:42, 11.30s/it]
 95%|█████████▌| 476/500 [1:18:43<03:25,  8.55s/it]
                                                   

 95%|█████████▌| 476/500 [1:18:43<03:25,  8.55s/it][2026-03-26 13:28:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:28:59] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=25.4

 95%|█████████▌| 477/500 [1:19:02<04:25, 11.55s/it]
                                                   

 95%|█████████▌| 477/500 [1:19:02<04:25, 11.55s/it]
 96%|█████████▌| 478/500 [1:19:03<03:10,  8.66s/it]
                                                   

 96%|█████████▌| 478/500 [1:19:03<03:10,  8.66s/it][2026-03-26 13:29:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:29:19] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.2

 96%|█████████▌| 479/500 [1:19:22<04:03, 11.62s/it]
                                                   

 96%|█████████▌| 479/500 [1:19:22<04:03, 11.62s/it]
 96%|█████████▌| 480/500 [1:19:24<02:54,  8.74s/it]
                                                   

 96%|█████████▌| 480/500 [1:19:24<02:54,  8.74s/it][2026-03-26 13:29:26] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:29:38] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=15.9

 96%|█████████▌| 481/500 [1:19:41<03:33, 11.25s/it]
                                                   

 96%|█████████▌| 481/500 [1:19:41<03:33, 11.25s/it]
 96%|█████████▋| 482/500 [1:19:43<02:32,  8.46s/it]
                                                   

 96%|█████████▋| 482/500 [1:19:43<02:32,  8.46s/it][2026-03-26 13:29:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:29:58] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=20.5

 97%|█████████▋| 483/500 [1:20:01<03:12, 11.34s/it]
                                                   

 97%|█████████▋| 483/500 [1:20:01<03:12, 11.34s/it]
 97%|█████████▋| 484/500 [1:20:03<02:17,  8.57s/it]
                                                   

 97%|█████████▋| 484/500 [1:20:03<02:17,  8.57s/it][2026-03-26 13:30:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:30:19] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=24.5

 97%|█████████▋| 485/500 [1:20:22<02:52, 11.50s/it]
                                                   

 97%|█████████▋| 485/500 [1:20:22<02:52, 11.50s/it]
 97%|█████████▋| 486/500 [1:20:24<02:01,  8.69s/it]
                                                   
{'loss': -0.3989, 'grad_norm': 0.486213356256485, 'learning_rate': 6.930880048822531e-08, 'num_tokens': 2895627.0, 'completions/mean_length': 1392.0, 'completions/min_length': 960.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.39375001192092896, 'rewards/game_reward_func/std': 0.20081673562526703, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.39375001192092896, 'reward_std': 0.20081673562526703, 'frac_reward_zero_std': 0.0, 'kl': 0.0183717361651361, 'entropy': 0.21915575116872787, 'clip_ratio/low_mean': 0.005418346729129553, 'clip_ratio/low_min': 0.005418346729129553, 'clip_ratio/high_mean': 0.004817708395421505, 'clip_ratio/high_max': 0.004817708395421505, 'clip_ratio/region_mean': 0.010236055124551058, 'step_time': 18.05837312699441, 'epoch': 0.12}
{'loss': 0.4051, 'grad_norm': 0.6296974420547485, 'learning_rate': 6.409108292774912e-08, 'kl': 0.01889330940321088, 'entropy': 0.26226720586419106, 'clip_ratio/low_mean': 0.004250006517395377, 'clip_ratio/low_min': 0.004250006517395377, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.008156256517395377, 'step_time': 1.9353525660044397, 'epoch': 0.12}
{'loss': 0.4323, 'grad_norm': 0.5126423835754395, 'learning_rate': 5.907626125022159e-08, 'num_tokens': 2910051.0, 'completions/mean_length': 1624.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3877499997615814, 'rewards/game_reward_func/std': 0.15949003398418427, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3877499997615814, 'reward_std': 0.15949003398418427, 'frac_reward_zero_std': 0.0, 'kl': 0.022889353334903717, 'entropy': 0.26392344012856483, 'clip_ratio/low_mean': 0.006450281653087586, 'clip_ratio/low_min': 0.006450281653087586, 'clip_ratio/high_mean': 0.0047433036379516125, 'clip_ratio/high_max': 0.0047433036379516125, 'clip_ratio/region_mean': 0.011193585291039199, 'step_time': 18.3521693809962, 'epoch': 0.12}
{'loss': -0.429, 'grad_norm': 0.6641908884048462, 'learning_rate': 5.426454159531913e-08, 'kl': 0.015432130778208375, 'entropy': 0.2353331446647644, 'clip_ratio/low_mean': 0.003073952393606305, 'clip_ratio/low_min': 0.003073952393606305, 'clip_ratio/high_mean': 0.003993442980572581, 'clip_ratio/high_max': 0.003993442980572581, 'clip_ratio/region_mean': 0.007067395374178886, 'step_time': 1.7486293879992445, 'epoch': 0.12}
{'loss': 0.3388, 'grad_norm': 0.5007304549217224, 'learning_rate': 4.9656121753990924e-08, 'num_tokens': 2923899.0, 'completions/mean_length': 1552.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.3889999985694885, 'rewards/game_reward_func/std': 0.2115790992975235, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.3889999985694885, 'reward_std': 0.2115790992975235, 'frac_reward_zero_std': 0.0, 'kl': 0.019938458688557148, 'entropy': 0.25136031582951546, 'clip_ratio/low_mean': 0.007219587103463709, 'clip_ratio/low_min': 0.007219587103463709, 'clip_ratio/high_mean': 0.0024857954122126102, 'clip_ratio/high_max': 0.0024857954122126102, 'clip_ratio/region_mean': 0.00970538251567632, 'step_time': 18.312325712002348, 'epoch': 0.12}
{'loss': -0.334, 'grad_norm': 0.38929301500320435, 'learning_rate': 4.52511911603265e-08, 'kl': 0.016378837637603283, 'entropy': 0.23113776370882988, 'clip_ratio/low_mean': 0.003272045636549592, 'clip_ratio/low_min': 0.003272045636549592, 'clip_ratio/high_mean': 0.0054510708432644606, 'clip_ratio/high_max': 0.0054510708432644606, 'clip_ratio/region_mean': 0.008723116479814053, 'step_time': 1.8528607570115128, 'epoch': 0.12}
{'loss': 0.3504, 'grad_norm': 0.7294067144393921, 'learning_rate': 4.104993088376974e-08, 'num_tokens': 2933459.0, 'completions/mean_length': 1016.0, 'completions/min_length': 448.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4375, 'rewards/game_reward_func/std': 0.05869777500629425, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4375, 'reward_std': 0.058697763830423355, 'frac_reward_zero_std': 0.0, 'kl': 0.029276788467541337, 'entropy': 0.2621674910187721, 'clip_ratio/low_mean': 0.012062156572937965, 'clip_ratio/low_min': 0.012062156572937965, 'clip_ratio/high_mean': 0.0027173913549631834, 'clip_ratio/high_max': 0.0027173913549631834, 'clip_ratio/region_mean': 0.014779547927901149, 'step_time': 16.924980686002527, 'epoch': 0.12}
{'loss': -0.3467, 'grad_norm': 0.8712191581726074, 'learning_rate': 3.705251362167484e-08, 'kl': 0.025214317720383406, 'entropy': 0.2504439800977707, 'clip_ratio/low_mean': 0.0062478999607264996, 'clip_ratio/low_min': 0.0062478999607264996, 'clip_ratio/high_mean': 0.006990131689235568, 'clip_ratio/high_max': 0.006990131689235568, 'clip_ratio/region_mean': 0.013238031649962068, 'step_time': 1.7781266339952708, 'epoch': 0.12}
{'loss': -0.1147, 'grad_norm': 0.24038660526275635, 'learning_rate': 3.325910369220975e-08, 'num_tokens': 2945387.0, 'completions/mean_length': 1312.0, 'completions/min_length': 320.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.44050002098083496, 'rewards/game_reward_func/std': 0.14082106947898865, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.44050002098083496, 'reward_std': 0.14082106947898865, 'frac_reward_zero_std': 0.0, 'kl': 0.01479291613213718, 'entropy': 0.23516176640987396, 'clip_ratio/low_mean': 0.0064538042061030865, 'clip_ratio/low_min': 0.0064538042061030865, 'clip_ratio/high_mean': 0.0021275111939758062, 'clip_ratio/high_max': 0.0021275111939758062, 'clip_ratio/region_mean': 0.008581315400078893, 'step_time': 17.862910484000167, 'epoch': 0.12}
{'loss': 0.1214, 'grad_norm': 0.308159738779068, 'learning_rate': 2.966985702759828e-08, 'kl': 0.030381083488464355, 'entropy': 0.2741697393357754, 'clip_ratio/low_mean': 0.011086309619713575, 'clip_ratio/low_min': 0.011086309619713575, 'clip_ratio/high_mean': 0.000732421875, 'clip_ratio/high_max': 0.000732421875, 'clip_ratio/region_mean': 0.011818731494713575, 'step_time': 1.927848185005132, 'epoch': 0.12}
{'loss': -0.6298, 'grad_norm': 0.7909207344055176, 'learning_rate': 2.6284921167712975e-08, 'num_tokens': 2959363.0, 'completions/mean_length': 1568.0, 'completions/min_length': 896.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.41200000047683716, 'rewards/game_reward_func/std': 0.19840219616889954, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.41200000047683716, 'reward_std': 0.19840218126773834, 'frac_reward_zero_std': 0.0, 'kl': 0.014891249593347311, 'entropy': 0.21689296886324883, 'clip_ratio/low_mean': 0.0027343749534338713, 'clip_ratio/low_min': 0.0027343749534338713, 'clip_ratio/high_mean': 0.006061056395992637, 'clip_ratio/high_max': 0.006061056395992637, 'clip_ratio/region_mean': 0.008795431349426508, 'step_time': 18.143936806001875, 'epoch': 0.12}

 97%|█████████▋| 486/500 [1:20:24<02:01,  8.69s/it][2026-03-26 13:30:25] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:30:38] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:8 D:0 (0% win) | avg_turns=17.1

 97%|█████████▋| 487/500 [1:20:41<02:24, 11.14s/it]
                                                   

 97%|█████████▋| 487/500 [1:20:41<02:24, 11.14s/it]
 98%|█████████▊| 488/500 [1:20:43<01:41,  8.50s/it]
                                                   

 98%|█████████▊| 488/500 [1:20:43<01:41,  8.50s/it][2026-03-26 13:30:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:30:58] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.4

 98%|█████████▊| 489/500 [1:21:01<02:05, 11.37s/it]
                                                   

 98%|█████████▊| 489/500 [1:21:01<02:05, 11.37s/it]
 98%|█████████▊| 490/500 [1:21:03<01:25,  8.58s/it]
                                                   

 98%|█████████▊| 490/500 [1:21:03<01:25,  8.58s/it][2026-03-26 13:31:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:31:18] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=19.5

 98%|█████████▊| 491/500 [1:21:21<01:42, 11.43s/it]
                                                   

 98%|█████████▊| 491/500 [1:21:21<01:42, 11.43s/it]
 98%|█████████▊| 492/500 [1:21:23<01:08,  8.58s/it]
                                                   

 98%|█████████▊| 492/500 [1:21:23<01:08,  8.58s/it][2026-03-26 13:31:25] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:31:39] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=22.2

 99%|█████████▊| 493/500 [1:21:41<01:20, 11.46s/it]
                                                   

 99%|█████████▊| 493/500 [1:21:41<01:20, 11.46s/it]
 99%|█████████▉| 494/500 [1:21:43<00:51,  8.63s/it]
                                                   

 99%|█████████▉| 494/500 [1:21:43<00:51,  8.63s/it][2026-03-26 13:31:45] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:31:58] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=17.2

 99%|█████████▉| 495/500 [1:22:01<00:57, 11.45s/it]
                                                   

 99%|█████████▉| 495/500 [1:22:01<00:57, 11.45s/it]
 99%|█████████▉| 496/500 [1:22:03<00:34,  8.62s/it]
                                                   

 99%|█████████▉| 496/500 [1:22:03<00:34,  8.62s/it][2026-03-26 13:32:05] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:32:19] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:7 D:1 (0% win) | avg_turns=19.0

 99%|█████████▉| 497/500 [1:22:21<00:34, 11.42s/it]
                                                   
{'loss': 0.6358, 'grad_norm': 0.8982511162757874, 'learning_rate': 2.3104435254008852e-08, 'kl': 0.019364329054951668, 'entropy': 0.2677430212497711, 'clip_ratio/low_mean': 0.003618647111579776, 'clip_ratio/low_min': 0.003618647111579776, 'clip_ratio/high_mean': 0.0016447368543595076, 'clip_ratio/high_max': 0.0016447368543595076, 'clip_ratio/region_mean': 0.005263383965939283, 'step_time': 1.9340580319985747, 'epoch': 0.12}
{'loss': -0.4225, 'grad_norm': 1.106363296508789, 'learning_rate': 2.012853002380466e-08, 'num_tokens': 2969563.0, 'completions/mean_length': 1096.0, 'completions/min_length': 128.0, 'completions/max_length': 1984.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.38099998235702515, 'rewards/game_reward_func/std': 0.13099181652069092, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.38099998235702515, 'reward_std': 0.13099181652069092, 'frac_reward_zero_std': 0.0, 'kl': 0.04323049425147474, 'entropy': 0.26949723064899445, 'clip_ratio/low_mean': 0.001953125, 'clip_ratio/low_min': 0.001953125, 'clip_ratio/high_mean': 0.017063591862097383, 'clip_ratio/high_max': 0.017063591862097383, 'clip_ratio/region_mean': 0.019016716862097383, 'step_time': 16.699984020997363, 'epoch': 0.12}
{'loss': 0.4306, 'grad_norm': 0.4367543160915375, 'learning_rate': 1.735732780490884e-08, 'kl': 0.02412146981805563, 'entropy': 0.25442226231098175, 'clip_ratio/low_mean': 0.007772436132654548, 'clip_ratio/low_min': 0.007772436132654548, 'clip_ratio/high_mean': 0.0049715908244252205, 'clip_ratio/high_max': 0.0049715908244252205, 'clip_ratio/region_mean': 0.012744026957079768, 'step_time': 2.1435261930018896, 'epoch': 0.12}
{'loss': 0.2099, 'grad_norm': 0.21515361964702606, 'learning_rate': 1.4790942510590767e-08, 'num_tokens': 2980915.0, 'completions/mean_length': 1240.0, 'completions/min_length': 768.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4000000059604645, 'rewards/game_reward_func/std': 0.23590797185897827, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4000000059604645, 'reward_std': 0.23590797185897827, 'frac_reward_zero_std': 0.0, 'kl': 0.023610908072441816, 'entropy': 0.2555149681866169, 'clip_ratio/low_mean': 0.005533854244276881, 'clip_ratio/low_min': 0.005533854244276881, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.009440104244276881, 'step_time': 17.892113784000685, 'epoch': 0.12}
{'loss': -0.2038, 'grad_norm': 0.32579460740089417, 'learning_rate': 1.2429479634897268e-08, 'kl': 0.019024748587980866, 'entropy': 0.2859521731734276, 'clip_ratio/low_mean': 0.003952752915211022, 'clip_ratio/low_min': 0.003952752915211022, 'clip_ratio/high_mean': 0.0030950730433687568, 'clip_ratio/high_max': 0.0030950730433687568, 'clip_ratio/region_mean': 0.007047825958579779, 'step_time': 1.862066434987355, 'epoch': 0.12}
{'loss': 0.2455, 'grad_norm': 0.45275622606277466, 'learning_rate': 1.0273036248318325e-08, 'num_tokens': 2992331.0, 'completions/mean_length': 1248.0, 'completions/min_length': 640.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.32100000977516174, 'rewards/game_reward_func/std': 0.22687189280986786, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.32100000977516174, 'reward_std': 0.22687189280986786, 'frac_reward_zero_std': 0.0, 'kl': 0.02862647664733231, 'entropy': 0.2616366446018219, 'clip_ratio/low_mean': 0.0065805286867544055, 'clip_ratio/low_min': 0.0065805286867544055, 'clip_ratio/high_mean': 0.008132102200761437, 'clip_ratio/high_max': 0.008132102200761437, 'clip_ratio/region_mean': 0.014712630887515843, 'step_time': 17.9250940560014, 'epoch': 0.12}
{'loss': -0.2382, 'grad_norm': 0.3538530766963959, 'learning_rate': 8.321700993795812e-09, 'kl': 0.019875663332641125, 'entropy': 0.25945261493325233, 'clip_ratio/low_mean': 0.004853219841606915, 'clip_ratio/low_min': 0.004853219841606915, 'clip_ratio/high_mean': 0.0048828125, 'clip_ratio/high_max': 0.0048828125, 'clip_ratio/region_mean': 0.009736032341606915, 'step_time': 1.7464512929946068, 'epoch': 0.12}
{'loss': -0.2117, 'grad_norm': 0.6108260154724121, 'learning_rate': 6.575554083078084e-09, 'num_tokens': 3005155.0, 'completions/mean_length': 1424.0, 'completions/min_length': 448.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.24150002002716064, 'rewards/game_reward_func/std': 0.20590358972549438, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.24150002002716064, 'reward_std': 0.2059035748243332, 'frac_reward_zero_std': 0.0, 'kl': 0.027503940742462873, 'entropy': 0.24392016977071762, 'clip_ratio/low_mean': 0.0013671874767169356, 'clip_ratio/low_min': 0.0013671874767169356, 'clip_ratio/high_mean': 0.005487351329065859, 'clip_ratio/high_max': 0.005487351329065859, 'clip_ratio/region_mean': 0.006854538805782795, 'step_time': 18.00928817599197, 'epoch': 0.12}
{'loss': 0.2196, 'grad_norm': 0.48659634590148926, 'learning_rate': 5.034667293427053e-09, 'kl': 0.018518929602578282, 'entropy': 0.22046158835291862, 'clip_ratio/low_mean': 0.004603794659487903, 'clip_ratio/low_min': 0.004603794659487903, 'clip_ratio/high_mean': 0.00406588043551892, 'clip_ratio/high_max': 0.00406588043551892, 'clip_ratio/region_mean': 0.008669675095006824, 'step_time': 1.838609753001947, 'epoch': 0.12}
{'loss': -0.2639, 'grad_norm': 0.7061721086502075, 'learning_rate': 3.6991039646616657e-09, 'num_tokens': 3015419.0, 'completions/mean_length': 1104.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4255000054836273, 'rewards/game_reward_func/std': 0.1610279530286789, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4255000054836273, 'reward_std': 0.1610279530286789, 'frac_reward_zero_std': 0.0, 'kl': 0.03771575796417892, 'entropy': 0.27812253683805466, 'clip_ratio/low_mean': 0.010470920242369175, 'clip_ratio/low_min': 0.010470920242369175, 'clip_ratio/high_mean': 0.015380859375, 'clip_ratio/high_max': 0.015380859375, 'clip_ratio/region_mean': 0.025851779617369175, 'step_time': 17.864062805994763, 'epoch': 0.12}
{'loss': 0.2706, 'grad_norm': 0.6266105771064758, 'learning_rate': 2.568918996560532e-09, 'kl': 0.01773937908001244, 'entropy': 0.25693894922733307, 'clip_ratio/low_mean': 0.0062184936250559986, 'clip_ratio/low_min': 0.0062184936250559986, 'clip_ratio/high_mean': 0.0015869140625, 'clip_ratio/high_max': 0.0015869140625, 'clip_ratio/region_mean': 0.0078054076875559986, 'step_time': 1.8376672540034633, 'epoch': 0.12}

 99%|█████████▉| 497/500 [1:22:21<00:34, 11.42s/it]
100%|█████████▉| 498/500 [1:22:23<00:17,  8.60s/it]
                                                   

100%|█████████▉| 498/500 [1:22:23<00:17,  8.60s/it][2026-03-26 13:32:25] INFO train_trl_v2.py:394: Rollout: playing 8 games (opponent=random)
[2026-03-26 13:32:39] INFO train_trl_v2.py:478: Rollout done: 8 games (0 failed) | W:0 L:6 D:2 (0% win) | avg_turns=20.5

100%|█████████▉| 499/500 [1:22:41<00:11, 11.47s/it]
                                                   

100%|█████████▉| 499/500 [1:22:41<00:11, 11.47s/it]
100%|██████████| 500/500 [1:22:43<00:00,  8.60s/it]
                                                   

100%|██████████| 500/500 [1:22:43<00:00,  8.60s/it]
                                                   

100%|██████████| 500/500 [1:22:44<00:00,  8.60s/it]
100%|██████████| 500/500 [1:22:44<00:00,  9.93s/it]
[2026-03-26 13:32:45] INFO train_trl_v2.py:799: ============================================================
[2026-03-26 13:32:45] INFO train_trl_v2.py:800: Training ended after 82.7 minutes (4965 seconds)
[2026-03-26 13:32:45] INFO train_trl_v2.py:805: Saving final LoRA adapter to /workspace/grpo-ginrummy-v2/checkpoints/final
[2026-03-26 13:32:46] INFO train_trl_v2.py:808: Model saved successfully.
wandb: updating run metadata
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
wandb: 
wandb: Run history:
wandb:                 profiling/Time taken: GRPOTrainer._calculate_rewards ▆▄▇▅▇▅▇▄▅▃▆▅▄▃▆▂█▁▄▂▅▅▆▇▅▄▄▅▅█▆▅▆▃▆▅▅▅▄▄
wandb: profiling/Time taken: GRPOTrainer._get_per_token_logps_and_entropies ▁█▁▁▁▁▁▁▁▇▁▁▁▁▁▁▁▁▁▇▇▁▁█▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁
wandb:                    profiling/Time taken: GRPOTrainer._prepare_inputs ██▁▁▇▁▁▁▁▁▁▁▁▁▁▁█▁▁▁█▁▁▁▇▁▁▁▁▁▁█▁▁▇▁▁▁▁▁
wandb:                       profiling/Time taken: GRPOTrainer.compute_loss ▂▂█▂▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▂▂▂▁▂▂▁▂▂▂▂▂▂▂▁▂▂▂▁▂▂
wandb:                 profiling/Time taken: GRPOTrainer.format_reward_func ▄▃▁▁▅▅▃▅▃▃▃▄▄▄▅▄▃▆▆▅▄▄▃▅▄▃▃▃▃▄▃▄▄▃▃█▂▄▄▅
wandb:                   profiling/Time taken: GRPOTrainer.game_reward_func ▇▇▁▂▄▂▄█▄▂▅▂▂▅▄▄▅▃▄▂▄▄▆▁▅▆▂▅▃▂▃▂▃▁▂▅▄▅█▃
wandb:                                                 rollout/avg_deadwood ▂▇▄▄▅▃▅▆▃▆▄▆▆▆▅▃▅█▆█▇▄▄▄█▇▄██▃▅▅▄▅█▅▆▆▆▁
wandb:                                              rollout/avg_game_length ▆▆▄▆▅▄▇▄▅▄▆▄▅█▄▆▁▁█▄▆▅▁▅▄▅▂▂▅▆▄▅▅▅▆▄▆▇▇▅
wandb:                                                        rollout/draws ▃▁█▅▅▁▃▆▃▃▃▅▁▃▆▆▅▁▁▅▅▃▆▃▃▁▅▃▁▅▃▃▁█▅▃▆▃▅▃
wandb:                                                 rollout/failed_games ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:                                                                  +36 ...
wandb: 
wandb: Run summary:
wandb:                 profiling/Time taken: GRPOTrainer._calculate_rewards 0.00429
wandb: profiling/Time taken: GRPOTrainer._get_per_token_logps_and_entropies 0.08853
wandb:                    profiling/Time taken: GRPOTrainer._prepare_inputs 0.0
wandb:                       profiling/Time taken: GRPOTrainer.compute_loss 0.13397
wandb:                 profiling/Time taken: GRPOTrainer.format_reward_func 0.00336
wandb:                   profiling/Time taken: GRPOTrainer.game_reward_func 0.00016
wandb:                                                 rollout/avg_deadwood 54
wandb:                                              rollout/avg_game_length 20.5
wandb:                                                        rollout/draws 2
wandb:                                                 rollout/failed_games 0
wandb:                                                                  +41 ...
wandb: 
wandb: 🚀 View run grpo-v2-20260326_120924 at: https://wandb.ai/good-start-labs/ginrummy-grpo/runs/mvhlef9f
wandb: ⭐️ View project at: https://wandb.ai/good-start-labs/ginrummy-grpo
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20260326_120924-mvhlef9f/logs
{'loss': 0.2766, 'grad_norm': 0.3811933994293213, 'learning_rate': 1.6441588466009627e-09, 'num_tokens': 3026579.0, 'completions/mean_length': 1216.0, 'completions/min_length': 256.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.4050000011920929, 'rewards/game_reward_func/std': 0.14274853467941284, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.4050000011920929, 'reward_std': 0.14274853467941284, 'frac_reward_zero_std': 0.0, 'kl': 0.026874777860939503, 'entropy': 0.23387732729315758, 'clip_ratio/low_mean': 0.007039387943223119, 'clip_ratio/low_min': 0.007039387943223119, 'clip_ratio/high_mean': 0.00390625, 'clip_ratio/high_max': 0.00390625, 'clip_ratio/region_mean': 0.010945637943223119, 'step_time': 17.762669515992457, 'epoch': 0.12}
{'loss': -0.2696, 'grad_norm': 0.41915178298950195, 'learning_rate': 9.248615280499362e-10, 'kl': 0.03727127565070987, 'entropy': 0.2431403435766697, 'clip_ratio/low_mean': 0.00881076417863369, 'clip_ratio/low_min': 0.00881076417863369, 'clip_ratio/high_mean': 0.00244140625, 'clip_ratio/high_max': 0.00244140625, 'clip_ratio/region_mean': 0.01125217042863369, 'step_time': 1.8349889699966297, 'epoch': 0.12}
{'loss': 0.6385, 'grad_norm': 0.690373420715332, 'learning_rate': 4.1105660840368154e-10, 'num_tokens': 3038507.0, 'completions/mean_length': 1312.0, 'completions/min_length': 128.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 1.0, 'completions/mean_terminated_length': 0.0, 'completions/min_terminated_length': 0.0, 'completions/max_terminated_length': 0.0, 'rewards/game_reward_func/mean': -0.31174999475479126, 'rewards/game_reward_func/std': 0.24622972309589386, 'rewards/format_reward_func/mean': 0.0, 'rewards/format_reward_func/std': 0.0, 'reward': -0.31174999475479126, 'reward_std': 0.24622972309589386, 'frac_reward_zero_std': 0.0, 'kl': 0.023132808972150087, 'entropy': 0.27511948347091675, 'clip_ratio/low_mean': 0.005211734329350293, 'clip_ratio/low_min': 0.005211734329350293, 'clip_ratio/high_mean': 0.009114583022892475, 'clip_ratio/high_max': 0.009114583022892475, 'clip_ratio/region_mean': 0.014326317352242768, 'step_time': 17.96897410399106, 'epoch': 0.12}
{'loss': -0.6307, 'grad_norm': 0.7578749656677246, 'learning_rate': 1.0276520816976388e-10, 'kl': 0.04273142293095589, 'entropy': 0.2925068140029907, 'clip_ratio/low_mean': 0.013671875, 'clip_ratio/low_min': 0.013671875, 'clip_ratio/high_mean': 0.0089111328125, 'clip_ratio/high_max': 0.0089111328125, 'clip_ratio/region_mean': 0.0225830078125, 'step_time': 1.7494091139960801, 'epoch': 0.12}
{'train_runtime': 4964.3126, 'train_samples_per_second': 0.403, 'train_steps_per_second': 0.101, 'train_loss': 0.002988333396613598, 'epoch': 0.12}