log.txt

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
2025-07-10 17:14:24,734	INFO worker.py:1812 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8266 
SYSTEM_PROMPT: You FIRST think about the reasoning process as an internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within <think> </think> tags. The final answer MUST BE put in \boxed{}.

  0%|                                                                      | 0/3940 [00:00<?, ?it/s]
  1%|▋                                                           | 46/3940 [00:00<00:09, 432.53it/s]
  3%|█▋                                                         | 109/3940 [00:00<00:07, 540.13it/s]
  4%|██▌                                                        | 174/3940 [00:00<00:06, 585.43it/s]
  6%|███▍                                                       | 233/3940 [00:00<00:07, 510.36it/s]
  8%|████▋                                                      | 309/3940 [00:00<00:06, 581.46it/s]
  9%|█████▌                                                     | 369/3940 [00:00<00:06, 586.18it/s]
 11%|██████▍                                                    | 429/3940 [00:00<00:05, 588.46it/s]
 12%|███████▎                                                   | 489/3940 [00:00<00:06, 547.34it/s]
 14%|████████▏                                                  | 545/3940 [00:00<00:06, 534.80it/s]
 16%|█████████▏                                                 | 613/3940 [00:01<00:05, 576.11it/s]
 17%|██████████▎                                                | 685/3940 [00:01<00:05, 617.34it/s]
 19%|███████████▎                                               | 753/3940 [00:01<00:05, 634.95it/s]
 21%|████████████▏                                              | 818/3940 [00:01<00:05, 586.41it/s]
 22%|█████████████▏                                             | 878/3940 [00:01<00:05, 578.40it/s]
 24%|██████████████▎                                            | 952/3940 [00:01<00:04, 610.40it/s]
 26%|██████████████▉                                           | 1014/3940 [00:01<00:04, 599.02it/s]
 28%|████████████████                                          | 1089/3940 [00:01<00:04, 635.13it/s]
 30%|█████████████████▏                                        | 1167/3940 [00:01<00:04, 674.98it/s]
 31%|██████████████████▏                                       | 1235/3940 [00:02<00:04, 613.71it/s]
 33%|███████████████████▎                                      | 1315/3940 [00:02<00:04, 653.93it/s]
 35%|████████████████████▍                                     | 1385/3940 [00:02<00:03, 657.72it/s]
 37%|█████████████████████▎                                    | 1452/3940 [00:02<00:04, 577.56it/s]
 39%|██████████████████████▌                                   | 1532/3940 [00:02<00:03, 635.11it/s]
 41%|███████████████████████▌                                  | 1598/3940 [00:02<00:04, 538.71it/s]
 42%|████████████████████████▌                                 | 1665/3940 [00:02<00:04, 562.15it/s]
 44%|█████████████████████████▍                                | 1731/3940 [00:02<00:03, 586.27it/s]
 46%|██████████████████████████▍                               | 1799/3940 [00:03<00:03, 606.29it/s]
 48%|███████████████████████████▌                              | 1872/3940 [00:03<00:03, 634.24it/s]
 49%|████████████████████████████▌                             | 1941/3940 [00:03<00:03, 649.47it/s]
 51%|█████████████████████████████▌                            | 2008/3940 [00:03<00:03, 610.92it/s]
 53%|██████████████████████████████▍                           | 2071/3940 [00:03<00:03, 530.59it/s]
 54%|███████████████████████████████▎                          | 2130/3940 [00:03<00:03, 532.79it/s]
 56%|████████████████████████████████▍                         | 2201/3940 [00:03<00:03, 571.37it/s]
 58%|█████████████████████████████████▎                        | 2267/3940 [00:03<00:02, 589.56it/s]
 60%|██████████████████████████████████▌                       | 2346/3940 [00:03<00:02, 643.67it/s]
 61%|███████████████████████████████████▋                      | 2421/3940 [00:04<00:02, 668.23it/s]
 63%|████████████████████████████████████▋                     | 2489/3940 [00:04<00:02, 654.41it/s]
 65%|█████████████████████████████████████▋                    | 2556/3940 [00:04<00:02, 550.73it/s]
 67%|██████████████████████████████████████▋                   | 2629/3940 [00:04<00:02, 594.99it/s]
 68%|███████████████████████████████████████▋                  | 2692/3940 [00:04<00:02, 585.95it/s]
 70%|████████████████████████████████████████▊                 | 2775/3940 [00:04<00:01, 647.23it/s]
 72%|█████████████████████████████████████████▊                | 2842/3940 [00:04<00:01, 626.41it/s]
 74%|██████████████████████████████████████████▊               | 2911/3940 [00:04<00:01, 630.79it/s]
 76%|████████████████████████████████████████████              | 2991/3940 [00:04<00:01, 676.88it/s]
 78%|█████████████████████████████████████████████▏            | 3067/3940 [00:05<00:01, 700.16it/s]
 80%|██████████████████████████████████████████████▏           | 3138/3940 [00:05<00:01, 644.79it/s]
 82%|███████████████████████████████████████████████▌          | 3228/3940 [00:05<00:01, 711.65it/s]
 84%|████████████████████████████████████████████████▌         | 3301/3940 [00:05<00:00, 669.29it/s]
 86%|█████████████████████████████████████████████████▌        | 3370/3940 [00:05<00:00, 643.05it/s]
 87%|██████████████████████████████████████████████████▌       | 3436/3940 [00:05<00:00, 614.63it/s]
 89%|███████████████████████████████████████████████████▌      | 3501/3940 [00:05<00:00, 621.44it/s]
 90%|████████████████████████████████████████████████████▍     | 3564/3940 [00:05<00:00, 547.63it/s]
 92%|█████████████████████████████████████████████████████▎    | 3621/3940 [00:06<00:00, 514.46it/s]
 94%|██████████████████████████████████████████████████████▎   | 3691/3940 [00:06<00:00, 558.83it/s]
 96%|███████████████████████████████████████████████████████▍  | 3764/3940 [00:06<00:00, 600.34it/s]
 97%|████████████████████████████████████████████████████████▍ | 3833/3940 [00:06<00:00, 622.67it/s]
 99%|█████████████████████████████████████████████████████████▎| 3897/3940 [00:06<00:00, 602.30it/s]
100%|██████████████████████████████████████████████████████████| 3940/3940 [00:07<00:00, 562.67it/s]
(infer pid=654445) 
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
(infer pid=654445) 
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:05<00:05,  5.99s/it]
(infer pid=654459) 
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s] [repeated 3x across cluster]
(infer pid=654445) 
(infer pid=654445) 
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:18<00:00,  9.39s/it] [repeated 5x across cluster]
(infer pid=654457) 
(infer pid=654443) 
(infer pid=654459) 
(infer pid=654457) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
(infer pid=654457) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
(infer pid=654459) 
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:15<00:00,  7.58s/it] [repeated 6x across cluster]
(infer pid=654445) Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. [repeated 3x across cluster]
(infer pid=654457) Token indices sequence length is longer than the specified maximum sequence length for this model (180224 > 131072). Running this sequence through the model will result in indexing errors
(infer pid=654457) 
Capturing CUDA graph shapes:   0%|                                           | 0/35 [00:00<?, ?it/s]
(infer pid=654445) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again. [repeated 3x across cluster]
(infer pid=654445) Token indices sequence length is longer than the specified maximum sequence length for this model (180224 > 131072). Running this sequence through the model will result in indexing errors [repeated 3x across cluster]
(infer pid=654457) 
Capturing CUDA graph shapes:   3%|█                                  | 1/35 [00:00<00:17,  1.99it/s]
(infer pid=654445) 
Capturing CUDA graph shapes:   0%|                                           | 0/35 [00:00<?, ?it/s] [repeated 3x across cluster]
(infer pid=654459) 
Capturing CUDA graph shapes:  29%|█████████▋                        | 10/35 [00:05<00:12,  1.96it/s] [repeated 33x across cluster]
(infer pid=654459) 
Capturing CUDA graph shapes:  57%|███████████████████▍              | 20/35 [00:10<00:07,  2.00it/s] [repeated 37x across cluster]
(infer pid=654445) 
Capturing CUDA graph shapes:  74%|█████████████████████████▎        | 26/35 [00:14<00:04,  1.90it/s] [repeated 39x across cluster]
(infer pid=654457) 
Capturing CUDA graph shapes: 100%|██████████████████████████████████| 35/35 [00:17<00:00,  1.98it/s]
Capturing CUDA graph shapes: 100%|██████████████████████████████████| 35/35 [00:17<00:00,  2.03it/s]
(infer pid=654443) 
Capturing CUDA graph shapes:  89%|██████████████████████████████    | 31/35 [00:19<00:02,  1.60it/s] [repeated 23x across cluster]
(infer pid=654445) 
Capturing CUDA graph shapes: 100%|██████████████████████████████████| 35/35 [00:19<00:00,  1.90it/s]
Capturing CUDA graph shapes: 100%|██████████████████████████████████| 35/35 [00:19<00:00,  1.84it/s] [repeated 2x across cluster]
(infer pid=654457) 
Processed prompts:   0%| | 0/985 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s
(infer pid=654443) 
Capturing CUDA graph shapes:  97%|█████████████████████████████████ | 34/35 [00:21<00:00,  1.60it/s] [repeated 3x across cluster]
(infer pid=654443) 
Capturing CUDA graph shapes: 100%|██████████████████████████████████| 35/35 [00:21<00:00,  1.58it/s]
Capturing CUDA graph shapes: 100%|██████████████████████████████████| 35/35 [00:21<00:00,  1.60it/s]
(infer pid=654443) 
Processed prompts:   0%| | 0/985 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s [repeated 3x across cluster]
(infer pid=654457) 
Processed prompts:   0%| | 1/985 [00:40<11:11:20, 40.94s/it, est. speed input: 32.10 toks/s, output:
(infer pid=654457) 
Processed prompts:   0%| | 2/985 [00:46<5:26:46, 19.95s/it, est. speed input: 36.05 toks/s, output:  [repeated 2x across cluster]
(infer pid=654459) 
Processed prompts:   1%| | 5/985 [00:50<1:36:10,  5.89s/it, est. speed input: 76.08 toks/s, output:  [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:   1%| | 5/985 [00:48<1:19:53,  4.89s/it, est. speed input: 45.92 toks/s, output:  [repeated 12x across cluster]
(infer pid=654445) 
Processed prompts:   1%| | 12/985 [00:57<18:37,  1.15s/it, est. speed input: 116.42 toks/s, output:  [repeated 15x across cluster]
(infer pid=654443) 
Processed prompts:   1%| | 12/985 [00:59<25:59,  1.60s/it, est. speed input: 110.61 toks/s, output:  [repeated 13x across cluster]
(infer pid=654445) 
Processed prompts:   2%| | 17/985 [01:08<32:55,  2.04s/it, est. speed input: 147.02 toks/s, output:  [repeated 15x across cluster]
(infer pid=654457) 
Processed prompts:   2%| | 22/985 [01:17<24:12,  1.51s/it, est. speed input: 182.76 toks/s, output:  [repeated 17x across cluster]
(infer pid=654457) 
Processed prompts:   3%| | 26/985 [01:23<21:06,  1.32s/it, est. speed input: 189.01 toks/s, output:  [repeated 15x across cluster]
(infer pid=654457) 
Processed prompts:   3%| | 31/985 [01:28<16:19,  1.03s/it, est. speed input: 220.31 toks/s, output:  [repeated 13x across cluster]
(infer pid=654459) 
Processed prompts:   4%| | 42/985 [01:33<09:52,  1.59it/s, est. speed input: 281.37 toks/s, output:  [repeated 16x across cluster]
(infer pid=654445) 
Processed prompts:   4%| | 36/985 [01:34<27:13,  1.72s/it, est. speed input: 230.85 toks/s, output:  [repeated 17x across cluster]
(infer pid=654445) 
Processed prompts:   4%| | 43/985 [01:39<11:51,  1.32it/s, est. speed input: 246.66 toks/s, output:  [repeated 17x across cluster]
(infer pid=654445) 
Processed prompts:   5%| | 46/985 [01:45<26:37,  1.70s/it, est. speed input: 254.20 toks/s, output:  [repeated 16x across cluster]
(infer pid=654459) 
Processed prompts:   6%| | 62/985 [01:54<15:32,  1.01s/it, est. speed input: 301.63 toks/s, output:  [repeated 13x across cluster]
(infer pid=654445) 
Processed prompts:   6%| | 56/985 [01:55<09:18,  1.66it/s, est. speed input: 289.48 toks/s, output:  [repeated 14x across cluster]
(infer pid=654445) 
Processed prompts:   6%| | 63/985 [02:01<16:08,  1.05s/it, est. speed input: 318.91 toks/s, output:  [repeated 16x across cluster]
(infer pid=654445) 
Processed prompts:   7%| | 66/985 [02:06<19:07,  1.25s/it, est. speed input: 324.58 toks/s, output:  [repeated 16x across cluster]
(infer pid=654445) 
Processed prompts:   7%| | 71/985 [02:11<17:18,  1.14s/it, est. speed input: 331.42 toks/s, output:  [repeated 11x across cluster]
(infer pid=654457) 
Processed prompts:   6%| | 61/985 [02:20<40:25,  2.63s/it, est. speed input: 254.79 toks/s, output:  [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:   9%| | 84/985 [02:25<29:02,  1.93s/it, est. speed input: 331.71 toks/s, output:  [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:   7%| | 67/985 [02:32<34:50,  2.28s/it, est. speed input: 257.38 toks/s, output:  [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:   6%| | 56/985 [02:25<1:08:27,  4.42s/it, est. speed input: 216.24 toks/s, output
(infer pid=654457) 
Processed prompts:   7%| | 72/985 [02:37<18:26,  1.21s/it, est. speed input: 261.89 toks/s, output:  [repeated 16x across cluster]
(infer pid=654445) 
Processed prompts:   9%| | 87/985 [02:38<12:08,  1.23it/s, est. speed input: 321.16 toks/s, output:  [repeated 17x across cluster]
(infer pid=654459) 
Processed prompts:  10%| | 99/985 [02:47<23:36,  1.60s/it, est. speed input: 331.69 toks/s, output:  [repeated 13x across cluster]
(infer pid=654443) 
Processed prompts:   7%| | 68/985 [02:45<28:42,  1.88s/it, est. speed input: 238.27 toks/s, output:  [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  11%| | 105/985 [02:58<21:37,  1.47s/it, est. speed input: 332.52 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  11%| | 108/985 [03:05<32:25,  2.22s/it, est. speed input: 324.95 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:   9%| | 90/985 [03:12<46:31,  3.12s/it, est. speed input: 264.39 toks/s, output:  [repeated 16x across cluster]
(infer pid=654457) 
Processed prompts:   9%| | 92/985 [03:17<40:04,  2.69s/it, est. speed input: 266.47 toks/s, output:  [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  12%| | 116/985 [03:21<46:28,  3.21s/it, est. speed input: 317.75 toks/s, output: [repeated 5x across cluster]
(infer pid=654445) 
Processed prompts:  10%| | 102/985 [03:20<1:02:02,  4.22s/it, est. speed input: 297.23 toks/s, outpu
(infer pid=654457) 
Processed prompts:  10%| | 97/985 [03:29<33:18,  2.25s/it, est. speed input: 261.84 toks/s, output:  [repeated 11x across cluster]
(infer pid=654459) 
Processed prompts:  13%|▏| 124/985 [03:31<18:31,  1.29s/it, est. speed input: 324.93 toks/s, output:
(infer pid=654445) 
Processed prompts:  11%| | 104/985 [03:28<1:02:47,  4.28s/it, est. speed input: 294.08 toks/s, outpu
(infer pid=654459) 
Processed prompts:  13%|▏| 125/985 [03:32<17:46,  1.24s/it, est. speed input: 324.90 toks/s, output:
(infer pid=654459) 
Processed prompts:  13%|▏| 127/985 [03:33<10:25,  1.37it/s, est. speed input: 328.68 toks/s, output:
(infer pid=654457) 
Processed prompts:  10%| | 100/985 [03:34<28:59,  1.97s/it, est. speed input: 260.88 toks/s, output: [repeated 6x across cluster]
(infer pid=654459) 
Processed prompts:  13%|▏| 128/985 [03:35<17:28,  1.22s/it, est. speed input: 326.28 toks/s, output:
(infer pid=654459) 
Processed prompts:  13%|▏| 129/985 [03:36<15:22,  1.08s/it, est. speed input: 331.43 toks/s, output:
(infer pid=654459) 
Processed prompts:  13%|▏| 130/985 [03:36<11:52,  1.20it/s, est. speed input: 332.85 toks/s, output:
(infer pid=654459) 
Processed prompts:  13%|▏| 131/985 [03:37<11:25,  1.24it/s, est. speed input: 337.74 toks/s, output:
(infer pid=654459) 
Processed prompts:  13%|▏| 132/985 [03:39<14:21,  1.01s/it, est. speed input: 337.15 toks/s, output:
(infer pid=654457) 
Processed prompts:  10%| | 102/985 [03:39<29:04,  1.98s/it, est. speed input: 259.26 toks/s, output: [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  14%|▏| 133/985 [03:39<12:21,  1.15it/s, est. speed input: 338.55 toks/s, output:
(infer pid=654459) 
Processed prompts:  14%|▏| 134/985 [03:43<24:16,  1.71s/it, est. speed input: 335.10 toks/s, output:
(infer pid=654457) 
Processed prompts:  11%| | 104/985 [03:45<33:22,  2.27s/it, est. speed input: 259.43 toks/s, output: [repeated 5x across cluster]
(infer pid=654459) 
Processed prompts:  14%|▏| 135/985 [03:49<41:18,  2.92s/it, est. speed input: 328.29 toks/s, output:
(infer pid=654457) 
Processed prompts:  11%| | 106/985 [03:50<37:49,  2.58s/it, est. speed input: 257.14 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  14%|▏| 136/985 [03:50<33:40,  2.38s/it, est. speed input: 328.80 toks/s, output:
(infer pid=654459) 
Processed prompts:  14%|▏| 137/985 [03:50<26:09,  1.85s/it, est. speed input: 329.61 toks/s, output:
(infer pid=654459) 
Processed prompts:  14%|▏| 138/985 [03:51<21:44,  1.54s/it, est. speed input: 334.26 toks/s, output:
(infer pid=654459) 
Processed prompts:  14%|▏| 139/985 [03:52<19:02,  1.35s/it, est. speed input: 334.44 toks/s, output:
(infer pid=654445) 
Processed prompts:  12%| | 115/985 [03:53<32:10,  2.22s/it, est. speed input: 287.31 toks/s, output: [repeated 14x across cluster]
(infer pid=654445) 
Processed prompts:  12%| | 117/985 [03:58<34:49,  2.41s/it, est. speed input: 287.41 toks/s, output: [repeated 6x across cluster]
(infer pid=654457) 
Processed prompts:  12%| | 120/985 [04:09<20:08,  1.40s/it, est. speed input: 276.86 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  14%|▏| 140/985 [04:09<1:26:36,  6.15s/it, est. speed input: 316.67 toks/s, outpu
(infer pid=654459) 
Processed prompts:  14%|▏| 141/985 [04:10<1:01:58,  4.41s/it, est. speed input: 317.89 toks/s, outpu
(infer pid=654459) 
Processed prompts:  14%|▏| 142/985 [04:13<58:23,  4.16s/it, est. speed input: 315.61 toks/s, output:
(infer pid=654445) 
Processed prompts:  12%| | 121/985 [04:09<36:07,  2.51s/it, est. speed input: 281.38 toks/s, output: [repeated 4x across cluster]
(infer pid=654459) 
Processed prompts:  15%|▏| 143/985 [04:15<46:24,  3.31s/it, est. speed input: 315.77 toks/s, output:
(infer pid=654443) 
Processed prompts:  11%| | 105/985 [04:11<29:02,  1.98s/it, est. speed input: 242.03 toks/s, output: [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  15%|▏| 144/985 [04:19<52:35,  3.75s/it, est. speed input: 311.52 toks/s, output:
(infer pid=654459) 
Processed prompts:  15%|▏| 145/985 [04:20<37:57,  2.71s/it, est. speed input: 312.55 toks/s, output:
(infer pid=654445) 
Processed prompts:  12%| | 123/985 [04:19<54:45,  3.81s/it, est. speed input: 276.15 toks/s, output: [repeated 2x across cluster]
(infer pid=654459) 
Processed prompts:  15%|▏| 148/985 [04:25<31:59,  2.29s/it, est. speed input: 311.87 toks/s, output: [repeated 6x across cluster]
(infer pid=654445) 
Processed prompts:  13%|▏| 124/985 [04:25<1:06:49,  4.66s/it, est. speed input: 270.78 toks/s, outpu
(infer pid=654443) 
Processed prompts:  11%| | 109/985 [04:22<34:43,  2.38s/it, est. speed input: 246.41 toks/s, output: [repeated 3x across cluster]
(infer pid=654445) 
Processed prompts:  13%|▏| 126/985 [04:27<37:27,  2.62s/it, est. speed input: 275.58 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  11%| | 111/985 [04:26<29:51,  2.05s/it, est. speed input: 249.65 toks/s, output: [repeated 2x across cluster]
(infer pid=654457) 
Processed prompts:  13%|▏| 130/985 [04:39<44:56,  3.15s/it, est. speed input: 270.66 toks/s, output: [repeated 10x across cluster]
(infer pid=654443) 
Processed prompts:  11%| | 112/985 [04:35<1:00:14,  4.14s/it, est. speed input: 243.09 toks/s, outpu
(infer pid=654445) 
Processed prompts:  14%|▏| 134/985 [04:41<29:06,  2.05s/it, est. speed input: 275.53 toks/s, output: [repeated 4x across cluster]
(infer pid=654443) 
Processed prompts:  11%| | 113/985 [04:38<57:40,  3.97s/it, est. speed input: 241.43 toks/s, output:
(infer pid=654443) 
Processed prompts:  12%| | 114/985 [04:40<47:07,  3.25s/it, est. speed input: 241.46 toks/s, output:
(infer pid=654443) 
Processed prompts:  12%| | 115/985 [04:42<44:06,  3.04s/it, est. speed input: 241.85 toks/s, output:
(infer pid=654457) 
Processed prompts:  14%|▏| 134/985 [04:50<33:16,  2.35s/it, est. speed input: 265.07 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  12%| | 116/985 [04:43<32:51,  2.27s/it, est. speed input: 243.91 toks/s, output:
(infer pid=654459) 
Processed prompts:  16%|▏| 159/985 [04:56<1:04:25,  4.68s/it, est. speed input: 303.64 toks/s, outpu
(infer pid=654457) 
Processed prompts:  14%|▏| 136/985 [04:55<33:58,  2.40s/it, est. speed input: 263.71 toks/s, output: [repeated 3x across cluster]
(infer pid=654443) 
Processed prompts:  12%| | 117/985 [04:51<59:14,  4.10s/it, est. speed input: 238.20 toks/s, output:
(infer pid=654445) 
Processed prompts:  14%|▏| 141/985 [04:58<36:55,  2.62s/it, est. speed input: 272.29 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  12%| | 118/985 [04:55<58:38,  4.06s/it, est. speed input: 236.24 toks/s, output:
(infer pid=654443) 
Processed prompts:  12%| | 119/985 [04:58<55:08,  3.82s/it, est. speed input: 235.04 toks/s, output:
(infer pid=654459) 
Processed prompts:  17%|▏| 165/985 [05:07<34:18,  2.51s/it, est. speed input: 300.04 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  12%| | 120/985 [05:04<1:01:54,  4.29s/it, est. speed input: 235.25 toks/s, outpu
(infer pid=654459) 
Processed prompts:  17%|▏| 169/985 [05:12<25:51,  1.90s/it, est. speed input: 304.05 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  12%| | 121/985 [05:09<1:05:52,  4.58s/it, est. speed input: 232.39 toks/s, outpu
(infer pid=654443) 
Processed prompts:  12%| | 122/985 [05:09<47:24,  3.30s/it, est. speed input: 233.36 toks/s, output:
(infer pid=654457) 
Processed prompts:  15%|▏| 145/985 [05:19<36:34,  2.61s/it, est. speed input: 260.49 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  12%| | 123/985 [05:12<43:33,  3.03s/it, est. speed input: 232.83 toks/s, output:
(infer pid=654459) 
Processed prompts:  18%|▏| 176/985 [05:23<24:13,  1.80s/it, est. speed input: 305.55 toks/s, output: [repeated 5x across cluster]
(infer pid=654445) 
Processed prompts:  15%|▏| 151/985 [05:25<48:14,  3.47s/it, est. speed input: 267.05 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  15%|▏| 150/985 [05:35<42:55,  3.08s/it, est. speed input: 253.68 toks/s, output: [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  18%|▏| 181/985 [05:40<57:34,  4.30s/it, est. speed input: 301.48 toks/s, output: [repeated 7x across cluster]
(infer pid=654445) 
Processed prompts:  16%|▏| 158/985 [05:44<58:19,  4.23s/it, est. speed input: 262.10 toks/s, output: [repeated 5x across cluster]
(infer pid=654459) 
Processed prompts:  19%|▏| 184/985 [05:54<1:08:58,  5.17s/it, est. speed input: 296.47 toks/s, outpu
(infer pid=654445) 
Processed prompts:  16%|▏| 162/985 [05:49<24:27,  1.78s/it, est. speed input: 268.72 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  13%|▏| 132/985 [05:51<1:22:42,  5.82s/it, est. speed input: 220.73 toks/s, outpu [repeated 2x across cluster]
(infer pid=654445) 
Processed prompts:  17%|▏| 166/985 [05:56<26:37,  1.95s/it, est. speed input: 271.59 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  14%|▏| 133/985 [05:52<1:03:04,  4.44s/it, est. speed input: 221.05 toks/s, outpu
(infer pid=654445) 
Processed prompts:  17%|▏| 169/985 [06:01<22:54,  1.68s/it, est. speed input: 274.31 toks/s, output: [repeated 10x across cluster]
(infer pid=654457) 
Processed prompts:  16%|▏| 160/985 [06:10<1:04:58,  4.73s/it, est. speed input: 244.16 toks/s, outpu
(infer pid=654445) 
Processed prompts:  17%|▏| 172/985 [06:03<15:51,  1.17s/it, est. speed input: 275.57 toks/s, output: [repeated 4x across cluster]
(infer pid=654443) 
Processed prompts:  14%|▏| 136/985 [06:05<1:02:04,  4.39s/it, est. speed input: 218.70 toks/s, outpu [repeated 2x across cluster]
(infer pid=654459) 
Processed prompts:  20%|▏| 194/985 [06:16<38:28,  2.92s/it, est. speed input: 293.18 toks/s, output: [repeated 10x across cluster]
(infer pid=654445) 
Processed prompts:  18%|▏| 177/985 [06:18<28:06,  2.09s/it, est. speed input: 271.87 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  15%|▏| 144/985 [06:19<26:20,  1.88s/it, est. speed input: 220.01 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  20%|▏| 196/985 [06:29<1:02:52,  4.78s/it, est. speed input: 287.89 toks/s, outpu
(infer pid=654443) 
Processed prompts:  15%|▏| 145/985 [06:25<44:11,  3.16s/it, est. speed input: 218.07 toks/s, output: [repeated 3x across cluster]
(infer pid=654459) 
Processed prompts:  20%|▏| 197/985 [06:35<1:07:35,  5.15s/it, est. speed input: 284.41 toks/s, outpu
(infer pid=654459) 
Processed prompts:  20%|▏| 201/985 [06:38<26:20,  2.02s/it, est. speed input: 286.67 toks/s, output: [repeated 7x across cluster]
(infer pid=654457) 
Processed prompts:  17%|▏| 170/985 [06:42<1:07:11,  4.95s/it, est. speed input: 235.06 toks/s, outpu
(infer pid=654459) 
Processed prompts:  21%|▏| 205/985 [06:43<21:00,  1.62s/it, est. speed input: 286.81 toks/s, output: [repeated 11x across cluster]
(infer pid=654459) 
Processed prompts:  21%|▏| 207/985 [06:49<29:00,  2.24s/it, est. speed input: 284.56 toks/s, output: [repeated 6x across cluster]
(infer pid=654457) 
Processed prompts:  18%|▏| 177/985 [06:56<32:48,  2.44s/it, est. speed input: 238.66 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  22%|▏| 214/985 [07:00<22:53,  1.78s/it, est. speed input: 285.88 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  18%|▏| 178/985 [07:06<1:03:53,  4.75s/it, est. speed input: 233.77 toks/s, outpu
(infer pid=654459) 
Processed prompts:  22%|▏| 216/985 [07:05<25:33,  1.99s/it, est. speed input: 284.52 toks/s, output: [repeated 5x across cluster]
(infer pid=654443) 
Processed prompts:  16%|▏| 154/985 [07:02<1:08:14,  4.93s/it, est. speed input: 209.32 toks/s, outpu [repeated 2x across cluster]
(infer pid=654459) 
Processed prompts:  22%|▏| 218/985 [07:11<30:10,  2.36s/it, est. speed input: 282.65 toks/s, output: [repeated 4x across cluster]
(infer pid=654457) 
Processed prompts:  18%|▏| 179/985 [07:14<1:18:14,  5.83s/it, est. speed input: 230.07 toks/s, outpu
(infer pid=654443) 
Processed prompts:  16%|▏| 158/985 [07:13<50:00,  3.63s/it, est. speed input: 207.44 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  18%|▏| 182/985 [07:25<1:06:38,  4.98s/it, est. speed input: 229.38 toks/s, outpu
(infer pid=654443) 
Processed prompts:  16%|▏| 161/985 [07:18<33:00,  2.40s/it, est. speed input: 207.74 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  17%|▏| 163/985 [07:24<39:00,  2.85s/it, est. speed input: 207.09 toks/s, output: [repeated 6x across cluster]
(infer pid=654445) 
Processed prompts:  20%|▏| 201/985 [07:28<1:15:37,  5.79s/it, est. speed input: 254.85 toks/s, outpu
(infer pid=654445) 
Processed prompts:  21%|▏| 206/985 [07:33<25:58,  2.00s/it, est. speed input: 256.23 toks/s, output: [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  23%|▏| 231/985 [07:41<30:13,  2.41s/it, est. speed input: 279.50 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  17%|▏| 165/985 [07:38<1:16:20,  5.59s/it, est. speed input: 202.06 toks/s, outpu
(infer pid=654445) 
Processed prompts:  21%|▏| 209/985 [07:43<44:03,  3.41s/it, est. speed input: 252.98 toks/s, output: [repeated 3x across cluster]
(infer pid=654443) 
Processed prompts:  17%|▏| 166/985 [07:40<1:00:09,  4.41s/it, est. speed input: 202.09 toks/s, outpu
(infer pid=654459) 
Processed prompts:  24%|▏| 236/985 [07:52<25:03,  2.01s/it, est. speed input: 281.43 toks/s, output: [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  24%|▏| 240/985 [07:57<19:27,  1.57s/it, est. speed input: 282.08 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  18%|▏| 173/985 [07:57<37:19,  2.76s/it, est. speed input: 203.16 toks/s, output: [repeated 11x across cluster]
(infer pid=654457) 
Processed prompts:  20%|▏| 195/985 [08:07<1:01:09,  4.65s/it, est. speed input: 227.12 toks/s, outpu
(infer pid=654445) 
Processed prompts:  22%|▏| 218/985 [08:06<33:41,  2.64s/it, est. speed input: 253.79 toks/s, output: [repeated 5x across cluster]
(infer pid=654459) 
Processed prompts:  25%|▎| 248/985 [08:15<19:14,  1.57s/it, est. speed input: 278.36 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  18%|▏| 175/985 [08:08<1:01:11,  4.53s/it, est. speed input: 200.14 toks/s, outpu
(infer pid=654443) 
Processed prompts:  18%|▏| 177/985 [08:14<49:38,  3.69s/it, est. speed input: 200.27 toks/s, output: [repeated 3x across cluster]
(infer pid=654445) 
Processed prompts:  23%|▏| 223/985 [08:23<47:53,  3.77s/it, est. speed input: 254.06 toks/s, output: [repeated 4x across cluster]
(infer pid=654457) 
Processed prompts:  20%|▏| 199/985 [08:31<1:39:16,  7.58s/it, est. speed input: 219.80 toks/s, outpu
(infer pid=654459) 
Processed prompts:  25%|▎| 251/985 [08:30<46:49,  3.83s/it, est. speed input: 272.47 toks/s, output: [repeated 2x across cluster]
(infer pid=654457) 
Processed prompts:  20%|▏| 201/985 [08:34<1:01:48,  4.73s/it, est. speed input: 220.15 toks/s, outpu [repeated 2x across cluster]
(infer pid=654445) 
Processed prompts:  23%|▏| 227/985 [08:34<40:25,  3.20s/it, est. speed input: 253.43 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  18%|▏| 181/985 [08:33<1:02:20,  4.65s/it, est. speed input: 197.18 toks/s, outpu
(infer pid=654459) 
Processed prompts:  26%|▎| 256/985 [08:43<30:29,  2.51s/it, est. speed input: 269.23 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  18%|▏| 182/985 [08:39<1:06:13,  4.95s/it, est. speed input: 195.70 toks/s, outpu
(infer pid=654443) 
Processed prompts:  19%|▏| 184/985 [08:41<41:45,  3.13s/it, est. speed input: 196.31 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  21%|▏| 208/985 [08:55<46:36,  3.60s/it, est. speed input: 222.03 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  19%|▏| 187/985 [08:52<40:11,  3.02s/it, est. speed input: 194.37 toks/s, output: [repeated 10x across cluster]
(infer pid=654443) 
Processed prompts:  19%|▏| 192/985 [08:57<16:03,  1.21s/it, est. speed input: 197.54 toks/s, output: [repeated 14x across cluster]
(infer pid=654443) 
Processed prompts:  20%|▏| 193/985 [09:03<33:36,  2.55s/it, est. speed input: 196.13 toks/s, output: [repeated 6x across cluster]
(infer pid=654457) 
Processed prompts:  22%|▏| 214/985 [09:17<45:49,  3.57s/it, est. speed input: 220.87 toks/s, output: [repeated 5x across cluster]
(infer pid=654445) 
Processed prompts:  24%|▏| 241/985 [09:15<1:08:44,  5.54s/it, est. speed input: 245.45 toks/s, outpu
(infer pid=654459) 
Processed prompts:  28%|▎| 277/985 [09:21<21:17,  1.81s/it, est. speed input: 270.55 toks/s, output: [repeated 11x across cluster]
(infer pid=654445) 
Processed prompts:  25%|▏| 244/985 [09:23<46:09,  3.74s/it, est. speed input: 245.82 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  21%|▏| 202/985 [09:24<37:36,  2.88s/it, est. speed input: 198.36 toks/s, output: [repeated 7x across cluster]
(infer pid=654457) 
Processed prompts:  23%|▏| 226/985 [09:39<31:25,  2.48s/it, est. speed input: 221.59 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  21%|▏| 206/985 [09:37<34:23,  2.65s/it, est. speed input: 198.32 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  21%|▏| 209/985 [09:42<27:16,  2.11s/it, est. speed input: 198.33 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  22%|▏| 215/985 [09:48<17:56,  1.40s/it, est. speed input: 203.46 toks/s, output: [repeated 13x across cluster]
(infer pid=654459) 
Processed prompts:  29%|▎| 288/985 [10:00<51:05,  4.40s/it, est. speed input: 262.75 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  27%|▎| 262/985 [10:02<19:22,  1.61s/it, est. speed input: 247.73 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  29%|▎| 289/985 [10:08<1:03:48,  5.50s/it, est. speed input: 259.88 toks/s, outpu
(infer pid=654443) 
Processed prompts:  22%|▏| 219/985 [10:05<46:58,  3.68s/it, est. speed input: 200.71 toks/s, output: [repeated 4x across cluster]
(infer pid=654445) 
Processed prompts:  27%|▎| 266/985 [10:13<26:51,  2.24s/it, est. speed input: 245.71 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  23%|▏| 222/985 [10:15<48:35,  3.82s/it, est. speed input: 200.64 toks/s, output: [repeated 4x across cluster]
(infer pid=654459) 
Processed prompts:  30%|▎| 293/985 [10:27<1:12:12,  6.26s/it, est. speed input: 254.50 toks/s, outpu
(infer pid=654443) 
Processed prompts:  23%|▏| 223/985 [10:21<56:00,  4.41s/it, est. speed input: 199.33 toks/s, output: [repeated 3x across cluster]
(infer pid=654459) 
Processed prompts:  30%|▎| 294/985 [10:31<1:01:27,  5.34s/it, est. speed input: 253.79 toks/s, outpu
(infer pid=654459) 
Processed prompts:  30%|▎| 296/985 [10:33<37:34,  3.27s/it, est. speed input: 253.88 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  28%|▎| 275/985 [10:37<24:41,  2.09s/it, est. speed input: 244.04 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  28%|▎| 277/985 [10:42<28:41,  2.43s/it, est. speed input: 243.27 toks/s, output: [repeated 4x across cluster]
(infer pid=654445) 
Processed prompts:  28%|▎| 279/985 [10:47<28:48,  2.45s/it, est. speed input: 243.02 toks/s, output: [repeated 5x across cluster]
(infer pid=654457) 
Processed prompts:  25%|▎| 247/985 [10:52<1:26:03,  7.00s/it, est. speed input: 211.22 toks/s, outpu
(infer pid=654457) 
Processed prompts:  25%|▎| 248/985 [10:54<1:04:14,  5.23s/it, est. speed input: 212.89 toks/s, outpu
(infer pid=654445) 
Processed prompts:  29%|▎| 282/985 [10:52<24:32,  2.09s/it, est. speed input: 244.49 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  23%|▏| 229/985 [10:48<1:03:34,  5.05s/it, est. speed input: 200.50 toks/s, outpu
(infer pid=654443) 
Processed prompts:  23%|▏| 230/985 [10:53<1:01:47,  4.91s/it, est. speed input: 199.69 toks/s, outpu
(infer pid=654457) 
Processed prompts:  26%|▎| 252/985 [11:01<33:36,  2.75s/it, est. speed input: 214.49 toks/s, output: [repeated 5x across cluster]
(infer pid=654445) 
Processed prompts:  29%|▎| 285/985 [11:03<38:10,  3.27s/it, est. speed input: 242.62 toks/s, output: [repeated 5x across cluster]
(infer pid=654457) 
Processed prompts:  26%|▎| 253/985 [11:14<1:08:33,  5.62s/it, est. speed input: 211.25 toks/s, outpu
(infer pid=654459) 
Processed prompts:  31%|▎| 310/985 [11:10<35:47,  3.18s/it, est. speed input: 253.61 toks/s, output: [repeated 4x across cluster]
(infer pid=654445) 
Processed prompts:  29%|▎| 286/985 [11:13<1:00:52,  5.23s/it, est. speed input: 239.89 toks/s, outpu
(infer pid=654445) 
Processed prompts:  29%|▎| 287/985 [11:15<48:36,  4.18s/it, est. speed input: 239.92 toks/s, output: [repeated 4x across cluster]
(infer pid=654457) 
Processed prompts:  26%|▎| 257/985 [11:24<46:54,  3.87s/it, est. speed input: 212.03 toks/s, output: [repeated 3x across cluster]
(infer pid=654443) 
Processed prompts:  24%|▏| 236/985 [11:19<1:03:59,  5.13s/it, est. speed input: 197.29 toks/s, outpu
(infer pid=654445) 
Processed prompts:  30%|▎| 293/985 [11:26<21:02,  1.82s/it, est. speed input: 243.37 toks/s, output: [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  32%|▎| 317/985 [11:35<39:15,  3.53s/it, est. speed input: 249.57 toks/s, output: [repeated 11x across cluster]
(infer pid=654445) 
Processed prompts:  30%|▎| 299/985 [11:37<22:55,  2.00s/it, est. speed input: 248.42 toks/s, output: [repeated 11x across cluster]
(infer pid=654443) 
Processed prompts:  25%|▏| 245/985 [11:40<42:06,  3.41s/it, est. speed input: 198.33 toks/s, output: [repeated 6x across cluster]
(infer pid=654459) 
Processed prompts:  33%|▎| 324/985 [11:52<31:14,  2.84s/it, est. speed input: 251.67 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  25%|▎| 251/985 [11:50<26:38,  2.18s/it, est. speed input: 198.71 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  26%|▎| 254/985 [11:56<23:41,  1.95s/it, est. speed input: 198.72 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  26%|▎| 257/985 [12:02<23:46,  1.96s/it, est. speed input: 198.63 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  28%|▎| 271/985 [12:11<1:02:43,  5.27s/it, est. speed input: 207.65 toks/s, outpu
(infer pid=654445) 
Processed prompts:  31%|▎| 309/985 [12:11<36:03,  3.20s/it, est. speed input: 247.02 toks/s, output: [repeated 6x across cluster]
(infer pid=654457) 
Processed prompts:  28%|▎| 272/985 [12:18<1:07:05,  5.65s/it, est. speed input: 206.36 toks/s, outpu
(infer pid=654445) 
Processed prompts:  32%|▎| 313/985 [12:16<18:37,  1.66s/it, est. speed input: 247.71 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  34%|▎| 335/985 [12:25<25:05,  2.32s/it, est. speed input: 249.28 toks/s, output: [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  34%|▎| 337/985 [12:31<28:22,  2.63s/it, est. speed input: 248.51 toks/s, output: [repeated 7x across cluster]
(infer pid=654445) 
Processed prompts:  32%|▎| 320/985 [12:33<22:05,  1.99s/it, est. speed input: 249.37 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  27%|▎| 269/985 [12:34<45:54,  3.85s/it, est. speed input: 197.31 toks/s, output: [repeated 4x across cluster]
(infer pid=654457) 
Processed prompts:  29%|▎| 283/985 [12:48<40:08,  3.43s/it, est. speed input: 205.84 toks/s, output: [repeated 7x across cluster]
(infer pid=654445) 
Processed prompts:  33%|▎| 322/985 [12:49<50:27,  4.57s/it, est. speed input: 245.14 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  28%|▎| 272/985 [12:50<1:03:11,  5.32s/it, est. speed input: 196.17 toks/s, outpu
(infer pid=654459) 
Processed prompts:  36%|▎| 350/985 [12:58<17:32,  1.66s/it, est. speed input: 247.47 toks/s, output: [repeated 12x across cluster]
(infer pid=654443) 
Processed prompts:  28%|▎| 273/985 [12:55<1:02:50,  5.30s/it, est. speed input: 195.33 toks/s, outpu
(infer pid=654443) 
Processed prompts:  28%|▎| 275/985 [12:56<37:33,  3.17s/it, est. speed input: 196.01 toks/s, output: [repeated 3x across cluster]
(infer pid=654457) 
Processed prompts:  30%|▎| 296/985 [13:10<27:36,  2.40s/it, est. speed input: 207.88 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  31%|▎| 301/985 [13:16<17:47,  1.56s/it, est. speed input: 209.62 toks/s, output: [repeated 10x across cluster]
(infer pid=654443) 
Processed prompts:  29%|▎| 281/985 [13:13<22:28,  1.92s/it, est. speed input: 196.59 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  34%|▎| 333/985 [13:22<48:40,  4.48s/it, est. speed input: 240.81 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  37%|▎| 363/985 [13:32<24:04,  2.32s/it, est. speed input: 247.06 toks/s, output: [repeated 10x across cluster]
(infer pid=654443) 
Processed prompts:  29%|▎| 289/985 [13:32<21:48,  1.88s/it, est. speed input: 200.48 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  30%|▎| 292/985 [13:37<20:22,  1.76s/it, est. speed input: 201.77 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  31%|▎| 310/985 [13:50<1:10:56,  6.31s/it, est. speed input: 206.70 toks/s, outpu
(infer pid=654443) 
Processed prompts:  30%|▎| 295/985 [13:41<16:02,  1.39s/it, est. speed input: 202.19 toks/s, output: [repeated 3x across cluster]
(infer pid=654457) 
Processed prompts:  32%|▎| 313/985 [13:55<35:04,  3.13s/it, est. speed input: 207.57 toks/s, output: [repeated 4x across cluster]
(infer pid=654445) 
Processed prompts:  35%|▎| 340/985 [13:52<1:04:00,  5.95s/it, est. speed input: 239.48 toks/s, outpu
(infer pid=654443) 
Processed prompts:  30%|▎| 297/985 [13:54<43:10,  3.76s/it, est. speed input: 201.15 toks/s, output: [repeated 7x across cluster]
(infer pid=654445) 
Processed prompts:  35%|▎| 342/985 [14:02<1:04:57,  6.06s/it, est. speed input: 237.40 toks/s, outpu
(infer pid=654459) 
Processed prompts:  38%|▍| 372/985 [14:06<24:56,  2.44s/it, est. speed input: 242.33 toks/s, output: [repeated 11x across cluster]
(infer pid=654457) 
Processed prompts:  33%|▎| 323/985 [14:12<21:32,  1.95s/it, est. speed input: 209.33 toks/s, output: [repeated 11x across cluster]
(infer pid=654459) 
Processed prompts:  38%|▍| 377/985 [14:16<17:42,  1.75s/it, est. speed input: 242.29 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  39%|▍| 382/985 [14:22<11:10,  1.11s/it, est. speed input: 243.31 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  36%|▎| 350/985 [14:24<41:04,  3.88s/it, est. speed input: 235.44 toks/s, output: [repeated 11x across cluster]
(infer pid=654457) 
Processed prompts:  34%|▎| 334/985 [14:34<23:50,  2.20s/it, est. speed input: 211.46 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  35%|▎| 340/985 [14:39<09:24,  1.14it/s, est. speed input: 214.98 toks/s, output: [repeated 12x across cluster]
(infer pid=654457) 
Processed prompts:  35%|▎| 345/985 [14:44<09:43,  1.10it/s, est. speed input: 215.97 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  36%|▎| 359/985 [14:46<24:00,  2.30s/it, est. speed input: 235.01 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  37%|▎| 365/985 [14:51<09:17,  1.11it/s, est. speed input: 237.47 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  40%|▍| 395/985 [15:00<26:19,  2.68s/it, est. speed input: 241.24 toks/s, output: [repeated 10x across cluster]
(infer pid=654445) 
Processed prompts:  38%|▍| 372/985 [15:03<16:37,  1.63s/it, est. speed input: 238.22 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  33%|▎| 325/985 [15:05<53:00,  4.82s/it, est. speed input: 202.67 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  41%|▍| 400/985 [15:20<32:56,  3.38s/it, est. speed input: 238.51 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  33%|▎| 329/985 [15:18<41:37,  3.81s/it, est. speed input: 202.68 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  34%|▎| 334/985 [15:24<18:27,  1.70s/it, est. speed input: 205.66 toks/s, output: [repeated 5x across cluster]
(infer pid=654445) 
Processed prompts:  38%|▍| 376/985 [15:28<1:04:13,  6.33s/it, est. speed input: 234.28 toks/s, outpu
(infer pid=654443) 
Processed prompts:  34%|▎| 338/985 [15:29<17:20,  1.61s/it, est. speed input: 208.31 toks/s, output: [repeated 10x across cluster]
(infer pid=654445) 
Processed prompts:  38%|▍| 378/985 [15:38<1:00:45,  6.01s/it, est. speed input: 232.85 toks/s, outpu
(infer pid=654443) 
Processed prompts:  35%|▎| 340/985 [15:34<21:49,  2.03s/it, est. speed input: 208.06 toks/s, output: [repeated 3x across cluster]
(infer pid=654445) 
Processed prompts:  39%|▍| 381/985 [15:45<38:26,  3.82s/it, est. speed input: 232.25 toks/s, output: [repeated 10x across cluster]
(infer pid=654445) 
Processed prompts:  39%|▍| 383/985 [15:51<34:36,  3.45s/it, est. speed input: 232.87 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  38%|▍| 376/985 [16:01<22:00,  2.17s/it, est. speed input: 219.82 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  40%|▍| 391/985 [16:02<15:07,  1.53s/it, est. speed input: 234.53 toks/s, output: [repeated 13x across cluster]
(infer pid=654459) 
Processed prompts:  42%|▍| 415/985 [16:11<29:27,  3.10s/it, est. speed input: 232.71 toks/s, output: [repeated 10x across cluster]
(infer pid=654443) 
Processed prompts:  36%|▎| 352/985 [16:09<40:38,  3.85s/it, est. speed input: 207.81 toks/s, output: [repeated 11x across cluster]
(infer pid=654443) 
Processed prompts:  36%|▎| 354/985 [16:14<32:21,  3.08s/it, est. speed input: 208.76 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  39%|▍| 387/985 [16:28<22:38,  2.27s/it, est. speed input: 219.47 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  39%|▍| 389/985 [16:33<24:58,  2.51s/it, est. speed input: 219.14 toks/s, output: [repeated 11x across cluster]
(infer pid=654443) 
Processed prompts:  37%|▎| 363/985 [16:31<26:22,  2.54s/it, est. speed input: 209.92 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  40%|▍| 394/985 [16:44<25:25,  2.58s/it, est. speed input: 220.95 toks/s, output: [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  44%|▍| 434/985 [16:49<21:09,  2.30s/it, est. speed input: 233.49 toks/s, output: [repeated 5x across cluster]
(infer pid=654459) 
Processed prompts:  44%|▍| 436/985 [16:54<23:30,  2.57s/it, est. speed input: 234.69 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  44%|▍| 437/985 [17:00<32:36,  3.57s/it, est. speed input: 233.75 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  41%|▍| 405/985 [17:06<17:13,  1.78s/it, est. speed input: 220.57 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  42%|▍| 415/985 [17:07<31:32,  3.32s/it, est. speed input: 233.59 toks/s, output: [repeated 11x across cluster]
(infer pid=654445) 
Processed prompts:  43%|▍| 419/985 [17:14<21:45,  2.31s/it, est. speed input: 234.83 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  42%|▍| 414/985 [17:24<18:10,  1.91s/it, est. speed input: 220.64 toks/s, output: [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  46%|▍| 450/985 [17:30<20:37,  2.31s/it, est. speed input: 233.79 toks/s, output: [repeated 12x across cluster]
(infer pid=654457) 
Processed prompts:  43%|▍| 420/985 [17:35<18:31,  1.97s/it, est. speed input: 220.85 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  39%|▍| 385/985 [17:33<22:03,  2.21s/it, est. speed input: 210.95 toks/s, output: [repeated 7x across cluster]
(infer pid=654445) 
Processed prompts:  43%|▍| 428/985 [17:42<25:10,  2.71s/it, est. speed input: 233.64 toks/s, output: [repeated 7x across cluster]
(infer pid=654457) 
Processed prompts:  43%|▍| 425/985 [17:51<33:48,  3.62s/it, est. speed input: 219.93 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  43%|▍| 426/985 [17:56<38:42,  4.15s/it, est. speed input: 219.22 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  47%|▍| 465/985 [18:01<15:58,  1.84s/it, est. speed input: 235.53 toks/s, output: [repeated 6x across cluster]
(infer pid=654459) 
Processed prompts:  47%|▍| 467/985 [18:06<19:12,  2.23s/it, est. speed input: 235.04 toks/s, output: [repeated 6x across cluster]
(infer pid=654459) 
Processed prompts:  48%|▍| 472/985 [18:12<12:34,  1.47s/it, est. speed input: 235.70 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  44%|▍| 436/985 [18:16<37:25,  4.09s/it, est. speed input: 230.40 toks/s, output: [repeated 6x across cluster]
(infer pid=654459) 
Processed prompts:  48%|▍| 475/985 [18:24<25:45,  3.03s/it, est. speed input: 234.90 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  41%|▍| 408/985 [18:23<25:00,  2.60s/it, est. speed input: 215.80 toks/s, output: [repeated 12x across cluster]
(infer pid=654443) 
Processed prompts:  42%|▍| 410/985 [18:29<26:19,  2.75s/it, est. speed input: 215.35 toks/s, output: [repeated 6x across cluster]
(infer pid=654457) 
Processed prompts:  44%|▍| 435/985 [18:42<44:28,  4.85s/it, est. speed input: 214.83 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  46%|▍| 452/985 [18:43<12:55,  1.45s/it, est. speed input: 233.15 toks/s, output: [repeated 11x across cluster]
(infer pid=654443) 
Processed prompts:  43%|▍| 419/985 [18:46<13:12,  1.40s/it, est. speed input: 215.42 toks/s, output: [repeated 12x across cluster]
(infer pid=654443) 
Processed prompts:  43%|▍| 422/985 [18:51<16:02,  1.71s/it, est. speed input: 215.43 toks/s, output: [repeated 6x across cluster]
(infer pid=654443) 
Processed prompts:  43%|▍| 427/985 [18:57<11:03,  1.19s/it, est. speed input: 216.03 toks/s, output: [repeated 17x across cluster]
(infer pid=654445) 
Processed prompts:  47%|▍| 460/985 [19:05<21:59,  2.51s/it, est. speed input: 232.22 toks/s, output: [repeated 11x across cluster]
(infer pid=654459) 
Processed prompts:  50%|▌| 497/985 [19:14<11:26,  1.41s/it, est. speed input: 237.11 toks/s, output: [repeated 4x across cluster]
(infer pid=654457) 
Processed prompts:  46%|▍| 456/985 [19:20<21:30,  2.44s/it, est. speed input: 216.43 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  48%|▍| 468/985 [19:23<14:14,  1.65s/it, est. speed input: 231.66 toks/s, output: [repeated 10x across cluster]
(infer pid=654445) 
Processed prompts:  48%|▍| 474/985 [19:29<12:52,  1.51s/it, est. speed input: 232.38 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  52%|▌| 511/985 [19:39<09:55,  1.26s/it, est. speed input: 240.18 toks/s, output: [repeated 10x across cluster]
(infer pid=654457) 
Processed prompts:  46%|▍| 458/985 [19:44<1:10:25,  8.02s/it, est. speed input: 213.44 toks/s, outpu
(infer pid=654459) 
Processed prompts:  52%|▌| 515/985 [19:44<11:44,  1.50s/it, est. speed input: 241.55 toks/s, output: [repeated 8x across cluster]
(infer pid=654443) 
Processed prompts:  45%|▍| 439/985 [19:42<34:03,  3.74s/it, est. speed input: 213.09 toks/s, output: [repeated 6x across cluster]
(infer pid=654459) 
Processed prompts:  53%|▌| 519/985 [19:56<23:01,  2.97s/it, est. speed input: 240.44 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  49%|▍| 483/985 [19:58<19:40,  2.35s/it, est. speed input: 230.01 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  47%|▍| 467/985 [20:07<22:01,  2.55s/it, est. speed input: 212.22 toks/s, output: [repeated 11x across cluster]
(infer pid=654445) 
Processed prompts:  50%|▍| 488/985 [20:08<18:19,  2.21s/it, est. speed input: 230.68 toks/s, output: [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  54%|▌| 529/985 [20:17<23:48,  3.13s/it, est. speed input: 240.53 toks/s, output: [repeated 6x across cluster]
(infer pid=654457) 
Processed prompts:  48%|▍| 472/985 [20:23<27:16,  3.19s/it, est. speed input: 212.60 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  46%|▍| 456/985 [20:21<24:50,  2.82s/it, est. speed input: 214.84 toks/s, output: [repeated 8x across cluster]
(infer pid=654445) 
Processed prompts:  50%|▍| 489/985 [20:29<1:03:37,  7.70s/it, est. speed input: 227.62 toks/s, outpu
(infer pid=654459) 
Processed prompts:  55%|▌| 538/985 [20:33<12:26,  1.67s/it, est. speed input: 240.54 toks/s, output: [repeated 7x across cluster]
(infer pid=654459) 
Processed prompts:  55%|▌| 543/985 [20:39<10:22,  1.41s/it, est. speed input: 241.23 toks/s, output: [repeated 10x across cluster]
(infer pid=654457) 
Processed prompts:  49%|▍| 482/985 [20:45<22:27,  2.68s/it, est. speed input: 213.48 toks/s, output: [repeated 9x across cluster]
(infer pid=654457) 
Processed prompts:  49%|▍| 485/985 [20:51<20:46,  2.49s/it, est. speed input: 213.52 toks/s, output: [repeated 10x across cluster]
(infer pid=654459) 
Processed prompts:  56%|▌| 551/985 [20:56<13:24,  1.85s/it, est. speed input: 241.45 toks/s, output: [repeated 12x across cluster]
(infer pid=654459) 
Processed prompts:  56%|▌| 554/985 [21:01<11:28,  1.60s/it, est. speed input: 241.43 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  57%|▌| 557/985 [21:07<15:00,  2.10s/it, est. speed input: 242.82 toks/s, output: [repeated 9x across cluster]
(infer pid=654459) 
Processed prompts:  57%|▌| 559/985 [21:12<15:47,  2.23s/it, est. speed input: 243.22 toks/s, output: [repeated 5x across cluster]
(infer pid=654443) 
Processed prompts:  48%|▍| 471/985 [21:11<25:22,  2.96s/it, est. speed input: 214.05 toks/s, output: [repeated 8x across cluster]
(infer pid=654459) 
Processed prompts:  57%|▌| 565/985 [21:24<13:49,  1.97s/it, est. speed input: 243.36 toks/s, output: [repeated 9x across cluster]
(infer pid=654445) 
Processed prompts:  52%|▌| 509/985 [21:26<38:32,  4.86s/it, est. speed input: 226.48 toks/s, output: [repeated 8x across cluster]
(infer pid=654457) 
Processed prompts:  51%|▌| 501/985 [21:36<32:30,  4.03s/it, est. speed input: 212.15 toks/s, output: [repeated 4x across cluster]
(infer pid=654457) 
Processed prompts:  51%|▌| 504/985 [21:41<18:38,  2.33s/it, est. speed input: 213.02 toks/s, output: [repeated 9x across cluster]
(infer pid=654443) 
Processed prompts:  50%|▍| 488/985 [21:39<11:06,  1.34s/it, est. speed input: 215.94 toks/s, output: [repeated 7x across cluster]
(infer pid=654443) 
Processed prompts:  50%|▍| 490/985 [21:44<15:38,  1.89s/it, est. speed input: 216.41 toks/s, output: [repeated 11x across cluster]
(infer pid=654445) 
Processed prompts:  53%|▌| 519/985 [21:54<17:41,  2.28s/it, est. speed input: 225.32 toks/s, output: [repeated 4x across cluster]
(infer pid=654445) 
Processed prompts:  53%|▌| 523/985 [22:02<17:40,  2.30s/it, est. speed input: 226.28 toks/s, output: [repeated 10x across cluster]
(infer pid=654457) 
Processed prompts:  52%|▌| 517/985 [22:12<14:38,  1.88s/it, est. speed input: 213.29 toks/s, output: [repeated 10x across cluster]