logs/train_host0_latest.log

d with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 20 x]
step  58900 | loss 5.6608 | text 2.0356 audio 5.2537 | grad 2.533 | 1.70s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=0.99 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  58925 | loss 5.6221 | text 2.0197 audio 5.2181 | grad 1.749 | 1.71s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.04 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.69 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  58950 | loss 5.6630 | text 2.0276 audio 5.2574 | grad 1.502 | 1.70s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.61 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  58975 | loss 5.6446 | text 2.0266 audio 5.2393 | grad 2.194 | 1.73s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.14 | spike L=+0.00 G=0.93 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59000 | loss 5.6219 | text 2.0364 audio 5.2146 | grad 3.274 | 1.63s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.22 lora=0.96 model_audio_embed=0.05 projection=0.04 text_embed=0.15 | spike L=-0.00 G=1.39 | nonfinite=0
  running validation at step 59000...
  val/composite=2.9213 (text=0.5062 audio=4.5314) val/loss=4.6326 cb0_acc=40.2% text_acc=94.0%
  [val] per-codebook acc: cb0=40.2% cb1=20.8% cb2=17.3% cb3=11.5% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 6/10 (best 2.9180)
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_tckzisgc/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_059000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 9 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_059000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c7be2-6044317c2da5935b3406d40a;d95c5902-677e-4824-a526-7a7c4569d816)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 2 x]
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 13 x]
step  59025 | loss 5.6500 | text 2.0341 audio 5.2432 | grad 2.070 | 20.98s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.85 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59050 | loss 5.6327 | text 2.0213 audio 5.2285 | grad 4.327 | 1.67s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.11 | spike L=-0.00 G=1.80 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59075 | loss 5.6556 | text 2.0260 audio 5.2504 | grad 1.889 | 1.66s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.73 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59100 | loss 5.6249 | text 2.0266 audio 5.2196 | grad 1.735 | 1.67s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.39 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.15 | spike L=-0.00 G=0.69 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59125 | loss 5.6116 | text 2.0126 audio 5.2091 | grad 2.174 | 1.63s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=-0.00 G=0.89 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59150 | loss 5.6244 | text 2.0356 audio 5.2173 | grad 1.652 | 1.69s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.68 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59175 | loss 5.6098 | text 1.9965 audio 5.2105 | grad 2.138 | 1.60s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=0.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59200 | loss 5.6291 | text 2.0268 audio 5.2238 | grad 1.939 | 1.67s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.83 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59225 | loss 5.6684 | text 2.0334 audio 5.2618 | grad 2.792 | 1.67s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=+0.01 G=1.22 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59250 | loss 5.6182 | text 2.0205 audio 5.2141 | grad 1.641 | 1.68s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.70 | nonfinite=0
  running validation at step 59250...
  val/composite=2.9188 (text=0.4993 audio=4.5317) val/loss=4.6316 cb0_acc=40.1% text_acc=94.1%
  [val] per-codebook acc: cb0=40.1% cb1=20.8% cb2=17.4% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 5/10 (best 2.9180)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59275 | loss 5.6865 | text 2.0419 audio 5.2781 | grad 1.870 | 2.59s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.01 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59300 | loss 5.6442 | text 2.0160 audio 5.2410 | grad 2.357 | 1.65s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.06 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59325 | loss 5.6251 | text 2.0111 audio 5.2228 | grad 1.896 | 1.71s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.00 G=0.85 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59350 | loss 5.6522 | text 2.0390 audio 5.2444 | grad 4.210 | 1.69s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.17 lora=0.97 model_audio_embed=0.06 projection=0.03 text_embed=0.15 | spike L=+0.00 G=1.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59375 | loss 5.6388 | text 2.0256 audio 5.2336 | grad 2.339 | 1.63s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.15 | spike L=-0.00 G=0.97 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59400 | loss 5.6460 | text 2.0530 audio 5.2354 | grad 1.963 | 1.70s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.07 projection=0.09 text_embed=0.15 | spike L=+0.00 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59425 | loss 5.6564 | text 2.0297 audio 5.2505 | grad 3.531 | 1.70s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.18 lora=0.97 model_audio_embed=0.06 projection=0.04 text_embed=0.15 | spike L=+0.00 G=1.50 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59450 | loss 5.6295 | text 2.0196 audio 5.2256 | grad 2.602 | 1.65s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.04 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.05 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59475 | loss 5.6404 | text 2.0069 audio 5.2390 | grad 2.088 | 1.72s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.84 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59500 | loss 5.6474 | text 2.0175 audio 5.2439 | grad 1.774 | 1.68s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.00 G=0.73 | nonfinite=0
  running validation at step 59500...
  val/composite=2.9230 (text=0.5061 audio=4.5342) val/loss=4.6354 cb0_acc=40.3% text_acc=94.2%
  [val] per-codebook acc: cb0=40.3% cb1=20.8% cb2=17.4% cb3=11.4% cb4=8.7% cb5=7.0% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 4/10 (best 2.9180)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59525 | loss 5.6267 | text 2.0178 audio 5.2231 | grad 2.594 | 2.56s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.09 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59550 | loss 5.6427 | text 2.0167 audio 5.2393 | grad 1.838 | 1.70s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.38 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.77 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59575 | loss 5.6327 | text 2.0241 audio 5.2279 | grad 1.872 | 1.63s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.80 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59600 | loss 5.6506 | text 2.0063 audio 5.2493 | grad 1.691 | 1.70s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.74 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59625 | loss 5.6375 | text 2.0416 audio 5.2292 | grad 2.602 | 1.65s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.16 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59650 | loss 5.6336 | text 2.0194 audio 5.2298 | grad 2.160 | 1.72s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.00 G=0.95 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59675 | loss 5.6508 | text 2.0226 audio 5.2463 | grad 3.325 | 1.69s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.19 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=+0.00 G=1.47 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59700 | loss 5.6646 | text 2.0284 audio 5.2589 | grad 2.617 | 1.63s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.11 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59725 | loss 5.6228 | text 2.0132 audio 5.2202 | grad 2.147 | 1.63s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.33 lora=0.92 model_audio_embed=0.05 projection=0.06 text_embed=0.18 | spike L=-0.00 G=0.90 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59750 | loss 5.6416 | text 2.0091 audio 5.2397 | grad 1.891 | 1.64s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.14 | spike L=+0.00 G=0.80 | nonfinite=0
  running validation at step 59750...
  val/composite=2.9185 (text=0.5016 audio=4.5298) val/loss=4.6301 cb0_acc=40.2% text_acc=94.0%
  [val] per-codebook acc: cb0=40.2% cb1=20.9% cb2=17.4% cb3=11.5% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 3/10 (best 2.9180)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59775 | loss 5.6283 | text 2.0157 audio 5.2252 | grad 1.414 | 2.45s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.15 | spike L=-0.00 G=0.61 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59800 | loss 5.6036 | text 2.0125 audio 5.2011 | grad 1.718 | 1.60s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.01 G=0.77 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59825 | loss 5.6446 | text 2.0277 audio 5.2391 | grad 1.797 | 1.60s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59850 | loss 5.6489 | text 2.0234 audio 5.2442 | grad 1.905 | 1.68s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.89 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59875 | loss 5.6442 | text 2.0206 audio 5.2400 | grad 2.162 | 1.65s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.02 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59900 | loss 5.6512 | text 2.0213 audio 5.2470 | grad 2.023 | 1.58s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.15 | spike L=+0.00 G=0.95 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59925 | loss 5.6497 | text 2.0397 audio 5.2417 | grad 1.466 | 1.65s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.00 G=0.69 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59950 | loss 5.6367 | text 2.0264 audio 5.2314 | grad 2.708 | 1.60s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.06 projection=0.06 text_embed=0.16 | spike L=-0.00 G=1.32 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  59975 | loss 5.6135 | text 2.0159 audio 5.2103 | grad 1.826 | 1.60s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.86 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60000 | loss 5.6253 | text 2.0357 audio 5.2182 | grad 1.752 | 1.71s/step | peak -1.0G | host_rss 53.9G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.84 | nonfinite=0
  [audio-demo] ar_cb0_acc=4.0% (39s)
  running validation at step 60000...

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

[repeated 2 x]
  ...es/step_060000/source.wav: 100%|██████████|  230kB /  230kB            

[repeated 2 x]
  ...es/step_060000/source.wav: 100%|██████████|  230kB /  230kB            
Processing Files (1 / 1)      : 100%|██████████|  230kB /  230kB,   ???B/s  

New Data Upload               : |          |  0.00B /  0.00B,   ???B/s  

  ...es/step_060000/source.wav: 100%|██████████|  230kB /  230kB            

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

[repeated 2 x]
  ...step_060000/target_gt.wav: 100%|██████████|  192kB /  192kB            

[repeated 2 x]
  ...step_060000/target_gt.wav: 100%|██████████|  192kB /  192kB            
Processing Files (1 / 1)      : 100%|██████████|  192kB /  192kB,   ???B/s  

New Data Upload               : |          |  0.00B /  0.00B,   ???B/s  

  ...step_060000/target_gt.wav: 100%|██████████|  192kB /  192kB            

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

[repeated 2 x]
  ...step_060000/generated.wav: 100%|██████████|  192kB /  192kB            

[repeated 2 x]
  ...step_060000/generated.wav: 100%|██████████|  192kB /  192kB            
Processing Files (1 / 1)      : 100%|██████████|  192kB /  192kB, 18.6kB/s  

New Data Upload               : 100%|██████████|  192kB /  192kB, 18.6kB/s  
Processing Files (1 / 1)      : 100%|██████████|  192kB /  192kB, 18.6kB/s  

New Data Upload               : 100%|██████████|  192kB /  192kB, 18.6kB/s  

  ...step_060000/generated.wav: 100%|██████████|  192kB /  192kB            
  pushed 3 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/samples/step_060000
  val/composite=2.9191 (text=0.4985 audio=4.5328) val/loss=4.6325 cb0_acc=40.2% text_acc=94.1%
  [val] per-codebook acc: cb0=40.2% cb1=20.8% cb2=17.3% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.1% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 2/10 (best 2.9180)
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_r4b9zu_b/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_060000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 9 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_060000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 4 x]
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c84ad-68c7f54f0aed8c7f492ca182;e6edbf94-4f52-433a-96a3-48c46c500147)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 11 x]
step  60025 | loss 5.6046 | text 2.0023 audio 5.2041 | grad 2.136 | 22.56s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.01 G=1.04 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60050 | loss 5.6328 | text 2.0258 audio 5.2276 | grad 1.592 | 1.57s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.77 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60075 | loss 5.6538 | text 2.0100 audio 5.2518 | grad 1.702 | 1.70s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.85 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60100 | loss 5.6264 | text 2.0122 audio 5.2240 | grad 2.605 | 1.61s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.31 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60125 | loss 5.6059 | text 2.0175 audio 5.2024 | grad 2.672 | 1.55s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.06 projection=0.05 text_embed=0.14 | spike L=-0.01 G=1.31 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60150 | loss 5.6483 | text 2.0357 audio 5.2411 | grad 1.795 | 1.68s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.39 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.16 | spike L=+0.00 G=0.85 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60175 | loss 5.6387 | text 2.0303 audio 5.2326 | grad 2.284 | 1.61s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=+0.00 G=1.10 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60200 | loss 5.6421 | text 2.0351 audio 5.2351 | grad 1.701 | 1.58s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.81 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60225 | loss 5.6729 | text 2.0525 audio 5.2624 | grad 2.159 | 1.67s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=+0.01 G=1.05 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60250 | loss 5.5956 | text 2.0121 audio 5.1931 | grad 1.960 | 1.70s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.01 G=0.95 | nonfinite=0
  running validation at step 60250...
  val/composite=2.9162 (text=0.4965 audio=4.5293) val/loss=4.6286 cb0_acc=40.4% text_acc=94.2%
  [val] per-codebook acc: cb0=40.4% cb1=20.9% cb2=17.5% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.1% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_35qpnyf2/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c8821-44955d660b9386db5bd9c052;ee9dc42b-0b2b-4a7e-9fa4-5be5370d81c6)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60275 | loss 5.6737 | text 2.0276 audio 5.2682 | grad 1.734 | 21.60s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.01 G=0.84 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60300 | loss 5.6303 | text 2.0205 audio 5.2262 | grad 1.959 | 1.68s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.37 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.13 | spike L=-0.00 G=0.97 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60325 | loss 5.6667 | text 2.0369 audio 5.2594 | grad 2.066 | 1.67s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=+0.01 G=1.02 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60350 | loss 5.6346 | text 2.0525 audio 5.2241 | grad 1.679 | 1.61s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.83 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60375 | loss 5.6148 | text 2.0264 audio 5.2095 | grad 1.622 | 1.61s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60400 | loss 5.6265 | text 2.0272 audio 5.2210 | grad 2.695 | 1.61s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.26 lora=0.94 model_audio_embed=0.06 projection=0.05 text_embed=0.18 | spike L=-0.00 G=1.38 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60425 | loss 5.6592 | text 2.0417 audio 5.2509 | grad 2.455 | 1.67s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.27 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.21 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60450 | loss 5.6348 | text 2.0262 audio 5.2296 | grad 2.854 | 1.70s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.14 | spike L=-0.00 G=1.38 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60475 | loss 5.6254 | text 2.0361 audio 5.2182 | grad 1.709 | 1.68s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.80 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60500 | loss 5.6632 | text 2.0707 audio 5.2490 | grad 3.524 | 1.68s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.20 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=+0.00 G=1.68 | nonfinite=0
  running validation at step 60500...
  val/composite=2.9146 (text=0.4916 audio=4.5300) val/loss=4.6283 cb0_acc=40.2% text_acc=94.2%
  [val] per-codebook acc: cb0=40.2% cb1=20.8% cb2=17.3% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.1% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_zh770a3p/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c8bb6-7b93d770713a975c255391c3;d16b151a-cf1c-403a-a104-4a2e88af84f9)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60525 | loss 5.6387 | text 2.0338 audio 5.2319 | grad 1.729 | 21.70s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.77 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60550 | loss 5.6260 | text 2.0189 audio 5.2222 | grad 2.771 | 1.55s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.26 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60575 | loss 5.6490 | text 2.0217 audio 5.2447 | grad 1.520 | 1.55s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.00 G=0.67 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60600 | loss 5.6140 | text 2.0086 audio 5.2123 | grad 2.629 | 1.62s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.21 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60625 | loss 5.6098 | text 2.0132 audio 5.2072 | grad 1.461 | 1.58s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.66 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60650 | loss 5.6262 | text 2.0319 audio 5.2198 | grad 2.149 | 1.64s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.00 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60675 | loss 5.6258 | text 2.0023 audio 5.2253 | grad 2.016 | 1.62s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.94 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60700 | loss 5.6354 | text 2.0332 audio 5.2288 | grad 2.595 | 1.55s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=+0.00 G=1.22 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60725 | loss 5.6421 | text 2.0227 audio 5.2376 | grad 2.456 | 1.56s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.13 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60750 | loss 5.6392 | text 2.0205 audio 5.2351 | grad 1.929 | 1.55s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.87 | nonfinite=0
  running validation at step 60750...
  val/composite=2.9120 (text=0.4915 audio=4.5256) val/loss=4.6239 cb0_acc=40.2% text_acc=94.2%
  [val] per-codebook acc: cb0=40.2% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_1vrjdv1y/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c8f3a-03c6136d4757fe625704cd46;a41a9c7e-a65b-4aa8-ab69-31d4da92f4d3)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60775 | loss 5.6375 | text 2.0385 audio 5.2298 | grad 1.605 | 21.73s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.74 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60800 | loss 5.6187 | text 1.9993 audio 5.2188 | grad 2.126 | 1.63s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.00 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60825 | loss 5.6698 | text 2.0585 audio 5.2581 | grad 3.554 | 1.64s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.21 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.09 | spike L=+0.01 G=1.67 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60850 | loss 5.6332 | text 2.0248 audio 5.2282 | grad 2.376 | 1.71s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.29 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.14 | spike L=-0.00 G=1.05 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60875 | loss 5.6428 | text 2.0225 audio 5.2383 | grad 1.801 | 1.68s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.79 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60900 | loss 5.6409 | text 2.0196 audio 5.2369 | grad 1.811 | 1.62s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.81 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60925 | loss 5.6408 | text 2.0341 audio 5.2340 | grad 1.813 | 1.62s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.83 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60950 | loss 5.6216 | text 2.0389 audio 5.2138 | grad 2.512 | 1.56s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.17 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  60975 | loss 5.6586 | text 2.0306 audio 5.2525 | grad 1.832 | 1.62s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.14 | spike L=+0.00 G=0.84 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61000 | loss 5.6484 | text 2.0268 audio 5.2431 | grad 1.397 | 1.62s/step | peak -1.0G | host_rss 54.0G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.65 | nonfinite=0
  running validation at step 61000...
  val/composite=2.9117 (text=0.4927 audio=4.5243) val/loss=4.6228 cb0_acc=40.3% text_acc=94.2%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.4% cb3=11.5% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_srnn_13x/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c92c2-0e7f6dd80ebf7af001909466;3c1b1587-c82b-4dd3-b53f-687addc54c33)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_ibt4i8_3/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_061000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 8 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_061000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 3 x]
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c94a4-5dc28e057f29b2b210b11506;91331fd6-f77a-451a-9829-0cf12fe40bd0)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 13 x]
step  61025 | loss 5.6428 | text 2.0056 audio 5.2416 | grad 2.801 | 40.08s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.35 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61050 | loss 5.6127 | text 1.9964 audio 5.2134 | grad 3.337 | 1.64s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.09 | spike L=-0.00 G=1.55 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61075 | loss 5.6284 | text 2.0223 audio 5.2240 | grad 2.862 | 1.65s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.26 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61100 | loss 5.6213 | text 2.0093 audio 5.2195 | grad 2.464 | 1.56s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.29 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=-0.00 G=1.06 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61125 | loss 5.6565 | text 2.0278 audio 5.2509 | grad 2.091 | 1.56s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.89 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61150 | loss 5.6400 | text 2.0092 audio 5.2381 | grad 1.858 | 1.57s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=+0.00 G=0.80 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61175 | loss 5.6276 | text 2.0315 audio 5.2213 | grad 2.374 | 1.63s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.29 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.17 | spike L=-0.00 G=1.05 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61200 | loss 5.6448 | text 2.0157 audio 5.2416 | grad 1.544 | 1.56s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.14 | spike L=+0.00 G=0.68 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61225 | loss 5.6341 | text 2.0345 audio 5.2272 | grad 3.151 | 1.63s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.08 | spike L=-0.00 G=1.43 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61250 | loss 5.6100 | text 2.0241 audio 5.2052 | grad 3.419 | 1.65s/step | peak -1.0G | host_rss 55.1G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.06 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.49 | nonfinite=0
  running validation at step 61250...
  val/composite=2.9109 (text=0.4932 audio=4.5227) val/loss=4.6213 cb0_acc=40.3% text_acc=94.3%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.8% cb5=7.1% cb6=6.0% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_jn5bsrv3/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c9817-4943bb871a2ce6996e47559e;3f102f5b-1d4f-45e5-93d6-823615b2dcd2)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61275 | loss 5.6501 | text 2.0370 audio 5.2427 | grad 1.835 | 21.62s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.76 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61300 | loss 5.5926 | text 2.0028 audio 5.1920 | grad 2.676 | 1.56s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.01 G=1.14 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61325 | loss 5.6382 | text 2.0304 audio 5.2321 | grad 1.931 | 1.56s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.81 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61350 | loss 5.6655 | text 2.0280 audio 5.2599 | grad 1.617 | 1.56s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.01 G=0.69 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61375 | loss 5.6112 | text 2.0141 audio 5.2084 | grad 3.038 | 1.56s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.34 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61400 | loss 5.6246 | text 2.0086 audio 5.2228 | grad 2.991 | 1.62s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.04 projection=0.05 text_embed=0.07 | spike L=-0.00 G=1.27 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61425 | loss 5.6382 | text 1.9996 audio 5.2382 | grad 2.010 | 1.63s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.83 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61450 | loss 5.6202 | text 2.0221 audio 5.2158 | grad 1.972 | 1.64s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.14 | spike L=-0.00 G=0.83 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61475 | loss 5.6332 | text 2.0228 audio 5.2286 | grad 2.922 | 1.64s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.24 lora=0.97 model_audio_embed=0.04 projection=0.05 text_embed=0.08 | spike L=+0.00 G=1.25 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61500 | loss 5.6317 | text 2.0201 audio 5.2277 | grad 2.457 | 1.55s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=1.03 | nonfinite=0
  running validation at step 61500...
  val/composite=2.9164 (text=0.4991 audio=4.5279) val/loss=4.6277 cb0_acc=40.2% text_acc=94.1%
  [val] per-codebook acc: cb0=40.2% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.1% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 9/10 (best 2.9109)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61525 | loss 5.6175 | text 2.0082 audio 5.2158 | grad 1.984 | 2.48s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.12 | spike L=-0.00 G=0.83 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61550 | loss 5.6448 | text 2.0119 audio 5.2424 | grad 3.046 | 1.59s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.12 | spike L=+0.00 G=1.29 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61575 | loss 5.6349 | text 2.0179 audio 5.2314 | grad 3.292 | 1.59s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.08 | spike L=+0.00 G=1.36 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61600 | loss 5.6162 | text 2.0119 audio 5.2138 | grad 2.950 | 1.65s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.06 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.17 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61625 | loss 5.6358 | text 2.0153 audio 5.2328 | grad 2.000 | 1.59s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.78 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61650 | loss 5.6410 | text 2.0144 audio 5.2381 | grad 2.750 | 1.66s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.10 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61675 | loss 5.6082 | text 2.0030 audio 5.2076 | grad 1.624 | 1.68s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.64 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61700 | loss 5.6254 | text 2.0288 audio 5.2196 | grad 2.377 | 1.59s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.32 lora=0.93 model_audio_embed=0.06 projection=0.06 text_embed=0.14 | spike L=-0.00 G=0.98 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61725 | loss 5.6230 | text 2.0264 audio 5.2177 | grad 2.649 | 1.59s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.09 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61750 | loss 5.6319 | text 2.0076 audio 5.2304 | grad 1.684 | 1.59s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.69 | nonfinite=0
  running validation at step 61750...
  val/composite=2.9113 (text=0.4940 audio=4.5228) val/loss=4.6216 cb0_acc=40.3% text_acc=94.4%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.5% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 8/10 (best 2.9109)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61775 | loss 5.6288 | text 2.0330 audio 5.2222 | grad 2.214 | 2.54s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.06 projection=0.07 text_embed=0.14 | spike L=-0.00 G=0.93 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61800 | loss 5.6172 | text 2.0186 audio 5.2135 | grad 2.229 | 1.65s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.34 lora=0.92 model_audio_embed=0.06 projection=0.07 text_embed=0.17 | spike L=-0.00 G=0.95 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61825 | loss 5.6338 | text 2.0178 audio 5.2302 | grad 1.599 | 1.71s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.68 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61850 | loss 5.6302 | text 2.0481 audio 5.2206 | grad 2.260 | 1.72s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=+0.00 G=1.00 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61875 | loss 5.6288 | text 2.0299 audio 5.2228 | grad 1.939 | 1.73s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.85 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61900 | loss 5.6285 | text 2.0383 audio 5.2208 | grad 2.259 | 1.74s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.08 | spike L=-0.00 G=1.01 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61925 | loss 5.6133 | text 2.0074 audio 5.2118 | grad 1.686 | 1.66s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.14 | spike L=-0.00 G=0.75 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61950 | loss 5.6403 | text 2.0244 audio 5.2355 | grad 2.342 | 1.66s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.07 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  61975 | loss 5.6325 | text 2.0290 audio 5.2267 | grad 2.339 | 1.65s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.06 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62000 | loss 5.6147 | text 2.0318 audio 5.2083 | grad 2.280 | 1.72s/step | peak -1.0G | host_rss 56.2G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.03 | nonfinite=0
  running validation at step 62000...
  val/composite=2.9086 (text=0.4899 audio=4.5210) val/loss=4.6190 cb0_acc=40.3% text_acc=94.3%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.7% cb5=7.2% cb6=6.0% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_kmg2jwib/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5c9eff-17bdad923b7eb47e105f68f0;fe5507d1-5c57-494a-8ff6-171c7f019db9)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_p4m82smv/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_062000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 8 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_062000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 4 x]
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5ca0df-77aa1290197e3b26627dd500;34c8dc17-0311-4669-95b3-707a8887b986)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 12 x]
step  62025 | loss 5.6517 | text 2.0292 audio 5.2459 | grad 2.919 | 39.96s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.31 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62050 | loss 5.6325 | text 2.0149 audio 5.2295 | grad 1.749 | 1.57s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.76 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62075 | loss 5.6338 | text 2.0016 audio 5.2335 | grad 1.739 | 1.69s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.00 G=0.78 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62100 | loss 5.6301 | text 2.0153 audio 5.2270 | grad 2.812 | 1.70s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.29 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62125 | loss 5.6316 | text 2.0207 audio 5.2275 | grad 1.923 | 1.71s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=+0.00 G=0.85 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62150 | loss 5.6285 | text 2.0245 audio 5.2236 | grad 1.774 | 1.63s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.80 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62175 | loss 5.6208 | text 2.0155 audio 5.2177 | grad 2.290 | 1.64s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.05 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62200 | loss 5.6690 | text 2.0318 audio 5.2626 | grad 1.666 | 1.63s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.01 G=0.76 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62225 | loss 5.6330 | text 2.0074 audio 5.2315 | grad 2.157 | 1.70s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=-0.00 G=1.01 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62250 | loss 5.6340 | text 2.0130 audio 5.2314 | grad 1.748 | 1.70s/step | peak -1.0G | host_rss 56.7G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.82 | nonfinite=0
  running validation at step 62250...
  val/composite=2.9082 (text=0.4844 audio=4.5241) val/loss=4.6210 cb0_acc=40.3% text_acc=94.3%
  [val] per-codebook acc: cb0=40.3% cb1=21.0% cb2=17.5% cb3=11.5% cb4=8.7% cb5=7.2% cb6=6.0% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt__vi7uku7/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5ca45d-556eff5c3ff2618b028b5acd;50e04c4d-abe6-4342-ac6d-504c4dab7c7c)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62275 | loss 5.5979 | text 1.9899 audio 5.1999 | grad 3.348 | 21.62s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.07 projection=0.05 text_embed=0.11 | spike L=-0.01 G=1.60 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62300 | loss 5.6241 | text 2.0135 audio 5.2214 | grad 2.046 | 1.63s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=-0.00 G=0.92 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62325 | loss 5.6503 | text 2.0214 audio 5.2460 | grad 1.815 | 1.59s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62350 | loss 5.6211 | text 2.0142 audio 5.2182 | grad 2.908 | 1.66s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.34 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62375 | loss 5.6323 | text 2.0194 audio 5.2285 | grad 2.077 | 1.57s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.93 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62400 | loss 5.6452 | text 2.0118 audio 5.2429 | grad 1.671 | 1.57s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.75 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62425 | loss 5.6508 | text 2.0077 audio 5.2492 | grad 2.484 | 1.56s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.09 | spike L=+0.00 G=1.15 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62450 | loss 5.6053 | text 1.9932 audio 5.2066 | grad 1.576 | 1.63s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.01 G=0.72 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62475 | loss 5.6367 | text 2.0163 audio 5.2334 | grad 2.017 | 1.63s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.14 | spike L=+0.00 G=0.94 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62500 | loss 5.6114 | text 2.0073 audio 5.2099 | grad 2.398 | 1.63s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=1.13 | nonfinite=0
  running validation at step 62500...
  val/composite=2.9077 (text=0.4855 audio=4.5224) val/loss=4.6195 cb0_acc=40.3% text_acc=94.5%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.3% cb3=11.5% cb4=8.8% cb5=7.2% cb6=6.1% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_znsn6zzh/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5ca7e5-61f2a2ae22273d320bb0949a;8607330c-1266-499a-b05f-47fe858efe18)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62525 | loss 5.6469 | text 2.0229 audio 5.2423 | grad 1.957 | 21.68s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62550 | loss 5.6262 | text 2.0273 audio 5.2207 | grad 2.322 | 1.71s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.09 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62575 | loss 5.6340 | text 2.0036 audio 5.2333 | grad 1.767 | 1.59s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.39 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.17 | spike L=+0.00 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62600 | loss 5.6388 | text 2.0187 audio 5.2351 | grad 1.736 | 1.65s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.38 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62625 | loss 5.6186 | text 1.9945 audio 5.2197 | grad 2.093 | 1.64s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=-0.00 G=1.01 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62650 | loss 5.6339 | text 2.0050 audio 5.2329 | grad 1.979 | 1.63s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.38 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.95 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62675 | loss 5.6059 | text 1.9938 audio 5.2072 | grad 3.201 | 1.70s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=-0.00 G=1.55 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62700 | loss 5.6193 | text 2.0020 audio 5.2189 | grad 1.658 | 1.70s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.76 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62725 | loss 5.6164 | text 2.0224 audio 5.2120 | grad 2.190 | 1.57s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.12 | spike L=-0.00 G=1.03 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62750 | loss 5.6582 | text 2.0456 audio 5.2490 | grad 1.821 | 1.63s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.37 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.01 G=0.85 | nonfinite=0
  running validation at step 62750...
  val/composite=2.9048 (text=0.4855 audio=4.5177) val/loss=4.6148 cb0_acc=40.4% text_acc=94.4%
  [val] per-codebook acc: cb0=40.4% cb1=21.0% cb2=17.4% cb3=11.5% cb4=8.8% cb5=7.2% cb6=6.1% cb7=5.9%
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_ktm0_bqi/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5cab72-3f1d200a23e0639057cab243;b2ca5168-f214-46c3-8a98-a738decdadab)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62775 | loss 5.5919 | text 2.0118 audio 5.1895 | grad 2.302 | 21.54s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.01 G=1.09 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62800 | loss 5.6366 | text 2.0037 audio 5.2358 | grad 2.000 | 1.64s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.94 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62825 | loss 5.6388 | text 2.0046 audio 5.2379 | grad 3.066 | 1.59s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.15 | spike L=+0.00 G=1.45 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62850 | loss 5.6341 | text 2.0062 audio 5.2329 | grad 1.740 | 1.58s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.79 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62875 | loss 5.6400 | text 2.0085 audio 5.2383 | grad 1.998 | 1.66s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.93 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62900 | loss 5.6607 | text 2.0177 audio 5.2571 | grad 2.056 | 1.56s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.01 G=0.96 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62925 | loss 5.5808 | text 2.0316 audio 5.1744 | grad 3.022 | 1.63s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.01 G=1.42 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62950 | loss 5.6386 | text 2.0349 audio 5.2316 | grad 1.986 | 1.64s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=+0.00 G=0.89 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  62975 | loss 5.6305 | text 2.0263 audio 5.2253 | grad 2.629 | 1.56s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.20 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63000 | loss 5.6155 | text 2.0334 audio 5.2088 | grad 2.718 | 1.63s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.13 | spike L=-0.00 G=1.21 | nonfinite=0
  running validation at step 63000...
  val/composite=2.9107 (text=0.4927 audio=4.5227) val/loss=4.6213 cb0_acc=40.3% text_acc=94.2%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.5% cb3=11.5% cb4=8.7% cb5=7.2% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 9/10 (best 2.9048)
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_2713ve_g/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_063000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 11 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_063000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 2 x]
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5caefd-56326671658195f47377e6bb;47d88419-c9fa-4157-83c2-6f74321372fa)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 11 x]
step  63025 | loss 5.6522 | text 2.0327 audio 5.2457 | grad 1.627 | 20.92s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.14 | spike L=+0.00 G=0.71 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63050 | loss 5.6141 | text 2.0055 audio 5.2130 | grad 2.248 | 1.64s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=-0.00 G=1.01 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63075 | loss 5.6132 | text 2.0134 audio 5.2105 | grad 3.396 | 1.58s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.11 | spike L=-0.00 G=1.53 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63100 | loss 5.6493 | text 2.0148 audio 5.2464 | grad 2.133 | 1.59s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=+0.00 G=0.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63125 | loss 5.6035 | text 2.0016 audio 5.2032 | grad 2.010 | 1.66s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.87 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63150 | loss 5.5969 | text 2.0316 audio 5.1905 | grad 1.859 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.01 G=0.81 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63175 | loss 5.6180 | text 1.9996 audio 5.2180 | grad 1.692 | 1.63s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.15 | spike L=-0.00 G=0.75 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63200 | loss 5.6155 | text 2.0311 audio 5.2093 | grad 1.971 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.14 | spike L=-0.00 G=0.90 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63225 | loss 5.6158 | text 1.9921 audio 5.2173 | grad 2.506 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.06 projection=0.06 text_embed=0.13 | spike L=-0.00 G=1.15 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63250 | loss 5.6148 | text 2.0121 audio 5.2123 | grad 1.778 | 1.63s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.81 | nonfinite=0
  running validation at step 63250...
  val/composite=2.9154 (text=0.4941 audio=4.5295) val/loss=4.6284 cb0_acc=40.3% text_acc=94.3%
  [val] per-codebook acc: cb0=40.3% cb1=20.8% cb2=17.3% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 8/10 (best 2.9048)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63275 | loss 5.5931 | text 2.0123 audio 5.1906 | grad 1.560 | 2.47s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.72 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63300 | loss 5.6089 | text 2.0157 audio 5.2058 | grad 1.984 | 1.66s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.37 lora=0.89 model_audio_embed=0.06 projection=0.07 text_embed=0.23 | spike L=-0.00 G=0.94 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63325 | loss 5.6182 | text 2.0011 audio 5.2180 | grad 1.522 | 1.61s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.73 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63350 | loss 5.6015 | text 1.9977 audio 5.2020 | grad 2.422 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=1.19 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63375 | loss 5.6207 | text 2.0354 audio 5.2136 | grad 2.968 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.12 | spike L=+0.00 G=1.43 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63400 | loss 5.6111 | text 2.0067 audio 5.2097 | grad 2.043 | 1.68s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.06 text_embed=0.14 | spike L=-0.00 G=0.95 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63425 | loss 5.6086 | text 2.0316 audio 5.2023 | grad 2.148 | 1.66s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=-0.00 G=1.00 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63450 | loss 5.6034 | text 2.0145 audio 5.2005 | grad 2.234 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=-0.00 G=1.04 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63475 | loss 5.6551 | text 2.0419 audio 5.2467 | grad 1.564 | 1.58s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.01 G=0.72 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63500 | loss 5.5945 | text 2.0059 audio 5.1933 | grad 1.988 | 1.59s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.95 | nonfinite=0
  running validation at step 63500...
  val/composite=2.9120 (text=0.4900 audio=4.5266) val/loss=4.6246 cb0_acc=40.4% text_acc=94.3%
  [val] per-codebook acc: cb0=40.4% cb1=20.8% cb2=17.3% cb3=11.4% cb4=8.7% cb5=7.2% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 7/10 (best 2.9048)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63525 | loss 5.6441 | text 2.0108 audio 5.2419 | grad 1.722 | 2.39s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.01 G=0.82 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63550 | loss 5.6387 | text 2.0096 audio 5.2368 | grad 2.114 | 1.52s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.06 projection=0.07 text_embed=0.14 | spike L=+0.00 G=1.03 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63575 | loss 5.6072 | text 1.9881 audio 5.2096 | grad 1.880 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=-0.00 G=0.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63600 | loss 5.6101 | text 1.9770 audio 5.2147 | grad 2.369 | 1.54s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=-0.00 G=1.16 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63625 | loss 5.6415 | text 2.0174 audio 5.2380 | grad 4.812 | 1.53s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.15 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.14 | spike L=+0.00 G=2.32 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63650 | loss 5.6533 | text 2.0362 audio 5.2460 | grad 1.682 | 1.53s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.01 G=0.72 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63675 | loss 5.6107 | text 2.0186 audio 5.2069 | grad 2.252 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=-0.00 G=0.99 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63700 | loss 5.6506 | text 2.0315 audio 5.2443 | grad 1.508 | 1.68s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.66 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63725 | loss 5.6014 | text 2.0237 audio 5.1967 | grad 1.945 | 1.52s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.88 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63750 | loss 5.6099 | text 2.0074 audio 5.2084 | grad 2.350 | 1.58s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.08 | nonfinite=0
  running validation at step 63750...
  val/composite=2.9073 (text=0.4841 audio=4.5228) val/loss=4.6196 cb0_acc=40.3% text_acc=94.3%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.8% cb5=7.1% cb6=6.1% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 6/10 (best 2.9048)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63775 | loss 5.6197 | text 2.0103 audio 5.2176 | grad 1.506 | 2.50s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.69 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63800 | loss 5.6260 | text 2.0226 audio 5.2215 | grad 1.587 | 1.60s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.75 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63825 | loss 5.6380 | text 2.0074 audio 5.2365 | grad 1.849 | 1.60s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=+0.00 G=0.89 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63850 | loss 5.6158 | text 2.0055 audio 5.2147 | grad 1.860 | 1.67s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63875 | loss 5.5893 | text 1.9992 audio 5.1895 | grad 1.788 | 1.60s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.01 G=0.88 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63900 | loss 5.6361 | text 2.0006 audio 5.2360 | grad 1.841 | 1.59s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.92 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63925 | loss 5.6586 | text 2.0081 audio 5.2570 | grad 2.330 | 1.59s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.01 G=1.17 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63950 | loss 5.6368 | text 2.0119 audio 5.2344 | grad 2.147 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=+0.00 G=1.06 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  63975 | loss 5.6202 | text 2.0225 audio 5.2157 | grad 2.422 | 1.67s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.19 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64000 | loss 5.6138 | text 2.0155 audio 5.2107 | grad 2.260 | 1.70s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.09 | nonfinite=0
  running validation at step 64000...
  val/composite=2.9096 (text=0.4881 audio=4.5240) val/loss=4.6216 cb0_acc=40.4% text_acc=94.3%
  [val] per-codebook acc: cb0=40.4% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.8% cb5=7.2% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 5/10 (best 2.9048)
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_uz4m4s3i/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_064000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 8 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_064000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 3 x]
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5cb75d-57bf9b6113e5afa36298c379;3a042a3f-7caf-4e18-94d3-f3c7229be818)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 13 x]
step  64025 | loss 5.6188 | text 2.0123 audio 5.2163 | grad 3.112 | 20.83s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.49 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64050 | loss 5.6119 | text 1.9986 audio 5.2122 | grad 1.410 | 1.66s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.54 lora=0.82 model_audio_embed=0.05 projection=0.10 text_embed=0.13 | spike L=-0.00 G=0.64 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64075 | loss 5.6502 | text 2.0223 audio 5.2457 | grad 2.963 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.13 | spike L=+0.00 G=1.40 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64100 | loss 5.6539 | text 2.0277 audio 5.2484 | grad 1.585 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.14 | spike L=+0.01 G=0.72 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64125 | loss 5.6187 | text 2.0059 audio 5.2175 | grad 3.355 | 1.64s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.57 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64150 | loss 5.6230 | text 1.9946 audio 5.2241 | grad 2.356 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=-0.00 G=1.04 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64175 | loss 5.6215 | text 2.0184 audio 5.2178 | grad 2.470 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=1.09 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64200 | loss 5.6170 | text 2.0116 audio 5.2146 | grad 1.790 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.78 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64225 | loss 5.6009 | text 2.0101 audio 5.1988 | grad 2.032 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.91 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64250 | loss 5.6112 | text 2.0126 audio 5.2087 | grad 2.493 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.12 | nonfinite=0
  running validation at step 64250...
  val/composite=2.9072 (text=0.4840 audio=4.5227) val/loss=4.6195 cb0_acc=40.5% text_acc=94.5%
  [val] per-codebook acc: cb0=40.5% cb1=20.9% cb2=17.4% cb3=11.5% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 4/10 (best 2.9048)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64275 | loss 5.6035 | text 1.9874 audio 5.2060 | grad 2.154 | 2.49s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=0.96 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64300 | loss 5.6162 | text 2.0233 audio 5.2115 | grad 1.474 | 1.67s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.66 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64325 | loss 5.6311 | text 2.0293 audio 5.2253 | grad 2.861 | 1.67s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.32 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64350 | loss 5.6531 | text 2.0139 audio 5.2503 | grad 1.909 | 1.62s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.01 G=0.86 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64375 | loss 5.5969 | text 2.0314 audio 5.1906 | grad 2.145 | 1.63s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.15 | spike L=-0.00 G=0.98 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64400 | loss 5.6525 | text 2.0445 audio 5.2436 | grad 1.660 | 1.62s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.14 | spike L=+0.01 G=0.76 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64425 | loss 5.6081 | text 2.0017 audio 5.2078 | grad 1.803 | 1.60s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.84 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64450 | loss 5.6270 | text 1.9874 audio 5.2295 | grad 2.192 | 1.52s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=1.04 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64475 | loss 5.6397 | text 2.0359 audio 5.2325 | grad 3.969 | 1.53s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.18 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=+0.00 G=1.88 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64500 | loss 5.6310 | text 1.9941 audio 5.2322 | grad 2.123 | 1.53s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.92 | nonfinite=0
  running validation at step 64500...
  val/composite=2.9083 (text=0.4880 audio=4.5218) val/loss=4.6194 cb0_acc=40.4% text_acc=94.5%
  [val] per-codebook acc: cb0=40.4% cb1=21.0% cb2=17.5% cb3=11.5% cb4=8.7% cb5=7.1% cb6=6.1% cb7=5.8%
  [early-stop] no val improvement (>0.0); patience 3/10 (best 2.9048)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64525 | loss 5.6266 | text 2.0098 audio 5.2246 | grad 2.304 | 2.34s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.01 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64550 | loss 5.6318 | text 2.0286 audio 5.2260 | grad 1.816 | 1.52s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.53 lora=0.84 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.79 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64575 | loss 5.6496 | text 2.0140 audio 5.2468 | grad 1.806 | 1.53s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.41 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.17 | spike L=+0.00 G=0.81 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64600 | loss 5.6733 | text 2.0341 audio 5.2665 | grad 2.425 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.01 G=1.10 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64625 | loss 5.6033 | text 2.0195 audio 5.1994 | grad 1.676 | 1.61s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.01 G=0.76 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64650 | loss 5.5907 | text 1.9986 audio 5.1910 | grad 2.841 | 1.53s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.06 projection=0.06 text_embed=0.12 | spike L=-0.01 G=1.31 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64675 | loss 5.6488 | text 2.0275 audio 5.2433 | grad 1.790 | 1.53s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.80 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64700 | loss 5.6577 | text 2.0338 audio 5.2509 | grad 1.648 | 1.59s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.01 G=0.75 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64725 | loss 5.6350 | text 2.0368 audio 5.2276 | grad 1.893 | 1.66s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.00 G=0.89 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64750 | loss 5.6456 | text 2.0235 audio 5.2409 | grad 3.744 | 1.69s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.19 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.12 | spike L=+0.00 G=1.77 | nonfinite=0
  running validation at step 64750...
  val/composite=2.9149 (text=0.4974 audio=4.5265) val/loss=4.6260 cb0_acc=40.3% text_acc=94.4%
  [val] per-codebook acc: cb0=40.3% cb1=20.9% cb2=17.4% cb3=11.5% cb4=8.8% cb5=7.2% cb6=6.1% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 2/10 (best 2.9048)
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64775 | loss 5.6252 | text 2.0151 audio 5.2222 | grad 3.697 | 2.39s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.63 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64800 | loss 5.6615 | text 2.0103 audio 5.2594 | grad 1.694 | 1.70s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.01 G=0.70 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64825 | loss 5.6548 | text 2.0157 audio 5.2517 | grad 2.555 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.13 | spike L=+0.00 G=1.09 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64850 | loss 5.6379 | text 2.0290 audio 5.2321 | grad 2.165 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.00 G=0.92 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64875 | loss 5.6061 | text 2.0162 audio 5.2029 | grad 2.290 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.01 G=0.98 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64900 | loss 5.5823 | text 2.0024 audio 5.1818 | grad 2.953 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=-0.01 G=1.26 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64925 | loss 5.6392 | text 1.9982 audio 5.2395 | grad 2.024 | 1.61s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.84 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64950 | loss 5.6649 | text 2.0426 audio 5.2564 | grad 3.466 | 1.67s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.20 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.11 | spike L=+0.01 G=1.47 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  64975 | loss 5.6204 | text 2.0196 audio 5.2164 | grad 2.397 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=-0.00 G=0.97 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65000 | loss 5.6121 | text 2.0097 audio 5.2102 | grad 1.893 | 1.60s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.14 | spike L=-0.00 G=0.77 | nonfinite=0
  [audio-demo] ar_cb0_acc=4.0% (39s)
  running validation at step 65000...

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

[repeated 2 x]
  ...es/step_065000/source.wav: 100%|██████████|  230kB /  230kB            

[repeated 2 x]
  ...es/step_065000/source.wav: 100%|██████████|  230kB /  230kB            
Processing Files (1 / 1)      : 100%|██████████|  230kB /  230kB,   ???B/s  

New Data Upload               : |          |  0.00B /  0.00B,   ???B/s  

  ...es/step_065000/source.wav: 100%|██████████|  230kB /  230kB            

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

[repeated 2 x]
  ...step_065000/target_gt.wav: 100%|██████████|  192kB /  192kB            

[repeated 2 x]
  ...step_065000/target_gt.wav: 100%|██████████|  192kB /  192kB            
Processing Files (1 / 1)      : 100%|██████████|  192kB /  192kB,   ???B/s  

New Data Upload               : |          |  0.00B /  0.00B,   ???B/s  

  ...step_065000/target_gt.wav: 100%|██████████|  192kB /  192kB            

Processing Files (0 / 0)      : |          |  0.00B /  0.00B            

New Data Upload               : |          |  0.00B /  0.00B            

[repeated 2 x]
  ...step_065000/generated.wav: 100%|██████████|  192kB /  192kB            

[repeated 2 x]
  ...step_065000/generated.wav: 100%|██████████|  192kB /  192kB            
Processing Files (1 / 1)      : 100%|██████████|  192kB /  192kB, 18.6kB/s  

New Data Upload               : 100%|██████████|  192kB /  192kB, 18.6kB/s  
Processing Files (1 / 1)      : 100%|██████████|  192kB /  192kB, 18.6kB/s  

New Data Upload               : 100%|██████████|  192kB /  192kB, 18.6kB/s  

  ...step_065000/generated.wav: 100%|██████████|  192kB /  192kB            
  pushed 3 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/samples/step_065000
  val/composite=2.9076 (text=0.4849 audio=4.5228) val/loss=4.6197 cb0_acc=40.5% text_acc=94.4%
  [val] per-codebook acc: cb0=40.5% cb1=20.9% cb2=17.4% cb3=11.4% cb4=8.7% cb5=7.1% cb6=6.1% cb7=5.9%
  [early-stop] no val improvement (>0.0); patience 1/10 (best 2.9048)
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_fs0ok6c0/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 14 x]
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065000
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 3 x]
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5cbfe9-352774fb1ead57df50f7955e;50c6709b-34d5-44d9-b26b-42c4eff95359)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 8 x]
step  65025 | loss 5.6225 | text 2.0151 audio 5.2195 | grad 1.950 | 22.47s/step | peak -1.0G | host_rss 57.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=-0.00 G=0.81 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65050 | loss 5.6364 | text 2.0332 audio 5.2298 | grad 1.442 | 1.57s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.61 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65075 | loss 5.6068 | text 2.0149 audio 5.2038 | grad 1.483 | 1.58s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.06 projection=0.09 text_embed=0.15 | spike L=-0.00 G=0.65 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65100 | loss 5.6431 | text 2.0142 audio 5.2402 | grad 2.730 | 1.65s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.25 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65125 | loss 5.6260 | text 2.0278 audio 5.2204 | grad 4.438 | 1.58s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.05 projection=0.04 text_embed=0.10 | spike L=-0.00 G=1.98 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65150 | loss 5.6308 | text 2.0263 audio 5.2255 | grad 4.187 | 1.58s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.11 | spike L=+0.00 G=1.70 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65175 | loss 5.6043 | text 2.0204 audio 5.2002 | grad 1.576 | 1.58s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.60 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65200 | loss 5.6304 | text 2.0002 audio 5.2304 | grad 1.903 | 1.58s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.75 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65225 | loss 5.6597 | text 2.0097 audio 5.2577 | grad 2.554 | 1.65s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.01 G=1.03 | nonfinite=0
sys:1: UserWarning: Full backward hook is firing when gradients are computed with respect to module outputs since no inputs require gradients. See https://docs.pytorch.org/docs/main/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook for more details.
[repeated 25 x]
step  65250 | loss 5.6217 | text 1.9978 audio 5.2221 | grad 3.196 | 1.60s/step | peak -1.0G | host_rss 57.3G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.06 projection=0.05 text_embed=0.14 | spike L=-0.00 G=1.29 | nonfinite=0
  running validation at step 65250...
  val/composite=2.9084 (text=0.4818 audio=4.5262) val/loss=4.6226 cb0_acc=40.4% text_acc=94.5%
  [val] per-codebook acc: cb0=40.4% cb1=20.9% cb2=17.4% cb3=11.5% cb4=8.7% cb5=7.2% cb6=6.1% cb7=5.9%
  [early-stop] no val improvement (>0.0); STOPPING (best 2.9048)
[ckpt] draining 163 background upload(s)...
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/ckpt_soxtngk7/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250
[ckpt] WARNING: hub push failed: (Request ID: Root=1-6a5cc355-5cc7dc296c1f46ed2864bfd6;eba70668-4506-42e9-8b41-f2c6562a1aa7)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.

[patch 19] entering canonical final save -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250_final
[patch 19] save_dir is GCS (gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250_final); staging to /tmp/canonical_final_no72ww1p
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/utils/save_and_load.py:356: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.
  warnings.warn("Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.")
[ckpt] uploading /tmp/canonical_final_no72ww1p/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250_final
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250_final
[patch 19] canonical final save complete

Training complete: 65250 steps in 3473.9 min
wandb: updating run metadata
wandb: uploading output_rank_0.log; uploading wandb-summary.json; uploading config.yaml
wandb: uploading output_rank_0.log
wandb: uploading summary, console lines 74336-74339
wandb: 
wandb: Run history:
wandb:                    audio/ar_cb0_acc ▁▁▅▁▁▃█▁█▁▃▁▁
wandb:                      audio/demo_sec █▁▁▂▂▂▂▂▄▄▄▄▃
wandb:             diag/adam_v_drift/depth ██▇▄▄▄▄▃▄▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:              diag/adam_v_drift/lora █▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: diag/adam_v_drift/model_audio_embed █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        diag/adam_v_drift/projection ██▄▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        diag/adam_v_drift/text_embed █▆▄▅▅▄▄▄▄▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:              diag/adam_v_mean/depth ▁▂▃▃▃▄▄▄▄▅▅▅▅▅▅▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████████
wandb:               diag/adam_v_mean/lora ▁▂▂▂▃▃▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇█████
wandb:  diag/adam_v_mean/model_audio_embed ▁▁▂▂▂▃▃▃▃▃▃▃▃▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇▇▇██████
wandb:                                +150 ...
wandb: 
wandb: Run summary:
wandb:                    audio/ar_cb0_acc 0.04
wandb:                      audio/demo_sec 38.75885
wandb:             diag/adam_v_drift/depth 1.00128
wandb:              diag/adam_v_drift/lora 1.00215
wandb: diag/adam_v_drift/model_audio_embed 1.00214
wandb:        diag/adam_v_drift/projection 1.00073
wandb:        diag/adam_v_drift/text_embed 1.00274
wandb:              diag/adam_v_mean/depth 0.0
wandb:               diag/adam_v_mean/lora 0.0
wandb:  diag/adam_v_mean/model_audio_embed 0.0
wandb:                                +150 ...
wandb: 
wandb: 🚀 View run v0.3-long-horizon-mh-r2 at: https://wandb.ai/cataluna84/tinyaya-stage2-tpu/runs/xzcb60bl
wandb: ⭐️ View project at: https://wandb.ai/cataluna84/tinyaya-stage2-tpu
wandb: Synced 5 W&B file(s), 39 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20260717_023948-xzcb60bl/logs
[2026-07-19T12:39:24+00:00] training exited with status 0
[2026-07-20T05:34:22+00:00] launching train_hierarchical.py [strategy=fsdpv2_lora]
   Building tinyaya-stage2 @ file:///opt/tinyaya
      Built tinyaya-stage2 @ file:///opt/tinyaya
Uninstalled 1 package in 0.47ms
Installed 1 package in 0.77ms
[bf16-mask-patch] AttentionMaskConverter patched (clamp >= -1e4; SDPA mask elision disabled; cohere2=True)
[tpu_backend] SPMD initialized: 16 devices, mesh_shape=(16,), axis=fsdp
[tpu_backend] WARNING: HBM telemetry unavailable; tpu-info returned no parseable hbm_usage output: 'TPU HBM Usage                    \n                                 \n| Device | HBM Usage (GiB)      |\n|--------|----------------------|\n| 2      | 0.00 GiB / 31'. mem/* metrics will be -1.
[tpu_backend][post-init] global=16 local=4 strategy=None hbm_used=-1.00GB/limit=-1.00GB peak=-1.00GB
[profiler] xp.start_server listening on :9012 (server obj retained: <_XLAC.profiler.ProfilerServer object at 0x6ffc4ec6b7f0>)
[batch] per_chip_batch=2 -> per_host loader batch 8 x 4 hosts = global 32 (before grad_accum)
[batch] val_per_chip_batch=8 -> per_host val loader batch 32 x 4 hosts = global 128

=== Effective config ===
{
  "backend": "tpu",
  "data": {
    "train_split": "/mnt/data/splits/train.jsonl",
    "val_split": "/mnt/data/splits/val.jsonl",
    "encoded_dir": "/mnt/data/encoded",
    "max_frames": 300,
    "bucket_frames": null,
    "audio_frame_rate": 12.5,
    "num_workers": 32,
    "pin_memory": false
  },
  "train": {
    "num_codebooks": 8,
    "batch_size": 8,
    "grad_accum": 1,
    "max_steps": 76250,
    "warmup_steps": 1100,
    "min_lr_ratio": 0.0,
    "depth_chunk_size": 100,
    "precision": "bfloat16",
    "max_grad_norm": 1.0,
    "seed": 42,
    "scheduler_total_steps": null,
    "per_chip_batch": 2,
    "schedule": "wsd",
    "wsd_anneal_steps": 11000,
    "wsd_anneal_frac": 0.1,
    "weight_decay": 0.01,
    "adam_beta1": 0.9,
    "adam_beta2": 0.999,
    "adam_eps": 1e-08,
    "allow_fresh_optimizer": false,
    "use_scan_layers": true,
    "xla_grad_checkpoint": true,
    "compile_warmup_steps": 1,
    "debug_input_sharding": false,
    "early_stop_patience": 0,
    "enable_clip_grad_norm": true,
    "clip_grad_norm": 1.0,
    "log_grad_norm": true,
    "fsdp_barrier_hook": false,
    "micro_mark_step": true,
    "final_canonical_save": true
  },
  "loss": {
    "text_weight": 0.2,
    "audio_weight": 1.0,
    "text_weight_curriculum_frac": 0.0,
    "composite_text_w": 0.4,
    "composite_audio_w": 0.6,
    "text_padding_weight": 0.01,
    "zero_padding_weight": 0.0,
    "label_smoothing": 0.1,
    "per_codebook_multipliers": [
      1.0,
[repeated 2 x]
      1.5,
[repeated 2 x]
      2.0,
[repeated 3 x]
      2.0
    ],
    "progressive_unmask_fraction": 0.1,
    "unmask_k0": 1
  },
  "optim": {
    "lr_lora": 0.0001716,
    "lr_full_ft": 5e-05,
    "lr_projection": 0.0005,
    "lr_depth": 0.0001,
    "lr_audio_embed": 0.0005,
    "lr_text_embed": 0.0005,
    "lr_depth_blocks": 1e-05
  },
  "logging": {
    "log_every": 25,
    "save_every": 1000,
    "audio_every": 5000,
    "audio_ar_frames": 50,
    "val_every": 250,
    "val_max_batches": 25,
    "val_on_tpu": true,
    "val_per_chip_batch": 8,
    "tpu_telemetry_every": 250,
    "save_dir": "gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2",
    "log_spaced_saves": true,
    "save_at_steps": null,
    "async_checkpoint_upload": true,
    "wandb_project": "tinyaya-stage2-tpu",
    "wandb_run_name": "v0.3-long-horizon-mh-r2",
    "use_wandb": true,
    "wandb_tags": [
      "v0.3",
      "multi-host",
      "wsd"
    ],
    "push_to_hub": true,
    "hub_repo_id": "tiny-aya-translate/tr-hi-s2st-v0.3",
    "hub_private": true,
    "diag_metrics": true,
    "keep_last_n": 0,
    "keep_local_checkpoints": false
  },
  "perf": {
    "enabled": true,
    "warmup_skip_steps": 25,
    "xprof_trace_labels": false
  },
  "lora": {
    "r": 32,
    "alpha": 64,
    "dropout": 0.05,
    "use_rslora": true,
    "target_modules": [
      "q_proj",
      "k_proj",
      "v_proj",
      "o_proj",
      "gate_proj",
      "up_proj",
      "down_proj",
      "embed_tokens"
    ],
    "lora_exclude_top": 0,
    "num_full_ft_layers": 0,
    "depth_unfreeze_blocks": 0
  }
}

=== Building composite model ===
============================================================
Step 1: Loading TinyAya backbone
============================================================

============================================================
Step 2: Creating projection layer
============================================================
Projection: Linear(2048, 4096, bias=False)

============================================================
Step 3: Extracting depth decoder from Moshiko
============================================================
Loading Moshiko for depth decoder extraction...
/opt/tinyaya/.venv/lib/python3.12/site-packages/transformers/models/mimi/modeling_mimi.py:164: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.register_buffer("padding_total", torch.tensor(kernel_size - stride, dtype=torch.int64), persistent=False)
Extracted 58 tensors from depth decoder
  input_projections: (8, 1024, 4096)
  lm_heads: (8, 2048, 1024)
Creating MoshiDepthDecoder: 8 codebooks, 6 layers
[depth_decoder] patched MoshiFlexibleLinear.forward -> equal-batch bmm (avoids XLA materialising the per-token weight broadcast)
  input_projections: (8, 1024, 4096)
  lm_heads: (8, 2048, 1024)
  embed_tokens: 7 audio embeddings
  Total params: 714M
[composite] scan homogeneity: forced full-attention on 27 sliding layers + eager attn on 36 (seq<=max_frames<<sliding_window, numerically identical)
[scan_utils] patched Cohere2Model.layers (n_layers=36 scan=True ckpt=True)
[scan_utils] patched MoshiDepthDecoder.layers (n_layers=6 scan=False ckpt=True)
/opt/tinyaya/.venv/lib/python3.12/site-packages/peft/tuners/tuners_utils.py:1348: UserWarning: Model has `tie_word_embeddings=True` and a tied layer is part of the adapter, but `ensure_weight_tying` is not set to True. This can lead to complications, for example when merging the adapter or converting your model to formats other than safetensors. Check the discussion here: https://github.com/huggingface/peft/issues/2777
  warnings.warn(msg)
[lora] r=32 alpha=64 rslora=True (alpha/sqrt(r)=11.314) targets=['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'embed_tokens'] lora_layers=0..35 (exclude_top=0) num_full_ft_layers=0 (+0.0M full-FT)
trainable params: 68,976,768 || all params: 3,422,406,784 || trainable%: 2.0154
Depth decoder: frozen 617M, trainable I/O 98M
[ckpt] downloading gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250/* -> /tmp/ckpt_load_t53q9w7y
Restored host RNG state from gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250
Loaded model weights from gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250 (step 65250); will resume W&B run xzcb60bl
[tpu_backend] cast model to bfloat16
[tpu_backend] wrap_model strategy=fsdpv2_lora
WARNING:absl:Tensorflow library not found, tensorflow.io.gfile operations will use native shim calls. GCS paths (i.e. 'gs://...') cannot be accessed.
[tpu_backend][post-wrap] global=16 local=4 strategy=fsdpv2_lora hbm_used=-1.00GB/limit=-1.00GB peak=-1.00GB
Total 5.24B, trainable 192M (3.7%)

=== Datasets ===
StreamingTranslationDataset: 1178302 rows from /mnt/data/splits/train.jsonl
StreamingTranslationDataset: alignment coverage 100.0% (1000/1000 sampled rows resolve both src+tgt)
StreamingTranslationDataset: 62036 rows from /mnt/data/splits/val.jsonl
StreamingTranslationDataset: alignment coverage 100.0% (1000/1000 sampled rows resolve both src+tgt)
[data] train_rows=1178302 train_batches=36822 drop_last=False pad_to=300 batch_pad_to=8

=== Loading Mimi for audio monitoring ===
/opt/tinyaya/.venv/lib/python3.12/site-packages/transformers/models/mimi/modeling_mimi.py:164: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.detach().clone() or sourceTensor.detach().clone().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.register_buffer("padding_total", torch.tensor(kernel_size - stride, dtype=torch.int64), persistent=False)

=== Parameter Groups ===
  lora: 507 tensors, 73.2M, lr=0.0001716
  projection: 1 tensors, 8.4M, lr=0.0005
  depth: 10 tensors, 97.8M, lr=0.0001
  text_embed: 2 tensors, 8.5M, lr=0.0005
  model_audio_embed: 1 tensors, 4.2M, lr=0.0005
Restored optimizer state from gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_065250
Resuming training from step 65250
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
wandb: Currently logged in as: cataluna84 to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: WARNING The `shared` mode feature is experimental and may change. Please contact support@wandb.com for guidance and to report any issues.
wandb: Tracking run with wandb version 0.27.2
wandb: Run data is saved locally in /opt/tinyaya/wandb/run-20260720_053620-xzcb60bl
wandb: Run `wandb offline` to turn off syncing.
wandb: Resuming run v0.3-long-horizon-mh-r2
wandb: ⭐️ View project at https://wandb.ai/cataluna84/tinyaya-stage2-tpu
wandb: 🚀 View run at https://wandb.ai/cataluna84/tinyaya-stage2-tpu/runs/xzcb60bl
[wandb] primary published run_id=xzcb60bl to gs://tinyaya-stage2-eu/wandb-rendezvous/tinyaya-v6e16-eu-1784525646.id
[hub] write access OK: tiny-aya-translate/tr-hi-s2st-v0.3 (private=True)
[bf16-mask-patch] AttentionMaskConverter patched (clamp >= -1e4; SDPA mask elision disabled; cohere2=True)
[repeated 32 x]
[resume] restored best_val=2.9048, patience_left=0

=== Training: 76250 steps, accum=1, batch=8, global_batch=32, clip=1.0 ===
[compile-warmup] skipped because resume start_step=65250
[scan_utils] patched FakeTensorMode default allow_non_fake_inputs=True
[scan_utils] swapped 252 nn.Dropout -> _ScanSafeDropout (bf16 mask; avoids scan's PRED/BF16 dynamic-update-slice mismatch)
step  65275 | loss 5.6198 | text 2.0089 audio 5.2180 | grad 1.994 | 18.94s/step | peak -1.0G | host_rss 37.5G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.00 | nonfinite=0
step  65300 | loss 5.5812 | text 1.9852 audio 5.1842 | grad 1.688 | 7.93s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.41 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.15 | spike L=-0.01 G=0.85 | nonfinite=0
step  65325 | loss 5.5998 | text 2.0151 audio 5.1968 | grad 2.595 | 1.39s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.32 | nonfinite=0
step  65350 | loss 5.5968 | text 1.9973 audio 5.1973 | grad 2.837 | 1.38s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.40 | nonfinite=0
step  65375 | loss 5.6255 | text 1.9842 audio 5.2286 | grad 2.739 | 1.40s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.30 | nonfinite=0
step  65400 | loss 5.6033 | text 1.9960 audio 5.2041 | grad 2.100 | 1.40s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.14 | spike L=-0.00 G=0.97 | nonfinite=0
step  65425 | loss 5.5730 | text 1.9810 audio 5.1768 | grad 1.706 | 1.38s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.14 | spike L=-0.01 G=0.79 | nonfinite=0
step  65450 | loss 5.6092 | text 2.0121 audio 5.2068 | grad 1.482 | 1.40s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.48 lora=0.85 model_audio_embed=0.05 projection=0.09 text_embed=0.15 | spike L=+0.00 G=0.70 | nonfinite=0
step  65475 | loss 5.6511 | text 2.0125 audio 5.2486 | grad 2.154 | 1.38s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.08 | spike L=+0.01 G=1.05 | nonfinite=0
step  65500 | loss 5.6100 | text 1.9929 audio 5.2115 | grad 3.505 | 1.40s/step | peak -1.0G | host_rss 34.9G
  [diag] gradnorm depth=0.21 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.10 | spike L=-0.00 G=1.70 | nonfinite=0
  running validation at step 65500...
[bf16-mask-patch] AttentionMaskConverter patched (clamp >= -1e4; SDPA mask elision disabled; cohere2=True)
[repeated 32 x]
  val/composite=2.9041 (text=0.4792 audio=4.5207) val/loss=4.6165 cb0_acc=40.4% text_acc=94.5%
  [val] per-codebook acc: cb0=40.4% cb1=20.9% cb2=17.3% cb3=11.6% cb4=8.7% cb5=7.1% cb6=6.0% cb7=5.8%
[ckpt] uploading /tmp/ckpt_k83sfte5/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  65525 | loss 5.5956 | text 2.0105 audio 5.1935 | grad 2.493 | 38.68s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.29 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.15 | spike L=-0.00 G=1.13 | nonfinite=0
step  65550 | loss 5.5883 | text 1.9826 audio 5.1917 | grad 5.111 | 1.41s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.13 lora=0.99 model_audio_embed=0.05 projection=0.03 text_embed=0.07 | spike L=-0.00 G=2.28 | nonfinite=0
step  65575 | loss 5.6011 | text 1.9926 audio 5.2026 | grad 2.093 | 1.39s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=-0.00 G=0.83 | nonfinite=0
step  65600 | loss 5.5925 | text 1.9793 audio 5.1966 | grad 1.593 | 1.40s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.64 | nonfinite=0
step  65625 | loss 5.6346 | text 1.9971 audio 5.2352 | grad 2.402 | 1.40s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.12 | spike L=+0.01 G=1.00 | nonfinite=0
step  65650 | loss 5.6065 | text 1.9982 audio 5.2069 | grad 1.881 | 1.41s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.79 | nonfinite=0
step  65675 | loss 5.6313 | text 2.0102 audio 5.2293 | grad 2.502 | 1.39s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.07 | nonfinite=0
step  65700 | loss 5.6073 | text 1.9959 audio 5.2081 | grad 2.658 | 1.41s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.27 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.13 | nonfinite=0
step  65725 | loss 5.5857 | text 2.0001 audio 5.1857 | grad 2.072 | 1.40s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.32 lora=0.93 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=-0.00 G=0.87 | nonfinite=0
step  65750 | loss 5.6292 | text 1.9936 audio 5.2305 | grad 2.394 | 1.41s/step | peak -1.0G | host_rss 37.4G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=+0.00 G=1.02 | nonfinite=0
  running validation at step 65750...
  val/composite=2.9006 (text=0.4812 audio=4.5135) val/loss=4.6097 cb0_acc=40.5% text_acc=94.7%
  [val] per-codebook acc: cb0=40.5% cb1=21.1% cb2=17.4% cb3=11.5% cb4=8.8% cb5=7.1% cb6=6.1% cb7=5.9%
[ckpt] uploading /tmp/ckpt_s8_0klk3/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  65775 | loss 5.6121 | text 2.0241 audio 5.2073 | grad 3.197 | 24.61s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=+0.00 G=1.35 | nonfinite=0
step  65800 | loss 5.5944 | text 2.0111 audio 5.1922 | grad 2.940 | 1.41s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.26 lora=0.95 model_audio_embed=0.06 projection=0.05 text_embed=0.13 | spike L=-0.00 G=1.20 | nonfinite=0
step  65825 | loss 5.6120 | text 1.9868 audio 5.2146 | grad 2.904 | 1.39s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=+0.00 G=1.16 | nonfinite=0
step  65850 | loss 5.5904 | text 1.9885 audio 5.1927 | grad 2.039 | 1.39s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=-0.00 G=0.80 | nonfinite=0
step  65875 | loss 5.6130 | text 2.0015 audio 5.2127 | grad 2.287 | 1.41s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.00 G=0.92 | nonfinite=0
step  65900 | loss 5.5932 | text 1.9910 audio 5.1950 | grad 2.590 | 1.39s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.29 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.16 | spike L=-0.00 G=1.05 | nonfinite=0
step  65925 | loss 5.6218 | text 1.9913 audio 5.2235 | grad 1.843 | 1.39s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.74 | nonfinite=0
step  65950 | loss 5.6039 | text 1.9842 audio 5.2071 | grad 2.937 | 1.41s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.22 | nonfinite=0
step  65975 | loss 5.6089 | text 2.0085 audio 5.2072 | grad 4.064 | 1.40s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.04 projection=0.04 text_embed=0.10 | spike L=+0.00 G=1.65 | nonfinite=0
step  66000 | loss 5.6447 | text 2.0096 audio 5.2428 | grad 2.735 | 1.41s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=+0.01 G=1.04 | nonfinite=0
  running validation at step 66000...
  val/composite=2.8922 (text=0.4681 audio=4.5082) val/loss=4.6018 cb0_acc=40.6% text_acc=94.7%
  [val] per-codebook acc: cb0=40.6% cb1=21.1% cb2=17.5% cb3=11.7% cb4=8.8% cb5=7.2% cb6=6.1% cb7=5.9%
[ckpt] uploading /tmp/ckpt_wtgid79o/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_pjcpqp42/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_066000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_066000
step  66025 | loss 5.6040 | text 1.9891 audio 5.2062 | grad 2.083 | 45.43s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=0.79 | nonfinite=0
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@step-66000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  66050 | loss 5.6047 | text 2.0068 audio 5.2033 | grad 1.919 | 1.41s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.15 | spike L=-0.00 G=0.74 | nonfinite=0
step  66075 | loss 5.5979 | text 1.9877 audio 5.2003 | grad 1.712 | 1.42s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.68 | nonfinite=0
step  66100 | loss 5.5788 | text 1.9810 audio 5.1826 | grad 2.081 | 1.40s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.34 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.20 | spike L=-0.01 G=0.85 | nonfinite=0
step  66125 | loss 5.5875 | text 1.9827 audio 5.1909 | grad 1.706 | 1.40s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.71 | nonfinite=0
step  66150 | loss 5.6084 | text 1.9830 audio 5.2118 | grad 3.390 | 1.41s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.08 | spike L=+0.00 G=1.45 | nonfinite=0
step  66175 | loss 5.6160 | text 1.9984 audio 5.2163 | grad 2.170 | 1.40s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.00 G=0.89 | nonfinite=0
step  66200 | loss 5.6025 | text 1.9847 audio 5.2056 | grad 2.762 | 1.40s/step | peak -1.0G | host_rss 37.1G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.15 | nonfinite=0
step  66225 | loss 5.6191 | text 2.0091 audio 5.2173 | grad 1.607 | 1.40s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.66 | nonfinite=0
step  66250 | loss 5.6127 | text 2.0033 audio 5.2120 | grad 2.109 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=+0.00 G=0.89 | nonfinite=0
  running validation at step 66250...
  val/composite=2.8932 (text=0.4655 audio=4.5117) val/loss=4.6048 cb0_acc=40.5% text_acc=94.9%
  [val] per-codebook acc: cb0=40.5% cb1=21.0% cb2=17.5% cb3=11.5% cb4=8.8% cb5=7.2% cb6=6.2% cb7=5.8%
step  66275 | loss 5.5842 | text 1.9902 audio 5.1861 | grad 3.584 | 2.09s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.21 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.09 | spike L=-0.00 G=1.53 | nonfinite=0
step  66300 | loss 5.6287 | text 1.9977 audio 5.2292 | grad 4.336 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.18 lora=0.98 model_audio_embed=0.05 projection=0.04 text_embed=0.08 | spike L=+0.00 G=1.76 | nonfinite=0
step  66325 | loss 5.6175 | text 1.9886 audio 5.2198 | grad 2.209 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.83 | nonfinite=0
step  66350 | loss 5.5798 | text 1.9891 audio 5.1819 | grad 1.942 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.01 G=0.75 | nonfinite=0
step  66375 | loss 5.6031 | text 1.9856 audio 5.2060 | grad 2.052 | 1.40s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.00 G=0.81 | nonfinite=0
step  66400 | loss 5.5937 | text 1.9729 audio 5.1991 | grad 1.670 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.51 lora=0.85 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.67 | nonfinite=0
step  66425 | loss 5.6075 | text 1.9726 audio 5.2130 | grad 1.618 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.67 | nonfinite=0
step  66450 | loss 5.6123 | text 1.9925 audio 5.2137 | grad 2.001 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.86 | nonfinite=0
step  66475 | loss 5.6372 | text 2.0062 audio 5.2359 | grad 1.572 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.43 lora=0.88 model_audio_embed=0.05 projection=0.08 text_embed=0.16 | spike L=+0.01 G=0.68 | nonfinite=0
step  66500 | loss 5.6192 | text 2.0075 audio 5.2177 | grad 1.661 | 1.43s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.75 | nonfinite=0
  running validation at step 66500...
  val/composite=2.8941 (text=0.4692 audio=4.5106) val/loss=4.6044 cb0_acc=40.5% text_acc=94.9%
  [val] per-codebook acc: cb0=40.5% cb1=20.9% cb2=17.5% cb3=11.5% cb4=8.7% cb5=7.2% cb6=6.0% cb7=5.9%
step  66525 | loss 5.6193 | text 2.0069 audio 5.2179 | grad 1.537 | 2.12s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.00 G=0.71 | nonfinite=0
step  66550 | loss 5.6227 | text 1.9988 audio 5.2229 | grad 1.976 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=+0.00 G=0.94 | nonfinite=0
step  66575 | loss 5.5758 | text 2.0011 audio 5.1756 | grad 1.633 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.01 G=0.78 | nonfinite=0
step  66600 | loss 5.6022 | text 1.9774 audio 5.2067 | grad 3.235 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.58 | nonfinite=0
step  66625 | loss 5.6159 | text 2.0028 audio 5.2154 | grad 2.164 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=1.00 | nonfinite=0
step  66650 | loss 5.5666 | text 1.9715 audio 5.1723 | grad 1.928 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.01 G=0.89 | nonfinite=0
step  66675 | loss 5.6198 | text 1.9821 audio 5.2234 | grad 2.411 | 1.41s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.13 | nonfinite=0
step  66700 | loss 5.5936 | text 1.9960 audio 5.1944 | grad 1.889 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.87 | nonfinite=0
step  66725 | loss 5.6368 | text 2.0001 audio 5.2367 | grad 3.237 | 1.43s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.01 G=1.51 | nonfinite=0
step  66750 | loss 5.5821 | text 1.9661 audio 5.1889 | grad 1.851 | 1.42s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.82 | nonfinite=0
  running validation at step 66750...
  val/composite=2.8882 (text=0.4663 audio=4.5029) val/loss=4.5961 cb0_acc=40.6% text_acc=95.0%
  [val] per-codebook acc: cb0=40.6% cb1=21.1% cb2=17.6% cb3=11.6% cb4=8.8% cb5=7.2% cb6=6.1% cb7=5.9%
[ckpt] uploading /tmp/ckpt_bxr9k2gw/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  66775 | loss 5.5991 | text 1.9619 audio 5.2068 | grad 1.723 | 24.76s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.78 | nonfinite=0
step  66800 | loss 5.6201 | text 1.9788 audio 5.2243 | grad 1.546 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.14 | spike L=+0.00 G=0.72 | nonfinite=0
step  66825 | loss 5.5988 | text 1.9751 audio 5.2037 | grad 2.193 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=-0.00 G=1.04 | nonfinite=0
step  66850 | loss 5.5797 | text 1.9622 audio 5.1873 | grad 1.685 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.80 | nonfinite=0
step  66875 | loss 5.5642 | text 1.9715 audio 5.1699 | grad 1.640 | 1.40s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.15 | spike L=-0.01 G=0.79 | nonfinite=0
step  66900 | loss 5.5683 | text 1.9644 audio 5.1754 | grad 2.637 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.26 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.14 | spike L=-0.01 G=1.30 | nonfinite=0
step  66925 | loss 5.5962 | text 1.9746 audio 5.2013 | grad 1.691 | 1.42s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.81 | nonfinite=0
step  66950 | loss 5.5905 | text 1.9757 audio 5.1954 | grad 1.488 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.73 | nonfinite=0
step  66975 | loss 5.6114 | text 1.9805 audio 5.2153 | grad 2.223 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=1.12 | nonfinite=0
step  67000 | loss 5.6308 | text 1.9980 audio 5.2312 | grad 2.708 | 1.41s/step | peak -1.0G | host_rss 36.9G
  [diag] gradnorm depth=0.26 lora=0.95 model_audio_embed=0.06 projection=0.05 text_embed=0.16 | spike L=+0.01 G=1.35 | nonfinite=0
  running validation at step 67000...
  val/composite=2.8843 (text=0.4567 audio=4.5027) val/loss=4.5941 cb0_acc=40.7% text_acc=95.3%
  [val] per-codebook acc: cb0=40.7% cb1=21.2% cb2=17.6% cb3=11.6% cb4=8.8% cb5=7.2% cb6=6.1% cb7=5.9%
[ckpt] uploading /tmp/ckpt_moullb1a/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_zgulbfi0/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_067000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_067000
step  67025 | loss 5.5685 | text 1.9933 audio 5.1699 | grad 1.948 | 45.53s/step | peak -1.0G | host_rss 36.7G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.01 G=0.94 | nonfinite=0
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@step-67000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  67050 | loss 5.5916 | text 1.9938 audio 5.1928 | grad 2.499 | 1.42s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.21 | nonfinite=0
step  67075 | loss 5.5886 | text 1.9776 audio 5.1931 | grad 1.401 | 1.43s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.13 | spike L=-0.00 G=0.66 | nonfinite=0
step  67100 | loss 5.6061 | text 1.9872 audio 5.2087 | grad 3.218 | 1.41s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.21 lora=0.97 model_audio_embed=0.06 projection=0.04 text_embed=0.11 | spike L=+0.00 G=1.58 | nonfinite=0
step  67125 | loss 5.5904 | text 1.9891 audio 5.1926 | grad 4.449 | 1.41s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.10 | spike L=-0.00 G=2.06 | nonfinite=0
step  67150 | loss 5.6054 | text 1.9695 audio 5.2115 | grad 2.171 | 1.41s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.91 | nonfinite=0
step  67175 | loss 5.5992 | text 1.9794 audio 5.2033 | grad 1.765 | 1.41s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.39 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.15 | spike L=+0.00 G=0.75 | nonfinite=0
step  67200 | loss 5.5636 | text 1.9872 audio 5.1661 | grad 2.703 | 1.40s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=-0.01 G=1.17 | nonfinite=0
step  67225 | loss 5.5965 | text 1.9752 audio 5.2014 | grad 1.595 | 1.40s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.00 G=0.68 | nonfinite=0
step  67250 | loss 5.5787 | text 1.9837 audio 5.1819 | grad 2.527 | 1.40s/step | peak -1.0G | host_rss 36.6G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=-0.00 G=1.11 | nonfinite=0
  running validation at step 67250...
  val/composite=2.8827 (text=0.4580 audio=4.4992) val/loss=4.5908 cb0_acc=40.6% text_acc=95.2%
  [val] per-codebook acc: cb0=40.6% cb1=21.2% cb2=17.6% cb3=11.6% cb4=8.9% cb5=7.2% cb6=6.1% cb7=5.9%
[ckpt] uploading /tmp/ckpt_t0w4ftow/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  67275 | loss 5.5889 | text 1.9865 audio 5.1916 | grad 1.713 | 24.87s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.06 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.75 | nonfinite=0
step  67300 | loss 5.5931 | text 1.9784 audio 5.1974 | grad 1.629 | 1.41s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.73 | nonfinite=0
step  67325 | loss 5.5857 | text 1.9841 audio 5.1889 | grad 1.745 | 1.42s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.04 projection=0.09 text_embed=0.09 | spike L=-0.00 G=0.80 | nonfinite=0
step  67350 | loss 5.5914 | text 1.9655 audio 5.1983 | grad 4.662 | 1.42s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.06 projection=0.03 text_embed=0.10 | spike L=-0.00 G=2.18 | nonfinite=0
step  67375 | loss 5.5848 | text 1.9670 audio 5.1914 | grad 1.638 | 1.41s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.06 projection=0.09 text_embed=0.14 | spike L=-0.00 G=0.69 | nonfinite=0
step  67400 | loss 5.5551 | text 1.9610 audio 5.1629 | grad 1.548 | 1.41s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.47 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.14 | spike L=-0.01 G=0.67 | nonfinite=0
step  67425 | loss 5.5869 | text 1.9666 audio 5.1936 | grad 2.134 | 1.41s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=-0.00 G=0.95 | nonfinite=0
step  67450 | loss 5.5351 | text 1.9742 audio 5.1403 | grad 2.155 | 1.41s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.01 G=0.97 | nonfinite=0
step  67475 | loss 5.6100 | text 1.9796 audio 5.2141 | grad 2.159 | 1.41s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.97 | nonfinite=0
step  67500 | loss 5.5608 | text 1.9646 audio 5.1679 | grad 1.540 | 1.43s/step | peak -1.0G | host_rss 36.5G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.70 | nonfinite=0
  running validation at step 67500...
  val/composite=2.8769 (text=0.4513 audio=4.4940) val/loss=4.5842 cb0_acc=40.6% text_acc=95.3%
  [val] per-codebook acc: cb0=40.6% cb1=21.3% cb2=17.6% cb3=11.7% cb4=8.9% cb5=7.2% cb6=6.1% cb7=6.0%
[ckpt] uploading /tmp/ckpt__jrc1fr2/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  67525 | loss 5.5978 | text 1.9703 audio 5.2037 | grad 1.646 | 25.11s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.77 | nonfinite=0
step  67550 | loss 5.5980 | text 1.9719 audio 5.2036 | grad 3.219 | 1.42s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.54 | nonfinite=0
step  67575 | loss 5.5652 | text 1.9624 audio 5.1727 | grad 1.530 | 1.42s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.69 | nonfinite=0
step  67600 | loss 5.5749 | text 1.9641 audio 5.1821 | grad 1.570 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.43 lora=0.88 model_audio_embed=0.06 projection=0.09 text_embed=0.15 | spike L=-0.00 G=0.73 | nonfinite=0
step  67625 | loss 5.5986 | text 1.9919 audio 5.2002 | grad 2.478 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.19 | nonfinite=0
step  67650 | loss 5.5614 | text 1.9656 audio 5.1682 | grad 4.225 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.06 projection=0.04 text_embed=0.12 | spike L=-0.00 G=1.99 | nonfinite=0
step  67675 | loss 5.5900 | text 1.9769 audio 5.1946 | grad 1.932 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.06 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.83 | nonfinite=0
step  67700 | loss 5.5970 | text 1.9689 audio 5.2032 | grad 2.364 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.03 | nonfinite=0
step  67725 | loss 5.5932 | text 1.9822 audio 5.1967 | grad 2.196 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.08 | spike L=+0.00 G=0.95 | nonfinite=0
step  67750 | loss 5.6015 | text 1.9867 audio 5.2041 | grad 1.779 | 1.41s/step | peak -1.0G | host_rss 36.4G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.14 | spike L=+0.00 G=0.78 | nonfinite=0
  running validation at step 67750...
  val/composite=2.8791 (text=0.4491 audio=4.4991) val/loss=4.5890 cb0_acc=40.8% text_acc=95.3%
  [val] per-codebook acc: cb0=40.8% cb1=21.3% cb2=17.6% cb3=11.7% cb4=8.9% cb5=7.2% cb6=6.1% cb7=5.9%
step  67775 | loss 5.6057 | text 1.9879 audio 5.2081 | grad 2.979 | 2.13s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.06 projection=0.05 text_embed=0.14 | spike L=+0.00 G=1.33 | nonfinite=0
step  67800 | loss 5.5914 | text 1.9907 audio 5.1932 | grad 1.420 | 1.42s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.61 | nonfinite=0
step  67825 | loss 5.6059 | text 1.9754 audio 5.2108 | grad 1.461 | 1.42s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.66 | nonfinite=0
step  67850 | loss 5.5992 | text 1.9719 audio 5.2048 | grad 2.984 | 1.41s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.26 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.14 | spike L=+0.00 G=1.39 | nonfinite=0
step  67875 | loss 5.5756 | text 1.9749 audio 5.1806 | grad 2.187 | 1.42s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=0.98 | nonfinite=0
step  67900 | loss 5.5828 | text 1.9757 audio 5.1877 | grad 2.471 | 1.43s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.11 | nonfinite=0
step  67925 | loss 5.5936 | text 1.9714 audio 5.1993 | grad 2.033 | 1.43s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.07 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.90 | nonfinite=0
step  67950 | loss 5.5637 | text 1.9618 audio 5.1714 | grad 1.829 | 1.42s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.06 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.82 | nonfinite=0
step  67975 | loss 5.5776 | text 1.9714 audio 5.1833 | grad 2.135 | 1.42s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.98 | nonfinite=0
step  68000 | loss 5.5694 | text 1.9471 audio 5.1799 | grad 1.476 | 1.42s/step | peak -1.0G | host_rss 36.3G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.68 | nonfinite=0
  running validation at step 68000...
  val/composite=2.8751 (text=0.4504 audio=4.4915) val/loss=4.5816 cb0_acc=40.8% text_acc=95.5%
  [val] per-codebook acc: cb0=40.8% cb1=21.3% cb2=17.7% cb3=11.7% cb4=8.9% cb5=7.3% cb6=6.1% cb7=6.0%
[ckpt] uploading /tmp/ckpt_r9ot59p2/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_nrnzr9np/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_068000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_068000
step  68025 | loss 5.5978 | text 1.9717 audio 5.2035 | grad 1.699 | 45.53s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.80 | nonfinite=0
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@step-68000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  68050 | loss 5.5598 | text 1.9628 audio 5.1672 | grad 1.467 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.71 | nonfinite=0
step  68075 | loss 5.5725 | text 1.9767 audio 5.1772 | grad 1.583 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.79 | nonfinite=0
step  68100 | loss 5.5916 | text 1.9577 audio 5.2001 | grad 1.472 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.75 | nonfinite=0
step  68125 | loss 5.6037 | text 1.9793 audio 5.2079 | grad 1.644 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.06 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.86 | nonfinite=0
step  68150 | loss 5.5666 | text 1.9573 audio 5.1752 | grad 2.778 | 1.43s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.47 | nonfinite=0
step  68175 | loss 5.5867 | text 1.9621 audio 5.1943 | grad 2.054 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=+0.00 G=1.04 | nonfinite=0
step  68200 | loss 5.5563 | text 1.9453 audio 5.1672 | grad 1.736 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.87 | nonfinite=0
step  68225 | loss 5.5794 | text 1.9579 audio 5.1878 | grad 1.456 | 1.41s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.74 | nonfinite=0
step  68250 | loss 5.5578 | text 1.9724 audio 5.1633 | grad 4.124 | 1.42s/step | peak -1.0G | host_rss 36.1G
  [diag] gradnorm depth=0.19 lora=0.97 model_audio_embed=0.06 projection=0.04 text_embed=0.14 | spike L=-0.00 G=2.16 | nonfinite=0
  running validation at step 68250...
  val/composite=2.8691 (text=0.4406 audio=4.4882) val/loss=4.5763 cb0_acc=40.9% text_acc=95.6%
  [val] per-codebook acc: cb0=40.9% cb1=21.4% cb2=17.7% cb3=11.7% cb4=8.9% cb5=7.2% cb6=6.1% cb7=6.0%
[ckpt] uploading /tmp/ckpt_h1vsd9vt/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  68275 | loss 5.5883 | text 1.9789 audio 5.1925 | grad 2.748 | 24.87s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.29 | nonfinite=0
step  68300 | loss 5.5944 | text 1.9666 audio 5.2011 | grad 1.488 | 1.41s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.00 G=0.68 | nonfinite=0
step  68325 | loss 5.5766 | text 1.9627 audio 5.1841 | grad 1.504 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.71 | nonfinite=0
step  68350 | loss 5.5825 | text 1.9728 audio 5.1879 | grad 1.586 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.77 | nonfinite=0
step  68375 | loss 5.5812 | text 1.9744 audio 5.1864 | grad 2.121 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=1.05 | nonfinite=0
step  68400 | loss 5.5765 | text 1.9519 audio 5.1862 | grad 1.721 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.04 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.85 | nonfinite=0
step  68425 | loss 5.5520 | text 1.9705 audio 5.1579 | grad 1.505 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.01 G=0.75 | nonfinite=0
step  68450 | loss 5.5784 | text 1.9660 audio 5.1852 | grad 3.388 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.21 lora=0.96 model_audio_embed=0.06 projection=0.04 text_embed=0.15 | spike L=+0.00 G=1.74 | nonfinite=0
step  68475 | loss 5.5835 | text 1.9757 audio 5.1884 | grad 3.389 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.13 | spike L=+0.00 G=1.62 | nonfinite=0
step  68500 | loss 5.5917 | text 1.9696 audio 5.1978 | grad 1.763 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.79 | nonfinite=0
  running validation at step 68500...
  val/composite=2.8662 (text=0.4334 audio=4.4880) val/loss=4.5747 cb0_acc=40.9% text_acc=95.6%
  [val] per-codebook acc: cb0=40.9% cb1=21.4% cb2=17.7% cb3=11.7% cb4=8.9% cb5=7.3% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_tz2n9nmo/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  68525 | loss 5.5927 | text 1.9834 audio 5.1960 | grad 1.919 | 24.90s/step | peak -1.0G | host_rss 35.9G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.88 | nonfinite=0
step  68550 | loss 5.5603 | text 1.9565 audio 5.1690 | grad 2.283 | 1.42s/step | peak -1.0G | host_rss 35.9G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=1.06 | nonfinite=0
step  68575 | loss 5.5807 | text 1.9565 audio 5.1894 | grad 2.500 | 1.42s/step | peak -1.0G | host_rss 35.9G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.05 text_embed=0.09 | spike L=+0.00 G=1.16 | nonfinite=0
step  68600 | loss 5.5720 | text 1.9381 audio 5.1844 | grad 2.349 | 1.42s/step | peak -1.0G | host_rss 35.9G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.07 | nonfinite=0
step  68625 | loss 5.5790 | text 1.9621 audio 5.1866 | grad 1.878 | 1.40s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.85 | nonfinite=0
step  68650 | loss 5.5620 | text 1.9565 audio 5.1707 | grad 1.988 | 1.44s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=-0.00 G=0.91 | nonfinite=0
step  68675 | loss 5.5471 | text 1.9455 audio 5.1580 | grad 1.990 | 1.42s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.37 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.14 | spike L=-0.01 G=0.92 | nonfinite=0
step  68700 | loss 5.5852 | text 1.9568 audio 5.1939 | grad 3.365 | 1.42s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.08 | spike L=+0.00 G=1.57 | nonfinite=0
step  68725 | loss 5.5576 | text 1.9542 audio 5.1667 | grad 1.420 | 1.40s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=-0.00 G=0.63 | nonfinite=0
step  68750 | loss 5.5925 | text 1.9716 audio 5.1982 | grad 3.301 | 1.42s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.12 | spike L=+0.00 G=1.51 | nonfinite=0
  running validation at step 68750...
  val/composite=2.8637 (text=0.4348 audio=4.4830) val/loss=4.5700 cb0_acc=40.9% text_acc=95.7%
  [val] per-codebook acc: cb0=40.9% cb1=21.5% cb2=17.8% cb3=11.7% cb4=9.0% cb5=7.3% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_tw9fexdt/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  68775 | loss 5.5876 | text 1.9760 audio 5.1924 | grad 2.087 | 25.17s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.91 | nonfinite=0
step  68800 | loss 5.5572 | text 1.9567 audio 5.1659 | grad 2.593 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.14 | nonfinite=0
step  68825 | loss 5.5697 | text 1.9648 audio 5.1768 | grad 2.518 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.09 | nonfinite=0
step  68850 | loss 5.5665 | text 1.9494 audio 5.1766 | grad 3.140 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.35 | nonfinite=0
step  68875 | loss 5.5522 | text 1.9523 audio 5.1618 | grad 1.946 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.06 projection=0.07 text_embed=0.13 | spike L=-0.00 G=0.81 | nonfinite=0
step  68900 | loss 5.5747 | text 1.9575 audio 5.1832 | grad 2.350 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=1.00 | nonfinite=0
step  68925 | loss 5.5843 | text 1.9653 audio 5.1913 | grad 1.908 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.81 | nonfinite=0
step  68950 | loss 5.5580 | text 1.9686 audio 5.1643 | grad 1.504 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.04 projection=0.10 text_embed=0.12 | spike L=-0.00 G=0.65 | nonfinite=0
step  68975 | loss 5.5641 | text 1.9642 audio 5.1713 | grad 4.390 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=-0.00 G=1.97 | nonfinite=0
step  69000 | loss 5.5575 | text 1.9563 audio 5.1663 | grad 1.649 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.67 | nonfinite=0
  running validation at step 69000...
  val/composite=2.8637 (text=0.4328 audio=4.4842) val/loss=4.5708 cb0_acc=40.9% text_acc=95.8%
  [val] per-codebook acc: cb0=40.9% cb1=21.4% cb2=17.7% cb3=11.8% cb4=9.0% cb5=7.3% cb6=6.1% cb7=6.0%
[ckpt] uploading /tmp/ckpt_gwwcksk_/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_w4lhfx_p/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_069000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_069000
step  69025 | loss 5.5376 | text 1.9513 audio 5.1474 | grad 1.590 | 45.54s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=-0.01 G=0.67 | nonfinite=0
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@step-69000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  69050 | loss 5.5730 | text 1.9458 audio 5.1839 | grad 1.575 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.69 | nonfinite=0
step  69075 | loss 5.5789 | text 1.9473 audio 5.1894 | grad 2.419 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.09 | nonfinite=0
step  69100 | loss 5.6176 | text 1.9659 audio 5.2244 | grad 1.934 | 1.40s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.01 G=0.86 | nonfinite=0
step  69125 | loss 5.5591 | text 1.9621 audio 5.1667 | grad 1.479 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.67 | nonfinite=0
step  69150 | loss 5.6073 | text 1.9582 audio 5.2156 | grad 1.960 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.01 G=0.92 | nonfinite=0
step  69175 | loss 5.5923 | text 1.9623 audio 5.1998 | grad 1.402 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=+0.00 G=0.66 | nonfinite=0
step  69200 | loss 5.5798 | text 1.9541 audio 5.1890 | grad 2.824 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=+0.00 G=1.38 | nonfinite=0
step  69225 | loss 5.5763 | text 1.9604 audio 5.1842 | grad 2.301 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.08 | nonfinite=0
step  69250 | loss 5.5665 | text 1.9626 audio 5.1739 | grad 1.598 | 1.40s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.75 | nonfinite=0
  running validation at step 69250...
  val/composite=2.8575 (text=0.4291 audio=4.4764) val/loss=4.5622 cb0_acc=41.0% text_acc=95.8%
  [val] per-codebook acc: cb0=41.0% cb1=21.6% cb2=17.8% cb3=11.8% cb4=9.0% cb5=7.3% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_4jtc8b3r/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  69275 | loss 5.5686 | text 1.9529 audio 5.1780 | grad 1.387 | 24.90s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.53 lora=0.83 model_audio_embed=0.05 projection=0.11 text_embed=0.10 | spike L=-0.00 G=0.66 | nonfinite=0
step  69300 | loss 5.5509 | text 1.9557 audio 5.1597 | grad 1.821 | 1.43s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.37 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.90 | nonfinite=0
step  69325 | loss 5.5567 | text 1.9607 audio 5.1646 | grad 2.062 | 1.40s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=1.03 | nonfinite=0
step  69350 | loss 5.5513 | text 1.9472 audio 5.1618 | grad 2.678 | 1.43s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.04 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.34 | nonfinite=0
step  69375 | loss 5.5619 | text 1.9427 audio 5.1733 | grad 3.237 | 1.43s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.56 | nonfinite=0
step  69400 | loss 5.5394 | text 1.9433 audio 5.1507 | grad 1.870 | 1.43s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.01 G=0.85 | nonfinite=0
step  69425 | loss 5.5266 | text 1.9378 audio 5.1391 | grad 2.077 | 1.41s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.01 G=0.96 | nonfinite=0
step  69450 | loss 5.5543 | text 1.9639 audio 5.1615 | grad 1.784 | 1.43s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.06 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.83 | nonfinite=0
step  69475 | loss 5.5566 | text 1.9467 audio 5.1673 | grad 1.482 | 1.43s/step | peak -1.0G | host_rss 35.5G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.70 | nonfinite=0
step  69500 | loss 5.5367 | text 1.9558 audio 5.1455 | grad 1.609 | 1.43s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.79 | nonfinite=0
  running validation at step 69500...
  val/composite=2.8574 (text=0.4272 audio=4.4775) val/loss=4.5629 cb0_acc=41.1% text_acc=95.9%
  [val] per-codebook acc: cb0=41.1% cb1=21.6% cb2=17.8% cb3=11.8% cb4=9.0% cb5=7.3% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_0i5pu_td/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub.hf_api:No files have been modified since last commit. Skipping to prevent empty commit.
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3@best
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  69525 | loss 5.5657 | text 1.9581 audio 5.1741 | grad 5.631 | 25.03s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.15 lora=0.98 model_audio_embed=0.04 projection=0.03 text_embed=0.10 | spike L=+0.00 G=2.81 | nonfinite=0
step  69550 | loss 5.5630 | text 1.9416 audio 5.1746 | grad 1.826 | 1.42s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.14 | spike L=+0.00 G=0.77 | nonfinite=0
step  69575 | loss 5.5518 | text 1.9486 audio 5.1621 | grad 2.890 | 1.43s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.25 | nonfinite=0
step  69600 | loss 5.5580 | text 1.9389 audio 5.1703 | grad 2.642 | 1.41s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.11 | nonfinite=0
step  69625 | loss 5.5662 | text 1.9460 audio 5.1770 | grad 2.259 | 1.43s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=0.94 | nonfinite=0
step  69650 | loss 5.5472 | text 1.9617 audio 5.1549 | grad 1.707 | 1.43s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.72 | nonfinite=0
step  69675 | loss 5.5624 | text 1.9483 audio 5.1728 | grad 1.892 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.82 | nonfinite=0
step  69700 | loss 5.5519 | text 1.9502 audio 5.1618 | grad 1.474 | 1.41s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.51 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.13 | spike L=-0.00 G=0.65 | nonfinite=0
step  69725 | loss 5.5608 | text 1.9450 audio 5.1718 | grad 2.037 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.37 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.00 G=0.93 | nonfinite=0
step  69750 | loss 5.5568 | text 1.9493 audio 5.1669 | grad 1.808 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.83 | nonfinite=0
  running validation at step 69750...
  val/composite=2.8546 (text=0.4292 audio=4.4715) val/loss=4.5573 cb0_acc=41.2% text_acc=95.9%
  [val] per-codebook acc: cb0=41.2% cb1=21.5% cb2=17.9% cb3=11.9% cb4=9.0% cb5=7.3% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_fq28c_ej/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] hub publishing DISABLED for this run: private storage limit reached; weights keep saving to GCS -- backfill post-run with publish_checkpoint_suite.py
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  69775 | loss 5.5485 | text 1.9510 audio 5.1583 | grad 2.488 | 23.82s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.16 | nonfinite=0
step  69800 | loss 5.5485 | text 1.9350 audio 5.1615 | grad 1.854 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.39 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.15 | spike L=-0.00 G=0.85 | nonfinite=0
step  69825 | loss 5.5572 | text 1.9313 audio 5.1710 | grad 1.490 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.15 | spike L=+0.00 G=0.70 | nonfinite=0
step  69850 | loss 5.5875 | text 1.9596 audio 5.1956 | grad 2.259 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=+0.01 G=1.09 | nonfinite=0
step  69875 | loss 5.5752 | text 1.9494 audio 5.1853 | grad 1.646 | 1.41s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.79 | nonfinite=0
step  69900 | loss 5.5611 | text 1.9562 audio 5.1699 | grad 2.816 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.27 lora=0.96 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.37 | nonfinite=0
step  69925 | loss 5.5706 | text 1.9495 audio 5.1807 | grad 1.479 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.70 | nonfinite=0
step  69950 | loss 5.5740 | text 1.9569 audio 5.1826 | grad 1.471 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.71 | nonfinite=0
step  69975 | loss 5.6016 | text 1.9607 audio 5.2094 | grad 1.523 | 1.41s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.47 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.01 G=0.76 | nonfinite=0
step  70000 | loss 5.5462 | text 1.9381 audio 5.1586 | grad 3.983 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.19 lora=0.98 model_audio_embed=0.04 projection=0.04 text_embed=0.09 | spike L=-0.00 G=2.04 | nonfinite=0
  [audio-demo] ar_cb0_acc=22.0% (51s)
  running validation at step 70000...
[ckpt] WARNING: async upload failed: (Request ID: Root=1-6a5dfdca-0de1e9db3fa6f03c2604cee8;5eaf654c-09f3-423d-9222-c1cb1186dd71)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  val/composite=2.8510 (text=0.4207 audio=4.4711) val/loss=4.5552 cb0_acc=41.2% text_acc=96.1%
  [val] per-codebook acc: cb0=41.2% cb1=21.5% cb2=17.8% cb3=11.8% cb4=9.0% cb5=7.4% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_02l5_jdv/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_qw4y7a80/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_070000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_070000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  70025 | loss 5.5718 | text 1.9698 audio 5.1779 | grad 2.772 | 46.63s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.28 | nonfinite=0
step  70050 | loss 5.5779 | text 1.9552 audio 5.1868 | grad 1.587 | 1.44s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.45 lora=0.87 model_audio_embed=0.06 projection=0.09 text_embed=0.15 | spike L=+0.00 G=0.71 | nonfinite=0
step  70075 | loss 5.5776 | text 1.9539 audio 5.1868 | grad 1.668 | 1.44s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.77 | nonfinite=0
step  70100 | loss 5.5338 | text 1.9491 audio 5.1440 | grad 3.637 | 1.44s/step | peak -1.0G | host_rss 35.4G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=-0.01 G=1.73 | nonfinite=0
step  70125 | loss 5.5830 | text 1.9546 audio 5.1921 | grad 1.598 | 1.42s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.71 | nonfinite=0
step  70150 | loss 5.5779 | text 1.9438 audio 5.1892 | grad 3.554 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.10 | spike L=+0.00 G=1.62 | nonfinite=0
step  70175 | loss 5.5573 | text 1.9612 audio 5.1650 | grad 1.957 | 1.42s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.84 | nonfinite=0
step  70200 | loss 5.5650 | text 1.9623 audio 5.1725 | grad 2.416 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.05 | nonfinite=0
step  70225 | loss 5.5815 | text 1.9627 audio 5.1889 | grad 1.522 | 1.43s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.66 | nonfinite=0
step  70250 | loss 5.5676 | text 1.9574 audio 5.1761 | grad 1.997 | 1.41s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.90 | nonfinite=0
  running validation at step 70250...
  val/composite=2.8485 (text=0.4179 audio=4.4689) val/loss=4.5525 cb0_acc=41.2% text_acc=96.2%
  [val] per-codebook acc: cb0=41.2% cb1=21.6% cb2=17.9% cb3=11.8% cb4=9.0% cb5=7.4% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_cibfcxzi/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  70275 | loss 5.5883 | text 1.9560 audio 5.1971 | grad 1.861 | 23.90s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.84 | nonfinite=0
step  70300 | loss 5.5431 | text 1.9459 audio 5.1540 | grad 1.512 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.70 | nonfinite=0
step  70325 | loss 5.5756 | text 1.9623 audio 5.1832 | grad 1.953 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.06 projection=0.07 text_embed=0.14 | spike L=+0.00 G=0.93 | nonfinite=0
step  70350 | loss 5.5853 | text 1.9499 audio 5.1953 | grad 1.732 | 1.42s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.83 | nonfinite=0
step  70375 | loss 5.5634 | text 1.9269 audio 5.1781 | grad 2.108 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=1.03 | nonfinite=0
step  70400 | loss 5.5471 | text 1.9431 audio 5.1585 | grad 1.370 | 1.42s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.53 lora=0.83 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.67 | nonfinite=0
step  70425 | loss 5.5494 | text 1.9411 audio 5.1611 | grad 1.564 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.79 | nonfinite=0
step  70450 | loss 5.5601 | text 1.9280 audio 5.1745 | grad 2.578 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.32 | nonfinite=0
step  70475 | loss 5.5235 | text 1.9429 audio 5.1349 | grad 1.972 | 1.42s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.01 G=0.98 | nonfinite=0
step  70500 | loss 5.5434 | text 1.9274 audio 5.1580 | grad 1.381 | 1.44s/step | peak -1.0G | host_rss 35.3G
  [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.12 | spike L=-0.00 G=0.69 | nonfinite=0
  running validation at step 70500...
  val/composite=2.8459 (text=0.4160 audio=4.4658) val/loss=4.5490 cb0_acc=41.2% text_acc=96.2%
  [val] per-codebook acc: cb0=41.2% cb1=21.6% cb2=17.9% cb3=11.9% cb4=9.1% cb5=7.4% cb6=6.2% cb7=6.1%
[ckpt] uploading /tmp/ckpt_1hz63srk/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  70525 | loss 5.5314 | text 1.9393 audio 5.1436 | grad 2.689 | 23.85s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=1.38 | nonfinite=0
step  70550 | loss 5.5951 | text 1.9619 audio 5.2028 | grad 2.403 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.11 | spike L=+0.01 G=1.19 | nonfinite=0
step  70575 | loss 5.5693 | text 1.9456 audio 5.1802 | grad 4.453 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.09 | spike L=+0.00 G=2.16 | nonfinite=0
step  70600 | loss 5.5598 | text 1.9520 audio 5.1694 | grad 1.652 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.72 | nonfinite=0
step  70625 | loss 5.5541 | text 1.9471 audio 5.1646 | grad 1.856 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.83 | nonfinite=0
step  70650 | loss 5.5804 | text 1.9393 audio 5.1925 | grad 1.457 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.66 | nonfinite=0
step  70675 | loss 5.5930 | text 1.9572 audio 5.2016 | grad 1.633 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.01 G=0.77 | nonfinite=0
step  70700 | loss 5.5351 | text 1.9408 audio 5.1469 | grad 1.860 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.01 G=0.90 | nonfinite=0
step  70725 | loss 5.5885 | text 1.9463 audio 5.1992 | grad 2.156 | 1.43s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=1.05 | nonfinite=0
step  70750 | loss 5.5500 | text 1.9479 audio 5.1605 | grad 1.854 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.90 | nonfinite=0
  running validation at step 70750...
  val/composite=2.8433 (text=0.4162 audio=4.4614) val/loss=4.5446 cb0_acc=41.2% text_acc=96.2%
  [val] per-codebook acc: cb0=41.2% cb1=21.6% cb2=18.0% cb3=11.9% cb4=9.1% cb5=7.3% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_upg1ebz4/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  70775 | loss 5.5456 | text 1.9386 audio 5.1579 | grad 4.672 | 23.67s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.17 lora=0.98 model_audio_embed=0.03 projection=0.03 text_embed=0.08 | spike L=-0.00 G=2.29 | nonfinite=0
step  70800 | loss 5.5486 | text 1.9443 audio 5.1598 | grad 1.872 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.39 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.81 | nonfinite=0
step  70825 | loss 5.5604 | text 1.9462 audio 5.1712 | grad 2.243 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.99 | nonfinite=0
step  70850 | loss 5.5683 | text 1.9427 audio 5.1797 | grad 1.585 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.70 | nonfinite=0
step  70875 | loss 5.5616 | text 1.9407 audio 5.1735 | grad 1.725 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.79 | nonfinite=0
step  70900 | loss 5.5541 | text 1.9424 audio 5.1656 | grad 2.096 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.12 | spike L=-0.00 G=0.98 | nonfinite=0
step  70925 | loss 5.5338 | text 1.9403 audio 5.1457 | grad 1.762 | 1.44s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.82 | nonfinite=0
step  70950 | loss 5.5628 | text 1.9417 audio 5.1745 | grad 1.707 | 1.42s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.81 | nonfinite=0
step  70975 | loss 5.5335 | text 1.9418 audio 5.1451 | grad 1.654 | 1.43s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.80 | nonfinite=0
step  71000 | loss 5.5601 | text 1.9440 audio 5.1713 | grad 1.432 | 1.41s/step | peak -1.0G | host_rss 35.2G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.71 | nonfinite=0
  running validation at step 71000...
  val/composite=2.8428 (text=0.4125 audio=4.4630) val/loss=4.5455 cb0_acc=41.2% text_acc=96.3%
  [val] per-codebook acc: cb0=41.2% cb1=21.7% cb2=18.0% cb3=11.9% cb4=9.1% cb5=7.4% cb6=6.3% cb7=6.0%
[ckpt] uploading /tmp/ckpt_nw1pwxxa/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_0p2x3qc3/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_071000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_071000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  71025 | loss 5.5647 | text 1.9477 audio 5.1751 | grad 2.150 | 44.67s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=+0.00 G=1.10 | nonfinite=0
step  71050 | loss 5.5630 | text 1.9342 audio 5.1761 | grad 2.067 | 1.44s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.37 lora=0.91 model_audio_embed=0.06 projection=0.07 text_embed=0.14 | spike L=+0.00 G=1.04 | nonfinite=0
step  71075 | loss 5.5595 | text 1.9417 audio 5.1712 | grad 3.075 | 1.42s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.00 G=1.55 | nonfinite=0
step  71100 | loss 5.5817 | text 1.9518 audio 5.1913 | grad 2.073 | 1.44s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.99 | nonfinite=0
step  71125 | loss 5.5689 | text 1.9352 audio 5.1819 | grad 1.817 | 1.42s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.16 | spike L=+0.00 G=0.87 | nonfinite=0
step  71150 | loss 5.5289 | text 1.9407 audio 5.1408 | grad 2.442 | 1.44s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.01 G=1.18 | nonfinite=0
step  71175 | loss 5.5698 | text 1.9400 audio 5.1818 | grad 1.964 | 1.41s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.14 | spike L=+0.00 G=0.93 | nonfinite=0
step  71200 | loss 5.5707 | text 1.9433 audio 5.1821 | grad 2.231 | 1.44s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=+0.00 G=1.07 | nonfinite=0
step  71225 | loss 5.5457 | text 1.9459 audio 5.1566 | grad 1.649 | 1.42s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.78 | nonfinite=0
step  71250 | loss 5.5370 | text 1.9395 audio 5.1491 | grad 1.640 | 1.44s/step | peak -1.0G | host_rss 35.8G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.09 text_embed=0.14 | spike L=-0.00 G=0.80 | nonfinite=0
  running validation at step 71250...
  val/composite=2.8383 (text=0.4099 audio=4.4572) val/loss=4.5392 cb0_acc=41.3% text_acc=96.4%
  [val] per-codebook acc: cb0=41.3% cb1=21.7% cb2=18.0% cb3=11.9% cb4=9.1% cb5=7.4% cb6=6.2% cb7=6.0%
[ckpt] uploading /tmp/ckpt_t5f0gnj7/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  71275 | loss 5.5678 | text 1.9445 audio 5.1788 | grad 1.685 | 23.73s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.00 G=0.84 | nonfinite=0
step  71300 | loss 5.5282 | text 1.9252 audio 5.1432 | grad 2.187 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.36 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.18 | spike L=-0.01 G=1.10 | nonfinite=0
step  71325 | loss 5.5467 | text 1.9298 audio 5.1607 | grad 2.385 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.12 | spike L=-0.00 G=1.19 | nonfinite=0
step  71350 | loss 5.5448 | text 1.9352 audio 5.1577 | grad 1.827 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.89 | nonfinite=0
step  71375 | loss 5.5977 | text 1.9543 audio 5.2068 | grad 2.233 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.01 G=1.10 | nonfinite=0
step  71400 | loss 5.5448 | text 1.9410 audio 5.1566 | grad 3.042 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.49 | nonfinite=0
step  71425 | loss 5.5331 | text 1.9240 audio 5.1483 | grad 2.610 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.03 projection=0.06 text_embed=0.08 | spike L=-0.00 G=1.22 | nonfinite=0
step  71450 | loss 5.5419 | text 1.9331 audio 5.1552 | grad 3.054 | 1.45s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.40 | nonfinite=0
step  71475 | loss 5.5501 | text 1.9240 audio 5.1653 | grad 2.648 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.16 | nonfinite=0
step  71500 | loss 5.5520 | text 1.9347 audio 5.1650 | grad 1.908 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.06 projection=0.08 text_embed=0.13 | spike L=-0.00 G=0.83 | nonfinite=0
  running validation at step 71500...
  val/composite=2.8382 (text=0.4130 audio=4.4550) val/loss=4.5376 cb0_acc=41.3% text_acc=96.4%
  [val] per-codebook acc: cb0=41.3% cb1=21.8% cb2=18.1% cb3=12.0% cb4=9.1% cb5=7.4% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_xrai_qb6/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  71525 | loss 5.5593 | text 1.9613 audio 5.1670 | grad 1.684 | 23.70s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.74 | nonfinite=0
step  71550 | loss 5.5397 | text 1.9274 audio 5.1542 | grad 1.533 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.69 | nonfinite=0
step  71575 | loss 5.5556 | text 1.9307 audio 5.1695 | grad 1.689 | 1.45s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.79 | nonfinite=0
step  71600 | loss 5.5653 | text 1.9386 audio 5.1775 | grad 1.626 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.77 | nonfinite=0
step  71625 | loss 5.5215 | text 1.9267 audio 5.1361 | grad 1.835 | 1.45s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.12 | spike L=-0.01 G=0.89 | nonfinite=0
step  71650 | loss 5.5593 | text 1.9409 audio 5.1711 | grad 4.816 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.16 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.10 | spike L=+0.00 G=2.37 | nonfinite=0
step  71675 | loss 5.5283 | text 1.9169 audio 5.1449 | grad 1.628 | 1.45s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.71 | nonfinite=0
step  71700 | loss 5.5801 | text 1.9304 audio 5.1940 | grad 1.549 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.11 | spike L=+0.01 G=0.69 | nonfinite=0
step  71725 | loss 5.5444 | text 1.9291 audio 5.1586 | grad 2.795 | 1.45s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.25 lora=0.95 model_audio_embed=0.06 projection=0.05 text_embed=0.16 | spike L=-0.00 G=1.29 | nonfinite=0
step  71750 | loss 5.5343 | text 1.9175 audio 5.1508 | grad 1.502 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.50 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=-0.00 G=0.67 | nonfinite=0
  running validation at step 71750...
  val/composite=2.8368 (text=0.4120 audio=4.4534) val/loss=4.5358 cb0_acc=41.3% text_acc=96.4%
  [val] per-codebook acc: cb0=41.3% cb1=21.8% cb2=18.0% cb3=12.0% cb4=9.1% cb5=7.4% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_j4mz98uc/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  71775 | loss 5.5560 | text 1.9399 audio 5.1680 | grad 3.653 | 23.80s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.20 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.11 | spike L=+0.00 G=1.69 | nonfinite=0
step  71800 | loss 5.5216 | text 1.9336 audio 5.1349 | grad 1.349 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.54 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.01 G=0.58 | nonfinite=0
step  71825 | loss 5.5248 | text 1.9247 audio 5.1399 | grad 1.339 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.53 lora=0.83 model_audio_embed=0.04 projection=0.10 text_embed=0.10 | spike L=-0.00 G=0.60 | nonfinite=0
step  71850 | loss 5.5308 | text 1.9299 audio 5.1448 | grad 2.133 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=1.00 | nonfinite=0
step  71875 | loss 5.5426 | text 1.9485 audio 5.1529 | grad 2.547 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.14 | spike L=-0.00 G=1.20 | nonfinite=0
step  71900 | loss 5.5196 | text 1.9278 audio 5.1340 | grad 1.562 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.72 | nonfinite=0
step  71925 | loss 5.5274 | text 1.9258 audio 5.1422 | grad 2.982 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.41 | nonfinite=0
step  71950 | loss 5.5389 | text 1.9439 audio 5.1502 | grad 2.114 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=-0.00 G=0.96 | nonfinite=0
step  71975 | loss 5.5317 | text 1.9279 audio 5.1461 | grad 1.599 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.73 | nonfinite=0
step  72000 | loss 5.5433 | text 1.9402 audio 5.1553 | grad 6.004 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.13 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.11 | spike L=+0.00 G=2.82 | nonfinite=0
  running validation at step 72000...
  val/composite=2.8324 (text=0.4037 audio=4.4516) val/loss=4.5323 cb0_acc=41.4% text_acc=96.4%
  [val] per-codebook acc: cb0=41.4% cb1=21.8% cb2=18.0% cb3=12.0% cb4=9.1% cb5=7.4% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_q_ubgzzq/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_0ecm1xu4/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_072000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_072000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  72025 | loss 5.5542 | text 1.9274 audio 5.1688 | grad 2.176 | 44.85s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.86 | nonfinite=0
step  72050 | loss 5.5576 | text 1.9273 audio 5.1721 | grad 1.472 | 1.42s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.14 | spike L=+0.00 G=0.59 | nonfinite=0
step  72075 | loss 5.5740 | text 1.9445 audio 5.1851 | grad 1.627 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.04 projection=0.09 text_embed=0.09 | spike L=+0.01 G=0.68 | nonfinite=0
step  72100 | loss 5.5656 | text 1.9261 audio 5.1803 | grad 1.550 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=+0.00 G=0.67 | nonfinite=0
step  72125 | loss 5.5281 | text 1.9342 audio 5.1413 | grad 1.717 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.77 | nonfinite=0
step  72150 | loss 5.5431 | text 1.9355 audio 5.1560 | grad 2.217 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.38 lora=0.90 model_audio_embed=0.07 projection=0.08 text_embed=0.16 | spike L=-0.00 G=1.02 | nonfinite=0
step  72175 | loss 5.5550 | text 1.9490 audio 5.1652 | grad 1.684 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.77 | nonfinite=0
step  72200 | loss 5.5602 | text 1.9149 audio 5.1772 | grad 1.635 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.77 | nonfinite=0
step  72225 | loss 5.4974 | text 1.9308 audio 5.1113 | grad 4.257 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.19 lora=0.97 model_audio_embed=0.05 projection=0.04 text_embed=0.11 | spike L=-0.01 G=2.04 | nonfinite=0
step  72250 | loss 5.5676 | text 1.9456 audio 5.1785 | grad 2.087 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.91 | nonfinite=0
  running validation at step 72250...
  val/composite=2.8326 (text=0.4070 audio=4.4497) val/loss=4.5311 cb0_acc=41.3% text_acc=96.4%
  [val] per-codebook acc: cb0=41.3% cb1=21.8% cb2=18.1% cb3=12.0% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
step  72275 | loss 5.5356 | text 1.9417 audio 5.1473 | grad 1.670 | 2.14s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.04 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.73 | nonfinite=0
step  72300 | loss 5.5141 | text 1.9423 audio 5.1257 | grad 3.002 | 1.47s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.01 G=1.35 | nonfinite=0
step  72325 | loss 5.5438 | text 1.9317 audio 5.1575 | grad 2.122 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.92 | nonfinite=0
step  72350 | loss 5.5406 | text 1.9364 audio 5.1534 | grad 1.714 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=-0.00 G=0.75 | nonfinite=0
step  72375 | loss 5.5403 | text 1.9446 audio 5.1513 | grad 1.574 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.71 | nonfinite=0
step  72400 | loss 5.5502 | text 1.9328 audio 5.1637 | grad 3.745 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.20 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.11 | spike L=+0.00 G=1.74 | nonfinite=0
step  72425 | loss 5.5456 | text 1.9273 audio 5.1601 | grad 1.384 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.51 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.60 | nonfinite=0
step  72450 | loss 5.5486 | text 1.9412 audio 5.1604 | grad 2.043 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=0.92 | nonfinite=0
step  72475 | loss 5.5183 | text 1.9360 audio 5.1311 | grad 1.363 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.54 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.00 G=0.62 | nonfinite=0
step  72500 | loss 5.5519 | text 1.9220 audio 5.1674 | grad 1.549 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.73 | nonfinite=0
  running validation at step 72500...
  val/composite=2.8316 (text=0.4062 audio=4.4485) val/loss=4.5297 cb0_acc=41.3% text_acc=96.5%
  [val] per-codebook acc: cb0=41.3% cb1=21.9% cb2=18.1% cb3=12.0% cb4=9.2% cb5=7.4% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_iz0_8dvf/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  72525 | loss 5.5617 | text 1.9329 audio 5.1752 | grad 1.511 | 23.82s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.13 | spike L=+0.00 G=0.73 | nonfinite=0
step  72550 | loss 5.5205 | text 1.9309 audio 5.1343 | grad 1.781 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=-0.00 G=0.89 | nonfinite=0
step  72575 | loss 5.5464 | text 1.9330 audio 5.1598 | grad 2.124 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=1.07 | nonfinite=0
step  72600 | loss 5.5477 | text 1.9366 audio 5.1604 | grad 2.823 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.07 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.41 | nonfinite=0
step  72625 | loss 5.5173 | text 1.9518 audio 5.1269 | grad 1.621 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=-0.00 G=0.78 | nonfinite=0
step  72650 | loss 5.5127 | text 1.9295 audio 5.1268 | grad 1.384 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.00 G=0.68 | nonfinite=0
step  72675 | loss 5.5265 | text 1.9232 audio 5.1419 | grad 2.243 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=-0.00 G=1.14 | nonfinite=0
step  72700 | loss 5.5092 | text 1.9326 audio 5.1227 | grad 2.713 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.28 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.36 | nonfinite=0
step  72725 | loss 5.5453 | text 1.9389 audio 5.1575 | grad 1.465 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.00 G=0.71 | nonfinite=0
step  72750 | loss 5.5515 | text 1.9512 audio 5.1613 | grad 2.076 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=1.03 | nonfinite=0
  running validation at step 72750...
  val/composite=2.8288 (text=0.4020 audio=4.4467) val/loss=4.5271 cb0_acc=41.4% text_acc=96.5%
  [val] per-codebook acc: cb0=41.4% cb1=21.9% cb2=18.2% cb3=12.0% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_eo2ctkd5/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  72775 | loss 5.5452 | text 1.9364 audio 5.1579 | grad 2.066 | 23.81s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.09 | spike L=+0.00 G=1.02 | nonfinite=0
step  72800 | loss 5.5051 | text 1.9252 audio 5.1201 | grad 3.669 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.20 lora=0.97 model_audio_embed=0.04 projection=0.04 text_embed=0.10 | spike L=-0.01 G=1.82 | nonfinite=0
step  72825 | loss 5.5381 | text 1.9208 audio 5.1539 | grad 1.783 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.82 | nonfinite=0
step  72850 | loss 5.5157 | text 1.9382 audio 5.1281 | grad 3.228 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.50 | nonfinite=0
step  72875 | loss 5.5416 | text 1.9174 audio 5.1581 | grad 1.664 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.44 lora=0.89 model_audio_embed=0.04 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.74 | nonfinite=0
step  72900 | loss 5.5286 | text 1.9342 audio 5.1417 | grad 2.595 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.30 lora=0.94 model_audio_embed=0.06 projection=0.06 text_embed=0.13 | spike L=-0.00 G=1.18 | nonfinite=0
step  72925 | loss 5.5590 | text 1.9392 audio 5.1711 | grad 1.949 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.87 | nonfinite=0
step  72950 | loss 5.5329 | text 1.9324 audio 5.1464 | grad 2.598 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.08 | spike L=-0.00 G=1.18 | nonfinite=0
step  72975 | loss 5.5504 | text 1.9357 audio 5.1633 | grad 2.230 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.99 | nonfinite=0
step  73000 | loss 5.5680 | text 1.9428 audio 5.1794 | grad 1.442 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=+0.01 G=0.64 | nonfinite=0
  running validation at step 73000...
  val/composite=2.8289 (text=0.4036 audio=4.4458) val/loss=4.5265 cb0_acc=41.4% text_acc=96.6%
  [val] per-codebook acc: cb0=41.4% cb1=21.9% cb2=18.2% cb3=12.0% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_k4heq9f5/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_073000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_073000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  73025 | loss 5.5528 | text 1.9422 audio 5.1643 | grad 1.785 | 23.10s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.82 | nonfinite=0
step  73050 | loss 5.5179 | text 1.9309 audio 5.1318 | grad 1.894 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.04 projection=0.09 text_embed=0.09 | spike L=-0.00 G=0.89 | nonfinite=0
step  73075 | loss 5.5153 | text 1.9396 audio 5.1274 | grad 2.073 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=-0.00 G=0.99 | nonfinite=0
step  73100 | loss 5.5060 | text 1.9336 audio 5.1193 | grad 1.588 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.01 G=0.76 | nonfinite=0
step  73125 | loss 5.5440 | text 1.9440 audio 5.1552 | grad 1.427 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.70 | nonfinite=0
step  73150 | loss 5.5519 | text 1.9363 audio 5.1646 | grad 2.849 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.06 projection=0.07 text_embed=0.13 | spike L=+0.00 G=1.43 | nonfinite=0
step  73175 | loss 5.5149 | text 1.9275 audio 5.1294 | grad 1.483 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.72 | nonfinite=0
step  73200 | loss 5.5287 | text 1.9382 audio 5.1411 | grad 3.456 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=-0.00 G=1.72 | nonfinite=0
  [ALERT] grad spike 3.2x (grad 6.95 vs EMA 2.16) step 73225
step  73225 | loss 5.5567 | text 1.9415 audio 5.1684 | grad 6.946 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.11 lora=0.98 model_audio_embed=0.06 projection=0.02 text_embed=0.12 | spike L=+0.00 G=3.22 | nonfinite=0
step  73250 | loss 5.4892 | text 1.9148 audio 5.1063 | grad 1.435 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.05 projection=0.11 text_embed=0.10 | spike L=-0.01 G=0.54 | nonfinite=0
  running validation at step 73250...
  val/composite=2.8271 (text=0.4035 audio=4.4428) val/loss=4.5235 cb0_acc=41.4% text_acc=96.6%
  [val] per-codebook acc: cb0=41.4% cb1=21.9% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_2nvinc1s/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  73275 | loss 5.5716 | text 1.9274 audio 5.1862 | grad 2.342 | 24.20s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.01 G=0.93 | nonfinite=0
step  73300 | loss 5.5336 | text 1.9240 audio 5.1488 | grad 1.551 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.62 | nonfinite=0
step  73325 | loss 5.5313 | text 1.9310 audio 5.1451 | grad 1.949 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.81 | nonfinite=0
step  73350 | loss 5.5332 | text 1.9225 audio 5.1487 | grad 1.485 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.53 lora=0.84 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.00 G=0.63 | nonfinite=0
step  73375 | loss 5.5315 | text 1.9466 audio 5.1421 | grad 1.664 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.73 | nonfinite=0
step  73400 | loss 5.5378 | text 1.9334 audio 5.1512 | grad 3.427 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.21 lora=0.96 model_audio_embed=0.06 projection=0.04 text_embed=0.14 | spike L=+0.00 G=1.55 | nonfinite=0
step  73425 | loss 5.5471 | text 1.9248 audio 5.1621 | grad 1.895 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=+0.00 G=0.81 | nonfinite=0
step  73450 | loss 5.5171 | text 1.9192 audio 5.1333 | grad 1.598 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.00 G=0.70 | nonfinite=0
step  73475 | loss 5.5443 | text 1.9278 audio 5.1587 | grad 1.364 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=+0.00 G=0.61 | nonfinite=0
step  73500 | loss 5.5385 | text 1.9211 audio 5.1543 | grad 1.753 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.82 | nonfinite=0
  running validation at step 73500...
  val/composite=2.8258 (text=0.4027 audio=4.4412) val/loss=4.5217 cb0_acc=41.4% text_acc=96.6%
  [val] per-codebook acc: cb0=41.4% cb1=21.9% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_o1o6yfgd/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  73525 | loss 5.5324 | text 1.9368 audio 5.1450 | grad 2.343 | 23.74s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.12 | nonfinite=0
step  73550 | loss 5.5716 | text 1.9315 audio 5.1853 | grad 3.492 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.09 | spike L=+0.01 G=1.65 | nonfinite=0
step  73575 | loss 5.5426 | text 1.9366 audio 5.1553 | grad 1.977 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.88 | nonfinite=0
step  73600 | loss 5.5388 | text 1.9382 audio 5.1511 | grad 2.030 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.38 lora=0.92 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.91 | nonfinite=0
step  73625 | loss 5.5302 | text 1.9208 audio 5.1461 | grad 1.735 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.04 projection=0.09 text_embed=0.09 | spike L=-0.00 G=0.79 | nonfinite=0
step  73650 | loss 5.5500 | text 1.9373 audio 5.1625 | grad 1.972 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.91 | nonfinite=0
step  73675 | loss 5.5312 | text 1.9350 audio 5.1442 | grad 2.625 | 1.42s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.22 | nonfinite=0
step  73700 | loss 5.5399 | text 1.9347 audio 5.1529 | grad 2.746 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.25 | nonfinite=0
step  73725 | loss 5.5269 | text 1.9173 audio 5.1434 | grad 2.515 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.08 | spike L=-0.00 G=1.12 | nonfinite=0
step  73750 | loss 5.5173 | text 1.9215 audio 5.1330 | grad 1.923 | 1.45s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.85 | nonfinite=0
  running validation at step 73750...
  val/composite=2.8230 (text=0.3989 audio=4.4390) val/loss=4.5188 cb0_acc=41.4% text_acc=96.6%
  [val] per-codebook acc: cb0=41.4% cb1=22.0% cb2=18.2% cb3=12.0% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt_22g4hijg/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  73775 | loss 5.5445 | text 1.9455 audio 5.1554 | grad 1.668 | 23.81s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.74 | nonfinite=0
step  73800 | loss 5.5787 | text 1.9452 audio 5.1897 | grad 2.395 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.30 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.01 G=1.10 | nonfinite=0
step  73825 | loss 5.5213 | text 1.9341 audio 5.1345 | grad 1.362 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.59 lora=0.79 model_audio_embed=0.05 projection=0.12 text_embed=0.12 | spike L=-0.00 G=0.62 | nonfinite=0
step  73850 | loss 5.5027 | text 1.9205 audio 5.1186 | grad 2.529 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.01 G=1.19 | nonfinite=0
step  73875 | loss 5.5265 | text 1.9347 audio 5.1396 | grad 1.945 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.90 | nonfinite=0
step  73900 | loss 5.5608 | text 1.9319 audio 5.1744 | grad 2.310 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.16 | spike L=+0.00 G=1.08 | nonfinite=0
step  73925 | loss 5.5237 | text 1.9255 audio 5.1386 | grad 1.929 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.89 | nonfinite=0
step  73950 | loss 5.5298 | text 1.9249 audio 5.1448 | grad 1.338 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.56 lora=0.81 model_audio_embed=0.05 projection=0.11 text_embed=0.13 | spike L=-0.00 G=0.63 | nonfinite=0
step  73975 | loss 5.5543 | text 1.9333 audio 5.1676 | grad 1.641 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.45 lora=0.87 model_audio_embed=0.06 projection=0.09 text_embed=0.15 | spike L=+0.00 G=0.80 | nonfinite=0
step  74000 | loss 5.5167 | text 1.9365 audio 5.1294 | grad 1.890 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.94 | nonfinite=0
  running validation at step 74000...
  val/composite=2.8233 (text=0.4004 audio=4.4386) val/loss=4.5187 cb0_acc=41.4% text_acc=96.6%
  [val] per-codebook acc: cb0=41.4% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.3% cb7=6.1%
[ckpt] uploading /tmp/ckpt__nzvxe3c/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_074000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_074000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  74025 | loss 5.5283 | text 1.9222 audio 5.1439 | grad 1.450 | 23.19s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.00 G=0.72 | nonfinite=0
step  74050 | loss 5.5187 | text 1.9372 audio 5.1312 | grad 1.754 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.17 | spike L=-0.00 G=0.90 | nonfinite=0
step  74075 | loss 5.5569 | text 1.9379 audio 5.1693 | grad 3.015 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.27 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.57 | nonfinite=0
step  74100 | loss 5.5387 | text 1.9368 audio 5.1513 | grad 1.797 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.88 | nonfinite=0
step  74125 | loss 5.5796 | text 1.9543 audio 5.1888 | grad 1.686 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.11 | spike L=+0.01 G=0.84 | nonfinite=0
step  74150 | loss 5.5883 | text 1.9488 audio 5.1985 | grad 3.209 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.26 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=+0.01 G=1.62 | nonfinite=0
step  74175 | loss 5.5452 | text 1.9207 audio 5.1610 | grad 4.235 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.18 lora=0.97 model_audio_embed=0.06 projection=0.04 text_embed=0.12 | spike L=+0.00 G=2.02 | nonfinite=0
step  74200 | loss 5.5087 | text 1.9260 audio 5.1235 | grad 2.167 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.36 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.01 G=0.94 | nonfinite=0
step  74225 | loss 5.5062 | text 1.9359 audio 5.1190 | grad 2.002 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.40 lora=0.88 model_audio_embed=0.06 projection=0.07 text_embed=0.25 | spike L=-0.01 G=0.87 | nonfinite=0
step  74250 | loss 5.5302 | text 1.9263 audio 5.1450 | grad 2.457 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=-0.00 G=1.08 | nonfinite=0
  running validation at step 74250...
  val/composite=2.8225 (text=0.3993 audio=4.4380) val/loss=4.5179 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_p0f5ccyw/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  74275 | loss 5.5209 | text 1.9302 audio 5.1348 | grad 1.996 | 23.89s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.87 | nonfinite=0
step  74300 | loss 5.5467 | text 1.9396 audio 5.1588 | grad 1.549 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.49 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=+0.00 G=0.69 | nonfinite=0
step  74325 | loss 5.5449 | text 1.9341 audio 5.1580 | grad 1.556 | 1.46s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.12 | spike L=+0.00 G=0.71 | nonfinite=0
step  74350 | loss 5.5482 | text 1.9284 audio 5.1625 | grad 1.940 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.40 lora=0.91 model_audio_embed=0.04 projection=0.08 text_embed=0.09 | spike L=+0.00 G=0.91 | nonfinite=0
step  74375 | loss 5.5148 | text 1.9471 audio 5.1253 | grad 2.086 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.99 | nonfinite=0
step  74400 | loss 5.5809 | text 1.9352 audio 5.1939 | grad 2.400 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.13 | spike L=+0.01 G=1.14 | nonfinite=0
step  74425 | loss 5.5197 | text 1.9402 audio 5.1317 | grad 2.059 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.96 | nonfinite=0
step  74450 | loss 5.5203 | text 1.9444 audio 5.1314 | grad 3.264 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.23 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.11 | spike L=-0.00 G=1.54 | nonfinite=0
step  74475 | loss 5.5067 | text 1.9128 audio 5.1242 | grad 1.664 | 1.43s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=-0.01 G=0.74 | nonfinite=0
step  74500 | loss 5.5534 | text 1.9423 audio 5.1649 | grad 1.917 | 1.44s/step | peak -1.0G | host_rss 35.6G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=+0.00 G=0.88 | nonfinite=0
  running validation at step 74500...
  val/composite=2.8217 (text=0.3992 audio=4.4367) val/loss=4.5165 cb0_acc=41.4% text_acc=96.6%
  [val] per-codebook acc: cb0=41.4% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_is7klo6i/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  74525 | loss 5.5087 | text 1.9247 audio 5.1237 | grad 1.410 | 23.81s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.56 lora=0.81 model_audio_embed=0.05 projection=0.11 text_embed=0.09 | spike L=-0.00 G=0.65 | nonfinite=0
step  74550 | loss 5.5113 | text 1.9318 audio 5.1250 | grad 1.651 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.79 | nonfinite=0
step  74575 | loss 5.5264 | text 1.9277 audio 5.1409 | grad 1.620 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.10 text_embed=0.09 | spike L=-0.00 G=0.79 | nonfinite=0
step  74600 | loss 5.5303 | text 1.9219 audio 5.1459 | grad 1.590 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.51 lora=0.84 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.80 | nonfinite=0
step  74625 | loss 5.4959 | text 1.9237 audio 5.1112 | grad 1.910 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.10 | spike L=-0.01 G=0.98 | nonfinite=0
step  74650 | loss 5.5261 | text 1.9334 audio 5.1394 | grad 3.505 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.22 lora=0.97 model_audio_embed=0.04 projection=0.05 text_embed=0.09 | spike L=-0.00 G=1.80 | nonfinite=0
step  74675 | loss 5.5356 | text 1.9365 audio 5.1483 | grad 1.925 | 1.46s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.15 | spike L=+0.00 G=0.91 | nonfinite=0
step  74700 | loss 5.4921 | text 1.9257 audio 5.1070 | grad 1.295 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.57 lora=0.80 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.01 G=0.62 | nonfinite=0
step  74725 | loss 5.5453 | text 1.9286 audio 5.1596 | grad 1.777 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.43 lora=0.89 model_audio_embed=0.06 projection=0.09 text_embed=0.11 | spike L=+0.00 G=0.88 | nonfinite=0
step  74750 | loss 5.4944 | text 1.9303 audio 5.1083 | grad 1.674 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.09 | spike L=-0.01 G=0.84 | nonfinite=0
  running validation at step 74750...
  val/composite=2.8216 (text=0.4000 audio=4.4359) val/loss=4.5159 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_hqrmpqrk/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  74775 | loss 5.5373 | text 1.9185 audio 5.1536 | grad 1.385 | 23.91s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.53 lora=0.83 model_audio_embed=0.05 projection=0.11 text_embed=0.12 | spike L=+0.00 G=0.71 | nonfinite=0
step  74800 | loss 5.5082 | text 1.9246 audio 5.1233 | grad 2.421 | 1.47s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=-0.00 G=1.28 | nonfinite=0
step  74825 | loss 5.5233 | text 1.9234 audio 5.1386 | grad 2.260 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=1.16 | nonfinite=0
step  74850 | loss 5.5278 | text 1.9364 audio 5.1405 | grad 2.981 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.25 lora=0.96 model_audio_embed=0.05 projection=0.05 text_embed=0.12 | spike L=+0.00 G=1.50 | nonfinite=0
step  74875 | loss 5.5562 | text 1.9380 audio 5.1686 | grad 1.443 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.54 lora=0.83 model_audio_embed=0.05 projection=0.11 text_embed=0.09 | spike L=+0.01 G=0.69 | nonfinite=0
step  74900 | loss 5.5219 | text 1.9253 audio 5.1368 | grad 2.414 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.09 | spike L=-0.00 G=1.20 | nonfinite=0
step  74925 | loss 5.4928 | text 1.9226 audio 5.1083 | grad 2.202 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.35 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.13 | spike L=-0.01 G=1.07 | nonfinite=0
step  74950 | loss 5.5433 | text 1.9407 audio 5.1551 | grad 2.236 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.09 | spike L=+0.00 G=1.08 | nonfinite=0
step  74975 | loss 5.4831 | text 1.9168 audio 5.0997 | grad 2.764 | 1.44s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.29 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.09 | spike L=-0.01 G=1.32 | nonfinite=0
step  75000 | loss 5.5227 | text 1.9276 audio 5.1371 | grad 2.085 | 1.43s/step | peak -1.0G | host_rss 35.7G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=+0.00 G=0.97 | nonfinite=0
  [audio-demo] ar_cb0_acc=6.0% (32s)
  running validation at step 75000...
[ckpt] WARNING: async upload failed: (Request ID: Root=1-6a5e4991-667b40c255bc9add72dc191f;736da97f-5ebd-4d1e-92d5-f6e43840850e)

403 Forbidden: Private repository storage limit reached, please upgrade your plan to increase your private storage limit.
Cannot access content at: https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3.git/info/lfs/objects/batch.
Make sure your token has the correct permissions.
  val/composite=2.8203 (text=0.3974 audio=4.4356) val/loss=4.5150 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_ui6_cnd7/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_1zqfzc9x/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_075000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3
  [ckpt-index] log failed: Cannot read the W&B step in shared mode.
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_075000
  pushed 1 file(s) -> https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3/tree/main/logs
step  75025 | loss 5.5113 | text 1.9266 audio 5.1260 | grad 2.047 | 46.33s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.37 lora=0.92 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.00 G=0.95 | nonfinite=0
step  75050 | loss 5.5299 | text 1.9337 audio 5.1432 | grad 1.896 | 1.42s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.09 | spike L=+0.00 G=0.89 | nonfinite=0
step  75075 | loss 5.5521 | text 1.9464 audio 5.1628 | grad 1.651 | 1.43s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.45 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=+0.01 G=0.78 | nonfinite=0
step  75100 | loss 5.5331 | text 1.9260 audio 5.1479 | grad 1.829 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=+0.00 G=0.88 | nonfinite=0
step  75125 | loss 5.5254 | text 1.9286 audio 5.1397 | grad 2.343 | 1.46s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.04 projection=0.06 text_embed=0.10 | spike L=+0.00 G=1.15 | nonfinite=0
step  75150 | loss 5.5237 | text 1.9321 audio 5.1373 | grad 1.384 | 1.43s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.10 | spike L=-0.00 G=0.67 | nonfinite=0
step  75175 | loss 5.5475 | text 1.9258 audio 5.1623 | grad 1.638 | 1.43s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.17 | spike L=+0.00 G=0.82 | nonfinite=0
step  75200 | loss 5.5273 | text 1.9302 audio 5.1413 | grad 2.348 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.32 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.11 | spike L=+0.00 G=1.19 | nonfinite=0
step  75225 | loss 5.5306 | text 1.9366 audio 5.1433 | grad 2.069 | 1.43s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.10 | spike L=+0.00 G=1.03 | nonfinite=0
step  75250 | loss 5.5234 | text 1.9136 audio 5.1407 | grad 2.088 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.38 lora=0.92 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=1.04 | nonfinite=0
  running validation at step 75250...
  val/composite=2.8200 (text=0.3978 audio=4.4348) val/loss=4.5144 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_rgf4lnds/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  75275 | loss 5.5107 | text 1.9105 audio 5.1286 | grad 1.871 | 23.85s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.42 lora=0.90 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=0.93 | nonfinite=0
step  75300 | loss 5.5376 | text 1.9252 audio 5.1526 | grad 1.485 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.58 lora=0.80 model_audio_embed=0.05 projection=0.12 text_embed=0.11 | spike L=+0.00 G=0.74 | nonfinite=0
step  75325 | loss 5.5086 | text 1.9409 audio 5.1204 | grad 1.995 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.09 | spike L=-0.00 G=1.02 | nonfinite=0
step  75350 | loss 5.5494 | text 1.9433 audio 5.1607 | grad 1.708 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.46 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.87 | nonfinite=0
step  75375 | loss 5.5282 | text 1.9310 audio 5.1420 | grad 1.602 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.48 lora=0.87 model_audio_embed=0.05 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.83 | nonfinite=0
step  75400 | loss 5.5590 | text 1.9338 audio 5.1723 | grad 1.925 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=+0.01 G=1.01 | nonfinite=0
step  75425 | loss 5.5113 | text 1.9421 audio 5.1229 | grad 1.688 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.44 lora=0.88 model_audio_embed=0.05 projection=0.09 text_embed=0.13 | spike L=-0.00 G=0.89 | nonfinite=0
step  75450 | loss 5.5119 | text 1.9224 audio 5.1274 | grad 1.479 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.55 lora=0.82 model_audio_embed=0.05 projection=0.11 text_embed=0.11 | spike L=-0.00 G=0.79 | nonfinite=0
step  75475 | loss 5.5237 | text 1.9185 audio 5.1400 | grad 1.534 | 1.46s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.50 lora=0.85 model_audio_embed=0.05 projection=0.10 text_embed=0.12 | spike L=-0.00 G=0.83 | nonfinite=0
step  75500 | loss 5.5312 | text 1.9159 audio 5.1480 | grad 2.075 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.38 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.13 | spike L=+0.00 G=1.15 | nonfinite=0
  running validation at step 75500...
  val/composite=2.8201 (text=0.3981 audio=4.4348) val/loss=4.5144 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.3% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
step  75525 | loss 5.5168 | text 1.9257 audio 5.1317 | grad 2.223 | 2.19s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.33 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.11 | spike L=-0.00 G=1.21 | nonfinite=0
step  75550 | loss 5.5370 | text 1.9294 audio 5.1511 | grad 3.139 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.24 lora=0.96 model_audio_embed=0.04 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.67 | nonfinite=0
step  75575 | loss 5.5511 | text 1.9267 audio 5.1658 | grad 2.855 | 1.44s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.27 lora=0.95 model_audio_embed=0.04 projection=0.06 text_embed=0.13 | spike L=+0.00 G=1.43 | nonfinite=0
step  75600 | loss 5.5343 | text 1.9287 audio 5.1485 | grad 1.969 | 1.43s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.40 lora=0.90 model_audio_embed=0.06 projection=0.08 text_embed=0.13 | spike L=+0.00 G=0.94 | nonfinite=0
step  75625 | loss 5.5495 | text 1.9244 audio 5.1646 | grad 1.777 | 1.46s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.49 lora=0.86 model_audio_embed=0.06 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.86 | nonfinite=0
step  75650 | loss 5.5488 | text 1.9337 audio 5.1621 | grad 1.607 | 1.46s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.47 lora=0.87 model_audio_embed=0.05 projection=0.09 text_embed=0.10 | spike L=+0.00 G=0.79 | nonfinite=0
step  75675 | loss 5.5252 | text 1.9181 audio 5.1416 | grad 1.760 | 1.46s/step | peak -1.0G | host_rss 36.0G
  [diag] gradnorm depth=0.42 lora=0.89 model_audio_embed=0.05 projection=0.08 text_embed=0.11 | spike L=-0.00 G=0.88 | nonfinite=0
step  75700 | loss 5.5314 | text 1.9281 audio 5.1458 | grad 2.620 | 5.18s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.31 lora=0.94 model_audio_embed=0.05 projection=0.06 text_embed=0.10 | spike L=-0.00 G=1.33 | nonfinite=0
step  75725 | loss 5.5297 | text 1.9285 audio 5.1440 | grad 1.616 | 1.47s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.48 lora=0.86 model_audio_embed=0.05 projection=0.10 text_embed=0.11 | spike L=-0.00 G=0.79 | nonfinite=0
step  75750 | loss 5.5077 | text 1.9324 audio 5.1212 | grad 1.941 | 1.47s/step | peak -1.0G | host_rss 37.0G
  [diag] gradnorm depth=0.39 lora=0.91 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.97 | nonfinite=0
  running validation at step 75750...
  val/composite=2.8199 (text=0.3977 audio=4.4347) val/loss=4.5143 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_4dfalb6j/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
step  75775 | loss 5.5134 | text 1.9207 audio 5.1293 | grad 2.277 | 23.96s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.34 lora=0.93 model_audio_embed=0.04 projection=0.07 text_embed=0.09 | spike L=-0.00 G=1.14 | nonfinite=0
step  75800 | loss 5.5151 | text 1.9138 audio 5.1324 | grad 4.870 | 1.47s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.16 lora=0.98 model_audio_embed=0.05 projection=0.03 text_embed=0.11 | spike L=-0.00 G=2.41 | nonfinite=0
step  75825 | loss 5.5010 | text 1.9115 audio 5.1187 | grad 1.475 | 1.45s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.55 lora=0.83 model_audio_embed=0.05 projection=0.11 text_embed=0.09 | spike L=-0.00 G=0.64 | nonfinite=0
step  75850 | loss 5.5127 | text 1.9282 audio 5.1271 | grad 1.796 | 1.45s/step | peak -1.0G | host_rss 37.3G
  [diag] gradnorm depth=0.41 lora=0.90 model_audio_embed=0.05 projection=0.08 text_embed=0.12 | spike L=-0.00 G=0.81 | nonfinite=0
step  75875 | loss 5.5467 | text 1.9228 audio 5.1621 | grad 3.450 | 1.44s/step | peak -1.0G | host_rss 37.2G
  [diag] gradnorm depth=0.23 lora=0.97 model_audio_embed=0.05 projection=0.05 text_embed=0.10 | spike L=+0.00 G=1.58 | nonfinite=0
step  75900 | loss 5.5455 | text 1.9195 audio 5.1616 | grad 2.258 | 1.44s/step | peak -1.0G | host_rss 37.2G
  [diag] gradnorm depth=0.36 lora=0.92 model_audio_embed=0.05 projection=0.07 text_embed=0.12 | spike L=+0.00 G=0.98 | nonfinite=0
step  75925 | loss 5.5288 | text 1.9143 audio 5.1460 | grad 1.378 | 4.47s/step | peak -1.0G | host_rss 37.6G
  [diag] gradnorm depth=0.52 lora=0.84 model_audio_embed=0.04 projection=0.10 text_embed=0.10 | spike L=+0.00 G=0.60 | nonfinite=0
step  75950 | loss 5.5223 | text 1.9249 audio 5.1373 | grad 2.064 | 1.47s/step | peak -1.0G | host_rss 37.6G
  [diag] gradnorm depth=0.35 lora=0.93 model_audio_embed=0.05 projection=0.07 text_embed=0.11 | spike L=-0.00 G=0.93 | nonfinite=0
step  75975 | loss 5.5103 | text 1.9413 audio 5.1221 | grad 2.314 | 1.46s/step | peak -1.0G | host_rss 37.6G
  [diag] gradnorm depth=0.46 lora=0.88 model_audio_embed=0.04 projection=0.08 text_embed=0.10 | spike L=-0.00 G=1.05 | nonfinite=0
step  76000 | loss 5.5270 | text 1.9096 audio 5.1451 | grad 2.212 | 1.50s/step | peak -1.0G | host_rss 37.6G
  [diag] gradnorm depth=0.33 lora=0.94 model_audio_embed=0.05 projection=0.07 text_embed=0.10 | spike L=+0.00 G=1.00 | nonfinite=0
  running validation at step 76000...
  val/composite=2.8199 (text=0.3978 audio=4.4346) val/loss=4.5142 cb0_acc=41.5% text_acc=96.6%
  [val] per-codebook acc: cb0=41.5% cb1=22.0% cb2=18.2% cb3=12.1% cb4=9.2% cb5=7.5% cb6=6.4% cb7=6.1%
[ckpt] uploading /tmp/ckpt_47wvo1aj/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] upload complete: gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
  * new best val — saved to gs:/tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/best_by_val
[ckpt] uploading /tmp/ckpt_6ga8pi01/* -> gs://tinyaya-stage2-eu/checkpoints/stage2-v6e16-mh-v03-r2/step_076000
  pushed to https://huggingface.co/tiny-aya-translate/tr-hi-s2st-v0.3