| W0808 10:31:20.445000 3495619 site-packages/torch/distributed/run.py:793] |
| W0808 10:31:20.445000 3495619 site-packages/torch/distributed/run.py:793] ***************************************** |
| W0808 10:31:20.445000 3495619 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. |
| W0808 10:31:20.445000 3495619 site-packages/torch/distributed/run.py:793] ***************************************** |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| CAUSAL_DISABLE_FLEX_ATTENTION=1 -> using segmented FlashAttention fallback |
| Rank 0 preloading generator from /data/fengjiaqi/causal_forcing/checkpoints/framewise/causal_cd.pt |
| [rank0]:[W808 10:31:34.439885095 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank5]:[W808 10:31:35.822494745 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank3]:[W808 10:31:35.832241850 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank4]:[W808 10:31:35.974846924 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank7]:[W808 10:31:35.208742648 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank6]:[W808 10:31:35.238435032 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank1]:[W808 10:31:35.245749668 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| [rank2]:[W808 10:31:35.304042470 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id. |
| DATASET TRAIN 6249 VALIDATION 256 SPLIT /home/jiaqi/NoPE/artifacts/data_splits/cpt_train_val_seed20260805.json |
| validation step=0 loss=0.26354822 alpha=0.000000 samples=32 seconds=22.517 |
| validation step=0 loss=0.15818384 alpha=0.250000 samples=32 seconds=22.643 |
| validation step=0 loss=0.09946489 alpha=0.500000 samples=32 seconds=23.028 |
| validation step=0 loss=0.06696399 alpha=0.750000 samples=32 seconds=23.197 |
| validation step=0 loss=0.06264185 alpha=1.000000 samples=32 seconds=23.181 |
|
|
