2026-03-01 07:36:18 INFO: Logging to: train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f/debug.log
2026-03-01 07:36:18 DEBUG: CMD: train.py configs/main_exp/ministral-3b/compact_only.yaml --model_name_or_path=mistralai/Ministral-3-3B-Instruct-2512 --target_modules=down_proj --lora_r=8 --eval_strategy=no --max_qas_len=2048 --max_qas_per_sample=1 --per_rank_gen=True --per_layer_processing=True --gen_lora_l1_reg_coef=0.1 --max_steps=4000 --gradient_accumulation_steps=8 --max_packed_inp_len=2048 --max_packed_ctx_len=2048 --use_per_ctx_average_loss=True --use_kl_loss=True --quantize_ctx_encoder=True --save_steps=250 --save_total_limit=5
2026-03-01 07:36:18 DEBUG: args: {'train_ds_names': ['self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact'], 'streaming': False, 'val_ds_names': ['squad', 'pwc', 'drop', 'ropes'], 'test_ds_names': None, 'max_train_samples_per_ds': None, 'max_val_samples_per_ds': 1000, 'max_test_samples_per_ds': 500, 'exp_setup': <ExperimentSetup.HYPERLORA: 'hyper_lora'>, 'from_pretrained_checkpoint': None, 'max_base_len': 8192, 'use_sequence_packing': True, 'max_ctx_len': -1, 'max_qas_len': 2048, 'max_qas_per_sample': 1, 'num_chunk_probs': None, 'max_ctx_chunk_len': -1, 'min_ctx_chunk_len': -1, 'max_ctx_chunk_num': None, 'max_packed_inp_len': 2048, 'max_packed_ctx_len': 2048, 'max_new_tokens': 256, 'gen_per_device_eval_batch_size': 1, 'notes': None, 'use_kl_loss': True, 'use_per_ctx_average_loss': True, 'gen_lora_l1_reg_coef': 0.1, 'add_negative_prompt': False, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'use_flash_attn': True, 'lora_r': 8, 'lora_dropout': 0.0, 'target_modules': ['down_proj'], 'output_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 64, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 8, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 4e-05, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 3.0, 'max_steps': 4000, 'lr_scheduler_type': <SchedulerType.COSINE_WITH_MIN_LR: 'cosine_with_min_lr'>, 'lr_scheduler_kwargs': {'min_lr': 1e-07}, 'warmup_ratio': 0.0, 'warmup_steps': 100, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': True, 'logging_steps': 100, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 250, 'save_total_limit': 5, 'save_safetensors': False, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': True, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': 1000, 'dataloader_num_workers': 8, 'dataloader_prefetch_factor': 16, 'past_index': -1, 'run_name': 'Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'disable_tqdm': False, 'remove_unused_columns': False, 'label_names': ('labels',), 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': AcceleratorConfig(split_batches=False, dispatch_batches=None, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH_FUSED: 'adamw_torch_fused'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': False, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': ('inputs',), 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1048576, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': 5.0, 'optim_target_modules': None, 'batch_eval_metrics': True, 'eval_on_start': False, 'use_liger_kernel': False, 'eval_use_gather_object': False, 'average_tokens_across_devices': False, 'distributed_state': Distributed environment: MULTI_GPU  Backend: nccl
Num processes: 4
Process index: 0
Local process index: 0
Device: cuda:0
, '_n_gpu': 1, '__cached__setup_devices': device(type='cuda', index=0), 'deepspeed_plugin': None, 'latent_size': 512, 'use_light_weight_lora': False, 'light_weight_latent_size': 128, 'dropout_rate': 0.0, 'extra_modules': None, 'per_rank_gen': True, 'use_bias': True, 'use_per_rank_bias': False, 'per_layer_processing': True, 'use_token_mixing': False, 'num_pre_head_layers': 1, 'aggregator_type': 'perceiver', 'pooling_type': 'mean', 'num_latent_factor': 8, 'n_latent_queries': 8, 'num_blocks': 9, 'num_self_attn_per_block': 0, 'shared_weights': False, 'ctx_encoder_model_name_or_path': None, 'ctx_encoder_type': 'per_layer_activations', 'layer_idx': None, 'quantize_ctx_encoder': True, 'ctx_encoder_last_layer': None}
2026-03-01 07:36:19 INFO: Logging to: train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f/debug.log
2026-03-01 07:36:19 DEBUG: CMD: train.py configs/main_exp/ministral-3b/compact_only.yaml --model_name_or_path=mistralai/Ministral-3-3B-Instruct-2512 --target_modules=down_proj --lora_r=8 --eval_strategy=no --max_qas_len=2048 --max_qas_per_sample=1 --per_rank_gen=True --per_layer_processing=True --gen_lora_l1_reg_coef=0.1 --max_steps=4000 --gradient_accumulation_steps=8 --max_packed_inp_len=2048 --max_packed_ctx_len=2048 --use_per_ctx_average_loss=True --use_kl_loss=True --quantize_ctx_encoder=True --save_steps=250 --save_total_limit=5
2026-03-01 07:36:19 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2'}
2026-03-01 07:36:19 DEBUG: args: {'train_ds_names': ['self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact'], 'streaming': False, 'val_ds_names': ['squad', 'pwc', 'drop', 'ropes'], 'test_ds_names': None, 'max_train_samples_per_ds': None, 'max_val_samples_per_ds': 1000, 'max_test_samples_per_ds': 500, 'exp_setup': <ExperimentSetup.HYPERLORA: 'hyper_lora'>, 'from_pretrained_checkpoint': None, 'max_base_len': 8192, 'use_sequence_packing': True, 'max_ctx_len': -1, 'max_qas_len': 2048, 'max_qas_per_sample': 1, 'num_chunk_probs': None, 'max_ctx_chunk_len': -1, 'min_ctx_chunk_len': -1, 'max_ctx_chunk_num': None, 'max_packed_inp_len': 2048, 'max_packed_ctx_len': 2048, 'max_new_tokens': 256, 'gen_per_device_eval_batch_size': 1, 'notes': None, 'use_kl_loss': True, 'use_per_ctx_average_loss': True, 'gen_lora_l1_reg_coef': 0.1, 'add_negative_prompt': False, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'use_flash_attn': True, 'lora_r': 8, 'lora_dropout': 0.0, 'target_modules': ['down_proj'], 'output_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 64, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 8, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 4e-05, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 3.0, 'max_steps': 4000, 'lr_scheduler_type': <SchedulerType.COSINE_WITH_MIN_LR: 'cosine_with_min_lr'>, 'lr_scheduler_kwargs': {'min_lr': 1e-07}, 'warmup_ratio': 0.0, 'warmup_steps': 100, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': True, 'logging_steps': 100, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 250, 'save_total_limit': 5, 'save_safetensors': False, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': True, 'local_rank': 2, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': 1000, 'dataloader_num_workers': 8, 'dataloader_prefetch_factor': 16, 'past_index': -1, 'run_name': 'Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'disable_tqdm': False, 'remove_unused_columns': False, 'label_names': ('labels',), 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': AcceleratorConfig(split_batches=False, dispatch_batches=None, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH_FUSED: 'adamw_torch_fused'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': False, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': ('inputs',), 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1048576, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': 5.0, 'optim_target_modules': None, 'batch_eval_metrics': True, 'eval_on_start': False, 'use_liger_kernel': False, 'eval_use_gather_object': False, 'average_tokens_across_devices': False, 'distributed_state': Distributed environment: MULTI_GPU  Backend: nccl
Num processes: 4
Process index: 2
Local process index: 2
Device: cuda:2
, '_n_gpu': 1, '__cached__setup_devices': device(type='cuda', index=2), 'deepspeed_plugin': None, 'latent_size': 512, 'use_light_weight_lora': False, 'light_weight_latent_size': 128, 'dropout_rate': 0.0, 'extra_modules': None, 'per_rank_gen': True, 'use_bias': True, 'use_per_rank_bias': False, 'per_layer_processing': True, 'use_token_mixing': False, 'num_pre_head_layers': 1, 'aggregator_type': 'perceiver', 'pooling_type': 'mean', 'num_latent_factor': 8, 'n_latent_queries': 8, 'num_blocks': 9, 'num_self_attn_per_block': 0, 'shared_weights': False, 'ctx_encoder_model_name_or_path': None, 'ctx_encoder_type': 'per_layer_activations', 'layer_idx': None, 'quantize_ctx_encoder': True, 'ctx_encoder_last_layer': None}
2026-03-01 07:36:19 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:19 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2'}
2026-03-01 07:36:19 DEBUG: Starting new HTTPS connection (1): huggingface.co:443
2026-03-01 07:36:19 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:19 DEBUG: Starting new HTTPS connection (1): huggingface.co:443
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 INFO: Logging to: train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f/debug.log
2026-03-01 07:36:19 DEBUG: CMD: train.py configs/main_exp/ministral-3b/compact_only.yaml --model_name_or_path=mistralai/Ministral-3-3B-Instruct-2512 --target_modules=down_proj --lora_r=8 --eval_strategy=no --max_qas_len=2048 --max_qas_per_sample=1 --per_rank_gen=True --per_layer_processing=True --gen_lora_l1_reg_coef=0.1 --max_steps=4000 --gradient_accumulation_steps=8 --max_packed_inp_len=2048 --max_packed_ctx_len=2048 --use_per_ctx_average_loss=True --use_kl_loss=True --quantize_ctx_encoder=True --save_steps=250 --save_total_limit=5
2026-03-01 07:36:19 DEBUG: args: {'train_ds_names': ['self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact'], 'streaming': False, 'val_ds_names': ['squad', 'pwc', 'drop', 'ropes'], 'test_ds_names': None, 'max_train_samples_per_ds': None, 'max_val_samples_per_ds': 1000, 'max_test_samples_per_ds': 500, 'exp_setup': <ExperimentSetup.HYPERLORA: 'hyper_lora'>, 'from_pretrained_checkpoint': None, 'max_base_len': 8192, 'use_sequence_packing': True, 'max_ctx_len': -1, 'max_qas_len': 2048, 'max_qas_per_sample': 1, 'num_chunk_probs': None, 'max_ctx_chunk_len': -1, 'min_ctx_chunk_len': -1, 'max_ctx_chunk_num': None, 'max_packed_inp_len': 2048, 'max_packed_ctx_len': 2048, 'max_new_tokens': 256, 'gen_per_device_eval_batch_size': 1, 'notes': None, 'use_kl_loss': True, 'use_per_ctx_average_loss': True, 'gen_lora_l1_reg_coef': 0.1, 'add_negative_prompt': False, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'use_flash_attn': True, 'lora_r': 8, 'lora_dropout': 0.0, 'target_modules': ['down_proj'], 'output_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 64, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 8, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 4e-05, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 3.0, 'max_steps': 4000, 'lr_scheduler_type': <SchedulerType.COSINE_WITH_MIN_LR: 'cosine_with_min_lr'>, 'lr_scheduler_kwargs': {'min_lr': 1e-07}, 'warmup_ratio': 0.0, 'warmup_steps': 100, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': True, 'logging_steps': 100, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 250, 'save_total_limit': 5, 'save_safetensors': False, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': True, 'local_rank': 1, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': 1000, 'dataloader_num_workers': 8, 'dataloader_prefetch_factor': 16, 'past_index': -1, 'run_name': 'Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'disable_tqdm': False, 'remove_unused_columns': False, 'label_names': ('labels',), 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': AcceleratorConfig(split_batches=False, dispatch_batches=None, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH_FUSED: 'adamw_torch_fused'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': False, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': ('inputs',), 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1048576, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': 5.0, 'optim_target_modules': None, 'batch_eval_metrics': True, 'eval_on_start': False, 'use_liger_kernel': False, 'eval_use_gather_object': False, 'average_tokens_across_devices': False, 'distributed_state': Distributed environment: MULTI_GPU  Backend: nccl
Num processes: 4
Process index: 1
Local process index: 1
Device: cuda:1
, '_n_gpu': 1, '__cached__setup_devices': device(type='cuda', index=1), 'deepspeed_plugin': None, 'latent_size': 512, 'use_light_weight_lora': False, 'light_weight_latent_size': 128, 'dropout_rate': 0.0, 'extra_modules': None, 'per_rank_gen': True, 'use_bias': True, 'use_per_rank_bias': False, 'per_layer_processing': True, 'use_token_mixing': False, 'num_pre_head_layers': 1, 'aggregator_type': 'perceiver', 'pooling_type': 'mean', 'num_latent_factor': 8, 'n_latent_queries': 8, 'num_blocks': 9, 'num_self_attn_per_block': 0, 'shared_weights': False, 'ctx_encoder_model_name_or_path': None, 'ctx_encoder_type': 'per_layer_activations', 'layer_idx': None, 'quantize_ctx_encoder': True, 'ctx_encoder_last_layer': None}
2026-03-01 07:36:19 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2'}
2026-03-01 07:36:19 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:19 DEBUG: Starting new HTTPS connection (1): huggingface.co:443
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 INFO: Logging to: train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f/debug.log
2026-03-01 07:36:19 DEBUG: CMD: train.py configs/main_exp/ministral-3b/compact_only.yaml --model_name_or_path=mistralai/Ministral-3-3B-Instruct-2512 --target_modules=down_proj --lora_r=8 --eval_strategy=no --max_qas_len=2048 --max_qas_per_sample=1 --per_rank_gen=True --per_layer_processing=True --gen_lora_l1_reg_coef=0.1 --max_steps=4000 --gradient_accumulation_steps=8 --max_packed_inp_len=2048 --max_packed_ctx_len=2048 --use_per_ctx_average_loss=True --use_kl_loss=True --quantize_ctx_encoder=True --save_steps=250 --save_total_limit=5
2026-03-01 07:36:19 DEBUG: args: {'train_ds_names': ['self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact', 'self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact'], 'streaming': False, 'val_ds_names': ['squad', 'pwc', 'drop', 'ropes'], 'test_ds_names': None, 'max_train_samples_per_ds': None, 'max_val_samples_per_ds': 1000, 'max_test_samples_per_ds': 500, 'exp_setup': <ExperimentSetup.HYPERLORA: 'hyper_lora'>, 'from_pretrained_checkpoint': None, 'max_base_len': 8192, 'use_sequence_packing': True, 'max_ctx_len': -1, 'max_qas_len': 2048, 'max_qas_per_sample': 1, 'num_chunk_probs': None, 'max_ctx_chunk_len': -1, 'min_ctx_chunk_len': -1, 'max_ctx_chunk_num': None, 'max_packed_inp_len': 2048, 'max_packed_ctx_len': 2048, 'max_new_tokens': 256, 'gen_per_device_eval_batch_size': 1, 'notes': None, 'use_kl_loss': True, 'use_per_ctx_average_loss': True, 'gen_lora_l1_reg_coef': 0.1, 'add_negative_prompt': False, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'use_flash_attn': True, 'lora_r': 8, 'lora_dropout': 0.0, 'target_modules': ['down_proj'], 'output_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': True, 'do_predict': False, 'eval_strategy': <IntervalStrategy.NO: 'no'>, 'prediction_loss_only': False, 'per_device_train_batch_size': 1, 'per_device_eval_batch_size': 64, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 8, 'eval_accumulation_steps': None, 'eval_delay': 0, 'torch_empty_cache_steps': None, 'learning_rate': 4e-05, 'weight_decay': 0.01, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 3.0, 'max_steps': 4000, 'lr_scheduler_type': <SchedulerType.COSINE_WITH_MIN_LR: 'cosine_with_min_lr'>, 'lr_scheduler_kwargs': {'min_lr': 1e-07}, 'warmup_ratio': 0.0, 'warmup_steps': 100, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': 'train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'logging_strategy': <IntervalStrategy.STEPS: 'steps'>, 'logging_first_step': True, 'logging_steps': 100, 'logging_nan_inf_filter': True, 'save_strategy': <SaveStrategy.STEPS: 'steps'>, 'save_steps': 250, 'save_total_limit': 5, 'save_safetensors': False, 'save_on_each_node': False, 'save_only_model': False, 'restore_callback_states_from_checkpoint': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': True, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': True, 'local_rank': 3, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': 1000, 'dataloader_num_workers': 8, 'dataloader_prefetch_factor': 16, 'past_index': -1, 'run_name': 'Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f', 'disable_tqdm': False, 'remove_unused_columns': False, 'label_names': ('labels',), 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, 'tp_size': 0, 'fsdp_transformer_layer_cls_to_wrap': None, 'accelerator_config': AcceleratorConfig(split_batches=False, dispatch_batches=None, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': <OptimizerNames.ADAMW_TORCH_FUSED: 'adamw_torch_fused'>, 'optim_args': None, 'adafactor': False, 'group_by_length': False, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': False, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': False, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': <HubStrategy.EVERY_SAVE: 'every_save'>, 'hub_token': None, 'hub_private_repo': None, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'include_for_metrics': ('inputs',), 'eval_do_concat_batches': True, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': None, 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1048576, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': 5.0, 'optim_target_modules': None, 'batch_eval_metrics': True, 'eval_on_start': False, 'use_liger_kernel': False, 'eval_use_gather_object': False, 'average_tokens_across_devices': False, 'distributed_state': Distributed environment: MULTI_GPU  Backend: nccl
Num processes: 4
Process index: 3
Local process index: 3
Device: cuda:3
, '_n_gpu': 1, '__cached__setup_devices': device(type='cuda', index=3), 'deepspeed_plugin': None, 'latent_size': 512, 'use_light_weight_lora': False, 'light_weight_latent_size': 128, 'dropout_rate': 0.0, 'extra_modules': None, 'per_rank_gen': True, 'use_bias': True, 'use_per_rank_bias': False, 'per_layer_processing': True, 'use_token_mixing': False, 'num_pre_head_layers': 1, 'aggregator_type': 'perceiver', 'pooling_type': 'mean', 'num_latent_factor': 8, 'n_latent_queries': 8, 'num_blocks': 9, 'num_self_attn_per_block': 0, 'shared_weights': False, 'ctx_encoder_model_name_or_path': None, 'ctx_encoder_type': 'per_layer_activations', 'layer_idx': None, 'quantize_ctx_encoder': True, 'ctx_encoder_last_layer': None}
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2'}
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:19 DEBUG: Starting new HTTPS connection (1): huggingface.co:443
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:19 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:27 DEBUG: Loading bitsandbytes native library from: /app/.venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda124.so
2026-03-01 07:36:27 DEBUG: Loading bitsandbytes native library from: /app/.venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda124.so
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:27 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:28 DEBUG: Loading bitsandbytes native library from: /app/.venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda124.so
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:28 DEBUG: Loading bitsandbytes native library from: /app/.venv/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda124.so
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2026-03-01 07:36:28 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512/cfcb068fa7c44114cf77a462357c6cdcd2c304b4/tokenizer_config.json HTTP/1.1" 200 0
2026-03-01 07:36:29 INFO: Using chat template from chat_templates/mistralai/Ministral-3-3B-Instruct-2512.jinja
2026-03-01 07:36:29 INFO: Using HyperLoRA
2026-03-01 07:36:29 INFO: Using the first 6 layers as the context encoder
2026-03-01 07:36:29 DEBUG: HyperLoRA config: HypernetConfig(latent_size=512, use_light_weight_lora=False, light_weight_latent_size=128, per_rank_gen=True, use_per_rank_bias=False, use_bias=True, per_layer_processing=True, use_token_mixing=False, num_pre_head_layers=1, dropout_rate=0.0, lora_config=LoraConfig(task_type='CAUSAL_LM', peft_type=<PeftType.LORA: 'LORA'>, auto_mapping=None, base_model_name_or_path='mistralai/Ministral-3-3B-Instruct-2512', revision=None, inference_mode=False, r=8, target_modules={'down_proj'}, exclude_modules=None, lora_alpha=45.254833995939045, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={}, megatron_config=None, megatron_core='megatron.core', trainable_token_indices=None, loftq_config={}, eva_config=None, corda_config=None, use_dora=False, layer_replication=None, runtime_config=LoraRuntimeConfig(ephemeral_gpu_offload=False), lora_bias=False), extra_modules=None, base_hidden_size=3072, layer_indices=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25], device='cuda:3'), feature_sizes=({'down_proj': 9216}, {'down_proj': 3072}), aggregator_config=AggregatorConfig(aggregator_type='perceiver', num_layers=26, num_modules=1, num_extra_modules=0, output_size=512, feature_size=3072, pooling_type='mean', num_latent_factor=8, lora_r=8, per_rank_gen=True, n_latent_queries=8, num_blocks=9, num_self_attn_per_block=0, shared_weights=False, layer_to_layer_ctx_encoder=True))
2026-03-01 07:36:29 INFO: Using chat template from chat_templates/mistralai/Ministral-3-3B-Instruct-2512.jinja
2026-03-01 07:36:29 INFO: Using HyperLoRA
2026-03-01 07:36:29 INFO: Using the first 6 layers as the context encoder
2026-03-01 07:36:29 DEBUG: HyperLoRA config: HypernetConfig(latent_size=512, use_light_weight_lora=False, light_weight_latent_size=128, per_rank_gen=True, use_per_rank_bias=False, use_bias=True, per_layer_processing=True, use_token_mixing=False, num_pre_head_layers=1, dropout_rate=0.0, lora_config=LoraConfig(task_type='CAUSAL_LM', peft_type=<PeftType.LORA: 'LORA'>, auto_mapping=None, base_model_name_or_path='mistralai/Ministral-3-3B-Instruct-2512', revision=None, inference_mode=False, r=8, target_modules={'down_proj'}, exclude_modules=None, lora_alpha=45.254833995939045, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={}, megatron_config=None, megatron_core='megatron.core', trainable_token_indices=None, loftq_config={}, eva_config=None, corda_config=None, use_dora=False, layer_replication=None, runtime_config=LoraRuntimeConfig(ephemeral_gpu_offload=False), lora_bias=False), extra_modules=None, base_hidden_size=3072, layer_indices=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25], device='cuda:1'), feature_sizes=({'down_proj': 9216}, {'down_proj': 3072}), aggregator_config=AggregatorConfig(aggregator_type='perceiver', num_layers=26, num_modules=1, num_extra_modules=0, output_size=512, feature_size=3072, pooling_type='mean', num_latent_factor=8, lora_r=8, per_rank_gen=True, n_latent_queries=8, num_blocks=9, num_self_attn_per_block=0, shared_weights=False, layer_to_layer_ctx_encoder=True))
2026-03-01 07:36:29 INFO: Using chat template from chat_templates/mistralai/Ministral-3-3B-Instruct-2512.jinja
2026-03-01 07:36:29 INFO: Using HyperLoRA
2026-03-01 07:36:29 INFO: Using the first 6 layers as the context encoder
2026-03-01 07:36:29 DEBUG: HyperLoRA config: HypernetConfig(latent_size=512, use_light_weight_lora=False, light_weight_latent_size=128, per_rank_gen=True, use_per_rank_bias=False, use_bias=True, per_layer_processing=True, use_token_mixing=False, num_pre_head_layers=1, dropout_rate=0.0, lora_config=LoraConfig(task_type='CAUSAL_LM', peft_type=<PeftType.LORA: 'LORA'>, auto_mapping=None, base_model_name_or_path='mistralai/Ministral-3-3B-Instruct-2512', revision=None, inference_mode=False, r=8, target_modules={'down_proj'}, exclude_modules=None, lora_alpha=45.254833995939045, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={}, megatron_config=None, megatron_core='megatron.core', trainable_token_indices=None, loftq_config={}, eva_config=None, corda_config=None, use_dora=False, layer_replication=None, runtime_config=LoraRuntimeConfig(ephemeral_gpu_offload=False), lora_bias=False), extra_modules=None, base_hidden_size=3072, layer_indices=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25], device='cuda:0'), feature_sizes=({'down_proj': 9216}, {'down_proj': 3072}), aggregator_config=AggregatorConfig(aggregator_type='perceiver', num_layers=26, num_modules=1, num_extra_modules=0, output_size=512, feature_size=3072, pooling_type='mean', num_latent_factor=8, lora_r=8, per_rank_gen=True, n_latent_queries=8, num_blocks=9, num_self_attn_per_block=0, shared_weights=False, layer_to_layer_ctx_encoder=True))
2026-03-01 07:36:30 INFO: Using chat template from chat_templates/mistralai/Ministral-3-3B-Instruct-2512.jinja
2026-03-01 07:36:30 INFO: Using HyperLoRA
2026-03-01 07:36:30 INFO: Using the first 6 layers as the context encoder
2026-03-01 07:36:30 DEBUG: HyperLoRA config: HypernetConfig(latent_size=512, use_light_weight_lora=False, light_weight_latent_size=128, per_rank_gen=True, use_per_rank_bias=False, use_bias=True, per_layer_processing=True, use_token_mixing=False, num_pre_head_layers=1, dropout_rate=0.0, lora_config=LoraConfig(task_type='CAUSAL_LM', peft_type=<PeftType.LORA: 'LORA'>, auto_mapping=None, base_model_name_or_path='mistralai/Ministral-3-3B-Instruct-2512', revision=None, inference_mode=False, r=8, target_modules={'down_proj'}, exclude_modules=None, lora_alpha=45.254833995939045, lora_dropout=0.0, fan_in_fan_out=False, bias='none', use_rslora=False, modules_to_save=None, init_lora_weights=True, layers_to_transform=None, layers_pattern=None, rank_pattern={}, alpha_pattern={}, megatron_config=None, megatron_core='megatron.core', trainable_token_indices=None, loftq_config={}, eva_config=None, corda_config=None, use_dora=False, layer_replication=None, runtime_config=LoraRuntimeConfig(ephemeral_gpu_offload=False), lora_bias=False), extra_modules=None, base_hidden_size=3072, layer_indices=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25], device='cuda:2'), feature_sizes=({'down_proj': 9216}, {'down_proj': 3072}), aggregator_config=AggregatorConfig(aggregator_type='perceiver', num_layers=26, num_modules=1, num_extra_modules=0, output_size=512, feature_size=3072, pooling_type='mean', num_latent_factor=8, lora_r=8, per_rank_gen=True, n_latent_queries=8, num_blocks=9, num_self_attn_per_block=0, shared_weights=False, layer_to_layer_ctx_encoder=True))
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: ctx_model_name: mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:36:31 DEBUG: base_model.config._attn_implementation: flash_attention_2
2026-03-01 07:36:31 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2', 'quantization_config': BitsAndBytesConfig {
  "_load_in_4bit": true,
  "_load_in_8bit": false,
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_storage": "uint8",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}
}
2026-03-01 07:36:31 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:31 DEBUG: ctx_model_name: mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:36:31 DEBUG: base_model.config._attn_implementation: flash_attention_2
2026-03-01 07:36:31 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2', 'quantization_config': BitsAndBytesConfig {
  "_load_in_4bit": true,
  "_load_in_8bit": false,
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_storage": "uint8",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}
}
2026-03-01 07:36:31 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:31 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:32 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:32 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:32 DEBUG: ctx_model_name: mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:36:32 DEBUG: base_model.config._attn_implementation: flash_attention_2
2026-03-01 07:36:32 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2', 'quantization_config': BitsAndBytesConfig {
  "_load_in_4bit": true,
  "_load_in_8bit": false,
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_storage": "uint8",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}
}
2026-03-01 07:36:32 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:32 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:32 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:32 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:32 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: Applying LoRA forward to mlp.down_proj
2026-03-01 07:36:33 DEBUG: ctx_model_name: mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:36:33 DEBUG: base_model.config._attn_implementation: flash_attention_2
2026-03-01 07:36:33 DEBUG: Model init kwargs: {'pretrained_model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'device_map': 'cuda', 'torch_dtype': torch.bfloat16, 'trust_remote_code': True, 'attn_implementation': 'flash_attention_2', 'quantization_config': BitsAndBytesConfig {
  "_load_in_4bit": true,
  "_load_in_8bit": false,
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_storage": "uint8",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}
}
2026-03-01 07:36:33 INFO: Loading multimodal mistralai/Ministral-3-3B-Instruct-2512-BF16 on CPU to extract language model...
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/config.json HTTP/1.1" 307 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/config.json HTTP/1.1" 200 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /mistralai/Ministral-3-3B-Instruct-2512-BF16/resolve/main/generation_config.json HTTP/1.1" 307 0
2026-03-01 07:36:33 DEBUG: https://huggingface.co:443 "HEAD /api/resolve-cache/models/mistralai/Ministral-3-3B-Instruct-2512-BF16/ecc3ba8b43a45610e709327c049d24b009bfec88/generation_config.json HTTP/1.1" 200 0
2026-03-01 07:36:39 DEBUG: peft_weights: {'down_proj': {'lora_dropout': Identity(), 'lora_A': Linear(in_features=9216, out_features=8, bias=False), 'lora_B': Linear(in_features=8, out_features=3072, bias=False)}}
2026-03-01 07:36:39 DEBUG: ModulatedPretrainedModel(
  (base_model): PeftModel(
    (base_model): LoraModel(
      (model): MistralForCausalLM(
        (model): MistralModel(
          (embed_tokens): Embedding(131072, 3072)
          (layers): ModuleList(
            (0-25): 26 x MistralDecoderLayer(
              (self_attn): MistralAttention(
                (q_proj): Linear(in_features=3072, out_features=4096, bias=False)
                (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (o_proj): Linear(in_features=4096, out_features=3072, bias=False)
              )
              (mlp): MistralMLP(
                (gate_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (up_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (down_proj): lora.Linear(
                  (base_layer): Linear(in_features=9216, out_features=3072, bias=False)
                  (lora_dropout): ModuleDict(
                    (default): Identity()
                  )
                  (lora_A): ModuleDict(
                    (default): Linear(in_features=9216, out_features=8, bias=False)
                  )
                  (lora_B): ModuleDict(
                    (default): Linear(in_features=8, out_features=3072, bias=False)
                  )
                  (lora_embedding_A): ParameterDict()
                  (lora_embedding_B): ParameterDict()
                  (lora_magnitude_vector): ModuleDict()
                )
                (act_fn): SiLU()
              )
              (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
              (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
            )
          )
          (norm): MistralRMSNorm((3072,), eps=1e-05)
          (rotary_emb): MistralRotaryEmbedding()
        )
        (lm_head): Linear(in_features=3072, out_features=131072, bias=False)
      )
    )
  )
  (hypernet): HyperLoRA(
    (aggregator): Perceiver(
      (perceiver): Idefics2Perceiver(
        (modality_projection): Idefics2MLP(
          (gate_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (up_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (down_proj): Linear(in_features=12288, out_features=512, bias=False)
          (act_fn): SiLU()
        )
        (encoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0-8): 9 x Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
        (decoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0): Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
      )
    )
    (layers): Sequential(
      (0): ResMLPBlockPerLayer(
        (layers): Sequential(
          (0): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (1): EinMix('bs n_layers n_modules r d_in -> bs n_layers n_modules r d_hid', 'n_layers d_in d_hid', 'n_layers d_hid', n_layers=26, d_in=512, d_hid=2048)
          (2): SiLU()
          (3): EinMix('bs n_layers n_modules r d_hid -> bs n_layers n_modules r d_out', 'n_layers d_hid d_out', 'n_layers d_out', n_layers=26, d_hid=2048, d_out=512)
          (4): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
    (bias_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x9216 (cuda:3)])
    (bias_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x3072 (cuda:3)])
    (scaler_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:3)])
    (scaler_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:3)])
    (head): EinMix('bs n_layers n_modules r d_latent -> bs n_layers n_modules r d_lora', 'n_layers d_latent d_lora', n_layers=26, d_latent=512, r=8, d_lora=12288)
  )
  (ctx_encoder): PerLayerActivations(
    (base_model): MistralModel(
      (embed_tokens): Embedding(131072, 3072)
      (layers): ModuleList(
        (0-24): 25 x MistralDecoderLayer(
          (self_attn): MistralAttention(
            (q_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
            (k_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (v_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (o_proj): Linear4bit(in_features=4096, out_features=3072, bias=False)
          )
          (mlp): MistralMLP(
            (gate_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (up_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (down_proj): Linear4bit(in_features=9216, out_features=3072, bias=False)
            (act_fn): SiLU()
          )
          (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
          (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
        )
      )
      (norm): MistralRMSNorm((3072,), eps=1e-05)
      (rotary_emb): MistralRotaryEmbedding()
    )
  )
)
2026-03-01 07:36:39 DEBUG: Trainable model parameters:
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.modality_projection.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.modality_projection.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.modality_projection.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.latents_q, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.encoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.latents_q, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.aggregator.perceiver.decoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.0.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.0.bias, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.1.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.1.bias, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.3.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.3.bias, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.4.weight, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.layers.0.layers.4.bias, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.bias_A.down_proj, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.bias_B.down_proj, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.scaler_A.down_proj, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.scaler_B.down_proj, dtype:torch.float32
2026-03-01 07:36:39 DEBUG: hypernet.head.weight, dtype:torch.float32
2026-03-01 07:36:39 INFO: trainable params: 360,224,160 || all params: 5,649,495,456 || trainable%: 6.3762
2026-03-01 07:36:39 INFO: Loading dataset...
2026-03-01 07:36:40 DEBUG: peft_weights: {'down_proj': {'lora_dropout': Identity(), 'lora_A': Linear(in_features=9216, out_features=8, bias=False), 'lora_B': Linear(in_features=8, out_features=3072, bias=False)}}
2026-03-01 07:36:40 DEBUG: ModulatedPretrainedModel(
  (base_model): PeftModel(
    (base_model): LoraModel(
      (model): MistralForCausalLM(
        (model): MistralModel(
          (embed_tokens): Embedding(131072, 3072)
          (layers): ModuleList(
            (0-25): 26 x MistralDecoderLayer(
              (self_attn): MistralAttention(
                (q_proj): Linear(in_features=3072, out_features=4096, bias=False)
                (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (o_proj): Linear(in_features=4096, out_features=3072, bias=False)
              )
              (mlp): MistralMLP(
                (gate_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (up_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (down_proj): lora.Linear(
                  (base_layer): Linear(in_features=9216, out_features=3072, bias=False)
                  (lora_dropout): ModuleDict(
                    (default): Identity()
                  )
                  (lora_A): ModuleDict(
                    (default): Linear(in_features=9216, out_features=8, bias=False)
                  )
                  (lora_B): ModuleDict(
                    (default): Linear(in_features=8, out_features=3072, bias=False)
                  )
                  (lora_embedding_A): ParameterDict()
                  (lora_embedding_B): ParameterDict()
                  (lora_magnitude_vector): ModuleDict()
                )
                (act_fn): SiLU()
              )
              (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
              (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
            )
          )
          (norm): MistralRMSNorm((3072,), eps=1e-05)
          (rotary_emb): MistralRotaryEmbedding()
        )
        (lm_head): Linear(in_features=3072, out_features=131072, bias=False)
      )
    )
  )
  (hypernet): HyperLoRA(
    (aggregator): Perceiver(
      (perceiver): Idefics2Perceiver(
        (modality_projection): Idefics2MLP(
          (gate_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (up_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (down_proj): Linear(in_features=12288, out_features=512, bias=False)
          (act_fn): SiLU()
        )
        (encoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0-8): 9 x Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
        (decoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0): Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
      )
    )
    (layers): Sequential(
      (0): ResMLPBlockPerLayer(
        (layers): Sequential(
          (0): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (1): EinMix('bs n_layers n_modules r d_in -> bs n_layers n_modules r d_hid', 'n_layers d_in d_hid', 'n_layers d_hid', n_layers=26, d_in=512, d_hid=2048)
          (2): SiLU()
          (3): EinMix('bs n_layers n_modules r d_hid -> bs n_layers n_modules r d_out', 'n_layers d_hid d_out', 'n_layers d_out', n_layers=26, d_hid=2048, d_out=512)
          (4): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
    (bias_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x9216 (cuda:1)])
    (bias_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x3072 (cuda:1)])
    (scaler_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:1)])
    (scaler_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:1)])
    (head): EinMix('bs n_layers n_modules r d_latent -> bs n_layers n_modules r d_lora', 'n_layers d_latent d_lora', n_layers=26, d_latent=512, r=8, d_lora=12288)
  )
  (ctx_encoder): PerLayerActivations(
    (base_model): MistralModel(
      (embed_tokens): Embedding(131072, 3072)
      (layers): ModuleList(
        (0-24): 25 x MistralDecoderLayer(
          (self_attn): MistralAttention(
            (q_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
            (k_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (v_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (o_proj): Linear4bit(in_features=4096, out_features=3072, bias=False)
          )
          (mlp): MistralMLP(
            (gate_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (up_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (down_proj): Linear4bit(in_features=9216, out_features=3072, bias=False)
            (act_fn): SiLU()
          )
          (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
          (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
        )
      )
      (norm): MistralRMSNorm((3072,), eps=1e-05)
      (rotary_emb): MistralRotaryEmbedding()
    )
  )
)
2026-03-01 07:36:40 DEBUG: Trainable model parameters:
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.modality_projection.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.modality_projection.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.modality_projection.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.latents_q, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.encoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.latents_q, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.aggregator.perceiver.decoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.0.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.0.bias, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.1.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.1.bias, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.3.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.3.bias, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.4.weight, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.layers.0.layers.4.bias, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.bias_A.down_proj, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.bias_B.down_proj, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.scaler_A.down_proj, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.scaler_B.down_proj, dtype:torch.float32
2026-03-01 07:36:40 DEBUG: hypernet.head.weight, dtype:torch.float32
2026-03-01 07:36:40 INFO: trainable params: 360,224,160 || all params: 5,649,495,456 || trainable%: 6.3762
2026-03-01 07:36:40 INFO: Loading dataset...
2026-03-01 07:36:41 DEBUG: peft_weights: {'down_proj': {'lora_dropout': Identity(), 'lora_A': Linear(in_features=9216, out_features=8, bias=False), 'lora_B': Linear(in_features=8, out_features=3072, bias=False)}}
2026-03-01 07:36:41 DEBUG: ModulatedPretrainedModel(
  (base_model): PeftModel(
    (base_model): LoraModel(
      (model): MistralForCausalLM(
        (model): MistralModel(
          (embed_tokens): Embedding(131072, 3072)
          (layers): ModuleList(
            (0-25): 26 x MistralDecoderLayer(
              (self_attn): MistralAttention(
                (q_proj): Linear(in_features=3072, out_features=4096, bias=False)
                (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (o_proj): Linear(in_features=4096, out_features=3072, bias=False)
              )
              (mlp): MistralMLP(
                (gate_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (up_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (down_proj): lora.Linear(
                  (base_layer): Linear(in_features=9216, out_features=3072, bias=False)
                  (lora_dropout): ModuleDict(
                    (default): Identity()
                  )
                  (lora_A): ModuleDict(
                    (default): Linear(in_features=9216, out_features=8, bias=False)
                  )
                  (lora_B): ModuleDict(
                    (default): Linear(in_features=8, out_features=3072, bias=False)
                  )
                  (lora_embedding_A): ParameterDict()
                  (lora_embedding_B): ParameterDict()
                  (lora_magnitude_vector): ModuleDict()
                )
                (act_fn): SiLU()
              )
              (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
              (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
            )
          )
          (norm): MistralRMSNorm((3072,), eps=1e-05)
          (rotary_emb): MistralRotaryEmbedding()
        )
        (lm_head): Linear(in_features=3072, out_features=131072, bias=False)
      )
    )
  )
  (hypernet): HyperLoRA(
    (aggregator): Perceiver(
      (perceiver): Idefics2Perceiver(
        (modality_projection): Idefics2MLP(
          (gate_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (up_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (down_proj): Linear(in_features=12288, out_features=512, bias=False)
          (act_fn): SiLU()
        )
        (encoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0-8): 9 x Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
        (decoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0): Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
      )
    )
    (layers): Sequential(
      (0): ResMLPBlockPerLayer(
        (layers): Sequential(
          (0): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (1): EinMix('bs n_layers n_modules r d_in -> bs n_layers n_modules r d_hid', 'n_layers d_in d_hid', 'n_layers d_hid', n_layers=26, d_in=512, d_hid=2048)
          (2): SiLU()
          (3): EinMix('bs n_layers n_modules r d_hid -> bs n_layers n_modules r d_out', 'n_layers d_hid d_out', 'n_layers d_out', n_layers=26, d_hid=2048, d_out=512)
          (4): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
    (bias_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x9216 (cuda:0)])
    (bias_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x3072 (cuda:0)])
    (scaler_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:0)])
    (scaler_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:0)])
    (head): EinMix('bs n_layers n_modules r d_latent -> bs n_layers n_modules r d_lora', 'n_layers d_latent d_lora', n_layers=26, d_latent=512, r=8, d_lora=12288)
  )
  (ctx_encoder): PerLayerActivations(
    (base_model): MistralModel(
      (embed_tokens): Embedding(131072, 3072)
      (layers): ModuleList(
        (0-24): 25 x MistralDecoderLayer(
          (self_attn): MistralAttention(
            (q_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
            (k_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (v_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (o_proj): Linear4bit(in_features=4096, out_features=3072, bias=False)
          )
          (mlp): MistralMLP(
            (gate_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (up_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (down_proj): Linear4bit(in_features=9216, out_features=3072, bias=False)
            (act_fn): SiLU()
          )
          (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
          (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
        )
      )
      (norm): MistralRMSNorm((3072,), eps=1e-05)
      (rotary_emb): MistralRotaryEmbedding()
    )
  )
)
2026-03-01 07:36:41 DEBUG: Trainable model parameters:
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.modality_projection.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.modality_projection.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.modality_projection.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.latents_q, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.encoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.latents_q, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.aggregator.perceiver.decoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.0.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.0.bias, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.1.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.1.bias, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.3.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.3.bias, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.4.weight, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.layers.0.layers.4.bias, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.bias_A.down_proj, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.bias_B.down_proj, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.scaler_A.down_proj, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.scaler_B.down_proj, dtype:torch.float32
2026-03-01 07:36:41 DEBUG: hypernet.head.weight, dtype:torch.float32
2026-03-01 07:36:41 INFO: trainable params: 360,224,160 || all params: 5,649,495,456 || trainable%: 6.3762
2026-03-01 07:36:41 INFO: Loading dataset...
2026-03-01 07:36:41 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact with split train...
2026-03-01 07:36:41 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:36:41 DEBUG: Dataset hash: 6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:36:41 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact with split train...
2026-03-01 07:36:41 DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
2026-03-01 07:36:41 DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/parquet/parquet.py HTTP/1.1" 404 0
2026-03-01 07:36:41 DEBUG: Attempting to acquire lock 139685795282080 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-762a5005adc5e6f2_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:41 DEBUG: Lock 139685795282080 acquired on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-762a5005adc5e6f2_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:41 DEBUG: Attempting to release lock 139685795282080 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-762a5005adc5e6f2_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:41 DEBUG: Lock 139685795282080 released on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-762a5005adc5e6f2_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:41 DEBUG: Attempting to acquire lock 139686235997136 on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:41 DEBUG: Lock 139686235997136 acquired on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:41 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/parquet-train-00000-00000-of-NNNNN.arrow
2026-03-01 07:36:42 DEBUG: peft_weights: {'down_proj': {'lora_dropout': Identity(), 'lora_A': Linear(in_features=9216, out_features=8, bias=False), 'lora_B': Linear(in_features=8, out_features=3072, bias=False)}}
2026-03-01 07:36:42 DEBUG: ModulatedPretrainedModel(
  (base_model): PeftModel(
    (base_model): LoraModel(
      (model): MistralForCausalLM(
        (model): MistralModel(
          (embed_tokens): Embedding(131072, 3072)
          (layers): ModuleList(
            (0-25): 26 x MistralDecoderLayer(
              (self_attn): MistralAttention(
                (q_proj): Linear(in_features=3072, out_features=4096, bias=False)
                (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
                (o_proj): Linear(in_features=4096, out_features=3072, bias=False)
              )
              (mlp): MistralMLP(
                (gate_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (up_proj): Linear(in_features=3072, out_features=9216, bias=False)
                (down_proj): lora.Linear(
                  (base_layer): Linear(in_features=9216, out_features=3072, bias=False)
                  (lora_dropout): ModuleDict(
                    (default): Identity()
                  )
                  (lora_A): ModuleDict(
                    (default): Linear(in_features=9216, out_features=8, bias=False)
                  )
                  (lora_B): ModuleDict(
                    (default): Linear(in_features=8, out_features=3072, bias=False)
                  )
                  (lora_embedding_A): ParameterDict()
                  (lora_embedding_B): ParameterDict()
                  (lora_magnitude_vector): ModuleDict()
                )
                (act_fn): SiLU()
              )
              (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
              (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
            )
          )
          (norm): MistralRMSNorm((3072,), eps=1e-05)
          (rotary_emb): MistralRotaryEmbedding()
        )
        (lm_head): Linear(in_features=3072, out_features=131072, bias=False)
      )
    )
  )
  (hypernet): HyperLoRA(
    (aggregator): Perceiver(
      (perceiver): Idefics2Perceiver(
        (modality_projection): Idefics2MLP(
          (gate_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (up_proj): Linear(in_features=3072, out_features=12288, bias=False)
          (down_proj): Linear(in_features=12288, out_features=512, bias=False)
          (act_fn): SiLU()
        )
        (encoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0-8): 9 x Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
        (decoder): Idefics2PerceiverResampler(
          (layers): ModuleList(
            (0): Idefics2PerceiverLayer(
              (input_latents_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (input_context_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (self_attn): Idefics2PerceiverFlashAttention2(
                (q_proj): Linear(in_features=512, out_features=2048, bias=False)
                (k_proj): Linear(in_features=512, out_features=512, bias=False)
                (v_proj): Linear(in_features=512, out_features=512, bias=False)
                (o_proj): Linear(in_features=2048, out_features=512, bias=False)
              )
              (post_attention_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (pre_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (post_ff_layernorm): Idefics2RMSNorm((512,), eps=1e-06)
              (mlp): Idefics2MLP(
                (gate_proj): Linear(in_features=512, out_features=2048, bias=False)
                (up_proj): Linear(in_features=512, out_features=2048, bias=False)
                (down_proj): Linear(in_features=2048, out_features=512, bias=False)
                (act_fn): SiLU()
              )
            )
          )
          (layernorm): Idefics2RMSNorm((512,), eps=1e-06)
        )
      )
    )
    (layers): Sequential(
      (0): ResMLPBlockPerLayer(
        (layers): Sequential(
          (0): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (1): EinMix('bs n_layers n_modules r d_in -> bs n_layers n_modules r d_hid', 'n_layers d_in d_hid', 'n_layers d_hid', n_layers=26, d_in=512, d_hid=2048)
          (2): SiLU()
          (3): EinMix('bs n_layers n_modules r d_hid -> bs n_layers n_modules r d_out', 'n_layers d_hid d_out', 'n_layers d_out', n_layers=26, d_hid=2048, d_out=512)
          (4): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        )
      )
    )
    (bias_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x9216 (cuda:2)])
    (bias_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 26x8x3072 (cuda:2)])
    (scaler_A): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:2)])
    (scaler_B): ParameterDict(  (down_proj): Parameter containing: [torch.cuda.FloatTensor of size 1x26x8x1 (cuda:2)])
    (head): EinMix('bs n_layers n_modules r d_latent -> bs n_layers n_modules r d_lora', 'n_layers d_latent d_lora', n_layers=26, d_latent=512, r=8, d_lora=12288)
  )
  (ctx_encoder): PerLayerActivations(
    (base_model): MistralModel(
      (embed_tokens): Embedding(131072, 3072)
      (layers): ModuleList(
        (0-24): 25 x MistralDecoderLayer(
          (self_attn): MistralAttention(
            (q_proj): Linear4bit(in_features=3072, out_features=4096, bias=False)
            (k_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (v_proj): Linear4bit(in_features=3072, out_features=1024, bias=False)
            (o_proj): Linear4bit(in_features=4096, out_features=3072, bias=False)
          )
          (mlp): MistralMLP(
            (gate_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (up_proj): Linear4bit(in_features=3072, out_features=9216, bias=False)
            (down_proj): Linear4bit(in_features=9216, out_features=3072, bias=False)
            (act_fn): SiLU()
          )
          (input_layernorm): MistralRMSNorm((3072,), eps=1e-05)
          (post_attention_layernorm): MistralRMSNorm((3072,), eps=1e-05)
        )
      )
      (norm): MistralRMSNorm((3072,), eps=1e-05)
      (rotary_emb): MistralRotaryEmbedding()
    )
  )
)
2026-03-01 07:36:42 DEBUG: Trainable model parameters:
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.modality_projection.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.modality_projection.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.modality_projection.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.latents_q, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.1.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.2.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.3.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.4.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.5.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.6.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.7.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layers.8.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.encoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.latents_q, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_latents_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.input_context_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.q_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.k_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.v_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.self_attn.o_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_attention_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.pre_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.post_ff_layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.gate_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.up_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layers.0.mlp.down_proj.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.aggregator.perceiver.decoder.layernorm.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.0.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.0.bias, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.1.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.1.bias, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.3.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.3.bias, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.4.weight, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.layers.0.layers.4.bias, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.bias_A.down_proj, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.bias_B.down_proj, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.scaler_A.down_proj, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.scaler_B.down_proj, dtype:torch.float32
2026-03-01 07:36:42 DEBUG: hypernet.head.weight, dtype:torch.float32
2026-03-01 07:36:42 INFO: trainable params: 360,224,160 || all params: 5,649,495,456 || trainable%: 6.3762
2026-03-01 07:36:42 INFO: Loading dataset...
2026-03-01 07:36:42 DEBUG: Attempting to acquire lock 139686235969840 on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:42 DEBUG: Lock 139686235969840 acquired on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:42 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/dataset_info.json
2026-03-01 07:36:42 DEBUG: Attempting to release lock 139686235969840 on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:42 DEBUG: Lock 139686235969840 released on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:42 DEBUG: Attempting to release lock 139686235997136 on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:42 DEBUG: Lock 139686235997136 released on /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:42 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpk4k_bx3k
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmqh0fdl2
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5ev5nt2l
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl3u8qcze
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp955q7d_m
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphnx4y4fu
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdcod07ir
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpeh4_rlw8
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_ttimp10
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqwllehwu
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpjqoirp6i
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpy3l5ijbs
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp918hybwf
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3cdjve0x
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvv7hga3l
2026-03-01 07:36:43 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0fl1mfun
2026-03-01 07:36:43 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqb868vd9
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp65tjtref
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp1lqy4yf4
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpt757lukz
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnc_ac_uo
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2capd1v5
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0c8z2hfo
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpif47kd8g
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqzuoc1p8
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpisy3ub56
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpjt_zvonq
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnbucm5kv
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5zetnimh
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpa_9r71ma
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpzyv814yq
2026-03-01 07:36:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpa8cvpkmb
2026-03-01 07:36:46 INFO: Constructing and tokenizing self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact with train split...
2026-03-01 07:36:46 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpw6ug3yq0
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpwaphsctc
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpm9zsp179
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2fj41itp
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpslue_tbi
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxlwveidn
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpkmzebevq
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpz_evn60g
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpc0itm11j
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpf2cdh978
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp1m1uaak8
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp267r0738
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9ytffcdl
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpx196rtrq
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3iy1r0t6
2026-03-01 07:36:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnvcg09gf
2026-03-01 07:36:47 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnokqfto0
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpr_m9eh3c
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmppy5yvyqz
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp69qu_x44
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptgwas8c6
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpgr60zu90
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpm4ag3hb9
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpihx6kl3j
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvm227iyb
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmtl79nm_
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpq50rpoiu
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxm923tck
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpib1gb9dw
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdbz80v6f
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp66nriwpk
2026-03-01 07:36:49 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpp2d8lxmm
2026-03-01 07:36:49 INFO: Chunking context with split_ctx_kwargs={'max_chunk_len': 262144, 'min_chunk_len': -1, 'num_chunk_probs': None, 'max_num_split': None, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'is_train': True}
2026-03-01 07:36:49 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9_lk_l21
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqa0bvh8y
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5crb344i
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptyczya43
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpgtqwyo71
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqp89gf9v
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxxp50t8g
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmplx5_7gks
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpx_pchpqb
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpusr7o1gt
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpepgevjqt
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpsh__mlf0
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpn8n6l_zg
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxlwvrf64
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdhxqbwmn
2026-03-01 07:36:50 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpw306ppm1
2026-03-01 07:36:50 INFO: Num samples after ctx chunking: 500
2026-03-01 07:36:50 INFO: Avg. num chunks per ctx: 1.0
2026-03-01 07:36:50 INFO: Split too long QAs with split_qa_kwargs={'max_qas_len': 2048, 'max_qas_per_sample': 1}
2026-03-01 07:36:50 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3518
2026-03-01 07:36:53 DEBUG: Longest new qas len: 287
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpw9cf9esn
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3523
2026-03-01 07:36:53 DEBUG: Longest new qas len: 290
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_jqopvgt
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3728
2026-03-01 07:36:53 DEBUG: Longest new qas len: 288
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphbi5tu83
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3322
2026-03-01 07:36:53 DEBUG: Longest new qas len: 286
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpprp2mu0g
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3432
2026-03-01 07:36:53 DEBUG: Longest new qas len: 289
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpwslz6ty5
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3575
2026-03-01 07:36:53 DEBUG: Longest new qas len: 292
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5xs7ee4z
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3552
2026-03-01 07:36:53 DEBUG: Longest new qas len: 287
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmppx7i2bk_
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3632
2026-03-01 07:36:53 DEBUG: Longest new qas len: 286
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpzalx82q_
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3600
2026-03-01 07:36:53 DEBUG: Longest new qas len: 295
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3427
2026-03-01 07:36:53 DEBUG: Longest new qas len: 292
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpthyuuc7d
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp23_k61zm
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3673
2026-03-01 07:36:53 DEBUG: Longest new qas len: 289
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptwlwwu8e
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3623
2026-03-01 07:36:53 DEBUG: Longest new qas len: 289
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8f7y72tb
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3647
2026-03-01 07:36:53 DEBUG: Longest new qas len: 290
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpa42ddjv6
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3711
2026-03-01 07:36:53 DEBUG: Longest new qas len: 288
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_46v39xq
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3601
2026-03-01 07:36:53 DEBUG: Longest new qas len: 289
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_m2aw9qn
2026-03-01 07:36:53 DEBUG: Longest old qas len: 3590
2026-03-01 07:36:53 DEBUG: Longest new qas len: 289
2026-03-01 07:36:53 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvf4t0g4k
2026-03-01 07:36:54 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-762a5005adc5e6f2/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpahol_22p
2026-03-01 07:36:54 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00000-of-00016.arrow
2026-03-01 07:36:54 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00001-of-00016.arrow
2026-03-01 07:36:54 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00002-of-00016.arrow
2026-03-01 07:36:54 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00003-of-00016.arrow
2026-03-01 07:36:54 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00004-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00005-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00006-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00007-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00008-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00009-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00010-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00011-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00012-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00013-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00014-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/data-00015-of-00016.arrow
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/state.json
2026-03-01 07:36:55 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/dataset_info.json
2026-03-01 07:36:55 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact with split train...
2026-03-01 07:36:55 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:36:55 DEBUG: Dataset hash: aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:36:55 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact with split train...
2026-03-01 07:36:55 DEBUG: Resetting dropped connection: s3.amazonaws.com
2026-03-01 07:36:55 DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/parquet/parquet.py HTTP/1.1" 404 0
2026-03-01 07:36:55 DEBUG: Attempting to acquire lock 139686236225152 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-f10d8cf6f9d5634c_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:55 DEBUG: Lock 139686236225152 acquired on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-f10d8cf6f9d5634c_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:55 DEBUG: Attempting to release lock 139686236225152 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-f10d8cf6f9d5634c_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:55 DEBUG: Lock 139686236225152 released on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-f10d8cf6f9d5634c_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:36:55 DEBUG: Attempting to acquire lock 139686235916848 on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:55 DEBUG: Lock 139686235916848 acquired on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:55 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/parquet-train-00000-00000-of-NNNNN.arrow
2026-03-01 07:36:55 DEBUG: Attempting to acquire lock 139685794949936 on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:55 DEBUG: Lock 139685794949936 acquired on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:55 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/dataset_info.json
2026-03-01 07:36:55 DEBUG: Attempting to release lock 139685794949936 on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:55 DEBUG: Lock 139685794949936 released on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:36:55 DEBUG: Attempting to release lock 139686235916848 on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:55 DEBUG: Lock 139686235916848 released on /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:36:55 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpaxnzrbgj
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpr5fpi9fv
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpfog1bqh4
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9igjbhxl
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl7nwiheu
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpjqyz0za6
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8giq_jtr
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpor71gmtk
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvhaansev
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptl4lmgbo
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnk7p1bwr
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6co28ey0
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpsb056phb
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmppzba7flv
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpotiw4nbu
2026-03-01 07:36:56 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpe12qpkoe
2026-03-01 07:36:56 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpuywd87zq
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3vc9mtt6
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpkm1lcpix
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpzxbioe_9
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqb7t5c_f
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvlxp0vaz
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp7fgc2hxq
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpb8k355lu
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxenws0zt
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpf9jrwxr3
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl_7rmayz
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp7v_f20_q
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpo_fm5sgb
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqnwl135e
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpctin5jur
2026-03-01 07:36:58 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp1rj39_vm
2026-03-01 07:36:59 INFO: Constructing and tokenizing self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact with train split...
2026-03-01 07:36:59 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpf30tns80
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_3bc1dnz
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpd6f5dlq3
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpd4sfzjcx
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpuus6ssun
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpb5re4x4y
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphxm5yalb
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6y41o_qe
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3y8wc0zk
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpa7axbg5c
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp00to00fp
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphecfik2t
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptv5oapxa
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpgkoccv7j
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp1yjyii5x
2026-03-01 07:37:00 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpkhgry99r
2026-03-01 07:37:00 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:01 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp4taka59w
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5d2jp1rx
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpw2dd55r5
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptpl4t5gr
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpork4uhhs
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6kg_w_kd
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0dbzt_7m
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpissom02p
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpfe37pg4c
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpkuesj2wf
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqncx6ua7
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpp8xqlbdk
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxh2e15ho
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp07tyinze
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpja8lmwyb
2026-03-01 07:37:02 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpssemmqlq
2026-03-01 07:37:02 INFO: Chunking context with split_ctx_kwargs={'max_chunk_len': 262144, 'min_chunk_len': -1, 'num_chunk_probs': None, 'max_num_split': None, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'is_train': True}
2026-03-01 07:37:02 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpsaoz5if6
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnush18v2
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpw91lmlh5
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0h7dhonv
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdxikxk3p
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpiyw_wldx
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvy9ten1p
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0gxbjjd1
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8abh6l9y
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvz13c5rb
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp99e0kejr
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpa2rxpf5z
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpno_vp__v
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpwhwdhbui
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2wk50a41
2026-03-01 07:37:03 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpm1jp4lzu
2026-03-01 07:37:04 INFO: Num samples after ctx chunking: 1500
2026-03-01 07:37:04 INFO: Avg. num chunks per ctx: 1.0
2026-03-01 07:37:04 INFO: Split too long QAs with split_qa_kwargs={'max_qas_len': 2048, 'max_qas_per_sample': 1}
2026-03-01 07:37:04 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:05 DEBUG: Longest old qas len: 880
2026-03-01 07:37:05 DEBUG: Longest new qas len: 283
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp7w9u4nol
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1367
2026-03-01 07:37:05 DEBUG: Longest new qas len: 284
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp571lxgip
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1595
2026-03-01 07:37:05 DEBUG: Longest new qas len: 285
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcib1xa78
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1127
2026-03-01 07:37:05 DEBUG: Longest new qas len: 284
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmplsv51fz_
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1308
2026-03-01 07:37:05 DEBUG: Longest new qas len: 294
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl6cisanz
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1226
2026-03-01 07:37:05 DEBUG: Longest new qas len: 274
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpoq7we8zw
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1465
2026-03-01 07:37:05 DEBUG: Longest new qas len: 290
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpb5_iswgo
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1304
2026-03-01 07:37:05 DEBUG: Longest new qas len: 284
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpal1hvl29
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1055
2026-03-01 07:37:05 DEBUG: Longest new qas len: 286
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmnlpj8di
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1429
2026-03-01 07:37:05 DEBUG: Longest new qas len: 289
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpseihastx
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1868
2026-03-01 07:37:05 DEBUG: Longest new qas len: 286
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9t99356m
2026-03-01 07:37:05 DEBUG: Longest old qas len: 1417
2026-03-01 07:37:05 DEBUG: Longest new qas len: 289
2026-03-01 07:37:05 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpow6gbalq
2026-03-01 07:37:06 DEBUG: Longest old qas len: 3278
2026-03-01 07:37:06 DEBUG: Longest new qas len: 286
2026-03-01 07:37:06 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdb62bvsl
2026-03-01 07:37:06 DEBUG: Longest old qas len: 1519
2026-03-01 07:37:06 DEBUG: Longest new qas len: 284
2026-03-01 07:37:06 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpuhh6egw_
2026-03-01 07:37:06 DEBUG: Longest old qas len: 3462
2026-03-01 07:37:06 DEBUG: Longest new qas len: 283
2026-03-01 07:37:06 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8p_p4obs
2026-03-01 07:37:06 DEBUG: Longest old qas len: 2276
2026-03-01 07:37:06 DEBUG: Longest new qas len: 295
2026-03-01 07:37:06 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcqdc49pn
2026-03-01 07:37:07 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-f10d8cf6f9d5634c/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcmpvj0p0
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00000-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00001-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00002-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00003-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00004-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00005-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00006-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00007-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00008-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00009-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00010-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00011-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00012-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00013-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00014-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/data-00015-of-00016.arrow
2026-03-01 07:37:07 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/state.json
2026-03-01 07:37:08 DEBUG: open file: /app/data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc/dataset_info.json
2026-03-01 07:37:08 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact with split train...
2026-03-01 07:37:08 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:08 DEBUG: Dataset hash: 1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:08 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact with split train...
2026-03-01 07:37:08 DEBUG: Resetting dropped connection: s3.amazonaws.com
2026-03-01 07:37:08 DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/parquet/parquet.py HTTP/1.1" 404 0
2026-03-01 07:37:08 DEBUG: Attempting to acquire lock 139686235997136 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-70bbd87b83938459_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:08 DEBUG: Lock 139686235997136 acquired on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-70bbd87b83938459_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:08 DEBUG: Attempting to release lock 139686235997136 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-70bbd87b83938459_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:08 DEBUG: Lock 139686235997136 released on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-70bbd87b83938459_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:08 DEBUG: Attempting to acquire lock 139686236225056 on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:08 DEBUG: Lock 139686236225056 acquired on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:08 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/parquet-train-00000-00000-of-NNNNN.arrow
2026-03-01 07:37:08 DEBUG: Attempting to acquire lock 139685794948784 on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:08 DEBUG: Lock 139685794948784 acquired on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:08 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/dataset_info.json
2026-03-01 07:37:08 DEBUG: Attempting to release lock 139685794948784 on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:08 DEBUG: Lock 139685794948784 released on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:08 DEBUG: Attempting to release lock 139686236225056 on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:08 DEBUG: Lock 139686236225056 released on /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:08 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvqo_4a49
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpu4b097ok
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpjp5ur92j
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3hkl_vul
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2x3vnfxp
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl4aaqtba
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpsxo00_xm
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_op69021
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpkcdqu7c_
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnetmmfxx
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpd5la1pvy
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmvq_k34u
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmph6uov79j
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmucvu1kx
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmplzmi612x
2026-03-01 07:37:09 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp29jhx7nn
2026-03-01 07:37:09 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:11 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3f89i4yl
2026-03-01 07:37:11 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphjvv4_3n
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp4c8juqmp
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_3c64azp
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9p8mp4bl
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp1fxe3_rx
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp4iysdt9k
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_4hkvif4
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpp8ex5dmh
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpbdk0kdzp
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmnr1di08
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpucuquaaf
2026-03-01 07:37:12 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcjek5dk0
2026-03-01 07:37:13 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpabc85a_g
2026-03-01 07:37:13 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpgzlayiry
2026-03-01 07:37:13 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmprvoker2e
2026-03-01 07:37:14 INFO: Constructing and tokenizing self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact with train split...
2026-03-01 07:37:14 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:14 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp64zqa31j
2026-03-01 07:37:14 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp33zzfj2r
2026-03-01 07:37:14 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpz1yknpw9
2026-03-01 07:37:14 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpc8lpbpb7
2026-03-01 07:37:14 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmprkl18d1m
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpgjy1c897
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpv1gsc9oa
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpv19879_f
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2ck4rxhh
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvjq1m6m6
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp52lmfhan
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpogk3vsyn
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmplccgjnki
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0cnzdmuu
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpt7bo5xtx
2026-03-01 07:37:15 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnfszow3v
2026-03-01 07:37:15 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:17 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpolvgn0ud
2026-03-01 07:37:17 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptujzn74p
2026-03-01 07:37:17 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpq5_w2nsq
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphmkdsi7_
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9uwwckee
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmplp_zf8wr
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpk95r4q9z
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp4qy4mrdz
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2uw93o40
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxzolo2pd
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpusgj5p_4
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmph54gmgo_
2026-03-01 07:37:18 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdpyql6lc
2026-03-01 07:37:19 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpa00_fck6
2026-03-01 07:37:19 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp7i9lnotn
2026-03-01 07:37:19 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdj4e5vj2
2026-03-01 07:37:20 INFO: Chunking context with split_ctx_kwargs={'max_chunk_len': 262144, 'min_chunk_len': -1, 'num_chunk_probs': None, 'max_num_split': None, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'is_train': True}
2026-03-01 07:37:20 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5_7qbutq
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqls9bssu
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6cl1ccfi
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpt4dgykz2
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxuj79ihf
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpd51sorli
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp4q6fajle
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcw1wou2j
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphxi5uniw
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6unceb_3
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpyta0_1gf
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpigztrtzk
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0gqqoxpn
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmppzhr9sox
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpihqjzij7
2026-03-01 07:37:20 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp64i6djyv
2026-03-01 07:37:21 INFO: Num samples after ctx chunking: 1410
2026-03-01 07:37:21 INFO: Avg. num chunks per ctx: 1.0
2026-03-01 07:37:21 INFO: Split too long QAs with split_qa_kwargs={'max_qas_len': 2048, 'max_qas_per_sample': 1}
2026-03-01 07:37:21 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:23 DEBUG: Longest old qas len: 1782
2026-03-01 07:37:23 DEBUG: Longest new qas len: 301
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_mthgjt1
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2398
2026-03-01 07:37:23 DEBUG: Longest new qas len: 304
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpg42qwrvu
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2162
2026-03-01 07:37:23 DEBUG: Longest new qas len: 275
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp7idns403
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2292
2026-03-01 07:37:23 DEBUG: Longest new qas len: 290
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpgqwggpyw
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2125
2026-03-01 07:37:23 DEBUG: Longest new qas len: 280
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0ynnrz4n
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2526
2026-03-01 07:37:23 DEBUG: Longest new qas len: 307
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpc1py5fek
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2369
2026-03-01 07:37:23 DEBUG: Longest new qas len: 282
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9nflkv_2
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2508
2026-03-01 07:37:23 DEBUG: Longest new qas len: 278
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpfmj5cd2b
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2317
2026-03-01 07:37:23 DEBUG: Longest new qas len: 282
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptx27h9fb
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2704
2026-03-01 07:37:23 DEBUG: Longest new qas len: 279
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpyk3h9wqm
2026-03-01 07:37:23 DEBUG: Longest old qas len: 2151
2026-03-01 07:37:23 DEBUG: Longest new qas len: 281
2026-03-01 07:37:23 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpmyfrt0me
2026-03-01 07:37:24 DEBUG: Longest old qas len: 2454
2026-03-01 07:37:24 DEBUG: Longest new qas len: 282
2026-03-01 07:37:24 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9tuaolj2
2026-03-01 07:37:24 DEBUG: Longest old qas len: 2419
2026-03-01 07:37:24 DEBUG: Longest new qas len: 277
2026-03-01 07:37:24 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxoxn1vf4
2026-03-01 07:37:25 DEBUG: Longest old qas len: 2729
2026-03-01 07:37:25 DEBUG: Longest new qas len: 285
2026-03-01 07:37:25 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_bttsj5v
2026-03-01 07:37:25 DEBUG: Longest old qas len: 2808
2026-03-01 07:37:25 DEBUG: Longest new qas len: 292
2026-03-01 07:37:25 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpk05lyop2
2026-03-01 07:37:25 DEBUG: Longest old qas len: 2766
2026-03-01 07:37:25 DEBUG: Longest new qas len: 291
2026-03-01 07:37:25 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9r2wiba1
2026-03-01 07:37:26 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-70bbd87b83938459/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpg4b6kmcv
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00000-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00001-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00002-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00003-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00004-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00005-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00006-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00007-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00008-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00009-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00010-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00011-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00012-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00013-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00014-of-00016.arrow
2026-03-01 07:37:27 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/data-00015-of-00016.arrow
2026-03-01 07:37:28 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/state.json
2026-03-01 07:37:28 DEBUG: open file: /app/data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde/dataset_info.json
2026-03-01 07:37:28 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact with split train...
2026-03-01 07:37:28 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:28 DEBUG: Dataset hash: 4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:28 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact with split train...
2026-03-01 07:37:28 DEBUG: Resetting dropped connection: s3.amazonaws.com
2026-03-01 07:37:28 DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/parquet/parquet.py HTTP/1.1" 404 0
2026-03-01 07:37:28 DEBUG: Attempting to acquire lock 139685795277040 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-5b001ed83121d69f_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:28 DEBUG: Lock 139685795277040 acquired on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-5b001ed83121d69f_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:28 DEBUG: Attempting to release lock 139685795277040 on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-5b001ed83121d69f_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:28 DEBUG: Lock 139685795277040 released on /root/.cache/huggingface/datasets/_root_.cache_huggingface_datasets_parquet_default-5b001ed83121d69f_0.0.0_9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.lock
2026-03-01 07:37:28 DEBUG: Attempting to acquire lock 139685795278816 on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:28 DEBUG: Lock 139685795278816 acquired on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:28 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/parquet-train-00000-00000-of-NNNNN.arrow
2026-03-01 07:37:28 DEBUG: Attempting to acquire lock 139685795639888 on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:28 DEBUG: Lock 139685795639888 acquired on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:28 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete/dataset_info.json
2026-03-01 07:37:28 DEBUG: Attempting to release lock 139685795639888 on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:28 DEBUG: Lock 139685795639888 released on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d.incomplete_info.lock
2026-03-01 07:37:28 DEBUG: Attempting to release lock 139685795278816 on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:28 DEBUG: Lock 139685795278816 released on /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d_builder.lock
2026-03-01 07:37:28 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_p51ylop
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdk6ctv35
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpwcos8jlm
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpnogjuiwy
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvkmfa3ja
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp0_ot6t3o
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6rg4bqi7
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmps5dg5s1y
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpf4br94q6
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpammg9jrz
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcf4kjzt6
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3oq0j2yw
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdaffu432
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpujjs6y3j
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmps7sp84px
2026-03-01 07:37:29 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp60s6pvff
2026-03-01 07:37:30 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:32 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5p2u_y51
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxhtdm3dm
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpo08ixsry
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpjpqh6_ad
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_697dw1k
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmprrdbpsgw
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpy6gx1g4d
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpspc5ki3y
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpu22rdt8a
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvsb_84wy
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpva9zybo_
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp69ahbc7r
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5qh5p8tl
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqzwlxfoh
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpur2_nfr3
2026-03-01 07:37:33 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9gjn0r44
2026-03-01 07:37:34 INFO: Constructing and tokenizing self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact with train split...
2026-03-01 07:37:34 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpy8c3i72h
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpwhnyf2y8
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpn1ap0rxg
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpg2_84rmz
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpopy54qfw
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpbwo7ziq6
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9q5sjg2f
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpvgfr_rrj
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpysdrr_sn
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpz6kajj41
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmprl02x1hi
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpxjhvvx8a
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpz1vb867v
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8xj7tnvf
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpqlfbx856
2026-03-01 07:37:35 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpe2hmyj32
2026-03-01 07:37:35 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:38 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2jxjvde2
2026-03-01 07:37:38 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpz0hk3j90
2026-03-01 07:37:38 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp5ni4qo3g
2026-03-01 07:37:38 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpt0hiyar_
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpim914stz
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpake4qt9e
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpq9c5dk07
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp87k8qf19
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmprevqrzqk
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpju297tk6
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2d4yjiw4
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpo3bbb19e
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpbneerfzj
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpia1dhqqu
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpadf_mayt
2026-03-01 07:37:39 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8wos812m
2026-03-01 07:37:39 INFO: Chunking context with split_ctx_kwargs={'max_chunk_len': 262144, 'min_chunk_len': -1, 'num_chunk_probs': None, 'max_num_split': None, 'model_name_or_path': 'mistralai/Ministral-3-3B-Instruct-2512', 'is_train': True}
2026-03-01 07:37:39 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmprzgnmo5d
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpbc24d2dg
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp6ckwax19
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpbqtk69w7
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp91n83zs8
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2f7suj_v
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp3s7l6mje
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8tveej9t
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp4cgr5vdc
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpp1_1_ozp
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpr2ov7xby
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpcp1bkupr
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp_cdn44bj
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmptasemkzy
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpbhu_aglr
2026-03-01 07:37:40 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpphl_da09
2026-03-01 07:37:41 INFO: Num samples after ctx chunking: 1500
2026-03-01 07:37:41 INFO: Avg. num chunks per ctx: 1.0
2026-03-01 07:37:41 INFO: Split too long QAs with split_qa_kwargs={'max_qas_len': 2048, 'max_qas_per_sample': 1}
2026-03-01 07:37:41 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:44 DEBUG: Longest old qas len: 9357
2026-03-01 07:37:44 DEBUG: Longest new qas len: 299
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmphidkk_ml
2026-03-01 07:37:44 DEBUG: Longest old qas len: 5350
2026-03-01 07:37:44 DEBUG: Longest new qas len: 285
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpz6qx8ih0
2026-03-01 07:37:44 DEBUG: Longest old qas len: 8814
2026-03-01 07:37:44 DEBUG: Longest new qas len: 309
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmplfqdbbcl
2026-03-01 07:37:44 DEBUG: Longest old qas len: 7685
2026-03-01 07:37:44 DEBUG: Longest new qas len: 295
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp9w2579uv
2026-03-01 07:37:44 DEBUG: Longest old qas len: 5258
2026-03-01 07:37:44 DEBUG: Longest new qas len: 296
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpq5db94jp
2026-03-01 07:37:44 DEBUG: Longest old qas len: 11734
2026-03-01 07:37:44 DEBUG: Longest new qas len: 293
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2ohc8g3h
2026-03-01 07:37:44 DEBUG: Longest old qas len: 8862
2026-03-01 07:37:44 DEBUG: Longest new qas len: 294
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl8q7xvqg
2026-03-01 07:37:44 DEBUG: Longest old qas len: 4676
2026-03-01 07:37:44 DEBUG: Longest new qas len: 314
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp68a1i0ty
2026-03-01 07:37:44 DEBUG: Longest old qas len: 7880
2026-03-01 07:37:44 DEBUG: Longest new qas len: 291
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpflqh7jeg
2026-03-01 07:37:44 DEBUG: Longest old qas len: 7534
2026-03-01 07:37:44 DEBUG: Longest new qas len: 294
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp8zrwxvjw
2026-03-01 07:37:44 DEBUG: Longest old qas len: 8252
2026-03-01 07:37:44 DEBUG: Longest new qas len: 316
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpyfr0w97c
2026-03-01 07:37:44 DEBUG: Longest old qas len: 10129
2026-03-01 07:37:44 DEBUG: Longest new qas len: 289
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpl9l544nc
2026-03-01 07:37:44 DEBUG: Longest old qas len: 8674
2026-03-01 07:37:44 DEBUG: Longest new qas len: 310
2026-03-01 07:37:44 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpkvm3s970
2026-03-01 07:37:45 DEBUG: Longest old qas len: 12616
2026-03-01 07:37:45 DEBUG: Longest new qas len: 294
2026-03-01 07:37:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmp2s0xkx5n
2026-03-01 07:37:45 DEBUG: Longest old qas len: 10678
2026-03-01 07:37:45 DEBUG: Longest new qas len: 335
2026-03-01 07:37:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpdru3pmrs
2026-03-01 07:37:45 DEBUG: Longest old qas len: 23952
2026-03-01 07:37:45 DEBUG: Longest new qas len: 292
2026-03-01 07:37:45 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmppe78vu9e
2026-03-01 07:37:46 DEBUG: open file: /root/.cache/huggingface/datasets/parquet/default-5b001ed83121d69f/0.0.0/9c460aabd2aa27d1496e5e38d2060760561f0ac2cd6a110134eefa5b3f153b8d/tmpuv4gtb3r
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00000-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00001-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00002-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00003-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00004-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00005-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00006-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00007-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00008-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00009-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00010-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00011-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00012-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00013-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00014-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/data-00015-of-00016.arrow
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/state.json
2026-03-01 07:37:47 DEBUG: open file: /app/data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc/dataset_info.json
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact with split train...
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact with split train...
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_0.0/pwc_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 INFO: train_ds: {'self_gen/pwc_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 7411
}), 'self_gen/squad_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 8234
}), 'self_gen/ropes_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 10924
}), 'self_gen/drop_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 23060
})}
2026-03-01 07:37:48 INFO: Total samples before packing: 49629
2026-03-01 07:37:48 INFO: Packing dataset
2026-03-01 07:37:48 INFO: Packing ds 3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603 with max_packed_inp_len=2048 and max_packed_ctx_len=2048
2026-03-01 07:37:48 DEBUG: Dataset probabilities: [0.14932801386286243, 0.1659110600656874, 0.2201132402426001, 0.4646476858288501]
2026-03-01 07:37:48 DEBUG: Dataset hash: 6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:37:48 DEBUG: Dataset hash: 6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:37:48 DEBUG: Dataset hash: 6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Dataset hash: aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:37:48 DEBUG: Dataset hash: aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/squad_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Dataset hash: aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/aefdafa463891ee24497e27fcf45df3a17c5e64328df4633119d6896a7c7b2cc
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Dataset hash: 1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:48 DEBUG: Dataset hash: 1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/ropes_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Dataset hash: 1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/1a2a921994c6d114ed31261cb190f9d77e3a4ab618f5ccf0c287962bbe044cde
2026-03-01 07:37:48 DEBUG: Dataset hash: 4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:48 DEBUG: Dataset hash: 4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:48 INFO: train_ds: {'self_gen/pwc_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 7411
}), 'self_gen/squad_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 8234
}), 'self_gen/ropes_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 10924
}), 'self_gen/drop_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 23060
})}
2026-03-01 07:37:48 INFO: train_ds: {'self_gen/pwc_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 7411
}), 'self_gen/squad_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 8234
}), 'self_gen/ropes_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 10924
}), 'self_gen/drop_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 23060
})}
2026-03-01 07:37:48 INFO: Loading dataset self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact with split train...
2026-03-01 07:37:48 DEBUG: Tokenizing dataset with kwargs: {"ds_name": "self_gen/mistralai/Ministral-3-3B-Instruct-2512_temp_0.0_closed_qa_prob_1.0/drop_compact", "split": "train", "add_negative_prompt": false, "max_qas_len": 2048, "max_qas_per_sample": 1, "base_model_max_len": 262144, "ctx_model_max_len": 262144, "add_ctx_to_chat": false, "max_ctx_chunk_len": 262144, "min_ctx_chunk_len": -1, "num_chunk_probs": null, "max_ctx_chunk_num": null, "need_ctx_ids": true, "max_new_tokens": 256, "set_format": null}mistralai/Ministral-3-3B-Instruct-2512mistralai/Ministral-3-3B-Instruct-2512
2026-03-01 07:37:48 DEBUG: Dataset hash: 4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:48 INFO: Loaded tokenized dataset from data/processed_datasets/4c12bdc7a868c367d0c4a0840a2a4ae66c4913a50b487d03c2e4f62fb159befc
2026-03-01 07:37:48 INFO: train_ds: {'self_gen/pwc_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 7411
}), 'self_gen/squad_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 8234
}), 'self_gen/ropes_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 10924
}), 'self_gen/drop_compact': Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels'],
    num_rows: 23060
})}
2026-03-01 07:37:50 INFO: Train dataset length: 49915
2026-03-01 07:37:50 WARNING: Setting TOKENIZERS_PARALLELISM=false for forked processes.
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9134708180147059,
  'avg_inp_packing_efficiency': 0.37944814644607844,
  'context_ids_length_stats': { 'avg': 1870.7882352941176,
                                'max': 2048,
                                'min': 686,
                                'std': 159.62277301661334},
  'input_ids_length_stats': { 'avg': 777.1098039215686,
                              'max': 1387,
                              'min': 120,
                              'std': 226.72151809351539},
  'original_samples': 1664,
  'packed_samples': 255}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9012186608840305,
  'avg_inp_packing_efficiency': 0.36906636169201523,
  'context_ids_length_stats': { 'avg': 1845.6958174904944,
                                'max': 2048,
                                'min': 382,
                                'std': 198.38014804511099},
  'input_ids_length_stats': { 'avg': 755.8479087452472,
                              'max': 1796,
                              'min': 56,
                              'std': 263.97920125143435},
  'original_samples': 1664,
  'packed_samples': 263}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9034063697318008,
  'avg_inp_packing_efficiency': 0.36842597581417624,
  'context_ids_length_stats': { 'avg': 1850.176245210728,
                                'max': 2048,
                                'min': 678,
                                'std': 182.83736353567207},
  'input_ids_length_stats': { 'avg': 754.536398467433,
                              'max': 1591,
                              'min': 127,
                              'std': 252.97404122436973},
  'original_samples': 1664,
  'packed_samples': 261}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9022021111641222,
  'avg_inp_packing_efficiency': 0.36282465112118323,
  'context_ids_length_stats': { 'avg': 1847.7099236641222,
                                'max': 2048,
                                'min': 435,
                                'std': 182.57024115379812},
  'input_ids_length_stats': { 'avg': 743.0648854961833,
                              'max': 1530,
                              'min': 173,
                              'std': 219.34351998483572},
  'original_samples': 1663,
  'packed_samples': 262}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9023754323711832,
  'avg_inp_packing_efficiency': 0.36756582478530536,
  'context_ids_length_stats': { 'avg': 1848.0648854961833,
                                'max': 2047,
                                'min': 949,
                                'std': 170.54667263126998},
  'input_ids_length_stats': { 'avg': 752.7748091603054,
                              'max': 1541,
                              'min': 175,
                              'std': 227.6116093456857},
  'original_samples': 1664,
  'packed_samples': 262}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9070696149553571,
  'avg_inp_packing_efficiency': 0.38252379402281744,
  'context_ids_length_stats': { 'avg': 1857.6785714285713,
                                'max': 2048,
                                'min': 329,
                                'std': 194.15986054171503},
  'input_ids_length_stats': { 'avg': 783.4087301587301,
                              'max': 1974,
                              'min': 111,
                              'std': 246.18154435482606},
  'original_samples': 1664,
  'packed_samples': 252}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9037331835183398,
  'avg_inp_packing_efficiency': 0.37749230815637064,
  'context_ids_length_stats': { 'avg': 1850.8455598455598,
                                'max': 2048,
                                'min': 1304,
                                'std': 156.12070647398934},
  'input_ids_length_stats': { 'avg': 773.1042471042471,
                              'max': 1620,
                              'min': 254,
                              'std': 253.31150558604907},
  'original_samples': 1664,
  'packed_samples': 259}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8965645405783582,
  'avg_inp_packing_efficiency': 0.3619913129664179,
  'context_ids_length_stats': { 'avg': 1836.1641791044776,
                                'max': 2046,
                                'min': 840,
                                'std': 170.56504838855122},
  'input_ids_length_stats': { 'avg': 741.3582089552239,
                              'max': 1855,
                              'min': 269,
                              'std': 235.18764826261523},
  'original_samples': 1664,
  'packed_samples': 268}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9042618588147411,
  'avg_inp_packing_efficiency': 0.37933617094123506,
  'context_ids_length_stats': { 'avg': 1851.9282868525897,
                                'max': 2046,
                                'min': 810,
                                'std': 160.50680080369318},
  'input_ids_length_stats': { 'avg': 776.8804780876494,
                              'max': 1536,
                              'min': 172,
                              'std': 234.60495246793192},
  'original_samples': 1664,
  'packed_samples': 251}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9073082593628404,
  'avg_inp_packing_efficiency': 0.38516080982490275,
  'context_ids_length_stats': { 'avg': 1858.1673151750972,
                                'max': 2048,
                                'min': 996,
                                'std': 156.64492805140995},
  'input_ids_length_stats': { 'avg': 788.8093385214008,
                              'max': 1551,
                              'min': 200,
                              'std': 246.41783547547544},
  'original_samples': 1664,
  'packed_samples': 257}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9052413881515444,
  'avg_inp_packing_efficiency': 0.3762782034266409,
  'context_ids_length_stats': { 'avg': 1853.934362934363,
                                'max': 2047,
                                'min': 1046,
                                'std': 170.83227887740486},
  'input_ids_length_stats': { 'avg': 770.6177606177606,
                              'max': 1384,
                              'min': 292,
                              'std': 233.47432256772922},
  'original_samples': 1664,
  'packed_samples': 259}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9056345037116859,
  'avg_inp_packing_efficiency': 0.37671927382662834,
  'context_ids_length_stats': { 'avg': 1854.7394636015326,
                                'max': 2048,
                                'min': 843,
                                'std': 180.0894018458153},
  'input_ids_length_stats': { 'avg': 771.5210727969348,
                              'max': 1552,
                              'min': 184,
                              'std': 240.22032618305383},
  'original_samples': 1664,
  'packed_samples': 261}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9120862154150198,
  'avg_inp_packing_efficiency': 0.39182157855731226,
  'context_ids_length_stats': { 'avg': 1867.9525691699605,
                                'max': 2047,
                                'min': 683,
                                'std': 159.8465102566379},
  'input_ids_length_stats': { 'avg': 802.4505928853755,
                              'max': 1593,
                              'min': 227,
                              'std': 256.32404941777344},
  'original_samples': 1664,
  'packed_samples': 253}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9023265165441177,
  'avg_inp_packing_efficiency': 0.38141276041666666,
  'context_ids_length_stats': { 'avg': 1847.964705882353,
                                'max': 2048,
                                'min': 915,
                                'std': 173.60880842332972},
  'input_ids_length_stats': { 'avg': 781.1333333333333,
                              'max': 1489,
                              'min': 155,
                              'std': 239.42630342391755},
  'original_samples': 1664,
  'packed_samples': 255}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9026445163973384,
  'avg_inp_packing_efficiency': 0.37002621494771865,
  'context_ids_length_stats': { 'avg': 1848.6159695817491,
                                'max': 2047,
                                'min': 259,
                                'std': 191.94555443313988},
  'input_ids_length_stats': { 'avg': 757.8136882129278,
                              'max': 1582,
                              'min': 133,
                              'std': 242.79633511433448},
  'original_samples': 1664,
  'packed_samples': 263}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8981684846698114,
  'avg_inp_packing_efficiency': 0.36749520931603774,
  'context_ids_length_stats': { 'avg': 1839.4490566037737,
                                'max': 2048,
                                'min': 806,
                                'std': 186.28837011132134},
  'input_ids_length_stats': { 'avg': 752.6301886792453,
                              'max': 1651,
                              'min': 140,
                              'std': 234.71514456601145},
  'original_samples': 1664,
  'packed_samples': 265}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8920679804104478,
  'avg_inp_packing_efficiency': 0.3651305241371269,
  'context_ids_length_stats': { 'avg': 1826.955223880597,
                                'max': 2048,
                                'min': 438,
                                'std': 203.5986271568985},
  'input_ids_length_stats': { 'avg': 747.7873134328358,
                              'max': 1763,
                              'min': 139,
                              'std': 234.1254995521495},
  'original_samples': 1664,
  'packed_samples': 268}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9067957261029411,
  'avg_inp_packing_efficiency': 0.37969898897058824,
  'context_ids_length_stats': { 'avg': 1857.1176470588234,
                                'max': 2048,
                                'min': 1026,
                                'std': 170.1134623736116},
  'input_ids_length_stats': { 'avg': 777.6235294117647,
                              'max': 1513,
                              'min': 159,
                              'std': 233.7658612324292},
  'original_samples': 1664,
  'packed_samples': 255}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp2itoyb8r
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp8184j7ag
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9022991920152091,
  'avg_inp_packing_efficiency': 0.37188651081273766,
  'context_ids_length_stats': { 'avg': 1847.9087452471483,
                                'max': 2048,
                                'min': 336,
                                'std': 196.22463082144185},
  'input_ids_length_stats': { 'avg': 761.6235741444867,
                              'max': 1495,
                              'min': 66,
                              'std': 233.14828041611148},
  'original_samples': 1664,
  'packed_samples': 263}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp_fd2zu6e
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp40r7txx0
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9062211644931102,
  'avg_inp_packing_efficiency': 0.378296859621063,
  'context_ids_length_stats': { 'avg': 1855.9409448818897,
                                'max': 2047,
                                'min': 662,
                                'std': 154.74973591079737},
  'input_ids_length_stats': { 'avg': 774.7519685039371,
                              'max': 1695,
                              'min': 97,
                              'std': 253.8613560325324},
  'original_samples': 1663,
  'packed_samples': 254}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp91sefes8
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9141358386857708,
  'avg_inp_packing_efficiency': 0.3913834764081028,
  'context_ids_length_stats': { 'avg': 1872.1501976284585,
                                'max': 2048,
                                'min': 274,
                                'std': 170.7368163127829},
  'input_ids_length_stats': { 'avg': 801.5533596837945,
                              'max': 1797,
                              'min': 178,
                              'std': 252.68758171141513},
  'original_samples': 1663,
  'packed_samples': 253}
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9116565034589694,
  'avg_inp_packing_efficiency': 0.3670831345419847,
  'context_ids_length_stats': { 'avg': 1867.0725190839694,
                                'max': 2048,
                                'min': 1243,
                                'std': 143.16103704305164},
  'input_ids_length_stats': { 'avg': 751.7862595419847,
                              'max': 1523,
                              'min': 190,
                              'std': 236.79984809005816},
  'original_samples': 1664,
  'packed_samples': 262}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpqs98mod9
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9066110662116859,
  'avg_inp_packing_efficiency': 0.3729608177681992,
  'context_ids_length_stats': { 'avg': 1856.7394636015326,
                                'max': 2048,
                                'min': 523,
                                'std': 187.1171317909559},
  'input_ids_length_stats': { 'avg': 763.823754789272,
                              'max': 1526,
                              'min': 205,
                              'std': 235.0403885788487},
  'original_samples': 1664,
  'packed_samples': 261}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpm7nm7it0
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpa48oiof7
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpe4ndp34l
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8965102687026515,
  'avg_inp_packing_efficiency': 0.3749426639441288,
  'context_ids_length_stats': { 'avg': 1836.0530303030303,
                                'max': 2043,
                                'min': 583,
                                'std': 177.4440506953696},
  'input_ids_length_stats': { 'avg': 767.8825757575758,
                              'max': 1475,
                              'min': 165,
                              'std': 229.20537436061508},
  'original_samples': 1664,
  'packed_samples': 264}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpxw585pvc
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8990164092092803,
  'avg_inp_packing_efficiency': 0.37488717743844696,
  'context_ids_length_stats': { 'avg': 1841.185606060606,
                                'max': 2048,
                                'min': 1141,
                                'std': 167.7470803758822},
  'input_ids_length_stats': { 'avg': 767.7689393939394,
                              'max': 1568,
                              'min': 197,
                              'std': 257.8359011348626},
  'original_samples': 1664,
  'packed_samples': 264}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpdvyn_m35
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpm7fz5y11
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8987914585269516,
  'avg_inp_packing_efficiency': 0.37008269923327136,
  'context_ids_length_stats': { 'avg': 1840.724907063197,
                                'max': 2048,
                                'min': 441,
                                'std': 209.08472556631216},
  'input_ids_length_stats': { 'avg': 757.9293680297397,
                              'max': 1513,
                              'min': 29,
                              'std': 252.72490927783335},
  'original_samples': 1664,
  'packed_samples': 269}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp8d8mb_ss
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmphbcejd9u
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpwnjg24t6
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.904163021838803,
  'avg_inp_packing_efficiency': 0.37430622586872586,
  'context_ids_length_stats': { 'avg': 1851.7258687258686,
                                'max': 2047,
                                'min': 1036,
                                'std': 168.34513106473514},
  'input_ids_length_stats': { 'avg': 766.5791505791506,
                              'max': 1458,
                              'min': 179,
                              'std': 258.12134711327525},
  'original_samples': 1663,
  'packed_samples': 259}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpu1g6bza8
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmppk5eub85
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpw0_q0ff2
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9063932794401545,
  'avg_inp_packing_efficiency': 0.3869657185086873,
  'context_ids_length_stats': { 'avg': 1856.2934362934363,
                                'max': 2047,
                                'min': 1210,
                                'std': 155.6528869987609},
  'input_ids_length_stats': { 'avg': 792.5057915057915,
                              'max': 1521,
                              'min': 106,
                              'std': 243.18857374245792},
  'original_samples': 1664,
  'packed_samples': 259}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpmjtclm_o
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.8982862785680971,
  'avg_inp_packing_efficiency': 0.3721905608675373,
  'context_ids_length_stats': { 'avg': 1839.6902985074628,
                                'max': 2046,
                                'min': 990,
                                'std': 188.72748748591636},
  'input_ids_length_stats': { 'avg': 762.2462686567164,
                              'max': 1532,
                              'min': 160,
                              'std': 254.7924217038624},
  'original_samples': 1664,
  'packed_samples': 268}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpjnbkniic
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmp94jgncwu
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpzbvhzhke
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpcjv7ep3d
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpvl_9l396
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpfse7tiyl
2026-03-01 07:37:57 DEBUG: Packing stats:
{ 'avg_ctx_packing_efficiency': 0.9043967507102273,
  'avg_inp_packing_efficiency': 0.3769253817471591,
  'context_ids_length_stats': { 'avg': 1852.2045454545455,
                                'max': 2047,
                                'min': 989,
                                'std': 161.16095091724597},
  'input_ids_length_stats': { 'avg': 771.9431818181819,
                              'max': 1701,
                              'min': 184,
                              'std': 242.27979808389514},
  'original_samples': 1663,
  'packed_samples': 264}
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpouddhiep
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpbwfadx72
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpb_vq9cxi
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpusvi9wwm
2026-03-01 07:37:57 DEBUG: open file: /app/data/processed_datasets/6fd8971ee8231a124bc8a8b42791602220c980e7bb647a6e198f8fa734c7c6a4/tmpmupppngu
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00000-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00001-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00002-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00003-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00004-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00005-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00006-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00007-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00008-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00009-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00010-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00011-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00012-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00013-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00014-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00015-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00016-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00017-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00018-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00019-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00020-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00021-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00022-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00023-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00024-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00025-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00026-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00027-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00028-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/data-00029-of-00030.arrow
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/state.json
2026-03-01 07:37:59 DEBUG: open file: /app/data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603/dataset_info.json
2026-03-01 07:37:59 INFO: Packed dataset length: 7809
2026-03-01 07:37:59 INFO: Avg. # of samples per packed sequence: 6.355359200922013
2026-03-01 07:37:59 INFO: Setting per_device_train_batch_size to 1
2026-03-01 07:37:59 INFO: train_ds: Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels', 'ctx_position_ids', 'position_ids', 'n_queries', 'n_ctx_chunks'],
    num_rows: 7809
})
2026-03-01 07:37:59 INFO: val_ds: {}
2026-03-01 07:37:59 INFO: Total samples before packing: 49629
2026-03-01 07:37:59 INFO: Total samples before packing: 49629
2026-03-01 07:37:59 INFO: Compiling base_model
2026-03-01 07:37:59 INFO: Total samples before packing: 49629
2026-03-01 07:37:59 INFO: Packing dataset
2026-03-01 07:37:59 INFO: Packing dataset
2026-03-01 07:37:59 INFO: Packing dataset
2026-03-01 07:37:59 INFO: Packing ds 3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603 with max_packed_inp_len=2048 and max_packed_ctx_len=2048
2026-03-01 07:37:59 INFO: Packing ds 3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603 with max_packed_inp_len=2048 and max_packed_ctx_len=2048
2026-03-01 07:37:59 INFO: Packing ds 3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603 with max_packed_inp_len=2048 and max_packed_ctx_len=2048
2026-03-01 07:37:59 INFO: Loading a cached packed dataset for data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603
2026-03-01 07:37:59 INFO: Loading a cached packed dataset for data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603
2026-03-01 07:37:59 INFO: Loading a cached packed dataset for data/processed_datasets/packed_3ff17973ccf4107411110c2272d7cc473c4e92dbe56b53b89feeb708e860c603
2026-03-01 07:37:59 DEBUG: Starting new HTTPS connection (1): 172.20.0.1:443
2026-03-01 07:37:59 INFO: Packed dataset length: 7809
2026-03-01 07:37:59 INFO: Avg. # of samples per packed sequence: 6.355359200922013
2026-03-01 07:37:59 INFO: Setting per_device_train_batch_size to 1
2026-03-01 07:37:59 INFO: train_ds: Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels', 'ctx_position_ids', 'position_ids', 'n_queries', 'n_ctx_chunks'],
    num_rows: 7809
})
2026-03-01 07:37:59 INFO: val_ds: {}
2026-03-01 07:37:59 INFO: Compiling base_model
2026-03-01 07:37:59 INFO: Packed dataset length: 7809
2026-03-01 07:37:59 INFO: Avg. # of samples per packed sequence: 6.355359200922013
2026-03-01 07:37:59 INFO: Setting per_device_train_batch_size to 1
2026-03-01 07:37:59 INFO: train_ds: Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels', 'ctx_position_ids', 'position_ids', 'n_queries', 'n_ctx_chunks'],
    num_rows: 7809
})
2026-03-01 07:37:59 INFO: val_ds: {}
2026-03-01 07:37:59 INFO: Compiling base_model
2026-03-01 07:37:59 INFO: Packed dataset length: 7809
2026-03-01 07:37:59 INFO: Avg. # of samples per packed sequence: 6.355359200922013
2026-03-01 07:37:59 INFO: Setting per_device_train_batch_size to 1
2026-03-01 07:37:59 INFO: train_ds: Dataset({
    features: ['ctx_ids', 'input_ids', 'logprobs_vals', 'logprobs_indices', 'labels', 'ctx_position_ids', 'position_ids', 'n_queries', 'n_ctx_chunks'],
    num_rows: 7809
})
2026-03-01 07:37:59 INFO: val_ds: {}
2026-03-01 07:37:59 INFO: Compiling base_model
2026-03-01 07:37:59 DEBUG: Starting new HTTPS connection (1): 172.20.0.1:443
2026-03-01 07:37:59 DEBUG: Starting new HTTPS connection (1): 172.20.0.1:443
2026-03-01 07:37:59 DEBUG: Starting new HTTPS connection (1): 172.20.0.1:443
2026-03-01 07:38:02 DEBUG: Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/app, stdin=None, shell=False, universal_newlines=False)
2026-03-01 07:38:02 DEBUG: Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/app, stdin=None, shell=False, universal_newlines=False)
2026-03-01 07:38:02 DEBUG: Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/app, stdin=None, shell=False, universal_newlines=False)
2026-03-01 07:38:02 DEBUG: Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/app, stdin=None, shell=False, universal_newlines=False)
2026-03-01 07:38:02 DEBUG: https://huggingface.co:443 "POST /api/repos/create HTTP/1.1" 409 159
2026-03-01 07:38:02 INFO: Training with modulated model.
2026-03-01 07:38:02 INFO: Training with distillation loss. Using DistillationTrainer.
2026-03-01 07:38:02 DEBUG: https://huggingface.co:443 "POST /api/repos/create HTTP/1.1" 409 159
2026-03-01 07:38:02 INFO: Training with modulated model.
2026-03-01 07:38:02 INFO: Training with distillation loss. Using DistillationTrainer.
2026-03-01 07:38:02 DEBUG: https://huggingface.co:443 "POST /api/repos/create HTTP/1.1" 409 159
2026-03-01 07:38:02 INFO: Training with modulated model.
2026-03-01 07:38:02 INFO: Training with distillation loss. Using DistillationTrainer.
2026-03-01 07:38:03 DEBUG: Starting new HTTPS connection (1): api.wandb.ai:443
2026-03-01 07:38:03 DEBUG: https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None
2026-03-01 07:38:03 DEBUG: https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 357
2026-03-01 07:38:03 DEBUG: Popen(['git', 'cat-file', '--batch-check'], cwd=/app, stdin=<valid stream>, shell=False, universal_newlines=False)
2026-03-01 07:38:04 DEBUG: https://huggingface.co:443 "POST /api/repos/create HTTP/1.1" 409 159
2026-03-01 07:38:04 INFO: Training with modulated model.
2026-03-01 07:38:04 INFO: Training with distillation loss. Using DistillationTrainer.
2026-03-01 07:38:17 WARNING: Using position ids for resampler
2026-03-01 07:38:17 WARNING: Using position ids for resampler
2026-03-01 07:38:17 WARNING: Using position ids for resampler
2026-03-01 07:38:17 WARNING: Using position ids for resampler
2026-03-01 08:07:42 INFO: Uploading checkpoint at step 250 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 08:07:52 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 08:07:52 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 1155
2026-03-01 08:07:52 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 08:07:52 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 08:08:09 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 08:08:09 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 08:37:46 INFO: Uploading checkpoint at step 500 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 08:38:05 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 08:38:05 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 2751
2026-03-01 08:38:06 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 08:38:06 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 08:38:22 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 08:38:22 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 09:07:53 INFO: Uploading checkpoint at step 750 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 09:08:22 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 09:08:23 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 4127
2026-03-01 09:08:23 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 09:08:23 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 09:08:40 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 09:08:40 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 09:38:09 INFO: Uploading checkpoint at step 1000 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 09:38:49 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 09:38:49 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 5512
2026-03-01 09:38:49 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 09:38:49 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 09:39:06 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 09:39:06 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 10:08:38 INFO: Uploading checkpoint at step 1250 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 10:09:28 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 10:09:28 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6897
2026-03-01 10:09:28 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 10:09:28 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 10:09:48 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 10:09:48 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 10:39:26 INFO: Uploading checkpoint at step 1500 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 10:40:16 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 10:40:17 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6906
2026-03-01 10:40:17 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 10:40:17 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 10:40:37 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 10:40:37 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 11:10:11 INFO: Uploading checkpoint at step 1750 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 11:11:00 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 11:11:01 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6915
2026-03-01 11:11:01 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 11:11:01 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 11:11:20 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 11:11:20 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 11:40:53 INFO: Uploading checkpoint at step 2000 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 11:41:42 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 11:41:43 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 11:41:43 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 11:41:43 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 11:42:05 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 11:42:05 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 12:11:38 INFO: Uploading checkpoint at step 2250 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 12:12:28 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 12:12:28 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 12:12:28 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 12:12:28 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 12:12:51 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 12:12:51 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 12:42:29 INFO: Uploading checkpoint at step 2500 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 12:43:19 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 12:43:19 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 12:43:19 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 12:43:19 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 12:43:42 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 12:43:42 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 13:13:21 INFO: Uploading checkpoint at step 2750 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 13:14:11 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 13:14:11 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 13:14:11 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 13:14:11 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 13:14:35 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 13:14:35 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 13:44:08 INFO: Uploading checkpoint at step 3000 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 13:44:57 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 13:44:57 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 13:44:57 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 13:44:57 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 13:45:20 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 13:45:20 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 14:14:54 INFO: Uploading checkpoint at step 3250 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 14:15:45 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 14:15:45 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 14:15:45 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 14:15:45 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 14:16:08 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 14:16:08 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 14:45:54 INFO: Uploading checkpoint at step 3500 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 14:46:43 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 14:46:43 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 14:46:44 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 14:46:44 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 14:47:09 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 14:47:09 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 15:16:47 INFO: Uploading checkpoint at step 3750 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 15:17:37 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 15:17:37 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 15:17:37 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 15:17:37 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 15:17:58 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 15:17:58 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 15:47:30 INFO: Uploading checkpoint at step 4000 to neopolita/doc-to-lora-ministral-3b-2512...
2026-03-01 15:48:20 DEBUG: Resetting dropped connection: huggingface.co
2026-03-01 15:48:20 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 6924
2026-03-01 15:48:20 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 15:48:20 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386
2026-03-01 15:48:44 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/commit/main HTTP/1.1" 200 212
2026-03-01 15:48:44 INFO: Checkpoint uploaded to https://huggingface.co/neopolita/doc-to-lora-ministral-3b-2512
2026-03-01 15:48:46 INFO: Training run finished and saved to train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f
2026-03-01 15:48:46 INFO: Training run finished and saved to train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f
2026-03-01 15:48:46 DEBUG: Attempting to acquire lock 140060076124784 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140060076124784 acquired on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to release lock 140060076124784 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140060076124784 released on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to acquire lock 140060076129920 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140060076129920 acquired on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to release lock 140060076129920 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140060076129920 released on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to acquire lock 139678300858992 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 139678300858992 acquired on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to release lock 139678300858992 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 139678300858992 released on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to acquire lock 139678300864128 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 139678300864128 acquired on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to release lock 139678300864128 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 139678300864128 released on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 INFO: Training run finished and saved to train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f
2026-03-01 15:48:46 DEBUG: Attempting to acquire lock 140361880136256 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140361880136256 acquired on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to release lock 140361880136256 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140361880136256 released on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to acquire lock 140361880141392 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140361880141392 acquired on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Attempting to release lock 140361880141392 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:46 DEBUG: Lock 140361880141392 released on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2026-03-01 15:48:48 INFO: Training run finished and saved to train_outputs/runs/Mar01_07-36-18_j-neopolita-69a3e406dfb316ac3f7c0a80-tuu1470p-784b1-cr29x_5298c41f
2026-03-01 15:49:40 DEBUG: https://huggingface.co:443 "POST /api/models/neopolita/doc-to-lora-ministral-3b-2512/preupload/main HTTP/1.1" 200 7695
2026-03-01 15:49:40 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/revision/main?expand=xetEnabled HTTP/1.1" 200 99
2026-03-01 15:49:40 DEBUG: https://huggingface.co:443 "GET /api/models/neopolita/doc-to-lora-ministral-3b-2512/xet-write-token/main HTTP/1.1" 200 386