llama-3.1-8b-instruct_LBox-category-prior-8x1-lora-moe

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7146
• Router Supervised Loss: 1.2566
• Router Load Balance Loss: 2.9630
• Router Z Loss: 1.6500

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• total_eval_batch_size: 2
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.03
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss	Supervised Loss	Load Balance Loss	Z Loss
9.9457	0.1830	500	1.2682	1.0739	2.7332	5.4282
6.7027	0.3660	1000	0.8868	1.1596	2.6544	7.1435
5.8974	0.5489	1500	0.8349	0.9402	2.9654	6.0732
5.8900	0.7319	2000	0.8071	1.3818	2.9513	5.3320
6.5693	0.9149	2500	0.7883	0.9038	2.9441	3.6248
5.8019	1.0977	3000	0.7796	0.9469	2.8268	2.8833
5.7989	1.2807	3500	0.7592	1.6139	3.5352	2.9172
5.6415	1.4637	4000	0.7490	1.2382	2.8955	1.9796
5.2636	1.6467	4500	0.7454	0.8895	3.0758	1.6344
5.7957	1.8296	5000	0.7356	1.1778	2.8222	1.5490
5.8623	2.0124	5500	0.7326	0.9802	2.8715	1.4671
5.1178	2.1954	6000	0.7344	0.9339	2.9024	1.5378
6.0174	2.3784	6500	0.7272	1.1811	2.9990	1.4460
5.5216	2.5614	7000	0.7250	1.0457	2.7300	1.6281
4.8988	2.7444	7500	0.7231	0.9169	2.9111	1.6735
5.4509	2.9274	8000	0.7195	1.3227	3.0895	1.4725
4.8223	3.1102	8500	0.7198	1.0807	2.7534	1.5616
5.0465	3.2931	9000	0.7202	1.2287	2.9593	1.5258
5.8587	3.4761	9500	0.7170	1.1436	2.8002	1.5982
5.7854	3.6591	10000	0.7163	1.4184	3.2401	1.5266
5.1895	3.8421	10500	0.7156	1.3455	3.0909	1.6244
5.6037	4.0249	11000	0.7150	1.2034	2.8747	1.6527
5.1878	4.2079	11500	0.7153	1.2657	2.9666	1.6544
5.1715	4.3909	12000	0.7149	1.2851	3.0077	1.6401
5.5300	4.5738	12500	0.7148	1.3079	3.0496	1.6346
5.3011	4.7568	13000	0.7146	1.2713	2.9880	1.6303
5.1349	4.9398	13500	0.7145	1.2590	2.9670	1.6471
5.3102	5.0	13665	0.7146	1.2566	2.9630	1.6500

Framework versions

• Transformers 5.9.0
• Pytorch 2.11.0+cu130
• Datasets 4.4.1
• Tokenizers 0.22.2