llama-3.1-8b-instruct_LBox-10x2-plus-generalist-lora-moe

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0368
• Router Supervised Loss: 0.7111
• Router Load Balance Loss: 3.1977
• Router Z Loss: 1.7449

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• total_eval_batch_size: 2
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.03
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss	Supervised Loss	Load Balance Loss	Z Loss
9.2157	0.3266	500	1.2531	0.6028	3.5467	4.5532
9.3029	0.6533	1000	1.1633	0.5261	3.7052	4.0598
8.8652	0.9799	1500	1.1281	0.6396	3.4210	3.4540
9.0210	1.3064	2000	1.0920	0.7815	3.0531	3.0569
9.4279	1.6330	2500	1.0720	0.5916	3.4984	2.6309
9.1875	1.9597	3000	1.0659	0.7595	3.1193	2.5054
8.8329	2.2861	3500	1.0522	0.5281	3.6835	2.2354
8.9111	2.6128	4000	1.0467	0.7138	3.2494	2.0908
7.8045	2.9394	4500	1.0469	0.6881	3.2693	2.0084
7.6419	3.2659	5000	1.0448	0.8645	2.9374	2.0494
7.9167	3.5925	5500	1.0425	0.6480	3.3811	1.8848
8.1925	3.9192	6000	1.0374	0.7151	3.1922	1.7747
7.7807	4.2456	6500	1.0373	0.6394	3.3661	1.7425
7.9786	4.5723	7000	1.0373	0.6628	3.3196	1.7385
8.0582	4.8989	7500	1.0369	0.7080	3.2079	1.7431
7.5259	5.0	7655	1.0368	0.7111	3.1977	1.7449

Framework versions

• Transformers 5.9.0
• Pytorch 2.11.0+cu130
• Datasets 4.4.1
• Tokenizers 0.22.2