llama-3.1-8b-instruct_SNI-ours-16x2-plus-generalist-lora-moe

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.5027
• Router Supervised Loss: 0.7194
• Router Load Balance Loss: 4.7610
• Router Z Loss: 3.7853

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• total_eval_batch_size: 2
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.03
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss	Load Balance Loss	Supervised Loss	Z Loss
15.6232	0.0154	100	2.1447	1.7013	2.0489	7.3349
13.4662	0.0308	200	1.8358	3.4900	1.1685	6.9430
12.5565	0.0462	300	1.7663	3.8867	1.0023	6.8321
12.9059	0.0617	400	1.7403	3.7354	1.0141	6.5768
13.0040	0.0771	500	1.7448	3.5863	1.0824	6.3713
12.2123	0.0925	600	1.7109	3.7762	1.0159	6.3466
12.2258	0.1079	700	1.6745	3.8608	0.9746	6.3237
12.2918	0.1233	800	1.6555	4.0183	0.9201	6.3320
12.6506	0.1387	900	1.6633	3.8044	0.9921	6.1764
11.8610	0.1542	1000	1.6537	3.9551	0.9743	6.2184
12.1072	0.1696	1100	1.6249	4.1888	0.8798	6.3181
11.0177	0.1850	1200	1.6279	4.0864	0.9200	6.1759
11.6769	0.2004	1300	1.6058	3.9728	0.9593	6.1361
11.0343	0.2158	1400	1.6189	4.2426	0.8534	6.1911
11.8778	0.2312	1500	1.6119	3.9320	0.9772	6.0295
12.3351	0.2466	1600	1.5961	4.1349	0.8686	6.0395
10.4334	0.2621	1700	1.5829	4.1259	0.8789	6.0559
11.5209	0.2775	1800	1.5949	3.9828	0.9480	5.9164
12.0791	0.2929	1900	1.5898	4.1122	0.8818	5.8331
11.3781	0.3083	2000	1.5814	4.0114	0.9481	5.7797
11.4371	0.3237	2100	1.5819	4.0377	0.9285	5.7558
10.9289	0.3391	2200	1.5729	4.1280	0.8972	5.7576
11.5314	0.3546	2300	1.5773	4.2842	0.8417	5.7084
10.8669	0.3700	2400	1.5678	4.2250	0.8569	5.6189
10.3766	0.3854	2500	1.5561	4.4027	0.8190	5.6738
11.5763	0.4008	2600	1.5584	4.2303	0.8578	5.5921
10.7845	0.4162	2700	1.5601	4.3294	0.8470	5.5831
10.1497	0.4316	2800	1.5608	4.3327	0.8534	5.5847
10.4831	0.4470	2900	1.5639	4.2652	0.8753	5.5136
10.7801	0.4625	3000	1.5662	4.4330	0.8180	5.5544
10.7479	0.4779	3100	1.5474	4.4952	0.7825	5.5159
11.0396	0.4933	3200	1.5607	4.4099	0.8096	5.4542
11.6400	0.5087	3300	1.5551	4.4644	0.8166	5.4356
10.8625	0.5241	3400	1.5509	4.3460	0.8360	5.3647
10.6702	0.5395	3500	1.5475	4.5231	0.8108	5.3802
10.8231	0.5550	3600	1.5522	4.4408	0.8238	5.2865
10.6207	0.5704	3700	1.5575	4.4695	0.7946	5.2747
11.0732	0.5858	3800	1.5404	4.7257	0.7215	5.3910
10.7991	0.6012	3900	1.5582	4.4051	0.8071	5.2275
12.0343	0.6166	4000	1.5391	4.5813	0.7715	5.2753
11.0641	0.6320	4100	1.5400	4.5042	0.7926	5.2278
11.3028	0.6474	4200	1.5484	4.5612	0.7708	5.2516
11.8669	0.6629	4300	1.5374	4.6484	0.7432	5.2637
9.9163	0.6783	4400	1.5390	4.6344	0.7418	5.2371
10.5348	0.6937	4500	1.5430	4.5655	0.7589	5.1622
11.2674	0.7091	4600	1.5387	4.5504	0.7798	5.1160
10.7514	0.7245	4700	1.5367	4.5458	0.7777	5.1131
11.5891	0.7399	4800	1.5371	4.4919	0.7959	5.1059
10.9617	0.7554	4900	1.5401	4.5074	0.7831	5.0999
11.5106	0.7708	5000	1.5405	4.4389	0.8076	5.0549
11.1462	0.7862	5100	1.5388	4.5169	0.7984	5.0897
11.8516	0.8016	5200	1.5351	4.5689	0.7810	5.1103
11.5424	0.8170	5300	1.5406	4.5283	0.7788	5.0760
10.7582	0.8324	5400	1.5416	4.5207	0.7789	5.0633
11.4588	0.8478	5500	1.5397	4.5252	0.7785	5.0557
11.0090	0.8633	5600	1.5388	4.5161	0.7812	5.0577
10.8893	0.8787	5700	1.5385	4.5208	0.7797	5.0507
10.2076	0.8941	5800	1.5407	4.4918	0.7878	5.0328
10.6553	0.9095	5900	1.5393	4.5185	0.7784	5.0425
11.4362	0.9249	6000	1.5383	4.5202	0.7777	5.0460
10.7311	0.9403	6100	1.5379	4.5282	0.7763	5.0600
11.3546	0.9558	6200	1.5376	4.5261	0.7760	5.0493
11.6983	0.9712	6300	1.5377	4.5190	0.7795	5.0523
10.4083	0.9866	6400	1.5381	4.5270	0.7780	5.0593
11.7906	1.0	6487	1.5382	4.5229	0.7782	5.0524
10.8294	1.0020	6500	1.5393	4.4302	0.8006	4.9811
11.8291	1.0174	6600	1.5535	4.5666	0.7647	5.0181
11.7838	1.0328	6700	1.5502	4.3838	0.8326	4.9356
10.7210	1.0483	6800	1.5318	4.6793	0.7888	4.9622
10.6957	1.0637	6900	1.5495	4.3961	0.8310	4.7859
10.0627	1.0791	7000	1.5383	4.6513	0.7562	4.9486
11.2647	1.0945	7100	1.5448	4.3967	0.8238	4.6997
10.8446	1.1099	7200	1.5478	4.5245	0.7824	4.7296
11.3806	1.1253	7300	1.5443	4.4200	0.8094	4.6557
10.9578	1.1407	7400	1.5381	4.5671	0.7684	4.6433
10.9091	1.1562	7500	1.5293	4.7389	0.7401	4.7983
10.1022	1.1716	7600	1.5236	4.7830	0.7201	4.7308
10.5942	1.1870	7700	1.5309	4.5007	0.8176	4.5444
10.5433	1.2024	7800	1.5217	4.5534	0.7931	4.5711
11.3332	1.2178	7900	1.5084	4.6348	0.7708	4.5580
11.1855	1.2332	8000	1.5212	4.4598	0.8135	4.4291
10.2708	1.2487	8100	1.5340	4.4995	0.7975	4.3767
10.3346	1.2641	8200	1.5289	4.3080	0.8660	4.2610
11.5252	1.2795	8300	1.5216	4.5079	0.8015	4.2823
10.8549	1.2949	8400	1.5367	4.5394	0.7862	4.1895
11.4078	1.3103	8500	1.5369	4.6780	0.7445	4.2274
10.6451	1.3257	8600	1.5279	4.9231	0.6734	4.2785
11.0626	1.3411	8700	1.5319	4.8115	0.7015	4.1644
10.5259	1.3566	8800	1.5196	4.8981	0.6782	4.2844
10.7047	1.3720	8900	1.5190	4.7310	0.7346	4.2221
11.5659	1.3874	9000	1.5061	4.8857	0.6805	4.2503
10.1941	1.4028	9100	1.5203	4.8367	0.7063	4.1631
10.9544	1.4182	9200	1.5190	4.7436	0.7316	4.1395
10.1546	1.4336	9300	1.5117	4.7279	0.7244	4.0631
10.6888	1.4491	9400	1.5119	4.7264	0.7406	4.0377
10.2189	1.4645	9500	1.5098	4.7562	0.7333	4.0063
11.6624	1.4799	9600	1.5149	4.7183	0.7305	4.0005
11.0747	1.4953	9700	1.5244	4.4375	0.7965	3.7650
10.0198	1.5107	9800	1.5041	4.7533	0.7142	3.8892
9.7294	1.5261	9900	1.5020	5.0568	0.6425	4.0504
12.2263	1.5415	10000	1.5152	4.6000	0.7638	3.7477
11.2295	1.5570	10100	1.5036	0.6986	4.8266	3.8638
12.4211	1.5724	10200	1.5026	0.7128	4.7729	3.8194
11.1475	1.5878	10300	1.5063	0.7279	4.7244	3.7999
10.6885	1.6032	10400	1.5089	0.7053	4.7981	3.8211
10.3050	1.6186	10500	1.5062	0.7167	4.7613	3.8073
11.0140	1.6340	10600	1.5108	0.7376	4.6783	3.7610
10.2874	1.6495	10700	1.5088	0.7195	4.7473	3.7827
10.7501	1.6649	10800	1.5055	0.7046	4.8034	3.7946
10.3340	1.6803	10900	1.5024	0.7000	4.8340	3.8170
10.2128	1.6957	11000	1.5024	0.7067	4.8058	3.7997
10.6650	1.7111	11100	1.5057	0.7232	4.7516	3.7784
10.5483	1.7265	11200	1.5011	0.7046	4.8222	3.8198
10.0481	1.7419	11300	1.5040	0.7196	4.7596	3.7796
11.3932	1.7574	11400	1.5058	0.7253	4.7338	3.7706
10.6396	1.7728	11500	1.5067	0.7228	4.7424	3.7680
10.3623	1.7882	11600	1.5021	0.7198	4.7590	3.7760
10.2864	1.8036	11700	1.5033	0.7199	4.7614	3.7771
10.6917	1.8190	11800	1.5043	0.7217	4.7528	3.7736
10.8664	1.8344	11900	1.5030	0.7227	4.7493	3.7761
10.1052	1.8499	12000	1.5034	0.7223	4.7527	3.7771
10.1901	1.8653	12100	1.5022	0.7197	4.7616	3.7799
11.5827	1.8807	12200	1.5039	0.7196	4.7605	3.7798
10.0638	1.8961	12300	1.5024	0.7175	4.7672	3.7861
10.7124	1.9115	12400	1.5029	0.7192	4.7592	3.7834
11.2595	1.9269	12500	1.5039	0.7188	4.7620	3.7820
10.1277	1.9423	12600	1.5038	0.7205	4.7563	3.7780
10.6804	1.9578	12700	1.5025	0.7168	4.7719	3.7896
10.5177	1.9732	12800	1.5036	0.7192	4.7621	3.7850
10.4430	1.9886	12900	1.5027	0.7188	4.7627	3.7890
10.5840	2.0	12974	1.5027	0.7194	4.7610	3.7853

Framework versions

• Transformers 5.9.0
• Pytorch 2.11.0+cu130
• Datasets 4.4.1
• Tokenizers 0.22.2