Ornith-1.5-35B-BF16.gguf > MTP ?
After several tests on the BF16 , i found a strange behavior on the MTP .
So, running the model with :
/path-to-llama-server/llama-server
-m /path-to-model/Ornith-1.5-35B-BF16.gguf
--mmproj /path-to-vision/Ornith-1.5-35B-A3B-mmproj-BF16.gguf
--host 0.0.0.0
--port 8080
--ctx-size 0
--temp 1.0
--top-p 0.95
--image-min-tokens 1024
--flash-attn on
--jinja
- Is loading the model:
0.00.339.609 I srv load_model: loading model '/path_to_model/Ornith-1.5-35B-BF16.gguf' - Is ignoring the MTP:
0.00.765.002 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
0.00.765.014 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
0.00.765.019 W model has unused tensor blk.40.attn_q.weight (size = 33554432 bytes) -- ignoring
0.00.765.020 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring
0.00.765.022 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring
0.00.765.026 W model has unused tensor blk.40.attn_output.weight (size = 16777216 bytes) -- ignoring
0.00.765.028 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
0.00.765.029 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
0.00.765.031 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring
0.00.765.032 W model has unused tensor blk.40.ffn_down_exps.weight (size = 536870912 bytes) -- ignoring
0.00.765.035 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 536870912 bytes) -- ignoring
0.00.765.036 W model has unused tensor blk.40.ffn_up_exps.weight (size = 536870912 bytes) -- ignoring
0.00.765.038 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring
0.00.765.039 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 2097152 bytes) -- ignoring
0.00.765.041 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 2097152 bytes) -- ignoring
0.00.765.042 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 2097152 bytes) -- ignoring
0.00.765.045 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 16777216 bytes) -- ignoring
0.00.765.047 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
0.00.765.048 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
0.00.765.052 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring - Continue loading the vision:
0.04.821.145 I cmn init: llama threadpool init, n_threads = 12
0.04.944.613 I srv load_model: loaded multimodal model, '/path_to_vision/Ornith-1.5-35B-A3B-mmproj-BF16.gguf'
0.04.964.243 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.04.966.725 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.04.966.761 I srv llama_server: model loaded
0.04.966.765 I srv llama_server: listening on http://0.0.0.0:8080 - Ready to be used. Simple prompt to model :
0.40.710.880 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.40.711.387 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.44.353.129 I slot print_timing: id 3 | task 0 | prompt eval time = 763.59 ms / 2534 tokens ( 0.30 ms per token, 3318.53 tokens per second)
0.44.353.131 I slot print_timing: id 3 | task 0 | eval time = 2877.80 ms / 504 tokens ( 5.72 ms per token, 174.79 tokens per second)
0.44.353.132 I slot print_timing: id 3 | task 0 | total time = 3641.40 ms / 3038 tokens
0.44.353.137 I slot print_timing: id 3 | task 0 | graphs reused = 501
0.44.353.325 I slot release: id 3 | task 0 | stop processing: n_tokens = 3037, truncated = 0 - Result : 174.79 tokens per second.
But if i load the model with MTP:
0.00.277.041 I srv load_model: loading model '/path_to_model/Ornith-1.5-35B-BF16.gguf'
0.41.426.217 I cmn init: llama threadpool init, n_threads = 12
0.41.474.635 I common_speculative_init_result: creating MTP draft context against the target model '/path_to_model/Ornith-1.5-35B-BF16.gguf'
0.41.902.079 I srv load_model: loaded multimodal model, '/path_to_vision/Ornith-1.5-35B-A3B-mmproj-BF16.gguf'
0.41.926.192 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.41.999.210 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.41.999.253 I srv llama_server: model loaded
0.41.999.257 I srv llama_server: listening on http://0.0.0.0:8080
Simple prompt to model:
1.09.757.609 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
1.09.758.402 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
1.13.688.890 I slot print_timing: id 3 | task 0 | n_gen = 343, tg = 113.96 t/s, tg_3s = 114.28 t/s
1.16.271.746 I slot print_timing: id 3 | task 0 | prompt eval time = 928.55 ms / 2534 tokens ( 0.37 ms per token, 2728.98 tokens per second)
1.16.271.749 I slot print_timing: id 3 | task 0 | eval time = 5583.97 ms / 636 tokens ( 8.79 ms per token, 113.72 tokens per second)
1.16.271.749 I slot print_timing: id 3 | task 0 | total time = 6512.52 ms / 3170 tokens
1.16.271.754 I slot print_timing: id 3 | task 0 | graphs reused = 419
1.16.271.756 I slot print_timing: id 3 | task 0 | draft acceptance = 0.16825 ( 213 accepted / 1266 generated), mean len = 1.50
1.16.271.922 I slot release: id 3 | task 0 | stop processing: n_tokens = 3169, truncated = 0
Result is a much lower generation speed: 113.72 tokens per second.
I have try to set --spec-draft-n-max 2 or 3 ... but result is similar, lower generation speed compared to not use MTP.
Also the draft acceptance is very low.
The tests have been done on a workstation with Ryzen9 9900X, 256Gb ram memory and 2 x RTX6000 PRO Workstation 96GB VRAM.
I am doing something wrong ?