unsloth

/

Qwen3.6-35B-A3B-MTP-GGUF

Like 940

Follow

Unsloth AI 35.9k

@Josephur can you explain how that allows other requests to use MTP? You've hard-coded no MTP into the model config for hermes.

Yes, this does work,

The crucial part is that this:

auxiliary:
vision:
extra_body:
speculative.n_max: 0

does not mean “configure this model with MTP permanently disabled.” It means, “when Hermes makes a request using its vision auxiliary model slot, add speculative.n_max: 0 to that particular HTTP request.” Hermes explicitly treats its main model and auxiliary models such as vision as separate independently configured slots. Its documentation also says auxiliary.<task>.extra_body is attached to that auxiliary request, and the OpenAI SDK turns extra_body entries into top-level JSON fields.

What is actually happening:

Suppose llama-server was started like this conceptually:

llama-server
...
--mmproj mmproj.gguf
--spec-type draft-mtp
--spec-draft-n-max 3

So its server default is MTP enabled, with up to 3 speculative tokens. Current llama.cpp still has draft-mtp as a supported speculative type and --spec-draft-n-max as the corresponding global default.

Then Hermes makes two different kinds of calls.

NORMAL HERMES REQUEST
│
│ no speculative override
▼
llama-server
server default: n_max = 3
│
▼
MTP ON

But when Hermes needs vision:

HERMES VISION AUXILIARY REQUEST
│
│ request contains:
│ "speculative.n_max": 0
▼
llama-server
server default: n_max = 3
request override: n_max = 0
│
▼
MTP OFF
for THIS request

Then the next regular request comes in without that override:

NEXT NORMAL REQUEST
│
│ no override
▼
server default: n_max = 3
│
▼
MTP ON

In current llama.cpp, each server request gets a task_params structure, and the speculative decoding configuration is part of those task parameters:

struct task_params {
...
struct common_params_sampling sampling;
struct common_params_speculative speculative;
...
};

Then, while llama-server is processing multiple generating slots, it calculates the draft-token limit separately for each slot:

const int n_draft_max = slot.get_n_draft_max();
if (n_draft_max > 0) {
...
}

So this isn't just some accidental behavior. The server architecture explicitly allows one active request/slot to have a different speculative configuration than another.

The llama.cpp documentation even shows speculative.n_max inside the parameters for individual slots.

So you could theoretically have this happening concurrently:

Slot 0: text agent request
speculative.n_max = 3
MTP ON

Slot 1: vision request
speculative.n_max = 0
MTP OFF

Slot 2: another text request
speculative.n_max = 3
MTP ON

All on the same loaded model/server.

The mmproj is still loaded into llama-server. You are not unloading the MTP head or mmproj.

You're merely telling llama.cpp:

For this generation, don't ask the MTP machinery to produce speculative tokens.

That makes sense as a workaround because multimodal processing inserts image-derived embeddings into the prompt rather than just ordinary text tokens. Current llama.cpp actually recognizes the combination of an mmproj and MTP/speculative model during server setup.

So you get:

Model weights
├── Qwen text model
├── MTP tensors/head loaded
└── mmproj loaded

For text:

text prompt
↓
Qwen
↓
MTP speculative decoding
↓
answer

For vision:

image
↓
mmproj
↓
image embeddings + text prompt
↓
Qwen
↓
ordinary autoregressive decoding
↓
answer

The MTP components remain sitting in memory, they just aren't invoked for that request.

The reply says:

“You've hard-coded no MTP into the model config for hermes.”

That would be true if you had done something like this for the overall main model:

model:
provider: custom
...
extra_body:
speculative.n_max: 0

Then every normal Hermes agent request could inherit it.

But you didn't.

You put it here:

auxiliary:
vision:
extra_body:
speculative.n_max: 0

Hermes specifically documents auxiliary jobs as independently configurable model slots, including Vision.

extra_body there isn't configuring llama.cpp globally. It's attached only to API requests made through Hermes' auxiliary.vision route. llama-server itself is still launched with MTP enabled. Vision requests override speculative.n_max to 0 for their individual request, while normal model requests omit the override and inherit the server's nonzero MTP setting.

Interestingly, Hermes did have a bug where some auxiliary.<task>.extra_body values from config.yaml were silently ignored. That bug was filed May 30, 2026 and has since been closed with a fix.

So on an older Hermes checkout, you could write:

auxiliary:
vision:
extra_body:
speculative.n_max: 0

and it might not actually reach llama.cpp.

If you enable llama.cpp's /slots and/or metrics/logging, you can watch the request parameters. Current llama.cpp's /slots output exposes speculative.n_max for each slot, and --metrics exposes speculative-draft statistics.

You should be able to observe the activity.

So yes, your workaround is mechanically sound, and better yet, current llama.cpp's per-task/per-slot implementation backs up why it works.

The really interesting implication is that you don't actually need a second Qwen server just to get stable vision. One Qwen3.6 instance can potentially serve text + MTP and vision without MTP simultaneously, with the request itself selecting whether speculative decoding runs.