NikiKrutan

/

Qwen3.8-27B-MTP-GGUF

Like 10

I've just tested this exact model in a 90K context coding session in Kilo Code and I have no issues. So I can't say for where the problem is. These quants are standard llama-quantize quants. Nothing special (aside from quant types allocation and imatrix).

I don't use LM Studio, only pure llama.cpp. But may I give you some advice looking at your config?

1. Unified KV cache. I believe it is for unified memory systems. Yours is obviously not.
2. Temperature 0.1 is probably not very good. Qwen recommends 0.6-1.0 from what I remember. I use 0.75 for hard agentic coding for example.
3. Max concurrent predictions. I believe it is -np in llama.cpp. If you don't need parallel inference you should set it to 1. Otherwise your effective context is 45545 tokens only (but you can inference 4 prompts in parallel, do you really need it?). May be it is your problem actually.
4. If you aim to fit the whole model to VRAM: take little lower quant, use context size of ~128K (if you need more, then even lower quant), lower ubatch (physical batch) - this will free some VRAM. I have used 3.6-27B with 3-bit quantization (I can't remember size and quant exactly) on single 16 Gb GPU with 132K context fully in VRAM for agentic coding. And 3.8 is much smarter, so good 3-bit quant should be ok. Experiment until you fit. CPU offloading is catastrophically slow for dense models.
5. q4_0 K-cache quantization is too aggressive, but you don't have other choice, so ok. Also I don't know if LM Studio ships llama.cpp compiled with -DGGML_CUDA_FA_ALL_QUANTS=ON (needed to run on CUDA with q4_1, q5_0, q5_1 cache quantizations, otherwise it falls back on CPU - very slow).
6. Set draft probability to 0.8. Then you may raise max draft tokens to 5.
7. If you after quality don't use NVFP4. They are speed/quality trade. And if you offload to CPU, then NVFP4 is poor choice for speed as well.

I think, my experience has the following:

1. You need to test your model for higher context length, test up to the limit. I have tested other models and they works fine albeit less token speed which is obvious as I have to give more ram toward that offloading more to CPU. Your model is not the first which is failing at higher context length for me, there are Q8 Quant version which are failing too. If you give LM Studio a try, the option has clarification for what they are meant for.
2. Temperature should not be higher for precision work, it should not be more than 0.5, higher the temperature, more random prediction or inference and inconsistent result which are fatal for variety of task needing precision work like debug, patching, codebase refactoring.
3. In OpenCode I have seen context token consumption and it did indeed reached to what I set it i.e. 201K without compacting so it is not 45545 context token as you think it would.
4. I will definitely try Q3 model, I think Qwen3.8 27b is capable to deliver decent results even at Q2
5. Need to check this point, what is the case with LM Studio.
6. As per my understanding, there is no guarantee that draft token will be accepted, others claims that usually 2-3 tokens are usually accepted so I have that for max tokens and rest remain as is. I will test your suggestion though.
7. NVFP4 is only for RTX50 series and its kind of architecture models, it is native nvidia FP4 which is faster than INT4. Infact, I have tested normal Q4K models in comparison which performs less compared to NVFP4, of course I have to use FP4 because I need long context length with Qwen3.8 model which eats lots of context token. More info on NVFP4: https://developers.redhat.com/articles/2026/02/04/accelerating-large-language-models-nvfp4-quantization

Thanks for your input!

Well.. Since I've started this, I have to continue =)

1. I've tested exactly this model again up to 262K until I had context overflow error. I see no problems. I use my 17300 quant on daily basis with 200K+ contexts, also no problems. Actually I can't remember that I had such problems with any quant I used (probably more than 100) other than most quants are not that good quality as advertised. May be I am just lucky. Or may be there is a problem with your setup.
2. Who told you this? Qwen's recommendations:

Best Practices
To achieve optimal performance, we recommend the following settings:

Sampling Parameters: We suggest using the following sets of sampling parameters:

Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

But if you are comfortable with temperature 0.1 that's ok. You tested it and it is best for you. No problems with that.
From my practice even 0.6 gives bad results on most hard logic tasks where model needs to think through the problem. While 1.0 gives bad random decisions and code (too high).
This is for this exact Qwen3.8-27B model. Another model - another story.

3. This doesn't make sense. If you set 182181 how it could show 201K? Well, I don't use OpenCode, but I use Kilo which is based on OpenCode. And token count is exactly what I see in llama.cpp logs. -np in llama.cpp divides context into slots for parallel inference. Since LM Studio uses llama.cpp as its engine it should be the same. But, well, they may use some fork or their own patch, so actually I don't know. Since I am not going to use LM Studio I won't investigate this.
4. More quantization = less quality. It depends on task. Some tasks can be done with Q2, some can't (or with lots of mistakes). Actually quant is not the same model as BF16/Q8_0. That's why I post graphs with RMS dp metric. It shows how much token probability distribution is away from the full model. But if you use near 0 temperature same top p metric is where you can easily see the difference. It shows exactly how often quantized model chooses different token than full model. For Q2 quants it is >10% tokens for temp=0. Anyway offloading to CPU is a pain on long contexts. Prefill speed is awful. So you should definitely try smaller quant. If it fits your VRAM and suits your tasks well you get job done with decent speed.
6. If you don't set p-min, then your draft will always generate max tokens. Even if it is totally sure that it is crap. Eating your speed. If you set p-min you can let draft generate more if it is sure enough or cut the crap immediately not wasting compute time. On my hardware and my tasks for this model p-min 0.8 and max 5 are optimal. I benched it and now I use it. It may differ for you.
7. Well, I know what is NVFP4, thanks. What is bad about it that quantisation quality is considerably lower than IQ4_XS/Q4_K. So I don't use it, since I need all the quality for my hard tasks (while I have Blackwell GPUs). And note that when offloaded on CPU it is worse because AMD CPU don't have native support for NVFP4. So it should only be used if fully loaded on Blackwell architecture and if speed is primary goal.

Also please note that to get to this quant quality level I had to do lots of research so I have better understanding of most aspects than average web poster.