I need advice
Good evening,
I am trying to run Qwen3.8-27B locally on my PC with an RTX 5070 12 GB GPU, 32 GB DDR5 RAM, and an Intel Core Ultra 5 245KF CPU on Windows 11. I mainly want to use it for coding help and debugging in my studies and projects.
Could you tell me which GGUF quantization is best for a 12 GB card, for example something like IQ2, IQ3, or Q3? What is a realistic maximum context length for this hardware, and is 8K a good default or can I go up to 16K with some trade-offs?
How much of the model should I try to keep in VRAM versus system RAM, and roughly how many GB of VRAM should be reserved for the KV cache and runtime? Are there any recommended GPU layer counts or LM Studio GPU offload settings for this setup?
Do you recommend using a quantized KV cache like q8_0 for K and V on this hardware, and are there any other flags or settings that help with stability and speed on a 12 GB GPU?
My goal is a stable and reasonably fast setup for programming tasks, not necessarily the maximum context. If a smaller model like a 14B to 16B Q4 would give a better experience than Qwen3.8-27B Q3 on this hardware, I am open to that advice as well.
Thank you.
At 12 gb its a hard sell, from unsloths UD-Q2_K_XL, you will have 2.17 gigs left.
If using llama.cpp, any quantized cache (q8_0 downwards) has the haddamard transform applied to it if using flash_attention, wich you should use.
So try using k q5_1, v q5_1. You will need to keep about 100mb of free memory for compute cache.
Dont offload anything to cpu.
Im assuming you have a 3060? If you can get your hands on a gtx1070 8gb you can pool the vram together and run q4km at kv q8 easily.
If you have the pcie slot (anything above 2x pcie gen 3) its worthwhile to get a cheaper card like that. Its going to run at about 600t/s infill and 20 t/s decode at the current llama.cpp main branch.
I know these numbers are correct because its what i run, rtx 3060 12gb and gtx 1070 8gb. Its plenty useful for programming.
I have RTX5080, Ryzen 9950X3D and 64 GB DDR5 ram and I am using NVFP4 version with MTP "qwen3.8-27b-nvfp4-mtp-low.gguf" on LM Studio with K and V Cache set to Q4, 45 layers to GPU and 150K context length. I'm also using custom chat template https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates and MTP draft token set to 3. It do take all 16GB, plus 25GB system ram plus load all cpu 32 threads.
I'm getting mere 8 tokens/s. The model eats up too much token in reasoning and thinking. In codebase refactoring where it has to read and analysis multiple files to refactor or create new codes, it may eat 80K token easy and won't generate a single line of code but it depends on the task for sure.
I don't think, it is feasible with this model and I will suggest you to try other models Qwen3.6 27B is also excellent model. But first try openai/gpt-oss-20b model which may be enough for your task because this model Q4 is around 12GB, with right context, KV Cache Quants and context length, it will be speed monster for variety of coding task.
But check and compare it will other models. Smaller model like 14B or 9B were not good in my experience but may well be good enough for you so you can check Qwen3 14B too.
Compare and find the best fit.
In summary, Qwen3.8 27B will be out of context or painfully slow or won't fit at all on your system.
Use q4 for kv, look for a distill into 9b, try escha now or bonsai when it releases
Wait for Bonsai version then you should be able to fit that in 12 GB
sorry for late but i think you should probably check this model page: https://www.whichllmmodel.com/models/alibaba-cloud-qwen-qwen38-27b