Is it possible? Yes. Q4_K_M should fit just fine. Give it a try. I use a RTX 5080 and gemma 12B is pure speed. One misunderstanding is getting greedy about a higher quant just to realize you didn't account for context size, KV caches, etc.
@Yoyo406 Yes — Mk2Oracle has it right. Q4_K_M (about 7.4 GB) is the sweet spot on 16 GB; it leaves room for the app + KV cache. The thing to watch (as he hinted) is context: Gemma 4 can go to 256K, but the KV cache grows with it and eats your headroom fast — so cap context around 8K and you'll be comfortable. Want more margin? Q3_K_M (about 6 GB) also runs fine. Skip Q6/Q8 on 16 GB — they fit on disk but leave nothing for context. Thanks @Mk2Oracle for the assist.
Running this model on a 4090 is insanely fast. Running it on a 4070 is still insanely fast, but context limit has to be much smaller and hits it real fast unless it's cache is heavily quantized too and then I find it breaks down/spits out hallucinations or garbled stuff way too fast.
Using Q8 w/mtp with context maxed out is great on the 4090 - but it makes me think people are expecting a lot more than they should when trying to fit models on low vram cards/systems.
I haven't attempted via system memory with this model, but I've tried with other larger models where 24gb vram wouldn't allow fits and the slowdown wasn't worth the trade off to me.
@bleaki Yeah, that's a fair read. Two things that line up with what you're seeing:
The "garbles/hallucinates fast" once you shrink context on the 4070 is usually the KV-cache quantization, not the model itself. Dropping KV to q4 is where it falls apart — try q8_0 KV cache instead (--cache-type-k q8_0 --cache-type-v q8_0). It saves almost as much memory as q4 but keeps quality basically intact, so you get more usable context on the 4070 without the garble.
On system-RAM offload: for a 12B I'd agree it's usually not worth it for interactive/agentic work — prompt processing is what tanks, and agentic loops re-read a lot of context, so the slowdown compounds. It's more defensible for a one-shot "leave it running overnight" generation than for anything interactive.
And you're right that expectations need calibrating for low-VRAM cards — a maxed 256K context on a small card was never going to be free. Q4_K_M at ~8K context is the honest sweet spot for most people. Appreciate you sharing the real numbers. 🙏
