NVFP4 without gguf possible in same size and quality for vllm?
Hi this looks great work wanted to test this against sakamakismile NVFP4 which is 20.6 GB is it possible to have only NVFP4 for vllm by any chance in safetensors format. As I really need those parallel request due to nature of my work. Thank you!
My main stack, which I work on and optimize pretty much daily, is llama.cpp. That’s why this quant is specifically optimized for that platform and released as GGUF.
I have a lot of respect for vLLM and have used it myself, but it’s not my first choice for my current workflows.
I spent quite a bit of time experimenting with the quantization mix and deliberately sized this one right up to the 24 GB VRAM ceiling… including CUDA overhead on Linux and a full 256K context at parallel=1. So the ~24 GB target wasn’t accidental.
I also tested parallel=2 and parallel=4 in my actual agent workflows. On my hardware, parallel=1 consistently makes more sense: individual inference is summary faster, and running tasks sequentially ends up completing the overall workload faster than splitting the available resources across concurrent requests.
Of course, there are plenty of setups and workloads where parallelism absolutely wins. This quant simply wasn’t designed for those… it was intentionally built around llama.cpp + 24 GB VRAM + 256K context + parallel=1.
When the smaller Qwen3.8-35B MoE variant lands, I may prepare a few different quantization flavors optimized for different use cases. Seeing how popular this build has become and how many people are actually using it makes me very happy… and definitely gives me a reason to experiment with more variants
Thanks for feedback @gameonvinay
Ok thanks for the quick reply I will try this with Llama.cpp. I loved the size that 2 GB could enable me to use Dflash2 on my 5090 while still having space for large context on q8 kv cache