Estimates
Hello! I noticed one of my GGUF conversions here, and while the estimate is informative, I think there's room for a more cost-effective option. For Cyber-frost-3.8 (https://huggingface.co/freakyskittle/CYBER-FROST-3.8-GGUF), you can actually run it on an RTX A6000 using custom kernels (https://github.com/Zapdev-labs/a6000-custom-kernels). Additionally, the RTX A6000 can be rented for just 0.62 cents an hour on RunPod, which might be a more budget-friendly solution.
Thanks Jackson, useful to hear from the person who made the conversions.
The calculator sizes a model with all of its weights in GPU memory, so it never offers a card smaller than the file. The smallest build in your repo is Q2_K at 80.1 GB (MXFP4_MOE is 96.6 GB, Q4_K_M 111.4 GB), and one RTX A6000 has 48 GB. Running it on a single A6000 means keeping a good part of the experts in system RAM, which can work for an MoE but is a different speed trade, and that is what custom kernels like yours are for. The calculator doesn't model offloading today.
If you pick Q2_K in the Weights menu, two A6000s (96 GB) hold the file with some room for context. Live A6000 rates right now: $0.28/hr on Vast.ai, $0.33/hr on RunPod community and $0.59/hr on RunPod secure, all on https://fastgpu.co/gpus/a6000?utm_source=huggingface&utm_medium=space&utm_campaign=gpu-cost-calculator (refreshed every 20 minutes).
Understood, thanks for the clarification! That makes sense. I'll go ahead and close the issue.