VRAM on v3
Well, loaded with llama.cpp in router mode:
[55903] 0.54.803.502 I srv llama_server: listening on http://127.0.0.1:55903
[55903] cmd_child_to_router:state:{"state":"ready","payload":{"id":"DeepSeek-V4-Flash-0731-Ombro-v3c","aliases":["DeepSeek-V4-Flash-0731-Ombro-v3c"],"tags":[],"object":"model","created":1786457198,"owned_by":"llamacpp","meta":{"vocab_type":2,"n_vocab":129280,"n_ctx":1048576,"n_ctx_train":1048576,"n_embd":4096,"n_params":284334567511,"size":85796235748,"ftype":"IQ2_XXS - 2.0625 bpw"}}}
VRAM at 89.5Gb / 95.6Gb and on some long prompts, is running consistent with 82 to 85T/S:
[55903] 2.54.378.945 I slot print_timing: id 0 | task 1346 | n_decoded = 250, tg = 83.33 t/s, tg_3s = 83.33 t/s
[55903] 2.57.379.025 I slot print_timing: id 0 | task 1346 | n_decoded = 504, tg = 84.00 t/s, tg_3s = 84.66 t/s
[55903] 3.33.458.230 I slot print_timing: id 0 | task 1346 | n_decoded = 3493, tg = 83.01 t/s, tg_3s = 83.05 t/s
[55903] 3.48.500.985 I slot print_timing: id 0 | task 1346 | n_decoded = 4730, tg = 82.81 t/s, tg_3s = 82.44 t/s
I would say is not bad. Much better then previous version. Did not do anything serious tasks with this one, will try this weekend.
I am curious what you packed in the new v4.
Haha I do my best.
I am running a massive script right now to invert the iquants, first few experts are already 34-39% faster while having the exact same precision.
Basically shrinking the matrixes but near lossless compared to the original. Better precision than v3c or v1.
I found a way to balance the bpw and size of weights.
Scaled some down from 4096x2048 to 4096x1024 at first. The results were shit.
Then I realized I made a mistake and turned my algorythim around, made it sort of train itself to get as close to the matrix I recorded from the full model.
Also went back to linear algebra, inserted variable names, named each cell in those matrixes from x1, x2…. xn.
And I am running some magic over night.
While typing one expert appears to have been folded and reduce around 60% of its size and calculations at full precision lol
Nice, nice.
As a question, what calibration imatrix you are using ?
I am running atm something on a derisked version from Blackfrost, converted the safetensors to f32 (got an output file of 1,1Tb) and building a calibration imatrix now.
10 different categories with 5 to 10 sub-categories , each around 25 to 35K. If all goes well, the end would be around 2 to 3M.
Just for fun. 🥸
I made my own based on 200k tokens of different contexts,
Both chatted with the model, codex and claude code turns, hermes agent and openclaw which contain multiple python, bash, node, whatever and many researching on my own machine.
And from my own benchmark I took almost all prompts for html file writing.
But I made multiple variants of that imatrix
Hello!
I’d like to thank you for sharing this build.
This is the first custom model out of about ten (including REAP/REAM — though honestly, I’m not sure if Ombro falls into those categories) that actually runs on my llama.cpp and delivers acceptable generation speed and quality.
The main issues I encountered when running custom models from the DeepSeek-V4-Flash-0731 family:
- Fitting DeepSeek-V4-Flash-0731 into 80 GB of VRAM on an A100. For instance, the v3c build is ~85.8 GB (79.91 GiB), so it doesn’t fit entirely.
- Compatibility with llama.cpp. Many custom builds are made for the DS framework and its forks, not for native llama.cpp.
- Chat template. Not all builds work correctly with the standard chat template.
- Support for languages other than English and Chinese. This matters for my use cases.
- Heavy compression. With overly aggressive quantizations, the model starts producing nonsense from the very first prompt.
The v3c build falls just short of fitting into VRAM (about 5 GB short), so I run it with:
llama-server -m DeepSeek-V4-Flash-0731-Ombro-v3c.gguf -ngl 99 -fa 1 -c 8000 --n-cpu-moe 2
Generation speed is around 30 t/s. For comparison, models that fit entirely into VRAM give about 50 t/s.
Questions:
- Are you planning to release a smaller build, e.g., 80–81 GB GGUF?
- Can I perform the quantization myself to fit my VRAM budget (using llama-quantize or other tools)?
Thanks for your answers — I’m a beginner in this area, so please don’t be too harsh 😊
Best regards,
zion0210
I don't see why you having problems with llama.cpp. I have run both models created by Wladastic , on last llama.cpp build and i did not find or had any problems.
The support for other languages would not be so easy, would mean to train the model for that and i do not think is worth. Maybe i am wrong, but Wladastic can say more.
For chat template, i used "--jinja" and was fine.
Also it is expected that model will start to spit out garbage, when a higher compression is made. So, do not expect to obtain quality as for a MXFP4 .
With your VRAM, you should be able to run it no problem, but not with the command you using.
Try this:
/path_to_llama_build/llama-server
--model /path_to_model/DeepSeek-V4-Flash-0731-Ombro-v3c.gguf
--host 127.0.0.1
--port 8080
--jinja
--ctx-size 0
--parallel 1
--flash-attn on
--cache-type-k bf16
--cache-type-v bf16
--load-mode mmap
--no-repack
--fit on
--split-mode layer
--threads 16
--batch-size 2048
--ubatch-size 512
--ctx-checkpoints 0
--cache-ram 0
--temp 1.0
--top-p 0.95
With this i could fit the model on a RTX6000 PRO 96Gb VRAM. Running at 80 - 82T/S.
If you have any questions, shout out. Best regards.
I think they mean that other models had problems.
But yes I have a smaller version coming, I am still experimenting with it but have 20% speed increase while having better quality.
The issue were the iquants but I am still testing.
I replaced some iquants with q2k and had to do some shenanigans.
I am rebuilding a new lossless gguf though and have to generate a new imatrix.
I found that the attentions were not converting 100% accurate and I think I made a fix there.
Btw. In case you use dspark, I did not manage to fit the lossless dspark on my machine and only made a iq2xxs fit, but it made my inference only slower with acceptance of 48%.
Once I have the dspark quant tuned it should bring some extra speed, at least for me I got from 15 to 20 t/s decode although I offload 60GB to ram and have two 16GB gpus running.
I found that using Q2_0 in some weights actually kept the same overall accuracy, which will also give a bit extra speed but that required me to edit llama.cpp a bit so I trashed that idea because they do not fully support that yet.
Sorry wrong button.
Well, to be honest, was a lot of fun playing with this. Still, i do not think i would make it as my daily driver.
I have also experimented with Q2, but i found that quality loss was more extreme compared to iQ.
For some chat is ok, but for a serious task, i would go with MXFP4, otherwise the time spent hunting errors later , is not worth.
As of right now, as my daily's drivers, i use DeepSeek-V4-Flash-0731-MXFP4-BF16 and Qwen3.8-27B-BF16. When is about coding i go with Qwen, but when i need large context i go with DeepSeek. Qwen 3.8 goes down to 7 > 8 T/S generation speed if i rise the context to 1M.
About DeepSeek V4 Flash, despite all the combines i have try for a quantization, like iQ2 or iQ3, the resulted model is cannot compare in quality with MXFP4-BF16.
Unless in the future we could have more development towards smaller models > higher intelligence, for moment i will stick with what we have.
Best regards.
Fair, I saw some degregation in reasoning with normal Q2 as well.
I first tried to instead resolve iquant tables to be faster, but now have a bpw of around 2.05, anything below screws with attention.
And Q2K_S required some fixing in the quantization code, I wrote my own cpp files for that because I think the llama.cpp is a bit messy with Deepseek V4 flash.
It performs better than iQ2_XXS AND iQ3_XXS now after running some calibration over the weights.
The official code caused some weird glitches, norm didn't really norm anything, Down was retrained a bit and works better.
But the main assumption I had was: Do not look at the original safetensors or MXFP4 as 100% quality reference, but like another representation.
Also after quantization you shouldn't treat it like "How can I make it return a close enough result" but rather like "What quantization would result in another truth?".
Basically treating the quantization like new information instead of "the same information but smaller"