How about a 3-bit quant?
I need a bit more space on my machine so that the model can actually compile and test code so a model that takes pretty much the full 128GB isn't going to work for me. A 3-bit quant would leave a bit more room.
Tell me a little more about your setup. Maybe I can help. Are you running this on a headless server? Is your Strix Halo machine your actual desktop?
It's not headless. I currently use Claude to work on this (but I have also tried some of the Qwen3.*-35B derivative models and Qwen3.8-27B with some success, though the latter is slow). "This" being a compiler that compiles a sub/superset of C to hardware. This means it's got to be able to compile the C++ codebase, run simulation, synthesis and place&route (the latter can easily take 10GB on it's own, sometimes more). So I need at least 20GB (preferrably more) headroom so that the model would be able to run these things as it iterates. Either a smaller quant of 3.8-flash-next or n-grams on SSD would help.
Yeah thats tough. Halogen is a memory hog. Did you try the 27B version of halogen and find it slow? There are a few people who have this requirement. I'd love to be able to support it. Let me see what I can do.
going to try your halogen 27B tomorrow. This variant of llama.cpp apparently has the n-grams on SSD, perhaps you can borrow from it? https://github.com/Aristo94/EngramHalo.cpp
Hi! Please let me know. I posted a comment on the offloading here: https://github.com/peonist-ai/halogen-flash-server/issues/7
Qwen 3.8 Flash Next is a big model. Let me know how the 27B goes for you. I hope it works out.
I seem to be getting about 11tok/sec with your 27b (full context). Was kind of hoping for better.
This is how I ran it:
podman run --rm -p 8731:8731
--device /dev/kfd --device /dev/dri --group-add keep-groups
--security-opt seccomp=unconfined --ipc=host
--security-opt label=disable
-v /mnt/LLMs/models/halogen-models:/models:ro
-v /mnt/LLMs/models/halogen-models/tokenizer:/tokenizer:ro
ghcr.io/peonist-ai/halogen:0.1.0
Is that going to do MTP or DFLASH?
DFlash2 should be enabled by default! Seems like that decode is exactly my measured serial number. Also, I need to update the hugging face card. There are a couple patches.
Try this:
podman run --rm -p 8731:8731
--device /dev/kfd --device /dev/dri --group-add keep-groups
--security-opt seccomp=unconfined --ipc=host
-v /path/to/models:/models:ro -v /path/to/tokenizer:/tokenizer:ro
ghcr.io/peonist-ai/halogen:0.1.3
27B is definitely slower, but the quality should be pretty good. Its higher bpw than flash.