Can the model size be further reduced to lower initial loading overhead
Hi, I have a question about this model. The README says that qwen38-flash-next-w4b.hgn is 115.55 GiB, but the actual usage is about 124 GB. Is there any way to further reduce the model size and ease the burden during initial loading? If so, what options would you recommend? Thanks!
Great question.
Before I answer, how long is your initial load? Are you running a headless server? If you're experiencing slow load times, it could be Halogen trying to find 68GB of continuous memory blocks to allocate.
Halogen pins ~68GB in memory (model weights) then uses another 35GB for the KV cache. The biggest lever you have for controlling memory is the kv cache pool. Send an agent to investigate the readme and issues. The repo is very friendly to LLMs and you should be able to find an answer. If you're still having trouble, please open up an issue in github.
Thank you for trying halogen out and the feedback.
I was seriously impressed with how quickly the model loads - in comparison with other Qwen3.8-Flash versions
Great question.
Before I answer, how long is your initial load? Are you running a headless server? If you're experiencing slow load times, it could be Halogen trying to find 68GB of continuous memory blocks to allocate.
Halogen pins ~68GB in memory (model weights) then uses another 35GB for the KV cache. The biggest lever you have for controlling memory is the kv cache pool. Send an agent to investigate the readme and issues. The repo is very friendly to LLMs and you should be able to find an answer. If you're still having trouble, please open up an issue in github.
Thank you for trying halogen out and the feedback.
I apologize that my description was inaccurate. I actually encountered an insufficient-memory issue during loading. My environment is WSL on Windows 11, and I created the container with the following command:
podman run --rm -it -p 8731:8731 --device /dev/dxg --group-add keep-groups --security-opt seccomp=unconfined --ipc=host --ulimit memlock=-1:-1 -v ~/halogen-models:/models -v /usr/lib/wsl/lib/libdxcore.so:/usr/lib/libdxcore.so:ro -v /opt/rocm/lib/librocdxg.so:/usr/lib/librocdxg.so:ro -v /opt/rocm/lib/libhsa-runtime64.so.1:/usr/lib/libhsa-runtime64.so.1:ro -v /opt/rocm/lib/libamdhip64.so.7:/usr/lib/libamdhip64.so.7:ro ghcr.io/peonist-ai/halogen-flash-server
After creation, it can be recognized normally. However, because of the compounded memory overhead caused by Windows, WSL, and Podman, the model cannot be loaded. Therefore, I would like to ask the author if you could advise on how to proceed or how to reduce the size of the model files. The log is as follows:
halogen: halogen-flash-server 0.11.7, mode all
halogen: quality sidecar present (2.4G) at /models/qwen38-flash-next-w4b.overlay.hgn
halogen: a vision sidecar is present but NOT loaded: /models/qwen38-flash-next-vision.hgn
set HALOGEN_VISION_TOWER=/models/qwen38-flash-next-vision.hgn to accept images.
halogen: KV budget 4 slot(s) over one 524288-position pool (each request up to 262144) = 14.9 GiB (~28 KiB/position incl. block scratch + ~115 MiB/slot) + 1.8 GiB prompt cache in RAM, on top of roughly 68 GiB of weights and 11 GiB of scratch. Those last two are estimates for this pre-flight check; the engine prints its measured figures once loaded, including the large lookup table it reads from disk and never holds. MemAvailable now 123.4 GiB.
halogen: no /tokenizer mount, using /models/tokenizer from the models volume
serve_api: request defaults none set (built-in)
halogen: waiting for engine on 8730 (cold load can take minutes)
kv pool: host RAM 125 GiB, less 67.7 resident weights and 20.0 reserved for the n-gram page cache and the OS = 37.3 GiB for the device
kv pool: 524288 positions, ~35.0 GiB (20.6 GiB of it for everything but the pool, at max_tok 32768)
checkpoint: mapped 124.1 GB, NOT pinned (Pin::None)
checkpoint: mapped 2.6 GB, NOT pinned (Pin::None)
overlay: 30 tensors upgraded to q8g64 in place of Q4C-P
overlay: 741 tensors, 2.40 GiB from /models/qwen38-flash-next-w4b.overlay.hgn (HALOGEN_CK_OVERLAY)
startup [ 0.0 s] host memory: 63169 contiguous 2 MiB blocks of host memory free (126338 MiB)
startup [ 0.0 s] pinning weights (reads the checkpoint from disk when it is not already in the page cache; this is usually the slowest step)
checkpoint: the GPU driver refused to register 0.33 GiB of the model file (invalid argument), after 0.00 GiB had registered.
This is the driver's limit on how much of this machine can be resident at once, not a problem with the file.
Free memory on the host: another process holding GPU memory (check for a second engine) is the usual cause.
Check also that the container has an unlimited memlock (--ulimit memlock=-1:-1).
halogen: the engine exited after 6s without listening.
Hi! There is a good writeup in the readme on github. Windows is not "officially supported", but I know people have gotten halogen running.