How is your fork running on Metal? I love ds4 and am stoked to see you working on this for 3.8-Flash, but I don't see any discussion of Metal? I am down to give it a try and work on the implementation, but I'd like to get an idea of the starting point.
EDIT: forgot to say cheers for being so responsive to my suggestion! I love this community. Unsloth revised their Q5+ quants to implement my llama.cpp SSD offloading implementation suggestion as well, and it works perfectly!
@nazeshinjite Thanks — and really appreciate the offer to the Metal side.
To be transparent, I don’t currently have enough Metal environment available for development or testing, so since forking ds4 I’ve been fairly focused on Blackwell/CUDA, particularly the DGX Spark. That’s also why there hasn’t been much discussion of Metal in the current Qwen3.8 work.
I do have a Rust rewrite of ds4 planned.('rust-host' in my forked ds4 branch.) One thing I want to be careful about there is preserving ds4’s philosophy of keeping abstractions minimal and the execution path explicit.
At the same time, as more model families are added(because of my personal interest😅), I think it makes sense to establish just enough portability at the model/backend boundaries so that bringing up a new target doesn’t require reshaping the whole runtime every time.
As part of that work, I’ll consider putting Metal support on the roadmap as an explicit milestone. I’ll probably start with a smaller model rather than Qwen3.8-Flash-Next due to a lack of resources, mainly to establish and validate the Metal execution path first.
So at this moment I unfortunately don’t have a useful Metal starting point to hand you, but I’d definitely like to make it one of the targets as the rewrite progresses. Contributions there would be very welcome.
And thanks again for the SSD-offload suggestion — it ended up pushing this implementation in a much more interesting direction than I originally expected.
