catplusplus/Qwen3.5-35B-A3B-heretic-v2-NVFP4

Quantization of Qwen 3.5 currently requires a lot of tinkering:

Installed git llm-compressor and cherry picked https://github.com/vllm-project/llm-compressor/pull/2383 Ran the provided example, it printed some errors at the end but still saved compressed model to current folder Saved model was missing a bunch of files, copied them from base, also copied base config.json and just added quantization config from compressed model (generated one was missing things like vision encoder) After that compressed model loaded into my built-from-git vllm + transformers + flashinfer venv. It complained about a bunch of things being unsupported during autotune, but nevertheless works well for text and images and tps looks like it's using NVFP4 acceleration. Doesn't refuse things in my brief testing. Calls tools well when run with --enable-auto-tool-choice --kv-cache-dtype fp8 --tool-call-parser qwen3_coder --reasoning-parser qwen3 --mm-encoder-tp-mode data --mm-processor-cache-type shm