Create README.md

README.md

@@ -0,0 +1,76 @@
+---
+library_name: transformers
+license: apache-2.0
+license_link: https://huggingface.co/Qwen/Qwen3.5-27B/blob/main/LICENSE
+pipeline_tag: image-text-to-text
+datasets:
+- HuggingFaceH4/ultrachat_200k
+- openai/gsm8k
+- sahil2801/CodeAlpaca-20k
+- CohereLabs/aya_dataset
+base_model:
+- Qwen/Qwen3.5-27B
+---
+
+# ig1/Qwen3.5-27B-NVFP4 by IG1
+
+## Quantization
+
+This model has been quantized using [llm-compressor](https://github.com/vllm-project/llm-compressor) v0.10.1.dev31+geb49917e (just after Qwen3.5 support was merged) and transformers v5.3.0.
+It is based on the [official example](https://github.com/vllm-project/llm-compressor/blob/36fab734a744b4f46b7b379b7c4b4b3f47fde3bf/docs/key-models/qwen3.5/nvfp4-vl-example.md) with a few modifications (see next section).
+
+### Quantization particularities
+
+The sequence length has been increased from 4096 to 8192 and the number of samples from 256 to 1024.
+The 1024 samples come from 4 differents datasets:
+
+* 256 general conversation samples ([UltraChat](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k))
+* 256 math reasoning samples ([GSM8K](https://huggingface.co/datasets/openai/gsm8k))
+* 256 code samples ([CodeAlpaca](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k))
+* 256 multilingual samples ([Aya](https://huggingface.co/datasets/CohereLabs/aya_dataset))
+
+You can find the quantization script [here](Qwen3.5-27B_nvfp4.py).
+
+While the quantization needed transformers v5, the original (transformers v4) tokenizer files has been put back for simple execution on current vLLM versions. The transformers v5 tokenizer files produced by llm-compressor can be found in the `transformers_v5` folder.
+
+## Qwen3.5 Profiles
+
+Alongside support for dynamic thinking and non-thinking modes, the Qwen team offers [4 sampling parameter profiles](https://huggingface.co/Qwen/Qwen3.5-27B#using-qwen35-via-the-chat-completions-api):
+
+* Thinking General
+* Thinking Coding
+* Instruct General
+* Instruct Reasoning (we prefer to call it Instruct Creative internally)
+
+Manually configuring these parameters for every AI client can be difficult.
+To solve this, we built a lightweight reverse proxy that exposes the 4 profiles as virtual model names.
+It handles request transformation on the fly using a single inference server as backend.
+View the project on [our GitHub](https://github.com/iguanesolutions/qwen35-rp).
+
+## Inference
+
+We run this model with vLLM, here is a sample execution command:
+
+```bash
+docker run --rm --name 'Qwen3.5-27B-NVFP4' \
+  --runtime=nvidia --gpus 'all' --ipc=host \
+  -e 'HF_TOKEN' \
+  -e 'VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1' \
+  -v '/srv/cache:/root/.cache' \
+  -p '127.0.0.1:8000:8000' \
+  'vllm/vllm-openai:v0.18.0-cu130' \
+  'ig1/Qwen3.5-27B-NVFP4' \
+  --served-model-name 'Qwen3.5-27B' \
+  --reasoning-parser 'qwen3' \
+  --enable-auto-tool-choice \
+  --tool-call-parser 'qwen3_coder' \
+  --max-model-len 'auto' \
+  --gpu-memory-utilization '0.9'
+```
+
+A few notes about some of the parameters:
+
+* Adapt the `/srv/cache:/root/.cache` mount point to your liking. It contains files you want to keep between multiples run (dynamo bytecode and AOT with torch compile but most importantly the huggingface folder for the model)
+* `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1` allows for more precise CUDA graph VRAM estimation. It should become the default once vLLM reaches v0.19.0 which at which point you can simply remove it
+* If you deploy the model into several GPUs using Tensor Parallelism, be sure to check the [official recipe](https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3.5.html) as others flags are needed.
+