--- base_model: Qwen/Qwen3.6-35B-A3B license: apache-2.0 library_name: transformers tags: - qwen3.6 - qwen3.6-moe - nvfp4 - fp8 - quantization - vllm - compressed-tensors - blackwell - rtx-5090 - mixture-of-experts pipeline_tag: image-text-to-text inference: false --- # Qwen3.6-35B-A3B-NVFP4 Mixed-precision NVFP4 (+ optional FP8 lm_head) quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — the Mixture-of-Experts variant with 35 B total parameters and ~3 B active per token — targeting native Blackwell (SM120) deployment, primarily RTX 5090 32 GB. The original BF16 checkpoint needs ~67 GiB of VRAM. This build fits a single RTX 5090 32 GB at 32 K context with usable KV cache, multi-turn reasoning, hybrid attention (self-attn + GDN linear-attn), and tool-call support. ## TL;DR - Use **`main` + `--kv-cache-dtype turboquant_4bit_nc`** for production workflow / tool-call / structured-output serving. - Use **`fp8-head` + `--kv-cache-dtype turboquant_4bit_nc`** when you want the highest concurrency and the workload is mostly free-form text. - Do **not** expect an `fp8-head-embed` branch here. FP8 embeddings were intentionally skipped after a regression canary on the 27 B sibling model. - The reported concurrency is measured on a single RTX 5090 32 GB with `max_model_len=32768`, `max_num_seqs=16`, and CUDA graphs enabled. ## Variants The repository hosts two branches: | Branch | `lm_head` dtype | `embed_tokens` dtype | Intended deployment | |---|---|---|---| | `main` | **BF16** | BF16 | Production default. Workflow / agent engine — preserves output-token precision for JSON, tool calls, enums. | | `fp8-head` | **FP8_BLOCK [128, 128]** | BF16 | Free-form text and chat at higher concurrency. ~10 % more KV-cache headroom than `main` at the cost of a small FP8 quantization of the output projection. | Inner Linear layers (every MoE expert projection plus self-attn projection) use the same uniform NVFP4 calibration on both branches. The calibrated NVFP4 inner weights are bit-identical across the two branches; only the output projection differs. ### VRAM and concurrency (RTX 5090 32 GB) Measured on a single RTX 5090 with `gpu_memory_utilization=0.93`, `max_model_len=32768`, `max_num_seqs=16`, `max_num_batched_tokens=4096`, `dtype=bfloat16`. | Branch | KV cache dtype | Weights | KV cache | GPU KV cache | Max concurrency @ 32 K | Decode tok/s | |---|---|---|---|---|---|---| | `main` (bf16head) | `fp8_e4m3` | 21.88 GiB | 4.63 GiB | 119,472 tok | 12.16× | 182 | | `main` (bf16head) | `turboquant_4bit_nc` | 21.96 GiB | 4.94 GiB | **249,856 tok** | **22.45×** | 175 | | `fp8-head` (fp8lm) | `fp8_e4m3` | 21.40 GiB | 5.10 GiB | 132,048 tok | 13.42× | 193 | | `fp8-head` (fp8lm) | `turboquant_4bit_nc` | 21.48 GiB | 5.41 GiB | **274,432 tok** | **24.55×** | 184 | TurboQuant 4-bit non-causal KV cache roughly doubles the KV pool versus `fp8_e4m3` at the cost of ~4–5 % single-stream decode. The recommended deploy mode is `turboquant_4bit_nc` for both branches. ### Scaling up to 262 K context (production config) The numbers above are the conservative Gate-4 baseline (32 K context, `max_num_seqs=16`, `gpu_memory_utilization=0.93`). For production workloads the same artifact serves cleanly at the full Qwen3.6-35B-A3B context window (262 144 tokens) with `gpu_memory_utilization=0.95` and `max_num_seqs=64`, which gives a substantially larger KV pool because the block-allocation strategy with longer contexts reduces partial-block waste: | Branch | KV cache dtype | Weights | KV cache | GPU KV cache | Max concurrency @ 262 K | |---|---|---|---|---|---| | `main` (bf16head) | `turboquant_4bit_nc` | 21.96 GiB | 5.48 GiB | **1,022,361 tok** | **3.90×** | | `main` (bf16head) | `fp8_e4m3` | 21.88 GiB | 4.97 GiB | 492,512 tok | 1.88× | | `fp8-head` (fp8lm) | `turboquant_4bit_nc` | 21.48 GiB | 5.48 GiB | ~1,022,000 tok | ~3.9× | A single endpoint at `max_model_len=262144` therefore serves both ordinary short requests (where `max_num_seqs=64` is the binding ceiling) and very long-context requests (3–4× concurrent at full 262 K) from the same engine, without needing a separate "long-context lane". ### Branch selection - **Workflow / agent engine, tool calls, structured output:** use `main`. BF16 lm_head keeps logit ordering intact for sampling sensitive tokens (JSON braces, enum values, IDs, dates). - **Free-form text generation, chat, summarization:** use `fp8-head` for the extra concurrency headroom at the same context length. ### Why no `fp8-head-embed` branch The sibling `inferRouter/Qwen3.6-27B-NVFP4` repository ships a third `fp8-head-embed` branch (FP8 lm_head **and** FP8 embed) marked **lab only** because a regression canary on a short Czech arithmetic prompt reproducibly flipped the model's final discount comparison while the intermediate calculation was still correct. The 35 B variant does **not** ship an `embed_tokens`-FP8 branch — the same risk applies and the saving on this architecture (~0.47 GiB) does not justify shipping a lab-only artifact. ## Architecture / quantization summary - Backbone: `Qwen3_5MoeForConditionalGeneration` - 40 transformer layers; hybrid attention (16 self-attention layers interleaved with 24 GDN linear-attention / Mamba-style layers) - 256 routed experts per MoE block, top-K routed - NVFP4 (W4A4, group size 16) on every `Linear` not in the ignore list - BF16 stays on: visual-encoder blocks (vision tower retained for image-text-to-text), every layer's `linear_attn.*`, every layer's `mlp.gate` and `mlp.shared_expert_gate`, MTP graft, and (on `main`) `lm_head` + `embed_tokens`. - On `fp8-head`: `lm_head` is FP8_BLOCK [128, 128] (block-quantized float-8 with a per-128-row scale grid). A frozen Multi-Token-Prediction head (`model_mtp.safetensors`, ~1.6 GB) is included for compatibility with vLLM speculative-decode setups; it is not loaded by default. ## Recipe and calibration This build follows the spirit of [Red Hat / Neural Magic's LLM Compressor recipes](https://github.com/vllm-project/llm-compressor) for NVFP4 MoE checkpoints, with an adjusted calibration mix tuned for Czech-language robustness and Czech legal-domain coverage in addition to the usual English chat / math / code / multilingual diet. The raw calibration corpus is not redistributed with this model card; the public artifact records the important reproducibility metadata in the checkpoint config and branch layout. Calibration ran one full pass at 1280 samples and 8192 max sequence length on the BF16 source checkpoint, producing the uniform NVFP4 inner used by both branches. The `fp8-head` branch keeps the calibrated FP8_BLOCK `lm_head`; the `main` branch surgically restores the original BF16 `lm_head` from the base checkpoint. This avoids a second 28-hour calibration run while preserving the same NVFP4 inner numerics. ## Files Per branch: ``` chat_template.jinja config.json generation_config.json model.safetensors (~22–23 GB; differs across branches by ~0.5 GB) model.safetensors.index.json model_mtp.safetensors (~1.6 GB; identical across branches) processor_config.json recipe.yaml tokenizer.json tokenizer_config.json ``` The `main` branch additionally hosts a [`vllm_patches/`](vllm_patches) folder with the source overlay needed to dispatch FP8 weights on `lm_head` through the compressed-tensors path when serving the `fp8-head` branch on vLLM versions that have not yet merged the upstream PR. ## Recommended vLLM serve config Single RTX 5090 32 GB, **full 262 K context, production-tuned**, `turboquant_4bit_nc` KV cache, max 64 in-flight sequences: ```bash vllm serve inferRouter/Qwen3.6-35B-A3B-NVFP4 \ --revision main \ --served-model-name qwen35b-a3b \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.95 \ --max-model-len 262144 \ --max-num-seqs 64 \ --max-num-batched-tokens 8192 \ --kv-cache-dtype turboquant_4bit_nc \ --enable-chunked-prefill \ --enable-prefix-caching \ --dtype bfloat16 \ --trust-remote-code ``` For the `fp8-head` branch, change `--revision` to `fp8-head` and keep everything else the same. If you do not need the full 262 K window and want a slightly safer VRAM headroom (extra ~0.5 GiB), drop `--max-model-len` to 32768 and `--gpu-memory-utilization` to 0.93 — that is the Gate-4 baseline whose numbers are in the upper VRAM table. `max_num_batched_tokens` must be at least 4096 because the GDN linear-attn layers in this architecture impose a Mamba block-size constraint of 3072 tokens. The recommended value of 8192 keeps long-prompt TTFT reasonable without exhausting per-iteration GPU memory. If your vLLM image does not include the upstream TurboQuant hybrid attention support yet, swap `--kv-cache-dtype turboquant_4bit_nc` for `--kv-cache-dtype fp8_e4m3` and you will get the `fp8_e4m3` baseline numbers from the tables above — both KV modes are tested and stable on both branches. Current vLLM `main` / nightly builds after 2026-05-05 should already include the TurboQuant hybrid patch; manual PR #39931 patching is mainly for pinned releases and older vendor images. ### Measured throughput under sustained load Numbers below come from a 13-minute sustained run on a single RTX 5090 32 GB with the production config above (262 K context, `max_num_seqs=64`, `turboquant_4bit_nc`, chunked prefill, `ignore_eos=true` so every request runs to `max_tokens`): **Production-shape workload** (25 concurrent, 30 K input + 2 000 forced output, 5 minutes sustained): | Metric | Value | |---|---| | Requests completed / errors | 100 / 0 | | Wall p50 / p95 | 92.3 s / 108.9 s | | TTFT p50 / p95 | 2.04 s / 19.8 s | | Aggregate decode | 542 tok/s | | Aggregate total (prefill + decode) | 7,005 tok/s | | Per-request decode | 22.2 tok/s | **Saturation ramp** (15 K input + 1 500 forced output, 60 s per step): | Concurrent | Wall p50 | TTFT p50 | Aggregate decode | |---|---|---|---| | 30 | 50.7 s | 1.81 s | **865 tok/s** (peak) | | 40 | 63.9 s | 8.6 s | 810 tok/s | | 50 | 71.7 s | 10.8 s | 793 tok/s | | 60 | 75.8 s | 12.9 s | 839 tok/s | | 64 | 77.1 s | 13.7 s | 854 tok/s | Sweet spot is around **30 concurrent**: aggregate decode peaks at ~865 tok/s with sub-second-class TTFT (1.8 s p50). Above 30 concurrent throughput plateaus around 800–850 tok/s — the engine is decode-bound at that point — and TTFT grows from 1.8 s to 13 s as the scheduler queue backs up. The container served 471 requests across the full 13-minute stress without errors or degraded responses. Very long single-request contexts (60 K, 120 K, 200 K tokens) **do** load and run in the same engine, but compete with the normal cohort for KV slots and prefill bandwidth. Under sustained 30-concurrent normal load, TTFT for a 60 K outlier was 73 s and a 200 K outlier did not complete in a 3-minute window. For batch workloads where outliers are rare (≤ 0.01 % of requests in our tests), either schedule them serially after the main cohort or route them through a dedicated low-priority worker; do not expect interactive latency on outliers while the engine is saturated by the normal cohort. ## Runtime requirements per branch Base requirement for both branches: a recent vLLM build with compressed-tensors NVFP4 support for Blackwell GPUs and the Qwen3.6 hybrid architecture. The table below lists the extra patch requirements on top of that base runtime. Three independent vLLM patches may be needed depending on which branch you serve and which KV cache dtype you choose: | Patch | Source | When required | |---|---|---| | **TurboQuant hybrid attention** | [vLLM PR #39931](https://github.com/vllm-project/vllm/pull/39931) | Any branch, **if** `--kv-cache-dtype turboquant_*` is set. Qwen3.6-35B-A3B is a hybrid architecture (self-attn + GDN linear-attn); the in-tree TurboQuant rejects hybrid models before this PR. The PR was merged to `vllm-project/vllm:main` on 2026-05-05, so current vLLM main/nightly builds after that date should already include it. Stock releases cut before that date, including v0.20.x, still need either the PR applied or a newer nightly/base image. | | **TurboQuant continuation-prefill workspace fix** | [vLLM PR #40798](https://github.com/vllm-project/vllm/pull/40798) | Any branch, **if** `--kv-cache-dtype turboquant_*` is set **and** chunked prefill is enabled (default in vLLM V1). Without this patch, prompts longer than `max_num_batched_tokens` trigger a `_continuation_prefill` workspace size assertion at runtime (see [upstream issue #41726](https://github.com/vllm-project/vllm/issues/41726), [#41565](https://github.com/vllm-project/vllm/issues/41565), [#40420](https://github.com/vllm-project/vllm/issues/40420)). The patch reserves the maximum-shape `continuation_prefill` workspace before locking it during CUDA-graph capture. **Still open upstream** at the time of this write-up; until it is merged, apply it as a source overlay on the vLLM Python package in the serving image. The `fp8_e4m3` KV cache mode uses a different attention backend and does not require this patch. | | **Compressed-tensors FP8 head dispatch** | [`vllm_patches/`](vllm_patches) | `fp8-head` branch only. The in-tree `compressed-tensors` dispatcher in vLLM 0.20.x routes only `LinearBase`, `ParallelLMHead`, `Attention`, and `FusedMoE` modules through quant schemes; FP8 weight loading on `lm_head` needs the additional dispatch patch shipped in this repo. An upstream PR for this is in progress. | Combined matrix: | Branch | KV cache dtype | Needs PR #39931? | Needs PR #40798? | Needs `vllm_patches/`? | |---|---|---|---|---| | `main` | `fp8_e4m3` (or default) | no | no | no | | `main` | `turboquant_4bit_nc` (recommended) | **yes** | **yes** | no | | `fp8-head` | `fp8_e4m3` | no | no | **yes** | | `fp8-head` | `turboquant_4bit_nc` (recommended) | **yes** | **yes** | **yes** | If your vLLM build or base image already includes PR #39931 — for example a vLLM main/nightly build from after 2026-05-05 — you only need PR #40798 (for `turboquant_*` KV modes) and, on the `fp8-head` branch, the `vllm_patches/` overlay. The `main` branch with `fp8_e4m3` KV cache runs out of the box on any recent vLLM build. ## What `vllm_patches/` adds (fp8-head only) The `vllm_patches/` folder on the `main` branch contains: - `apply_ct_fp8_lmhead_patch.py` — idempotent source patcher. Adds the `lm_head` branch to `CompressedTensorsConfig.get_quant_method`, wires the FP8 weight + block-scale parameters into `ParallelLMHead`, and adjusts `vocab_parallel_embedding.py` so the FP8 scale companion loads cleanly. The patcher detects when upstream PR #41000-style wire-up is already present and skips the relevant patch points. - `compressed_tensors_embedding.py` — companion runtime module for the compressed-tensors FP8 embedding experiment. It is included because the patch bundle is shared with sibling checkpoints, but this 35 B repository does **not** ship an FP8-embedding branch. - `Dockerfile.turboquant` — minimal Docker overlay that applies the patcher on top of a vLLM image that already includes PR #39931 plus the upstream FP8 lm_head work (Red Hat / Neural Magic vLLM nightly). Builds in ~30 seconds. ### Apply the patch (local dev) ```bash # Clone a vLLM source tree. Stock v0.20.2 is fine for fp8_e4m3 KV-cache # testing, but does NOT include TurboQuant hybrid attention support. For # --kv-cache-dtype turboquant_* use vLLM main/nightly from 2026-05-05 or # later, or a base image with PR #39931. git clone --depth 1 --branch v0.20.2 https://github.com/vllm-project/vllm.git vllm-src cd vllm-src # Place the patcher and companion in /tmp and run the patcher against # your vLLM install path. cp /path/to/vllm_patches/apply_ct_fp8_lmhead_patch.py /tmp/ cp /path/to/vllm_patches/compressed_tensors_embedding.py /tmp/ python3 /tmp/apply_ct_fp8_lmhead_patch.py /path/to/site-packages python3 -m py_compile \ /path/to/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py \ /path/to/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py \ /path/to/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_embedding.py ``` The patcher is idempotent and detects when PR #41000 has already wired `quant_config` into `ParallelLMHead`; in that case it skips the first patch point. For the compressed-tensors FP8 head dispatch itself, it works on both stock `v0.20.2` and on [Red Hat / Neural Magic vLLM nightlies](https://github.com/vllm-project/vllm) that already include the upstream lm_head FP8 work. TurboQuant hybrid support is separate and still requires PR #39931 or a newer vLLM build that includes it. ### Docker overlay (recommended) ```bash # inside vllm_patches/ docker build -f Dockerfile.turboquant -t local/vllm-qwen36-35b-a3b:patched . ``` The base image referenced in `Dockerfile.turboquant` already includes the TurboQuant hybrid patch (PR #39931) and the upstream FP8 lm_head work; the overlay only adds the compressed-tensors dispatch. ## Inference example ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="sk-anything") resp = client.chat.completions.create( model="qwen35b-a3b", messages=[ {"role": "user", "content": "Napis mi tri vety o vyznamu vcel pro zemedelstvi."}, ], max_tokens=200, temperature=0.7, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(resp.choices[0].message.content) ``` The Qwen3.6 chat template supports both `enable_thinking=true` and `enable_thinking=false`. For tests and short prompts prefer `false` — thinking mode emits a reasoning trace before the final answer and a low `max_tokens` cap will cut the trace mid-flight, returning `content=null`. ## Quality notes The 35 B `fp8-head` branch passed the same short smoke set used for the production branch, including the Czech discount canary that caught the FP8-embedding regression on the 27 B sibling model. That result should not be read as proof that FP8 output projection is universally harmless; it is why the recommended workflow-engine branch remains `main` with BF16 `lm_head` and BF16 `embed_tokens`. For structured output, tool routing, JSON, enum selection, or long-running agent workflows, prefer `main`. For summarization, chat, and prose-heavy generation where a small output-projection quantization risk is acceptable, `fp8-head` gives about 9 % more full-32K concurrency with TurboQuant. ## Credits - [Qwen team](https://huggingface.co/Qwen) for the base [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) checkpoint and the MoE architecture. - [Red Hat / Neural Magic LLM Compressor](https://github.com/vllm-project/llm-compressor) team for the NVFP4 calibration recipe this build is derived from. - [vLLM project](https://github.com/vllm-project/vllm) for the serving runtime, the TurboQuant hybrid attention work ([PR #39931](https://github.com/vllm-project/vllm/pull/39931)), and the upstream FP8 lm_head work. ## License Apache-2.0, inherited from the base [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) model.