File size: 3,049 Bytes
e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b e50a71c 5fde52b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
license: apache-2.0
license_name: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive
base_model_relation: quantized
tags:
- fp8
- llm-compressor
- compressed-tensors
- vllm
- moe
- qwen3_5_moe
- roleplay
language:
- ru
- en
- zh
---
# JoyFox-Qwen3.6-35B-A3B-RP-Aggressive — FP8 Dynamic
FP8_DYNAMIC (W8A8) quantization of
[joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive](https://huggingface.co/joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive)
(architecture `Qwen3_5MoeForConditionalGeneration`, base model `Qwen/Qwen3.6-35B-A3B`) —
a 35B-A3B MoE roleplay finetune.
* **Source:** `joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive`
* **License:** apache-2.0 (inherited from the source and base models)
* **Tool:** [llm-compressor](https://github.com/vllm-project/llm-compressor), data-free one-shot
* **Scheme:** FP8 per-channel weights (static), FP8 per-token activations (dynamic)
* **Weights size:** 36.66 GiB (down from ~70 GiB BF16) — fits a single 96 GB GPU with ample room for KV cache
## Kept in BF16
The following modules were excluded from quantization: `lm_head`, embeddings,
the vision tower (`model.visual.*`), the Gated DeltaNet linear-attention layers
(`linear_attn.*`), MoE routers (`mlp.gate`), the shared-expert gate
(`shared_expert_gate`) and the MTP head (`mtp.*`). The ignore list matches the
reference `RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic` quantization.
## Recipe
```python
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
MODEL_ID = "joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=[
"re:.*lm_head",
"re:visual.*",
"re:model.visual.*",
"re:.*mlp.gate$",
"re:.*embed_tokens$",
"re:.*shared_expert_gate$",
"re:.*linear_attn.*",
"re:^mtp.*",
],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained("out")
processor.save_pretrained("out")
```
## Serving with vLLM
```bash
vllm serve cloud19/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive-FP8-Dynamic \
--served-model-name joyfox \
--enable-chunked-prefill \
--max-model-len 16384 --kv-cache-dtype fp8 \
--limit-mm-per-prompt '{"image": 0, "audio": 0, "video": 0}' \
--default-chat-template-kwargs '{"enable_thinking": false}'
```
Notes:
* **Chunked prefill is required.** The hybrid attention stack (Gated DeltaNet
linear-attention layers) uses vLLM's mamba cache in mode `align`; passing
`--no-enable-chunked-prefill` makes the engine refuse to start.
* The MTP head is shipped as `model_mtp.safetensors`, but speculative decoding
is intentionally **not** used here.
* Thinking (`<think>`) is disabled at the chat-template level via
`enable_thinking: false`.
|