---
license: apache-2.0
license_name: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive
base_model_relation: quantized
tags:
- fp8
- llm-compressor
- compressed-tensors
- vllm
- moe
- qwen3_5_moe
- roleplay
language:
- ru
- en
- zh
---

# JoyFox-Qwen3.6-35B-A3B-RP-Aggressive — FP8 Dynamic

FP8_DYNAMIC (W8A8) quantization of
[joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive](https://huggingface.co/joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive)
(architecture `Qwen3_5MoeForConditionalGeneration`, base model `Qwen/Qwen3.6-35B-A3B`) —
a 35B-A3B MoE roleplay finetune.

* **Source:** `joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive`
* **License:** apache-2.0 (inherited from the source and base models)
* **Tool:** [llm-compressor](https://github.com/vllm-project/llm-compressor), data-free one-shot
* **Scheme:** FP8 per-channel weights (static), FP8 per-token activations (dynamic)
* **Weights size:** 36.66 GiB (down from ~70 GiB BF16) — fits a single 96 GB GPU with ample room for KV cache

## Kept in BF16

The following modules were excluded from quantization: `lm_head`, embeddings,
the vision tower (`model.visual.*`), the Gated DeltaNet linear-attention layers
(`linear_attn.*`), MoE routers (`mlp.gate`), the shared-expert gate
(`shared_expert_gate`) and the MTP head (`mtp.*`). The ignore list matches the
reference `RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic` quantization.

## Recipe

```python
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
        "re:.*embed_tokens$",
        "re:.*shared_expert_gate$",
        "re:.*linear_attn.*",
        "re:^mtp.*",
    ],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained("out")
processor.save_pretrained("out")
```

## Serving with vLLM

```bash
vllm serve cloud19/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive-FP8-Dynamic \
  --served-model-name joyfox \
  --enable-chunked-prefill \
  --max-model-len 16384 --kv-cache-dtype fp8 \
  --limit-mm-per-prompt '{"image": 0, "audio": 0, "video": 0}' \
  --default-chat-template-kwargs '{"enable_thinking": false}'
```

Notes:

* **Chunked prefill is required.** The hybrid attention stack (Gated DeltaNet
  linear-attention layers) uses vLLM's mamba cache in mode `align`; passing
  `--no-enable-chunked-prefill` makes the engine refuse to start.
* The MTP head is shipped as `model_mtp.safetensors`, but speculative decoding
  is intentionally **not** used here.
* Thinking (`<think>`) is disabled at the chat-template level via
  `enable_thinking: false`.