---
license: apache-2.0
language:
- en
- zh
base_model:
- nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill
pipeline_tag: text-generation
tags:
- qwen3
- moe
- distillation
- reasoning
- thinking
- agentic
- nvfp4
- modelopt
- vision
- multimodal
---

# Qwen3.6-35B-A3B-DSV4Pro-FP4

## Model Summary
This is the **NVFP4 (W4A16)** quantized version of the **Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill** model. 

### Key Characteristics:
*   **Vision Tower Kept**: Both the speculative decoding MTP heads have been pruned to reduce model overhead, but the **visual tower (`model.visual`) is fully preserved**. To guarantee multimodal capability and high accuracy, the visual tower is kept unquantized in high-precision `bfloat16`.
*   **NVIDIA ModelOpt FP4 (GS16) Quantization**: Quantized using NVIDIA ModelOpt with **Group Size 16 (GS16)**.
    *   *Note on Group Size:* While we tested Group Size 128 (GS128) to shrink the footprint further, the vLLM Marlin FP4 CUDA kernel (`marlin_mm`) only supports a group size of 16. Attempts to serve GS128 will result in a serving engine crash (`Invalid thread config`). Therefore, GS16 is the only viable serving configuration.
*   **Disk Footprint**: ~21 GB.

---

## Evaluation Benchmarks (GS16, thinking-on)
Evaluated using the local test suite (temperature 0.6, top_p 0.95):
*   **GSM8K Accuracy**: **85.00%** (17/20 correct)
*   **ARC-Challenge Accuracy**: **80.00%** (16/20 correct)
*   **GPU KV Cache (RTX 5060 Ti / vLLM)**: **534,588 tokens** (FP8 KV Cache enabled). At a target concurrency of 32, this allocates a physical slice of **16,705 tokens** per concurrent request (prefix caching enabled).

---

## Original Distill Model Information (`nerkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill`)

> On **Qwen3.6-35B-A3B** (**MoE, 3B active**), we use LoRA to **distill the *way DeepSeek-V4-Pro reasons* (with thinking-on) plus its agentic behavior** — purpose-built as a fast **task orchestrator** (decompose → delegate → verify) for Lynn Agent.
>
> This is the **MoE counterpart of the 27B Dense sister model**: same R6000 GPU, same teacher, same recipe, on a sparse architecture.
>
> ⚠️ **Distilling a thinking style ≠ distilling knowledge/capability**: the goal is "learn *how to reason and how to converge*", not to inject knowledge or raise the capability ceiling.

### Training details
- **Base**: Qwen3.6-35B-A3B (MoE, 3B active, **BF16** base)
- **Method**: **LoRA**, **r = 64, α = 128, dropout = 0.05**, targets = all attention + MLP projections
- **Optim**: paged_adamw_8bit, cosine LR, warmup 0.03, **~1 epoch**
- **Teacher**: DeepSeek-V4-Pro (thinking-on + agentic)
- **Data**: **~1,842 distillation samples** (lynn_prod spec). Trajectories = DS-V4-Pro multi-step reasoning under thinking-on (`<think>`) + ReAct-style tool calls (think one step → call one tool → observe → loop).
  - **The tool "execution results" are SIMULATED, not actually run** — each result line is improvised by a small fast model (DeepSeek-V4-Flash) role-playing the runtime, not real sandbox execution.
  - **Training masks those fabricated results** — the model learns only "how to think / how to call tools", never the made-up outputs (so it doesn't learn to fabricate tool returns).

### Attribution (the method is not original — it is a combination of published techniques)
- **ReAct** (interleaved reasoning + acting): Yao et al., **2022**, arXiv:2210.03629 (ICLR 2023)
- **STaR** (bootstrapping reasoning traces): Zelikman et al., 2022, arXiv:2203.14465
- **Self-Instruct / Baize self-chat**: Wang et al., 2022; Xu et al., 2023, arXiv:2304.01196
- **AgentTuning**: Zeng et al., 2023, arXiv:2310.12823
- **ToolBench / ToolLLM**: Qin et al., 2023, arXiv:2307.16789
- **DeepSeek-R1 reasoning distillation**: DeepSeek-AI, 2025, arXiv:2501.12948

---

## Serving & Inference with vLLM
Load using the standard `compressed-tensors` path inside vLLM. Here is the recommended serving configuration:

```bash
vllm serve /path/to/Qwen3.6-35B-A3B-DSV4Pro-Vision-FP4-GS16 \
  --served-model-name nvidia/Agents-A1-FP4 \
  --enforce-eager \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --max-model-len 128000 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching
```

---

## 🇨🇳 中文版 (Chinese Version)

### 模型摘要
这是 **Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill** 模型的 **NVFP4 (W4A16)** 量化版本。

*   **保留视觉塔 (Visual Tower)**: 我们移除了投机解码的 MTP 头以降低运行开销，但**完整保留了视觉塔 (`model.visual`)**。视觉塔保持未量化的 `bfloat16` 高精度格式，以确信优秀的多模态理解能力。
*   **NVIDIA ModelOpt FP4 (GS16) 量化**: 使用 NVIDIA ModelOpt 以 **Group Size 16 (GS16)** 进行 PTQ 校准。
    *   *关于组大小的说明:* 虽然我们测试了 Group Size 128 (GS128) 以进一步减小模型体积，但在 vLLM 中，Marlin FP4 CUDA 算子 (`marlin_mm`) 目前仅支持 group size 为 16。尝试加载 GS128 模形将导致服务崩溃（提示 `Invalid thread config`）。因此，GS16 是唯一可用的运行服务配置。
*   **磁盘大小**: ~21 GB。