Upload folder using huggingface_hub

README.md

@@ -1,214 +1,52 @@
 ---
-pipeline_tag: text-generation
-base_model:
-- Qwen/Qwen3.6-35B-A3B
 license: apache-2.0
-library_name: Model Optimizer
+base_model: Cadododoom/Qwen3.6-35B-A3B-DSV4Pro-FP4
 tags:
-- nvidia
-- ModelOpt
-- Qwen3.6
-- quantized
-- FP4
+- modelopt
+- nvfp4
 - fp4
+- gs16
+- compressed-tensors
+- vllm
+- moe
+- tensor-parallel
 ---
 
-# Model Overview
-
-## Description:
-The NVIDIA Qwen3.6-35B-A3B-NVFP4 model is the quantized version of Alibaba's Qwen3.6-35B-A3B model, which is an auto-regressive language model that uses an optimized transformer architecture. For more information, please check [here](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). The NVIDIA Qwen3.6-35B-A3B-NVFP4 model is quantized with [Model Optimizer](https://github.com/NVIDIA/Model-Optimizer).
-
-This model is ready for commercial/non-commercial use.  <br>
-
-## Third-Party Community Consideration
-This model is not owned or developed by NVIDIA. This model has been developed and built to a third-party’s requirements for this application and use case; see link to Non-NVIDIA [(Qwen3.6-35B-A3B) Model Card](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) from Alibaba.
-
-## References
-NVIDIA Model Optimizer: https://github.com/NVIDIA/Model-Optimizer
-
-### License/Terms of Use:
-[Apache license 2.0](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE)
-
-### Deployment Geography:
-Global <br>
-
-### Use Case: <br>
-Developers looking to take off-the-shelf, pre-quantized models for deployment in AI Agent systems, chatbots, RAG systems, and other AI-powered applications. <br>
-
-### Release Date:  <br>
-Hugging Face on 05/28/2026 via https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4 <br>
-
-## Model Architecture:
-**Architecture Type:** Transformers  <br>
-**Network Architecture:** Mixture-of-Experts (MoE) with Hybrid Attention <br>
-**Number of Model Parameters:** 35B in total and 3B activated <br>
-
-## Input:
-**Input Type(s):** Text, Image, Video <br>
-**Input Format(s):** String, Red, Green, Blue (RGB), Video (MP4/WebM) <br>
-**Input Parameters:** One-Dimensional (1D), Two-Dimensional (2D), Three-Dimensional (3D) <br>
-**Other Properties Related to Input:** Context length up to 262K <br>
-
-## Output:
-**Output Type(s):** Text <br>
-**Output Format:** String <br>
-**Output Parameters:** One-Dimensional(1D): Sequences <br>
-**Other Properties Related to Output:** None <br>
-
-Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>
-
-## Software Integration:
-**Supported Runtime Engine(s):** <br>
-* vLLM <br>
-
-**Supported Hardware Microarchitecture Compatibility:** <br>
-* NVIDIA Hopper, NVIDIA Blackwell <br>
-
-**Preferred Operating System(s):** <br>
-* Linux <br>
-
-The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
-
-## Model Version(s):
-The model version is NVFP4 1.0 version and is Quantized with nvidia-modelopt v0.44.0 <br>
-
-## Training and Evaluation Datasets:
-
-## Calibration Dataset:
-**Link:** [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail), [Nemotron-Post-Training-Dataset-v2](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2) <br>
-**Data Collection Method by dataset:** Automated. <br>
-**Labeling Method by dataset:** Automated. <br>
-**Properties:** The cnn_dailymail dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. The Nemotron-Post-Training-Dataset-v2 is a post-training dataset curated by NVIDIA containing multi-turn conversations across diverse topics. <br>
-
-## Training Dataset:
-**Data Modality:** Undisclosed <br>
-**Data Collection Method by dataset:** Undisclosed <br>
-**Labeling Method by dataset:** Undisclosed <br>
-**Data Size:**  Undisclosed <br>
-**Properties:** Undisclosed
-
-## Evaluation Dataset:
-**Datasets:** MMLU Pro, GPQA Diamond, τ²-Bench Telecom,  MMMU Pro, SciCode, AIME 2025, AA-LCR, IFBench <br>
-**Data Collection Method by dataset:** Hybrid: Automated, Human <br>
-**Labeling Method by dataset:** Hybrid: Human, Automated <br>
-**Properties:** We evaluated the model on text-based reasoning and coding benchmarks: MMLU Pro is a multi-task language understanding benchmark with challenging multiple-choice questions across diverse academic domains; GPQA Diamond contains 448 graduate-level multiple-choice questions written by domain experts in biology, physics, and chemistry; τ²-Bench Telecom evaluates agentic tool-use and policy-adherence capabilities in dual-control telecom customer-service scenarios where the model interacts with a simulated user and external tools to resolve account issues; MMMU Pro is the more challenging version of the Massive Multi-discipline Multimodal Understanding benchmark, measuring college-level multimodal reasoning across diverse disciplines with expanded answer choices and a vision-only input setting; SciCode evaluates scientific coding capabilities; AIME 2025 contains problems from the American Invitational Mathematics Examination; AA-LCR (Artificial Analysis Long Context Recall) evaluates a model's ability to accurately retrieve and recall information from long input contexts; IFBench is a benchmark for evaluating instruction-following capabilities across diverse and structured task constraints. <br>
-
-## Inference:
-**Acceleration Engine:** vLLM <br>
-**Test Hardware:** NVIDIA GB300 <br>
-
-## Post Training Quantization
-This model was obtained by quantizing the weights of Qwen3.6-35B-A3B to NVFP4 data type, ready for inference with vLLM. Only the weights and activations of the linear operators within transformer blocks in MoE are quantized. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 3.06x.
-
-## Usage
-
-To serve this checkpoint with [vLLM](https://github.com/vllm-project/vllm), you can start the docker `vllm/vllm-openai:nightly` and run the sample command below:
-
-```sh
-vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --port 8000 --quantization modelopt --max-model-len 262144 --reasoning-parser qwen3
-```
-
-For NVIDIA DGX Spark, we recommend using this `vllm serve` command:
-
-```sh
-vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4  \
-   --host 0.0.0.0 \
-   --port 8000 \
-   --tensor-parallel-size 1 \
-   --trust-remote-code \
-   --kv-cache-dtype fp8  \
-   --attention-backend flashinfer  \
-   --moe-backend marlin \
-   --gpu-memory-utilization 0.4  \
-   --max-model-len 262144 \
-   --max-num-seqs 4 \
-   --max-num-batched-tokens 8192  \
-   --enable-chunked-prefill \
-   --async-scheduling \
-   --enable-prefix-caching \
-   --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'  \
-   --load-format fastsafetensors \
-   --reasoning-parser qwen3 \
-   --tool-call-parser qwen3_xml \
-   --enable-auto-tool-choice
+# Qwen3.6-35B-A3B-ModelOpt-NVFP4
+
+This repository contains an optimized **NVFP4 (Group Size 16)** quantized checkpoint of `Qwen3.6-35B-A3B-DSV4Pro-FP4` (a 35B parameter sparse Mixture-of-Experts model) produced using **NVIDIA ModelOpt** and formatted for native **vLLM** serving.
+
+## Model Summary
+* **Base Architecture**: Qwen 3.5 MoE (`Qwen3_5MoeForCausalLM`)
+* **Quantization Method**: NVIDIA ModelOpt NVFP4 (W4A16 per-group quantization)
+* **Group Size**: **GS16** (Required configuration for vLLM Marlin FP4 CUDA kernel execution)
+* **Total Model Size**: ~19.66 GB safetensors weight shards (6 shards)
+* **Vocabulary Size**: 248,320 tokens (intact)
+* **Vision Encoder & MTP**: Vision encoder intact (`visual.*`), vocabulary intact, MTP heads stripped (`mtp.*`)
+
+## Group Size & vLLM Serving Note
+> **Critical Serving Note**: While Group Size 128 (GS128) shrinks footprint further, the vLLM Marlin FP4 CUDA kernel (`marlin_mm`) only supports a group size of 16. Attempts to serve GS128 will result in a engine crash (`Invalid thread config`). Therefore, **GS16 is the only viable serving configuration**.
+
+## Measured Benchmark Results
+* **GSM8K Math Reasoning (20 Representative Samples, temp=0.6, top_p=0.95)**: Evaluated under local test suite
+* **ARC-Challenge Science Reasoning (20 Representative Samples, temp=0.6, top_p=0.95)**: Evaluated under local test suite
+
+## Serving with vLLM
+
+```bash
+vllm serve Cadododoom/qwen3.6-35b-a3b-modelopt-nvfp4 \
+  --served-model-name qwen3.6-35b-a3b-modelopt-nvfp4 \
+  --tensor-parallel-size 2 \
+  --quantization compressed-tensors \
+  --moe-backend marlin \
+  --attention-backend flashinfer \
+  --kv-cache-dtype fp8 \
+  --max-model-len 112000 \
+  --max-num-seqs 4 \
+  --max-num-batched-tokens 4096 \
+  --gpu-memory-utilization 0.96 \
+  --enable-prefix-caching \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 30000
 ```
-
-
-## Evaluation
-The accuracy benchmark results are presented in the table below:
-<table>
-  <tr>
-   <td><strong>Precision</strong>
-   </td>
-   <td><strong>MMLU Pro</strong>
-   </td>
-   <td><strong>GPQA Diamond</strong>
-   </td>
-   <td><strong>τ²-Bench Telecom</strong>
-   </td>
-   <td><strong>SciCode</strong>
-   </td>
-   <td><strong>AIME 2025</strong>
-   </td>
-   <td><strong>AA-LCR</strong>
-   </td>
-   <td><strong>IFBench</strong>
-   </td>
-   <td><strong>MMMU PRO</strong>
-   </td>
-  </tr>
-  <tr>
-   <td>BF16
-   </td>
-   <td><strong>85.6</strong>
-   </td>
-   <td><strong>84.9</strong>
-   </td>
-   <td><strong>95.5</strong>
-   </td>
-   <td><strong>40.8</strong>
-   </td>
-   <td><strong>89.2</strong>
-   </td>
-   <td><strong>62.0</strong>
-   </td>
-   <td><strong>62.3</strong>
-   </td>
-   <td><strong>74.1</strong>
-   </td>
-  </tr>
-  <tr>
-   <td>NVFP4
-   </td>
-   <td><strong>85.0</strong>
-   </td>
-   <td><strong>84.8</strong>
-   </td>
-   <td><strong>94.7</strong>
-   </td>
-   <td><strong>40.6</strong>
-   </td>
-   <td><strong>88.8</strong>
-   </td>
-   <td><strong>62.0</strong>
-   </td>
-   <td><strong>62.8</strong>
-   </td>
-   <td><strong>74.5</strong>
-   </td>
-  </tr>
-</table>
-
-> Baseline: [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B).
-> SciCode with temperature=0.6, top_p=0.95, max num tokens 131072; the others with temperature=1.0, top_p=0.95, max num tokens 131072
-
-## Model Limitations:
-The base model was trained on data that contains toxic language and societal biases originally crawled from the internet. Therefore, the model may amplify those biases and return toxic responses especially when prompted with toxic prompts. The model may generate answers that may be inaccurate, omit key information, or include irrelevant or redundant text producing socially unacceptable or undesirable text, even if the prompt itself does not include anything explicitly offensive.
-
-## Ethical Considerations
-
-NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
-
-Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
-
-Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).

config.json

@@ -6,7 +6,7 @@
   "image_token_id": 248056,
   "model_type": "qwen3_5_moe",
   "tie_word_embeddings": false,
-  "transformers_version": "5.7.0.dev0",
+  "transformers_version": "5.14.0.dev0",
   "video_token_id": 248057,
   "vision_config": {
     "deepstack_visual_indexes": [],
@@ -30,156 +30,6 @@
   "quantization_config": {
     "config_groups": {
       "group_0": {
-        "input_activations": {
-          "dynamic": false,
-          "num_bits": 8,
-          "type": "float"
-        },
-        "weights": {
-          "dynamic": false,
-          "num_bits": 8,
-          "type": "float"
-        },
-        "targets": [
-          "model.language_model.layers.0.linear_attn.in_proj_qkv",
-          "model.language_model.layers.0.linear_attn.in_proj_z",
-          "model.language_model.layers.0.linear_attn.out_proj",
-          "model.language_model.layers.1.linear_attn.in_proj_qkv",
-          "model.language_model.layers.1.linear_attn.in_proj_z",
-          "model.language_model.layers.1.linear_attn.out_proj",
-          "model.language_model.layers.10.linear_attn.in_proj_qkv",
-          "model.language_model.layers.10.linear_attn.in_proj_z",
-          "model.language_model.layers.10.linear_attn.out_proj",
-          "model.language_model.layers.11.self_attn.k_proj",
-          "model.language_model.layers.11.self_attn.o_proj",
-          "model.language_model.layers.11.self_attn.q_proj",
-          "model.language_model.layers.11.self_attn.v_proj",
-          "model.language_model.layers.12.linear_attn.in_proj_qkv",
-          "model.language_model.layers.12.linear_attn.in_proj_z",
-          "model.language_model.layers.12.linear_attn.out_proj",
-          "model.language_model.layers.13.linear_attn.in_proj_qkv",
-          "model.language_model.layers.13.linear_attn.in_proj_z",
-          "model.language_model.layers.13.linear_attn.out_proj",
-          "model.language_model.layers.14.linear_attn.in_proj_qkv",
-          "model.language_model.layers.14.linear_attn.in_proj_z",
-          "model.language_model.layers.14.linear_attn.out_proj",
-          "model.language_model.layers.15.self_attn.k_proj",
-          "model.language_model.layers.15.self_attn.o_proj",
-          "model.language_model.layers.15.self_attn.q_proj",
-          "model.language_model.layers.15.self_attn.v_proj",
-          "model.language_model.layers.16.linear_attn.in_proj_qkv",
-          "model.language_model.layers.16.linear_attn.in_proj_z",
-          "model.language_model.layers.16.linear_attn.out_proj",
-          "model.language_model.layers.17.linear_attn.in_proj_qkv",
-          "model.language_model.layers.17.linear_attn.in_proj_z",
-          "model.language_model.layers.17.linear_attn.out_proj",
-          "model.language_model.layers.18.linear_attn.in_proj_qkv",
-          "model.language_model.layers.18.linear_attn.in_proj_z",
-          "model.language_model.layers.18.linear_attn.out_proj",
-          "model.language_model.layers.19.self_attn.k_proj",
-          "model.language_model.layers.19.self_attn.o_proj",
-          "model.language_model.layers.19.self_attn.q_proj",
-          "model.language_model.layers.19.self_attn.v_proj",
-          "model.language_model.layers.2.linear_attn.in_proj_qkv",
-          "model.language_model.layers.2.linear_attn.in_proj_z",
-          "model.language_model.layers.2.linear_attn.out_proj",
-          "model.language_model.layers.20.linear_attn.in_proj_qkv",
-          "model.language_model.layers.20.linear_attn.in_proj_z",
-          "model.language_model.layers.20.linear_attn.out_proj",
-          "model.language_model.layers.21.linear_attn.in_proj_qkv",
-          "model.language_model.layers.21.linear_attn.in_proj_z",
-          "model.language_model.layers.21.linear_attn.out_proj",
-          "model.language_model.layers.22.linear_attn.in_proj_qkv",
-          "model.language_model.layers.22.linear_attn.in_proj_z",
-          "model.language_model.layers.22.linear_attn.out_proj",
-          "model.language_model.layers.23.self_attn.k_proj",
-          "model.language_model.layers.23.self_attn.o_proj",
-          "model.language_model.layers.23.self_attn.q_proj",
-          "model.language_model.layers.23.self_attn.v_proj",
-          "model.language_model.layers.24.linear_attn.in_proj_qkv",
-          "model.language_model.layers.24.linear_attn.in_proj_z",
-          "model.language_model.layers.24.linear_attn.out_proj",
-          "model.language_model.layers.25.linear_attn.in_proj_qkv",
-          "model.language_model.layers.25.linear_attn.in_proj_z",
-          "model.language_model.layers.25.linear_attn.out_proj",
-          "model.language_model.layers.26.linear_attn.in_proj_qkv",
-          "model.language_model.layers.26.linear_attn.in_proj_z",
-          "model.language_model.layers.26.linear_attn.out_proj",
-          "model.language_model.layers.27.self_attn.k_proj",
-          "model.language_model.layers.27.self_attn.o_proj",
-          "model.language_model.layers.27.self_attn.q_proj",
-          "model.language_model.layers.27.self_attn.v_proj",
-          "model.language_model.layers.28.linear_attn.in_proj_qkv",
-          "model.language_model.layers.28.linear_attn.in_proj_z",
-          "model.language_model.layers.28.linear_attn.out_proj",
-          "model.language_model.layers.29.linear_attn.in_proj_qkv",
-          "model.language_model.layers.29.linear_attn.in_proj_z",
-          "model.language_model.layers.29.linear_attn.out_proj",
-          "model.language_model.layers.3.self_attn.k_proj",
-          "model.language_model.layers.3.self_attn.o_proj",
-          "model.language_model.layers.3.self_attn.q_proj",
-          "model.language_model.layers.3.self_attn.v_proj",
-          "model.language_model.layers.30.linear_attn.in_proj_qkv",
-          "model.language_model.layers.30.linear_attn.in_proj_z",
-          "model.language_model.layers.30.linear_attn.out_proj",
-          "model.language_model.layers.31.self_attn.k_proj",
-          "model.language_model.layers.31.self_attn.o_proj",
-          "model.language_model.layers.31.self_attn.q_proj",
-          "model.language_model.layers.31.self_attn.v_proj",
-          "model.language_model.layers.32.linear_attn.in_proj_qkv",
-          "model.language_model.layers.32.linear_attn.in_proj_z",
-          "model.language_model.layers.32.linear_attn.out_proj",
-          "model.language_model.layers.33.linear_attn.in_proj_qkv",
-          "model.language_model.layers.33.linear_attn.in_proj_z",
-          "model.language_model.layers.33.linear_attn.out_proj",
-          "model.language_model.layers.34.linear_attn.in_proj_qkv",
-          "model.language_model.layers.34.linear_attn.in_proj_z",
-          "model.language_model.layers.34.linear_attn.out_proj",
-          "model.language_model.layers.35.self_attn.k_proj",
-          "model.language_model.layers.35.self_attn.o_proj",
-          "model.language_model.layers.35.self_attn.q_proj",
-          "model.language_model.layers.35.self_attn.v_proj",
-          "model.language_model.layers.36.linear_attn.in_proj_qkv",
-          "model.language_model.layers.36.linear_attn.in_proj_z",
-          "model.language_model.layers.36.linear_attn.out_proj",
-          "model.language_model.layers.37.linear_attn.in_proj_qkv",
-          "model.language_model.layers.37.linear_attn.in_proj_z",
-          "model.language_model.layers.37.linear_attn.out_proj",
-          "model.language_model.layers.38.linear_attn.in_proj_qkv",
-          "model.language_model.layers.38.linear_attn.in_proj_z",
-          "model.language_model.layers.38.linear_attn.out_proj",
-          "model.language_model.layers.39.self_attn.k_proj",
-          "model.language_model.layers.39.self_attn.o_proj",
-          "model.language_model.layers.39.self_attn.q_proj",
-          "model.language_model.layers.39.self_attn.v_proj",
-          "model.language_model.layers.4.linear_attn.in_proj_qkv",
-          "model.language_model.layers.4.linear_attn.in_proj_z",
-          "model.language_model.layers.4.linear_attn.out_proj",
-          "model.language_model.layers.5.linear_attn.in_proj_qkv",
-          "model.language_model.layers.5.linear_attn.in_proj_z",
-          "model.language_model.layers.5.linear_attn.out_proj",
-          "model.language_model.layers.6.linear_attn.in_proj_qkv",
-          "model.language_model.layers.6.linear_attn.in_proj_z",
-          "model.language_model.layers.6.linear_attn.out_proj",
-          "model.language_model.layers.7.self_attn.k_proj",
-          "model.language_model.layers.7.self_attn.o_proj",
-          "model.language_model.layers.7.self_attn.q_proj",
-          "model.language_model.layers.7.self_attn.v_proj",
-          "model.language_model.layers.8.linear_attn.in_proj_qkv",
-          "model.language_model.layers.8.linear_attn.in_proj_z",
-          "model.language_model.layers.8.linear_attn.out_proj",
-          "model.language_model.layers.9.linear_attn.in_proj_qkv",
-          "model.language_model.layers.9.linear_attn.in_proj_z",
-          "model.language_model.layers.9.linear_attn.out_proj"
-        ]
-      },
-      "group_1": {
-        "input_activations": {
-          "dynamic": false,
-          "num_bits": 4,
-          "type": "float",
-          "group_size": 16
-        },
         "weights": {
           "dynamic": false,
           "num_bits": 4,
@@ -187,1219 +37,193 @@
           "group_size": 16
         },
         "targets": [
-          "lm_head",
-          "model.language_model.layers.0.mlp.experts",
-          "model.language_model.layers.0.mlp.shared_expert.down_proj",
-          "model.language_model.layers.0.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.0.mlp.shared_expert.up_proj",
-          "model.language_model.layers.1.mlp.experts",
-          "model.language_model.layers.1.mlp.shared_expert.down_proj",
-          "model.language_model.layers.1.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.1.mlp.shared_expert.up_proj",
-          "model.language_model.layers.10.mlp.experts",
-          "model.language_model.layers.10.mlp.shared_expert.down_proj",
-          "model.language_model.layers.10.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.10.mlp.shared_expert.up_proj",
-          "model.language_model.layers.11.mlp.experts",
-          "model.language_model.layers.11.mlp.shared_expert.down_proj",
-          "model.language_model.layers.11.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.11.mlp.shared_expert.up_proj",
-          "model.language_model.layers.12.mlp.experts",
-          "model.language_model.layers.12.mlp.shared_expert.down_proj",
-          "model.language_model.layers.12.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.12.mlp.shared_expert.up_proj",
-          "model.language_model.layers.13.mlp.experts",
-          "model.language_model.layers.13.mlp.shared_expert.down_proj",
-          "model.language_model.layers.13.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.13.mlp.shared_expert.up_proj",
-          "model.language_model.layers.14.mlp.experts",
-          "model.language_model.layers.14.mlp.shared_expert.down_proj",
-          "model.language_model.layers.14.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.14.mlp.shared_expert.up_proj",
-          "model.language_model.layers.15.mlp.experts",
-          "model.language_model.layers.15.mlp.shared_expert.down_proj",
-          "model.language_model.layers.15.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.15.mlp.shared_expert.up_proj",
-          "model.language_model.layers.16.mlp.experts",
-          "model.language_model.layers.16.mlp.shared_expert.down_proj",
-          "model.language_model.layers.16.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.16.mlp.shared_expert.up_proj",
-          "model.language_model.layers.17.mlp.experts",
-          "model.language_model.layers.17.mlp.shared_expert.down_proj",
-          "model.language_model.layers.17.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.17.mlp.shared_expert.up_proj",
-          "model.language_model.layers.18.mlp.experts",
-          "model.language_model.layers.18.mlp.shared_expert.down_proj",
-          "model.language_model.layers.18.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.18.mlp.shared_expert.up_proj",
-          "model.language_model.layers.19.mlp.experts",
-          "model.language_model.layers.19.mlp.shared_expert.down_proj",
-          "model.language_model.layers.19.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.19.mlp.shared_expert.up_proj",
-          "model.language_model.layers.2.mlp.experts",
-          "model.language_model.layers.2.mlp.shared_expert.down_proj",
-          "model.language_model.layers.2.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.2.mlp.shared_expert.up_proj",
-          "model.language_model.layers.20.mlp.experts",
-          "model.language_model.layers.20.mlp.shared_expert.down_proj",
-          "model.language_model.layers.20.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.20.mlp.shared_expert.up_proj",
-          "model.language_model.layers.21.mlp.experts",
-          "model.language_model.layers.21.mlp.shared_expert.down_proj",
-          "model.language_model.layers.21.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.21.mlp.shared_expert.up_proj",
-          "model.language_model.layers.22.mlp.experts",
-          "model.language_model.layers.22.mlp.shared_expert.down_proj",
-          "model.language_model.layers.22.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.22.mlp.shared_expert.up_proj",
-          "model.language_model.layers.23.mlp.experts",
-          "model.language_model.layers.23.mlp.shared_expert.down_proj",
-          "model.language_model.layers.23.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.23.mlp.shared_expert.up_proj",
-          "model.language_model.layers.24.mlp.experts",
-          "model.language_model.layers.24.mlp.shared_expert.down_proj",
-          "model.language_model.layers.24.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.24.mlp.shared_expert.up_proj",
-          "model.language_model.layers.25.mlp.experts",
-          "model.language_model.layers.25.mlp.shared_expert.down_proj",
-          "model.language_model.layers.25.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.25.mlp.shared_expert.up_proj",
-          "model.language_model.layers.26.mlp.experts",
-          "model.language_model.layers.26.mlp.shared_expert.down_proj",
-          "model.language_model.layers.26.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.26.mlp.shared_expert.up_proj",
-          "model.language_model.layers.27.mlp.experts",
-          "model.language_model.layers.27.mlp.shared_expert.down_proj",
-          "model.language_model.layers.27.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.27.mlp.shared_expert.up_proj",
-          "model.language_model.layers.28.mlp.experts",
-          "model.language_model.layers.28.mlp.shared_expert.down_proj",
-          "model.language_model.layers.28.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.28.mlp.shared_expert.up_proj",
-          "model.language_model.layers.29.mlp.experts",
-          "model.language_model.layers.29.mlp.shared_expert.down_proj",
-          "model.language_model.layers.29.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.29.mlp.shared_expert.up_proj",
-          "model.language_model.layers.3.mlp.experts",
-          "model.language_model.layers.3.mlp.shared_expert.down_proj",
-          "model.language_model.layers.3.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.3.mlp.shared_expert.up_proj",
-          "model.language_model.layers.30.mlp.experts",
-          "model.language_model.layers.30.mlp.shared_expert.down_proj",
-          "model.language_model.layers.30.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.30.mlp.shared_expert.up_proj",
-          "model.language_model.layers.31.mlp.experts",
-          "model.language_model.layers.31.mlp.shared_expert.down_proj",
-          "model.language_model.layers.31.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.31.mlp.shared_expert.up_proj",
-          "model.language_model.layers.32.mlp.experts",
-          "model.language_model.layers.32.mlp.shared_expert.down_proj",
-          "model.language_model.layers.32.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.32.mlp.shared_expert.up_proj",
-          "model.language_model.layers.33.mlp.experts",
-          "model.language_model.layers.33.mlp.shared_expert.down_proj",
-          "model.language_model.layers.33.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.33.mlp.shared_expert.up_proj",
-          "model.language_model.layers.34.mlp.experts",
-          "model.language_model.layers.34.mlp.shared_expert.down_proj",
-          "model.language_model.layers.34.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.34.mlp.shared_expert.up_proj",
-          "model.language_model.layers.35.mlp.experts",
-          "model.language_model.layers.35.mlp.shared_expert.down_proj",
-          "model.language_model.layers.35.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.35.mlp.shared_expert.up_proj",
-          "model.language_model.layers.36.mlp.experts",
-          "model.language_model.layers.36.mlp.shared_expert.down_proj",
-          "model.language_model.layers.36.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.36.mlp.shared_expert.up_proj",
-          "model.language_model.layers.37.mlp.experts",
-          "model.language_model.layers.37.mlp.shared_expert.down_proj",
-          "model.language_model.layers.37.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.37.mlp.shared_expert.up_proj",
-          "model.language_model.layers.38.mlp.experts",
-          "model.language_model.layers.38.mlp.shared_expert.down_proj",
-          "model.language_model.layers.38.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.38.mlp.shared_expert.up_proj",
-          "model.language_model.layers.39.mlp.experts",
-          "model.language_model.layers.39.mlp.shared_expert.down_proj",
-          "model.language_model.layers.39.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.39.mlp.shared_expert.up_proj",
-          "model.language_model.layers.4.mlp.experts",
-          "model.language_model.layers.4.mlp.shared_expert.down_proj",
-          "model.language_model.layers.4.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.4.mlp.shared_expert.up_proj",
-          "model.language_model.layers.5.mlp.experts",
-          "model.language_model.layers.5.mlp.shared_expert.down_proj",
-          "model.language_model.layers.5.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.5.mlp.shared_expert.up_proj",
-          "model.language_model.layers.6.mlp.experts",
-          "model.language_model.layers.6.mlp.shared_expert.down_proj",
-          "model.language_model.layers.6.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.6.mlp.shared_expert.up_proj",
-          "model.language_model.layers.7.mlp.experts",
-          "model.language_model.layers.7.mlp.shared_expert.down_proj",
-          "model.language_model.layers.7.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.7.mlp.shared_expert.up_proj",
-          "model.language_model.layers.8.mlp.experts",
-          "model.language_model.layers.8.mlp.shared_expert.down_proj",
-          "model.language_model.layers.8.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.8.mlp.shared_expert.up_proj",
-          "model.language_model.layers.9.mlp.experts",
-          "model.language_model.layers.9.mlp.shared_expert.down_proj",
-          "model.language_model.layers.9.mlp.shared_expert.gate_proj",
-          "model.language_model.layers.9.mlp.shared_expert.up_proj"
+          "Linear"
         ]
       }
     },
-    "quantized_layers": {
-      "model.language_model.layers.0.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.0.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.0.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.0.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.0.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.0.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.0.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.1.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.1.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.1.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.2.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.2.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.2.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.4.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.4.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.4.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.5.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.5.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.5.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.6.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.6.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.6.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.8.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.8.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.8.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.9.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.9.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.9.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.10.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.10.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.10.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.12.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.12.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.12.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.13.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.13.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.13.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.14.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.14.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.14.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.16.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.16.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.16.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.17.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.17.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.17.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.18.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.18.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.18.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.20.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.20.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.20.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.21.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.21.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.21.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.22.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.22.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.22.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.24.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.24.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.24.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.25.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.25.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.25.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.26.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.26.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.26.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.28.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.28.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.28.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.29.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.29.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.29.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.30.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.30.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.30.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.32.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.32.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.32.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.33.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.33.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.33.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.34.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.34.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.34.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.36.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.36.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.36.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.37.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.37.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.37.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.38.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.38.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.38.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "lm_head": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      }
-    },
     "ignore": [
-      "mtp.layers.0*",
-      "mtp*",
-      "mtp",
-      "re:.*mtp.*"
+      "lm_head",
+      "model.language_model.embed_tokens",
+      "model.language_model.layers.0.linear_attn.conv1d",
+      "model.language_model.layers.0.linear_attn.in_proj_a",
+      "model.language_model.layers.0.linear_attn.in_proj_b",
+      "model.language_model.layers.0.mlp.gate",
+      "model.language_model.layers.0.mlp.shared_expert_gate",
+      "model.language_model.layers.1.linear_attn.conv1d",
+      "model.language_model.layers.1.linear_attn.in_proj_a",
+      "model.language_model.layers.1.linear_attn.in_proj_b",
+      "model.language_model.layers.1.mlp.gate",
+      "model.language_model.layers.1.mlp.shared_expert_gate",
+      "model.language_model.layers.10.linear_attn.conv1d",
+      "model.language_model.layers.10.linear_attn.in_proj_a",
+      "model.language_model.layers.10.linear_attn.in_proj_b",
+      "model.language_model.layers.10.mlp.gate",
+      "model.language_model.layers.10.mlp.shared_expert_gate",
+      "model.language_model.layers.11.mlp.gate",
+      "model.language_model.layers.11.mlp.shared_expert_gate",
+      "model.language_model.layers.12.linear_attn.conv1d",
+      "model.language_model.layers.12.linear_attn.in_proj_a",
+      "model.language_model.layers.12.linear_attn.in_proj_b",
+      "model.language_model.layers.12.mlp.gate",
+      "model.language_model.layers.12.mlp.shared_expert_gate",
+      "model.language_model.layers.13.linear_attn.conv1d",
+      "model.language_model.layers.13.linear_attn.in_proj_a",
+      "model.language_model.layers.13.linear_attn.in_proj_b",
+      "model.language_model.layers.13.mlp.gate",
+      "model.language_model.layers.13.mlp.shared_expert_gate",
+      "model.language_model.layers.14.linear_attn.conv1d",
+      "model.language_model.layers.14.linear_attn.in_proj_a",
+      "model.language_model.layers.14.linear_attn.in_proj_b",
+      "model.language_model.layers.14.mlp.gate",
+      "model.language_model.layers.14.mlp.shared_expert_gate",
+      "model.language_model.layers.15.mlp.gate",
+      "model.language_model.layers.15.mlp.shared_expert_gate",
+      "model.language_model.layers.16.linear_attn.conv1d",
+      "model.language_model.layers.16.linear_attn.in_proj_a",
+      "model.language_model.layers.16.linear_attn.in_proj_b",
+      "model.language_model.layers.16.mlp.gate",
+      "model.language_model.layers.16.mlp.shared_expert_gate",
+      "model.language_model.layers.17.linear_attn.conv1d",
+      "model.language_model.layers.17.linear_attn.in_proj_a",
+      "model.language_model.layers.17.linear_attn.in_proj_b",
+      "model.language_model.layers.17.mlp.gate",
+      "model.language_model.layers.17.mlp.shared_expert_gate",
+      "model.language_model.layers.18.linear_attn.conv1d",
+      "model.language_model.layers.18.linear_attn.in_proj_a",
+      "model.language_model.layers.18.linear_attn.in_proj_b",
+      "model.language_model.layers.18.mlp.gate",
+      "model.language_model.layers.18.mlp.shared_expert_gate",
+      "model.language_model.layers.19.mlp.gate",
+      "model.language_model.layers.19.mlp.shared_expert_gate",
+      "model.language_model.layers.2.linear_attn.conv1d",
+      "model.language_model.layers.2.linear_attn.in_proj_a",
+      "model.language_model.layers.2.linear_attn.in_proj_b",
+      "model.language_model.layers.2.mlp.gate",
+      "model.language_model.layers.2.mlp.shared_expert_gate",
+      "model.language_model.layers.20.linear_attn.conv1d",
+      "model.language_model.layers.20.linear_attn.in_proj_a",
+      "model.language_model.layers.20.linear_attn.in_proj_b",
+      "model.language_model.layers.20.mlp.gate",
+      "model.language_model.layers.20.mlp.shared_expert_gate",
+      "model.language_model.layers.21.linear_attn.conv1d",
+      "model.language_model.layers.21.linear_attn.in_proj_a",
+      "model.language_model.layers.21.linear_attn.in_proj_b",
+      "model.language_model.layers.21.mlp.gate",
+      "model.language_model.layers.21.mlp.shared_expert_gate",
+      "model.language_model.layers.22.linear_attn.conv1d",
+      "model.language_model.layers.22.linear_attn.in_proj_a",
+      "model.language_model.layers.22.linear_attn.in_proj_b",
+      "model.language_model.layers.22.mlp.gate",
+      "model.language_model.layers.22.mlp.shared_expert_gate",
+      "model.language_model.layers.23.mlp.gate",
+      "model.language_model.layers.23.mlp.shared_expert_gate",
+      "model.language_model.layers.24.linear_attn.conv1d",
+      "model.language_model.layers.24.linear_attn.in_proj_a",
+      "model.language_model.layers.24.linear_attn.in_proj_b",
+      "model.language_model.layers.24.mlp.gate",
+      "model.language_model.layers.24.mlp.shared_expert_gate",
+      "model.language_model.layers.25.linear_attn.conv1d",
+      "model.language_model.layers.25.linear_attn.in_proj_a",
+      "model.language_model.layers.25.linear_attn.in_proj_b",
+      "model.language_model.layers.25.mlp.gate",
+      "model.language_model.layers.25.mlp.shared_expert_gate",
+      "model.language_model.layers.26.linear_attn.conv1d",
+      "model.language_model.layers.26.linear_attn.in_proj_a",
+      "model.language_model.layers.26.linear_attn.in_proj_b",
+      "model.language_model.layers.26.mlp.gate",
+      "model.language_model.layers.26.mlp.shared_expert_gate",
+      "model.language_model.layers.27.mlp.gate",
+      "model.language_model.layers.27.mlp.shared_expert_gate",
+      "model.language_model.layers.28.linear_attn.conv1d",
+      "model.language_model.layers.28.linear_attn.in_proj_a",
+      "model.language_model.layers.28.linear_attn.in_proj_b",
+      "model.language_model.layers.28.mlp.gate",
+      "model.language_model.layers.28.mlp.shared_expert_gate",
+      "model.language_model.layers.29.linear_attn.conv1d",
+      "model.language_model.layers.29.linear_attn.in_proj_a",
+      "model.language_model.layers.29.linear_attn.in_proj_b",
+      "model.language_model.layers.29.mlp.gate",
+      "model.language_model.layers.29.mlp.shared_expert_gate",
+      "model.language_model.layers.3.mlp.gate",
+      "model.language_model.layers.3.mlp.shared_expert_gate",
+      "model.language_model.layers.30.linear_attn.conv1d",
+      "model.language_model.layers.30.linear_attn.in_proj_a",
+      "model.language_model.layers.30.linear_attn.in_proj_b",
+      "model.language_model.layers.30.mlp.gate",
+      "model.language_model.layers.30.mlp.shared_expert_gate",
+      "model.language_model.layers.31.mlp.gate",
+      "model.language_model.layers.31.mlp.shared_expert_gate",
+      "model.language_model.layers.32.linear_attn.conv1d",
+      "model.language_model.layers.32.linear_attn.in_proj_a",
+      "model.language_model.layers.32.linear_attn.in_proj_b",
+      "model.language_model.layers.32.mlp.gate",
+      "model.language_model.layers.32.mlp.shared_expert_gate",
+      "model.language_model.layers.33.linear_attn.conv1d",
+      "model.language_model.layers.33.linear_attn.in_proj_a",
+      "model.language_model.layers.33.linear_attn.in_proj_b",
+      "model.language_model.layers.33.mlp.gate",
+      "model.language_model.layers.33.mlp.shared_expert_gate",
+      "model.language_model.layers.34.linear_attn.conv1d",
+      "model.language_model.layers.34.linear_attn.in_proj_a",
+      "model.language_model.layers.34.linear_attn.in_proj_b",
+      "model.language_model.layers.34.mlp.gate",
+      "model.language_model.layers.34.mlp.shared_expert_gate",
+      "model.language_model.layers.35.mlp.gate",
+      "model.language_model.layers.35.mlp.shared_expert_gate",
+      "model.language_model.layers.36.linear_attn.conv1d",
+      "model.language_model.layers.36.linear_attn.in_proj_a",
+      "model.language_model.layers.36.linear_attn.in_proj_b",
+      "model.language_model.layers.36.mlp.gate",
+      "model.language_model.layers.36.mlp.shared_expert_gate",
+      "model.language_model.layers.37.linear_attn.conv1d",
+      "model.language_model.layers.37.linear_attn.in_proj_a",
+      "model.language_model.layers.37.linear_attn.in_proj_b",
+      "model.language_model.layers.37.mlp.gate",
+      "model.language_model.layers.37.mlp.shared_expert_gate",
+      "model.language_model.layers.38.linear_attn.conv1d",
+      "model.language_model.layers.38.linear_attn.in_proj_a",
+      "model.language_model.layers.38.linear_attn.in_proj_b",
+      "model.language_model.layers.38.mlp.gate",
+      "model.language_model.layers.38.mlp.shared_expert_gate",
+      "model.language_model.layers.39.mlp.gate",
+      "model.language_model.layers.39.mlp.shared_expert_gate",
+      "model.language_model.layers.4.linear_attn.conv1d",
+      "model.language_model.layers.4.linear_attn.in_proj_a",
+      "model.language_model.layers.4.linear_attn.in_proj_b",
+      "model.language_model.layers.4.mlp.gate",
+      "model.language_model.layers.4.mlp.shared_expert_gate",
+      "model.language_model.layers.5.linear_attn.conv1d",
+      "model.language_model.layers.5.linear_attn.in_proj_a",
+      "model.language_model.layers.5.linear_attn.in_proj_b",
+      "model.language_model.layers.5.mlp.gate",
+      "model.language_model.layers.5.mlp.shared_expert_gate",
+      "model.language_model.layers.6.linear_attn.conv1d",
+      "model.language_model.layers.6.linear_attn.in_proj_a",
+      "model.language_model.layers.6.linear_attn.in_proj_b",
+      "model.language_model.layers.6.mlp.gate",
+      "model.language_model.layers.6.mlp.shared_expert_gate",
+      "model.language_model.layers.7.mlp.gate",
+      "model.language_model.layers.7.mlp.shared_expert_gate",
+      "model.language_model.layers.8.linear_attn.conv1d",
+      "model.language_model.layers.8.linear_attn.in_proj_a",
+      "model.language_model.layers.8.linear_attn.in_proj_b",
+      "model.language_model.layers.8.mlp.gate",
+      "model.language_model.layers.8.mlp.shared_expert_gate",
+      "model.language_model.layers.9.linear_attn.conv1d",
+      "model.language_model.layers.9.linear_attn.in_proj_a",
+      "model.language_model.layers.9.linear_attn.in_proj_b",
+      "model.language_model.layers.9.mlp.gate",
+      "model.language_model.layers.9.mlp.shared_expert_gate",
+      "model.visual*"
     ],
     "quant_algo": "W4A16_NVFP4",
     "producer": {
       "name": "modelopt",
-      "version": "0.37.0"
+      "version": "0.46.0.dev135+gb0ee95363"
     },
     "quant_method": "modelopt"
   },
+  "max_position_embeddings": 112000,
   "attention_bias": false,
   "attention_dropout": 0.0,
   "attn_output_gate": true,
@@ -1458,7 +282,6 @@
   "linear_num_value_heads": 32,
   "linear_value_head_dim": 128,
   "mamba_ssm_dtype": "float32",
-  "max_position_embeddings": 262144,
   "moe_intermediate_size": 512,
   "num_attention_heads": 16,
   "num_experts": 256,

generation_config.json

@@ -1,12 +1,13 @@
 {
-    "bos_token_id": 248044,
-    "do_sample": true,
-    "eos_token_id": [
-        248046,
-        248044
-    ],
-    "pad_token_id": 248044,
-    "temperature": 1.0,
-    "top_k": 20,
-    "top_p": 0.95
+  "bos_token_id": 248044,
+  "do_sample": true,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
+  "pad_token_id": 248044,
+  "temperature": 1.0,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
 }

hf_quant_config.json

@@ -1,1050 +1,186 @@
 {
-  "producer": {
-    "name": "modelopt",
-    "version": "0.44.0"
-  },
-  "quantization": {
-    "quant_algo": "MIXED_PRECISION",
-    "kv_cache_quant_algo": "FP8",
-    "quantized_layers": {
-      "model.language_model.layers.0.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.0.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.0.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.0.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.0.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.0.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.0.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.1.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.1.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.1.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.1.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.2.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.2.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.2.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.2.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.3.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.3.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.4.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.4.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.4.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.4.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.5.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.5.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.5.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.5.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.6.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.6.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.6.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.6.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.7.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.7.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.8.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.8.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.8.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.8.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.9.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.9.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.9.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.9.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.10.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.10.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.10.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.10.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.11.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.11.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.12.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.12.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.12.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.12.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.13.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.13.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.13.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.13.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.14.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.14.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.14.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.14.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.15.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.15.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.16.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.16.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.16.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.16.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.17.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.17.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.17.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.17.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.18.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.18.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.18.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.18.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.19.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.19.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.20.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.20.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.20.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.20.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.21.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.21.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.21.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.21.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.22.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.22.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.22.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.22.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.23.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.23.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.24.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.24.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.24.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.24.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.25.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.25.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.25.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.25.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.26.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.26.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.26.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.26.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.27.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.27.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.28.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.28.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.28.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.28.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.29.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.29.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.29.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.29.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.30.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.30.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.30.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.30.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.31.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.31.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.32.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.32.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.32.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.32.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.33.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.33.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.33.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.33.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.34.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.34.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.34.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.34.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.35.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.35.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.36.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.36.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.36.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.36.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.37.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.37.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.37.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.37.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.linear_attn.out_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.38.linear_attn.in_proj_qkv": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.38.linear_attn.in_proj_z": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.38.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.38.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.self_attn.q_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.self_attn.k_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.self_attn.v_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.self_attn.o_proj": {
-        "quant_algo": "FP8"
-      },
-      "model.language_model.layers.39.mlp.experts": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.mlp.shared_expert.gate_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.mlp.shared_expert.up_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "model.language_model.layers.39.mlp.shared_expert.down_proj": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      },
-      "lm_head": {
-        "quant_algo": "W4A16_NVFP4",
-        "group_size": 16
-      }
+    "producer": {
+        "name": "modelopt",
+        "version": "0.46.0.dev135+gb0ee95363"
     },
-    "exclude_modules": [
-      "mtp.layers.0*",
-      "mtp*"
-    ]
-  }
-}
+    "quantization": {
+        "quant_algo": "W4A16_NVFP4",
+        "kv_cache_quant_algo": null,
+        "group_size": 16,
+        "exclude_modules": [
+            "lm_head",
+            "model.language_model.embed_tokens",
+            "model.language_model.layers.0.linear_attn.conv1d",
+            "model.language_model.layers.0.linear_attn.in_proj_a",
+            "model.language_model.layers.0.linear_attn.in_proj_b",
+            "model.language_model.layers.0.mlp.gate",
+            "model.language_model.layers.0.mlp.shared_expert_gate",
+            "model.language_model.layers.1.linear_attn.conv1d",
+            "model.language_model.layers.1.linear_attn.in_proj_a",
+            "model.language_model.layers.1.linear_attn.in_proj_b",
+            "model.language_model.layers.1.mlp.gate",
+            "model.language_model.layers.1.mlp.shared_expert_gate",
+            "model.language_model.layers.10.linear_attn.conv1d",
+            "model.language_model.layers.10.linear_attn.in_proj_a",
+            "model.language_model.layers.10.linear_attn.in_proj_b",
+            "model.language_model.layers.10.mlp.gate",
+            "model.language_model.layers.10.mlp.shared_expert_gate",
+            "model.language_model.layers.11.mlp.gate",
+            "model.language_model.layers.11.mlp.shared_expert_gate",
+            "model.language_model.layers.12.linear_attn.conv1d",
+            "model.language_model.layers.12.linear_attn.in_proj_a",
+            "model.language_model.layers.12.linear_attn.in_proj_b",
+            "model.language_model.layers.12.mlp.gate",
+            "model.language_model.layers.12.mlp.shared_expert_gate",
+            "model.language_model.layers.13.linear_attn.conv1d",
+            "model.language_model.layers.13.linear_attn.in_proj_a",
+            "model.language_model.layers.13.linear_attn.in_proj_b",
+            "model.language_model.layers.13.mlp.gate",
+            "model.language_model.layers.13.mlp.shared_expert_gate",
+            "model.language_model.layers.14.linear_attn.conv1d",
+            "model.language_model.layers.14.linear_attn.in_proj_a",
+            "model.language_model.layers.14.linear_attn.in_proj_b",
+            "model.language_model.layers.14.mlp.gate",
+            "model.language_model.layers.14.mlp.shared_expert_gate",
+            "model.language_model.layers.15.mlp.gate",
+            "model.language_model.layers.15.mlp.shared_expert_gate",
+            "model.language_model.layers.16.linear_attn.conv1d",
+            "model.language_model.layers.16.linear_attn.in_proj_a",
+            "model.language_model.layers.16.linear_attn.in_proj_b",
+            "model.language_model.layers.16.mlp.gate",
+            "model.language_model.layers.16.mlp.shared_expert_gate",
+            "model.language_model.layers.17.linear_attn.conv1d",
+            "model.language_model.layers.17.linear_attn.in_proj_a",
+            "model.language_model.layers.17.linear_attn.in_proj_b",
+            "model.language_model.layers.17.mlp.gate",
+            "model.language_model.layers.17.mlp.shared_expert_gate",
+            "model.language_model.layers.18.linear_attn.conv1d",
+            "model.language_model.layers.18.linear_attn.in_proj_a",
+            "model.language_model.layers.18.linear_attn.in_proj_b",
+            "model.language_model.layers.18.mlp.gate",
+            "model.language_model.layers.18.mlp.shared_expert_gate",
+            "model.language_model.layers.19.mlp.gate",
+            "model.language_model.layers.19.mlp.shared_expert_gate",
+            "model.language_model.layers.2.linear_attn.conv1d",
+            "model.language_model.layers.2.linear_attn.in_proj_a",
+            "model.language_model.layers.2.linear_attn.in_proj_b",
+            "model.language_model.layers.2.mlp.gate",
+            "model.language_model.layers.2.mlp.shared_expert_gate",
+            "model.language_model.layers.20.linear_attn.conv1d",
+            "model.language_model.layers.20.linear_attn.in_proj_a",
+            "model.language_model.layers.20.linear_attn.in_proj_b",
+            "model.language_model.layers.20.mlp.gate",
+            "model.language_model.layers.20.mlp.shared_expert_gate",
+            "model.language_model.layers.21.linear_attn.conv1d",
+            "model.language_model.layers.21.linear_attn.in_proj_a",
+            "model.language_model.layers.21.linear_attn.in_proj_b",
+            "model.language_model.layers.21.mlp.gate",
+            "model.language_model.layers.21.mlp.shared_expert_gate",
+            "model.language_model.layers.22.linear_attn.conv1d",
+            "model.language_model.layers.22.linear_attn.in_proj_a",
+            "model.language_model.layers.22.linear_attn.in_proj_b",
+            "model.language_model.layers.22.mlp.gate",
+            "model.language_model.layers.22.mlp.shared_expert_gate",
+            "model.language_model.layers.23.mlp.gate",
+            "model.language_model.layers.23.mlp.shared_expert_gate",
+            "model.language_model.layers.24.linear_attn.conv1d",
+            "model.language_model.layers.24.linear_attn.in_proj_a",
+            "model.language_model.layers.24.linear_attn.in_proj_b",
+            "model.language_model.layers.24.mlp.gate",
+            "model.language_model.layers.24.mlp.shared_expert_gate",
+            "model.language_model.layers.25.linear_attn.conv1d",
+            "model.language_model.layers.25.linear_attn.in_proj_a",
+            "model.language_model.layers.25.linear_attn.in_proj_b",
+            "model.language_model.layers.25.mlp.gate",
+            "model.language_model.layers.25.mlp.shared_expert_gate",
+            "model.language_model.layers.26.linear_attn.conv1d",
+            "model.language_model.layers.26.linear_attn.in_proj_a",
+            "model.language_model.layers.26.linear_attn.in_proj_b",
+            "model.language_model.layers.26.mlp.gate",
+            "model.language_model.layers.26.mlp.shared_expert_gate",
+            "model.language_model.layers.27.mlp.gate",
+            "model.language_model.layers.27.mlp.shared_expert_gate",
+            "model.language_model.layers.28.linear_attn.conv1d",
+            "model.language_model.layers.28.linear_attn.in_proj_a",
+            "model.language_model.layers.28.linear_attn.in_proj_b",
+            "model.language_model.layers.28.mlp.gate",
+            "model.language_model.layers.28.mlp.shared_expert_gate",
+            "model.language_model.layers.29.linear_attn.conv1d",
+            "model.language_model.layers.29.linear_attn.in_proj_a",
+            "model.language_model.layers.29.linear_attn.in_proj_b",
+            "model.language_model.layers.29.mlp.gate",
+            "model.language_model.layers.29.mlp.shared_expert_gate",
+            "model.language_model.layers.3.mlp.gate",
+            "model.language_model.layers.3.mlp.shared_expert_gate",
+            "model.language_model.layers.30.linear_attn.conv1d",
+            "model.language_model.layers.30.linear_attn.in_proj_a",
+            "model.language_model.layers.30.linear_attn.in_proj_b",
+            "model.language_model.layers.30.mlp.gate",
+            "model.language_model.layers.30.mlp.shared_expert_gate",
+            "model.language_model.layers.31.mlp.gate",
+            "model.language_model.layers.31.mlp.shared_expert_gate",
+            "model.language_model.layers.32.linear_attn.conv1d",
+            "model.language_model.layers.32.linear_attn.in_proj_a",
+            "model.language_model.layers.32.linear_attn.in_proj_b",
+            "model.language_model.layers.32.mlp.gate",
+            "model.language_model.layers.32.mlp.shared_expert_gate",
+            "model.language_model.layers.33.linear_attn.conv1d",
+            "model.language_model.layers.33.linear_attn.in_proj_a",
+            "model.language_model.layers.33.linear_attn.in_proj_b",
+            "model.language_model.layers.33.mlp.gate",
+            "model.language_model.layers.33.mlp.shared_expert_gate",
+            "model.language_model.layers.34.linear_attn.conv1d",
+            "model.language_model.layers.34.linear_attn.in_proj_a",
+            "model.language_model.layers.34.linear_attn.in_proj_b",
+            "model.language_model.layers.34.mlp.gate",
+            "model.language_model.layers.34.mlp.shared_expert_gate",
+            "model.language_model.layers.35.mlp.gate",
+            "model.language_model.layers.35.mlp.shared_expert_gate",
+            "model.language_model.layers.36.linear_attn.conv1d",
+            "model.language_model.layers.36.linear_attn.in_proj_a",
+            "model.language_model.layers.36.linear_attn.in_proj_b",
+            "model.language_model.layers.36.mlp.gate",
+            "model.language_model.layers.36.mlp.shared_expert_gate",
+            "model.language_model.layers.37.linear_attn.conv1d",
+            "model.language_model.layers.37.linear_attn.in_proj_a",
+            "model.language_model.layers.37.linear_attn.in_proj_b",
+            "model.language_model.layers.37.mlp.gate",
+            "model.language_model.layers.37.mlp.shared_expert_gate",
+            "model.language_model.layers.38.linear_attn.conv1d",
+            "model.language_model.layers.38.linear_attn.in_proj_a",
+            "model.language_model.layers.38.linear_attn.in_proj_b",
+            "model.language_model.layers.38.mlp.gate",
+            "model.language_model.layers.38.mlp.shared_expert_gate",
+            "model.language_model.layers.39.mlp.gate",
+            "model.language_model.layers.39.mlp.shared_expert_gate",
+            "model.language_model.layers.4.linear_attn.conv1d",
+            "model.language_model.layers.4.linear_attn.in_proj_a",
+            "model.language_model.layers.4.linear_attn.in_proj_b",
+            "model.language_model.layers.4.mlp.gate",
+            "model.language_model.layers.4.mlp.shared_expert_gate",
+            "model.language_model.layers.5.linear_attn.conv1d",
+            "model.language_model.layers.5.linear_attn.in_proj_a",
+            "model.language_model.layers.5.linear_attn.in_proj_b",
+            "model.language_model.layers.5.mlp.gate",
+            "model.language_model.layers.5.mlp.shared_expert_gate",
+            "model.language_model.layers.6.linear_attn.conv1d",
+            "model.language_model.layers.6.linear_attn.in_proj_a",
+            "model.language_model.layers.6.linear_attn.in_proj_b",
+            "model.language_model.layers.6.mlp.gate",
+            "model.language_model.layers.6.mlp.shared_expert_gate",
+            "model.language_model.layers.7.mlp.gate",
+            "model.language_model.layers.7.mlp.shared_expert_gate",
+            "model.language_model.layers.8.linear_attn.conv1d",
+            "model.language_model.layers.8.linear_attn.in_proj_a",
+            "model.language_model.layers.8.linear_attn.in_proj_b",
+            "model.language_model.layers.8.mlp.gate",
+            "model.language_model.layers.8.mlp.shared_expert_gate",
+            "model.language_model.layers.9.linear_attn.conv1d",
+            "model.language_model.layers.9.linear_attn.in_proj_a",
+            "model.language_model.layers.9.linear_attn.in_proj_b",
+            "model.language_model.layers.9.mlp.gate",
+            "model.language_model.layers.9.mlp.shared_expert_gate",
+            "model.visual*"
+        ]
+    }
+}

model-00001-of-00006.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1d4519295ca88253a9f5fb3b97210ab0cdad76d25080c6782a09e0bbd5200282
-size 4297423948
+oid sha256:b40599ab51ebd513d300d9c3a92b968c23d1bd2624db5ab0581783290e53ece1
+size 3759770576

model-00002-of-00006.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:526c50fed25b0e1d8a574cbfc3ee2c8d03c25fd7f52d3ab086736d9aeeb9ded4
-size 4298505928
+oid sha256:6c77d607bbdd2c0dd17df8d32deff523cbfc343c3945b277c1571430f5c34d29
+size 3760338424

model-00003-of-00006.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2ea66a8ce452abff70025bb9d6f7613e74c1f9febcfff15da363c65e3c0e412d
-size 4297877120
+oid sha256:0073177cc4d16ea87c078894c398b572e0c620634c5281764f2854c47c22934e
+size 3759740724

model-00004-of-00006.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6c23f109d6089d1df7bb1562987b927cb2a7952ffcc8a3138911185ea7ab9af7
-size 4298497912
+oid sha256:707a3a3cd73f241eece902be9e3d107eead1865669a82cf8d72b14ac57565c25
+size 3760164508

model-00005-of-00006.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e806cf2529e1c39740a6a7277865abca7a08aa2b3af9fd9529711bb45e0f93a7
-size 4298289192
+oid sha256:63de08db56f84ac79f037294f7420e5cc358a32c86b82fba37def0fe3cc41d30
+size 3760164396

model-00006-of-00006.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3727863b41808570770eab25c741e27d6b4d081c0f342d6aa7877b44a9909c46
-size 244398212
+oid sha256:38ce759165dd781fa0e52c5a014ef34d7d3b416e2413526e07b5943c3d40d47c
+size 3102188676

model.safetensors.index.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0ca224e0ca265402bb8c53a37aa9f54af83a3226cff17a32208a46fefc88c1ad
-size 13724689
+oid sha256:cd3ed671cfae49154def31781222b7053030f4beaab92fc552f2e8b3be160b79
+size 10312036

processor_config.json

@@ -0,0 +1,60 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42
-size 12807982
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -1,305 +1,32 @@
 {
-    "add_prefix_space": false,
-    "added_tokens_decoder": {
-        "248044": {
-            "content": "<|endoftext|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248045": {
-            "content": "<|im_start|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248046": {
-            "content": "<|im_end|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248047": {
-            "content": "<|object_ref_start|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248048": {
-            "content": "<|object_ref_end|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248049": {
-            "content": "<|box_start|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248050": {
-            "content": "<|box_end|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248051": {
-            "content": "<|quad_start|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248052": {
-            "content": "<|quad_end|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248053": {
-            "content": "<|vision_start|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248054": {
-            "content": "<|vision_end|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248055": {
-            "content": "<|vision_pad|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248056": {
-            "content": "<|image_pad|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248057": {
-            "content": "<|video_pad|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248058": {
-            "content": "<tool_call>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248059": {
-            "content": "</tool_call>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248060": {
-            "content": "<|fim_prefix|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248061": {
-            "content": "<|fim_middle|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248062": {
-            "content": "<|fim_suffix|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248063": {
-            "content": "<|fim_pad|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248064": {
-            "content": "<|repo_name|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248065": {
-            "content": "<|file_sep|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248066": {
-            "content": "<tool_response>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248067": {
-            "content": "</tool_response>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248068": {
-            "content": "<think>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248069": {
-            "content": "</think>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": false
-        },
-        "248070": {
-            "content": "<|audio_start|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248071": {
-            "content": "<|audio_end|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248072": {
-            "content": "<tts_pad>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248073": {
-            "content": "<tts_text_bos>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248074": {
-            "content": "<tts_text_eod>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248075": {
-            "content": "<tts_text_bos_single>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        },
-        "248076": {
-            "content": "<|audio_pad|>",
-            "lstrip": false,
-            "normalized": false,
-            "rstrip": false,
-            "single_word": false,
-            "special": true
-        }
-    },
-    "additional_special_tokens": [
-        "<|im_start|>",
-        "<|im_end|>",
-        "<|object_ref_start|>",
-        "<|object_ref_end|>",
-        "<|box_start|>",
-        "<|box_end|>",
-        "<|quad_start|>",
-        "<|quad_end|>",
-        "<|vision_start|>",
-        "<|vision_end|>",
-        "<|vision_pad|>",
-        "<|image_pad|>",
-        "<|video_pad|>"
-    ],
-    "bos_token": null,
-    "chat_template": "{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n    {%- if content is string %}\n        {{- content }}\n    {%- elif content is iterable and content is not mapping %}\n        {%- for item in content %}\n            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}\n                {%- if is_system_content %}\n                    {{- raise_exception('System message cannot contain images.') }}\n                {%- endif %}\n                {%- if do_vision_count %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                {%- endif %}\n                {%- if add_vision_id %}\n                    {{- 'Picture ' ~ image_count.value ~ ': ' }}\n                {%- endif %}\n                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}\n            {%- elif 'video' in item or item.type == 'video' %}\n                {%- if is_system_content %}\n                    {{- raise_exception('System message cannot contain videos.') }}\n                {%- endif %}\n                {%- if do_vision_count %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                {%- endif %}\n                {%- if add_vision_id %}\n                    {{- 'Video ' ~ video_count.value ~ ': ' }}\n                {%- endif %}\n                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}\n            {%- elif 'text' in item %}\n                {{- item.text }}\n            {%- else %}\n                {{- raise_exception('Unexpected item type in content.') }}\n            {%- endif %}\n        {%- endfor %}\n    {%- elif content is none or content is undefined %}\n        {{- '' }}\n    {%- else %}\n        {{- raise_exception('Unexpected content type.') }}\n    {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n    {{- raise_exception('No messages provided.') }}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n    {{- '<|im_start|>system\\n' }}\n    {{- \"# Tools\\n\\nYou have access to the following functions:\\n\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\" }}\n    {{- '\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n<tool_call>\\n<function=example_function_name>\\n<parameter=example_parameter_1>\\nvalue_1\\n</parameter>\\n<parameter=example_parameter_2>\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n</parameter>\\n</function>\\n</tool_call>\\n\\n<IMPORTANT>\\nReminder:\\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n</IMPORTANT>' }}\n    {%- if messages[0].role == 'system' %}\n        {%- set content = render_content(messages[0].content, false, true)|trim %}\n        {%- if content %}\n            {{- '\\n\\n' + content }}\n        {%- endif %}\n    {%- endif %}\n    {{- '<|im_end|>\\n' }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {%- set content = render_content(messages[0].content, false, true)|trim %}\n        {{- '<|im_start|>system\\n' + content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" %}\n        {%- set content = render_content(message.content, false)|trim %}\n        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}\n            {%- set ns.multi_step_tool = false %}\n            {%- set ns.last_query_index = index %}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n    {{- raise_exception('No user query found in messages.') }}\n{%- endif %}\n{%- for message in messages %}\n    {%- set content = render_content(message.content, true)|trim %}\n    {%- if message.role == \"system\" %}\n        {%- if not loop.first %}\n            {{- raise_exception('System message must be at the beginning.') }}\n        {%- endif %}\n    {%- elif message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is string %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in content %}\n                {%- set reasoning_content = content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n                {%- set content = content.split('</think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- set reasoning_content = reasoning_content|trim %}\n        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}\n            {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content + '\\n</think>\\n\\n' + content }}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if tool_call.function is defined %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {%- if loop.first %}\n                    {%- if content|trim %}\n                        {{- '\\n\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                    {%- else %}\n                        {{- '<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                    {%- endif %}\n                {%- else %}\n                    {{- '\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                {%- endif %}\n                {%- if tool_call.arguments is defined %}\n                    {%- for args_name, args_value in tool_call.arguments|items %}\n                        {{- '<parameter=' + args_name + '>\\n' }}\n                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}\n                        {{- args_value }}\n                        {{- '\\n</parameter>\\n' }}\n                    {%- endfor %}\n                {%- endif %}\n                {{- '</function>\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.previtem and loop.previtem.role != \"tool\" %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- content }}\n        {{- '\\n</tool_response>' }}\n        {%- if not loop.last and loop.nextitem.role != \"tool\" %}\n            {{- '<|im_end|>\\n' }}\n        {%- elif loop.last %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- else %}\n        {{- raise_exception('Unexpected message role.') }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- else %}\n        {{- '<think>\\n' }}\n    {%- endif %}\n{%- endif %}",
-    "clean_up_tokenization_spaces": false,
-    "eos_token": "<|im_end|>",
-    "errors": "replace",
-    "model_max_length": 262144,
-    "pad_token": "<|endoftext|>",
-    "split_special_tokens": false,
-    "tokenizer_class": "Qwen2Tokenizer",
-    "unk_token": null,
-    "add_bos_token": false,
-    "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
-    "extra_special_tokens": {
-        "audio_bos_token": "<|audio_start|>",
-        "audio_eos_token": "<|audio_end|>",
-        "audio_token": "<|audio_pad|>",
-        "image_token": "<|image_pad|>",
-        "video_token": "<|video_pad|>",
-        "vision_bos_token": "<|vision_start|>",
-        "vision_eos_token": "<|vision_end|>"
-    }
-}
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}