Instructions to use sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit")
config = load_config("sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

sluttybutfast commited on Aug 6

Commit

ce94aa4

verified ·

1 Parent(s): c232200

Upload folder using huggingface_hub

Browse files

Files changed (21) hide show

.gitattributes +1 -0
README.md +164 -0
chat_template.jinja +159 -0
chat_template.jinja.org +154 -0
chat_template.jinja.unsloth +158 -0
config.json +0 -0
generation_config.json +12 -0
model-00001-of-00006.safetensors +3 -0
model-00002-of-00006.safetensors +3 -0
model-00003-of-00006.safetensors +3 -0
model-00004-of-00006.safetensors +3 -0
model-00005-of-00006.safetensors +3 -0
model-00006-of-00006.safetensors +3 -0
model.safetensors.index.json +0 -0
optiq_metadata.json +2056 -0
processor_config.json +60 -0
scripts/merge.py +155 -0
scripts/split.py +162 -0
scripts/verify.py +91 -0
tokenizer.json +3 -0
tokenizer_config.json +34 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,164 @@

+---
+license: apache-2.0
+library_name: mlx
+tags:
+  - mlx
+  - qwen3_5_moe
+  - vision-language
+  - multimodal
+  - image-text-to-text
+  - code
+base_model:
+  - mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit
+  - unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit
+pipeline_tag: image-text-to-text
+---
+# sluttybutfast/KAT-Coder-V2.5-Dev-Vision-OptiQ-4bit
+A vision-language model created by grafting the vision tower of
+/unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit onto the language model of  mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit.
+The finetuned model was language-only (its vision tower had been stripped
+during training), but was the best model for agentic coding i ever tested for 32gb systems (very
+good multi turn tool calling and small thinking blocks for a qwen variant).
+This merge fully restores multimodal capability by re-attaching the original vision encoder and
+projector from the base VLM, while keeping the finetuned language weights intact.
+- Architecture: Qwen3_5MoeForConditionalGeneration (qwen3_5_moe)
+- Language model: from  mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit (finetuned, quantized)
+- Vision tower: from unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit (unmodified, bf16)
+- Format: MLX
+- Runs with: mlx-vlm (https://github.com/Blaizzy/mlx-vlm), oMLX prefered
+## Verify image capability
+Install:
+    pip install mlx-vlm
+Command line:
+    python -m mlx_vlm.generate \
+      --model YOUR_MODEL_NAME \
+      --image path/to/image.jpg \
+      --prompt "Describe this image. Extract all text" \
+      --max-tokens 5000
+Python:
+    from mlx_vlm import load, generate
+    from mlx_vlm.prompt_utils import apply_chat_template
+    model, processor = load("YOUR_MODEL_NAME")
+    config = model.config
+    prompt = apply_chat_template(
+        processor, config,
+        "Describe this image. Extract all text",
+        num_images=1,
+    )
+    output = generate(model, processor, prompt, ["path/to/image.jpg"],
+                      max_tokens=5000, verbose=True)
+    print(output)
+## How the merge was done
+The two source models share the same qwen3_5_moe architecture and an
+identical namespace convention, which made a clean weight-level merge
+possible without any retraining, fine-tuning, or projector re-alignment.
+### 1. Namespace split
+All tensors in both models fall into two disjoint prefixes:
+- language_model.*  -> Language model + LM head  -> taken from the FINETUNE
+- vision_tower.*    -> Vision encoder + merger    -> taken from the BASE VLM
+The merge is a straightforward union: every language_model.* tensor from the
+finetune, and every vision_tower.* tensor from the base VLM.
+### 2. Dimension compatibility
+The vision projector (vision_tower.merger) outputs
+vision_config.out_hidden_size = 2048, which matches the language model's
+text_config.hidden_size = 2048. Vision features therefore project directly
+into the LM embedding space with no adapter needed. Image features are
+injected at the image_token_id position (single-point injection;
+deepstack_visual_indexes is empty).
+### 3. Precision and quantization
+- Language model: retains the finetune's mixed quantization scheme --
+  4-bit switch_mlp experts, 8-bit attention / shared experts / embeddings /
+  LM head (group size 64, affine). The finetune's quantization map was
+  carried over verbatim.
+- Vision tower: kept at full precision (bfloat16), exactly as shipped in the
+  base VLM. MLX supports this mixed quantized-LM / bf16-vision setup.
+### 4. Configuration
+The merged config.json uses the finetune's config as the base (it holds the
+correct quantization map and text_config), with the following vision-related
+fields grafted in from the base VLM:
+- vision_config
+- image_token_id, video_token_id
+- vision_start_token_id, vision_end_token_id
+The multi-token-prediction head was not carried over
+(mtp_num_hidden_layers remains 0, matching the finetune). The image preprocessor
+(preprocessor_config.json) were taken from the base VLM, since the finetune's tokenizer
+configuration was text-only. The tokenizer vocabulary is identical between both sources
+(vocab_size = 248320, same special-token IDs), so the base VLM's template is
+fully compatible.
+### 5. Verification
+The merged weights were verified byte-for-byte via SHA-256 hashes of each
+tensor, comparing the merged output against both sources:
+- All 1757 language_model.* tensors are byte-identical to the finetune.
+- All 333 vision_tower.* tensors are byte-identical to the base VLM.
+This model is a derivative combining:
+- Language model: mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit
+- Vision tower: unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit
+License: apache-2.0. You must comply with the licenses of BOTH source
+models. Please cite and credit the original authors of both the base VLM and
+the finetune.
+## Reproduction
+The merge was performed with a script (scripts/merge.py) that:
+1. Loads all safetensors from both source directories natively in MLX
+   (preserving bfloat16 and packed-quant dtypes).
+2. Selects language_model.* tensors from the finetune and vision_tower.*
+   tensors from the base VLM.
+3. Builds the merged config (finetune config + base vision fields).
+4. Copies the image preprocessor config from the base VLM, rest from finetune.
+5. Saves the merged safetensors with a model.safetensors.index.json.
+The merge was split into Hugging Face splittensors using the scripts/split.py.
+The model was verified for correct tensors (checking text tensors were untouched) using the scripts/verify.py.
+You should be able to do your own merges with the script as long as the two models are compatible (check for
+same dimensions, same tokenizer size). Verify and check vision capabilities (f.e. ocr a text).
+The model uses unsloth chat_template with default reasoning_preserve=true.
+If you prefer the original chat_template replace chat_template.json with chat_templat.json.org or any you like.
+## Recommended settings (coding/thinking):
+Though finetuned, this is still Qwen3.6, so stick close to the official Qwen recommendations
+(temp 0.7, top_p=0.95, min_p=0.0, top_k=20, repetition_penalty=1.0, presence_penalty=0.0).
+Avoid temp < 0.4, as this can cause the model to loop!!!
+My personal recommendation (forced! in oMLX when required, as f.e. copilot likes to use temp 0.0):
+ temp 0.6, top_p=0.85, min_p=0.0, top_k=20, repetition_penalty=1.0, presence_penalty=0.0,
+ thinking_budget=none, enable_thinking=true, preserve_thinking=true, output_token_limit=16384

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,159 @@

+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- set preserve_thinking = preserve_thinking if preserve_thinking is defined else true %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set num_sys = 0 %}
+{%- set merged_system = '' %}
+{%- if messages[0].role == 'system' or messages[0].role == 'developer' %}
+    {%- set first = render_content(messages[0].content, false, true)|trim %}
+    {%- if messages|length > 1 and (messages[1].role == 'system' or messages[1].role == 'developer') %}
+        {%- set second = render_content(messages[1].content, false, true)|trim %}
+        {%- set merged_system = first + '\n' + second %}
+        {%- set num_sys = 2 %}
+    {%- else %}
+        {%- set merged_system = first %}
+        {%- set num_sys = 1 %}
+    {%- endif %}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if merged_system %}
+        {{- '\n\n' + merged_system }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if merged_system %}
+        {{- '<|im_start|>system\n' + merged_system + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if loop.index0 >= num_sys and message.role != "system" and message.role != "developer" %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is mapping %}
+                    {%- for args_name in tool_call.arguments %}
+                        {%- set args_value = tool_call.arguments[args_name] %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}
+{#- Unsloth fixes - developer role, tool calling #}

chat_template.jinja.org ADDED Viewed

	@@ -0,0 +1,154 @@

+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- '<|im_start|>system\n' + content + '<|im_end|>' + '\n' }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

chat_template.jinja.unsloth ADDED Viewed

	@@ -0,0 +1,158 @@

+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set num_sys = 0 %}
+{%- set merged_system = '' %}
+{%- if messages[0].role == 'system' or messages[0].role == 'developer' %}
+    {%- set first = render_content(messages[0].content, false, true)|trim %}
+    {%- if messages|length > 1 and (messages[1].role == 'system' or messages[1].role == 'developer') %}
+        {%- set second = render_content(messages[1].content, false, true)|trim %}
+        {%- set merged_system = first + '\n' + second %}
+        {%- set num_sys = 2 %}
+    {%- else %}
+        {%- set merged_system = first %}
+        {%- set num_sys = 1 %}
+    {%- endif %}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if merged_system %}
+        {{- '\n\n' + merged_system }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if merged_system %}
+        {{- '<|im_start|>system\n' + merged_system + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if loop.index0 >= num_sys and message.role != "system" and message.role != "developer" %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is mapping %}
+                    {%- for args_name in tool_call.arguments %}
+                        {%- set args_value = tool_call.arguments[args_name] %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}
+{#- Unsloth fixes - developer role, tool calling #}

config.json ADDED Viewed

The diff for this file is too large to render. See raw diff

generation_config.json ADDED Viewed

	@@ -0,0 +1,12 @@

+{
+    "bos_token_id": 248044,
+    "do_sample": true,
+    "eos_token_id": [
+        248046,
+        248044
+    ],
+    "pad_token_id": 248044,
+    "temperature": 1.0,
+    "top_k": 20,
+    "top_p": 0.95
+}

model-00001-of-00006.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c723e5a413fc11f858c79f809f660ba94d39d1e51006ca709fa8856cf1fea0b5
+size 4290266206

model-00002-of-00006.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:21d6ef300e21c4c011b9ee4d8160d1d03d2369b929b5a420450541341f5a8bcd
+size 4279959581

model-00003-of-00006.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a4f65095227500325ae56e53f3761f29b596e55fc4f8e4022203e687611c958f
+size 4293322632

model-00004-of-00006.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:319f5eae052376b9b096417e2b02370bab56552c7fe7843815873c256305a03f
+size 4165945826

model-00005-of-00006.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e3cd1ae0b12b76b5df6c6e3a9b96204245d19982b00d3589bd5fa3480676e283
+size 4167598879

model-00006-of-00006.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bd623e36304007051d0e541fb1c4a2f89f38c62fe55ff34b097b2d05a6293a25
+size 1626141612

model.safetensors.index.json ADDED Viewed

The diff for this file is too large to render. See raw diff

optiq_metadata.json ADDED Viewed

	@@ -0,0 +1,2056 @@

+{
+  "method": "static_mixed_precision",
+  "base_model": "/Users/codelion/.cache/huggingface/hub/models--Kwaipilot--KAT-Coder-V2.5-Dev/snapshots/7a4a45dc6426b4951be99a9f230557c0b0b77761/",
+  "reference": "structural_rules",
+  "target_bpw": 4.5,
+  "achieved_bpw": 4.510037156886497,
+  "n_high_bits": 400,
+  "n_low_bits": 111,
+  "threshold": 0.0,
+  "per_layer": {
+    "language_model.lm_head": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.switch_mlp.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.switch_mlp.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.switch_mlp.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.39.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.38.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.37.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.36.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.35.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.34.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.33.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.32.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.31.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.30.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.29.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.28.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.27.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.26.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.25.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.24.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.23.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.22.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.21.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.20.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.19.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.18.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.17.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.16.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.15.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.14.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.13.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.12.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.11.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.10.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.9.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.8.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.7.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.6.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.5.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.4.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.self_attn.o_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.self_attn.v_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.self_attn.k_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.3.self_attn.q_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.switch_mlp.down_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.switch_mlp.up_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.switch_mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.2.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.switch_mlp.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.switch_mlp.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.switch_mlp.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.1.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.shared_expert_gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.shared_expert.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.shared_expert.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.shared_expert.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.switch_mlp.down_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.switch_mlp.up_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.switch_mlp.gate_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.mlp.gate": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.linear_attn.out_proj": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_a": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_b": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_z": {
+      "bits": 8,
+      "group_size": 64
+    },
+    "language_model.model.layers.0.linear_attn.in_proj_qkv": {
+      "bits": 8,
+      "group_size": 64
+    }
+  }
+}

processor_config.json ADDED Viewed

	@@ -0,0 +1,60 @@

+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

scripts/merge.py ADDED Viewed

	@@ -0,0 +1,155 @@

+#!/usr/bin/env python3
+"""
+Merge Qwen3.5-MoE vision_tower (dir B) into the finetuned
+language-only quantized model (dir A).
+Verified clean namespace split:
+  language_model.*  -> LM (from A, keeps A's quantization map)
+  vision_tower.*    -> vision (from B, unquantized bf16)
+Config: LM + quantization from A (finetune), vision fields from B.
+Chat template: from B by default (vision-aware).
+"""
+import argparse
+import json
+import shutil
+from pathlib import Path
+import mlx.core as mx
+def load_all(model_dir: Path) -> dict:
+    """Load every tensor from all safetensors shards, natively in MLX.
+    mx.load reads safetensors directly and preserves bfloat16 (unlike a
+    NumPy round-trip, which has no native bf16 dtype).
+    """
+    weights = {}
+    files = sorted(model_dir.glob("*.safetensors"))
+    if not files:
+        raise FileNotFoundError(f"No safetensors in {model_dir}")
+    for f in files:
+        weights.update(mx.load(str(f)))
+    return weights
+def is_lm_key(k: str) -> bool:
+    return k.startswith("language_model.")
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--finetune", "-a", required=True,
+                    help="Dir A: finetuned language-only quantized model")
+    ap.add_argument("--base", "-b", required=True,
+                    help="Dir B: base VLM with vision_tower")
+    ap.add_argument("--out", "-o", required=True,
+                    help="Output directory for merged model")
+    ap.add_argument("--template-from", choices=["a", "b"], default="a",
+                    help="Which model's chat template to use "
+                         "(default b: vision-aware).")
+    args = ap.parse_args()
+    dir_a, dir_b, out = Path(args.finetune), Path(args.base), Path(args.out)
+    out.mkdir(parents=True, exist_ok=True)
+    print("Loading A (finetuned LM)…")
+    a = load_all(dir_a)
+    print("Loading B (base VLM)…")
+    b = load_all(dir_b)
+    lm_weights = {k: v for k, v in a.items() if is_lm_key(k)}
+    vision_weights = {k: v for k, v in b.items() if not is_lm_key(k)}
+    # Verify the vision split is what we expect (all vision_tower.*).
+    non_vision_b = [k for k in vision_weights
+                    if not k.startswith("vision_tower.")]
+    if non_vision_b:
+        print("WARNING: B has non-LM keys that are NOT vision_tower.*:")
+        for k in non_vision_b[:20]:
+            print("   ", k)
+        print("   -> Decide whether these belong in the merge. "
+              "Currently they WILL be included.")
+    stray_a = [k for k in a if not is_lm_key(k)]
+    if stray_a:
+        print(f"NOTE: A has {len(stray_a)} non-LM keys (ignored). "
+              f"e.g. {stray_a[:5]}")
+    # Drop MTP tensors if present in B (A config sets mtp layers = 0).
+    mtp = [k for k in vision_weights if "mtp" in k.lower()]
+    for k in mtp:
+        vision_weights.pop(k)
+    if mtp:
+        print(f"Dropped {len(mtp)} MTP tensors from B.")
+    print(f"LM tensors (A):     {len(lm_weights)}")
+    print(f"Vision tensors (B): {len(vision_weights)}")
+    if not vision_weights:
+        print("ERROR: no vision tensors. Aborting.")
+        return
+    merged = {**lm_weights, **vision_weights}
+    print(f"Total merged:       {len(merged)}")
+    mx.save_safetensors(str(out / "model.safetensors"),
+                        merged, metadata={"format": "mlx"})
+    print("Saved model.safetensors")
+    build_config(dir_a, dir_b, out)
+    copy_aux(dir_a, dir_b, out, template_from=args.template_from)
+    print("\nDone. Test:")
+    print(f"  python -m mlx_vlm.generate --model {out} "
+          f"--prompt 'Hi' --max-tokens 30")
+    print(f"  python -m mlx_vlm.generate --model {out} "
+          f"--image test.jpg --prompt 'Describe this.' --max-tokens 100")
+def build_config(dir_a: Path, dir_b: Path, out: Path):
+    ca = json.loads((dir_a / "config.json").read_text())
+    cb = json.loads((dir_b / "config.json").read_text())
+    # A is authoritative for LM: it has the CORRECT quantization map
+    # (A and B were quantized differently) and text_config (mtp=0).
+    merged = dict(ca)
+    # Graft vision-specific fields from B.
+    for field in ("vision_config", "image_token_id", "video_token_id",
+                  "vision_start_token_id", "vision_end_token_id"):
+        if field in cb:
+            merged[field] = cb[field]
+            print(f"config: added '{field}' from B")
+    (out / "config.json").write_text(json.dumps(merged, indent=2))
+    print("config: wrote merged config.json "
+          "(LM/quant from A, vision from B)")
+def copy_aux(dir_a: Path, dir_b: Path, out: Path, template_from="a"):
+    tmpl_dir = dir_b if template_from == "b" else dir_a
+    print(f"aux: chat template / tokenizer_config from '{template_from}'")
+    # Vocab files: identical between A and B; take from A.
+    for n in ["tokenizer.json", "vocab.json", "merges.txt",
+              "special_tokens_map.json", "added_tokens.json",
+              "generation_config.json", "optiq_metadata.json"]:
+        if (dir_a / n).exists():
+            shutil.copy(dir_a / n, out / n)
+    # Template-bearing files: from chosen source (default B, vision-aware).
+    for n in ["tokenizer_config.json", "chat_template.jinja"]:
+        if (tmpl_dir / n).exists():
+            shutil.copy(tmpl_dir / n, out / n)
+    # Vision preprocessing: always from B.
+    for n in ["preprocessor_config.json", "processor_config.json",
+              "image_processor_config.json", "video_processor_config.json"]:
+        if (dir_b / n).exists():
+            shutil.copy(dir_b / n, out / n)
+            print(f"aux: copied {n} from B")
+if __name__ == "__main__":
+    main()

scripts/split.py ADDED Viewed

	@@ -0,0 +1,162 @@

+#!/usr/bin/env python3
+"""
+Split a single MLX model.safetensors into sharded safetensors for
+HuggingFace upload, generating model.safetensors.index.json.
+Preserves dtypes (bf16, uint32 quant packs, etc.) since it loads and
+saves natively with MLX.
+"""
+import argparse
+import json
+import shutil
+from pathlib import Path
+import mlx.core as mx
+def human_to_bytes(s: str) -> int:
+    s = s.strip().upper()
+    units = {"B": 1, "KB": 1024, "MB": 1024**2,
+             "GB": 1024**3, "TB": 1024**4}
+    for u in ("TB", "GB", "MB", "KB", "B"):
+        if s.endswith(u):
+            return int(float(s[:-len(u)]) * units[u])
+    return int(s)  # raw byte count
+def dtype_size(arr: mx.array) -> int:
+    """Bytes per element for the array's dtype."""
+    # mx.array.nbytes gives total bytes directly.
+    return arr.nbytes
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--input", "-i", required=True,
+                    help="Merged model dir OR path to a single "
+                         "model.safetensors")
+    ap.add_argument("--out", "-o", required=True,
+                    help="Output directory for sharded model")
+    ap.add_argument("--max-shard-size", default="5GB",
+                    help="Max size per shard (e.g. 5GB, 4GB). Default 5GB.")
+    ap.add_argument("--copy-aux", action="store_true",
+                    help="Copy config/tokenizer/etc. from input dir to out.")
+    args = ap.parse_args()
+    in_path = Path(args.input)
+    out = Path(args.out)
+    out.mkdir(parents=True, exist_ok=True)
+    # Resolve the single-file source and its parent dir (for aux files).
+    if in_path.is_dir():
+        src_file = in_path / "model.safetensors"
+        src_dir = in_path
+        if not src_file.exists():
+            # Maybe it's already sharded — bail with guidance.
+            shards = sorted(in_path.glob("model-*.safetensors"))
+            if shards:
+                print("Input dir already contains sharded safetensors:")
+                for s in shards:
+                    print("  ", s.name)
+                print("This script expects a single model.safetensors. "
+                      "Point --input at that file, or consolidate first.")
+                return
+            raise FileNotFoundError(f"No model.safetensors in {in_path}")
+    else:
+        src_file = in_path
+        src_dir = in_path.parent
+    max_bytes = human_to_bytes(args.max_shard_size)
+    print(f"Loading {src_file} …")
+    weights = mx.load(str(src_file))
+    print(f"Loaded {len(weights)} tensors.")
+    # Compute total size and per-tensor sizes.
+    sizes = {k: dtype_size(v) for k, v in weights.items()}
+    total = sum(sizes.values())
+    print(f"Total weight size: {total / 1024**3:.2f} GB")
+    print(f"Target max shard size: {max_bytes / 1024**3:.2f} GB")
+    # Greedy bin-packing into shards, preserving insertion order.
+    # (Keeps related tensors together reasonably well.)
+    shards = []           # list of dict[name -> array]
+    current = {}
+    current_size = 0
+    for name, arr in weights.items():
+        sz = sizes[name]
+        if sz > max_bytes:
+            # A single tensor exceeds the shard limit; it gets its own shard.
+            if current:
+                shards.append(current)
+                current, current_size = {}, 0
+            shards.append({name: arr})
+            print(f"  NOTE: '{name}' ({sz/1024**3:.2f} GB) exceeds shard "
+                  f"limit; placed in its own shard.")
+            continue
+        if current_size + sz > max_bytes and current:
+            shards.append(current)
+            current, current_size = {}, 0
+        current[name] = arr
+        current_size += sz
+    if current:
+        shards.append(current)
+    n = len(shards)
+    print(f"Splitting into {n} shard(s).")
+    if n == 1:
+        # Single shard: HF convention is just model.safetensors (no index).
+        out_file = out / "model.safetensors"
+        mx.save_safetensors(str(out_file), shards[0],
+                            metadata={"format": "mlx"})
+        print(f"Wrote {out_file.name} (single shard, no index needed).")
+    else:
+        # Multi-shard: model-00001-of-000NN.safetensors + index.
+        weight_map = {}
+        for i, shard in enumerate(shards, start=1):
+            fname = f"model-{i:05d}-of-{n:05d}.safetensors"
+            mx.save_safetensors(str(out / fname), shard,
+                                metadata={"format": "mlx"})
+            for k in shard:
+                weight_map[k] = fname
+            shard_bytes = sum(sizes[k] for k in shard)
+            print(f"  Wrote {fname} "
+                  f"({len(shard)} tensors, {shard_bytes/1024**3:.2f} GB)")
+        index = {
+            "metadata": {"total_size": total},
+            "weight_map": weight_map,
+        }
+        idx_file = out / "model.safetensors.index.json"
+        idx_file.write_text(json.dumps(index, indent=2))
+        print(f"Wrote {idx_file.name}")
+    if args.copy_aux:
+        copy_aux(src_dir, out)
+    print("\nDone. Upload with:")
+    print(f"  huggingface-cli upload <repo_id> {out} .")
+def copy_aux(src_dir: Path, out: Path):
+    aux = [
+        "config.json",
+        "tokenizer.json", "tokenizer_config.json", "vocab.json",
+        "merges.txt", "special_tokens_map.json", "added_tokens.json",
+        "chat_template.jinja", "generation_config.json",
+        "preprocessor_config.json", "processor_config.json",
+        "image_processor_config.json", "video_processor_config.json",
+    ]
+    copied = 0
+    for n in aux:
+        src = src_dir / n
+        if src.exists():
+            shutil.copy(src, out / n)
+            copied += 1
+    print(f"aux: copied {copied} auxiliary file(s) from {src_dir}")
+if __name__ == "__main__":
+    main()

scripts/verify.py ADDED Viewed

	@@ -0,0 +1,91 @@

+#!/usr/bin/env python3
+"""
+Memory-efficient verification. Streams tensors one at a time via
+safetensors mmap instead of loading whole models into RAM.
+Verifies: merged LM == A, merged vision == B.
+"""
+import hashlib
+import argparse
+import json
+from pathlib import Path
+import numpy as np
+from safetensors import safe_open
+def build_key_index(model_dir: Path) -> dict:
+    """Map each tensor name -> the shard file that contains it."""
+    idx = {}
+    for f in sorted(model_dir.glob("*.safetensors")):
+        with safe_open(f, framework="numpy") as sf:  # metadata only
+            for k in sf.keys():
+                idx[k] = f
+    return idx
+def get_tensor_np(index: dict, key: str):
+    """Fetch a single tensor as float32-normalized numpy for hashing."""
+    f = index[key]
+    with safe_open(f, framework="pt") as sf:   # pt handles bf16
+        t = sf.get_tensor(key)
+    # torch tensor -> float32 numpy (lossless for bf16; passthrough others)
+    import torch
+    if t.dtype == torch.bfloat16:
+        t = t.to(torch.float32)
+    return t.numpy()
+def h(np_arr) -> str:
+    return hashlib.sha256(np_arr.tobytes()).hexdigest()[:16]
+def check(name, merged_idx, ref_idx, prefix):
+    keys = sorted(k for k in merged_idx if k.startswith(prefix))
+    mismatches = []
+    for i, k in enumerate(keys):
+        if k not in ref_idx:
+            mismatches.append((k, "missing in reference"))
+            continue
+        hm = h(get_tensor_np(merged_idx, k))
+        hr = h(get_tensor_np(ref_idx, k))
+        if hm != hr:
+            mismatches.append((k, "hash differs"))
+        if (i + 1) % 100 == 0:
+            print(f"  {name}: checked {i+1}/{len(keys)}…")
+    print(f"\n=== {name} ===")
+    print(f"Checked {len(keys)}  mismatches: {len(mismatches)}")
+    for k, why in mismatches[:30]:
+        print("  !!", k, "-", why)
+    return not mismatches
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("-a", "--finetune", required=True)
+    ap.add_argument("-b", "--base", required=True)
+    ap.add_argument("-m", "--merged", required=True)
+    ap.add_argument("--only", choices=["lm", "vision", "both"],
+                    default="both")
+    args = ap.parse_args()
+    print("Indexing (metadata only, no tensor loads)…")
+    a_idx = build_key_index(Path(args.finetune))
+    b_idx = build_key_index(Path(args.base))
+    m_idx = build_key_index(Path(args.merged))
+    lm_ok = vis_ok = True
+    if args.only in ("lm", "both"):
+        lm_ok = check("LM (merged vs A)", m_idx, a_idx, "language_model.")
+    if args.only in ("vision", "both"):
+        vis_ok = check("VISION (merged vs B)", m_idx, b_idx, "vision_tower.")
+    print("\n=== SUMMARY ===")
+    if args.only in ("lm", "both"):
+        print(f"LM  == A: {'PASS' if lm_ok else 'FAIL'}")
+    if args.only in ("vision", "both"):
+        print(f"VIS == B: {'PASS' if vis_ok else 'FAIL'}")
+    print("✅ VERIFIED" if (lm_ok and vis_ok) else "❌ MISMATCH")
+if __name__ == "__main__":
+    main()

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,34 @@

+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "tool_parser_type": "qwen3_coder",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}