ryugyosoft commited on
Commit
a092008
·
verified ·
1 Parent(s): ef859e6

Qwen3.6-35B-A3B with 50% of the experts pruned (REAP) for onw

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright 2026 Alibaba Cloud
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: Qwen/Qwen3.6-35B-A3B
4
+ pipeline_tag: image-text-to-text
5
+ library_name: openvino
6
+ tags:
7
+ - openvino
8
+ - npu
9
+ - intel
10
+ - intel-npu
11
+ - qwen3.6
12
+ - moe
13
+ - reap
14
+ - expert-pruning
15
+ - gated-deltanet
16
+ - onw
17
+ language:
18
+ - en
19
+ - ja
20
+ ---
21
+
22
+ # Qwen3.6-REAP-18B-A3B for onw — Qwen3.6-35B-A3B at half the size, entirely on the Intel NPU
23
+
24
+ [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) with **half of its experts pruned**
25
+ (256 → 128 per layer, REAP saliency) and converted for **[onw](https://huggingface.co/ryugyosoft/onw)**
26
+ (俺のNPUがこんなに動くわけない), the engine that runs LLMs entirely on the Intel NPU. ~18B total / ~3B active
27
+ parameters, text + image. The unpruned model is [ryugyosoft/Qwen3.6-35B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-35B-A3B-onw).
28
+
29
+ ```bash
30
+ hf download ryugyosoft/onw --local-dir onw && cd onw
31
+ start.bat ryugyosoft/Qwen3.6-REAP-18B-A3B-onw # Windows (or setup.bat for the tray app)
32
+ bash start.sh ryugyosoft/Qwen3.6-REAP-18B-A3B-onw # Ubuntu
33
+ ```
34
+
35
+ | | this model (REAP 50%) | unpruned Qwen3.6-35B-A3B-onw |
36
+ |---|---|---|
37
+ | download | **9.6 GB** | 17 GB |
38
+ | memory in use (working set, NPU 3720) | **~15 GB** | ~23 GB |
39
+ | decode, NPU 3720 | 6.8 tok/s | 6.6 tok/s |
40
+ | perplexity (chat data, bf16 reference) | 14.30 (+8.9%) | 13.13 |
41
+ | experts per layer / active | 128 / 8 + shared | 256 / 8 + shared |
42
+
43
+ ## How it was pruned
44
+
45
+ REAP (Router-weighted Expert Activation Pruning): each expert's saliency is the mean, over the tokens routed to
46
+ it, of gate weight x ||expert output||, measured on 16k tokens of chat-formatted instruction data in Japanese,
47
+ English and code (databricks-dolly-15k-ja, databricks-dolly-15k, CodeAlpaca-20k). In every layer the 128 least
48
+ salient experts are removed together with their router rows; the router's softmax then spreads over the
49
+ survivors. No retraining. Perplexity on held-out data of the same mix: 13.13 → 13.31 (75% kept) → 14.30 (50%).
50
+ Reproduce with `python -m onw.prune calibrate` / `eval` and `onw convert --prune saliency.json --keep 0.5`.
51
+
52
+ Expect some loss on rare domains and languages that the calibration mix does not cover.
53
+
54
+ Memory: ~15 GB in use on the test machine (includes memory-mapped weight files the OS can page out). That is
55
+ comfortable on 32 GB; on a 16 GB machine it is tight and untested.
56
+
57
+ ## Files
58
+
59
+ | file | what |
60
+ |---|---|
61
+ | `seg*.bin` + `seg*_S{1,16}.xml` | 41 segments + an LM-head segment; one weights file per segment shared by the 1-token and 16-token graphs |
62
+ | `experts.bin` (+ `.json`) | 40 layers x 128 experts, channel-wise INT4, bound into the segments by the host per token |
63
+ | `shared.bin` | INT8 LM head, INT4 token embedding (host lookup) |
64
+ | `vision.xml` | INT8 vision tower (512x512 -> 256 tokens) |
65
+
66
+ ## License
67
+
68
+ Apache 2.0, same as the base model. The weights are pruned, re-quantized and restructured from it.
chat_template.jinja ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set image_count = namespace(value=0) %}
2
+ {%- set video_count = namespace(value=0) %}
3
+ {%- macro render_content(content, do_vision_count, is_system_content=false) %}
4
+ {%- if content is string %}
5
+ {{- content }}
6
+ {%- elif content is iterable and content is not mapping %}
7
+ {%- for item in content %}
8
+ {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
9
+ {%- if is_system_content %}
10
+ {{- raise_exception('System message cannot contain images.') }}
11
+ {%- endif %}
12
+ {%- if do_vision_count %}
13
+ {%- set image_count.value = image_count.value + 1 %}
14
+ {%- endif %}
15
+ {%- if add_vision_id %}
16
+ {{- 'Picture ' ~ image_count.value ~ ': ' }}
17
+ {%- endif %}
18
+ {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
19
+ {%- elif 'video' in item or item.type == 'video' %}
20
+ {%- if is_system_content %}
21
+ {{- raise_exception('System message cannot contain videos.') }}
22
+ {%- endif %}
23
+ {%- if do_vision_count %}
24
+ {%- set video_count.value = video_count.value + 1 %}
25
+ {%- endif %}
26
+ {%- if add_vision_id %}
27
+ {{- 'Video ' ~ video_count.value ~ ': ' }}
28
+ {%- endif %}
29
+ {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
30
+ {%- elif 'text' in item %}
31
+ {{- item.text }}
32
+ {%- else %}
33
+ {{- raise_exception('Unexpected item type in content.') }}
34
+ {%- endif %}
35
+ {%- endfor %}
36
+ {%- elif content is none or content is undefined %}
37
+ {{- '' }}
38
+ {%- else %}
39
+ {{- raise_exception('Unexpected content type.') }}
40
+ {%- endif %}
41
+ {%- endmacro %}
42
+ {%- if not messages %}
43
+ {{- raise_exception('No messages provided.') }}
44
+ {%- endif %}
45
+ {%- if tools and tools is iterable and tools is not mapping %}
46
+ {{- '<|im_start|>system\n' }}
47
+ {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
48
+ {%- for tool in tools %}
49
+ {{- "\n" }}
50
+ {{- tool | tojson }}
51
+ {%- endfor %}
52
+ {{- "\n</tools>" }}
53
+ {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
54
+ {%- if messages[0].role == 'system' %}
55
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
56
+ {%- if content %}
57
+ {{- '\n\n' + content }}
58
+ {%- endif %}
59
+ {%- endif %}
60
+ {{- '<|im_end|>\n' }}
61
+ {%- else %}
62
+ {%- if messages[0].role == 'system' %}
63
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
64
+ {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
65
+ {%- endif %}
66
+ {%- endif %}
67
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
68
+ {%- for message in messages[::-1] %}
69
+ {%- set index = (messages|length - 1) - loop.index0 %}
70
+ {%- if ns.multi_step_tool and message.role == "user" %}
71
+ {%- set content = render_content(message.content, false)|trim %}
72
+ {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
73
+ {%- set ns.multi_step_tool = false %}
74
+ {%- set ns.last_query_index = index %}
75
+ {%- endif %}
76
+ {%- endif %}
77
+ {%- endfor %}
78
+ {%- if ns.multi_step_tool %}
79
+ {{- raise_exception('No user query found in messages.') }}
80
+ {%- endif %}
81
+ {%- for message in messages %}
82
+ {%- set content = render_content(message.content, true)|trim %}
83
+ {%- if message.role == "system" %}
84
+ {%- if not loop.first %}
85
+ {{- raise_exception('System message must be at the beginning.') }}
86
+ {%- endif %}
87
+ {%- elif message.role == "user" %}
88
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
89
+ {%- elif message.role == "assistant" %}
90
+ {%- set reasoning_content = '' %}
91
+ {%- if message.reasoning_content is string %}
92
+ {%- set reasoning_content = message.reasoning_content %}
93
+ {%- else %}
94
+ {%- if '</think>' in content %}
95
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
96
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
97
+ {%- endif %}
98
+ {%- endif %}
99
+ {%- set reasoning_content = reasoning_content|trim %}
100
+ {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
101
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
102
+ {%- else %}
103
+ {{- '<|im_start|>' + message.role + '\n' + content }}
104
+ {%- endif %}
105
+ {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
106
+ {%- for tool_call in message.tool_calls %}
107
+ {%- if tool_call.function is defined %}
108
+ {%- set tool_call = tool_call.function %}
109
+ {%- endif %}
110
+ {%- if loop.first %}
111
+ {%- if content|trim %}
112
+ {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
113
+ {%- else %}
114
+ {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
115
+ {%- endif %}
116
+ {%- else %}
117
+ {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
118
+ {%- endif %}
119
+ {%- if tool_call.arguments is defined %}
120
+ {%- for args_name, args_value in tool_call.arguments|items %}
121
+ {{- '<parameter=' + args_name + '>\n' }}
122
+ {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
123
+ {{- args_value }}
124
+ {{- '\n</parameter>\n' }}
125
+ {%- endfor %}
126
+ {%- endif %}
127
+ {{- '</function>\n</tool_call>' }}
128
+ {%- endfor %}
129
+ {%- endif %}
130
+ {{- '<|im_end|>\n' }}
131
+ {%- elif message.role == "tool" %}
132
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
133
+ {{- '<|im_start|>user' }}
134
+ {%- endif %}
135
+ {{- '\n<tool_response>\n' }}
136
+ {{- content }}
137
+ {{- '\n</tool_response>' }}
138
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
139
+ {{- '<|im_end|>\n' }}
140
+ {%- elif loop.last %}
141
+ {{- '<|im_end|>\n' }}
142
+ {%- endif %}
143
+ {%- else %}
144
+ {{- raise_exception('Unexpected message role.') }}
145
+ {%- endif %}
146
+ {%- endfor %}
147
+ {%- if add_generation_prompt %}
148
+ {{- '<|im_start|>assistant\n' }}
149
+ {%- if enable_thinking is defined and enable_thinking is false %}
150
+ {{- '<think>\n\n</think>\n\n' }}
151
+ {%- else %}
152
+ {{- '<think>\n' }}
153
+ {%- endif %}
154
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3_5MoeForConditionalGeneration"
4
+ ],
5
+ "image_token_id": 248056,
6
+ "model_type": "qwen3_5_moe",
7
+ "text_config": {
8
+ "attention_bias": false,
9
+ "attention_dropout": 0.0,
10
+ "attn_output_gate": true,
11
+ "bos_token_id": 248044,
12
+ "dtype": "bfloat16",
13
+ "eos_token_id": 248044,
14
+ "full_attention_interval": 4,
15
+ "head_dim": 256,
16
+ "hidden_act": "silu",
17
+ "hidden_size": 2048,
18
+ "initializer_range": 0.02,
19
+ "layer_types": [
20
+ "linear_attention",
21
+ "linear_attention",
22
+ "linear_attention",
23
+ "full_attention",
24
+ "linear_attention",
25
+ "linear_attention",
26
+ "linear_attention",
27
+ "full_attention",
28
+ "linear_attention",
29
+ "linear_attention",
30
+ "linear_attention",
31
+ "full_attention",
32
+ "linear_attention",
33
+ "linear_attention",
34
+ "linear_attention",
35
+ "full_attention",
36
+ "linear_attention",
37
+ "linear_attention",
38
+ "linear_attention",
39
+ "full_attention",
40
+ "linear_attention",
41
+ "linear_attention",
42
+ "linear_attention",
43
+ "full_attention",
44
+ "linear_attention",
45
+ "linear_attention",
46
+ "linear_attention",
47
+ "full_attention",
48
+ "linear_attention",
49
+ "linear_attention",
50
+ "linear_attention",
51
+ "full_attention",
52
+ "linear_attention",
53
+ "linear_attention",
54
+ "linear_attention",
55
+ "full_attention",
56
+ "linear_attention",
57
+ "linear_attention",
58
+ "linear_attention",
59
+ "full_attention"
60
+ ],
61
+ "linear_conv_kernel_dim": 4,
62
+ "linear_key_head_dim": 128,
63
+ "linear_num_key_heads": 16,
64
+ "linear_num_value_heads": 32,
65
+ "linear_value_head_dim": 128,
66
+ "mamba_ssm_dtype": "float32",
67
+ "max_position_embeddings": 262144,
68
+ "model_type": "qwen3_5_moe_text",
69
+ "moe_intermediate_size": 512,
70
+ "mtp_num_hidden_layers": 1,
71
+ "mtp_use_dedicated_embeddings": false,
72
+ "num_attention_heads": 16,
73
+ "num_experts": 256,
74
+ "num_experts_per_tok": 8,
75
+ "num_hidden_layers": 40,
76
+ "num_key_value_heads": 2,
77
+ "output_router_logits": false,
78
+ "pad_token_id": null,
79
+ "partial_rotary_factor": 0.25,
80
+ "rms_norm_eps": 1e-06,
81
+ "rope_parameters": {
82
+ "mrope_interleaved": true,
83
+ "mrope_section": [
84
+ 11,
85
+ 11,
86
+ 10
87
+ ],
88
+ "partial_rotary_factor": 0.25,
89
+ "rope_theta": 10000000,
90
+ "rope_type": "default"
91
+ },
92
+ "router_aux_loss_coef": 0.001,
93
+ "shared_expert_intermediate_size": 512,
94
+ "tie_word_embeddings": false,
95
+ "use_cache": true,
96
+ "vocab_size": 248320
97
+ },
98
+ "tie_word_embeddings": false,
99
+ "transformers_version": "4.57.1",
100
+ "video_token_id": 248057,
101
+ "vision_config": {
102
+ "deepstack_visual_indexes": [],
103
+ "depth": 27,
104
+ "hidden_act": "gelu_pytorch_tanh",
105
+ "hidden_size": 1152,
106
+ "in_channels": 3,
107
+ "initializer_range": 0.02,
108
+ "intermediate_size": 4304,
109
+ "model_type": "qwen3_5_moe",
110
+ "num_heads": 16,
111
+ "num_position_embeddings": 2304,
112
+ "out_hidden_size": 2048,
113
+ "patch_size": 16,
114
+ "spatial_merge_size": 2,
115
+ "temporal_patch_size": 2
116
+ },
117
+ "vision_end_token_id": 248054,
118
+ "vision_start_token_id": 248053
119
+ }
engine.json ADDED
@@ -0,0 +1,817 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "arch": "qwen3_5_moe",
3
+ "T": 1024,
4
+ "sizes": [
5
+ 1,
6
+ 16
7
+ ],
8
+ "config": {
9
+ "attention_bias": false,
10
+ "attention_dropout": 0.0,
11
+ "attn_output_gate": true,
12
+ "bos_token_id": 248044,
13
+ "dtype": "bfloat16",
14
+ "eos_token_id": 248044,
15
+ "full_attention_interval": 4,
16
+ "head_dim": 256,
17
+ "hidden_act": "silu",
18
+ "hidden_size": 2048,
19
+ "initializer_range": 0.02,
20
+ "layer_types": [
21
+ "linear_attention",
22
+ "linear_attention",
23
+ "linear_attention",
24
+ "full_attention",
25
+ "linear_attention",
26
+ "linear_attention",
27
+ "linear_attention",
28
+ "full_attention",
29
+ "linear_attention",
30
+ "linear_attention",
31
+ "linear_attention",
32
+ "full_attention",
33
+ "linear_attention",
34
+ "linear_attention",
35
+ "linear_attention",
36
+ "full_attention",
37
+ "linear_attention",
38
+ "linear_attention",
39
+ "linear_attention",
40
+ "full_attention",
41
+ "linear_attention",
42
+ "linear_attention",
43
+ "linear_attention",
44
+ "full_attention",
45
+ "linear_attention",
46
+ "linear_attention",
47
+ "linear_attention",
48
+ "full_attention",
49
+ "linear_attention",
50
+ "linear_attention",
51
+ "linear_attention",
52
+ "full_attention",
53
+ "linear_attention",
54
+ "linear_attention",
55
+ "linear_attention",
56
+ "full_attention",
57
+ "linear_attention",
58
+ "linear_attention",
59
+ "linear_attention",
60
+ "full_attention"
61
+ ],
62
+ "linear_conv_kernel_dim": 4,
63
+ "linear_key_head_dim": 128,
64
+ "linear_num_key_heads": 16,
65
+ "linear_num_value_heads": 32,
66
+ "linear_value_head_dim": 128,
67
+ "mamba_ssm_dtype": "float32",
68
+ "max_position_embeddings": 262144,
69
+ "model_type": "qwen3_5_moe_text",
70
+ "moe_intermediate_size": 512,
71
+ "mtp_num_hidden_layers": 1,
72
+ "mtp_use_dedicated_embeddings": false,
73
+ "num_attention_heads": 16,
74
+ "num_experts": 128,
75
+ "num_experts_per_tok": 8,
76
+ "num_hidden_layers": 40,
77
+ "num_key_value_heads": 2,
78
+ "output_router_logits": false,
79
+ "pad_token_id": null,
80
+ "partial_rotary_factor": 0.25,
81
+ "rms_norm_eps": 1e-06,
82
+ "rope_parameters": {
83
+ "mrope_interleaved": true,
84
+ "mrope_section": [
85
+ 11,
86
+ 11,
87
+ 10
88
+ ],
89
+ "partial_rotary_factor": 0.25,
90
+ "rope_theta": 10000000,
91
+ "rope_type": "default"
92
+ },
93
+ "router_aux_loss_coef": 0.001,
94
+ "shared_expert_intermediate_size": 512,
95
+ "tie_word_embeddings": false,
96
+ "use_cache": true,
97
+ "vocab_size": 248320
98
+ },
99
+ "segments": {
100
+ "seg00_S1.xml": {
101
+ "index": 0,
102
+ "consume_moe": null,
103
+ "router_layer": 0,
104
+ "S": 1,
105
+ "slots": 8,
106
+ "bin": "seg00.bin"
107
+ },
108
+ "seg01_S1.xml": {
109
+ "index": 1,
110
+ "consume_moe": 0,
111
+ "router_layer": 1,
112
+ "S": 1,
113
+ "slots": 8,
114
+ "bin": "seg01.bin"
115
+ },
116
+ "seg02_S1.xml": {
117
+ "index": 2,
118
+ "consume_moe": 1,
119
+ "router_layer": 2,
120
+ "S": 1,
121
+ "slots": 8,
122
+ "bin": "seg02.bin"
123
+ },
124
+ "seg03_S1.xml": {
125
+ "index": 3,
126
+ "consume_moe": 2,
127
+ "router_layer": 3,
128
+ "S": 1,
129
+ "slots": 8,
130
+ "bin": "seg03.bin"
131
+ },
132
+ "seg04_S1.xml": {
133
+ "index": 4,
134
+ "consume_moe": 3,
135
+ "router_layer": 4,
136
+ "S": 1,
137
+ "slots": 8,
138
+ "bin": "seg04.bin"
139
+ },
140
+ "seg05_S1.xml": {
141
+ "index": 5,
142
+ "consume_moe": 4,
143
+ "router_layer": 5,
144
+ "S": 1,
145
+ "slots": 8,
146
+ "bin": "seg05.bin"
147
+ },
148
+ "seg06_S1.xml": {
149
+ "index": 6,
150
+ "consume_moe": 5,
151
+ "router_layer": 6,
152
+ "S": 1,
153
+ "slots": 8,
154
+ "bin": "seg06.bin"
155
+ },
156
+ "seg07_S1.xml": {
157
+ "index": 7,
158
+ "consume_moe": 6,
159
+ "router_layer": 7,
160
+ "S": 1,
161
+ "slots": 8,
162
+ "bin": "seg07.bin"
163
+ },
164
+ "seg08_S1.xml": {
165
+ "index": 8,
166
+ "consume_moe": 7,
167
+ "router_layer": 8,
168
+ "S": 1,
169
+ "slots": 8,
170
+ "bin": "seg08.bin"
171
+ },
172
+ "seg09_S1.xml": {
173
+ "index": 9,
174
+ "consume_moe": 8,
175
+ "router_layer": 9,
176
+ "S": 1,
177
+ "slots": 8,
178
+ "bin": "seg09.bin"
179
+ },
180
+ "seg10_S1.xml": {
181
+ "index": 10,
182
+ "consume_moe": 9,
183
+ "router_layer": 10,
184
+ "S": 1,
185
+ "slots": 8,
186
+ "bin": "seg10.bin"
187
+ },
188
+ "seg11_S1.xml": {
189
+ "index": 11,
190
+ "consume_moe": 10,
191
+ "router_layer": 11,
192
+ "S": 1,
193
+ "slots": 8,
194
+ "bin": "seg11.bin"
195
+ },
196
+ "seg12_S1.xml": {
197
+ "index": 12,
198
+ "consume_moe": 11,
199
+ "router_layer": 12,
200
+ "S": 1,
201
+ "slots": 8,
202
+ "bin": "seg12.bin"
203
+ },
204
+ "seg13_S1.xml": {
205
+ "index": 13,
206
+ "consume_moe": 12,
207
+ "router_layer": 13,
208
+ "S": 1,
209
+ "slots": 8,
210
+ "bin": "seg13.bin"
211
+ },
212
+ "seg14_S1.xml": {
213
+ "index": 14,
214
+ "consume_moe": 13,
215
+ "router_layer": 14,
216
+ "S": 1,
217
+ "slots": 8,
218
+ "bin": "seg14.bin"
219
+ },
220
+ "seg15_S1.xml": {
221
+ "index": 15,
222
+ "consume_moe": 14,
223
+ "router_layer": 15,
224
+ "S": 1,
225
+ "slots": 8,
226
+ "bin": "seg15.bin"
227
+ },
228
+ "seg16_S1.xml": {
229
+ "index": 16,
230
+ "consume_moe": 15,
231
+ "router_layer": 16,
232
+ "S": 1,
233
+ "slots": 8,
234
+ "bin": "seg16.bin"
235
+ },
236
+ "seg17_S1.xml": {
237
+ "index": 17,
238
+ "consume_moe": 16,
239
+ "router_layer": 17,
240
+ "S": 1,
241
+ "slots": 8,
242
+ "bin": "seg17.bin"
243
+ },
244
+ "seg18_S1.xml": {
245
+ "index": 18,
246
+ "consume_moe": 17,
247
+ "router_layer": 18,
248
+ "S": 1,
249
+ "slots": 8,
250
+ "bin": "seg18.bin"
251
+ },
252
+ "seg19_S1.xml": {
253
+ "index": 19,
254
+ "consume_moe": 18,
255
+ "router_layer": 19,
256
+ "S": 1,
257
+ "slots": 8,
258
+ "bin": "seg19.bin"
259
+ },
260
+ "seg20_S1.xml": {
261
+ "index": 20,
262
+ "consume_moe": 19,
263
+ "router_layer": 20,
264
+ "S": 1,
265
+ "slots": 8,
266
+ "bin": "seg20.bin"
267
+ },
268
+ "seg21_S1.xml": {
269
+ "index": 21,
270
+ "consume_moe": 20,
271
+ "router_layer": 21,
272
+ "S": 1,
273
+ "slots": 8,
274
+ "bin": "seg21.bin"
275
+ },
276
+ "seg22_S1.xml": {
277
+ "index": 22,
278
+ "consume_moe": 21,
279
+ "router_layer": 22,
280
+ "S": 1,
281
+ "slots": 8,
282
+ "bin": "seg22.bin"
283
+ },
284
+ "seg23_S1.xml": {
285
+ "index": 23,
286
+ "consume_moe": 22,
287
+ "router_layer": 23,
288
+ "S": 1,
289
+ "slots": 8,
290
+ "bin": "seg23.bin"
291
+ },
292
+ "seg24_S1.xml": {
293
+ "index": 24,
294
+ "consume_moe": 23,
295
+ "router_layer": 24,
296
+ "S": 1,
297
+ "slots": 8,
298
+ "bin": "seg24.bin"
299
+ },
300
+ "seg25_S1.xml": {
301
+ "index": 25,
302
+ "consume_moe": 24,
303
+ "router_layer": 25,
304
+ "S": 1,
305
+ "slots": 8,
306
+ "bin": "seg25.bin"
307
+ },
308
+ "seg26_S1.xml": {
309
+ "index": 26,
310
+ "consume_moe": 25,
311
+ "router_layer": 26,
312
+ "S": 1,
313
+ "slots": 8,
314
+ "bin": "seg26.bin"
315
+ },
316
+ "seg27_S1.xml": {
317
+ "index": 27,
318
+ "consume_moe": 26,
319
+ "router_layer": 27,
320
+ "S": 1,
321
+ "slots": 8,
322
+ "bin": "seg27.bin"
323
+ },
324
+ "seg28_S1.xml": {
325
+ "index": 28,
326
+ "consume_moe": 27,
327
+ "router_layer": 28,
328
+ "S": 1,
329
+ "slots": 8,
330
+ "bin": "seg28.bin"
331
+ },
332
+ "seg29_S1.xml": {
333
+ "index": 29,
334
+ "consume_moe": 28,
335
+ "router_layer": 29,
336
+ "S": 1,
337
+ "slots": 8,
338
+ "bin": "seg29.bin"
339
+ },
340
+ "seg30_S1.xml": {
341
+ "index": 30,
342
+ "consume_moe": 29,
343
+ "router_layer": 30,
344
+ "S": 1,
345
+ "slots": 8,
346
+ "bin": "seg30.bin"
347
+ },
348
+ "seg31_S1.xml": {
349
+ "index": 31,
350
+ "consume_moe": 30,
351
+ "router_layer": 31,
352
+ "S": 1,
353
+ "slots": 8,
354
+ "bin": "seg31.bin"
355
+ },
356
+ "seg32_S1.xml": {
357
+ "index": 32,
358
+ "consume_moe": 31,
359
+ "router_layer": 32,
360
+ "S": 1,
361
+ "slots": 8,
362
+ "bin": "seg32.bin"
363
+ },
364
+ "seg33_S1.xml": {
365
+ "index": 33,
366
+ "consume_moe": 32,
367
+ "router_layer": 33,
368
+ "S": 1,
369
+ "slots": 8,
370
+ "bin": "seg33.bin"
371
+ },
372
+ "seg34_S1.xml": {
373
+ "index": 34,
374
+ "consume_moe": 33,
375
+ "router_layer": 34,
376
+ "S": 1,
377
+ "slots": 8,
378
+ "bin": "seg34.bin"
379
+ },
380
+ "seg35_S1.xml": {
381
+ "index": 35,
382
+ "consume_moe": 34,
383
+ "router_layer": 35,
384
+ "S": 1,
385
+ "slots": 8,
386
+ "bin": "seg35.bin"
387
+ },
388
+ "seg36_S1.xml": {
389
+ "index": 36,
390
+ "consume_moe": 35,
391
+ "router_layer": 36,
392
+ "S": 1,
393
+ "slots": 8,
394
+ "bin": "seg36.bin"
395
+ },
396
+ "seg37_S1.xml": {
397
+ "index": 37,
398
+ "consume_moe": 36,
399
+ "router_layer": 37,
400
+ "S": 1,
401
+ "slots": 8,
402
+ "bin": "seg37.bin"
403
+ },
404
+ "seg38_S1.xml": {
405
+ "index": 38,
406
+ "consume_moe": 37,
407
+ "router_layer": 38,
408
+ "S": 1,
409
+ "slots": 8,
410
+ "bin": "seg38.bin"
411
+ },
412
+ "seg39_S1.xml": {
413
+ "index": 39,
414
+ "consume_moe": 38,
415
+ "router_layer": 39,
416
+ "S": 1,
417
+ "slots": 8,
418
+ "bin": "seg39.bin"
419
+ },
420
+ "seg40_S1.xml": {
421
+ "index": 40,
422
+ "consume_moe": 39,
423
+ "router_layer": null,
424
+ "S": 1,
425
+ "slots": 8,
426
+ "bin": "seg40.bin"
427
+ },
428
+ "seg41_S1.xml": {
429
+ "index": 41,
430
+ "consume_moe": null,
431
+ "router_layer": null,
432
+ "S": 1,
433
+ "slots": 0,
434
+ "head": true,
435
+ "bin": "seg41.bin"
436
+ },
437
+ "seg00_S16.xml": {
438
+ "index": 0,
439
+ "consume_moe": null,
440
+ "router_layer": 0,
441
+ "S": 16,
442
+ "slots": 128,
443
+ "bin": "seg00.bin"
444
+ },
445
+ "seg01_S16.xml": {
446
+ "index": 1,
447
+ "consume_moe": 0,
448
+ "router_layer": 1,
449
+ "S": 16,
450
+ "slots": 128,
451
+ "bin": "seg01.bin"
452
+ },
453
+ "seg02_S16.xml": {
454
+ "index": 2,
455
+ "consume_moe": 1,
456
+ "router_layer": 2,
457
+ "S": 16,
458
+ "slots": 128,
459
+ "bin": "seg02.bin"
460
+ },
461
+ "seg03_S16.xml": {
462
+ "index": 3,
463
+ "consume_moe": 2,
464
+ "router_layer": 3,
465
+ "S": 16,
466
+ "slots": 128,
467
+ "bin": "seg03.bin"
468
+ },
469
+ "seg04_S16.xml": {
470
+ "index": 4,
471
+ "consume_moe": 3,
472
+ "router_layer": 4,
473
+ "S": 16,
474
+ "slots": 128,
475
+ "bin": "seg04.bin"
476
+ },
477
+ "seg05_S16.xml": {
478
+ "index": 5,
479
+ "consume_moe": 4,
480
+ "router_layer": 5,
481
+ "S": 16,
482
+ "slots": 128,
483
+ "bin": "seg05.bin"
484
+ },
485
+ "seg06_S16.xml": {
486
+ "index": 6,
487
+ "consume_moe": 5,
488
+ "router_layer": 6,
489
+ "S": 16,
490
+ "slots": 128,
491
+ "bin": "seg06.bin"
492
+ },
493
+ "seg07_S16.xml": {
494
+ "index": 7,
495
+ "consume_moe": 6,
496
+ "router_layer": 7,
497
+ "S": 16,
498
+ "slots": 128,
499
+ "bin": "seg07.bin"
500
+ },
501
+ "seg08_S16.xml": {
502
+ "index": 8,
503
+ "consume_moe": 7,
504
+ "router_layer": 8,
505
+ "S": 16,
506
+ "slots": 128,
507
+ "bin": "seg08.bin"
508
+ },
509
+ "seg09_S16.xml": {
510
+ "index": 9,
511
+ "consume_moe": 8,
512
+ "router_layer": 9,
513
+ "S": 16,
514
+ "slots": 128,
515
+ "bin": "seg09.bin"
516
+ },
517
+ "seg10_S16.xml": {
518
+ "index": 10,
519
+ "consume_moe": 9,
520
+ "router_layer": 10,
521
+ "S": 16,
522
+ "slots": 128,
523
+ "bin": "seg10.bin"
524
+ },
525
+ "seg11_S16.xml": {
526
+ "index": 11,
527
+ "consume_moe": 10,
528
+ "router_layer": 11,
529
+ "S": 16,
530
+ "slots": 128,
531
+ "bin": "seg11.bin"
532
+ },
533
+ "seg12_S16.xml": {
534
+ "index": 12,
535
+ "consume_moe": 11,
536
+ "router_layer": 12,
537
+ "S": 16,
538
+ "slots": 128,
539
+ "bin": "seg12.bin"
540
+ },
541
+ "seg13_S16.xml": {
542
+ "index": 13,
543
+ "consume_moe": 12,
544
+ "router_layer": 13,
545
+ "S": 16,
546
+ "slots": 128,
547
+ "bin": "seg13.bin"
548
+ },
549
+ "seg14_S16.xml": {
550
+ "index": 14,
551
+ "consume_moe": 13,
552
+ "router_layer": 14,
553
+ "S": 16,
554
+ "slots": 128,
555
+ "bin": "seg14.bin"
556
+ },
557
+ "seg15_S16.xml": {
558
+ "index": 15,
559
+ "consume_moe": 14,
560
+ "router_layer": 15,
561
+ "S": 16,
562
+ "slots": 128,
563
+ "bin": "seg15.bin"
564
+ },
565
+ "seg16_S16.xml": {
566
+ "index": 16,
567
+ "consume_moe": 15,
568
+ "router_layer": 16,
569
+ "S": 16,
570
+ "slots": 128,
571
+ "bin": "seg16.bin"
572
+ },
573
+ "seg17_S16.xml": {
574
+ "index": 17,
575
+ "consume_moe": 16,
576
+ "router_layer": 17,
577
+ "S": 16,
578
+ "slots": 128,
579
+ "bin": "seg17.bin"
580
+ },
581
+ "seg18_S16.xml": {
582
+ "index": 18,
583
+ "consume_moe": 17,
584
+ "router_layer": 18,
585
+ "S": 16,
586
+ "slots": 128,
587
+ "bin": "seg18.bin"
588
+ },
589
+ "seg19_S16.xml": {
590
+ "index": 19,
591
+ "consume_moe": 18,
592
+ "router_layer": 19,
593
+ "S": 16,
594
+ "slots": 128,
595
+ "bin": "seg19.bin"
596
+ },
597
+ "seg20_S16.xml": {
598
+ "index": 20,
599
+ "consume_moe": 19,
600
+ "router_layer": 20,
601
+ "S": 16,
602
+ "slots": 128,
603
+ "bin": "seg20.bin"
604
+ },
605
+ "seg21_S16.xml": {
606
+ "index": 21,
607
+ "consume_moe": 20,
608
+ "router_layer": 21,
609
+ "S": 16,
610
+ "slots": 128,
611
+ "bin": "seg21.bin"
612
+ },
613
+ "seg22_S16.xml": {
614
+ "index": 22,
615
+ "consume_moe": 21,
616
+ "router_layer": 22,
617
+ "S": 16,
618
+ "slots": 128,
619
+ "bin": "seg22.bin"
620
+ },
621
+ "seg23_S16.xml": {
622
+ "index": 23,
623
+ "consume_moe": 22,
624
+ "router_layer": 23,
625
+ "S": 16,
626
+ "slots": 128,
627
+ "bin": "seg23.bin"
628
+ },
629
+ "seg24_S16.xml": {
630
+ "index": 24,
631
+ "consume_moe": 23,
632
+ "router_layer": 24,
633
+ "S": 16,
634
+ "slots": 128,
635
+ "bin": "seg24.bin"
636
+ },
637
+ "seg25_S16.xml": {
638
+ "index": 25,
639
+ "consume_moe": 24,
640
+ "router_layer": 25,
641
+ "S": 16,
642
+ "slots": 128,
643
+ "bin": "seg25.bin"
644
+ },
645
+ "seg26_S16.xml": {
646
+ "index": 26,
647
+ "consume_moe": 25,
648
+ "router_layer": 26,
649
+ "S": 16,
650
+ "slots": 128,
651
+ "bin": "seg26.bin"
652
+ },
653
+ "seg27_S16.xml": {
654
+ "index": 27,
655
+ "consume_moe": 26,
656
+ "router_layer": 27,
657
+ "S": 16,
658
+ "slots": 128,
659
+ "bin": "seg27.bin"
660
+ },
661
+ "seg28_S16.xml": {
662
+ "index": 28,
663
+ "consume_moe": 27,
664
+ "router_layer": 28,
665
+ "S": 16,
666
+ "slots": 128,
667
+ "bin": "seg28.bin"
668
+ },
669
+ "seg29_S16.xml": {
670
+ "index": 29,
671
+ "consume_moe": 28,
672
+ "router_layer": 29,
673
+ "S": 16,
674
+ "slots": 128,
675
+ "bin": "seg29.bin"
676
+ },
677
+ "seg30_S16.xml": {
678
+ "index": 30,
679
+ "consume_moe": 29,
680
+ "router_layer": 30,
681
+ "S": 16,
682
+ "slots": 128,
683
+ "bin": "seg30.bin"
684
+ },
685
+ "seg31_S16.xml": {
686
+ "index": 31,
687
+ "consume_moe": 30,
688
+ "router_layer": 31,
689
+ "S": 16,
690
+ "slots": 128,
691
+ "bin": "seg31.bin"
692
+ },
693
+ "seg32_S16.xml": {
694
+ "index": 32,
695
+ "consume_moe": 31,
696
+ "router_layer": 32,
697
+ "S": 16,
698
+ "slots": 128,
699
+ "bin": "seg32.bin"
700
+ },
701
+ "seg33_S16.xml": {
702
+ "index": 33,
703
+ "consume_moe": 32,
704
+ "router_layer": 33,
705
+ "S": 16,
706
+ "slots": 128,
707
+ "bin": "seg33.bin"
708
+ },
709
+ "seg34_S16.xml": {
710
+ "index": 34,
711
+ "consume_moe": 33,
712
+ "router_layer": 34,
713
+ "S": 16,
714
+ "slots": 128,
715
+ "bin": "seg34.bin"
716
+ },
717
+ "seg35_S16.xml": {
718
+ "index": 35,
719
+ "consume_moe": 34,
720
+ "router_layer": 35,
721
+ "S": 16,
722
+ "slots": 128,
723
+ "bin": "seg35.bin"
724
+ },
725
+ "seg36_S16.xml": {
726
+ "index": 36,
727
+ "consume_moe": 35,
728
+ "router_layer": 36,
729
+ "S": 16,
730
+ "slots": 128,
731
+ "bin": "seg36.bin"
732
+ },
733
+ "seg37_S16.xml": {
734
+ "index": 37,
735
+ "consume_moe": 36,
736
+ "router_layer": 37,
737
+ "S": 16,
738
+ "slots": 128,
739
+ "bin": "seg37.bin"
740
+ },
741
+ "seg38_S16.xml": {
742
+ "index": 38,
743
+ "consume_moe": 37,
744
+ "router_layer": 38,
745
+ "S": 16,
746
+ "slots": 128,
747
+ "bin": "seg38.bin"
748
+ },
749
+ "seg39_S16.xml": {
750
+ "index": 39,
751
+ "consume_moe": 38,
752
+ "router_layer": 39,
753
+ "S": 16,
754
+ "slots": 128,
755
+ "bin": "seg39.bin"
756
+ },
757
+ "seg40_S16.xml": {
758
+ "index": 40,
759
+ "consume_moe": 39,
760
+ "router_layer": null,
761
+ "S": 16,
762
+ "slots": 128,
763
+ "bin": "seg40.bin"
764
+ },
765
+ "seg41_S16.xml": {
766
+ "index": 41,
767
+ "consume_moe": null,
768
+ "router_layer": null,
769
+ "S": 16,
770
+ "slots": 0,
771
+ "head": true,
772
+ "bin": "seg41.bin"
773
+ }
774
+ },
775
+ "rope": {
776
+ "dim": 64,
777
+ "theta": 10000000,
778
+ "mrope_section": [
779
+ 11,
780
+ 11,
781
+ 10
782
+ ],
783
+ "mrope_interleaved": true
784
+ },
785
+ "vision": {
786
+ "type": "qwen",
787
+ "grid": [
788
+ 32,
789
+ 32
790
+ ],
791
+ "tokens": 256,
792
+ "merge": 2,
793
+ "patch": 16,
794
+ "temporal": 2,
795
+ "mean": [
796
+ 0.5,
797
+ 0.5,
798
+ 0.5
799
+ ],
800
+ "std": [
801
+ 0.5,
802
+ 0.5,
803
+ 0.5
804
+ ],
805
+ "image_token_id": 248056,
806
+ "image_px": [
807
+ 512,
808
+ 512
809
+ ]
810
+ },
811
+ "tables": {
812
+ "embed": {
813
+ "bits": 4,
814
+ "group": 32
815
+ }
816
+ }
817
+ }
experts.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cc93a890e847d9873bfd12b53b158777e3cb4b2eeff7698702d3b829740a864a
3
+ size 8084520960
experts.bin.json ADDED
The diff for this file is too large to render. See raw diff
 
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 248044,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 248046,
6
+ 248044
7
+ ],
8
+ "pad_token_id": 248044,
9
+ "temperature": 1.0,
10
+ "top_k": 20,
11
+ "top_p": 0.95
12
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
preprocessor_config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "size": {
3
+ "longest_edge": 16777216,
4
+ "shortest_edge": 65536
5
+ },
6
+ "patch_size": 16,
7
+ "temporal_patch_size": 2,
8
+ "merge_size": 2,
9
+ "image_mean": [
10
+ 0.5,
11
+ 0.5,
12
+ 0.5
13
+ ],
14
+ "image_std": [
15
+ 0.5,
16
+ 0.5,
17
+ 0.5
18
+ ],
19
+ "processor_class": "Qwen3VLProcessor",
20
+ "image_processor_type": "Qwen2VLImageProcessorFast"
21
+ }
seg00.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ad89bb3c5d0e1930cd8b109074b856c8687201196ae18d753d75e49091ef31cd
3
+ size 19870268
seg00_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg00_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg01.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0881d8211afa9b2fc96ff5130609e28d47368feea1a8febb6a6c5454ecc27a89
3
+ size 19871308
seg01_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg01_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg02.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93f88d6dfc42098d0bf6b3e821a042e66095271d8d3e2c0b820aa8149c0ffbe5
3
+ size 19871308
seg02_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg02_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg03.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b45b56ea69a6e1f0c30e805103c457ff366a2c14895b21694d531a59969d04b
3
+ size 16228044
seg03_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg03_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg04.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1095b8ea8ac42665a2b69c9596a0c9ed345d68391fa1a826ec993307be478685
3
+ size 19871308
seg04_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg04_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg05.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:10ab15ef04d4eddee97d1742b6312c555039556470f8572e4a8f4b911893c0ec
3
+ size 19871308
seg05_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg05_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg06.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc9ea9be28bb3630102d59e0f7fb230cffa8faf066c69aa1fb89e9e669e7f863
3
+ size 19871308
seg06_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg06_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg07.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e874d8e50f1915ba965c098f933a6fb8cf2dfbc127e0ea0a984ed87dcc79923
3
+ size 16228044
seg07_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg07_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg08.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b096f1164ba4c67fb3a6edffa474e4cb3a626278256f869428eb780c94af5f61
3
+ size 19871308
seg08_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg08_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg09.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a873659987363108faa67795dda82f96e031e9d1ae2ef0c733fa7e9615acbe8f
3
+ size 19871308
seg09_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg09_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg10.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:698e578843e9d871e2c4f461c2ace65fc92402787da985ba806cfdd570e8cbd8
3
+ size 19871308
seg10_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg10_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg11.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:688a1f8762de14eed15ac30ecd654fec2eb2ef494b0fb62c1e0404c7e943db0e
3
+ size 16228044
seg11_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg11_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg12.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:05f4bb9e4f8aaa033dc514eefa34e97433ea92d63bea39d918c16ee03bd73f19
3
+ size 19871308
seg12_S1.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg12_S16.xml ADDED
The diff for this file is too large to render. See raw diff
 
seg13.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aadbcf9a12995a3ef2b326cd7a44dfe27959ab5b9f9e257b72f97e28b8dd1ae3
3
+ size 19871308