diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..aa7aacd0134a92c3c1943fdecc75cd8b7420cce6 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text +tokenizer.json filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md index d00eece498b767d7afb5267c3948897991765863..9050fb5f2f8e35b3dbf4f57ee6264b485ad0ac77 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ --- license: other -base_model: zai-org/GLM-5.3 +base_model: cyankiwi/GLM-5.3-AWQ-INT4 +base_model_relation: quantized pipeline_tag: text-generation tags: - glm @@ -8,36 +9,36 @@ tags: - moe - awq - w4a16 + - int4 - compressed-tensors - reap - expert-pruning + - hopper --- > [!TIP] > **[Support this work →](https://donate.sybilsolutions.ai)** · [X](https://x.com/0xsero) · [GitHub](https://github.com/0xsero) · [REAP paper](https://arxiv.org/abs/2510.13999) · [Cerebras REAP](https://huggingface.co/collections/cerebras/cerebras-reap) -# GLM-5.3-569B-W4A16 — REAP keep-192 (AWQ / W4A16) +# GLM-5.3-569B — REAP keep-192 (W4A16 / AWQ-INT4) -> A **25%-expert-pruned GLM-5.3** (192 / 256 routed experts per layer, ~569B params) in **W4A16** -> (compressed-tensors, AWQ-family) — **324 GB**, built for **Hopper (4× H200)** serving in vLLM/SGLang. +> A **25%-expert-pruned GLM-5.3** (192 / 256 routed experts per layer, ~569B params) in **INT4 W4A16** +> (compressed-tensors, AWQ), built for **Hopper (4x H200)** serving in vLLM / SGLang. -The Hopper-servable sibling of [`GLM-5.3-569B-EXL3-3.0bpw`](https://huggingface.co/0xSero/GLM-5.3-569B-EXL3-3.0bpw). -Pruned with the same **domain-protective max-over-domain saliency criterion** (each expert scored by its -largest share of any single domain's routed mass, so every domain's specialists survive), applied -directly to a W4A16 checkpoint. +Pruned directly on the [cyankiwi/GLM-5.3-AWQ-INT4](https://huggingface.co/cyankiwi/GLM-5.3-AWQ-INT4) checkpoint +with a **domain-protective max-over-domain saliency criterion** (each expert scored by its largest share of +any single domains routed mass, so every domains specialist experts survive). Whole experts are dropped, +kept experts renumbered, and the routers + MTP layer sliced to match. | | | |---|---| -| Base | zai-org/GLM-5.3 → W4A16 (compressed-tensors, group 128) | -| Prune | REAP, keep-192/256, all 75 MoE layers + MTP, routers sliced | -| Size | 324 GB (fits 4× H200) | -| KL vs BF16 | **0.349 nats** (prose 0.54 / code 0.20 / multilingual 0.25) | +| Base | [cyankiwi/GLM-5.3-AWQ-INT4](https://huggingface.co/cyankiwi/GLM-5.3-AWQ-INT4) (compressed-tensors INT4 / AWQ) | +| Prune | REAP, keep-192/256 (max-over-domain routed-mass saliency), all 75 MoE layers + MTP | +| Format | INT4 W4A16, compressed-tensors (Marlin MoE kernel in vLLM) | -At keep-192 the W4A16 and EXL3 3.0bpw cuts measure the same fidelity (0.349 vs 0.361 KLD on the same -25-window full-vocabulary panel) — the prune dominates, so pick the format your hardware wants: W4A16 -for Hopper, EXL3 for Blackwell/consumer. +Criterion and fidelity: [`0xSero/glm-5.3-reap-fidelity-study`](https://huggingface.co/datasets/0xSero/glm-5.3-reap-fidelity-study). +This is the Hopper-servable sibling of [`GLM-5.3-569B-EXL3-3.0bpw`](https://huggingface.co/0xSero/GLM-5.3-569B-EXL3-3.0bpw). -## Serving (vLLM, 4× H200) +## Serving (vLLM, 4x H200) ```bash vllm serve 0xSero/GLM-5.3-569B-W4A16 --tensor-parallel-size 4 --trust-remote-code --max-model-len 131072 @@ -46,18 +47,18 @@ vllm serve 0xSero/GLM-5.3-569B-W4A16 --tensor-parallel-size 4 --trust-remote-cod ## Acknowledgements - **[Z.AI / zai-org](https://huggingface.co/zai-org)** for [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3), the base model. -- **[Cerebras Research](https://github.com/CerebrasResearch/reap)** for REAP — [arXiv:2510.13999](https://arxiv.org/abs/2510.13999). -- W4A16 base quantization via **compressed-tensors** (AWQ-family). +- **[cyankiwi](https://huggingface.co/cyankiwi)** for the [GLM-5.3-AWQ-INT4](https://huggingface.co/cyankiwi/GLM-5.3-AWQ-INT4) W4A16 base quantization this prune is built on. +- **[Cerebras Research](https://github.com/CerebrasResearch/reap)** for REAP (Router-weighted Expert Activation Pruning) — [arXiv:2510.13999](https://arxiv.org/abs/2510.13999). -Observations: [`0xSero/glm-5.3-nvfp4-reap-observations-v1`](https://huggingface.co/datasets/0xSero/glm-5.3-nvfp4-reap-observations-v1). Study: [`0xSero/glm-5.3-reap-fidelity-study`](https://huggingface.co/datasets/0xSero/glm-5.3-reap-fidelity-study). Pruning + evaluation on 8× NVIDIA RTX PRO 6000 Blackwell. +Observations: [`0xSero/glm-5.3-reap-observations-v1`](https://huggingface.co/datasets/0xSero/glm-5.3-reap-observations-v1). Fidelity study: [`0xSero/glm-5.3-reap-fidelity-study`](https://huggingface.co/datasets/0xSero/glm-5.3-reap-fidelity-study). Pruning and evaluation ran on 8x NVIDIA RTX PRO 6000 Blackwell. ## Citation ```bibtex @misc{lasby2025reap, - title = {REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression}, - author = {Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa}, - year = {2025}, eprint = {2510.13999}, archivePrefix = {arXiv} + title = {{REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression}}, + author = {{Mike Lasby and Ivan Lazarevich and Nish Sinnadurai and Sean Lie and Yani Ioannou and Vithursan Thangarasa}}, + year = {{2025}}, eprint = {{2510.13999}}, archivePrefix = {{arXiv}} } ``` diff --git a/chat_template.jinja b/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..d226a6928e6ba040e2eadccaf832c037bc4cd98f --- /dev/null +++ b/chat_template.jinja @@ -0,0 +1,251 @@ +[gMASK] +{%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%} +{%- if effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%} +{%- set clear_thinking = clear_thinking if clear_thinking is defined else false -%} +{%- if tools -%} +{%- macro tool_to_json(tool) -%} + {%- set ns_tool = namespace(first=true) -%} + {{ '{' -}} + {%- for k, v in tool.items() -%} + {%- if k != 'defer_loading' and k != 'strict' -%} + {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%} + {%- set ns_tool.first = false -%} + "{{ k }}": {{ v | tojson(ensure_ascii=False) }} + {%- endif -%} + {%- endfor -%} + {{- '}' -}} +{%- endmacro -%} +{%- macro tool_references_to_response(refs) -%} + {{- '\n' -}} + {%- for tr in refs -%} + {%- for tool in tools -%} + {%- if 'function' in tool -%} + {%- set tool = tool['function'] -%} + {%- endif -%} + {%- if tool.name == tr.name -%} + {{- tool_to_json(tool) + '\n' -}} + {%- endif -%} + {%- endfor -%} + {%- endfor -%} + {{- '' -}} +{%- endmacro -%} +<|system|> +# Tools + +You may call one or more functions to assist with the user query. + +You are provided with function signatures within XML tags: + +{% for tool in tools %} +{%- if 'function' in tool -%} + {%- set tool = tool['function'] -%} +{%- endif -%} +{% if tool.defer_loading is not defined or not tool.defer_loading %} +{{ tool_to_json(tool) }} +{% endif %} +{% endfor %} + + +For each function call, output the function name and arguments within the following XML format: +{function-name}{arg-key-1}{arg-value-1}{arg-key-2}{arg-value-2}...{%- endif -%} +{%- macro visible_text(content) -%} + {%- if content is string -%} + {{- content }} + {%- elif content is iterable and content is not mapping -%} + {%- for item in content -%} + {%- if item is mapping and item.type == 'text' -%} + {{- item.text }} + {%- elif item is string -%} + {{- item }} + {%- elif item is mapping and item.type in ['image', 'image_url', 'video', 'video_url', 'audio', 'audio_url', 'input_audio'] -%} + {%- set media_type = item.type | replace('_url', '') | replace('input_', '') -%} + {{- "You are unable to process this " ~ media_type ~ " because you don't have multi-modal input ability. Try different methods." }} + {%- endif -%} + {%- endfor -%} + {%- else -%} + {{- content }} + {%- endif -%} +{%- endmacro -%} +{%- macro tool_response(text) -%} +{{- '' + text + '' -}} +{%- endmacro -%} +{%- macro render_tool_response(m) -%} +{%- if m.content is string -%} + {{- tool_response(m.content) -}} +{%- elif m.content and m.content is not mapping and m.content.0.type == "tool_reference" -%} + {{- tool_references_to_response(m.content) -}} +{%- elif is_list_of_outputs(m) -%} + {%- for tr in m.content -%} + {%- if tr.output is iterable and tr.output is not string and tr.output is not mapping and tr.output and tr.output.0.type == "tool_reference" -%} + {{- tool_references_to_response(tr.output) -}} + {%- else -%} + {{- tool_response(visible_text(tr.output)) -}} + {%- endif -%} + {%- endfor -%} +{%- else -%} + {{- tool_response(visible_text(m.content)) -}} +{%- endif -%} +{%- endmacro -%} +{%- macro id_of(obj) -%} + {%- if obj.tool_call_id -%} + {{- obj.tool_call_id -}} + {%- elif obj.id -%} + {{- obj.id -}} + {%- endif -%} +{%- endmacro -%} +{%- macro is_list_of_outputs(m) -%} + {%- if m.content and m.content.0.output is defined -%}1{%- endif -%} +{%- endmacro -%} +{%- macro has_dup_tool_result_id(lo, hi, target) -%} + {%- set ns_cnt = namespace(n=0) -%} + {%- for k in range(lo, hi + 1) -%} + {%- set m = messages[k] -%} + {%- if is_list_of_outputs(m) -%} + {%- for entry in m.content -%} + {%- if id_of(entry) == target -%} + {%- set ns_cnt.n = ns_cnt.n + 1 -%} + {%- endif -%} + {%- endfor -%} + {%- elif id_of(m) == target -%} + {%- set ns_cnt.n = ns_cnt.n + 1 -%} + {%- endif -%} + {%- if ns_cnt.n > 1 -%}{%- break -%}{%- endif -%} + {%- endfor -%} + {%- if ns_cnt.n > 1 -%}1{%- endif -%} +{%- endmacro -%} +{%- macro tc_id_exists(tcs, target) -%} + {%- set ns_f = namespace(found=false) -%} + {%- for tc in tcs -%} + {%- if id_of(tc) == target -%} + {%- set ns_f.found = true -%} + {%- break -%} + {%- endif -%} + {%- endfor -%} + {%- if ns_f.found -%}1{%- endif -%} +{%- endmacro -%} +{%- set ns = namespace(last_user_index=-1) -%} +{%- for m in messages %} + {%- if m.role == 'user' %} + {%- set ns.last_user_index = loop.index0 -%} + {%- endif %} +{%- endfor %} +{%- for m in messages -%} +{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }} +{%- elif m.role == 'assistant' -%} +<|assistant|> +{%- set content = visible_text(m.content) %} +{%- if m.reasoning_content is string %} + {%- set reasoning_content = m.reasoning_content %} +{%- elif '' in content %} + {%- set reasoning_content = content.split('')[0].split('')[-1] %} + {%- set content = content.split('')[-1] %} +{%- endif %} +{%- if (not clear_thinking or loop.index0 > ns.last_user_index) and reasoning_content is defined -%} +{{ '' + reasoning_content + ''}} +{%- else -%} +{{ '' }} +{%- endif -%} +{%- if content.strip() -%} +{{ content.strip() }} +{%- endif -%} +{% if m.tool_calls %} +{% for tc in m.tool_calls %} +{%- if tc.function %} + {%- set tc = tc.function %} +{%- endif %} +{{- '' + tc.name -}} +{% set _args = tc.arguments %}{% for k, v in _args.items() %}{{ k }}{{ v | tojson(ensure_ascii=False) if v is not string else v }}{% endfor %}{% endfor %} +{% endif %} +{%- elif m.role == 'tool' -%} +{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|observation|>' -}} + {%- set block_start = loop.index0 -%} + {%- set ns_blk = namespace(end=block_start) -%} + {%- for j in range(block_start, messages|length) -%} + {%- if messages[j].role == 'tool' -%} + {%- set ns_blk.end = j -%} + {%- else -%} + {%- break -%} + {%- endif -%} + {%- endfor -%} + {%- set ns_a = namespace(tool_calls=none) -%} + {%- if block_start > 0 and messages[block_start - 1].role == 'assistant' and messages[block_start - 1].tool_calls -%} + {%- set ns_a.tool_calls = messages[block_start - 1].tool_calls -%} + {%- endif -%} + {%- set ns_chk = namespace(can_sort=true) -%} + {%- if not ns_a.tool_calls -%} + {%- set ns_chk.can_sort = false -%} + {%- else -%} + {%- for k in range(block_start, ns_blk.end + 1) -%} + {%- set m = messages[k] -%} + {%- if is_list_of_outputs(m) -%} + {%- for entry in m.content -%} + {%- set eid = id_of(entry) -%} + {%- if not eid -%} + {%- set ns_chk.can_sort = false -%} + {%- elif has_dup_tool_result_id(block_start, ns_blk.end, eid) -%} + {%- set ns_chk.can_sort = false -%} + {%- elif not tc_id_exists(ns_a.tool_calls, eid) -%} + {%- set ns_chk.can_sort = false -%} + {%- endif -%} + {%- endfor -%} + {%- else -%} + {%- set tk_id = id_of(m) -%} + {%- if not tk_id -%} + {%- set ns_chk.can_sort = false -%} + {%- elif has_dup_tool_result_id(block_start, ns_blk.end, tk_id) -%} + {%- set ns_chk.can_sort = false -%} + {%- elif not tc_id_exists(ns_a.tool_calls, tk_id) -%} + {%- set ns_chk.can_sort = false -%} + {%- endif -%} + {%- endif -%} + {%- endfor -%} + {%- for i in range(ns_a.tool_calls | length) -%} + {%- set tc_id = id_of(ns_a.tool_calls[i]) -%} + {%- if not tc_id -%} + {%- set ns_chk.can_sort = false -%} + {%- endif -%} + {%- for j in range(i + 1, ns_a.tool_calls | length) -%} + {%- if id_of(ns_a.tool_calls[j]) == tc_id -%} + {%- set ns_chk.can_sort = false -%} + {%- endif -%} + {%- endfor -%} + {%- endfor -%} + {%- endif -%} + {%- if ns_chk.can_sort -%} + {%- for tc in ns_a.tool_calls -%} + {%- set tc_id = id_of(tc) -%} + {%- for k in range(block_start, ns_blk.end + 1) -%} + {%- set m = messages[k] -%} + {%- if is_list_of_outputs(m) -%} + {%- for entry in m.content -%} + {%- set eid = id_of(entry) -%} + {%- if eid == tc_id -%} + {%- if entry.output is iterable and entry.output is not string and entry.output is not mapping and entry.output and entry.output.0.type == "tool_reference" -%} + {{- tool_references_to_response(entry.output) -}} + {%- else -%} + {{- tool_response(visible_text(entry.output)) -}} + {%- endif -%} + {%- endif -%} + {%- endfor -%} + {%- else -%} + {%- set tk_id = id_of(m) -%} + {%- if tk_id == tc_id -%} + {{- render_tool_response(m) -}} + {%- endif -%} + {%- endif -%} + {%- endfor -%} + {%- endfor -%} + {%- else -%} + {%- for k in range(block_start, ns_blk.end + 1) -%} + {{- render_tool_response(messages[k]) -}} + {%- endfor -%} + {%- endif -%} +{% endif -%} +{%- elif m.role == 'system' -%} +<|system|>{{ visible_text(m.content) }} +{%- endif -%} +{%- endfor -%} +{%- if add_generation_prompt -%} + <|assistant|>{{- '' -}} +{%- endif -%} diff --git a/config.json b/config.json new file mode 100644 index 0000000000000000000000000000000000000000..9cbceed72616895b1e75ad54f7187f18bb8fd59e --- /dev/null +++ b/config.json @@ -0,0 +1,3205 @@ +{ + "architectures": [ + "GlmMoeDsaForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 0, + "dtype": "bfloat16", + "eos_token_id": [ + 154820, + 154827, + 154829 + ], + "ep_size": 1, + "first_k_dense_replace": 3, + "head_dim": 64, + "hidden_act": "silu", + "hidden_size": 6144, + "index_head_dim": 128, + "index_n_heads": 32, + "index_share_for_mtp_iteration": true, + "index_skip_topk_offset": 3, + "index_topk": 2048, + "index_topk_freq": 4, + "index_topk_pattern": null, + "indexer_rope_interleave": true, + "indexer_types": [ + "full", + "full", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared", + "full", + "shared", + "shared", + "shared" + ], + "initializer_range": 0.02, + "intermediate_size": 12288, + "kv_lora_rank": 512, + "layer_types": [ + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "deepseek_sparse_attention" + ], + "max_position_embeddings": 1048576, + "mlp_bias": false, + "mlp_layer_types": [ + "dense", + "dense", + "dense", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse" + ], + "model_type": "glm_moe_dsa", + "moe_intermediate_size": 2048, + "moe_layer_freq": 1, + "moe_router_dtype": "float32", + "n_group": 1, + "n_routed_experts": 192, + "n_shared_experts": 1, + "norm_topk_prob": true, + "num_attention_heads": 64, + "num_experts_per_tok": 8, + "num_hidden_layers": 78, + "num_key_value_heads": 64, + "num_nextn_predict_layers": 1, + "pad_token_id": 154820, + "pretraining_tp": 1, + "q_lora_rank": 2048, + "qk_head_dim": 256, + "qk_nope_head_dim": 192, + "qk_rope_head_dim": 64, + "quantization_config": { + "config_groups": { + "group_0": { + "format": "pack-quantized", + "input_activations": null, + "output_activations": null, + "targets": [ + "Linear", + "re:.*fused_qkv_a_proj$" + ], + "weights": { + "actorder": null, + "block_structure": null, + "dynamic": false, + "group_size": 32, + "num_bits": 4, + "observer": "mse", + "observer_kwargs": {}, + "scale_dtype": null, + "strategy": "group", + "symmetric": false, + "type": "int", + "zp_dtype": "torch.int8" + } + } + }, + "format": "pack-quantized", + "global_compression_ratio": null, + "ignore": [ + "model.layers.0.self_attn.q_a_proj", + "model.layers.0.self_attn.q_b_proj", + "model.layers.0.self_attn.kv_a_proj_with_mqa", + "model.layers.0.self_attn.kv_b_proj", + "model.layers.0.self_attn.o_proj", + "model.layers.0.self_attn.indexer.wq_b", + "model.layers.0.self_attn.indexer.wk", + "model.layers.0.self_attn.indexer.weights_proj", + "model.layers.0.mlp.gate_proj", + "model.layers.0.mlp.up_proj", + "model.layers.0.mlp.down_proj", + "model.layers.1.self_attn.q_a_proj", + "model.layers.1.self_attn.q_b_proj", + "model.layers.1.self_attn.kv_a_proj_with_mqa", + "model.layers.1.self_attn.kv_b_proj", + "model.layers.1.self_attn.o_proj", + "model.layers.1.self_attn.indexer.wq_b", + "model.layers.1.self_attn.indexer.wk", + "model.layers.1.self_attn.indexer.weights_proj", + "model.layers.1.mlp.gate_proj", + "model.layers.1.mlp.up_proj", + "model.layers.1.mlp.down_proj", + "model.layers.2.self_attn.q_a_proj", + "model.layers.2.self_attn.q_b_proj", + "model.layers.2.self_attn.kv_a_proj_with_mqa", + "model.layers.2.self_attn.kv_b_proj", + "model.layers.2.self_attn.o_proj", + "model.layers.2.self_attn.indexer.wq_b", + "model.layers.2.self_attn.indexer.wk", + "model.layers.2.self_attn.indexer.weights_proj", + "model.layers.2.mlp.gate_proj", + "model.layers.2.mlp.up_proj", + "model.layers.2.mlp.down_proj", + "model.layers.3.self_attn.q_a_proj", + "model.layers.3.self_attn.q_b_proj", + "model.layers.3.self_attn.kv_a_proj_with_mqa", + "model.layers.3.self_attn.kv_b_proj", + "model.layers.3.self_attn.o_proj", + "model.layers.3.mlp.gate", + "model.layers.3.mlp.experts.0.gate_proj", + "model.layers.3.mlp.experts.0.up_proj", + "model.layers.3.mlp.experts.0.down_proj", + "model.layers.3.mlp.experts.1.gate_proj", + "model.layers.3.mlp.experts.1.up_proj", + "model.layers.3.mlp.experts.1.down_proj", + "model.layers.3.mlp.experts.2.gate_proj", + "model.layers.3.mlp.experts.2.up_proj", + "model.layers.3.mlp.experts.2.down_proj", + "model.layers.3.mlp.experts.3.gate_proj", + "model.layers.3.mlp.experts.3.up_proj", + "model.layers.3.mlp.experts.3.down_proj", + "model.layers.3.mlp.experts.4.gate_proj", + "model.layers.3.mlp.experts.4.up_proj", + "model.layers.3.mlp.experts.4.down_proj", + "model.layers.3.mlp.experts.5.gate_proj", + "model.layers.3.mlp.experts.5.up_proj", + "model.layers.3.mlp.experts.5.down_proj", + "model.layers.3.mlp.experts.6.gate_proj", + "model.layers.3.mlp.experts.6.up_proj", + "model.layers.3.mlp.experts.6.down_proj", + "model.layers.3.mlp.experts.7.gate_proj", + "model.layers.3.mlp.experts.7.up_proj", + "model.layers.3.mlp.experts.7.down_proj", + "model.layers.3.mlp.experts.8.gate_proj", + "model.layers.3.mlp.experts.8.up_proj", + "model.layers.3.mlp.experts.8.down_proj", + "model.layers.3.mlp.experts.9.gate_proj", + "model.layers.3.mlp.experts.9.up_proj", + "model.layers.3.mlp.experts.9.down_proj", + "model.layers.3.mlp.experts.10.gate_proj", + "model.layers.3.mlp.experts.10.up_proj", + "model.layers.3.mlp.experts.10.down_proj", + "model.layers.3.mlp.experts.11.gate_proj", + "model.layers.3.mlp.experts.11.up_proj", + "model.layers.3.mlp.experts.11.down_proj", + "model.layers.3.mlp.experts.12.gate_proj", + "model.layers.3.mlp.experts.12.up_proj", + "model.layers.3.mlp.experts.12.down_proj", + "model.layers.3.mlp.experts.13.gate_proj", + "model.layers.3.mlp.experts.13.up_proj", + "model.layers.3.mlp.experts.13.down_proj", + "model.layers.3.mlp.experts.14.gate_proj", + "model.layers.3.mlp.experts.14.up_proj", + "model.layers.3.mlp.experts.14.down_proj", + "model.layers.3.mlp.experts.15.gate_proj", + "model.layers.3.mlp.experts.15.up_proj", + "model.layers.3.mlp.experts.15.down_proj", + "model.layers.3.mlp.experts.16.gate_proj", + "model.layers.3.mlp.experts.16.up_proj", + "model.layers.3.mlp.experts.16.down_proj", + "model.layers.3.mlp.experts.17.gate_proj", + "model.layers.3.mlp.experts.17.up_proj", + "model.layers.3.mlp.experts.17.down_proj", + "model.layers.3.mlp.experts.18.gate_proj", + "model.layers.3.mlp.experts.18.up_proj", + "model.layers.3.mlp.experts.18.down_proj", + "model.layers.3.mlp.experts.19.gate_proj", + "model.layers.3.mlp.experts.19.up_proj", + "model.layers.3.mlp.experts.19.down_proj", + "model.layers.3.mlp.experts.20.gate_proj", + "model.layers.3.mlp.experts.20.up_proj", + "model.layers.3.mlp.experts.20.down_proj", + "model.layers.3.mlp.experts.21.gate_proj", + "model.layers.3.mlp.experts.21.up_proj", + "model.layers.3.mlp.experts.21.down_proj", + "model.layers.3.mlp.experts.22.gate_proj", + "model.layers.3.mlp.experts.22.up_proj", + "model.layers.3.mlp.experts.22.down_proj", + "model.layers.3.mlp.experts.23.gate_proj", + "model.layers.3.mlp.experts.23.up_proj", + "model.layers.3.mlp.experts.23.down_proj", + "model.layers.3.mlp.experts.24.gate_proj", + "model.layers.3.mlp.experts.24.up_proj", + "model.layers.3.mlp.experts.24.down_proj", + "model.layers.3.mlp.experts.25.gate_proj", + "model.layers.3.mlp.experts.25.up_proj", + "model.layers.3.mlp.experts.25.down_proj", + "model.layers.3.mlp.experts.26.gate_proj", + "model.layers.3.mlp.experts.26.up_proj", + "model.layers.3.mlp.experts.26.down_proj", + "model.layers.3.mlp.experts.27.gate_proj", + "model.layers.3.mlp.experts.27.up_proj", + "model.layers.3.mlp.experts.27.down_proj", + "model.layers.3.mlp.experts.28.gate_proj", + "model.layers.3.mlp.experts.28.up_proj", + "model.layers.3.mlp.experts.28.down_proj", + "model.layers.3.mlp.experts.29.gate_proj", + "model.layers.3.mlp.experts.29.up_proj", + "model.layers.3.mlp.experts.29.down_proj", + "model.layers.3.mlp.experts.30.gate_proj", + "model.layers.3.mlp.experts.30.up_proj", + "model.layers.3.mlp.experts.30.down_proj", + "model.layers.3.mlp.experts.31.gate_proj", + "model.layers.3.mlp.experts.31.up_proj", + "model.layers.3.mlp.experts.31.down_proj", + "model.layers.3.mlp.experts.32.gate_proj", + "model.layers.3.mlp.experts.32.up_proj", + "model.layers.3.mlp.experts.32.down_proj", + "model.layers.3.mlp.experts.33.gate_proj", + "model.layers.3.mlp.experts.33.up_proj", + "model.layers.3.mlp.experts.33.down_proj", + "model.layers.3.mlp.experts.34.gate_proj", + "model.layers.3.mlp.experts.34.up_proj", + "model.layers.3.mlp.experts.34.down_proj", + "model.layers.3.mlp.experts.35.gate_proj", + "model.layers.3.mlp.experts.35.up_proj", + "model.layers.3.mlp.experts.35.down_proj", + "model.layers.3.mlp.experts.36.gate_proj", + "model.layers.3.mlp.experts.36.up_proj", + "model.layers.3.mlp.experts.36.down_proj", + "model.layers.3.mlp.experts.37.gate_proj", + "model.layers.3.mlp.experts.37.up_proj", + "model.layers.3.mlp.experts.37.down_proj", + "model.layers.3.mlp.experts.38.gate_proj", + "model.layers.3.mlp.experts.38.up_proj", + "model.layers.3.mlp.experts.38.down_proj", + "model.layers.3.mlp.experts.39.gate_proj", + "model.layers.3.mlp.experts.39.up_proj", + "model.layers.3.mlp.experts.39.down_proj", + "model.layers.3.mlp.experts.40.gate_proj", + "model.layers.3.mlp.experts.40.up_proj", + "model.layers.3.mlp.experts.40.down_proj", + "model.layers.3.mlp.experts.41.gate_proj", + "model.layers.3.mlp.experts.41.up_proj", + "model.layers.3.mlp.experts.41.down_proj", + "model.layers.3.mlp.experts.42.gate_proj", + "model.layers.3.mlp.experts.42.up_proj", + "model.layers.3.mlp.experts.42.down_proj", + "model.layers.3.mlp.experts.43.gate_proj", + "model.layers.3.mlp.experts.43.up_proj", + "model.layers.3.mlp.experts.43.down_proj", + "model.layers.3.mlp.experts.44.gate_proj", + "model.layers.3.mlp.experts.44.up_proj", + "model.layers.3.mlp.experts.44.down_proj", + "model.layers.3.mlp.experts.45.gate_proj", + "model.layers.3.mlp.experts.45.up_proj", + "model.layers.3.mlp.experts.45.down_proj", + "model.layers.3.mlp.experts.46.gate_proj", + "model.layers.3.mlp.experts.46.up_proj", + "model.layers.3.mlp.experts.46.down_proj", + "model.layers.3.mlp.experts.47.gate_proj", + "model.layers.3.mlp.experts.47.up_proj", + "model.layers.3.mlp.experts.47.down_proj", + "model.layers.3.mlp.experts.48.gate_proj", + "model.layers.3.mlp.experts.48.up_proj", + "model.layers.3.mlp.experts.48.down_proj", + "model.layers.3.mlp.experts.49.gate_proj", + "model.layers.3.mlp.experts.49.up_proj", + "model.layers.3.mlp.experts.49.down_proj", + "model.layers.3.mlp.experts.50.gate_proj", + "model.layers.3.mlp.experts.50.up_proj", + "model.layers.3.mlp.experts.50.down_proj", + "model.layers.3.mlp.experts.51.gate_proj", + "model.layers.3.mlp.experts.51.up_proj", + "model.layers.3.mlp.experts.51.down_proj", + "model.layers.3.mlp.experts.52.gate_proj", + "model.layers.3.mlp.experts.52.up_proj", + "model.layers.3.mlp.experts.52.down_proj", + "model.layers.3.mlp.experts.53.gate_proj", + "model.layers.3.mlp.experts.53.up_proj", + "model.layers.3.mlp.experts.53.down_proj", + "model.layers.3.mlp.experts.54.gate_proj", + "model.layers.3.mlp.experts.54.up_proj", + "model.layers.3.mlp.experts.54.down_proj", + "model.layers.3.mlp.experts.55.gate_proj", + "model.layers.3.mlp.experts.55.up_proj", + "model.layers.3.mlp.experts.55.down_proj", + "model.layers.3.mlp.experts.56.gate_proj", + "model.layers.3.mlp.experts.56.up_proj", + "model.layers.3.mlp.experts.56.down_proj", + "model.layers.3.mlp.experts.57.gate_proj", + "model.layers.3.mlp.experts.57.up_proj", + "model.layers.3.mlp.experts.57.down_proj", + "model.layers.3.mlp.experts.58.gate_proj", + "model.layers.3.mlp.experts.58.up_proj", + "model.layers.3.mlp.experts.58.down_proj", + "model.layers.3.mlp.experts.59.gate_proj", + "model.layers.3.mlp.experts.59.up_proj", + "model.layers.3.mlp.experts.59.down_proj", + "model.layers.3.mlp.experts.60.gate_proj", + "model.layers.3.mlp.experts.60.up_proj", + "model.layers.3.mlp.experts.60.down_proj", + "model.layers.3.mlp.experts.61.gate_proj", + "model.layers.3.mlp.experts.61.up_proj", + "model.layers.3.mlp.experts.61.down_proj", + "model.layers.3.mlp.experts.62.gate_proj", + "model.layers.3.mlp.experts.62.up_proj", + "model.layers.3.mlp.experts.62.down_proj", + "model.layers.3.mlp.experts.63.gate_proj", + "model.layers.3.mlp.experts.63.up_proj", + "model.layers.3.mlp.experts.63.down_proj", + "model.layers.3.mlp.experts.64.gate_proj", + "model.layers.3.mlp.experts.64.up_proj", + "model.layers.3.mlp.experts.64.down_proj", + "model.layers.3.mlp.experts.65.gate_proj", + "model.layers.3.mlp.experts.65.up_proj", + "model.layers.3.mlp.experts.65.down_proj", + "model.layers.3.mlp.experts.66.gate_proj", + "model.layers.3.mlp.experts.66.up_proj", + "model.layers.3.mlp.experts.66.down_proj", + "model.layers.3.mlp.experts.67.gate_proj", + "model.layers.3.mlp.experts.67.up_proj", + "model.layers.3.mlp.experts.67.down_proj", + "model.layers.3.mlp.experts.68.gate_proj", + "model.layers.3.mlp.experts.68.up_proj", + "model.layers.3.mlp.experts.68.down_proj", + "model.layers.3.mlp.experts.69.gate_proj", + "model.layers.3.mlp.experts.69.up_proj", + "model.layers.3.mlp.experts.69.down_proj", + "model.layers.3.mlp.experts.70.gate_proj", + "model.layers.3.mlp.experts.70.up_proj", + "model.layers.3.mlp.experts.70.down_proj", + "model.layers.3.mlp.experts.71.gate_proj", + "model.layers.3.mlp.experts.71.up_proj", + "model.layers.3.mlp.experts.71.down_proj", + "model.layers.3.mlp.experts.72.gate_proj", + "model.layers.3.mlp.experts.72.up_proj", + "model.layers.3.mlp.experts.72.down_proj", + "model.layers.3.mlp.experts.73.gate_proj", + "model.layers.3.mlp.experts.73.up_proj", + "model.layers.3.mlp.experts.73.down_proj", + "model.layers.3.mlp.experts.74.gate_proj", + "model.layers.3.mlp.experts.74.up_proj", + "model.layers.3.mlp.experts.74.down_proj", + "model.layers.3.mlp.experts.75.gate_proj", + "model.layers.3.mlp.experts.75.up_proj", + "model.layers.3.mlp.experts.75.down_proj", + "model.layers.3.mlp.experts.76.gate_proj", + "model.layers.3.mlp.experts.76.up_proj", + "model.layers.3.mlp.experts.76.down_proj", + "model.layers.3.mlp.experts.77.gate_proj", + "model.layers.3.mlp.experts.77.up_proj", + "model.layers.3.mlp.experts.77.down_proj", + "model.layers.3.mlp.experts.78.gate_proj", + "model.layers.3.mlp.experts.78.up_proj", + "model.layers.3.mlp.experts.78.down_proj", + "model.layers.3.mlp.experts.79.gate_proj", + "model.layers.3.mlp.experts.79.up_proj", + "model.layers.3.mlp.experts.79.down_proj", + "model.layers.3.mlp.experts.80.gate_proj", + "model.layers.3.mlp.experts.80.up_proj", + "model.layers.3.mlp.experts.80.down_proj", + "model.layers.3.mlp.experts.81.gate_proj", + "model.layers.3.mlp.experts.81.up_proj", + "model.layers.3.mlp.experts.81.down_proj", + "model.layers.3.mlp.experts.82.gate_proj", + "model.layers.3.mlp.experts.82.up_proj", + "model.layers.3.mlp.experts.82.down_proj", + "model.layers.3.mlp.experts.83.gate_proj", + "model.layers.3.mlp.experts.83.up_proj", + "model.layers.3.mlp.experts.83.down_proj", + "model.layers.3.mlp.experts.84.gate_proj", + "model.layers.3.mlp.experts.84.up_proj", + "model.layers.3.mlp.experts.84.down_proj", + "model.layers.3.mlp.experts.85.gate_proj", + "model.layers.3.mlp.experts.85.up_proj", + "model.layers.3.mlp.experts.85.down_proj", + "model.layers.3.mlp.experts.86.gate_proj", + "model.layers.3.mlp.experts.86.up_proj", + "model.layers.3.mlp.experts.86.down_proj", + "model.layers.3.mlp.experts.87.gate_proj", + "model.layers.3.mlp.experts.87.up_proj", + "model.layers.3.mlp.experts.87.down_proj", + "model.layers.3.mlp.experts.88.gate_proj", + "model.layers.3.mlp.experts.88.up_proj", + "model.layers.3.mlp.experts.88.down_proj", + "model.layers.3.mlp.experts.89.gate_proj", + "model.layers.3.mlp.experts.89.up_proj", + "model.layers.3.mlp.experts.89.down_proj", + "model.layers.3.mlp.experts.90.gate_proj", + "model.layers.3.mlp.experts.90.up_proj", + "model.layers.3.mlp.experts.90.down_proj", + "model.layers.3.mlp.experts.91.gate_proj", + "model.layers.3.mlp.experts.91.up_proj", + "model.layers.3.mlp.experts.91.down_proj", + "model.layers.3.mlp.experts.92.gate_proj", + "model.layers.3.mlp.experts.92.up_proj", + "model.layers.3.mlp.experts.92.down_proj", + "model.layers.3.mlp.experts.93.gate_proj", + "model.layers.3.mlp.experts.93.up_proj", + "model.layers.3.mlp.experts.93.down_proj", + "model.layers.3.mlp.experts.94.gate_proj", + "model.layers.3.mlp.experts.94.up_proj", + "model.layers.3.mlp.experts.94.down_proj", + "model.layers.3.mlp.experts.95.gate_proj", + "model.layers.3.mlp.experts.95.up_proj", + "model.layers.3.mlp.experts.95.down_proj", + "model.layers.3.mlp.experts.96.gate_proj", + "model.layers.3.mlp.experts.96.up_proj", + "model.layers.3.mlp.experts.96.down_proj", + "model.layers.3.mlp.experts.97.gate_proj", + "model.layers.3.mlp.experts.97.up_proj", + "model.layers.3.mlp.experts.97.down_proj", + "model.layers.3.mlp.experts.98.gate_proj", + "model.layers.3.mlp.experts.98.up_proj", + "model.layers.3.mlp.experts.98.down_proj", + "model.layers.3.mlp.experts.99.gate_proj", + "model.layers.3.mlp.experts.99.up_proj", + "model.layers.3.mlp.experts.99.down_proj", + "model.layers.3.mlp.experts.100.gate_proj", + "model.layers.3.mlp.experts.100.up_proj", + "model.layers.3.mlp.experts.100.down_proj", + "model.layers.3.mlp.experts.101.gate_proj", + "model.layers.3.mlp.experts.101.up_proj", + "model.layers.3.mlp.experts.101.down_proj", + "model.layers.3.mlp.experts.102.gate_proj", + "model.layers.3.mlp.experts.102.up_proj", + "model.layers.3.mlp.experts.102.down_proj", + "model.layers.3.mlp.experts.103.gate_proj", + "model.layers.3.mlp.experts.103.up_proj", + "model.layers.3.mlp.experts.103.down_proj", + "model.layers.3.mlp.experts.104.gate_proj", + "model.layers.3.mlp.experts.104.up_proj", + "model.layers.3.mlp.experts.104.down_proj", + "model.layers.3.mlp.experts.105.gate_proj", + "model.layers.3.mlp.experts.105.up_proj", + "model.layers.3.mlp.experts.105.down_proj", + "model.layers.3.mlp.experts.106.gate_proj", + "model.layers.3.mlp.experts.106.up_proj", + "model.layers.3.mlp.experts.106.down_proj", + "model.layers.3.mlp.experts.107.gate_proj", + "model.layers.3.mlp.experts.107.up_proj", + "model.layers.3.mlp.experts.107.down_proj", + "model.layers.3.mlp.experts.108.gate_proj", + "model.layers.3.mlp.experts.108.up_proj", + "model.layers.3.mlp.experts.108.down_proj", + "model.layers.3.mlp.experts.109.gate_proj", + "model.layers.3.mlp.experts.109.up_proj", + "model.layers.3.mlp.experts.109.down_proj", + "model.layers.3.mlp.experts.110.gate_proj", + "model.layers.3.mlp.experts.110.up_proj", + "model.layers.3.mlp.experts.110.down_proj", + "model.layers.3.mlp.experts.111.gate_proj", + "model.layers.3.mlp.experts.111.up_proj", + "model.layers.3.mlp.experts.111.down_proj", + "model.layers.3.mlp.experts.112.gate_proj", + "model.layers.3.mlp.experts.112.up_proj", + "model.layers.3.mlp.experts.112.down_proj", + "model.layers.3.mlp.experts.113.gate_proj", + "model.layers.3.mlp.experts.113.up_proj", + "model.layers.3.mlp.experts.113.down_proj", + "model.layers.3.mlp.experts.114.gate_proj", + "model.layers.3.mlp.experts.114.up_proj", + "model.layers.3.mlp.experts.114.down_proj", + "model.layers.3.mlp.experts.115.gate_proj", + "model.layers.3.mlp.experts.115.up_proj", + "model.layers.3.mlp.experts.115.down_proj", + "model.layers.3.mlp.experts.116.gate_proj", + "model.layers.3.mlp.experts.116.up_proj", + "model.layers.3.mlp.experts.116.down_proj", + "model.layers.3.mlp.experts.117.gate_proj", + "model.layers.3.mlp.experts.117.up_proj", + "model.layers.3.mlp.experts.117.down_proj", + "model.layers.3.mlp.experts.118.gate_proj", + "model.layers.3.mlp.experts.118.up_proj", + "model.layers.3.mlp.experts.118.down_proj", + "model.layers.3.mlp.experts.119.gate_proj", + "model.layers.3.mlp.experts.119.up_proj", + "model.layers.3.mlp.experts.119.down_proj", + "model.layers.3.mlp.experts.120.gate_proj", + "model.layers.3.mlp.experts.120.up_proj", + "model.layers.3.mlp.experts.120.down_proj", + "model.layers.3.mlp.experts.121.gate_proj", + "model.layers.3.mlp.experts.121.up_proj", + "model.layers.3.mlp.experts.121.down_proj", + "model.layers.3.mlp.experts.122.gate_proj", + "model.layers.3.mlp.experts.122.up_proj", + "model.layers.3.mlp.experts.122.down_proj", + "model.layers.3.mlp.experts.123.gate_proj", + "model.layers.3.mlp.experts.123.up_proj", + "model.layers.3.mlp.experts.123.down_proj", + "model.layers.3.mlp.experts.124.gate_proj", + "model.layers.3.mlp.experts.124.up_proj", + "model.layers.3.mlp.experts.124.down_proj", + "model.layers.3.mlp.experts.125.gate_proj", + "model.layers.3.mlp.experts.125.up_proj", + "model.layers.3.mlp.experts.125.down_proj", + "model.layers.3.mlp.experts.126.gate_proj", + "model.layers.3.mlp.experts.126.up_proj", + "model.layers.3.mlp.experts.126.down_proj", + "model.layers.3.mlp.experts.127.gate_proj", + "model.layers.3.mlp.experts.127.up_proj", + "model.layers.3.mlp.experts.127.down_proj", + "model.layers.3.mlp.experts.128.gate_proj", + "model.layers.3.mlp.experts.128.up_proj", + "model.layers.3.mlp.experts.128.down_proj", + "model.layers.3.mlp.experts.129.gate_proj", + "model.layers.3.mlp.experts.129.up_proj", + "model.layers.3.mlp.experts.129.down_proj", + "model.layers.3.mlp.experts.130.gate_proj", + "model.layers.3.mlp.experts.130.up_proj", + "model.layers.3.mlp.experts.130.down_proj", + "model.layers.3.mlp.experts.131.gate_proj", + "model.layers.3.mlp.experts.131.up_proj", + "model.layers.3.mlp.experts.131.down_proj", + "model.layers.3.mlp.experts.132.gate_proj", + "model.layers.3.mlp.experts.132.up_proj", + "model.layers.3.mlp.experts.132.down_proj", + "model.layers.3.mlp.experts.133.gate_proj", + "model.layers.3.mlp.experts.133.up_proj", + "model.layers.3.mlp.experts.133.down_proj", + "model.layers.3.mlp.experts.134.gate_proj", + "model.layers.3.mlp.experts.134.up_proj", + "model.layers.3.mlp.experts.134.down_proj", + "model.layers.3.mlp.experts.135.gate_proj", + "model.layers.3.mlp.experts.135.up_proj", + "model.layers.3.mlp.experts.135.down_proj", + "model.layers.3.mlp.experts.136.gate_proj", + "model.layers.3.mlp.experts.136.up_proj", + "model.layers.3.mlp.experts.136.down_proj", + "model.layers.3.mlp.experts.137.gate_proj", + "model.layers.3.mlp.experts.137.up_proj", + "model.layers.3.mlp.experts.137.down_proj", + "model.layers.3.mlp.experts.138.gate_proj", + "model.layers.3.mlp.experts.138.up_proj", + "model.layers.3.mlp.experts.138.down_proj", + "model.layers.3.mlp.experts.139.gate_proj", + "model.layers.3.mlp.experts.139.up_proj", + "model.layers.3.mlp.experts.139.down_proj", + "model.layers.3.mlp.experts.140.gate_proj", + "model.layers.3.mlp.experts.140.up_proj", + "model.layers.3.mlp.experts.140.down_proj", + "model.layers.3.mlp.experts.141.gate_proj", + "model.layers.3.mlp.experts.141.up_proj", + "model.layers.3.mlp.experts.141.down_proj", + "model.layers.3.mlp.experts.142.gate_proj", + "model.layers.3.mlp.experts.142.up_proj", + "model.layers.3.mlp.experts.142.down_proj", + "model.layers.3.mlp.experts.143.gate_proj", + "model.layers.3.mlp.experts.143.up_proj", + "model.layers.3.mlp.experts.143.down_proj", + "model.layers.3.mlp.experts.144.gate_proj", + "model.layers.3.mlp.experts.144.up_proj", + "model.layers.3.mlp.experts.144.down_proj", + "model.layers.3.mlp.experts.145.gate_proj", + "model.layers.3.mlp.experts.145.up_proj", + "model.layers.3.mlp.experts.145.down_proj", + "model.layers.3.mlp.experts.146.gate_proj", + "model.layers.3.mlp.experts.146.up_proj", + "model.layers.3.mlp.experts.146.down_proj", + "model.layers.3.mlp.experts.147.gate_proj", + "model.layers.3.mlp.experts.147.up_proj", + "model.layers.3.mlp.experts.147.down_proj", + "model.layers.3.mlp.experts.148.gate_proj", + "model.layers.3.mlp.experts.148.up_proj", + "model.layers.3.mlp.experts.148.down_proj", + "model.layers.3.mlp.experts.149.gate_proj", + "model.layers.3.mlp.experts.149.up_proj", + "model.layers.3.mlp.experts.149.down_proj", + "model.layers.3.mlp.experts.150.gate_proj", + "model.layers.3.mlp.experts.150.up_proj", + "model.layers.3.mlp.experts.150.down_proj", + "model.layers.3.mlp.experts.151.gate_proj", + "model.layers.3.mlp.experts.151.up_proj", + "model.layers.3.mlp.experts.151.down_proj", + "model.layers.3.mlp.experts.152.gate_proj", + "model.layers.3.mlp.experts.152.up_proj", + "model.layers.3.mlp.experts.152.down_proj", + "model.layers.3.mlp.experts.153.gate_proj", + "model.layers.3.mlp.experts.153.up_proj", + "model.layers.3.mlp.experts.153.down_proj", + "model.layers.3.mlp.experts.154.gate_proj", + "model.layers.3.mlp.experts.154.up_proj", + "model.layers.3.mlp.experts.154.down_proj", + "model.layers.3.mlp.experts.155.gate_proj", + "model.layers.3.mlp.experts.155.up_proj", + "model.layers.3.mlp.experts.155.down_proj", + "model.layers.3.mlp.experts.156.gate_proj", + "model.layers.3.mlp.experts.156.up_proj", + "model.layers.3.mlp.experts.156.down_proj", + "model.layers.3.mlp.experts.157.gate_proj", + "model.layers.3.mlp.experts.157.up_proj", + "model.layers.3.mlp.experts.157.down_proj", + "model.layers.3.mlp.experts.158.gate_proj", + "model.layers.3.mlp.experts.158.up_proj", + "model.layers.3.mlp.experts.158.down_proj", + "model.layers.3.mlp.experts.159.gate_proj", + "model.layers.3.mlp.experts.159.up_proj", + "model.layers.3.mlp.experts.159.down_proj", + "model.layers.3.mlp.experts.160.gate_proj", + "model.layers.3.mlp.experts.160.up_proj", + "model.layers.3.mlp.experts.160.down_proj", + "model.layers.3.mlp.experts.161.gate_proj", + "model.layers.3.mlp.experts.161.up_proj", + "model.layers.3.mlp.experts.161.down_proj", + "model.layers.3.mlp.experts.162.gate_proj", + "model.layers.3.mlp.experts.162.up_proj", + "model.layers.3.mlp.experts.162.down_proj", + "model.layers.3.mlp.experts.163.gate_proj", + "model.layers.3.mlp.experts.163.up_proj", + "model.layers.3.mlp.experts.163.down_proj", + "model.layers.3.mlp.experts.164.gate_proj", + "model.layers.3.mlp.experts.164.up_proj", + "model.layers.3.mlp.experts.164.down_proj", + "model.layers.3.mlp.experts.165.gate_proj", + "model.layers.3.mlp.experts.165.up_proj", + "model.layers.3.mlp.experts.165.down_proj", + "model.layers.3.mlp.experts.166.gate_proj", + "model.layers.3.mlp.experts.166.up_proj", + "model.layers.3.mlp.experts.166.down_proj", + "model.layers.3.mlp.experts.167.gate_proj", + "model.layers.3.mlp.experts.167.up_proj", + "model.layers.3.mlp.experts.167.down_proj", + "model.layers.3.mlp.experts.168.gate_proj", + "model.layers.3.mlp.experts.168.up_proj", + "model.layers.3.mlp.experts.168.down_proj", + "model.layers.3.mlp.experts.169.gate_proj", + "model.layers.3.mlp.experts.169.up_proj", + "model.layers.3.mlp.experts.169.down_proj", + "model.layers.3.mlp.experts.170.gate_proj", + "model.layers.3.mlp.experts.170.up_proj", + "model.layers.3.mlp.experts.170.down_proj", + "model.layers.3.mlp.experts.171.gate_proj", + "model.layers.3.mlp.experts.171.up_proj", + "model.layers.3.mlp.experts.171.down_proj", + "model.layers.3.mlp.experts.172.gate_proj", + "model.layers.3.mlp.experts.172.up_proj", + "model.layers.3.mlp.experts.172.down_proj", + "model.layers.3.mlp.experts.173.gate_proj", + "model.layers.3.mlp.experts.173.up_proj", + "model.layers.3.mlp.experts.173.down_proj", + "model.layers.3.mlp.experts.174.gate_proj", + "model.layers.3.mlp.experts.174.up_proj", + "model.layers.3.mlp.experts.174.down_proj", + "model.layers.3.mlp.experts.175.gate_proj", + "model.layers.3.mlp.experts.175.up_proj", + "model.layers.3.mlp.experts.175.down_proj", + "model.layers.3.mlp.experts.176.gate_proj", + "model.layers.3.mlp.experts.176.up_proj", + "model.layers.3.mlp.experts.176.down_proj", + "model.layers.3.mlp.experts.177.gate_proj", + "model.layers.3.mlp.experts.177.up_proj", + "model.layers.3.mlp.experts.177.down_proj", + "model.layers.3.mlp.experts.178.gate_proj", + "model.layers.3.mlp.experts.178.up_proj", + "model.layers.3.mlp.experts.178.down_proj", + "model.layers.3.mlp.experts.179.gate_proj", + "model.layers.3.mlp.experts.179.up_proj", + "model.layers.3.mlp.experts.179.down_proj", + "model.layers.3.mlp.experts.180.gate_proj", + "model.layers.3.mlp.experts.180.up_proj", + "model.layers.3.mlp.experts.180.down_proj", + "model.layers.3.mlp.experts.181.gate_proj", + "model.layers.3.mlp.experts.181.up_proj", + "model.layers.3.mlp.experts.181.down_proj", + "model.layers.3.mlp.experts.182.gate_proj", + "model.layers.3.mlp.experts.182.up_proj", + "model.layers.3.mlp.experts.182.down_proj", + "model.layers.3.mlp.experts.183.gate_proj", + "model.layers.3.mlp.experts.183.up_proj", + "model.layers.3.mlp.experts.183.down_proj", + "model.layers.3.mlp.experts.184.gate_proj", + "model.layers.3.mlp.experts.184.up_proj", + "model.layers.3.mlp.experts.184.down_proj", + "model.layers.3.mlp.experts.185.gate_proj", + "model.layers.3.mlp.experts.185.up_proj", + "model.layers.3.mlp.experts.185.down_proj", + "model.layers.3.mlp.experts.186.gate_proj", + "model.layers.3.mlp.experts.186.up_proj", + "model.layers.3.mlp.experts.186.down_proj", + "model.layers.3.mlp.experts.187.gate_proj", + "model.layers.3.mlp.experts.187.up_proj", + "model.layers.3.mlp.experts.187.down_proj", + "model.layers.3.mlp.experts.188.gate_proj", + "model.layers.3.mlp.experts.188.up_proj", + "model.layers.3.mlp.experts.188.down_proj", + "model.layers.3.mlp.experts.189.gate_proj", + "model.layers.3.mlp.experts.189.up_proj", + "model.layers.3.mlp.experts.189.down_proj", + "model.layers.3.mlp.experts.190.gate_proj", + "model.layers.3.mlp.experts.190.up_proj", + "model.layers.3.mlp.experts.190.down_proj", + "model.layers.3.mlp.experts.191.gate_proj", + "model.layers.3.mlp.experts.191.up_proj", + "model.layers.3.mlp.experts.191.down_proj", + "model.layers.3.mlp.experts.192.gate_proj", + "model.layers.3.mlp.experts.192.up_proj", + "model.layers.3.mlp.experts.192.down_proj", + "model.layers.3.mlp.experts.193.gate_proj", + "model.layers.3.mlp.experts.193.up_proj", + "model.layers.3.mlp.experts.193.down_proj", + "model.layers.3.mlp.experts.194.gate_proj", + "model.layers.3.mlp.experts.194.up_proj", + "model.layers.3.mlp.experts.194.down_proj", + "model.layers.3.mlp.experts.195.gate_proj", + "model.layers.3.mlp.experts.195.up_proj", + "model.layers.3.mlp.experts.195.down_proj", + "model.layers.3.mlp.experts.196.gate_proj", + "model.layers.3.mlp.experts.196.up_proj", + "model.layers.3.mlp.experts.196.down_proj", + "model.layers.3.mlp.experts.197.gate_proj", + "model.layers.3.mlp.experts.197.up_proj", + "model.layers.3.mlp.experts.197.down_proj", + "model.layers.3.mlp.experts.198.gate_proj", + "model.layers.3.mlp.experts.198.up_proj", + "model.layers.3.mlp.experts.198.down_proj", + "model.layers.3.mlp.experts.199.gate_proj", + "model.layers.3.mlp.experts.199.up_proj", + "model.layers.3.mlp.experts.199.down_proj", + "model.layers.3.mlp.experts.200.gate_proj", + "model.layers.3.mlp.experts.200.up_proj", + "model.layers.3.mlp.experts.200.down_proj", + "model.layers.3.mlp.experts.201.gate_proj", + "model.layers.3.mlp.experts.201.up_proj", + "model.layers.3.mlp.experts.201.down_proj", + "model.layers.3.mlp.experts.202.gate_proj", + "model.layers.3.mlp.experts.202.up_proj", + "model.layers.3.mlp.experts.202.down_proj", + "model.layers.3.mlp.experts.203.gate_proj", + "model.layers.3.mlp.experts.203.up_proj", + "model.layers.3.mlp.experts.203.down_proj", + "model.layers.3.mlp.experts.204.gate_proj", + "model.layers.3.mlp.experts.204.up_proj", + "model.layers.3.mlp.experts.204.down_proj", + "model.layers.3.mlp.experts.205.gate_proj", + "model.layers.3.mlp.experts.205.up_proj", + "model.layers.3.mlp.experts.205.down_proj", + "model.layers.3.mlp.experts.206.gate_proj", + "model.layers.3.mlp.experts.206.up_proj", + "model.layers.3.mlp.experts.206.down_proj", + "model.layers.3.mlp.experts.207.gate_proj", + "model.layers.3.mlp.experts.207.up_proj", + "model.layers.3.mlp.experts.207.down_proj", + "model.layers.3.mlp.experts.208.gate_proj", + "model.layers.3.mlp.experts.208.up_proj", + "model.layers.3.mlp.experts.208.down_proj", + "model.layers.3.mlp.experts.209.gate_proj", + "model.layers.3.mlp.experts.209.up_proj", + "model.layers.3.mlp.experts.209.down_proj", + "model.layers.3.mlp.experts.210.gate_proj", + "model.layers.3.mlp.experts.210.up_proj", + "model.layers.3.mlp.experts.210.down_proj", + "model.layers.3.mlp.experts.211.gate_proj", + "model.layers.3.mlp.experts.211.up_proj", + "model.layers.3.mlp.experts.211.down_proj", + "model.layers.3.mlp.experts.212.gate_proj", + "model.layers.3.mlp.experts.212.up_proj", + "model.layers.3.mlp.experts.212.down_proj", + "model.layers.3.mlp.experts.213.gate_proj", + "model.layers.3.mlp.experts.213.up_proj", + "model.layers.3.mlp.experts.213.down_proj", + "model.layers.3.mlp.experts.214.gate_proj", + "model.layers.3.mlp.experts.214.up_proj", + "model.layers.3.mlp.experts.214.down_proj", + "model.layers.3.mlp.experts.215.gate_proj", + "model.layers.3.mlp.experts.215.up_proj", + "model.layers.3.mlp.experts.215.down_proj", + "model.layers.3.mlp.experts.216.gate_proj", + "model.layers.3.mlp.experts.216.up_proj", + "model.layers.3.mlp.experts.216.down_proj", + "model.layers.3.mlp.experts.217.gate_proj", + "model.layers.3.mlp.experts.217.up_proj", + "model.layers.3.mlp.experts.217.down_proj", + "model.layers.3.mlp.experts.218.gate_proj", + "model.layers.3.mlp.experts.218.up_proj", + "model.layers.3.mlp.experts.218.down_proj", + "model.layers.3.mlp.experts.219.gate_proj", + "model.layers.3.mlp.experts.219.up_proj", + "model.layers.3.mlp.experts.219.down_proj", + "model.layers.3.mlp.experts.220.gate_proj", + "model.layers.3.mlp.experts.220.up_proj", + "model.layers.3.mlp.experts.220.down_proj", + "model.layers.3.mlp.experts.221.gate_proj", + "model.layers.3.mlp.experts.221.up_proj", + "model.layers.3.mlp.experts.221.down_proj", + "model.layers.3.mlp.experts.222.gate_proj", + "model.layers.3.mlp.experts.222.up_proj", + "model.layers.3.mlp.experts.222.down_proj", + "model.layers.3.mlp.experts.223.gate_proj", + "model.layers.3.mlp.experts.223.up_proj", + "model.layers.3.mlp.experts.223.down_proj", + "model.layers.3.mlp.experts.224.gate_proj", + "model.layers.3.mlp.experts.224.up_proj", + "model.layers.3.mlp.experts.224.down_proj", + "model.layers.3.mlp.experts.225.gate_proj", + "model.layers.3.mlp.experts.225.up_proj", + "model.layers.3.mlp.experts.225.down_proj", + "model.layers.3.mlp.experts.226.gate_proj", + "model.layers.3.mlp.experts.226.up_proj", + "model.layers.3.mlp.experts.226.down_proj", + "model.layers.3.mlp.experts.227.gate_proj", + "model.layers.3.mlp.experts.227.up_proj", + "model.layers.3.mlp.experts.227.down_proj", + "model.layers.3.mlp.experts.228.gate_proj", + "model.layers.3.mlp.experts.228.up_proj", + "model.layers.3.mlp.experts.228.down_proj", + "model.layers.3.mlp.experts.229.gate_proj", + "model.layers.3.mlp.experts.229.up_proj", + "model.layers.3.mlp.experts.229.down_proj", + "model.layers.3.mlp.experts.230.gate_proj", + "model.layers.3.mlp.experts.230.up_proj", + "model.layers.3.mlp.experts.230.down_proj", + "model.layers.3.mlp.experts.231.gate_proj", + "model.layers.3.mlp.experts.231.up_proj", + "model.layers.3.mlp.experts.231.down_proj", + "model.layers.3.mlp.experts.232.gate_proj", + "model.layers.3.mlp.experts.232.up_proj", + "model.layers.3.mlp.experts.232.down_proj", + "model.layers.3.mlp.experts.233.gate_proj", + "model.layers.3.mlp.experts.233.up_proj", + "model.layers.3.mlp.experts.233.down_proj", + "model.layers.3.mlp.experts.234.gate_proj", + "model.layers.3.mlp.experts.234.up_proj", + "model.layers.3.mlp.experts.234.down_proj", + "model.layers.3.mlp.experts.235.gate_proj", + "model.layers.3.mlp.experts.235.up_proj", + "model.layers.3.mlp.experts.235.down_proj", + "model.layers.3.mlp.experts.236.gate_proj", + "model.layers.3.mlp.experts.236.up_proj", + "model.layers.3.mlp.experts.236.down_proj", + "model.layers.3.mlp.experts.237.gate_proj", + "model.layers.3.mlp.experts.237.up_proj", + "model.layers.3.mlp.experts.237.down_proj", + "model.layers.3.mlp.experts.238.gate_proj", + "model.layers.3.mlp.experts.238.up_proj", + "model.layers.3.mlp.experts.238.down_proj", + "model.layers.3.mlp.experts.239.gate_proj", + "model.layers.3.mlp.experts.239.up_proj", + "model.layers.3.mlp.experts.239.down_proj", + "model.layers.3.mlp.experts.240.gate_proj", + "model.layers.3.mlp.experts.240.up_proj", + "model.layers.3.mlp.experts.240.down_proj", + "model.layers.3.mlp.experts.241.gate_proj", + "model.layers.3.mlp.experts.241.up_proj", + "model.layers.3.mlp.experts.241.down_proj", + "model.layers.3.mlp.experts.242.gate_proj", + "model.layers.3.mlp.experts.242.up_proj", + "model.layers.3.mlp.experts.242.down_proj", + "model.layers.3.mlp.experts.243.gate_proj", + "model.layers.3.mlp.experts.243.up_proj", + "model.layers.3.mlp.experts.243.down_proj", + "model.layers.3.mlp.experts.244.gate_proj", + "model.layers.3.mlp.experts.244.up_proj", + "model.layers.3.mlp.experts.244.down_proj", + "model.layers.3.mlp.experts.245.gate_proj", + "model.layers.3.mlp.experts.245.up_proj", + "model.layers.3.mlp.experts.245.down_proj", + "model.layers.3.mlp.experts.246.gate_proj", + "model.layers.3.mlp.experts.246.up_proj", + "model.layers.3.mlp.experts.246.down_proj", + "model.layers.3.mlp.experts.247.gate_proj", + "model.layers.3.mlp.experts.247.up_proj", + "model.layers.3.mlp.experts.247.down_proj", + "model.layers.3.mlp.experts.248.gate_proj", + "model.layers.3.mlp.experts.248.up_proj", + "model.layers.3.mlp.experts.248.down_proj", + "model.layers.3.mlp.experts.249.gate_proj", + "model.layers.3.mlp.experts.249.up_proj", + "model.layers.3.mlp.experts.249.down_proj", + "model.layers.3.mlp.experts.250.gate_proj", + "model.layers.3.mlp.experts.250.up_proj", + "model.layers.3.mlp.experts.250.down_proj", + "model.layers.3.mlp.experts.251.gate_proj", + "model.layers.3.mlp.experts.251.up_proj", + "model.layers.3.mlp.experts.251.down_proj", + "model.layers.3.mlp.experts.252.gate_proj", + "model.layers.3.mlp.experts.252.up_proj", + "model.layers.3.mlp.experts.252.down_proj", + "model.layers.3.mlp.experts.253.gate_proj", + "model.layers.3.mlp.experts.253.up_proj", + "model.layers.3.mlp.experts.253.down_proj", + "model.layers.3.mlp.experts.254.gate_proj", + "model.layers.3.mlp.experts.254.up_proj", + "model.layers.3.mlp.experts.254.down_proj", + "model.layers.3.mlp.experts.255.gate_proj", + "model.layers.3.mlp.experts.255.up_proj", + "model.layers.3.mlp.experts.255.down_proj", + "model.layers.3.mlp.shared_experts.gate_proj", + "model.layers.3.mlp.shared_experts.up_proj", + "model.layers.3.mlp.shared_experts.down_proj", + "model.layers.4.self_attn.q_a_proj", + "model.layers.4.self_attn.kv_a_proj_with_mqa", + "model.layers.4.mlp.gate", + "model.layers.4.mlp.shared_experts.gate_proj", + "model.layers.4.mlp.shared_experts.up_proj", + "model.layers.4.mlp.shared_experts.down_proj", + "model.layers.5.self_attn.q_a_proj", + "model.layers.5.self_attn.kv_a_proj_with_mqa", + "model.layers.5.mlp.gate", + "model.layers.5.mlp.shared_experts.gate_proj", + "model.layers.5.mlp.shared_experts.up_proj", + "model.layers.5.mlp.shared_experts.down_proj", + "model.layers.6.self_attn.q_a_proj", + "model.layers.6.self_attn.kv_a_proj_with_mqa", + "model.layers.6.self_attn.indexer.wq_b", + "model.layers.6.self_attn.indexer.wk", + "model.layers.6.self_attn.indexer.weights_proj", + "model.layers.6.mlp.gate", + "model.layers.6.mlp.shared_experts.gate_proj", + "model.layers.6.mlp.shared_experts.up_proj", + "model.layers.6.mlp.shared_experts.down_proj", + "model.layers.7.self_attn.q_a_proj", + "model.layers.7.self_attn.kv_a_proj_with_mqa", + "model.layers.7.mlp.gate", + "model.layers.7.mlp.shared_experts.gate_proj", + "model.layers.7.mlp.shared_experts.up_proj", + "model.layers.7.mlp.shared_experts.down_proj", + "model.layers.8.self_attn.q_a_proj", + "model.layers.8.self_attn.kv_a_proj_with_mqa", + "model.layers.8.mlp.gate", + "model.layers.8.mlp.shared_experts.gate_proj", + "model.layers.8.mlp.shared_experts.up_proj", + "model.layers.8.mlp.shared_experts.down_proj", + "model.layers.9.self_attn.q_a_proj", + "model.layers.9.self_attn.kv_a_proj_with_mqa", + "model.layers.9.mlp.gate", + "model.layers.9.mlp.shared_experts.gate_proj", + "model.layers.9.mlp.shared_experts.up_proj", + "model.layers.9.mlp.shared_experts.down_proj", + "model.layers.10.self_attn.q_a_proj", + "model.layers.10.self_attn.kv_a_proj_with_mqa", + "model.layers.10.self_attn.indexer.wq_b", + "model.layers.10.self_attn.indexer.wk", + "model.layers.10.self_attn.indexer.weights_proj", + "model.layers.10.mlp.gate", + "model.layers.10.mlp.shared_experts.gate_proj", + "model.layers.10.mlp.shared_experts.up_proj", + "model.layers.10.mlp.shared_experts.down_proj", + "model.layers.11.self_attn.q_a_proj", + "model.layers.11.self_attn.kv_a_proj_with_mqa", + "model.layers.11.mlp.gate", + "model.layers.11.mlp.shared_experts.gate_proj", + "model.layers.11.mlp.shared_experts.up_proj", + "model.layers.11.mlp.shared_experts.down_proj", + "model.layers.12.self_attn.q_a_proj", + "model.layers.12.self_attn.kv_a_proj_with_mqa", + "model.layers.12.mlp.gate", + "model.layers.12.mlp.shared_experts.gate_proj", + "model.layers.12.mlp.shared_experts.up_proj", + "model.layers.12.mlp.shared_experts.down_proj", + "model.layers.13.self_attn.q_a_proj", + "model.layers.13.self_attn.kv_a_proj_with_mqa", + "model.layers.13.mlp.gate", + "model.layers.13.mlp.shared_experts.gate_proj", + "model.layers.13.mlp.shared_experts.up_proj", + "model.layers.13.mlp.shared_experts.down_proj", + "model.layers.14.self_attn.q_a_proj", + "model.layers.14.self_attn.kv_a_proj_with_mqa", + "model.layers.14.self_attn.indexer.wq_b", + "model.layers.14.self_attn.indexer.wk", + "model.layers.14.self_attn.indexer.weights_proj", + "model.layers.14.mlp.gate", + "model.layers.14.mlp.shared_experts.gate_proj", + "model.layers.14.mlp.shared_experts.up_proj", + "model.layers.14.mlp.shared_experts.down_proj", + "model.layers.15.self_attn.q_a_proj", + "model.layers.15.self_attn.kv_a_proj_with_mqa", + "model.layers.15.mlp.gate", + "model.layers.15.mlp.shared_experts.gate_proj", + "model.layers.15.mlp.shared_experts.up_proj", + "model.layers.15.mlp.shared_experts.down_proj", + "model.layers.16.self_attn.q_a_proj", + "model.layers.16.self_attn.kv_a_proj_with_mqa", + "model.layers.16.mlp.gate", + "model.layers.16.mlp.shared_experts.gate_proj", + "model.layers.16.mlp.shared_experts.up_proj", + "model.layers.16.mlp.shared_experts.down_proj", + "model.layers.17.self_attn.q_a_proj", + "model.layers.17.self_attn.kv_a_proj_with_mqa", + "model.layers.17.mlp.gate", + "model.layers.17.mlp.shared_experts.gate_proj", + "model.layers.17.mlp.shared_experts.up_proj", + "model.layers.17.mlp.shared_experts.down_proj", + "model.layers.18.self_attn.q_a_proj", + "model.layers.18.self_attn.kv_a_proj_with_mqa", + "model.layers.18.self_attn.indexer.wq_b", + "model.layers.18.self_attn.indexer.wk", + "model.layers.18.self_attn.indexer.weights_proj", + "model.layers.18.mlp.gate", + "model.layers.18.mlp.shared_experts.gate_proj", + "model.layers.18.mlp.shared_experts.up_proj", + "model.layers.18.mlp.shared_experts.down_proj", + "model.layers.19.self_attn.q_a_proj", + "model.layers.19.self_attn.kv_a_proj_with_mqa", + "model.layers.19.mlp.gate", + "model.layers.19.mlp.shared_experts.gate_proj", + "model.layers.19.mlp.shared_experts.up_proj", + "model.layers.19.mlp.shared_experts.down_proj", + "model.layers.20.self_attn.q_a_proj", + "model.layers.20.self_attn.kv_a_proj_with_mqa", + "model.layers.20.mlp.gate", + "model.layers.20.mlp.shared_experts.gate_proj", + "model.layers.20.mlp.shared_experts.up_proj", + "model.layers.20.mlp.shared_experts.down_proj", + "model.layers.21.self_attn.q_a_proj", + "model.layers.21.self_attn.kv_a_proj_with_mqa", + "model.layers.21.mlp.gate", + "model.layers.21.mlp.shared_experts.gate_proj", + "model.layers.21.mlp.shared_experts.up_proj", + "model.layers.21.mlp.shared_experts.down_proj", + "model.layers.22.self_attn.q_a_proj", + "model.layers.22.self_attn.kv_a_proj_with_mqa", + "model.layers.22.self_attn.indexer.wq_b", + "model.layers.22.self_attn.indexer.wk", + "model.layers.22.self_attn.indexer.weights_proj", + "model.layers.22.mlp.gate", + "model.layers.22.mlp.shared_experts.gate_proj", + "model.layers.22.mlp.shared_experts.up_proj", + "model.layers.22.mlp.shared_experts.down_proj", + "model.layers.23.self_attn.q_a_proj", + "model.layers.23.self_attn.kv_a_proj_with_mqa", + "model.layers.23.mlp.gate", + "model.layers.23.mlp.shared_experts.gate_proj", + "model.layers.23.mlp.shared_experts.up_proj", + "model.layers.23.mlp.shared_experts.down_proj", + "model.layers.24.self_attn.q_a_proj", + "model.layers.24.self_attn.kv_a_proj_with_mqa", + "model.layers.24.mlp.gate", + "model.layers.24.mlp.shared_experts.gate_proj", + "model.layers.24.mlp.shared_experts.up_proj", + "model.layers.24.mlp.shared_experts.down_proj", + "model.layers.25.self_attn.q_a_proj", + "model.layers.25.self_attn.kv_a_proj_with_mqa", + "model.layers.25.mlp.gate", + "model.layers.25.mlp.shared_experts.gate_proj", + "model.layers.25.mlp.shared_experts.up_proj", + "model.layers.25.mlp.shared_experts.down_proj", + "model.layers.26.self_attn.q_a_proj", + "model.layers.26.self_attn.kv_a_proj_with_mqa", + "model.layers.26.self_attn.indexer.wq_b", + "model.layers.26.self_attn.indexer.wk", + "model.layers.26.self_attn.indexer.weights_proj", + "model.layers.26.mlp.gate", + "model.layers.26.mlp.shared_experts.gate_proj", + "model.layers.26.mlp.shared_experts.up_proj", + "model.layers.26.mlp.shared_experts.down_proj", + "model.layers.27.self_attn.q_a_proj", + "model.layers.27.self_attn.kv_a_proj_with_mqa", + "model.layers.27.mlp.gate", + "model.layers.27.mlp.shared_experts.gate_proj", + "model.layers.27.mlp.shared_experts.up_proj", + "model.layers.27.mlp.shared_experts.down_proj", + "model.layers.28.self_attn.q_a_proj", + "model.layers.28.self_attn.kv_a_proj_with_mqa", + "model.layers.28.mlp.gate", + "model.layers.28.mlp.shared_experts.gate_proj", + "model.layers.28.mlp.shared_experts.up_proj", + "model.layers.28.mlp.shared_experts.down_proj", + "model.layers.29.self_attn.q_a_proj", + "model.layers.29.self_attn.kv_a_proj_with_mqa", + "model.layers.29.mlp.gate", + "model.layers.29.mlp.shared_experts.gate_proj", + "model.layers.29.mlp.shared_experts.up_proj", + "model.layers.29.mlp.shared_experts.down_proj", + "model.layers.30.self_attn.q_a_proj", + "model.layers.30.self_attn.kv_a_proj_with_mqa", + "model.layers.30.self_attn.indexer.wq_b", + "model.layers.30.self_attn.indexer.wk", + "model.layers.30.self_attn.indexer.weights_proj", + "model.layers.30.mlp.gate", + "model.layers.30.mlp.shared_experts.gate_proj", + "model.layers.30.mlp.shared_experts.up_proj", + "model.layers.30.mlp.shared_experts.down_proj", + "model.layers.31.self_attn.q_a_proj", + "model.layers.31.self_attn.kv_a_proj_with_mqa", + "model.layers.31.mlp.gate", + "model.layers.31.mlp.shared_experts.gate_proj", + "model.layers.31.mlp.shared_experts.up_proj", + "model.layers.31.mlp.shared_experts.down_proj", + "model.layers.32.self_attn.q_a_proj", + "model.layers.32.self_attn.kv_a_proj_with_mqa", + "model.layers.32.mlp.gate", + "model.layers.32.mlp.shared_experts.gate_proj", + "model.layers.32.mlp.shared_experts.up_proj", + "model.layers.32.mlp.shared_experts.down_proj", + "model.layers.33.self_attn.q_a_proj", + "model.layers.33.self_attn.kv_a_proj_with_mqa", + "model.layers.33.mlp.gate", + "model.layers.33.mlp.shared_experts.gate_proj", + "model.layers.33.mlp.shared_experts.up_proj", + "model.layers.33.mlp.shared_experts.down_proj", + "model.layers.34.self_attn.q_a_proj", + "model.layers.34.self_attn.kv_a_proj_with_mqa", + "model.layers.34.self_attn.indexer.wq_b", + "model.layers.34.self_attn.indexer.wk", + "model.layers.34.self_attn.indexer.weights_proj", + "model.layers.34.mlp.gate", + "model.layers.34.mlp.shared_experts.gate_proj", + "model.layers.34.mlp.shared_experts.up_proj", + "model.layers.34.mlp.shared_experts.down_proj", + "model.layers.35.self_attn.q_a_proj", + "model.layers.35.self_attn.kv_a_proj_with_mqa", + "model.layers.35.mlp.gate", + "model.layers.35.mlp.shared_experts.gate_proj", + "model.layers.35.mlp.shared_experts.up_proj", + "model.layers.35.mlp.shared_experts.down_proj", + "model.layers.36.self_attn.q_a_proj", + "model.layers.36.self_attn.kv_a_proj_with_mqa", + "model.layers.36.mlp.gate", + "model.layers.36.mlp.shared_experts.gate_proj", + "model.layers.36.mlp.shared_experts.up_proj", + "model.layers.36.mlp.shared_experts.down_proj", + "model.layers.37.self_attn.q_a_proj", + "model.layers.37.self_attn.kv_a_proj_with_mqa", + "model.layers.37.mlp.gate", + "model.layers.37.mlp.shared_experts.gate_proj", + "model.layers.37.mlp.shared_experts.up_proj", + "model.layers.37.mlp.shared_experts.down_proj", + "model.layers.38.self_attn.q_a_proj", + "model.layers.38.self_attn.kv_a_proj_with_mqa", + "model.layers.38.self_attn.indexer.wq_b", + "model.layers.38.self_attn.indexer.wk", + "model.layers.38.self_attn.indexer.weights_proj", + "model.layers.38.mlp.gate", + "model.layers.38.mlp.shared_experts.gate_proj", + "model.layers.38.mlp.shared_experts.up_proj", + "model.layers.38.mlp.shared_experts.down_proj", + "model.layers.39.self_attn.q_a_proj", + "model.layers.39.self_attn.kv_a_proj_with_mqa", + "model.layers.39.mlp.gate", + "model.layers.39.mlp.shared_experts.gate_proj", + "model.layers.39.mlp.shared_experts.up_proj", + "model.layers.39.mlp.shared_experts.down_proj", + "model.layers.40.self_attn.q_a_proj", + "model.layers.40.self_attn.kv_a_proj_with_mqa", + "model.layers.40.mlp.gate", + "model.layers.40.mlp.shared_experts.gate_proj", + "model.layers.40.mlp.shared_experts.up_proj", + "model.layers.40.mlp.shared_experts.down_proj", + "model.layers.41.self_attn.q_a_proj", + "model.layers.41.self_attn.kv_a_proj_with_mqa", + "model.layers.41.mlp.gate", + "model.layers.41.mlp.shared_experts.gate_proj", + "model.layers.41.mlp.shared_experts.up_proj", + "model.layers.41.mlp.shared_experts.down_proj", + "model.layers.42.self_attn.q_a_proj", + "model.layers.42.self_attn.kv_a_proj_with_mqa", + "model.layers.42.self_attn.indexer.wq_b", + "model.layers.42.self_attn.indexer.wk", + "model.layers.42.self_attn.indexer.weights_proj", + "model.layers.42.mlp.gate", + "model.layers.42.mlp.shared_experts.gate_proj", + "model.layers.42.mlp.shared_experts.up_proj", + "model.layers.42.mlp.shared_experts.down_proj", + "model.layers.43.self_attn.q_a_proj", + "model.layers.43.self_attn.kv_a_proj_with_mqa", + "model.layers.43.mlp.gate", + "model.layers.43.mlp.shared_experts.gate_proj", + "model.layers.43.mlp.shared_experts.up_proj", + "model.layers.43.mlp.shared_experts.down_proj", + "model.layers.44.self_attn.q_a_proj", + "model.layers.44.self_attn.kv_a_proj_with_mqa", + "model.layers.44.mlp.gate", + "model.layers.44.mlp.shared_experts.gate_proj", + "model.layers.44.mlp.shared_experts.up_proj", + "model.layers.44.mlp.shared_experts.down_proj", + "model.layers.45.self_attn.q_a_proj", + "model.layers.45.self_attn.kv_a_proj_with_mqa", + "model.layers.45.mlp.gate", + "model.layers.45.mlp.shared_experts.gate_proj", + "model.layers.45.mlp.shared_experts.up_proj", + "model.layers.45.mlp.shared_experts.down_proj", + "model.layers.46.self_attn.q_a_proj", + "model.layers.46.self_attn.kv_a_proj_with_mqa", + "model.layers.46.self_attn.indexer.wq_b", + "model.layers.46.self_attn.indexer.wk", + "model.layers.46.self_attn.indexer.weights_proj", + "model.layers.46.mlp.gate", + "model.layers.46.mlp.shared_experts.gate_proj", + "model.layers.46.mlp.shared_experts.up_proj", + "model.layers.46.mlp.shared_experts.down_proj", + "model.layers.47.self_attn.q_a_proj", + "model.layers.47.self_attn.kv_a_proj_with_mqa", + "model.layers.47.mlp.gate", + "model.layers.47.mlp.shared_experts.gate_proj", + "model.layers.47.mlp.shared_experts.up_proj", + "model.layers.47.mlp.shared_experts.down_proj", + "model.layers.48.self_attn.q_a_proj", + "model.layers.48.self_attn.kv_a_proj_with_mqa", + "model.layers.48.mlp.gate", + "model.layers.48.mlp.shared_experts.gate_proj", + "model.layers.48.mlp.shared_experts.up_proj", + "model.layers.48.mlp.shared_experts.down_proj", + "model.layers.49.self_attn.q_a_proj", + "model.layers.49.self_attn.kv_a_proj_with_mqa", + "model.layers.49.mlp.gate", + "model.layers.49.mlp.shared_experts.gate_proj", + "model.layers.49.mlp.shared_experts.up_proj", + "model.layers.49.mlp.shared_experts.down_proj", + "model.layers.50.self_attn.q_a_proj", + "model.layers.50.self_attn.kv_a_proj_with_mqa", + "model.layers.50.self_attn.indexer.wq_b", + "model.layers.50.self_attn.indexer.wk", + "model.layers.50.self_attn.indexer.weights_proj", + "model.layers.50.mlp.gate", + "model.layers.50.mlp.shared_experts.gate_proj", + "model.layers.50.mlp.shared_experts.up_proj", + "model.layers.50.mlp.shared_experts.down_proj", + "model.layers.51.self_attn.q_a_proj", + "model.layers.51.self_attn.kv_a_proj_with_mqa", + "model.layers.51.mlp.gate", + "model.layers.51.mlp.shared_experts.gate_proj", + "model.layers.51.mlp.shared_experts.up_proj", + "model.layers.51.mlp.shared_experts.down_proj", + "model.layers.52.self_attn.q_a_proj", + "model.layers.52.self_attn.kv_a_proj_with_mqa", + "model.layers.52.mlp.gate", + "model.layers.52.mlp.shared_experts.gate_proj", + "model.layers.52.mlp.shared_experts.up_proj", + "model.layers.52.mlp.shared_experts.down_proj", + "model.layers.53.self_attn.q_a_proj", + "model.layers.53.self_attn.kv_a_proj_with_mqa", + "model.layers.53.mlp.gate", + "model.layers.53.mlp.shared_experts.gate_proj", + "model.layers.53.mlp.shared_experts.up_proj", + "model.layers.53.mlp.shared_experts.down_proj", + "model.layers.54.self_attn.q_a_proj", + "model.layers.54.self_attn.kv_a_proj_with_mqa", + "model.layers.54.self_attn.indexer.wq_b", + "model.layers.54.self_attn.indexer.wk", + "model.layers.54.self_attn.indexer.weights_proj", + "model.layers.54.mlp.gate", + "model.layers.54.mlp.shared_experts.gate_proj", + "model.layers.54.mlp.shared_experts.up_proj", + "model.layers.54.mlp.shared_experts.down_proj", + "model.layers.55.self_attn.q_a_proj", + "model.layers.55.self_attn.kv_a_proj_with_mqa", + "model.layers.55.mlp.gate", + "model.layers.55.mlp.shared_experts.gate_proj", + "model.layers.55.mlp.shared_experts.up_proj", + "model.layers.55.mlp.shared_experts.down_proj", + "model.layers.56.self_attn.q_a_proj", + "model.layers.56.self_attn.kv_a_proj_with_mqa", + "model.layers.56.mlp.gate", + "model.layers.56.mlp.shared_experts.gate_proj", + "model.layers.56.mlp.shared_experts.up_proj", + "model.layers.56.mlp.shared_experts.down_proj", + "model.layers.57.self_attn.q_a_proj", + "model.layers.57.self_attn.kv_a_proj_with_mqa", + "model.layers.57.mlp.gate", + "model.layers.57.mlp.shared_experts.gate_proj", + "model.layers.57.mlp.shared_experts.up_proj", + "model.layers.57.mlp.shared_experts.down_proj", + "model.layers.58.self_attn.q_a_proj", + "model.layers.58.self_attn.kv_a_proj_with_mqa", + "model.layers.58.self_attn.indexer.wq_b", + "model.layers.58.self_attn.indexer.wk", + "model.layers.58.self_attn.indexer.weights_proj", + "model.layers.58.mlp.gate", + "model.layers.58.mlp.shared_experts.gate_proj", + "model.layers.58.mlp.shared_experts.up_proj", + "model.layers.58.mlp.shared_experts.down_proj", + "model.layers.59.self_attn.q_a_proj", + "model.layers.59.self_attn.kv_a_proj_with_mqa", + "model.layers.59.mlp.gate", + "model.layers.59.mlp.shared_experts.gate_proj", + "model.layers.59.mlp.shared_experts.up_proj", + "model.layers.59.mlp.shared_experts.down_proj", + "model.layers.60.self_attn.q_a_proj", + "model.layers.60.self_attn.kv_a_proj_with_mqa", + "model.layers.60.mlp.gate", + "model.layers.60.mlp.shared_experts.gate_proj", + "model.layers.60.mlp.shared_experts.up_proj", + "model.layers.60.mlp.shared_experts.down_proj", + "model.layers.61.self_attn.q_a_proj", + "model.layers.61.self_attn.kv_a_proj_with_mqa", + "model.layers.61.mlp.gate", + "model.layers.61.mlp.shared_experts.gate_proj", + "model.layers.61.mlp.shared_experts.up_proj", + "model.layers.61.mlp.shared_experts.down_proj", + "model.layers.62.self_attn.q_a_proj", + "model.layers.62.self_attn.kv_a_proj_with_mqa", + "model.layers.62.self_attn.indexer.wq_b", + "model.layers.62.self_attn.indexer.wk", + "model.layers.62.self_attn.indexer.weights_proj", + "model.layers.62.mlp.gate", + "model.layers.62.mlp.shared_experts.gate_proj", + "model.layers.62.mlp.shared_experts.up_proj", + "model.layers.62.mlp.shared_experts.down_proj", + "model.layers.63.self_attn.q_a_proj", + "model.layers.63.self_attn.kv_a_proj_with_mqa", + "model.layers.63.mlp.gate", + "model.layers.63.mlp.shared_experts.gate_proj", + "model.layers.63.mlp.shared_experts.up_proj", + "model.layers.63.mlp.shared_experts.down_proj", + "model.layers.64.self_attn.q_a_proj", + "model.layers.64.self_attn.kv_a_proj_with_mqa", + "model.layers.64.mlp.gate", + "model.layers.64.mlp.shared_experts.gate_proj", + "model.layers.64.mlp.shared_experts.up_proj", + "model.layers.64.mlp.shared_experts.down_proj", + "model.layers.65.self_attn.q_a_proj", + "model.layers.65.self_attn.kv_a_proj_with_mqa", + "model.layers.65.mlp.gate", + "model.layers.65.mlp.shared_experts.gate_proj", + "model.layers.65.mlp.shared_experts.up_proj", + "model.layers.65.mlp.shared_experts.down_proj", + "model.layers.66.self_attn.q_a_proj", + "model.layers.66.self_attn.kv_a_proj_with_mqa", + "model.layers.66.self_attn.indexer.wq_b", + "model.layers.66.self_attn.indexer.wk", + "model.layers.66.self_attn.indexer.weights_proj", + "model.layers.66.mlp.gate", + "model.layers.66.mlp.shared_experts.gate_proj", + "model.layers.66.mlp.shared_experts.up_proj", + "model.layers.66.mlp.shared_experts.down_proj", + "model.layers.67.self_attn.q_a_proj", + "model.layers.67.self_attn.kv_a_proj_with_mqa", + "model.layers.67.mlp.gate", + "model.layers.67.mlp.shared_experts.gate_proj", + "model.layers.67.mlp.shared_experts.up_proj", + "model.layers.67.mlp.shared_experts.down_proj", + "model.layers.68.self_attn.q_a_proj", + "model.layers.68.self_attn.kv_a_proj_with_mqa", + "model.layers.68.mlp.gate", + "model.layers.68.mlp.shared_experts.gate_proj", + "model.layers.68.mlp.shared_experts.up_proj", + "model.layers.68.mlp.shared_experts.down_proj", + "model.layers.69.self_attn.q_a_proj", + "model.layers.69.self_attn.kv_a_proj_with_mqa", + "model.layers.69.mlp.gate", + "model.layers.69.mlp.shared_experts.gate_proj", + "model.layers.69.mlp.shared_experts.up_proj", + "model.layers.69.mlp.shared_experts.down_proj", + "model.layers.70.self_attn.q_a_proj", + "model.layers.70.self_attn.kv_a_proj_with_mqa", + "model.layers.70.self_attn.indexer.wq_b", + "model.layers.70.self_attn.indexer.wk", + "model.layers.70.self_attn.indexer.weights_proj", + "model.layers.70.mlp.gate", + "model.layers.70.mlp.shared_experts.gate_proj", + "model.layers.70.mlp.shared_experts.up_proj", + "model.layers.70.mlp.shared_experts.down_proj", + "model.layers.71.self_attn.q_a_proj", + "model.layers.71.self_attn.kv_a_proj_with_mqa", + "model.layers.71.mlp.gate", + "model.layers.71.mlp.shared_experts.gate_proj", + "model.layers.71.mlp.shared_experts.up_proj", + "model.layers.71.mlp.shared_experts.down_proj", + "model.layers.72.self_attn.q_a_proj", + "model.layers.72.self_attn.kv_a_proj_with_mqa", + "model.layers.72.mlp.gate", + "model.layers.72.mlp.shared_experts.gate_proj", + "model.layers.72.mlp.shared_experts.up_proj", + "model.layers.72.mlp.shared_experts.down_proj", + "model.layers.73.self_attn.q_a_proj", + "model.layers.73.self_attn.kv_a_proj_with_mqa", + "model.layers.73.mlp.gate", + "model.layers.73.mlp.shared_experts.gate_proj", + "model.layers.73.mlp.shared_experts.up_proj", + "model.layers.73.mlp.shared_experts.down_proj", + "model.layers.74.self_attn.q_a_proj", + "model.layers.74.self_attn.kv_a_proj_with_mqa", + "model.layers.74.self_attn.indexer.wq_b", + "model.layers.74.self_attn.indexer.wk", + "model.layers.74.self_attn.indexer.weights_proj", + "model.layers.74.mlp.gate", + "model.layers.74.mlp.shared_experts.gate_proj", + "model.layers.74.mlp.shared_experts.up_proj", + "model.layers.74.mlp.shared_experts.down_proj", + "model.layers.75.self_attn.q_a_proj", + "model.layers.75.self_attn.kv_a_proj_with_mqa", + "model.layers.75.mlp.gate", + "model.layers.75.mlp.shared_experts.gate_proj", + "model.layers.75.mlp.shared_experts.up_proj", + "model.layers.75.mlp.shared_experts.down_proj", + "model.layers.76.self_attn.q_a_proj", + "model.layers.76.self_attn.kv_a_proj_with_mqa", + "model.layers.76.mlp.gate", + "model.layers.76.mlp.shared_experts.gate_proj", + "model.layers.76.mlp.shared_experts.up_proj", + "model.layers.76.mlp.shared_experts.down_proj", + "model.layers.77.self_attn.q_a_proj", + "model.layers.77.self_attn.q_b_proj", + "model.layers.77.self_attn.kv_a_proj_with_mqa", + "model.layers.77.self_attn.kv_b_proj", + "model.layers.77.self_attn.o_proj", + "model.layers.77.mlp.gate", + "model.layers.77.mlp.experts.0.gate_proj", + "model.layers.77.mlp.experts.0.up_proj", + "model.layers.77.mlp.experts.0.down_proj", + "model.layers.77.mlp.experts.1.gate_proj", + "model.layers.77.mlp.experts.1.up_proj", + "model.layers.77.mlp.experts.1.down_proj", + "model.layers.77.mlp.experts.2.gate_proj", + "model.layers.77.mlp.experts.2.up_proj", + "model.layers.77.mlp.experts.2.down_proj", + "model.layers.77.mlp.experts.3.gate_proj", + "model.layers.77.mlp.experts.3.up_proj", + "model.layers.77.mlp.experts.3.down_proj", + "model.layers.77.mlp.experts.4.gate_proj", + "model.layers.77.mlp.experts.4.up_proj", + "model.layers.77.mlp.experts.4.down_proj", + "model.layers.77.mlp.experts.5.gate_proj", + "model.layers.77.mlp.experts.5.up_proj", + "model.layers.77.mlp.experts.5.down_proj", + "model.layers.77.mlp.experts.6.gate_proj", + "model.layers.77.mlp.experts.6.up_proj", + "model.layers.77.mlp.experts.6.down_proj", + "model.layers.77.mlp.experts.7.gate_proj", + "model.layers.77.mlp.experts.7.up_proj", + "model.layers.77.mlp.experts.7.down_proj", + "model.layers.77.mlp.experts.8.gate_proj", + "model.layers.77.mlp.experts.8.up_proj", + "model.layers.77.mlp.experts.8.down_proj", + "model.layers.77.mlp.experts.9.gate_proj", + "model.layers.77.mlp.experts.9.up_proj", + "model.layers.77.mlp.experts.9.down_proj", + "model.layers.77.mlp.experts.10.gate_proj", + "model.layers.77.mlp.experts.10.up_proj", + "model.layers.77.mlp.experts.10.down_proj", + "model.layers.77.mlp.experts.11.gate_proj", + "model.layers.77.mlp.experts.11.up_proj", + "model.layers.77.mlp.experts.11.down_proj", + "model.layers.77.mlp.experts.12.gate_proj", + "model.layers.77.mlp.experts.12.up_proj", + "model.layers.77.mlp.experts.12.down_proj", + "model.layers.77.mlp.experts.13.gate_proj", + "model.layers.77.mlp.experts.13.up_proj", + "model.layers.77.mlp.experts.13.down_proj", + "model.layers.77.mlp.experts.14.gate_proj", + "model.layers.77.mlp.experts.14.up_proj", + "model.layers.77.mlp.experts.14.down_proj", + "model.layers.77.mlp.experts.15.gate_proj", + "model.layers.77.mlp.experts.15.up_proj", + "model.layers.77.mlp.experts.15.down_proj", + "model.layers.77.mlp.experts.16.gate_proj", + "model.layers.77.mlp.experts.16.up_proj", + "model.layers.77.mlp.experts.16.down_proj", + "model.layers.77.mlp.experts.17.gate_proj", + "model.layers.77.mlp.experts.17.up_proj", + "model.layers.77.mlp.experts.17.down_proj", + "model.layers.77.mlp.experts.18.gate_proj", + "model.layers.77.mlp.experts.18.up_proj", + "model.layers.77.mlp.experts.18.down_proj", + "model.layers.77.mlp.experts.19.gate_proj", + "model.layers.77.mlp.experts.19.up_proj", + "model.layers.77.mlp.experts.19.down_proj", + "model.layers.77.mlp.experts.20.gate_proj", + "model.layers.77.mlp.experts.20.up_proj", + "model.layers.77.mlp.experts.20.down_proj", + "model.layers.77.mlp.experts.21.gate_proj", + "model.layers.77.mlp.experts.21.up_proj", + "model.layers.77.mlp.experts.21.down_proj", + "model.layers.77.mlp.experts.22.gate_proj", + "model.layers.77.mlp.experts.22.up_proj", + "model.layers.77.mlp.experts.22.down_proj", + "model.layers.77.mlp.experts.23.gate_proj", + "model.layers.77.mlp.experts.23.up_proj", + "model.layers.77.mlp.experts.23.down_proj", + "model.layers.77.mlp.experts.24.gate_proj", + "model.layers.77.mlp.experts.24.up_proj", + "model.layers.77.mlp.experts.24.down_proj", + "model.layers.77.mlp.experts.25.gate_proj", + "model.layers.77.mlp.experts.25.up_proj", + "model.layers.77.mlp.experts.25.down_proj", + "model.layers.77.mlp.experts.26.gate_proj", + "model.layers.77.mlp.experts.26.up_proj", + "model.layers.77.mlp.experts.26.down_proj", + "model.layers.77.mlp.experts.27.gate_proj", + "model.layers.77.mlp.experts.27.up_proj", + "model.layers.77.mlp.experts.27.down_proj", + "model.layers.77.mlp.experts.28.gate_proj", + "model.layers.77.mlp.experts.28.up_proj", + "model.layers.77.mlp.experts.28.down_proj", + "model.layers.77.mlp.experts.29.gate_proj", + "model.layers.77.mlp.experts.29.up_proj", + "model.layers.77.mlp.experts.29.down_proj", + "model.layers.77.mlp.experts.30.gate_proj", + "model.layers.77.mlp.experts.30.up_proj", + "model.layers.77.mlp.experts.30.down_proj", + "model.layers.77.mlp.experts.31.gate_proj", + "model.layers.77.mlp.experts.31.up_proj", + "model.layers.77.mlp.experts.31.down_proj", + "model.layers.77.mlp.experts.32.gate_proj", + "model.layers.77.mlp.experts.32.up_proj", + "model.layers.77.mlp.experts.32.down_proj", + "model.layers.77.mlp.experts.33.gate_proj", + "model.layers.77.mlp.experts.33.up_proj", + "model.layers.77.mlp.experts.33.down_proj", + "model.layers.77.mlp.experts.34.gate_proj", + "model.layers.77.mlp.experts.34.up_proj", + "model.layers.77.mlp.experts.34.down_proj", + "model.layers.77.mlp.experts.35.gate_proj", + "model.layers.77.mlp.experts.35.up_proj", + "model.layers.77.mlp.experts.35.down_proj", + "model.layers.77.mlp.experts.36.gate_proj", + "model.layers.77.mlp.experts.36.up_proj", + "model.layers.77.mlp.experts.36.down_proj", + "model.layers.77.mlp.experts.37.gate_proj", + "model.layers.77.mlp.experts.37.up_proj", + "model.layers.77.mlp.experts.37.down_proj", + "model.layers.77.mlp.experts.38.gate_proj", + "model.layers.77.mlp.experts.38.up_proj", + "model.layers.77.mlp.experts.38.down_proj", + "model.layers.77.mlp.experts.39.gate_proj", + "model.layers.77.mlp.experts.39.up_proj", + "model.layers.77.mlp.experts.39.down_proj", + "model.layers.77.mlp.experts.40.gate_proj", + "model.layers.77.mlp.experts.40.up_proj", + "model.layers.77.mlp.experts.40.down_proj", + "model.layers.77.mlp.experts.41.gate_proj", + "model.layers.77.mlp.experts.41.up_proj", + "model.layers.77.mlp.experts.41.down_proj", + "model.layers.77.mlp.experts.42.gate_proj", + "model.layers.77.mlp.experts.42.up_proj", + "model.layers.77.mlp.experts.42.down_proj", + "model.layers.77.mlp.experts.43.gate_proj", + "model.layers.77.mlp.experts.43.up_proj", + "model.layers.77.mlp.experts.43.down_proj", + "model.layers.77.mlp.experts.44.gate_proj", + "model.layers.77.mlp.experts.44.up_proj", + "model.layers.77.mlp.experts.44.down_proj", + "model.layers.77.mlp.experts.45.gate_proj", + "model.layers.77.mlp.experts.45.up_proj", + "model.layers.77.mlp.experts.45.down_proj", + "model.layers.77.mlp.experts.46.gate_proj", + "model.layers.77.mlp.experts.46.up_proj", + "model.layers.77.mlp.experts.46.down_proj", + "model.layers.77.mlp.experts.47.gate_proj", + "model.layers.77.mlp.experts.47.up_proj", + "model.layers.77.mlp.experts.47.down_proj", + "model.layers.77.mlp.experts.48.gate_proj", + "model.layers.77.mlp.experts.48.up_proj", + "model.layers.77.mlp.experts.48.down_proj", + "model.layers.77.mlp.experts.49.gate_proj", + "model.layers.77.mlp.experts.49.up_proj", + "model.layers.77.mlp.experts.49.down_proj", + "model.layers.77.mlp.experts.50.gate_proj", + "model.layers.77.mlp.experts.50.up_proj", + "model.layers.77.mlp.experts.50.down_proj", + "model.layers.77.mlp.experts.51.gate_proj", + "model.layers.77.mlp.experts.51.up_proj", + "model.layers.77.mlp.experts.51.down_proj", + "model.layers.77.mlp.experts.52.gate_proj", + "model.layers.77.mlp.experts.52.up_proj", + "model.layers.77.mlp.experts.52.down_proj", + "model.layers.77.mlp.experts.53.gate_proj", + "model.layers.77.mlp.experts.53.up_proj", + "model.layers.77.mlp.experts.53.down_proj", + "model.layers.77.mlp.experts.54.gate_proj", + "model.layers.77.mlp.experts.54.up_proj", + "model.layers.77.mlp.experts.54.down_proj", + "model.layers.77.mlp.experts.55.gate_proj", + "model.layers.77.mlp.experts.55.up_proj", + "model.layers.77.mlp.experts.55.down_proj", + "model.layers.77.mlp.experts.56.gate_proj", + "model.layers.77.mlp.experts.56.up_proj", + "model.layers.77.mlp.experts.56.down_proj", + "model.layers.77.mlp.experts.57.gate_proj", + "model.layers.77.mlp.experts.57.up_proj", + "model.layers.77.mlp.experts.57.down_proj", + "model.layers.77.mlp.experts.58.gate_proj", + "model.layers.77.mlp.experts.58.up_proj", + "model.layers.77.mlp.experts.58.down_proj", + "model.layers.77.mlp.experts.59.gate_proj", + "model.layers.77.mlp.experts.59.up_proj", + "model.layers.77.mlp.experts.59.down_proj", + "model.layers.77.mlp.experts.60.gate_proj", + "model.layers.77.mlp.experts.60.up_proj", + "model.layers.77.mlp.experts.60.down_proj", + "model.layers.77.mlp.experts.61.gate_proj", + "model.layers.77.mlp.experts.61.up_proj", + "model.layers.77.mlp.experts.61.down_proj", + "model.layers.77.mlp.experts.62.gate_proj", + "model.layers.77.mlp.experts.62.up_proj", + "model.layers.77.mlp.experts.62.down_proj", + "model.layers.77.mlp.experts.63.gate_proj", + "model.layers.77.mlp.experts.63.up_proj", + "model.layers.77.mlp.experts.63.down_proj", + "model.layers.77.mlp.experts.64.gate_proj", + "model.layers.77.mlp.experts.64.up_proj", + "model.layers.77.mlp.experts.64.down_proj", + "model.layers.77.mlp.experts.65.gate_proj", + "model.layers.77.mlp.experts.65.up_proj", + "model.layers.77.mlp.experts.65.down_proj", + "model.layers.77.mlp.experts.66.gate_proj", + "model.layers.77.mlp.experts.66.up_proj", + "model.layers.77.mlp.experts.66.down_proj", + "model.layers.77.mlp.experts.67.gate_proj", + "model.layers.77.mlp.experts.67.up_proj", + "model.layers.77.mlp.experts.67.down_proj", + "model.layers.77.mlp.experts.68.gate_proj", + "model.layers.77.mlp.experts.68.up_proj", + "model.layers.77.mlp.experts.68.down_proj", + "model.layers.77.mlp.experts.69.gate_proj", + "model.layers.77.mlp.experts.69.up_proj", + "model.layers.77.mlp.experts.69.down_proj", + "model.layers.77.mlp.experts.70.gate_proj", + "model.layers.77.mlp.experts.70.up_proj", + "model.layers.77.mlp.experts.70.down_proj", + "model.layers.77.mlp.experts.71.gate_proj", + "model.layers.77.mlp.experts.71.up_proj", + "model.layers.77.mlp.experts.71.down_proj", + "model.layers.77.mlp.experts.72.gate_proj", + "model.layers.77.mlp.experts.72.up_proj", + "model.layers.77.mlp.experts.72.down_proj", + "model.layers.77.mlp.experts.73.gate_proj", + "model.layers.77.mlp.experts.73.up_proj", + "model.layers.77.mlp.experts.73.down_proj", + "model.layers.77.mlp.experts.74.gate_proj", + "model.layers.77.mlp.experts.74.up_proj", + "model.layers.77.mlp.experts.74.down_proj", + "model.layers.77.mlp.experts.75.gate_proj", + "model.layers.77.mlp.experts.75.up_proj", + "model.layers.77.mlp.experts.75.down_proj", + "model.layers.77.mlp.experts.76.gate_proj", + "model.layers.77.mlp.experts.76.up_proj", + "model.layers.77.mlp.experts.76.down_proj", + "model.layers.77.mlp.experts.77.gate_proj", + "model.layers.77.mlp.experts.77.up_proj", + "model.layers.77.mlp.experts.77.down_proj", + "model.layers.77.mlp.experts.78.gate_proj", + "model.layers.77.mlp.experts.78.up_proj", + "model.layers.77.mlp.experts.78.down_proj", + "model.layers.77.mlp.experts.79.gate_proj", + "model.layers.77.mlp.experts.79.up_proj", + "model.layers.77.mlp.experts.79.down_proj", + "model.layers.77.mlp.experts.80.gate_proj", + "model.layers.77.mlp.experts.80.up_proj", + "model.layers.77.mlp.experts.80.down_proj", + "model.layers.77.mlp.experts.81.gate_proj", + "model.layers.77.mlp.experts.81.up_proj", + "model.layers.77.mlp.experts.81.down_proj", + "model.layers.77.mlp.experts.82.gate_proj", + "model.layers.77.mlp.experts.82.up_proj", + "model.layers.77.mlp.experts.82.down_proj", + "model.layers.77.mlp.experts.83.gate_proj", + "model.layers.77.mlp.experts.83.up_proj", + "model.layers.77.mlp.experts.83.down_proj", + "model.layers.77.mlp.experts.84.gate_proj", + "model.layers.77.mlp.experts.84.up_proj", + "model.layers.77.mlp.experts.84.down_proj", + "model.layers.77.mlp.experts.85.gate_proj", + "model.layers.77.mlp.experts.85.up_proj", + "model.layers.77.mlp.experts.85.down_proj", + "model.layers.77.mlp.experts.86.gate_proj", + "model.layers.77.mlp.experts.86.up_proj", + "model.layers.77.mlp.experts.86.down_proj", + "model.layers.77.mlp.experts.87.gate_proj", + "model.layers.77.mlp.experts.87.up_proj", + "model.layers.77.mlp.experts.87.down_proj", + "model.layers.77.mlp.experts.88.gate_proj", + "model.layers.77.mlp.experts.88.up_proj", + "model.layers.77.mlp.experts.88.down_proj", + "model.layers.77.mlp.experts.89.gate_proj", + "model.layers.77.mlp.experts.89.up_proj", + "model.layers.77.mlp.experts.89.down_proj", + "model.layers.77.mlp.experts.90.gate_proj", + "model.layers.77.mlp.experts.90.up_proj", + "model.layers.77.mlp.experts.90.down_proj", + "model.layers.77.mlp.experts.91.gate_proj", + "model.layers.77.mlp.experts.91.up_proj", + "model.layers.77.mlp.experts.91.down_proj", + "model.layers.77.mlp.experts.92.gate_proj", + "model.layers.77.mlp.experts.92.up_proj", + "model.layers.77.mlp.experts.92.down_proj", + "model.layers.77.mlp.experts.93.gate_proj", + "model.layers.77.mlp.experts.93.up_proj", + "model.layers.77.mlp.experts.93.down_proj", + "model.layers.77.mlp.experts.94.gate_proj", + "model.layers.77.mlp.experts.94.up_proj", + "model.layers.77.mlp.experts.94.down_proj", + "model.layers.77.mlp.experts.95.gate_proj", + "model.layers.77.mlp.experts.95.up_proj", + "model.layers.77.mlp.experts.95.down_proj", + "model.layers.77.mlp.experts.96.gate_proj", + "model.layers.77.mlp.experts.96.up_proj", + "model.layers.77.mlp.experts.96.down_proj", + "model.layers.77.mlp.experts.97.gate_proj", + "model.layers.77.mlp.experts.97.up_proj", + "model.layers.77.mlp.experts.97.down_proj", + "model.layers.77.mlp.experts.98.gate_proj", + "model.layers.77.mlp.experts.98.up_proj", + "model.layers.77.mlp.experts.98.down_proj", + "model.layers.77.mlp.experts.99.gate_proj", + "model.layers.77.mlp.experts.99.up_proj", + "model.layers.77.mlp.experts.99.down_proj", + "model.layers.77.mlp.experts.100.gate_proj", + "model.layers.77.mlp.experts.100.up_proj", + "model.layers.77.mlp.experts.100.down_proj", + "model.layers.77.mlp.experts.101.gate_proj", + "model.layers.77.mlp.experts.101.up_proj", + "model.layers.77.mlp.experts.101.down_proj", + "model.layers.77.mlp.experts.102.gate_proj", + "model.layers.77.mlp.experts.102.up_proj", + "model.layers.77.mlp.experts.102.down_proj", + "model.layers.77.mlp.experts.103.gate_proj", + "model.layers.77.mlp.experts.103.up_proj", + "model.layers.77.mlp.experts.103.down_proj", + "model.layers.77.mlp.experts.104.gate_proj", + "model.layers.77.mlp.experts.104.up_proj", + "model.layers.77.mlp.experts.104.down_proj", + "model.layers.77.mlp.experts.105.gate_proj", + "model.layers.77.mlp.experts.105.up_proj", + "model.layers.77.mlp.experts.105.down_proj", + "model.layers.77.mlp.experts.106.gate_proj", + "model.layers.77.mlp.experts.106.up_proj", + "model.layers.77.mlp.experts.106.down_proj", + "model.layers.77.mlp.experts.107.gate_proj", + "model.layers.77.mlp.experts.107.up_proj", + "model.layers.77.mlp.experts.107.down_proj", + "model.layers.77.mlp.experts.108.gate_proj", + "model.layers.77.mlp.experts.108.up_proj", + "model.layers.77.mlp.experts.108.down_proj", + "model.layers.77.mlp.experts.109.gate_proj", + "model.layers.77.mlp.experts.109.up_proj", + "model.layers.77.mlp.experts.109.down_proj", + "model.layers.77.mlp.experts.110.gate_proj", + "model.layers.77.mlp.experts.110.up_proj", + "model.layers.77.mlp.experts.110.down_proj", + "model.layers.77.mlp.experts.111.gate_proj", + "model.layers.77.mlp.experts.111.up_proj", + "model.layers.77.mlp.experts.111.down_proj", + "model.layers.77.mlp.experts.112.gate_proj", + "model.layers.77.mlp.experts.112.up_proj", + "model.layers.77.mlp.experts.112.down_proj", + "model.layers.77.mlp.experts.113.gate_proj", + "model.layers.77.mlp.experts.113.up_proj", + "model.layers.77.mlp.experts.113.down_proj", + "model.layers.77.mlp.experts.114.gate_proj", + "model.layers.77.mlp.experts.114.up_proj", + "model.layers.77.mlp.experts.114.down_proj", + "model.layers.77.mlp.experts.115.gate_proj", + "model.layers.77.mlp.experts.115.up_proj", + "model.layers.77.mlp.experts.115.down_proj", + "model.layers.77.mlp.experts.116.gate_proj", + "model.layers.77.mlp.experts.116.up_proj", + "model.layers.77.mlp.experts.116.down_proj", + "model.layers.77.mlp.experts.117.gate_proj", + "model.layers.77.mlp.experts.117.up_proj", + "model.layers.77.mlp.experts.117.down_proj", + "model.layers.77.mlp.experts.118.gate_proj", + "model.layers.77.mlp.experts.118.up_proj", + "model.layers.77.mlp.experts.118.down_proj", + "model.layers.77.mlp.experts.119.gate_proj", + "model.layers.77.mlp.experts.119.up_proj", + "model.layers.77.mlp.experts.119.down_proj", + "model.layers.77.mlp.experts.120.gate_proj", + "model.layers.77.mlp.experts.120.up_proj", + "model.layers.77.mlp.experts.120.down_proj", + "model.layers.77.mlp.experts.121.gate_proj", + "model.layers.77.mlp.experts.121.up_proj", + "model.layers.77.mlp.experts.121.down_proj", + "model.layers.77.mlp.experts.122.gate_proj", + "model.layers.77.mlp.experts.122.up_proj", + "model.layers.77.mlp.experts.122.down_proj", + "model.layers.77.mlp.experts.123.gate_proj", + "model.layers.77.mlp.experts.123.up_proj", + "model.layers.77.mlp.experts.123.down_proj", + "model.layers.77.mlp.experts.124.gate_proj", + "model.layers.77.mlp.experts.124.up_proj", + "model.layers.77.mlp.experts.124.down_proj", + "model.layers.77.mlp.experts.125.gate_proj", + "model.layers.77.mlp.experts.125.up_proj", + "model.layers.77.mlp.experts.125.down_proj", + "model.layers.77.mlp.experts.126.gate_proj", + "model.layers.77.mlp.experts.126.up_proj", + "model.layers.77.mlp.experts.126.down_proj", + "model.layers.77.mlp.experts.127.gate_proj", + "model.layers.77.mlp.experts.127.up_proj", + "model.layers.77.mlp.experts.127.down_proj", + "model.layers.77.mlp.experts.128.gate_proj", + "model.layers.77.mlp.experts.128.up_proj", + "model.layers.77.mlp.experts.128.down_proj", + "model.layers.77.mlp.experts.129.gate_proj", + "model.layers.77.mlp.experts.129.up_proj", + "model.layers.77.mlp.experts.129.down_proj", + "model.layers.77.mlp.experts.130.gate_proj", + "model.layers.77.mlp.experts.130.up_proj", + "model.layers.77.mlp.experts.130.down_proj", + "model.layers.77.mlp.experts.131.gate_proj", + "model.layers.77.mlp.experts.131.up_proj", + "model.layers.77.mlp.experts.131.down_proj", + "model.layers.77.mlp.experts.132.gate_proj", + "model.layers.77.mlp.experts.132.up_proj", + "model.layers.77.mlp.experts.132.down_proj", + "model.layers.77.mlp.experts.133.gate_proj", + "model.layers.77.mlp.experts.133.up_proj", + "model.layers.77.mlp.experts.133.down_proj", + "model.layers.77.mlp.experts.134.gate_proj", + "model.layers.77.mlp.experts.134.up_proj", + "model.layers.77.mlp.experts.134.down_proj", + "model.layers.77.mlp.experts.135.gate_proj", + "model.layers.77.mlp.experts.135.up_proj", + "model.layers.77.mlp.experts.135.down_proj", + "model.layers.77.mlp.experts.136.gate_proj", + "model.layers.77.mlp.experts.136.up_proj", + "model.layers.77.mlp.experts.136.down_proj", + "model.layers.77.mlp.experts.137.gate_proj", + "model.layers.77.mlp.experts.137.up_proj", + "model.layers.77.mlp.experts.137.down_proj", + "model.layers.77.mlp.experts.138.gate_proj", + "model.layers.77.mlp.experts.138.up_proj", + "model.layers.77.mlp.experts.138.down_proj", + "model.layers.77.mlp.experts.139.gate_proj", + "model.layers.77.mlp.experts.139.up_proj", + "model.layers.77.mlp.experts.139.down_proj", + "model.layers.77.mlp.experts.140.gate_proj", + "model.layers.77.mlp.experts.140.up_proj", + "model.layers.77.mlp.experts.140.down_proj", + "model.layers.77.mlp.experts.141.gate_proj", + "model.layers.77.mlp.experts.141.up_proj", + "model.layers.77.mlp.experts.141.down_proj", + "model.layers.77.mlp.experts.142.gate_proj", + "model.layers.77.mlp.experts.142.up_proj", + "model.layers.77.mlp.experts.142.down_proj", + "model.layers.77.mlp.experts.143.gate_proj", + "model.layers.77.mlp.experts.143.up_proj", + "model.layers.77.mlp.experts.143.down_proj", + "model.layers.77.mlp.experts.144.gate_proj", + "model.layers.77.mlp.experts.144.up_proj", + "model.layers.77.mlp.experts.144.down_proj", + "model.layers.77.mlp.experts.145.gate_proj", + "model.layers.77.mlp.experts.145.up_proj", + "model.layers.77.mlp.experts.145.down_proj", + "model.layers.77.mlp.experts.146.gate_proj", + "model.layers.77.mlp.experts.146.up_proj", + "model.layers.77.mlp.experts.146.down_proj", + "model.layers.77.mlp.experts.147.gate_proj", + "model.layers.77.mlp.experts.147.up_proj", + "model.layers.77.mlp.experts.147.down_proj", + "model.layers.77.mlp.experts.148.gate_proj", + "model.layers.77.mlp.experts.148.up_proj", + "model.layers.77.mlp.experts.148.down_proj", + "model.layers.77.mlp.experts.149.gate_proj", + "model.layers.77.mlp.experts.149.up_proj", + "model.layers.77.mlp.experts.149.down_proj", + "model.layers.77.mlp.experts.150.gate_proj", + "model.layers.77.mlp.experts.150.up_proj", + "model.layers.77.mlp.experts.150.down_proj", + "model.layers.77.mlp.experts.151.gate_proj", + "model.layers.77.mlp.experts.151.up_proj", + "model.layers.77.mlp.experts.151.down_proj", + "model.layers.77.mlp.experts.152.gate_proj", + "model.layers.77.mlp.experts.152.up_proj", + "model.layers.77.mlp.experts.152.down_proj", + "model.layers.77.mlp.experts.153.gate_proj", + "model.layers.77.mlp.experts.153.up_proj", + "model.layers.77.mlp.experts.153.down_proj", + "model.layers.77.mlp.experts.154.gate_proj", + "model.layers.77.mlp.experts.154.up_proj", + "model.layers.77.mlp.experts.154.down_proj", + "model.layers.77.mlp.experts.155.gate_proj", + "model.layers.77.mlp.experts.155.up_proj", + "model.layers.77.mlp.experts.155.down_proj", + "model.layers.77.mlp.experts.156.gate_proj", + "model.layers.77.mlp.experts.156.up_proj", + "model.layers.77.mlp.experts.156.down_proj", + "model.layers.77.mlp.experts.157.gate_proj", + "model.layers.77.mlp.experts.157.up_proj", + "model.layers.77.mlp.experts.157.down_proj", + "model.layers.77.mlp.experts.158.gate_proj", + "model.layers.77.mlp.experts.158.up_proj", + "model.layers.77.mlp.experts.158.down_proj", + "model.layers.77.mlp.experts.159.gate_proj", + "model.layers.77.mlp.experts.159.up_proj", + "model.layers.77.mlp.experts.159.down_proj", + "model.layers.77.mlp.experts.160.gate_proj", + "model.layers.77.mlp.experts.160.up_proj", + "model.layers.77.mlp.experts.160.down_proj", + "model.layers.77.mlp.experts.161.gate_proj", + "model.layers.77.mlp.experts.161.up_proj", + "model.layers.77.mlp.experts.161.down_proj", + "model.layers.77.mlp.experts.162.gate_proj", + "model.layers.77.mlp.experts.162.up_proj", + "model.layers.77.mlp.experts.162.down_proj", + "model.layers.77.mlp.experts.163.gate_proj", + "model.layers.77.mlp.experts.163.up_proj", + "model.layers.77.mlp.experts.163.down_proj", + "model.layers.77.mlp.experts.164.gate_proj", + "model.layers.77.mlp.experts.164.up_proj", + "model.layers.77.mlp.experts.164.down_proj", + "model.layers.77.mlp.experts.165.gate_proj", + "model.layers.77.mlp.experts.165.up_proj", + "model.layers.77.mlp.experts.165.down_proj", + "model.layers.77.mlp.experts.166.gate_proj", + "model.layers.77.mlp.experts.166.up_proj", + "model.layers.77.mlp.experts.166.down_proj", + "model.layers.77.mlp.experts.167.gate_proj", + "model.layers.77.mlp.experts.167.up_proj", + "model.layers.77.mlp.experts.167.down_proj", + "model.layers.77.mlp.experts.168.gate_proj", + "model.layers.77.mlp.experts.168.up_proj", + "model.layers.77.mlp.experts.168.down_proj", + "model.layers.77.mlp.experts.169.gate_proj", + "model.layers.77.mlp.experts.169.up_proj", + "model.layers.77.mlp.experts.169.down_proj", + "model.layers.77.mlp.experts.170.gate_proj", + "model.layers.77.mlp.experts.170.up_proj", + "model.layers.77.mlp.experts.170.down_proj", + "model.layers.77.mlp.experts.171.gate_proj", + "model.layers.77.mlp.experts.171.up_proj", + "model.layers.77.mlp.experts.171.down_proj", + "model.layers.77.mlp.experts.172.gate_proj", + "model.layers.77.mlp.experts.172.up_proj", + "model.layers.77.mlp.experts.172.down_proj", + "model.layers.77.mlp.experts.173.gate_proj", + "model.layers.77.mlp.experts.173.up_proj", + "model.layers.77.mlp.experts.173.down_proj", + "model.layers.77.mlp.experts.174.gate_proj", + "model.layers.77.mlp.experts.174.up_proj", + "model.layers.77.mlp.experts.174.down_proj", + "model.layers.77.mlp.experts.175.gate_proj", + "model.layers.77.mlp.experts.175.up_proj", + "model.layers.77.mlp.experts.175.down_proj", + "model.layers.77.mlp.experts.176.gate_proj", + "model.layers.77.mlp.experts.176.up_proj", + "model.layers.77.mlp.experts.176.down_proj", + "model.layers.77.mlp.experts.177.gate_proj", + "model.layers.77.mlp.experts.177.up_proj", + "model.layers.77.mlp.experts.177.down_proj", + "model.layers.77.mlp.experts.178.gate_proj", + "model.layers.77.mlp.experts.178.up_proj", + "model.layers.77.mlp.experts.178.down_proj", + "model.layers.77.mlp.experts.179.gate_proj", + "model.layers.77.mlp.experts.179.up_proj", + "model.layers.77.mlp.experts.179.down_proj", + "model.layers.77.mlp.experts.180.gate_proj", + "model.layers.77.mlp.experts.180.up_proj", + "model.layers.77.mlp.experts.180.down_proj", + "model.layers.77.mlp.experts.181.gate_proj", + "model.layers.77.mlp.experts.181.up_proj", + "model.layers.77.mlp.experts.181.down_proj", + "model.layers.77.mlp.experts.182.gate_proj", + "model.layers.77.mlp.experts.182.up_proj", + "model.layers.77.mlp.experts.182.down_proj", + "model.layers.77.mlp.experts.183.gate_proj", + "model.layers.77.mlp.experts.183.up_proj", + "model.layers.77.mlp.experts.183.down_proj", + "model.layers.77.mlp.experts.184.gate_proj", + "model.layers.77.mlp.experts.184.up_proj", + "model.layers.77.mlp.experts.184.down_proj", + "model.layers.77.mlp.experts.185.gate_proj", + "model.layers.77.mlp.experts.185.up_proj", + "model.layers.77.mlp.experts.185.down_proj", + "model.layers.77.mlp.experts.186.gate_proj", + "model.layers.77.mlp.experts.186.up_proj", + "model.layers.77.mlp.experts.186.down_proj", + "model.layers.77.mlp.experts.187.gate_proj", + "model.layers.77.mlp.experts.187.up_proj", + "model.layers.77.mlp.experts.187.down_proj", + "model.layers.77.mlp.experts.188.gate_proj", + "model.layers.77.mlp.experts.188.up_proj", + "model.layers.77.mlp.experts.188.down_proj", + "model.layers.77.mlp.experts.189.gate_proj", + "model.layers.77.mlp.experts.189.up_proj", + "model.layers.77.mlp.experts.189.down_proj", + "model.layers.77.mlp.experts.190.gate_proj", + "model.layers.77.mlp.experts.190.up_proj", + "model.layers.77.mlp.experts.190.down_proj", + "model.layers.77.mlp.experts.191.gate_proj", + "model.layers.77.mlp.experts.191.up_proj", + "model.layers.77.mlp.experts.191.down_proj", + "model.layers.77.mlp.experts.192.gate_proj", + "model.layers.77.mlp.experts.192.up_proj", + "model.layers.77.mlp.experts.192.down_proj", + "model.layers.77.mlp.experts.193.gate_proj", + "model.layers.77.mlp.experts.193.up_proj", + "model.layers.77.mlp.experts.193.down_proj", + "model.layers.77.mlp.experts.194.gate_proj", + "model.layers.77.mlp.experts.194.up_proj", + "model.layers.77.mlp.experts.194.down_proj", + "model.layers.77.mlp.experts.195.gate_proj", + "model.layers.77.mlp.experts.195.up_proj", + "model.layers.77.mlp.experts.195.down_proj", + "model.layers.77.mlp.experts.196.gate_proj", + "model.layers.77.mlp.experts.196.up_proj", + "model.layers.77.mlp.experts.196.down_proj", + "model.layers.77.mlp.experts.197.gate_proj", + "model.layers.77.mlp.experts.197.up_proj", + "model.layers.77.mlp.experts.197.down_proj", + "model.layers.77.mlp.experts.198.gate_proj", + "model.layers.77.mlp.experts.198.up_proj", + "model.layers.77.mlp.experts.198.down_proj", + "model.layers.77.mlp.experts.199.gate_proj", + "model.layers.77.mlp.experts.199.up_proj", + "model.layers.77.mlp.experts.199.down_proj", + "model.layers.77.mlp.experts.200.gate_proj", + "model.layers.77.mlp.experts.200.up_proj", + "model.layers.77.mlp.experts.200.down_proj", + "model.layers.77.mlp.experts.201.gate_proj", + "model.layers.77.mlp.experts.201.up_proj", + "model.layers.77.mlp.experts.201.down_proj", + "model.layers.77.mlp.experts.202.gate_proj", + "model.layers.77.mlp.experts.202.up_proj", + "model.layers.77.mlp.experts.202.down_proj", + "model.layers.77.mlp.experts.203.gate_proj", + "model.layers.77.mlp.experts.203.up_proj", + "model.layers.77.mlp.experts.203.down_proj", + "model.layers.77.mlp.experts.204.gate_proj", + "model.layers.77.mlp.experts.204.up_proj", + "model.layers.77.mlp.experts.204.down_proj", + "model.layers.77.mlp.experts.205.gate_proj", + "model.layers.77.mlp.experts.205.up_proj", + "model.layers.77.mlp.experts.205.down_proj", + "model.layers.77.mlp.experts.206.gate_proj", + "model.layers.77.mlp.experts.206.up_proj", + "model.layers.77.mlp.experts.206.down_proj", + "model.layers.77.mlp.experts.207.gate_proj", + "model.layers.77.mlp.experts.207.up_proj", + "model.layers.77.mlp.experts.207.down_proj", + "model.layers.77.mlp.experts.208.gate_proj", + "model.layers.77.mlp.experts.208.up_proj", + "model.layers.77.mlp.experts.208.down_proj", + "model.layers.77.mlp.experts.209.gate_proj", + "model.layers.77.mlp.experts.209.up_proj", + "model.layers.77.mlp.experts.209.down_proj", + "model.layers.77.mlp.experts.210.gate_proj", + "model.layers.77.mlp.experts.210.up_proj", + "model.layers.77.mlp.experts.210.down_proj", + "model.layers.77.mlp.experts.211.gate_proj", + "model.layers.77.mlp.experts.211.up_proj", + "model.layers.77.mlp.experts.211.down_proj", + "model.layers.77.mlp.experts.212.gate_proj", + "model.layers.77.mlp.experts.212.up_proj", + "model.layers.77.mlp.experts.212.down_proj", + "model.layers.77.mlp.experts.213.gate_proj", + "model.layers.77.mlp.experts.213.up_proj", + "model.layers.77.mlp.experts.213.down_proj", + "model.layers.77.mlp.experts.214.gate_proj", + "model.layers.77.mlp.experts.214.up_proj", + "model.layers.77.mlp.experts.214.down_proj", + "model.layers.77.mlp.experts.215.gate_proj", + "model.layers.77.mlp.experts.215.up_proj", + "model.layers.77.mlp.experts.215.down_proj", + "model.layers.77.mlp.experts.216.gate_proj", + "model.layers.77.mlp.experts.216.up_proj", + "model.layers.77.mlp.experts.216.down_proj", + "model.layers.77.mlp.experts.217.gate_proj", + "model.layers.77.mlp.experts.217.up_proj", + "model.layers.77.mlp.experts.217.down_proj", + "model.layers.77.mlp.experts.218.gate_proj", + "model.layers.77.mlp.experts.218.up_proj", + "model.layers.77.mlp.experts.218.down_proj", + "model.layers.77.mlp.experts.219.gate_proj", + "model.layers.77.mlp.experts.219.up_proj", + "model.layers.77.mlp.experts.219.down_proj", + "model.layers.77.mlp.experts.220.gate_proj", + "model.layers.77.mlp.experts.220.up_proj", + "model.layers.77.mlp.experts.220.down_proj", + "model.layers.77.mlp.experts.221.gate_proj", + "model.layers.77.mlp.experts.221.up_proj", + "model.layers.77.mlp.experts.221.down_proj", + "model.layers.77.mlp.experts.222.gate_proj", + "model.layers.77.mlp.experts.222.up_proj", + "model.layers.77.mlp.experts.222.down_proj", + "model.layers.77.mlp.experts.223.gate_proj", + "model.layers.77.mlp.experts.223.up_proj", + "model.layers.77.mlp.experts.223.down_proj", + "model.layers.77.mlp.experts.224.gate_proj", + "model.layers.77.mlp.experts.224.up_proj", + "model.layers.77.mlp.experts.224.down_proj", + "model.layers.77.mlp.experts.225.gate_proj", + "model.layers.77.mlp.experts.225.up_proj", + "model.layers.77.mlp.experts.225.down_proj", + "model.layers.77.mlp.experts.226.gate_proj", + "model.layers.77.mlp.experts.226.up_proj", + "model.layers.77.mlp.experts.226.down_proj", + "model.layers.77.mlp.experts.227.gate_proj", + "model.layers.77.mlp.experts.227.up_proj", + "model.layers.77.mlp.experts.227.down_proj", + "model.layers.77.mlp.experts.228.gate_proj", + "model.layers.77.mlp.experts.228.up_proj", + "model.layers.77.mlp.experts.228.down_proj", + "model.layers.77.mlp.experts.229.gate_proj", + "model.layers.77.mlp.experts.229.up_proj", + "model.layers.77.mlp.experts.229.down_proj", + "model.layers.77.mlp.experts.230.gate_proj", + "model.layers.77.mlp.experts.230.up_proj", + "model.layers.77.mlp.experts.230.down_proj", + "model.layers.77.mlp.experts.231.gate_proj", + "model.layers.77.mlp.experts.231.up_proj", + "model.layers.77.mlp.experts.231.down_proj", + "model.layers.77.mlp.experts.232.gate_proj", + "model.layers.77.mlp.experts.232.up_proj", + "model.layers.77.mlp.experts.232.down_proj", + "model.layers.77.mlp.experts.233.gate_proj", + "model.layers.77.mlp.experts.233.up_proj", + "model.layers.77.mlp.experts.233.down_proj", + "model.layers.77.mlp.experts.234.gate_proj", + "model.layers.77.mlp.experts.234.up_proj", + "model.layers.77.mlp.experts.234.down_proj", + "model.layers.77.mlp.experts.235.gate_proj", + "model.layers.77.mlp.experts.235.up_proj", + "model.layers.77.mlp.experts.235.down_proj", + "model.layers.77.mlp.experts.236.gate_proj", + "model.layers.77.mlp.experts.236.up_proj", + "model.layers.77.mlp.experts.236.down_proj", + "model.layers.77.mlp.experts.237.gate_proj", + "model.layers.77.mlp.experts.237.up_proj", + "model.layers.77.mlp.experts.237.down_proj", + "model.layers.77.mlp.experts.238.gate_proj", + "model.layers.77.mlp.experts.238.up_proj", + "model.layers.77.mlp.experts.238.down_proj", + "model.layers.77.mlp.experts.239.gate_proj", + "model.layers.77.mlp.experts.239.up_proj", + "model.layers.77.mlp.experts.239.down_proj", + "model.layers.77.mlp.experts.240.gate_proj", + "model.layers.77.mlp.experts.240.up_proj", + "model.layers.77.mlp.experts.240.down_proj", + "model.layers.77.mlp.experts.241.gate_proj", + "model.layers.77.mlp.experts.241.up_proj", + "model.layers.77.mlp.experts.241.down_proj", + "model.layers.77.mlp.experts.242.gate_proj", + "model.layers.77.mlp.experts.242.up_proj", + "model.layers.77.mlp.experts.242.down_proj", + "model.layers.77.mlp.experts.243.gate_proj", + "model.layers.77.mlp.experts.243.up_proj", + "model.layers.77.mlp.experts.243.down_proj", + "model.layers.77.mlp.experts.244.gate_proj", + "model.layers.77.mlp.experts.244.up_proj", + "model.layers.77.mlp.experts.244.down_proj", + "model.layers.77.mlp.experts.245.gate_proj", + "model.layers.77.mlp.experts.245.up_proj", + "model.layers.77.mlp.experts.245.down_proj", + "model.layers.77.mlp.experts.246.gate_proj", + "model.layers.77.mlp.experts.246.up_proj", + "model.layers.77.mlp.experts.246.down_proj", + "model.layers.77.mlp.experts.247.gate_proj", + "model.layers.77.mlp.experts.247.up_proj", + "model.layers.77.mlp.experts.247.down_proj", + "model.layers.77.mlp.experts.248.gate_proj", + "model.layers.77.mlp.experts.248.up_proj", + "model.layers.77.mlp.experts.248.down_proj", + "model.layers.77.mlp.experts.249.gate_proj", + "model.layers.77.mlp.experts.249.up_proj", + "model.layers.77.mlp.experts.249.down_proj", + "model.layers.77.mlp.experts.250.gate_proj", + "model.layers.77.mlp.experts.250.up_proj", + "model.layers.77.mlp.experts.250.down_proj", + "model.layers.77.mlp.experts.251.gate_proj", + "model.layers.77.mlp.experts.251.up_proj", + "model.layers.77.mlp.experts.251.down_proj", + "model.layers.77.mlp.experts.252.gate_proj", + "model.layers.77.mlp.experts.252.up_proj", + "model.layers.77.mlp.experts.252.down_proj", + "model.layers.77.mlp.experts.253.gate_proj", + "model.layers.77.mlp.experts.253.up_proj", + "model.layers.77.mlp.experts.253.down_proj", + "model.layers.77.mlp.experts.254.gate_proj", + "model.layers.77.mlp.experts.254.up_proj", + "model.layers.77.mlp.experts.254.down_proj", + "model.layers.77.mlp.experts.255.gate_proj", + "model.layers.77.mlp.experts.255.up_proj", + "model.layers.77.mlp.experts.255.down_proj", + "model.layers.77.mlp.shared_experts.gate_proj", + "model.layers.77.mlp.shared_experts.up_proj", + "model.layers.77.mlp.shared_experts.down_proj", + "lm_head", + "model.layers.78.eh_proj", + "model.layers.78.mlp.experts.0.down_proj", + "model.layers.78.mlp.experts.0.gate_proj", + "model.layers.78.mlp.experts.0.up_proj", + "model.layers.78.mlp.experts.1.down_proj", + "model.layers.78.mlp.experts.1.gate_proj", + "model.layers.78.mlp.experts.1.up_proj", + "model.layers.78.mlp.experts.10.down_proj", + "model.layers.78.mlp.experts.10.gate_proj", + "model.layers.78.mlp.experts.10.up_proj", + "model.layers.78.mlp.experts.100.down_proj", + "model.layers.78.mlp.experts.100.gate_proj", + "model.layers.78.mlp.experts.100.up_proj", + "model.layers.78.mlp.experts.101.down_proj", + "model.layers.78.mlp.experts.101.gate_proj", + "model.layers.78.mlp.experts.101.up_proj", + "model.layers.78.mlp.experts.102.down_proj", + "model.layers.78.mlp.experts.102.gate_proj", + "model.layers.78.mlp.experts.102.up_proj", + "model.layers.78.mlp.experts.103.down_proj", + "model.layers.78.mlp.experts.103.gate_proj", + "model.layers.78.mlp.experts.103.up_proj", + "model.layers.78.mlp.experts.104.down_proj", + "model.layers.78.mlp.experts.104.gate_proj", + "model.layers.78.mlp.experts.104.up_proj", + "model.layers.78.mlp.experts.105.down_proj", + "model.layers.78.mlp.experts.105.gate_proj", + "model.layers.78.mlp.experts.105.up_proj", + "model.layers.78.mlp.experts.106.down_proj", + "model.layers.78.mlp.experts.106.gate_proj", + "model.layers.78.mlp.experts.106.up_proj", + "model.layers.78.mlp.experts.107.down_proj", + "model.layers.78.mlp.experts.107.gate_proj", + "model.layers.78.mlp.experts.107.up_proj", + "model.layers.78.mlp.experts.108.down_proj", + "model.layers.78.mlp.experts.108.gate_proj", + "model.layers.78.mlp.experts.108.up_proj", + "model.layers.78.mlp.experts.109.down_proj", + "model.layers.78.mlp.experts.109.gate_proj", + "model.layers.78.mlp.experts.109.up_proj", + "model.layers.78.mlp.experts.11.down_proj", + "model.layers.78.mlp.experts.11.gate_proj", + "model.layers.78.mlp.experts.11.up_proj", + "model.layers.78.mlp.experts.110.down_proj", + "model.layers.78.mlp.experts.110.gate_proj", + "model.layers.78.mlp.experts.110.up_proj", + "model.layers.78.mlp.experts.111.down_proj", + "model.layers.78.mlp.experts.111.gate_proj", + "model.layers.78.mlp.experts.111.up_proj", + "model.layers.78.mlp.experts.112.down_proj", + "model.layers.78.mlp.experts.112.gate_proj", + "model.layers.78.mlp.experts.112.up_proj", + "model.layers.78.mlp.experts.113.down_proj", + "model.layers.78.mlp.experts.113.gate_proj", + "model.layers.78.mlp.experts.113.up_proj", + "model.layers.78.mlp.experts.114.down_proj", + "model.layers.78.mlp.experts.114.gate_proj", + "model.layers.78.mlp.experts.114.up_proj", + "model.layers.78.mlp.experts.115.down_proj", + "model.layers.78.mlp.experts.115.gate_proj", + "model.layers.78.mlp.experts.115.up_proj", + "model.layers.78.mlp.experts.116.down_proj", + "model.layers.78.mlp.experts.116.gate_proj", + "model.layers.78.mlp.experts.116.up_proj", + "model.layers.78.mlp.experts.117.down_proj", + "model.layers.78.mlp.experts.117.gate_proj", + "model.layers.78.mlp.experts.117.up_proj", + "model.layers.78.mlp.experts.118.down_proj", + "model.layers.78.mlp.experts.118.gate_proj", + "model.layers.78.mlp.experts.118.up_proj", + "model.layers.78.mlp.experts.119.down_proj", + "model.layers.78.mlp.experts.119.gate_proj", + "model.layers.78.mlp.experts.119.up_proj", + "model.layers.78.mlp.experts.12.down_proj", + "model.layers.78.mlp.experts.12.gate_proj", + "model.layers.78.mlp.experts.12.up_proj", + "model.layers.78.mlp.experts.120.down_proj", + "model.layers.78.mlp.experts.120.gate_proj", + "model.layers.78.mlp.experts.120.up_proj", + "model.layers.78.mlp.experts.121.down_proj", + "model.layers.78.mlp.experts.121.gate_proj", + "model.layers.78.mlp.experts.121.up_proj", + "model.layers.78.mlp.experts.122.down_proj", + "model.layers.78.mlp.experts.122.gate_proj", + "model.layers.78.mlp.experts.122.up_proj", + "model.layers.78.mlp.experts.123.down_proj", + "model.layers.78.mlp.experts.123.gate_proj", + "model.layers.78.mlp.experts.123.up_proj", + "model.layers.78.mlp.experts.124.down_proj", + "model.layers.78.mlp.experts.124.gate_proj", + "model.layers.78.mlp.experts.124.up_proj", + "model.layers.78.mlp.experts.125.down_proj", + "model.layers.78.mlp.experts.125.gate_proj", + "model.layers.78.mlp.experts.125.up_proj", + "model.layers.78.mlp.experts.126.down_proj", + "model.layers.78.mlp.experts.126.gate_proj", + "model.layers.78.mlp.experts.126.up_proj", + "model.layers.78.mlp.experts.127.down_proj", + "model.layers.78.mlp.experts.127.gate_proj", + "model.layers.78.mlp.experts.127.up_proj", + "model.layers.78.mlp.experts.128.down_proj", + "model.layers.78.mlp.experts.128.gate_proj", + "model.layers.78.mlp.experts.128.up_proj", + "model.layers.78.mlp.experts.129.down_proj", + "model.layers.78.mlp.experts.129.gate_proj", + "model.layers.78.mlp.experts.129.up_proj", + "model.layers.78.mlp.experts.13.down_proj", + "model.layers.78.mlp.experts.13.gate_proj", + "model.layers.78.mlp.experts.13.up_proj", + "model.layers.78.mlp.experts.130.down_proj", + "model.layers.78.mlp.experts.130.gate_proj", + "model.layers.78.mlp.experts.130.up_proj", + "model.layers.78.mlp.experts.131.down_proj", + "model.layers.78.mlp.experts.131.gate_proj", + "model.layers.78.mlp.experts.131.up_proj", + "model.layers.78.mlp.experts.132.down_proj", + "model.layers.78.mlp.experts.132.gate_proj", + "model.layers.78.mlp.experts.132.up_proj", + "model.layers.78.mlp.experts.133.down_proj", + "model.layers.78.mlp.experts.133.gate_proj", + "model.layers.78.mlp.experts.133.up_proj", + "model.layers.78.mlp.experts.134.down_proj", + "model.layers.78.mlp.experts.134.gate_proj", + "model.layers.78.mlp.experts.134.up_proj", + "model.layers.78.mlp.experts.135.down_proj", + "model.layers.78.mlp.experts.135.gate_proj", + "model.layers.78.mlp.experts.135.up_proj", + "model.layers.78.mlp.experts.136.down_proj", + "model.layers.78.mlp.experts.136.gate_proj", + "model.layers.78.mlp.experts.136.up_proj", + "model.layers.78.mlp.experts.137.down_proj", + "model.layers.78.mlp.experts.137.gate_proj", + "model.layers.78.mlp.experts.137.up_proj", + "model.layers.78.mlp.experts.138.down_proj", + "model.layers.78.mlp.experts.138.gate_proj", + "model.layers.78.mlp.experts.138.up_proj", + "model.layers.78.mlp.experts.139.down_proj", + "model.layers.78.mlp.experts.139.gate_proj", + "model.layers.78.mlp.experts.139.up_proj", + "model.layers.78.mlp.experts.14.down_proj", + "model.layers.78.mlp.experts.14.gate_proj", + "model.layers.78.mlp.experts.14.up_proj", + "model.layers.78.mlp.experts.140.down_proj", + "model.layers.78.mlp.experts.140.gate_proj", + "model.layers.78.mlp.experts.140.up_proj", + "model.layers.78.mlp.experts.141.down_proj", + "model.layers.78.mlp.experts.141.gate_proj", + "model.layers.78.mlp.experts.141.up_proj", + "model.layers.78.mlp.experts.142.down_proj", + "model.layers.78.mlp.experts.142.gate_proj", + "model.layers.78.mlp.experts.142.up_proj", + "model.layers.78.mlp.experts.143.down_proj", + "model.layers.78.mlp.experts.143.gate_proj", + "model.layers.78.mlp.experts.143.up_proj", + "model.layers.78.mlp.experts.144.down_proj", + "model.layers.78.mlp.experts.144.gate_proj", + "model.layers.78.mlp.experts.144.up_proj", + "model.layers.78.mlp.experts.145.down_proj", + "model.layers.78.mlp.experts.145.gate_proj", + "model.layers.78.mlp.experts.145.up_proj", + "model.layers.78.mlp.experts.146.down_proj", + "model.layers.78.mlp.experts.146.gate_proj", + "model.layers.78.mlp.experts.146.up_proj", + "model.layers.78.mlp.experts.147.down_proj", + "model.layers.78.mlp.experts.147.gate_proj", + "model.layers.78.mlp.experts.147.up_proj", + "model.layers.78.mlp.experts.148.down_proj", + "model.layers.78.mlp.experts.148.gate_proj", + "model.layers.78.mlp.experts.148.up_proj", + "model.layers.78.mlp.experts.149.down_proj", + "model.layers.78.mlp.experts.149.gate_proj", + "model.layers.78.mlp.experts.149.up_proj", + "model.layers.78.mlp.experts.15.down_proj", + "model.layers.78.mlp.experts.15.gate_proj", + "model.layers.78.mlp.experts.15.up_proj", + "model.layers.78.mlp.experts.150.down_proj", + "model.layers.78.mlp.experts.150.gate_proj", + "model.layers.78.mlp.experts.150.up_proj", + "model.layers.78.mlp.experts.151.down_proj", + "model.layers.78.mlp.experts.151.gate_proj", + "model.layers.78.mlp.experts.151.up_proj", + "model.layers.78.mlp.experts.152.down_proj", + "model.layers.78.mlp.experts.152.gate_proj", + "model.layers.78.mlp.experts.152.up_proj", + "model.layers.78.mlp.experts.153.down_proj", + "model.layers.78.mlp.experts.153.gate_proj", + "model.layers.78.mlp.experts.153.up_proj", + "model.layers.78.mlp.experts.154.down_proj", + "model.layers.78.mlp.experts.154.gate_proj", + "model.layers.78.mlp.experts.154.up_proj", + "model.layers.78.mlp.experts.155.down_proj", + "model.layers.78.mlp.experts.155.gate_proj", + "model.layers.78.mlp.experts.155.up_proj", + "model.layers.78.mlp.experts.156.down_proj", + "model.layers.78.mlp.experts.156.gate_proj", + "model.layers.78.mlp.experts.156.up_proj", + "model.layers.78.mlp.experts.157.down_proj", + "model.layers.78.mlp.experts.157.gate_proj", + "model.layers.78.mlp.experts.157.up_proj", + "model.layers.78.mlp.experts.158.down_proj", + "model.layers.78.mlp.experts.158.gate_proj", + "model.layers.78.mlp.experts.158.up_proj", + "model.layers.78.mlp.experts.159.down_proj", + "model.layers.78.mlp.experts.159.gate_proj", + "model.layers.78.mlp.experts.159.up_proj", + "model.layers.78.mlp.experts.16.down_proj", + "model.layers.78.mlp.experts.16.gate_proj", + "model.layers.78.mlp.experts.16.up_proj", + "model.layers.78.mlp.experts.160.down_proj", + "model.layers.78.mlp.experts.160.gate_proj", + "model.layers.78.mlp.experts.160.up_proj", + "model.layers.78.mlp.experts.161.down_proj", + "model.layers.78.mlp.experts.161.gate_proj", + "model.layers.78.mlp.experts.161.up_proj", + "model.layers.78.mlp.experts.162.down_proj", + "model.layers.78.mlp.experts.162.gate_proj", + "model.layers.78.mlp.experts.162.up_proj", + "model.layers.78.mlp.experts.163.down_proj", + "model.layers.78.mlp.experts.163.gate_proj", + "model.layers.78.mlp.experts.163.up_proj", + "model.layers.78.mlp.experts.164.down_proj", + "model.layers.78.mlp.experts.164.gate_proj", + "model.layers.78.mlp.experts.164.up_proj", + "model.layers.78.mlp.experts.165.down_proj", + "model.layers.78.mlp.experts.165.gate_proj", + "model.layers.78.mlp.experts.165.up_proj", + "model.layers.78.mlp.experts.166.down_proj", + "model.layers.78.mlp.experts.166.gate_proj", + "model.layers.78.mlp.experts.166.up_proj", + "model.layers.78.mlp.experts.167.down_proj", + "model.layers.78.mlp.experts.167.gate_proj", + "model.layers.78.mlp.experts.167.up_proj", + "model.layers.78.mlp.experts.168.down_proj", + "model.layers.78.mlp.experts.168.gate_proj", + "model.layers.78.mlp.experts.168.up_proj", + "model.layers.78.mlp.experts.169.down_proj", + "model.layers.78.mlp.experts.169.gate_proj", + "model.layers.78.mlp.experts.169.up_proj", + "model.layers.78.mlp.experts.17.down_proj", + "model.layers.78.mlp.experts.17.gate_proj", + "model.layers.78.mlp.experts.17.up_proj", + "model.layers.78.mlp.experts.170.down_proj", + "model.layers.78.mlp.experts.170.gate_proj", + "model.layers.78.mlp.experts.170.up_proj", + "model.layers.78.mlp.experts.171.down_proj", + "model.layers.78.mlp.experts.171.gate_proj", + "model.layers.78.mlp.experts.171.up_proj", + "model.layers.78.mlp.experts.172.down_proj", + "model.layers.78.mlp.experts.172.gate_proj", + "model.layers.78.mlp.experts.172.up_proj", + "model.layers.78.mlp.experts.173.down_proj", + "model.layers.78.mlp.experts.173.gate_proj", + "model.layers.78.mlp.experts.173.up_proj", + "model.layers.78.mlp.experts.174.down_proj", + "model.layers.78.mlp.experts.174.gate_proj", + "model.layers.78.mlp.experts.174.up_proj", + "model.layers.78.mlp.experts.175.down_proj", + "model.layers.78.mlp.experts.175.gate_proj", + "model.layers.78.mlp.experts.175.up_proj", + "model.layers.78.mlp.experts.176.down_proj", + "model.layers.78.mlp.experts.176.gate_proj", + "model.layers.78.mlp.experts.176.up_proj", + "model.layers.78.mlp.experts.177.down_proj", + "model.layers.78.mlp.experts.177.gate_proj", + "model.layers.78.mlp.experts.177.up_proj", + "model.layers.78.mlp.experts.178.down_proj", + "model.layers.78.mlp.experts.178.gate_proj", + "model.layers.78.mlp.experts.178.up_proj", + "model.layers.78.mlp.experts.179.down_proj", + "model.layers.78.mlp.experts.179.gate_proj", + "model.layers.78.mlp.experts.179.up_proj", + "model.layers.78.mlp.experts.18.down_proj", + "model.layers.78.mlp.experts.18.gate_proj", + "model.layers.78.mlp.experts.18.up_proj", + "model.layers.78.mlp.experts.180.down_proj", + "model.layers.78.mlp.experts.180.gate_proj", + "model.layers.78.mlp.experts.180.up_proj", + "model.layers.78.mlp.experts.181.down_proj", + "model.layers.78.mlp.experts.181.gate_proj", + "model.layers.78.mlp.experts.181.up_proj", + "model.layers.78.mlp.experts.182.down_proj", + "model.layers.78.mlp.experts.182.gate_proj", + "model.layers.78.mlp.experts.182.up_proj", + "model.layers.78.mlp.experts.183.down_proj", + "model.layers.78.mlp.experts.183.gate_proj", + "model.layers.78.mlp.experts.183.up_proj", + "model.layers.78.mlp.experts.184.down_proj", + "model.layers.78.mlp.experts.184.gate_proj", + "model.layers.78.mlp.experts.184.up_proj", + "model.layers.78.mlp.experts.185.down_proj", + "model.layers.78.mlp.experts.185.gate_proj", + "model.layers.78.mlp.experts.185.up_proj", + "model.layers.78.mlp.experts.186.down_proj", + "model.layers.78.mlp.experts.186.gate_proj", + "model.layers.78.mlp.experts.186.up_proj", + "model.layers.78.mlp.experts.187.down_proj", + "model.layers.78.mlp.experts.187.gate_proj", + "model.layers.78.mlp.experts.187.up_proj", + "model.layers.78.mlp.experts.188.down_proj", + "model.layers.78.mlp.experts.188.gate_proj", + "model.layers.78.mlp.experts.188.up_proj", + "model.layers.78.mlp.experts.189.down_proj", + "model.layers.78.mlp.experts.189.gate_proj", + "model.layers.78.mlp.experts.189.up_proj", + "model.layers.78.mlp.experts.19.down_proj", + "model.layers.78.mlp.experts.19.gate_proj", + "model.layers.78.mlp.experts.19.up_proj", + "model.layers.78.mlp.experts.190.down_proj", + "model.layers.78.mlp.experts.190.gate_proj", + "model.layers.78.mlp.experts.190.up_proj", + "model.layers.78.mlp.experts.191.down_proj", + "model.layers.78.mlp.experts.191.gate_proj", + "model.layers.78.mlp.experts.191.up_proj", + "model.layers.78.mlp.experts.192.down_proj", + "model.layers.78.mlp.experts.192.gate_proj", + "model.layers.78.mlp.experts.192.up_proj", + "model.layers.78.mlp.experts.193.down_proj", + "model.layers.78.mlp.experts.193.gate_proj", + "model.layers.78.mlp.experts.193.up_proj", + "model.layers.78.mlp.experts.194.down_proj", + "model.layers.78.mlp.experts.194.gate_proj", + "model.layers.78.mlp.experts.194.up_proj", + "model.layers.78.mlp.experts.195.down_proj", + "model.layers.78.mlp.experts.195.gate_proj", + "model.layers.78.mlp.experts.195.up_proj", + "model.layers.78.mlp.experts.196.down_proj", + "model.layers.78.mlp.experts.196.gate_proj", + "model.layers.78.mlp.experts.196.up_proj", + "model.layers.78.mlp.experts.197.down_proj", + "model.layers.78.mlp.experts.197.gate_proj", + "model.layers.78.mlp.experts.197.up_proj", + "model.layers.78.mlp.experts.198.down_proj", + "model.layers.78.mlp.experts.198.gate_proj", + "model.layers.78.mlp.experts.198.up_proj", + "model.layers.78.mlp.experts.199.down_proj", + "model.layers.78.mlp.experts.199.gate_proj", + "model.layers.78.mlp.experts.199.up_proj", + "model.layers.78.mlp.experts.2.down_proj", + "model.layers.78.mlp.experts.2.gate_proj", + "model.layers.78.mlp.experts.2.up_proj", + "model.layers.78.mlp.experts.20.down_proj", + "model.layers.78.mlp.experts.20.gate_proj", + "model.layers.78.mlp.experts.20.up_proj", + "model.layers.78.mlp.experts.200.down_proj", + "model.layers.78.mlp.experts.200.gate_proj", + "model.layers.78.mlp.experts.200.up_proj", + "model.layers.78.mlp.experts.201.down_proj", + "model.layers.78.mlp.experts.201.gate_proj", + "model.layers.78.mlp.experts.201.up_proj", + "model.layers.78.mlp.experts.202.down_proj", + "model.layers.78.mlp.experts.202.gate_proj", + "model.layers.78.mlp.experts.202.up_proj", + "model.layers.78.mlp.experts.203.down_proj", + "model.layers.78.mlp.experts.203.gate_proj", + "model.layers.78.mlp.experts.203.up_proj", + "model.layers.78.mlp.experts.204.down_proj", + "model.layers.78.mlp.experts.204.gate_proj", + "model.layers.78.mlp.experts.204.up_proj", + "model.layers.78.mlp.experts.205.down_proj", + "model.layers.78.mlp.experts.205.gate_proj", + "model.layers.78.mlp.experts.205.up_proj", + "model.layers.78.mlp.experts.206.down_proj", + "model.layers.78.mlp.experts.206.gate_proj", + "model.layers.78.mlp.experts.206.up_proj", + "model.layers.78.mlp.experts.207.down_proj", + "model.layers.78.mlp.experts.207.gate_proj", + "model.layers.78.mlp.experts.207.up_proj", + "model.layers.78.mlp.experts.208.down_proj", + "model.layers.78.mlp.experts.208.gate_proj", + "model.layers.78.mlp.experts.208.up_proj", + "model.layers.78.mlp.experts.209.down_proj", + "model.layers.78.mlp.experts.209.gate_proj", + "model.layers.78.mlp.experts.209.up_proj", + "model.layers.78.mlp.experts.21.down_proj", + "model.layers.78.mlp.experts.21.gate_proj", + "model.layers.78.mlp.experts.21.up_proj", + "model.layers.78.mlp.experts.210.down_proj", + "model.layers.78.mlp.experts.210.gate_proj", + "model.layers.78.mlp.experts.210.up_proj", + "model.layers.78.mlp.experts.211.down_proj", + "model.layers.78.mlp.experts.211.gate_proj", + "model.layers.78.mlp.experts.211.up_proj", + "model.layers.78.mlp.experts.212.down_proj", + "model.layers.78.mlp.experts.212.gate_proj", + "model.layers.78.mlp.experts.212.up_proj", + "model.layers.78.mlp.experts.213.down_proj", + "model.layers.78.mlp.experts.213.gate_proj", + "model.layers.78.mlp.experts.213.up_proj", + "model.layers.78.mlp.experts.214.down_proj", + "model.layers.78.mlp.experts.214.gate_proj", + "model.layers.78.mlp.experts.214.up_proj", + "model.layers.78.mlp.experts.215.down_proj", + "model.layers.78.mlp.experts.215.gate_proj", + "model.layers.78.mlp.experts.215.up_proj", + "model.layers.78.mlp.experts.216.down_proj", + "model.layers.78.mlp.experts.216.gate_proj", + "model.layers.78.mlp.experts.216.up_proj", + "model.layers.78.mlp.experts.217.down_proj", + "model.layers.78.mlp.experts.217.gate_proj", + "model.layers.78.mlp.experts.217.up_proj", + "model.layers.78.mlp.experts.218.down_proj", + "model.layers.78.mlp.experts.218.gate_proj", + "model.layers.78.mlp.experts.218.up_proj", + "model.layers.78.mlp.experts.219.down_proj", + "model.layers.78.mlp.experts.219.gate_proj", + "model.layers.78.mlp.experts.219.up_proj", + "model.layers.78.mlp.experts.22.down_proj", + "model.layers.78.mlp.experts.22.gate_proj", + "model.layers.78.mlp.experts.22.up_proj", + "model.layers.78.mlp.experts.220.down_proj", + "model.layers.78.mlp.experts.220.gate_proj", + "model.layers.78.mlp.experts.220.up_proj", + "model.layers.78.mlp.experts.221.down_proj", + "model.layers.78.mlp.experts.221.gate_proj", + "model.layers.78.mlp.experts.221.up_proj", + "model.layers.78.mlp.experts.222.down_proj", + "model.layers.78.mlp.experts.222.gate_proj", + "model.layers.78.mlp.experts.222.up_proj", + "model.layers.78.mlp.experts.223.down_proj", + "model.layers.78.mlp.experts.223.gate_proj", + "model.layers.78.mlp.experts.223.up_proj", + "model.layers.78.mlp.experts.224.down_proj", + "model.layers.78.mlp.experts.224.gate_proj", + "model.layers.78.mlp.experts.224.up_proj", + "model.layers.78.mlp.experts.225.down_proj", + "model.layers.78.mlp.experts.225.gate_proj", + "model.layers.78.mlp.experts.225.up_proj", + "model.layers.78.mlp.experts.226.down_proj", + "model.layers.78.mlp.experts.226.gate_proj", + "model.layers.78.mlp.experts.226.up_proj", + "model.layers.78.mlp.experts.227.down_proj", + "model.layers.78.mlp.experts.227.gate_proj", + "model.layers.78.mlp.experts.227.up_proj", + "model.layers.78.mlp.experts.228.down_proj", + "model.layers.78.mlp.experts.228.gate_proj", + "model.layers.78.mlp.experts.228.up_proj", + "model.layers.78.mlp.experts.229.down_proj", + "model.layers.78.mlp.experts.229.gate_proj", + "model.layers.78.mlp.experts.229.up_proj", + "model.layers.78.mlp.experts.23.down_proj", + "model.layers.78.mlp.experts.23.gate_proj", + "model.layers.78.mlp.experts.23.up_proj", + "model.layers.78.mlp.experts.230.down_proj", + "model.layers.78.mlp.experts.230.gate_proj", + "model.layers.78.mlp.experts.230.up_proj", + "model.layers.78.mlp.experts.231.down_proj", + "model.layers.78.mlp.experts.231.gate_proj", + "model.layers.78.mlp.experts.231.up_proj", + "model.layers.78.mlp.experts.232.down_proj", + "model.layers.78.mlp.experts.232.gate_proj", + "model.layers.78.mlp.experts.232.up_proj", + "model.layers.78.mlp.experts.233.down_proj", + "model.layers.78.mlp.experts.233.gate_proj", + "model.layers.78.mlp.experts.233.up_proj", + "model.layers.78.mlp.experts.234.down_proj", + "model.layers.78.mlp.experts.234.gate_proj", + "model.layers.78.mlp.experts.234.up_proj", + "model.layers.78.mlp.experts.235.down_proj", + "model.layers.78.mlp.experts.235.gate_proj", + "model.layers.78.mlp.experts.235.up_proj", + "model.layers.78.mlp.experts.236.down_proj", + "model.layers.78.mlp.experts.236.gate_proj", + "model.layers.78.mlp.experts.236.up_proj", + "model.layers.78.mlp.experts.237.down_proj", + "model.layers.78.mlp.experts.237.gate_proj", + "model.layers.78.mlp.experts.237.up_proj", + "model.layers.78.mlp.experts.238.down_proj", + "model.layers.78.mlp.experts.238.gate_proj", + "model.layers.78.mlp.experts.238.up_proj", + "model.layers.78.mlp.experts.239.down_proj", + "model.layers.78.mlp.experts.239.gate_proj", + "model.layers.78.mlp.experts.239.up_proj", + "model.layers.78.mlp.experts.24.down_proj", + "model.layers.78.mlp.experts.24.gate_proj", + "model.layers.78.mlp.experts.24.up_proj", + "model.layers.78.mlp.experts.240.down_proj", + "model.layers.78.mlp.experts.240.gate_proj", + "model.layers.78.mlp.experts.240.up_proj", + "model.layers.78.mlp.experts.241.down_proj", + "model.layers.78.mlp.experts.241.gate_proj", + "model.layers.78.mlp.experts.241.up_proj", + "model.layers.78.mlp.experts.242.down_proj", + "model.layers.78.mlp.experts.242.gate_proj", + "model.layers.78.mlp.experts.242.up_proj", + "model.layers.78.mlp.experts.243.down_proj", + "model.layers.78.mlp.experts.243.gate_proj", + "model.layers.78.mlp.experts.243.up_proj", + "model.layers.78.mlp.experts.244.down_proj", + "model.layers.78.mlp.experts.244.gate_proj", + "model.layers.78.mlp.experts.244.up_proj", + "model.layers.78.mlp.experts.245.down_proj", + "model.layers.78.mlp.experts.245.gate_proj", + "model.layers.78.mlp.experts.245.up_proj", + "model.layers.78.mlp.experts.246.down_proj", + "model.layers.78.mlp.experts.246.gate_proj", + "model.layers.78.mlp.experts.246.up_proj", + "model.layers.78.mlp.experts.247.down_proj", + "model.layers.78.mlp.experts.247.gate_proj", + "model.layers.78.mlp.experts.247.up_proj", + "model.layers.78.mlp.experts.248.down_proj", + "model.layers.78.mlp.experts.248.gate_proj", + "model.layers.78.mlp.experts.248.up_proj", + "model.layers.78.mlp.experts.249.down_proj", + "model.layers.78.mlp.experts.249.gate_proj", + "model.layers.78.mlp.experts.249.up_proj", + "model.layers.78.mlp.experts.25.down_proj", + "model.layers.78.mlp.experts.25.gate_proj", + "model.layers.78.mlp.experts.25.up_proj", + "model.layers.78.mlp.experts.250.down_proj", + "model.layers.78.mlp.experts.250.gate_proj", + "model.layers.78.mlp.experts.250.up_proj", + "model.layers.78.mlp.experts.251.down_proj", + "model.layers.78.mlp.experts.251.gate_proj", + "model.layers.78.mlp.experts.251.up_proj", + "model.layers.78.mlp.experts.252.down_proj", + "model.layers.78.mlp.experts.252.gate_proj", + "model.layers.78.mlp.experts.252.up_proj", + "model.layers.78.mlp.experts.253.down_proj", + "model.layers.78.mlp.experts.253.gate_proj", + "model.layers.78.mlp.experts.253.up_proj", + "model.layers.78.mlp.experts.254.down_proj", + "model.layers.78.mlp.experts.254.gate_proj", + "model.layers.78.mlp.experts.254.up_proj", + "model.layers.78.mlp.experts.255.down_proj", + "model.layers.78.mlp.experts.255.gate_proj", + "model.layers.78.mlp.experts.255.up_proj", + "model.layers.78.mlp.experts.26.down_proj", + "model.layers.78.mlp.experts.26.gate_proj", + "model.layers.78.mlp.experts.26.up_proj", + "model.layers.78.mlp.experts.27.down_proj", + "model.layers.78.mlp.experts.27.gate_proj", + "model.layers.78.mlp.experts.27.up_proj", + "model.layers.78.mlp.experts.28.down_proj", + "model.layers.78.mlp.experts.28.gate_proj", + "model.layers.78.mlp.experts.28.up_proj", + "model.layers.78.mlp.experts.29.down_proj", + "model.layers.78.mlp.experts.29.gate_proj", + "model.layers.78.mlp.experts.29.up_proj", + "model.layers.78.mlp.experts.3.down_proj", + "model.layers.78.mlp.experts.3.gate_proj", + "model.layers.78.mlp.experts.3.up_proj", + "model.layers.78.mlp.experts.30.down_proj", + "model.layers.78.mlp.experts.30.gate_proj", + "model.layers.78.mlp.experts.30.up_proj", + "model.layers.78.mlp.experts.31.down_proj", + "model.layers.78.mlp.experts.31.gate_proj", + "model.layers.78.mlp.experts.31.up_proj", + "model.layers.78.mlp.experts.32.down_proj", + "model.layers.78.mlp.experts.32.gate_proj", + "model.layers.78.mlp.experts.32.up_proj", + "model.layers.78.mlp.experts.33.down_proj", + "model.layers.78.mlp.experts.33.gate_proj", + "model.layers.78.mlp.experts.33.up_proj", + "model.layers.78.mlp.experts.34.down_proj", + "model.layers.78.mlp.experts.34.gate_proj", + "model.layers.78.mlp.experts.34.up_proj", + "model.layers.78.mlp.experts.35.down_proj", + "model.layers.78.mlp.experts.35.gate_proj", + "model.layers.78.mlp.experts.35.up_proj", + "model.layers.78.mlp.experts.36.down_proj", + "model.layers.78.mlp.experts.36.gate_proj", + "model.layers.78.mlp.experts.36.up_proj", + "model.layers.78.mlp.experts.37.down_proj", + "model.layers.78.mlp.experts.37.gate_proj", + "model.layers.78.mlp.experts.37.up_proj", + "model.layers.78.mlp.experts.38.down_proj", + "model.layers.78.mlp.experts.38.gate_proj", + "model.layers.78.mlp.experts.38.up_proj", + "model.layers.78.mlp.experts.39.down_proj", + "model.layers.78.mlp.experts.39.gate_proj", + "model.layers.78.mlp.experts.39.up_proj", + "model.layers.78.mlp.experts.4.down_proj", + "model.layers.78.mlp.experts.4.gate_proj", + "model.layers.78.mlp.experts.4.up_proj", + "model.layers.78.mlp.experts.40.down_proj", + "model.layers.78.mlp.experts.40.gate_proj", + "model.layers.78.mlp.experts.40.up_proj", + "model.layers.78.mlp.experts.41.down_proj", + "model.layers.78.mlp.experts.41.gate_proj", + "model.layers.78.mlp.experts.41.up_proj", + "model.layers.78.mlp.experts.42.down_proj", + "model.layers.78.mlp.experts.42.gate_proj", + "model.layers.78.mlp.experts.42.up_proj", + "model.layers.78.mlp.experts.43.down_proj", + "model.layers.78.mlp.experts.43.gate_proj", + "model.layers.78.mlp.experts.43.up_proj", + "model.layers.78.mlp.experts.44.down_proj", + "model.layers.78.mlp.experts.44.gate_proj", + "model.layers.78.mlp.experts.44.up_proj", + "model.layers.78.mlp.experts.45.down_proj", + "model.layers.78.mlp.experts.45.gate_proj", + "model.layers.78.mlp.experts.45.up_proj", + "model.layers.78.mlp.experts.46.down_proj", + "model.layers.78.mlp.experts.46.gate_proj", + "model.layers.78.mlp.experts.46.up_proj", + "model.layers.78.mlp.experts.47.down_proj", + "model.layers.78.mlp.experts.47.gate_proj", + "model.layers.78.mlp.experts.47.up_proj", + "model.layers.78.mlp.experts.48.down_proj", + "model.layers.78.mlp.experts.48.gate_proj", + "model.layers.78.mlp.experts.48.up_proj", + "model.layers.78.mlp.experts.49.down_proj", + "model.layers.78.mlp.experts.49.gate_proj", + "model.layers.78.mlp.experts.49.up_proj", + "model.layers.78.mlp.experts.5.down_proj", + "model.layers.78.mlp.experts.5.gate_proj", + "model.layers.78.mlp.experts.5.up_proj", + "model.layers.78.mlp.experts.50.down_proj", + "model.layers.78.mlp.experts.50.gate_proj", + "model.layers.78.mlp.experts.50.up_proj", + "model.layers.78.mlp.experts.51.down_proj", + "model.layers.78.mlp.experts.51.gate_proj", + "model.layers.78.mlp.experts.51.up_proj", + "model.layers.78.mlp.experts.52.down_proj", + "model.layers.78.mlp.experts.52.gate_proj", + "model.layers.78.mlp.experts.52.up_proj", + "model.layers.78.mlp.experts.53.down_proj", + "model.layers.78.mlp.experts.53.gate_proj", + "model.layers.78.mlp.experts.53.up_proj", + "model.layers.78.mlp.experts.54.down_proj", + "model.layers.78.mlp.experts.54.gate_proj", + "model.layers.78.mlp.experts.54.up_proj", + "model.layers.78.mlp.experts.55.down_proj", + "model.layers.78.mlp.experts.55.gate_proj", + "model.layers.78.mlp.experts.55.up_proj", + "model.layers.78.mlp.experts.56.down_proj", + "model.layers.78.mlp.experts.56.gate_proj", + "model.layers.78.mlp.experts.56.up_proj", + "model.layers.78.mlp.experts.57.down_proj", + "model.layers.78.mlp.experts.57.gate_proj", + "model.layers.78.mlp.experts.57.up_proj", + "model.layers.78.mlp.experts.58.down_proj", + "model.layers.78.mlp.experts.58.gate_proj", + "model.layers.78.mlp.experts.58.up_proj", + "model.layers.78.mlp.experts.59.down_proj", + "model.layers.78.mlp.experts.59.gate_proj", + "model.layers.78.mlp.experts.59.up_proj", + "model.layers.78.mlp.experts.6.down_proj", + "model.layers.78.mlp.experts.6.gate_proj", + "model.layers.78.mlp.experts.6.up_proj", + "model.layers.78.mlp.experts.60.down_proj", + "model.layers.78.mlp.experts.60.gate_proj", + "model.layers.78.mlp.experts.60.up_proj", + "model.layers.78.mlp.experts.61.down_proj", + "model.layers.78.mlp.experts.61.gate_proj", + "model.layers.78.mlp.experts.61.up_proj", + "model.layers.78.mlp.experts.62.down_proj", + "model.layers.78.mlp.experts.62.gate_proj", + "model.layers.78.mlp.experts.62.up_proj", + "model.layers.78.mlp.experts.63.down_proj", + "model.layers.78.mlp.experts.63.gate_proj", + "model.layers.78.mlp.experts.63.up_proj", + "model.layers.78.mlp.experts.64.down_proj", + "model.layers.78.mlp.experts.64.gate_proj", + "model.layers.78.mlp.experts.64.up_proj", + "model.layers.78.mlp.experts.65.down_proj", + "model.layers.78.mlp.experts.65.gate_proj", + "model.layers.78.mlp.experts.65.up_proj", + "model.layers.78.mlp.experts.66.down_proj", + "model.layers.78.mlp.experts.66.gate_proj", + "model.layers.78.mlp.experts.66.up_proj", + "model.layers.78.mlp.experts.67.down_proj", + "model.layers.78.mlp.experts.67.gate_proj", + "model.layers.78.mlp.experts.67.up_proj", + "model.layers.78.mlp.experts.68.down_proj", + "model.layers.78.mlp.experts.68.gate_proj", + "model.layers.78.mlp.experts.68.up_proj", + "model.layers.78.mlp.experts.69.down_proj", + "model.layers.78.mlp.experts.69.gate_proj", + "model.layers.78.mlp.experts.69.up_proj", + "model.layers.78.mlp.experts.7.down_proj", + "model.layers.78.mlp.experts.7.gate_proj", + "model.layers.78.mlp.experts.7.up_proj", + "model.layers.78.mlp.experts.70.down_proj", + "model.layers.78.mlp.experts.70.gate_proj", + "model.layers.78.mlp.experts.70.up_proj", + "model.layers.78.mlp.experts.71.down_proj", + "model.layers.78.mlp.experts.71.gate_proj", + "model.layers.78.mlp.experts.71.up_proj", + "model.layers.78.mlp.experts.72.down_proj", + "model.layers.78.mlp.experts.72.gate_proj", + "model.layers.78.mlp.experts.72.up_proj", + "model.layers.78.mlp.experts.73.down_proj", + "model.layers.78.mlp.experts.73.gate_proj", + "model.layers.78.mlp.experts.73.up_proj", + "model.layers.78.mlp.experts.74.down_proj", + "model.layers.78.mlp.experts.74.gate_proj", + "model.layers.78.mlp.experts.74.up_proj", + "model.layers.78.mlp.experts.75.down_proj", + "model.layers.78.mlp.experts.75.gate_proj", + "model.layers.78.mlp.experts.75.up_proj", + "model.layers.78.mlp.experts.76.down_proj", + "model.layers.78.mlp.experts.76.gate_proj", + "model.layers.78.mlp.experts.76.up_proj", + "model.layers.78.mlp.experts.77.down_proj", + "model.layers.78.mlp.experts.77.gate_proj", + "model.layers.78.mlp.experts.77.up_proj", + "model.layers.78.mlp.experts.78.down_proj", + "model.layers.78.mlp.experts.78.gate_proj", + "model.layers.78.mlp.experts.78.up_proj", + "model.layers.78.mlp.experts.79.down_proj", + "model.layers.78.mlp.experts.79.gate_proj", + "model.layers.78.mlp.experts.79.up_proj", + "model.layers.78.mlp.experts.8.down_proj", + "model.layers.78.mlp.experts.8.gate_proj", + "model.layers.78.mlp.experts.8.up_proj", + "model.layers.78.mlp.experts.80.down_proj", + "model.layers.78.mlp.experts.80.gate_proj", + "model.layers.78.mlp.experts.80.up_proj", + "model.layers.78.mlp.experts.81.down_proj", + "model.layers.78.mlp.experts.81.gate_proj", + "model.layers.78.mlp.experts.81.up_proj", + "model.layers.78.mlp.experts.82.down_proj", + "model.layers.78.mlp.experts.82.gate_proj", + "model.layers.78.mlp.experts.82.up_proj", + "model.layers.78.mlp.experts.83.down_proj", + "model.layers.78.mlp.experts.83.gate_proj", + "model.layers.78.mlp.experts.83.up_proj", + "model.layers.78.mlp.experts.84.down_proj", + "model.layers.78.mlp.experts.84.gate_proj", + "model.layers.78.mlp.experts.84.up_proj", + "model.layers.78.mlp.experts.85.down_proj", + "model.layers.78.mlp.experts.85.gate_proj", + "model.layers.78.mlp.experts.85.up_proj", + "model.layers.78.mlp.experts.86.down_proj", + "model.layers.78.mlp.experts.86.gate_proj", + "model.layers.78.mlp.experts.86.up_proj", + "model.layers.78.mlp.experts.87.down_proj", + "model.layers.78.mlp.experts.87.gate_proj", + "model.layers.78.mlp.experts.87.up_proj", + "model.layers.78.mlp.experts.88.down_proj", + "model.layers.78.mlp.experts.88.gate_proj", + "model.layers.78.mlp.experts.88.up_proj", + "model.layers.78.mlp.experts.89.down_proj", + "model.layers.78.mlp.experts.89.gate_proj", + "model.layers.78.mlp.experts.89.up_proj", + "model.layers.78.mlp.experts.9.down_proj", + "model.layers.78.mlp.experts.9.gate_proj", + "model.layers.78.mlp.experts.9.up_proj", + "model.layers.78.mlp.experts.90.down_proj", + "model.layers.78.mlp.experts.90.gate_proj", + "model.layers.78.mlp.experts.90.up_proj", + "model.layers.78.mlp.experts.91.down_proj", + "model.layers.78.mlp.experts.91.gate_proj", + "model.layers.78.mlp.experts.91.up_proj", + "model.layers.78.mlp.experts.92.down_proj", + "model.layers.78.mlp.experts.92.gate_proj", + "model.layers.78.mlp.experts.92.up_proj", + "model.layers.78.mlp.experts.93.down_proj", + "model.layers.78.mlp.experts.93.gate_proj", + "model.layers.78.mlp.experts.93.up_proj", + "model.layers.78.mlp.experts.94.down_proj", + "model.layers.78.mlp.experts.94.gate_proj", + "model.layers.78.mlp.experts.94.up_proj", + "model.layers.78.mlp.experts.95.down_proj", + "model.layers.78.mlp.experts.95.gate_proj", + "model.layers.78.mlp.experts.95.up_proj", + "model.layers.78.mlp.experts.96.down_proj", + "model.layers.78.mlp.experts.96.gate_proj", + "model.layers.78.mlp.experts.96.up_proj", + "model.layers.78.mlp.experts.97.down_proj", + "model.layers.78.mlp.experts.97.gate_proj", + "model.layers.78.mlp.experts.97.up_proj", + "model.layers.78.mlp.experts.98.down_proj", + "model.layers.78.mlp.experts.98.gate_proj", + "model.layers.78.mlp.experts.98.up_proj", + "model.layers.78.mlp.experts.99.down_proj", + "model.layers.78.mlp.experts.99.gate_proj", + "model.layers.78.mlp.experts.99.up_proj", + "model.layers.78.mlp.gate", + "model.layers.78.mlp.shared_experts.down_proj", + "model.layers.78.mlp.shared_experts.gate_proj", + "model.layers.78.mlp.shared_experts.up_proj", + "model.layers.78.self_attn.indexer.weights_proj", + "model.layers.78.self_attn.indexer.wk", + "model.layers.78.self_attn.indexer.wq_b", + "model.layers.78.self_attn.kv_a_proj_with_mqa", + "model.layers.78.self_attn.kv_b_proj", + "model.layers.78.self_attn.o_proj", + "model.layers.78.self_attn.q_a_proj", + "model.layers.78.self_attn.q_b_proj", + "model.layers.78.self_attn.fused_qkv_a_proj" + ], + "kv_cache_scheme": null, + "quant_method": "compressed-tensors", + "quantization_status": "compressed", + "sparsity_config": {}, + "transform_config": {}, + "version": "0.1.dev537+g1cee215" + }, + "rms_norm_eps": 1e-05, + "rope_interleave": true, + "rope_parameters": { + "rope_theta": 8000000, + "rope_type": "default" + }, + "routed_scaling_factor": 2.5, + "scoring_func": "sigmoid", + "tie_word_embeddings": false, + "topk_group": 1, + "topk_method": "noaux_tc", + "transformers_version": "5.16.0.dev0", + "use_cache": true, + "v_head_dim": 256, + "vocab_size": 154880 +} \ No newline at end of file diff --git a/generation_config.json b/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..626fd861c588006b81e2fdc84298a76f4ab324e0 --- /dev/null +++ b/generation_config.json @@ -0,0 +1,13 @@ +{ + "_from_model_config": true, + "do_sample": true, + "eos_token_id": [ + 154820, + 154827, + 154829 + ], + "pad_token_id": 154820, + "temperature": 1.0, + "top_p": 0.95, + "transformers_version": "5.16.0.dev0" +} diff --git a/model-00001.safetensors b/model-00001.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c69c828e43d1d8ce216e49dd6dac63ab98ec7f23 --- /dev/null +++ b/model-00001.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8e69135e3dfd9f6dae38c5fc74adce38c15a74a54441794b308e8719873ae0ed +size 8595006928 diff --git a/model-00002.safetensors b/model-00002.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..17391635a061235ac1f57ca7898d0626b0eb8e30 --- /dev/null +++ b/model-00002.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:aa00b52978946d0ced7aaa24fa4efd81942da3895b278a2559f7093418af0223 +size 8692606664 diff --git a/model-00003.safetensors b/model-00003.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e55d1d931d101edb1efb7006283b8bba6569b1fd --- /dev/null +++ b/model-00003.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:61b77806c7716ff1b17f4df4b0c7a9c3a44bf3efb71a2e6fb40c8e8a9b39a784 +size 8600230816 diff --git a/model-00004.safetensors b/model-00004.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4152fb981c915a4a14c11cde94df2ad4662bbd52 --- /dev/null +++ b/model-00004.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:44b1bf3c17950bc1ded6c87f0ad7e92e80078c85ee105da9871cf43e8ffab54f +size 8606754392 diff --git a/model-00005.safetensors b/model-00005.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..42b76b7137da1e0bc120d5e68b3b8fe8c65fb49d --- /dev/null +++ b/model-00005.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b38d7f1e20997934c24c5095d516b882ec07fa3433885cf85f73aa0737a1aaca +size 8591724016 diff --git a/model-00006.safetensors b/model-00006.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4d21325139efb5afb3fba94b7ccd881a19fcd107 --- /dev/null +++ b/model-00006.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:59a48f2991c5d7741e954deec27f560d91f132fc8b0dd6ff0ff4b58e98f8b285 +size 8594866696 diff --git a/model-00007.safetensors b/model-00007.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..219f744ea6bc05b17156d32cb417ce2a53712398 --- /dev/null +++ b/model-00007.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cfd98ec4282c630ad3c1fa008024f903102e368ad557ee6fa1f6912fc1f3d1f4 +size 8594866808 diff --git a/model-00008.safetensors b/model-00008.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..cda18b3e9610e83c327b02315d52d622412592fd --- /dev/null +++ b/model-00008.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1a67f933cf739ee393f14ace8a6c0cf2528288fe45899c5f7429fe9363d92e9b +size 8593585472 diff --git a/model-00009.safetensors b/model-00009.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..3d51dd4e6526f5f35e41071219de27b098b40d20 --- /dev/null +++ b/model-00009.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:50062f4b71e47911d1321adc40fe5f266b7ba2fd30ba998b03b9b2449d0078b1 +size 8593194288 diff --git a/model-00010.safetensors b/model-00010.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e5f270cefba6c5b023f7b67cdc41500d1a5f2ace --- /dev/null +++ b/model-00010.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e4992f30a7bbfa38f5c799cfeae8e2d8cf65a04cec64893a6e79eba7b2aa784 +size 8596869448 diff --git a/model-00011.safetensors b/model-00011.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..3e001528675abee5068032db5ed3ff7ff728cc9e --- /dev/null +++ b/model-00011.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cc0c91cc2a8aae7ee925912a79503be2a553af11b2f9a766d00bf6aa7a29e478 +size 8594866704 diff --git a/model-00012.safetensors b/model-00012.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..54e2d90021689fbf916c444f1c526ff39691d6c8 --- /dev/null +++ b/model-00012.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c2f09703fbfadde14e1f3f689242154f86b7277968e7a08ca30b8fa0fc986ec9 +size 8593782824 diff --git a/model-00013.safetensors b/model-00013.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..41e4289b51bc7bbb0a86bc375baf92222cf27509 --- /dev/null +++ b/model-00013.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6b087fa344da1261b61185d0463cb9a0053aaeaa8e0c69cb881f3a43b9500f4e +size 8590832064 diff --git a/model-00014.safetensors b/model-00014.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..bcd6854eaafbff59991324a7b3f087246158bebd --- /dev/null +++ b/model-00014.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7aa8804441cc35c028ac1bb22e622e9e3fc6232ba5ea4e0cab70c6553a2b262e +size 8591966280 diff --git a/model-00015.safetensors b/model-00015.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1db517fddf90dff138aa8ec6540a94c7253cfee1 --- /dev/null +++ b/model-00015.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:01a34c98d21580fb7360cda568814f0f8c496773e9f0ebd23881cdbf86315f92 +size 8594866808 diff --git a/model-00016.safetensors b/model-00016.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..34f8a84b6fb189ed4e821c0327a18408dfed7740 --- /dev/null +++ b/model-00016.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cb087fc9fa324030e79cb5ef608a977cb6008e18b6ae57427ddc6d7c4747d9d9 +size 8594866728 diff --git a/model-00017.safetensors b/model-00017.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1c60e36599f324d0a9ab3d5a50d086d78cdcd3d8 --- /dev/null +++ b/model-00017.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4ca8e936d734185ff5577b4f09d4cbfca1789e3f233d49ca79ad616d0a484622 +size 8590418576 diff --git a/model-00018.safetensors b/model-00018.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..ec92da23b048ea17352e851f5cafef18c294dbb0 --- /dev/null +++ b/model-00018.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ea638f35c68c0d95115e5c99da9d6da4c1a5b68f2f8d5d82ee061e830b64f8ef +size 8590695768 diff --git a/model-00019.safetensors b/model-00019.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..332199581c5d2604f8dd8b26f211835f973a7587 --- /dev/null +++ b/model-00019.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7c0dbe3874327609ff2dbebe53352d11bf6d4471f310017e1fe85e196146c796 +size 8596799920 diff --git a/model-00020.safetensors b/model-00020.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..16ac8dd63b30635d54877f023ee99c215fe88db4 --- /dev/null +++ b/model-00020.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cdd108961b4be3ad7476f25db1b4aa6d6f3c43469404c349cff0eae33ddf1993 +size 8594866824 diff --git a/model-00021.safetensors b/model-00021.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9b87185ba356f166790f4b6df4143f1bd8bd1d3f --- /dev/null +++ b/model-00021.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:809c5ff00c8af0966021c07a56352ab361477b2a9a7a83e7f8980cf05fd53707 +size 8594866784 diff --git a/model-00022.safetensors b/model-00022.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..45d0fc940c44b23ec50c07d231a4788e124c7930 --- /dev/null +++ b/model-00022.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:551e50f7570c0e4c9630216a6aa2c5ffac95ddd027800c97dc71cae749810bdf +size 8591160696 diff --git a/model-00023.safetensors b/model-00023.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..aafbad56606671c00949a74957c5a4cc5d76340f --- /dev/null +++ b/model-00023.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:912897e6ab49b9bdd040d2e6dd59f23e8f1ca9b710b268c71dd2201e1d8b4c34 +size 8596054424 diff --git a/model-00024.safetensors b/model-00024.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..7916750924cde3d6f3a29ba1bbc7c40ef8c31917 --- /dev/null +++ b/model-00024.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e9f664dd1f7a84683497d256e9d98dae480286fd894e5651eed568d96ab3f8d6 +size 8593782824 diff --git a/model-00025.safetensors b/model-00025.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..edd61c9c6e7b33f97a6fffb29493e3879bdb1ac3 --- /dev/null +++ b/model-00025.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f8db519f3ef3fbbc622e118cfbfa37fc38b97ecfc50d9b4a4ef52987b145bc24 +size 8594866808 diff --git a/model-00026.safetensors b/model-00026.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e1f44e4562699cdac1336be87f4867d3e7804807 --- /dev/null +++ b/model-00026.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:549a322be812e4706896fd86efd2230cfa4169f337655301c3bb48d0d075ad4c +size 8594189520 diff --git a/model-00027.safetensors b/model-00027.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c7983a537a40b8704bef30d7db6e2902d5b3c00f --- /dev/null +++ b/model-00027.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:941f6898bf1d74d04ee41b07d3f1222860ff36e9e138e1bec7d59ec5f37b04fa +size 8594506760 diff --git a/model-00028.safetensors b/model-00028.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..960588f827ed27220415214b296b4c9145c6eb27 --- /dev/null +++ b/model-00028.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:eed316d92934d98a733601ae9a4ef046407b968557f3fc98623aab5d93f37ab9 +size 8594866592 diff --git a/model-00029.safetensors b/model-00029.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b54d01327e786cc13481f730981b71cee2b3ae76 --- /dev/null +++ b/model-00029.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b298b0702152433374fac7367afcc92955a4a9e08ebb9b229bcf84bd427731e5 +size 8593782816 diff --git a/model-00030.safetensors b/model-00030.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..145e05cab0621c8edb3c8df6641aa2a31fb30c1a --- /dev/null +++ b/model-00030.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5b75a333fac8279f12f90c0ac4e092b2f8a10261acfaa17f7e0d7eb0f1c8664b +size 8594866824 diff --git a/model-00031.safetensors b/model-00031.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1c392936caf5b27052aceec9a3bcc87f01aecd2b --- /dev/null +++ b/model-00031.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:636705fb45ca4389b633caa3a440834de5df8b565053ae18cb1b932a5a7cb6dc +size 8596428280 diff --git a/model-00032.safetensors b/model-00032.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..18a00e5d1c71ba64fe85115d45a6a66296db6dee --- /dev/null +++ b/model-00032.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a4b1caa7efb22a68d9d616b46201aa4be87900a2577b7e34c3ad9d12a03c0be2 +size 8590982032 diff --git a/model-00033.safetensors b/model-00033.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9b7793382a2068579067db3a910312eb530dd64e --- /dev/null +++ b/model-00033.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:12d4a864bcc5a1208833480baab5dd6d266a3d6fea4dd6f14e5a5002f90e2e8f +size 8594669936 diff --git a/model-00034.safetensors b/model-00034.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1a4d1838e96e6aaaffbef7d58440da77a0f67d93 --- /dev/null +++ b/model-00034.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:173467d6a28ee084562e2c08959fe9a24cfea370366c56492e6c61690178951c +size 8593782816 diff --git a/model-00035.safetensors b/model-00035.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..cb33a990ea0813ff9be1ef69d2b0fc324c58a798 --- /dev/null +++ b/model-00035.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0249c1835f358e4d19d6d9ab7f998c738026ab75ca41e94923322e5372f44fb3 +size 8603796416 diff --git a/model-00036.safetensors b/model-00036.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..dff10f7ced981ead6203972701ac0afc6b7e294f --- /dev/null +++ b/model-00036.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:47240a56be8d7c9323b2e99cf92533ac843f00d773f11fb8c91864598dbeb749 +size 8594883232 diff --git a/model-00037.safetensors b/model-00037.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..bc2984fe107f53f910e286ff7f6b53c7edac37cc --- /dev/null +++ b/model-00037.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b72c1e9d1b90917da67ba00d2ec7a5834e7323d4252afedf6ae215bdf7efdae9 +size 8594866832 diff --git a/model-00038.safetensors b/model-00038.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..14be05ba3248487d02425baf1f68dd36b8564962 --- /dev/null +++ b/model-00038.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e39818dc9a058e8d6091d9739b562e597dbd1a898cd3997222390c1eaa42d8e7 +size 8594866680 diff --git a/model-00039.safetensors b/model-00039.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..2fc8f3b510c161d16ed9d852d02eeede5e2d2c16 --- /dev/null +++ b/model-00039.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3e6a29e4c7d5dd410a8e286e15de8379d29f7d11b3294247eb0b8098b6ee875e +size 8590578600 diff --git a/model-00040.safetensors b/model-00040.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9f3f1bdc8c757db3267f0e96a957e85fe7616035 --- /dev/null +++ b/model-00040.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3d13cb5fac2d5d7da28c37eeabcdadebcf263d49db7c9ca6cd79c34168a6a31a +size 8599473872 diff --git a/model-00041.safetensors b/model-00041.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..8a08a37255d2dc2d63f1e5ce822ace09b2ee89a6 --- /dev/null +++ b/model-00041.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0c8950b4b97794c9952e15d1fd8a7b2dd68798f6512f06406069e2961fa5ce31 +size 8612002216 diff --git a/model-00042.safetensors b/model-00042.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..59c4d43b19e8f05a8765f3a8f529ea48dcd0bfa3 --- /dev/null +++ b/model-00042.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4deb21abc128d4892b932a189682ca646ac219668e8dcd3a5827cf4752635095 +size 8606816048 diff --git a/model-00043.safetensors b/model-00043.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1f628e92c4939fde558d379d3753b0eb19704d02 --- /dev/null +++ b/model-00043.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cc89b21550c7295f5613de22ed6e3ae51e74ae4bc006107e65ce6c9cb3ea44fb +size 8606754800 diff --git a/model-00044.safetensors b/model-00044.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..21ddc9072c8402a0d981f4b28230940eb5b15ed1 --- /dev/null +++ b/model-00044.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4d8d0602518a179f5495f3dd52253075a93a4544ad074f36b786d186fb06a5d5 +size 1936629544 diff --git a/model.safetensors.index.json b/model.safetensors.index.json new file mode 100644 index 0000000000000000000000000000000000000000..4cb09db647a6d5b1d3f83f16b1157301a7a036f9 --- /dev/null +++ b/model.safetensors.index.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3d22f89a6897c8decc1fae7a21e6e0580646ac94d2503ce59ea92f0d2eda1dbe +size 14948995 diff --git a/reap_prune_plan.json b/reap_prune_plan.json new file mode 100644 index 0000000000000000000000000000000000000000..faf160b8b4951b3661a93b0c15a24556410790c7 --- /dev/null +++ b/reap_prune_plan.json @@ -0,0 +1 @@ +{"keep": 192, "criterion": "massmax", "source": "/data/models/GLM-5.3-AWQ-INT4-cyankiwi", "layers": {"3": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 154, 155, 156, 157, 158, 159, 161, 162, 168, 173, 176, 188, 189, 192, 194, 195, 196, 198, 200, 207, 209, 216, 219, 221, 222, 228, 231, 232, 235, 236, 240, 243, 244, 245, 247, 250, 253, 254, 255], "4": [0, 1, 3, 4, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 25, 26, 28, 29, 30, 31, 35, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 50, 51, 53, 54, 55, 56, 57, 58, 61, 62, 63, 64, 66, 67, 68, 69, 70, 71, 72, 73, 74, 76, 77, 79, 80, 81, 82, 85, 86, 87, 88, 89, 91, 92, 93, 96, 98, 99, 100, 101, 102, 104, 105, 107, 109, 110, 112, 113, 115, 116, 117, 118, 119, 120, 122, 123, 125, 126, 127, 128, 132, 133, 134, 135, 136, 137, 139, 140, 141, 143, 144, 145, 146, 147, 148, 149, 150, 152, 154, 155, 157, 159, 160, 162, 164, 166, 168, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 181, 182, 183, 184, 185, 186, 187, 188, 189, 191, 193, 194, 195, 196, 200, 203, 204, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 219, 220, 221, 222, 223, 224, 225, 226, 227, 229, 230, 231, 232, 233, 234, 235, 237, 240, 241, 242, 243, 244, 245, 247, 248, 249, 250, 251, 254, 255], "5": [2, 4, 6, 8, 11, 12, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 32, 33, 34, 36, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 52, 53, 55, 56, 57, 58, 59, 61, 62, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 84, 88, 89, 90, 91, 92, 94, 95, 97, 98, 99, 100, 101, 102, 104, 107, 112, 114, 115, 117, 118, 119, 120, 122, 123, 125, 126, 127, 128, 131, 133, 134, 135, 136, 137, 138, 139, 140, 141, 144, 146, 147, 149, 150, 151, 152, 153, 154, 156, 157, 158, 159, 160, 161, 162, 163, 166, 167, 168, 169, 170, 171, 172, 174, 175, 177, 179, 180, 182, 183, 184, 187, 188, 189, 190, 191, 192, 194, 195, 196, 198, 199, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 216, 217, 218, 219, 220, 221, 222, 223, 225, 226, 227, 229, 232, 233, 234, 235, 236, 238, 239, 240, 241, 242, 244, 245, 246, 247, 248, 249, 250, 251, 254, 255], "6": [1, 2, 3, 4, 6, 8, 10, 11, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 24, 25, 29, 30, 33, 34, 35, 37, 38, 41, 42, 43, 44, 47, 48, 49, 50, 51, 52, 54, 56, 57, 58, 59, 60, 61, 62, 63, 65, 66, 68, 69, 70, 71, 72, 73, 74, 76, 77, 78, 81, 82, 84, 85, 86, 87, 88, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 103, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 118, 119, 120, 121, 123, 124, 125, 128, 129, 132, 133, 135, 136, 137, 138, 139, 142, 143, 144, 145, 146, 148, 150, 151, 152, 155, 157, 158, 159, 160, 161, 162, 164, 165, 166, 167, 168, 169, 170, 172, 173, 174, 175, 177, 179, 180, 181, 182, 183, 184, 186, 187, 188, 189, 191, 193, 195, 196, 197, 199, 200, 201, 202, 203, 204, 207, 208, 210, 212, 213, 214, 215, 216, 218, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 232, 234, 235, 236, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 251, 253, 255], "7": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 28, 30, 32, 33, 34, 35, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 49, 51, 52, 54, 56, 59, 60, 65, 66, 67, 69, 70, 71, 72, 73, 74, 76, 77, 78, 80, 81, 82, 84, 85, 87, 88, 89, 91, 93, 95, 98, 99, 100, 101, 102, 103, 104, 105, 107, 108, 109, 111, 113, 114, 115, 116, 117, 118, 119, 120, 122, 124, 125, 126, 128, 129, 131, 132, 133, 134, 136, 137, 138, 139, 141, 142, 143, 144, 146, 150, 152, 153, 154, 156, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 171, 172, 173, 175, 176, 177, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 195, 196, 197, 200, 201, 202, 203, 204, 205, 207, 209, 210, 211, 212, 214, 216, 217, 218, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 238, 239, 241, 242, 243, 244, 245, 246, 248, 249, 251, 253, 254, 255], "8": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 19, 20, 22, 23, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 55, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 70, 71, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 90, 91, 93, 94, 95, 96, 97, 98, 99, 102, 103, 104, 105, 106, 107, 109, 110, 111, 112, 113, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 128, 130, 133, 135, 137, 138, 141, 142, 144, 145, 148, 149, 150, 151, 154, 155, 156, 157, 158, 159, 160, 161, 163, 166, 168, 170, 171, 172, 174, 175, 176, 177, 179, 180, 182, 183, 184, 185, 187, 190, 193, 194, 195, 196, 197, 198, 200, 201, 202, 204, 205, 206, 208, 209, 212, 213, 214, 215, 216, 217, 219, 221, 223, 224, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 240, 243, 245, 246, 247, 249, 253, 254, 255], "9": [0, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 17, 18, 20, 21, 24, 25, 26, 27, 28, 29, 30, 31, 34, 36, 37, 39, 40, 41, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 54, 55, 56, 59, 63, 64, 65, 66, 67, 68, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 82, 83, 84, 85, 86, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 101, 102, 105, 106, 109, 110, 112, 113, 115, 117, 118, 119, 120, 121, 123, 125, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 142, 143, 144, 145, 146, 147, 148, 149, 151, 152, 153, 154, 155, 156, 157, 158, 160, 161, 163, 164, 165, 166, 167, 168, 169, 170, 172, 173, 176, 177, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 193, 195, 197, 198, 199, 201, 202, 203, 205, 206, 208, 210, 211, 212, 213, 214, 215, 216, 217, 219, 220, 221, 222, 223, 224, 225, 229, 230, 231, 233, 234, 235, 237, 239, 240, 241, 242, 243, 245, 248, 249, 250, 251, 253, 254], "10": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 18, 19, 20, 22, 23, 24, 25, 27, 29, 30, 31, 32, 33, 34, 36, 37, 38, 40, 41, 44, 45, 47, 48, 49, 50, 51, 53, 55, 56, 57, 58, 59, 60, 61, 63, 64, 66, 67, 71, 75, 76, 79, 80, 81, 82, 84, 87, 88, 89, 91, 92, 93, 94, 95, 96, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 108, 110, 111, 112, 115, 116, 117, 118, 119, 123, 124, 125, 126, 128, 130, 131, 132, 133, 134, 135, 136, 137, 139, 141, 142, 143, 144, 145, 147, 148, 149, 151, 152, 153, 154, 155, 157, 158, 160, 164, 165, 166, 169, 170, 171, 172, 173, 174, 175, 176, 179, 180, 181, 182, 183, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 201, 202, 203, 204, 205, 206, 207, 209, 210, 211, 212, 213, 215, 216, 217, 218, 219, 221, 223, 224, 225, 226, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 241, 243, 244, 245, 246, 248, 249, 251, 252, 253, 254], "11": [1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 37, 39, 40, 41, 44, 45, 46, 47, 48, 50, 51, 52, 53, 54, 55, 56, 58, 60, 61, 62, 63, 64, 65, 67, 68, 70, 71, 72, 73, 74, 79, 80, 81, 84, 85, 86, 88, 89, 90, 92, 93, 94, 97, 98, 101, 102, 104, 105, 106, 108, 109, 110, 111, 112, 113, 114, 115, 118, 119, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 136, 137, 138, 139, 140, 141, 142, 144, 146, 147, 148, 149, 151, 152, 155, 156, 157, 159, 160, 161, 162, 163, 164, 165, 166, 167, 169, 170, 171, 172, 173, 174, 175, 176, 177, 179, 180, 182, 183, 184, 185, 186, 188, 189, 190, 193, 194, 195, 196, 197, 199, 201, 203, 204, 205, 206, 207, 208, 210, 211, 213, 214, 216, 218, 219, 220, 221, 224, 226, 227, 229, 230, 231, 234, 235, 236, 238, 240, 241, 242, 243, 244, 246, 247, 248, 249, 251, 252, 253, 254, 255], "12": [0, 1, 2, 3, 5, 7, 9, 13, 17, 18, 20, 21, 22, 23, 24, 25, 26, 27, 29, 30, 32, 33, 34, 35, 36, 39, 40, 41, 42, 43, 45, 46, 48, 52, 54, 56, 57, 58, 59, 60, 61, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 77, 78, 79, 81, 82, 83, 85, 86, 87, 88, 89, 90, 91, 93, 94, 95, 96, 97, 98, 99, 100, 101, 103, 104, 106, 107, 108, 109, 110, 113, 114, 115, 116, 118, 119, 120, 121, 122, 123, 124, 125, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 144, 145, 147, 149, 150, 152, 153, 154, 158, 159, 160, 161, 162, 163, 164, 167, 169, 170, 171, 172, 173, 174, 175, 176, 177, 179, 180, 181, 182, 183, 184, 185, 188, 189, 191, 192, 194, 196, 197, 198, 199, 200, 201, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 214, 216, 217, 218, 219, 221, 222, 223, 224, 226, 227, 228, 229, 230, 231, 234, 236, 239, 241, 244, 245, 246, 248, 249, 250, 251, 252, 253, 254], "13": [0, 1, 2, 4, 6, 7, 9, 10, 11, 13, 14, 16, 17, 18, 19, 22, 23, 24, 25, 28, 29, 30, 31, 32, 34, 36, 37, 38, 42, 44, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 61, 62, 63, 64, 65, 66, 67, 68, 69, 71, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 93, 94, 95, 96, 97, 98, 99, 100, 103, 104, 106, 107, 109, 110, 111, 112, 113, 114, 117, 119, 120, 121, 122, 123, 125, 128, 129, 130, 132, 133, 134, 135, 136, 137, 139, 140, 141, 142, 143, 144, 145, 146, 147, 149, 151, 152, 153, 154, 156, 157, 158, 159, 160, 163, 164, 165, 166, 167, 168, 170, 171, 172, 173, 174, 176, 177, 178, 179, 180, 182, 183, 184, 186, 187, 188, 190, 191, 192, 193, 194, 196, 197, 198, 199, 200, 203, 204, 205, 206, 207, 208, 209, 211, 213, 214, 215, 216, 218, 219, 220, 221, 222, 223, 227, 229, 230, 231, 233, 235, 236, 237, 238, 242, 243, 244, 246, 248, 250, 251, 252, 253, 254], "14": [1, 2, 6, 7, 9, 10, 11, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 35, 37, 38, 39, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 72, 73, 74, 75, 76, 78, 80, 81, 82, 83, 85, 87, 88, 91, 92, 93, 94, 95, 96, 97, 100, 102, 103, 104, 105, 106, 110, 112, 113, 114, 115, 116, 118, 119, 120, 122, 123, 124, 125, 126, 127, 128, 129, 132, 133, 134, 135, 136, 138, 139, 140, 141, 142, 143, 145, 146, 147, 148, 149, 150, 151, 152, 154, 157, 159, 161, 162, 163, 164, 165, 166, 167, 168, 169, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 187, 188, 190, 191, 192, 194, 195, 196, 197, 198, 199, 200, 201, 203, 204, 206, 207, 209, 210, 212, 217, 218, 219, 220, 221, 222, 223, 225, 227, 228, 229, 230, 231, 232, 234, 235, 236, 237, 238, 240, 241, 243, 244, 245, 246, 247, 248, 249, 250, 252, 254, 255], "15": [0, 1, 2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 29, 31, 32, 33, 35, 36, 38, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 52, 53, 54, 56, 57, 59, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 75, 77, 78, 79, 81, 83, 84, 85, 86, 88, 89, 91, 92, 93, 94, 95, 96, 98, 100, 101, 102, 103, 104, 109, 110, 112, 113, 116, 117, 118, 119, 121, 124, 125, 127, 129, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 142, 143, 144, 145, 147, 148, 149, 151, 152, 153, 154, 155, 157, 158, 159, 160, 161, 162, 163, 164, 165, 167, 168, 170, 171, 173, 174, 175, 177, 179, 181, 182, 183, 185, 186, 187, 188, 189, 190, 191, 192, 193, 195, 197, 198, 200, 201, 202, 203, 204, 205, 206, 207, 209, 210, 212, 213, 214, 215, 216, 217, 218, 220, 221, 222, 224, 226, 227, 229, 231, 233, 234, 237, 238, 239, 240, 242, 243, 244, 245, 246, 247, 249, 250, 252, 253, 254, 255], "16": [0, 1, 2, 3, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 56, 58, 59, 60, 61, 65, 66, 69, 71, 72, 73, 74, 76, 77, 78, 79, 80, 81, 82, 83, 87, 88, 89, 90, 91, 93, 94, 96, 97, 99, 100, 103, 104, 105, 107, 108, 109, 110, 111, 112, 114, 115, 116, 117, 119, 120, 121, 123, 124, 127, 128, 129, 130, 132, 134, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 148, 149, 152, 153, 154, 155, 156, 159, 160, 161, 163, 165, 166, 167, 168, 169, 171, 173, 174, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 189, 190, 191, 192, 194, 196, 197, 199, 201, 202, 203, 204, 205, 206, 207, 208, 209, 211, 212, 213, 216, 217, 218, 219, 220, 225, 226, 228, 229, 230, 231, 232, 233, 234, 237, 238, 239, 240, 241, 242, 243, 244, 245, 247, 249, 251, 253, 254, 255], "17": [0, 1, 2, 3, 5, 7, 8, 9, 10, 13, 14, 15, 18, 21, 22, 23, 24, 25, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 46, 47, 48, 49, 50, 52, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 68, 70, 71, 72, 74, 75, 76, 77, 78, 79, 80, 81, 83, 84, 85, 86, 87, 88, 89, 90, 91, 95, 96, 99, 101, 102, 103, 104, 105, 106, 107, 108, 109, 111, 113, 114, 117, 118, 119, 121, 122, 123, 124, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 138, 139, 140, 141, 142, 145, 147, 148, 149, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 162, 163, 164, 166, 167, 168, 169, 171, 172, 173, 174, 175, 176, 177, 179, 180, 181, 182, 184, 185, 186, 187, 189, 190, 191, 192, 193, 194, 195, 197, 198, 200, 203, 204, 207, 208, 209, 211, 213, 216, 217, 219, 220, 222, 223, 225, 226, 227, 228, 231, 233, 234, 235, 236, 237, 239, 240, 241, 242, 243, 244, 245, 247, 249, 250, 251, 252, 254, 255], "18": [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 43, 44, 46, 47, 48, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 65, 66, 67, 68, 70, 71, 72, 74, 76, 77, 79, 81, 82, 83, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 100, 101, 103, 105, 106, 107, 108, 110, 111, 112, 113, 117, 118, 120, 122, 123, 124, 125, 126, 128, 129, 130, 131, 132, 135, 136, 138, 139, 140, 141, 143, 144, 145, 146, 148, 149, 151, 154, 155, 156, 157, 158, 159, 161, 162, 163, 164, 165, 166, 168, 170, 171, 172, 173, 174, 175, 176, 178, 179, 182, 183, 184, 185, 186, 189, 190, 192, 193, 195, 197, 199, 201, 202, 203, 206, 207, 208, 209, 210, 211, 213, 215, 217, 219, 220, 222, 223, 225, 227, 229, 230, 231, 232, 234, 235, 237, 238, 239, 241, 243, 244, 245, 246, 248, 249, 252, 253, 254, 255], "19": [0, 1, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 41, 42, 44, 45, 46, 47, 48, 50, 53, 54, 56, 57, 58, 59, 60, 62, 63, 64, 65, 68, 69, 71, 73, 74, 75, 79, 81, 82, 83, 85, 86, 88, 90, 91, 92, 94, 96, 97, 98, 99, 100, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 121, 122, 124, 125, 127, 130, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 150, 151, 153, 154, 155, 156, 157, 158, 159, 161, 162, 163, 164, 165, 168, 169, 171, 172, 173, 174, 175, 176, 178, 179, 181, 182, 183, 184, 187, 189, 192, 194, 197, 198, 199, 200, 201, 203, 205, 207, 208, 209, 210, 212, 214, 216, 217, 219, 220, 221, 222, 226, 227, 228, 229, 230, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 243, 244, 245, 246, 247, 249, 250, 253, 254], "20": [0, 1, 2, 3, 4, 5, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 34, 36, 38, 39, 40, 41, 42, 44, 45, 47, 48, 49, 51, 54, 55, 56, 57, 58, 61, 62, 63, 64, 65, 66, 67, 68, 71, 72, 74, 75, 77, 78, 79, 81, 82, 83, 84, 85, 86, 87, 88, 89, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 109, 111, 112, 114, 115, 117, 118, 120, 121, 122, 127, 128, 129, 130, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 160, 162, 163, 164, 166, 167, 168, 169, 170, 171, 172, 173, 175, 176, 177, 179, 180, 181, 182, 183, 185, 186, 189, 190, 193, 195, 196, 197, 198, 199, 202, 203, 205, 206, 207, 208, 210, 213, 215, 216, 217, 218, 220, 222, 223, 224, 225, 226, 227, 228, 229, 231, 232, 233, 234, 235, 236, 237, 238, 241, 242, 243, 245, 248, 250, 253, 254, 255], "21": [1, 2, 4, 6, 7, 8, 11, 12, 13, 14, 15, 17, 19, 20, 21, 22, 23, 24, 26, 27, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 51, 52, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 67, 69, 70, 73, 76, 77, 78, 79, 80, 82, 83, 85, 87, 88, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 104, 105, 107, 108, 110, 113, 114, 115, 116, 117, 119, 121, 122, 124, 125, 127, 129, 130, 131, 132, 134, 135, 136, 137, 138, 139, 140, 141, 143, 144, 145, 147, 148, 149, 150, 151, 153, 154, 155, 158, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 174, 175, 176, 179, 180, 181, 183, 184, 185, 186, 187, 188, 189, 190, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 205, 206, 208, 209, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 223, 224, 226, 228, 229, 231, 233, 234, 235, 236, 238, 239, 241, 242, 244, 245, 248, 249, 250, 251, 252, 253, 254], "22": [0, 2, 3, 4, 8, 10, 12, 13, 14, 15, 17, 18, 19, 20, 21, 23, 24, 27, 29, 30, 31, 32, 35, 36, 37, 38, 39, 40, 41, 43, 44, 47, 48, 49, 50, 52, 53, 55, 56, 59, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 94, 95, 96, 97, 98, 99, 100, 104, 107, 108, 109, 110, 111, 113, 114, 115, 116, 118, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 137, 138, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 154, 156, 157, 158, 159, 160, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 173, 174, 175, 176, 177, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 191, 192, 193, 194, 197, 198, 199, 200, 202, 206, 207, 208, 209, 210, 213, 214, 215, 218, 219, 221, 223, 226, 228, 229, 231, 232, 234, 235, 236, 237, 238, 239, 240, 241, 242, 244, 245, 246, 248, 249, 250, 251, 252, 253, 254, 255], "23": [0, 1, 2, 4, 5, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33, 35, 37, 38, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 59, 60, 61, 62, 63, 65, 67, 68, 69, 70, 71, 72, 73, 75, 76, 78, 80, 83, 85, 86, 87, 88, 89, 90, 91, 93, 94, 98, 99, 100, 101, 102, 105, 107, 108, 110, 111, 113, 114, 115, 117, 118, 120, 121, 122, 124, 125, 126, 127, 128, 129, 130, 131, 132, 135, 136, 137, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 151, 152, 154, 156, 158, 160, 162, 164, 165, 166, 167, 169, 170, 171, 172, 174, 175, 176, 177, 178, 179, 180, 181, 182, 184, 186, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 205, 206, 207, 209, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 224, 225, 226, 228, 229, 232, 233, 234, 235, 237, 240, 241, 242, 244, 245, 246, 247, 248, 250, 252, 253, 254, 255], "24": [0, 3, 4, 6, 7, 10, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22, 23, 26, 27, 28, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 43, 44, 45, 46, 47, 48, 51, 53, 54, 55, 57, 59, 60, 61, 62, 63, 64, 65, 67, 68, 69, 70, 71, 73, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 88, 89, 90, 91, 92, 93, 94, 95, 96, 98, 99, 100, 101, 102, 104, 105, 106, 107, 109, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 129, 131, 132, 133, 134, 135, 136, 137, 138, 140, 141, 143, 145, 146, 147, 149, 150, 151, 153, 154, 155, 157, 159, 160, 161, 162, 163, 164, 165, 166, 167, 169, 170, 172, 174, 175, 177, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 192, 193, 195, 196, 197, 198, 201, 202, 203, 204, 206, 207, 209, 210, 211, 212, 213, 214, 215, 218, 220, 221, 225, 226, 228, 229, 230, 233, 234, 236, 237, 238, 239, 242, 245, 246, 247, 248, 249, 250, 251, 252, 255], "25": [0, 1, 2, 3, 5, 6, 7, 8, 9, 13, 14, 16, 17, 18, 19, 21, 22, 23, 24, 25, 27, 28, 29, 33, 34, 35, 37, 38, 40, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 64, 65, 67, 68, 69, 70, 71, 72, 76, 77, 79, 80, 81, 82, 83, 84, 88, 89, 90, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 103, 106, 108, 109, 110, 111, 112, 113, 114, 116, 119, 120, 122, 123, 124, 125, 126, 127, 128, 129, 130, 132, 133, 134, 135, 136, 137, 139, 140, 141, 142, 143, 144, 146, 147, 148, 150, 151, 153, 154, 156, 157, 158, 159, 160, 162, 163, 164, 165, 167, 168, 169, 170, 171, 172, 173, 175, 177, 179, 180, 182, 183, 184, 185, 186, 187, 188, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 202, 204, 205, 206, 208, 210, 211, 212, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 227, 228, 230, 231, 232, 233, 235, 240, 241, 243, 244, 245, 246, 247, 249, 250, 255], "26": [0, 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 16, 17, 19, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35, 39, 40, 42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 53, 54, 55, 57, 58, 60, 61, 63, 66, 67, 68, 69, 70, 72, 74, 75, 76, 77, 79, 80, 81, 82, 83, 84, 85, 86, 87, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 101, 102, 104, 105, 106, 108, 110, 111, 113, 114, 118, 119, 120, 121, 123, 124, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 141, 142, 143, 144, 146, 147, 148, 149, 150, 151, 153, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 167, 168, 169, 171, 172, 173, 175, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 190, 191, 192, 194, 195, 196, 197, 198, 200, 201, 202, 203, 205, 207, 208, 209, 211, 212, 214, 215, 216, 217, 218, 219, 221, 225, 226, 229, 230, 232, 233, 234, 236, 237, 240, 241, 242, 243, 244, 245, 248, 249, 250, 251, 252, 253, 255], "27": [1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 27, 28, 29, 30, 31, 34, 35, 36, 37, 38, 39, 40, 42, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 65, 67, 68, 69, 70, 71, 72, 75, 76, 78, 80, 81, 82, 83, 84, 85, 87, 88, 89, 90, 91, 93, 94, 96, 97, 98, 99, 102, 103, 105, 106, 109, 111, 114, 116, 117, 119, 120, 121, 122, 124, 125, 126, 127, 128, 130, 131, 132, 133, 135, 136, 138, 139, 140, 141, 143, 144, 145, 146, 147, 148, 149, 150, 153, 155, 157, 158, 160, 161, 162, 164, 166, 167, 168, 169, 170, 171, 172, 173, 175, 176, 177, 178, 179, 180, 181, 184, 185, 188, 190, 192, 193, 194, 196, 198, 200, 201, 202, 203, 204, 205, 206, 207, 208, 210, 211, 213, 214, 215, 216, 217, 218, 219, 220, 223, 224, 225, 227, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 247, 248, 249, 250, 251, 252, 253, 255], "28": [1, 3, 4, 5, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 20, 21, 23, 25, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 39, 40, 42, 44, 46, 47, 48, 50, 51, 55, 57, 58, 59, 60, 61, 62, 63, 65, 66, 68, 69, 70, 71, 72, 73, 75, 77, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 92, 93, 94, 95, 97, 99, 100, 101, 102, 103, 104, 108, 109, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 131, 132, 133, 134, 135, 138, 139, 140, 141, 143, 144, 145, 146, 148, 149, 150, 151, 152, 153, 155, 157, 158, 159, 160, 161, 162, 163, 165, 168, 169, 170, 171, 173, 174, 175, 176, 177, 178, 179, 180, 181, 183, 184, 186, 187, 189, 190, 193, 194, 195, 197, 198, 199, 200, 201, 202, 203, 204, 206, 207, 208, 209, 213, 215, 216, 217, 218, 219, 220, 221, 222, 225, 228, 229, 230, 231, 232, 233, 235, 236, 238, 240, 243, 244, 245, 246, 247, 248, 249, 250, 252, 254, 255], "29": [2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 23, 24, 25, 28, 29, 30, 32, 33, 35, 36, 37, 38, 39, 41, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 62, 63, 64, 65, 68, 70, 71, 72, 74, 75, 76, 78, 79, 80, 82, 83, 85, 86, 88, 89, 91, 92, 93, 94, 97, 100, 104, 105, 106, 107, 109, 111, 112, 113, 114, 116, 117, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 129, 130, 132, 133, 134, 136, 137, 138, 139, 140, 142, 143, 144, 146, 147, 149, 151, 152, 153, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 166, 167, 168, 169, 170, 171, 173, 175, 176, 177, 178, 179, 180, 182, 184, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 211, 212, 213, 214, 215, 217, 219, 220, 221, 222, 223, 225, 227, 229, 230, 231, 232, 234, 235, 239, 240, 242, 243, 244, 245, 246, 247, 250, 251, 253, 254, 255], "30": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 30, 31, 32, 33, 34, 35, 37, 38, 39, 40, 41, 42, 43, 44, 45, 47, 48, 49, 50, 51, 54, 55, 56, 58, 59, 61, 62, 63, 64, 66, 68, 69, 72, 73, 75, 76, 77, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 90, 94, 95, 98, 99, 100, 101, 103, 104, 105, 106, 107, 109, 110, 112, 113, 114, 115, 116, 117, 119, 121, 122, 125, 126, 127, 128, 129, 130, 131, 132, 134, 136, 137, 138, 139, 140, 141, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 155, 156, 160, 161, 162, 165, 166, 167, 169, 170, 171, 172, 174, 177, 179, 180, 183, 184, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 202, 203, 204, 211, 212, 213, 214, 215, 216, 217, 218, 221, 222, 223, 224, 225, 226, 228, 229, 230, 231, 232, 233, 234, 235, 237, 239, 240, 241, 242, 243, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255], "31": [0, 2, 5, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 20, 23, 24, 25, 26, 28, 30, 34, 35, 36, 37, 38, 40, 41, 42, 43, 44, 45, 46, 48, 49, 51, 52, 53, 56, 57, 58, 59, 60, 61, 62, 63, 65, 68, 69, 70, 71, 72, 73, 74, 76, 77, 79, 81, 83, 84, 85, 86, 87, 89, 91, 92, 93, 94, 96, 97, 99, 100, 101, 102, 103, 104, 105, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 122, 123, 124, 125, 126, 130, 132, 133, 134, 136, 137, 139, 140, 141, 142, 144, 145, 146, 147, 149, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 163, 165, 166, 168, 169, 170, 172, 174, 175, 177, 178, 179, 180, 181, 183, 184, 186, 187, 188, 189, 190, 191, 192, 193, 196, 198, 199, 200, 201, 202, 204, 206, 207, 208, 209, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 252, 253, 255], "32": [0, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 35, 38, 39, 40, 42, 43, 44, 46, 47, 51, 53, 54, 55, 57, 58, 59, 61, 62, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 89, 90, 91, 92, 93, 94, 95, 96, 102, 103, 104, 105, 106, 107, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 137, 138, 139, 140, 141, 142, 143, 144, 146, 147, 148, 150, 152, 155, 156, 158, 159, 160, 161, 164, 166, 167, 169, 170, 173, 174, 175, 176, 178, 179, 181, 182, 183, 186, 187, 190, 191, 192, 194, 198, 199, 200, 201, 202, 203, 204, 205, 206, 210, 211, 212, 213, 214, 216, 217, 218, 219, 220, 221, 223, 224, 226, 227, 229, 230, 231, 232, 233, 234, 235, 237, 239, 240, 241, 242, 244, 246, 248, 251, 252, 253, 254, 255], "33": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 42, 43, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 60, 61, 62, 63, 64, 66, 67, 69, 70, 72, 73, 74, 75, 78, 81, 82, 83, 84, 85, 86, 88, 89, 90, 92, 93, 94, 95, 98, 99, 100, 101, 102, 105, 106, 107, 109, 110, 111, 112, 114, 115, 117, 118, 119, 122, 124, 125, 126, 127, 129, 130, 131, 132, 133, 135, 136, 137, 138, 140, 141, 142, 143, 144, 145, 146, 149, 152, 154, 155, 156, 157, 158, 159, 161, 162, 163, 165, 166, 167, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 203, 204, 205, 206, 207, 208, 211, 212, 215, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 230, 232, 234, 236, 237, 238, 239, 241, 242, 245, 246, 248, 250, 251, 252, 255], "34": [0, 1, 3, 4, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 25, 28, 29, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 44, 46, 47, 48, 49, 50, 51, 52, 53, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 97, 98, 99, 100, 101, 102, 103, 106, 107, 108, 109, 110, 111, 112, 113, 115, 117, 118, 120, 121, 122, 125, 129, 130, 132, 133, 137, 139, 141, 142, 143, 144, 145, 147, 152, 153, 154, 156, 157, 158, 159, 161, 162, 163, 164, 165, 166, 167, 170, 172, 173, 174, 175, 176, 177, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 194, 195, 196, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 213, 214, 215, 216, 217, 218, 219, 220, 222, 223, 224, 228, 229, 230, 232, 233, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 248, 249, 253, 254], "35": [0, 1, 2, 3, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37, 38, 39, 40, 41, 42, 43, 45, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 61, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 76, 78, 80, 81, 82, 83, 84, 85, 87, 88, 89, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 106, 107, 108, 110, 112, 113, 115, 117, 118, 119, 120, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 133, 134, 138, 139, 140, 141, 142, 145, 147, 149, 150, 151, 152, 154, 155, 158, 159, 160, 161, 163, 165, 166, 167, 168, 169, 170, 172, 173, 174, 175, 176, 178, 179, 181, 183, 185, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 200, 201, 203, 205, 206, 207, 208, 210, 211, 212, 213, 214, 216, 217, 218, 220, 221, 222, 223, 224, 227, 228, 230, 232, 235, 236, 237, 238, 239, 240, 243, 244, 245, 248, 251], "36": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 14, 15, 17, 20, 22, 23, 25, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 40, 41, 42, 43, 44, 45, 47, 48, 49, 50, 51, 53, 55, 57, 58, 60, 62, 63, 65, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 81, 82, 83, 88, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 102, 103, 104, 105, 106, 107, 109, 111, 112, 113, 114, 116, 119, 120, 121, 122, 123, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 136, 138, 139, 141, 142, 144, 147, 148, 149, 152, 153, 154, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 171, 173, 174, 175, 176, 177, 178, 179, 180, 182, 183, 184, 187, 188, 190, 191, 192, 195, 196, 198, 200, 201, 202, 203, 204, 206, 207, 208, 209, 210, 211, 212, 213, 216, 217, 218, 222, 223, 224, 226, 227, 228, 229, 230, 232, 235, 237, 238, 239, 240, 241, 243, 244, 245, 246, 247, 248, 249, 250, 252, 253, 254, 255], "37": [0, 1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 16, 18, 19, 20, 21, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 37, 38, 39, 40, 41, 43, 46, 47, 48, 49, 50, 52, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 65, 66, 70, 72, 73, 74, 75, 76, 77, 79, 80, 81, 83, 84, 87, 88, 89, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 108, 109, 110, 111, 112, 113, 114, 115, 117, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 132, 133, 134, 135, 136, 138, 139, 140, 143, 144, 147, 148, 149, 150, 151, 152, 154, 155, 156, 157, 158, 159, 160, 162, 164, 165, 167, 168, 169, 170, 171, 172, 173, 174, 176, 177, 178, 179, 180, 182, 183, 184, 185, 186, 187, 188, 190, 191, 195, 197, 198, 199, 200, 202, 205, 206, 207, 208, 209, 210, 212, 213, 215, 216, 217, 218, 219, 220, 222, 224, 226, 227, 229, 230, 233, 234, 236, 238, 240, 241, 242, 243, 244, 247, 249, 250, 252, 253, 255], "38": [0, 1, 2, 4, 5, 6, 8, 10, 11, 13, 14, 15, 16, 18, 20, 21, 23, 26, 27, 30, 31, 32, 33, 34, 36, 38, 39, 41, 44, 45, 46, 47, 49, 51, 54, 55, 56, 58, 59, 61, 62, 63, 64, 67, 69, 71, 73, 74, 76, 77, 78, 79, 80, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 95, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 115, 116, 117, 118, 119, 120, 121, 123, 125, 126, 128, 129, 130, 131, 132, 134, 136, 138, 139, 140, 142, 145, 146, 147, 148, 149, 150, 151, 152, 154, 155, 157, 158, 159, 160, 161, 164, 165, 166, 167, 170, 171, 172, 173, 174, 175, 177, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 215, 216, 217, 218, 220, 221, 223, 224, 225, 227, 230, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 243, 245, 246, 248, 249, 250, 251, 252, 253], "39": [0, 1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 20, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 34, 35, 36, 38, 39, 40, 42, 43, 44, 45, 46, 47, 48, 49, 52, 53, 56, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 83, 84, 85, 87, 88, 89, 90, 92, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 106, 107, 108, 111, 112, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 127, 128, 129, 130, 132, 134, 135, 136, 137, 138, 140, 142, 143, 144, 145, 146, 148, 149, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 163, 164, 168, 170, 171, 173, 174, 175, 177, 179, 181, 182, 190, 193, 194, 196, 197, 200, 201, 202, 203, 204, 205, 206, 207, 208, 210, 211, 212, 213, 215, 216, 218, 221, 223, 224, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 241, 242, 243, 244, 245, 246, 247, 248, 251, 252, 253, 254, 255], "40": [0, 1, 4, 5, 6, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 23, 25, 26, 27, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 67, 69, 70, 71, 72, 73, 74, 75, 76, 79, 80, 81, 82, 83, 84, 85, 86, 89, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 106, 107, 108, 109, 111, 114, 116, 117, 118, 120, 121, 123, 124, 125, 127, 130, 131, 132, 133, 134, 136, 137, 138, 139, 140, 141, 144, 146, 147, 148, 150, 151, 152, 153, 154, 155, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 172, 173, 174, 175, 176, 179, 183, 184, 186, 187, 188, 190, 192, 195, 196, 199, 200, 203, 204, 205, 206, 207, 208, 210, 211, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 228, 229, 231, 232, 233, 234, 236, 238, 239, 242, 243, 244, 245, 246, 247, 248, 250, 252, 253, 255], "41": [0, 3, 4, 5, 6, 7, 8, 14, 16, 17, 18, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 32, 33, 34, 35, 36, 37, 38, 39, 41, 42, 44, 46, 47, 48, 49, 50, 51, 52, 54, 56, 58, 59, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 102, 103, 105, 106, 107, 109, 110, 114, 116, 117, 118, 121, 122, 123, 124, 125, 126, 128, 129, 130, 132, 133, 134, 135, 136, 137, 138, 140, 141, 142, 145, 146, 147, 148, 149, 150, 152, 153, 154, 155, 156, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 174, 175, 176, 177, 178, 180, 182, 183, 184, 186, 187, 188, 189, 190, 191, 192, 193, 194, 196, 197, 199, 202, 205, 206, 207, 208, 209, 210, 212, 213, 214, 216, 217, 221, 222, 223, 224, 225, 226, 227, 229, 230, 231, 232, 233, 235, 236, 237, 238, 240, 241, 242, 243, 244, 245, 246, 249, 251, 253, 255], "42": [0, 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 37, 38, 39, 40, 41, 44, 45, 47, 48, 49, 50, 51, 52, 54, 58, 59, 60, 61, 62, 66, 67, 68, 69, 70, 71, 73, 76, 78, 80, 81, 83, 85, 86, 87, 88, 90, 91, 93, 94, 95, 96, 97, 100, 101, 102, 103, 105, 107, 108, 109, 110, 111, 112, 113, 116, 117, 120, 121, 122, 123, 124, 125, 126, 127, 128, 131, 132, 133, 134, 136, 137, 138, 139, 140, 141, 142, 143, 144, 146, 150, 151, 152, 155, 156, 157, 158, 159, 160, 161, 162, 164, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 177, 179, 180, 181, 182, 183, 184, 186, 187, 188, 190, 191, 192, 194, 195, 196, 200, 201, 202, 203, 204, 205, 206, 207, 208, 210, 212, 213, 215, 219, 220, 221, 222, 223, 224, 227, 228, 229, 230, 231, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 244, 245, 246, 248, 249, 250, 252, 253, 254, 255], "43": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16, 19, 20, 21, 24, 26, 27, 28, 29, 30, 33, 35, 37, 38, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 54, 56, 57, 58, 59, 60, 61, 62, 63, 64, 66, 67, 68, 69, 70, 71, 72, 73, 74, 76, 78, 79, 81, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 112, 113, 115, 118, 119, 120, 121, 123, 125, 126, 127, 128, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 144, 146, 147, 148, 149, 150, 152, 153, 154, 155, 156, 157, 160, 161, 162, 163, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 178, 179, 180, 181, 183, 184, 185, 187, 188, 189, 191, 192, 194, 195, 196, 198, 199, 201, 202, 206, 208, 210, 211, 213, 214, 215, 216, 219, 220, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 242, 244, 245, 246, 247, 248, 249, 250, 252, 253, 254], "44": [1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 21, 22, 23, 24, 25, 26, 28, 30, 31, 32, 33, 34, 35, 36, 38, 41, 42, 43, 45, 46, 47, 48, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 86, 87, 89, 91, 92, 94, 95, 97, 99, 103, 104, 105, 106, 107, 108, 112, 113, 114, 117, 118, 119, 122, 123, 124, 126, 128, 129, 130, 132, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 150, 151, 152, 155, 156, 157, 158, 159, 160, 162, 163, 164, 166, 167, 168, 169, 170, 172, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 187, 188, 189, 190, 192, 193, 194, 196, 197, 198, 200, 201, 202, 204, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 219, 220, 223, 224, 225, 227, 228, 229, 230, 235, 236, 237, 239, 240, 241, 242, 244, 245, 247, 249, 250, 251, 254, 255], "45": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 63, 65, 68, 69, 70, 71, 72, 74, 76, 77, 79, 83, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 97, 100, 101, 102, 103, 104, 105, 107, 109, 110, 111, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 132, 133, 134, 135, 136, 138, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 154, 155, 158, 161, 163, 166, 167, 169, 170, 171, 173, 174, 175, 176, 177, 178, 180, 182, 184, 186, 187, 189, 190, 191, 192, 193, 195, 197, 200, 201, 204, 207, 208, 210, 211, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 231, 232, 235, 236, 238, 239, 240, 243, 244, 245, 248, 249, 250, 251, 252, 253, 255], "46": [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 35, 36, 37, 38, 39, 41, 42, 43, 45, 46, 47, 48, 49, 51, 52, 53, 55, 56, 57, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 70, 71, 72, 73, 74, 75, 77, 78, 79, 80, 81, 82, 84, 85, 87, 88, 90, 92, 93, 94, 95, 96, 98, 99, 102, 105, 107, 108, 109, 112, 113, 114, 115, 116, 117, 120, 122, 127, 128, 130, 131, 132, 133, 134, 137, 138, 140, 141, 142, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 171, 173, 175, 176, 177, 178, 180, 181, 182, 183, 186, 188, 189, 190, 191, 192, 193, 195, 196, 197, 198, 199, 201, 205, 206, 207, 208, 209, 212, 215, 216, 218, 221, 224, 225, 226, 227, 228, 229, 230, 231, 232, 234, 236, 237, 239, 240, 241, 242, 243, 244, 246, 247, 248, 250, 251, 252, 253, 254], "47": [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 14, 16, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 42, 43, 44, 48, 49, 50, 52, 54, 55, 56, 57, 58, 59, 60, 61, 62, 65, 66, 67, 69, 71, 72, 74, 77, 79, 80, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 96, 99, 100, 102, 103, 104, 105, 106, 108, 109, 110, 111, 112, 113, 114, 115, 117, 118, 119, 121, 122, 123, 124, 125, 126, 127, 128, 129, 133, 135, 136, 138, 141, 143, 145, 146, 147, 148, 150, 151, 152, 153, 155, 156, 158, 159, 162, 163, 164, 165, 166, 167, 168, 171, 172, 173, 175, 176, 177, 179, 180, 181, 183, 184, 185, 186, 187, 188, 189, 190, 192, 193, 194, 195, 198, 199, 200, 202, 204, 205, 207, 208, 209, 210, 211, 213, 215, 216, 217, 218, 219, 221, 222, 223, 225, 226, 228, 229, 230, 231, 233, 234, 236, 237, 238, 239, 240, 242, 243, 244, 247, 248, 249, 250, 252, 253, 254, 255], "48": [1, 2, 3, 4, 7, 8, 11, 12, 13, 14, 15, 16, 17, 20, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 35, 36, 37, 39, 40, 41, 43, 44, 46, 47, 48, 50, 51, 52, 53, 54, 55, 56, 58, 59, 60, 62, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 88, 89, 90, 91, 92, 94, 96, 97, 99, 100, 101, 102, 103, 105, 106, 107, 109, 111, 114, 116, 117, 119, 122, 128, 129, 130, 131, 133, 134, 135, 136, 137, 138, 141, 142, 143, 144, 145, 146, 147, 148, 150, 154, 155, 156, 157, 159, 160, 161, 162, 163, 164, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 177, 178, 180, 181, 183, 186, 187, 188, 189, 190, 193, 194, 196, 197, 199, 200, 201, 202, 203, 204, 206, 207, 208, 209, 210, 211, 213, 215, 217, 218, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 233, 234, 235, 236, 238, 239, 240, 241, 242, 244, 246, 247, 248, 249, 250, 251, 252, 254, 255], "49": [0, 1, 2, 3, 4, 5, 8, 9, 10, 12, 13, 14, 16, 17, 19, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 43, 44, 45, 46, 47, 48, 50, 51, 54, 55, 56, 57, 59, 61, 62, 63, 65, 66, 67, 68, 70, 72, 73, 74, 76, 77, 78, 79, 80, 84, 86, 87, 88, 89, 91, 92, 93, 94, 95, 98, 101, 102, 103, 104, 105, 106, 107, 108, 110, 111, 112, 113, 114, 115, 116, 118, 119, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 138, 140, 141, 142, 143, 144, 145, 146, 147, 149, 150, 151, 152, 153, 154, 157, 158, 159, 160, 161, 163, 164, 165, 166, 167, 168, 169, 170, 172, 173, 179, 180, 181, 182, 183, 184, 187, 188, 189, 191, 192, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 212, 213, 214, 215, 217, 218, 219, 221, 222, 223, 224, 225, 226, 229, 230, 231, 232, 235, 236, 238, 239, 242, 243, 246, 247, 249, 250, 252, 253, 254], "50": [0, 2, 6, 8, 9, 10, 11, 12, 13, 15, 16, 17, 19, 20, 22, 23, 24, 25, 26, 28, 29, 30, 31, 34, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 53, 55, 56, 58, 59, 60, 61, 62, 63, 65, 67, 70, 71, 72, 73, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 99, 101, 102, 103, 104, 106, 107, 109, 110, 112, 113, 114, 115, 116, 118, 119, 121, 123, 124, 125, 126, 128, 129, 130, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 142, 143, 145, 146, 147, 148, 149, 150, 151, 152, 153, 155, 156, 158, 159, 161, 162, 163, 164, 165, 167, 168, 169, 170, 171, 172, 173, 176, 177, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 196, 197, 198, 201, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 218, 220, 221, 222, 224, 226, 227, 229, 230, 232, 233, 234, 235, 239, 243, 244, 245, 246, 247, 248, 249, 250, 252, 254], "51": [1, 3, 4, 5, 7, 9, 11, 13, 15, 16, 17, 18, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 54, 55, 57, 58, 59, 60, 61, 63, 64, 66, 67, 68, 69, 71, 72, 74, 75, 76, 79, 80, 82, 83, 84, 85, 87, 89, 90, 91, 92, 93, 95, 96, 97, 98, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 113, 115, 116, 118, 119, 121, 122, 123, 124, 125, 126, 127, 128, 129, 131, 132, 133, 134, 135, 136, 137, 138, 139, 142, 143, 144, 145, 146, 147, 149, 150, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 167, 168, 169, 172, 174, 182, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 199, 200, 201, 203, 204, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 218, 219, 220, 221, 223, 225, 226, 227, 228, 230, 232, 233, 234, 235, 237, 238, 240, 241, 242, 243, 244, 246, 247, 248, 249, 251, 252, 254, 255], "52": [0, 2, 3, 4, 5, 7, 8, 9, 11, 12, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 29, 30, 31, 32, 35, 36, 38, 39, 41, 42, 43, 44, 47, 48, 49, 51, 52, 53, 54, 55, 57, 59, 60, 61, 62, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 75, 76, 77, 78, 81, 83, 84, 86, 87, 89, 90, 91, 93, 94, 95, 96, 97, 98, 101, 102, 103, 104, 106, 107, 108, 109, 110, 111, 112, 113, 115, 117, 118, 120, 121, 123, 124, 126, 127, 128, 130, 131, 133, 134, 136, 138, 139, 140, 141, 142, 143, 144, 145, 147, 148, 149, 150, 151, 152, 153, 154, 157, 158, 161, 163, 164, 166, 167, 169, 170, 171, 173, 174, 176, 177, 178, 179, 180, 182, 184, 185, 187, 188, 189, 190, 191, 192, 193, 195, 196, 197, 198, 199, 201, 202, 204, 205, 206, 207, 209, 210, 211, 212, 213, 214, 215, 217, 219, 220, 221, 222, 223, 224, 225, 226, 228, 229, 230, 232, 233, 235, 236, 237, 238, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 255], "53": [1, 2, 3, 4, 5, 7, 8, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 33, 34, 36, 37, 38, 40, 41, 42, 43, 45, 47, 49, 50, 51, 52, 53, 54, 56, 58, 59, 61, 62, 63, 64, 65, 67, 68, 69, 72, 73, 74, 75, 76, 77, 78, 79, 82, 83, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 98, 99, 100, 101, 102, 106, 110, 112, 113, 114, 116, 117, 118, 120, 121, 122, 123, 125, 126, 127, 128, 132, 133, 135, 136, 137, 138, 139, 140, 142, 143, 144, 145, 146, 147, 149, 150, 152, 154, 156, 157, 158, 159, 161, 162, 163, 165, 166, 167, 171, 172, 175, 176, 178, 179, 180, 182, 183, 184, 185, 187, 188, 189, 190, 192, 193, 194, 195, 196, 199, 200, 201, 202, 203, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 216, 217, 218, 219, 221, 222, 223, 224, 226, 227, 229, 230, 231, 232, 233, 234, 235, 236, 239, 240, 241, 243, 244, 245, 246, 247, 248, 249, 251, 252, 253, 254, 255], "54": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35, 38, 39, 40, 41, 43, 48, 49, 50, 51, 52, 53, 54, 55, 58, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 81, 82, 84, 85, 86, 87, 88, 89, 90, 92, 93, 95, 98, 99, 100, 101, 102, 105, 107, 108, 109, 110, 111, 112, 113, 114, 116, 118, 120, 121, 124, 125, 126, 127, 128, 129, 131, 132, 134, 135, 136, 138, 139, 140, 143, 144, 145, 146, 148, 149, 150, 152, 153, 154, 158, 159, 161, 162, 163, 164, 165, 166, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 191, 194, 195, 196, 199, 200, 204, 205, 206, 207, 209, 210, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 223, 224, 225, 226, 228, 229, 230, 232, 233, 234, 235, 236, 237, 238, 239, 242, 244, 245, 248, 249, 251, 252, 253, 254], "55": [0, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 34, 35, 40, 41, 42, 43, 44, 45, 46, 47, 48, 50, 51, 53, 54, 58, 59, 60, 61, 63, 64, 65, 66, 69, 70, 71, 72, 74, 75, 76, 77, 78, 79, 80, 81, 83, 84, 85, 86, 87, 88, 89, 90, 92, 93, 94, 95, 96, 97, 100, 101, 104, 105, 106, 107, 109, 111, 112, 113, 114, 116, 117, 118, 120, 121, 122, 123, 124, 126, 127, 128, 129, 131, 132, 133, 134, 135, 136, 137, 140, 141, 142, 143, 144, 145, 149, 150, 151, 152, 153, 154, 155, 156, 158, 160, 161, 162, 163, 164, 165, 166, 169, 171, 172, 173, 174, 175, 176, 177, 178, 181, 184, 185, 186, 187, 189, 190, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 208, 209, 210, 211, 212, 213, 215, 218, 219, 221, 222, 223, 224, 225, 226, 227, 228, 229, 232, 234, 235, 236, 237, 238, 239, 240, 242, 245, 249, 250, 251, 253, 254], "56": [1, 6, 8, 9, 10, 13, 15, 16, 18, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 61, 63, 64, 65, 66, 67, 69, 70, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 85, 87, 88, 89, 90, 91, 93, 94, 95, 96, 98, 102, 103, 104, 106, 107, 108, 109, 110, 111, 113, 114, 115, 116, 117, 119, 120, 121, 122, 123, 124, 126, 128, 129, 130, 132, 133, 134, 135, 136, 137, 140, 141, 142, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 157, 158, 160, 161, 162, 163, 164, 165, 168, 169, 170, 172, 174, 175, 178, 180, 182, 183, 184, 185, 186, 187, 188, 190, 191, 193, 194, 195, 196, 197, 198, 199, 200, 201, 203, 204, 205, 206, 207, 209, 211, 214, 216, 217, 218, 219, 220, 222, 223, 224, 225, 227, 229, 230, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 246, 247, 248, 249, 250, 251, 253, 254], "57": [1, 3, 4, 5, 7, 8, 9, 11, 12, 13, 16, 17, 18, 19, 20, 22, 23, 25, 26, 27, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 53, 54, 55, 58, 60, 63, 64, 65, 66, 69, 70, 71, 73, 74, 75, 76, 77, 78, 79, 82, 84, 85, 86, 87, 89, 91, 92, 94, 95, 97, 98, 99, 100, 101, 103, 104, 106, 107, 109, 111, 113, 114, 115, 116, 118, 119, 120, 122, 123, 124, 125, 126, 127, 129, 131, 132, 133, 134, 135, 136, 137, 138, 139, 141, 142, 143, 144, 145, 146, 147, 150, 155, 156, 157, 158, 159, 160, 161, 164, 165, 166, 167, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 185, 187, 188, 192, 193, 194, 195, 196, 197, 199, 200, 202, 203, 204, 205, 206, 207, 208, 209, 211, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 230, 231, 232, 233, 234, 235, 236, 237, 238, 240, 241, 243, 244, 245, 247, 249, 250, 253, 255], "58": [0, 1, 2, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 21, 23, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 36, 39, 40, 41, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 57, 61, 63, 64, 65, 66, 67, 68, 70, 72, 73, 74, 76, 77, 78, 79, 80, 81, 83, 84, 85, 86, 87, 89, 90, 91, 93, 94, 96, 97, 98, 99, 100, 103, 104, 105, 106, 107, 108, 110, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 125, 126, 127, 128, 129, 131, 136, 137, 138, 139, 143, 144, 145, 147, 148, 149, 150, 151, 152, 154, 156, 157, 158, 159, 161, 163, 164, 165, 166, 167, 169, 171, 173, 174, 176, 177, 179, 180, 181, 182, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 195, 196, 197, 198, 200, 201, 203, 204, 206, 207, 210, 211, 213, 214, 215, 217, 218, 219, 220, 221, 222, 223, 224, 225, 227, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 245, 248, 249, 251, 252, 253, 255], "59": [0, 2, 4, 5, 6, 7, 9, 10, 12, 13, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 31, 33, 34, 35, 36, 39, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 57, 58, 59, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 73, 75, 76, 77, 79, 80, 81, 83, 84, 86, 87, 88, 89, 90, 91, 93, 95, 96, 99, 100, 101, 102, 103, 104, 105, 106, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 122, 123, 124, 125, 127, 128, 129, 130, 131, 132, 133, 135, 136, 139, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 154, 155, 157, 158, 159, 160, 161, 162, 163, 164, 167, 168, 169, 170, 173, 174, 175, 176, 177, 178, 179, 181, 183, 184, 185, 187, 188, 191, 193, 195, 197, 198, 199, 201, 202, 203, 204, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 217, 218, 221, 222, 223, 224, 226, 228, 230, 231, 234, 235, 236, 237, 238, 240, 242, 243, 244, 245, 246, 247, 248, 249, 251, 252, 253, 254], "60": [0, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24, 25, 26, 27, 30, 31, 32, 33, 35, 37, 38, 39, 40, 41, 42, 46, 48, 49, 50, 51, 53, 54, 55, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 70, 71, 72, 73, 74, 75, 76, 78, 79, 80, 81, 82, 83, 85, 87, 88, 89, 90, 91, 92, 93, 94, 95, 97, 99, 100, 101, 103, 105, 106, 107, 108, 109, 110, 111, 116, 117, 118, 119, 120, 121, 122, 124, 125, 127, 128, 129, 130, 131, 133, 135, 136, 137, 138, 139, 140, 141, 142, 144, 147, 150, 152, 154, 155, 156, 157, 160, 161, 162, 163, 164, 165, 166, 167, 168, 170, 172, 174, 178, 179, 180, 181, 182, 183, 185, 186, 187, 188, 189, 192, 193, 195, 196, 199, 200, 201, 203, 205, 206, 207, 209, 210, 211, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 227, 228, 229, 231, 232, 233, 234, 235, 236, 237, 238, 239, 241, 242, 243, 244, 245, 246, 247, 249, 250, 251, 253, 254], "61": [1, 2, 3, 5, 6, 7, 11, 12, 13, 15, 17, 18, 19, 20, 23, 24, 25, 26, 28, 29, 30, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 47, 48, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 63, 65, 67, 69, 70, 71, 73, 74, 75, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 89, 90, 91, 92, 93, 96, 98, 99, 100, 101, 104, 105, 106, 107, 108, 109, 110, 111, 113, 115, 118, 119, 120, 121, 123, 124, 125, 126, 128, 129, 130, 131, 132, 134, 135, 136, 137, 138, 140, 141, 143, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 157, 158, 160, 161, 162, 163, 165, 166, 168, 170, 171, 172, 173, 175, 176, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 190, 191, 193, 194, 197, 198, 199, 200, 203, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 218, 221, 222, 224, 225, 227, 228, 229, 230, 231, 232, 233, 236, 237, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 251, 252, 254, 255], "62": [0, 1, 2, 4, 6, 7, 10, 11, 13, 14, 15, 16, 17, 18, 21, 22, 24, 25, 26, 27, 29, 30, 31, 32, 35, 36, 37, 38, 39, 40, 41, 43, 45, 46, 47, 48, 49, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 62, 63, 64, 65, 66, 70, 72, 73, 74, 76, 79, 80, 81, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 96, 97, 98, 99, 101, 102, 103, 104, 106, 107, 109, 111, 112, 113, 114, 115, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 135, 136, 137, 138, 140, 141, 143, 144, 147, 148, 149, 150, 151, 152, 153, 154, 156, 159, 160, 163, 164, 165, 166, 168, 169, 170, 171, 173, 174, 175, 176, 177, 179, 181, 182, 183, 185, 186, 187, 188, 190, 192, 193, 194, 195, 196, 198, 202, 205, 206, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 224, 225, 227, 228, 229, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 242, 245, 246, 247, 248, 249, 250, 251, 254, 255], "63": [1, 2, 3, 4, 5, 6, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39, 40, 44, 45, 46, 47, 48, 50, 52, 54, 55, 56, 57, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 74, 76, 78, 80, 81, 82, 83, 86, 87, 88, 90, 91, 92, 93, 94, 96, 97, 98, 99, 100, 101, 102, 104, 106, 108, 109, 111, 112, 113, 114, 116, 118, 119, 120, 121, 122, 123, 126, 127, 129, 130, 131, 133, 134, 135, 136, 137, 138, 140, 141, 143, 144, 146, 148, 149, 151, 152, 153, 154, 155, 156, 157, 158, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 171, 172, 173, 174, 176, 177, 179, 181, 183, 184, 185, 187, 190, 192, 194, 195, 196, 197, 198, 200, 201, 202, 204, 206, 207, 208, 209, 210, 211, 212, 213, 215, 216, 217, 218, 222, 225, 226, 228, 229, 230, 231, 232, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 246, 249, 250, 251, 252, 254, 255], "64": [1, 2, 3, 5, 7, 8, 9, 10, 11, 13, 15, 16, 18, 20, 21, 22, 24, 27, 28, 30, 31, 32, 33, 34, 36, 37, 38, 39, 42, 43, 44, 45, 48, 49, 50, 51, 53, 54, 55, 58, 59, 60, 62, 63, 65, 68, 69, 70, 71, 72, 73, 74, 75, 77, 78, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 93, 94, 95, 96, 98, 100, 101, 102, 103, 104, 106, 107, 108, 109, 110, 111, 113, 114, 115, 118, 119, 120, 121, 123, 124, 125, 126, 127, 128, 130, 133, 134, 135, 136, 139, 140, 141, 142, 144, 145, 146, 147, 148, 150, 152, 154, 155, 156, 157, 158, 159, 160, 162, 163, 164, 165, 166, 167, 168, 172, 173, 174, 176, 177, 178, 179, 180, 181, 182, 183, 184, 186, 187, 188, 189, 190, 192, 193, 195, 196, 198, 199, 200, 201, 203, 204, 206, 207, 208, 210, 211, 214, 215, 216, 217, 220, 221, 222, 223, 224, 228, 229, 230, 232, 233, 234, 235, 236, 237, 239, 240, 241, 242, 243, 244, 245, 247, 248, 249, 250, 251, 252, 253, 254, 255], "65": [0, 1, 2, 4, 5, 6, 8, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 23, 24, 25, 26, 27, 28, 29, 32, 33, 35, 37, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 55, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 71, 72, 73, 74, 76, 78, 79, 80, 81, 82, 84, 85, 87, 89, 90, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 106, 107, 108, 109, 110, 111, 112, 113, 115, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 131, 132, 133, 134, 137, 138, 139, 140, 141, 143, 144, 146, 147, 148, 149, 152, 154, 155, 156, 159, 160, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 175, 176, 177, 180, 181, 182, 183, 184, 187, 188, 189, 190, 192, 193, 194, 196, 199, 201, 202, 205, 207, 211, 212, 213, 214, 215, 219, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 234, 236, 238, 239, 240, 241, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254], "66": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 14, 15, 16, 17, 18, 20, 23, 24, 25, 26, 29, 31, 32, 33, 34, 35, 39, 40, 41, 43, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 68, 70, 71, 72, 73, 74, 76, 82, 83, 84, 85, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 112, 113, 114, 115, 116, 117, 120, 123, 124, 125, 126, 128, 129, 130, 131, 133, 134, 135, 137, 138, 140, 141, 143, 144, 145, 147, 148, 149, 150, 151, 153, 156, 159, 160, 161, 162, 163, 166, 167, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 185, 186, 187, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 200, 201, 202, 204, 205, 206, 207, 209, 210, 213, 214, 215, 216, 217, 218, 219, 221, 222, 223, 226, 228, 229, 230, 231, 232, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 247, 248, 250, 252, 253, 255], "67": [0, 1, 3, 4, 7, 8, 9, 10, 12, 16, 17, 18, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 40, 41, 42, 43, 46, 48, 49, 50, 51, 52, 53, 54, 55, 57, 58, 59, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 74, 75, 76, 78, 79, 80, 83, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 97, 99, 100, 102, 103, 104, 106, 107, 108, 109, 110, 111, 112, 115, 116, 118, 119, 120, 121, 124, 125, 126, 127, 130, 131, 132, 133, 134, 136, 139, 140, 143, 144, 145, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 176, 177, 178, 182, 183, 184, 188, 190, 192, 193, 194, 196, 197, 198, 199, 200, 201, 203, 206, 207, 208, 210, 211, 212, 213, 215, 216, 217, 218, 219, 221, 222, 223, 224, 226, 228, 229, 230, 231, 232, 233, 234, 235, 237, 238, 239, 242, 244, 245, 246, 247, 248, 250, 251, 252, 253, 254, 255], "68": [0, 1, 2, 3, 5, 6, 7, 8, 9, 11, 12, 13, 16, 18, 20, 21, 22, 25, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 43, 45, 46, 48, 50, 52, 53, 54, 57, 58, 59, 60, 61, 63, 65, 66, 68, 69, 71, 72, 74, 76, 78, 80, 81, 82, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 95, 96, 97, 99, 100, 101, 102, 103, 104, 105, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 121, 123, 124, 125, 126, 127, 128, 129, 132, 133, 134, 136, 139, 140, 142, 143, 144, 145, 146, 147, 148, 149, 151, 153, 154, 155, 157, 158, 159, 160, 162, 163, 164, 167, 169, 174, 175, 176, 177, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 190, 191, 193, 194, 195, 197, 198, 199, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 219, 222, 223, 224, 225, 226, 228, 229, 230, 231, 232, 234, 235, 236, 237, 238, 239, 241, 242, 243, 244, 245, 247, 248, 249, 250, 251, 253, 254], "69": [0, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 38, 41, 42, 43, 44, 45, 47, 48, 49, 51, 53, 54, 57, 60, 61, 63, 64, 66, 67, 68, 69, 71, 72, 73, 76, 77, 79, 82, 83, 84, 86, 87, 88, 89, 90, 92, 93, 94, 95, 96, 97, 98, 99, 100, 103, 104, 105, 106, 108, 109, 110, 111, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 128, 130, 131, 132, 133, 135, 136, 138, 139, 141, 142, 143, 144, 145, 146, 148, 149, 150, 151, 152, 153, 154, 155, 156, 159, 161, 163, 165, 166, 167, 168, 169, 170, 171, 173, 174, 176, 177, 178, 180, 181, 183, 184, 185, 186, 189, 192, 194, 195, 196, 197, 198, 201, 202, 203, 204, 205, 207, 208, 209, 210, 211, 214, 215, 216, 217, 218, 219, 221, 222, 224, 225, 226, 227, 228, 229, 231, 232, 233, 234, 235, 236, 237, 240, 241, 244, 245, 246, 248, 249, 250, 251, 252, 254, 255], "70": [1, 2, 3, 4, 5, 8, 9, 10, 13, 14, 16, 17, 18, 19, 20, 21, 23, 24, 25, 28, 29, 32, 33, 34, 35, 36, 37, 39, 40, 42, 43, 44, 45, 46, 49, 50, 51, 52, 53, 54, 55, 56, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 72, 73, 75, 76, 77, 79, 81, 82, 83, 84, 87, 88, 89, 94, 97, 98, 99, 101, 102, 104, 105, 106, 107, 109, 110, 112, 113, 114, 115, 117, 119, 121, 122, 123, 124, 125, 126, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 141, 142, 143, 145, 146, 147, 149, 150, 152, 153, 154, 155, 156, 157, 158, 160, 161, 162, 164, 167, 168, 169, 170, 171, 172, 173, 174, 176, 178, 179, 181, 183, 184, 185, 186, 187, 188, 189, 190, 193, 195, 196, 197, 198, 199, 200, 203, 204, 205, 206, 207, 208, 210, 211, 212, 213, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 240, 241, 242, 243, 245, 246, 248, 249, 250, 251, 252, 255], "71": [0, 1, 2, 3, 5, 7, 9, 10, 12, 13, 14, 16, 18, 19, 20, 23, 24, 25, 26, 27, 28, 32, 33, 34, 35, 36, 37, 39, 41, 42, 44, 45, 46, 47, 48, 50, 51, 52, 54, 55, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 73, 74, 76, 77, 78, 79, 80, 81, 82, 83, 84, 86, 88, 89, 90, 91, 94, 95, 96, 97, 98, 99, 101, 102, 103, 105, 106, 108, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 123, 125, 126, 129, 130, 131, 132, 133, 134, 136, 137, 138, 139, 140, 141, 142, 144, 146, 147, 148, 149, 151, 152, 153, 155, 156, 157, 161, 162, 163, 164, 165, 167, 168, 169, 170, 171, 173, 174, 175, 176, 177, 178, 181, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 211, 214, 215, 216, 217, 218, 219, 221, 222, 223, 225, 226, 227, 228, 229, 231, 232, 234, 235, 237, 238, 240, 241, 242, 243, 245, 246, 249, 251, 252, 253, 255], "72": [0, 1, 3, 4, 5, 7, 8, 9, 10, 11, 14, 15, 16, 17, 18, 19, 20, 23, 24, 26, 27, 28, 29, 30, 32, 33, 37, 38, 39, 41, 44, 46, 47, 48, 50, 51, 52, 53, 55, 56, 59, 60, 61, 62, 65, 68, 70, 71, 72, 73, 75, 76, 77, 79, 80, 83, 84, 85, 86, 87, 88, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 103, 104, 105, 106, 108, 109, 110, 111, 113, 115, 118, 119, 120, 121, 122, 124, 125, 126, 127, 129, 130, 131, 132, 133, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 147, 148, 149, 150, 153, 154, 155, 157, 158, 159, 160, 162, 163, 164, 165, 166, 167, 170, 171, 172, 174, 177, 178, 179, 181, 182, 184, 185, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 199, 200, 201, 203, 204, 205, 206, 207, 208, 211, 212, 213, 214, 216, 217, 218, 219, 220, 221, 223, 225, 226, 227, 228, 229, 230, 231, 233, 234, 235, 236, 237, 238, 239, 240, 241, 243, 245, 246, 247, 248, 249, 250, 251, 253, 254, 255], "73": [0, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 43, 44, 45, 46, 47, 48, 50, 52, 53, 54, 55, 56, 57, 59, 62, 64, 65, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 79, 80, 81, 82, 84, 85, 87, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 102, 103, 104, 106, 107, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 122, 123, 124, 126, 127, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 146, 147, 148, 149, 151, 152, 153, 154, 157, 158, 159, 161, 162, 163, 164, 165, 168, 169, 170, 171, 172, 178, 180, 181, 182, 183, 184, 185, 186, 187, 188, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 203, 206, 211, 212, 214, 215, 218, 219, 220, 222, 223, 224, 228, 230, 231, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 245, 247, 249, 250, 251, 252, 253, 254, 255], "74": [1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 13, 14, 15, 16, 17, 18, 22, 23, 25, 26, 27, 28, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 58, 59, 61, 62, 63, 64, 65, 69, 70, 71, 72, 73, 74, 75, 77, 78, 80, 81, 82, 83, 84, 86, 87, 89, 90, 92, 95, 96, 97, 98, 99, 100, 102, 103, 104, 105, 106, 107, 109, 110, 111, 113, 115, 116, 117, 118, 120, 123, 125, 126, 127, 128, 129, 131, 132, 134, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 147, 148, 149, 151, 152, 155, 156, 157, 158, 159, 160, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 174, 175, 176, 177, 178, 180, 181, 182, 184, 185, 186, 187, 188, 189, 190, 192, 194, 196, 197, 201, 203, 204, 205, 206, 207, 208, 209, 210, 212, 213, 214, 217, 219, 220, 221, 223, 224, 225, 226, 227, 228, 229, 232, 233, 234, 235, 238, 241, 242, 244, 246, 247, 248, 249, 252, 253, 254, 255], "75": [0, 1, 2, 3, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 22, 24, 26, 27, 28, 29, 30, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 50, 53, 54, 55, 58, 60, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 75, 77, 78, 79, 80, 81, 82, 83, 84, 85, 87, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 105, 106, 107, 108, 109, 111, 112, 113, 114, 118, 119, 120, 121, 123, 124, 125, 126, 128, 130, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 143, 144, 147, 149, 150, 151, 152, 154, 155, 156, 160, 161, 163, 164, 165, 166, 167, 170, 171, 172, 173, 174, 176, 178, 180, 181, 182, 183, 184, 185, 187, 188, 189, 190, 191, 193, 194, 196, 197, 200, 201, 202, 203, 204, 205, 206, 207, 209, 210, 211, 212, 213, 215, 216, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 233, 234, 239, 240, 242, 244, 246, 248, 249, 251, 252, 253, 255], "76": [0, 1, 3, 5, 6, 8, 9, 10, 12, 16, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 29, 30, 31, 33, 35, 36, 37, 38, 39, 40, 41, 43, 44, 47, 48, 50, 52, 53, 56, 57, 59, 60, 61, 64, 66, 69, 70, 71, 72, 75, 76, 78, 79, 80, 81, 82, 83, 85, 86, 87, 88, 89, 90, 91, 93, 95, 96, 97, 98, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 124, 125, 126, 127, 128, 130, 131, 132, 133, 134, 135, 137, 138, 139, 140, 141, 143, 144, 146, 147, 148, 149, 150, 151, 152, 154, 158, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 180, 181, 183, 184, 185, 186, 188, 189, 190, 191, 192, 193, 195, 196, 197, 199, 200, 201, 202, 203, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 217, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 234, 236, 237, 238, 241, 242, 243, 244, 245, 246, 247, 250, 251, 252, 254], "77": [0, 1, 2, 3, 4, 6, 7, 9, 10, 12, 14, 16, 17, 18, 19, 20, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 35, 37, 39, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 57, 59, 60, 61, 62, 63, 64, 65, 68, 69, 70, 71, 72, 73, 74, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 95, 96, 97, 98, 100, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 114, 116, 119, 122, 123, 125, 126, 127, 128, 132, 133, 134, 136, 137, 138, 139, 140, 141, 142, 145, 146, 147, 148, 149, 151, 152, 153, 154, 155, 156, 157, 159, 160, 161, 162, 163, 164, 166, 168, 169, 171, 173, 174, 175, 176, 177, 180, 183, 184, 185, 186, 188, 190, 191, 192, 193, 194, 195, 196, 197, 199, 200, 201, 202, 203, 205, 206, 207, 210, 211, 212, 214, 216, 218, 222, 223, 225, 226, 227, 228, 229, 230, 231, 234, 235, 237, 238, 239, 240, 241, 242, 243, 244, 246, 248, 249, 250, 251, 252, 253, 254, 255], "78": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 18, 19, 20, 21, 22, 26, 27, 28, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 46, 48, 49, 50, 53, 54, 55, 56, 57, 58, 59, 61, 62, 63, 65, 66, 68, 69, 70, 72, 73, 75, 76, 77, 79, 80, 83, 84, 87, 88, 89, 91, 92, 95, 96, 97, 98, 99, 100, 101, 102, 104, 105, 107, 109, 110, 111, 112, 113, 114, 115, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 132, 133, 134, 135, 136, 137, 139, 140, 141, 142, 143, 144, 145, 146, 147, 149, 150, 151, 153, 154, 155, 157, 159, 160, 162, 163, 164, 165, 166, 168, 169, 171, 172, 176, 177, 181, 182, 183, 185, 186, 189, 190, 191, 193, 194, 195, 196, 197, 198, 199, 200, 202, 204, 205, 207, 208, 211, 212, 214, 215, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 230, 234, 235, 237, 238, 239, 240, 241, 242, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254]}, "mtp_ranking": "weight magnitude from /data/reap/analysis_v2/mtp_expert_norms.json"} \ No newline at end of file diff --git a/tokenizer.json b/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..aba40197a4cdb5607f4ab7a05fb0a4ee8054fd6d --- /dev/null +++ b/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d +size 20217442 diff --git a/tokenizer_config.json b/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..0891a18eef6fc3d26dd2e87796ed39c7f91e3a77 --- /dev/null +++ b/tokenizer_config.json @@ -0,0 +1,34 @@ +{ + "backend": "tokenizers", + "clean_up_tokenization_spaces": false, + "do_lower_case": false, + "eos_token": "<|endoftext|>", + "extra_special_tokens": [ + "<|endoftext|>", + "[MASK]", + "[gMASK]", + "[sMASK]", + "", + "", + "<|system|>", + "<|user|>", + "<|assistant|>", + "<|observation|>", + "<|begin_of_image|>", + "<|end_of_image|>", + "<|begin_of_video|>", + "<|end_of_video|>", + "<|begin_of_audio|>", + "<|end_of_audio|>", + "<|begin_of_transcription|>", + "<|end_of_transcription|>" + ], + "is_local": true, + "local_files_only": false, + "model_max_length": 1048576, + "model_specific_special_tokens": {}, + "pad_token": "<|endoftext|>", + "padding_side": "left", + "remove_space": false, + "tokenizer_class": "TokenizersBackend" +}