diff --git a/.gitattributes b/.gitattributes
index a6344aac8c09253b3b630fb776ae94478aa0275b..52373fe24473b1aa44333d318f578ae6bf04b49b 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
diff --git a/README.md b/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..d68a2a7d5d487b69951fb93603a9210ac9ce6e54
--- /dev/null
+++ b/README.md
@@ -0,0 +1,95 @@
+---
+license: apache-2.0
+base_model: thinkingmachines/Inkling-Small
+base_model_relation: quantized
+pipeline_tag: image-text-to-text
+library_name: mlx
+tags:
+- mlx
+- moe
+- multimodal
+- inkling
+- thinking-machines
+---
+
+# Inkling-Small-MLX-6bit
+
+**Built with Inkling (Thinking Machines Lab).**
+
+MLX (Apple Silicon) conversion of
+[thinkingmachines/Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small),
+quantized to **6-bit** (affine group quant, group size 64).
+
+**Code / loader:** [github.com/PipeNetwork/inkling-mlx](https://github.com/PipeNetwork/inkling-mlx)
+
+Inkling Small is a **276B-total / 12B-active**
+sparse-MoE, natively multimodal model (text + image/video + audio → text). This is the
+**full multimodal** conversion: all three towers (text backbone, HMLP vision, dMel audio)
+are ported; the multi-token-prediction head is dropped (inference-irrelevant).
+
+## Builds
+
+| Variant | Size | Text ppl | Notes |
+|---|---:| ---: |---|
+| [bf16](https://huggingface.co/pipenetwork/Inkling-Small-MLX-bf16) | ~527 GB | — | reference precision |
+| [8bit](https://huggingface.co/pipenetwork/Inkling-Small-MLX-8bit) | ~280 GB | 5.569 | near-lossless |
+| [6bit](https://huggingface.co/pipenetwork/Inkling-Small-MLX-6bit) | ~214 GB | 5.569 | high quality |
+| [4bit](https://huggingface.co/pipenetwork/Inkling-Small-MLX-4bit) | ~148 GB | 5.452 | balanced default |
+| [3bit](https://huggingface.co/pipenetwork/Inkling-Small-MLX-3bit) | ~115 GB | 6.706 | ⚠️ experimental — visibly degraded |
+
+Perplexity is teacher-forcing over one fixed held-out set (prose / code / reasoning / multilingual) — identical inputs across builds, so the columns compare directly. 4-bit shows no measurable loss vs 8-bit.
+
+## Quantization scheme: affine int4 (not NVFP4 / MXFP4)
+
+MLX supports FP4 modes and Thinking Machines ships an
+[Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4) checkpoint — so for
+the record, we benchmarked round-trip reconstruction error (‖W − Ŵ‖ / ‖W‖ vs bf16) on real
+Inkling expert weights:
+
+| Scheme | bits/weight | reconstruction error |
+|---|---:|---:|
+| **affine int4** (group 64) | 4.50 | **~9.1%** |
+| nvfp4 (group 16) | 4.50 | ~10.2% |
+| mxfp4 (group 32) | 4.25 | ~12.3% |
+
+Affine int4 is the most faithful: it is *asymmetric* (per-group scale **and** zero-point, 16
+uniform levels), which centers on Inkling's near-Gaussian expert weights better than
+symmetric FP4's fixed non-uniform levels. FP4's real payoff is heavy-tailed *activations* and
+native Blackwell FP4 tensor cores — neither helps weight fidelity on Apple Silicon, where MLX
+would dequantize FP4 anyway. So these builds use affine int4.
+
+## ⚠️ Loading requires the bundled `inkling_mlx` loader
+
+The `inkling_mm_model` architecture is **not** in stock `mlx-lm` / `mlx-vlm`, so this
+repo bundles a minimal, numerically-validated MLX implementation under `inkling_mlx/`.
+
+```bash
+pip install mlx mlx-lm transformers
+```
+```python
+from inkling_mlx.load import load
+from inkling_mlx.generate import greedy_generate
+from transformers import AutoTokenizer
+
+model, config = load("/path/to/this/repo")
+tok = AutoTokenizer.from_pretrained("/path/to/this/repo", trust_remote_code=True)
+ids = tok("The capital of France is")["input_ids"]
+print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))
+```
+
+Needs an Apple-Silicon Mac with enough unified memory to hold the weights (≈ the
+size above).
+
+## Status & caveats
+
+- **Text generation** works end-to-end via an incremental KV + short-convolution cache.
+- **Multimodal** is supported end-to-end: the vision/audio towers and their
+  preprocessing (`InklingProcessor` — image patchify/normalize, audio log-mel→dMel,
+  validated ~1e-7 vs the reference) are included. Pass images/audio via the processor.
+- Quantized: attention / MLP / expert projections, token embed+unembed, and the
+  vision/audio matmuls. Kept in higher precision: the MoE router, RMSNorms, the four
+  short-convolutions per layer, and the relative-position bias.
+
+Conversion is streaming (tensor-by-tensor; the ~527 GB bf16 model never fully
+loads into RAM) and was validated with fp32 numerical parity against transformers PR #47347.
+License: Apache-2.0 (inherits the base model).
diff --git a/chat_template.jinja b/chat_template.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..04c53c7d1f9832f8cce2dab9cbde247a1c66b680
--- /dev/null
+++ b/chat_template.jinja
@@ -0,0 +1,129 @@
+{%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "max": 0.99} -%}
+{%- set role_token = {"user": "<|message_user|>", "assistant": "<|message_model|>", "system": "<|message_system|>", "tool": "<|message_tool|>"} -%}
+
+{%- macro emit_thinking_effort() -%}
+    {%- set eff = reasoning_effort if reasoning_effort is defined and reasoning_effort is not none else 0.9 -%}
+    {%- if eff is string -%}
+        {%- set key = eff | trim -%}
+        {%- if key not in effort_map -%}
+            {{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
+        {%- endif -%}
+        {%- set num = effort_map[key] -%}
+    {%- else -%}
+        {%- set num = eff | float -%}
+    {%- endif -%}
+    {%- if num < 0.0 or num > 0.99 -%}
+        {{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
+    {%- endif -%}
+    {{- "<|message_system|><|content_text|>Thinking effort level: " -}}
+    {%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
+    {{- "<|end_message|>" -}}
+{%- endmacro -%}
+
+{%- if tools -%}
+    {%- set tool_state = namespace(specs=[]) -%}
+    {%- for tool in tools -%}
+        {%- set fn = tool.function if tool.function is defined else tool -%}
+        {%- set spec = {
+            "description": (fn.description if fn.description is defined and fn.description else ""),
+            "name": fn.name,
+            "parameters": (fn.parameters if fn.parameters is defined and fn.parameters else {}),
+            "type": (tool.type if tool.type is defined and tool.type else "function"),
+        } -%}
+        {%- set tool_state.specs = tool_state.specs + [spec] -%}
+    {%- endfor -%}
+    {{- "<|message_system|>tool_declare<|content_xml|>" -}}
+    {{- tool_state.specs | tojson(sort_keys=true, separators=(",", ":")) -}}
+    {{- "<|end_message|>" -}}
+{%- endif -%}
+
+{%- set state = namespace(effort_emitted=false) -%}
+{%- for message in messages -%}
+    {%- if message.role not in role_token -%}
+        {{- raise_exception("Unknown message role: " ~ message.role) -}}
+    {%- endif -%}
+    {%- if not state.effort_emitted and message.role != "system" -%}
+        {{- emit_thinking_effort() -}}
+        {%- set state.effort_emitted = true -%}
+    {%- endif -%}
+
+    {%- set rtok = role_token[message.role] -%}
+
+    {%- if message.role == "tool" -%}
+        {%- set tool_name_state = namespace(name="") -%}
+        {%- if message.name is defined and message.name -%}
+            {%- set tool_name_state.name = message.name -%}
+        {%- elif message.tool_call_id is defined and message.tool_call_id -%}
+            {%- for prev in messages -%}
+                {%- if prev.role == "assistant" and prev.tool_calls -%}
+                    {%- for tc in prev.tool_calls -%}
+                        {%- if tc.id is defined and tc.id == message.tool_call_id and tc.function.name is defined -%}
+                            {%- set tool_name_state.name = tc.function.name -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+        {{- rtok -}}
+        {%- if tool_name_state.name -%}{{- tool_name_state.name -}}{%- endif -%}
+        {{- "<|content_text|>" -}}
+        {%- if message.content is string -%}{{- message.content -}}{%- endif -%}
+        {{- "<|end_message|>" -}}
+
+    {%- else -%}
+        {%- if message.role == "assistant" and message.reasoning_content is defined and message.reasoning_content -%}
+            {{- "<|message_model|><|content_thinking|>" ~ message.reasoning_content ~ "<|end_message|>" -}}
+        {%- endif -%}
+
+        {%- if message.content is string -%}
+            {{- rtok ~ "<|content_text|>" ~ message.content ~ "<|end_message|>" -}}
+        {%- elif message.content -%}
+            {%- for part in message.content -%}
+                {%- if part is string -%}
+                    {{- rtok ~ "<|content_text|>" ~ part ~ "<|end_message|>" -}}
+                {%- elif part.type is not defined or part.type in ("text", "input_text") -%}
+                    {%- set text_part = (part.text if part.text is defined and part.text is string else "") -%}
+                    {{- rtok ~ "<|content_text|>" ~ text_part ~ "<|end_message|>" -}}
+                {%- elif part.type in ("image", "input_image", "image_url") -%}
+                    {{- rtok ~ "<|content_image|><|unused_200054|><|end_message|>" -}}
+                {%- elif part.type in ("audio", "input_audio", "audio_url") -%}
+                    {{- rtok ~ "<|content_audio_input|><|unused_200053|><|audio_end|><|end_message|>" -}}
+                {%- else -%}
+                    {{- raise_exception("Unsupported content part type: " ~ part.type) -}}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+
+        {%- if message.role == "assistant" and message.tool_calls -%}
+            {%- for tc in message.tool_calls -%}
+                {%- set fn = tc.function -%}
+                {%- if fn.name is not defined or fn.name is not string -%}
+                    {{- raise_exception("tool call function name must be a string") -}}
+                {%- endif -%}
+                {%- set args = fn.arguments if fn.arguments is defined and fn.arguments else {} -%}
+                {%- if args is string -%}
+                    {{- raise_exception("tool call arguments must be a parsed object, not a JSON string; canonicalize upstream") -}}
+                {%- endif -%}
+                {%- if args is not mapping -%}
+                    {{- raise_exception("tool call arguments must be an object") -}}
+                {%- endif -%}
+                {{- "<|message_model|>" ~ fn.name ~ "<|content_invoke_tool_json|>" -}}
+                {{- '{"name":' ~ (fn.name | tojson(sort_keys=true, separators=(",", ":"))) ~ ',"args":' -}}
+                {{- (args | tojson(sort_keys=true, separators=(",", ":"))) -}}
+                {{- "}<|end_message|>" -}}
+            {%- endfor -%}
+        {%- endif -%}
+
+        {%- if message.role == "assistant" -%}
+            {{- "<|content_model_end_sampling|>" -}}
+        {%- endif -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if not state.effort_emitted -%}
+    {{- emit_thinking_effort() -}}
+{%- endif -%}
+
+{%- if add_generation_prompt -%}
+    {{- "<|message_model|>" -}}
+{%- endif -%}
diff --git a/config.json b/config.json
new file mode 100644
index 0000000000000000000000000000000000000000..b5578bc6e8dc1fb92f0285b4d2e9aa37b34e5770
--- /dev/null
+++ b/config.json
@@ -0,0 +1,119 @@
+{
+  "architectures": [
+    "InklingForConditionalGeneration"
+  ],
+  "model_type": "inkling_mm_model",
+  "eos_token_id": 200006,
+  "text_config": {
+    "model_max_length": 1048576,
+    "torch_dtype": "bfloat16",
+    "hidden_size": 4096,
+    "num_hidden_layers": 42,
+    "vocab_size": 201024,
+    "num_attention_heads": 32,
+    "num_key_value_heads": 8,
+    "head_dim": 128,
+    "d_rel": 16,
+    "rel_extent": 1024,
+    "q_bias": false,
+    "o_bias": false,
+    "log_scaling_n_floor": 128000,
+    "log_scaling_alpha": 0.1,
+    "rms_norm_eps": 1e-06,
+    "use_embed_norm": true,
+    "local_layer_ids": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      6,
+      7,
+      8,
+      9,
+      10,
+      12,
+      13,
+      14,
+      15,
+      16,
+      18,
+      19,
+      20,
+      21,
+      22,
+      24,
+      25,
+      26,
+      27,
+      28,
+      30,
+      31,
+      32,
+      33,
+      34,
+      36,
+      37,
+      38,
+      39,
+      40
+    ],
+    "dense_mlp_idx": 2,
+    "use_sconv": true,
+    "sconv_kernel_size": 4,
+    "unpadded_vocab_size": 200058,
+    "logits_mup_width_multiplier": 16.0,
+    "final_logit_softcapping": null,
+    "swa_head_dim": 128,
+    "swa_num_attention_heads": 32,
+    "swa_num_key_value_heads": 8,
+    "sliding_window_size": 512,
+    "n_routed_experts": 256,
+    "num_experts_per_tok": 6,
+    "n_shared_experts": 2,
+    "shared_expert_sink": true,
+    "dense_intermediate_size": 16384,
+    "intermediate_size": 2048,
+    "route_scale": 8.0,
+    "use_gate_bias": true,
+    "gate_activation": "sigmoid",
+    "norm_after_topk": true,
+    "use_global_scale": true
+  },
+  "audio_config": {
+    "decoder_dmodel": 4096,
+    "n_mel_bins": 80,
+    "mel_vocab_size": 16,
+    "bias": false,
+    "dmel_min_value": -7.0,
+    "dmel_max_value": 2.0,
+    "use_audio_norm": true,
+    "audio_mode": "dmel"
+  },
+  "vision_config": {
+    "vision_encoder_type": "hmlp",
+    "decoder_dmodel": 4096,
+    "patch_size": 40,
+    "temporal_patch_size": 2,
+    "n_channels": 3,
+    "n_layers": 4,
+    "use_vision_norm": true
+  },
+  "mtp_config": {
+    "num_nextn_predict_layers": 8,
+    "chain_hidden_post_norm": false,
+    "local_layer_ids": [
+      0,
+      2,
+      4,
+      5,
+      6,
+      7
+    ]
+  },
+  "quantization": {
+    "group_size": 64,
+    "bits": 6,
+    "recipe": "uniform"
+  }
+}
\ No newline at end of file
diff --git a/inkling_mlx/__init__.py b/inkling_mlx/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..febf41b1d21e148136935345d731796b5d8b6c76
--- /dev/null
+++ b/inkling_mlx/__init__.py
@@ -0,0 +1,14 @@
+"""MLX port of thinkingmachines/Inkling (975B MoE, natively multimodal)."""
+
+from .config import AudioConfig, InklingConfig, TextConfig, VisionConfig
+from .model import InklingForConditionalGeneration
+from .text import TextModel
+
+__all__ = [
+    "InklingConfig",
+    "TextConfig",
+    "VisionConfig",
+    "AudioConfig",
+    "InklingForConditionalGeneration",
+    "TextModel",
+]
diff --git a/inkling_mlx/attention.py b/inkling_mlx/attention.py
new file mode 100644
index 0000000000000000000000000000000000000000..2113158a6b7eff4078b3833e9d2cf29284cdf66e
--- /dev/null
+++ b/inkling_mlx/attention.py
@@ -0,0 +1,131 @@
+"""Inkling attention: hybrid local/global, per-head q/k RMSNorm, relative-position
+logits bias, optional log-scaling, and short-convolution on k/v.
+
+Mirrors ``InklingAttention`` + ``InklingRelativeLogits`` from transformers PR #47347.
+This implementation is prefill-oriented (full-sequence, no KV cache); an incremental
+cache (including the 4 per-layer conv states) can be layered on top later.
+"""
+
+from __future__ import annotations
+
+import math
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from .common import RMSNorm, ShortConvolution
+from .config import TextConfig
+
+NEG_INF = -1e30
+
+
+class RelativeLogits(nn.Module):
+    """Hidden-state-conditioned relative position bias.
+
+    ``proj`` is a bank of bias-vs-distance profiles ``[d_rel, rel_extent]``. Each
+    query's ``d_rel`` relative-state vector mixes them into one bias value per
+    backward distance; the bias is zero outside ``0 <= distance < rel_extent``.
+    """
+
+    def __init__(self, d_rel: int, rel_extent: int):
+        super().__init__()
+        self.rel_extent = rel_extent
+        self.proj = mx.zeros((d_rel, rel_extent))
+
+    def __call__(self, relative_states, q_pos, kv_pos):
+        # relative_states: [B, Lq, heads, d_rel]
+        # rel_logits: [B, Lq, heads, rel_extent] -> [B, heads, Lq, rel_extent]
+        rel_logits = mx.swapaxes(relative_states @ self.proj, 1, 2)
+        B, H, Lq, _ = rel_logits.shape
+        distance = q_pos[:, None] - kv_pos[None, :]          # [Lq, Lkv]
+        gather = mx.clip(distance, 0, self.rel_extent - 1)   # [Lq, Lkv]
+        gather = mx.broadcast_to(gather[None, None], (B, H, Lq, gather.shape[-1]))
+        bias = mx.take_along_axis(rel_logits, gather, axis=-1)  # [B, H, Lq, Lkv]
+        valid = (distance >= 0) & (distance < self.rel_extent)  # [Lq, Lkv]
+        return mx.where(valid[None, None], bias, 0.0)
+
+
+class Attention(nn.Module):
+    def __init__(self, config: TextConfig, layer_idx: int):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.is_sliding = config.layer_types[layer_idx] == "hybrid_sliding"
+
+        self.head_dim = config.swa_head_dim if self.is_sliding else config.head_dim
+        self.num_heads = config.swa_num_attention_heads if self.is_sliding else config.num_attention_heads
+        self.num_kv_heads = config.swa_num_key_value_heads if self.is_sliding else config.num_key_value_heads
+        self.n_rep = self.num_heads // self.num_kv_heads
+        self.sliding_window = config.sliding_window_size if self.is_sliding else None
+        self.rel_extent = config.sliding_window_size if self.is_sliding else config.rel_extent
+        self.d_rel = config.d_rel
+        # q/k are per-head RMS-normalized, hence 1/d rather than 1/sqrt(d)
+        self.scaling = 1.0 / self.head_dim
+
+        h = config.hidden_size
+        self.wq_du = nn.Linear(h, self.num_heads * self.head_dim, bias=False)
+        self.wk_dv = nn.Linear(h, self.num_kv_heads * self.head_dim, bias=False)
+        self.wv_dv = nn.Linear(h, self.num_kv_heads * self.head_dim, bias=False)
+        self.wr_du = nn.Linear(h, self.num_heads * self.d_rel, bias=False)
+        self.wo_ud = nn.Linear(self.num_heads * self.head_dim, h, bias=False)
+
+        self.k_sconv = ShortConvolution(self.num_kv_heads * self.head_dim, config.sconv_kernel_size)
+        self.v_sconv = ShortConvolution(self.num_kv_heads * self.head_dim, config.sconv_kernel_size)
+        self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
+        self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
+        self.rel_logits_proj = RelativeLogits(self.d_rel, self.rel_extent)
+
+    def __call__(self, hidden_states, start_pos=0, kv_cache=None,
+                 k_conv=None, v_conv=None, conv_mask=None):
+        B, L, _ = hidden_states.shape
+
+        q = self.wq_du(hidden_states)
+        k = self.k_sconv(self.wk_dv(hidden_states), mask=conv_mask, cache=k_conv)
+        v = self.v_sconv(self.wv_dv(hidden_states), mask=conv_mask, cache=v_conv)
+        rel = self.wr_du(hidden_states)
+
+        q = self.q_norm(q.reshape(B, L, self.num_heads, self.head_dim))
+        k = self.k_norm(k.reshape(B, L, self.num_kv_heads, self.head_dim))
+        v = v.reshape(B, L, self.num_kv_heads, self.head_dim)
+
+        # -> [B, heads, L, head_dim]
+        q = q.transpose(0, 2, 1, 3)
+        k = k.transpose(0, 2, 1, 3)
+        v = v.transpose(0, 2, 1, 3)
+
+        q_pos = mx.arange(L) + start_pos
+        if kv_cache is not None:
+            k, v = kv_cache.update(k, v)      # full history
+        kv_pos = mx.arange(k.shape[2])
+
+        rel = rel.reshape(B, L, self.num_heads, self.d_rel)
+        position_bias = self.rel_logits_proj(rel, q_pos, kv_pos)  # [B, heads, Lq, Lkv]
+
+        # log-scaling (global layers only; no-op for context <= n_floor)
+        if not self.is_sliding and self.config.log_scaling_n_floor is not None:
+            n_floor = self.config.log_scaling_n_floor
+            eff_n = (q_pos + 1).astype(mx.float32)
+            tau = 1.0 + self.config.log_scaling_alpha * mx.log(
+                mx.maximum(eff_n / n_floor, 1.0)
+            )
+            tau_q = tau.reshape(1, 1, -1, 1)
+            q = (q.astype(mx.float32) * tau_q).astype(q.dtype)
+            position_bias = (position_bias.astype(mx.float32) * tau_q).astype(position_bias.dtype)
+
+        # Fused SDPA: O = softmax(scale·QKᵀ + mask)·V, softmax in fp32, native GQA
+        # (k/v passed un-tiled). mask = relative-position bias + causal/sliding, additive.
+        mask = position_bias + self._causal_mask(q_pos, kv_pos)   # [B, heads, Lq, Lkv]
+        out = mx.fast.scaled_dot_product_attention(
+            q, k, v, scale=self.scaling, mask=mask.astype(q.dtype)
+        )                                                        # [B, heads, Lq, head_dim]
+
+        out = out.transpose(0, 2, 1, 3).reshape(B, L, self.num_heads * self.head_dim)
+        return self.wo_ud(out)
+
+    def _causal_mask(self, q_pos, kv_pos):
+        distance = q_pos[:, None] - kv_pos[None, :]  # [Lq, Lkv]
+        allowed = distance >= 0
+        if self.sliding_window is not None:
+            allowed = allowed & (distance < self.sliding_window)
+        mask = mx.where(allowed, 0.0, NEG_INF)
+        return mask[None, None].astype(mx.float32)
diff --git a/inkling_mlx/audio.py b/inkling_mlx/audio.py
new file mode 100644
index 0000000000000000000000000000000000000000..c0a7472a6209176b449bf0e299bdf301aedaefcc
--- /dev/null
+++ b/inkling_mlx/audio.py
@@ -0,0 +1,34 @@
+"""Inkling audio tower: discrete dMel-token embedding + norm.
+
+Each audio frame is ``n_mel_bins`` discretized bins (values in ``[0, mel_vocab_size)``);
+each bin is embedded from its own slice of a shared table (offset ``bin * mel_vocab_size``)
+and the per-bin embeddings are summed. Mirrors ``InklingAudioModel`` /
+``InklingAudioModelEmbeddings``. Checkpoint keys: ``audio.encoder.weight`` (the
+``[n_mel_bins*mel_vocab_size, hidden]`` table) and ``audio.final_norm.weight``.
+"""
+
+from __future__ import annotations
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from .common import RMSNorm
+from .config import AudioConfig
+
+
+class AudioModel(nn.Module):
+    def __init__(self, config: AudioConfig):
+        super().__init__()
+        self.config = config
+        self.encoder = nn.Embedding(
+            config.n_mel_bins * config.mel_vocab_size, config.text_hidden_size
+        )
+        self.final_norm = RMSNorm(config.text_hidden_size, eps=config.rms_norm_eps)
+        # non-persistent: arange(n_mel_bins) * mel_vocab_size
+        self._offsets = mx.arange(config.n_mel_bins) * config.mel_vocab_size
+
+    def __call__(self, audio_input_ids: mx.array) -> mx.array:
+        # audio_input_ids: [..., n_mel_bins] with values in [0, mel_vocab_size)
+        embeds = self.encoder(audio_input_ids + self._offsets)  # [..., n_mel_bins, hidden]
+        embeds = embeds.sum(axis=-2)                            # [..., hidden]
+        return self.final_norm(embeds)
diff --git a/inkling_mlx/cache.py b/inkling_mlx/cache.py
new file mode 100644
index 0000000000000000000000000000000000000000..e20af1f55d29a5b85e649e151acc6e1af7841fd3
--- /dev/null
+++ b/inkling_mlx/cache.py
@@ -0,0 +1,66 @@
+"""Incremental caches for Inkling generation.
+
+Two kinds of per-layer state must persist across decode steps:
+
+* ``KVCache``   — the appended key/value tensors for attention.
+* ``ConvCache`` — the last ``kernel-1`` inputs of each depthwise short-convolution
+  (there are 4 per layer: k, v, post-attn, post-mlp).
+
+A ``LayerCache`` bundles one KVCache + the 4 ConvCaches; ``make_cache`` builds one
+per decoder layer. Absolute key positions are always ``arange(kv_len)`` because the
+cache holds every key from position 0 (KVCache keeps the full history — correct for
+both global and sliding layers, since the sliding-window constraint is enforced by
+the attention mask).
+"""
+
+from __future__ import annotations
+
+import mlx.core as mx
+
+
+class ConvCache:
+    """Holds the last ``kernel-1`` inputs of a short convolution."""
+
+    __slots__ = ("state",)
+
+    def __init__(self):
+        self.state = None  # [B, kernel-1, C] or None
+
+
+class KVCache:
+    """Appends keys/values along the sequence axis (full history)."""
+
+    __slots__ = ("keys", "values")
+
+    def __init__(self):
+        self.keys = None    # [B, heads, T, d]
+        self.values = None
+
+    @property
+    def offset(self) -> int:
+        return 0 if self.keys is None else self.keys.shape[2]
+
+    def update(self, k: mx.array, v: mx.array):
+        if self.keys is None:
+            self.keys, self.values = k, v
+        else:
+            self.keys = mx.concatenate([self.keys, k], axis=2)
+            self.values = mx.concatenate([self.values, v], axis=2)
+        return self.keys, self.values
+
+
+class LayerCache:
+    __slots__ = ("kv", "k_conv", "v_conv", "attn_conv", "mlp_conv")
+
+    def __init__(self):
+        self.kv = KVCache()
+        self.k_conv = ConvCache()
+        self.v_conv = ConvCache()
+        self.attn_conv = ConvCache()
+        self.mlp_conv = ConvCache()
+
+
+def make_cache(model) -> list[LayerCache]:
+    """One LayerCache per text decoder layer."""
+    n = len(model.model.llm.layers) if hasattr(model, "model") else len(model.layers)
+    return [LayerCache() for _ in range(n)]
diff --git a/inkling_mlx/common.py b/inkling_mlx/common.py
new file mode 100644
index 0000000000000000000000000000000000000000..8ecd3dcf78b3057ceb7a31a8282ea2fce0fd74fe
--- /dev/null
+++ b/inkling_mlx/common.py
@@ -0,0 +1,63 @@
+"""Shared low-level modules for the Inkling MLX port."""
+
+from __future__ import annotations
+
+import mlx.core as mx
+import mlx.nn as nn
+
+
+class RMSNorm(nn.Module):
+    """Llama-style RMSNorm (compute in fp32, weight is a gain).
+
+    Matches ``LlamaRMSNorm``: ``x_fp32 * rsqrt(mean(x^2) + eps) * weight``.
+    """
+
+    def __init__(self, dims: int, eps: float = 1e-6):
+        super().__init__()
+        self.weight = mx.ones((dims,))
+        self.eps = eps
+
+    def __call__(self, x: mx.array) -> mx.array:
+        return mx.fast.rms_norm(x, self.weight, self.eps)
+
+
+class ShortConvolution(nn.Module):
+    """Depthwise causal 1-D convolution with a residual add, computed in fp32.
+
+    Mirrors ``InklingShortConvolution``: a per-channel (groups == channels) causal
+    conv1d of ``kernel_size`` taps, no bias, no activation, then ``out + input``.
+    The reference keeps this module in fp32 regardless of the model dtype
+    (``_keep_in_fp32_modules_strict``), so we upcast here too.
+
+    Weight layout (MLX ``conv1d``): ``[channels, kernel_size, 1]``.
+    """
+
+    def __init__(self, channels: int, kernel_size: int):
+        super().__init__()
+        self.channels = channels
+        self.kernel_size = kernel_size
+        # [C_out, K, C_in // groups] with groups == channels -> [C, K, 1]
+        self.weight = mx.zeros((channels, kernel_size, 1))
+
+    def __call__(self, x: mx.array, mask: mx.array | None = None, cache=None) -> mx.array:
+        # x: [batch, seq, channels]
+        in_dtype = x.dtype
+        xf = x.astype(mx.float32)
+        residual = xf
+        if mask is not None:
+            xf = xf * mask.astype(mx.float32)
+        k = self.kernel_size
+        B, seq, C = xf.shape
+        w = self.weight.astype(mx.float32)
+        if cache is not None:
+            # left-context = cached last (k-1) inputs (zeros on the first call);
+            # a "valid" conv over [left, xf] yields exactly `seq` causal outputs.
+            left = cache.state if cache.state is not None else mx.zeros((B, k - 1, C), dtype=mx.float32)
+            x_in = mx.concatenate([left, xf], axis=1)
+            out = mx.conv1d(x_in, w, padding=0, groups=self.channels)
+            cache.state = x_in[:, -(k - 1):, :]
+        else:
+            # causal: left-pad by (k-1), keep first `seq` outputs (== zero left-context)
+            out = mx.conv1d(xf, w, padding=k - 1, groups=self.channels)[:, :seq, :]
+        out = out + residual
+        return out.astype(in_dtype)
diff --git a/inkling_mlx/config.py b/inkling_mlx/config.py
new file mode 100644
index 0000000000000000000000000000000000000000..ab24c4c5a69fa4b28401381782a2289da6bc1bec
--- /dev/null
+++ b/inkling_mlx/config.py
@@ -0,0 +1,207 @@
+"""Configuration for the Inkling multimodal model (MLX port).
+
+Mirrors ``thinkingmachines/Inkling`` ``config.json`` and the transformers PR #47347
+reference (``InklingConfig`` / ``InklingTextConfig`` / ``InklingVisionConfig`` /
+``InklingAudioConfig``). We parse the *checkpoint* config layout (top-level
+``text_config`` / ``vision_config`` / ``audio_config`` / ``mtp_config``), not the
+flattened transformers layout.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+
+def _get(d: dict, *names, default=None):
+    for n in names:
+        if n in d and d[n] is not None:
+            return d[n]
+    return default
+
+
+@dataclass
+class TextConfig:
+    hidden_size: int = 6144
+    num_hidden_layers: int = 66
+    vocab_size: int = 201024
+    unpadded_vocab_size: int | None = 200058
+
+    # global (full) attention
+    num_attention_heads: int = 64
+    num_key_value_heads: int = 8
+    head_dim: int = 128
+    # sliding-window attention
+    swa_num_attention_heads: int = 64
+    swa_num_key_value_heads: int = 16
+    swa_head_dim: int = 128
+    sliding_window_size: int = 512
+
+    # relative-position logits
+    d_rel: int = 16
+    rel_extent: int = 1024
+    log_scaling_n_floor: int | None = 128000
+    log_scaling_alpha: float = 0.1
+
+    rms_norm_eps: float = 1e-6
+    use_embed_norm: bool = True
+
+    # short convolution
+    sconv_kernel_size: int = 4
+
+    # dense vs MoE MLP
+    dense_mlp_idx: int = 2
+    dense_intermediate_size: int = 24576  # dense MLP intermediate
+    moe_intermediate_size: int = 3072     # per-expert intermediate
+
+    # MoE routing
+    n_routed_experts: int = 256
+    num_experts_per_tok: int = 6
+    n_shared_experts: int = 2
+    shared_expert_sink: bool = True
+    route_scale: float = 8.0
+    use_gate_bias: bool = True
+    norm_after_topk: bool = True
+    use_global_scale: bool = True
+
+    logits_mup_width_multiplier: float = 24.0
+    hidden_act: str = "silu"
+
+    max_position_embeddings: int = 1048576
+
+    # which layer indices use sliding-window ("local") attention
+    local_layer_ids: list[int] = field(default_factory=list)
+
+    # MTP head (dropped for inference)
+    num_mtp_layers: int | None = None
+
+    @property
+    def layer_types(self) -> list[str]:
+        local = set(self.local_layer_ids)
+        return [
+            "hybrid_sliding" if i in local else "hybrid"
+            for i in range(self.num_hidden_layers)
+        ]
+
+    @property
+    def mlp_layer_types(self) -> list[str]:
+        return [
+            "dense" if i < self.dense_mlp_idx else "sparse"
+            for i in range(self.num_hidden_layers)
+        ]
+
+    @classmethod
+    def from_dict(cls, tc: dict) -> "TextConfig":
+        return cls(
+            hidden_size=_get(tc, "hidden_size", default=6144),
+            num_hidden_layers=_get(tc, "num_hidden_layers", default=66),
+            vocab_size=_get(tc, "vocab_size", default=201024),
+            unpadded_vocab_size=_get(tc, "unpadded_vocab_size"),
+            num_attention_heads=_get(tc, "num_attention_heads", default=64),
+            num_key_value_heads=_get(tc, "num_key_value_heads", default=8),
+            head_dim=_get(tc, "head_dim", default=128),
+            swa_num_attention_heads=_get(tc, "swa_num_attention_heads", default=64),
+            swa_num_key_value_heads=_get(tc, "swa_num_key_value_heads", default=16),
+            swa_head_dim=_get(tc, "swa_head_dim", default=128),
+            sliding_window_size=_get(tc, "sliding_window_size", default=512),
+            d_rel=_get(tc, "d_rel", default=16),
+            rel_extent=_get(tc, "rel_extent", default=1024),
+            log_scaling_n_floor=_get(tc, "log_scaling_n_floor"),
+            log_scaling_alpha=_get(tc, "log_scaling_alpha", default=0.1),
+            rms_norm_eps=_get(tc, "rms_norm_eps", default=1e-6),
+            use_embed_norm=_get(tc, "use_embed_norm", default=True),
+            sconv_kernel_size=_get(tc, "sconv_kernel_size", default=4),
+            dense_mlp_idx=_get(tc, "dense_mlp_idx", default=2),
+            dense_intermediate_size=_get(tc, "dense_intermediate_size", default=24576),
+            # checkpoint labels the *MoE* intermediate as `intermediate_size`
+            moe_intermediate_size=_get(tc, "intermediate_size", default=3072),
+            n_routed_experts=_get(tc, "n_routed_experts", default=256),
+            num_experts_per_tok=_get(tc, "num_experts_per_tok", default=6),
+            n_shared_experts=_get(tc, "n_shared_experts", default=2),
+            shared_expert_sink=_get(tc, "shared_expert_sink", default=True),
+            route_scale=_get(tc, "route_scale", default=8.0),
+            use_gate_bias=_get(tc, "use_gate_bias", default=True),
+            norm_after_topk=_get(tc, "norm_after_topk", default=True),
+            use_global_scale=_get(tc, "use_global_scale", default=True),
+            logits_mup_width_multiplier=_get(tc, "logits_mup_width_multiplier", default=24.0),
+            max_position_embeddings=_get(tc, "model_max_length", "max_position_embeddings", default=1048576),
+            local_layer_ids=list(_get(tc, "local_layer_ids", default=[]) or []),
+        )
+
+
+@dataclass
+class VisionConfig:
+    text_hidden_size: int = 6144
+    patch_size: int = 40
+    temporal_patch_size: int = 2
+    num_channels: int = 3
+    n_layers: int = 4
+    rms_norm_eps: float = 1e-6
+    use_vision_norm: bool = True
+
+    @classmethod
+    def from_dict(cls, vc: dict, text_hidden: int) -> "VisionConfig":
+        return cls(
+            text_hidden_size=text_hidden,
+            patch_size=_get(vc, "patch_size", default=40),
+            temporal_patch_size=_get(vc, "temporal_patch_size", default=2),
+            num_channels=_get(vc, "n_channels", "num_channels", default=3),
+            n_layers=_get(vc, "n_layers", "num_hidden_layers", default=4),
+            rms_norm_eps=_get(vc, "rms_norm_eps", default=1e-6),
+            use_vision_norm=_get(vc, "use_vision_norm", default=True),
+        )
+
+
+@dataclass
+class AudioConfig:
+    text_hidden_size: int = 6144
+    n_mel_bins: int = 80
+    mel_vocab_size: int = 16
+    rms_norm_eps: float = 1e-6
+
+    @classmethod
+    def from_dict(cls, ac: dict, text_hidden: int) -> "AudioConfig":
+        return cls(
+            text_hidden_size=text_hidden,
+            n_mel_bins=_get(ac, "n_mel_bins", default=80),
+            mel_vocab_size=_get(ac, "mel_vocab_size", default=16),
+            rms_norm_eps=_get(ac, "rms_norm_eps", default=1e-6),
+        )
+
+
+@dataclass
+class InklingConfig:
+    text: TextConfig
+    vision: VisionConfig
+    audio: AudioConfig
+    image_token_id: int = 200054
+    audio_token_id: int = 200053
+    image_bos_token_id: int = 200005
+    audio_bos_token_id: int = 200020
+    eos_token_id: int = 200006
+    model_type: str = "inkling_mm_model"
+
+    @classmethod
+    def from_dict(cls, cfg: dict) -> "InklingConfig":
+        tc = dict(cfg.get("text_config", {}))
+        mtp = cfg.get("mtp_config") or {}
+        if mtp.get("num_nextn_predict_layers") is not None:
+            tc.setdefault("num_mtp_layers", mtp.get("num_nextn_predict_layers"))
+        text = TextConfig.from_dict(tc)
+        vision = VisionConfig.from_dict(cfg.get("vision_config", {}) or {}, text.hidden_size)
+        audio = AudioConfig.from_dict(cfg.get("audio_config", {}) or {}, text.hidden_size)
+        return cls(
+            text=text,
+            vision=vision,
+            audio=audio,
+            image_token_id=_get(cfg, "image_token_id", default=200054),
+            audio_token_id=_get(cfg, "audio_token_id", default=200053),
+            image_bos_token_id=_get(cfg, "image_bos_token_id", default=200005),
+            audio_bos_token_id=_get(cfg, "audio_bos_token_id", default=200020),
+            eos_token_id=_get(cfg, "eos_token_id", default=200006),
+            model_type=_get(cfg, "model_type", default="inkling_mm_model"),
+        )
+
+    @property
+    def raw(self) -> dict[str, Any]:
+        return {"model_type": self.model_type}
diff --git a/inkling_mlx/convert.py b/inkling_mlx/convert.py
new file mode 100644
index 0000000000000000000000000000000000000000..15e0b7b4518ff3aab5147ad10d157323c36dd244
--- /dev/null
+++ b/inkling_mlx/convert.py
@@ -0,0 +1,254 @@
+"""Streaming HF -> MLX conversion + quantization for Inkling.
+
+The model is far too large (~1.9 TB bf16) to instantiate in RAM, so we convert
+tensor-by-tensor: read each source shard (mmap), remap the name, apply the layout
+transform, optionally affine-quantize the weight, and write output shards. Affine
+quantization has no cross-tensor dependency, so per-tensor streaming is exactly
+equivalent to ``nn.quantize(model)``.
+
+Name/layout transforms vs. the checkpoint:
+  * ``*_sconv.weight``  [C,1,K] -> [C,K,1]            (MLX conv1d layout)
+  * ``mlp.w13_dn``      [2I,H]  -> gate_proj/up_proj   (split dense fused gate+up)
+  * ``experts.w13_weight``        [E,2I,H] -> gate_proj/up_proj (split)
+  * ``experts.w2_weight``         [E,H,I]  -> down_proj          (identity)
+  * ``shared_experts.shared_w13`` [2,2I,H] -> gate_proj/up_proj  (split)
+  * ``model.mtp.*``               dropped (inference-irrelevant)
+  * everything else: identity
+"""
+
+from __future__ import annotations
+
+import glob
+import json
+import os
+import re
+import shutil
+
+import mlx.core as mx
+
+_LAYER_RE = re.compile(r"model\.llm\.layers\.(\d+)\.")
+_N_SHARED = 2   # Inkling: 2 shared experts (router rows after the routed ones)
+
+
+def prune_moe_tensor(name: str, w: mx.array, keep, dense_mlp_idx: int) -> mx.array:
+    """Subset a MoE tensor to the kept experts (REAP). ``keep`` maps sparse-layer
+    index -> array of kept routed-expert indices. Applied BEFORE map_name/transform.
+    Routed experts + the router (gate) are subset; shared experts are untouched."""
+    m = _LAYER_RE.search(name)
+    if m is None:
+        return w
+    L = int(m.group(1))
+    if L < dense_mlp_idx:                        # dense layer — no routed experts
+        return w
+    kidx = mx.array(keep[L - dense_mlp_idx])     # [K]
+    if name.endswith(("experts.w13_weight", "experts.w2_weight")) and "shared" not in name:
+        return w[kidx]                           # [E, ...] -> [K, ...]
+    if name.endswith("mlp.gate.weight"):         # [n_routed + n_shared, hidden]
+        n_routed = w.shape[0] - _N_SHARED        # rows: [routed... , shared...]
+        return mx.concatenate([w[kidx], w[n_routed:]], axis=0)   # [K + n_shared, hidden]
+    if name.endswith("mlp.gate.bias"):           # [n_routed] correction bias
+        return w[kidx]
+    return w                                     # shared_experts.*, gate.global_scale, etc.
+
+
+def map_name(name: str):
+    """HF checkpoint tensor name -> list of (out_name, kind) for the MLX model."""
+    if name.startswith("model.mtp."):
+        return []  # drop MTP head
+
+    if name.endswith(("k_sconv.weight", "v_sconv.weight", "attn_sconv.weight", "mlp_sconv.weight")):
+        return [(name, "sconv")]
+
+    # dense MLP fused gate+up / down
+    if name.endswith("mlp.w13_dn.weight"):
+        base = name[: -len("w13_dn.weight")]
+        return [(base + "gate_proj.weight", "w13_gate"), (base + "up_proj.weight", "w13_up")]
+    if name.endswith("mlp.w2_md.weight"):
+        return [(name[: -len("w2_md.weight")] + "down_proj.weight", "identity")]
+
+    # routed experts fused
+    if name.endswith("experts.w13_weight"):
+        base = name[: -len("w13_weight")]
+        return [(base + "gate_proj.weight", "w13_gate"), (base + "up_proj.weight", "w13_up")]
+    if name.endswith("experts.w2_weight"):
+        return [(name[: -len("w2_weight")] + "down_proj.weight", "identity")]
+
+    # shared experts fused
+    if name.endswith("shared_experts.shared_w13_weight"):
+        base = name[: -len("shared_w13_weight")]
+        return [(base + "gate_proj.weight", "w13_gate"), (base + "up_proj.weight", "w13_up")]
+    if name.endswith("shared_experts.shared_w2_weight"):
+        return [(name[: -len("shared_w2_weight")] + "down_proj.weight", "identity")]
+
+    return [(name, "identity")]
+
+
+def transform(w: mx.array, kind: str) -> mx.array:
+    if kind == "identity":
+        return w
+    if kind == "sconv":
+        # [C, 1, K] -> [C, K, 1]
+        return mx.swapaxes(w, 1, 2)
+    if kind in ("w13_gate", "w13_up"):
+        # The checkpoint stores gate/up INTERLEAVED row-wise: [g0, u0, g1, u1, ...]
+        # (SGLang `deinterleave_w13`). De-interleave: gate = rows 0::2, up = rows 1::2.
+        # A contiguous [:half]/[half:] split scrambles gate<->up in every MLP.
+        n = w.shape[-2] // 2
+        g = w.reshape(*w.shape[:-2], n, 2, w.shape[-1])
+        return g[..., 0, :] if kind == "w13_gate" else g[..., 1, :]
+    raise ValueError(kind)
+
+
+# ---- quantization target predicate (must be identical in convert and load) ----
+
+# Quant "recipes" — which module leaves get affine-quantized.
+#   uniform       : everything (attention, MLP/experts, embed/unembed, audio, vision)
+#   experts_only  : ONLY the MLP/expert matmuls (+ audio/vision); attention and
+#                   embed/unembed stay bf16. Inkling attention dominates 4-bit error
+#                   (~58% per layer vs ~15% for experts), so this keeps a 4-bit-sized
+#                   build coherent while the ~927 B experts still fit in 512 GB.
+_RECIPES = {
+    "uniform": {"wq_du", "wk_dv", "wv_dv", "wr_du", "wo_ud",
+                "gate_proj", "up_proj", "down_proj", "embed", "unembed", "encoder"},
+    "experts_only": {"gate_proj", "up_proj", "down_proj", "encoder"},
+}
+
+
+def is_quant_target(out_name: str, quant_axis_size: int, group_size: int, recipe: str = "uniform") -> bool:
+    """Whether ``out_name`` (a converted param path) should be affine-quantized."""
+    if not out_name.endswith(".weight"):
+        return False
+    leaf = out_name[: -len(".weight")].rsplit(".", 1)[-1]
+    leaves = _RECIPES[recipe]
+    # vision projection layers (linear_0 .. linear_3) — quantized in both recipes
+    is_vision_linear = leaf.startswith("linear_") and ".visual." in out_name
+    if leaf not in leaves and not is_vision_linear:
+        return False
+    # router gate stays fp (leaf == "gate", excluded above); norms/sconv excluded by leaf
+    # can only group-quantize when the input dim is a multiple of group_size
+    return quant_axis_size % group_size == 0
+
+
+# ------------------------------ streaming driver ------------------------------
+
+_SHARD_CAP_BYTES = 5_000_000_000  # ~5 GB per output shard
+
+
+def _process_tensor(name, w, bits, group_size, out_dtype, recipe="uniform", keep=None, dmi=2):
+    """Yield (out_name, array) pairs for one source tensor."""
+    if keep is not None:
+        w = prune_moe_tensor(name, w, keep, dmi)   # REAP: subset to kept experts
+    for out_name, kind in map_name(name):
+        wt = transform(w, kind)
+        quantize = bits is not None and is_quant_target(out_name, wt.shape[-1], group_size, recipe)
+        if quantize:
+            qw, scales, biases = mx.quantize(wt, group_size=group_size, bits=bits)
+            base = out_name[: -len(".weight")]
+            yield out_name, qw
+            yield base + ".scales", scales
+            yield base + ".biases", biases
+        else:
+            # keep norms/router/sconv/rel-proj in fp32-safe dtype; matmul weights in out_dtype
+            keep_hi = wt.dtype == mx.float32 and (".global_scale" in out_name or ".bias" in out_name
+                                                  or out_name.endswith(("_norm.weight", "norm.weight")))
+            yield out_name, wt.astype(mx.float32 if keep_hi else out_dtype)
+
+
+def convert_model(src: str, dst: str, bits=None, group_size: int = 64, out_dtype=mx.bfloat16,
+                  recipe: str = "uniform", keep_path=None):
+    """Stream-convert an Inkling checkpoint from ``src`` to ``dst``.
+
+    ``bits=None`` -> plain dtype cast (bf16). ``bits in {4,6,8}`` -> affine quant.
+    ``recipe`` selects which modules are quantized (see ``_RECIPES``).
+    ``keep_path`` (REAP): npz with ``keep`` [n_sparse_layers, K] + ``dense_mlp_idx`` ->
+    prune each MoE layer to its K kept experts and set ``n_routed_experts=K`` in config.
+    Processes one source shard at a time; never holds the whole model in RAM.
+    """
+    os.makedirs(dst, exist_ok=True)
+    keep = dmi = new_ne = None
+    if keep_path is not None:
+        import numpy as np
+        kd = np.load(keep_path)
+        keep = kd["keep"]; dmi = int(kd["dense_mlp_idx"]); new_ne = int(kd["K"])
+    index = json.load(open(os.path.join(src, "model.safetensors.index.json")))
+    weight_map = index["weight_map"]
+
+    shard_to_names: dict[str, list[str]] = {}
+    for n, s in weight_map.items():
+        shard_to_names.setdefault(s, []).append(n)
+
+    out_index: dict[str, str] = {}
+    buffer: dict[str, mx.array] = {}
+    buffer_bytes = 0
+    out_shard_id = 0
+    total_out_shards_placeholder = "{:05d}"
+
+    def flush(final=False):
+        nonlocal buffer, buffer_bytes, out_shard_id
+        if not buffer:
+            return
+        out_shard_id += 1
+        fname = f"model-{total_out_shards_placeholder.format(out_shard_id)}.safetensors"
+        mx.save_safetensors(os.path.join(dst, fname), buffer, metadata={"format": "mlx"})
+        for k in buffer:
+            out_index[k] = fname
+        buffer = {}
+        buffer_bytes = 0
+
+    for shard in sorted(shard_to_names):
+        path = os.path.join(src, shard)
+        tensors = mx.load(path)  # mmap
+        for name in shard_to_names[shard]:
+            w = tensors[name]
+            for out_name, arr in _process_tensor(name, w, bits, group_size, out_dtype, recipe, keep, dmi):
+                mx.eval(arr)
+                buffer[out_name] = arr
+                buffer_bytes += arr.nbytes
+                if buffer_bytes >= _SHARD_CAP_BYTES:
+                    flush()
+        del tensors
+    flush(final=True)
+
+    # rename shards with correct total, build index.json
+    _finalize_index(dst, out_index, out_shard_id)
+    _write_config(src, dst, bits, group_size, recipe, new_ne)
+    _copy_aux(src, dst)
+    return dst
+
+
+def _finalize_index(dst, out_index, n_shards):
+    # rewrite shard filenames to model-XXXXX-of-YYYYY.safetensors
+    remap = {}
+    for i in range(1, n_shards + 1):
+        old = f"model-{i:05d}.safetensors"
+        new = f"model-{i:05d}-of-{n_shards:05d}.safetensors"
+        if old != new and os.path.exists(os.path.join(dst, old)):
+            os.rename(os.path.join(dst, old), os.path.join(dst, new))
+        remap[old] = new
+    weight_map = {k: remap[v] for k, v in out_index.items()}
+    total = sum(os.path.getsize(os.path.join(dst, f)) for f in set(weight_map.values()))
+    with open(os.path.join(dst, "model.safetensors.index.json"), "w") as f:
+        json.dump({"metadata": {"total_size": total}, "weight_map": weight_map}, f, indent=2)
+
+
+def _write_config(src, dst, bits, group_size, recipe="uniform", new_ne=None):
+    cfg = json.load(open(os.path.join(src, "config.json")))
+    if bits is not None:
+        cfg["quantization"] = {"group_size": group_size, "bits": bits, "recipe": recipe}
+    if new_ne is not None:                       # REAP: fewer routed experts
+        cfg["text_config"]["n_routed_experts"] = new_ne
+        cfg.setdefault("reap", {})["kept_experts"] = new_ne
+    with open(os.path.join(dst, "config.json"), "w") as f:
+        json.dump(cfg, f, indent=2)
+
+
+def _copy_aux(src, dst):
+    for pat in ("tokenizer*", "special_tokens_map.json", "*.tiktoken", "tiktoken",
+                "chat_template.jinja", "processor_config.json"):
+        for p in glob.glob(os.path.join(src, pat)):
+            base = os.path.basename(p)
+            target = os.path.join(dst, base)
+            if os.path.isdir(p):
+                shutil.copytree(p, target, dirs_exist_ok=True)
+            else:
+                shutil.copy2(p, target)
diff --git a/inkling_mlx/convert_cli.py b/inkling_mlx/convert_cli.py
new file mode 100644
index 0000000000000000000000000000000000000000..3f020c3257c029204af9d0a1195279872895bed3
--- /dev/null
+++ b/inkling_mlx/convert_cli.py
@@ -0,0 +1,47 @@
+"""CLI: convert/quantize an Inkling checkpoint to MLX.
+
+Examples:
+  python -m inkling_mlx.convert_cli --src /path/Inkling-src --dst out-bf16
+  python -m inkling_mlx.convert_cli --src /path/Inkling-src --dst out-4bit --bits 4
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+
+import mlx.core as mx
+
+from .convert import convert_model
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--src", required=True, help="Inkling bf16 source dir (HF layout)")
+    ap.add_argument("--dst", required=True, help="output dir")
+    ap.add_argument("--bits", type=int, default=None, choices=[2, 3, 4, 5, 6, 8],
+                    help="quantization bits; omit for bf16 passthrough")
+    ap.add_argument("--group-size", type=int, default=64)
+    ap.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16"])
+    ap.add_argument("--device", default="gpu", choices=["gpu", "cpu"],
+                    help="cpu avoids the Metal GPU-timeout watchdog on huge tensors (slower, robust)")
+    ap.add_argument("--recipe", default="uniform", choices=["uniform", "experts_only"],
+                    help="experts_only keeps attention + embed/unembed at bf16 (coherent 4-bit-sized build)")
+    ap.add_argument("--prune", default=None,
+                    help="REAP keep-indices npz (from prune_experts.py) -> prune experts during convert")
+    args = ap.parse_args()
+
+    if args.device == "cpu":
+        mx.set_default_device(mx.cpu)
+        print("[convert] using CPU device (avoids Metal command-buffer timeout)")
+
+    dtype = {"bfloat16": mx.bfloat16, "float16": mx.float16}[args.dtype]
+    t0 = time.time()
+    print(f"[convert] {args.src} -> {args.dst}  bits={args.bits} group_size={args.group_size} dtype={args.dtype} recipe={args.recipe} prune={args.prune}")
+    convert_model(args.src, args.dst, bits=args.bits, group_size=args.group_size, out_dtype=dtype,
+                  recipe=args.recipe, keep_path=args.prune)
+    print(f"[convert] done in {time.time()-t0:.0f}s -> {args.dst}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/inkling_mlx/generate.py b/inkling_mlx/generate.py
new file mode 100644
index 0000000000000000000000000000000000000000..b422b521aaea87becac366e13dbc62dc79f8923d
--- /dev/null
+++ b/inkling_mlx/generate.py
@@ -0,0 +1,90 @@
+"""Greedy generation for an Inkling MLX model, using an incremental KV + conv-state
+cache: the prompt is prefilled once, then each new token is a single-position step.
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+
+import mlx.core as mx
+
+from .cache import make_cache
+from .load import load
+
+
+def load_tokenizer(path: str):
+    try:
+        from transformers import AutoTokenizer
+        return AutoTokenizer.from_pretrained(path, trust_remote_code=True)
+    except Exception:
+        from transformers import PreTrainedTokenizerFast
+        import os
+        return PreTrainedTokenizerFast(tokenizer_file=os.path.join(path, "tokenizer.json"))
+
+
+def greedy_generate(model, config, input_ids, max_new_tokens=32, eos_id=None,
+                    pixel_values=None, audio_input_ids=None):
+    """Greedy decode. For multimodal, pass ``pixel_values`` / ``audio_input_ids``
+    (from ``InklingProcessor``); they are consumed only by the prompt prefill."""
+    eos_id = eos_id if eos_id is not None else config.eos_token_id
+    caches = make_cache(model)
+    prompt = list(input_ids)
+
+    # prefill the whole prompt (with any media) in one pass
+    logits = model(mx.array([prompt]), caches=caches, start_pos=0, last_logit_only=True,
+                   pixel_values=pixel_values, audio_input_ids=audio_input_ids)
+    next_id = int(mx.argmax(logits[0, -1]).item())
+    out = [next_id]
+    pos = len(prompt)
+
+    for _ in range(max_new_tokens - 1):
+        if next_id == eos_id:
+            break
+        logits = model(mx.array([[next_id]]), caches=caches, start_pos=pos, last_logit_only=True)
+        next_id = int(mx.argmax(logits[0, -1]).item())
+        out.append(next_id)
+        pos += 1
+
+    return prompt + out
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model", required=True, help="converted MLX model dir")
+    ap.add_argument("--prompt", default="The capital of France is")
+    ap.add_argument("--max-new-tokens", type=int, default=32)
+    ap.add_argument("--wired-limit-gb", type=float, default=500.0,
+                    help="mx wired-memory limit; needs `sudo sysctl iogpu.wired_limit_mb` set too")
+    ap.add_argument("--lazy", action="store_true",
+                    help="mmap weights instead of eager-loading (lower peak RAM, but forwards "
+                         "re-read from disk and thrash near the memory ceiling)")
+    args = ap.parse_args()
+
+    # eager load pins the weights wired-resident so prefill/decode don't re-read the
+    # mmap (the big win for near-capacity models); pass --lazy to opt out.
+    try:
+        mx.set_wired_limit(int(args.wired_limit_gb * 1e9))
+    except Exception as e:
+        print(f"[warn] set_wired_limit: {e}")
+
+    print(f"[load] {args.model} ({'lazy mmap' if args.lazy else 'eager, wired-resident'})")
+    t0 = time.time()
+    model, config = load(args.model, lazy=args.lazy)
+    print(f"[load] ready in {time.time()-t0:.0f}s")
+
+    tok = load_tokenizer(args.model)
+    input_ids = tok(args.prompt)["input_ids"]
+    print(f"[prompt] {args.prompt!r} -> {len(input_ids)} tokens")
+
+    t0 = time.time()
+    out_ids = greedy_generate(model, config, input_ids, args.max_new_tokens)
+    dt = time.time() - t0
+    text = tok.decode(out_ids)
+    n_new = len(out_ids) - len(input_ids)
+    print(f"\n{text}\n")
+    print(f"[gen] {n_new} tokens in {dt:.1f}s ({n_new/dt:.2f} tok/s)")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/inkling_mlx/layers.py b/inkling_mlx/layers.py
new file mode 100644
index 0000000000000000000000000000000000000000..89050e1e393902244b18de35718b18b991d499c4
--- /dev/null
+++ b/inkling_mlx/layers.py
@@ -0,0 +1,47 @@
+"""Inkling decoder layer: attention + MLP, each wrapped by a pre-norm and a
+trailing short-convolution, with residual adds. Mirrors ``InklingDecoderLayer``.
+"""
+
+from __future__ import annotations
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from .attention import Attention
+from .common import RMSNorm, ShortConvolution
+from .config import TextConfig
+from .moe import DenseMLP, MoE
+
+
+class DecoderLayer(nn.Module):
+    def __init__(self, config: TextConfig, layer_idx: int):
+        super().__init__()
+        self.attn = Attention(config, layer_idx)
+        self.attn_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.mlp_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        if config.mlp_layer_types[layer_idx] == "sparse":
+            self.mlp = MoE(config)
+        else:
+            self.mlp = DenseMLP(config)
+        self.attn_sconv = ShortConvolution(config.hidden_size, config.sconv_kernel_size)
+        self.mlp_sconv = ShortConvolution(config.hidden_size, config.sconv_kernel_size)
+
+    def __call__(self, x, start_pos=0, cache=None, conv_mask=None):
+        kv = cache.kv if cache is not None else None
+        residual = x
+        h = self.attn_norm(x)
+        h = self.attn(
+            h, start_pos=start_pos, kv_cache=kv,
+            k_conv=cache.k_conv if cache is not None else None,
+            v_conv=cache.v_conv if cache is not None else None,
+            conv_mask=conv_mask,
+        )
+        h = self.attn_sconv(h, mask=conv_mask, cache=cache.attn_conv if cache is not None else None)
+        x = residual + h
+
+        residual = x
+        h = self.mlp_norm(x)
+        h = self.mlp(h)
+        h = self.mlp_sconv(h, mask=conv_mask, cache=cache.mlp_conv if cache is not None else None)
+        x = residual + h
+        return x
diff --git a/inkling_mlx/load.py b/inkling_mlx/load.py
new file mode 100644
index 0000000000000000000000000000000000000000..f6a9ee2a589a25a78e250a27e6738f8422da889c
--- /dev/null
+++ b/inkling_mlx/load.py
@@ -0,0 +1,69 @@
+"""Load a (possibly quantized) Inkling MLX model produced by ``convert_model``."""
+
+from __future__ import annotations
+
+import glob
+import json
+import os
+
+import mlx.core as mx
+import mlx.nn as nn
+from mlx.utils import tree_flatten
+
+from .config import InklingConfig
+from .model import InklingForConditionalGeneration
+
+
+def quant_predicate(group_size: int, recipe: str = "uniform"):
+    """Quantize exactly the modules the converter did, by delegating to
+    ``convert.is_quant_target`` with the same ``recipe``. Guarantees the loaded
+    module set matches the checkpoint (e.g. under ``experts_only``, attention and
+    embed/unembed stay bf16 and must NOT be re-quantized here)."""
+    from .convert import is_quant_target
+
+    def pred(path, module):
+        if not hasattr(module, "to_quantized"):
+            return False
+        w = getattr(module, "weight", None)
+        if w is None:
+            return False
+        return is_quant_target(path + ".weight", w.shape[-1], group_size, recipe)
+
+    return pred
+
+
+def load(path: str, lazy: bool = False):
+    cfg_dict = json.load(open(os.path.join(path, "config.json")))
+    config = InklingConfig.from_dict(cfg_dict)
+    model = InklingForConditionalGeneration(config)
+
+    q = cfg_dict.get("quantization")
+    if q:
+        nn.quantize(model, group_size=q["group_size"], bits=q["bits"],
+                    class_predicate=quant_predicate(q["group_size"], q.get("recipe", "uniform")))
+
+    # Stream shards: assign each, then release its handle. We do NOT eagerly
+    # mx.eval() the whole parameter tree — for a ~500 GB model that builds one
+    # enormous eval graph and trips a Metal resource limit. Weights stay lazy
+    # (mmap-backed) and materialize on demand during the forward pass, exactly
+    # like mlx-lm loads large models.
+    loaded = set()
+    shards = sorted(glob.glob(os.path.join(path, "*.safetensors")))
+    for shard in shards:
+        w = mx.load(shard)
+        model.load_weights(list(w.items()), strict=False)
+        if not lazy:
+            # materialize THIS shard's tensors now (bounded graph) and keep them
+            # resident. Avoids one enormous eval over all ~500 GB of params, which
+            # trips a Metal resource limit; also prevents per-token disk paging.
+            mx.eval(list(w.values()))
+        loaded.update(w.keys())
+        del w
+
+    expected = {k for k, _ in tree_flatten(model.parameters())}
+    missing = expected - loaded
+    if missing:
+        raise ValueError(f"{len(missing)} params not found in checkpoint, e.g. {sorted(missing)[:3]}")
+
+    model.eval()
+    return model, config
diff --git a/inkling_mlx/model.py b/inkling_mlx/model.py
new file mode 100644
index 0000000000000000000000000000000000000000..de0f6ec6ed82ba2e7d130a854f314eac0fc8d07b
--- /dev/null
+++ b/inkling_mlx/model.py
@@ -0,0 +1,79 @@
+"""Top-level Inkling multimodal model.
+
+Checkpoint layout: ``model.llm.*`` (text backbone + untied unembed), ``model.visual.*``
+(HMLP vision tower), ``model.audio.*`` (dMel audio tower). Image/audio features are
+scattered into the token-embedding stream at their placeholder-token positions, then
+the text backbone runs and the untied unembed head produces (muP-scaled) logits.
+The MTP head (``model.mtp.*``) is intentionally not loaded (inference-irrelevant).
+"""
+
+from __future__ import annotations
+
+import mlx.core as mx
+import mlx.nn as nn
+import numpy as np
+
+from .audio import AudioModel
+from .config import InklingConfig
+from .text import TextModel
+from .vision import VisionModel
+
+
+def _scatter_features(embeds, input_ids, token_id, features):
+    """Replace ``embeds`` rows where ``input_ids == token_id`` with ``features``
+    (in sequence order). ``input_ids`` is host-known so we resolve positions on CPU."""
+    B, L, H = embeds.shape
+    ids = np.array(input_ids).reshape(-1)
+    pos = np.nonzero(ids == token_id)[0]
+    if pos.size == 0:
+        return embeds
+    flat = embeds.reshape(B * L, H)
+    flat[mx.array(pos)] = features.astype(flat.dtype)
+    return flat.reshape(B, L, H)
+
+
+class InnerModel(nn.Module):
+    """The ``model.`` level holding the three towers."""
+
+    def __init__(self, config: InklingConfig):
+        super().__init__()
+        self.llm = TextModel(config.text)
+        self.visual = VisionModel(config.vision)
+        self.audio = AudioModel(config.audio)
+
+
+class InklingForConditionalGeneration(nn.Module):
+    def __init__(self, config: InklingConfig):
+        super().__init__()
+        self.config = config
+        self.model = InnerModel(config)
+
+    # --- convenience accessors ---
+    @property
+    def llm(self) -> TextModel:
+        return self.model.llm
+
+    def __call__(
+        self,
+        input_ids: mx.array,
+        pixel_values: mx.array | None = None,
+        audio_input_ids: mx.array | None = None,
+        conv_mask=None,
+        caches=None,
+        start_pos: int = 0,
+        last_logit_only: bool = False,
+    ) -> mx.array:
+        embeds = self.model.llm.embed_tokens(input_ids)
+
+        if pixel_values is not None:
+            img = self.model.visual(pixel_values)
+            embeds = _scatter_features(embeds, input_ids, self.config.image_token_id, img)
+
+        if audio_input_ids is not None:
+            aud = self.model.audio(audio_input_ids)
+            embeds = _scatter_features(embeds, input_ids, self.config.audio_token_id, aud)
+
+        hidden = self.model.llm.backbone(embeds, conv_mask=conv_mask, caches=caches, start_pos=start_pos)
+        if last_logit_only:
+            hidden = hidden[:, -1:, :]
+        return self.model.llm.logits(hidden)
diff --git a/inkling_mlx/moe.py b/inkling_mlx/moe.py
new file mode 100644
index 0000000000000000000000000000000000000000..ef18601ff58e3f152faef023b57f9ed6b1ca4e5b
--- /dev/null
+++ b/inkling_mlx/moe.py
@@ -0,0 +1,117 @@
+"""Inkling MLP variants: dense SwiGLU (with a learned output scale) and the
+sparse MoE (sigmoid router with correction bias, softmax-over-selected weights,
+route/global scaling, and 2 always-on shared experts forming a routing "sink").
+
+Mirrors ``InklingMLP`` / ``InklingTopkRouter`` / ``InklingExperts`` /
+``InklingSharedExperts`` / ``InklingMoE`` from transformers PR #47347.
+"""
+
+from __future__ import annotations
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from mlx_lm.models.switch_layers import SwitchGLU
+
+from .config import TextConfig
+
+
+class DenseMLP(nn.Module):
+    """SwiGLU MLP with a learned scalar output gain (``global_scale``).
+
+    The checkpoint fuses gate+up into ``w13_dn``; the converter splits it into
+    ``gate_proj``/``up_proj`` so the standard MLX quantizer sees plain ``nn.Linear``s.
+    """
+
+    def __init__(self, config: TextConfig):
+        super().__init__()
+        h = config.hidden_size
+        inter = config.dense_intermediate_size
+        self.gate_proj = nn.Linear(h, inter, bias=False)
+        self.up_proj = nn.Linear(h, inter, bias=False)
+        self.down_proj = nn.Linear(inter, h, bias=False)
+        self.global_scale = mx.ones((1,))
+
+    def __call__(self, x: mx.array) -> mx.array:
+        y = self.down_proj(nn.silu(self.gate_proj(x)) * self.up_proj(x))
+        return y * self.global_scale
+
+
+class Router(nn.Module):
+    """Sigmoid top-k router with a correction bias and a shared-expert sink.
+
+    Kept in full precision (tiny). Returns per-token routed weights/indices plus
+    the two shared-expert gammas produced by the same softmax (the "sink").
+    """
+
+    def __init__(self, config: TextConfig):
+        super().__init__()
+        self.num_experts = config.n_routed_experts
+        self.n_shared = config.n_shared_experts
+        self.n_total = self.num_experts + self.n_shared
+        self.top_k = config.num_experts_per_tok
+        self.route_scale = config.route_scale
+        self.hidden = config.hidden_size
+        self.weight = mx.zeros((self.n_total, config.hidden_size))
+        self.bias = mx.zeros((self.num_experts,))              # e_score_correction_bias
+        self.global_scale = mx.ones((1,))
+
+    def __call__(self, x: mx.array):
+        # Routing (esp. the top-k selection) is precision-sensitive: in bf16 the
+        # rounding of near-tied expert scores flips which experts fire, and a wrong
+        # choice compounds over 64 MoE layers into incoherent output. Compute the
+        # whole router in fp32.
+        flat = x.reshape(-1, self.hidden).astype(mx.float32)
+        router_logits = flat @ self.weight.T.astype(mx.float32)  # [T, n_total]
+        scores = mx.sigmoid(router_logits)
+        routed_scores = scores[:, : self.num_experts]
+        scores_for_choice = routed_scores + self.bias
+
+        # top-k experts (order within the top-k is irrelevant downstream)
+        topk_idx = mx.argpartition(-scores_for_choice, kth=self.top_k - 1, axis=-1)[:, : self.top_k]
+
+        routed_logits = router_logits[:, : self.num_experts]
+        shared_logits = router_logits[:, self.num_experts :]  # [T, n_shared]
+        gathered = mx.take_along_axis(routed_logits, topk_idx, axis=-1)  # [T, top_k]
+        topk_logits = mx.concatenate([gathered, shared_logits], axis=-1)  # [T, top_k+n_shared]
+
+        # softmax over the selected (+shared) logits, computed in the log domain
+        log_probs = -mx.logaddexp(mx.zeros_like(topk_logits), -topk_logits)  # logsigmoid
+        weights = mx.softmax(log_probs, axis=-1)
+        weights = weights * self.route_scale * self.global_scale
+
+        shared_gammas = weights[:, self.top_k :].astype(x.dtype)   # [T, n_shared]
+        topk_weights = weights[:, : self.top_k].astype(x.dtype)    # [T, top_k]
+        return topk_weights, topk_idx, shared_gammas
+
+
+class MoE(nn.Module):
+    def __init__(self, config: TextConfig):
+        super().__init__()
+        self.config = config
+        self.n_shared = config.n_shared_experts
+        self.gate = Router(config)
+        self.experts = SwitchGLU(
+            config.hidden_size, config.moe_intermediate_size, config.n_routed_experts, bias=False
+        )
+        self.shared_experts = SwitchGLU(
+            config.hidden_size, config.moe_intermediate_size, config.n_shared_experts, bias=False
+        )
+
+    def __call__(self, x: mx.array) -> mx.array:
+        B, L, H = x.shape
+        topk_weights, topk_idx, shared_gammas = self.gate(x)
+        xf = x.reshape(-1, H)                                  # [T, H]
+        T = xf.shape[0]
+
+        routed = self.experts(xf, topk_idx)                   # [T, top_k, H]
+        routed = (routed * topk_weights[..., None]).sum(axis=1)
+
+        shared_idx = mx.broadcast_to(
+            mx.arange(self.n_shared)[None], (T, self.n_shared)
+        )
+        shared = self.shared_experts(xf, shared_idx)          # [T, n_shared, H]
+        shared = (shared.astype(mx.float32) * shared_gammas[..., None].astype(mx.float32)).sum(axis=1)
+        shared = shared.astype(routed.dtype)
+
+        return (routed + shared).reshape(B, L, H)
diff --git a/inkling_mlx/processing.py b/inkling_mlx/processing.py
new file mode 100644
index 0000000000000000000000000000000000000000..bf982ab03ecf946523b88ad38cb62994bd4d4d58
--- /dev/null
+++ b/inkling_mlx/processing.py
@@ -0,0 +1,188 @@
+"""Image + audio preprocessing for Inkling (MLX), ported from the reference
+`InklingImageProcessor` / `InklingFeatureExtractor` / `InklingProcessor`.
+
+  * image  -> pixel_values [num_patches, T=2, 40, 40, 3]  (feeds `VisionModel`)
+  * audio  -> audio_input_ids [num_frames, 80] dMel bins  (feeds `AudioModel`)
+
+`InklingProcessor.apply` builds the full multimodal input (input_ids + features)
+from a chat message list, inserting the right number of placeholder soft-tokens.
+Uses numpy/PIL + transformers' mel filterbank; no torch needed at inference.
+"""
+
+from __future__ import annotations
+
+import math
+
+import numpy as np
+
+# CLIP normalization (OPENAI_CLIP_MEAN / STD), per processor_config.json
+CLIP_MEAN = np.array([0.48145466, 0.4578275, 0.40821073], dtype=np.float32)
+CLIP_STD = np.array([0.26862954, 0.26130258, 0.27577711], dtype=np.float32)
+
+PATCH = 40           # image patch size (== vision patch_size)
+TEMPORAL = 2         # temporal_patch_size (images duplicated across 2 frames)
+
+# audio (processor_config.json / feature_extraction_inkling.py)
+SR = 16000
+HOP = 800            # audio_token_duration_s (0.05) * SR
+WIN = 1600           # * window_size_multiplier (2.0)
+N_FFT = 1600
+N_MEL = 80
+DMEL_BINS = 16
+DMEL_MIN, DMEL_MAX = -7.0, 2.0
+
+# special tokens
+IMAGE_TOKEN_ID = 200054      # <|unused_200054|>  (soft-token slot)
+AUDIO_TOKEN_ID = 200053      # <|unused_200053|>
+IMAGE_BOS = "<|content_image|>"
+AUDIO_BOS = "<|content_audio_input|>"
+
+
+# ------------------------------- image -------------------------------
+
+def preprocess_image(image, max_long_edge: int | None = None) -> tuple[np.ndarray, int]:
+    """PIL.Image or HxWx3 uint8 array -> (pixel_values [N,2,40,40,3] float32, N).
+
+    ``max_long_edge`` (optional) downscales the image (LANCZOS, aspect preserved) so its
+    long edge is at most this many pixels *before* patchify. Each 40px patch is one
+    vision soft-token, so this directly cuts the prompt length / prefill cost for big
+    images (e.g. cap 512 -> ~130 patches vs ~450 at full 960px). ``None`` keeps native
+    resolution (the reference default)."""
+    from PIL import Image
+    if not hasattr(image, "convert"):
+        image = Image.fromarray(np.asarray(image).astype(np.uint8))
+    image = image.convert("RGB")
+    if max_long_edge and max(image.size) > max_long_edge:
+        r = max_long_edge / max(image.size)
+        image = image.resize((max(1, round(image.width * r)), max(1, round(image.height * r))), Image.LANCZOS)
+    image = np.asarray(image)
+    if image.ndim == 2:
+        image = np.stack([image] * 3, axis=-1)
+    img = image[..., :3].astype(np.float32).transpose(2, 0, 1)  # -> [C, H, W]
+    C, H, W = img.shape
+
+    num_rows = (H + PATCH - 1) // PATCH
+    num_cols = W // PATCH + 1                       # reference: W//P + 1
+    patches = []
+    for i in range(num_rows):
+        for j in range(num_cols):
+            p = img[:, i * PATCH:(i + 1) * PATCH, j * PATCH:(j + 1) * PATCH]  # may be < 40
+            padded = np.full((C, PATCH, PATCH), -1.0, dtype=np.float32)       # pad value -1.0
+            padded[:, : p.shape[1], : p.shape[2]] = p
+            patches.append(padded)
+    patches = np.stack(patches, axis=0)             # [N, C, 40, 40]
+
+    # rescale (1/255) + CLIP normalize per channel
+    patches = patches / 255.0
+    patches = (patches - CLIP_MEAN[None, :, None, None]) / CLIP_STD[None, :, None, None]
+
+    # add temporal dim, duplicate x2, then -> [N, T, H, W, C]
+    patches = np.repeat(patches[..., None], TEMPORAL, axis=-1)   # [N, C, 40, 40, 2]
+    pixel_values = patches.transpose(0, 4, 2, 3, 1)              # [N, 2, 40, 40, C]
+    return pixel_values.astype(np.float32), pixel_values.shape[0]
+
+
+# ------------------------------- audio -------------------------------
+
+_mel_fb = None
+def _mel_filters() -> np.ndarray:
+    global _mel_fb
+    if _mel_fb is None:
+        from transformers.audio_utils import mel_filter_bank
+        fb = mel_filter_bank(num_frequency_bins=N_FFT // 2 + 1, num_mel_filters=N_MEL,
+                             min_frequency=0.0, max_frequency=SR / 2.0, sampling_rate=SR,
+                             norm="slaney", mel_scale="slaney")          # [801, 80]
+        _mel_fb = np.ascontiguousarray(fb.T, dtype=np.float32)           # [80, 801]
+    return _mel_fb
+
+
+def _log_mel(waveform: np.ndarray) -> np.ndarray:
+    """raw mono waveform -> log10-mel spectrogram [num_frames, 80]."""
+    wav = np.asarray(waveform, dtype=np.float32).reshape(-1)
+    right = math.ceil(wav.shape[0] / HOP) * HOP - wav.shape[0]
+    left = max(N_FFT - HOP, 0)
+    wav = np.pad(wav, (left, right))
+    window = np.hanning(WIN + 1)[:-1].astype(np.float32)                 # periodic Hann
+    n_frames = 1 + (wav.shape[0] - N_FFT) // HOP                          # center=False
+    frames = np.stack([wav[i * HOP: i * HOP + N_FFT] * window for i in range(n_frames)])  # [T, N_FFT]
+    mag = np.abs(np.fft.rfft(frames, n=N_FFT, axis=-1))                   # [T, 801]
+    mag = np.maximum(mag, 1e-10)
+    mel = _mel_filters() @ mag.T                                         # [80, T]
+    mel = np.log10(np.maximum(mel, 1e-10))
+    return mel.T                                                         # [T, 80]
+
+
+def preprocess_audio(waveform: np.ndarray, sampling_rate: int = SR) -> np.ndarray:
+    """raw 16 kHz mono waveform -> dMel bin ids [num_frames, 80] (int32, 0..15)."""
+    if sampling_rate != SR:
+        raise ValueError(f"Inkling audio expects {SR} Hz, got {sampling_rate}")
+    mel = _log_mel(waveform)                                             # [T, 80] log10
+    n_valid = math.ceil(len(np.asarray(waveform).reshape(-1)) / HOP)
+    mel = mel[:n_valid]                                                  # drop trailing pad frames
+    centers = np.linspace(DMEL_MIN, DMEL_MAX, DMEL_BINS)                 # 16 bin centers
+    clamped = np.clip(mel.astype(np.float64), DMEL_MIN, DMEL_MAX)
+    bins = np.abs(clamped[..., None] - centers).argmin(-1)              # nearest center
+    return bins.astype(np.int32)                                        # [T, 80]
+
+
+# --------------------------- prompt assembly ---------------------------
+
+class InklingProcessor:
+    """Assembles multimodal model inputs from chat messages with image/audio parts.
+
+    Content parts: {"type":"text","text":...}, {"type":"image","image":PIL/array},
+    {"type":"audio","audio":waveform, "sampling_rate":16000}.
+    """
+
+    def __init__(self, tokenizer, chat_template: str):
+        self.tok = tokenizer
+        self.chat_template = chat_template
+        self.image_bos_id = tokenizer.encode(IMAGE_BOS, add_special_tokens=False)[0]
+        self.audio_bos_id = tokenizer.encode(AUDIO_BOS, add_special_tokens=False)[0]
+
+    def apply(self, messages, reasoning_effort: str = "none", max_long_edge: int | None = None):
+        import mlx.core as mx
+        pixel_values, audio_ids = [], []
+        # Render text via the chat template with placeholders stripped to a sentinel,
+        # then splice media spans in. We build ids directly for robustness.
+        ids: list[int] = []
+
+        def emit_text(s):
+            ids.extend(self.tok.encode(s, add_special_tokens=False))
+
+        # header: thinking-effort system message (matches chat_template)
+        eff = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "max": 0.99}[reasoning_effort]
+        emit_text(f"<|message_system|><|content_text|>Thinking effort level: {0 if eff == 0 else eff}<|end_message|>")
+
+        for msg in messages:
+            role = {"user": "<|message_user|>", "assistant": "<|message_model|>",
+                    "system": "<|message_system|>"}[msg["role"]]
+            content = msg["content"]
+            if isinstance(content, str):
+                content = [{"type": "text", "text": content}]
+            for part in content:
+                t = part.get("type", "text")
+                if t == "text":
+                    emit_text(role + "<|content_text|>" + part["text"] + "<|end_message|>")
+                elif t == "image":
+                    pv, n = preprocess_image(part["image"], max_long_edge=max_long_edge)
+                    pixel_values.append(pv)
+                    ids.append(self.tok.encode(role, add_special_tokens=False)[0])
+                    ids.append(self.image_bos_id)
+                    ids.extend([IMAGE_TOKEN_ID] * n)
+                    ids.extend(self.tok.encode("<|end_message|>", add_special_tokens=False))
+                elif t == "audio":
+                    aid = preprocess_audio(part["audio"], part.get("sampling_rate", SR))
+                    audio_ids.append(aid)
+                    ids.append(self.tok.encode(role, add_special_tokens=False)[0])
+                    ids.append(self.audio_bos_id)
+                    ids.extend([AUDIO_TOKEN_ID] * aid.shape[0])
+                    ids.extend(self.tok.encode("<|end_message|>", add_special_tokens=False))
+        emit_text("<|message_model|>")   # generation prompt
+
+        out = {"input_ids": ids}
+        if pixel_values:
+            out["pixel_values"] = mx.array(np.concatenate(pixel_values, axis=0))
+        if audio_ids:
+            out["audio_input_ids"] = mx.array(np.concatenate(audio_ids, axis=0))
+        return out
diff --git a/inkling_mlx/text.py b/inkling_mlx/text.py
new file mode 100644
index 0000000000000000000000000000000000000000..b2ffeb8a9e22b0997c96aa1eea40c8b9ed880097
--- /dev/null
+++ b/inkling_mlx/text.py
@@ -0,0 +1,50 @@
+"""Inkling text backbone (``model.llm.*``): token embedding + embed-norm,
+66 decoder layers, final norm, and the (untied) unembed head.
+
+Mirrors ``InklingTextModel`` + the unembed / muP-logit scaling from
+``InklingForConditionalGeneration``.
+"""
+
+from __future__ import annotations
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from .common import RMSNorm
+from .config import TextConfig
+from .layers import DecoderLayer
+
+
+class TextModel(nn.Module):
+    def __init__(self, config: TextConfig):
+        super().__init__()
+        self.config = config
+        self.embed = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.embed_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.layers = [DecoderLayer(config, i) for i in range(config.num_hidden_layers)]
+        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.unembed = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+    def embed_tokens(self, input_ids: mx.array) -> mx.array:
+        return self.embed_norm(self.embed(input_ids))
+
+    def backbone(self, inputs_embeds: mx.array, conv_mask=None, caches=None, start_pos=0) -> mx.array:
+        h = inputs_embeds
+        for i, layer in enumerate(self.layers):
+            h = layer(h, start_pos=start_pos,
+                      cache=caches[i] if caches is not None else None,
+                      conv_mask=conv_mask)
+        return self.norm(h)
+
+    def logits(self, hidden: mx.array) -> mx.array:
+        hidden = hidden / self.config.logits_mup_width_multiplier
+        logits = self.unembed(hidden)
+        uv = self.config.unpadded_vocab_size
+        if uv is not None and uv < logits.shape[-1]:
+            logits = logits[..., :uv]
+        return logits
+
+    def __call__(self, input_ids: mx.array, conv_mask=None) -> mx.array:
+        h = self.embed_tokens(input_ids)
+        h = self.backbone(h, conv_mask=conv_mask)
+        return self.logits(h)
diff --git a/inkling_mlx/vision.py b/inkling_mlx/vision.py
new file mode 100644
index 0000000000000000000000000000000000000000..79d2e056e033203d80d5167c1707dbaf0bbabfb2
--- /dev/null
+++ b/inkling_mlx/vision.py
@@ -0,0 +1,126 @@
+"""Inkling vision tower: an HMLP (hierarchical MLP) patch encoder.
+
+No attention — each layer folds space/time into the channel dim then projects
+(Linear -> RMSNorm -> GELU), progressively growing channels up to the text hidden
+size. Mirrors ``InklingVisionModel`` / ``InklingVisionEncoderLayer`` /
+``plan_out_scales``. Checkpoint keys are flat: ``visual.layers.linear_{i}`` and
+``visual.layers.norm_{i}`` plus ``visual.final_norm``.
+"""
+
+from __future__ import annotations
+
+import math
+
+import mlx.core as mx
+import mlx.nn as nn
+
+from .common import RMSNorm
+from .config import VisionConfig
+
+
+def _prime_factors(n: int) -> list[int]:
+    factors = []
+    while n % 2 == 0:
+        factors.append(2)
+        n //= 2
+    p = 3
+    while p * p <= n:
+        while n % p == 0:
+            factors.append(p)
+            n //= p
+        p += 2
+    if n > 1:
+        factors.append(n)
+    return factors
+
+
+def plan_out_scales(temporal_patch_size: int, patch_size: int, n_layers: int, n_channels: int):
+    """Port of the reference ``plan_out_scales`` (returns an ``(n_layers+1, 4)``
+    array of (t, h, w, c) grid sizes). Uses numpy + scipy for the assignment."""
+    import numpy as np
+    from scipy.optimize import linear_sum_assignment
+
+    h = np.cumprod(np.array(_prime_factors(patch_size)[::-1]))
+    t = np.cumprod(np.array(_prime_factors(temporal_patch_size)[::-1]))
+
+    h_ch = np.ceil(h**2 * n_channels / 64).astype(np.int64) * 64
+    t_ch = (np.ceil(h[-1] ** 2 * n_channels * t)).astype(np.int64) * 64
+
+    base = np.array([[1, 1, 1, n_channels]], dtype=np.int64)
+    spatial = np.stack([np.ones_like(h), h, h, h_ch], axis=1)
+    temporal = np.stack([t, np.full_like(t, h[-1]), np.full_like(t, h[-1]), t_ch], axis=1)
+    scales = np.concatenate([base, spatial, temporal], axis=0).astype(np.int64)
+
+    size_reduction = np.prod(scales[:, :-1], axis=1).astype(np.float64)
+    total_elements = patch_size * patch_size * temporal_patch_size * n_channels
+    log_ideal = np.linspace(0.0, math.log(total_elements), n_layers + 1)
+    cost = np.abs(log_ideal[:, None] - np.log(size_reduction)[None, :])
+
+    if n_layers >= scales.shape[0]:
+        idxs = np.argmin(cost, axis=1)
+    else:
+        _, idxs = linear_sum_assignment(cost)
+    idxs = np.array(idxs)
+    idxs[0] = 0
+    idxs[-1] = scales.shape[0] - 1
+    return scales[idxs]
+
+
+def _fold_timespace_to_depth(x, t_fold, hw_fold):
+    # x: [B, T, H, W, C] -> [B, T//t, H//hw, W//hw, C*t*hw*hw]
+    B, T, H, W, C = x.shape
+    t_new, h_new, w_new = T // t_fold, H // hw_fold, W // hw_fold
+    x = x.reshape(B, t_new, t_fold, h_new, hw_fold, w_new, hw_fold, C)
+    x = x.transpose(0, 1, 3, 5, 2, 4, 6, 7)
+    x = x.reshape(B, t_new, h_new, w_new, t_fold * hw_fold * hw_fold * C)
+    return x
+
+
+class _VisionLayers(nn.Module):
+    """Holds ``linear_{i}`` / ``norm_{i}`` to match checkpoint keys."""
+
+    def __init__(self, config: VisionConfig):
+        super().__init__()
+        scales = plan_out_scales(
+            config.temporal_patch_size, config.patch_size, config.n_layers, config.num_channels
+        )
+        self.n_layers = config.n_layers
+        self.folds = []  # (t_fold, hw_fold, add_norm)
+        for i in range(config.n_layers):
+            start, end = scales[i], scales[i + 1]
+            shuffle = (
+                (end[0] // start[0]) * (end[1] // start[1]) * (end[2] // start[2])
+            )
+            hw_fold = int(end[1] // start[1])
+            t_fold = int(end[0] // start[0])
+            in_dim = int(start[3]) * int(shuffle)
+            add_norm = i != config.n_layers - 1
+            out_dim = config.text_hidden_size if i == config.n_layers - 1 else int(end[3])
+            setattr(self, f"linear_{i}", nn.Linear(in_dim, out_dim, bias=False))
+            if add_norm:
+                setattr(self, f"norm_{i}", RMSNorm(out_dim, eps=config.rms_norm_eps))
+            self.folds.append((t_fold, hw_fold, add_norm))
+
+    def __call__(self, x):
+        for i, (t_fold, hw_fold, add_norm) in enumerate(self.folds):
+            if hw_fold > 1 or t_fold > 1:
+                x = _fold_timespace_to_depth(x, t_fold, hw_fold)
+            x = getattr(self, f"linear_{i}")(x)
+            if add_norm:
+                x = getattr(self, f"norm_{i}")(x)
+                x = nn.gelu(x)
+        return x
+
+
+class VisionModel(nn.Module):
+    def __init__(self, config: VisionConfig):
+        super().__init__()
+        self.config = config
+        self.layers = _VisionLayers(config)
+        self.final_norm = RMSNorm(config.text_hidden_size, eps=config.rms_norm_eps)
+
+    def __call__(self, pixel_values: mx.array) -> mx.array:
+        num_patches = pixel_values.shape[0]
+        h = self.layers(pixel_values)
+        h = self.final_norm(h)
+        return h.reshape(num_patches, -1)
diff --git a/model-00001-of-00041.safetensors b/model-00001-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..4650f68b4d324cb84e4d869be6ea6fc1c77577ba
--- /dev/null
+++ b/model-00001-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5b4cd845159002bbfb01f464183da4a7fd8e30edd1cfb85d69b4df0fd2b61b0
+size 5154869256
diff --git a/model-00002-of-00041.safetensors b/model-00002-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..46a634f158bf7b7729c60890a12ba9a2d1f33391
--- /dev/null
+++ b/model-00002-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad388b047e62a23a56cae26a15c581025f5b3ad0ab803f79b78fcd8863e0bbe1
+size 5250330809
diff --git a/model-00003-of-00041.safetensors b/model-00003-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..8d0c89bb68d65c410f9a883e962e9d5b9e2b8baf
--- /dev/null
+++ b/model-00003-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ef52d86c28697040a85913baffbe717ceba59e37f6f7266af380519df6fc9e5
+size 5264010250
diff --git a/model-00004-of-00041.safetensors b/model-00004-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..80eeb70951ecd627655a4aa5712838da492c39a6
--- /dev/null
+++ b/model-00004-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95126cdbd77d8d3be412f6228126ea26bcdd6a20a1795cf4e6558f4ed8e09ccf
+size 5316394081
diff --git a/model-00005-of-00041.safetensors b/model-00005-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..4edd3956bfdfd9c15ed506531e6097a4285ab603
--- /dev/null
+++ b/model-00005-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:397cb07c6dafb8be8429119a9532f392fd1c190959ebd2835ef33e9e01295e4c
+size 5234492564
diff --git a/model-00009-of-00041.safetensors b/model-00009-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..e6b0988000639115d40300e34b271d8ca1dda33a
--- /dev/null
+++ b/model-00009-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d38b4b06b45d78b81e09d0ac72cd61774780db3c8f2921f41d76b07091f15584
+size 5378655813
diff --git a/model-00010-of-00041.safetensors b/model-00010-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..bbd84eb318e6d806fbc13788ab5c6d0346345ee2
--- /dev/null
+++ b/model-00010-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b7c9c07f81571bb80b87ec9664c9b39df7ba388243b2b17f68821257b863a09
+size 5503899023
diff --git a/model-00013-of-00041.safetensors b/model-00013-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..fc4149bdfba41016ba35231de831a673b08e31a3
--- /dev/null
+++ b/model-00013-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af18786e6fe0b99b0b229a1ad5963e463416f47b0d71eded42c243cc615df6d5
+size 5279257887
diff --git a/model-00015-of-00041.safetensors b/model-00015-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..ff60d4e1c36eadac89bdb7b2a20da0ee05664b83
--- /dev/null
+++ b/model-00015-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd99fbd550d2794a843f470c9f5fc62cac8d0e40e7d14d904529dfc452260554
+size 5343572973
diff --git a/model-00016-of-00041.safetensors b/model-00016-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..865bda238c18350b9bf4ce301143c4986d6a6f9f
--- /dev/null
+++ b/model-00016-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7af62e6d10629541953309e37cad6c721c1353eda477a65616f98cecc72a699
+size 5427988950
diff --git a/model-00019-of-00041.safetensors b/model-00019-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..f68f1167324fb83e9a1bd694efb3ccc6e06518e1
--- /dev/null
+++ b/model-00019-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8b04c6185967314e94ee48d1676f4557c2ca374bf94bc27864bcf2e2c64554f
+size 5299287758
diff --git a/model-00020-of-00041.safetensors b/model-00020-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..06b6c9b9e3a58dc0b30b82a4a80eb3a62db8a162
--- /dev/null
+++ b/model-00020-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:856674758e2ce23230afe7cc356487035d042f2fd49add4e8347e287087d72e2
+size 5267394964
diff --git a/model-00021-of-00041.safetensors b/model-00021-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..2fe460e76c4e7bbca2771a6daf9068bc40d76a1c
--- /dev/null
+++ b/model-00021-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:611411ef5ab0fc89ef4021e99022aedc4956b2f41aa342ad4d6384205cf2f0e3
+size 5381631974
diff --git a/model-00024-of-00041.safetensors b/model-00024-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..cc86ba0c91a57a17b64b9493c98c5c302a47fc73
--- /dev/null
+++ b/model-00024-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09a261f23a1619210e4419a3c45d9b97d756e78f338fe9a23a23dd66234d3c89
+size 5248124467
diff --git a/model-00027-of-00041.safetensors b/model-00027-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..d1f37479033cef2f5401e123a07d4c7cca71a666
--- /dev/null
+++ b/model-00027-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4757b3d3e59014673421800c0ced70035f0d27a8ca0ff792e66601ea8956d153
+size 5345427509
diff --git a/model-00028-of-00041.safetensors b/model-00028-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..4dabbdb2267789efc3897efd5622a0b54ad648c6
--- /dev/null
+++ b/model-00028-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e995fc54d9aeb5ff1616f98564f45dd520b1717d6dbd2b824392e1da273904f
+size 5315526266
diff --git a/model-00031-of-00041.safetensors b/model-00031-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..5546af445703d3fc80e6b35ea20a014e08e97eb7
--- /dev/null
+++ b/model-00031-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c33841c1f2afa9420a88003ba422dacf42d7a81cbbfea1a6b04d081d376a7ee7
+size 5307946509
diff --git a/model-00032-of-00041.safetensors b/model-00032-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..796c8e9002d9b85f52329c322bfcddcc7d81fa8f
--- /dev/null
+++ b/model-00032-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0267d920ff5657d0601dc63db0e1658cd88240794f1b5ed6f6db22ac1e3d405
+size 5352655999
diff --git a/model-00033-of-00041.safetensors b/model-00033-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..2d5b3d0d6ab35460e95eb81ed5df0fc94b6c8a51
--- /dev/null
+++ b/model-00033-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9754ff697f32c99eecb19943957cf5061b171f890baf96fc77be3f6b44c3f40
+size 5261816618
diff --git a/model-00034-of-00041.safetensors b/model-00034-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..624397418f3694fa13ab0f10633e938cd6795558
--- /dev/null
+++ b/model-00034-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f5ca9e6660bd3823db2b90e6ee056e0e9a3111e6c9446e98300acbd2456e705
+size 5324087881
diff --git a/model-00035-of-00041.safetensors b/model-00035-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..62a46848569143a2159e16d1a12cea0048cbb519
--- /dev/null
+++ b/model-00035-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7f7f69363707911dbdcc18b760e298bb2efeee0ebbd669d676586965383a039
+size 5403218715
diff --git a/model-00036-of-00041.safetensors b/model-00036-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..9a47d591c9fd8f2b2a211d1f7e31b4187104536c
--- /dev/null
+++ b/model-00036-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a06706f8c93cfdf35f835dec2f28f791a05f0c2baf4a49fd9f4da3560007d75d
+size 5241819789
diff --git a/model-00037-of-00041.safetensors b/model-00037-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..7b019abcb8dba018edc817f61b95099a813f0c0c
--- /dev/null
+++ b/model-00037-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fafd75bb26ad8409e8d8f4f55e3de7c93db53a4d3d7d3ebf03366e0cd89a627
+size 5921516804
diff --git a/model-00039-of-00041.safetensors b/model-00039-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..21b45f7c959ba2b0ce747baaa901ba463324ef6f
--- /dev/null
+++ b/model-00039-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b76671d80e3bec5a1ebe728a31ba3495ebedc9edede65f5ef3bf03952877edc0
+size 5236624700
diff --git a/model-00040-of-00041.safetensors b/model-00040-of-00041.safetensors
new file mode 100644
index 0000000000000000000000000000000000000000..baf991e340104c12dc36e275cdaf2a41b35e01e2
--- /dev/null
+++ b/model-00040-of-00041.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38e9ef34d8fd9cb3d51a91014b8dfd1acf54ae6a575ee5b60d8acb142293fcc4
+size 5265216153
diff --git a/model.safetensors.index.json b/model.safetensors.index.json
new file mode 100644
index 0000000000000000000000000000000000000000..ee28917d1d86da47123bf1cd50a041137b6e5081
--- /dev/null
+++ b/model.safetensors.index.json
@@ -0,0 +1,1901 @@
+{
+  "metadata": {
+    "total_size": 214329612243
+  },
+  "weight_map": {
+    "model.llm.layers.11.attn_sconv.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.13.attn.k_sconv.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.18.mlp_norm.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.attn.wq_du.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.attn.wq_du.scales": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.attn.wq_du.biases": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.gate_proj.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.gate_proj.scales": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.gate_proj.biases": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.up_proj.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.up_proj.scales": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.up_proj.biases": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.gate.bias": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.gate_proj.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.gate_proj.scales": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.gate_proj.biases": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.up_proj.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.up_proj.scales": "model-00001-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.up_proj.biases": "model-00001-of-00041.safetensors",
+    "model.llm.layers.21.attn_norm.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.down_proj.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.down_proj.scales": "model-00001-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.down_proj.biases": "model-00001-of-00041.safetensors",
+    "model.llm.layers.26.attn.v_sconv.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.gate_proj.weight": "model-00001-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.gate_proj.scales": "model-00002-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.gate_proj.biases": "model-00002-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.up_proj.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.up_proj.scales": "model-00002-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.up_proj.biases": "model-00002-of-00041.safetensors",
+    "model.llm.layers.32.attn.q_norm.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.34.mlp.gate.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.7.mlp.gate.global_scale": "model-00002-of-00041.safetensors",
+    "model.llm.layers.15.attn_norm.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.16.attn.v_sconv.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.16.attn_norm.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.17.attn_sconv.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.22.attn.q_norm.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.22.attn_sconv.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.down_proj.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.down_proj.scales": "model-00002-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.down_proj.biases": "model-00002-of-00041.safetensors",
+    "model.llm.layers.22.mlp.gate.bias": "model-00002-of-00041.safetensors",
+    "model.llm.layers.26.attn.wq_du.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.26.attn.wq_du.scales": "model-00002-of-00041.safetensors",
+    "model.llm.layers.26.attn.wq_du.biases": "model-00002-of-00041.safetensors",
+    "model.llm.layers.26.mlp.gate.global_scale": "model-00002-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.gate_proj.weight": "model-00002-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.gate_proj.scales": "model-00003-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.gate_proj.biases": "model-00003-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.up_proj.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.up_proj.scales": "model-00003-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.up_proj.biases": "model-00003-of-00041.safetensors",
+    "model.llm.layers.27.mlp_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.29.mlp.gate.global_scale": "model-00003-of-00041.safetensors",
+    "model.llm.layers.30.attn.v_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.down_proj.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.down_proj.scales": "model-00003-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.down_proj.biases": "model-00003-of-00041.safetensors",
+    "model.llm.layers.31.attn.k_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.32.mlp.gate.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.35.attn.k_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.37.attn.k_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.37.attn_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.5.attn.wo_ud.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.5.attn.wo_ud.scales": "model-00003-of-00041.safetensors",
+    "model.llm.layers.5.attn.wo_ud.biases": "model-00003-of-00041.safetensors",
+    "model.llm.layers.0.attn_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.15.attn.k_sconv.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.gate_proj.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.gate_proj.scales": "model-00003-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.gate_proj.biases": "model-00003-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.up_proj.weight": "model-00003-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.up_proj.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.up_proj.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.16.attn.wv_dv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.16.attn.wv_dv.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.16.attn.wv_dv.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.17.attn.wq_du.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.17.attn.wq_du.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.17.attn.wq_du.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.19.attn.wq_du.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.19.attn.wq_du.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.19.attn.wq_du.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.2.mlp.gate.global_scale": "model-00004-of-00041.safetensors",
+    "model.llm.layers.20.attn.k_norm.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.22.mlp_sconv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.gate_proj.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.gate_proj.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.gate_proj.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.up_proj.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.up_proj.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.up_proj.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.25.attn_norm.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.27.attn.k_norm.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.27.attn.wv_dv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.27.attn.wv_dv.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.27.attn.wv_dv.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.29.attn.wo_ud.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.29.attn.wo_ud.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.29.attn.wo_ud.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.33.attn.v_sconv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.33.mlp.gate.bias": "model-00004-of-00041.safetensors",
+    "model.llm.layers.33.mlp_norm.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.36.mlp.gate.global_scale": "model-00004-of-00041.safetensors",
+    "model.llm.layers.37.attn_norm.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.38.attn.wv_dv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.38.attn.wv_dv.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.38.attn.wv_dv.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.39.attn.wk_dv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.39.attn.wk_dv.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.39.attn.wk_dv.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.4.attn.wo_ud.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.4.attn.wo_ud.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.4.attn.wo_ud.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.4.attn_sconv.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.5.attn_norm.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.down_proj.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.down_proj.scales": "model-00004-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.down_proj.biases": "model-00004-of-00041.safetensors",
+    "model.llm.layers.13.mlp.gate.global_scale": "model-00004-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.gate_proj.weight": "model-00004-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.gate_proj.scales": "model-00005-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.gate_proj.biases": "model-00005-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.up_proj.weight": "model-00005-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.up_proj.scales": "model-00005-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.up_proj.biases": "model-00005-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.gate_proj.weight": "model-00005-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.gate_proj.scales": "model-00005-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.gate_proj.biases": "model-00005-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.up_proj.weight": "model-00005-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.up_proj.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.up_proj.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.10.attn.k_sconv.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.11.attn.rel_logits_proj.proj": "model-00006-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.down_proj.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.down_proj.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.down_proj.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.16.attn.q_norm.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.v_sconv.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.wk_dv.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.wk_dv.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.wk_dv.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.wv_dv.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.wv_dv.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.17.attn.wv_dv.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.22.attn.wr_du.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.22.attn.wr_du.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.22.attn.wr_du.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.3.attn.q_norm.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.33.attn_norm.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.36.attn.q_norm.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.gate_proj.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.gate_proj.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.gate_proj.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.up_proj.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.up_proj.scales": "model-00006-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.up_proj.biases": "model-00006-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.down_proj.weight": "model-00006-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.down_proj.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.down_proj.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.40.mlp.gate.global_scale": "model-00007-of-00041.safetensors",
+    "model.llm.layers.41.attn.k_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.1.attn.k_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.1.attn.wr_du.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.1.attn.wr_du.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.1.attn.wr_du.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.11.attn.wo_ud.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.11.attn.wo_ud.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.11.attn.wo_ud.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.13.attn.k_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.16.mlp.gate.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.16.mlp_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.18.attn.wr_du.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.18.attn.wr_du.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.18.attn.wr_du.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.18.mlp_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.19.attn_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.2.attn.rel_logits_proj.proj": "model-00007-of-00041.safetensors",
+    "model.llm.layers.20.attn.k_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.21.attn.q_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.down_proj.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.down_proj.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.down_proj.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.attn_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.down_proj.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.down_proj.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.down_proj.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.gate_proj.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.gate_proj.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.gate_proj.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.up_proj.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.up_proj.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.up_proj.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.24.attn.q_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.26.attn.wo_ud.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.26.attn.wo_ud.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.26.attn.wo_ud.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.26.attn.wv_dv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.26.attn.wv_dv.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.26.attn.wv_dv.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.27.attn.k_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.27.attn.v_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.27.mlp_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.28.mlp_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.29.attn_norm.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.3.attn.v_sconv.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.gate_proj.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.gate_proj.scales": "model-00007-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.gate_proj.biases": "model-00007-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.up_proj.weight": "model-00007-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.up_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.up_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.31.attn.wq_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.31.attn.wq_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.31.attn.wq_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.33.mlp_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.gate_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.gate_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.gate_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.up_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.up_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.up_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.35.attn.rel_logits_proj.proj": "model-00008-of-00041.safetensors",
+    "model.llm.layers.36.attn.wq_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.36.attn.wq_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.36.attn.wq_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.37.attn.wq_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.37.attn.wq_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.37.attn.wq_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.gate_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.gate_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.gate_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.up_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.up_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.up_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.39.attn.k_norm.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.39.attn.wq_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.39.attn.wq_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.39.attn.wq_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.4.attn.q_norm.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.down_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.down_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.down_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.40.attn.k_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.40.attn.wq_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.40.attn.wq_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.40.attn.wq_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.40.mlp.gate.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.5.attn.wr_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.5.attn.wr_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.5.attn.wr_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.6.attn.k_norm.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.6.mlp_norm.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.7.attn.k_norm.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.8.attn.wv_dv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.8.attn.wv_dv.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.8.attn.wv_dv.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.9.attn.v_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.9.mlp.gate.global_scale": "model-00008-of-00041.safetensors",
+    "model.llm.layers.9.mlp.gate.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.0.attn.v_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.0.mlp.global_scale": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.attn.wk_dv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.attn.wk_dv.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.attn.wk_dv.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.down_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.down_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.down_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.11.mlp_norm.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.12.attn.wq_du.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.12.attn.wq_du.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.12.attn.wq_du.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.13.attn.v_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.14.attn.wo_ud.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.14.attn.wo_ud.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.14.attn.wo_ud.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.15.attn.wk_dv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.15.attn.wk_dv.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.15.attn.wk_dv.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.15.mlp_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.attn.rel_logits_proj.proj": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.gate_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.gate_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.gate_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.up_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.up_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.up_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.19.mlp.gate.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.2.attn.wo_ud.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.2.attn.wo_ud.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.2.attn.wo_ud.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.2.mlp_sconv.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.21.mlp.gate.bias": "model-00008-of-00041.safetensors",
+    "model.llm.layers.24.mlp.gate.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.gate_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.gate_proj.scales": "model-00008-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.gate_proj.biases": "model-00008-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.up_proj.weight": "model-00008-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.up_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.up_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.28.attn.wo_ud.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.28.attn.wo_ud.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.28.attn.wo_ud.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.28.mlp.gate.global_scale": "model-00009-of-00041.safetensors",
+    "model.llm.layers.3.attn.wk_dv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.3.attn.wk_dv.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.3.attn.wk_dv.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.3.mlp_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.attn_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.gate_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.gate_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.gate_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.up_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.up_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.mlp.shared_experts.up_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.32.attn.rel_logits_proj.proj": "model-00009-of-00041.safetensors",
+    "model.llm.layers.32.mlp_sconv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.33.attn.wr_du.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.33.attn.wr_du.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.33.attn.wr_du.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.34.attn.wr_du.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.34.attn.wr_du.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.34.attn.wr_du.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.35.attn_sconv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.35.mlp.gate.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.gate_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.gate_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.gate_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.up_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.up_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.up_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.38.attn.k_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.4.mlp_sconv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.6.attn.wo_ud.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.6.attn.wo_ud.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.6.attn.wo_ud.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.down_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.down_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.8.mlp.experts.down_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.0.attn.q_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.12.mlp_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.13.mlp_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.gate_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.gate_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.gate_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.up_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.up_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.up_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.15.mlp.gate.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.down_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.down_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.down_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.19.attn.q_norm.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.22.mlp.gate.global_scale": "model-00009-of-00041.safetensors",
+    "model.llm.layers.23.attn.wk_dv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.23.attn.wk_dv.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.23.attn.wk_dv.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.25.attn.k_sconv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.26.attn.rel_logits_proj.proj": "model-00009-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.down_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.down_proj.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.27.mlp.experts.down_proj.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.attn.wr_du.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.attn.wr_du.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.30.attn.wr_du.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.34.attn.wv_dv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.34.attn.wv_dv.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.34.attn.wv_dv.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.attn.wr_du.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.attn.wr_du.scales": "model-00009-of-00041.safetensors",
+    "model.llm.layers.36.attn.wr_du.biases": "model-00009-of-00041.safetensors",
+    "model.llm.layers.37.mlp_sconv.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.gate_proj.weight": "model-00009-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.gate_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.gate_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.up_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.up_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.up_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.gate_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.gate_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.gate_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.up_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.up_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.up_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.4.mlp.gate.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.6.attn.wr_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.6.attn.wr_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.6.attn.wr_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.6.mlp_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.8.attn.wr_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.8.attn.wr_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.8.attn.wr_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.9.attn.wr_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.9.attn.wr_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.9.attn.wr_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.attn.k_norm.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.attn.wq_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.attn.wq_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.attn.wq_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.mlp.gate_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.mlp.gate_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.mlp.gate_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.mlp.up_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.mlp.up_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.0.mlp.up_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.1.attn_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.11.attn.k_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.13.attn.rel_logits_proj.proj": "model-00010-of-00041.safetensors",
+    "model.llm.layers.13.mlp.gate.bias": "model-00010-of-00041.safetensors",
+    "model.llm.layers.14.mlp.gate.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.15.attn.rel_logits_proj.proj": "model-00010-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.down_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.down_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.down_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.16.attn.rel_logits_proj.proj": "model-00010-of-00041.safetensors",
+    "model.llm.layers.16.mlp.gate.global_scale": "model-00010-of-00041.safetensors",
+    "model.llm.layers.17.attn.k_norm.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.18.attn_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.19.mlp.gate.bias": "model-00010-of-00041.safetensors",
+    "model.llm.layers.20.attn.wv_dv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.20.attn.wv_dv.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.20.attn.wv_dv.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.21.mlp.gate.global_scale": "model-00010-of-00041.safetensors",
+    "model.llm.layers.22.mlp_norm.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wo_ud.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wo_ud.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wo_ud.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wq_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wq_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wq_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wv_dv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wv_dv.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.23.attn.wv_dv.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.24.attn.wq_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.24.attn.wq_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.24.attn.wq_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.26.attn.q_norm.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.26.mlp.gate.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.27.attn.wq_du.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.27.attn.wq_du.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.27.attn.wq_du.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.28.attn.k_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.down_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.down_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.28.mlp.experts.down_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.29.mlp_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.32.attn_norm.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.32.mlp.gate.global_scale": "model-00010-of-00041.safetensors",
+    "model.llm.layers.34.attn.v_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.down_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.down_proj.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.34.mlp.shared_experts.down_proj.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.35.attn.wk_dv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.35.attn.wk_dv.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.35.attn.wk_dv.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.36.attn.wo_ud.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.36.attn.wo_ud.scales": "model-00010-of-00041.safetensors",
+    "model.llm.layers.36.attn.wo_ud.biases": "model-00010-of-00041.safetensors",
+    "model.llm.layers.37.attn.v_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.37.mlp.gate.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.38.attn.v_sconv.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.39.attn.q_norm.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.gate_proj.weight": "model-00010-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.gate_proj.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.gate_proj.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.up_proj.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.up_proj.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.4.mlp.experts.up_proj.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.4.mlp.gate.bias": "model-00011-of-00041.safetensors",
+    "model.llm.layers.41.attn.rel_logits_proj.proj": "model-00011-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.down_proj.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.down_proj.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.down_proj.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.41.mlp.gate.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.5.attn.k_sconv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.5.mlp.gate.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.5.mlp_norm.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.7.attn.rel_logits_proj.proj": "model-00011-of-00041.safetensors",
+    "model.llm.layers.7.attn.wr_du.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.7.attn.wr_du.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.7.attn.wr_du.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.8.attn.k_sconv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.9.attn.rel_logits_proj.proj": "model-00011-of-00041.safetensors",
+    "model.visual.layers.linear_2.weight": "model-00011-of-00041.safetensors",
+    "model.visual.layers.linear_2.scales": "model-00011-of-00041.safetensors",
+    "model.visual.layers.linear_2.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.1.attn.wk_dv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.1.attn.wk_dv.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.1.attn.wk_dv.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.1.mlp.down_proj.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.1.mlp.down_proj.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.1.mlp.down_proj.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.10.attn.rel_logits_proj.proj": "model-00011-of-00041.safetensors",
+    "model.llm.layers.10.attn.wk_dv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.10.attn.wk_dv.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.10.attn.wk_dv.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.17.attn.wr_du.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.17.attn.wr_du.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.17.attn.wr_du.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.22.attn.wk_dv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.22.attn.wk_dv.scales": "model-00011-of-00041.safetensors",
+    "model.llm.layers.22.attn.wk_dv.biases": "model-00011-of-00041.safetensors",
+    "model.llm.layers.24.attn_norm.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.24.attn_sconv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.24.mlp.gate.global_scale": "model-00011-of-00041.safetensors",
+    "model.llm.layers.26.mlp_sconv.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.gate_proj.weight": "model-00011-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.gate_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.gate_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.up_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.up_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.up_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.34.mlp_sconv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.35.attn_norm.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.down_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.down_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.down_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.36.mlp_norm.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.38.mlp.gate.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.39.mlp_sconv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.40.mlp_norm.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.40.mlp_sconv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.attn.wr_du.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.attn.wr_du.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.attn.wr_du.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.attn.wv_dv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.attn.wv_dv.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.attn.wv_dv.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.gate_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.gate_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.gate_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.up_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.up_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.up_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.8.attn.wk_dv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.8.attn.wk_dv.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.8.attn.wk_dv.biases": "model-00012-of-00041.safetensors",
+    "model.visual.layers.linear_0.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.0.attn_norm.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.1.attn.k_norm.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.12.attn.k_sconv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.13.attn.wq_du.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.13.attn.wq_du.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.13.attn.wq_du.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.14.attn.rel_logits_proj.proj": "model-00012-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.down_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.down_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.14.mlp.shared_experts.down_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.15.attn.wq_du.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.15.attn.wq_du.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.15.attn.wq_du.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.16.attn.wq_du.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.16.attn.wq_du.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.16.attn.wq_du.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.16.mlp.gate.bias": "model-00012-of-00041.safetensors",
+    "model.llm.layers.18.attn.wk_dv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.18.attn.wk_dv.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.18.attn.wk_dv.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.2.mlp_norm.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.20.attn.wo_ud.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.20.attn.wo_ud.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.20.attn.wo_ud.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.20.mlp.gate.bias": "model-00012-of-00041.safetensors",
+    "model.llm.layers.20.mlp.gate.global_scale": "model-00012-of-00041.safetensors",
+    "model.llm.layers.20.mlp_sconv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.21.attn_sconv.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.22.attn.wo_ud.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.22.attn.wo_ud.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.22.attn.wo_ud.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.gate_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.gate_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.gate_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.up_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.up_proj.scales": "model-00012-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.up_proj.biases": "model-00012-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.gate_proj.weight": "model-00012-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.gate_proj.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.gate_proj.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.up_proj.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.up_proj.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.up_proj.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.27.attn.wr_du.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.27.attn.wr_du.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.27.attn.wr_du.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.27.mlp.gate.bias": "model-00013-of-00041.safetensors",
+    "model.llm.layers.29.mlp.gate.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.down_proj.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.down_proj.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.down_proj.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.32.attn.k_sconv.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.35.attn.q_norm.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.35.mlp_sconv.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.38.attn.q_norm.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.down_proj.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.down_proj.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.39.mlp.experts.down_proj.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.gate_proj.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.gate_proj.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.gate_proj.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.up_proj.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.up_proj.scales": "model-00013-of-00041.safetensors",
+    "model.llm.layers.4.mlp.shared_experts.up_proj.biases": "model-00013-of-00041.safetensors",
+    "model.llm.layers.40.attn.k_norm.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.41.attn.v_sconv.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.down_proj.weight": "model-00013-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.down_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.6.mlp.experts.down_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.7.attn.v_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.8.attn.k_norm.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.8.attn.rel_logits_proj.proj": "model-00014-of-00041.safetensors",
+    "model.visual.layers.norm_0.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.0.attn.wo_ud.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.0.attn.wo_ud.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.0.attn.wo_ud.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.1.attn_norm.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.10.attn.wq_du.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.10.attn.wq_du.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.10.attn.wq_du.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.10.mlp.gate.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.12.attn.rel_logits_proj.proj": "model-00014-of-00041.safetensors",
+    "model.llm.layers.13.attn_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.down_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.down_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.down_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.17.mlp_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.18.attn.k_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.19.attn.k_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.19.attn.wv_dv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.19.attn.wv_dv.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.19.attn.wv_dv.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.gate_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.gate_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.gate_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.up_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.up_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.up_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.21.attn.wo_ud.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.21.attn.wo_ud.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.21.attn.wo_ud.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.gate_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.gate_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.gate_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.up_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.up_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.up_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.25.attn.rel_logits_proj.proj": "model-00014-of-00041.safetensors",
+    "model.llm.layers.27.attn_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.28.attn.k_norm.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.28.attn.wr_du.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.28.attn.wr_du.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.28.attn.wr_du.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.gate_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.gate_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.gate_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.up_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.up_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.3.mlp.shared_experts.up_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.30.attn.wo_ud.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.30.attn.wo_ud.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.30.attn.wo_ud.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.30.mlp.gate.bias": "model-00014-of-00041.safetensors",
+    "model.llm.layers.30.mlp_norm.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.31.attn.v_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.31.mlp_sconv.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.down_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.down_proj.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.down_proj.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.35.attn.wq_du.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.35.attn.wq_du.scales": "model-00014-of-00041.safetensors",
+    "model.llm.layers.35.attn.wq_du.biases": "model-00014-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.gate_proj.weight": "model-00014-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.gate_proj.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.gate_proj.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.up_proj.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.up_proj.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.35.mlp.experts.up_proj.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.35.mlp_norm.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.gate_proj.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.gate_proj.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.gate_proj.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.up_proj.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.up_proj.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.up_proj.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.40.attn.wo_ud.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.40.attn.wo_ud.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.40.attn.wo_ud.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.7.attn.q_norm.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.8.attn.wq_du.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.8.attn.wq_du.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.8.attn.wq_du.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.9.mlp.gate.bias": "model-00015-of-00041.safetensors",
+    "model.llm.layers.0.mlp.down_proj.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.0.mlp.down_proj.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.0.mlp.down_proj.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.1.attn.rel_logits_proj.proj": "model-00015-of-00041.safetensors",
+    "model.llm.layers.10.attn.k_norm.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.gate_proj.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.gate_proj.scales": "model-00015-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.gate_proj.biases": "model-00015-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.up_proj.weight": "model-00015-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.up_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.up_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.14.attn.v_sconv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.18.attn.wv_dv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.18.attn.wv_dv.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.18.attn.wv_dv.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.20.attn.wq_du.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.20.attn.wq_du.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.20.attn.wq_du.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.down_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.down_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.20.mlp.shared_experts.down_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.24.attn.wr_du.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.24.attn.wr_du.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.24.attn.wr_du.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.25.attn.wk_dv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.25.attn.wk_dv.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.25.attn.wk_dv.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.gate_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.gate_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.gate_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.up_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.up_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.up_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.down_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.down_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.down_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.3.attn.wq_du.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.3.attn.wq_du.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.3.attn.wq_du.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.33.attn.wq_du.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.33.attn.wq_du.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.33.attn.wq_du.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.33.mlp.gate.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.37.mlp.gate.global_scale": "model-00016-of-00041.safetensors",
+    "model.llm.layers.38.attn_norm.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.39.attn.wo_ud.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.39.attn.wo_ud.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.39.attn.wo_ud.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.4.attn.v_sconv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.down_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.down_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.down_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.gate_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.gate_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.gate_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.up_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.up_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.up_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.6.attn.q_norm.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.6.attn_sconv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.6.mlp.gate.global_scale": "model-00016-of-00041.safetensors",
+    "model.llm.layers.9.attn_norm.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.10.attn_norm.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.11.attn.wv_dv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.11.attn.wv_dv.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.11.attn.wv_dv.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.11.mlp.gate.bias": "model-00016-of-00041.safetensors",
+    "model.llm.layers.12.attn_norm.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.12.mlp.gate.global_scale": "model-00016-of-00041.safetensors",
+    "model.llm.layers.12.mlp.gate.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.gate_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.gate_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.gate_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.up_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.up_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.13.mlp.shared_experts.up_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.14.mlp.gate.global_scale": "model-00016-of-00041.safetensors",
+    "model.llm.layers.14.mlp_sconv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.attn.k_sconv.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.gate_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.gate_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.gate_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.up_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.up_proj.scales": "model-00016-of-00041.safetensors",
+    "model.llm.layers.16.mlp.shared_experts.up_proj.biases": "model-00016-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.gate_proj.weight": "model-00016-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.gate_proj.scales": "model-00017-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.gate_proj.biases": "model-00017-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.up_proj.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.up_proj.scales": "model-00017-of-00041.safetensors",
+    "model.llm.layers.17.mlp.experts.up_proj.biases": "model-00017-of-00041.safetensors",
+    "model.llm.layers.17.mlp.gate.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.19.attn.wk_dv.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.19.attn.wk_dv.scales": "model-00017-of-00041.safetensors",
+    "model.llm.layers.19.attn.wk_dv.biases": "model-00017-of-00041.safetensors",
+    "model.llm.layers.19.attn.wr_du.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.19.attn.wr_du.scales": "model-00017-of-00041.safetensors",
+    "model.llm.layers.19.attn.wr_du.biases": "model-00017-of-00041.safetensors",
+    "model.llm.layers.2.attn.k_sconv.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.2.attn_norm.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.23.attn.k_sconv.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.28.attn.wv_dv.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.28.attn.wv_dv.scales": "model-00017-of-00041.safetensors",
+    "model.llm.layers.28.attn.wv_dv.biases": "model-00017-of-00041.safetensors",
+    "model.llm.layers.3.mlp.gate.bias": "model-00017-of-00041.safetensors",
+    "model.llm.layers.31.attn_norm.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.gate_proj.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.gate_proj.scales": "model-00017-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.gate_proj.biases": "model-00017-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.up_proj.weight": "model-00017-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.up_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.32.mlp.experts.up_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.down_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.down_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.down_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.33.attn.wk_dv.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.33.attn.wk_dv.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.33.attn.wk_dv.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.33.attn.wo_ud.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.33.attn.wo_ud.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.33.attn.wo_ud.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.34.attn.rel_logits_proj.proj": "model-00018-of-00041.safetensors",
+    "model.llm.layers.35.mlp.gate.bias": "model-00018-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.down_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.down_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.down_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.37.attn.rel_logits_proj.proj": "model-00018-of-00041.safetensors",
+    "model.llm.layers.39.mlp_norm.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.40.attn.wv_dv.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.40.attn.wv_dv.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.40.attn.wv_dv.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.40.attn_sconv.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.40.mlp.gate.bias": "model-00018-of-00041.safetensors",
+    "model.llm.layers.5.attn.v_sconv.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.gate_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.gate_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.gate_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.up_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.up_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.up_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.8.mlp_norm.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.9.attn.q_norm.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.1.mlp.global_scale": "model-00018-of-00041.safetensors",
+    "model.llm.layers.11.mlp.gate.global_scale": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.attn.q_norm.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.down_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.down_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.experts.down_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.gate_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.gate_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.gate_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.up_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.up_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.up_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.gate_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.gate_proj.scales": "model-00018-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.gate_proj.biases": "model-00018-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.up_proj.weight": "model-00018-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.up_proj.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.up_proj.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.22.attn_norm.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.29.attn.q_norm.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.down_proj.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.down_proj.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.down_proj.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.31.attn.wk_dv.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.31.attn.wk_dv.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.31.attn.wk_dv.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.31.attn.wv_dv.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.31.attn.wv_dv.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.31.attn.wv_dv.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.attn.wk_dv.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.attn.wk_dv.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.attn.wk_dv.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.gate_proj.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.gate_proj.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.gate_proj.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.up_proj.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.up_proj.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.32.mlp.shared_experts.up_proj.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.34.attn.q_norm.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.37.mlp.gate.bias": "model-00019-of-00041.safetensors",
+    "model.llm.layers.7.attn_sconv.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.gate_proj.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.gate_proj.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.gate_proj.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.up_proj.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.up_proj.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.up_proj.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.15.attn.wo_ud.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.15.attn.wo_ud.scales": "model-00019-of-00041.safetensors",
+    "model.llm.layers.15.attn.wo_ud.biases": "model-00019-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.gate_proj.weight": "model-00019-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.gate_proj.scales": "model-00020-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.gate_proj.biases": "model-00020-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.up_proj.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.up_proj.scales": "model-00020-of-00041.safetensors",
+    "model.llm.layers.23.mlp.experts.up_proj.biases": "model-00020-of-00041.safetensors",
+    "model.llm.layers.3.mlp.gate.global_scale": "model-00020-of-00041.safetensors",
+    "model.llm.layers.30.mlp.gate.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.31.attn.wo_ud.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.31.attn.wo_ud.scales": "model-00020-of-00041.safetensors",
+    "model.llm.layers.31.attn.wo_ud.biases": "model-00020-of-00041.safetensors",
+    "model.llm.layers.36.attn_sconv.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.5.attn_sconv.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.0.mlp_norm.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.1.attn.q_norm.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.1.mlp_sconv.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.10.mlp.gate.bias": "model-00020-of-00041.safetensors",
+    "model.llm.layers.12.attn.v_sconv.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.12.attn.wk_dv.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.12.attn.wk_dv.scales": "model-00020-of-00041.safetensors",
+    "model.llm.layers.12.attn.wk_dv.biases": "model-00020-of-00041.safetensors",
+    "model.llm.layers.13.attn.wo_ud.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.13.attn.wo_ud.scales": "model-00020-of-00041.safetensors",
+    "model.llm.layers.13.attn.wo_ud.biases": "model-00020-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.gate_proj.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.gate_proj.scales": "model-00020-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.gate_proj.biases": "model-00020-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.up_proj.weight": "model-00020-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.up_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.up_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.16.attn.wo_ud.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.16.attn.wo_ud.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.16.attn.wo_ud.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.19.mlp_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.down_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.down_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.2.mlp.experts.down_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.20.attn.q_norm.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.20.attn.v_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.21.mlp.gate.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.23.attn.k_norm.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.23.mlp_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.down_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.down_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.25.mlp.shared_experts.down_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.28.attn.wk_dv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.28.attn.wk_dv.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.28.attn.wk_dv.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.28.attn_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.29.attn.wr_du.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.29.attn.wr_du.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.29.attn.wr_du.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.3.attn.rel_logits_proj.proj": "model-00021-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.gate_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.gate_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.gate_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.up_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.up_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.up_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.34.attn.wo_ud.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.34.attn.wo_ud.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.34.attn.wo_ud.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.36.attn_norm.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.attn.wo_ud.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.attn.wo_ud.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.attn.wo_ud.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.attn.wr_du.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.attn.wr_du.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.attn.wr_du.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.down_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.down_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.mlp.experts.down_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.down_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.down_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.38.mlp.shared_experts.down_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.4.attn.wr_du.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.4.attn.wr_du.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.4.attn.wr_du.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.4.attn.wv_dv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.4.attn.wv_dv.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.4.attn.wv_dv.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.41.attn.k_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.6.attn.v_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.7.attn.wv_dv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.7.attn.wv_dv.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.7.attn.wv_dv.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.8.mlp.gate.global_scale": "model-00021-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.down_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.down_proj.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.down_proj.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.0.mlp_sconv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.12.mlp.gate.bias": "model-00021-of-00041.safetensors",
+    "model.llm.layers.14.attn.wv_dv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.14.attn.wv_dv.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.14.attn.wv_dv.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.16.attn.wk_dv.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.16.attn.wk_dv.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.16.attn.wk_dv.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.17.attn.q_norm.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.18.attn.wo_ud.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.18.attn.wo_ud.scales": "model-00021-of-00041.safetensors",
+    "model.llm.layers.18.attn.wo_ud.biases": "model-00021-of-00041.safetensors",
+    "model.llm.layers.18.mlp.gate.bias": "model-00021-of-00041.safetensors",
+    "model.llm.layers.19.attn.k_norm.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.down_proj.weight": "model-00021-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.down_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.down_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.attn.wv_dv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.attn.wv_dv.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.attn.wv_dv.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.gate_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.gate_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.gate_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.up_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.up_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.22.mlp.shared_experts.up_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.23.attn_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.24.attn.v_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.down_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.down_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.24.mlp.shared_experts.down_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.26.attn.k_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.gate_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.gate_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.gate_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.up_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.up_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.up_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.28.attn.q_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.28.mlp.gate.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.29.attn.v_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.down_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.down_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.down_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.30.attn_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.31.mlp.gate.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.39.attn.v_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.39.attn_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.40.attn.q_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.40.attn.v_sconv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.40.attn.wk_dv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.40.attn.wk_dv.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.40.attn.wk_dv.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.41.mlp.gate.global_scale": "model-00022-of-00041.safetensors",
+    "model.llm.layers.41.mlp_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.5.attn.wv_dv.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.5.attn.wv_dv.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.5.attn.wv_dv.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.6.mlp.gate.bias": "model-00022-of-00041.safetensors",
+    "model.llm.layers.7.attn.wo_ud.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.7.attn.wo_ud.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.7.attn.wo_ud.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.8.mlp.gate.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.attn.k_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.down_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.down_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.down_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.gate_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.gate_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.gate_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.up_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.up_proj.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp.shared_experts.up_proj.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.9.mlp_norm.weight": "model-00022-of-00041.safetensors",
+    "model.visual.layers.linear_1.weight": "model-00022-of-00041.safetensors",
+    "model.visual.layers.linear_1.scales": "model-00022-of-00041.safetensors",
+    "model.visual.layers.linear_1.biases": "model-00022-of-00041.safetensors",
+    "model.visual.layers.linear_3.weight": "model-00022-of-00041.safetensors",
+    "model.visual.layers.linear_3.scales": "model-00022-of-00041.safetensors",
+    "model.visual.layers.linear_3.biases": "model-00022-of-00041.safetensors",
+    "model.visual.layers.norm_1.weight": "model-00022-of-00041.safetensors",
+    "model.audio.encoder.weight": "model-00022-of-00041.safetensors",
+    "model.audio.encoder.scales": "model-00022-of-00041.safetensors",
+    "model.audio.encoder.biases": "model-00022-of-00041.safetensors",
+    "model.llm.embed_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.13.attn.wr_du.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.13.attn.wr_du.scales": "model-00022-of-00041.safetensors",
+    "model.llm.layers.13.attn.wr_du.biases": "model-00022-of-00041.safetensors",
+    "model.llm.layers.14.attn.k_norm.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.gate_proj.weight": "model-00022-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.gate_proj.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.gate_proj.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.up_proj.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.up_proj.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.up_proj.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.16.mlp_sconv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.20.mlp_norm.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.24.attn.k_sconv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.29.attn_sconv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.3.mlp.gate.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.down_proj.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.down_proj.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.30.mlp.experts.down_proj.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.34.attn.wk_dv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.34.attn.wk_dv.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.34.attn.wk_dv.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.37.attn.wk_dv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.37.attn.wk_dv.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.37.attn.wk_dv.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.37.attn.wo_ud.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.37.attn.wo_ud.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.37.attn.wo_ud.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.38.mlp.gate.global_scale": "model-00023-of-00041.safetensors",
+    "model.llm.unembed.weight": "model-00023-of-00041.safetensors",
+    "model.llm.unembed.scales": "model-00023-of-00041.safetensors",
+    "model.llm.unembed.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.11.attn.wq_du.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.11.attn.wq_du.scales": "model-00023-of-00041.safetensors",
+    "model.llm.layers.11.attn.wq_du.biases": "model-00023-of-00041.safetensors",
+    "model.llm.layers.13.mlp_sconv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.15.mlp.gate.global_scale": "model-00023-of-00041.safetensors",
+    "model.llm.layers.17.attn.k_sconv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.22.attn.v_sconv.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.gate_proj.weight": "model-00023-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.gate_proj.scales": "model-00024-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.gate_proj.biases": "model-00024-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.up_proj.weight": "model-00024-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.up_proj.scales": "model-00024-of-00041.safetensors",
+    "model.llm.layers.29.mlp.experts.up_proj.biases": "model-00024-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.down_proj.weight": "model-00024-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.down_proj.scales": "model-00024-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.down_proj.biases": "model-00024-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.gate_proj.weight": "model-00024-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.gate_proj.scales": "model-00024-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.gate_proj.biases": "model-00024-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.up_proj.weight": "model-00024-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.up_proj.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.up_proj.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.37.attn.wv_dv.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.37.attn.wv_dv.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.37.attn.wv_dv.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.4.mlp_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.41.attn.wk_dv.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.41.attn.wk_dv.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.41.attn.wk_dv.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.5.attn.k_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.gate_proj.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.gate_proj.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.gate_proj.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.up_proj.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.up_proj.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.up_proj.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.7.mlp_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.8.attn_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.10.attn.wo_ud.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.10.attn.wo_ud.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.10.attn.wo_ud.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.down_proj.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.down_proj.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.down_proj.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.11.attn.k_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.11.attn_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.14.attn.wr_du.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.14.attn.wr_du.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.14.attn.wr_du.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.14.mlp_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.15.attn.k_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.15.attn.wv_dv.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.15.attn.wv_dv.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.15.attn.wv_dv.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.16.attn_sconv.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.gate_proj.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.gate_proj.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.gate_proj.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.up_proj.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.up_proj.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.up_proj.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.19.attn.rel_logits_proj.proj": "model-00025-of-00041.safetensors",
+    "model.llm.layers.19.mlp.gate.global_scale": "model-00025-of-00041.safetensors",
+    "model.llm.layers.2.attn.wk_dv.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.2.attn.wk_dv.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.2.attn.wk_dv.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.20.attn_norm.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.23.attn.rel_logits_proj.proj": "model-00025-of-00041.safetensors",
+    "model.llm.layers.24.attn.wk_dv.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.24.attn.wk_dv.scales": "model-00025-of-00041.safetensors",
+    "model.llm.layers.24.attn.wk_dv.biases": "model-00025-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.down_proj.weight": "model-00025-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.24.mlp.experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.24.mlp_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.attn.wv_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.attn.wv_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.attn.wv_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.26.attn.wr_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.26.attn.wr_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.26.attn.wr_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.26.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.28.attn_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.k_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.mlp_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.31.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.31.mlp.gate.global_scale": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wr_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wr_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wr_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wv_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wv_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wv_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.gate_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.gate_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.gate_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.up_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.up_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.shared_experts.up_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.34.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.attn.k_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.attn.wo_ud.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.attn.wo_ud.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.attn.wo_ud.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.gate.global_scale": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.gate_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.gate_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.gate_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.up_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.up_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.35.mlp.shared_experts.up_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.38.attn.k_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.40.attn.rel_logits_proj.proj": "model-00026-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.41.mlp.shared_experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.5.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.5.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.5.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wq_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wq_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wq_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wv_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wv_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.6.attn.wv_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.7.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.8.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.9.attn.k_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.9.attn.wq_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.9.attn.wq_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.9.attn.wq_du.biases": "model-00026-of-00041.safetensors",
+    "model.audio.final_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.10.attn.q_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.gate.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.gate_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.gate_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.gate_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.up_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.up_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.11.mlp.shared_experts.up_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.13.attn.q_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.13.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.13.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.13.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.14.attn.q_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.gate_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.gate_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.gate_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.up_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.up_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.15.mlp.shared_experts.up_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.17.mlp.shared_experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.attn.q_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.attn.v_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.attn.wq_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.attn.wq_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.attn.wq_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.19.attn.wo_ud.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.19.attn.wo_ud.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.19.attn.wo_ud.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.19.mlp.experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.2.attn.wv_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.2.attn.wv_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.2.attn.wv_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.2.mlp.gate.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.20.attn.rel_logits_proj.proj": "model-00026-of-00041.safetensors",
+    "model.llm.layers.20.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.20.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.20.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.21.attn.k_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.21.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.21.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.21.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.21.mlp_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.22.attn.k_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.22.attn.wq_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.22.attn.wq_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.22.attn.wq_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.23.attn.wr_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.23.attn.wr_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.23.attn.wr_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.23.mlp.gate.global_scale": "model-00026-of-00041.safetensors",
+    "model.llm.layers.24.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.attn.wo_ud.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.attn.wo_ud.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.attn.wo_ud.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.mlp.gate.global_scale": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.mlp_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.25.mlp_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.26.attn_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.attn.wk_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.attn.wk_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.attn.wk_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.attn_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.mlp.gate.global_scale": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.down_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.down_proj.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.27.mlp.shared_experts.down_proj.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.k_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.wv_dv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.wv_dv.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.29.attn.wv_dv.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.k_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.wo_ud.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.wo_ud.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.wo_ud.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.wr_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.wr_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.3.attn.wr_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.31.attn.rel_logits_proj.proj": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wo_ud.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wo_ud.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wo_ud.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wq_du.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wq_du.scales": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.attn.wq_du.biases": "model-00026-of-00041.safetensors",
+    "model.llm.layers.32.mlp.gate.bias": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.attn.k_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.attn.rel_logits_proj.proj": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.attn_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.33.mlp.gate.global_scale": "model-00026-of-00041.safetensors",
+    "model.llm.layers.34.attn.k_sconv.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.34.attn_norm.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.gate_proj.weight": "model-00026-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.gate_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.gate_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.up_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.up_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.up_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.34.mlp.gate.global_scale": "model-00027-of-00041.safetensors",
+    "model.llm.layers.35.attn.v_sconv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.36.attn.k_sconv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.36.mlp.gate.bias": "model-00027-of-00041.safetensors",
+    "model.llm.layers.36.mlp_sconv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.37.attn.wr_du.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.37.attn.wr_du.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.37.attn.wr_du.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.down_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.down_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.37.mlp.shared_experts.down_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.38.attn.rel_logits_proj.proj": "model-00027-of-00041.safetensors",
+    "model.llm.layers.38.mlp_sconv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.4.attn.wk_dv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.4.attn.wk_dv.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.4.attn.wk_dv.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.4.attn.wq_du.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.4.attn.wq_du.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.4.attn.wq_du.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.40.attn.wr_du.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.40.attn.wr_du.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.40.attn.wr_du.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.41.attn_norm.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.gate.global_scale": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.gate_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.gate_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.gate_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.up_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.up_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.up_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.down_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.down_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.5.mlp.shared_experts.down_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.down_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.down_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.down_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.8.attn_sconv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.9.mlp_sconv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.attn.wv_dv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.attn.wv_dv.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.attn.wv_dv.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.gate.global_scale": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.gate_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.gate_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.gate_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.up_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.up_proj.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.10.mlp.shared_experts.up_proj.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.11.attn.wr_du.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.11.attn.wr_du.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.11.attn.wr_du.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.12.attn.wv_dv.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.12.attn.wv_dv.scales": "model-00027-of-00041.safetensors",
+    "model.llm.layers.12.attn.wv_dv.biases": "model-00027-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.gate_proj.weight": "model-00027-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.gate_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.gate_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.up_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.up_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.up_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.21.attn.wr_du.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.21.attn.wr_du.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.21.attn.wr_du.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.22.mlp.gate.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.23.attn.v_sconv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.down_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.down_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.23.mlp.shared_experts.down_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.25.attn.q_norm.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.down_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.down_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.26.mlp.shared_experts.down_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.26.mlp_norm.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.3.attn.wv_dv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.3.attn.wv_dv.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.3.attn.wv_dv.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.32.attn.k_norm.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.32.attn_sconv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.33.attn.k_sconv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.37.attn.q_norm.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.39.mlp.gate.global_scale": "model-00028-of-00041.safetensors",
+    "model.llm.layers.41.attn.q_norm.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.41.mlp.gate.bias": "model-00028-of-00041.safetensors",
+    "model.llm.layers.5.mlp_sconv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.6.mlp.gate.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.attn.wk_dv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.attn.wk_dv.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.attn.wk_dv.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.down_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.down_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.mlp.shared_experts.down_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.7.mlp_sconv.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.gate_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.gate_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.gate_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.up_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.up_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.up_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.gate_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.gate_proj.scales": "model-00028-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.gate_proj.biases": "model-00028-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.up_proj.weight": "model-00028-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.up_proj.scales": "model-00029-of-00041.safetensors",
+    "model.llm.layers.9.mlp.experts.up_proj.biases": "model-00029-of-00041.safetensors",
+    "model.llm.layers.11.attn.v_sconv.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.15.attn.v_sconv.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.2.attn.wr_du.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.2.attn.wr_du.scales": "model-00029-of-00041.safetensors",
+    "model.llm.layers.2.attn.wr_du.biases": "model-00029-of-00041.safetensors",
+    "model.llm.layers.2.attn_sconv.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.20.attn_sconv.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.gate_proj.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.gate_proj.scales": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.gate_proj.biases": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.up_proj.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.up_proj.scales": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.experts.up_proj.biases": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.gate_proj.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.gate_proj.scales": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.gate_proj.biases": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.up_proj.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.up_proj.scales": "model-00029-of-00041.safetensors",
+    "model.llm.layers.21.mlp.shared_experts.up_proj.biases": "model-00029-of-00041.safetensors",
+    "model.llm.layers.22.attn.rel_logits_proj.proj": "model-00029-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.gate_proj.weight": "model-00029-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.gate_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.gate_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.up_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.up_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.22.mlp.experts.up_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.25.attn.k_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.attn.rel_logits_proj.proj": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.gate_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.gate_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.gate_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.up_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.up_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.29.mlp.shared_experts.up_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.3.mlp_sconv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.30.attn.wv_dv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.30.attn.wv_dv.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.30.attn.wv_dv.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.34.attn.k_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.36.attn.k_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.37.attn.k_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.38.attn_sconv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.38.mlp_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.39.attn.k_sconv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.39.attn.wv_dv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.39.attn.wv_dv.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.39.attn.wv_dv.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.39.mlp.gate.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.7.mlp.gate.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.8.mlp_sconv.weight": "model-00030-of-00041.safetensors",
+    "model.visual.final_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.0.attn.wk_dv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.0.attn.wk_dv.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.0.attn.wk_dv.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.10.attn.wr_du.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.10.attn.wr_du.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.10.attn.wr_du.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.13.attn.wv_dv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.13.attn.wv_dv.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.13.attn.wv_dv.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.15.attn.wr_du.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.15.attn.wr_du.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.15.attn.wr_du.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.15.mlp_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.down_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.down_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.16.mlp.experts.down_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.17.attn.wo_ud.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.17.attn.wo_ud.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.17.attn.wo_ud.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.gate_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.gate_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.gate_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.up_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.up_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.up_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.down_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.down_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.19.mlp.shared_experts.down_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.2.attn.k_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.2.attn.q_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.2.attn.v_sconv.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.down_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.down_proj.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.2.mlp.shared_experts.down_proj.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.20.attn.wr_du.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.20.attn.wr_du.scales": "model-00030-of-00041.safetensors",
+    "model.llm.layers.20.attn.wr_du.biases": "model-00030-of-00041.safetensors",
+    "model.llm.layers.23.mlp_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.24.mlp_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.30.mlp.gate.global_scale": "model-00030-of-00041.safetensors",
+    "model.llm.layers.31.attn.q_norm.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.down_proj.weight": "model-00030-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.down_proj.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.down_proj.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.32.attn.v_sconv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.36.attn.wv_dv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.36.attn.wv_dv.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.36.attn.wv_dv.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.gate_proj.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.gate_proj.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.gate_proj.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.up_proj.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.up_proj.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.up_proj.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.38.attn.wk_dv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.38.attn.wk_dv.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.38.attn.wk_dv.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.41.attn.wo_ud.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.41.attn.wo_ud.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.41.attn.wo_ud.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.5.attn.rel_logits_proj.proj": "model-00031-of-00041.safetensors",
+    "model.llm.layers.6.attn.rel_logits_proj.proj": "model-00031-of-00041.safetensors",
+    "model.llm.layers.9.attn.wo_ud.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.9.attn.wo_ud.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.9.attn.wo_ud.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.9.attn.wv_dv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.9.attn.wv_dv.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.9.attn.wv_dv.biases": "model-00031-of-00041.safetensors",
+    "model.llm.norm.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.0.attn.wr_du.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.0.attn.wr_du.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.0.attn.wr_du.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.0.attn.wv_dv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.0.attn.wv_dv.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.0.attn.wv_dv.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.attn.wo_ud.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.attn.wo_ud.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.attn.wo_ud.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.attn.wv_dv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.attn.wv_dv.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.attn.wv_dv.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.1.mlp_norm.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.10.mlp_sconv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.12.attn_sconv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.down_proj.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.down_proj.scales": "model-00031-of-00041.safetensors",
+    "model.llm.layers.12.mlp.shared_experts.down_proj.biases": "model-00031-of-00041.safetensors",
+    "model.llm.layers.14.attn_norm.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.14.attn_sconv.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.down_proj.weight": "model-00031-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.down_proj.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.14.mlp.experts.down_proj.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.16.attn.wr_du.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.16.attn.wr_du.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.16.attn.wr_du.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.17.attn_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.18.attn.k_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.18.attn_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.19.attn.v_sconv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.19.attn_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.19.mlp_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.21.attn.wq_du.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.21.attn.wq_du.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.21.attn.wq_du.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.21.attn.wv_dv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.21.attn.wv_dv.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.21.attn.wv_dv.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.23.attn.q_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.23.mlp.gate.bias": "model-00032-of-00041.safetensors",
+    "model.llm.layers.23.mlp.gate.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.24.attn.rel_logits_proj.proj": "model-00032-of-00041.safetensors",
+    "model.llm.layers.24.attn.wv_dv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.24.attn.wv_dv.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.24.attn.wv_dv.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.25.attn.wq_du.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.25.attn.wq_du.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.25.attn.wq_du.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.26.attn.wk_dv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.26.attn.wk_dv.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.26.attn.wk_dv.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.27.attn.wo_ud.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.27.attn.wo_ud.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.27.attn.wo_ud.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.attn.v_sconv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.gate_proj.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.gate_proj.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.gate_proj.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.up_proj.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.up_proj.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.28.mlp.shared_experts.up_proj.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.29.attn.wq_du.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.29.attn.wq_du.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.29.attn.wq_du.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.3.attn.k_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.30.attn.k_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.30.attn.q_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.30.mlp_sconv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.31.attn.k_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.31.mlp_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.33.attn.wv_dv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.33.attn.wv_dv.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.33.attn.wv_dv.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.down_proj.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.down_proj.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.33.mlp.experts.down_proj.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.36.attn.wk_dv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.36.attn.wk_dv.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.36.attn.wk_dv.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.attn.rel_logits_proj.proj": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.attn.wr_du.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.attn.wr_du.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.attn.wr_du.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.down_proj.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.down_proj.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.39.mlp.shared_experts.down_proj.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.4.attn.rel_logits_proj.proj": "model-00032-of-00041.safetensors",
+    "model.llm.layers.4.mlp.gate.global_scale": "model-00032-of-00041.safetensors",
+    "model.llm.layers.41.attn_sconv.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.5.attn.q_norm.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.gate_proj.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.gate_proj.scales": "model-00032-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.gate_proj.biases": "model-00032-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.up_proj.weight": "model-00032-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.up_proj.scales": "model-00033-of-00041.safetensors",
+    "model.llm.layers.5.mlp.experts.up_proj.biases": "model-00033-of-00041.safetensors",
+    "model.llm.layers.5.mlp.gate.bias": "model-00033-of-00041.safetensors",
+    "model.llm.layers.7.attn.k_sconv.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.down_proj.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.down_proj.scales": "model-00033-of-00041.safetensors",
+    "model.llm.layers.8.mlp.shared_experts.down_proj.biases": "model-00033-of-00041.safetensors",
+    "model.visual.layers.norm_2.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.1.attn.wq_du.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.1.attn.wq_du.scales": "model-00033-of-00041.safetensors",
+    "model.llm.layers.1.attn.wq_du.biases": "model-00033-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.down_proj.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.down_proj.scales": "model-00033-of-00041.safetensors",
+    "model.llm.layers.10.mlp.experts.down_proj.biases": "model-00033-of-00041.safetensors",
+    "model.llm.layers.11.mlp_sconv.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.13.attn_norm.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.gate_proj.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.gate_proj.scales": "model-00033-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.gate_proj.biases": "model-00033-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.up_proj.weight": "model-00033-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.up_proj.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.up_proj.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.down_proj.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.down_proj.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.13.mlp.experts.down_proj.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.14.mlp.gate.bias": "model-00034-of-00041.safetensors",
+    "model.llm.layers.15.attn_sconv.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.17.mlp.gate.bias": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.gate.global_scale": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.gate_proj.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.gate_proj.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.gate_proj.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.up_proj.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.up_proj.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.18.mlp.shared_experts.up_proj.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.20.mlp.gate.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.21.attn.k_sconv.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.21.mlp_sconv.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.24.attn.wo_ud.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.24.attn.wo_ud.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.24.attn.wo_ud.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.25.mlp.gate.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.30.attn.k_sconv.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.33.attn.q_norm.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.0.attn.rel_logits_proj.proj": "model-00034-of-00041.safetensors",
+    "model.llm.layers.12.attn.wo_ud.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.12.attn.wo_ud.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.12.attn.wo_ud.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.14.attn.wq_du.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.14.attn.wq_du.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.14.attn.wq_du.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.16.attn.k_norm.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.25.attn.v_sconv.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.down_proj.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.down_proj.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.25.mlp.experts.down_proj.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.27.attn.q_norm.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.28.attn.rel_logits_proj.proj": "model-00034-of-00041.safetensors",
+    "model.llm.layers.28.attn.wq_du.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.28.attn.wq_du.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.28.attn.wq_du.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.3.attn_sconv.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.31.attn.wr_du.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.31.attn.wr_du.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.31.attn.wr_du.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.34.mlp_norm.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.35.attn.wr_du.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.35.attn.wr_du.scales": "model-00034-of-00041.safetensors",
+    "model.llm.layers.35.attn.wr_du.biases": "model-00034-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.down_proj.weight": "model-00034-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.down_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.36.mlp.experts.down_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.down_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.down_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.36.mlp.shared_experts.down_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.38.attn.wq_du.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.38.attn.wq_du.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.38.attn.wq_du.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.gate_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.gate_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.gate_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.up_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.up_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.41.mlp.experts.up_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.down_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.down_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.6.mlp.shared_experts.down_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.8.attn.q_norm.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.8.attn.wo_ud.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.8.attn.wo_ud.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.8.attn.wo_ud.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.9.attn.wk_dv.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.9.attn.wk_dv.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.9.attn.wk_dv.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.1.mlp.gate_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.1.mlp.gate_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.1.mlp.gate_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.1.mlp.up_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.1.mlp.up_proj.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.1.mlp.up_proj.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.10.attn_sconv.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.12.attn.k_norm.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.12.attn.wr_du.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.12.attn.wr_du.scales": "model-00035-of-00041.safetensors",
+    "model.llm.layers.12.attn.wr_du.biases": "model-00035-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.down_proj.weight": "model-00035-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.down_proj.scales": "model-00036-of-00041.safetensors",
+    "model.llm.layers.15.mlp.experts.down_proj.biases": "model-00036-of-00041.safetensors",
+    "model.llm.layers.21.attn.v_sconv.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.24.attn.k_norm.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.25.attn.wr_du.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.25.attn.wr_du.scales": "model-00036-of-00041.safetensors",
+    "model.llm.layers.25.attn.wr_du.biases": "model-00036-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.down_proj.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.down_proj.scales": "model-00036-of-00041.safetensors",
+    "model.llm.layers.26.mlp.experts.down_proj.biases": "model-00036-of-00041.safetensors",
+    "model.llm.layers.27.attn.rel_logits_proj.proj": "model-00036-of-00041.safetensors",
+    "model.llm.layers.28.mlp_norm.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.31.attn_sconv.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.35.attn.wv_dv.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.35.attn.wv_dv.scales": "model-00036-of-00041.safetensors",
+    "model.llm.layers.35.attn.wv_dv.biases": "model-00036-of-00041.safetensors",
+    "model.llm.layers.36.mlp.gate.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.37.mlp_norm.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.39.mlp.gate.bias": "model-00036-of-00041.safetensors",
+    "model.llm.layers.4.attn.k_norm.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.40.attn_norm.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.6.attn.k_sconv.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.6.attn_norm.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.gate_proj.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.gate_proj.scales": "model-00036-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.gate_proj.biases": "model-00036-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.up_proj.weight": "model-00036-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.up_proj.scales": "model-00037-of-00041.safetensors",
+    "model.llm.layers.7.mlp.experts.up_proj.biases": "model-00037-of-00041.safetensors",
+    "model.llm.embed.weight": "model-00037-of-00041.safetensors",
+    "model.llm.embed.scales": "model-00037-of-00041.safetensors",
+    "model.llm.embed.biases": "model-00037-of-00041.safetensors",
+    "model.llm.layers.0.attn.k_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.1.attn.v_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.10.attn.v_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.gate_proj.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.gate_proj.scales": "model-00037-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.gate_proj.biases": "model-00037-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.up_proj.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.up_proj.scales": "model-00037-of-00041.safetensors",
+    "model.llm.layers.11.mlp.experts.up_proj.biases": "model-00037-of-00041.safetensors",
+    "model.llm.layers.12.mlp_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.14.attn.k_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.18.attn.rel_logits_proj.proj": "model-00037-of-00041.safetensors",
+    "model.llm.layers.18.mlp.gate.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.21.attn.rel_logits_proj.proj": "model-00037-of-00041.safetensors",
+    "model.llm.layers.22.attn.k_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.26.attn.k_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.26.attn_sconv.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.27.mlp.gate.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.3.attn_norm.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.30.attn.wq_du.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.30.attn.wq_du.scales": "model-00037-of-00041.safetensors",
+    "model.llm.layers.30.attn.wq_du.biases": "model-00037-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.gate_proj.weight": "model-00037-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.gate_proj.scales": "model-00038-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.gate_proj.biases": "model-00038-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.up_proj.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.up_proj.scales": "model-00038-of-00041.safetensors",
+    "model.llm.layers.31.mlp.experts.up_proj.biases": "model-00038-of-00041.safetensors",
+    "model.llm.layers.32.mlp_norm.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.34.attn_sconv.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.36.attn.rel_logits_proj.proj": "model-00038-of-00041.safetensors",
+    "model.llm.layers.36.attn.v_sconv.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.39.attn_norm.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.4.attn_norm.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.9.attn_sconv.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.10.mlp_norm.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.14.attn.wk_dv.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.14.attn.wk_dv.scales": "model-00038-of-00041.safetensors",
+    "model.llm.layers.14.attn.wk_dv.biases": "model-00038-of-00041.safetensors",
+    "model.llm.layers.17.mlp.gate.global_scale": "model-00038-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.down_proj.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.down_proj.scales": "model-00038-of-00041.safetensors",
+    "model.llm.layers.20.mlp.experts.down_proj.biases": "model-00038-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.down_proj.weight": "model-00038-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.down_proj.scales": "model-00039-of-00041.safetensors",
+    "model.llm.layers.34.mlp.experts.down_proj.biases": "model-00039-of-00041.safetensors",
+    "model.llm.layers.38.mlp.gate.bias": "model-00039-of-00041.safetensors",
+    "model.llm.layers.4.attn.k_sconv.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.gate_proj.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.gate_proj.scales": "model-00039-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.gate_proj.biases": "model-00039-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.up_proj.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.up_proj.scales": "model-00039-of-00041.safetensors",
+    "model.llm.layers.40.mlp.experts.up_proj.biases": "model-00039-of-00041.safetensors",
+    "model.llm.layers.11.attn.q_norm.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.13.mlp.gate.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.15.attn.q_norm.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.17.mlp_norm.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.down_proj.weight": "model-00039-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.down_proj.scales": "model-00040-of-00041.safetensors",
+    "model.llm.layers.18.mlp.experts.down_proj.biases": "model-00040-of-00041.safetensors",
+    "model.llm.layers.25.attn_sconv.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.28.mlp.gate.bias": "model-00040-of-00041.safetensors",
+    "model.llm.layers.29.mlp.gate.bias": "model-00040-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.gate_proj.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.gate_proj.scales": "model-00040-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.gate_proj.biases": "model-00040-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.up_proj.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.up_proj.scales": "model-00040-of-00041.safetensors",
+    "model.llm.layers.3.mlp.experts.up_proj.biases": "model-00040-of-00041.safetensors",
+    "model.llm.layers.30.attn.rel_logits_proj.proj": "model-00040-of-00041.safetensors",
+    "model.llm.layers.30.attn.wk_dv.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.30.attn.wk_dv.scales": "model-00040-of-00041.safetensors",
+    "model.llm.layers.30.attn.wk_dv.biases": "model-00040-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.down_proj.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.down_proj.scales": "model-00040-of-00041.safetensors",
+    "model.llm.layers.31.mlp.shared_experts.down_proj.biases": "model-00040-of-00041.safetensors",
+    "model.llm.layers.34.attn.wq_du.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.34.attn.wq_du.scales": "model-00040-of-00041.safetensors",
+    "model.llm.layers.34.attn.wq_du.biases": "model-00040-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.down_proj.weight": "model-00040-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.down_proj.scales": "model-00041-of-00041.safetensors",
+    "model.llm.layers.37.mlp.experts.down_proj.biases": "model-00041-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.down_proj.weight": "model-00041-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.down_proj.scales": "model-00041-of-00041.safetensors",
+    "model.llm.layers.40.mlp.shared_experts.down_proj.biases": "model-00041-of-00041.safetensors",
+    "model.llm.layers.41.attn.wq_du.weight": "model-00041-of-00041.safetensors",
+    "model.llm.layers.41.attn.wq_du.scales": "model-00041-of-00041.safetensors",
+    "model.llm.layers.41.attn.wq_du.biases": "model-00041-of-00041.safetensors",
+    "model.llm.layers.41.mlp_sconv.weight": "model-00041-of-00041.safetensors",
+    "model.llm.layers.5.attn.wq_du.weight": "model-00041-of-00041.safetensors",
+    "model.llm.layers.5.attn.wq_du.scales": "model-00041-of-00041.safetensors",
+    "model.llm.layers.5.attn.wq_du.biases": "model-00041-of-00041.safetensors",
+    "model.llm.layers.7.attn.wq_du.weight": "model-00041-of-00041.safetensors",
+    "model.llm.layers.7.attn.wq_du.scales": "model-00041-of-00041.safetensors",
+    "model.llm.layers.7.attn.wq_du.biases": "model-00041-of-00041.safetensors",
+    "model.llm.layers.7.attn_norm.weight": "model-00041-of-00041.safetensors",
+    "model.llm.layers.8.attn.v_sconv.weight": "model-00041-of-00041.safetensors"
+  }
+}
\ No newline at end of file
diff --git a/processor_config.json b/processor_config.json
new file mode 100644
index 0000000000000000000000000000000000000000..5119c4247a30de9fea35fe269cc786db37b8d67b
--- /dev/null
+++ b/processor_config.json
@@ -0,0 +1,46 @@
+{
+  "audio_token": "<|unused_200053|>",
+  "audio_bos_token": "<|content_audio_input|>",
+  "dmel_max_value": 2.0,
+  "dmel_min_value": -7.0,
+  "feature_extractor": {
+    "audio_token_duration_s": 0.05,
+    "feature_extractor_type": "InklingFeatureExtractor",
+    "feature_size": 80,
+    "hop_length": 800,
+    "n_fft": 1600,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "return_attention_mask": true,
+    "sampling_rate": 16000,
+    "window_size": 1600,
+    "window_size_multiplier": 2.0
+  },
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.48145466,
+      0.4578275,
+      0.40821073
+    ],
+    "image_processor_type": "InklingImageProcessor",
+    "image_std": [
+      0.26862954,
+      0.26130258,
+      0.27577711
+    ],
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "height": 40,
+      "width": 40
+    }
+  },
+  "image_token": "<|unused_200054|>",
+  "image_bos_token": "<|content_image|>",
+  "num_dmel_bins": 16,
+  "processor_class": "InklingProcessor"
+}
diff --git a/special_tokens_map.json b/special_tokens_map.json
new file mode 100644
index 0000000000000000000000000000000000000000..0c6ed62743194dc98ba50d3ca338bf906503b80b
--- /dev/null
+++ b/special_tokens_map.json
@@ -0,0 +1,22 @@
+{
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<|message_user|>",
+    "<|message_model|>",
+    "<|message_system|>",
+    "<|message_tool|>",
+    "<|content_text|>",
+    "<|content_image|>",
+    "<|content_model_end_sampling|>",
+    "<|content_thinking|>",
+    "<|end_message|>",
+    "<|content_audio_input|>",
+    "<|content_tool_error|>",
+    "<|audio|>",
+    "<|content_xml|>",
+    "<|begin_of_text|>",
+    "<|audio_end|>",
+    "<|content_invoke_tool_json|>",
+    "<|content_invoke_tool_text|>"
+  ]
+}
diff --git a/tiktoken/tokenizer.model b/tiktoken/tokenizer.model
new file mode 100644
index 0000000000000000000000000000000000000000..9367ddda1a28202d89d24db9b93f34e1834ea01c
--- /dev/null
+++ b/tiktoken/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc253fd2b702f7a6da7105eaa8f3463b2f1247e83614f23e5323b921088bed2a
+size 3615874
diff --git a/tokenizer.json b/tokenizer.json
new file mode 100644
index 0000000000000000000000000000000000000000..8375e6ba74d80af765dbaf2acbaffd33885ad379
--- /dev/null
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fb6333a7db8fe5da90728e741e4a3ee4ac2ae12c5dd4958cc6f31688787d3c2
+size 27875797
diff --git a/tokenizer_config.json b/tokenizer_config.json
new file mode 100644
index 0000000000000000000000000000000000000000..3f7b53bb76be489908376923167e78d060fa20c6
--- /dev/null
+++ b/tokenizer_config.json
@@ -0,0 +1,508 @@
+{
+  "added_tokens_decoder": {
+    "199998": {
+      "content": "<|unused|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "199999": {
+      "content": "<|endoftext|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200000": {
+      "content": "<|message_user|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200001": {
+      "content": "<|message_model|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200002": {
+      "content": "<|message_system|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200003": {
+      "content": "<|message_tool|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200004": {
+      "content": "<|content_text|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200005": {
+      "content": "<|content_image|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200006": {
+      "content": "<|content_model_end_sampling|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200007": {
+      "content": "<|unused_200007|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200008": {
+      "content": "<|content_thinking|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200009": {
+      "content": "<|unused_200009|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200010": {
+      "content": "<|end_message|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200011": {
+      "content": "<|unused_200011|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200012": {
+      "content": "<|unused_200012|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200013": {
+      "content": "<|unused_200013|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200014": {
+      "content": "<|unused_200014|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200015": {
+      "content": "<|unused_200015|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200016": {
+      "content": "<|unused_200016|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200017": {
+      "content": "<|unused_200017|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200018": {
+      "content": "<|unused_200018|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200019": {
+      "content": "<|unused_200019|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200020": {
+      "content": "<|content_audio_input|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200021": {
+      "content": "<|unused_200021|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200022": {
+      "content": "<|content_tool_error|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200023": {
+      "content": "<|audio|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200024": {
+      "content": "<|content_xml|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200025": {
+      "content": "<|unused_200025|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200026": {
+      "content": "<|unused_200026|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200027": {
+      "content": "<|unused_200027|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200028": {
+      "content": "<|begin_of_text|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200029": {
+      "content": "<|unused_200029|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200030": {
+      "content": "<|unused_200030|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200031": {
+      "content": "<|unused_200031|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200032": {
+      "content": "<|unused_200032|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200033": {
+      "content": "<|unused_200033|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200034": {
+      "content": "<|unused_200034|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200035": {
+      "content": "<|unused_200035|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200036": {
+      "content": "<|unused_200036|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200037": {
+      "content": "<|unused_200037|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200038": {
+      "content": "<|unused_200038|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200039": {
+      "content": "<|unused_200039|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200040": {
+      "content": "<|unused_200040|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200041": {
+      "content": "<|unused_200041|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200042": {
+      "content": "<|unused_200042|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200043": {
+      "content": "<|audio_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200044": {
+      "content": "<|unused_200044|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200045": {
+      "content": "<|unused_200045|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200046": {
+      "content": "<|unused_200046|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200047": {
+      "content": "<|unused_200047|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200048": {
+      "content": "<|unused_200048|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200049": {
+      "content": "<|content_invoke_tool_json|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200050": {
+      "content": "<|unused_200050|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200051": {
+      "content": "<|unused_200051|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200052": {
+      "content": "<|unused_200052|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200053": {
+      "content": "<|unused_200053|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200054": {
+      "content": "<|unused_200054|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200055": {
+      "content": "<|unused_200055|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200056": {
+      "content": "<|unused_200056|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "200057": {
+      "content": "<|content_invoke_tool_text|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": false,
+  "fix_mistral_regex": false,
+  "extra_special_tokens": {
+    "endoftext": "<|endoftext|>",
+    "message_user": "<|message_user|>",
+    "message_model": "<|message_model|>",
+    "message_system": "<|message_system|>",
+    "message_tool": "<|message_tool|>",
+    "content_text": "<|content_text|>",
+    "content_image": "<|content_image|>",
+    "content_model_end_sampling": "<|content_model_end_sampling|>",
+    "content_thinking": "<|content_thinking|>",
+    "end_message": "<|end_message|>",
+    "content_audio_input": "<|content_audio_input|>",
+    "content_tool_error": "<|content_tool_error|>",
+    "audio": "<|audio|>",
+    "content_xml": "<|content_xml|>",
+    "begin_of_text": "<|begin_of_text|>",
+    "audio_end": "<|audio_end|>",
+    "content_invoke_tool_json": "<|content_invoke_tool_json|>",
+    "content_invoke_tool_text": "<|content_invoke_tool_text|>"
+  },
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}