Initial Upload

.quantization_complete

@@ -0,0 +1,31 @@
+{
+  "schema_version": 1,
+  "model_id": "tiiuae/Falcon3-3B-Instruct",
+  "format": "auto_awq",
+  "bits": 4,
+  "group_size": 64,
+  "sym": false,
+  "symmetry_label": "asym",
+  "duration_s": 19986.7,
+  "completed_at_utc": "2026-07-02T11:32:27.433950+00:00",
+  "auto_round_version": "0.13.1",
+  "transformers_version": "4.55.3",
+  "torch_version": "2.12.1+cpu",
+  "torch_dtype": "torch.bfloat16",
+  "device": "cpu",
+  "calibration": {
+    "nsamples": 128,
+    "iters": 200,
+    "seqlen": 512,
+    "batch_size": 4
+  },
+  "base_model_type": "llama",
+  "quality_metrics": {
+    "schema_version": 1,
+    "per_block": [],
+    "aggregate": {
+      "n_blocks": 0,
+      "available": false
+    }
+  }
+}

README.md

@@ -0,0 +1,165 @@
+---
+language:
+- en
+tags:
+- awq
+- auto-awq
+- autoawq
+- pytorch
+- causal-lm
+- autoround
+- auto-round
+- intel-autoround
+- intel
+- woq
+- weights-only-quantization
+- falcon
+- falcon3
+- tii
+- 4-bit
+license: apache-2.0
+license_link: https://choosealicense.com/licenses/apache-2.0/
+library_name: transformers
+model_name: Falcon3 3B Instruct
+base_model:
+- tiiuae/Falcon3-3B-Instruct
+base_model_relation: quantized
+model_type: llama
+inference: false
+model_creator: fbaldassarri
+pipeline_tag: text-generation
+quantized_by: fbaldassarri
+quantization_config:
+  method: auto_awq
+  bits: 4
+  group_size: 64
+  sym: false
+  auto_round_version: 0.13.1
+  torch_dtype: torch.bfloat16
+  device: cpu
+  nsamples: 128
+  iters: 200
+  seqlen: 512
+  batch_size: 4
+---
+
+## Model Information
+
+Quantized version of [tiiuae/Falcon3-3B-Instruct](https://huggingface.co/tiiuae/Falcon3-3B-Instruct) using `torch.bfloat16` for quantization tuning.
+- 4 bits (INT4)
+- group size = 64
+- Asymmetrical Quantization
+- Method: WoQ — AWQ (AutoAWQ algorithm)
+
+Fast and low memory, 2-3X speedup (slight accuracy drop at W4G64)
+
+Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round) v0.13.1
+
+Note: this INT4 version of `Falcon3 3B Instruct` has been quantized for inference on Intel CPU, Intel iGPU (Arc) via intel-extension-for-pytorch, Intel NPU (AI Boost on Core Ultra series) via OpenVINO.
+
+## Usage
+
+This is an **instruct / chat** model — use its built-in chat template rather than raw text prompts:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "fbaldassarri/tiiuae_Falcon3-3B-Instruct-auto_awq-int4-gs64-asym"
+model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained(repo)
+
+messages = [{"role": "user", "content": "Hello!"}]
+inputs = tokenizer.apply_chat_template(
+    messages, return_tensors="pt", add_generation_prompt=True
+).to(model.device)
+outputs = model.generate(inputs, max_new_tokens=128)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Replication Recipe
+
+The recommended way to reproduce this exact quantization is via the [auto-round-pipeline](https://git.epicdynamic.com/auto-round-pipeline) — the same orchestration tool that produced this artifact.
+
+### Step 1 — Bootstrap the auto-round-pipeline
+
+Set up a dedicated conda environment using the pipeline's `setup.sh`. Any of the install modes below produces an environment that can reproduce this quantization; pick the one that matches your goals:
+
+```
+git clone https://git.epicdynamic.com/auto-round-pipeline
+cd auto-round-pipeline
+
+# Pinned PyPI wheel (fastest; matches what this pipeline used by default):
+bash setup.sh --pip-version 0.13.1
+
+# Or build from intel/auto-round at the same tag (byte-identical reproducibility):
+bash setup.sh --source-tag v0.13.1
+
+# Intel Arc iGPU acceleration (e.g. Core Ultra 185H) — append to either of the above:
+#   ... --intel-xpu
+# NVIDIA / AMD opt-in: --cuda / --rocm
+```
+
+The script prints the resulting conda env name (something like `auto-round-pipeline-v0.13.1[-src][-xpu|-cuda|-rocm]`) at the end.
+
+### Step 2 — Quantize just this model
+
+Activate the env that `setup.sh` created, then invoke the runner with the same job filters that produced this artifact:
+
+```
+conda activate <env-name-printed-by-setup.sh>
+python runner.py \
+    --model 'tiiuae/Falcon3-3B-Instruct' \
+    --quant 'INT4-gs64' \
+    --format auto_awq \
+    --no-upload      # drop this to also push to HuggingFace Hub
+```
+
+### Step 3 — (Optional) standalone Python recipe
+
+If you'd rather call auto-round directly without the orchestration wrapper, this is the exact call the pipeline made:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from auto_round import AutoRound
+
+model_name = "tiiuae/Falcon3-3B-Instruct"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+bits, group_size, sym = 4, 64, False
+autoround = AutoRound(
+    model, tokenizer,
+    bits=bits, group_size=group_size, sym=sym,
+    device_map="cpu",
+    nsamples=128, iters=200, seqlen=512, batch_size=4,
+)
+autoround.quantize_and_save("./AutoRound/tiiuae_Falcon3-3B-Instruct-auto_awq-int4-gs64-asym", format="auto_awq")
+```
+
+## Actual Run Conditions
+
+Recorded by the auto-round-pipeline at quantization time:
+
+| Field | Value |
+|---|---|
+| Intel auto-round version | 0.13.1 |
+| transformers version | 4.55.3 |
+| torch version | 2.12.1+cpu |
+| torch_dtype (load) | torch.bfloat16 |
+| calibration device | `cpu` |
+| calibration samples | 128 |
+| tuning iterations | 200 |
+| calibration seq len | 512 |
+| calibration batch size | 4 |
+| quantization duration | 19986.7s (333.1 min) |
+| completed at (UTC) | 2026-07-02T11:32:27.433950+00:00 |
+
+
+## License
+
+[Apache 2.0 License](https://choosealicense.com/licenses/apache-2.0/)
+
+## Disclaimer
+
+This quantized model comes with no warranty. It has been developed only for research purposes.

chat_template.jinja

@@ -0,0 +1,45 @@
+{%- if tools %}
+{{- '<|system|>\n' }}
+{%- if messages[0]['role'] == 'system' %}
+{{- messages[0]['content'] }}
+{%- set remaining_messages = messages[1:] %}
+{%- else %}
+{%- set remaining_messages = messages %}
+{%- endif %}
+{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\n\n# Tools\n\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\n\n<tools>' + tools|tojson(indent=2) + '</tools>\n\n# Output Format\n\nYour response MUST follow this format when making function calls:\n<tool_call>\n[\n  {"name": "function_name", "arguments": {"arg1": "value1", "arg2": "value2"}},\n  {"name": "another_function", "arguments": {"arg": "value"}}\n]\n</tool_call>\nIf no function calls are needed, respond normally without the tool_call tags.\n' }}
+{%- for message in remaining_messages %}
+{%- if message['role'] == 'user' %}
+{{- '<|user|>\n' + message['content'] + '\n' }}
+{%- elif message['role'] == 'assistant' %}
+{%- if message.content %}
+{{- '<|assistant|>\n' + message['content'] }}
+{%- endif %}
+{%- if message.tool_calls %}
+{{- '\n<tool_call>\n' }}
+{{- message.tool_calls|tojson(indent=2) }}
+{{- '\n</tool_call>' }}
+{%- endif %}
+{{- eos_token + '\n' }}
+{%- elif message['role'] == 'tool' %}
+{{- '<|assistant|>\n<tool_response>\n' + message['content'] + '\n</tool_response>\n' }}
+{%- endif %}
+{%- endfor %}
+{{- '<|assistant|>\n' if add_generation_prompt }}
+{%- else %}
+{%- for message in messages %}
+{%- if message['role'] == 'system' %}
+{{- '<|system|>\n' + message['content'] + '\n' }}
+{%- elif message['role'] == 'user' %}
+{{- '<|user|>\n' + message['content'] + '\n' }}
+{%- elif message['role'] == 'assistant' %}
+{%- if not loop.last %}
+{{- '<|assistant|>\n' + message['content'] + eos_token + '\n' }}
+{%- else %}
+{{- '<|assistant|>\n' + message['content'] + eos_token }}
+{%- endif %}
+{%- endif %}
+{%- if loop.last and add_generation_prompt %}
+{{- '<|assistant|>\n' }}
+{%- endif %}
+{%- endfor %}
+{%- endif %}

config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 11,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 3072,
+  "initializer_range": 0.02,
+  "intermediate_size": 9216,
+  "max_position_embeddings": 32768,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "num_key_value_heads": 4,
+  "pretraining_tp": 1,
+  "quantization_config": {
+    "amp": false,
+    "autoround_version": "0.13.1",
+    "batch_size": 4,
+    "bits": 4,
+    "data_type": "int",
+    "group_size": 64,
+    "modules_to_not_convert": [
+      "lm_head"
+    ],
+    "provider": "auto-round",
+    "quant_method": "awq",
+    "seqlen": 512,
+    "sym": false,
+    "to_quant_block_names": "model.layers",
+    "version": "gemm",
+    "zero_point": true
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000042,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.55.3",
+  "use_cache": true,
+  "vocab_size": 131072
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 11,
+  "eos_token_id": 11,
+  "transformers_version": "4.55.3"
+}

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3662a5f525f8121fbce2509687290a2be9a5ae1a053494748b6dbea2b0eca4ed
+size 1073599848

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ac43be3ac16183cb853ec727e544ff46138ae71ee6040b07c9a8d324ab7ef97
+size 232720136

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1340a022f29c4f15cbb239608baa7fe70a3d4388b65f55f9c3c2ac393c227860
+size 1610612848

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf6ef0f538e1e729eb1c4ed6c0408604db2a61a690d7323438dce14b52a5ceed
+size 12376

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd1aa9509eab4f8a3ab7574a7d78ffb8863bbaee02503150f91328c0a40216d0
+size 1610612832

model.safetensors.index.json

@@ -0,0 +1,519 @@
+{
+  "metadata": {
+    "format": "safetensors",
+    "total_shards": 5,
+    "total_parameters": 1150798848,
+    "total_size": 4527501312
+  },
+  "weight_map": {
+    "model.layers.0.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.0.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.1.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.2.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.3.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.4.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.5.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.6.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.7.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.8.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.9.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.10.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.11.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.12.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.13.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.14.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.15.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.16.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.q_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.k_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.k_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.k_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.v_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.v_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.v_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.o_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.o_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.self_attn.o_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.gate_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.gate_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.gate_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.up_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.up_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.up_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.down_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.down_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.17.mlp.down_proj.scales": "model-00001-of-00005.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00005.safetensors",
+    "model.layers.18.self_attn.q_proj.qweight": "model-00001-of-00005.safetensors",
+    "model.layers.18.self_attn.q_proj.qzeros": "model-00001-of-00005.safetensors",
+    "model.layers.18.self_attn.q_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.k_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.k_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.k_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.v_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.v_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.v_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.o_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.o_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.18.self_attn.o_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.gate_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.gate_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.gate_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.up_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.up_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.up_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.down_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.down_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.18.mlp.down_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.q_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.q_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.q_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.k_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.k_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.k_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.v_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.v_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.v_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.o_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.o_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.self_attn.o_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.gate_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.gate_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.gate_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.up_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.up_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.up_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.down_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.down_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.19.mlp.down_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.q_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.q_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.q_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.k_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.k_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.k_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.v_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.v_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.v_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.o_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.o_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.self_attn.o_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.gate_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.gate_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.gate_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.up_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.up_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.up_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.down_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.down_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.20.mlp.down_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.q_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.q_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.q_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.k_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.k_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.k_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.v_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.v_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.v_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.o_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.o_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.self_attn.o_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.gate_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.gate_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.gate_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.up_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.up_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.up_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.down_proj.qweight": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.down_proj.qzeros": "model-00002-of-00005.safetensors",
+    "model.layers.21.mlp.down_proj.scales": "model-00002-of-00005.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00002-of-00005.safetensors",
+    "model.embed_tokens.weight": "model-00003-of-00005.safetensors",
+    "model.norm.weight": "model-00004-of-00005.safetensors",
+    "lm_head.weight": "model-00005-of-00005.safetensors"
+  }
+}

quantization_config.json

@@ -0,0 +1,18 @@
+{
+  "bits": 4,
+  "data_type": "int",
+  "group_size": 64,
+  "sym": false,
+  "amp": false,
+  "batch_size": 4,
+  "seqlen": 512,
+  "to_quant_block_names": "model.layers",
+  "autoround_version": "0.13.1",
+  "provider": "auto-round",
+  "quant_method": "awq",
+  "zero_point": true,
+  "version": "gemm",
+  "modules_to_not_convert": [
+    "lm_head"
+  ]
+}

special_tokens_map.json

@@ -0,0 +1,41 @@
+{
+  "additional_special_tokens": [
+    ">>TITLE<<",
+    ">>ABSTRACT<<",
+    ">>INTRODUCTION<<",
+    ">>SUMMARY<<",
+    ">>COMMENT<<",
+    ">>ANSWER<<",
+    ">>QUESTION<<",
+    ">>DOMAIN<<",
+    ">>EMAIL_ADDRESS<<",
+    ">>IP_ADDRESS<<",
+    "<|startoftext|>",
+    ">>IP_ADDRESS_0<<",
+    ">>IP_ADDRESS_1<<",
+    ">>IP_ADDRESS_2<<",
+    ">>IP_ADDRESS_3<<",
+    ">>IP_ADDRESS_4<<",
+    ">>IP_ADDRESS_5<<",
+    ">>IP_ADDRESS_6<<",
+    ">>IP_ADDRESS_7<<",
+    ">>IP_ADDRESS_8<<",
+    ">>IP_ADDRESS_9<<",
+    ">>PASSWORD<<",
+    ">>KEY<<"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}