KronQ W4A16 packed int4 (L2-13B)

README.md

@@ -0,0 +1,44 @@
+---
+base_model: meta-llama/Llama-2-13b-hf
+license: llama2
+tags:
+  - kronq
+  - quantization
+  - int4
+  - weight-only
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# Llama-2-13b — KronQ W4A16 (packed int4)
+
+[Llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b-hf) quantized to **4-bit weights / 16-bit activations** with **KronQ** (Kronecker-factored Hessian quantization). Weights are stored **packed int4** (~6.9 GB vs 26 GB fp16); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel unpacks them on the fly.
+
+## Results (WikiText-2, seqlen 2048)
+
+**Perplexity:** **4.96**
+
+**Zero-shot accuracy:**
+
+| PIQA | HellaSwag | ARC-E | ARC-C | WinoGrande | BoolQ | Average |
+|---|---|---|---|---|---|---|
+| 80.20 | 79.14 | 74.71 | 47.95 | 71.82 | 82.14 | **72.66** |
+
+(lm-evaluation-harness, 0-shot. `acc_norm` for PIQA/HellaSwag/ARC, `acc` for WinoGrande/BoolQ.)
+
+## Usage
+
+KronQ-packed checkpoint (`model.safetensors` carries `biip_w_codes`/scale/zero + BiIP buffers, see `kronq_packed_config.json`). Load with the KronQ runtime:
+
+```bash
+python eval_pretrained.py meta-llama/Llama-2-13b-hf donghyunli/Llama-2-13b-KronQ-W4A16 --ppl --zs
+```
+
+## Recipe
+
+Per-channel asymmetric W4, weight-only (a_bits=16), `--alpha 0.25`, bidirectional incoherence processing (BiIP), `act_order`, raw H_G. Calibrated on 128 WikiText-2 sequences.
+
+## License
+
+Derivative of Llama-2-13b — subject to the [Llama 2 Community License](https://ai.meta.com/llama/license/).

kronq_packed_config.json

@@ -0,0 +1,5 @@
+{
+  "format": "kronq-packed",
+  "bits": 4,
+  "n_compressed": 280
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:826580592e5965f924b76aab5d1a037f768d4cb24f2a4090df7defa0953f36e7
+size 7352355120

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,35 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}