Card on 4.5bpw

README.md

@@ -1,70 +1,71 @@
 ---
+license: llama3
 language:
 - sv
 - da
-- 'no'
-license: llama3
-base_model: AI-Sweden-Models/Llama-3-8B
+- "no"
+- en
+base_model: AI-Sweden-Models/Llama-3-8B-instruct
+base_model_relation: quantized
+library_name: exllamav2
+tags:
+- exl2
+- llama
+- swedish
+- quantized
 ---
 
-# Checkpoint 1
-
-## Training setup
-The training was perfomed on the [LUMI supercomputer](https://lumi-supercomputer.eu/) within the [DeployAI EU project](https://www.ai.se/en/project/deployai). 
-Based of the base model [AI-Sweden-Models/Llama-3-8B](https://huggingface.co/AI-Sweden-Models/Llama-3-8B).
-
-## How to use
-
-```python
-import transformers
-import torch
-
-model_id = "AI-Sweden-Models/Llama-3-8B-instruct"
-
-pipeline = transformers.pipeline(
-    "text-generation",
-    model=model_id,
-    model_kwargs={"torch_dtype": torch.bfloat16},
-    device_map="auto",
-)
-
-messages = [
-    {"role": "system", "content": "Du är en hjälpsam assistant som svarar klokt och vänligt."},
-    {"role": "user", "content": "Hur gör man pannkakor? Och vad behöver man handla? Undrar också vad 5+6 är.."},
-]
-
-terminators = [
-    pipeline.tokenizer.eos_token_id,
-    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
-]
-
-outputs = pipeline(
-    messages,
-    max_new_tokens=256,
-    eos_token_id=terminators,
-    do_sample=True,
-    temperature=0.6,
-    top_p=0.9,
-)
-print(outputs[0]["generated_text"][-1])
+# AI-Sweden Llama-3-8B-instruct — EXL2
+
+ExLlamaV2 / EXL2 quants of [AI-Sweden-Models/Llama-3-8B-instruct](https://huggingface.co/AI-Sweden-Models/Llama-3-8B-instruct).  
+That checkpoint is a Swedish/Nordic instruct tune of Llama 3 8B. Official Hub files are BF16 (and a separate GGUF); there were no EXL2 uploads.
+
+Converted with **ExLlamaV2 0.3.2**, `lm_head` at 6-bit, **built-in default calibration** (same recipe as turboderp-style quants). One measurement pass, then each bitrate from `measurement.json`.
+
+## Branches
+
+| Revision | Target bpw | Size (approx.) |
+|---|---|---|
+| [`4.0bpw`](../../tree/4.0bpw) | 4.0 | 4.7 GB |
+| [`4.5bpw`](../../tree/4.5bpw) | 4.5 | 5.1 GB |
+| [`5.0bpw`](../../tree/5.0bpw) | 5.0 | 5.5 GB |
+| [`5.5bpw`](../../tree/5.5bpw) | 5.5 | 5.9 GB |
+| [`6.0bpw`](../../tree/6.0bpw) | 6.0 | 6.3 GB |
+
+[`measurement.json`](../../blob/main/measurement.json) is on `main` if you want to roll another bitrate yourself.
+
+Download a revision:
+
+```bash
+hf download oxfrug/Llama-3-8B-instruct-exl2 --revision 5.0bpw --local-dir ./Llama-3-8B-instruct-exl2-5.0bpw
 ```
-```python
->>> "För att göra pannkakor behöver du följande ingredienser:
-- 1 kopp vetemjöl
-- 1 tesked bakpulver
-- 1/4 tesked salt
-- 1 kopp mjölk
-- 1 stort ägg
-- 2 matskedar smält smör eller olja
-
-För att börja, blanda vetemjölet, bakpulvret och saltet i en bunke. I en annan skål, vispa ihop mjölken, ägget och smöret eller oljan. 
-Tillsätt de våta ingredienserna till de torra ingredienserna och blanda tills det är väl blandat. 
-Låt smeten vila i cirka 10 minuter.
-
-För att göra pannkakorna, värm en non-stick-panna eller stekpanna över medelvärme. 
-När den är varm, häll smeten på pannan och grädda tills kanterna börjar torka ut och toppen är fast. 
-Vänd pannkakan med en stekspade och grädda den andra sidan tills den är gyllenbrun. 
-Upprepa med resten av smeten.
-
-När det gäller 5+6 är svaret 11."
-```
+
+TabbyAPI / ExUI: point the model path at a checked-out branch, or set the HF revision to `5.0bpw`.
+
+## Notes
+
+- **License:** [Meta Llama 3 Community License](https://llama.meta.com/llama3/license). Keep the `NOTICE` file. This is a derivative of Meta Llama 3 via AI Sweden’s instruct tune.
+- **Loader:** ExLlamaV2 (TabbyAPI, text-generation-webui `exllamav2`, ExUI). Not GGUF / llama.cpp.
+- On **PyTorch 2.13** without Flash Attention 2.5.7+, set `config.no_sdpa = True` before load. Default SDPA + `causal_lower_right` produced collapsed output in our tests; the explicit matmul path matched BF16 (6.0bpw raw-EN first 16 tokens identical). Older torch + flash-attn (paged attention) is the usual community stack and was not the convert path.
+- These are **not** the experimental Swedish-heavy calibration files. Default EXL2 cal is what other EXL2 repos ship.
+
+## Source
+
+```
+AI-Sweden-Models/Llama-3-8B-instruct
+  ← AI-Sweden-Models/Llama-3-8B
+    ← meta-llama/Meta-Llama-3-8B
+```
+
+Convert: ExLlamaV2 `convert.py` 0.3.2, RTX 3090.
+
+## Smoke (greedy, `no_sdpa=True`)
+
+6.0bpw, same chat template as the BF16 card:
+
+| Prompt | Output |
+|---|---|
+| `Vad heter Sveriges huvudstad? Ett ord.` | `Huvudstaden i Sverige är Stockholm.` |
+| `Name the capital of Sweden in one word.` | `Stockholm` |
+
+4.0bpw is coherent on the same prompts (Swedish-first on the English question, like the BF16 base). This is not a leaderboard score.