File size: 2,722 Bytes
30d08d5 91dcc06 30d08d5 91dcc06 30d08d5 91dcc06 30d08d5 91dcc06 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
license: llama3
language:
- sv
- da
- "no"
- en
base_model: AI-Sweden-Models/Llama-3-8B-instruct
base_model_relation: quantized
library_name: exllamav2
tags:
- exl2
- llama
- swedish
- quantized
---
# AI-Sweden Llama-3-8B-instruct — EXL2
ExLlamaV2 / EXL2 quants of [AI-Sweden-Models/Llama-3-8B-instruct](https://huggingface.co/AI-Sweden-Models/Llama-3-8B-instruct).
That checkpoint is a Swedish/Nordic instruct tune of Llama 3 8B. Official Hub files are BF16 (and a separate GGUF); there were no EXL2 uploads.
Converted with **ExLlamaV2 0.3.2**, `lm_head` at 6-bit, **built-in default calibration** (same recipe as turboderp-style quants). One measurement pass, then each bitrate from `measurement.json`.
## Branches
| Revision | Target bpw | Size (approx.) |
|---|---|---|
| [`4.0bpw`](../../tree/4.0bpw) | 4.0 | 4.7 GB |
| [`4.5bpw`](../../tree/4.5bpw) | 4.5 | 5.1 GB |
| [`5.0bpw`](../../tree/5.0bpw) | 5.0 | 5.5 GB |
| [`5.5bpw`](../../tree/5.5bpw) | 5.5 | 5.9 GB |
| [`6.0bpw`](../../tree/6.0bpw) | 6.0 | 6.3 GB |
[`measurement.json`](../../blob/main/measurement.json) is on `main` if you want to roll another bitrate yourself.
Download a revision:
```bash
hf download oxfrug/Llama-3-8B-instruct-exl2 --revision 5.0bpw --local-dir ./Llama-3-8B-instruct-exl2-5.0bpw
```
TabbyAPI / ExUI: point the model path at a checked-out branch, or set the HF revision to `5.0bpw`.
## Notes
- **License:** [Meta Llama 3 Community License](https://llama.meta.com/llama3/license). Keep the `NOTICE` file. This is a derivative of Meta Llama 3 via AI Sweden’s instruct tune.
- **Loader:** ExLlamaV2 (TabbyAPI, text-generation-webui `exllamav2`, ExUI). Not GGUF / llama.cpp.
- On **PyTorch 2.13** without Flash Attention 2.5.7+, set `config.no_sdpa = True` before load. Default SDPA + `causal_lower_right` produced collapsed output in our tests; the explicit matmul path matched BF16 (6.0bpw raw-EN first 16 tokens identical). Older torch + flash-attn (paged attention) is the usual community stack and was not the convert path.
- These are **not** the experimental Swedish-heavy calibration files. Default EXL2 cal is what other EXL2 repos ship.
## Source
```
AI-Sweden-Models/Llama-3-8B-instruct
← AI-Sweden-Models/Llama-3-8B
← meta-llama/Meta-Llama-3-8B
```
Convert: ExLlamaV2 `convert.py` 0.3.2, RTX 3090.
## Smoke (greedy, `no_sdpa=True`)
6.0bpw, same chat template as the BF16 card:
| Prompt | Output |
|---|---|
| `Vad heter Sveriges huvudstad? Ett ord.` | `Huvudstaden i Sverige är Stockholm.` |
| `Name the capital of Sweden in one word.` | `Stockholm` |
4.0bpw is coherent on the same prompts (Swedish-first on the English question, like the BF16 base). This is not a leaderboard score.
|