| --- |
| quantized_by: LLMJapan |
| pipeline_tag: text-generation |
| license: cc-by-nc-4.0 |
| language: |
| - en |
| tags: |
| - nvidia |
| - AceInstruct |
| - code |
| - math |
| - general_domain |
| - instruct_model |
| base_model: nvidia/AceInstruct-72B |
| --- |
| ## Exllama v2 Quantizations of AceInstruct-72B by nvidia |
|
|
| Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.2.8">turboderp's ExLlamaV2 v0.2.8</a> for quantization. |
|
|
| Original model: https://huggingface.co/nvidia/AceInstruct-72B |
|
|
| Quantization Command Example for creating other bpw quantization |
| ``` |
| cd {your git clone directory} |
| python convert.py -i {path to}/AceInstruct-72B -o {path to}/AceInstruct-72B/workingdir -cf {path to}/AceInstruct-72B/AceInstruct-72B-3bpw -b 3.0 |
| ``` |
|
|
| ## Prompt format |
|
|
| ``` |
| <|im_start|>system |
| {system_prompt}<|im_end|> |
| <|im_start|>user |
| {prompt}<|im_end|> |
| <|im_start|>assistant |
| ``` |
|
|
| ## How to add your system prompt |
|
|
| Copy the following json and replace the "You are AceInstruct developed by NVIDIA. You are helpful assistant." sentence with your original system prompt. |
| The default tokenizer_config.json does not have system prompt. |
| |
| tokenizer_config.json |
| ``` |
| "chat_template": "{{- '<|im_start|>system\\nYou are AceInstruct developed by NVIDIA. You are helpful assistant.<|im_end|>\\n' }}\n {%- for message in messages %}\n{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}\n{%- endfor %}\n{%- if add_generation_prompt %}\n{{- '<|im_start|>assistant\n' }}\n{%- endif %}\n", |
| ``` |
|
|
| ## File information |
|
|
| | quantization type | file size | |
| | ----------------------- | ----------: | |
| | 3.0bpw | 27.8 GiB | |
|
|
| ## Benchmark Results |
|
|
| | | Qwen2.5-1.5B-Instruct | AceInstruct-1.5B | Qwen2.5-7B-Instruct | AceInstruct-7B | Qwen2.5-72B-Instruct | AceInstruct-72B | |
| | --------- |:-----:|:-----:|:-----:|:-----:|:-----:|:-----:| |
| | HumanEval | 61.60 | 73.17 | 84.80 | 85.37 | 86.60 | 89.63 | |
| | MBPP | 63.20 | 65.76 | 79.20 | 74.32 | 88.20 | 83.66 | |
| | GSM8K | 73.20 | 80.44 | 91.60 | 93.10 | 95.80 | 96.36 | |
| | MATH | 55.20 | 60.34 | 75.50 | 76.40 | 83.10 | 84.50 | |
| | MMLU | 58.37 | 58.17 | 74.51 | 74.68 | 84.67 | 83.88 | |
| | MMLU Pro | 32.40 | 33.78 | 56.30 | 54.50 | 71.10 | 66.10 | |
| | Average | 57.33 | 61.94 | 76.99 | 76.40 | 84.91 | 84.02 | |
|
|
| ## Credits |
|
|
| Thanks to NVIDIA team. |
|
|
| --- |
| license: cc-by-nc-4.0 |
| --- |
|
|
