Llama-3.1-8B GPTQ 4-bit (group size 128)
Built with Llama.
GPTQ-quantized meta-llama/Llama-3.1-8B, produced with AutoGPTQ 0.7.1.
Quantization settings
| bits | 4 |
| group_size | 128 |
| damp_percent | 0.01 |
| desc_act | false |
| sym | true |
| true_sequential | true |
| static_groups | false |
| format | safetensors, fp16 |
Calibration
- Dataset:
wikitext/wikitext-2-raw-v1, train split - 128 samples, sequence length 2048, seed 0
- Sampling follows
examples/quantization/basic_usage_wikitext2.pyfrom AutoGPTQ
Usage
from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer
repo = "fmlini251/Llama-3.1-8B-GPTQ-4bit-128g"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoGPTQForCausalLM.from_quantized(repo, device="cuda:0", use_triton=False)
License
Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
See LICENSE and USE_POLICY.md in this repository.
- Downloads last month
- 34
Model tree for fmlini251/Llama-3.1-8B-GPTQ-4bit-128g
Base model
meta-llama/Llama-3.1-8B