Instructions to use TheBloke/Llama-2-70B-Chat-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/Llama-2-70B-Chat-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/Llama-2-70B-Chat-GPTQ")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-70B-Chat-GPTQ")
model = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-70B-Chat-GPTQ", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheBloke/Llama-2-70B-Chat-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/Llama-2-70B-Chat-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Llama-2-70B-Chat-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/Llama-2-70B-Chat-GPTQ

SGLang

How to use TheBloke/Llama-2-70B-Chat-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/Llama-2-70B-Chat-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Llama-2-70B-Chat-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/Llama-2-70B-Chat-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Llama-2-70B-Chat-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TheBloke/Llama-2-70B-Chat-GPTQ with Docker Model Runner:
```
docker model run hf.co/TheBloke/Llama-2-70B-Chat-GPTQ
```

update to latest transformers and exllama, still loading fail

#17

by yiouyou - opened Jul 23, 2023

Discussion

yiouyou

Jul 23, 2023

•

edited Jul 23, 2023

update the transformers to 4.32.0.dev0, exllama to 0.0.8+cu117, still get errors when loading with text-generation-webui as below:

Traceback (most recent call last): File “/home/abc/text-generation-webui/server.py”, line 68, in load_model_wrapper shared.model, shared.tokenizer = load_model(shared.model_name, loader) File “/home/abc/text-generation-webui/modules/models.py”, line 79, in load_model output = load_func_maploader File “/home/abc/text-generation-webui/modules/models.py”, line 320, in ExLlama_loader model, tokenizer = ExllamaModel.from_pretrained(model_name) File “/home/abc/text-generation-webui/modules/exllama.py”, line 49, in from_pretrained config = ExLlamaConfig(str(model_config_path)) File “/home/abc/miniconda3/envs/textgen/lib/python3.10/site-packages/exllama/model.py”, line 46, in init read_config = json.load(f) File “/home/abc/miniconda3/envs/textgen/lib/python3.10/json/init.py”, line 293, in load return loads(fp.read(), File “/home/abc/miniconda3/envs/textgen/lib/python3.10/json/init.py”, line 346, in loads return _default_decoder.decode(s) File “/home/abc/miniconda3/envs/textgen/lib/python3.10/json/decoder.py”, line 340, in decode raise JSONDecodeError(“Extra data”, s, end) json.decoder.JSONDecodeError: Extra data: line 25 column 5 (char 632)

Need some help to get it running, Thanks for help!

yiouyou changed discussion title from update the transformers to 4.32.0.dev0, exllama to 0.0.8+cu117, still get errors when loading with text-generation-webui to update to latest transformers and exllama, still get errors when loading with text-generation-webui Jul 23, 2023

yiouyou changed discussion title from update to latest transformers and exllama, still get errors when loading with text-generation-webui to update to latest transformers and exllama, text-generation-webui still loading fail Jul 23, 2023

yiouyou changed discussion title from update to latest transformers and exllama, text-generation-webui still loading fail to update to latest transformers and exllama, still loading fail Jul 23, 2023

TheBloke

Owner Jul 23, 2023

•

edited Jul 23, 2023

Hmm, it seems like it's failing on reading config.json.

The files are definitely fine, I tested this model with ExLlama earlier today actually.

Can you try triggering a download of the repo again, in case your files have got corrupted or one file is missing or something like that.

yiouyou

Jul 24, 2023

Thanks for your help! After re-download all small files, it's loaded.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment