Instructions to use TheBloke/WizardLM-30B-Uncensored-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/WizardLM-30B-Uncensored-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/WizardLM-30B-Uncensored-GPTQ")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TheBloke/WizardLM-30B-Uncensored-GPTQ")
model = AutoModelForCausalLM.from_pretrained("TheBloke/WizardLM-30B-Uncensored-GPTQ", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TheBloke/WizardLM-30B-Uncensored-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/WizardLM-30B-Uncensored-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardLM-30B-Uncensored-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/WizardLM-30B-Uncensored-GPTQ

SGLang

How to use TheBloke/WizardLM-30B-Uncensored-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/WizardLM-30B-Uncensored-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardLM-30B-Uncensored-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/WizardLM-30B-Uncensored-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/WizardLM-30B-Uncensored-GPTQ",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TheBloke/WizardLM-30B-Uncensored-GPTQ with Docker Model Runner:
```
docker model run hf.co/TheBloke/WizardLM-30B-Uncensored-GPTQ
```

Model exits in Windows Booga

by 0spr4y - opened May 24, 2023

Discussion

0spr4y

May 24, 2023

I have a 4090 and 64gb of CPU ram. Upon loading this model in the
text-generation-webui, the webui exits with

Loading model ...

Done!
Press any key to continue . . .

Has anyone found a solution to this on windows.

crainto

May 24, 2023

Same

0spr4y

May 24, 2023

So turns out on windows you need alot of Pagefile... like 90GB of page file. After setting this the model loads -_-.

TheBloke

Owner May 24, 2023

Ahh yeah you're running out of RAM and it can't load it onto the GPU. It's annoying it doesn't print a better error message.

Thanks for posting the solution! Lots of people are having this issue and I couldn't figure out what was wrong.

crainto

May 24, 2023

So turns out on windows you need alot of Pagefile... like 90GB of page file. After setting this the model loads -_-.

Do you mean increasing pagefile on the system disk C? And which drive is your model located?

0spr4y

May 24, 2023

I have a second NVM SSD which I used because 90GB is alot. Win + R "sysdm.cpl" -> Advanced -> Performance Settings -> Advanced -> Virtual Memory. Then set a fast SSD to have a min and max pagefile of 94000 save and reboot and try to load the model again.

crainto

May 24, 2023

I have a second NVM SSD which I used because 90GB is alot. Win + R "sysdm.cpl" -> Advanced -> Performance Settings -> Advanced -> Virtual Memory. Then set a fast SSD to have a min and max pagefile of 94000 save and reboot and try to load the model again.

Yep, it works, thank you

Itwasthellamas78

May 24, 2023

Can confirm. Was puzzled for several days why the model was not loading. Changing the page file as suggested resolved the issue. Thank you so much!

Wildstar50

May 25, 2023

•

edited May 25, 2023

Interesting, this worked for me the first time. 4090 / 64GB of CPU memory, Virtual Memory size at default setting of 36GB. Windows 11. Perhaps just the combined CPU + VM needs to exceed 90GB?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment