Instructions to use DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2")
model = AutoModelForCausalLM.from_pretrained("DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2

SGLang

How to use DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2 with Docker Model Runner:
```
docker model run hf.co/DrNicefellow/Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2
```

Qwen2.5-Coder-32B-Instruct-4.25bpw-exl2

File size: 1,547 Bytes

bc2b5a2

url: https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct
branch: main
download date: 2024-11-12 05:14:04
sha256sum:
    5746929d771eb06ba0105671a2018dca3c648d4664ca5a5ce2077df82fabad92 model-00001-of-00014.safetensors
    12aaad653a85d7212a9cc7722937fff7b2bd060ab8522399b4a74d2bf736df8d model-00002-of-00014.safetensors
    9bac433064b2c29f3e2a891539993b74b7a1da5b0a78c900b2e22fc20fe66059 model-00003-of-00014.safetensors
    4e859e864a61c45dc2e38650359540e5f35f09d4a6993ef66d7274760e7d1905 model-00004-of-00014.safetensors
    57000579cf2e87cd575333288c25588a45816fc406ace74a52efed1e7bc910d5 model-00005-of-00014.safetensors
    72dd80465092fad2665c9705332efa99c20b9fb22ad1944e7fe7429a222ede42 model-00006-of-00014.safetensors
    b91130f8049d3446221000a1d44cdf7bd344f01a2b5843f9bb81ac8193beeae8 model-00007-of-00014.safetensors
    522db6c67390557f2964a4a0094a463a067c91992be164b6e141aa1267a225e5 model-00008-of-00014.safetensors
    7c03b072b4aad4da68016b75b78c6eb3ea6b35e18ac0dae4906146603fa32e23 model-00009-of-00014.safetensors
    c5ee9a4665ae7d5c9e6d37f77d93c47244a80ee41284060efde69d223472a87a model-00010-of-00014.safetensors
    49f60205e190369145352de972e3510bcfeef6b5a26041573032edbe132c1e06 model-00011-of-00014.safetensors
    9a53a0d4a02dcb5f036da4fcd50e408107144ff4c4b09357cf1829be2ef382f9 model-00012-of-00014.safetensors
    8b26127defc68bff2e0744ae9afc9c8ab1730fee071e0400b4bfe8f07fadda5b model-00013-of-00014.safetensors
    983ebbded19c19264d08845710c9993714d60e17f81c62e1e15c8f174af79847 model-00014-of-00014.safetensors