Instructions to use RedHatAI/granite-3.1-2b-base-quantized.w8a8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/granite-3.1-2b-base-quantized.w8a8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/granite-3.1-2b-base-quantized.w8a8")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/granite-3.1-2b-base-quantized.w8a8")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/granite-3.1-2b-base-quantized.w8a8", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/granite-3.1-2b-base-quantized.w8a8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/granite-3.1-2b-base-quantized.w8a8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-base-quantized.w8a8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RedHatAI/granite-3.1-2b-base-quantized.w8a8

SGLang

How to use RedHatAI/granite-3.1-2b-base-quantized.w8a8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/granite-3.1-2b-base-quantized.w8a8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-base-quantized.w8a8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/granite-3.1-2b-base-quantized.w8a8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-base-quantized.w8a8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RedHatAI/granite-3.1-2b-base-quantized.w8a8 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/granite-3.1-2b-base-quantized.w8a8
```

granite-3.1-2b-base-quantized.w8a8

Commit History

Update README.md

1413be9
verified

shubhrapandit commited on Feb 28, 2025

Update README.md

4032746
verified

nm-research commited on Jan 30, 2025

Update README.md

b89cdac
verified

nm-research commited on Jan 30, 2025

Update README.md

1541725
verified

nm-research commited on Jan 30, 2025

Update README.md

7ec5743
verified

shubhrapandit commited on Jan 28, 2025

Update README.md

9326acd
verified

nm-research commited on Jan 28, 2025

Update README.md

c9bad45
verified

nm-research commited on Jan 25, 2025

Update README.md

0cd5367
verified

nm-research commited on Jan 24, 2025

Update README.md

6268ee6
verified

nm-research commited on Jan 20, 2025

Update README.md

8203e2f
verified

nm-research commited on Jan 17, 2025

Create README.md

96f22b7
verified

nm-research commited on Jan 16, 2025

Upload model files

d4bbdf0

Shubhra Pandit commited on Jan 16, 2025

initial commit

df6e349
verified

nm-research commited on Jan 16, 2025

Commit History

Update README.md 1413be9 verified

Update README.md 4032746 verified

Update README.md b89cdac verified

Update README.md 1541725 verified

Update README.md 7ec5743 verified

Update README.md 9326acd verified

Update README.md c9bad45 verified

Update README.md 0cd5367 verified

Update README.md 6268ee6 verified

Update README.md 8203e2f verified

Create README.md 96f22b7 verified

Upload model files d4bbdf0

initial commit df6e349 verified

Update README.md

1413be9
verified

Update README.md

4032746
verified

Update README.md

b89cdac
verified

Update README.md

1541725
verified

Update README.md

7ec5743
verified

Update README.md

9326acd
verified

Update README.md

c9bad45
verified

Update README.md

0cd5367
verified

Update README.md

6268ee6
verified

Update README.md

8203e2f
verified

Create README.md

96f22b7
verified

Upload model files

d4bbdf0

initial commit

df6e349
verified