Instructions to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp")
model = AutoModelForCausalLM.from_pretrained("newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp

SGLang

How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp with Docker Model Runner:
```
docker model run hf.co/newmindai/Llama-3.1-8B-Instruct_w16a8_rw_with_gw_hp
```

nmmursit commited on Nov 12, 2025

Commit

dc76f2b

verified ·

1 Parent(s): 1faa658

Update README.md

Browse files

Files changed (1) hide show

README.md +7 -6

README.md CHANGED Viewed

@@ -118,12 +118,13 @@ if use_fp8:
 |---------|---------|---------|---------|---------|
 |![W&B Chart 11_6_2025, 1_28_06 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/DOvLRT82pszUOYStDOBRG.png)| ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/27loZAUgU-V0gQ0jAZz4o.png) | ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/NX_Ni-U6spKd4R7PzBi8i.png)|![W&B Chart 11_6_2025, 1_24_01 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/YezrVMnv1CoJchKNhDA1V.png)| ![W&B Chart 11_6_2025, 1_25_41 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/qPkBYPC9rNKeyz0ONHmlO.png)|
-| Method               | Total Training Time | Avg GPU Utilization | Final Val Loss | Final Val Perplexity | Final Val Accuracy | Memory Allocated (avg %) |     **Speed vs Tensorwise**     |
-| :------------------- | :------------------ | :-----------------: | :------------: | :------------------: | :----------------: | :----------------------: | :-----------------------------: |
-| **w16a8 Tensorwise** | **123 min 50 s**    |        ≈ 98 %       |     0.8394     |        2.3151        |       0.7934       |        ~68 – 80 %        |                —                |
-| **w16a8 Rowwise**    | **115 min 25 s**    |        ≈ 99 %       |     0.8392     |        2.3144        |       0.7941       |        ~66 – 78 %        |                7%               |
 ## Implementation

 |---------|---------|---------|---------|---------|
 |![W&B Chart 11_6_2025, 1_28_06 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/DOvLRT82pszUOYStDOBRG.png)| ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/27loZAUgU-V0gQ0jAZz4o.png) | ![image](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/NX_Ni-U6spKd4R7PzBi8i.png)|![W&B Chart 11_6_2025, 1_24_01 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/YezrVMnv1CoJchKNhDA1V.png)| ![W&B Chart 11_6_2025, 1_25_41 PM](https://cdn-uploads.huggingface.co/production/uploads/683d4880e639f8d647355997/qPkBYPC9rNKeyz0ONHmlO.png)|
+| Model                                               | Training Time (mins) | Memory Allocated (avg %) | GPU Utilization (avg %) | Speed vs bf16 |
+| :-------------------------------------------------- | --------------------: | -----------------------: | -----------------------: | -------------: |
+| **Llama-3.1-8B-Instruct_w16a16**                    | 138.75267            | 74.4189                  | 56.6059%                 | _             |
+| **Llama-3.1-8B-Instruct-w16a8-tw**                  | 123.75267            | 68.8982                  | 97.5364%                 | 12.11%        |
+| **Llama-3.1-8B-Instruct_w16a8_rw**                  | 115.75364            | 69.6132                  | 97.7689%                 | 19.87%        |
+| **Llama-3.1-8B-Instruct_w16a8_rowwise_with_gw_hp**  | 109.00364            | 69.4806                  | 97.3312%                 | 27.33%        |
+| **Llama-3.1-8B-Instruct-w16a8-mxtw**                | 64.00328             | 68.8982                  | 95.5661%                 | 116.82%       |
 ## Implementation