Instructions to use petra312/GLM-4.6-Derestricted-v3-NF4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use petra312/GLM-4.6-Derestricted-v3-NF4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="petra312/GLM-4.6-Derestricted-v3-NF4", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("petra312/GLM-4.6-Derestricted-v3-NF4", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("petra312/GLM-4.6-Derestricted-v3-NF4", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use petra312/GLM-4.6-Derestricted-v3-NF4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "petra312/GLM-4.6-Derestricted-v3-NF4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "petra312/GLM-4.6-Derestricted-v3-NF4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/petra312/GLM-4.6-Derestricted-v3-NF4

SGLang

How to use petra312/GLM-4.6-Derestricted-v3-NF4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "petra312/GLM-4.6-Derestricted-v3-NF4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "petra312/GLM-4.6-Derestricted-v3-NF4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "petra312/GLM-4.6-Derestricted-v3-NF4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "petra312/GLM-4.6-Derestricted-v3-NF4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use petra312/GLM-4.6-Derestricted-v3-NF4 with Docker Model Runner:
```
docker model run hf.co/petra312/GLM-4.6-Derestricted-v3-NF4
```

GLM-4.6-Derestricted-v3 NF4 (bitsandbytes)

4-bit NF4 / double-quant conversion of ArliAI/GLM-4.6-Derestricted-v3 (GLM-4.6, 92 layers, 160 routed experts).

Intended layout on a single 24–32 GB GPU + lots of CPU RAM (Unsloth / llama.cpp MoE-offload style):

GPU: embeddings, attention, dense MLPs (first 3 layers), shared experts, router, lm_head
CPU RAM: routed expert NF4 weights (~160 experts × 89 MoE layers)

Experts are not fused 3D gate_up_proj tensors. They stay as per-expert gate_proj / up_proj / down_proj so bitsandbytes can NF4 them. Use transformers 4.57.x, not 5.x (5.x fuses GLM experts and bnb cannot quantize the 3D parameters).

Load (dense GPU, experts CPU)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

model_id = "petra312/GLM-4.6-Derestricted-v3-NF4"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)

device_map = {
    "model.embed_tokens": 0,
    "model.norm": 0,
    "model.rotary_emb": 0,
    "lm_head": 0,
}
for i in range(config.num_hidden_layers):
    device_map[f"model.layers.{i}.self_attn"] = 0
    device_map[f"model.layers.{i}.input_layernorm"] = 0
    device_map[f"model.layers.{i}.post_attention_layernorm"] = 0
    if i < config.first_k_dense_replace:
        device_map[f"model.layers.{i}.mlp"] = 0
    else:
        device_map[f"model.layers.{i}.mlp.gate"] = 0
        device_map[f"model.layers.{i}.mlp.shared_experts"] = 0
        device_map[f"model.layers.{i}.mlp.experts"] = "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map=device_map,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)

Requires bitsandbytes, accelerate, transformers==4.57.6.

Notes

MTP layer 92 is omitted (not used by Glm4MoeForCausalLM).
modeling_glm4_moe.py in this repo returns router logits so REAP-style expert observers still work.
Original BF16 weights are unchanged on the source model card; this repo is a quantized derivative.

Downloads last month: 260

Safetensors

Model size

353B params

Tensor type

BF16

F32

Model tree for petra312/GLM-4.6-Derestricted-v3-NF4

Base model

zai-org/GLM-4.6

Finetuned

ArliAI/GLM-4.6-Derestricted-v3

Quantized

(5)

this model