Instructions to use kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid")
model = AutoModelForCausalLM.from_pretrained("kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid

SGLang

How to use kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid with Docker Model Runner:
```
docker model run hf.co/kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid
```

SuperGLM-5.2-abliterated — MXFP8 / NVFP4 / NF3 Hybrid

A single-node hybrid-quantized build of the abliterated GLM-5.2 (753B, MoE) that fits on 4×96 GB GPUs at tensor-parallel 4, serving the full 256-expert model at 262K context — weights preserved, refusals removed, reasoning intact.

~341 GB on disk, 92 safetensors shards. Runs where the BF16/NVFP4-only builds need 8 cards.
Uncensored: the abliteration lives in the BF16 non-expert tier and survives quantization (measured 0/12 refusals post-build and post-healing).
Capability retained: GSM8K 94.2% (flexible) / 93.9% (strict) after the healing pass.

Research artifact. It inherits the behavior and the license of its upstreams — read Provenance and Intended use below before deploying.

What this is

GLM-5.2 is a Mixture-of-Experts model (78 layers, 256 experts, first_k_dense=3, glm_moe_dsa). This build applies a per-expert mixed-precision scheme so the whole model fits four cards instead of eight, without dropping any experts (no structural pruning):

Tier	Precision	What it covers
Top-damage experts	NVFP4 (4-bit, per-16 scale + global)	the 64 highest-damage experts per MoE layer
Remaining experts	NF3 (3-bit, group-32 e4m3 scale)	the other 192 experts per layer
Attention KV-B / shared	BF16	kept full precision
All non-expert weights	BF16	attention, norms, embeddings, `lm_head` — where the abliteration lives

A bit_map (75 MoE layers × 256 experts → 4,800 NVFP4 / 14,400 NF3) drives the per-expert allocation. This reproduces the hybrid scheme published by madeby561 for the stock GLM-5.2 checkpoint, applied here to the abliterated weights.

Healing pass

After quantization, expert weights are error-corrected with GPTQ-style error feedback (healed against the source BF16, gate/up sharing a Hessian, down-proj reconstructed from the SwiGLU intermediate). 36,591 matrices (~85%; low-traffic experts left naive) were healed, cutting per-expert output error from 14.2% → 1.1%. The healing only touches experts, so the abliteration (non-expert tier) is untouched.

Provenance & credits

This is a derivative work. Full credit to the upstreams:

zai-org/GLM-5.2 — the original model and architecture.
Jiunsong/SuperGLM-5.2-abliterated-NVFP4 — the abliterated (refusal-removed) NVFP4 source weights this build quantizes from.
madeby561 — author of the NVFP4 + NF3 + BF16 hybrid quantization scheme and the vLLM serving image; the NF3/NVFP4 quantizers and bit_map here reproduce that method.

The build reproduces the two quantizers (NF3 and NVFP4) and reuses the upstream hybrid bit_map; the NF3 output format was validated byte-exact against the reference implementation before the full run.

Requirements

4× NVIDIA Blackwell GPUs (SM 120 / RTX PRO 6000-class), ~96 GB each (~384 GB total VRAM). The NVFP4 experts and the B12X MoE / sparse-MLA kernels are Blackwell-native — this build will not run unchanged on Ampere/Hopper.
Docker + the NVIDIA container runtime.
The hybrid-aware vLLM image madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3. Its loader reads the bit_map and reconstructs each precision tier at load — a stock transformers/vllm load will not reconstruct the mixed-precision experts.

Download

HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid \
  --local-dir ./SuperGLM-5.2-abliterated-hybrid

Serving (vLLM, tensor-parallel 4)

Point MODEL_DIR at the downloaded checkpoint. docker-compose.yml:

services:
  glm52:
    image: madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3
    network_mode: host
    ipc: host
    shm_size: 32gb
    init: true
    ulimits:
      memlock: -1
      stack: 67108864
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ${MODEL_DIR:?set MODEL_DIR to the downloaded checkpoint dir}:/model:ro
      - vllm-cache:/cache
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:8000/health"]
      start_period: 600s
      interval: 15s
    environment:
      CUDA_VISIBLE_DEVICES: "0,1,2,3"
      CUDA_DEVICE_ORDER: PCI_BUS_ID
      CUTE_DSL_ARCH: sm_120a            # Blackwell
      PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
      NCCL_IB_DISABLE: "1"
      NCCL_P2P_LEVEL: SYS
      GLOO_SOCKET_IFNAME: lo
      TP_SOCKET_IFNAME: lo
      VLLM_USE_B12X_FP8_GEMM: "1"
      VLLM_USE_B12X_MOE: "1"
      VLLM_USE_B12X_SPARSE_INDEXER: "1"
      VLLM_USE_V2_MODEL_RUNNER: "1"
      B12X_W4A16_TC_DECODE: "1"
      B12X_MOE_FORCE_A16: "1"
      VLLM_DCP_GLOBAL_TOPK: "1"
      VLLM_DCP_SHARD_DRAFT: "1"
      VLLM_ENABLE_PCIE_ALLREDUCE: "1"
      VLLM_PCIE_ALLREDUCE_BACKEND: b12x
      VLLM_PCIE_ONESHOT_MAX_BYTES: "65536"
      B12X_DENSE_SPLITK_TURBO: "1"
      XDG_CACHE_HOME: /cache/jit
      CUDA_CACHE_PATH: /cache/jit
      # hybrid loader
      HYBRID_TIER: both
      HYBRID_KEPT: b12x_nf3
      HYBRID_NF3: b12x_nf3
      HYBRID_B12X_MAX_TOKENS: "2048"
      HYBRID_MXFP8_NATIVE: "1"
    command: >
      vllm serve /model
      --served-model-name GLM-5.2 --host 0.0.0.0 --port 8000
      --trust-remote-code --tensor-parallel-size 4
      --decode-context-parallel-size 4 --dcp-comm-backend ag_rs --dcp-kv-cache-interleave-size 1
      --kv-cache-dtype fp8
      --attention-backend B12X_MLA_SPARSE
      --moe-backend b12x
      --load-format safetensors
      -cc.pass_config.fuse_allreduce_rms=True
      --gpu-memory-utilization 0.968
      --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 2048
      --max-cudagraph-capture-size 64
      --async-scheduling --enable-chunked-prefill --enable-prefix-caching
      --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45
      --default-chat-template-kwargs '{"reasoning_effort":"high"}'
      --hf-overrides '{"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'
      --speculative-config '{"method":"mtp","num_speculative_tokens":5,"moe_backend":"b12x","draft_sample_method":"probabilistic"}'
volumes:
  vllm-cache:

MODEL_DIR=./SuperGLM-5.2-abliterated-hybrid docker compose up -d

First boot takes several minutes (engine init ~1–2 min + JIT compile; the healthcheck has a 600 s grace).

Verify the boot — two checks that actually matter

Sparse-MLA indexing and the KV dtype are load-bearing for long-context correctness. Check the container log:

grep -c "skip sparse MLA indexer" must be 57 (one per sparse layer). A different count means the index_topk_pattern hf-override didn't apply → silent long-context corruption.
grep "fp8_ds_mla KV cache" must be present. Missing it (i.e. BF16 KV) → garbage output on the B12X_MLA_SPARSE backend.

Inference

OpenAI-compatible /v1. GLM pins temperature 1.0 — do not lower it (its default inverts the usual advice):

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"GLM-5.2","messages":[{"role":"user","content":"..."}],"temperature":1.0}'

Served id is GLM-5.2; 262K context; MTP speculative decoding (5 draft tokens), chunked prefill and prefix caching on; tool-calling (glm47 parser) and reasoning (glm45 parser) enabled.

Evaluation

Metric	Result	Notes
GSM8K (flexible)	94.24%	lm-evaluation-harness, served endpoint
GSM8K (strict)	93.86%	"
Refusals	0 / 12	abliteration preserved through quant + healing
Coherence / reasoning	intact	arithmetic, ordering, multi-step held
Throughput	~46–54 tok/s	single-stream, TP4

Intended use & limitations

Research use. This model has had safety refusals removed (abliterated). It will attempt most requests. You are responsible for how you deploy and prompt it; apply your own safety layer for any user-facing use.
Quantization is lossy. NF3 3-bit experts trade some fidelity for the 4-card fit; the healing pass recovers most, not all, of that gap.
License: inherits MIT from GLM-5.2. Attribution to the upstreams above is required.

Format

Standard safetensors (92 shards) + config.json carrying the hybrid metadata (hybrid_bit_map, hybrid_scheme, per-tier layouts) + mxfp8_tier_nokvb.json. Load with the hybrid-aware vLLM image above; a stock transformers load will not reconstruct the mixed-precision tiers.

Troubleshooting

Symptom	Cause / fix
OOM at load	Needs 4×~96 GB. Lower `--gpu-memory-utilization`, or you don't have the VRAM for TP4.
Garbage / repetition at long context	The `index_topk_pattern` override didn't apply — confirm the 57 sparse-indexer skips in the boot log.
Garbage output generally	KV must be `fp8_ds_mla` — confirm that line in the boot log; BF16 KV breaks `B12X_MLA_SPARSE`.
`unknown kernel` / illegal instruction at load	Not on Blackwell (SM 120). The B12X kernels are Blackwell-native; Ampere/Hopper need a different build.
Stock `vllm serve` fails to load the experts	Use the `madeby561/vllm-glm52-nvfp4-nf3-hybrid:v3` image — the hybrid loader is required.

Downloads last month: 60

Safetensors

Model size

347B params

Tensor type

BF16

F8_E4M3

F32

Model tree for kachowtowmater/SuperGLM-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid

Base model

zai-org/GLM-5.2

Quantized

(145)

this model