how to serving sglang?
I only using sglang, This model serving for slgang?
My GPU pro 6000 workstatoin x2
It's not a standard quantization format — it's a custom mixed-precision vector quantization (AQLM-style, 1/2/3-bit per expert; quant_method: auto-round) with a bespoke grouped-MoE Triton kernel. Inference engines can only serve quant formats they have a kernel/loader for, and both vLLM and SGLang natively support only the standard ones (uniform GPTQ, AWQ, compressed-tensors, FP8). For this model the kernel exists only as a vLLM plugin:
-> https://github.com/mmzz164/OneCompression
check out tag glm-serving-v1, then see example/glm-5.2/ for the full serving recipe. The plugin lives in vllm_plugins/gptq (MIT). vLLM itself is unpatched — the plugin just adds the mixed-VQ MoE kernel + loader.