RUNTIME_ADAPTER.md

LingBot SDNQ runtime adapter

lingbot_sdnq_runtime is part of this model repository. It loads ordinary SDNQ-packed Linear weights and also reconstructs LingBot's 3-D MoE w1, w2, and w3 tensors from packed UINT4 storage.

On the official RunPod Torch 2.8 / CUDA 12.8 image, torch._grouped_mm only accepts compute capability 9.0 and the upstream sglang-kernel wheel targets a newer Torch ABI. The adapter therefore supplies Torch implementations of only SGLang's token-alignment and top-k sum-reduction helpers. Expert GEMMs and activation stay on the pinned SGLang Triton fused-MoE implementation.

For MoE inference install the upstream-pinned runtime without dependency resolution so it does not replace the tested Torch build:

pip install --no-deps sglang==0.5.13.post1 apache-tvm-ffi==0.1.9 tilelang==0.1.8 triton==3.6.0

The adapter intentionally does not cache dequantized bf16 expert weights; reported residency therefore remains the SDNQ packed representation.