Detailed results
Hardware: 2× RTX PRO 6000 Blackwell Max-Q (sm_120, 96 GB, 300 W cap), PCIe, TP2. A side process held 2.4–2.9 GB per
GPU throughout. Raw outputs are in raw/ and Nsight Compute summaries in ncu/.
Dense MXFP8 backend
Server A/B with CUDA graphs and Engram in pinned RAM, before any MoE kernel changes (raw/ab-marlin.txt,
raw/ab-humming.txt).
| Dense backend | Decode, 2K, 1 stream | Decode, 2K, 4 streams | Prefill, 46K, 1 stream | Decode, 46K, 1 stream | Prefill, 46K, 4 streams | GSM8K-200 |
|---|---|---|---|---|---|---|
| FlashInfer W8A8 (vLLM default) | 4,026 | 68.0 | 97.5% | |||
| Marlin W8A16 | 101.6 | 210.2 | 3,804 | 90.5 | 7,617 | 97.0% |
| Humming 0.1.12 | 87.2 | 227.6 | 3,937 | 87.6 | 7,940 | 97.5% |
Kernel profiles in the server
One 33,863-token real-code prompt plus 64 decode tokens under the torch profiler (bench/profile-run.sh,
raw/prof-*.txt). Times are summed GPU kernel time.
| Prefill, 33.9K tokens | Marlin, stock MoE | + multi-row prefill | + flat scheduler |
|---|---|---|---|
| Total | 9,938 ms | 8,731 ms | 8,719 ms |
| MoE | 5,771 ms | 4,455 ms | 4,470 ms |
| Dense GEMM | 2,002 ms | 2,003 ms | 2,038 ms |
| All-reduce | 749 ms | 857 ms | 783 ms |
| Sparse attention | 555 ms | 555 ms | 568 ms |
| Decode, per step | Multi-row prefill kernel | + flat scheduler |
|---|---|---|
| Total | 28.9 ms | 23.5 ms |
| MoE | 16.1 ms | 8.9 ms |
| Dense GEMM | 8.0 ms | 8.2 ms |
| NCCL all-reduce | 0.6 ms | 1.6 ms |
All-reduce time per call grew with the flat scheduler, most likely one rank waiting for the other. Dense GEMMs now cost as much as the MoE per decode step.
Nsight Compute on the stock kernel
kernels/ncu runs came from ncu --set full on single exl3_moe launches (ncu/*.summary.txt).
Stock exl3_moe |
6 tokens | 4,096 tokens |
|---|---|---|
| DRAM throughput | 18% | 10% |
| Issue slots busy | 51% | 56% |
| Registers per thread | 128 | 128 |
| Achieved occupancy | 33% | 33% |
| Integer ALU pipe | 31% | 34% |
| Tensor pipe | 11% | 13% |
| Top stall, cycles per issued instruction | barrier 1.61 | wait 1.38 |
Barrier stalls were 36–38% of stall samples in both cases. In decode, a serial scan over 384 expert counts was the single hottest spot, about 8% of samples.
Things that did not help
| Change | Effect |
|---|---|
| Shared-memory pipeline depth 3 → 5, 8, 12 stages | At most 4% faster (raw/stage-sweep.txt) |
| Expert group count 4–23 instead of 23 | Between 13% faster and 61% slower depending on batch size (raw/bench-na.txt) |
| 4× multi-row prefill (64 rows per tile decode) | Slower than 2× at every size: register spills at the 128-register limit |
| Wider N tiles on the decode path | Deadlocked in split-K |
Hybrid dense GEMM, controlled A/B
Both runs use the flat scheduler (raw/ab-flat.txt, raw/ab-flathyb.txt).
| Default | DENSE_HYBRID_M=128, 6 GiB KV |
|
|---|---|---|
| Prefill, 46K, 1 / 4 streams | 4,044 / 8,085 | 4,260 / 8,503 |
| Decode, 46K, 1 stream | 116.6 | 114.9 |
| DSpark mean acceptance length | 3.66 | 3.59 |
| GSM8K-200 | 98.5% | 95.5% |