XiaomiMiMo

/

MiMo-V2.6-Flash-RL

Like 468

Follow

Xiaomi MiMo 4.54k

Yes, it runs well on 2× RTX PRO 6000 Blackwell (TP2, PCIe). We've published a full recipe for exactly that setup, using the official weights unchanged:

https://huggingface.co/diffbot/MiMo-V2.6-Flash-RL-FP8KV-W4A8-2x-RTX-PRO-6000

What it does on top of the stock vllm/vllm-openai:mimo-v26 image: FP8 KV cache for the DiffKV attention layers (stock vLLM only allows bf16 there), Marlin MoE with FP8 activations, three fixes to vLLM's Triton DiffKV kernel (the big one: split-KV was disabled for the speculative-decode verify step, which made long-context decode ~2× slower than it should be), a custom sm_120 prefill attention kernel, DFlash with 3 draft tokens, and vLLM's CPU KV tier. All four input modes (text/image/video/audio) are on, 256K context, KV pool ≈479K tokens.

Measured on our pair (Max-Q, 300 W): at 46K-token prompts, 10.2K tok/s prefill, 4.5 s TTFT, ~190 tok/s single-stream decode, 416 tok/s aggregate at 4 streams; the stock image on the same cards does 6.2K prefill / 7.4 s / 87–94 tok/s. GSM8K-200 unchanged at 98%. The card has the full step-by-step table, and recipe/README.md is three commands to run.

Note: vLLM v0.30.0 (released today) does not include the MiMo-V2.6 support PRs — use the mimo-v26* image tags.

what about single 6k pro?

fastest tp2 setup out there right now getting up streamed into LIL VLLM I believe

https://github.com/local-inference-lab/vllm/issues/882

services:
  mm26f:
    image: madeby561/vllm:mimo-v26-flash-b12x-20260923-rc4
    container_name: mm26f
    network_mode: "host"
    gpus: all
    shm_size: "32g"
    ipc: "host"
    init: true
    security_opt:
      - seccomp=unconfined
    ulimits:
      memlock: -1
      stack: 67108864
    environment:
      - HF_HUB_OFFLINE=1
      - TRANSFORMERS_OFFLINE=1
      - CUDA_VISIBLE_DEVICES=0,1
      - VLLM_CACHE_ROOT=/cache
      - VLLM_LOGGING_LEVEL=INFO
      - VLLM_KV_CACHE_LAYOUT=BLHNC
      - VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=32768
      - VLLM_ENABLE_PCIE_ALLREDUCE=1
      - VLLM_PCIE_ALLREDUCE_BACKEND=b12x
      - VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE=64KB
      - VLLM_PCIE_TWOSHOT_ALLREDUCE_MAX_SIZE=2MB
      - VLLM_ADAPTIVE_VERIFICATION_COST_SCALE=1.0
      - VLLM_PCIE_DMA_MIN_BYTES=off
      - NCCL_IB_DISABLE=1
      - NCCL_P2P_LEVEL=SYS
      - NCCL_PROTO=LL,LL128,Simple
      - CUTE_DSL_ARCH=sm_120a
      - VLLM_B12X_MOE_FP4_FORCE_A16=1
      - OPUS55_ATTN=1
      - OPUS55_L2PF=1
      - TRITON_CACHE_DIR=/cache/triton
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0
    volumes:
      - /data1/MiMo-V2.6-Flash-RL:/model:ro
      - /data1/MiMo-V2.6-Flash-RL-tp2.cache:/root/.cache:rw
      - /data1/MiMo-V2.6-Flash-RL-tp2.cache:/cache:rw
    entrypoint:
      - /bin/sh
      - -c
      - "unset NCCL_GRAPH_FILE NCCL_GRAPH_DUMP_FILE VLLM_B12X_MLA_EXTEND_MAX_CHUNKS && exec /opt/venv/bin/vllm serve \"$@\""
      - --
    command:
      - /model
      - --served-model-name=mm26f
      - --trust-remote-code
      - '--hf-overrides={"architectures":["MiMoV2OmniForCausalLM"]}'
      - '--compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"],"compile_sizes":[1024]}'
      - --kv-cache-dtype=fp8
      - --block-size=64
      - --tensor-parallel-size=2
#      - --gpu-memory-utilization=0.985
      - --kv-cache-memory=8042543104
      - --cpu-offload-gb=1.5  # karmic-kraken KV for each image 3sec load
      - --cpu-offload-params=visual  # karmic-kraken
      - --attention-backend=b12x
      - --moe-backend=b12x
      - --linear-backend=b12x
      - --max-num-seqs=10
      - --max-cudagraph-capture-size=80
      - --max-num-batched-tokens=2048
      - --max-num-scheduled-tokens=2048
      - --long-prefill-token-threshold=1024
      - --prefill-compute-share=0.8
      - --async-scheduling
      - --no-scheduler-reserve-full-isl
      - --enable-chunked-prefill
      - --enable-prefix-caching
      - --generation-config=vllm  # checkpoint generation_config.json defaults max_tokens to 2048
      - --reasoning-parser=mimo
      - --tool-call-parser=mimo
      - --enable-auto-tool-choice
      - '--override-generation-config={"temperature":1,"top_p":0.95}' #   ,"repetition_penalty":1.18}'
      - '--speculative-config={"method":"dflash","model":"/opt/mimo-v26/dflash-fixed","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"enable_adaptive_verification":true,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"B12X","kv_cache_dtype":"fp8"}' 
      - --host=0.0.0.0
      - --port=8001```