lukealonso
/

MiniMax-M2.5-NVFP4

8-bit precision

Model card Files Files and versions

lukealonso commited on Feb 17

Commit

4ef3ab7

·

verified ·

1 Parent(s): 0664adf

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -49,7 +49,7 @@ python3 -m sglang.launch_server \
   --tp 2 --ep 2 \
   --mem-fraction-static 0.9 \
   --max-running-requests 16 \
-  --kv-cache-dtype fp8_e4m3 \
   --quantization modelopt_fp4 \
   --attention-backend flashinfer \
   --moe-runner-backend flashinfer_cutlass \

   --tp 2 --ep 2 \
   --mem-fraction-static 0.9 \
   --max-running-requests 16 \
+  --kv-cache-dtype bf16 # or fp8_e4m3 \
   --quantization modelopt_fp4 \
   --attention-backend flashinfer \
   --moe-runner-backend flashinfer_cutlass \