Update DFlash recommendation
Browse files
README.md
CHANGED
|
@@ -111,7 +111,7 @@ The full vLLM recipe is on the main [Laguna S 2.1 model card](https://huggingfac
|
|
| 111 |
> INT4 support landed in vLLM with [vllm-project/vllm#47154](https://github.com/vllm-project/vllm/pull/47154). This checkpoint mixes INT4 and group-quantized INT8 experts, which earlier builds rejected in the Marlin WNA16 MoE path. Use a vLLM build that includes this fix.
|
| 112 |
|
| 113 |
> [!NOTE]
|
| 114 |
-
> **Optional: speculative decoding with DFlash.** Pair with the quantization-matched draft model [poolside/Laguna-S-2.1-DFlash-INT4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4) by adding `--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-INT4","num_speculative_tokens":
|
| 115 |
|
| 116 |
#### SGLang
|
| 117 |
|
|
|
|
| 111 |
> INT4 support landed in vLLM with [vllm-project/vllm#47154](https://github.com/vllm-project/vllm/pull/47154). This checkpoint mixes INT4 and group-quantized INT8 experts, which earlier builds rejected in the Marlin WNA16 MoE path. Use a vLLM build that includes this fix.
|
| 112 |
|
| 113 |
> [!NOTE]
|
| 114 |
+
> **Optional: speculative decoding with DFlash.** Pair with the quantization-matched draft model [poolside/Laguna-S-2.1-DFlash-INT4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4) by adding `--speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash-INT4","num_speculative_tokens":7,"method":"dflash"}'` to the serve command.
|
| 115 |
|
| 116 |
#### SGLang
|
| 117 |
|