Pin OneComp serving plugin to immutable tag nex-serving-v1 (was mutable main); content identical, repro-stable
Browse files
README.md
CHANGED
|
@@ -31,7 +31,7 @@ serves on a single workstation with **2×96 GB GPUs**.
|
|
| 31 |
|
| 32 |
> ⚠️ **This checkpoint uses a custom mixed-bit MoE format and does NOT load with
|
| 33 |
> stock vLLM / Transformers / llama.cpp.** Serving requires the mixed-bit MoE plugin
|
| 34 |
-
> ([**mmzz164/OneCompression**](https://github.com/mmzz164/OneCompression/tree/
|
| 35 |
> MIT) — see *How to serve*. If you want a plug-and-play quant, use one of the standard
|
| 36 |
> GGUF/AWQ builds instead.
|
| 37 |
|
|
@@ -110,10 +110,10 @@ This is a **custom mixed-bit MoE checkpoint**. Stock loaders cannot read it. To
|
|
| 110 |
quantization/serving base.
|
| 111 |
2. Add the **mixed-bit grouped fused-MoE serving plugin** (the part that actually loads
|
| 112 |
and runs per-expert 2/3/4/8-bit MoE — not in upstream) from
|
| 113 |
-
**[mmzz164/OneCompression](https://github.com/mmzz164/OneCompression/tree/
|
| 114 |
`mixed_moe.py` (the `mixed_gptq` `FusedMoEMethodBase`), plus `grouped_moe.py` and
|
| 115 |
`fused_dq_gemm.py` (the Triton kernels). See
|
| 116 |
-
[`MIXED_BIT_MOE_SERVING.md`](https://github.com/mmzz164/OneCompression/blob/
|
| 117 |
3. Serve with vLLM using pipeline-parallel across your two GPUs and the `mixed_gptq`
|
| 118 |
quantization, e.g. (illustrative):
|
| 119 |
```bash
|
|
@@ -160,7 +160,7 @@ model nex-agi/Nex-N2-Pro; only the weight precision was changed.
|
|
| 160 |
- **Fujitsu Research** for [OneComp / OneCompression](https://github.com/FujitsuResearch/OneCompression)
|
| 161 |
(MIT), including the AutoBit allocator and QEP.
|
| 162 |
- Mixed-bit MoE serving kernels:
|
| 163 |
-
[mmzz164/OneCompression](https://github.com/mmzz164/OneCompression/tree/
|
| 164 |
|
| 165 |
## Citation
|
| 166 |
|
|
|
|
| 31 |
|
| 32 |
> ⚠️ **This checkpoint uses a custom mixed-bit MoE format and does NOT load with
|
| 33 |
> stock vLLM / Transformers / llama.cpp.** Serving requires the mixed-bit MoE plugin
|
| 34 |
+
> ([**mmzz164/OneCompression**](https://github.com/mmzz164/OneCompression/tree/nex-serving-v1/vllm_plugins/gptq),
|
| 35 |
> MIT) — see *How to serve*. If you want a plug-and-play quant, use one of the standard
|
| 36 |
> GGUF/AWQ builds instead.
|
| 37 |
|
|
|
|
| 110 |
quantization/serving base.
|
| 111 |
2. Add the **mixed-bit grouped fused-MoE serving plugin** (the part that actually loads
|
| 112 |
and runs per-expert 2/3/4/8-bit MoE — not in upstream) from
|
| 113 |
+
**[mmzz164/OneCompression](https://github.com/mmzz164/OneCompression/tree/nex-serving-v1/vllm_plugins/gptq)**:
|
| 114 |
`mixed_moe.py` (the `mixed_gptq` `FusedMoEMethodBase`), plus `grouped_moe.py` and
|
| 115 |
`fused_dq_gemm.py` (the Triton kernels). See
|
| 116 |
+
[`MIXED_BIT_MOE_SERVING.md`](https://github.com/mmzz164/OneCompression/blob/nex-serving-v1/vllm_plugins/gptq/MIXED_BIT_MOE_SERVING.md).
|
| 117 |
3. Serve with vLLM using pipeline-parallel across your two GPUs and the `mixed_gptq`
|
| 118 |
quantization, e.g. (illustrative):
|
| 119 |
```bash
|
|
|
|
| 160 |
- **Fujitsu Research** for [OneComp / OneCompression](https://github.com/FujitsuResearch/OneCompression)
|
| 161 |
(MIT), including the AutoBit allocator and QEP.
|
| 162 |
- Mixed-bit MoE serving kernels:
|
| 163 |
+
[mmzz164/OneCompression](https://github.com/mmzz164/OneCompression/tree/nex-serving-v1/vllm_plugins/gptq) (MIT).
|
| 164 |
|
| 165 |
## Citation
|
| 166 |
|