README: pin neutral serving branch serving/arvq-v4-v5 (same commit d7ada6d)
Browse files
README.md
CHANGED
|
@@ -52,11 +52,13 @@ regression; treat general-English quality accordingly.
|
|
| 52 |
## Serving (vLLM)
|
| 53 |
|
| 54 |
Serve with the SM120 fork `github.com/jarrelscy/vllm-glm52-sm120`, branch
|
| 55 |
-
`
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
|
|
|
|
|
|
| 60 |
format version 4 to match. Earlier branches cannot load this repo:
|
| 61 |
`experiment/arvq-fp16-block-scales` (00f57ca51) registers uint8 E4M3 scales
|
| 62 |
and predates the v4 format string; the `glm52-sm120` main branch and the v3
|
|
|
|
| 52 |
## Serving (vLLM)
|
| 53 |
|
| 54 |
Serve with the SM120 fork `github.com/jarrelscy/vllm-glm52-sm120`, branch
|
| 55 |
+
`serving/arvq-v4-v5`, commit `d7ada6d` or later (the same commit is the tip
|
| 56 |
+
of `experiment/arvq-mcbook16`; one serving line loads v3, v4 and v5
|
| 57 |
+
checkpoints). That build loads the v4 cold-expert format
|
| 58 |
+
`rvq256_256x8_expert_fp16block` used by all PV-replaced layers here (fitted
|
| 59 |
+
FP16 block scales serialized directly, applied in FP32 after the FP4 MMA;
|
| 60 |
+
experimental ABI — rebuild the ARVQ libraries together with the loader via
|
| 61 |
+
arvq/build.sh). The config.json `arvq` marker declares
|
| 62 |
format version 4 to match. Earlier branches cannot load this repo:
|
| 63 |
`experiment/arvq-fp16-block-scales` (00f57ca51) registers uint8 E4M3 scales
|
| 64 |
and predates the v4 format string; the `glm52-sm120` main branch and the v3
|