Document Snowball long-context checkpoint

README.md

@@ -0,0 +1,30 @@
+---
+license: openmdw-1.1
+language:
+- en
+pipeline_tag: text-generation
+library_name: vllm
+tags:
+- moe
+- marin
+- grug
+- base-model
+---
+# Snowball 67B-A2B, 262K context, QK 1.75, skew 4×
+
+BF16 export of step 157,000 from the [Snowball long-context comparison](https://github.com/marin-community/marin/issues/8977).
+The run extended context to 262,144 tokens from step 156,000 through step 157,000 with `qk_mult=1.75`.
+Long-context documents were upsampled 4× during extension.
+
+The model has 67B total parameters, approximately 2B active parameters per token, 26 layers, 256 experts with four selected per token, and a 128,256-token vocabulary.
+This is a base model. The tokenizer's bundled chat template does not indicate instruction tuning; use text completions.
+
+Serving requires the [Marin vLLM fork](https://github.com/marin-community/vllm), which registers `GrugMoeForCausalLM` / `grug_moe`.
+The existing serving configuration uses eight H100 GPUs, tensor parallelism 1, data parallelism 8, and expert parallelism because the model has five KV heads.
+See the [serving and evaluation record](https://github.com/marin-community/marin/issues/8702) for the QK 1.57 and 1.75 baseline exports.
+This upload validates tensor names, BF16 dtype, config, sizes, and SHA-256 checksums. No new generation or numerical parity test has been run for this upload.
+
+The [export manifest](export-manifest.json) records each file’s size and SHA-256 checksum, the exporter revision, and the verified Hub weight revision.
+The exporter applies pending QB router biases before conversion. Optimizer state is excluded.
+
+Model materials are licensed under [OpenMDW 1.1](LICENSE).