Document Snowball long-context checkpoint

README.md

@@ -0,0 +1,30 @@
+---
+language:
+- en
+pipeline_tag: text-generation
+library_name: vllm
+tags:
+- moe
+- marin
+- grug
+- base-model
+---
+# Snowball 67B-A2B, 262K context, QK 1.57, skew 1×
+
+BF16 export of step 157,000 from the [Snowball long-context comparison](https://github.com/marin-community/marin/issues/8977).
+The run extended context to 262,144 tokens from step 156,000 through step 157,000 with `qk_mult=1.57`. Unchanged long-context document sampling.
+
+The model has 67B total parameters, approximately 2B active parameters per token, 26 layers, 256 experts with four selected per token, and a 128,256-token vocabulary.
+This is a base model. The tokenizer's bundled chat template does not indicate instruction tuning; use text completions.
+
+Serving requires the [Marin vLLM fork](https://github.com/marin-community/vllm), which registers `GrugMoeForCausalLM` / `grug_moe`.
+The existing serving configuration uses eight H100 GPUs, tensor parallelism 1, data parallelism 8, and expert parallelism because the model has five KV heads.
+See the [serving and evaluation record](https://github.com/marin-community/marin/issues/8702) for the QK 1.57 and 1.75 baseline exports.
+These uploads validate tensor names, BF16 dtype, config, sizes, and SHA-256 checksums. No new generation or numerical parity test has been run for this upload.
+
+Source checkpoint: `gs://marin-us-central2/grug/moe_67b_a2b_d2560_ep1_rep1_ctx4_bs256_seq262144_ctxext_step156k_qk157-0997fb/checkpoints/step-157000/`
+
+Export source: `gs://marin-us-central2/marin/exports/grug/mrcr-8701/step-157000-qk157/hf-bf16-vllm`
+
+The [export manifest](export-manifest.json) records each source object's generation and SHA-256 checksum, the exporter revision, and the verified Hub weight revision.
+The exporter applies pending QB router biases before conversion. Optimizer state is excluded.