README.md

---
license: openmdw-1.1
language:
- en
pipeline_tag: text-generation
library_name: vllm
tags:
- moe
- marin
- grug
- base-model
---
# Snowball 67B-A2B, 262K context, QK 1.75, skew 4×

BF16 export of step 157,000 from the [Snowball long-context comparison](https://github.com/marin-community/marin/issues/8977).
The run extended context to 262,144 tokens from step 156,000 through step 157,000 with `qk_mult=1.75`.
Long-context documents were upsampled 4× during extension.

The model has 67B total parameters, approximately 2B active parameters per token, 26 layers, 256 experts with four selected per token, and a 128,256-token vocabulary.
This is a base model. The tokenizer's bundled chat template does not indicate instruction tuning; use text completions.

Serving requires the [Marin vLLM fork](https://github.com/marin-community/vllm), which registers `GrugMoeForCausalLM` / `grug_moe`.
The existing serving configuration uses eight H100 GPUs, tensor parallelism 1, data parallelism 8, and expert parallelism because the model has five KV heads.
See the [serving and evaluation record](https://github.com/marin-community/marin/issues/8702) for the QK 1.57 and 1.75 baseline exports.
This upload validates tensor names, BF16 dtype, config, sizes, and SHA-256 checksums. No new generation or numerical parity test has been run for this upload.

The [export manifest](export-manifest.json) records each file’s size and SHA-256 checksum, the exporter revision, and the verified Hub weight revision.
The exporter applies pending QB router biases before conversion. Optimizer state is excluded.

Model materials are licensed under [OpenMDW 1.1](LICENSE).