Muse-Glimmer-30B-W8A8-Int8

An inference-oriented W8A8 quantization of Muse-Glimmer-30B, using 8-bit weights and dynamic 8-bit activations. This release is designed to retain the behavior of the original BF16 model while substantially reducing the memory bandwidth, storage costs of the language model and achieve the best speeds possible on dual RTX 3090s. Weight-only comparisons are already very close to the BF16 reference. A broader activation-side validation campaign is still in progress; see Validation status for the exact status of the current claims.

This is an early validation release. The weight quantization itself closely matches the BF16 model, but the full effect of INT8 activations is still being characterized across layers, sequence lengths, workloads, and generation regimes. Results will be added to this model card as they become available.

Model summary

Property	Value
Base model	Muse-Glimmer-30B
Quantization	W8A8
Weight format	INT8, symmetric, per-channel
Activation format	INT8, symmetric, dynamic per-token
Weight reconstruction	LogEq + round-to-nearest (RTN)
Compute precision	FP16 or BF16, depending on the inference backend
Primary use	Efficient local and server-side inference
Recommended backend	vLLM with compressed-tensors support

The language-model linear layers are quantized. Components that are especially sensitive or not part of the main W8A8 execution path—such as embeddings, the language-model head, and non-language components—remain outside the INT8 quantization scope.

Why this quantization exists

The objective of this release is not merely to reduce the checkpoint size. It is to provide a practical W8A8 version of Muse-Glimmer-30B that can benefit from optimized INT8 kernels while preserving the model's BF16 behavior as closely as possible.

The quantization process therefore prioritizes:

• low divergence from the BF16 reference
• stable behavior across individual layers, layer groups, and the complete model
• preservation of instruction following, coding, tool use, and long-context behavior
• high-throughput inference on hardware with efficient INT8 tensor operations
• reproducible, workload-aware validation rather than relying on a single aggregate similarity metric

Validation status

Validation is deliberately split into two separate questions:

1. Weight quantization fidelity: how closely the dequantized INT8 weights reproduce the BF16 weights and their weight-only outputs.
2. End-to-end W8A8 fidelity: how dynamic INT8 activations affect hidden states, logits, token decisions, generation trajectories, and downstream capabilities.

The first is already very close to the BF16 reference. The second is still being evaluated comprehensively and should not be inferred from weight-only measurements.

Weight quantization

BF16 vs W8A8 validation results

Metric	Result
Samples	128
Evaluated tokens	152,347
BF16 NLL	1.40303408
BF16 perplexity	4.06752244
W8A8 NLL delta	+0.00230600
Logit cosine similarity	0.99810028
Mean KL divergence	0.01006362
Top-1 token agreement	90.5512%
Top-k token overlap	87.4016%
Hidden-state cosine — state 21	0.99638128
Hidden-state cosine — state 52	0.96988422

Static audits of the INT8 weights, quantization scales, saturation behavior, and corrected LogEq effective weights were all successful. They indicate that the checkpoint itself is healthy and remains very close to the BF16 reference.

Dynamic INT8 activation validation

The following section is intentionally reserved for the full activation evaluations. The final report will cover individual layers, architectural groups, and complete end-to-end behavior.

Evaluation axis	Metric or protocol	Result
Activation distributions	Range, mean, variance, RMS, percentiles, skewness, kurtosis, clipping and saturation rates	TBD
Per-layer fidelity	Cosine similarity, normalized RMSE, relative L1/L2 error, SQNR	TBD
Layer-group fidelity	Attention, MLP, normalization, residual stream, local/global layer groups	TBD
Error propagation	Divergence growth by depth and residual accumulation	TBD
Logit fidelity	Cosine similarity, RMSE, KL divergence, Jensen-Shannon divergence	TBD
Token decisions	Top-1 agreement, top-k overlap, rank correlation, margin changes	TBD
Distribution tails	Rare-token and low-probability logit behavior	TBD
Sequence-length stability	Short, medium, long, and maximum-context evaluations	TBD
Generation stability	Greedy and stochastic decoding across multiple seeds	TBD
Capability retention	Code, reasoning, instruction following, tool use, and agentic tasks	TBD
Robustness	Outliers, adversarial calibration cases, repeated tokens, and unusual formatting	TBD
Throughput and memory	Prefill, decode, peak VRAM, concurrency, and context capacity	TBD

Usage

vLLM server

docker run --rm \
  --name muse-glimmer-w8a8 \
  --gpus all \
  --ipc=host \
  -p 8080:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  AlexanderKyng/Muse-Glimmer-30B-W8A8-Int8 \
  --generation-config vllm \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":64}' \
  --tensor-parallel-size 2 \
  --max-num-seqs 5 \
  --language-model-only \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Docker might not be strictly necessary but as of today, I only tried using this command, not vLLM directly.

OpenAI-compatible client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local",
)

response = client.chat.completions.create(
    model="AlexanderKyng/Muse-Glimmer-30B-W8A8-Int8",
    messages=[
        {
            "role": "user",
            "content": "Write a Python function that merges overlapping intervals.",
        }
    ],
    temperature=1.0,
)

print(response.choices[0].message.content)

Hardware notes

W8A8 speedups depend on the inference backend, GPU architecture, available INT8 kernels, batch size, sequence length, and tensor-parallel configuration. It was 100% developed with the Ampere GPUs (and espacially the RTX 3090s) and RTX 4090s in mind.

Intended use

This model is intended for users who want to evaluate or deploy Muse-Glimmer-30B with a lower-bandwidth W8A8 execution path, especially for:

• local inference
• coding and agentic workloads
• long-context experiments
• throughput, memory, and quantization research

It is also suitable as a reproducible target for studying activation quantization and layerwise divergence in modern large language models.

Limitations

• Full activation-side equivalence with BF16 is not yet proven.
• Small numerical errors can accumulate across layers and alter generation trajectories, even when aggregate similarity metrics appear excellent.
• Stochastic generations are not expected to match BF16 token-for-token; distributional comparisons require repeated sampling and controlled seeds.
• Backend support and realized performance vary by hardware and software version.
• This quantization does not change the underlying model's knowledge, safety characteristics, biases, or license obligations.

Reproducibility

The final quantization and validation release will document:

• the exact quantization configuration
• calibration data composition and preprocessing
• software and kernel versions
• deterministic weight-level checks
• layerwise activation captures
• end-to-end BF16 versus W8A8 comparisons
• throughput and memory measurements

Acknowledgements

Thanks to the Meta, the creators of Muse-Glimmer-30B and to the open-source communities behind vLLM, compressed-tensors, Transformers, and the wider quantization ecosystem.

Citation

If this quantized model is useful in your work, please cite the original Muse-Glimmer-30B model and link to this repository.

@misc{muse_glimmer_30b_w8a8_logeq_rtn_v1,
  title        = {Muse-Glimmer-30B-W8A8-Int8},
  author       = {AlexanderKyng},
  year         = {2026},
  howpublished = {Hugging Face model repository}
}