tmax-27b-MTP-vision-BF16-8bit

Mixed-precision 8-bit quantization of TMax 27B with vision and MTP restored from the base model — a terminal-agent fine-tune of Qwen3.6-27B, optimized for efficient local inference with minimal quality loss.

Quick Stats

Metric	Value
Base model	allenai/tmax-27b
Architecture	Qwen3.6-27B (64-layer hybrid attention)
Quantization	MLP-only 8-bit, group-size=64
BF16 size	~52 GB
Mixed 8-bit size	~37 GB (29% reduction)
License	Apache 2.0

Quantization Strategy

This model uses surgical mixed-precision quantization: only the SwiGLU MLP projection matrices (gate_proj, up_proj, down_proj) in the language model are quantized to 8-bit. All other components remain in BF16:

• ✅ Quantized (8-bit): LLM MLP layers (192 tensors, ~66% of model mass)
• ✅ Protected (BF16): Linear attention, full attention, vision tower, MTP head, embeddings, LM head, all normalization layers

Rationale: MLP layers are "knowledge lookup" / feature transformation layers that are extremely quantization-tolerant. Attention mechanisms carry reasoning and context-window recall critical for agentic work and coding.

Benchmark Results

Code generation benchmarks (fixed seed sampling, no chain-of-thought):

Benchmark	Qwen3.6-27B-MTP-Mixed-8bit	tmax-27b-Mixed-8bit	Δ
HumanEval	93.3%	93.3%	=
MBPP	86.7%	93.3%	+6.6%
LiveCodeBench	60.0%	63.3%	+3.3%

Speed improvements (same tasks, lower latency):

Benchmark	Qwen3.6-27B (s)	tmax-27b (s)	Speedup
HumanEval	487.0	395.5	1.23x
MBPP	304.2	274.6	1.11x
LiveCodeBench	2027.9	1683.6	1.20x

All benchmarks: 30 samples, sampled decoding, no thinking mode.

Model Architecture

• 64 dense layers alternating [linear_attention × 3, full_attention × 1] × 16
• Hybrid attention: SSM-style linear attention (3/4 of layers) + full MHA with GQA 6:1 (1/4 of layers)
• Vision tower: 27-block ViT (BF16, restored from base model)
• MTP head: 1-layer multi-token prediction (BF16, restored from base model)
• Hidden size: 5120, Intermediate: 17408, Heads: 24Q / 4KV, Head dim: 256
• Vocab: 248,320, Context: 262,144 tokens

Restoring Vision + MTP from Base Model

The original allenai/tmax-27b checkpoint was distributed as a text-only model with the vision tower (333 tensors) and MTP head (15 tensors) stripped during DPPO training. This quantization restores both components from the base Qwen3.6-27B model.

The tmax-27b checkpoint (851 keys) is a strict subset of the base Qwen3.6-27B checkpoint (1199 keys). No keys were added or renamed during fine-tuning.

Merge process

1. Load both indexes: Parse model.safetensors.index.json from both tmax-27b and Qwen3.6-27B
2. Categorize keys: Identify which 348 keys are missing from tmax-27b (333 vision + 15 MTP)
3. Stream tensors: Load LLM weights from tmax-27b (fine-tuned), vision + MTP from base (untouched)
4. Verify shapes: Confirm all 851 shared keys have matching tensor shapes between models
5. Save merged checkpoint: Write 1199 tensors to new sharded safetensors (~52 GB BF16)

Reproducing the merge

# Conceptual — full script at quantization_scripts/merge_tmax_27b_vision_mtp.py
from safetensors import safe_open

# Keys to source from each model
FROM_TMAX = {k for k in tmax_index if not k.startswith(('model.visual.', 'mtp.'))}  # 851 keys
FROM_BASE = {k for k in base_index if k.startswith(('model.visual.', 'mtp.'))}      # 348 keys

# Merge: fine-tuned LLM + base vision/MTP
merged = {}
for k in FROM_TMAX:
    merged[k] = load_from_tmax(k)      # DPPO-fine-tuned weights
for k in FROM_BASE:
    merged[k] = load_from_base(k)      # Untouched base weights

save_sharded(merged, 'tmax-27b-merged/')  # 1199 tensors, 52 GB

Caveats

• Vision tower is not fine-tuned: The 27-block ViT and merger/projector are from the base Qwen3.6-27B. They may not be fully aligned with the DPPO-adapted LLM for vision-grounded terminal tasks.
• MTP is not fine-tuned: The single MTP prediction layer is from the base model. Multi-token prediction quality reflects the base model, not the tmax fine-tune.
• For text-only inference: If you only need text generation, you can skip the merge and quantize tmax-27b directly (~50 GB → ~36 GB).

Usage

oMLX (Recommended)

Loads with no hacks or workarounds.

MLX

from mlx_vlm.utils import load

model, processor = load("your-username/tmax-27b-Mixed-8bit", lazy=True)
# Model loads in ~37 GB vs ~52 GB for BF16

vLLM

vllm serve your-username/tmax-27b-Mixed-8bit \
  --served-model-name tmax-27b-mixed-8bit \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --quantization bitsandbytes \
  --max-model-len 65536

Hugging Face Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "your-username/tmax-27b-Mixed-8bit",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("your-username/tmax-27b-Mixed-8bit")

Files

File	Description
model-00001.safetensors – model-00008.safetensors	Sharded model weights (5 GB each)
model.safetensors.index.json	Weight index
config.json	Model configuration with quantization metadata
tokenizer.json / tokenizer_config.json	Tokenizer
chat_template.jinja	Chat template
generation_config.json	Generation parameters
preprocessor_config.json	Vision preprocessor config
vocab.json / merges.txt	Tokenizer vocabulary

Training Provenance

This model is a quantization of allenai/tmax-27b, which was trained using DPPO on Qwen3.6-27B for terminal-agent tasks:

• Base model: Qwen3.6-27B
• Fine-tuning: DPPO on TMax-15K
• Training steps: 160 (best checkpoint on Terminal Bench)
• Vision/MTP: Restored from base Qwen3.6-27B (not trained during DPPO)

Limitations

• Mixed quantization may show minor quality degradation on edge cases compared to BF16
• Vision tower and MTP head are from the base model (not fine-tuned), so vision understanding and multi-token prediction may not be optimized for terminal-agent tasks
• The vision tower was stripped during the original DPPO training; these weights are restored from the base model and may not be fully aligned with the fine-tuned LLM

License

Apache 2.0 — same as allenai/tmax-27b

Citation

@misc{ivison2026tmaxsimplerecipeterminal,
      title={Tmax: A simple recipe for terminal agents},
      author={Hamish Ivison and Junjie Oscar Yin and Rulin Shao and Teng Xiao and Nathan Lambert and Hannaneh Hajishirzi},
      year={2026},
      eprint={2606.23321},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.23321},
}