pplx-decider-v1.1-27b-bnb-nf4

Derivative of perplexity-ai/pplx-decider-v1.1-27b, quantized to 4-bit NF4 with bitsandbytes. 15.4 GB.

Base Model

This is a derivative of pplx-decider-v1.1-27b (a decision model built on Qwen/Qwen3.8-27B), quantized from revision 3b45dead91dfa6d95aad6b95764a606fab2bf7a6. All credit for the model, its training and its Decision Index results goes to the original authors. See their card for what the model does and how it scores. The base model's license (Apache 2.0) applies to this derivative. LICENSE and NOTICE are carried over, with an addendum at the end of NOTICE listing what changed.

Quantization Method

• bitsandbytes NF4 with double quantization and bf16 compute (quantization_config in config.json)
• Of the backbone's 1,183 tensors, 606 (the linear layers) are 4-bit; the other 577 (embeddings, norms and the like) stay bf16
• The decision head (readout.safetensors, a bf16 [255, 5120] linear layer), decision_config.json (including the calibration temperature), the tokenizer, the processor config and source/ are unchanged copies of the base revision
• The saved tensors are byte-identical to what bitsandbytes produces when it quantizes the bf16 checkpoint in memory (4,213 tensors, 0 differences)

Files

File	Size	Quant	Note
model-00001-of-00004.safetensors to model-00004-of-00004.safetensors	15.4 GB	NF4 + bf16	backbone, 0.29x the 52.2 GB bf16 original
readout.safetensors	2.6 MB	bf16 (unquantized)	255-option decision head
nf4_loader.py			required loader, see Usage
requirements-nf4.txt			bitsandbytes pin, on top of upstream's requirements.txt
SHA256SUMS			checksums for every file

Measurements

	AMD Strix Halo (Ryzen AI MAX+ 395, gfx1151)	NVIDIA RTX 3090 (24 GB)
Load time	7.4 s	25 s
GPU memory	14.6 GiB (PyTorch peak allocated, one request)	16.2 to 16.5 GiB total in use (nvidia-smi), about 14.5 to 15.0 GiB without other processes on the card
First request (Triton kernels compiling)	53.5 s	33.7 s
Later requests	1.7 s per decision	0.3 s per decision

Both runs gave identical probabilities on the same request.

• Strix Halo software: PyTorch 2.14.0 (ROCm 7.2), transformers 5.17.0, bitsandbytes 0.50.2, flash-linear-attention 0.5.2, causal-conv1d 1.7.0
• RTX 3090 software: PyTorch 2.14.0 (CUDA 13.0), transformers 5.17.0, bitsandbytes 0.50.2, under WSL2 Ubuntu 22.04

Usage

pip install -r requirements.txt -r requirements-nf4.txt

import os
import sys
from pathlib import Path
from huggingface_hub import snapshot_download

checkpoint = Path(snapshot_download("lmcoleman/pplx-decider-v1.1-27b-bnb-nf4"))
os.environ["AUTOJEV_QUANTIZE"] = "nf4"  # turns on the loader
sys.path[:0] = [str(checkpoint), str(checkpoint / "source" / "src")]
import nf4_loader  # noqa: F401  (must be imported before DecisionModel is built)
from autojev.model import DecisionModel, answer

model = DecisionModel(checkpoint, device="cuda")
question = {
    "type": "choice",
    "instructions": "Which team should handle this request?",
    "criteria": {
        "billing": "Charges and refunds",
        "support": "Technical integration errors",
        "sales": "Questions about buying a product",
    },
}
row = {"state": "My integration keeps failing. Please help.", "question": question}
probabilities = model.predict([row])[0]
print(answer(question, probabilities))

On ROCm, device="cuda" is correct; PyTorch's ROCm build uses the cuda device name.

Caveats

• The loader is required. Upstream's DecisionModel calls .to(device) after loading, which bitsandbytes 4-bit models don't support. nf4_loader.py swaps in a loader that uses device_map="auto", only when AUTOJEV_QUANTIZE=nf4 is set.
• 512x512 image cap. The loader matches revision 3b45dead. Upstream later raised the image cap in model.py and processor_config.json; that change isn't in this repo.

Limitations

• The Decision Index hasn't been run on the NF4 weights
• This model inherits any limitations and biases present in the base model
• Not affiliated with or endorsed by Perplexity or Qwen