Instructions to use darrellbest/Qwen3.8-27B-Heretic-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use darrellbest/Qwen3.8-27B-Heretic-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="darrellbest/Qwen3.8-27B-Heretic-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("darrellbest/Qwen3.8-27B-Heretic-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("darrellbest/Qwen3.8-27B-Heretic-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use darrellbest/Qwen3.8-27B-Heretic-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "darrellbest/Qwen3.8-27B-Heretic-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "darrellbest/Qwen3.8-27B-Heretic-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/darrellbest/Qwen3.8-27B-Heretic-NVFP4

SGLang

How to use darrellbest/Qwen3.8-27B-Heretic-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "darrellbest/Qwen3.8-27B-Heretic-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "darrellbest/Qwen3.8-27B-Heretic-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "darrellbest/Qwen3.8-27B-Heretic-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "darrellbest/Qwen3.8-27B-Heretic-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use darrellbest/Qwen3.8-27B-Heretic-NVFP4 with Docker Model Runner:
```
docker model run hf.co/darrellbest/Qwen3.8-27B-Heretic-NVFP4
```

Qwen3.8-27B Heretic NVFP4

NVFP4 (4-bit) build of darrellbest/Qwen3.8-27B-Heretic, for vLLM on NVIDIA Blackwell, which runs NVFP4 natively. 27 GB instead of 51 GB.

What is quantized

Part	Precision
Linear layers of the MLPs and the 16 full-attention layers	NVFP4, 16-value groups, FP8 scales
Gated DeltaNet (`linear_attn`) layers, vision tower, embeddings, `lm_head`, MTP	bf16 (unchanged)

This model is 64 layers of which 48 are Gated DeltaNet, and those keep a recurrent state that low precision damages, so they stay in bf16 along with the vision tower. That is why the file is 27 GB rather than roughly 15 GB. Multi-token-prediction weights are preserved in model-auxiliary.safetensors, which save_pretrained drops for this architecture.

Made with llm-compressor 0.13.0 (scheme="NVFP4"), calibrated on 64 harmless chat prompts. Format: compressed-tensors, nvfp4-pack-quantized.

Use

vllm serve darrellbest/Qwen3.8-27B-Heretic-NVFP4

Ablation

0/100 refusals against 98/100 for the original, KL divergence 0.0465. Method and parameters are on the parent card. The 4-bit weights were not re-measured for refusals, and the 4-bit activation path is vLLM's, which was not exercised here.

Reduced safety guardrails by design. You are responsible for what you do with it.

The family

Repository	Format	Size	Use it with
Qwen3.8-27B-Heretic	bf16 safetensors	51 GB	transformers, vLLM, SGLang
Qwen3.8-27B-Heretic-GGUF	GGUF BF16 / Q8_0 / Q4_K_M (+ vision)	51 / 27 / 16 GB	llama.cpp, Ollama
Qwen3.8-27B-Heretic-FP8	FP8 W8A8, compressed-tensors	35 GB	vLLM
Qwen3.8-27B-Heretic-NVFP4	NVFP4, compressed-tensors	27 GB	vLLM on Blackwell

Downloads last month: 35

Safetensors

Model size

19B params

Tensor type

BF16

Model tree for darrellbest/Qwen3.8-27B-Heretic-NVFP4

Base model

Qwen/Qwen3.8-27B

Finetuned

darrellbest/Qwen3.8-27B-Heretic

Quantized

(3)

this model