darrellbest/Qwen3.5-4B-Heretic-NVFP4

Qwen3.5-4B-Heretic NVFP4

NVFP4 build of darrellbest/Qwen3.5-4B-Heretic for vLLM on NVIDIA Blackwell, which runs NVFP4 natively. 5.67 GB instead of 9.35 GB. Qwen/Qwen3.5-4B with its refusal behaviour removed by Heretic using full-weight Arbitrary-Rank Ablation: 6/100 refusals (original: 99/100) at KL divergence 0.0220. See the main repository for how it was made and measured.

What is quantized

Part	Precision
MLP linear layers, and the attention projections of the full-attention layers	NVFP4, 16-value groups, FP8 scales
Everything else	unchanged

Unchanged: the vision encoder, the Gated DeltaNet (linear_attn) layers, the multi-token-prediction block, the embeddings (tied to lm_head) and norms stay bf16 (the DeltaNet A_log/norm parameters float32, as in the original); the recurrent DeltaNet state is sensitive to low precision, and the 248k-token embedding table is a large share of a model this size. Made with llm-compressor 0.13.0 (scheme="NVFP4", calibrated on 64 harmless chat prompts). The multi-token-prediction weights, which the quantized save drops, were copied back unchanged into model-auxiliary.safetensors.

Checked

Loaded in vLLM 0.30.0 on an RTX PRO 6000 Blackwell: ordinary prompts answered correctly and the test image (a red circle and a blue square) described correctly. Thinking-mode reasoning, 4 arithmetic and word problems x 10 seeds with Qwen's recommended sampling: 39/40 finished and correct (bf16 Heretic 40/40, original 40/40). Throughput: ~140 tok/s single-stream, ~3,100 tok/s aggregate at batch 32 (512-token generations; bf16: ~100 / ~2,500).

Use

vllm serve darrellbest/Qwen3.5-4B-Heretic-NVFP4

The NVFP4 weights were not re-measured for refusals.

Reduced safety guardrails by design. You are responsible for what you do with it.

The family

Repository	Format	Size	Use it with
Qwen3.5-4B-Heretic	bf16 safetensors	9.35 GB	transformers, vLLM, SGLang
Qwen3.5-4B-Heretic-GGUF	GGUF BF16 / Q8_0 / Q4_K_M + vision mmproj	8.67 / 4.61 / 2.78 GB + 0.67 GB	llama.cpp, Ollama
Qwen3.5-4B-Heretic-FP8	FP8 W8A8, compressed-tensors	6.79 GB	vLLM
Qwen3.5-4B-Heretic-NVFP4	NVFP4, compressed-tensors	5.67 GB	vLLM on Blackwell